NUREG/IA-0137

* • International 

Agreement Report 

A Study of 

Control Room Staffing Levels 

for Advanced Reactors 

Prepared by 

B. P. Hallbert1 , A. Sebok, Institutt for Energiteknikk, OECD Halden Reactor Project 

D. Morisseau, U.S. Nuclear Regulatory Commission 2 

Institutt for Energiteknikk 

OECD Halden Reactor Project 

P.O. Box 173 

N-1751 Halden 

NORWAY 

1Now with the Idaho National Engineering & Environmental Laboratory 2

Now with the Institutt for Energiteknikk, OECD Halden Reactor Project 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

November 2000 

Prepared as part of 

The Agreement on Research Participation and Technical Exchange 

under the International Code Application and Maintenance Program (CAMP) 

Published by 

U.S. Nuclear Regulatory Commission
AVAILABILITY OF REFERENCE MATERIALS 

IN NRC PUBLICATIONS

NRC Reference Material 

As of November 1999, you may electronically access NUREG-series publications and other NRC records at 

NRC's Public Electronic Reading Room at www. nrc. gov/N RC/ADAMS/index.html. 

Publicly released records include, to name afew, 

NUREG-series publications; Federal Register notices; 

applicant, licensee, and vendor documents and 

correspondence; NRC correspondence and internal 

memoranda; bulletins and information notices; 

inspection and investigative reports; licensee event 

reports; and Commission papers and their 

attachments. 

NRC publications in the NUREG series, NRC 

regulations, and Title 10, Energy, in the Code of 

Federal Regulations may also be purchased from one 

of these two sources. 1. The Superintendent of Documents 

U.S. Government Printing Office 

P. 0. Box 37082 

Washington, DC 20402-9328 

www.access.gpo.gov/su-docs 

202-512-1800 

2. The National Technical Information Service 

Springfield, VA 22161-0002 

www.ntis.gov 

1-800-533-6847 or, locally, 703-805-6000 

A single copy of each NRC draft report for comment is 

available free, to the extent of supply, upon written 

request as follows: 

Address: Office of the Chief Information Officer, 

Reproduction and Distribution 

Services Section 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

E-mail: DISTRIBUTION@nrc.gov 

Facsimile: 301-415-2289 

Some publications in the NUREG series that are 

posted at NRC's Web site address 

www.nrc.gov/NRC/NUREGS/indexnum.html 

are updated periodically and may differ from the last 

printed version. Although references to material found on a Web site bear the date the material was accessed, the material available on the date cited may 

subsequently be removed from the site.

Non-NRC Reference Material 

Documents available from public and special technical 

libraries include all open literature items, such as 

books, journal articles, and transactions, Federal 

Register notices, Federal and State legislation, and 

congressional reports. Such documents as theses, 

dissertations, foreign reports and translations, and 

non-NRC conference proceedings may be purchased 

from their sponsoring organization. 

Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are 

maintained at

The NRC Technical Library 

Two White Flint North 

11545 Rockville Pike 

Rockville, MD 20852-2738 

These standards are available in the library for 

reference use by the public. Codes and standards are usually copyrighted and may be purchased from the 

originating organization or, if they are American 

National Standards, from

American National Standards Institute 

11 West 4 2 n Street 

New York, NY 10036-8002 

www.ansi.org 

212-642-4900

The NUREG series comprises (1) technical and 

administrative reports and books prepared by the 

staff (NUREG-XXXX) or agency contractors 

(NUREG/CR-XXXX), (2) proceedings of 

conferences (NUREG/CP-XXXX), (3) reports 

resulting from international agreements 

(NUREG/IA-XXXX), (4) brochures 

(NUREG/BR-XXXX), and (5) compilations of legal 

decisions and orders of the Commission and 

Atomic and Safety Licensing Boards and of 

Directors' decisions under Section 2.206 of NRC's 

regulations (NUREG-0750).

DISCLAIMER: This report was prepared under an international cooperative agreement for the exchange of 

technical information. Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the 

results of such use, of any information, apparatus, product or process disclosed in this publication, or represents that its use by such third party would not infringe privately owned rights.

I
NUREG/IA-0137

International 

Agreement Report 

A Study of 

Control Room Staffing Levels 

for Advanced Reactors 

Prepared by 

B. P. Hallbert1

, A. Sebok, Institutt for Energiteknikk, OECD Halden Reactor Project 

D. Morisseau, U.S. Nuclear Regulatory Commission 2 

Institutt for Energiteknikk 

OECD Halden Reactor Project 

P.O. Box 173 

N-1751 Halden 

NORWAY 

'Now with the Idaho National Engineering & Environmental Laboratory 

2Now with the Institutt for Energiteknikk, OECD Halden Reactor Project 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

November 2000 

Prepared as part of 

The Agreement on Research Participation and Technical Exchange 

under the International Code Application and Maintenance Program (CAMP) 

Published by 

U.S. Nuclear Regulatory Commission
ABSTRACT 

This report documents the results of an empirical study of operator and plant performance in simulator 

based settings. The simulator settings were designed to be representative of conventional and advanced 

plants. The advanced plant design employed passive systems. The control room architectures were also 

designed to represent both plant types. Two control room staffing configurations were employed in each 

plant setting: a staffing configuration reflecting the requirements of 10 CFR 50.54 (m); and a staffing 

configuration that involved a reduced number of control room operators. A series of five design basis 

scenarios, relevant for both conventional and advanced passive plants were chosen to evaluate the effects of 

plant type and crew size on operator performance. The scenarios were: 1) steam generator tube rupture 

with a stuck open steam generator safety relief valve in the affected steam generator, preceded by a fire in 

the turbine hall; 2) interfacing systems loss of coolant accident with compounded instrument failures due to 

the incident; 3) sustained total loss of feedwater; 4) loss of off-site power with a single steam generator 

safety relief valve stuck open, and; 5) steam generator overfill. Eight crews of operators from the Loviisa 

nuclear power station in Loviisa, Finland participated in the study: four crews in the conventional plant 

setting; and four crews in the advanced plant setting. Measures of objective performance were obtained, 

including ratings of crew performance and transient management. Measures of operator workload, situation 

awareness, and team interaction were obtained repeatedly during each scenario.. The findings of the study 

revealed a number of effects of crew size and plant type, and their combination on operator performance. 

This report documents the study and discusses the implications and issues raised by this performance-based 

evaluation of control room staffing requirements for advanced passive plants. 

iii 

NUREG/IA-0137
CONTENTS 

ABSTRACT ...................................................................................................... iii 

LIST OF FIGURES ........................................................................................... vii 

LIST OF TABLES ............................................................................................. viii 

EXECUTIVE SUM M ARY .................................................................................. ix 

ACKNOW LEDGEM ENTS .................................................................................... A 

ACRONYM S ..................................................................................................... xiii 

1. INTRODUCTION ................................................................................................................................ 1 

1.1 Purpose and Objectives ................................................................................................................... 1 

1.2 Current Staffing Requirements for Operating Nuclear Power Units ......................................... 1 

1.3 Review of New Reactor Designs ............................................................................................... 1 

1.4 Report Structure .............................................................................................................................. 2 

2. BACKGROUND .................................................................................................................................. 4 

2.1 Review of Advanced Reactor Submittals .................................................................................. 4 

2.1.1 Staffing Levels for the M ain Control Room ....................................................................... 4 

2.1.2 Required Operator Activities ............................................................................................... 4 

2.1.3 Control Room Instrumentation and Control (I&C) Systems and 

Human-System Interfaces .................................................................................................. 6 

2.1.4 Summary .................................................................................................................................. 7 

2.2 Study Issues .................................................................................................................................... 7 

2.3 Performance measures .................................................................................................................... 8 

2.3.1 Evaluating Crew Size, Plant Performance and Crew Performance ..................................... 8 

2.3.1.1 Objective Performance .................................................................................................. 8 

2.3.1.2 Rated Crew Perform ance ............................................................................................ 10 

2.3.1.3 Situation Awareness .................................................................................................... 10 

2.3.1.4 W orkload ........................................................................................................................ 13 

2.3.1.5 Team Interaction ............................................................................................................. 14 

3. ME THOD ........................................................................................................................................... 17 

3.1 Experimental Design ..................................................................................................................... 17 

3.1.1 Participants ............................................................................................................................ 19 

3.1.1.1 Training ........................................................................................................................... 20 

3.1.2 Scenarios ................................................................................................................................ 20 

3.1.3 Simulator Performance Characteristics .............................................................................. 21 

4. ANALYSIS ......................................................................................................................................... 23 

5. RESULTS ........................................................................................................................................... 24 

5.1 Transient M anagement and Task Performance ......................................................................... 24 

5.1.1 Crew Size and Plant Type Effects .................................................................................... 24 

5.2 Rated Crew Perform ance .............................................................................................................. 26 

5.2.1 Inter-Rater Reliability ............................................................................................................ 26 

5.2.2 Crew Size and Plant Type Effects ..................................................................................... 26 

5.3 Situation Awareness, Subjective Workload, and Team Interaction ......................................... 28 

5.4 Situation Awareness ...................................................................................................................... 29 

5.4.1 Scenario Type and Period Effects .................................................................................... 30 

5.4.2 Scenario Effects ..................................................................................................................... 31 

5A.3 Crew Size, Plant Type, and Position Effects ..................................................................... 32 

5.4.4 Crew Size by Scenario Period Effects ............................................................................... 33 

5.4.5 Position Effects ...................................................................................................................... 33 

5.4.6 Crew Size, Plant Type and Scenario Type Effects ............................................................ 34 

5.5 W orkload ...................................................................................................................................... 36 

5.5.1 Scenario and Scenario Period Effects ................................................................................ 36 

5.5.2 Crew Size, Plant Type, and Position Effects ..................................................................... 38 

5.5.3 Crew Size and Plant Type Effects .................................................................................... 38 

5.5.4 Plant Type and Position Effects ......................................................................................... 40 

V 

NUTREG/IA-0137
5.5.5 Crew Size and Plant Type Effects .................................................................................... 41 

5.6 Team Interaction ........................................................................................................................... 43 

5.6.1 Inter-Rater Reliability ............................................................................................................ 43 

5.6.2 Scenario and Scenario Period Effects ............................................................................... 44 

5.6.3 Crew Size and Plant Type Effects .................................................................................... 45 

6. DISCUSSION ..................................................................................................................................... 49 

6.1 General Trends .............................................................................................................................. 49 

6.2 Effects of Crew Size ..................................................................................................................... 50 

6.3 Effects of Plant Type .................................................................................................................... 50 

6.4 Interactions of Crew Size and Plant Type and Effects on Operator Performance ..................... 52 

7. INSIGHTS .......................................................................................................................................... 55 

8. REFERENCES ................................................................................................................................... 57 

APPENDIX A: SCENARIO DESCRIPTIONS ............................................................................. A-1 

A. 1 Scenario 1: Steam Generator Tube Rupture With A Fire In The Turbine 

Hall And A Stuck Open Steam Generator Safety Relief Valve ................................................. A-2 

A.I.1 Synopsis ............................................................................................................................. A-2 

A. 1.2 Probable course of events ................................................................................................... A-3 

A.2 Scenario 2: Interfacing Systems Loss Of Coolant Accident: Primary Leakage Outside The 

Containment With Ensuing Control Room Instrument Malfunctions ............................................... A-7 

A.2.1 Synopsis ............................................................................................................................. A-7 

A.2.2 Probable course of events .................................................................................................. A-8 

A.3 Scenario 3: Total Loss Of Feedwater ................................................................................ A-11 

A.3.1 Synopsis ........................................................................................................................... A-11 

A.3.2 Probable course of events .......................................................................................... A- 12 

A.4 Scenario 4: Loss Of Off-site Power With A Stuck Open Steam Generator 

Safety Relief Valve ................................................................................................................. A-15 

A.4.1 Synopsis ........................................................................................................................... A-15 

A.4.2 Probable course of events .......................................................................................... A-16 

A.5 Scenario 5. Steam Generator Over-fill ............................................................................. A-18 

A.5.1 Synopsis .................................... : ...................................................................................... A-18 

A.5.2 Probable course of events ................................................................................................ A-19 

APPENDIX B: INSTRUCTIONS TO PARTICIPANTS ................................................................ B-1 

B.1 General Instructions to Participants .......................................................................................... B-2 

B.2 Instructions for Situation Awareness Questionnaire .............................................................. B4 

B.3 Participant Instructions: W orkload Rating Scales ................................................................ B-5 

B.4 Instructions for the Behaviorally Anchored Rating Scale ......................................................... B-6 

APPENDIX C: INVENTORIES ..................................................................................................... C-1 

C.A Situation Awareness Control Room Inventory Example Inventory ........................................... C-2 

C.2 NASA TLX Rating Scale Definitions ................................................................................. C-4 

C.3 Behaviorally Anchored Rating Scales Example Inventory ........................................................ C-6 

C.4 Rated Crew Performance Example Inventory ......................................................................... C- 12 

APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI) ...... D-1 

D. 1 SACRI: The Inventory of Questions ........................................................................................ D-2 

D.2 W ording of the SACRI Questions ............................................................................................. D-3 

D.3 Answer Categories .................................................................................................................... D-3 

D.4 Signal Detection Theory Calculations .................................................................................. D-4 

D.5 Scoring the Situation Awareness Inventories ...................................................................... D-5 

APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS .................... E-1 

APPENDIX F: DESCRIPTIVE STATISTICS ................................................................................ F-i 

vi 

NUREG/IA-0137
LIST OF FIGURES 

No. Title Page 

Figure 3.1 Side view of the Loviisa nuclear power station main control room ................................. 18 

Figure 3.2 Side view of the HAMMLAB experimental control room .............................................. 19 

Figure 5.1 Plot of transient management performance measures by crew size ................................ 25 

Figure 5.2 Plot of transient management performance measures by plant type ................................ 25 

Figure 5.3 Plot of rated crew performance by crew size and plant type ........................................... 27 

Figure 5.4 Plot of average situation awareness by scenario period ................................................... 30 

Figure 5.5 Plot of situation awareness by scenario type and period ................................................... 31 

Figure 5.6 Plot of situation awareness by crew size, plant type, and scenario period ........................ 32 

Figure 5.7 Plot of change in situation awareness from baseline ....................................................... 33 

Figure 5.8 Plot of situation awareness by crew size and plant type .................................................. 34 

Figure 5.9 Plot of situation awareness by crew size, plant type, and scenario type ......................... 35 

Figure 5.10 Plot of average subjective workload across scenario periods ....................................... 37 

Figure 5.11 Plot of average subjective workload by crew size and position ..................................... 38 

Figure 5.12 Plot of subjective workload by plant type and position ................................................ 39 

Figure 5.13 Plot of subjective workload by control room crew member ......................................... 39 

Figure 5.14 Plot of operator workload by plant type, crew member, and scenario period ................ 40 

Figure 5.15 Plot of operator workload by crew size, plant type and scenario period ....................... 41 

Figure 5.16 Plot of operator workload change from baseline ........................................................... 42 

Figure 5.17 Plot of average team interaction across scenario periods ............................................. 44 

Figure 5.18 Plot of team interaction by crew size and plant type .................................................... 45 

Figure 5.19 Plot of team interaction by plant type and scenario period ............................................ 46 

Figure 5.20 Plot of team interaction by crew size, plant type, and scenario period .......................... 47 

Figure 5.21 Plot of team interaction change from baseline .............................................................. 48 

vii 

NUREG/IA-0137
LIST OF TABLES 

No. Title Paeg 

Table 2.1 Summary of advanced control room characteristics taken from vendor descriptions ...... 6 

Table 3.1 Experimental study design ............................................................................................... 17 

Table 5.1 Effects of scenario, crew size, and plant type on transient management and task 

performance ............................ ........................................................................................... 24 

Table 5.2 Effects of scenario, crew size, and plant type on rated crew performance ........................ 26 

Table 5.3 Results of repeated measures MANOVA on situation awareness, workload, 

and team interaction performance measures .................................................................... 28 

Table 5.4 Effects of scenario type, crew size, plant type, position, and scenario period on 

situation aw areness ............................................................................................................... 29 

Table 5.5 Effects of scenario, crew size, plant type, position, and scenario period on operator 

w orkload ................................................................................................................................ 36 

Table 5.6 Effects of scenario, crew size, plant type, and scenario period on team interaction ...... 43 

Table E- 1 Ratings of transient management tasks on the Steam Generator Tube 

R upture scenario .................................................................................................................. E-2 

Table E-2 Ratings of transient management tasks on the Interfacing Systems Loss of 

Coolant scenario .................................................................................................................. E-3 

Table E-3 Ratings of transient management tasks on the Loss of Feedwater scenario ........................ E-4 

Table E-4 Ratings of transient management tasks on the Loss of Off-Site Power scenario ................. E-5 

Table E-5 Ratings of transient management tasks on the Steam Generator overfill scenario ............ E-6 

Table F-I Summary table of situation awareness data: scenario type by scenario period ................... F-2 

Table F-2 Summary table of situation awareness data: crew size by scenario period ......................... F-2 

Table F-3 Summary table of situation awareness data: plant type by scenario period ........................ F-2 

Table F-4 Summary table of situation awareness data: crew size and plant type by 

scenario period ..................................................................................................................... F-3 

Table F-5 Summary table of situation awareness data: position and plant type by 

scenario period ..................................................................................................................... F-3 

Table F-6 Summary table of situation awareness data: crew size, plant type, and 

scenario type by scenario period .......................................................................................... F-4 

Table F-7 Summary table of subjective workload data: scenario by scenario period .......................... F-4 

Table F-8 Summary Table of Subjective Workload data: Crew Size by Scenario Period .................. F-4 

Table F-9 Summary table of subjective workload data: plant type by scenario period ................... F-5 

Table F-10 Summary table of subjective workload data: position by scenario period .................... F-5 

Table F-11 Summary table of subjective workload data: position by plant type by 

scenario period ................................................................................................................... F-5 

Table F-12 Summary table of workload data: crew size by plant type by scenario period ................. F-6 

Table F-13 Summary table of subjective workload data: crew size by position by 

scenario period ................................................................................................................... F-6 

Table F-14 Summary table of team interaction data: scenario by scenario period .............................. F-6 

Table F-15 Summary table of team interaction data: crew size by scenario period ............................ F-7 

Table F- 16 Summary table of team interaction data: plant type by scenario period ........................... F-7 

Table F-17 Summary table of team interaction data: crew size and plant type by 

scenario period ................................................................................................................... F-7 

Table F-18 Summary table of rated crew performance data by scenario ............................................. F-8 

Table F-19 Summary table of rated crew performance data: crew size by plant type ......................... F-8 

Table F-20 Summary table of objective performance rating data by scenario ..................................... F-9 

Table F-21 Summary table of objective performance rating data: crew size and plant type 

by scenario ......................................................................................................................... F-9 

viii 

NUREG/IA-0137
EXECUTIVE SUMMARY 

An empirical study was conducted to evaluate issues affecting staffing requirements in advanced nuclear 

power plants. New plant designs rely on passive systems to achieve safety objectives. With improvements 

in ease of plant operation, some vendors expect staffing requirements for advanced plants to differ from 

current regulations. Since the advanced passive plants under consideration are not yet built, no opportunity 

exists for observation of plant operation and the demands placed on the operating staff. 

The study was conducted to provide data to support design review guidance. Two factors were evaluated 

across a range of plant operating conditions: 1) control room crew staffing sizes; and, 2) the effects of 

characteristics of the operating facility itself, whether employing conventional or advanced features. 

Design factors influenced the performance of crews in this study. Crews in the conventional plant normal 

staffing complement exhibited better performance than minimum sized crews. Such plants have been 

designed with a larger control room staffing complement in mind. The functions, allocations, size, and 

automation in such control rooms require more crew members to maintain control of the plant. In the 

advanced plant, the minimum sized crews performed better than the normal sized crews. The advanced 

plant control room setting was designed for a smaller control room complement, being more compact and 

taking advantage of computer-based technologies more than the conventional plant. The compact control 

room was designed to support a more limited crew size than the conventional plant, being smaller, having 

greater automation, passive design features, and displays that better integrate control room information. 

Therefore, decisions about control room staffing should be based upon design features including: function 

allocation, automation, integration, and plant-specific characteristics (e.g., passive system performance). 

Validation and verification using measures of operator and crew performance are necessary to determine the 

staffing complement needed to operate the plant. 

This study was carried out in two phases by collecting data on control room crew performance. The first 

phase was conducted at the Loviisa nuclear power station, in Loviisa, Finland. The second phase was 

carried out at the Halden Human-Machine Laboratory (HAMMLAB) at the OECD Halden Reactor Project in Halden, Norway. The Loviisa plant served as the conventional plant for this study; HAMMLAB served 

as the advanced plant. Both facilities are driven by a simulated plant model of the Loviisa nuclear power 

plant. Two crew configurations participated in each of the plant types: a crew consisting of four members, 

and a crew of 2 in the advanced plant; 3 in the conventional plant. 

Eight crews of operators from the Loviisa nuclear power station participated in the study. All crews 

included a reactor operator (RO) and a balance of plant operator (BOP operator). In the 2-man crew, one 

operator performed supervisor functions. The 3-man crew included a dedicated Control Room Supervisor 

(CRS). The 4-man crew had a CRS as well as a control room technician (CT) who is a licensed control 

room operator. 

Each crew participated in five scenarios: a Steam Generator Tube Rupture (SGTR), a sustained Total Loss 

of Feedwater (LOFW), a Loss of Off-site Power (LOOP), an Interfacing Systems Loss of Coolant Accident 

(ISLOCA), and a Steam Generator Overfill (SGOF). The scenarios were presented in random order. All 

scenarios contained several faults and events that occurred regardless of operator actions, but scenario 

outcome varied depending on the crew's responses. The scenarios required the crew to coordinate with 

external personnel, and required an hour to two-and-a-half hours to complete. All scenarios were divided 

into segments separated by data collection interrupts during which the simulator was frozen. During this 

pause, questionnaire data were collected from the crew and raters. 

Scenarios required different problem solving techniques to mitigate disturbances. Three scenarios, the 

SGTR, LOFW, and LOOP were straight-forward to diagnose, each having a number of salient symptoms, 

and could be mitigated by following procedures. These are referred to as rule-based scenarios. The other 

two scenarios, the ISLOCA and the SGOF contained misleading or hidden symptoms, were difficult to 

ix 

NUREGAIA-0137
diagnose, or included features that complicated operators' access to procedures (e.g., temporal demand for 

operator response, etc.). These are referred to as knowledge-based scenarios. 

A number of metrics were used to assess operator and plant performance. Objective performance measures 

included task completion, critical task initiation time, and control of plant parameters. Other measures 

included situation awareness, operator workload, team interaction, and rated crew performance. Data were 

collected across time periods in the scenarios. 

All crews received simulator-specific training on Instrumentation and Control (I&C) systems and plant performance characteristics. They also were informed about the overall purpose of the study, and were 

instructed on completing the inventories used in the study. The raters who evaluated team interaction and 

crew performance received inventory-specific training. 

The minimum sized crews in the advanced plant demonstrated the best rated crew performance. In terms of 

objective performance, advanced plant crews performed better than conventional plant crews. While crews 

in the advanced plant experienced significantly higher workload than crews in the conventional plant, their 

performance was unimpaired. 

Normal sized crews performed better than minimum size crews on cooldown and stabilization tasks. This 

difference may be explained by workload and task shedding. The normal sized crews experienced lower 

workload than the minimum sized crews, and thus were able to complete more tasks. The minimum sized 

crews shed tasks that were not perceived as critical. However, both crew sizes performed similarly on 

critical task completion. 

Crews in the advanced plant setting achieved higher situation awareness than crews in the conventional 

plant. Minimum sized crews performed better than normal sized crews in the advanced plant; conversely, normal sized crews performed better than minimum sized crews in the conventional plant. This interaction 

may be explained by design aspects of the control rooms that better support different crew staffing 

complements. 

Minimum sized crews experienced more workload than normal sized crews, with the control room 

supervisor experiencing most of the additional workload. Crew members are resources available to the 

control room supervisor, and their absence increases tasks for the supervisor. Advanced plant crews 

experienced significantly higher workload than their conventional plant counterparts. During debriefings, 

operators indicated that using the automated systems in HAMMLAB, different from their normal work 

environment, created additional workload. In the advanced plant setting, the control room supervisor also 

served as one of the operators, and this dual role CRS experienced much higher workload than all other 

operators. During the debriefings, the dual role CRSs expressed concerns about working effectively in such 

a stressful capacity. 

Crews in the advanced plant setting exhibited better and more stable team interaction than crews in the 

conventional plant. This improvement possibly relates to design differences between the conventional and 

advanced plant settings used in this study. Many conventional plants have relatively large control rooms in 

which operators may be distant from one another. The advanced plant configuration provided close 

operator workstations centered around a common overview display, which served as a focus of operator 

discussions. 

x 

NUREG/IA-0137
ACKNOWLEDGEMENTS 

The authors appreciate the dedication and effort extended to this study by a number of key persons; firstly, the staff of Imatran Voima OY's Loviisa nuclear power generating station. We thank all of the operators who volunteered to participate in this study; whose hard work under challenging circumstances made this 

work possible. We also thank Mr. Pekka Sammatti and Dr. Jussi K. Vaurio for their assistance in arranging 

for this study with the Operations group at the plant, and for their assistance throughout the entire study. Pekka Sammatti and Ray Saarni from the Loviisa Plant and Pekka Pyy from the Technical Research Center 

in Finland, VTT, provided invaluable assistance throughout this study, beginning with the development of 

the study scenarios, to issues of experimental design and realism, through data collection and analysis. They provided a much needed and valued liaison with the operators in this study on behalf of the 

researchers and assisted in the development of analysis techniques. The simulator staff at both facilities, the 

Loviisa training simulator and the Halden Human-Machine Laboratory, worked diligently to make the 

necessary modifications to the simulator models and I&C systems, and conducted operations expertly during data collection under very dynamic conditions. We noted many times that the simulator operators' workload appeared nearly to match that of the operators in the simulator during periods of high interaction 

and role play with the operators as they worked to maintain the realism of the scenarios. Finally, we 

acknowledge the assistance and support from Dr. Persensky of the U.S. Nuclear Regulatory Commission. 

We asked for and received a great deal of input on the aims and objectives, as well as background and 

design aspects of this study that greatly facilitated 

the work presented in this report. 

xi 

NUREG/IA-0137
ACRONYMS

ABB 

AECL 

ANOVA 

AP600 

BARS 

BOP 

BWR 

CANDU 

CFR 

CRS 

CRT 

CT 

EOP 

ERG 

FR 

GE 

HAMMLAB 

HILOAS 

I&C 

ISLOCA 

LOCA 

LOOP 

LOFW 

LOTS 

LWR 

MANOVA 

NASA 

NPP 

NRC 

NSSS 

OECD 

PIUS 

PWR 

RCS 

RO 

S/G 

SA 

SACRI 

SAR 

SBWR 

SGOF 

SGTR 

SME 

SRO 

SSAR 

STA 

TLX 

VDU 

Vrr

Xiii

NUREG/IA-0137

Asea, Brown, Boveri 

Atomic Energy of Canada Limited 

Analysis of Variance 

Advanced Plant 600 

Behaviorally-Anchored Rating Scale 

Balance of Plant 

Boiling Water Reactor 

Canadian Deuterium Uranium 

Code of Federal Regulations 

Control Room Supervisor 

Cathode Ray Tube 

Control Room Technician 

Emergency Operating Procedure 

Emergency Response Guidelines 

Federal Register 

General Electric 

Halden Human Machine Interaction Laboratory 

High Level Operator Action Strategies 

Instrumentation and Control 

Interfacing Systems Loss of Coolant Accident 

Loss of Coolant Accident 

Loss of Off-site Power 

Loss of Feedwater 

Loviisa Training Simulator 

Light Water Reactor 

Multivariate Analysis of Variance 

National Aeronautics and Space Administration 

Nuclear Power Plant 

Nuclear Regulatory Commission 

Nuclear Steam Supply System 

Organization for Economic Cooperation and Development 

Process Inherent Ultimately Safe Plant 

Pressurized Water Reactor 

Reactor Coolant System 

Reactor Operator 

Steam Generator 

Situation Awareness 

Situation Awareness Control Room Inventory 

Safety Analysis Report 

Simplified Boiling Water Reactor 

Steam Generator Overfill 

Steam Generator Tube Rupture 

Subject Matter Expert 

Senior Reactor Operator 

Standard Safety Analysis Report 

Shift Technical Advisor 

Task Loading Index 

Video Display Unit 

Technical Research Center of Finland
INTRODUCTION

1. INTRODUCTION 

1.1 Purpose and Objectives 

The purpose of this study was to evaluate the impact(s) of advanced passive plant design and staffing of 

control room crews on operator and team performance. It was conducted to contribute to the understanding 

of potential safety issues and provide data to support the review of proposed staffing levels for advanced 

passive reactors. Two factors were evaluated across a range of plant operating conditions: 1) control room 

crew staffing complement; and, 2) the effect of characteristics of the operating facility itself, whether 

employing conventional or advanced, passive features on operator performance. 

This study was carried out by collecting data on control room crew performance. The first phase of this 

study was conducted at the Loviisa nuclear power station, in Loviisa, Finland. The second phase was 

conducted at the Halden Human-Machine Laboratory (HAMMLAB) at the OECD Halden Reactor Project 

in Halden, Norway. The Loviisa plant served as the conventional plant for this study; HAMMLAB served 

as the advanced plant. Both facilities are driven by a simulated plant model of the Loviisa nuclear power 

plant. The control room systems at Loviisa used in this study are representative of those in conventional 

control rooms, as are the time constants of the plant's thermal-hydraulic performance. The control room 

systems used in HAMMLAB consist of video display unit-based information and control systems, similar in 

form and function to those of advanced plant control rooms. In addition, some of the time constants of the 

thermal-hydraulic model in HAMMLAB were longer than at Loviisa to emulate the longer thermal

hydraulic time constants in advanced plants. These longer time constants are due to the performance of 

passive systems or other advanced design features. 

1.2 Current Staffing Requirements for Operating Nuclear Power Units. 

The operation and management of a nuclear power plant require a sustained, coordinated effort to achieve 

both power production goals and compliance with established safety requirements. The focus of these 

efforts is usually the main control room, where operations are conducted by licensed personnel. The control 

room staff has the responsibility for oversight and decisions about the operation of the plant. Because of the 

dependence on these personnel, qualification requirements and minimum licensed operator staffing of 

nuclear power plants are specified by federal regulation, found in lOCFR 50.54(m). 

1.3 Review of New Reactor Designs 

In addition to ensuring compliance of existing nuclear units with regulatory requirements, the U.S. Nuclear 

Regulatory Commission (NRC) also conducts Design Certification reviews of proposed reactor designs to 

ensure the compliance of a vendor's proposed design with regulatory requirements. New plant designs have 

been produced by a number of vendors. Some of these advanced plant designs rely on new design or 

application approaches using passive systems to achieve increased redundancy and/or diversity of methods 

to achieve safety objectives. Although they use proven technologies to a large extent, these advanced 

passive plant designs also differ in many respects from conventional plant design. 

The Advanced Reactor Policy Statement (51 FR 24643) and NUREG-1226, "Development and Utilization 

of the NRC Policy Statement on the Regulation of Advanced Nuclear Power Plants," define advanced 

reactors as those with innovative designs for which licensing requirements will differ significantly from 

existing light-water reactor (LWR) requirements. These documents also provide guidance for the 

NUREG/IA-0137
INTRODUCTION

development of new regulatory requirements to support advanced designs. To the extent practical, staff 

reviews are based on existing regulations. When new requirements are necessary, there is a preference for 

performance-based rather than prescriptive regulations. 

With improvements in ease of plant operation gained through passive design and digital instrumentation and 

control systems, vendors expect differences in operator performance needed to achieve plant safety 

objectives for credible events. These expected differences have led to a need to reconsider the requirements 

for minimum shift staffing of licensed Reactor Operators and Senior Reactor Operators contained in 10 

CFR 50.54(m). 

Since the advanced passive plants under consideration are not yet built, no opportunity exists for 

observation of plant operation and the demands placed on the operating staff. Operating experience with 

these plants and systems is limited to similar systems in other plants (e.g., BWR isolation condenser 

designs, PWR N2 injection accumulators) and designs incorporating similar features (i.e., introduction of 

digital control systems into existing analog-based control rooms). Although the basic principles of 

operating these new plants are similar to existing plants, the range of operating conditions and response 

requirements may be quite different. Hence, the demands placed on the operating crew itself may be 

different. 

1.4 Report Structure 

The following sections present the background, method, analysis approach, results, conclusions, and a 

summary of this study. The Background section provides a summary of the review of advanced reactor 

submittals that was conducted for this study, and the key information that was obtained from this review. 

The Background section also presents the study issues that were identified for this research project, and a 

discussion of the performance measures that were used to assess the study issues. 

The Method section includes a discussion of nuclear power plant staffing practices. Current staffing 

requirements and proposed advanced design staffing levels are compared. Issues such as crew size and 

team performance are addressed. Issues expected to influence staffing requirements in advanced plants, 

automation, passive plant features, and crew size, are also evaluated. The study design and dependent 

measures, the scenarios in which these measures were assessed, and how the data collection was performed 

are discussed. 

The Analysis section provides a summary of the approach and rationale used to conduct the statistical 

analyses of data that were gathered. This section includes a discussion of how the level of statistical 

significance was adopted for this study, and the general approach to the evaluation of study issues using 

tests of statistical significance. 

The Results section presents the results of the analyses of performance measures used to evaluate the effects 

of staffing and plant type on operator performance. Significant interactions are also discussed. The results 

of analyses are presented in tables that show the particular effects tested and the resulting test results. This 

section also contains a number of graphs of main results, to assist in the interpretation of study results. 

The Discussion addresses the results of statistical analyses of the effects of crew size and plant technology 

on operator performance. This section contains separate discussions about general trends observed in the 

study, and the effects of crew size, plant type, and their interaction. Implications of these results are outlined 

in the Insights section, which discusses control room staffing issues for advanced plants that were identified 

by this study. 

References and appendices are included at the end of the report. Appendices contain: 

", synopses and detailed descriptions of the scenarios used in this study; 

"* instructions given to participants; 

"* copies of inventories administered; 

2 

NIJREGJEA-0137
INTRODUCTION

"* information regarding the development and scoring of the situation awareness 

inventory; 

"* transient management performance rating forms; and 

"* descriptive statistics of performance measures collected in the study. 

3 

NUREG/IA-0137
BACKGROUND

2. BACKGROUND 

2.1 Review of Advanced Reactor Submittals 

Vendor submittals provide information to be used in the process of design certification and address a wide 

range of issues. To develop an approach for the study of crew size and crew performance, advanced reactor 

submittals were reviewed. The reviews were conducted to determine: the types of staffing levels 

envisioned; activities required of the main control room crew in operating these new plants; and the types of 

instrumentation and control systems employed in the human-system interface for carrying out information 

gathering and control activities. The advanced reference plant designs used include the Westinghouse 

AP600, ABB Atom PIUS Plant, General Electric Simplified Boiling Water Reactor (SBWR), and the 

Atomic Energy of Canada Limited CANDU 3 plant designs. The following summarizes the main 

conclusions of the review. 

2.1.1 Staffing Levels for the Main Control Room 

For most power operations, a reduced-staff control room crew is envisioned to be sufficient by most 

vendors. This is attributed to improvements gained through automation (e.g., function allocation) of many 

activities that are manually conducted in today's plants. In both the Westinghouse AP600 and GE SBWR 

plant designs, staffing considerations, including accident situations, follow the requirements of 10 CFR 

50.54 (m) (Westinghouse AP600 SSAR Section 18.7.1 "Basis for Staffing Requirements," and GE SBWR 

SSAR Section 18.2 "Design Goals and Design Bases"). The AP600 submittal allows for a shift staffing 

complement that would be larger than that required by 10 CFR 50.54 (in). This includes additional licensed 

operating personnel to interface with other plant staff and a separate position for the shift technical advisor, 

resulting in up to six licensed operators (Westinghouse AP600 SSAR Section 18.7.2.1 "Main Control 

Area"). 

At present, the staffing requirements for the ABB PIUS plant differ from the current federal regulations. 

I&C systems in the ABB PIUS plant are designed to require only one operator in the control room for 

normal operation (Section 501.2.1). Furthermore, the design goal of the plant for operations is not 

envisioned to exceed two operators in the control room, though one should be fully adequate for controlling 

the plant (Section 511.2). Staffing issues were not addressed in AECL's CANDU 3 submittal. 

Thus, the shift staffing plans vary considerably for different advanced plants. This variation in itself may 

not be at conflict with safety objectives, depending on the actual demands for human intervention in both 

preventing, detecting, and responding to accident conditions. Passive system operation and automation 

may reduce demands on the operating crew for direct intervention and minimize the likelihood for errors. 

Were this the case, a smaller staffing complement may be capable of operating a plant safely. Thus, staffing 

plans for advanced plants should also be considered in light of the activities required of the operating staff. 

2.1.2 Required Operator Activities 

One of the reasons given by vendors for reduced control room staffing is a general reduction in the number 

of tasks allocated to operating personnel. Increased allocation of functions to plant control systems, and 

improvements in the ease of performing remaining tasks due to advanced I&C systems reduce the number 

of staff required. This expectation appears more valid for activities carried out during normal operations 

than with emergency operations. Two of the vendor submittals contained information about operator 

activities required for emergencies: the Westinghouse and GE SSARs. To evaluate differences in the 

anticipated operator actions between advanced and conventional plants, procedures or operator action 

4 

NUREG/IIA-0137
BACKGROUND

strategies were compared. The High Level Operator Action Strategies (HILOAS) of the Westinghouse 

AP600 were compared with the EOPs of a conventional 3-loop Westinghouse PWR. A similar comparison 

had also been made by GE in the Emergency Response Guidelines (ERGs) for the SBWR with Revision 4 

ERGs for conventional BWRs. Differences were found in operator activities between conventional and 

advanced plants. 

Most importantly, the EOP structures of the advanced plants are similar to those of operating plants, often 

having the same EOP titles, general structure, and individual steps. A detailed comparison was made of 

operator activities contained in these documents. However, the HILOAS are activity descriptions, not 

actual procedures, and they may change when they are converted into procedures. Hence, this comparison 

of AP600 HILOAS with EOPs is actually a rough determination of activity differences. Some of the 

differences observed may thus be attributed to the level of detail provided in the conventional plant's 

procedures. They are written for an operational facility, in contrast to operator action strategies written at a 

stated "high level." 

Bearing this in mind, a number of differences were observed between the conventional and advanced plant 

documents used to determine operator activities. The differences were characterized as wording changes, 

sequence changes, and activity changes. Wording changes do not result in significant changes in the 

activity of the operating crew, or in the intent of the step (i.e., "Check RCS Pressure" to "Check RCS 

Pressure within Plant Specific Limits"). A sequence change merely involves changing the order of a set of 

actions, though the balance of activities and, ostensibly, their objective remains the same. An activity 

change represents a likely change in the task(s) or intention of the activity. Examples of activity changes 

include the deletion of steps for establishing feed and bleed or once-through core cooling due to changes in 

plant operation, or the deletion of a task requiring a manual action to one requiring verification (e.g., 

"Check if Passive Containment Cooling is Required"). A total of 32 EOPs/ERGs was reviewed. On 

average, two to three wording changes, one to two sequence changes, and four activity changes were found 

in each EOP/ERG. 

In summary, two of the vendors provided material about planned operator activities that was sufficiently 

detailed to allow comparison with operator activities in conventional plants. The strategies and safety 

functions for these advanced LWR plants appear analogous with those of the conventional LWR plants 

though some differences were found at the activity level. These differences are primarily due to changes in 

the systems employed in the advanced plants for maintaining critical plant functions, and not due to a 

change in the operational philosophies or functional response strategies. Unfortunately, the one vendor that 

expects the biggest differences in the necessary operational staff and systems and strategies for achieving 

plant safety did not furnish sufficient information to make a similar comparison of operator activities. 

Hence, the basis for expected differences in operator activities between conventional and advanced plants is 

limited to the insights gained from a review of material provided by two LWR vendors. 

The tasks that operators must perform in the two types of plant may be more different than indicated by the 

activity descriptions. In conventional plants, analog information and manual control devices dominate the 

I&C systems in the control room. The use of advanced technology has a potential to change the ways in 

which operators gather information, the types of information presented, operator response modes (e.g., 

motor, perceptual, cognitive, etc.) and the types of operator-system interactions that occur when conducting 

control room activities. Therefore, vendor plans for main control room I&C systems were reviewed. 

5 

NUREG/IA-0137
BACKGROUND

2.1.3 Control Room Instrumentation and Control (I&C) Systems and Human-System 

Interfaces 

All vendors described the general layout; work areas provided for crew members; information systems; 

methods of interaction; and, to some extent, automatic modes of operation provided in the main control 

room of their plant design. The control rooms in each of the advanced plants rely heavily upon advanced 

I&C technology, compared to conventional plants. Table 2.1 summarizes the main features of the control 

room as described by the vendors. 

Table 2.1 Summary of advanced control room characteristics taken from vendor descriptions 

Operator Control * Soft control units. 

"* Tracker ball, touch screens, keyboards, mouse. 

"* Function keyboards. 

"* Dedicated fixed-position or hard-wired function 

switches. 

"* Display selection controls. 

"* Dedicated safety system controls. 

"* Functionally-arranged physical control panels for 

backup (1 vendor only). 

"* Procedure displays (1 vendor only). 

Information Systems VDUs presenting: 

a Physical mimic displays 

* Alarms 

• SPDS 

* Overview Displays (e.g., process mimics) with 

important alarm information 

* Parameter trend VDUs. 

Configurations a 2 Operator Workstations on a single integrated 

console or separate RO and BOP workstations. 

* 1 supervisor's console, with or without control 

capabilities, located aft of operator workstations. 

* Overview and alarm displays in a fixed wall 

position. 

Alarm Processing 0 Processing techniques to reduce alarms including 

suppression or filtration via: 

"* prioritization; 

"* mode; 

"* subsidiary; 

"* redundancy; 

"* system. 

Operators interact with the information and control systems via tracker ball, mouse, or other dedicated input 

device (i.e., touch screen). Requests for information such as process formats and the navigation through the 

information system are largely accomplished by requests through either menus, touch screens, or function 

keyboards. VDUs provide information to the crew about process parameters, integrated as part of either 

plant mimic diagrams, detailed process diagrams, trends, or in connection with alarms. The integration of 

6 

NUREGJIA-0137
BACKGROUND

this information and the specific systems developed for providing this information are vendor-specific and 

differ somewhat from plant to plant. 

Operations in the advanced plants will be carried out primarily using computer-based technologies to obtain 

information and to issue commands to the equipment and systems in the plant for both normal and 

emergency operations. Work areas for the operating crew are divided into separate computer-based 

workstation areas either physically separate from each other or functionally grouped for each operator. In 

contrast to existing control rooms, the control room supervisor also has a workstation for information 

gathering. Each operator typically has access to the same information and displays as the other operators. 

Each operator's work area or workstation typically has between three and seven VDUs for display and 

control, in addition to the wall panel overview and alarm displays. In most cases, operators also have the 

ability to obtain additional information about alarms using some of their VDUs. In some cases, specific 

VDUs are allocated in an operator's workstation or area for this purpose. 

2.1.4 Summary 

Due to the way advanced passive plant vendors expect these plants to be operated, compared to existing 

plants, some differences were noted in the amount of manual control required to safely operate these plants. 

In some cases, a change in the size of the minimum control room staff is suggested by a vendor. Some 

vendors, though, expect no change to the minimum control room staff compared to the current federal 

regulation. Descriptions of emergency operations indicate that the types of activities operators will carry 

out in the advanced plants are similar to activities carried out in existing plants. However, the vendor 

expecting the largest change in the amount of operator intervention to operate its plant did not provide 

sufficient information to allow comparison with either existing plant or other advanced plant activity 

descriptions. The ways in which operators will perform activities in the advanced plants will change 

significantly from existing plants due to the use of advanced computer-based systems for information 

display and control functions. 

2.2 Study Issues 

This study was concerned with the effects of advanced plant technologies and reductions in the minimum 

control room crew staffing complement on operator performance. Performance measures were defined for 

the study (see Section 2.3) and an experimental design employed that would permit collection of data to 

evaluate these effects. To evaluate the effects of the independent variables, a number of study issues were 

identified. These are described below. 

" Do changes in the staffing complement of the main control room produce measurable differences in the 

performance of operators? 

"* Compared to the reference (i.e., normal) crew size, do smaller sized crews exhibit poorer 

performance on transient management and mitigation activities? 

"* Is the workload of crews in a smaller crew staffing complement different than that of a normal 

staffing complement? If so, does the workload of smaller sized crews exceed a level beyond 

which performance on other measures degrades? 

"* Do any particular crew members appear to be affected most by a reduction in crew size? 

"* Are crew members able to maintain an awareness of the plant when one or more members of 

the control room crew are removed from the control room complement? 

"* Are crew performance characteristics, i.e., how the crew works as a team, negatively affected 

by a reduction in crew size? 

"* Do changes in the type of plant automation and control systems produce measurable differences in the 

performance of operators? 

7 

NUREGAIA-0137
BACKGROUND

"* Are operators in the advanced plant able to manage the demands of the transients as well as 

operators in the conventional plant? 

"* Is the workload of crews in the advanced plant different than the workload of crews in the 

conventional plant? If so, does it exceed or fall below an optimal level such that performance 

on other measures degrades? 

"* Compared to the conventional plant, are operators in the advanced plant able to maintain their 

awareness of the plant? 

"* Are any particular crew members affected most by a change in the task demands of operating 

the plant using a higher degree of automation? 

"* Are crews able to maintain their teamwork skills when using the more highly automated 

systems? 

* Does the combination of changes in control room staffing together with changes in plant automation 

and control systems produce measurable differences in the performance of operators? 

* Does the interaction of crew size and plant type effect the performance of crews on transient 

management tasks? 

0 Does the interaction of crew size and plant type effect the workload of crews? 

* Does the interaction of crew size and plant type effect the ability of crews to maintain an 

awareness of the plant? 

0 Does the interaction of crew size and plant type effect the way in which crews work together 

as teams? 

2.3 Performance measures 

2.3.1 Evaluating Crew Size, Plant Performance and Crew Performance 

To evaluate the effects of crew size and plant type, performance criteria were defined. Firstly, since one of 

the objectives of this study was to identify potential safety issues associated with either a reduced staffing 

complement or features of the advanced plants (e.g., passive plant performance, required operator activities, 

human-system interfaces) the study employed operating environments representative of both conventional 

and advanced plants. The issue of safety was to be partly addressed by a comparison of operator 

performance in the advanced plant setting with that in the conventional plant. A concern might exist, or 

warrant further consideration, if operator performance on some criteria in the advanced plant degraded 

below that exhibited by crews in the conventional plant setting, in different control room staffing 

complements, or by the interaction (e.g., combination) of plant type and crew size. 

Secondly, measures of operator performance must be sensitive to changes that can be attributed to the 

different staffing and operating environments. They must also provide meaningful information about the 

ability of operators to manage the plant under simulated emergencies. The measurements taken should 

provide information about the objective performance of crews, and contextual (e.g., subjective) factors that 

influence their performance. The measurements should further provide information about operator and 

crew performance so that an assessment can be made about the effects that a reduction in crew size may 

have on the remaining operators in the team, and whether the effects of such reductions degrade 

performance. 

Five metrics were used to assess crew performance: objective performance indicating the crew's 

management of transient mitigation activities; rated crew performance; situation awareness; operator 

workload; and team interaction. These are discussed in more detail below. 

2.3.1.1 Objective Performance 

8 

NUREG/IA-0137
BACKGROUND

Data indicating crews' management of the scenarios were collected. Important tasks were identified prior 

to the simulator portion of the study via task analyses of each scenario (e.g., manual reactor trip, pressurizer 

cooldown) and initiation times for these tasks were obtained during the experiment. Plant parameters were 

logged from the simulator and stored for later analysis. Simulator event logs list key system events (e.g., 

reactor protection system signals, pressurizer heater bank actuations), automatic and manual system 

actions, and alarms. In addition, audio-video recordings of each crew's performance were stored on video 

cassette for later evaluation. 

Task initiation times were obtained for important tasks identified prior to the simulator portion of the study 

from task analyses of each scenario (e.g., manual reactor trip, pressurizer cooldown). Plant parameters 

were logged from the simulator and stored for later analysis. Simulator event logs were also used to 

identify key system events (e.g., reactor protection system signals, simulator instructor-initiated faults and 

malfunctions, etc.), automatic and manual system actions, and alarms (e.g., accumulator level low level or 

gradient alarm, critical function alarms). In addition, audio-video recordings of each crew's performance 

were stored for later use. 

A licensed shift supervisor (SRO) from the Loviisa plant reviewed the performance of crews using event 

logs from scenarios, plant parameter data, and audio and video tapes of crew performance during the 

scenarios. Three groups of tasks were identified, each necessary for successful management of transients: 

Announcements and Notifications, Critical Task Completion, and Stabilization and Cooldown. In all five 

scenarios, operators were required to make a standard set of announcements and notifications. Simulator 

instructors noted announcements and notifications made by the crew. Each scenario also contained separate 

Critical Tasks that had to be completed. For example, in the SGTR scenario, a critical task was "isolating 

the faulted steam generator." Similarly, Stabilization and Cooldown tasks had to be accomplished in each 

scenario. Specific details of how these were best accomplished varied among the scenarios, but the overall 

objectives were the same. 

To perform the analyses, a list of sub-tasks was identified for each task group. For Announcements and 

Notifications, for example, a list of the required calls was developed based on operating procedures (e.g., 

emergency organization activation,), technical specifications (e.g., requirements for notifications to the 

safety authority,), and scenario-specific requirements (e.g., mobilizing the fire department, etc.). Each 

crew's performance of announcements and notifications was compared to this list. If a crew made a 

required call, they received a check for making that call. For a crew to receive credit for making a call, 

some announcements and notifications had to be made in a timely manner (e.g. fire notifications). Only 

calls made at an appropriate time were credited. In analyzing the performance of all the crews, the checks 

each crew received were tallied and the crew received a score. 

Analysis of Critical Task Completion was conducted differently. Tasks necessary to mitigate the effects of 

the initiating events for each scenario were identified, resulting in a number of tasks in each scenario. Some 

tasks only needed to be completed (e.g., stop a pump); some needed to be completed within a certain time 

(e.g., close a valve before an overfill occurs); and some required a plant parameter to be controlled (e.g., 

safe filling of steam generators after dryout). Tasks that only needed to be completed were rated with a 

check if they were completed. Tasks that had to be performed within a certain period of time were rated 

based on occurrence and timeliness. Tasks that required complex control actions were rated more 

qualitatively, based on a process expert's evaluation of plant parameters. The process expert evaluated the 

parameters by ensuring that the crew controlled the plant by maintaining the parameters (e.g., margin of 

departure from nucleate boiling) in a safe range. 

Each scenario placed a different set of challenges on the plant and on the control room crew's ability to 

mitigate these events. Some scenarios resulted in overheating of the primary circuit, and eventually 

inadequate core cooling (i.e., the total loss of feedwater scenario). Some resulted in overcooling of the 

reactor coolant system (RCS) (e.g., the steam generator overfill and loss of off-site power with Steam 

Generator (S/G) safety valve stuck open). Other scenarios resulted in a loss of RCS inventory, unfiltered 

radioactive release to the environment, and a threat to the availability and operability of equipment in both 

units due to a common cause external event (e.g., steam generator tube rupture with a stuck open S/G safety 

9 

NUREGIIA-0137
BACKGROUND

valve and a fire in the turbine hall). Thus, the necessary mitigation actions differed in each of the scenarios. 

For each scenario, the necessary actions were identified to mitigate the event and prevent additional 

challenges to equipment or failures, and crew performance was assessed in terms of accomplishing these 

actions. 

Stabilization and Cooldown tasks were evaluated similarly to Critical Task Completion. All of the 

scenarios were of sufficient length that crews should have begun taking the plant from post-disturbance 

conditions to a hot standby or shutdown state. In the transition from important transient-specific event 

mitigation tasks to longer term stabilization activities, crews must be cognizant of many plant conditions. 

Cooldown rates, boiling margins, inventory levels in safety and auxiliary systems, and pressure differentials 

must be maintained within target ranges in order to prevent unintended challenges to equipment and 

systems. Plant parameters and event logs were studied to determine when stabilization activities were 

begun, and how well these tasks were accomplished. The specific tasks that had to be completed for each 

scenario were identified. Crews were evaluated (objectively) in terms of whether or not they completed the 

tasks and a weighting scale was applied (subjectively) in terms of the quality with which they controlled 

various critical plant parameters 

Appendix E contains the transient management rating forms for each of the five scenarios. All crews' 

ratings are presented, as are the criteria for these ratings. 

2.3.1.2 Rated Crew Performance 

A crew performance rating technique was also used to evaluate how well the crew performed on a variety of 

dimensions. The dimensions on which crews were rated were deemed necessary to achieve and maintain 

control of plant performance. This rating technique was taken from a previous study that focused on the 

performance of nuclear power plant control room crews (Hanson et al., 1987). Process experts rated the 

crew's performance on four dimensions: solution path, control of plant, communication, and confidence. 

These dimensions are intuitively linked to the types of behavior required to achieve and maintain control of 

plant performance. Solution path refers to the crew's use of time in recognizing the event and selection of 

the correct mitigation procedure(s). Control of plant refers to the crew's demonstrated understanding of 

procedures in their analysis of the transient and the extent to which they challenged safety equipment. 

Communication refers to the extent to which the information exchange among the crew members facilitated 

transient mitigation. Confidence refers to the ease with which the crew completed transient mitigation 

without hesitation, and self-statements about the sureness of their own actions and decisions. Licensed 

operator examiner personnel created these dimensions through previous evaluations of crew performance. 

This technique uses the observations of knowledgeable or trained observers to provide an evaluation of 

how well the crew performs on these dimensions. Immediately after a crew completed a scenario, three 

process experts rated the crew's performance. The process experts who provided ratings in this study 

include two licensed SRO's, and one engineer knowledgeable on plant operation and plant safety. Each 

dimension was rated on a scale of 1-10, with 1 being the worst performance, and 10 being optimal 

performance. An example of the rating form and a description of the behavioral categories is found in 

Appendix C. 

2.3.1.3 Situation Awareness 

Situation Awareness (SA) is a measure of the operator's understanding of the current process state. Current 

understanding of operator performance indicates that developing and maintaining SA is an integral and 

important aspect of operator activities (Adams, Tenney, & Pew, 1995). During routine monitoring, 

operators periodically gather information to assess the plant state or operations being conducted. Using 

their knowledge of the plant (i.e., their mental model of the process, systems and how they interact), 

operators integrate the information they gather to develop and maintain situation awareness. This part of 

10 

NUREG/IA-0137
BACKGROUND

the feedback loop is necessary to understand the state of the process at a given point in time, to determine 

the need for and type of control activities, and to predict the process state in the future. SA is necessary, but 

not sufficient, for optimal performance (Endsley, 1990). Although intuitively connected, understanding the 

process state (i.e., possessing good situation awareness) and taking the appropriate control actions are not 

inexorably linked. Decision-making in many instances may also require balancing economic goals with 

prescriptive requirements for operation (i.e., procedures, technical specifications, etc.). Situation awareness 

is required as an input to decision making, but must also be coupled with other types of knowledge to 

achieve safe and effective performance. 

One may argue the need for situation awareness when the operator need only follow procedures. This may 

be especially so for emergency situations where extensive efforts have been directed to develop and ensure 

the adequacy of emergency operating procedures (EOPs) for all foreseeable circumstances (U.S. NRC 

1980, 1981, 1982a, 1982b). Recent studies have shown, however, that active situation assessment is often a 

part of using EOPs . It enables operators to detect faults ahead of procedurally-driven checks, detect 

important information, and identify situations or problems that are not addressed by procedures (Roth, 

Mumaw, & Lewis, 1994). Although procedures are in principle applicable to a wide range of foreseeable 

events, the execution of procedures, especially during infrequent or rare circumstances (i.e., accidents) may 

involve significant planning and situation assessment to achieve procedural goals. 

SA assessment methodologies have been applied in the aviation industry and serve as basic evaluation and 

major design criteria for the design of new pilot support systems (Selcon and Taylor, 1989). SA is 

measured to assess whether or not pilot support systems do, in fact, provide the pilot with an enhanced 

understanding of the state of the aircraft. SA is also relevant to the nuclear process control industry because 

of the need to evaluate new complex system designs in terms of how well they support operator SA (AIAA, 

1992). 

Changes in control room automation, evidenced by increased function allocation to machine systems or 

changes in the human-system interfaces used for information gathering and control have the potential to 

affect operator situation awareness. Some of these effects may be desirable. The way information is 

presented to operators may influence SA by increasing the amount of information operators can acquire 

visually. It may also minimize the potential for misunderstanding through improved integration of process 

measurements, and by presenting information in a manner more consistent with the operator's 

understanding of the plant and processes. 

Improved I&C systems in advanced plants may also support SA by integrating information from various 

sources or by minimizing the amount of information an operator must attend to in processing disturbance 

information (e.g., alarm filtering features). Improved information presentation and less workload (e.g., via 

function allocation), especially in challenging situations, should result in greater reserve performance 

capacity of the operating crew. This reserve capacity should afford operators both the time needed and 

information necessary to build and maintain situation awareness. Hence, one of the advantages of advanced 

control room design is expected to be enhanced operator SA (O'Hara, 1990). However, no studies have 

assessed this expectation. 

Increased automation may also create conflicts in maintaining SA. Greater function allocation to automated 

processes may place an additional mental burden on the operating crew to understand the functioning of 

automated systems. That is, their mental models also need to account for the activities of automation in 

order to correctly integrate information about the state of the plant and processes (Sarter & Woods, 1995). 

In other industries, a number of incidents have occurred in which misunderstanding or miscomprehension of 

automated systems have contributed directly to catastrophic outcomes (Rodgers, Mogford, and Mogford, 

1995; Sarter &Woods, 1995; Strauch, 1995). 

Increased automation may result in the operating crew being out of the control loop. As a result of design 

decisions to support operators by reducing workload, the operator may become side-lined. Operator 

performance has been shown to degrade in more highly-automated environments due to a diminished ability 

to detect system-level or automation errors and intervene to carry out the correct manual task (Carmody & 

11 

NIJREG/IA-0137
BACKGROUND

Gluckman, 1993; Wiener & Curry, 1980). This performance degradation may be due to a number of 

factors, including loss of operator vigilance, a loss of skills in performing tasks over time, or the failure of 

automated systems to provide relevant information to operators concerning automation failures (e.g., self

checking or diagnostic information). 

Thus, some factors influencing SA are likely to differ between conventional and advanced plants. Situation 

awareness may also be affected by changes in the staffing of control rooms, independent of the technologies 

employed for information presentation and control. Operators in a nuclear power plant control room work 

as a team. To some extent, crew members also work independently of each other, owing to a division of 

labor among crew members (e.g., reactor operator and balance of plant operator positions). As Endsley 

points out (Endsley, 1995) communication and sharing of SA-relevant information by individual members 

is crucial for the team to achieve SA, as well as for individual members to acquire responsibility-specific 

SA. Team interaction processes also serve as a vehicle by which the situation awareness of individual 

members is shaped (Salas et aL, 1995). 

Given the relationship between individual members, the team, and situation awareness, changes in the way 

crews are staffed may also produce changes in operator situation awareness. Reducing the number of crew 

members from a nominal staffing level, and the accompanying increased workload demands placed on the 

remaining crew members may affect operator SA. Additionally, team processes and workload factors may 

be affected by changes in control room staffing, and may potentially affect SA. 

As Endsley (1993) demonstrated in a series of studies, workload influences SA, though SA and workload 

differ in a number of ways. With low workload, operators may have low SA due to boredom, lack of 

attention, and vigilance problems. However, if an operator with low workload attends to information in the 

environment and if that information is presented in an easily-understood manner, the operator may have 

high SA. In high workload situations, the operator may have such low SA due to being so overburdened 

with tasks, the pace of incoming information, etc., that s/he is unable to formulate an accurate assessment of 

what is occurring in the plant. Differences in workload experienced by operators in the different staffing 

complements used in this study may also influence the situation awareness between the normal and 

minimum shift staffing complements. 

The Situation Awareness Control Room Inventory, SACRI, a technique developed by Hogg et aL, 1994, 

was used to assess situation awareness. The questions in SACRI were specifically chosen to be relevant to 

nuclear power plant control rooms, and ask about the trends of different process parameters. Each 

administration contained 18 questions, and are divided six ways. Questions are asked about the past, 

present and future state of the plant and are divided between primary (i.e., reactor coolant system) and 

secondary (i.e., balance of plant) side parameters. The questions are evenly distributed so three questions 

are asked in each group. 

The questions used in SACRI are in multiple-choice format. Depending on the type of question, either 

three or six possible answers are presented. If the SA question asks about a single parameter, such as the 

pressure in the pressurizer, three possible answers are presented: increase, decrease, or same. If the SA 

question asks about multiple parameters, such as the levels in the steam generators, six possible answers are 

presented: increase in one, increase in more than one, same, decrease in one, decrease in more than one, 

drift in both directions. The participants circle the answer they feel best characterizes the plant status for 

that parameter or group of parameters. 

To grade the inventory, the operators' answers are compared with the applicable plant parameters during a 

given time interval. Simulator data were collected in event and variable logs. Event logs recorded when 

operator and simulator instructor actions occurred (such as freezing the simulator for a data-collection 

period). Variable logs recorded process parameters every 15 seconds during the scenario. 

Using simulator data from the variable logs, a scoring copy of the SA test was generated. This provided a 

list of the correct answers for the inventory. SA inventories collected from operators during the scenarios 

were scored by comparing their responses to the correct answers. All answers were characterized as one of 

12 

NIJREG/IA-0137
BACKGROUND

four types: hit, correct acceptance, miss, or false alarm. A hit indicates that a change occurred in the 

process and the operator correctly recognized this change. A correct acceptance indicates that no change 

occurred in the process, and the operator correctly noted no change. A miss indicates that a change 

occurred in the process, but the operator said that no change occurred. A false alarm indicates that no 

change occurred in the process, but the operator said that a change had occurred. 

This classification scheme creates some ambiguity in grading, so more specific guidelines were developed. 

In the case where a change occurs, and the operator notes that a change occurs, but has noted the wrong 

direction (e.g. the process parameter increased and the operator indicated "decrease"), the answer was 

characterized as a miss. Appendix D provides a more detailed description of the scoring criteria. 

The Situation Awareness score is calculated based on the number of correct responses. The formula used to 

calculate SA is also presented in Appendix D. Two measures are calculated: operator sensitivity to the 

detection of process drifts and the response bias. Sensitivity and response bias scores were calculated for 

the primary side, the secondary side, and the entire process (i.e., both primary and secondary sides). For 

every questionnaire administered, these six scores were calculated. Analyses of data used only the overall 

sensitivity measure, which indicates how accurately the operator assessed the situation of the entire process 

(Hogg et al., 1994). 

2.3.1.4 Workload 

Workload is of particular importance in complex systems, especially in nuclear power plant control rooms, 

because of its effect on human error and performance. A certain amount of workload is considered optimal 

by both operators and designers. Some amount of workload is necessary to keep the operator engaged in 

overseeing the process and in the process control loop. If workload is much below this level, operators 

become under-stimulated and may become inattentive, lose vigilance, SA, and demonstrate degraded 

performance when the situation demands intervention. Also, overloaded operators generally suffer 

performance degradation as they are simply unable to match resources with the demands of the task. Some 

expert operators may be able to perform satisfactorily, at least temporarily, in an over-loaded condition but 

over time even their performance degrades. Between these two extremes is the optimal workload which 

provides sufficient challenge to keep operators stimulated without excessively taxing their capabilities 

(Huey and Wickens, 1993). 

As nuclear power plant I&C systems become more automated, vendors expect that operator workload will 

decrease. Intuitively, this expectation seems reasonable: operators have less to do as automated systems 

are allocated more functions. However, the opposite effect may occur. Operators may have higher mental 

workload to understand a system with which they are not actively involved. In addition, the changes in I&C 

systems may shift the load in the types of resources required to interact with the systems. Some activities 

utilizing motor and physical capabilities of operators in conventional nuclear power plants may shift to 

perceptual and information processing functions of the operator in advanced plants. Although overall 

workload in some respects may be decreased, some processes (e.g., cognitive) may be more taxed in 

advanced plants. 

The relationship between workload and performance is not linear or entirely predictable. As workload 

increases to a breaking point, operator performance may remain unchanged. Then suddenly, operator 

performance may deteriorate drastically (Bergstroem, 1993). In summarizing the determinants of workload 

and the consequences of workload on human performance, Huey and Wickens (1993) state: 

"In some cases, it is apparent that human limitations reflect the consequences of poorly 

designed controls, displays, and automatic subsystems. In others, task demands simply 

exceed the operator's capabilities either momentarily or for extended periods. Despite 

their limitations, humans are remarkably flexible, adaptable, and capable. They can 

improvise, compensate for inadequate information and system or human failures, adjust to 

13 

NUREG/IA-0137
BACKGROUND

novel situations, exhibit graceful (rather than catastrophic) degradation, plan ahead, predict 

the outcome of familiar and unfamiliar events, and learn from experience. However, the 

consequence of extreme demands and requirements to act creatively and adaptively impose 

significant workload on the human operators of complex systems." (National Research 

Council, p. 85) 

A further consideration about workload and performance is that changes in workload, especially sudden 

transitions from low to high workload, have been shown to produce the most pronounced decrements in 

performance (Kantowitz and Casper, 1988). This aspect of the relationship between workload and 

performance is especially significant for advanced NPP control room evaluation. In the advanced plants 

reviewed for this study, many routine operations of the plant during normal power operation will be 

automated. Sudden increases in workload when the crew is called upon to respond to an unanticipated 

situation may result in performance problems. A companion issue in the present study is whether the degree 

of workload produced by the staffing complements will produce marked differences in the workload 

experienced by the remaining members of the minimum control room crew complements. Members of the 

minimally-staffed crews which lack one or more licensed operators to assist during peak activity periods 

may experience an increase in workload. 

The NASA Task Load Index (TLX) is a subjective workload measurement technique that has been 

validated in several studies and widely applied (Wierwille and Eggemeier, 1993; Hill, Iavecchia, Byers, 

Bittner, Zaklad, and Christ, 1992; and Moroney, Biers, Eggemeier, and Mitchell, 1992). This technique 

offers several advantages over other subjective measurement techniques. It is particularly suited for use in 

complex systems (Weirwille and Eggemeier, 1993) and applied settings (Nygren, 1991). It has been shown 

to be a globally sensitive measure of workload (Moroney et aL, 1992). Weirwille and Eggemeier (1993) 

recommend that TLX should be given strong consideration in test and evaluation applications. 

Furthermore, Hill et al. (1992) found TLX to be one of the best subjective workload assessment techniques 

in terms of operator acceptance and validity. 

The NASA Task Load Index (TLX) subjective workload rating scale was used for the measurement of 

operator workload in this study. NASA TLX measures six components of workload: mental demand, 

physical demand, temporal demand, performance, effort, and frustration. Operators use a scale to rate 

workload for each component. The rating scale is from 0 to 100, with 0 being "no workload" and 100 being 

"highest possible workload." Participants were instructed to rate only the segment of the scenario prior to 

the interrupt. Participants were not given specific anchors, but were instructed to try to maintain 

consistency with their own use of the scale. Examples of the measurement scale and definitions provided 

participants are provided in Appendix C. 

To analyze subjective workload, ratings of the six workload components were averaged. All six rating 

scales were weighted identically. One of the sub-scales, performance, was inverted for purposes of 

averaging. This is because the other workload component scales are generally at 0 indicating "no 

workload" and 100 indicating "extremely high workload." So, for most scales, a lower score is more 

desirable. However, in the "performance" scale, a higher score is more desirable. A low score indicates 

poor perceived performance. Since the NASA TLX rating scale was designed and validated in this manner, 

it was administered in this format. 

2.3.1.5 Team Interaction 

Most vendors of advanced plants envisage control room operators working in teams, though they differ to 

some extent in how many team members they consider necessary to safely operate their plants. The 

reduction by one or two crew members in a shift staffing complement represents a significant reduction in 

overall control room resources. The ultimate reference criterion about the adequacy of a staffing decision is 

the performance of the crew. Crew performance involves not only objective performance of the necessary 

control room tasks (e.g., procedural tasks), but also how well the crew works as a team in coordinating 

14 

NUREG/IA-0137
BACKGROUND

activities, developing goals and objectives, and communicating together and with others outside the control 

room. 

Operators perform as members of a team in the control room of a nuclear power plant. Few studies have 

considered team interaction in complex, dynamic systems (Coury and Terranova, 1991). Those few studies 

that have looked at team interaction have shown teams to perform better than individuals in controlling a 

complex, dynamic system, and diagnosing failures. Teams control deviations better, require fewer control 

actions to achieve control, and are more efficient in information gathering than are individuals (Hooper, 

Coury, and Terranova, 1991; Montgomery and Hauth, 1991). 

However, team interaction may have certain drawbacks. In a team situation, individual members may 

become complacent, relying on other members to do certain tasks without necessarily verifying that the 

tasks have been performed. Teams may also develop overly cohesive or dependent thinking habits in which 

members are hesitant to introduce or entertain discrepant information for fear of creating conflict or 

disturbing group cohesion. Pooling the abilities of a number of highly-skilled individuals should result in a 

better product, but this effect does not always occur. Group processes can be slow, cumbersome, 

inefficient, and lead to individual member frustration. Such problems must be considered in determining 

the correct balance between crew tasks and system dependence on humans (Foushee and Helmreich, 1983; 

Cooper, White, and Lauber, 1979). 

Differences between conventional and advanced control rooms may influence the team interaction process. 

In conventional control rooms, for example, the work area typically occupies a physically larger area than in 

advanced control rooms. This larger area and the division of responsibilities between reactor and balance 

of plant operators may result in operators working at some distance from one another, although still within 

visual range. While the division of responsibilities may remain the same in advanced control rooms, the 

operators will not be working as far from one another due to the smaller control room areas in the designs 

reviewed for this study. Physically closer working areas may support team interaction. Alternatively, some 

advanced design features have been cited as potentially detracting from team performance. Intensive work 

with VDU-based systems may result in focusing or narrowing of an individual operator's attention on the 

VDU system, at the expense of information and activities outside this periphery. Such cognitive tunnel 

vision may be produced by some computer-based activities (e.g., via menu or system navigation) and result 

in reduced communication between operators and hence, decreased team interaction and performance. 

Team interaction is important in this study because of the potential for variations in task load, staffing, and 

control room design to influence the adequacy of the performance of the control room crew. Several 

measurement techniques have been developed to assess the quality of team interaction. These measurement 

techniques use ratings of communication and other aspects of teams. One technique, the Behaviorally 

Anchored Rating Scale (BARS) has been used to assess team interaction though it is not a test per se 

(Meister, 1985). Using a BARS measurement technique, dimensions of team interaction have been assessed 

previously (Baker and Salas, 1992; Montgomery and Hauth, 1992; Montgomery, Gaddy, and Toquam, 

1991). The actual crew interaction dimensions assessed vary in different studies, but in general they are 

team behaviors (e.g., communication, cooperation, task coordination, team spirit, maintaining task focus, 

and adaptability, etc.). 

Team interaction was measured in this study by using a Behaviorally Anchored Rating Scale (BARS) 

developed with experienced plant personnel. Five behavioral categories were used to assess team 

interaction: communication, openness, coordination as a crew, team spirit, and task focus and decision 

making. Specific behaviors for each of these dimensions were obtained from previous studies that used 

similar dimensions (Glickman et al., 1987; Morgan et al., 1986). Each dimension was rated on a scale of 1

7, with one being the lowest rating and 7 being the highest. On the rating forms, specific examples of 

positive and negative behaviors were provided to the raters for reference purposes. Raters had these 

examples with them as they observed crew performance. Participants were rated as crews, not as 

individuals. Nonetheless, some crew members might exhibit behavior that would lead to a high rating on a 

specific dimension, while others might exhibit behavior that would lead to a low rating on the same 

15 

NUREG/AA-0137
BACKGROUND

dimension. Such situations could produce an intermediate rating of the behavioral category for the crew. 

An example of a BARS rating scale and form used by raters in this study is provided in Appendix C. 

As with the workload data, the five team interaction scales were averaged to obtain an overall rating of team 

interaction. All BARS scales were designed so a higher score indicates more desirable team performance, 

so no inversions were necessary. Further, after checking the inter-rater reliability, the scores were averaged 

across raters. 

16 

NUREG/IA-0137
METHOD

3. METHOD 

3.1 Experimental Design 

Eight crews of operators participated in this study. Four crews participated in the normal staffing 

complement (4-person crews), and four in the minimum staffing complement (3-person crews in the 

conventional plant and 2-person crews in the advanced plant). Table 3.1 shows the experimental design of 

the study. 

Table 3.1 Experimental study design 

I Crew Size

Plant Type Normal Staffing Level Minimum Staffing Level 

Conventional 4-person 3-person 

Advanced 4-person 2-person

The normal staffing complement is based on current federal regulation (i.e., 10 CFR 50.54(m)) for staffing 

of a single operating unit at a single unit site, which is similar to the current staffing practice at the Loviisa 

plant. In the normal staffing complement for both plant types, the control room crew consisted of a control 

room supervisor/shift supervisor (CRS), a reactor operator (RO), balance of plant operator (BOP operator) 

and a control room technician (CT). The minimum staffing complement is based on the minimum crew size 

considered by an advanced passive plant vendor, or the minimum crew composition that would typically 

occur at Loviisa, which served as the conventional plant. In the conventional plant, the minimum crew 

complement of 3 included a CRS, RO, and BOP operator. In the advanced plant, the minimum crew 

complement of 2 included a CRS and either an RO or BOP operator, the CRS himself acting as a control 

room supervisor and an operator (either RO or BOP operator). 

Four crews participated in the conventional plant condition, and four participated in the advanced plant 

condition. The conventional plant was modeled at the Loviisa Nuclear Power Station Training Simulator. 

The advanced plant was modeled in the HAlden Man-Machine LABoratory (HAMMLAB), in Halden, 

Norway. The Loviisa plant main control room, upon which the training simulator is modeled, and the 

HAMMLAB simulator control room are shown in Figures 3.1 and 3.2. 

Each crew was presented with the same five scenarios over a period of two to three days. The scenarios 

were presented in random order to the crews. Scenario duration ranged from one hour to two and a half 

hours. All scenarios contained several faults and events that occurred regardless of operator actions, but 

scenario outcome could vary significantly depending on the crew's responses. Operator actions had the 

influence they would in the plant. The scenarios were nearly identical between both plant types, but 

modifications were made to simulate plant type-specific performance differences (i.e., passive system or 

increased automation). 

Scenarios were divided into segments based on plant state and critical crew actions. The crews participated 

in the scenario, and, when a segment was complete, the simulator was temporarily frozen. During this 

pause, situation awareness and subjective workload data were collected from the crew, and team interaction 

data were gathered from the raters. These pauses lasted approximately 5-7 minutes. Observers who 

provided ratings of team interaction sat in the simulator area behind and to the side of the shift crew in an 

17 

NUREG/IA-0137
METHOD

area that would not interfere with the crew's activities. These two raters (one a licensed SRO from Loviisa, 

the other a systems analyst with a background in plant operations, thermal hydraulics, and risk assessment) 

assessed team interaction. At the end of the scenario, these two raters, together with a third rater (also a 

licensed SRO in charge of performance-based training at Loviisa) provided evaluations of the crew's 

performance.

Figure 3.1 Side view of the Loviisa nuclear power station main control room 

Three of the scenarios employed in the study (i.e., the SGTR, ISLOCA, and LOFW scenarios) contained 

five data collection segments; the other two scenarios (i.e., the LOOP and SGOF scenarios) contained four 

segments (see Section 3.1.2 for a description of the scenarios). The first segment of each scenario contained 

a normal operating task, for example, load following. No disturbances occurred during this scenario 

segment. In the second segment, the simulated initiating event was introduced. After the crew noticed a 

malfunction in the plant and had begun taking steps to respond, the next interrupt occurred. During the 

ensuing segment(s), the crews proceeded with functional response strategies, event diagnosis (i.e., for event

based procedure selection), and began stabilization and cooldown of the plant. Novel disturbances were 

generally not introduced, unless a crew's action resulted in a fault (e.g., operating a plant system in such a 

way that eventual damage or failure of equipment would result). 

During each scenario, crews were required to make off-site notifications and to communicate with operators 

and other plant staff outside the main control room. The crews simulated performance of these notifications 

and communications by phoning or radioing to simulator instructors. The instructors made notes listing the 

agency or person to whom the call was made and the time the call was received. These data were later used 

in the evaluations of transient management. 

18 

NUREG/IA-0137

7777ýýL
METHOD

Figure 3.2 Side view of the HAMMLAB experimental control room

Performance measures were collected throughout the scenario. The measures collected during the first 

segment provided a baseline of operator performance data during normal operation. Operator performance 

was also assessed following the onset of the disturbance, during mitigation efforts, and during the resolution phase. Data collected across time periods in the scenario permitted comparisons of these types of design basis scenarios on baseline measures. Comparisons could also be made between crew staffing complements and plant types, and whether these differences increase as a function of scenario length, demands of specific scenarios, etc. The baselines also provided a measure for evaluating the comparability of crews, to ensure that any potential pre-study differences between crews did not affect the outcomes and inferences drawn 

from the study of performance data. 

3.1.1 Participants 

Eight crews of operators from the Loviisa nuclear power station in Loviisa, Finland, participated in the 

study. Crews were selected from actual operating crews at the plant. All participants were male and all 

were licensed plant operators, having the equivalent of an RO or SRO license. Control room/shift 

supervisors at Loviisa are degreed technical personnel (e.g., engineers). All operators spoke Finnish during 

the study. 

All crews included a reactor operator (RO) and a balance of plant operator (BOP operator). In the 2-man 

crew, one of the operators was also a licensed supervisor and performed supervisor functions. The 3-man 

crew included a dedicated Control Room Supervisor (CRS) apart from the RO and BOP operators. The 4-man crew had a CRS as well as an additional licensed operator, a control room technician (CT). The CT 

19 

NUREG/IA-0137
METHOD

performed necessary supporting control room tasks as directed by the CRS, and assisted in monitoring the 

status of critical plant functions. 

All crew activities and interactions were carried out in Finnish, the native language of the operators who 

participated in this study. Procedures that were used at both test sites, as well as all general instructions, 

and data collection inventories were written in or translated to Finnish. 

3.1.1.1 Training 

All crews were trained for participation in the study. Participants were informed about the overall purpose 

of the study and shown copies of the inventories used for data collection. They received verbal instruction 

on requirements for completing the SA and workload inventories, with emphasis on the need to work 

individually. Copies of the instructions to subjects are presented in Appendix B. 

Further, crews received simulator-specific training. In the conventional plant setting, this training required 

approximately 2 hours and addressed modifications of the Instrumentation and Control (I&C) systems and 

plant performance characteristics. In the advanced plant condition, the crews received two and a half days 

of training on use of the HAMMLAB simulator and the interface. The difference in the amount of 

simulator-specific training between the two conditions was due to the operators' differing familiarity with 

the two simulators. Operators were trained on the Loviisa simulator, and work on the actual Loviisa plant. 

Only a couple of hours of instruction and practice on minor modifications was needed. In HAMMLAB, the 

simulator models the Loviisa process, but with some differences. Also, the interface in HAMMLAB differs 

significantly from Loviisa simulator interface. Operators needed a much longer training period for the 

advanced plant condition, modeled in HAMMLAB. Following the training, the operators had a 15 minute 

break prior to participating in the scenarios and data collection. 

The raters of team interaction were trained on the use of these inventories. These raters had participated in 

the design of the team interaction assessment inventory and field-tested the inventory during a pilot study. 

A written copy of the verbal instructions the expert raters received is presented in Appendix B, and the 

inventory reference aid to which they referred during scenarios is presented in Appendix C. 

3.1.2 Scenarios 

The scenarios in which the crews participated were delivered on full-scale nuclear power plant simulators. 

The training simulator at the Loviisa Nuclear Power Plant served as the conventional plant; HAMMLAB 

served as the advanced plant. Data were collected during five design-basis scenarios. 

"* Steam Generator Tube Rupture (SGTR) with a fire in the turbine building and a stuck-open steam 

generator safety relief valve in the affected steam generator; 

"* Total Loss Of FeedWater (LOFW); 

"* Loss Of Off-site Power (LOOP) with a stuck open steam generator safety relief valve; 

"* Interfacing Systems Loss Of Coolant Accident (ISLOCA), and; 

"* Steam Generator Over-Fill (SGOF). 

The scenarios were chosen to cover a range of design basis conditions including overheating of the primary 

coolant system, overcooling of the primary coolant system, and loss of coolant from the primary system. 

These conditions correspond to those analyzed both in conventional plant safety analyses, and in the safety 

20 

NUREGAIA-0137
METHOD

analysis reports produced by the nuclear steam supply system (NSSS) vendors for the plant designs 

considered in this study. 

The first three scenarios listed above were considered to be rule-based. That is, the crews at Loviisa are 

well trained on the procedures for these scenarios and have experienced them during training. These three 

scenarios are not difficult to diagnose, each having a number of salient symptoms. 

The last two scenarios may be considered to be knowledge-based scenarios. The ISLOCA included a 

number of secondary instrument failures that were produced by the initiating event and that complicated the 

determination of the source of the leak. In the SGOF scenario, the failures of the feedwater flow control 

valves and ensuing steam generator level increase occurred in quick succession. The pace of these events 

may interfere with the crew's ability to identify and select the correct system procedure for responding to 

this scenario. 

Several features of the scenarios improved the realism and the degree of challenge. The scenarios required 

the crew to coordinate with external personnel; the scenarios were of long duration; and by their control 

actions, the operators could influence the outcome of the scenario. Crews made radio and/or phone calls to 

simulator instructors who acted as field operators, maintenance personnel, instrumentation technicians, 

radiation protection personnel, and firefighters. Further, crews made off-site emergency notifications in the 

scenarios. The scenarios generally required an hour to two and a half hours. This duration allowed the 

researchers to observe and measure crew performance issues over time. 

Appendix A provides more detail about the initial plant conditions, a brief synopsis of each scenario, the 

specific event sequence, expected or typical crew activities, and typical scenario pause times used for 

collection of SA, workload, and team interaction data. 

3.1.3 Simulator Performance Characteristics 

In specifying performance characteristics of the two simulators used for this research, the aim was to 

produce performance representative of the reference plants identified for the study. The Loviisa nuclear 

power plant model was used in the simulations both at Loviisa and in HAMMLAB. Since this nuclear 

power plant is a Russian light-water PWR design, the simulators were modified slightly to produce 

performance representative of both conventional and advanced plants. For the portion of the study 

conducted at Loviisa, U.S. conventional plant performance characteristics were identified from published 

information. For the portion of the study conducted in HAMMLAB, advanced passive plant performance 

characteristics were identified from advanced plants' safety analysis reports. 

For the conventional plant model, key event and time constants for specific thermal-hydraulic behavior of 

conventional plants were identified from Wheatley, et al. (1987). The reference study provides information 

on three design basis scenarios similar to, or having features in common with, three of the scenarios used in 

this study. Wheatley, et al. include modeling of plant response to a total loss of feedwater, steam generator 

overfill, and small-break loss of coolant accident for a Babcock & Wilcox, Combustion Engineering, and 

Westinghouse plant. Using key events or thermal hydraulic time constants from the reference study meant 

that for the overheating and overcooling scenarios, the steam generator models in the Loviisa Training 

Simulator (LOTS) were reduced to about 1/3 of their actual capacity. This ensured that dryout of the steam 

generators and heatup of the primary coolant system in the loss of feedwater scenario, for example, 

occurred at about the same rate and in the same range of time as were predicted to occur in the conventional 

plants in the reference study. 

For the advanced plant model, the safety analysis reports (SARs) provided to the NRC by the vendors were 

reviewed to find time constants, key events (e.g., passive system actuations) and a description of thermal

hydraulic performance for scenarios similar to those used in this study. Of the four safety analyses used, 

only two describe design basis or accident analyses to a level of detail that allowed extrapolation of key 

21 

NUREG/IA-0137
METHOD

events or thermal hydraulic time constants to the level of detail required. Of these two safety analyses, only 

one is a PWR. However, the accident analyses in the SAR for the particular scenarios include different 

assumptions about equipment availability and recovery than the designed scenarios. As a result, best 

estimates, based on the advanced plant SAR were used to determine the significant characteristics to be 

used in emulating advanced plant performance. This meant, for example, that dryout of the steam 

generators and heatup of the primary coolant system in the loss of feedwater scenario would take 

approximately twice as long to occur in HAMMLAB than at LOTS. In addition, some safety system 

actuations were programmed to occur automatically (e.g., HPSI) when certain conditions existed in the 

plant (e.g., inventory loss) to emulate passive system actuations. 

In addition to simulator thermal-hydraulic performance characteristics, the instrumentation and control 

(I&C) systems at Loviisa and in the Halden Man-Machine Laboratory (HAMMLAB) were reviewed and 

adjusted to make them representative of conventional and advanced plants, respectively. 

22 

NUREG/IA-0137
ANALYSIS

4. ANALYSIS 

Analyses of data were performed on performance measures to determine whether overall performance of 

crews and specific aspects of crew performance were affected by staffing levels (i.e., normal vs. minimum 

crew complements) and plant type (i.e., conventional vs. advanced plants). Analyses were also performed 

to determine whether crew performance varied as a function of scenarios or scenario types (i.e., rule-based, 

knowledge-based). The experimental design of the study comprised a between-within design, having both 

between groups factors (i.e., crew size, plant type) and within-subjects factors (i.e., repeated measurement 

of subjective performance measures, scenarios). 

The data analyses from this study were performed using the Statistica software package. Applying 

statistical tests to experimental data introduces the risk of two fundamental errors. A Type I error occurs 

when the researcher concludes that main experimental effects exist when actually none are present. In 

contrast, Type II errors occur when the researcher concludes that no experimental effects exist when such 

effects actually are present. In dynamic simulator studies such as these, the likelihood of committing a Type 

II error is larger than committing a Type I error. According to Wiener (1971), too much emphasis has been 

put on the significance level convention of 0.05 and 0.01: 

"When the power of tests is likely to be low under these levels of significance (0.05 

and 0.01), and when type I and type II errors are of approximately the same 

importance, then .30 and .20 levels of significance may be more appropriate than the 

.05 and .01 levels." (Wiener, 1971 page 14) 

In this study, it was as important to avoid erroneously concluding that the performance of different-sized 

crews was equal when, in fact, it was different, as it was to avoid concluding that it was different when it 

was, in fact, equal. Thus, to avoid type II errors, results approaching the 0.20 level of statistical 

significance were also taken into account and are discussed. 

The general approach to the analysis of data in this study was to first conduct multivariate analyses of 

variance (MANOVAs). Assuming that a significant difference was obtained in the MANOVA, univariate 

analyses of variance (ANOVAs) were conducted to estimate specific effects. Some of the specific study 

issues entailed a comparison of particular study groups. In such cases, following the univariate ANOVA, a 

post-hoc test was conducted. 

To evaluate study issues, repeated measures Multivariate Analyses of Variance (MANOVA) were 

performed on the transient management, situation awareness, subjective workload, and team interaction 

data. As mentioned earlier, data on each of these measures were obtained a number of times during each 

scenario; these are the repeated measures in this analysis. The independent variables in the analysis were: 

scenario, crew size (normal or minimum crew complement), plant type (conventional or advanced), and 

scenario period (i.e., period in the scenario in which measures were obtained). Because of the potential for 

a large number of interactions between the main effects, and to simplify the explanation of results, a Type 

III approach was used to estimate the effects of the independent variables in all analyses of variance (SAS 

1982; StatSoft, 1994). In the Type III approach, tests are conducted on specific effects, including 

interactions, after controlling for all other possible effects specified in the analysis. A significant effect 

represents a unique effect for a specific factor after controlling for all other effects or factors. Post hoc 

analyses of data were performed using the Tukey Test for Honestly Significant Differences for unequal 

sample sizes; the Spjotvoll and Stoline test (Spjotvoll & Stoline, 1973, StatSoft 1994, p. 1589). 

23 

NUREG/IA-0137
RESULTS

5. RESULTS 

5.1 Transient Management and Task Performance 

Crew performance was analyzed using objective criteria to evaluate their management of the transients and 

their performance of control room tasks. Analyses were performed on three performance categories: 1) 

announcements and notifications, 2) transient mitigation, and 3) cooldown and stabilization. Scenario

specific tasks were identified for each group of activities, and each crew received a rating from 1 (poor 

performance) to 5 (best performance) based on performance of these tasks. Ratings were combined and a 

score for each task category obtained for each crew. Ratings were subsequently analyzed. The independent 

variables used in the analysis were scenario, crew size, and plant type. A 3-way repeated measures (i.e., by 

scenario) Multivariate Analysis of Variance (MANOVA) was conducted, and showed significant effects of 

crew size and plant type. The results are summarized in Table 5.1. 

Table 5.1 Effects of scenario, crew size, and plant type on transient management and task 

performance 

Summary of Effects: 

Effect Rao's R/F df (effect) df (error) p-level (a) 

Scenario 0.84 (F) 4 35 not significant 

Crew size 20.65 (Rao's R) 3 2 < 0.05 

Plant Type 204.51 (Rao's R) 3 2 < 0.005 

Objective performance measures did not vary significantly as a function of scenarios. Crews performed the 

necessary tasks about as well in all scenarios. This indicates that the crews coped equally with the study 

scenarios, regardless of the differences. This is similar to findings from the analyses of workload, team 

interaction, and crew performance ratings 

5.1.1 Crew Size and Plant Type Effects 

Both crew size and plant type exhibited a significant effect on the crews' performance of their tasks. In 

contrast to many of the other findings, there were no significant interactions of crew size and plant type on 

these task performance measures. Further post hoc analyses of data were conducted on the individual 

measures of transient management and task performance for which significant results were found. 

Figures 5.1 and 5.2 show plots of transient management performance measures for the two different staffing 

complements and plant types, respectively. Both crew complements performed important mitigation tasks 

equally well. However, the crews in the normal crew complement conducted cooldown and plant 

stabilization tasks better than crews in the minimum crew size complement (p < 0.02). The normal sized 

crews also carried out announcement and notification tasks slightly better than the smaller crews, though the 

difference is not statistically significant. 

Crews in the advanced plant showed improvements in all aspects of mitigation activities, relative to 

performance in the conventional plant. Crews showed a slight improvement in identification and 

notification tasks, and larger improvements in mitigation tasks and cooldown and stabilization tasks relative 

to the performance of crews in the conventional plant. The differences between the performance of the 

crews in the two plant settings are significant for both important task performance (p < 0.01) and cooldown 

and stabilization tasks (p < 0.001). 

24 

NUREG/IA-0137
RESULTS 

3.4 

3.2 

3.1 .

3.2.9 ....... .........--------- _._.......__.......-------

a.O 

28 .............-

S2.7 . ....-.................................... . ... . ".. . ... Announcements&.Notifications . . . . .

-"-0- Important Task Performance 

2.6 ....-. -. - Cooldown & Stabilization I 

2.5 

Minimum Normal 

Crew Complement 

Figure 5.1 Plot of transient management performance measures by crew size 

4.5 

4 .0 ....................... ................... .. .. .. ........................ .................... ... . . . . . . .............. ... . ............................. .. . .. .......... ...... .......... ................. 

3 ............................ .5 '. ... 

350 ..... ...... .......... ...... ....... ... 4....... ..... 

23 . .......... ........ .... ...... ........ .. ........ ......................... .. .......... 

, , [ -O-Announcements & Notifications: 

i ./ / 2.0 

2.0 ... - ----.................... .-- -l- Important Task Performance 

- Cooldown & Stabilization 

1.5', 

Conventional Plant Advanced Plant 

Figure 5.2 Plot of transient management performance measures by plant type 

25 

NUREG/IA-0137
RESULTS

5.2 Rated Crew Performance 

Ratings of crew performance obtained from the trained observers following each scenario were analyzed. 

The independent variables used in the analysis are: scenario, crew size, and plant type. A 3-way Analysis 

of Variance (ANOVA) was conducted on the data, and the results are summarized in Table 5.2. 

Table 5.2 Effects of scenario, crew size, and plant type on rated crew performance 

Summary of Effects: 

Effect F df (effect) df (error) p-level (cx) 

Scenario 0.81 4 98 not significant 

Crew size 4.55 1 98 < 0.05 

Plant Type 23.26 1 98 < 0.001 

Interaction Effect 

Crew size X 

plant type 17.72 1 98 < 0.001 

5.2.1 Inter-Rater Reliability 

Three trained observers made ratings of crew performance using the behavioral scales adopted in this study. 

Following the collection of data, the inter-rater reliability or degree of correspondence among ratings made 

by the raters was assessed. A Pearson product moment correlation coefficient of 0.65 was obtained among 

the raters. 

5.2.2 Crew Size and Plant Type Effects 

The staffing levels, plant types, and their interaction exerted significant effects on crew performance. There 

were, however, no significant differences between ratings of crew performance for the different scenarios 

used in the study. 

Figure 5.3 is a plot of the ratings obtained on crew performance measures for the different crew 

complements in the conventional and advanced plants. Ratings on the measures of crew performance range 

from I (i.e., poor performance) to 10 (i.e., excellent performance). The analyses indicate that normal sized 

crews performed better than minimum sized crews, and that crews in the advanced plant performed better 

than in the conventional plant. However, the interaction between these two factors, as shown in Figure 5.3, 

indicates that the minimum crew complement in the conventional plant primarily influences these outcomes. 

The crews in the other conditions demonstrated better control of the plant, adopted consistently better 

solution paths in mitigating the disturbance, communicated better, and displayed more confidence in their 

performance and decisions than did minimum sized crews in the conventional plant (p < .001). 

The plot of rated crew performance indicates that the minimum sized crews achieved better performance in 

the advanced plant than in the conventional plant. Their performance was also slightly better than the 

normal sized crews in the advanced plant. Crews participating in the normal shift staffing complements 

maintained their performance levels on these performance criteria in the advanced plant. 

26 

NUREG/IA-0137
RESULTS 

8.5 

E750------ ........ . ... .............. ...... . . . 6.0 

.0 

0 .5 

"5.5 6.0 ------------------ - ... normal crew complement. ... 

-[--- minimum crew complement 

5.0 

Conventional Plant Advanced Plant 

Figure 5.3 Plot of rated crew performance by crew size and plant type 

27

NUREG/IA-0137
RESULTS

5.3 Situation Awareness, Subjective Workload, and Team Interaction 

The results of the analysis demonstrate significant effects of scenario, crew size, plant type, and scenario 

period on some or all of these performance measures. These are shown in Table 5.3. The conditions that 

are the focus of this study, crew size and plant type, appear to influence operator and crew performance. 

So, too, do the different scenarios developed for this study as well as the period within the scenario (e.g., 

stage of the transient in which the performance measures were obtained). The effects of plant type, crew 

size, scenario, and period are further modified by interactions with some or all of the performance 

measures. The significant crew size by plant type interaction indicates that the observed differences on 

some performance measure due to crew size were not constant across plant types. A specific crew 

complement that exhibited better performance on a measure in one plant setting may demonstrate either 

better or worse performance in the other plant setting. Further, this second order interaction (i.e., crew size 

by plant type) is modified by scenario period and the different scenarios themselves. The effects of crew 

size and plant type may therefore be different across scenarios and scenario periods. Each of the 

performance measures shows an interaction with scenario period, indicating significant deviations in the 

performance measures as a function of different stages of the transients. Means, standard deviations, and 

cell sizes for each of the performance measures analyzed are provided in Appendix F. 

Table 5.3 Results of repeated measures MANOVA on situation awareness, workload, and team 

interaction performance measures 

Summary of Effects: 

Rao's R df (effect) df (error) p-level (a) 

Effect 

Scenario 2.41 12 47 < 0.02 

Crew size 5.35 3 18 < 0.01 

Plant type 11.78 3 18 < 0.001 

Period 59.2 9 12 < 0.001 

Interaction 

Effects 

Crew size X 

plant type 9.64 3 18 < 0.001 

Scenario X 

period 6.27 36 46 < 0.001 

Crew size X 

period 3.32 9 12 < 0.05 

Plant type X 

period 4.90 9 12 < 0.01 

Scenario X 

plant type X 2.05 36 46 < 0.02 

period 

Crew size X 

plant type X 3.90 9 12 < 0.02 

period 

Scenario X 

crew size X 

plant type X 1.98 36 46 < 0.02 

period I I

28

NUREG/IA-0137
RESULTS

The shift staffing complements in the two plant settings differ on the dependent measures, though the results 

do not indicate how (i.e., which measures or which crew size complement had higher or lower scores). The 

significant interaction effects of scenario and scenario period indicate that performance measures do not 

differ across scenarios alone, but also across some portion(s) of the scenarios. This means that aggregating 

performance data across scenarios or scenario periods may not be warranted for some measures. In 

subsequent univariate analyses of individual performance measures, the effect of scenario is tested 

separately, prior to aggregating and analyzing data. 

5.4 Situation Awareness 

Previous analysis of situation awareness data (Hallbert et al., 1995) demonstrated significant differences 

between rule-based and knowledge-based scenarios. Prior to the analysis of main effects in this study, 

scenarios were therefore grouped into these two categories, which are referred to in the table of effects and 

ensuing discussion as "scenario type." The independent variables used in the analysis are: scenario type, 

crew size, plant type, crew member position (e.g., control room supervisor, reactor operator, balance of 

plant operator), and scenario period. A 5-way repeated measures Analysis of Variance (ANOVA) was 

conducted on the situation awareness data. Table 5.4 summarizes the results of this analysis. 

Table 5.4 Effects of scenario type, crew size, plant type, position, and scenario period on situation 

awareness 

Summary of Effects: 

Effect F df (effect) df (error) p-level (a) 

Scenario type 6.65 1 86 < 0.01 

Crew size 0.40 1 86 not significant 

Plant type 2.10 1 86 < 0.20 

Position 1.75 2 86 < 0.20 

Period 2.68 3 258 < 0.05 

Interaction Effect 

Scenario type X 

plant type 1.76 1 86 < 0.20 

Scenario type X 

period 5.35 3 258 < 0.005 

Crew size X 

plant type 25.23 1 86 < 0.001 

Crew size X 

period 2.0 3 258 < 0.15 

Scenario type X 

crew size X 2.05 1 86 < 0.20 

plant type 

Scenario type X 

plant type X 1.96 3 258 < 0.15 

period 

Position X 

plant type X 1.54 6 258 < 0.20 

period

29

NUREG/IA-0137
RESULTS 

5.4.1 Scenario Type and Period Effects 

The analyses show that the situation awareness of crews, in general, differed across scenario types, and 

scenario periods. Situation awareness follows a transition curve. This is shown in Figure 5.4. The situation 

awareness score ranges from a low of 0 (lacking situation awareness) to 1.0 (complete situation awareness). 

During normal conditions, crews possessed a relatively high level of situation awareness. Following 

disturbance onset, SA dropped markedly. The loss of SA from baseline conditions to its trough represents a 

loss of approximately 20% of the baseline SA. Following a sharp decline in SA, crew members gradually 

regained their SA. This occurred after they assessed the effects of the disturbance, began implementation of 

the necessary mitigation actions, and began to re-establish control of plant parameters. However, post

accident SA levels (i.e., period 5 measurements) were still lower than pre-accident SA levels. Figure 5.4 

also includes the three study scenarios that contained five data collection periods (i.e., the SGTR, ISLOCA, 

and LOFW scenarios). In these scenarios, crews experienced a drop in situation awareness from period 4 to 

period 5. This drop, however was not statistically significant.

1 2 3 

Scenario Period

4 5

Figure 5.4 Plot of average situation awareness by scenario period 

30 

NUREG/IA-0137

0.82 

0.80 

0.78 

0.76 

0.74 

0.72 

0.70 

0.68 

0.66

0 (U 

0 

U) 

r

_a

---------------------- -------- ....... 

--- ------- ...... ------- -- .... ..................... 

.......... -------- - -- -------- ............ 

---------- ......... -- ---- -------. ....... ... ...... ------- ---- ------ --- ------ ...... 

- - - --- --- ---- -- ----------

.. ........ . . . . . . . . . . . ...... ............... .... .... ..... .. 

............ ........ ...... .. . ............ .. .
RESULTS 

5.4.2 Scenario Effects 

Crews demonstrated better situation awareness on rule-based scenarios than knowledge-based scenarios. 

Table 5A also shows a significant interaction between scenario type and scenario period. Figure 5.5 shows 

a plot of SA for the two scenario types across scenario periods. The figure shows that crews experienced 

more variations in their situation awareness across scenario periods in rule-based scenarios than in 

knowledge-based scenarios. Post hoc analyses yielded significant differences between periods 1 and 2 (p < 

0.001), 1 and 3 (p < 0.001), and periods 3 and 4 (p < 0.05) within the rule based scenarios. No significant 

differences were found between scenario periods within knowledge-based scenarios. One significant difference was also identified in period 1 between the rule-based and knowledge-based scenarios. Overall, 

crews possessed lower SA in knowledge-based scenarios than in rule-based scenarios. However, the 

transient-induced SA loss that accompanied the rule-based scenarios left the crews with roughly the same 

situation awareness as knowledge-based scenarios during the most disturbed phases of the scenario (i.e., 

periods 2-4).

0

0.850 

0.825 

0.800 

0.775 

0.750 

0.725 

0.700 

0.675 

0.650 

0.625 

0.600

1 2 3 4 

Scenario Period 

Figure 5.5 Plot of situation awareness by scenario type and period 

31 

NUREG/IA-0137
RESULTS 

5.4.3 Crew Size, Plant Type, and Position Effects 

Figure 5.6 plots the situation awareness of crews in the different plant and staffing complements across 

scenario periods. Figure 5.7 plots the change in situation awareness from baseline of the crew complements 

in the two plant types across scenario periods. These two figures show a loss in SA experienced by all 

crews during the immediate post-disturbance period (i.e., period 2). Following the loss of SA, both the 

normal sized crews in the conventional plant and the minimum sized crews in the advanced plant tended to 

recover SA. The minimum crew complement in the conventional plant tended to lose SA over the scenario 

periods without recovery. Following an initial large drop, the normal crew complement in the advanced 

plant recovered SA. 

In addition, both plant type and position had significant effects on operator situation awareness. By itself, 

crew size did not exert a significant effect on crew situation awareness, though differences in the SA of 

different crew complements were noted across scenario periods. Interactions among all three of these 

factors had significant effects on the situation awareness of operators in this study.

2 3 

Conventional Plant

4 period 1 2 3 

Advanced Plant

Figure 5.6 Plot of situation awareness by crew size, plant type, and scenario period 

32 

NUREG/IA-0137

0.85 

0.80 --....-

0.75 

0.7 

0 

P 0.70 .... 

~0 ,0.65 

0.60 

0.55 

0.50 . 

periodI

• _x /, 

S...... '- ----..---- - - .. 

............ .............. ... ........... ........ 

i i \J61

4
RESULTS 

6 

4 -..... ---- - -----1 1---. ---- ......- .. . -..... .. ...... -.. 

2 * Normal crew complement 2 --- -- ~------- --.----- j-- 1 Minimum crew compleme I----- 0 ........ ................... -- .... . [ .: .• ............ : :;iM i m c : e w o p i m ::................ r .• .......... 

Z -2 ... ...------------.--.... . . . .. ----- 

-4~ . . .~~~~~ . .. .. .. . .; .. . . . .. ~~~ . ... ...... .... * -2 e.0 ------ - -- - - - - --- ---

-8 2 -- - - - -

--------

-1 0 ... .... ..... .. ..... . .. .. ------ . .... .. . 4 ...... -

-12~~~~ ------------- -------- ------ ---.-------- --------------- ------------

-14 .. . . . .. . . . .. . . . .. . .. .. -.-. -. . . . . . . . . . ..... -. .. .... .. .. { .. . .... .. .. . •.. ....---

-16 .______I___ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ period 2 period 3 period 4 period 2 peoa3 period 4 

Covnentlonal Plant Advanced Plant 

Figure 5.7 Plot of change in situation awareness from baseline 

5.4.4 Crew Size by Scenario Period Effects 

Significant differences were found in the situation awareness of all crews across scenario periods. Post hoc 

analyses revealed the scenario periods for which SA differed. In the normal sized crews, SA was 

significantly different between periods 3 and 1 (p < 0.20) and between periods 3 and 4 (p < 0.05). Between 

the two crew sizes, situation awareness was significantly different in period 1 (p < 0.05). T7hese results were 

probably due to the considerable variation in situation awareness demonstrated by the normal sized crews in 

the advanced plant, dominating the performance of normal sized crews in the conventional plant. This 

effect is shown in Figure 5.6. 

5.4.5 Position Effects 

A significant difference was found between the SA of individual control room crew positions. Post hoc 

analyses revealed a significant difference between the SA of the shift supervisor and* that of the reactor 

operator position (p < 0. 15). However, as this difference was not demonstrated across scenarios, scenario 

periods, crew complements, or plant types it should be interpreted with caution. 

33

NUREG/IA-0137
RESULTS 

5.4.6 Crew Size, Plant Type and Scenario Type Effects 

Although crew size did not exhibit a main effect, in combination with plant type it exhibited an effect on 

situation awareness. Figure 5.8 shows a plot of the crew size by plant type interaction. Post hoc analyses 

identified significant differences between the SA of crews in the normal crew complement in the 

conventional plant with both the minimum crew complement in the conventional plant (p < 0.005) and with 

the normal crew complement in the advanced plant (p < 0.001). In addition, the SA of crews in the 

minimum crew complement in the advanced plant were significantly different than those in the minimum 

crew complement in the conventional plant (p < 0.15) and in the normal crew complement in the advanced 

plant (p < 0.001). Normal sized crews in the conventional plant and minimum sized crews in the advanced 

plant demonstrated similar levels of situation awareness, and were significantly higher than the other crews.

Conventional Plant Advanced Plant

Figure 5.8 Plot of situation awareness by crew size and plant type 

34 

NUREG/IA-0137

0= 

== 

0=

0.80 

0.78 

0.76 

0.74 

0.72 

0.70 

0.68 

0.66 

0.64 

0.62

. ~ ..... ~ ...... ----- ...... ----- -------

0- normal crew complement o 

-- minimum crew complement
RESULTS 

Moreover, these trends in performance on SA extended to scenario type. Figure 5.9 shows the situation 

awareness of the different crew complements in both the conventional and advanced plant on rule- and 

knowledge-based scenarios. As can be seen in the plots, crews in the different crew complements and plant settings demonstrated lower situation awareness in knowledge-based scenarios than in rule-based scenarios. 

The exception to this is the minimum-sized crews in the advanced plant, which showed slightly better SA on 

knowledge-based scenarios. Post-hoc analyses showed that the normal sized crews in the conventional 

plant possessed higher situation awareness in rule-based scenarios than the normal sized crews in the 

advanced plant (p < 0.01). They also demonstrated higher SA on knowledge-based scenarios than both the 

minimum sized crews in the conventional plant (p < 0.15), and the normal sized crews in the advanced plant (p < 0.05). Crews in the minimum shift staffing complement in the advanced plant demonstrated 

significantly higher situation awareness on knowledge-based scenarios than normal sized crews in the 

advanced plant (p < 0.10) and minimum sized crews in the conventional plant (p < 0.20). No differences 

were found between the situation awareness of crews in the normal staffing complement in the conventional 

plant and the minimum crew complement in the advanced plant for either rule- or knowledge-based 

scenarios. 

A significant effect was found in the interaction between scenario type and plant type. Post hoc analyses showed that crews in the conventional plant demonstrated better SA on rule-based scenarios than crews in 

the advanced plant (p < 0.15). In addition, their own SA was significantly higher on rule-based than 

knowledge-based scenarios (p < 0.05). Table 5.4 also shows a significant interaction between scenario 

type, plant type, and scenario period. Post hoc analyses, however, did not identify differences in SA on rule 

based scenarios between crews in the conventional plant and crews in the advanced plant across scenario 

periods. Neither were any significant differences found in the SA of crews on knowledge-based scenarios 

in the conventional and advanced plants. The interaction appears, rather, to be based on the variation in SA 

between rule- and knowledge-based scenarios (see Figure 5.5).

Rule-based Knowledge-based 

Conventional Plant

Rule-based Knowledge-based 

Advanced Plant

Figure 5.9 Plot of situation awareness by crew size, plant type, and scenario type 

35 

NUREG/IA-0137

0.85 

0.80 

' 0.75 

0. 

= 0.70 

0 

' 0.65 

0.60

0.55
RESULTS

5.5 Workload 

Subjective workload measures obtained from individual crew members during the data collection intervals 

from all scenarios were analyzed. The independent variables used in the analysis were: scenario, crew size, 

plant type, crew position, and scenario period. A 5-way repeated measures Analysis of Variance (ANOVA) 

was conducted on the subjective workload data. Table 5.5 summarizes the results of these analyses. 

Table 5.5 Effects of scenario, crew size, plant type, position, and scenario period on operator 

workload

5.5.1 Scenario and Scenario Period Effects 

The results of the analysis show that the scenarios used in this study, in general, did not produce 

significantly different levels of workload on crews. All of the scenarios used in this study were design basis 

events, and appear to have created roughly equivalent amounts of workload. Some of the crew members 

experienced more demand in some scenarios than other crew members (i.e., the significant Scenario X 

Position interaction). This makes intuitive sense, since some scenarios resulted in challenges to the primary 

systems, while others were initiated in the balance of the plant. Thus, specific crew members should 

experience some difference in the amount of workload placed upon them by the demands of the scenario. 

For example, post hoc tests showed significant differences in the subjective workload experienced by the 

RO in the ISLOCA and SGOF scenarios. In the ISLOCA scenario the RO experienced more workload. 

This was likely due to the fact that the ISLOCA produced more problems in the primary circuit than the 

SGOF, which was a balance of plant-induced transient. Similarly, differences in workload were observed 

between crew members in a few cases. The ISLOCA scenario, for example, placed greater demand on the 

RO than the BOP operator. This was shown in these analyses. 

36 

NUREG/IA-0137

Summary of Effects:

Effect F df (effect) df (error) p-level (c) 

Scenario 1.45 4 50 not significant 

Crew size 5.15 1 50 < 0.05 

Plant type 39.24 1 50 < 0.001 

Position 4.52 2 50 < 0.02 

Period 220.7 3 150 < 0.001 

Interaction Effects 

Scenario X position 1.96 8 50 < 0.10 

Scenario X period 9.64 12 150 < 0.001 

Crew size X position 4.24 2 50 < 0.02 

Plant type Xperiod 10.59 3 150 < 0.001 

Plant type X position 4.09 2 50 < 0.05 

Crew size X 

position X period 2.17 6 150 < 0.05 

Crew size X 

plant type X period 2.86 3 150 < 0.05 

Position X 

plant type X period 1.95 6 150 < 0.10 

Crew size X 

position X 1.92 6 150 < 0.10 

plant type X period
RESULTS 

The effect of scenario period was found to be significant. Figure 5.10, a plot of the subjective workload for 

all crews in the study, shows the development of workload over scenario periods. This shows a marked 

effect of workload transition produced by the demands of the scenarios. The subjective workload scale has a 

range from 0 (total absence of workload) to 100 (extremely high workload). As can be seen, workload 

increased dramatically following the onset of the disturbance. The amount of subjective workload 

experienced by the operator following a major disturbance is approximately twice the amount experienced 

during normal operations. After reaching a peak during the middle of the scenario, workload subsided, but 

did not fall to pre-disturbance levels. 

In summary, the results indicate that the scenarios produced roughly the same amount of physical, mental, 

and temporal demand on the crews. As these were all design basis scenarios requiring significant crew 

efforts to manage their effects, this result is not surprising. Two additional findings related to position and 

scenario period effects were observed. These indicate that the technique used for measuring subjective 

workload was sensitive to differences in workload experienced by different crew members in some 

scenarios, and during different periods of the scenarios. Although no differences in subjective workload 

were observed between the different scenarios on the whole, workload was higher in some scenario periods 

than in others.

1 2 3 

Scenario Period

4 5

Figure 5.10 Plot of average subjective workload across scenario periods 

37 

NUREG/IA-0137

50 

45 

40

0 

S35 0 

30 

0 

25 

20

.... ... .......... ------ ....... ------ ----- ---------

............ .............. . . . . .. . . . . . . . . . . . . ... ...... .. . . . . . .. .. . . . . . . . .. .............. . . . . . . . . . . . . . . . . . . .................. . . . . . . . . .. . . . . . . . . . .. . .. . .. . . . . . . . . . . . . ..... ........... 

........ .... . . . . .. . . . ... ..... ...... . . . . . . . . .. . .. . . . 

.. ....... . .................... ..z ............ . . . . . . . . . . . . .. .. . . ............. ...... . . . . . ..................... 

---- ------- ------------------- .. ......
RESULTS

5.51 Crew Size, Plant Type, and Position Effects 

The analyses show that crew size, plant type, and position themselves and by their interaction (i.e., crew 

size by position and plant type by position interactions) significantly affected the workload of operators in 

this study. The interactions between these factors were also modified by scenario period. This is 

demonstrated by the significant interaction between crew size by position by period, plant type by position 

by period, and crew size by plant type by period. The four-way interaction between crew size, plant type, 

position, and scenario period was also significant. 

5.5.3 Crew Size and Plant Type Effects 

Crew members in the normal crew complement experienced less workload on average than crew members 

in the minimum staffing complement. Figure 5.11 shows the average subjective workload of the control 

room crew members in the two crew complements. The figure shows that the shift supervisor experienced 

significantly more workload in the minimum crew complement than in the normal crew complement (p < 

0.001). However, this relationship did not hold true between the conventional and advanced plants: Figure 

5.12 shows that shift supervisors in the two plants experienced similar demand. However, differences in 

workload for both RO and the BOP operators were observed between the conventional and advanced plants 

(p < 0.001 and p < 0.05, respectively). Both ROs and BOP operators experienced significantly more 

workload in the advanced plant than in the conventional plant. 

A significant interaction was also found between position, crew complement, and scenario. Differences 

were limited to the workload of shift supervisors in the conventional and advanced plants during the first 3 

periods of the scenarios (p < 0.001, p <0.001, and p <0.20, respectively). Both the ROs and BOP operators 

experienced similar amounts of workload across scenario periods in the normal and minimum staffing 

complements.

52 

50 

48 

46 

44 

42 

40 

38 

36

Supervisor Reactor Operator BOP Operator 

Control Room Crew Member 

Figure 5.11 Plot of average subjective workload by crew size and position

38

NUREG/IA-0137

0 

S 

0 

CL 

0
RESULTS

55 

50 

45 

40 

35 

30

Supervisor Reactor Operator BOP Operator 

Control Room Crew Member

Figure 5.12 Plot of subjective workload by plant type and position 

A particular issue in the study concerned the workload of the control room supervisor serving in the two

person crew. The workload of the dual-role CRS was compared with that of other crew members, and 

contrasted specifically with that of other CRSs in other staffing complements. When compared with other 

CRSs, ROs, and BOP operators, a significant difference was observed in the amount of workload these 

operators experienced (F 3,106 = 5.02, p < .005), a planned comparison showing that the dual role CRS 

experienced significantly more workload than all others. In contrast to all other control room supervisors, a 

planned comparison showed that the dual role CRS also experienced higher workload than other control 

room supervisors (F 1,106 = 6.22, p < .02). Figure 5.13 shows a plot of the workload of individual control 

room crew members.

YO BOP operator 

Control Room Crew Member

Dual Role CRS

Figure 5.13 Plot of subjective workload by control room crew member 

39 

NUREG/IA-0137

0 

0. 

0

5C

V 

01 

E 4z. 

0 

44C

34
RESULTS

5.5.4 Plant Type and Position Effects 

Workload of operators in the advanced plant was significantly higher across scenario periods. Post hoc 

analyses of the plant type by scenario period interactions revealed significant differences between crew 

members' workload across scenario periods: in each scenario period most crew members experienced more 

workload in the advanced plant than in the conventional plant. However, this relationship does not hold for 

all crew members. Figure 5.14 shows the subjective workload of the different control room crew members 

in both the conventional and advanced plants across the different scenario periods. Post hoc analyses 

showed that differences in subjective workload experienced by the shift supervisor were statistically 

significant only in period 4 of the scenario (p < 0.001). Significant differences exist in workload 

experienced by the BOP operator in periods 2 and 4 of the scenario (p < 0.05 and p < 0.001, respectively). 

The reactor operators in the study, however, experienced significantly higher workload in the advanced 

plant across all portions of the scenarios (i.e., from period 1 to 4: p < 0.05, p < 0.001, p < 0.001, and p < 

0.001). 

65 

60 ----- ... .. 

55 -- ----- --- .... 

50 

t4O .... . . ...... .. --

1 2 3 4 1 2 3 4 

Supervisor Reactor Operator

Figure 5.14 Plot of operator workload by plant type, crew member, and scenario period 

40 

NUREG/IA-0137
RESULTS 

5.5.5 Crew Size and Plant Type Effects 

Figure 5.15 shows a plot of the interaction between crew size, plant type, and scenario period on subjective 

workload. Post hoe analyses showed that crews in the normal complement in the conventional plant 

experienced significantly lower workload than both crew complements in the advanced plant across all 

scenario periods (p between < 0.01 and < 0.001). Minimum sized crews in the conventional plant 

experienced significantly less workload than minimum sized crews in the advanced plant following 

disturbance onset (i.e., scenario period 2) (p < 0.01). They also experienced less workload than both crew 

complements in the advanced plant during scenario periods 3 and 4 (p < 0.001). The only intra-plant type 

workload differences were found to be between the normal and minimum crew complements in the 

conventional plant, but only for scenario periods I and 2 (p < 0.01 and p < 0.10, respectively). No 

significant differences were detected between the normal and minimum sized crews in the advanced plant, 

though the crews serving in the minimum crew complement appear to have experienced slightly more 

workload. 

Figure 5.16 shows a plot of the change in workload experienced by the different crews across scenario 

periods, as compared to their baseline levels of workload. This figure shows that the crews in the advanced 

plant experienced higher workload transition than crews in the conventional plant (e.g., period 2 change 

from baseline). The results also show that the workload of crews in the advanced plant tended to increase 

almost linearly over the scenario. Workload of crews in the conventional plant tended to stabilize, and even 

decrease in the case of the normal crew complements.

2 3 4 Period 

Conventional Plant

j •z normal crew complement 

•ck minimum crew complement 

1 2 3 4 

Advanced Plant

Figure 5.15 Plot of operator workload by crew size, plant type and scenario period 

41 

NUREGAIA-0137

65 

60 

55 

50 

m45 

0 

040 

0 35 

0 30 

25 

20 

15

10 1. 

Period
RESULTS

perid 2 perod 3 peiod 4 period 2 period 3 period 4 

Conventional Plant Advanced Plant 

Figure 5.16 Plot of operator workload change from baseline

42

NUREG/IA-0137

40 

35 

30 

0 

025 

20 

10 

15 15
RESULTS

5.6 Team Interaction 

Team interaction measures were based on ratings obtained from the trained observers during the data 

collection intervals from all scenarios. The independent variables used in the analysis were: scenario, crew 

size, plant type, and scenario period. A 4-way repeated measures Analysis of Variance (ANOVA) was 

conducted on the team interaction data. Table 5.6. summarizes the results of this analysis. 

Table 5.6 Effects of scenario, crew size, plant type, and scenario period on team interaction 

Summary of Effects: 

Effect F df (effect) df (error) p-level (a) 

Scenario 1.41 4 60 not significant 

Crew size 1.13 1 76 not significant 

Plant type 18.49 1 76 < 0.001 

Period 3.70 3 228 < 0.02 

Interaction Effect 

Crew size X 

plant type 3.59 1 76 < 0.06 

Crew size X period 2.58 3 228 < 0.06 

Plant type X period 2.59 3 228 < 0.06 

Crew size X 

plant type X 2.54 3 228 < 0.06 

period 

5.6.1 Inter-Rater Reliability 

Two trained observers rated crew interactions. As previously discussed, the raters also had participated in 

the development of the rating scales and had examples of behavioral anchors for reference purposes. Each 

observer rated crew interactions during the same scenario periods. After collecting data from both phases 

of the study, the inter-rater reliability was assessed. A Pearson-product moment correlation coefficient was 

used to evaluate the reliability or degree of correspondence between the two raters. Analysis of data 

indicated a moderate degree of correspondence or inter-rater reliability (r=0.55). This is due to variation in 

ratings made on the individual crew interaction sub-scales. Due to this variability, no analyses of team 

interaction sub-scales were performed. This is because some of the variability that might influence results 

and inferences drawn could be due to either honest differences of opinions held by the raters, criterion shift, 

or both. 

43 

NUREG/IA-0137
RESULTS 

5.6.2 Scenario and Scenario Period Effects 

The results show that team interaction did not differ significantly across scenarios, but as with other 

performance measures, it differed significantly across scenario periods. This is shown in Figure 5.17. Team 

interaction was rated on a scale from 1 (poor team interaction) to 7 (optimum team interaction). The data 

indicate that during normal conditions, crews showed a moderate level of team interaction (i.e., a score 

between 4 and 5 on a scale from 1 to 7). Following the initiating event, crews mobilized, and team 

interaction increased. This is probably because during normal operating activities, tasks do not require as 

close a coupling of activities. During an abnormal event, especially during a design basis event, the 

challenges to the plant require much closer and more sustained coordination of activities. Following this 

initial mobilization, team interaction gradually declined. By the fourth data collection period, team 

interaction dropped to pre-disturbance levels. By the end of the scenario, team interaction decreased to 

slightly below pre-disturbance levels.

I 2 3 4 5

Scenario Period 

Figure 5.17 Plot of average team interaction across scenario periods 

44 

NUREG/IA-0137

6.0 

5.5 

5.0 

4.5 

4.0 

3.5 

3.0

C 

S 

0 C 

2: 

E

........................... ....... 

.............. ............. ---- ----- ......... . . ......... 

----------------------- ------------ --- -- -- -------

- ----- --- ------ -- - ----

.......... -- - --------------- - ------- ------ -- --- ------------------ - - -----
RESULTS

5.6.3 Crew Size and Plant Type Effects 

The results of the analysis show a main effect for plant type, but not for crew size. However, a number of 

interactions between crew size, plant type, and scenario period occurred that affected team interaction. 

Figure 5.18 shows the interaction of crew complement and plant type on team interaction. The figure shows 

an improvement in team interaction in the advanced plant as compared to the conventional plant. In 

addition, the most improvement in team interaction was achieved by the minimum crew complement. Post 

hoc analyses demonstrated significant differences between: 1) normal and minimum sized crews in the 

conventional plant (p < 0.20); 2) the normal sized crews in the conventional plant and the minimum sized 

crews in the advanced plant (p < 0.10); and, 3) the minimum shift staffing complement in the conventional 

plant and both the normal and minimum sized crews in the advanced plant (p < 0.005 for both). Crews 

tended to perform better as a team in the advanced plant setting; the more so for minimum sized crews. 

Whereas the smaller crew complements exhibited poorer team interaction in the conventional plant in 

comparison to the larger crews, in the advanced plant they performed at least as well as a team.

C a 

0 

S 

S 

C 

E 

U 

S I5.6 

5.4 

5.2 

5.0 

4.8 

4.6

4.4 

4.2 

4.0

Conventional Plant Advanced Plant

Figure 5.18 Plot of team interaction by crew size and plant type 

45 

NUREG/IA-0137
RESULTS 

The data indicate that the effects of crew size and plant type varied across scenario periods (i.e., the crew 

size by scenario period and plant type by scenario period interactions). Figure 5.19 is a plot of the interaction effect of plant type and scenario period on team interaction. This figure shows that crews in the advanced plant exhibited better team interaction than their counterparts in the conventional plant. Post hoc analyses indicated that this S,'ý,-.;,urred across each of the scenario periods (for periods 1-4: p< 0.001, 

p< 0.02, p< 0.001, and p< 0.001, respectively). During some scenario periods, crews in both plants demonstrate slightly better team interaction relative to other scenario periods, but these differences were not 

statistically significant.

5.8 

5.6 

5.4 

5.2 

5.0 

4.8 

4.6 

4.4 

4.2 

4.0

1 2 3 4 

Scenario Period 

Figure 5.19 Plot of team interaction by plant type and scenario period 

46 

NUREG/IA-0137

C 

0 

E
RESULTS 

Both crew size and plant type interacted with scenario period (i.e., the crew size by plant type by scenario 

period interaction). Figure 5.20 plots the team interaction for the different crew complements in the 

conventional and advanced plants across scenario periods. Figure 5.21 plots the change in team interaction 

of crews in the two plants from baseline conditions. The better team interaction ratings of the crews in the 

advanced plant can be seen in these figures. The crews in the conventional plant exhibited more variation 

in their team interaction over the course of the scenario, especially towards the latter portions of the 

scenario. Post hoc analyses confirm this. Analyses yielded significant differences in team interaction 

between scenario periods 2 and 3, and periods 2 and 4 for the minimum crew complement (p < 0.05, and p 

< 0.15, respectively). Significant differences were also found between periods 3 and 4 for the normal crew 

complement in the conventional plant (p < 0.10). Analyses also indicate a difference in team interaction 

between the normal and minimum sized crews in the conventional plant during scenario period 3 (p < 

0.001) and scenario period 4 (p < 0.10). No significant differences were observed in team interaction across 

scenario periods or between either crew complement in the advanced plant. 

Further analyses identified team interaction differences of crew complements during some scenario periods 

across plant types. Both crew complements in the advanced plant exhibited better team interaction during 

the first and third periods of the scenarios than the minimum crew complement in the conventional plant (p 

< 0.01 and p < 0.05, respectively). During the second scenario period the minimum crew complements in 

the advanced plant exhibited better team interaction than the minimum sized crews in the conventional plant 

(p < 0.05). The minimum sized crews in the advanced plant showed better team interaction than both crew 

complements in the conventional plant during the fourth period of the scenarios (p < 0.001).

2 3 

Conventional Plant

4

Sz normal crew complement 1 

minimum crew complement

1 2 3 

Advanced Plant

4

Figure 5.20 Plot of team interaction by crew size, plant type, and scenario period 

47 

NUREG/TA-0137

6.0 

5.5 

0 5.0 0 

0 

0 

.4.5 

4.0

3.5

1
period 3 period 4 period 2 period 3 

Conventional Plant Advanced Plant 

Figure 5.21 Plot of team interaction change from baseline

48

period 4

NUREG/IA-0137

RESULTS

8 

6 

4 

2 

0 

S-2 

C 

U 

-4 

-8 

-10

... .. ..... ... ........ ........... } .. . 

- -*-*------- - -~-*----------

S > . .. . . ..... . .... ...... . . . . . . . . . . t . . . . - - .

--- ------ ----- 0

period 2

----------------------- -- -- ---

S.................. ....... i .. .. ... ........... ...... 

0 

/ Normaicrew .omplement 

S... , Minimum crew compleren .........
DISCUSSION

6. DISCUSSION 

6.1 General Trends 

General trends in the performance measures of situation awareness, workload, and team interaction were 

observed and merit consideration. The findings demonstrate the effect of large plant disturbances on the 

performance of operating crews. Over the course of these scenarios, situation awareness varied 

significantly: following a disturbance, crews, on average experience a marked loss in situation awareness, 

i.e., approximately a 20 percent loss of baseline SA. Crews gradually regain SA though this recovery does 

not approach the level of SA found in pre-disturbance conditions. In some of the longer scenarios that 

included five data collection periods, SA dropped in the last period. Thus, the recovery of SA, both in 

terms of the amount and rate of recovery, may be affected by scenario-specific factors such as length, 

complexity and task demand. Although the differences in SA between the last and preceding period were 

not statistically significant, the trend by itself is noteworthy. Crews may be required to make critical 

decisions after one or more hours following the initiating event in a real situation, and this requires, among 

other things, adequate situation awareness. 

The technique used to measure situation awareness demonstrated sensitivity to differences in the type of 

scenario, (i.e., whether rule-based or knowledge-based). Crews possessed better SA in rule-based than 

knowledge-based scenarios on the whole. The results further show that transient-induced SA losses in rule

based scenarios may leave crews with approximately the same levels of SA during the most disturbed 

phases as in knowledge-based scenarios. This demonstrates, again, that the loss and recovery of SA is not 

invariant. Rather, it appears to be subject to many things, such as: a crew's previous experience with 

similar types of scenarios; their ability to identify procedural information to assist in stabilizing the plant; 

whether they direct their attention to relevant information; and the rate of change in plant parameters. 

On average, ratings of workload during transient mitigation were nearly double the ratings in baseline 

conditions. Most of the workload change increase occurs during the onset of the disturbance, and persists 

over the scenario. Even later in the scenario, after critical mitigation actions have been accomplished, 

crews experienced a high degree of workload as they brought the plant to a more stable, safer state. Thus, 

crews worked in a condition of sustained, high demand over the length of the scenarios in this study. 

Although each scenario was different, each produced similar workload and workload transition effects. 

This finding is not surprising since all of the scenarios used in this study were design basis events, requiring 

significant intervention to minimize challenges to the safety of the plant and, in some cases, the 

environment. 

Team interaction also demonstrates the effects of these challenging and lengthy scenarios. During the initial 

phase of these scenarios that involved routine tasks, crews demonstrated a moderate degree of team 

interaction. The coupling between the primary and secondary sides of the nuclear power plant require this 

coordination and communication between crew members to balance the generation and transfer of energy 

while carrying out plant evolutions. Following a disturbance, crews mobilize by increasing communication, 

coordination, and emphasis on task requirements (i.e., establishing goals and objectives, discussing options, 

and decisions). Following this mobilization, team interaction subsides somewhat over the course of the 

scenario. In most situations, this is probably because the individual crew members focus on specific 

activities related to their area of responsibility (e.g., primary circuit) and interact to communicate 

instructions, information, and status as needed. 

This trend merits consideration. While the demands of energy generation and transfer during normal 

activities require close coordination among crew members, there is likely to be more need during large plant 

upsets. Not all crews sustained team activities over the course of these scenarios. Perhaps this was due to 

task demands, design, or other factors. Nevertheless, team interaction is one of the ways that crews develop 

49 

NUREG/IA-0137
DISCUSSION

and maintain situation awareness, and is necessary to achieve effective performance in a nuclear power 

plant control room. 

6.2 Effects of Crew Size 

Although most other metrics did not demonstrate significant effects due to crew size alone, one aspect of 

transient management, cooldown and stabilization did. Control room crews are accustomed to working 

under different workload conditions, and learn to adapt their performance to match the demands of the 

operation being conducted. Under conditions of elevated workload, they, like skilled operators of other 

technological processes, may be able to perform for a period with no notable degradation in performance. 

The effect of workload may evidence itself not only early in a transient as workload transition, but also later 

as a cumulative effect. This may explain the difference between the different crew complements in carrying 

out cooldown and stabilization tasks, since no differences between the two crew complements were 

observed in their performance of other mitigation tasks. The minimum and normal crew complements both 

identified the correct procedural activities to mitigate the transients. However, the cooldown and plant 

stabilization activities were carried out under conditions of workload that were sustained typically for an 

hour or more. The absence of some crew member(s) may have affected the smaller crews' abilities to bring 

the plant to safe shutdown either in as timely a manner or within technical constraints (i.e., while 

maintaining boiling margins, pressure differentials between systems, etc.). 

Minimum sized crews experienced more workload over the course of scenarios than normal sized crews. 

Further analyses revealed that the control room supervisor experienced the bulk of the additional workload. 

Crew members are resources available to the control room supervisor. Their absence results in a significant 

increase in activities that he is either not able to allocate to others, or which require additional planning and 

interaction on his part to accomplish. This may interfere with the effective completion of the control room 

supervisor's tasks, coordination of resources, or simply place the control room supervisor in a sustained 

situation of higher workload. 

6.3 Effects of Plant Type 

In all aspects of transient management, crews in the advanced plant setting performed better than crews in 

the conventional plant, particularly so for transient mitigation and cooldown and stabilization tasks. These 

findings indicate that although crews in the advanced plant experienced significantly higher workload than 

crews in the conventional plant, it did not exceed a threshold beyond which performance degraded. 

All of the operators in the study experienced an increase in workload extending across scenario periods. By 

the fourth scenario period, all crew members in the advanced plant setting experienced significantly higher 

workload than their counterparts in the conventional plant setting. During debriefings, operators indicated 

that they had some difficulties in accomplishing their work using the different automated systems in 

HAMMLAB. Their relative unfamiliarity with the interfaces and automated systems in HAMMLAB, 

despite the training they received, may have affected their workload. This seems likely, since the largest 

demonstrated differences in workload were found among the RO and BOP operators between the two plant 

settings. These are the two crew members who interact most directly with the plant. Hence, some of the 

difference in workload observed between the conventional and advanced plant is probably related to the 

additional effort expended by operators to accomplish their tasks using different interfaces and automated 

systems. 

Approximately two to three days training was provided to crews coming to Halden to participate in the 

advanced plant phase of the study. During that time, each crew learned how to interact with the advanced 

I&C systems in the simulator. Baseline measures of workload showed that crews in both plant settings 

experience roughly equivalent levels of workload. However, this did not hold true during the transient 

50 

NUREG/IA-0137
DISCUSSION

phases of scenarios. Then, there was evidence of differences between HAMMLAB and their own plant 

where they have developed their skills and habits in operating a nuclear power plant. Crews experienced 

more workload in the advanced plant over all disturbance phases. Moreover, their experience of workload 

transition was greater and did not level off later in the scenarios as did the workload of crews in the 

conventional plant. A question remaining is whether these differences in workload are temporary and 

would, over time and trials, subside. This question is beyond the scope of the present study. 

A further concern was identified with the workload of the dual role control room supervisor/operator. In 

the advanced plant setting, the control room supervisor also served as one of the operators. The study 

design included conditions in which this dual role CRS served as the RO in one crew and the BOP operator 

in the other. Comparisons of his workload with that of other operators in the study showed that the dual 

role CRS experienced much higher workload. This includes CRSs in the conventional plant. During the 

debriefings, both of the licensed shift supervisors who served as dual role CRSs expressed concerns about 

working in this capacity. The shift supervisors expressed concerns about: being able to maintain an 

overview of the entire operation and its goals while being in the midst of the mitigation efforts; demands of 

communication with external personnel (e.g., local and national authorities, plant management and the 

emergency organization); and occasions in which the shift supervisor was left alone in the control room 

while the other licensed operator went to check something in the plant. 

One of the concerns associated with the finding of higher workload in the advanced plant is the effect this 

may have on crews' abilities to manage the demands of the disturbance. Crews in the advanced plant, 

however, demonstrated better performance than crews in the conventional plant. Two separate measures of 

task performance were used in this study. One was a subjective evaluation based on criteria suggested by 

licensed operator examiners and used previously in other studies. The other was a method that divided 

transient management into distinct activities, each having tasks and goals that were more objectively 

evaluated. On measures of rated crew performance, the minimum sized crews in the advanced plant 

demonstrated the best performance. These crews also experienced the greatest workload levels in the study. 

Crews in the advanced plant setting exhibited better team interaction than crews in the conventional plant. 

These differences extended across all scenario periods, during both normal and disturbance phases of 

scenarios. The averaged plot of team interaction of all crews across scenario periods (Figure 5.17) shows a 

drop in team performance following mobilization. However, the plot of team interaction across scenario 

periods in both plant settings (Figure 5.19) shows that this must be due to lower ratings of crews' team 

performance in the conventional plant. Crews in the advanced plant, on average, demonstrated more stable 

team interaction. They also showed no marked drop in performance later in the scenarios. This effect was 

modified by crew size and will be discussed further in the discussion of interactions. 

One explanation for the better and more consistent team interaction of crews in the advanced plant relates to 

design differences between the conventional and advanced plant settings used in this study. Many 

conventional plants have relatively large control rooms in which operators carry out activities. The division 

of labor in NPPs typically dictates that the RO and BOP operators work on different, though 

complementary activities (i.e., different tasks, same goals). Depending on the design of the control room, 

this could place the operators at some distance from one another. The advanced plant complement used in 

this study embodied a wrap-around design in which each operator sat close to the other around a central 

overview and SPDS. Each operator could see what the other did and what displays were used. The control 

room supervisor in the advanced plant setting often left his workstation to sit between and a little behind the 

RO and BOP operators. He stayed very close to them for much, if not all, of the scenario. The crews in the 

advanced plant also used the common overview display as a focus of their discussions, both during the 

initial phases of the disturbance and during recovery. This specific type of interaction did not occur in the 

conventional plant due to the absence of such a display. 

All of the crews in this study experienced situations of intense workload transition. During this transition, 

crews also experienced a loss of situation awareness. This can mean that some aspects of decision making 

are likely to be impaired. Symptom-based emergency operating procedures represent an advance in the 

prescriptive guidance provided to control room crews since they require much less determination of causes 

51 

NUREG/IA-0137
DISCUSSION

to be applied. However, operators actively engage in re-establishing and maintaining their SA following 

disturbances, and SA is itself an important element in using EOPs. Operators in the advanced plant, on 

average, possessed higher SA than crews in the conventional plant, though this was shown to depend on 

crew size. 

6.4 Interactions of Crew Size and Plant Type and Effects on Operator Performance 

Some of the measures used to assess the effects of crew performance in normal and minimum staffing 

complements in the two plant types revealed significant interactions (i.e., situation awareness, team 

interaction, rated crew performance). In these cases this means that the effect of crew size and plant type 

cannot be considered separately. The differences in performances on these measures were due, rather, to a 

combination of the two factors. 

Rated crew performance on mitigation tasks nearly mirror the interaction of team interaction, discussed 

later. In the conventional plant, the normal sized crews demonstrated better performance on task-oriented 

performance measures than the minimum sized crews. The minimum sized crews in the conventional plant 

received the lowest ratings on these performance measures of all the crews. In the advanced plant setting, 

the normal sized crews demonstrated performance on these measures at levels equivalent to normal sized 

crews in the conventional plant. The minimum sized crews in the advanced plant, though, showed 

significantly better selection of mitigation paths, control of the plant, communication and confidence in their 

decision-making compared to minimum sized crews in the conventional plant. In fact, the smaller sized 

crews in the advanced plant performed slightly better than the normal sized crews in both plant settings. 

This difference, though, was not statistically significant. 

Crews in the advanced plant setting achieved higher situation awareness on average than crews in the 

conventional plant. However, the difference between the situation awareness of the minimum sized crews 

in the conventional and advanced plant settings contributed significantly to this result (see Figure 5.8). In 

the conventional plant, the minimum sized crews demonstrated lower SA than normal sized crews. In the 

advanced plant, the minimum sized crews possessed significantly higher SA than the normal sized crews. 

Hence, there were marked differences in the SA of the crew complements between the conventional and 

advanced plants. The normal sized crews in the conventional plant and the minimum sized crews in the 

advanced plant demonstrated roughly equivalent levels of SA (i.e., the differences between them were not 

statistically significant). Their SA was higher than that of the other crews. 

All crews experienced a drop in SA following transient initiation. However, crews demonstrated different 

amounts of SA loss and rates of SA recovery. The normal sized crews in the conventional plant and the 

minimum sized crews in the advanced plant both experienced a drop in their SA following the onset of and 

during the peak of the disturbances. Over the course of the scenario, however, both groups regained their 

SA. Figures 5.6 and 5.7 show the change in SA of the crews from their pre-disturbance SA. 

In contrast, the minimum sized crews in the conventional plant lost SA over the course of the scenario 

without recovery. These crews' SA, in fact, showed a down-turn at the last scenario period. The normal 

sized crews in the advanced plant, on the other hand, recovered the SA they lost during the most disturbed 

portions of the scenarios. They also possessed slightly better SA at the end of the scenarios than at the 

beginning. However, this group also experienced the largest loss of SA of all the crews. At it's lowest, 

their SA approached a level near which their assessments of plant performance (e.g., detections of change 

and direction, predictions into the future, etc.) were only slightly better than chance. 

The differences in the SA of the normal sized crews between the conventional and advanced plant settings 

was unexpected. HAMMLAB was designed for a somewhat smaller shift complement, though CRTs and 

other operator support systems were provided for each crew member in this study commensurate with the 

plans suggested by advanced plant vendors. However, the normal sized crews in the conventional plant, 

and the minimum sized crews in the advanced plant possessed higher SA than the other groups. Hence, 

52 

NUREG/IA-0137
DISCUSSION

there may be some design aspects in each of the control rooms that better support different crew staffing 

complements. 

These results extend to performance on rule- and knowledge-based scenarios. Most crew complements 

demonstrated lower SA on knowledge-based scenarios than rule-based scenarios. The exceptions to this are 

the minimum sized crews in the advanced plant who demonstrated equivalent levels of SA in both types of 

scenarios. 

Maintaining situation awareness is an integral part of operator performance, during both simulated and 

actual emergencies. Situation awareness is needed by crew members to evaluate progress in procedurally

directed activities, to evaluate different alternatives in the various planning phases that occur during 

transient mitigation, to establish appropriate performance goals, and to use as input to decision making. 

Situation awareness of the control room crew may also influence the quality of information provided to 

external agencies. During an emergency, crews must direct not only on-site activities towards mitigating the 

disturbance but also communicate and provide information to local and federal authorities. These agencies 

use the information provided by the plant as part of the basis for making decisions about important 

collateral activities. This includes the activation of emergency contingencies, public evacuation, dose rate 

or source term prediction (in the case of the most serious accidents). Some of this information may come 

directly to agencies via data links with the plant where such links exist. Some information will almost 

certainly come from the verbal reports by control room crew members. In some phases of a design basis 

event, reported information may be subject to large uncertainties depending on whether it taps into SA

related information or not. The findings of this study show that control room crew staffing and plant design 

features may contribute to such uncertainties. 

The largest differences in the ratings of team interaction were found between the minimum sized crews in 

the conventional and advanced plant settings. In the conventional plant, the minimum sized crews 

demonstrated fewer team interaction characteristics than the normal sized crews. In the advanced plant, 

though, both the normal sized crews and the minimum sized crews exhibited equivalent team interaction 

characteristics. 

The trends in the team interaction over scenario periods are also relevant. Crews in the conventional plant 

demonstrated more variability in their team interaction than crews in the advanced plant. The minimum 

sized crews in the conventional plant demonstrated more variability in team interaction during the latter 

portions of the scenarios, and exhibited poorer team performance than other crews in the conventional plant. 

The differences in team interaction compared to baseline further underscore these results. Figure 5.21, a 

plot of the change in team interaction from baseline, shows that both the normal and minimum crews in the 

conventional plant mobilized to about the same extent. Following this mobilization, the minimum sized 

crews showed a significant drop in team interaction. They also showed little improvement at the end of the 

scenario. The normal sized crews demonstrated a slight increase in team interaction following mobilization, 

but did not maintain team performance throughout the scenario. This was shown by a drop in team 

interaction during the last phase of the disturbance. In contrast, both crew complements in the advanced 

plant exhibited similar team interaction. Their team interaction was better than in the conventional plant, 

and exhibited less variability across periods of the scenarios. On average, crews in the advanced plant 

maintained team interaction at about the same levels during the scenarios 

An objective of control room design is to support the performance of individual operators as a team. 

Design features should be pursued that lead to improvements in crew coordination, and in the availability 

and presentation of information needed to support crew task performance and decision making. An aim of 

such design improvements should be to achieve levels of team interaction that result in acceptable 

performance. Such performance characteristics can be considered in relative terms (i.e., compared to 

another operating environment) as well as in terms of the consistency crews demonstrate in their 

performance. The findings of the study show that performance as a team is affected by both crew size and 

control room design features. Crews interacted better as a team in the advanced plant setting, especially the 

smaller sized crews who demonstrated poorer performance in the conventional plant setting. Variability 

between the different crew complements was reduced in the advanced plant setting. The interaction of 

53 

NUREG/IA-0137
DISCUSSION

effects, thus, appears to have more to do with equalizing the performance between the different crew 

complements (i.e., minimizing differences between them). Moreover, the levels of performance on these 

measures were, in the advanced plant, higher than in the conventional plant. In this respect, the interactions 

indicate that some design objectives were better achieved in the advanced plant, evidenced by the 

interaction and performance of the crews. 

54 

NUREG/IA-0137
INSIGHTS

7. INSIGHTS 

This study involved the analysis of crew performance in normal and minimum crew staffing complements, 

carried out in simulator settings that were made to be representative of both conventional and advanced 

passive plants. Modifications were made to the plant model and part of the instrumentation and control 

systems at the Loviisa simulator prior to data collection to provide a control room environment more similar 

to U.S. plants. In HAMMLAB, the instrumentation, control, and automated systems were configured to 

produce a work environment similar to those that operators in advanced passive plants are expected to 

encounter. Results obtained on all of the performance measures used in this study revealed a number of 

effects of crew size, plant type, and their interaction. 

In considering the effects of crew size and plant type on performance measures, several issues concerning 

the validity of the study itself should be mentioned. Simulator studies are sometimes criticized because they 

take place outside the real control room, involve scenarios that crews are highly trained on, don't include 

off-site notifications and communications with outside control room staff, or performance of activities 

outside the control room. Simulator studies, however, represent the best approximation of the actual control 

room environment, and are the environment in which crew performance is tested as part of licensing and re

qualification. The scenarios used in this study represented a broad range of design basis events, involving 

scenarios for which crews were well trained, and some for which they had less training. In addition, all 

necessary notification and communication tasks with external personnel, whether on-site or off-site, were 

included in the scope of these scenarios to maintain a high degree of realism in the scenarios. Feedback 

from crew members and training personnel experienced with observing crews supports the conclusion that 

the scenarios contained a high degree of realism and challenge. 

Considering analyses of the task performance, situation awareness, subjective workload, and team 

interaction measures, these scenarios were deemed to challenge the resources of the crews to mobilize and 

mitigate the disturbances. Situation awareness, subjective workload, and team interaction all demonstrated 

the effects of transitions from normal to abnormal operating conditions. The effects were further associated 

with the crew size and plant settings used in this study. Simple effects of these factors on operator, crew, or 

plant performance were observed in some cases. In many cases, though, significant effects were observed 

by the combination of these factors. 

The findings of this study, thus, demonstrate significant effects of control room crew staffing and plant 

technology on crew performance. Some of these are in accord with the expectations of previous research 

into human performance with advanced technologies (i.e., better situation awareness). Some were not (i.e., 

lower SA of normal sized crews in advanced plant relative to that of same-sized crews in the conventional 

plant). Though improvements occurred in many aspects of performance based on differences between 

conventional and advanced plant features, the improvements did not extend equally to the different crew 

complements. 

The findings underscore the benefit of using a broad range of performance measures to evaluate issues that 

involve human performance, such as control room staffing and design issues. Were one to consider the 

differences in workload alone between the two plant types and staffing complements, then perhaps the 

current minimum staffing complement in a conventional plant might have appeared adequate based on 

statistically significant differences alone. However, differences in team interaction, situation awareness, 

and task performance in the advanced plant underscore the importance of using both subjective and 

objective performance measures in evaluating potential changes that affect aspects of the work environment 

and that may, therefore, affect the performance of the crew. 

The designs of new nuclear steam supply systems that incorporate passive system performance, together 

with new control room technologies offer the possibility to automate much of what is done manually today. 

55 

NUREG/IA-0137
INSIGHTS

However to the extent that this is done, consideration must be given to the best methods of involving 

nuclear power plant operators in the control of the process. The reasons for this are threefold, at least. 

Firstly, the operator serves as both a first and last line of defense in case of automation failure, and may, 

perhaps, sometimes be an initiator of it. Maintaining an awareness of the process is the most important part 

of the operator's role, and is the basis for many important control room decisions and actions. Maintaining 

an awareness of the plant may become more difficult if the operator becomes removed from the process 

control loop through automation of operator activities. 

Secondly, inappropriate function allocation may produce a set of activities that may be difficult or error

prone. This may be the result of allocating to machines those functions that are easiest in terms of 

engineering approaches, and leaving those that are not easy to allocate to the control room crew. Not all of 

the vendor submittals that served as the basis of this study (which, admittedly are at different stages of 

completion) included the decision criteria used for allocation of control room functions. Hence, the final set 

of control room activities for which operators in advanced passive plants will be responsible is still 

uncertain. 

Thirdly, even though many functions may be under automated system control, intervention by the operating 

crew must be anticipated. As discussed earlier, operators in advanced passive plants will carry out many 

activities similar to those in plants today, though the timing and methods for carrying them out may change 

considerably. The ability to intervene with automated systems is necessary if the crew is to serve as a 

backup to automation, and to initiate automation sequences (e.g., change from one mode of operation to 

another). To supervise automated systems, the crew must understand the goals of automation (e.g., 

programmed sequences) and how it functions (e.g., receive the appropriate feedback from the system). 

Else, operator intervention may be ill-timed or erroneous, based on incorrect assumptions, and knowledge 

of goals and means. 

At the same time that changes in the design and operation of nuclear power plants are under consideration, 

some vendors also anticipate that a smaller control room complement may be sufficient to control the 

nuclear power production process. Some of the design factors that have led to proposed reductions in 

staffing relate to changes in allocation of functions (i.e., activities usually performed by operators allocated 

to machine systems) or introduction of passive systems. To date, none of the vendor submittals cite 

empirical studies or demonstrations of their systems that show the ability of crews to operate their systems 

as envisioned by the vendors. The findings of this study demonstrate that changes in the control room crew 

complement must take into account the balance of activities allocated to the control room crew for 

foreseeable circumstances. The findings also underscore the importance of adopting appropriate measures 

of operator performance. Such measures are needed to ensure that the demands imposed by the system are 

manageable by the control room crew and do not have a negative impact on important aspects of crew and 

plant performance. 

In summary, design factors influenced the performance of crews in this study. Crews in the conventional 

plant normal staffing complement exhibited better performance than minimum sized crews. Such plants 

have been designed with a larger control room staffing complement in mind. The functions, allocations, 

size, and automation in such control rooms require more crew members to maintain control of the plant. In 

the advanced plant, the minimum sized crews performed better than the normal sized crews. The advanced 

plant control room setting was more compact, designed for a smaller control room complement. It made 

greater use of computer-based technologies and automation than the conventional plant, including passive 

design features and displays that better integrate control room information. Therefore, decisions about 

control room staffing should be based upon design features including function allocation, automation, 

integration, and plant-specific characteristics (e.g., passive system performance). Validation and 

verification using measures of operator and crew performance are necessary to determine the staffing 

complement needed to operate the plant. 

56 

NUIREG/IA-0137
REFERENCES

8. REFERENCES 

"Policy Statement on Engineering Expertise on Shift," Federal Register (50 FR 53621), October 21, 1985. 

10 CFR 50.54(m), "Energy: Conditions of Licensees," Code of Federal Regulations, Washington, D.C., 

January 1, 1992. 

ABB Atom Preliminary Safety Information Document (PSID). PIUS: The PIUS Arrangement Principle 

for a Simplified Passive Advanced Light Water Reactor Plant., December, 1989. 

Adams, M.J., Tenney, Y.J., & Pew, R.W. "Situation awareness and the cognitive management of complex 

systems." Human Factors, 37 (1), pp. 85-104, 1995. 

American Institute of Aeronautics and Astronautics. Guide to Human Performance Measurement. 

American National Standards Institute. BSRIAIAA G-035-1992. 

Atomic Energy of Canada Limited, CANDU Operations. The CANDU 3 Technical Description. Technical 

Report 74-01371-TED-001, Rev. 3, September 15, 1989. 

Atomic Energy of Canada Limited, CANDU Operations. The CANDU 3 Technical Outline. Technical 

Report 74-01010-TED-001, Rev. 9, September 15, 1989. 

Baker, D.P. & Salas, E. Principles for measuring teamwork skills. Human Factors. 34(4). pp. 469-475, 

1992. 

Bergstroem, B. Task demand, workload, and performance. National Defense Research Establishment 

Department for Human Studies. FOA report A 50016-5.2. ISSN 0281-0239, 1993. 

Carmody, M.A., & Gluckman, J.P. "Task-specific effects of automation and automation failure on 

performance, workload and situational awareness." In R.S. Jensen and D. Neumeister (Eds.), 

Proceedings of the Seventh International Symposium on Aviation Psychology (pp. 167-17 1). 

Columbus, OH: Ohio State University, 1993. 

Casali, J.G. & Wierwille, W.W. "A comparison of rating scale, secondary-task, physiological, and primary

task workload estimation techniques in a simulated flight task emphasizing communications load." 

Human Factors. 25(6). pp. 623-64, 1993. 

Cooper, G.E., White, M.D. & Lauber, J.K. (Eds.). Resource Management on the flight deck. NASA 

Conference Publication 2120, Moffet Field, CA: NASA-Ames Research Center, 1979. 

Coury, B. G., & Terranova, M. "Collaborative Decision Making in Dynamic Systems." In Proceedings of 

the Human Factors Society 35th Annual Meeting, September 2-6, San Francisco, CA, 1991. 

Endsley, M.R. "Toward a theory of situation awareness in dynamic systems." Human Factors, 37(1), pp. 

32-64, 1995. 

Endsley, M.R. "Design and evaluation for situation awareness enhancement." Proceedings of the Human 

Factors Society 32nd Annual Meeting. pp. 97-101, 1988-A. 

Endsley, M.R. "Situation awareness global assessment technique (SAGAT)." Proceedings of the IEEE 

National Aerospace and Electronics Conference, NAECON. Dayton, Ohio. pp. 789-795, 1988-B. 

Endsley, M.R. "Pilot situation awareness: the challenge for the training community." Proceedings of the 

11th Interservice llndustry Training Systems Conference. November, 1989. pp. 111-117, 1989. 

57 

NUREG/IA-0137
REFERENCES

Endsley, M.R. "Predictive utility of an objective measure of situation awareness." Proceedings of the 

Human Factors Society 34th Annual Meeting. pp.4 1-

4 5 , 1990. 

Endsley, M.R. "A survey of situation awareness requirements in air-to-air combat fighters." The 

International Journal of Aviation Psychology. 3(2). pp. 157-168, 1993-A. 

Endsley, M.R. Towards a Theory of Situation Awareness in Dynamic Systems. TTU-IE-93-02. Lubbock, 

TX: Texas Tech University, Department of Industrial Engineering (internal technical report), 1993-B. 

Endsley, M.R. Measurement of Situation Awareness in Dynamic Systems. TTU-IE-94-04. Lubbock, TX: 

Texas Tech University, Department of Industrial Engineering (internal technical report), 1994. 

Endsley, M.R. "Predictive Utility of an Objective Measure of Situation Awareness." In Proceedings of the 

Human Factors Society 34th Annual Meeting, pp. 41-45, 1992. 

Endsley, M.R. "The application of human factors to the development of expert systems for advanced 

cockpits." Proceedings of the Human Factors Society, 31st Annual Meeting. pp. 1388-1392, 1987. 

Endsley, M.R. & Bolstad, C.A. "Human capabilities and limitations in situation awareness. Individual 

differences in pilot situation awareness." International Journal of Aviation Psychology. pp. 19-1 - 19

10, 1992. 

Ericsson, K.A. & Simon, H.A. Protocol Analysis: Verbal Reports as Data. London, England: The MIT 

Press, 1984. 

Fontenelle, G. & Laughery, K.R. "A workload assessment aid for human engineering design." 

Proceedings of the Human Factors Society - 32nd Annual Meeting, 1988. 

Foushee, H.C. "Dyads and triads at 35,000 feet: factors affecting group process and aircrew performance." 

American Psychologist. 39(8). pp. 885-893, 1984. 

Foushee, H.C., & Helmreich, R.L. Group Interaction and Flight Crew Performance. in Human Factors in 

Aviation, E. Wiener & D. Nagel (eds.), New York, NY: Academic Press, 1988. 

Fracker, M.L. "A theory of situation assessment: Implications for situation awareness." Proceedings of 

the Human Factors Society 32nd Annual Meeting. pp.102-106, 1988. 

Fracker, M.L. "Attention gradients in situation awareness." NATO AGARD Aerospace Medical Panel 

Symposium. pp. 6-1 - 6-10, 1989. 

GE Nuclear Energy. SBWR Standard Safety Analysis Report. Report 25A5113 Rev. A, August 1992. 

Glickman, A.S.; Zimmer, S.; Montero, R.C.; Guerette, P.J.; Campbell, W.J.; Morgan, B.B., Jr.; Salas, E. 

The Evolution of Teamwork Skills: An Empirical Assessment with Implications for Training. Technical 

Report 87-016. Prepared for the Office of Naval Research and Human Factors Division (Naval Training 

Systems Center). Orlando, FL, 1987. 

Hahler, B.; Dahl, S.; Laughery, R.; Lockett, J.; & Thein, B. "Crewcut - A tool for modeling the effects of 

high workload on human performance." Proceedings of the Human Factors Society 35th Annual 

Meeting, pp. 1210-1214, 1991. 

Hallbert, B.P., Sebok, A.L., Haugset, K., Morisseau, D.S., & Persensky, J.L "Interim Results of the Study 

of Control Room Crew Staffing for Advanced Passive Reactor Plants." In Proceedings of the 23rd 

58 

NUREG/IA-0137
REFERENCES

Water Reactor Safety Meeting, U.S. Nuclear Regulatory Commission, October 23-25, 1995, Bethesda, 

Maryland. 

Hanson, D.J., Meyer, O.R., Blackman, H.S., Nelson, W.R., & Hallbert, B.P. Evaluation of Operation 

Safety at Babcock and Wilcox Plants Volume I - Results Overview. NUREG/CR-4966, Volume 1, U.S. 

Nuclear Regulatory Commission, Washington, D.C., October 1987. 

Hart, S.G. & Hauser, J.R. (1987). "Inflight application of three pilot workload measurement techniques." 

Aviation, Space and Environmental Medicine, pp. 402-410 May, 1987. 

Hendy, K.C.; Hamilton, K.M.; & Landry, L.N. "Measuring subjective workload: When is one scale better 

than many?" Human Factors. 35(4). pp. 579-60, 1993. 

Hill, S.G.; Iavecchia, H.P.; Byers, J.C.; Bittner, A.C.; Zaklad, A.L.; Christ, R.E. "Comparison of four 

subjective workload rating scales." Human Factors. 34(4). pp. 429-439, 1992. 

Hogg, D.N.; Folles0, K.; Torralba, B.; & Volden, F.S. "Development of a situation awareness measure to 

evaluate advanced alarm systems in nuclear power plant control rooms." Ergonomics. 38(11), pp. 

2394-2413, 1995. 

Hooper, K. N., Coury, B.G., & Terranova, M. "Team Performance in Dynamic Systems." Proceedings of 

the Human Factors Society 3 5Ih Annual Meeting, September 2-6, San Francisco, CA, 1991. 

Huey, B.M. & Wickens, C.D. (Eds.). Workload Transition: Implications for Individual and Team 

Performance. Washington, DC: National Academy Press, 1993. 

Idaszak, J.R. "Human operators in automated systems: The impact of active participation and 

communication." Proceedings of the Human Factors Society 33rd Annual Meeting. pp. 778-782, 

1989. 

James, L.R.; Demaree, R.G.; & Wolf, G. "Estimating within-group interrater reliability with and without 

response bias." Journal of Applied Psychology. 69(1). pp. 85-98, 1984. 

Kantowitz, B.H. & Casper, P.A. "Human workload in aviation." In Human Factors in Aviation (Ed. 

Weiner, E. &Nagel, D.) San Diego, CA: Academic Press. pp.157-187, 1988. 

Kibbe, M.P. "Information transfer from intelligent EW displays." Proceedings of the Human Factors 

Society 32nd Annual Meeting. pp.107-110, 1988. 

Komaki, J.L.; Desselles, M.L.; & Bowman, E.D. "Definitely not a breeze: Extending an operant model of 

effective supervision to teams." Journal of Applied Psychology. Vol. 74, No. 3, pp. 522-529, 1989. 

Meister, D. Behavioral Analysis and Measurement Methods. New York: John Wiley and Sons, Inc, 1985. 

Melber, B.; Roussel, A.; Baker, K.; Durbin, N.; Hunt, P.; & Hauth, J. Staffing decision processes and 

issues: Case studies of seven US Nuclear Power Plants. NUREG/CR-6122. Washington, DC: 

Nuclear Regulatory Commission, 1994. 

Montgomery, J.; Gaddy, C.; & Toquam, J. "Team interaction skills evaluation criteria for nuclear power 

plan control room operators." Proceedings of the Human Factors Society 35th Annual Meeting. pp. 

918-922, 1991. 

Montgomery, J.C., & Hauth, J.T. Team Interaction Skills Evaluation Criteria for Nuclear Power Plant 

Control Room Operators. NUREG/CP-0 114, Vol. 1, 1991. 

59 

NUREGAIA-0137
REFERENCES

Montgomery, J.C.; Hauth, J.T. "Team interaction skills evaluation criteria for nuclear power plant control 

room operators." Proceedings of the U.S. Nuclear Regulator Commission Eighteenth Water Reactor 

Safety Information Meeting. U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory 

Research, 1992. 

Moray, N. "Subjective mental workload." Human Factors. 24(1). pp. 25-40, 1982. 

Morgan, B.B., Jr.; Glickman, A.S.; Woodard, E.A.; Salas, E.; Blaiwes, A. Measurement of Team 

Behaviors in a Navy Training Environment. Technical Report 86-014. Prepared for the Office of 

Naval Research and Human Factors Division (Naval Training Systems Center). Orlando, FL, 1986. 

Moroney, W.F., Biers, D.W., Eggemeier, F.T, & Mitchell, F.A. "A Comparison of Two Scoring 

Procedures with the NASA Task Load Index in a Simulated Flight Task." IEEE Proceedings, pp. 734

740, 1992. 

Mosier, K.L. & Chidester, T.R. "Situation Assessment and Situation Awareness in a Team Setting." In Y. 

Queinnec & F. Daniellou (Eds.), Designing for Everyone. London: Taylor and Francis. pp. 798-800, 

1991. 

Murphy, K.R.; Martin, C. & Garcia, M. "Do Behavioral Observation Scales Measure Observation?" 

Journal of Applied Psychology. Vol. 67, No. 5, 562-567, 1982. 

NUREG-0660. NRC action plan developed as a result of the TMI-2 accident, Volume 1. Washington, 

D.C.: U.S. Nuclear Regulatory Commission, 1980. 

NUREG-0737. Requirements for emergency response capability, supplement 1 (Generic Letter 82-33). 

Washington, D.C.: U.S. Nuclear Regulatory Commission, 1982. 

NUREG-0799. Draft criteria for preparation of emergency operating procedures. Washington, D.C.: U.S. 

Nuclear Regulatory Commission, 1981. 

NUREG-0899. Guidelines for the preparation of emergency operating procedures. Washington, D.C.: 

U.S. Nuclear Regulatory Commission, 1982. 

Nygren, T.E. "Psychometric properties of subjective workload measurement techniques: implications for 

their use in the assessment of perceived mental workload." Human Factors, 33(1), pp. 17-31, 1991. 

O'Hara, J.M. "Advanced human-system interface design review guidelines." Proceedings of the U.S. 

Nuclear Regulatory Commission Eighteenth Water Reactor Safety Information Meeting. Prepared by 

Brookhaven National Laboratory. Holiday Inn Crowne Plaza, Rockville, MD. October 22-24, 1990. 

Olsen, L.M. "Operator modeling of a loss-of-pumping accident using MicroSAINT." Proceedings of the 

ANS Annual Meeting, Boston, 1992. 

Parasuraman, R.; Molloy, R. & Singh, I.L. "Performance Consequences of automation-induced 

"complacency." International Journal of Aviation Psychology, 3(1), 1-23, 1993. 

Regal, D.M.; Rogers, W.H.; Boucek, G.P., Jr. "Situational awareness in the commercial flight deck: 

definition, measurement, and enhancement." SAE Technical Paper Series: Aerospace Technological 

Conference and Exposition. Anaheim Convention Center, California October 3-6, 1988. pp. 1-5, 

1988. 

60 

NLTREG/IA-0137
REFERENCES

Roth, E.M., Mumaw, R.J., & Lewis, P.M. An empirical investigation of operator performance in 

cognitively demanding simulated emergencies. NUREG/CR-6208, U.S. Nuclear Regulatory 

Commission, Washington, D.C., July 1994. 

Sanders, M.S. & McCormick, E.J. Human Factors in Engineering and Design, Sixth Edition. London: 

McGraw-Hill Book Company, 1987. 

Sarter, N.B., & Woods, D.D. "How in the world did we ever get into that mode? Mode error and 

awareness in supervisory control." Human Factors, 37(11), pp. 5-19, 1995. 

SAS Institute, Inc. SAS User's Guide: Statistics. SAS Institute, Inc., Box 8000, Cary, North Carolina, 

1982. 

Salas, E., Prince, C., Baker, D.P., & Shrestha, L. "Situation awareness in team performance: Implications 

for measurement and training." Human Factors, 37(1), pp. 123-136, 1995. 

Sebok, A.L., Hallbert, B.P. & Morisseau, D.S. "Crew Performance Issues in Nuclear Power Plant Process 

Disturbances." Proceedings of the 1996 American Nuclear Society International Topical Meeting on 

Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies (NPIC & HMIT). 

Pennsylvania State University, State College, Pennsylvania. pp. 645-651, 1996. 

Selcon, S.J. & Taylor, R.M. "Evaluation of the situational awareness rating technique (SART) as a tool for 

aircrew systems design." NATO AGARD Aerospace Medical Panel Symposium. pp. 5-1 - 5-8, 1989. 

Selcon, S.J.; Taylor, R.M.; & Koritsas, E. "Workload or situational awareness?: TLX vs. SART for 

aerospace systems design evaluation." Proceedings of the Human Factors Society 35th Annual 

Meeting. pp. 62-66, 1991. 

Spjotvoll, E., & Stoline, M.R. "An Extension of the T-Method of Multiple Comparison to Include the 

Cases with Unequal Sample Sizes." Journal of the American Statistical Association, 68, pp. 976-978, 

1973. 

Spurgin, A.J. & Beveridge, R.L. "A model of a control-room crew." Power Plant Dynamics, Control and 

Testing. pp. 69.01-69.11, 1986. 

Tsang, P.S. & Johnson, W.W. "Cognitive demands in automation." Aviation, Space, and Environmental 

Medicine. February, 1989. pp. 130-135, 1989. 

Vidulich, M.A. & Hughes, E.R. "Testing a subjective metric of situation awareness." Proceedings of the 

Human Factors Society 35th Annual Meeting. pp. 1307-1311, 1991. 

Westinghouse Electric Corporation. AP600 Standard Safety Analysis Report, Rev. 0, Simplified Passive 

Advanced Light Water Reactor Plant Program Report DE-AC03-90SF18495, Prepared for U.S. 

Department of Energy, San Francisco Operations Office, June 16, 1992. 

Wheatley, P.D.; Davis, C.B.; Callow, R.A.; Fletcher, C.D.; Dobbe, C.A.; & Beelman, R.J. (1987). 

Evaluation of Operational Safety at Babcock and Wilcox Plants, Volume 2 - Thermal-Hydraulic 

Results. NUREG/CR-4966, EGG-2515, U.S. Nuclear Regulatory Commission, Washington, D:C:, 

November, 1987. 

Wierwille, W.W. & Eggemeier, F.T. "Recommendations for mental workload measurement in a test and 

evaluation environment." Human Factors. 35(2). pp. 263-281, 1993. 

61 

NUREG/IA-0137
REFERENCES 

Wiener, E.L., & Curry, R.E. "Flight deck automation: Promises and problems." Ergonomics, 23, pp. 995

1101, 1980. 

Winer, B.J. Statistical Principles in Experimental Design (2nd Edition). McGraw-Hill Book Company: 

New York, 1971. 

62 

NUREGAIA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A-1

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.1 Scenario 1: Steam Generator Tube Rupture With A Fire In The Turbine Hall 

And A Stuck Open Steam Generator Safety Relief Valve. 

A.1.1 Synopsis 

The scenario begins after a shift turnover. A control rod has inadvertently dropped earlier. An electrical 

fault has been repaired and main control room (MCR) staff will start returning the plant to full power. 

Turbine oil separation had been started before the shift change, so the level of the oil tank is rising slightly. 

The only symptom of the leak in the oil tank, apparent to operators in the MCR after separation is complete, 

is that the oil level of the main oil tank is not rising. After pumping is stopped, a decreasing oil level 

indicates the leak. A Field Operator (FO) will inform the MCR of the fire. Very soon after getting 

information about the leak, operators will probably trip the turbine. 

The fire in the turbine hall will be initiated simultaneously with the turbine trip (due to electric spark and oil 

fumes). The oil leak will continue until the turbine is tripped. Thereafter the scenario continued as a small 

turbine hall fire. In the event of a turbine hall fire, the reactor should be scrammed according to procedure. 

Just after the scram a SGTR (5 kg/s) will occur increasing to 35 kg/s ten minutes later. All indicators work 

normally, and the crew should have no difficulty to diagnose this transient. 

Subsequent to the steam generator leakage, a steam generator safety relief valve will fail open, after cycling 

open and closed a few times. This results in an event that leads to an unfiltered release of primary circuit 

radiation to the atmosphere. 

A-2 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.1.2 Probable course of events 

Note: Entries preceded by an * symbol indicate planned malfunctions implemented by the simulator 

instructor. 

Time (min.) 3

4 Description 

Initial condition (IC): 

Smaller steam generators; secondary volume is 1/3 nominal size (10 nP 

each). Measurement fault in SC1OL003 gradient, its parameter +3 

mmn/nin. 

Power level is about 90 % due to control rod drop. MCR staff start 

withdrawal of the rod and increasing the output to full power over 1 1/2 

hours. 

(11: 00 p.m. simulated time of day) 

0 min. Low level alarm of the main oil tank < 0 mm. It is between 10-50 mm 

below normal level. An alarm has been active for some hours but this is 

normal during oil separation. 

1 Oil separation is going on (during IC), and normally increases the level 

of oil tank, +3 mm/min. This task is conducted by the FO. When oil tank 

is full, pumping will be regenerated back to main oil tank from leaking 

oil tank at a rate of +50 I/min. over a 15 minute period. 

5 *A small control oil pipe (return oil line) under turbine 1 breaks. 

Leaking oil drops on a steam line of super pre-heater and a portion of it 

will vaporize. 

Oil leak initiation, SCIOL03 = 0 mnmi/min. Level will not rise because of 

the leak, nor will it decrease. The leakage is masked by simultaneous oil 

pumping from the oil separator which pumps purified oil back to the 

main oil tank compensating for the level decrease 

Leak onto the floor is at a rate of between 40-50 /mrin. 

Low level alarm does not disappear despite oil pumping back to tank. 

MCR staff may detect this and suspect something strange. 

7 Separation is over and the leak will now start decreasing the level of the 

oil tank at a rate of -3 mra/min. 

8 Pause for data collection 

3 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., 

emulation of passive system performance. 

4 Times for data collection are approximate and, in some cases, depend on a crew activity 

A-3 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

11 Alarm in MCR: "SC 10L803 (gradient) -3 > -1 mm/min." 

12 BOP operator probably asks FO: "Is there something unusual in oil 

separation?", and will get an answer: "Everything is OK." 

14 After the first low level alarm of the oil tank BOP operator will ask the 

field operator to check the turbine hall for leakages. 

16 FO finds the leak and reports the situation like: 'There is a rather big 

leak under turbine 1." It is very difficult for FO to determine which of 

several control oil pipes is leaking. 

17 CRS gives oil fire fighting alarm to the fire brigade. 

18 CRS orders trip of the Number 1 turbine. 

18 Turbine I tripped. 

19 Oil fumes will catch fire in the basement. A small oil fire starts in the 

turbine hall on level +3 - 0 m. 

20 alarm *Automatic fire alarm. 

20 Oil leak is terminated by turbine trip 

20 FO informs BOP operator by radio telephone about the fire. 

21 CRS performs routine procedures for activating fire brigade. 

23 CRS probably orders RO to trip the reactor according to the EOP, "Fire 

in turbine hall". 

23 Turbine 2 trips automatically due to reactor trip. 

24 CRS should now use symptom based "General Emergency Operating 

Procedure" (GEOP) to make post trip checks. 

24 *SGTR, parameter 0.54 cm2 activation in SG2 (YB52WO0). 

Just after the scram SGTR (5 kg/s) will occur. 

24 RL72S03 closed (automatic on high level). 

25 SC1OL803 (gradient) LG <-1 mm/min. alarm off. 

25 Extra makeup pump (2 of 3)started. 

26 Extra makeup pump (3 of 3) started. 

30 Fire is over. MCR is informed by fire brigade. 

30 Pause for data collection 

34 *SGTR leak increases to 4 cm2

, 35 kg/s. 

A-4 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS 

34 alarm Radioactivity in evacuation pumps of condensers. 

34 alarm RA52R001, high radiation 0.637 > 0.01 mSv/h 

35 HPSI on 

36 The staff will probably transition to event based EOP, "Steam Generator 

Tube Rupture." 

36 Stop of RCP, YD12D001. 

36 *An attempt to close the main primary gate valves is unsuccessful due to 

YA12SO01 torque failure, stuck open 70%. 

36 Automatic closure of SG steam and feedwater isolation valves in 

affected SG (activated on high SG level). 

37 Manual start of high volume boron injection. 

38 Steam dumping to the condenser on both turbines; by-pass 2x26 % open 

due to reduced SG volumes in conventional plant. 

38 Staff should start primary circuit depressurization and cooldown by 

PZR-spray from make-up-system to minimize the integrated break flow 

from primary to secondary circuit, and to environment. 

38 Primary pressure decrease by opening PORV (1.3 kg/s). 

40 *After automatic isolation of the affected steam line when the pressure 

rises and the SG safety valve had opened several times, the SG safety 

relief valve finally sticks open. 

41 Manual override of steam isolation signal from 30 bars and change-over 

to gradient mode at 5 bars/min. 

41 Containment general isolation due to low level of pressurizer (< 1.4 m) 

and low primary pressure (<100 bar). RCPs trip. 

45 Pause for data collection 

46 Steam dumping will probably be decreased to between 2-10% (valve 

position) due to low steam pressure. Stuck open safety relief valve 

causes high primary circuit cooldown rate. 

50 Staff begin to maintain emergency cooling water from exceptional 

reserve pools (this activity is taken outside of MCR). 

50 The second HPSI pump will probably be stopped. 

50 Decision to try manually closing the sticking main gate valve. 

51 Core cooling accumulators isolated manually. 

52 Staff should start cooling of the feedwater tanks based on the 

temperature of the primary circuit. 

A-5 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

55 The third HPSI pump will probably be stopped. 

55 Restart of intermediate cooling system (TF60) to facilitate the letdown

flow. 

56 Restart of service water system (VF62) to cool TF60. 

56 Opening of the letdown isolation valves. 

57 Let-down will probably be opened for controlling the level in the 

Pressurizer. 

60 Core exit temperature is about 235°C, pressure 45 bars, PZR-level 8 m, 

steam pressure 10 bars. 

60 Pause for data collection 

75 Pause for data collection 

90 Cooling down shall continue to cold shut-down. 

A-6 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.2 Scenario 2: Interfacing Systems Loss Of Coolant Accident: Primary Leakage 

Outside The Containment With Ensuing Control Room 

Instrument Malfunctions 

A.2.1 Synopsis 

In initial conditions the turbines are operating at full power, PZR-level 4.6 m. An instrument line breaks in 

the instrumentation room (IR) in the reactor building basement. Rupture of the reference line in the 

pressurizer steam space means that a steam leak takes place as long as the PZR-level is under 5.3 meters; 

thereafter it is in liquid phase or theoretically as two phase flow. Leak flow was permanently set at 0.88 

kg/s as steam. Additional demands on the MCR staff are caused by a defective measurement YD14FO1, 

measurement fault (i.e., RCP seal injection flow) with gradient -0.1 kg/s. This occurs because the steam 

leakage in the instrument room damages this instrument. This results in a spurious signal starting the 

emergency injection water pump, YD30D01, and tripping RCP YD14DO1. This incident will probably lead 

the staff to suspect a leak in RCP injection water system. 

After all this process information the CRS or RO will send a FO to check the rooms in the reactor building 

basement to find the possible leak. The Loviisa staff will expect some symptoms (alarms) in the ventilation 

and floor drain well levels. The FO will pass the defected instrumentation room on the way to YD-injection 

water room on the lower levels. Some steam may penetrate through the door seals, which are not leak-tight. 

The assumption in this early phase is that the FO does not notice anything strange. 

After the FO has checked the basement, the simulator instructor will call to the RO and report that he found 

nothing. 

Simultaneously with the call from the FO, the MCR staff will receive a specific fire alarm from the 

instrumentation room. The same signal will automatically close the exit ventilation line. It will further 

increase temperature and the pressure rise in the instrument room resulting in visible steam penetration 

through the door. 

The room-specific humidity alarm would be the first alarm. A steam leak of 1 kg/s will probably increase 

the humidity (alarm limit > 60%, normal value is 6 %) very early. Closed trap of the floor will also produce 

a specific flood alarm > 30 mm. The latter alarm was assumed to come later than the humidity alarm in the 

case of a steam leak. 

The exit ventilation line of the instrumentation room has a humidity measurement alarm down-stream, but it 

probably is slower than the other alarms. There are also radiation measurements, which will cause a change

over in the vent line from exhaust to filtration in case of high radiation. 

Additional instrumentation faults occur as a consequence of continued steam spraying into the instrument 

room, e.g., injection water disturbance of the same kind as earlier or spurious low level signal of an 

emergency accumulator closing the discharge line. These have little importance for the operation of the 

plant, but will produce uncertainties in the control room. 

A-7 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTONS

A.22 Probable course of events 

Note: Entries preceded by an * symbol indicate planned malfunctions implemented by the simulator 

instructor. 

Time (min.) 5'

6 Description 

(7:57 p.m. simulated time of day) 

0 Unit at full power. 

3 * Leak of PZR-level measurement +line (steam space), parameter 1 

cm2 . * Measurement fault YP10L001 +0.3 i/min. over 12 minutes. 

3 alarm High PZR-level > +200 mm. 

3 alarm Unrecognized leak > 1 kg/s. This is a common alarm, even in normal 

power operation or small evolutions at the plant. 

4 alarm PZR-heater banks automatically actuated. 

4 Normal letdown valve opened on high level signal. 

5 Normal level control switched to manual mode by RO. 

5 RO probably notices that letdown should not be open and closes the 

valve. 

6 *Measurement fault YD 14FO1, gradient -0.1 kg/s. Alarm and pump 

trip in 3 minutes. 

7 RO takes the makeup pump automatics off to make it possible to 

operate makeup pumps. 

7 RO probably starts both TK52/53D001 additional makeup pumps. 

8 Emergency RCP injection water initiated. 

9 alarm YD14FO1 (RCP YD14 seal injection flow) < 0.19 kg/s, YD14DO1 

(RCP) trip. 

9 Plant automatic power reduction to 80 %. 

10 Pause for data collection 

5 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., 

emulation of passive system performance. 

6Times for data collection are approximate and, in some cases, depend on a crew activity. 

A-8 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

11 alarm *Fire alarm in the instrumentation room UX00K401. 

12 Fire alarm sounds and CRS goes to the fire monitoring panel and 

checks the identification code of fire location. CRS alerts the fire 

department by the loud speaker, and they will access the reactor 

building. 

13 *TH80L003 (accumulator) measurement fault, gradient -I m/main. 

Alarm will be produced in 5 minutes. Erroneous instrumentation 

signal due to high humidity/temperature. This will close the 

accumulator discharge line valve in five minutes. Alarm will 

indicate that accumulators have been injecting to primary circuit, 

though primary circuit is above accumulator injection pressure. 

18 TH80L003 Accumulator gradient alarm. 

19 *YB54L061(Steam Generator Low level) level measurement fault 

due to humidity problems in instrumentation room. This plant 

protection signal initiates inadvertent EFW startup into the YB54 

steam generator. Because it is a plant protection signal, BOP 

operator may follow up to attempt to determine the cause of the 

alarm. 

20 Pause for data collection 

24 Firemen inform MCR about what they see in the reactor building 

basement: "There is no fire, but the instrumentation room is full of 

steam". They cannot see anything nor access the compartment. 

26 Firemen have a radiation meter. They will ask CRS instructions what 

to do, i.e., Are they allowed to access the instrument room? CRS 

will probably ask how high the radiation is. If it is > 10 mSv/h, 

access is not allowed. 

In case of access, they have respirators, and protective clothes are 

available in the fire truck. 

CRS will probably ask firemen to rapidly check the room inside and 

identify the leaking pipeline, if they have appropriate clothing and 

respirators. 

28 The MCR staff may diagnose an ISLOCA, though the source of the 

leak will not yet be clear. Earlier RCP trip due to low flow may 

indicate leaking flow injection line. However, the temperature of 

RCP seal water is not high enough to produce steam, so they will 

have to check all possibilities, and look through alarm logs, 

instrument diagrams, and procedures. 

CRS takes the event based procedure of small LOCA. 

29 The firemen cannot see clearly enough to tell the code of the leaking 

line. 

29 Manual start of boration. 

A-9 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

32 Reactor manual scram. 

Pause for data collection (Manual SCRAM plus 5 minutes, or at -32 

minutes). 

35 Cooling down by turbine by-pass according to the EOP. 

33-40 Operation of PZR-level control to adjust the level to normal. 

40 YZ70 manual overriding to gradient mode with the limit of 5 bars. 

This is done when steam pressure < 35 bars. This may be done later 

depending on the shutdown rate, etc. 

50 Leak may be locally identified in the SG compartment of the 

containment. Access into the compartment is possible immediately 

after reactor scram. But the crew must decide that this is the place to 

look. 

The isolation is not complete, YP1OLO continues leaking, 0.1 cmr2 . 

Pause for data collection 

53 Start of the pressurizer cooling by spraying. 

55 Shut down goes on. 

60 Attempts to stop the rest of leak, crimping the leaking pipe flat, for 

example. 

65 Pause for data collection 

A-10 

NUTREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.3 Scenario 3: Total Loss Of Feedwater 

A.3.1 Synopsis 

Initial conditions: 

"* Unit is at 100 % power, 

"* Reactor status: End Of Cycle (or high decay heat rate) 

"* Emergency Main Feedwater-pump RL92DO1 is out of service. 

"* MFW-pump RL91DO1 is out of service. 

"* Lubricating oil pump RL22D03 out of service. 

In this scenario, a main feedwater line leakage of 20 kg/s occurs in the room where the main feedwater 

pump lubricating oil pumps are located. This is a common-cause fault (i.e., a single fault that results in 

multiple failures) that produces electrical short-circuiting of the lubricating oil pumps. As a consequence, 

individual oil pumps trip until, one by one, all of the main feedwater pumps have tripped. Only one of the 

emergency feedwater pumps is available, since the other is out of service for some kind of maintenance or 

repair. 

The operating crew will use the remaining EFW pump to supply feedwater to the steam generators. 

However, due to a recurrent over-speed fault, even the remaining EFW pump is not able to supply the 

necessary feedwater to remove decay heat. Over the course of the scenario, the crew will inject water from 

emergency sources into the RCS while they attempt to recover at least one feedwater or emergency 

feedwater pump. Heatup of the RCS will occur. It is important to note here that, unlike U.S. plants, the 

Loviisa plant (like other Russian-designed PWRs of it's type) does not have sufficient PORV capacity to 

use once-through core cooling or "feed and bleed" cooling to successfully remove core decay heat. Hence, 

the crew is dependent upon recovery of a secondary system heat sink to remove decay heat. 

After approximately 1 hour (conventional plant) to 2 1⁄2 hours (advanced plant) later, as maintenance and 

electrician personnel arrive at the plant on the morning shift, a main feedwater pump lubricating oil pump 

will become available to place back in service. This occurs only after the MCR staff isolate the feedwater 

leak and direct personnel to dry the pump. Thereafter, MCR staff must evaluate a situation for which 

procedural guidance is unclear: the recommissioning of an empty steam generator. The crew must 

determine the best way to supply water to the empty steam generator(s) without inducing thermal shock to 

the collector sheet in the steam generator. This latter situation (i.e., uncontrolled feed of cold water to the 

collector sheet) could result in a tube rupture. 

A-11 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.3.2 Probable course of events 

Note: Entries preceded by an * symbol indicate planned malfunctions implemented by the simulator 

instructor. 

Time (min.sec) 7,8 Description 

(simulated time of day) 

-10.0 Unit on full power. 

(06:10 a.m.) 

0.0 *Leak of main feed water line RL20, parameter 1.3 = 20 kg/s 

(06:20 a.m.)

2.00

5.10 alarm 

5.17 

5.18 alarm 

6.11 alarm 

alarm 

6.29 alarm 

7.07 alarm 

7.13 

7.40 

7.42 alarm 

7.47 alarm 

9.13 alarm 

10.06 alarm

Decreasing level of the FW-tanks. 

Increasing secondary makeup water flow. 

Pause for data collection 

RL22D02 oil pump tripped. This is caused by electrical short 

circuit because of wetting from the leak. 

MFW-pump RL22DO1 tripped, due to no lubrication. 

MFW-pressure < 65 bar. 

Fire alarm due to hot steam and water spraying in room. 

Fire extinguisher pumps automatically started. 

RL21D02 oil pump tripped (electrical short circuit, because of 

wetting). 

RL21D03 reserve oil pump starts automatically. 

RL21D03 tripped (electrical short circuit, because of wetting). 

RL21D01, MFW-pump tripped, due to no lubrication oil. 

Steam pressure > 47 bar. 

MFW-pressure > 55 bars. 

RC (turbine bypass) > 25% open, due to automatic turbine power 

decrease. 

RL61D02 oil pump tripped. (electrical short circuit because of 

wetting). 

RL61D03 started automatically. 

RL61D03 tripped. (electrical short circuit, because of wetting).

7 Times are different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation 

of passive system performance. 

8 Times for data collection are approximate and, in some cases, depend on a crew activity. 

A-12 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

alarm 

alarm 

SPDS-alarm 

prior 

alarm

10.12 

10.13

10.24 

10.25 

10.25 

10.40 

10.35 

10.50 

11.14 

11.55 

14.07 

16.16 

17.17 

24.04 

25.13 

29.07 

30.06 

37.35 

(Note: 

40 

42

A-13

NUREG/IA-0137

RL61DO1 MFW-pump tripped, due to no lubrication. 

RL62DO1 tripped because of low MFW-pressure < 52 bars. 

There is now a loss of MFW. 

The leak decreased from 20 kg/s to 6 kg/s after Loss of MFW. 

Manual reactor trip. 

EFW-pump RL93DO1 started automatically, on low SG level. 

Turbine trips due to reactor scram. 

YZ51 started EFW-signal due to low level -140 ram. 

Primary cooling is abnormal, category 1. 

SG-level < 1.95 m.

SPDS-alarm Low SG level 

SPDS-alarm SG level 2/6 -300 mm. 

SPDS-alarm Automatic diagnosis: Leak of secondary circuit (suppressed). 

alarm *Over-current trip of EFW-pump, RL93D01. 

Theie is now a total loss of feedwater. 

Pause for data collection 

alarm All RCPs tripped due to low SG levels, -500 mm. 

SPDS-alarm Primary cooling is abnormal, category 2. 

SG-level -800 mm and T < 300"C. 

SPDS-alarm SG-levels < 1.2 m (= <-900 mm below nominal). 

RL93DO1 started again. 

*RL93D01 trips again. (Starting and tripping occurs repeatedly, 

depending on attempts to re-start requested by MCR staff). 

Pause for data collection 

RO Initiates large volume boration. 

Time scale between conventional and advanced plants differ now) 

Primary circuit heatup 20°C/h. 

SPDS-alarm Primary cooling is abnormal, category 3. 

SG-level -800 mm and T > 300°C.
APPENDIX A: SCENARIO DESCRIPTIONS

Pause for data collection 

60 Startup of RL62D03, MFW oil pump after drying the electrical 

switching box of the pump motor. 

62 Startup of RL62DO1, MFW pump 

64 Initiation of MFW to one SG with small flow rate in the beginning. 

Pause for data collection 

A-14 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.4 Scenario 4: Loss Of Off-site Power With A Stuck Open Steam Generator 

Safety Relief Valve 

A.4.1 Synopsis 

Initial conditions: 

* 90% power, load following. 

* Reactor status: End Of Cycle, boron concentration 77 PPM, burnup 322 full power days. 

* 400 kV grid in overhaul, circle line of Southern Finland is disconnected at Korea switch yard. 

* A core exit temperature measurement of reactor protection is faulted. It has one day earlier exceeded 

the fast trip limit, 3 10°C. Reactor protection takes place by 2/3 logic. 

The scenario begins with a request from the dispatch center for the plant to go to full power. After 

beginning this task, a load rejection on one of the plant's two turbine generators occurs due to a controller 

failure. The staff will be able to repair this malfunction rather easily, and begin a return to power on both 

turbines. During the return to power, the external supply of power to the plant is lost. 

The loss of off-site power produces a number of alarms, mainly on the secondary side of the process. In 

addition to the normal alarms, an additional malfunction occurs: a stuck-open steam generator safety relief 

valve (SRV). Normally an over-cooling event, the stuck open SRV will increase the rate of over-cooling. 

An important task of the MCR staff is to detect this spurious valve failure and isolate it. 

Because of a pre-existing reactor protection signal fault, the reactor will trip. Even after the station lost it's 

external power supply, it was supplied with power by it's own power production. With the trip of the 

reactor due to the protection signal fault, the station is without power, aside from it's emergency diesel 

generators. These will provide the necessary electrical supply for the MCR staff to bring the plant to a safe 

and stable shutdown state. 

A-15 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.4.2 Probable course of events 

Note: Entries preceded by an * symbol indicate planned malfunctions implemented by the simulator 

instructor. 

Time (min.sec) 9,10 Description

0 

(06:00 a.m. simulated time of day) 

7 

8

10 alarm

14 

15 

20 

25 

26 

32 

34

34.25 

35.00 

35.15 

35.16

alarms

Unit power at 90% power.

Dispatch center asks Unit to go back to full power, increasing at 100 

MW/h. 

BOP operator begins to carry out the task. 

The electrical output starts increasing. 

Pause for data collection 

*Turbine power controller alarm. 

BOP operator/CRS acknowledges the alarm in the instrumentation 

room. 

Turbine 2 power reduction due to the power controller failure. The 

turbine output is 30-40 MW controlled by the frequency controller. 

Staff will repair the malfunction and begin the return to power. 

Pause for data collection 

*Loss of off-site power, 400 & 100 kV grids are lost. 

*RA52S02, SG safety relief valve sticks open simultaneously with 

secondary pressure spike. 

Reactor fast trip due to faulted core exit temperature over 31 0VC, 2/3 

logic. 

(Reactor scram is not a consequence of turbine trips.) 

Power imbalance I, between reactor and turbines, +5 bar addition to the 

fresh steam set value of the turbine power controller. 

Turbine trips due to power imbalance II. 

RO starts emergency boration. 

Low voltage alarms of auxiliary power supplies. 

Frequency = 45 Hz, generator trips.

9 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., 

emulation of passive system performance. 

' Times for data collection are approximate and, in some cases, depend on a crew activity. 

A-16 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

35.18

35.20 

35.20 

35.22

SPDS-alarm

RCP trips due to low voltage. 

6 and 0.4 kV loads trip, breakers of busbars open. 

Reactor trip due to loss of RCPs. 

Inventory alarm, PZR-level < 1.7 m 

Diesel generators start. 

Automatic restart of safety related loads. 

HPSI initiation. 

At this time approximately the staff will probably recognize stuck open 

SG safety relief valve. 

EFW automatic initiation due to loss of main feedwater. 

RO probably starts closing YA12SO01, the main gate valve and also the 

corresponding TC-valves connected to the loop with the affected SG. 

Pause for data collection 

EFW-pump trip due to pressure spike of makeup water in the pump 

suction. 

Steam pressure < 35 bar. 

YZ51 plant protection signal on reinitiating EFW. 

BOP isolates the faulted SG resulting in a fast pressure drop in the SG. 

Automatic isolation of SG, YB52 due to steam pressure difference > 5 

bars between YB52 and the main steam header. 

Now the staff will dry out the faulted SG and let it cool down. 

RO should stop HPSI, 2/4, TJ12DO01 and TJ52D00l. 

TJ51DO01 stopped by RO. 

Pause for data collection 

TJ1 1DO01 stopped by RO. 

Process is stable and cooling is maintained by natural circulation and 

using atmospheric steam header relief valves. 

A-17 

NUREG/IA-0137

35.22-35 

36.30 

36.36 

36.40 

36.46 

37.10 

.37.28 

37.44 

40.11 

47.18 

51.04
APPENDIX A: SCENARIO DESCRIPTIONS

A.5 Scenario 5. Steam Generator Over-fill 

A.5.1 Synopsis 

Initial condition: Full power, 100%, normal size of SG. 

The scenario begins after a shift change-over has taken place. A small de-watering line leak in the high 

pressure pre-heaters has been detected earlier and it is necessary to shut down one train of the high pressure 

pre-heater system (RD50) for repair. It is a written order given by the operations engineer. 

During the shut-down of the affected train of high pressure pre-heaters, an alarm is received in the MCR. A 

faulty current has resulted in an interruption of the normal power supply to one of the RCPs. The pump has 

not tripped, but is supplied by station batteries. When the RO goes to the instrument room and 

acknowledges the alarm, a trip of the RCP occurs together with the ensuing power run-back of the reactor. 

After some time, the cause of the malfunction is repaired and the crew places the RCP back on-line. 

During the preparations for, and together with the re-start of the RCP, the BOP operator's task is to supply 

feedwater to the steam generator in the loop of the affected RCP. While doing so, a malfunction of the 

steam generator feedwater regulating supply valve system occurs. The regulating valve sticks open at 80%, 

and subsequent supply valve failures produce a situation in which the steam generator is supplied with water 

in excess of the capacity to boil it off. This results in a rising steam generator level in the affected steam 

generator. There is no specific procedure for this malfunction, only system knowledge and system-level 

operating procedures. 

When the BOP and other MCR staff realize that they have an uncontrolled supply of feedwater to a steam 

generator, they will begin attempts, both in and outside the control room to manually isolate the steam 

generator. This is needed to prevent water from entering the steam lines, and causing thermal shock to the 

highly-heated pipes and to further guard against the possibility of water approaching the turbine generators 

which may be destroyed by such an event. 

It is important, while attempting to isolate the affected steam generator, that water continues to be supplied 

to some of the unaffected steam generators to remove core decay heat. 

A-18 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

A.5.2 Probable course of events 

Time (min.sec) "," Description 

(10:00 p.m. simulated time of day) 

0 Unit on full power. 

Instrument technician and electrician are on-site during evening shift. 

3 BOP operator starts decreasing turbine 2 output by 8 MW to match high 

pressure pre-heater train RD50 shut-down. 

5 BOP gives an automatic shutdown command for RD50. 

8 He opens the FW-bypass of RD50 and closes the normal lineup (manual 

command of remote control valves.) 

9 The staff probably start planning the remaining isolations needed for 

shut down and repairs. 

Pause for data collection 

10 alarm (very *During the isolation planning, RCP YD12DO1 gives an alarm: 

important) "Magnet is supplied by batteries". The reason is a faulty current 

measurement. 

RO should go to the instrument room in the same building, lower level 

to check the status of the magnet. 

13 The simulator instructor describes the situation to the RO: "The RCP 

relief magnet is now supplied by the batteries." 

15 When the RO acknowledges the alarm it results in a trip of the whole 

relief magnet and RCP, too. 

15- Automatic power decrease occurs. 

20 The staff monitor that everything goes as planned: 

- reactor power reduction to 75-80 % 

- no reverse rotation of the RCP (1-2 minute after the trip) 

- electric power reduction to 350-370 MW 

- Plant parameters, i.e.: 

- SG-levels 

- Steam pressure 

- Primary pressure, etc. 

20- Electrical and instrumentation technician diagnose the fault and 

30 correct it replacing the defective instrumentation card. 

Pause for data collection 

'1 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., 

emulation of passive system performance. 

12 Times for data collection are approximate and, in some cases, depend on a crew activity. 

A-19 

NUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

BOP operator adjusts the set value of the power controller to match the 

new situation. 

He also acknowledges an additional pressure signal from the reactor 

power limitation due to the RCP trip. 

His duty is now especially to monitor the power balance between reactor 

and the turbines. 

32 - The staff starts preparing for the RCP restart: 

- PZR-level increased to +50 mm. 

- SG-level decreased -50 mm to avoid overfill during start-up. 

This is done by opening the periodic SG-blow down for a while. 

- BOP operator switches the SG-level controller to manual mode and 

closed. 

35.39 RO restarts the tripped reactor coolant pump. 

35.51 BOP maintains the SG-level by operating manually the FW-control 

valve. 

36.46 RL72S002 (feedwater flow control valve) is 70 % open. 

36.54 65% open. 

37.10 When the level corresponds to the set value, BOP switches the level 

controller to automatic mode. 

37.20 BOP gives the power controller a new set value, 458 MW, and a 

gradient of 300 MW/h, which is used up to 405 MW; after that 120 

MW/h. 

37.36 *Simultaneously with power adjustment the specific main feedwater 

flow controller of SG52 inadvertently opens the control valve and on the 

way it sticks open 80% due to mechanical failure of the actuator. 

37.57 alarm SG level is rising (high level) 

38.12 RL72S03 block valve (operation time is 19 sec.) closes automatically 

due to high level, +100 mm. 

After RL72S03 closure the SG level drops fast and at +50 mm, the valve 

is opened again by automatics. 

Pause for data collection 

38.24 BOP switches the control circuit to manual mode and tries to close the 

control valve, but is unsuccessful. 

38.41 *The valve cycles 2-3 times and finally sticks 30 % open due to a torque 

trip. 

39 BOP will now try to close the serial block valve, RL72SO01 (operation 

time is 78 sec.) 

A-20 

NLUREG/IA-0137
APPENDIX A: SCENARIO DESCRIPTIONS

39.45 *RL72SO01 sticks 13 % open, torque trip. 

41.09 *Torque trip will finally stick RL72S03 to the position of 30-50% 

resulting in further rising level, three stuck open serial valves. 

41.45 Turbine trips will occur at a level of +20 Mm. 

42.24 At the level +340 mm a new protection signal will isolate the steam line, 

MFW-line (RL72S03, but it stuck open 30 %) and trip the RCP. 

No reactor trip takes place. Reactor power will be decreased 

automatically (if in automatic mode) or the RO may do it manually 

faster, if quick enough. 

The SG level continues rising. 

43.00 BOP tries to avoid the torque trip of RL72S03 by trying to open it, no 

success. 

43.30 BOP tries to avoid the torque trip of RL72SO1 by trying to open and re

close it, no success. 

43.45 BOP sends FO to locally and manually close RL72S03. The closure will 

take about in 10 minutes. The SG level will continue to rise during this 

time, gradually approaching the level of the MFW lines. 

44.00 RO trips the reactor. 

44.00 BOP stops all the MFW-pumps to avoid excessive overfilling. 

SG safety valve may open, depending on the time scale. 

44 An alternative: BOP can try to split the MFW-system to two sections 

(closing RL61S001, operation time 125 sec.) and stop half of the MFW

pumps and corresponding RCPs leaving the other half running and 

maintaining 50 % power. 

45 After MFW-pumps stop it is possible that the crew try to re-open and re

close the faulted block valve. It will be logical to let them succeed in 

closing because the pressure difference across the valves is gone. 

Pause for data collection 

46 Scenario continues until valves are returned to normal (e.g., 10-20 

minutes) 

A-21 

NUREG/IA-0137
APPENDIX B: INSTRUCTIONS TO PARTICIPANTS 

B-1 

NUREG/IA-0137
B.1 General Instructions to Participants

The Halden Reactor Project together with the Loviisa Nuclear Power Station are conducting a research 

project for the U.S. Nuclear Regulatory Commission concerning the staffing of advanced passive plant 

control rooms. The purpose of this research is to evaluate operator performance in representative, 

conventional plant environments and advanced plant-type environments. For similar challenging situations 

we wish to evaluate the effect of different staffing complements on crew performance. Eight crews from the 

Loviisa Nuclear Power Station are assisting us with this research, together with personnel from the Loviisa 

Nuclear Power Station simulator facility. Four of the crews will participate in the study at the Loviisa 

Nuclear Power Station. The other four crews will come to Halden (HAMMLAB - HAlden Man-Machine 

Laboratory) to participate in the study. 

(The following paragraph was only in the General Instructions given to Loviisa /Conventional plant 

participants) 

At Loviisa some changes will be made in the simulation facility as you are accustomed to seeing it. This is 

being done to make Loviisa more representative of conventional western style PWR-control rooms and 

plant performance. Before the study that you are about to participate in, you will also receive some 

additional training to acquaint you with some of these differences and to become more familiarized with 

these differences. All of your previous training, experience and procedures for operating the plant will still 

be applicable during the study. The main differences are that some of the information systems in the control 

room, e.g., the advanced computerized systems, will be somewhat unavailable. After the training we will go 

through a series of five challenging scenarios. The purpose of the scenarios is to gather a baseline of 

operator performance in a conventional type plant complement for comparison purposes with advanced 

plant type complements, which will be conducted in the autumn at Halden. 

(The following paragraph was only in the General Instructions given to Halden/Advanced plant 

participants) 

Before the study in HAMMLAB, you will receive a couple of days of training. The process modeled here 

is the Loviisa process, with minor modifications. Your training will focus on the differences between the 

interfaces, the systems in HAMMLAB which may be unfamiliar to you, and the way to interact with the 

simulator. After the training we will go through a series of five challenging scenarios. The purpose of the 

scenarios is to collect data on operator performance in an advanced type plant complement for comparison 

purposes with conventional plant type complements, which was conducted in the spring at Loviisa. 

During each scenario, interruptions will be made in which members of our staff will give you a set of 

questionnaires with some very straight-forward questions. In the interruptions, please simply answer the 

questions on the questionnaires; do not discuss questions with the other crew members. During the actual 

scenarios themselves, though, the part in which you are actually operating together as a crew, discussions 

should be carried out as normal. 

Following each scenario you will have a break. During the break there will be a short debrief in which we 

will ask you for some feedback about the scenarios, your impressions and thoughts about how specific 

things like procedures and training and the instrumentation in the control room helped you in controlling the 

situation. We will also ask you some general questions about what factors made the scenario difficult or 

easy to control. There are no right or wrong answers to the questions that we will ask you during the 

debriefing. We are only asking for your opinions about things which you perceive as affecting your 

performance. 

B-2 

NUREG/IA-0137
At the end of the study in the autumn, all the data will be analyzed from both Loviisa and from Halden, and 

a report produced. The report will be sent to Loviisa for review and analysis, before it will be sent to the 

NRC. If you are curious about the results of the different crews' performance in both the conventional and 

the advanced plant type complements, please contact (name of the plant-specific contact person) for details 

or even perhaps a copy of the report. 

One word should be mentioned about confidentiality. It is very important for obvious reasons that other 

members of the plant not be aware of the specific scenarios we are using. We ask, therefore, that you do not 

discuss the scenarios with other members of the plant or the operations staff, or in the areas where other 

crew members could overhear your conversations about this research. 

To guarantee your anonymity, data will not be provided which could be used to identify crew members. 

There will be no direct link between crew performance as presented in our final report and the crews which 

participated in the study nor individual crew members. 

Thank you for your participation, support and offering us your expertise in carrying out this very important 

research for a very important issue for advanced reactors. If you have specific questions about the study or 

about data from the study, please ask either (name of the plant-specific contact person) or (name of the 

study leader). 

B-3 

NUTREG/IA-0137
B.2 Instructions for Situation Awareness Questionnaire

This questionnaire has been developed in Halden with assistance from Loviisa operators to investigate how 

operators form and maintain a good overview of the process status. We typically use it in situations where 

we want to see if different process control systems are good or poor at giving the operator overview of the 

process. 

During the scenario, the simulator will be frozen several times and you will be given a couple of 

questionnaires. Please turn away from the screens: do not look at the panels. During these interruptions 

you will be given a questionnaire with a set of 18 questions. Each set of questions will be new for each 

interruption, although some questions might be repeated several times during a scenario. Please note that: 

"* For each interruption, the questions are picked at random from a large question-base. 

"* The questions sometimes focus on important parameters related to the disturbance, other times 

the parameters are not related to the disturbance. 

The questions will be given to you in written form. Each of the questions will be focused on a set of key 

process parameters (such as pressures and flows) and components (such as pumps and valves). The first 6 

questions are related to how the process has developed the last three minutes, the next 6 questions focus on 

the current state of the process in comparison to the first five minutes of the scenario, and the last 6 

questions are related to how you think the plant will develop over the next three minutes. For each of the 

questions, there is a limited number of answers to choose from. The alternatives are: 

"* The parameter remains the same. 

"* The parameter has decreased. 

"* The parameter has increased. 

Some questions ask you about several parameters at the same time (e.g. steam generators), and you can 

then answer deviations in one, more than one, or deviations in both directions. Please circle the answer 

category you think is appropriate for each question. 

Sometimes it can be difficult to decide if a parameter is just fluctuating or actually decreasing or increasing. 

We want to report changes only when: 

"* A parameter is deviating from its normal value 

"* There is a steady increasing or decreasing trend in the parameter. 

We will ask you how parameters have developed from the 'recent past' and how they will develop in the 

'near future'. By 'recent past' and 'near future' we mean approximately three minutes ago, or 

approximately three minutes into the future. After the experiment, all your answers will be compared to the 

simulator log. 

Please answer all questions. We want you to give your best judgment about the status of the parameter and 

component, regardless if you know the actual status of the component, or you have to infer this from other 

information you have. If you have no idea about the actual parameter, simply give us your best guess. 

3-4 

NUREG/IA-0137
B.3 Participant Instructions: Workload Rating Scales

We are not only interested in assessing your performance but also the experiences you had during the 

different task conditions. Right now we are going to describe the technique that will be used to examine 

your experiences. In the most general sense we are examining the "workload" you experienced. Workload 

is a difficult concept to define precisely, but a simple one to understand generally. The factors that 

influence your experience of workload may come from the task itself, your feelings about your own 

performance, how much effort you put in, or the stress and frustration you felt. The workload contributed 

by different task elements may change as you get more familiar with a task, perform easier or harder 

versions of it, or move from one task to another. Physical components of workload are relatively easy to 

conceptualize and evaluate. However, the mental components of workload may be more difficult to 

measure. 

Since workload is something that is experienced individually by each person, there are no effective "rulers" 

that can be used to estimate the workload of different activities. One way to find out about workload is to 

ask people to describe the feelings they experienced. Because workload may be caused by many different 

factors, we would like you to evaluate several of them individually rather than lumping them into a single 

global evaluation of overall workload. This set of six rating scales was developed for you to use in 

evaluating your experiences during different tasks. Please read the descriptions of the scales carefully. If 

you have a question about any of the scales in the table, please ask me about it. It is extremely important 

that they be clear to you. You may keep the descriptions with you for reference during the experiment. 

After performing each of the tasks, you will be given a sheet of rating scales. You will evaluate the task by 

putting an "X" on each of the six scales at the point which matches your experience. Each line has two 

endpoint descriptors that describe the scale. Please consider your responses carefully, and try to use the 

scales to distinguish among the different task con-ditions. Consider each scale individually. The ratings you 

give us will play an important role in the evaluation being conducted. 

Your active participation is essential to the success of this experiment. Your willingness to participate is 

greatly appreciated by all of us. 

B-5 

NUREG/IA-0137
B.4 Instructions for the Behaviorally Anchored Rating Scale

During this experiment, you will watch eight crews of operators performing in five process disturbance 

scenarios. All crews will participate in the same five scenarios. You will be seated in the control room with 

the crew, so you can hear and see their communications and behaviors. You will be seated at the side of the 

room, so you do not disturb the crews. Please remain seated and do not talk with the crew members during 

the sessions. Each scenario will be divided by interrupts. During these interrupts, the crew will complete 

questionnaires. Also, during these interrupts, please complete the rating form for evaluating the crew's 

team interaction. Please only evaluate the crew's performance on the last segment of the scenario (from the 

last interrupt to the present). 

The rating form divides team interaction into five dimensions: task focus/decision making, coordination as 

a crew, communication, openness, and team spirit. For each of these dimensions, a table lists behaviors to 

watch for when evaluating the crews. Please make notes as you feel relevant, to help you accurately 

evaluate the crew. 

At the bottom of each page (for each dimension of team interaction) is a rating scale, from 1 (worst) to 7 

(best). Please draw an X over the part of the scale which you feel best represents the crew's performance 

on that dimension for the scenario segment immediately preceding the interrupt. To help you make your 

evaluation, each rating scale contains examples of behaviors that would be rated as 1, 4, or 7. 

Thank you for your participation and assistance. 

B-6 

NUREG/IA-0137
APPENDIX C: INVENTORIES

C-1

NUREG/IA-0137
APPENDIX C: INVENTORIES

C.1 Situation Awareness Control Room Inventory Example Inventory 

Test subject: RO Scenario:01 Administration :1 

Team: 1 

HISTORY 

In comparison with the recent past, how has the insertion of L rods 

(strong adjustment effect) developed? 

Increase Same Decrease 

In comparison with the recent past, how has the number of active primary 

circulation pumps developed? 

Increase Same Decrease 

In comparison with the recent past, how has the number of active pumps 

in the TK make up system developed? 

Increase Same Decrease 

In comparison with the recent past, how has the temperatures after 

the low-pressure pre-heaters developed? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

In comparison with the recent past, how has the number of active condensate 

pumps developed? 

Increase Same Decrease 

In comparison with the recent past, how has the pressures in the condensers 

developed? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

NOW 

In comparison with the normal status, how would you describe the current 

temperatures in the hot legs of the primary circuit ? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

In comparison with the normal status, how would you describe the current 

neutron flux of the reactor (power range detectors) ? 

Increase Same Decrease 

In comparison with the normal status, how would you describe the current 

number of active pressurizer heater banks ? 

C-2 

NUREG/IA-0137
APPENDIX C: INVENTORIES

Increase Same Decrease 

In comparison with the normal status, how would you describe the current 

temperatures in the condensers ? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

In comparison with the normal status, how would you describe the current 

flows into the steam generators ? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

In comparison with the normal status, how would you describe the current 

electrical power outputs from the generators ? 

Increase > 1 Increase I Same Decrease I Decrease > 1 Drift In Both Directions 

FUTURE 

In comparison with now, predict how the insertion of D rods (weak adjustment 

effect) will develop? 

Increase Same Decrease 

In comparison with now, predict how the level in the pressurizer will develop? 

Increase Same Decrease 

In comparison with now, predict how the temperatures in the cold legs 

of the primary circuit will develop? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > I Drift In Both Directions 

In comparison with now, predict how the steam pressures in the secondary 

loops will develop? 

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions 

In comparison with now, predict how the levels in the condensers will develop? 

Increase > 1 Increase 1 Same Decrease I Decrease > I Drift In Both Directions 

In comparison with now, predict how the openings of the turbine bypass 

valves will develop? 

Increase Same Decrease 

C-3 

NUREG/IA-0137
APPENDIX C: INVENTORIES

C.2 NASA TLX Rating Scale Definitions

Title

Mental Demand 

Physical Demand 

Temporal Demand

Performance

Effort

Frustration

Endpoints 

Low / High 

Low / High 

Low / High

Failure / Perfect

Low / High 

Low / High

Description

How much mental and perceptual activity was 

required (thinking, deciding, calculating, 

remembering?) 

How much physical activity was required (e.g. 

pushing, pulling, turning, controlling, activating, 

etc.)? Was the task slow or brisk, slack or 

strenuous, restful or laborious? 

How much time pressure did you feel due to the 

rate or pace at which the task elements occurred? 

Was the pace slow and leisurely or rapid and 

frantic? 

How successful do you think you were in 

accomplishing the goals of the task set by the 

experimenter (or yourself)? How satisfied were 

you with your performance in accomplishing 

these goals? 

How hard did you have to work (mentally and 

physically) to accomplish your level of 

performance? 

How insecure, discouraged, irritated, stressed 

and annoyed versus secure, gratified, content, 

relaxed and complacent did you feel during the 

task? 

C-4 

NUREG/IA-0137
APPENDIX C: INVENTORIES

Job Title in Scenario: 

Date: 

Name of examiner:

Scenario Number: 

Pause Number:

Mental Demand

L =1

Low 50 High

Physical Demand

I- I I

Low 50

.1

High

Temporal Demand

I I 1-

Low 50

I I

-I

High

Performance

i Failure

I I I I I I I

50

Effort

I- I I I I I

Low 50

Frustration Level

I I I I I I I I

High

C-5

NUREGAA-0137

Perfect

-I

High

!.

Low 50

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I I

I

!

I

I

I

I

I

I

I

I I

I

I

I

I

I

I

I

I

I

I I

!

I

I

I

I

I

I

I

I

I

I

I I

I

I

I
APPENDIX C: INVENTORIES 

C.3 Behaviorally Anchored Rating Scales Example Inventory 

Scenario 

Segment of scenario 

Date Time 

Crew 

Evaluator 

C-6 

NUREGAIA-0137
APPENDIX C: INVENTORIES

During the last set of exercises that you observed, did you see any of these things happen that significantly 

affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of 

the individuals who did what you marked. 

Communication Reactor Turbine Shift Control Room 

Operator Operator Supervisor Technician 

1. Sufficient amount of communication 

information that needed to be stated was stated. 

2. Information was informative, not chatter. 

3. Information was relevant to the situation. 

4. Information was correct. 

5. Information had the desired effect - the targeted 

person responded appropriately. 

6. Information was given at the appropriate time in 

the scenario. 

If the incident took place more than once, please indicate the number of occurrences. 

Did you observe any examples of extremely effective or extremely poor communication? (describe below I 

other side) 

Please place an X over the rating which you feel best describes the crew's performance on this dimension.

I I I I I I

Communication 1

I I I 

4

I I 7 

7

1: failed to communicate or communicated misleading information 

4: communicated about important issues, but neglected some 

7: optimal communication - communication was informative, effective, timely

C-7

NUREG/IA-0137

I
APPENDIX C: INVENTORIES

During the last set of exercises that you observed, did you see any of these things happen that significantly 

affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of 

the individuals who did what you marked. 

Team Spirit Reactor Turbine Shift Supervisor Control Room 

Operator Operator Technician 

1. Joking in a positive manner. 

2. Appropriate degree of informal discussion. 

3. Silences due to hostility. (- negative) 

4. Gestures of hostility. (- negative) 

5. Arguing over trivial issues. (- negative) 

If the incident took place more than once, please indicate the number of occurrences. 

Did you observe any examples of extremely good or extremely poor team spirit? (describe below/other 

side) 

Please place an X over the rating which you feel best describes the crew's performance on this dimension.

I I I I I I

Team Spirit

1: were openly negative, hostile 

4: were neutral 

7: demonstrated encouraging, positive and reinforcing behaviors 

C-8

NUREG/IA-0137

1 4 7

| | | | | |

I

I
APPENDIX C: INVENTORIES

During the last set of exercises that you observed, did you see any of these things happen that significantly 

affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of 

the individuals who did what you marked. 

Openness Reactor Turbine Shift Control Room 

Operator Operator Supervisor Technician 

1. Helpful in giving information to crew 

members. 

2. Asking for/providing assistance when needed. 

3. Seeking confirmation when appropriate. 

If the incident took place more than once, please indicate the number of occurrences. 

Did you observe any examples of extremely open or extremely closed team openness? (describe below / 

other side) 

Please place an X over the rating which you feel best describes the crew's performance on this dimension. 

Openness

I I I I I I

4 7

1: were not willing to give assistance or ask for assistance 

4: moderate openness - gave/received some assistance, but more would have been better 

7: showed optimal openness - giving/receiving assistance as often as needed 

C-9 

NUREG/IA-0137

1

I I I I I I
APPENDIX C: INVENTORIES

During the last set of exercises that you observed, did you see any of these things happen that significantly 

affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of 

the individuals who did what you marked. 

Coordination as a Crew Reactor Turbine Shift Control Room 

Operator Operator Supervisor Technician 

1. Good ability to work together. 

2. Able to shift roles when needed. 

3. Work together to find facts needed for 

decision making. 

4. Appropriate degree of delegation of 

tasks. 

5. Good utilization of technical support. 

If the incident took place more than once, please indicate the number of occurrences. 

Did you observe any examples of extremely good or extremely poor crew coordination? (describe below / 

other side) 

Please place an X over the rating which you feel best describes the crew's performance on this dimension.

Coordination as a Crew I I I I

III I I

I I I 4 I I I 

7

1: unable to work together 

4: moderately effective as a crew - they worked together reasonably effectively 

7: delegated wisely, shifted roles as needed - they worked together highly effectively 

C-10

NUREG/IA-0137

1
APPENDIX C: INVENTORIES 

During the last set of exercises that you observed, did you see any of these things happen that significantly 

affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of 

the individuals who did what you marked. 

Task Focus / Decision Making Reactor Turbine Shift Control Room 

Operator Operator Supervisor Technician 

1. Considering alternative actions. 

2. Staying with chosen primary task (remaining 

focused on main task). 

3. Prioritizing tasks. 

4. Solving the correct problem. 

5. Decisions appropriate to diagnosis. 

If the incident took place more than once, please indicate the number of occurrences. 

Did you observe any examples of extremely good or extremely poor task focus/decision making? (describe 

below / other side) 

Please place an X over the rating which you feel best describes the crew's performance on this dimension. 

Task Focus / Decision Making

I I I 4

' I 

7

1: unfocused, poor decision making 

4: moderately effective decision making 

7: optimal degree of consideration of alternatives and focus on task 

C-11

NUREG/IA-0137

1

I I I

I IJ

I

II
APPENDIX C: INVENTORIES

C.4 Rated Crew Performance Example Inventory 

Additional Performance Factors 

Explanation: 

Solution Path refers to the crew's use of time in recognizing the event (e.g., loss of heat 

transfer, etc.) and selection of the correct mitigation procedure(s). 

Control of plant refers to the crew's understanding of procedures in their analysis of the 

transient and the extent to which they challenged safety equipment (e.g., PORVs, code safety 

valves, etc.). 

Communication refers to the extent to which the information exchange among the crew 

members facilitated transient mitigation. 

Confidence refers to the ease with which the crew completed transient mitigation without 

hesitation, and self-statements about the sureness of their own actions and decisions. 

Please circle the number below which corresponds best to your rating of the crew on the above 

factors for this scenario only. A score of 1 represents a low score (poor performance on the 

factor) while a 10 is a high score (best performance on the factor).

Solution Path 

Control of Plant 

Communication 

Confidence 

Crew Number: 

Rater:

1 2 3 4 5 6 7 8 9 10 

1 2 3 4 5 6 7 8 9 10 

1 2 3 4 5 6 7 8 9 10 

1 2 3 4 5 6 7 8 9 10 

Scenario:

C-12

NUREG/IA-0137
APPENDIX D: SITUATION AWARENESS CONTROL ROOM 

INVENTORY (SACRI) 

D-1 

NUIREG/IA-0137
APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI)

D.1 SACRI: The Inventory of Questions 

Parameters included in the inventory 

Primary Circuit 

"* Average reactor temperature 

"* Flows through the TC purification systems 

"* Insertion of D rods (weak adjustment effect) 

"* Insertion of L rods (strong adjustment effect) 

"* Level in the pressurizer 

"* Level in the TE let-down system tank 

"* Level in the TH emergency water supply tank 

"* Margin of departure from Nucleate Boiling (DNB) 

"* Neutron flux of the reactor (power range monitors) 

"* Number of active pressurizer heater banks 

"* Number of active primary circulation pumps 

"* Number of active pumps in the TK make-up system 

"* Pressure in the containment 

"* Pressure in the pressurizer 

"* Pressure in the primary circuit 

"* Temperature in the pressurizer 

"* Temperatures in the cold legs of the primary circuit 

"* Temperatures in the hot legs of the primary circuit 

Secondary Circuit 

"* Electrical power outputs from the generators 

"* Flows into the steam generators 

"* Levels in the condensers 

"* Levels in the feedwater tanks 

"* Levels in the steam generators 

"* Number of active main ejectors 

"• Number of active condensate pumps 

"* Number of active emergency feedwater pumps 

"• Number of active main feedwater pumps 

"* Openings of the condensate systems' three-way control valves 

"• Openings of the turbine bypass valves 

"* Pressures in the condensers 

"* Steam line pressure 

"* Steam line temperature 

"* Temperatures after the high-pressure pre-heaters 

"* Temperatures after the low-pressure pre-heaters 

"• Temperatures in the condensers 

D-2

NUREG/IA-0137
APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI)

D.2 Wording of the SACRI Questions 

The wording of the questions is as follows (with the parameter inserted in the blank space): 

Questions comparing the present with the recent past 

e In comparison with the recent past, how have the __ developed? 

Questions comparing the present with the status during normal operations 

o In comparison with the normal status, how would you describe the -? 

Questions comparing the present with the predicted near future 

o In comparison with now, predict how the - will develop over the next few minutes. 

D.3 Answer Categories 

The answer categories used are: 

* Increase / Decrease I Same 

* Increase in more than one / Increase in one / Same / Decrease in one / Decrease in more than one / Drift 

in both directions. 

A computer program generates a series of 18 questions, including 9 from the primary side and 9 from the 

secondary side. The 9 questions can then be subdivided into 3 groups of 3 questions each, asking about the 

past, present, and future states of the plant. A parameter is only selected once for each questionnaire 

generation. Each question uses the appropriate answer category. 

D-3 

NUREG/IA-0137
APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI)

D.4 Signal Detection Theory Calculations 

Abbreviations used in the formulas 

pCA probability of Correct Acceptance: number of correct acceptances divided by the total number of 

correct acceptances and false alarms 

pFA probability of False Alarm: number of false alarms divided by the total number of correct 

acceptances and false alarms 

pH probability of Hit: number of hits divided by total number of hits and misses 

pM probability of Miss: number of misses divided by total number of hits and misses 

pDe probability of Deviation: total number of hits and misses divided by total number of questions 

pNDe probability of Non Deviations: total number of correct acceptances and false alarms divided by the 

total number of questions 

A' Non parametric measure of operator's ability to discriminate stable parameters from those which are 

fluctuating 

A' = I - 0.25 {(pFA/pH) + (pM/pCA)} 

An A' score of 1.0 indicates that the operator was perfect in his/her ability to discriminate parameter drifts 

from stability, i.e. s/he has made no errors. A performance score of 0.5 meant the operator performed no 

better than if s/he had been responding according to chance. A performance score of below 0.5 meant the 

operator's responses were worse than if they had been determined by chance. In fact, it is mathematically 

possible to derive a negative score: however, due to psychological factors involved in the detection of 

parameter deviations (on which the underlying principles of Signal Detection Theory are based), an expert 

operator who is genuinely responding to the best of his/her ability is unlikely to score below the 0.5 mark. 

R:S Ratio Non-parametric measure of response bias 

R:S = { (pNDe * pFA) + (pDe * pH)} / pDe 

A value of I means the operator showed no bias towards either under-estimating or overestimating the 

extent of non-stable parameters; below 1, the operator under-estimate; above 1, the operator over

estimated. 

D-4 

NUREG/AI-0137
APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI)

D.5 Scoring the Situation Awareness Inventories 

Operator responses should be scored as one of the following four categories: hit, miss, false alarm, or 

correct acceptance. These categories are based on the following situation: 

Process Parameter

Change 

Hit

No change 

False Alarm 

Correct Acceptance

Use the following table for more specific guidance.

The operator says...

Same

The master copy says...

Same

Score

Correct Acceptance

Increase, 

Decrease, 

Increase >1, 

Increase in only one, 

Decrease > 1, 

Decrease in only one, 

Drift in both directions 

(a change)

Same

Increase, 

Decrease, 

Increase >1, 

Increase in only one, 

Decrease > 1, 

Decrease in only one, 

Drift in both directions 

(same change)

any change

any change Same

Drift in both directions 

any other change, Same 

Increase, Increase > 1, Increase in 

only one, Same 

Decrease, Decrease > 1, Decrease 

in only one, Same 

Increase >1, Decrease >1 

Increase in only one, Decrease in 

only one

any other change, Same 

Drift in both directions 

Decrease, Decrease > 1, Decrease 

in only one 

Increase, Increase > 1, Increase in 

only one 

Increase in only one, Decrease in 

only one 

Increase >1, Decrease >1 

D-5

Hit 

Hit 

Hit 

Hit 

Hit 

Hit 

Hit 

(score as Hit) 

Miss 

False Alarm 

False Alarm 

Miss 

Miss 

Miss 

False Alarm 

Miss

NUREG/IA-0137

Operator 

response 

Change

No change Miss
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE 

RATING FORMS 

E-1 

NUREG/IA-0137
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

Table E-1 Ratings of transient management tasks on the Steam Generator Tube Rupture scenario 

Announcements and Notifications Critical Task Completion Stabilization and Cooldown 

Crew 1 2 3 413 5 6 7 8 Crew 122 31456 718 Crew 1 2 3 4 56 7 8 

Initiate Response 3 4 4 3 3 4 4 5 Identification of SGTR 2 4 1 2 3 4 2 5 -Handling of Cooldown 1 4 3 2 3 5 3 1 

Sound Fire Alarm XXXXXX X X Response toSGTR 3 5 1 2 4 5 2 5 Pressurizer Level 2 2 2 1 45 4 5 

Alert Fire Brigade -- XX-- -- X XX Stop Pump X X -- X X X-- X Primary Pressure 1 2 3 2 3 3 3 1 

Make Fire Announcement 3 4 3 3 2 4 5 5 Close Valves X X XX X Xj-- X RateofCooling 2 4 3 1 215 2 1 

Declare Plant Emergency X X XIX -- X X X Reactor Scram 3 4 3 3 2 4 4 5 Departure from Nucleate 1 4 3 2 3 3 3 1 

1_ Boiling 

Notify Regulatory Agency X XX X -- -- X X InitiateBoration 4 2 3 2 4 4 4 41 

Alert Shift Technical Advisor X X X X X X X X 

Alert Emergency Preparedness X X ----- X X X 

Organization 

Alert Dispatch Center -- X X X -- X X X 

Call Radiation Personnel r X ..... -

Overall rating: 34332445 Overall rating: 3422314 35 Overall rating: 1 4 3 2 35 3 1 

Key 

Rating Scale: 1= minimal performance, 5=optimal performance 

X = task completed 

-- = task not completed 

Crews 1-4: Crews 5-8: 

Conventional Plant Advanced Plant 

Crews 1 & 2: 4-man crews Crews 5 & 6: 4-man 

Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man

E-2

NUREG/IA-0137
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

Table E-2 Ratings of transient management tasks on the Interfacing Systems Loss of Coolant scenario 

Announcements & Notifications Critical Task Completion Stabilization & Cooldown 

Crew 1 2 3 4 5 6 7 8 Crew 1 213141516178 Crew 1 2 3 4516 7 8 

Sound Fire Alarm --........- X X X X Identify Leak in Primary 4 1 3 1 2 1 2 4 Handling of Cooldown 3 4 1 11313 4 4 

Circuit 

Alert Fire Brigade Verification of Pump 2 1 3 3 2 3 4 3 Primary Pressure 3 4 1 21514 5 4 

Stop 

Make Fire Announcement Investigation of Other 3 2 2 2 4 3 2 4 Pressurizer Level 3 3 1 1 5 5 5 5 

Signals 

Declare Plant Emergency X X -- X X -- X Reactor Scram 1 1 3 5 3 3 4 5 Rate of Cooldown 3 4-- 4 4 5-

Notify Regulatory Agency X X X -- X -- X X Initiate Boration 4 4 4 4 4 4 4 -- Departure from Nucleate 4 4 --... 4 2 2 4 

Boiling 

Alert Shift Technical Advisor X X XXX -- X X Isolating Leak 4 1 2 1 .--........ 

Alert Emergency Preparedness X X ---- X X X X 

Organization 

Alert Dispatch Center -- X ..... X -.... X 

Call Radiation Personnel -- X X -- X I--I--I

Overall rating: 3 5 3 1 5 3 3 4 Overall rating: 2 1313 3 3 4 5 -Overall rating: 314 1 154 4 4 

Key: 

Rating Scale: 1= minimal performance, 5=optimal performance 

X = task completed 

-- = task not completed 

Crews 1-4: Crews 5-8: 

Conventional Plant Advanced Plant 

Crews 1 & 2: 4-man crews Crews 5 & 6: 4-man 

Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man

E-3

NUREG/IA-0137
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

Table E-3 Ratings of transient management tasks on the Loss of Feedwater scenario 

Announcements & Notifications Critical Task Completion Stabilization & Cooldown 

Crew 1 2314156718 Crew 1234 5 6 7 8 Crew 1 2 3 4 5 6 7 8 

Sound Fire Alarm ..--------- X X X X Identifying leak before 1 1 1 --------- Handling of cooldown 2 3 2 2 4 3 4 5 

pump trip __PrimaryPr 

Alert Fire Brigade ---------------------- Identifying leak location 1 1 1 1-- ---- Primary Pressure 2 3 2 2 4 4 4 4 

Make Fire Announcement ------------------ Isolating leak 2 2 2 1 1 2 2 1 Pressurizer Level 1 3 1 1 4 3 4 4 

Declare Plant Emergency -- X X---- X-- Restarting of RL-pumps 3 3 2 2 3 4 4 3 Cooldown gradient 

Notify Regulatory Agency X X X .-- - X X X Reactor scram 4 5 4 4 5 5 5 5 Departure from Nucleate 3 3 1 2 3 4 4 4 

Boiling 

Alert Shift Technical Advisor X X X X X X X X InitiateBoration 4 4 4 2 5 4 4 4 

Alert Emergency Preparedness -- X X -- X X X X Refilling of dry steam 1 3 1 2 3 4 4 5 

Organization generator(s) 

Alert Dispatch Center XlXl--X-- X X X 

Call Maintenance Personnel I--1---- ---

Overall rating: 2 5 5 1 3 4 4 5 0verallrating: 1*3 2 2 3 4 4 5 Overall rating: 2 3 2 2 4 4 4 5 

Key 

Rating Scale: 1= minimal performance, 5=optimal performance 

X = task completed 

-- = task not completed 

Crews 1-4: Crews 5-8: 

Conventional Plant Advanced Plant 

Crews 1 & 2: 4-man crews Crews 5 & 6: 4-man 

Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man

E-4

NUREGJIA-0137
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

Table E-4 Ratings of transient management tasks on the Loss of Off-Site Power scenario 

Announcements & Notifications Critical Task Completion Stabilization & Cooldown 

Crew 1 2 3 4 5 6 7 8 Crew 123456 7 8 Crew 1 2 6 

Make Fire Announcement --.--.--.--.--.--.--.-- Smooth Power Reduction 2 1 2 1 4 4 4 4 Handling of Cooldown 1 1 4 4 4 4 

Declare Plant Emergency -- X X ..... X X X Assessing LOOP, 2 1 2 1 4 4 3 3 Primary Pressure 1 111 2 4 4 4 

Attempting Back-up Power 

Sources 

Notify Regulatory Agency X X X ..... X X-- Reactor scram (from 5 1 5 1 5 5 5 5 Pressurizer Level 2 1 3 4 4 4 

protection vs. by hand) 

Alert Shift Technical X X X -- X X X X Initiate Boration 3 1 3 1 3 5 4 4 Rate of Cooldown 1 111 3 3 3 3 

Advisor I 

Alert Emergency X X X ..... X X X Identify andAttemptto 2 1 2 1 5 5 5 5 Departure from 2 1 2 1 3 3 3 3 

Preparedness Close Stuck-Open Safety Nucleate Boiling 

Organization Relief Valve I 

Alert Dispatch Center -- X X X -- X X X Minimize Loss Through 2 1 2 1 5 5 4 2 

Valve: Stop Pump and 

Close Valve 

Call Radiation Personnel -------------k.. .--

Overall rating: 3 4 4 1 1 5 5 3 Overallrating: 2 1 2 1 4 5 4 4 Overall rating: 2 1213 4 4 4 

Key: 

Rating Scale: 1= minimal performance, 5=optimal performance 

X = task completed 

-- = task not completed 

Crews 1-4: Crews 5-8: 

Conventional Plant Advanced Plant 

Crews 1 & 2: 4-man Crews 5 & 6: 4-man 

Crews'3 & 4: 3-man Crews 7 & 8: 2-man

E-5

NUREG/IA-0137
APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

Table E-5 Ratings of transient management tasks on the Steam Generator Over-fill scenario 

Announcements and Notifications Critical Task Completion Stabilization and Cooldown 

Crew 1 2 3 4 5 6 7 8 Crew 1 213 4 5 6 7 8 Crew 1 213 4 5 6 7 8 

Make Fire Announcement .................- Identify and Attempt to Close Stuck- --.--.-- 1 4 4 4 4 Handling of Cooldown 4 4 4 4 4 4 4 4 

Open Valve 

Declare Plant Emergency Manually Closing Alternative 1 4 4 4 4 Primary Pressure 4 2 2 2 5 4 4 4 

Valves 

Notify Regulatory Agency X X X ------- X Reactor Scram 2-- 1 1 4 4 3 4 Pressurizer Level 4 3 3 3 3 3 3 3 

Alert Shift Technical Advisor X X X X X X X X Stopping Feed Water Pumps 2 2 1 1 1 4 2 4 Rate of Cooldown I .... 

Alert Emergency Preparedness -- X X --------- Stop Overfilling by Separating 2 2 1 1 -- 4 4 Departure from Nucleate 4 4 4 4 4 4 4 4 

Organization Feedwater Lines Boiling 

Alert Dispatch Center Initiate Boration F-- I----

Call Maintenance Personnel -- X X ...... X -- Prevent Safety Relief Valve From 1 2 2 1 3 4 1 4 

Hammering 

Overall rating: 2 4 41 1 1 2 1 2 Overall rating: 2 2 1 1 3 4 3 4 Overall rating: 4 21 2 2 4 33 33 

Key 

Rating Scale: 1= minimal performance, 5=optimal performance 

X = task completed 

-- = task not completed 

Crews 1-4: Crews 5-8: 

Conventional Plant Advanced Plant 

Crews I & 2: 4-man crews Crews 5 & 6: 4-man crews 

Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man crews

E-6

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

F-1

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-1 Summary table of situation awareness data: scenario type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Scenario Type Mean I N I SD Mean I N SD Mean I N [SD Mean I N [SD Mean N SD 

Rule-based 0.83 66 0.14 0.69 66 0.14 0.65 66 0.24 0.77 66 0.15 0.65 41 0.22 

Knowledge-based 0.66 44 0.24 0.69 44 0.19 0.69 44 0.19 0.68 44 0.26 0.90 22 0.06 

All Scenarios 0.76 110 0.21 10.69 110 0.16 0.67 110 0.22 10.73 110 0.21 0.74 63* 0.22 

*Indicates a loss of 3 records from the data set. This was a 3 person (minimum crew size) crew from the conventional plant in the last scenario period of the Loss of Feedwater scenario. 

No parametric statistical analyses used data from this fifth scenario period.

Table F-2 Summary table of situation awareness data: crew size by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

CrewSize Mean I N ISD Mean I N I SD Mean I N I SD Mean I N I SD Mean N SD 

Normal 0.75 60 0.23 0.69 60 0.17 0.64 60 0.26 0.76 60 0.20 0.76 36 0.20 

Minimum 0.78 50 0.17 0.70 50 0.15 0.70 50 0.17 0.69 50 0.21 0.71 27 0.25 

All Groups 0.76 110 0.21 0.69 110 0.16 10.67 110 0.22 0.73 110 0.21 10.74 63* 0.22 

Table F-3 Summary table of situation awareness data: plant type by scenario period 

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Plant Type Mean N ISD Mean I N I SD Mean N SD Mean IN ISD Mean I N ISD 

Conventional 0.79 60 0.22 0.71 60 0.15 0.71 60 0.17 0.72 60 0.19 0.77 33 0.21 

Advanced 0.73 50 0.19 0.67 50 0.17 0.62 50 0.27 0.75 50 0.23 0.70 30 0.23 

All Groups 0.76 110 0.21 0.69 110 0.16 0.67 110 0.22 10.73 110 0.21 0.74 63* 0.22

F-2

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-4 Summary table of situation awareness data: crew size and plant type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Plant Type Mean N SD Mean I N SD Mean N SD Mean I N SD Mean N SD 

Normal Conventional 0.81 30 0.23 0.74 30 0.11 0.75 30 0.15 0.79 30 0.14 0.80 18 0.15 

Normal Advanced 0.69 30 0.22 0.64 30 0.21 0.53 30 0.29 0.73 30 0.25 0.72 18 0.24 

Minimum Conventional 0.77 30 0.21 0.69 30 0.18 0.67 30 0.17 0.65 30 0.22 0.74 15 0.26 

Minimum Advanced 0.79 20 0.09 0.71 20 0.10 0.76 20 0.14 0.76 20 0.20 0.67 12 0.24 

AllGroups 1 0.76 110 0.21 0.69 110 0.16 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22 

Table F-5 Summary table of situation awareness data: position and plant type by scenario period 

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Operator Plant Type Mean I N SD Mean I N I SD Mean I N ISD Mean I N I SD Mean I N SD 

CRS Conventional 0.84 20 0.14 0.72 20 0.17 0.70 20 0.18 0.74 20 0.12 0.81 11 0.14 

CRS Advanced 0.73 10 0.14 0.66 10 0.16 0.62 10 0.19 0.79 10 0.09 0.78 6 0.14 

RO Conventional 0.73 20 0.29 0.68 20 0.17 0.70 20 0.17 0.70 20 0.24 0.80 11 0.14 

RO Advanced 0.78 15 0.11 0.68 15 0.18 0.61 15 0.28 0.66 15 0.35 0.64 9 0.27 

BOP Conventional 0.81 20 0.19 0.74 20 0.10 0.73 20 0.16 0.72 20 0.20 0.71 11 0.30 

BOP Advanced 0.64 15 0.28 0.64 15 0.22 0.52 15 0.34 0.79 15 0.11 0.71 9 0.27 

Dual Role Advanced 0.79 10 0.07 0,70 10 0.10 0.79 10 0.07 0.76 10 0.21 0.69 6 0.21 

CRS 

AllGroups 0.76 110 0.21 .69 110 0.16 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22

F-3

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-6 Summary table of situation awareness data: crew size, plant type, and scenario type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Plant Type Scenario Type Mean N SD MeanI N I SD Mean N SD MeanI N I SD Mean N I SD 

Normal Conventional Rule 0.91 18 0.13 0.73 18 0.11 0.74 18 0.18 0.79 18 0.14 0.74 12 0.15 

Normal Conventional Knowledge 0.66 12 0.27 0.74 12 0.11 0.77 12 0.11 0.80 12 0.13 0.94 6 0.01 

Normal Advanced Rule 0.74 18 0.14 0.63 18 0.17 0.52 18 0.33 0.79 18 0.15 0.61 12 0.23 

Normal Advanced Knowledge 0.62 12 0.30 0.65 12 0.26 0.54 12 0.23 0.65 12 0.33 0.92 6 0.05 

Minimum Conventional Rule 0.87 18 0.13 0.73 18 0.16 0.65 18 0.18 0.69 18 0.17 0.66 9 0.31 

Minimum Conventional Knowledge 0.62 12 0.22 0.63 12 0.20 0.70 12 0.17 0.59 12 0.26 0.85 6 0.06 

Minimum Advanced Rule 0.80 12 0.08 0.68 12 0.09 0.74 12 0.17 0.81 12 0.10 0.54 8 0.17 

Minimum Advanced Knowledge 0.77 8 0.11 0.76 8 0.10 0.79 8 0.08 0.70 8 0.28 0.93 4 0.06 

AllGroups 0.76 110 0.21 0.69 110 .016 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22 

Table F-7 Summary table of subjective workload data: scenario by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Scenario Mean I N SD_ MeanI N SD MeanI N TSD Mean I N SD 

SGTR 23.36 22 10.14 46.15 22 13.23 49.64 22 8.62 50.14 22 12.38 47.63 19 10.35 

ISLOCA 37.09 22 12.85 43.04 22 15.84 44.42 22 13.49 43.87 22 13.26 42.16 22 14.40 

LOOP 18.22 22 10.24 41.89 22 12.85 46.88 22 13.46 48.61 22 10.57 45.34 22 11.23 

LOFW 20.76 22 10.29 38.67 22 14.15 50.41 22 13.48 51.84 22 12.59 -- 0 0.00 

SGOF 26.17 22 11.98 33.22 22 11.90 42.56 22 12.41 45.67 22 15.49 41.67 3 10.14 

All Scenarios 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07 

Table F-8 Summary Table of Subjective Workload data: Crew Size by Scenario Period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Mean I N ISD Mean N ISD Mean I N ISD Mean I N -SD MeanI N .1 SD 

Normal 23.40 60 12.43 38.17 60 13.94 46.09 60 12.93 46.82 60 13.61 42.18 39 12.86 

Minimum 27.18 50 13.00 43.50 50 13.91 47.61 50 12.20 49.48 50 12.32 48.52 27 9.90 

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

F-4

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-9 Summary table of subjective workload data: plant type by scenario period 

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

PlantType Mean I N I SD Mean I N I SD Mean I N I SD Mean N FSD Mean I N I SD 

Conventional 22.98 60 12.97 34.80 60 11.98 41.91 60 11.63 41.29 60 10.30 38.21 33 8.97 

Advanced 27.68 50 12.16 47.55 50 13.42 52.63 50 11.17 56.12 50 11.34 51.34 33 11.26 

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07 

Table F-10 Summary table of subjective workload data: position by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Position Meanl N I SD Mean 7-N K SD Mean I N I SD MeanI N ISD Mean N SD 

CRS 26.73 40 14.77 45.13 40 14.22 50.94 40 12.67 52.21 40 12.11 45.03 24 10.31 

RO 24.74 35 12.22 39.06 35 16.44 45.93 35 12.91 46.57 35 13.90 47.42 21 14.29 

TO 23.66 35 10.86 36.95 35 9.83 42.89 35 10.92 44.70 35 12.26 41.83 21 11.43 

All Groups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-11 Summary table of subjective workload data: position by plant type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Position Plant Type Mean N SD Mean I N I SD Mean I N SD Mean N SD Mean I N ISD 

CRS Conventional 28.50 20 16.07 42.25 20 11.97 47.25 20 12.51 46.42 20 10.61 39.70 11 8.29 

CRS Advanced 24.96 20 13.54 48.00 20 15.95 54.63 20 12.01 58.00 20 10.86 49.55 13 9.91 

RO Conventional 20.13 20 11.12 29.27 20 10.63 38.46 20 8.31 39.00 20 10.39 39.17 11 8.50 

RO Advanced 30.89 15 11.12 52.11 15 13.54 55.89 15 11.17 56.67 15 11.46 56.50 10 14.08 

BOP Conventional 20.33 20 9.63 32.88 20 9.74 40.02 20 12.17 38.44 20 8,19 35.76 11 10.29 

BOP Advanced 28.11 15 11.10 42.39 15 7.12 46.71 15 7.83 53.06 15 11.96 48.50 10 8.84 

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

F-5

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-12 Summary table of workload data: crew size by plant type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Plant Type Mean I N ISD Mean I N ISD MeanI N SD Mean N SD Mean I N ISD 

Normal Conventional 18.67 30 10.22 31.18 30 11.12 40.71 30 12.86 39.36 30 10.94 33.75 18 6.32 

Normal Advanced 28.14 30 12.78 45.17 30 13.06 51.47 30 10.72 54.28 30 11.91 49.40 21 12.71 

Minimum Conventional 27.30 30 14.12 38.42 30 11.88 43.11 30 10.33 43.21 30 9.41 43.56 15 8.89 

Minimum Advanced 27.00 20 11.46 51.13 20 13.48 54.37 20 11.87 58.87 20 10.07 54.72 12 7.46 

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07 

Table F-13 Summary table of subjective workload data: crew size by position by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Operator Mean I N I SD Mean I N S MSDean N SD Mean I N Mean IN SD 

Normal CRS 19.96 20 10.94 38.42 20 12.23 46.54 20 11.85 49.08 20 13.63 39.68 13 9.95 

Normal RO 24.33 20 13.82 39.10 20 17.89 47.75 20 14.19 46.92 20 16.34 45.38 13 17.32 

Normal BOP 25.92 20 12.19 37.00 20 11.47 43.98 20 13.02 44.46 20 10.52 41.47 13 10.30 

Minimum CRS 33.50 20 15.23 51.83 20 13.06 55.33 20 12.17 55.33 20 9.74 51.36 11 6.66 

Minimum RO 25.28 15 10.13 39.00 15 14.89 43.50 15 10.97 46.11 15 10.30 50.73 8 6.98 

Minimum BOP 20.66 15 8.22 36.89 15 7.50 41.42 15 7.44 45.03 15 14.64 42.40 8 13.83 

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07 

Table F-14 Summary table of team interaction data: scenario by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Scenario Mean I N ISD Mean I N ]SD MeanI N I SD Mean I N I SD Mean I N ISD. 

SGTR 5.20 16 1.05 5.59 16 0,88 5.56 16 1.23 4.55 16 1.56 4.44 14 1.38 

ISLOCA 4.72 16 1.28 4.72 16 0.98 4.98 16 1.07 4.83 16 1,11 4.54 16 1.28 

LOFW 4.84 16 0.98 5.46 16 0.83 5.06 16 1.12 5.22 16 1.21 4.80 16 1.16 

LOOP 5.04 16 1.17 4.61 16 1.22 4.38 16 1.60 4.43 16 1.37 -- 0 -

SGOF 5.02 16 1.00 5.25 16 0.84 5.36 16 0.88 4.85 16 1.01 -- 0 -

All Scenarios 4.97 90 1.09 5.13 90 1.02 5.07 90 1.24 4.78 90 1.26 4,60 46 1.25

F-6

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-15 Summary table of team interaction data: crew size by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Mean N SD Mean I N I SD Mean I N ISD Mean I N I SD Mean I N J'SD 

Normal 5.05 40 0.94 5.14 40 0.98 5.36 40 1.09 4.79 40 1.14 4.59 24 1.17 

Minimum 4.87 40 1.22 5.11 40 1.07 4.78 40 1.33 4.76 40 1.39 4.61 22 1.36 

All Groups 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25 

Table F-16 Summary table of team interaction data: plant type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Plant Type Mean I N ISD Mean N SD Mean N SD Mean N SD Mean N SD 

Conventional 4.62 40 1.00 4.86 40 1.13 4.56 40 1.39 4.24 40 1.34 3.84 22 1.13 

Advanced 5.32 40 1.07 5.39 40 0.83 5.57 40 0.82 5.31 40 0.93 5.30 24 0.92 

All Groups , 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25

Table F-17 Summary table of team interaction data: crew size and plant type by scenario period 

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 

Crew Size Plant Type Mean I N I SD Mean I N ISD Mean I N SD Mean I N [SD Mean I N I SD 

Normal Conventional 4.75 20 0.99 5.01 20 1.22 5.19 20 1.41 4.47 20 1.42 3.98 12 1.23 

Normal Advanced 5.36 20 0.80 5.28 20 0.65 5.52 20 0.63 5.11 20 0.66 5.20 12 0.73 

Minimum Conventional 4.48 20 1.01 4.72 20 1.03 3.93 20 1.06 4.01 20 1.25 3.67 10 1.03 

Minimum Advanced 5.27 20 1.31 5.50 20 0.97 5.63 20 0.99 5.51 20 1.11 5.40 12 1.10 

All Groups 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25

F-7

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-18 Summary table of rated crew performance data by scenario 

Scenario Mean N SD 

SGTR 6.97 23 2.03 

ISLOCA 6.95 23 1.52 

LOFW 7.50 24 1.60 

LOOP 6.78 24 2.26 

SGOF 7.19 24 1.62 

All Scenarios 7.08 118 1.81 

Table F-19 Summary table of rated crew performance data: crew size by plant type 

Crew Size Plant Type Mean N SD 

Normal Conventional 7.30 30 1.47 

Normal Advanced 7.47 30 1.28 

Minimum Conventional 5.57 30 2.01 

Minimum Advanced 8.04 28 1.46 

All Groups 7.08 118 1.81

F-8

NUREG/IA-0137
APPENDIX F: DESCRIPTIVE STATISTICS

Table F-20 Summary table of objective performance rating data by scenario 

Announcements and Notifications Important Task Performance Cooldown and Stabilization 

Scenario Mean N SD Mean N SD Mean N SD 

SGTR 3.63 8 0.74 3.13 8 1.55 2.75 8 1.39 

ISLOCA 3.38 8 1.30 3.00 8 1.20 3.25 8 1.49 

LOFW 3.63 8 1.51 3.00 8 1.31 3.25 8 1.16 

LOOP 3.25 8 1.58 2.88 8 1.55 2.63 8 1.30 

SGOF 2.13 8 1.25 2.50 8 1.20 2.88 8 0.83 

All Scenarios 3.20 40 1.36 2.90 40 1.32 2.95 40 1.22 

Table F-21 Summary table of objective performance rating data: crew size and plant type by scenario 

Announcements and Notifications Important Task Performance Cooldown and Stabilization 

Crew Size Plant Type Mean N SD Mean N SD Mean N SD 

Minimum Conventional 2.60 10 1.51 1.80 10 0.79 1.80 10 0.63 

Normal Conventional 3.50 10 1.08 2.00 10 0.94 2.60 10 1.17 

Minimum Advanced 3.60 10 1.35 4.00 10 0.94 3.50 10 1.08 

Normal Advanced 3.10 10 1.45 3.80 10 0.79 3.90 10 0.74 

All Groups 3.20 40 1.36 2.90 40 1.32 2.95 40 1,22

F-9

NUREG/IA-0137
NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER 

(2-89) (Assigned by NRC, Add Vol., Supp., Rev., 

NRCM 1102, and Addendum Numbers, If any.) 

3201,3202 BIBLIOGRAPHIC DATA SHEET 

(See instnuctions on the reverse) 

2. TITLE AND SUBTITLE NUREG/IA-0137 

A Study of Control Room Staffing Levels for Advanced Reactors 

3. DATE REPORT PUBLISHED 

MONTH I YEAR 

November 2000 

4. FIN OR GRANT NUMBER 

5. AUTHOR(S) 6. TYPE OF REPORT 

B. P. Hallbert, A. L. Sebok, D. Morisseau Technical 

7. PERIOD COVERED (inclusive Dates) 

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (If NRC, provide Division. Office or Region, U.S. Nuclear Regulatory Commission. and mailing address. if contractor, 

provide name and mailing address.) 

Institutt for Energiteknikk 

Kjell Hangset 

P.O. Box 173, N-1751 Halden 

NORWAY

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (If NRC, type "Same as above'; ifcontractor, provide NRC Division, Office or Region, U.S. Nuclear Regulatory Commission, 

and mailing address.)

Divsion of Systems Analysis and Regulatory Effectiveness 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

10. SUPPLEMENTARY NOTES 

11. ABSTRACT (200 words or less) 

This report documents the results of an empirical study of operator and plant performance in simulator based settings. The 

simulator settings were designed to be representative of conventional and advanced plants. The advanced plant design 

employed passive systems. The control room architectures were also designed to represent both plant types. Two control 

room staffing configurations were employed in each plant setting: a staffing configuration reflecting the requirements of 10 CFR 

50.54 (m); and a staffing configuration that involved a reduced number of control room operators. A series of five design basis 

scenarios, relevant for both conventional and advanced passive plants were chosen to evaluate the effects of plant type and 

crew size on operator performance. The scenarios were: 1) steam generator tube rupture with a stuck open steam generator 

safety relief valve in the affected steam generator, preceded by a fire in the turbine hall; 2) interfacing systems loss of coolant 

accident with compounded instrument failures due to the incident; 3) sustained total loss of feedwater; 4) loss of off-site power 

with a single steam generator safety relief valve stuck open, and; 5) steam generator overfill. Eight crews of operators from the 

Loviisa nuclear power station in Loviisa, Finland participated in the study: four crews in the conventional plant setting; and four 

crews in the advanced plant setting. Measures of objective performance were obtained, including ratings of crew performance 

and transient management. Measures of operator workload, situation awareness, and team interaction were obtained 

repeatedly during each scenario. The findings of the study revealed a number of effects of crew size and plant type, and their 

combination on operator performance. This report documents the study and discusses the implications and issues raised by 

this performance-based evaluation of control room staffing requirements for advanced passive plants.

12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers in locating the report.) 13. AVAILABILITY STATEMENT 

OECD Halden unlimited 

human factors 14. SECURITY CLASSIFICATION 

control room (This Page) 

crew size unclassified 

human performance (This Report) 

unclassified 

15. NUMBER OF PAGES 

16. PRICE

NRC FORM 335 (2-89) This form was electronically produced by Elite Federal Forms, Inc.
kpaPrinted 

1n recycled• 

pFaper 

Federal Recycling Program
UNITED STATES 

NUCLEAR REGULATORY COMMISSION 

WASHINGTON, D.C. 20555-0001

years