NUREGICR-6749 

BNL-NUREG-5263 8 

Integrating Digital and 

Conventional Human-System 

Interfaces: Lessons Learned from 

a Control Room Modernization 

Program 

Brookhaven National Laboratory 

U.S. Nuclear Regulatory Commission 

Office of Nuclear Regulatory Research 

Washington, DC 20555-0001 
AVAILABILITY OF REFERENCE MATERIALS 

IN NRC PUBLICATIONS 

NRC Reference Material 

As of November 1999, you may electronically access 

NUREG-series publications and other NRC records at 

NRC's Public Electronic Reading Room at 

http://www.nrc.qov/readinq-rm. html. Publicly released 

records include, to name a few, NUREG-series 

publications; Federal Register notices; applicant, 

licensee, and vendor documents and correspondence: 

NRC correspondence and internal memoranda; 

bulletins and information notices; inspection and 

investigative reports: licensee event reports; and 

Sommission papers and their attachments. 

VRC publications in the NUREG series, NRC 

sgulations, and Tifle 70, €negy, in the Code of 

%detal Regulations may also be purchased from one 

if these two sources. 

I. The Superintendent of Documents 

US. Government Printing Office 

Mail Stop SSOP 

Washington, DC 2040240011 

Internet: bookstore.gpo.gov 

Telephone: 202-592-1 800 

Fax: 202-51 2-2250 

!. The National Technical Information Sewice 

Springfield, \/A 22161-0002 

w.ntis.gov 

1-800-55343847 or, locally, 7034OWOOO 

L single copy of each NRC draft report for comment is 

vailable free, to the extent of supply, upon written 

zquest as follows: 

iddress: Office of the Chief Information Officer, 

Reproduction and Distribution 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

Services Section 

-mail: DISTRIBUTION@nrc.gov 

acsimile: 30141 5-2289 

iome publications in the NUREG series that are 

iosted at NRC's Web site address 

ittp://www.nrc.qov/readinq-rm/doc-collections/nureqs 

ire updated periodically and may differ from the last 

lrinted version. Although references to material found 

n a Web site bear the date the material was accessed, 

le material available on the date cited may 

ubsequently be removed from the site. 

Non-NRC Reference Material 

Documents available from public and special technica 

libraries include all open literature items, such as 

books, journal articles, and transactions, Federal 

Register notices, Federal and State legislation, and 

mgressional reports. Such documents as theses, 

jissertations, foreign reports and translations, and 

ion-NRC conference proceedings may be purchased 

'rom their sponsoring organization. 

2opies of industry codes and standards used in a 

substantive manner in the NRC regulatory process arc 

naintained atThe NRC Technical Library 

Two White Flint North 

11545 Rockville Pike 

Rockville, MD 20852-2738 

-hese standards are available in the library for 

eference use by the public. Codes and standards are 

rsually copyrighted and may be purchased from the 

iriginating organization or, if they are American 

tlakional Standards, hmAmerican National Standards Institute 

9 1 West 42nd Street 

New Ysh, NY 1003643002 

w.ansi.org 

2124424900 

Legally binding regulatory requirements are stated 

only in laws; NRC regulations; licenses, including 

technical specifications: or orders, not in 

NUREG-series publications. The views expressed 

in contractor-prepared publications in this series are 

not necessarily those of the NRC. 

The NUREG series comprises (1) technical and 

administrative reports and books prepared by the 

staff (NUREGX)OUO or agency contractors 

(NUREGICR-XXXX), (2) proceedings of 

conferences (NUREGKP-XX>Oo, (3) reports 

resulting from international agreements 

(NUREGAA-)(XXX), (4) brochures 

(NUREGIBR-)0000, and (5) compilations of legal 

decisions and orders of the Commission and Atomic 

and Safely Licensing Boards and of Directors' 

decisions under Section 2.206 of NRC's regulations 

(NUREG-0750). 

r 

BISCWMER This report was prepared as an account of work sponsored by an agency of the US. Government. 

Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or 

implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any 

information, apparatus, product, or process disclosed in this publication, or represents that its use by such third I party would not infringe privately owned rights. 
NUREG/CR-6749 

BNL-NUREG-52638 

Integrating Digital and 

Conventional Human-System 

Interfaces: Lessons Learned from 

a Control Room Modernization 

Program 

- 

Manuscript Completed: August 2001 

Date Published: September 2002 

Prepared by 

Emilie Roth, Roth Cognitive Engineering 

John O’Hara, Brookhaven National Laboratory 

Principid Contractor: 

Brookhaven National Laboratory 

Energy Sciences and Technology Department 

Upton, NY 11973-5000 

Subcontractor: 

Roth Cognitive Engineering 

89 Rawson Road 

Brookline, MA 02445-4509 

Paul M. Lewis, NRC Project Manager 

Prepared for 

Divisioin of Systems Analysis and Regulatory Effectiveness 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

NRC Job Code W6546 
ABSTRACT 

This study examined the impact of introducing advanced human-system interfaces (HSI) into a 

conventional nuclear power plant control room. The advanced HSIs include a computer-based 

procedure system, an advanced alarm system, and a graphic-based plant information display 

system. The impact of the new systems on the cognitive functioning of individual crew members 

and on the structure and functioning of the crew as a team was examined. Information on crew 

performance was obtained by observing crews during full-scope training simulations of plant 

disturbances. In addition, interviews were conducted with operators and other utility and vendor 

personnel. The general findings were that the new HSIs provided positive support for crew 

performance, reduced workload, and were well accepted by the crews. One of the more 

interesting and significant effects introduced by the advanced HSI systems was on crew structure 

and communication. The computer-based procedure system enabled the shift supervisor to 

access plant state information directly, reducing the need to ask board operators for plant 

parameter values. In turn, the board operators had access to a richer source of plant state 

information via the advanced alarm system and graphic-based plant information system than 

previously was available, enabling them to monitor plant state more broadly. These changes 

have potential implications for human performance and reliability. The fact that crews use 

multiple independent sources of information and multiple independent perspectives may increase 

the crew reliability by increasing the probability of detecting and correcting errors in situation 

assessment, thus reducing the potential for errors of intention. However, the improvement in 

human performance and reliability depends on the crew’s ability to effectively communicate and 

maintain a shared situation awareness. The study provides illustrative cases where successful 

performance is dependent on effective communication among crew members. While this study 

began exploring these issues, further research is required to establish how new HSIs affect crew 

structure, communication, decision-making and reliability. The lessons learned from this 

investigation will be used to support human factors guidance development addressing these 

topics. 

... 

111 
CONTENTS 

& 

... ABSTRACT ................................................................. m 

ACKNOWLEDGMENTS ...................................................... Vii 

ACRONYMS ................................................................ ix 

1 INTRODUCTION .......................................................... 1 

2 OBJECTIVE .............................................................. 3 

3 METHODOLOGY ......................................................... 7 

3.1 Participants ....................................................... 7 

3.2 Simulator ......................................................... 7 

3.3 Advanced HSIs ................................................... -8 

3.3.1 Alarmsystem ............................................... 8 

3.3.2 Computer-Based Procedure System ............................... 9 

3.3.3 Graphic Display System ....................................... 11 

3.3.4 Operations Philosophy and Philosophy of Use of the HSI Systems ....... 12 

3.4 Interview Questionnaire ............................................. 13 

3.5 StudyProcedures .................................................. 14 

3.5.1 Simulator Exercises and Observations ............................. 14 

3.5.2 Interviews ................................................. 15 

4 RESULTS ............................................................... 17 

4.1 Observations ..................................................... 17 

4.1.1 UseoftheAAS ............................................. 17 

4.1.2 Use ofthe CBP ............................................. 17 

4.1.3 Use of the Graphic Display System ............................... 18 

4.1.4 Utilization of Available VDU Display Real Estate and Coordination of HSI 

Systems ................................................... 19 

4.1.5 Crew Structure and Division of Responsibility ....................... 20 

4.1.6 Transitioning fiom CBP to Paper EOP ............................ 20 

4.2 IllustrativeCases .................................................. 21 

4.2.1 Cases Where Crews Correctly Chose to Redirect the CBP ............ 22 

4.2.2 A Case Illustrating Importance of Good Crew Communication and Crew 

SituationAwareness ......................................... 26 

4.3 Operator Interviews ................................................ 28 

4.3.1 Advanced Alarm System ...................................... 28 

4.3.2 Computer-Based Procedures ................................... 32 

4.3.3 Graphic Display System ....................................... 34 

V 
CONTENTS 

(continued) 

4.3.4 VDU Display Real Estate ...................................... 36 

4.3.5 Confidence in Software ....................................... 37 

4.3.6 Impact on Crew Performance .................................. -38 

4.3.7 Training ................................................... 43 

Interviews with Head of Operations and Head of Training ................... 44 

5 DISCUSSION ............................................................ 47 

4.4 

5.1 Overall Jmpact of Introduction of Computerized HSIs ...................... 47 

5.2 AlarmSystems .................................................. -48 

5.3 Computer-Based Procedures ......................................... 50 

Graphic Display System ............................................. 54 

Confidence in Software ............................................. 55 

Impact on Operations and Crew Structure ............................... 56 

Implications for Training ............................................. 58 

5.4 

5.5 

5.6 

5.7 

6 CONCLUSIONS AND RECOMMENDATIONS ................................ 61 

7 REFERENCES ........................................................... 63 

APPENDIXA ............................................................. A-1 

APPENDIXB .............................................................. B-1 

vi 
ACKNOWLEDGMENTS 

The authors wish to express their sincere gratitude to the Westinghouse design personnel and 

Nordostschweizerische Kraftwerke AG (NOK) utility personnel who granted us permission to 

conduct this study and who gave generously of their time and patience while it was being 

conducted. Specifically we would like to thank Garry Brassart, Jim Easter, Me1 Lipner, 

Craig Watson, and Mike Halverson of Westinghouse and Fritz Portmann and Eberhard Wyrsch 

of NOK. Special thanks is given to the utility operators who provided insightful comments and 

perspectives on the new systems and to the vendor’s training instructor whose knowledge and 

understanding significantly contributed to the study. 

Thanks are also extended to ow NRC colleagues Joel Kraxner and Paul Lewis who made it 

possible to conduct the study and who provided thoughtful comments on the development, 

conduct, and interpretation of the study. 

Finally, we wish to thank Mary Anne Corwin for her technical editing and improvement of the 

report. 

vii 
ACRONYMS 

AAS 

BNL 

CBP 

CRT 

EOP 

HFE 

HSI 

I&C 

GDS 

LOCA 

MSIV 

NPP 

NRC 

PWR 

RCS 

RNO 

SG 

SI 

SRO 

VDU 

V&V 

advanced alarm system 

Brookhaven National Laboratory 

computer-based procedure 

cathode ray tube 

emergency operating procedure 

human factors engineering 

human-system interface 

instrumentation and control 

graphic display system 

loss of coolant accident 

main steam isolation valve 

nuclear power plant 

U.S. Nuclear Regulatory Commission 

pressurized water reactor 

reactor coolant system 

response not obtained 

steam generator 

safety injection 

senior reactor operator 

video display unit 

verification and validation 

ix 
1 INTRODUCTION 

The Human-System Interface Design Review Guideline (NUREG-0700, Revision 1) 

(O'Hara, Brown, Stubler, Wachtel, and Persensky, 1996) was developed to provide human 

factors engineering (HFE) guidance to the U.S. Nuclear Regulatory Commission (NRC). 

It describes those aspects of the human-system interface (HSI) design review process that are 

important to the identification and resolution of human engineering discrepancies that could 

adversely affect plant safety. NUREG-0700 also provides detailed HFE guidelines for the 

assessment of HSI design implementations. 

In the development of NUREG-0700, Rev. 1 , several topics were identified as "gaps" because 

there was an insufficient technical basis upon which to develop guidance. One such topic is the 

integration of advanced HSI technology into conventional nuclear power plants (NPPs). The 

NRC is currently sponsoring research at Brookhaven National Laboratory (BNL) to: (1) better 

define the effects of changes to plant HSIs brought about by the application of digital technology 

on personnel performance and plant safety and (2) develop human factors engineering (HFE) 

guidance to support safety reviews in the event that a review of plant modifications involving a 

safety significant aspect of HSIs is necessary. Such guidance is currently being integrated into 

NUREG-0700 and will be used to provide NRC staff with the technical basis to help ensure that 

the modifications do not compromise safety. 

Based upon literature, interviews, and site visits, changes in HSI technology and the potential 

effects those changes have on personnel performance were identified (O'Hara, Stubler, and 

Higgins, 1996). The topics were then evaluated and prioritized with respect to their potential 

safety significance (Stubler, Higgins, and OWara, 1996). Based on this analysis, several topics 

were included in those selected for guidance development, including computer-based procedures 

(CBPs) and information design and organization. In addition, the NRC has developed guidance 

for advanced alarm systems (Brown, O'Hara, and Higgins, 2000) and interface management. 

The purpose of the study described in this report was to develop lessons learned related to these 

topics based upon a study of the integration of advanced HSIs into a NPP that currently uses 

primarily conventional HSIs. In connection with a computer replacement, the Swiss utility 

Nordostschweizerische Kraftwerke AG (NOK), modernized the control rooms at Beznau. 

The two units are two loop, 350 MWe pressurized water reactors that began operation in the late 

1960s and early 1970s. The modifications were designed by Westinghouse and include a CBP 

system, an advanced alarm system (AAS), and a graphic based plant display system. Operating 

crews were trained on the use of these systems at the Westinghouse Waltz Mill Site in Madison, 

Pennsylvania. The training was conducted in a full-scope dynamic plant simulator and involved 

using the computerized HSI to respond to dynamic plant disturbances. The control room 

modernization project represented a unique data collection opportunity to address human 

performance issues associated with regard to the new HSI technologies and but also any related 

issues such as the design, evaluation, and implementation of the new systems. The latter are 

critical to the successful introduction of new technology. 

1 
1 INTRODUCTION 

The information obtained was based on observations of crews during their initial training with 

the new systems and interviews with operators and other utility and vendor personnel. The 

lessons learned will be used to support guidance development in the topic areas. 

2 
2 OBJECTIVE 

The objective of this study was to obtain information and lessons learned regarding a wide range 

of topics related to the integration of advanced HSI technology with conventional technology in a 

control room modernization program, and more particularly, to identify the strengths and 

weaknesses (opportunities for improvement) of the specific technologies: 

Computer-based procedure system 

Alarm system 

Display system 

Systems-integration across the new systems and with the existing HSIs 

In addition, information on the following general topic areas was sought: 

Impact on operator training 

.Configuration, maintenance, and upgrade 

Scope and objectives of the HSI modernization program 

Design goals, basis, methods, and evaluation 

Implementation and transition to the new systems 

Impact on crew operations, staffing, qualifications, communication, and coordination 

An area of particular interest in the study was the impact of the new HSIs on crew performance. 

Early discussions with designers of the HSIs and plant management suggested that the 

combination of new HSIs strongly impacted how the operators functioned as a crew. The CBP 

system was developed for the plant’s emergency operating procedures (EOPs). It provided the 

shift supervisor with the plant parameter data required for him to work through the procedures. 

This had two direct effects: (1) it reduced the need for ‘low level’ communication between shift 

supervisor and the board operators that was previously required to provide the shift supervisor 

the plant parameter data called out in the procedures; and, (2) as a consequence, it freed the board 

operators’ cognitive resources. The board operators no longer needed to serve as the ’eyes’ of the 

shift supervisor. It should be noted that these CBP-related observations may reflect the fact that 

it was EOPs that were computerized and not some simpler type of procedure. 

The AAS and graphic display system (GDS) also created new sources of information that, if 

utilized, could improve the crew’s understanding of, and response to, plant conditions. 

The utility, recognizing the opportunities afforded by the new computerized system, instituted a 

change in crew structure and distribution of responsibilities. Instead of the traditional crew 

structure, where board operators only take action at the direction of the shift supervisor and 

function essentially as one cognitive unit, the utility decided to allow the board operators to work 

independently and in parallel with the shift supervisor. For the immediate post-trip actions, the 

shift supervisor works through the procedures, while the board operators assess plant conditions 

based on the AAS (as well as the graphical display system and traditional board indicators). 

3 
2 OBJECTIVE 

We were interested in exploring the impact of the new HSIs and crew structure on crew 

performance. The new crew structure has the potential to increase the reliability and 

effectiveness of the crew. This is because the board operators, by working independently from 

the shift supervisor (using different information sources and methods), may provide an 

additional, independent (redundant and diverse) check on plant state and appropriateness of 

response actions. However, the improvement in crew performance and reliability depend on 

effective communication among crew members. Paper-based procedures necessitate 

communication between the shift supervisor and the board operators. By providing plant 

parameter data directly to the shift supervisor, the CBP eliminates that forcing function. The 

question is, how do the new HSIs impact crew communication and, in turn, the ability of the 

crew to maintain a shared understanding of the state of the plant and of the procedures. 

It is possible that the CBP could improve communication and the ability of the crew to maintain 

a high level of situation awareness by shifting the content of communication to a higher level. 

With paper-based procedures the focus of communication is on the status of individual parameter 

values. It is possible that CBP could shift the content of communication to a higher level that 

more directly addresses the status of the plant relative to the goals of the procedures and 

achievement and maintenance of safety functions. It is also possible that by removing the forcing 

function to communicate, the CBP results in reduced communication and a consequent loss of 

situation awareness among crew members. A reduction in communication per se is not a signal 

of degraded team performance. It is possible that more effective teams, in some cases, actually 

talk less than less effective teams (Orasanu and Fischer, 1992; Orasanu, 1994). The key is that 

the more effective teams know what information needs to be passed to their team-mates and 

when. One of the goals of this study was to explore the impact of the new HSIs, particularly the 

CBP, on crew communication. 

We were interested in obtaining the operator’s perspective on the impact of the new HSIs on 

(1) cognitive load, (2) roles and functions, and (3) communication among team members. In 

addition we were interested in how the changes in cognitive load, operator roles, and 

communication affected the situation awareness of each of the crew members and the quality of 

team decision-making. 

We were also interested in the impact of the new HSIs on the ability of crews to work through 

emergency response procedures in pace with dynamically evolving emergency scenarios. 

Of particular interest was the impact of the new HSIs on the ability of crews to monitor the 

effectiveness of the emergency procedures in handling emergency scenarios as well as the ability 

of the crews to detect and respond to cases where the actions specified in the emergency response 

procedures are not fully appropriate to the specific situation. Several studies examining both 

actual and simulated incidents have shown that conditions sometimes arise where response 

guidance in the procedures are not fully appropriate to the situation (Kauffman, Lanik, Trager, 

4 
2 OBJECTIVE 

and Spence, 1992; Wreathall, Reason, and Dougherty, 1993; Roth, Mumaw, and Lewis, 1994). 

In those cases the ability of the crews to recognize that the actions specified in the procedures are 

not fully appropriate to the specific plant conditions and to take corrective action are important 

cognitive activities. Roth, Mumaw, and Lewis (1994) showed that with paper-based emergency 

response procedures, operator crews actively monitor the appropriateness of procedural steps to 

the evolving plant conditions and occasionally will deviate from the literal interpretation of a 

procedural step when it is determined to be necessary to achieve the goals intended by the 

procedural step and/or to maintain plant safety. We were interested in determining how CBP 

affected operators ability to monitor the effectiveness of the procedures, and detect and respond 

to situations where the actions specified by a procedural step were not fully appropriate to the 

situation. 

It is important to point out the limitations and constraints of the study. First, the observations 

were made during the first training week using the new systems. Thus, we were observing 

operators using the HSIs early in their learning curve. Second, the systems were not completely 

debugged at the time the study was conducted. Thus many of the specific problems encountered 

by the crews were known to management and were in the process of being corrected. Third, the 

scope of the scenarios was limited to relatively straightforward events from a cognitive 

perspective. Since the operators were being exposed to the HSIs for the first time, the training 

staff felt that for pedagogical reasons, it was important to use simulator scenarios that were 

straightforward and that and exercised a core set of procedures. Thus, use of the HSIs was not 

observed in the full range of complex situations that could arise. 

In addition, as will be discussed in the next section, the study relied on interviews and naturalistic 

observational methods. No systematic manipulations of HSI features and characteristics were 

attempted and no quantitative assessment of performance was obtained. 

The main objective of the study was to capture lessons learned from the introduction of these 

new HSIs to serve as a source of input to designers, implementers, trainers and evaluators of new 

HSIs. A goal of the study was to gain an understanding of operators’ perception of the 

computerized HSIs relative to traditional hardwired control boards. We were interested in 

gathering information on which aspects of the new HSIs the operators felt were clear 

improvements over traditional hardwired control boards as well as which areas they felt 

introduced new challenges or raised new issues. The study aimed to provide information on 

what features of the new computerized HSIs appeared to be most beneficial to the operators, to 

identify areas that designers, implementers and evaluators of new HSIs need to pay careful 

attention to, and to identify areas where further research is required. 

5 
3 METHODOLOGY 

3.1 Participants 

The main participants in the study were five crews of professional NOK operators observed and 

interviewed between July and September of 1997. The crews varied in size between four and 

six. Each contained minimally a shift supervisor, reactor operator, a turbinehalance-of-plant 

operator, and shift engineer. 

The observations and interviews took place in a full-scope training simulator. The operators 

were at the simulator for a week of refresher training. 

In addition to the observation and interviews of the crews, interviews were conducted with: 

Utility head of operations 

Utility head of training 

Vendor project manager 

Principal training instructor 

Principal designer of the CBP system 

Principal designer of the AAS 

Principal designer of the graphic display system 

3.2 Simulator 

The simulator was Westinghouse’s full-scope training simulator of a pressurized water reactor 

(PWR) in Madison, Pennsylvania. The control room was modified to include the new CBP, 

AAS, and display systems. 

In addition to the standard control board with analog indicators, there was an Alarm Overview 

Panel made up of vacuum-fluorescent display devices installed at the top of the control board for 

presentation of advanced alarm messages. 

There were also five workstations distributed around the control room. Each workstation had 

two visual display units (VDUs) for a total of 10 VDUs. One pair of VDUs was at the shift 

supervisor’s desk. A second pair was located in the back of the room and was intended for use by 

the shift engineer. The three remaining pairs of VDU were located by the control board and were 

intended to be used by the board operators. The total number of VDUs was the same as in the 

actual control room. 

7 
3 METHODOLOGY 

3.3 Advanced HSIs 

A brief description of each of the three advanced HSIs follows (references to additional 

information are provided). While the systems were mature enough to support initial training, it 

should be noted that the CBP and AAS were still in the final stages of implementation and, so, 

were still undergoing fine-tuning and corrections to the databases. 

3.3.1 Alarm System 

The AAS is composed of two main components: (1) an Overview Panel and (2) a Support Panel. 

The Overview Panel is composed of a set of alarm message windows presented at the top of the 

control board. The alarm message windows are vacuum-fluorescent eighty-character alpha 

numeric display devices that display computer-generated alarm text messages. The alarm 

windows are analogous to engraved tile light boxes of current annunciator systems, but allow for 

changes in the text presented in the alarm window. 

The alarm system attempts to preserve some of the strengths of a conventional tile-based 

annunciator system, namely the value of having multiple, parallel-presented, dedicated position, 

alarm windows, while taking advantage of the flexibility afforded by use of computer-generated 

alarm text. 

The Overview Panel is composed of 254 alarm message windows. This is approximately 

one-third to one-fourth the number of tiles in the usual annunciator tile system. At the time of 

this study, the database for the AAS contained a total of approximately 6000 distinct alarm 

messages. The alarm messages are grouped and assigned to alarm windows on the Overview 

Panel based on a plant function organization scheme. 

Only one alarm message can be displayed in an alarm window at a time. However, it is possible 

for more than one alarm message associated with a given alarm window to be active at the same 

time. Therefore, a prioritization scheme was defined to determine which alarm message will be 

displayed in an Overview Panel window when more than one alarm message is active. The 

alarms not displayed are stored in a queue of active alarm messages associated with a given 

Overview Panel alarm window. If there are alarm messages in the queue, a symbol appears in 

the alarm message window to alert the operators that queued messages exist. The lower-priority 

alarm messages in the queue can be accessed from the Support Panel. 

The AAS consisted of Westinghouse’s advanced alarm system called AWARE. Additional 

information regarding the AWARE system design can be found elsewhere (Camera et al., 1996; 

Easter and Lot, 1991; Levett et al., 1995). A key feature of this AAS that distinguishes it from 

other computer-based alarm systems is that it is organized functionally. The alarm layout is 

8 
3 METHODOLOGY 

based on a functional goal-means decomposition of the plant (Rasmussen, 1986). Further, 

prioritization among alarm messages is only performed within narrowly-defined queues of alarms 

that all relate to the same plant function. No attempt is made to prioritize alarms across 

functions. This contrasts with many other computerized alarm systems that assign each alarm a 

predefined indication of urgency for operator action, with some alarms always coded as "high" 

urgency for action and other alarms always coded as "low" urgency. In this alarm system, 

operators do not have to consciously consider relative alarm priority. The alarms that appear in 

the alarm windows at any given point in time are expected to be addressed by the operators. 

The Support Panel is displayed on the dual-headed VDU workstations. One display provides a 

reduced version of the Overview Panel that allows the user to see in which areas there are alarms. 

From the Support Panel, operators can also access all active alarms in a queue. They can 

examine the list of alarm messages whose priority did not permit them to be displayed on the 

Overview Panel or that have been recently "bumped" off the Overview Panel by messages of 

higher priority. They can also see the logic behind any alarm message, the set-points and inputs 

to the logic, and various parsings of the chronological darm message list. 

In addition to the AAS, there were two other alarm systems available to operators. One was the 

original tile-based alarm system that was implemented at the time the plant was built. The tiles 

are typical of conventional alarm tiles that are organized into matrices by plant functions and 

systems. 

The third alarm system encompassed a set of "base" alarms that came with the plant computer 

upgrade as a standard feature. This alarm system is a VDU message list display that is 

chronologically organized. It is accessible from the two-headed workstation VDUs. While the 

alarm processing and alarm display capabilities are more sophisticated in the AAS, the base 

alarm system had greater coverage in that it contained alarm setpoints associated with every plant 

parameter on the plant data highway. 

3.3.2 Computer-Based Procedure System 

The EOPs were implemented in Westinghouse's Computerized Procedure (COMPRO) system. 

Additional information regarding the COMPRO system design can be found elsewhere 

(Lipner, 1994; Lipner and Kerch, 1994; Lipner and Orendi, 1992). 

The CBP is based on the standard PWR two-column format EOPs. The CBP uses the written 

procedures as the basis for textual displays. Prompts guide the operator step-by-step through the 

EOPs. 

The CBP is connected to the plant computer database. As a result, it is able to present the 

operator with current plant parameters. Key features of the CBP include: 

9 
3 METHODOLOGY 

Resolution of procedural step logic for the operator and indication of whether a 

high-level step is satisfied or violated; 

Presentation of the detailed sub-steps and supporting plant parameter data that 

provide the basis for whether the high-level step is satisfied or violated; 

If a high level step is violated, the contingent step information is presented in the 

adjacent column (ie., the "Response Not Obtained" column); 

Indication of when criteria to transition to another procedure have been met and 

provision of a link to the new procedure; 

Continuously monitoring of plant parameters associated with the following procedure 

elements, and alerts the operator when associated criteria are satisfied 

- Fold-out pages 

- Critical safety functions 

- Continuous action steps 

- Notes and Cautions 

These functions are performed by the operator in paper procedure use and impose additional 

monitoring and memory demands. 

The CBP is completely user-paced and user-directed. It is user-paced in the sense that the system 

only advances to the next step at the prompting of the user. It is user-directed in the sense that it 

is the user who has ultimate control and responsibility for how the procedures are utilized. 

While, the CBP indicates whether a procedural step is satisfied or violated, and what action this 

implies (e.g., taking a control action at the board, following the steps in the Response Not 

Obtained column, transitioning to another procedure) the operator can always override the CBP. 

While the CBP presents procedural steps sequentially, there are navigation facilities that allow 

the operator to get to any step in any EOP procedure at any time. 

The CBP operates on a double monitor workstation. The procedural information appears on the 

first VDU and supporting or supplementary procedural information appears on the second VDU 

(e.g., graphs, critical safety-function status trees). The procedural information VDU is divided 

into four areas, which appear from top to bottom as follows: 

Pull down menus 

Parallel information 

Current procedure information 

User prompts 

10 
3 METHODOLOGY 

The current procedure information area provides the status of the current high level step and the 

supporting sub-steps. The two-column format of the EOPs is preserved. If the high level step is 

violated, then the contingency actions in the 'response not obtained' (RNO) column appear. If the 

high level step is satisfied, the information in the "0' column is not displayed unless the 

operator requests to see it. 

The parallel information area is used to display information that needs to be monitored in parallel 

(e.g., status of the critical safety functions, fold-out page criteria, and cautions). 

The user prompt area is the primary navigation mechanism. Phrases such as 'continue to Step X' 

or Transition to Procedure X Step Y'are displayed in this area to indicate the recommended next 

movement within the EOPs. The user moves through the EOP steps by clicking in the 'prompt' 

area. 

The pull down menu provides a mechanism for accessing supplemental infomation as well as a 

mechanism for navigating to any procedural step in any procedure at any point in time. 

The CBP was used by the shift supervisor who was the senior reactor operator (SRO) on the 

crew. Since the CBP was connected to the plant computer, the shift supervisor could go through 

the procedure without having to ask board operators for the values of plant parameters. 

There was no automatic control of plant equipment by the CBP and no soft controls associated 

with either the CBP or the Graphic Display System. Control actions could only be made by 

operators at the control board. Thus, when the procedures specified control actions to be taken, 

the shift supervisor had to instruct the operators to carry them out. 

3.3.3 Graphic Display System 

The graphic display system contains a variety of display types including: 

Graphic process displays that provide plant parameter information organized around 

plant system mimics 

Predefined trend displays of key plant parameters 

Operator-defined tailored trend plots of chosen plant parameters 

Additional information regarding the general approach to the display system design can be found 

elsewhere (Easter and Watson, 1997). 

11 
3 METHODOLOGY 

There are several navigation mechanisms to bring up displays of interest including: 

Hierarchically organized pull-down menus 

Pokefields that allow direct access from one display to another related display by 

clicking on a pokefield within the first display 

Paging (up/down/left/right) between displays 

Another key feature of the Graphic Display System is that it includes zoom and pan capabilities. 

This allows the operators to ‘blow-up’ areas of interest to see more detail or to make it visible 

from farther away. 

The Graphic Display System is accessible from any of the workstation VDUs. It is implemented 

in a windowing system that allows operators to bring up multiple displays in different windows 

on a given VDU. The size and location of the display windows can be manipulated and 

overlapped. Operators can switch between displays and retrieve those that were hidden. 

3.3.4 Operations Philosophy and Philosophy of Use of the HSI Systems 

The introduction of computer-based HSI systems into the control room changed the amount of 

plant state information available to operators. It also changed the accessibility of the information 

to different members of the crew. In particular, CBP provided the shift supervisor with the plant 

state information required to work through the procedural steps. It was no longer necessary for 

the shift supervisor to ask the board operators to read plant parameter indications off the control 

board and say them aloud in order for the shift supervisor to determine whether a procedural step 

is satisfied or violated. The CBP provided that information directly. 

This change in technology introduces the opportunity to change the traditional roles assigned to 

crew members in responding to emergencies. This became apparent to plant management during 

the simulator testing of the HSI systems that occurred several months prior to the beginning of 

the training period. At the time of the study, plant management was in the process of developing 

a philosophy of use for the HSI systems. 

At the time our observations were conducted, the philosophy of use that had been developed and 

trained was as follows: 

Following a trip, the shift supervisor was to utilize the CBP to work through the EOP 

steps. He did not need to ask the board operators for plant state information. He 

could work independently of the board operators until a control action was needed, at 

which point he would ask the board operators to take the actions. Minimal guidance 

12 
3 METHODOLOGY 

was provided with respect to requirements to communicate progress through the 

EOPs to the board operators. In general, the shift operator was asked to call out 

"important steps" including any procedure transitions. 

During the first EOP immediately post-trip (the Reactor Trip or Safety Injection 

procedure referred to as E-0), the board operators were told to focus on monitoring 

the AAS. The AAS provided alarm messages if any of the automatic safety 

equipment that was supposed to turn on following a trip did not come on. The 

operators were told that if any alarms came in indicating that automatic systems had 

not come on as required, they should take appropriate manual control actions to 

recover the systems. They were not required to inform the shift supervisor or get his 

approval prior to taking the control actions. 

All operators were responsible for attending to all alarms. In general, it was 

understood that the reactor operator would tend to focus on reactor related alarms, and 

the balance-of-plant operator would tend to focus on secondary side alarms. In this 

stage of the introduction of the new systems, there was no specific rule about division 

of responsibilities. 

The shift engineer was responsible for monitoring the critical safety functions and 

maintaining a broad overview of plant state. His role was to provide advice to the 

shift supervisor and ensure that any procedure transitions that were made were 

appropriate to the plant situation. 

3.4 Interview Questionnaire 

A principal source of data was crew and other personnel responses to interview questions. .Prior 

to the actual data collection, an extensive list of interview topics was developed. The topic list 

was based on general findings and issues identified in an earlier phase of the project (O'Hara, 

Stubler, and Higgins, 1996), the general experience of the authors, and comments on a draft set 

of questions by utility, vendor and sponsor personnel. In addition, background discussions were 

held with the vendor personnel to gain a better understanding of the control room modernization 

program and the detailed design considerations of the individual HSI systems. 

This complete list of topics was broad and addressed detailed considerations of each area 

identified in the study objectives. This list and the detailed considerations under each topic is 

contained in Appendix A. Addressing such diverse topics required interviews with a range of 

both vendor and utility personnel. The specific subset of questions addressed in interviews with 

operators is contained in Appendix B. 

13 
3 METHODOLOGY 

3.5 Study Procedures 

3.5.1 Simulator Exercises and Observations 

The crews were observed and interviewed during their regularly-scheduled week of 

refresher-training on a high-fidelity training simulator. This one-week training session was the 

first time the operators were being trained on the use of the AAS and CBP in a high fidelity plant 

simulator.' The operators were already familiar with the Graphic Display System since it had 

been installed in the plant for several years and was routinely used. 

During the training week operators utilized the computerized HSIs to respond to simulated 

accident scenarios. For pedagogical reasons all the accident scenarios run during the first week 

of training with the new HSIs were designed to be familiar events that exercised a core set of 

procedures and were straightforward to diagnose and respond to. 

Each of the crews was observed over a two-day period. Typically, one observer attended the 

training session on the third and fourth day of the training week. This allowed the operators to 

gain some familiarity in the use of the HSI systems prior to the observation sessions. 

Each crew was observed in four training exercise scenarios: A steam line break outside 

containment, a steam generator tube rupture, a small break loss of coolant accident (LOCA), and 

a loss of all feedwater event. Each scenario typically included a main malfunction that led to a 

reactor trip as well as several additional small complicating malfunctions (e. g., a controlling 

Pressurizer Pressure channel failed high). 

Following a trip, the shift supervisor was responsible for using the CBP to work through the 

EOP. The board operators were responsible for monitoring and responding to the AAS and 

taking control actions at the board. 

The shift engineer played a role equivalent to that of a Shift Technical Advisor. He was typically 

called out of the control room prior to the reactor trip (so that he would not be aware of the early 

evolution of the event) and then called back in approximately 10 minutes into the event. 

For each crew, one scenario was included where the computer system failed and the operators 

had to transition from use of the new HSI systems to use of the analog HSIs only. The scenario 

was a small break LOCA. The crew used the CBP, AAS and Graphic Display System through 

Step 10 of E-0, the initial post-trip EOP procedure. At that point, all the computerized systems 

Prior to full-scope simulator training, the crews were given preliminary training on the basic design of 

the CBP and MS. This included both classroom and limited-scope simulator training. 

14 
3 METHODOLOGY 

were turned off and the crew was told that they had lost the plant computer. In the case of the 

EOPs, the shift supervisor had to transition from the CBP to the paper procedures. 

Crews were observed unobtrusively from the instructor’s area. Although crews were aware of the 

observer’s presence, a one-way viewing capability allowed the observer to view crew 

performance during training scenarios without disruption or distraction to the crews. The 

observer came into the simulated control room during the post-scenario exercise debriefing when 

the training instructor discussed what went well and what could have been done better with the 

crews. The debriefing also included discussions of strengths and weaknesses of the 

computer-based HSI systems in supporting operator performance in the scenario. The observer 

posed clarifying questions during the debriefing. 

The observations were qualitative, i.e., no formal data collection was attempted. The 

observations provided the opportunity to see how the operators interacted with the new systems 

and each other. In addition, they provided the opportunity to observe and document cases that 

illustrate the kinds of complexities that can arise in accident situations and how the new HSIs 

impacted the ability of the crews to identify and respond to those complications. These notes 

often provided information to discuss during operator interviews. 

3.5.2 Interviews 

Operators were interviewed in crews. The crews were typically available for one hour at the end 

of each of the two days. Given this limited amount of time, no effort was made to address each 

question in Appendix B with each crew. Instead, for individual crews, questions were selected 

based on (1) the overall goal to address all questions across crews, (2) the unique events that 

occurred for a particular crew, and (3) the aspects of the new HSIs that the crews felt were 

important. 

The primary purpose of the interviews was to obtain the operators’ perspective on how they used 

the new HSI systems and how the new systems had affected their performance as individuals and 

as a team. Questions probed the perceived impact of the new systems on operator workload, 

situation awareness, distribution of tasks and responsibility among team members, and 

communication and coordination among the team members. 

Other topics that were covered included: 

Whether they felt the available display real estate was sufficient and how it was 

utilized 

The extent to which they understood and had confidence in the software processing 

15 
3 METHODOLOGY 

How they felt about the training they had received on the new systems and what 

additional training they felt would be desirable 

In general, the interviewer attempted to elicit reactions to the new HSI systems from the 

perspective of the shift supervisor, the board operators and the shift engineer. 

16 
4 RESULTS 

The observations of crew performance and crew interview comments provided complimentary 

and reinforcing perspectives on how the operators utilized the new HSIs. They provided the 

basis for drawing broader conclusions about the impact of HSI systems on operator performance, 

implications for training, and the technology introduction process. 

4.1 Observations 

In general, operators were able to handle the emergency scenario exercises and had no difficulty 

using the computerized HSIs to handle the events. 

4.1.1 Use of the AAS 

The board operators utilized the Overview Panel alarm messages to detect plant malfunctions. 

Consistent with their training on philosophy of use, the board operators focused on the AAS 

during E-0. If an alarm message came in indicating a failure of an automatic system, they took 

the appropriate manual backup action without getting approval from the shift supervisor. 

Consistent with training, all operators attended to all alarms on the Overview Panel. There 

seemed to be no clear division of labor with respect to which operators acknowledged which 

alarms. 

The Alarm Support Panel, available from the workstation VDUs, was generally not used by the 

board operators. Board operators generally did not check the Alarm Support Panel to see what 

lower-priority active alarms were present but not displayed on the Overview Panel. In the case of 

one of the crews, the turbinehalance-of-plant operator, who had a lower workload during the 

scenario, did have the Alarm Support Panel displayed on one of his VDUs and consulted it. 

The Alarm Support Panel was used mainly by the shift supervisor and the shift engineer. Shift 

supervisors typically consulted the alarm overview display or a display that provided a 

chronological listing of alarms. 

4.1.2 Use of the CBP 

In all the scenarios we observed, the shift supervisors were able to utilize the CBP to work 

through the EOPs with little trouble. They were able to keep pace with the event and transition 

smoothly between procedures. 

Prior work in human factors and human computer interaction (Elm and Woods, 1985; O'Hara, 

Stubler, and Higgins, 1996; Roth, Mumaw, and Stubler, 1992; Woods and Watts, 1997; Roth, 

Malin, and Schreckenghost, 1997) suggested that operators might become disoriented due to the 

17 
4 RESULTS 

narrow ‘keyhole’ provided by a computerized procedure as compared to paper (i.e., because they 

can see fewer steps at a time with computerized procedures than with paper) and/or that they 

might experience difficulties navigating among procedures (Le., the ’getting lost’ problem). No 

evidence of getting disoriented, losing place in the procedure, or having difficulty finding or 

navigating to other procedures was observed during the study. Shift supervisors appeared to be 

able to move readily within and across procedures. 

Cases were observed where the procedures (as presented by the CBP) gave operators the option 

to take actions or make procedure transitions that were inappropriate to the situation.2 In those 

instances, operators were able to detect that the indicated action was inappropriate and override 

the CBP. These cases are discussed in greater detail in the section on illustrative cases. 

4.1.3 Use of the Graphic Display System 

The main type of display that operators utilized during the emergency scenarios were trend 

graphics. Operators rarely selected system mimic displays (process displays). 

Operators tended to put up multiple trend plots on their VDUs. It was not unusual to see 

operators divide a VDU screen into two windows and place four trend plots in each. Since each 

trend plot could have up to four parameters trended, this meant that operators could have up to 

32 parameters trended per VDU. 

One crew was observed to put up a unique trend display that they had tailored themselves. The 

operators used the llzoomll feature to create a bold graphic display that allowed them to monitor a 

parameter trend from a distance. The parameter they were monitoring was Tave. They used one 

color trend (blue) to define the target value (Tref) and a second color trend (red) to define the 

parameter value (Tave). They modified the display so that the trends were represented not as thin 

lines but as solid color areas. As a result, the relative change in shape of the two colored areas 

provided an indication of the relative value Tave to Tref, and the change over time that could be 

easily seen and interpreted from a distance. 

The display created independently by the operators illustrates two interesting points. First, it 

provides an example of an ’ecological interface’ (Vicente and Rasmussen, 1992) that is designed 

to make emergent and perceptually salient information that is of importance to the user. In this 

case, the operators were interested in the difference between Tave and Tref, and how that 

difference changed over time. They exploited the relative shape of the colored areas for Tave 

and Tref to make the difference between the two values perceptually salient. Second, it provides 

These cases were similar to the types of situations that have been observed to arise with paper-based 

procedures (e.g., Roth, Mumaw & Lewis, 1994). 

18 
4 RESULTS 

an example of how users tailor computerized systems to their needs in ways that are not always 

anticipated by system designers. This phenomenon has been observed repeatedly (eg, Roth, 

Mumaw, Vicente, and Burns, 1997; Vicente, Mumaw, and Roth, 1997). Vicente has coined the 

term "finishing the design" to describe the phenomenon (Vicente, Mumaw, Roth, and Burns, 

1996). 

Operators rarely modified or pulled up new displays once a scenario was started. Rather, they 

tended to select a set of trend plots to display at the start of the session and kept those up 

throughout the session. This supports the point that operators are reluctant to engage in interface 

management tasks when workload gets high preferring instead to go with the information that is 

available rather than taking the time and effort to retrieve new information (O'Hara, Stubler, and 

Nasta, 1997). 

4.1.4 Utilization of Available VDU Display Real Estate and Coordination of HSI Systems 

The AAS, CBP and Graphic Display System all included displays that could be brought up on 

any of the workstation VDUs. One of the questions of interest was what displays different crew 

members chose to put up and how they utilized the VDU display real estate available to them. 

Operators generally packed the available VDU display real estate with information. Operators 

were observed putting up multiple windows per VDU, occasionally overlapping and/or covering 

up one display with another. Covering up information by overlapping or covering one display 

with another was not found to create any problems. 

As mentioned previously, board operators tended to put up multiple trend displays on their 

workstation VDUs. 

The shift supervisors put up the CBP on one VDU. The other VDU was used to display 

information on plant state. Typically, the shift supervisor would put up an alarm display in one 

window and some trend displays in a second overlapping window. 

The shift engineers, because of their broader range of responsibility, were observed to make the 

most intense use of the two VDU screens allocated to them. For example, in one case a shift 

engineer was observed displaying the CBP on one VDU so that he could monitor the critical 

safety functions (which are displayed at the top of the CBP screen). He placed two more 

windows on the same VDU that presented plant parameter trends. These two windows were 

overlapped on top of the CBP display so that the critical safety function display on the CBP was 

still visible. On the second display was a chronological list of alarms in one window, a set of 

additional trend plots in a second window, and an overview system mimic display. 

19 
4 RESULTS 

4.1.5 Crew Structure and Division of Responsibility 

Observation of operator performance during the scenarios indicated that they followed the 

operations philosophy presented during training; i.e. , during E-0, the shift supervisor and control 

board operators operated in parallel and largely independent of each other. When the board 

operators took manual action when alarm messages indicated that automatic systems had not 

come on as required, they generally informed the shift supervisor of the action they took, but this 

was not always the case (see critical incident described below). 

The shift supervisor worked through the procedures using the CBP. There appeared to be 

considerable variability between shift supervisors in the extent to which they communicated with 

the board operators as they worked through the procedure and what they communicated to them. 

One shift supervisor tended to read out loud all the high-level steps, indicating whether the step 

was satisfied or violated. Other shift supervisors only called out 'important steps' such as 

procedure transitions, steps where actions were to be taken , and the step goals (e.g., 'we are 

leaving this procedure because we have to stop safety injection'). A third style observed was a 

shift supervisor who tended to paraphrase and summarize procedural steps to keep the crew 

informed of his progress. He would state the problem (what is wrong), the solution path, and the 

current goal (of the portion of the procedure they were in). He would also read major procedural 

steps and procedure transitions. 

4.1.6 Transitioning from CBP to Paper EOP 

Each crew was observed during a scenario where a failure of the plant computer was simulated, 

so that all the computer systems failed and operators had to revert back to conventional board 

indicators and paper procedures. The training objectives of the scenario were to provide practice 

and to assess whether: 

The shift supervisor would pick up the paper-based EOPs and continue through the 

EOP steps from where he left off with the CBP 

The shift supervisor would pull out the fold-out page and monitor it in parallel with 

following the procedure in effect 

The shift engineer would pick up the paper status trees and begin to monitor the 

critical safety functions manually 

The board operators would coordinate with the shift supervisor to provide him the 

plant parameter information needed to follow the procedures that would have been 

provided by the CBP 

20 
4 RESULTS 

The crews successfully demonstrated all these behaviors. The operators made the transition 

smoothly and had little difficulty handling the events either before, during, or subsequent to 

losing the HSI systems. 

In the debriefing that immediately followed the scenario where the computerized systems were 

lost, operators indicated that they had no trouble responding to the scenario without the 

computerized procedure but that they did miss them. One crew elaborated on the features of the 

HSI systems that they missed. They indicated that the capability they missed the most was the 

trending capability of the Graphic Display System. Next they missed the CBP, especially its 

support with respect to automatic calculation of subcooling, automatic monitoring and alert of 

critical safety function, and automatic monitoring of foldout pages. One shift engineer indicated 

that he missed the AAS and CBP that helped him get quickly oriented when he is called in late in 

the event. These comments reflect the opinions of the crews in general. 

4.2 Illustrative Cases 

One of the areas of particular interest was the ability of crew members to maintain awareness of: 

The state of the plant 

The state of the procedure (i.e., where they are in the procedure and what they are 

trying to achieve) 

Whether the procedurally-directed actions are appropriate to the current situation 

Roth, Mumaw, and Lewis (1994) demonstrated that these are important cognitive activities that 

operators engage in when handling emergencies with paper procedures. An important question is 

whether the new computerized HSIs enhance, leave alone or interfere with these important 

cognitive functions of operators. 

The existing literature has raised concerns that CBPs might interfere with the ability of operators 

to follow the logic behind the procedure and detect situations where the procedures do not fully 

apply (e.g., Elm and Woods, 1985). Reasons given include that CBPs provide a narrower ‘key 

hole’ on the procedure (i.e., show fewer steps at a time) and that they potentially reduce 

communication among operators. In the observations, we looked for illustrative cases that 

provided evidence regarding whether or not the operators demonstrated the ability to monitor the 

appropriateness of the procedure to the current situation and overrode it in cases where it was 

inappropriate. 

Since the new HSIs were expected to impact communication among the crew, we also looked for 

cases that illustrated good communication as well as cases that illustrated breakdowns in 

21 
4 RESULTS 

communication and consequent loss in situation awareness. These cases help illuminate the 

impact of the new HSIs on team communication and team situation awareness. 

Illustrative cases related to these issues are discussed below. 

4.2.1 Cases Where Crews Correctly Chose to Redirect the CBP 

In the large majority of the scenarios observed, the CBP provided correct information and 

indicated actions for operators to take that were appropriate for the situation. In those cases, the 

shift supervisor worked through the procedure smoothly and the crew was able to handle the 

event effectively. 

Three cases were observed, however, where the CBP provided misleading information and/or 

specified inappropriate a~tion.~ In all those cases that we observed, the operators correctly 

detected that the indicated action was inappropriate to the situation and overrode it. This 

demonstrates the crews’ ability to maintain broad situation awareness, monitor the procedural 

path, and detect when an action indicated by a procedural step is not fully appropriate to the 

situation. It also demonstrates that the operators recognized that the CBP was a tool to support 

them and they were comfortable redirecting it when they judged it was appropriate. This is 

consistent with the design philosophy of the CBP. 

The three cases are briefly described below. 

Case 1: Steam Line Break Outside Containment Scenario 

This case illustrates a situation where operator judgment was required to override an action 

specified by the CBP, and they did so methodically and conservatively. 

When the crew got to the safety injection (SI) termination criteria step, the CBP indicated that the 

criteria for terminating SI was not met. The operators cycled through the procedure loop twice 

and the CBP still indicated that the criteria for terminating SI was not met. 

The shift engineer determined that it was appropriate to terminate SI based on indications from 

the Graphic Display System and the conventional plant HSIs. After discussion, the shift 

supervisor announced to the crew that they would terminate SI and that they could reinstate SI if 

it was needed. The crew double-checked that SI was not needed and then terminated it. They 

As is described below, these cases resulted from a literal interpretation of the paper-based procedures, 

and would occur with computer-based procedures as well if they are followed literally (cf. Roth, Mumaw, and 

Lewis, 1994). 

22 
4 RESULTS 

then moved through the CBP. Thus, this is a case where the crew correctly determined that SI 

should be terminated even though the procedure indicated that the literal SI termination criteria 

had not been fully met. 

The issue here was that the procedural step criteria asked if a parameter was ’stable or increasing.’ 

Since the plant was cooling down, the parameter was slightly decreasing (but the decrease was 

accounted for by the cooldown). The procedures (whether paper-based or computerized) do not 

consider all the plant data that the operators use in making their determination (e.g., the EOP 

does not take into account decay heat). 

As has been observed with paper-based procedures (Roth, Mumaw, and Lewis, 1994) some 

procedural steps require operator judgment to determine whether the step has been satisfied. In 

those cases, operators utilize their knowledge of the plant state and the intent of a procedural step 

in determining whether a procedural step is satisfied. Case 1 illustrates that the same types of 

situations can arise with CBPs and that operators are able to recognize those situations and 

redirect the procedure path just as they would with paper-based procedures. 

Case 2: Small Break LOCA 

This is a second case where the operators decided to redirect a procedural path. It provides 

another instance demonstrating the importance of operators’ to use multiple divergent sources of 

information as double-checks. In this case, they used Graphic Display System trends and board 

indications to independently evaluate the action specified by the CBP. It also illustrates the 

importance of crew communications and joint decision making in monitoring the appropriateness 

of the solution path and ensuring that the actions taken are appropriate. Further, it provides an 

instance demonstrating that situations can arise where the actions specified by the CBP are not 

fully appropriate given the local context. 

The crew was in Procedure ES-1.2 when they reached procedural Step 5 to initiate reactor 

coolant system (RCS) cooldown. By technical specifications, the system must not cool down at a 

rate greater than 55 degrees per hour. The CBP computed the cooldown rate over the last hour 

and determined that it was at a ramp of 55 degrees or less. Thus the CBP determined it was 

acceptable to initiate cooldown. The CBP resolved the step logic to indicate that the operators 

should dump steam to cool down. 

The shift supervisor asked the board operators to use the steam generators to cool down based on 

the CBP step. The board operators decided that this was not a good idea based on the trend of 

the current cooldown rate. They saw that the rate was greater than 60 degrees (when taken over a 

shorter period of time) and that they were already cooling down rapidly because of SI flow. 

Therefore, they decided not to increase cooldown rate. Although they had not yet exceeded the 

tech spec cooldown rate when averaged over an hour at that point in time, if they increased the 

cooldown rate, the resulting trajectory may have exceeded the tech spec cooldown rate when 

23 
4 RESULTS 

averaged over an hour at a later point in time? The crew discussed it, and decided to override the 

CBP and redirect the procedural path.5 

Interestingly, the training instructor indicated that other crews when coming across the same 

situation, followed the recommendation of the CBP and took action to increase the cooldown 

rate. In the instructor’s opinion, increasing the cooldown rate was the wrong action to take in that 

situation. While the observers in the study did not directly observe any cases of this type, the 

instructor’s comment suggests the existence of cases where operators failed to recognize a 

situation where they should have overridden a CBP recommendation. 

Case 3: Small Break LOCA 

This is a case where the CBP directed the crews to transition to a function restoration guideline 

for a spurious reason (a failed sensor). The crew recognized that they should not be in that 

procedure and exited it. It provides an additional example of the ability of the crews to utilize 

available control room resources to maintain broad situation awareness and provide an 

independent check on the appropriateness of procedural directives. 

In this scenario, a single channel failure from power range was failed high (sensor failure).6 

At a later point, the plant tripped due to the LOCA. When the operators went through the CBP, 

a critical safety function became red and the CBP indicated that the criteria had been met to 

transition to Procedure FR-S. 1, entitled ’Response from ECR to Nuclear Power Generation.’ 

The red path was spurious and strictly due to the failed sensor input. 

The shift supervisor transitioned to FR-S. 1. When in FR-S. 1 , a procedural step indicated to 

emergency borate. The crew recognized that emergency boration was not required and that they 

were in the wrong procedure. The crew recognized they did not have a critical safety function 

The CBP in computing cooldown rate, computed the rate over the past hour. This turns out to be an 

insensitive predictor of future cooldown rate. It may be that a more accurate predictor would result if a shorter 

temporal period were used (e.g., last 5 minutes), but it is also possible that more factors (e.g., what else is going on 

at the time, or is likely to happen next) need to be considered in projecting cooldown rate. 

cooldown rate. Most importantly, they consider current and upcoming activities that are likely to contribute to 

cooldown rate (e.g., SI injection will cool down the plant). The computerized procedure does not do this. There is 

often a great deal of human judgment involved in steps that on the surface seem like they would be straightforward 

to program. 

important to cool down slowly. 

malfunction is detected so that it does not effect the remainder of the scenario. On this occasion, the instructor 

didn‘t take the channel out of scan. 

As was demonstrated in this case, operators take more factors into account in predicting future 

This scenario has only one coolant pump running. If only one coolant pump is running it is especially 

Normally when this scenario is run, the channel is taken out of scan by the training instructor after the 

24 
4 RESULTS 

violation. They recognized that the red path was due to the failed channel and checked other 

parameters to establish definitively that the critical safety function was not violated. 

This case provides another illustration of the importance of the crew's understanding of plant 

state, the objectives of the procedures, and the consequences of actions in identifying situations 

where the actions specified by the procedure are inappropriate and should be overridden. 

During the debriefing, the crew also provided some evidence that there were difficulties in team 

communication, and in both individual and crew situation awareness during the scenario. The 

shift engineer indicated that he recognized all along that they were not really in a red path. But 

he did not realize that the crew was in FR-S. 1 until Step 4, when the procedure indicated that 

emergency boration should be initiated. This meant that the transition to FR-S. 1 had not been 

effectively communicated to the crew. 

There is also some evidence that the shift supervisor may have not been aware of the transition to 

FR-S. 1. What apparently happened was that the CBP, once out of E-0, automatically checked 

the status trees. It computed a red path.7 As a consequence, the button at the bottom of the 

screen normally used to go on to the next step switched to become the button to transition to 

FR-S. 1 .' It is possible that the shift supervisor transitioned to FR-S. 1 without realizing it. He 

may have just thought he was pressing the 'Next Step' button. The shift supervisor's recollection 

was that CBP 'automatically' transitioned to the FR-S. 1 procedure? 

It should be noted that the difficulties that the shift supervisor had in recognizing the transition to 

the FR-S. 1 ,may well have been due to lack of familiarity with the CBP interface since it was 

early in the training period. The main point of the case is not that the shift supervisor 

inadvertently transitioned to an inappropriate procedure but that the crew recognized they were in 

the wrong procedure and recovered. This is an important cognitive function that arises with 

paper-based procedures and needs to be preserved when shifting to the new computerized 

medium. 

The ability to detect and correct an inappropriate procedure transition depended on the 

knowledge of the board operators regarding plant state, and the appropriateness of procedural 

actions under different plant conditions. Most importantly, it depended on the ability of the 

It should be noted that the CBP does display the status of the critical safety function as a bar graphic. 

The critical safety function bars are coded for data quality. Had the shift supervisor noticed the data quality coding, 

he would have realized that the critical safety function "red path" criteria had not been met and was a function of 

poor data quality (i.e., the failed sensor). 

move on to the next step. The shift supervisor may not have noticed the difference, possibly due to the fact that it 

was early in the training process. 

The CBP does not automatically transition to another procedure, so this would not have been possible. 

' In actuality, the button to transition to another procedure differs in shape and position from the button to 

25 
4 RESULTS 

board operators to actively monitor the plant state and evaluate the appropriateness of actions 

requested by the shift supervisor. 

In summary, these three cases provide illustration of the operators’ ability to recognize when they 

are on a wrong procedural path and to take action to redirect the procedural path. This is an 

important cognitive function that has been observed with paper-based procedures and needs to be 

preserved when computerizing the procedures. 

These three cases reveal that: 

As in the case of paper-based procedures, situations can arise where the CBP provides 

misleading information and/or indicates inappropriate action. 

In these cases, operators are able to recognize that the CBP is off-track and redirect it. 

The operators double-check the CBP by utilizing the other control room systems 

(alarms, graphic displays, board indicators). 

Communication among crew members and team decision making are important 

contributors to the ability of crews to detect these situations and determine 

appropriate action. 

While only three cases were directly observed during the study, discussion with the training 

instructor indicated that these cases are not isolated examples; other similar cases have arisen 

with other crews. While in the three cases directly observed in the study, the crews correctly 

detected when the CBP was off track and redirected it. The training manager also indicated that 

there was at least one case where a crew did not detect that the CBP was off-track and followed 

the actions it specified. In that specific situation the actions were inappropriate. 

4.2.2 A Case Illustrating Importance of Good Crew Communication and Crew Situation 

Awareness 

In general, crews effectively communicated among themselves and maintained good situation 

awareness. That is, shift supervisors kept crews informed of progress through the procedures, 

and the board operators kept the shift supervisor informed of their actions. 

There was one case that arose during the study where a significant communication breakdown 

occurred. The board operators failed to communicate to the shift supervisor that a major 

malfunction had occurred and that they had taken an important action in response that had 

terminated the malfunction. As a result, the shift supervisor remained unaware throughout the 

scenario that a major malfunction had occurred that the crew had detected and corrected. Given 

26 
4 RESULTS 

that the malfunction had been corrected, the failure in communication was not a serious problem, 

but the case is discussed because it illustrates the importance of keeping everyone in the crew 

informed of major malfunctions, actions, and changes in plant state. 

In this event, a steam line break occurred outside containment. The board operators identified 

the break early in the scenario, partly based on the AAS alarms (the alarm message: 'SG Pressure 

Not Normal') and closed the main steam isolation valves (MSIVS).'~ As a result, when the CBP 

went through the E-0 procedure, everything looked normal. The shift supervisor was not told 

about the steam generator problem by the board operators and could not find out from the CBP 

because by the time the CBP reached the step that checked for this condition, the problem had 

been corrected. As a result, he did not see or hear about the steam line break and was not aware 

of it throughout the scenario. When the shift engineer arrived, he was able to detect that the 

steam line break had occurred based on the alarm history and the panels. 

This example illustrates the importance of good crew communication in ensuring accurate 

situation awareness on the part of all crew members. Neither the board operators nor the shift 

engineer informed the shift supervisor that a steam line break had occurred and had been 

terminated. This is an important piece of information that the shift supervisor should have been 

made aware of. 

This example highlights the need for board operators to be cognizant that they need to take an 

active role in keeping the shift supervisor aware of actions they take and major changes in plant 

state. Board operators need to keep the shift supervisor informed of plant changes and major 

actions in conventional control rooms as well. Effective communication on the part of the board 

operators becomes even more important in computerized control rooms where information may 

be more compartmentalized. For example, in this case, the primary information source for the 

shift supervisor was the CBP, and information about the steam line break and the corrective 

action taken was not available through that medium. 

The example also illustrates the value of having multiple sources of redundant information in the 

control room and multiple crew members observing the information from multiple perspectives. 

While no one informed the shift engineer that there had been a steam line break, he was able to 

personally assess the situation based on the alarm history and the board indications. In contrast, 

the shift supervisor, who had only access to the CBP, had no way of knowing that a steam line 

break had occurred and been terminated. 

lo Closing MSIVs early is a good idea in a steam break outside containment because of concern for the 

safety of people. Some crews close the MSIVs as soon as they get an SI, early in E-0. This is acceptable at this 

plant. ' 

27 
4 RESULTS 

4.3 Operator Interviews 

A summary of the operators’responses is provided below. Operator opinions on the individual 

HSI systems are presented first. In each case, the positive aspects of the computerized system 

that operators mentioned are presented first, followed by opportunities for improvements that the 

operators mentioned. The operator comments presented are generally close paraphrases of what 

the operators said. In some cases, we quote the operators’ own words. In those cases, we use 

quotation marks to indicate a direct quote. 

As pointed out in the introduction, the new HSIs had recently been installed at the plant and were 

still undergoing testing and fine tuning. Many of the limitations mentioned by the operators were 

known to the designers and utility management and were in the process of being addressed. 

Nevertheless, these limitations are worth documenting here because they represent the types of 

issues that can arise in the introduction of a new computerized system that should be addressed in 

design and evaluation of new systems. 

It should also be noted that some of the suggestions for improvement that were mentioned by 

operators and are reported below may not be practical or may not be the best design solution for 

dealing with the issue. The comments are reported because they serve as pointers to interface 

design issues that may be worth further exploration. 

4.3.1 Advanced Alarm System 

Positive Aspects 

Reduced Number of Alarms 

A positive feature repeatedly mentioned is that the AAS reduces the number of alarms relative to 

the original tile-based system implemented at the plant. 

Post-Trip Support 

A strength of the AAS that was repeatedly mentioned is that it provides alerts when automatic 

safety systems do not come on as expected following a reactor trip (e.g., failures in SI, failures in 

Reactor Trip, failures of the instrumentation and control (I&C) system, a pump problem, a valve 

problem). This allows the board operators to detect a problem and take corrective action 

promptly. 

28 
4 RESULTS 

Unexpected Event Detection 

It was felt that the AAS was most useful in aiding operators to detect when an event was not 

proceeding as expected (e.g., when automatic systems did not come on as expected, when there 

were secondary malfunctions or complications). As one operator put it, "It is a good help for the 

out of the ordinary." 

Messages are Specific 

The alarm system provides more specific alarm messages (eg, exactly what channel has failed) 

than was possible with the traditional tile system. Operators said this reduces the mental and 

physical activity required to localize a problem in comparison to the original tile based alarm 

system. As one operator stated, "It makes some connections that we had done before in our head. 

Before (in the case of a channel failure), we had to look at more parameters to determine the 

channel failure." 

Functional Organization 

Operators indicated that the functional organization of AAS is helpful. After a trip, they can 

focus on the center panels of AAS to get information on the reactor protection. 

An operator commented that the old tile-based alarm system was not organized in terms of goals, 

but operators still thought that way. Operators think in terms of plant protection goals. In 

contrast, the old alarm system was organized by physical location. The operator indicated that 

the AAS organization is very helpful. 

Understanding of Plant State 

Operators indicated that the AAS enhances their understanding of plant state. 

Opportunities for Improvement 

Number of Alarm Messages Can Be Reduced 

While operators said the AAS reduces the number of alarms relative to the original tile based 

system, they still felt there were too many alarms. Comments made were: 

In some cases, many alarm display windows light up at once, and it is hard to read 

them all. 

29 
4 RESULTS 

In some cases, there may be many messages in a queue and only one is presented in 

the alarm display window (the highest priority message). The operators indicated that 

they did not have time to go back and look at the other messages in the queue on the 

Support Panel. 

When many alarms come in, it is hard to figure out which is the newest alarm. 

Operators felt that the AAS logic needed to be refined to further reduce the number of alarms. It 

should be noted that the utility was in the process of doing this. 

Indications of Status and Expected Conditions Need Not Be Alarmed 

Operators felt that some of the alarm messages that come up on the Overview Panel are 'status 

indicators' that alert operators to conditions that are appropriate and expected given the event. 

They thought these alarms should not be presented. A repeated coment was that alarms should 

only be displayed on the Overview Panel to alert operators to unexpected abnormal conditions. 

Examples operators gave of unnecessary alarms include: 

When there is an SI, there is an alarm that comes in to indicate that two main 

feedwater pumps have tripped. The operators felt it was not necessary to have this 

alarm since they expect these pumps to trip on SI. 

An alarm message comes up saying that steam dumps are open in situations where the 

steam dumps should be open. In an operator's own words, "The operator knows he is 

using the steam dumps. The message should not come in. It is taking away time from 

the operator in order to read the message." 

When an operator shuts down a pump, they felt there does not need to be a message 

that indicates the pump was stopped. 

When auto makeup comes on, they felt they did not need an alarm. 

If there is high temperature in a system and blowdown actuates, they felt an alarm is 

not needed to indicate that it actuated. Operators indicated it would be better to create 

an alarm if blowdown does not actuate in this situation. 

Operators generally felt that while these types of messages provide information that, as one 

operator put it, 'is nice to know,' there are a lot of other messages that come on at the same time. 

In another operator's words, "The important messages can get lost among the less important 

messages." 

30 
4 RESULTS 

Operators repeatedly mentioned that systems that come on as expected, especially systems that 

are manually operated, should not result in alarm messages. One operator said, "An operator 

under high workload conditions does not want status messages telling him that systems are 

coming on as expected. He only wants to be alerted to the unexpected - if a system is supposed 

to come in and it doesn't." 

It should be noted that at this plant a distinction had been made between status messages and 

alarms, and the general intent was to place status messages on the Support Panel and only alarms 

on the Overview Panel. However, the design team that included operations staff had decided that 

some status messages were sufficiently important that they should appear on the Overview Panel. 

The operators interviewed appeared to disagree with this decision. 

Reading Alarm Messages 

While the Overview Panel provides some spatial dedication cues (messages on a particular 

function always appear in a specific display window), the operators pointed out that, unlike the 

traditional tile system, there is no absolute one-to-one space dedication of alarm message to 

position on the alarm panel. As a result, the operators have to read the message. They cannot 

rely on position of the window to infer the content of the alarm message. 

A second concern is that after a reactor trip, many messages are displayed in a window. When a 

higher priority alarm comes in, there is concern that the operator may not notice the new alarm or 

may forget the message that was displayed. 

Relation to Other Alarm Systems 

As noted earlier, three alarm systems are available in the control room. We asked operators what 

they felt the role of each alarm system was and when they used the different alarm systems. 

Operators indicated that the AAS was preferable to the original tile-based system in the case of 

major plant disturbances and post-trip situations. The perceived benefits of the AAS in 

comparison to the original tile-based system appeared to outweigh the perceived drawbacks. 

Interestingly, the chronological-list base-ala system was preferred for use during normal 

operations. Because the base alarm system contains set points on more plant parameters than the 

AAS, operators indicated that it is more sensitive and provides earlier alerts for the types of small 

problems that arise day-to-day (eg, minor equipment malfunctions, status changes in auxiliary 

equipment). However, as has been documented before, once a major plant malfunction occurs, 

the chronologically-organized, message-based alarm system becomes ineffective and is 

abandoned (O'Hara, Brown, Higgins, and Stubler, 1994). 

31 
4 RESULTS 

4.3.2 Computer-Based Procedures 

The CBP was used by the shift supervisor. It had a significant effect on the shift supervisor's task 

of working through the EOP. It also had a strong effect on crew communication and 

coordination because it reduced the need for the board operators to provide the shift supervisor 

with parameter values called out in the EOPs. 

In this section, we focus on the impact of the CBP on the shift supervisor. The impact of the 

CBP on the team is discussed in the later section on impact of computerized systems on crew 

performance. 

Positive Aspects 

Improved Speed and Accuracy 

Shift supervisors felt the CBP is a beneficial support system. They felt it allows them to work 

through the procedure faster and more accurately than with paper-based procedures. One shift 

supervisor commented that the CBP "gives you a lot of information without asking the operators. 

You know the information is correct, and it's faster because the operator doesn't have to run 

around." As a consequence, the crew is faster handling transients with CBP than with 

paper-based procedures. With some transients, such as a Tube Rupture, it is very important to 

get through the procedure quickly. 

Reduced Supervisor Cognitive Workload 

Shift supervisors noted that much of the 'mental overhead' associated with following paper 

procedures are handled by the CBP, thus reducing the shift supervisor's cognitive load. Among 

the things automatically done by the CBP that were mentioned by operators were: 

Plant checking is done automatically by CBP. Only if some checks fail is it necessary 

to go to the panel operators. With the paper version, the panel operators have to 

check every step. 

There are some difficult EOP steps (with complex logic); the CBP resolves the logic 

and identifies whether a step is satisfied or violated. 

The CBP keeps track of the status of critical safety functions, continuous action steps 

and fold-out page criteria. 

Shift supervisors noted that they missed having the CBP in the scenario where it is unavailable 

and they have to transition to paper procedures. 

32 
4 RESULTS 

Reduced Board Operator Cognitive Workload 

From the board operators' perspective, since they no longer have to serve as the eyes and ears for 

the shift supervisor, they can do their jobs more efficiently without being interrupted. 

Ouportunities for Immovement 

Opportunities to Streamline Wording 

Several operators felt that the CBP provides too much detail. They did not feel it was necessary 

to repeat all the detailed sub-steps that appear in paper-based procedures. There was a general 

consensus that the power of the computer has the advantage of providing results without having 

to show all the details that appear in the paper-based procedures. 

The primary concern was that there is too much to read. One operator pointed out that it was not 

necessary to transfer the entire wording contained in the paper procedures as was attempted in the 

CBP. The wording could be adapted to the computerized system. For example, if a step is 

satisfied, there is no need to present the detailed criteria; it is enough to indicate the status 

(e.g., Pumps running). It should be noted that the designers were careful to leave the wording of 

the procedures the same as the paper versions, in part to avoid changing their content which 

might have additional regulatory implications. 

Another operator suggested that the items that are phrased as questions in the paper procedures 

(eg, "Check that pressure is greater than XI') could be converted to statements in the CBP 

(eg, "Pressure is greater than XI'). In the present system, the phraseology of the paper-based 

procedure is preserved. 

Several operators noted that if a high-level step is satisfied, they do not feel it is necessary to read 

through all the sub-steps. They felt that, with the current CBP, they must page through too much 

to get through the sub-steps. 

They feel it is unnecessary to 'double-check' what the CBP is doing by reading the sub-steps 

because they assume it is processing the logic correctly. One operator noted, "You want to 

double-check with the Board, not just by reading the details of CBP." 

Wording Issues 

Whle the conventions used by the CBP to convert the paper procedures into computerized text 

generally worked well, operators noted that in some cases the resulting text is awkward or 

confusing. An example given by one operator is a step that is intended to have operators cool 

down the plant, preferably by using steam-dump valves. 

33 
4 RESULTS 

Narrow Field of View 

One shift supervisor commented that he could not look ahead as much with the CBP as he would 

like to, and normally can with paper procedures. He felt that, as a result, operators really have to 

know the procedures because it is not easy to look ahead. He noted that he would like to see a 

larger window dedicated to steps so that he could see more steps in parallel (both previous steps 

and future steps). 

Transition to Paper Procedures 

One shift supervisor expressed concern that with the CBP, the shift supervisor may be less likely 

to keep track of what procedure and step he is on. If it was necessary to shift to paper in 

midstream, the shift supervisor was concerned that he might not remember what step he had been 

on.'' 

Suggested Modifications 

Operators suggested changes or additions to the CBP as follows: 

Highlight the steps that the shift supervisor is supposed to announce to the crew 

There is added information that pops up in the parallel information window such as 

Cautions and Warnings that are important and should not be ignored. One operator 

felt the cautions and warnings would be less likely to be overlooked if they appeared 

in the main text window.I2 

4.3.3 Graphic Display System 

Unlike the CBP and AAS, the graphic display system has no specific phlosophy of use 

associated with emergency operations. There are no conditions under which operators are 

required to use it, nor specific guidance as to what displays should be used at specific times. 

It should be pointed out that all instances of shifts from computerized procedures to paper procedures 

l2 It should be pointed out that no instances were observed during the simulated training where cautions or 

that were observed during the simulated training showed no deficiencies. 

warnings were missed. In addition, the suggestion to move cautions and wanlings into the main text window was 

only mentioned by one operator. The comment is reported because operators missing cautions and warnings has 

been a chronic problem with paper-based procedures for which there has been no adequate solution. The computer 

offers a new medium that has the potential to reduce the problem of missed cautions and warnings. The operator's 

comment suggests that the appropriate timing and placement of cautions and warning in computer-based procedures 

may warrant further investigation. 

34 
4 RESULTS 

As a result, one of our main interests was to understand who used the graphic displays, what 

displays were used, and when these displays were used. 

Operators indicated that each operator will select the data that he needs on his VDUs and change 

it as necessary. During normal operations, the shift supervisor is likely to have the base 

(chronological list) alarm system, trends, and a system mimic display up on the VDUs. The 

board operators would typically put up trends on their VDUs. One operator noted that, during 

normal operations, each operator would typically have two trends displayed on one cathode ray 

tube (CRT) and two schematics displayed on the other CRT. 

Positive Aspects 

Value of Trends 

Operators indicated that trend displays are particularly useful and often used by the entire crew of 

board operators, shift supervisors and shift engineers. One of the operators said he tailors his 

own trends at the start of a shift. 

Value of System Mimic Displays 

Operators indicated that schematic displays are useful for performing or supervising specific 

maneuvers, system tests, and surveillances rather than for general monitoring of plant state. For 

example, the shift supervisor can see the flow path and monitor whether an operator is doing a 

switchover correctly. Another example is that during a surveillance of pressure channels, the 

operators can bring up a display with all four pressure channels in parallel. The availability of a 

graphic display makes it easier to check all four prior to disconnecting one of the channels, and 

then check the other three after one is disconnected. All the information needed is on one 

display. 

Opportunities for Improvement 

Navigation and Setup Requirements 

It was repeatedly pointed out that operators had no time to bring up or tailor displays under 

real-time changing conditions. The problem was not that the menu structures were difficult to 

understand or use. Rather, in dynamically changing conditions, any navigation and/or setup 

requirements impose too great a workload. 

Operator comments included: 

35 
4 RESULTS 

When a reactor trip occurs and you are in a transient, the most important parameter 

trends should come up automatically or with just one pushbutton. In an emergency, 

operators do not have time to navigate for the information. Now it can take four or 

five mouse clicks to get to the right trend. In dynamic conditions, that is too long. 

The problem is not one of understanding how to navigate to the right display. 

Operators know how to get to the right display but do not have time to do it. It takes 

away from monitoring. 

In emergencies, if operators have to set up a trend, they will not take the time to do so. 

If it is preloaded, they will look at it. 

’ Even though the menu system is straightforward, in a dynamic event, operators don’t 

have time to navigate to the desired trend. If it takes time to get to the picture, 

operators will use it less. 

Operators indicated that the utility is now in the process of creating icons allowing operators to 

get quickly (with one click of the mouse) to desired trend displays (e.g., trends tailored to 

particular transients such as a primary trip or a secondary trip). 

Difficult to See from a Distance 

Several operators mentioned that, while the trend displays are very useful, the trend lines are thin 

which makes them difficult to see from a distance. The problem is exacerbated by the fact that 

operators try to maximize the number of trend plots they can put on a VDU, since parameter 

trends are a valuable aid to monitoring. 

Some operators found ways to tailor the trend plots to make them more useable from a distance. 

One particularly good example is documented in Section 4.1.3. 

Premature Introduction 

One crew felt that the graphic display system had been installed in the plant prematurely and the 

process of testing the displays was still going on. They felt that, as a result of possible software 

bugs, some of the operators may be skeptical in using it. 

4.3.4 VDU Display Real Estate 

Operators indicated that the screen real estate provided by the VDUs in the control room is 

usually sufficient. However, they did indicate that they could make good use of additional VDUs 

if they were available. 

36 
4 RESULTS 

One of the crews discussed the use of VDU display real estate at length. The shift supervisor felt 

he could use more VDUs. He felt that in an emergency situation the CBP should have a 

dedicated VDU. He also wanted to put up trends and the AAS but did not feel he had sufficient 

display area to put everything up he wanted to. The shift engineer also felt he could use more 

than two screens. To compensate, he had to put up multiple overlapping windows. The board 

operators indicated that they could use more screens as well. In particular they pointed out that it 

is important to make the trends as large as possible so that you can see them from a distance. 

A second crew indicated that they tend to use the VDUs at the Shift Engineer's workstation and 

the spare station for special tasks. Thus, the board operators did not limit themselves to two 

VDUs per operator. They used all the VDU real estate available. 

4.3.5 Confidence in Software 

The operators were asked whether it was easy to understand how the software processes data and 

what the displayed information means. They were also asked about their confidence in the 

accuracy of the information provided by the new systems. 

The operators indicated that they did not feel they needed to understand the software processes. 

They felt that accurate software processing should be 'a given.' Their confidence in the accuracy 

of the system appeared to be a function of how accurately it performed in the past, rather than 

how transparent the underlying logic was. If there was a need to 'double-check' the accuracy of 

the computer system, they indicated that they would want to compare the output with that of an 

independent, redundant source of information in the control room. Related comments include: 

Operators felt that they did not need to 'double-check' what the CBP is doing by 

reading the sub-steps because they assume it is processing the logic correctly. One 

operator said "You want to double-check with the board, not just by reading the 

details of the CBP." 

Operators indicated that the CBP is straightforward, and it is easy to understand what 

it is doing. The logic behind the AAS alarm messages is less transparent. 

One operator pointed out that because there are some known problems with the 

current database of the AAS (messages that come on that are clearly incorrect), he 

was not sure how much to trust it, and therefore, would want to double-check with the 

original tile-based alarm system. However, he indicated that "once the little problems 

are ironed out," he would trust the information coming in. 

37 
4 RESULTS 

4.3.6 Impact on Crew Performance 

Operators were asked if they felt the new HSIs changed how they worked as a team when 

responding to incidents. The operators' responses are summarized below. 

Board Operators Have More Responsibility 

Shift supervisors as well as board operators confirmed that the new HSI systems have resulted in 

an increase of responsibility for the board operators. This, in turn, has increased the importance 

of their knowledge and skill. Operator comments included: 

"Before the CBP was introduced, the shift supervisor had the lead of the group and 

could supervise operators. Now, board operators have more responsibility. If the 

AAS alerts them to a system misalignment, they can take the action to correct it. But 

still, everyone must work within the rules of the EOP procedures." 

"Skills of the operator are more important than before. The board operator must 

know what he is doing." 

"If you have reliable operators, it is a relief for the shift supervisor. But if the shift 

supervisor can't be sure that operators are knowledgeable, there can be problems." 

"Board operators are not so guided by the supervisor. It is a great change for them. 

Before they were more guided by the shift supervisor, and now they must do the work 

by themselves. They may discuss with each other, but the shift supervisor is working 

independently. 'I 

"With the paper version of procedures, you are guided ...y ou know what you have to 

do. With the CBP, you have to watch the plant yourself." 

Irnpact on Communication 

The operators also confirmed that the HSI systems have had a strong impact on crew 

communication. The general consensus was that the amount of communication between the shift 

supervisor and the board operators went down. What the shift supervisor chose to communicate 

to the board operators, and the level at which he chose to communicate to them, varied across 

crews. It was also felt that the quality of communication between the shift supervisor and the 

board operators improved over the course of the week of training. The impact of crew 

communication on situation awareness of the various members of the crew is discussed in the 

next section. 

38 
4 RESULTS 

Operators were explicitly asked about their communication style during the scenarios that were 

carried out that week. Based on a comparison of operator comments from the different crews, 

there appeared to be wide variability across crews in the nature of communication between the 

shift supervisor and the board operators. In general, shift supervisors tended to communicate 

with the board operators more than was strictly necessary for the shift supervisor to obtain 

information needed for him to follow the procedure. There was an attempt by the shift 

supervisors to keep the crew synchronized by requesting andor calling out information that they 

felt was important for all crew members to know &e., that needed to be comrnon knowledge). 

However, what information the shift supervisor chose to call out varied from crew to crew. 

While the content of communication appeared to vary across crews, there was consensus that the 

quality of communication improved over the course of the one-week training period, suggesting 

that there was a "learning curve." 

Operator comments included: 

"The major impact is on communication. It is more quiet with the CBP." 

"Communication is clearly different with CBP. The shift supervisor asks fewer 

questions." 

"As a result of the CBP, communication is reduced. The shift supervisor has to be 

careful that everyone is following along." 

Communication Style Varied Across Crews 

One operator explained that in this early phase of the introduction of the CBP, the operators do 

not have approved rules and training regarding the shift supervisor's communication. They do 

have a list of 'highlight' steps that should be announced. They also have guidance that they 

should announce transition steps and steps when actions need to be taken. In general different 

shift supervisors communicated with board operators at different levels of detail. 

One shift supervisor said that he tended to read the high-level step titles out loud along with the 

'answer'that he was getting from the CBP (whether the step is satisfied or violated), unless it was 

a minor step (unimportant step). 

Another shift supervisor said he called out the important steps, such as transitions. He let the 

crew know the goal (e.g., leaving this procedure because SI has to be stopped). 

Another shift supervisor indicated that he asked fewer questions with the CBP than with paper 

procedures and that the board operators preferred being asked fewer questions. He indicated that 

39 
4 RESULTS 

he would ask the crew for the values of parameters when he thought it was important for all the 

crew members to know the value. He said, "You ask the board operator to make sure everyone 

knows the answer even if the CBP gave the shift supervisor the answer." 

One crew indicated that the communication style had improved over the course of the week. 

When asked how they developed the improved communication style over the span of the week, 

they indicated that the crew discussed the problem as a team, but that it was the shift supervisor 

who developed the more effective communication style. When asked to describe the 

communication style he developed, the shift supervisor said that he was not just reading out steps 

in the procedure. He was abstracting out the goals of the procedure and communicating those to 

the operators. He said that he would state the problem (what is wrong), the solution path, and the 

current goal - "We are doing this. And, this is our goal." He used his own words, not the 

explicit steps in the procedure. He would also read out when he was switching procedures and 

the major procedural steps. The shift supervisor and the board operators agreed that 

communication was improved as a result. 

Communication Required Conscious Effort 

Because the CBP provides the shift supervisor with the plant parameter data required to resolve 

procedural step logic, the inherent forcing function for communication that exists with 

paper-based procedures is no longer as strong. Communication between the shift supervisor and 

the board operators is no longer an integral a part of task performance. It has become a distinct, 

parallel activity that imposes its own cognitive demands on both the shift supervisor and the 

board operators. The shift supervisor needs to consciously remember to communicate with the 

board operators and decide what to communicate to them. With paper-based procedures, 

communication was, by necessity, at a very detailed level, with specific plant parameter values 

requested and passed. Computer-based procedures free the shift supervisor to communicate with 

the board operators at a higher level, but he must consciously decide when to communicate and 

what to say. In turn, because there is less constant communication, the board operators need to 

make a conscious effort to listen for the shift supervisor. They must also make conscious effort 

to report to him findings that are relevant but that he would not know about through the CBP 

because they are not called out in the procedures. 

One shift supervisor pointed out that as a result of introduction of the CBP, the shift supervisor 

"needs discipline" in his communication style. He elaborated, indicating that during the 

scenarios he had to consciously remember that the operators don't see what he sees with the CBP 

and explicitly remember to keep them informed of the situation. He felt that the tendency not to 

communicate is potentially problematic, and therefore, should be addressed in training. 

A second shift supervisor noted that sometimes with the new systems, the panel operators were 

monitoring independently and not paying as much attention to the shift supervisor. He indicated 

40 
4 RESULTS 

that on occasion, when he required the board operators to take an action, he had to say it more 

than once before he got their attention. 

Communication Improved with Practice 

Several crews noted that at the beginning of the week, crew communication had been poor; 

however, by the fourth day, the communication became more natural and was no longer a 

problem. 

Impact on Situation Awareness of Crew Members 

While the introduction of the new HSI systems, and in particular the CBP, clearly affected 

communication, the effect on operator situation awareness was less clear. Shift supervisors 

generally felt that their situation awareness has improved. There was more variability of opinion 

among board operators with respect to whether the new HSIs have positively or negatively 

impacted their situation awareness. The majority of the board operators felt that their 

understanding of plant state and their ability to effectively respond to events was improved 

because they had the freedom to monitor the AAS and board indicators. They also pointed out 

that with the CBP, while they communicated less with the shift supervisor, the board operators 

were able to communicate and coordinate more with each other. As a result, they felt that their 

situation awareness had improved. A couple of board operators felt that the reduced 

communication with the shift supervisor negatively impacted their ability to maintain the big 

picture and keep up with the shift supervisor as he worked through the procedure. 

The general opinion was that situation awareness improved over the course of the training week 

as communication between the shift supervisor and the board operators improved. 

Impact on situation awareness of shift supervisor 

Comments on situation awareness by shift supervisors included: 

"With the CBP the shift supervisor has better situation awareness than without." 

"With the CBP the shift supervisor feels more in control" 

"The CBP neither helps nor hurts from the perspective of situation awareness. It is 

the same as with paper." 

41 
4 RESULTS 

Impact on situation awareness of board operators 

Many of the board operators' comments stressed improvement in situation awareness because of 

the attentional resources freed up by the introduction of the CBP and the improved information 

provided by the AAS. Comments included: 

"For board operators, it improved understanding and handling of events because they 

now have more attention to pay to the board." (Ths operator did not feel that reduced 

communication with the shift supervisor impaired his ability to keep track of 

situation). 

"Without the CBP, the board operator was so busy giving back data to the shift 

supervisor that he did not have the mental resources to keep track of the process. 

Now, the board operator has more attention to pay to the process." 

"Operators have more time to watch the panels because they do not have to provide 

the shift supervisor with the detailed parameter information required by the EOP 

steps. It 

"Board operators get information from the AAS that they otherwise would not have 

from conventional plant indicators and alarm systems (thus improving situation 

awareness). It 

A related comment made by one crew was that they felt the board operators coordinated among 

themselves better with CBP than with paper. The board operators on this crew felt that while 

they communicated less with the shift supervisor, they had more time and attentional resources to 

communicate and coordinate with each other. 

Other comments suggested that the change in communication with the shift supervisor negatively 

impacted their situation awareness: 

One of the board operators said that he was able to follow better with the paper 

version because every step was said out loud. Interestingly, the other board operator 

in the crew disagreed, attributing the difference in opinion between them to different 

levels of experience. The operator who did not think the CBP reduced his situation 

awareness thought he was more familiar with the structure of the EOPs than the other 

operator. 

Several crews indicated that, at the beginning of the training week, the board 

operators felt they had lost situation awareness, but that by the fourth day, they felt 

that their situation awareness was just as good with the CBP as with paper. Ths is 

42 
4 RESULTS 

because of the increased discipline of the shift supervisor in communicating with 

them. 

4.3.7 Training 

The operators were asked their opinion of the training they received on the new HSI systems. 

They were asked, "Do you think, based on the training you have gotten so far, that you would feel 

confident using these new systems in an actual incident? They were also asked, "What additional 

activities or training do you think is needed before you will feel confident in using these new 

systems in an actual incident?" 

Their answers were thoughtful and informative. While they expressed general satisfaction with 

the training they had received, they had a number of positive suggestions for additional types of 

training that would provide broader experience in use of the HSI systems. Their suggestions 

included: 

Training on more complex scenarios that provided the opportunity to experience a 

greater variety of procedures and experience using the CBP in multi-fault conditions 

Training using the plant's part-task simulator to provide more opportunity to try out 

the systems 

Training using a data-base of "canned scenarios" to drive the HSI (One operator said, 

"It would be nice if there was a way to practice using the CBP on your own, using a 

simple simulation or database to drive the CBP. Right now, the CBP is available at 

the plant, but it cannot be seen in action in the control room when everything is 

normal. 'I) 

Many of the operator comments focused on the need for more practice on crew-interaction skills. 

'They felt there was a need for focused training on the new crew roles and responsibilities, 

communication, and coordination. One shift engineer pointed out that the board operators in that 

\crew used the advantage of extra attentional resources to keep track of the event, but whether 

other board operators do so depends on their level of expertise and training. He felt there may be 

;a need to train operators to take advantage of the attentional resources that have been freed as a 

iresult of the introduction of the CBP. 

Several crews pointed out that the need to train as a crew is more important with the new HSI 

systems because the operators now have more freedom in how they do their job. As a result, it 

becomes more important to understand how your team-mates perform their tasks. As an 

example, the different shift supervisors verbalize the EOP steps as they went through them to 

43 
4 RESULTS 

different degrees. At this plant, depending on scheduling constraints, operators did not always 

train as a crew, sometimes they trained with operators from a different crew. 

Operators also stressed the value of continuing to train in the use of paper-based procedures both 

as a basis for developing a deeper understanding of the EOPs and as a precaution in case they 

ever needed to revert back to paper-based procedures. Comments on the importance of training 

on paper procedures as well as the CBP included: 

"Even when the CBP is used in the plant, it will still be important to train operators to 

use the paper versions of the procedures. Procedures are more difficult in paper form. 

Operators would not want to lose that skill. It is especially important for new 

operators who have not had lots of training on computerized procedures." 

"If the computer fails, you will have to go back to paper, so it is important to train on 

the paper version of procedure as well." 

4.4 Interviews with Head of Operations and Head of Training 

The head of operations at the plant and the head of training were interviewed. The primary 

purpose of these interviews was to provide perspective on: 

The motivation for installing these new HSI systems in the plant and the benefits 

expected 

The impact of the introduction of these systems on operator performance from the 

management perspective 

The involvement of the plant personnel in the development and maintenance of these 

systems 

The introduction and transition process by which these HSI systems were 

implemented in the control room 

The impact on training 

The emphasis was on learning the plant management's perspective on what things they felt 

worked well and what lessons were learned. Each is discussed below. 

,Motivation for Installing Advanced Alarm System and CBP 

Plant management indicated that the AAS was designed to support alarm reduction. The CBP 

was developed in an effort to move through the EOPs faster. The plant management wanted the 

44 
4 RESULTS 

crews to be able to get through E-0 faster, since for some procedures and events, such as a Steam 

Generator Tube Rupture, it is important to get through these steps quickly. It was pointed out 

that paper procedures are very time-consuming; for example, indication for some of the valves to 

be checked is not available in the control room. With the new computer system, all the data 

points are on the plant data highway and are available to the CBP. Thus, it is possible to go 

faster. 

Impact on Operator Performance 

Plant management felt that operator performance using the AAS is improved because it reduces 

the number of alarms and covers all of the immediate action steps in E-0 and alerts the operators 

if automatic systems have not responded as expected. 

The CBP improves operator performance because operators can move faster. The goal is to 

finish E-0 in ten minutes. For tube ruptures, it is important to reach pressure equalization 

between primary and secondary in 20 minutes. The CBP also resolves step logic and points out 

procedure transitions. Disadvantages include a narrower field of view than paper. In addition, 

operators cannot flip forward and back as easily. 

With respect to the Graphic Display System, plant management indicated that operators have 

become so accustomed to using it that if they turned it off now, they would get protest screams 

from the operators. During normal operations, they mostly look at the graphic display system 

and not the hard-wired instrumentation. With the graphic display system, operators can detect 

deviations more quickly and take control actions. The Graphic display system is heavily used in 

checking systems and allows the operators to find a problem "before there is a problem." 

Impact of HSI Systems on Crew Communication 

Plant management indicated that during initial testing, they identified a potential problem with 

communication because it was not occurring naturally. They decided that training had to stress 

the importance of the SRO communicating to synchronize the crew. They initially came up with 

a list of "most important steps" in the EOPs and asked shift supervisors to announce them. With 

experience however, they found that that was not useful and it is no longer required. They now 

require that the shift supervisor announce all the transitions and exits from procedures. 

With respect to synchronizing the crew (i.e., making sure everyone "is on the same page" and 

shares common knowledge), plant management felt that synchronization of the crew happens 

naturally due to the work that has to be performed. It was felt that once the crew exits E-0 and 

enters another procedure, there are action steps that require the shift supervisor to ask the board 

,operators to take action. The shift supervisor calls out the action request and then sees the result 

#of the action request in the CBP. It was felt that this was sufficient to synchronize the crew. 

45 
4 RESULTS 

Imuact of HSI Systems on Training 

Plant management felt that with the new HSI systems, they have basically the same training 

objectives as with the older technology, but training should go faster and more efficiently. The 

first year, they are training crews on basic EOP procedures. In the second year, they plan to 

introduce more complicated transients in the procedures (such as compound malfunction 

transients and larger number of procedures covered). 

Plant management plans to train operators on the CBP and on paper. In every training period 

they plan to go through one scenario where the operators will need to transition to paper 

procedure. 

Involvement of Plant Personnel in Development and Maintenance 

Plant management indicated that plant personnel were actively involved in the development and 

upgrade of the new HSI systems. They felt the plant had initially underestimated the level of 

participation of plant personnel that would be required. As the project went on, they realized the 

demands were greater than anticipated. They would have liked to assign more operations people 

to the design effort. 

In terms of 'lessons learned' it was felt that it was important to have much more involvement of 

operations personnel in development of the HSI systems. It was also felt that it was important to 

install the systems in the simulator first to familiarize the operators, before installing the HSIs in 

the control room. 

Plant management indicated that maintenance of AAS and CBP databases will be done by the 

plant operations staff. They indicated that the software tools to maintain the AAS and CBP are 

easy to use. The utility is currently trying to set up a systematic and formal approach to capture 

"problems and suggestions." The issues identified will be reviewed by a small group that will 

decide whether to make changes or not. 

The plant management also expressed the desirability of establishing an "Owner's Group" to 

share experiences and information about the new HSI systems. 

Verification and Validation Activities 

The plant is currently negotiating verification and validation (V&V) requirements with their 

regulatory agency. 

46 
5 DISCUSSION 

This study examined the impact of introducing computer-based HSI systems into a conventional 

nuclear power plant control room. The study examined the impact of the new systems on the 

cognitive functioning of individual crew members as well as on the structure and functioning of 

the crew as a team. The study was one of the first to look at text-based CBPs (the previous CBP 

studies were flow-chart based) and one of the first to look at transition to paper-based procedures 

upon loss of the CBP system. In addition, the process by which the systems were developed and 

introduced was considered. Finally, the effect of the new systems on training was examined. 

In considering the results of the study, it is important to re-emphasize the constraints: 

(1) the observations were made during the first training week using the new systems, 

(2) the systems were not completely debugged at the time the study was conducted, and 

(3) the scope of the scenarios was limited to relatively straightforward events. 

In addition, the study relied on interviews and naturalistic observational methods. No systematic 

manipulations of HSI features and characteristics were attempted and no quantitative assessment 

of performance was obtained. 

With these limitations acknowledged, the lessons learned from the study have potential 

implications for design, training, and implementation of advanced HSIs in a conventional plant. 

These lessons are discussed below. 

5.1 Overall Impact of Introduction of Computerized HSIs 

Overall the results of the study suggest that introduction of computerized HSIs positively impacts 

crew performance. The results show that the computerized HSIs expanded the range of data 

available to the operators and provided increased power and flexibility. They reduced the 

workload associated with gathering and integrating plant parameter information needed to work 

through the emergency response procedures, freeing attentional resources to more broadly 

.monitor plant state. The operators were uniformly positive about the new HSIs and would not 

.want to revert back to the conventional control board, alarm system, and paper-based procedures. 

Further, the training period to utilize the systems was minimal. The operators were able to use 

ithe HSIs to handle simulated dynamic emergency events in the first week of training on the 

;simulator. 

'While the general results were positive, the study identified HSI design issues that warrant 

consideration in the design and evaluation of new HSIs. The study also identified HSI issues for 

which there is currently minimal guidance available in the human factors literature, suggesting 

the need for additional research. These are discussed in the following sections. 

47 
5 DISCUSSION 

5.2 Alarm Systems 

The AAS was found to enhance the operators’ awareness of the plant and to support their 

performance. In this regard, two especially-positive features stood out. First, the alarm messages 

were more specific and detailed than is possible with a conventional tile system. Second, the 

alarm messages after a trip were well coordinated with Procedure E-0 and allowed operators to 

rapidly assess whether the automatic safety systems came on as they were supposed to, or 

whether manual backup action was required. 

The study also reinforced the need for significant alarm reduction during plant disturbances and 

the difficulty of achieving it. In spite of the fact that the AAS reduced the number of alarms 

substantially from the tile-based system originally implemented at the plant, and that this was 

viewed positively by the operators, operators reported that there were still too many alarms 

activated during a serious plant transient. 

A related point that was strongly articulated by the operators is the importance of distinguishing 

status messages (things expected to happen) from alarms (things expected to happen that did not, 

and things that should not have happened but did). While it may be important to alert operators 

to significant changes in equipment status (e.g., an automatic safety system coming on), clear 

cues should be used to distinguish between alarm messages and status messages. This may 

include placing them in different physical locations and/or using different audio tones as alerts. 

In a rapidly evolving event, operators do not have sufficient time to read alarm messages. 

Consequently, alarm presentation strategies should be utilized that allow operators to 

characterize or identify the content of an alarm without having to read the detailed text. 

Traditional tile-based systems achieved this goal because of the one-to-one mapping between the 

location of a tile and the content of the alarm (i.e., complete spatial dedication). Other 

approaches might be to use color or graphic symbols to enable operators to more rapidly process 

the content of an alarm. 

The AAS design goal was to utilize the physical position of the alarm display windows to enable 

the operators to identify the nature of the alarm without having to read the text. While it is likely 

that, with training and experience, the operators’ ability to establish such a rapid perceptual 

mapping will improve, their initial comments were that they felt they needed to take too much 

time to read the alarm message in order to understand the nature of the alert. 

Another finding of the study is that during a dynamically evolving event, directly involved board 

coperators do not have time to consult Support Panels to review ’overflow’ (lower priority) alarms. 

‘They may do so in special cases, or later in the event during low tempo periods, but in general 

they rely on the alarm prioritization scheme to present them with the most important alarms they 

should be aware of. This increases the importance of having a robust alarm prioritization scheme 

48 
5 DISCUSSION 

that is broadly applicable across contexts. In the current study, everyone interviewed, including 

the operators and training instructors, appeared satisfied with the prioritization scheme. 

A final lesson learned' related to alarm systems is that alarm system requirements differ during 

normal operations and emergency operations. In this plant, operators relied on the 

chronological-list alarm system during normal operations because it included alarm limits on 

virtually all the data points on the highway. It was useful for picking up early signs of minor 

malfunctions (e.g., equipment problems). In an emergency, the large number of alarms generated 

and chronological list organization made the system ineffective. For emergencies, the AAS that 

organized alarms into categories (in this case by plant functions) and presented the alarm 

categories in parallel on a physically distributed board was preferred. This general finding has 

been repeatedly observed (e.g., Mumaw, Vicente, Roth, and Burns, 1995). 

The implication of these results is not that chronological list alarm systems are the most 

appropriate for normal operations. It is likely that the preference for the chronological list alarm 

system was due to the fact that the AAS did not include as many specific alarms relevant to 

normal operations. As the AAS database expands, operators may come to rely on it for normal 

operations as well as emergency operations. The main point to be drawn from the result is that 

the goals and uses of alarm systems are very different in normal and emergency conditions, and 

that these differences should be explicitly considered in the design and evaluation of new alarm 

systems. 

An important aspect of the present study is that it has provided converging support for AAS 

design and NRC findings previously reported (O'Hara, Brown, Hallbert, Skrhing, Persensky, 

and Wachtel, 2000). The two studies were based on different objectives, different research 

methods (naturalistic observation and interviews vs. controlled experiment), and different alarm 

system design. Despite these differences, the general lessons learned from the two studies are 

strikingly similar, especially with regard to the following: 

There were difficulties in achieving meaningful alarm reduction (in the other study, 

the number of alarms during simulated process disturbances was reduced as much as 

75 percent by alarm processing; despite the considerable alarm reduction, its impact 

on objective performance was small). 

Spatial dedication is important. 

Message systems have limitations (as discussed in Section 4.3.1 , Advanced Alarm 

System, and in Section 5.2, Alarm Systems; unlike tiles, which can be recognized 

quickly based on the operator's knowledge of their spatial location, alarm messages 

have to be individually read, which may create high workload when there are many 

alarms). 

49 
5 DISCUSSION 

Operators were reluctant to access alarm support in high workload situations. 

Alarm system requirements were different under normal, disturbance, and 

post-disturbance conditions. 

5.3 Computer-Based Procedures 

The most significant findings of the study relate to computerized procedures. This is an area 

where there have been relatively few studies, and virtually no studies that have examined 

text-based CBP. 

The study provided a positive illustration of the ways in which CBPs can be an effective aid in 

helping shift supervisors work through the procedures. Observations of performance indicated 

that operators were able to utilize the CBP to work through the procedures in pace with the 

events. Interviews with operators, training instructors, and plant managers provided consistent 

evidence that the computerized procedures improved the speed and accuracy of following 

procedures. 

One of the most important positive aspects of the CBP was that it reduced the shift supervisor’s 

cognitive workload. The CBP reduced the need for the shift supervisor to get plant parameter 

data from the board operators. Further, the CBP performed many low-level cognitive tasks 

associated with analyzing and following procedures, including: 

Resolving step logic 

Keeping track of location in the procedure 

Keeping track of steps of continuous applicability 

Assessing cautions, critical safety function status trees, and fold-out page criteria. 

As a result, the shift supervisor was able to work through the procedures faster and was less 

likely to make an error (e.g., skip a step, misinterpret a step logic, miss a transition). Shift 

supervisors reported that they felt ’more in control’ as a result. 

To the operators, these positive features appear to outweigh some of the less desirable features of 

the CBP system. These include the nmower ’field of view’provided by CBP (Le., in terms of 

number of steps that can be viewed in parallel as compared to paper). While operators noted that 

the CBP provided a narrower window on the procedure and made ‘looking ahead’ (i.e., the ability 

to flip forward and back through the procedure) more difficult, they did not perceive these as 

serious problems. 

50 
5 DISCUSSION 

There was some evidence that on occasion the CBP could provide misleading information or 

direct the operators down the wrong procedural path. It should be noted that the problem of the 

procedures occasionally misdirecting operators is not unique to CBP, but occurs with 

paper-based procedures as well (cf. Roth, Mumaw, and Lewis, 1994). CBP may accentuate the 

problem, however. One reason is because the computer tends to be more precise and literal in its 

interpretation of steps. Steps that are left to operator judgment in paper procedures may be 

operationalized too narrowly when computerized. 

As an example, conditions such as "stable or increasing" are sometimes difficult to specify 

precisely. In the written EOPs, these are left to operator judgment. In converting to 

computerized procedures, there is a tendency to try to define terms more precisely and in some 

cases the criteria used by the computerized procedure may be too narrow or may not take into 

account as many factors as an operator would in the situation. Operators consider not only the 

behavior of the parameter in question, but also: (1) other current situational factors (such as a 

cooldown that is temporarily decreasing pressurizer level), and (2) situations that are likely to 

happen in the near future (such as if the crew is about to take some action that will increase the 

cooldown rate) that will affect the behavior of the parameter. A prime example is that cooldown 

will affect RCS level and pressure, as well as SG level and pressure. While operators consider 

cooldown effects in interpreting and projecting parameter trends (Roth, Mumaw, and Lewis, 

1994), the CBP does not. 

These observations suggest that caution should be taken in computerizing procedural steps that 

have classically been left to operator judgment in paper procedure. One option is to follow the 

approach taken by paper procedures, and leave these steps to operator judgment (i.e., not resolve 

the step logic for the operator). The CBP examined in this study used this strategy in some cases. 

There was a category of steps that was defined as 'no evaluation' steps. These steps explicitly 

required the operator to evaluate the step. The steps were highlighted by the CBP when they 

came up and no evaluation was provided by the CBP for these steps. 

If designers choose to computerize steps that have been classically left to operator judgment, care 

should be taken to ensure that the operational definition of the step is not overly simplistic and 

that it takes into account the variety of factors that operators do under the same circumstances. 

This may require input from operations and training experts, in addition to engineering expertise. 

Further, the resulting step logic should be subject to extensive testing. 

A second factor that may make CBP more prone to misdirect operators than paper-based 

procedures is that operators may tend to accept the CBP assessment without sufficient reflection. 

If the computerized procedure says that a step is satisfied, the operator may be less likely to 

scrutinize and double-check that the step is in fact satisfied (Mosier, Palmer, and Degani, 1992; 

Layton, Smith, and McCoy, 1994; Roth, Malin, and Schreckenghost, 1997). In contrast, if the 

[operator has to assess whether a step is satisfied on his own (e.g., as happens with paper 

procedures), he may be more likely to look closely at parameter values and trend information in 

51 
5 DISCUSSION 

formulating his judgment. There is a significant body of literature in domains other than nuclear 

power that suggest that this is the case (Layton, Smith, and McCoy, 1994; Roth, Malin, and 

Schreckenghost, 1997). Further our findings are consistent with this claim. In two of the three 

cases where the CBP was off-track, it was not the shift supervisor who detected the problem. 

The shift supervisor called out actions for the board operators to take. It was the board operators 

and shift engineer who recognized that the actions were inappropriate to the situation. The 

question of whether CBPs encourage complacency in the user and how to structure crews to 

optimize the possibility of catching when a CBP is off-track is an important area for further 

research. 

It should be recognized that the particular CBP under study was in the early stages of 

introduction and that its database was still in the process of being fine tuned. Most of the 

specific cases observed where the CBP misdirected the operators may be corrected through 

changes to the database. Further, while the focus here is on cases where the CBP has the 

potential to misdirect, it should be stressed that overall the CBP is likely to reduce the potential 

for operators going off track by skipping or mis-evaluating a procedural step. 

The main reason for focusing on cases where the CBP is potentially misleading is that experience 

with decision aids in other media and other domains suggest that situations where a decision aid 

provides misdirection will occasionally arise (Billings, 1996; Mosier, 1997; Roth, Bennett, and 

Woods, 1988; Roth, Malin, and Schreckenghost, 1997; Roth, Mumaw, and Lewis, 1994). Given 

that situations may arise where the decision aid is off-track, important questions are: 

(1) Can operators detect when the decision aid is off track? (2) Are they able to redirect the 

decision aid and get back on track? 

In the three cases we observed in the study, operators were able to correctly detect that the CBP 

direction was inappropriate to the situation and overrode it. The examples illustrate important 

positive features of the CBP, and raise questions about the conditions that are necessary to foster 

the ability of crews to detect that a CBP is off-track and redirect it. 

The first point is that the CBP was designed with the recognition that the person using the CBP is 

the cognizant decision-maker. The design philosophy was that the CBP is a tool to support that 

person. As a result, it included a number of features that enabled operators to redirect the CBP 

.when they determined it was off-track. 

:For example, the CBP allowed the operator to override it at any point. At any point, the operator 

could move to any other place in the same procedure or to any other procedure. The fact that we 

observed cases where operators needed to redirect the procedural path reinforced the importance 

of including features that allow operators to override the CBP and move to any other procedural 

step of their choosing in the CBP. 

52 
5 DISCUSSION 

The second point relates to the conditions necessary to foster the ability of crews to detect that a 

CBP is off-track and take action to redirect it. The findings suggest the importance of having 

(1) multiple diverse sources of information available to operators in the control room, and 

(2) effective communication among the operators in order to detect and correct cases where the 

CBP is off-track. In some of the cases we observed, it was not the shift supervisor, but rather one 

of the other crew members who pointed out that the actions specified by the CBP were 

inappropriate given the situation. The ability of the operators to recognize that the actions 

specified were inappropriate depended on (1) accurate understanding of current plant state, 

(2) solid knowledge of the goals and assumptions of the procedures and the consequences of the 

actions indicated by the procedure, and (3) strong communication between the shift supervisor 

and the board operators, that allowed the board operators to keep track of the procedural path that 

the shift supervisor was on. 

While the study provides suggestive evidence of the kinds of factors that contribute to the ability 

of crews to detect if a CBP was off-track, more research is required to fully address this issue. 

The study also has implications for training. The findings highlight the importance of training 

operators to utilize multiple independent sources of information in the control room (i.e., alarms, 

graphic displays, and board indicators) to double-check the CBP. This was the training 

philosophy adopted by the plant observed in this study. 

The study also suggests the importance of training good communication skills to make sure that 

the crew maintains common understanding of the current state of the plant and the objectives and 

current state of the procedures. Our observations reinforced the importance of developing better 

guidance on what makes for effective team communication. While the plant management and 

training staff recognized that good crew communication was important, the guidance for the 

operators as to what constitutes good communication was not finalized and was in development. 

In fact, these training sessions were being used as the basis for the guidance. As a consequence, 

there was wide variation among the crews in communication style. In some cases, breakdowns 

in communication were observed. The lack of guidance on what is good communication reflects 

a gap in the research literature as to what constitutes good crew communication. There is 

currently active research in the military on what factors are associated with high reliability crews, 

what constitutes effective team communication, and what training is required to develop good 

communication skills (e.g., Salas, Canon-Bowers, and Johnston, 1997). Additional work in this 

area is needed to synthesize the available research results into guidance as well as to extend the 

:results to nuclear power plant applications. 

:Finally, the results reinforce the importance of operators having a thorough understanding of the 

goals and response strategies underlying the procedures. The crews observed in the study had 

detailed training on the procedures and extensive experience using paper-based procedures. This 

imay have facilitated their ability to detect when actions specified in the procedure were not 

53 
5 DISCUSSION 

appropriate to the specific situation and override them. A question arises whether the same 

would be true of crews that were not initially trained with paper-based procedures. Can the shift 

supervisor pick up the goals and logic behind the EOPs given the narrower field of view and 

automatic logic resolution provided by the CBP? Can board operators pick up the goals and 

logic behind the EOPs given that dl the procedural steps are no longer voiced aloud to the crew 

as happens with paper? 

A question that arises with respect to CBP is what happens if the CBP system fails and the 

operators must revert back to paper-based procedures. Concern has been expressed in the 

literature about whether operators would be able to locate their place in the paper-based 

procedures and take-over the tasks that had been done automatically by the CBP. This study 

provided one of the first opportunities to examine the transition from CBP to paper-based 

procedures. The results indicated that operators were able to transition smoothly to paper-based 

procedures in a simulated failure of the plant computer. The operators were able to find their 

place in the paper procedure and take over tasks that had been done automatically by the CBP, 

such as reviewing the critical safety function status trees and the fold-out page criteria, without 

difficulty. It should be noted, however, that the particular scenario where the computer failure 

was simulated was a simple accident situation and the transition to paper occurred early in the 

event (while still in E-0), before many procedure transitions were made. Also these crews were 

highly practiced on the use of paper-based procedures, and only recently introduced to the CBPs. 

While the finding that operators had no difficulty transitioning to paper under these conditions is 

encouraging, more study is required to determine whether transfer to paper can occur smoothly in 

complex situations by operators whose primary experience has been in the use of computerized 

procedures. 

5.4 Graphic Display System 

Consistent with previous research, the results reiterated the importance of trend displays to 

support monitoring of plant state during both normal and emergency operations (cf. Mumaw, 

Vicente, Roth, and Burns, 1995; O'Hara et al., 2000; Roth, Brockhoff, Rusnica, Kenney, and 

Kerch, 1997; Vicente, Mumaw, and Roth, 1997). The trend displays were mentioned by all the 

crews as the most valuable displays during both normal and emergency operations. They were 

also mentioned as the displays they most missed in the scenario that simulated a loss of the plant 

computer. Trend displays support broad situation awareness by enabling operators to see at a 

glance whether parameters are staying constant or changing, the direction and speed of change, 

and approaches to limits. 

.An important observation is that because of the high-tempo, dynamic nature of process control, 

#and the need for operators to move around the plant, it is important to create displays that can be 

:seen from a distance. Operators relied on putting up trend displays as their overview displays. 

54 
5 DISCUSSION 

The general outline of a trend can be seen from a distance, even though the exact values cannot. 

Operators indicated that they wished the trend lines were thicker so that they could be viewed 

from farther away. To overcome this problem, operators created their own graphic display that 

could be readily interpreted from a distance. The display exploited the ability of people to detect 

changes in the shape of objects to allow operators to quickly assess from a distance the deviation 

of a parameter from target and the direction of change over time. This is an excellent example of 

an ecological interface display (Vicente and Rasmussen, 1992). 

It was also found that in dynamic emergency conditions any requirement for display navigation 

or display set up may be too much. Any requirement for set-up or navigation is an added burden 

on operators during dynamic emergencies and should be minimized. While operators liked and 

used the trend displays, they indicated that in high tempo situations such as an emergency, they 

could not afford to take the time to set up a particular trend display. There was an expressed 

desire to predefine displays and to have them come up with just one pushbutton mouse click. 

This is a more general representation of the reluctance noted above for operators to access alarm 

support displays. 

This finding is consistent with the findings of a recent NRC study of interface management, such 

as display configuration navigation (O'Hara, Stubler, and Nasta, 1997). That study found that 

operators were reluctant to engage in interface management tasks in high workload situations. 

O'Hara et al. have noted that this reluctance to access information not presently available and the 

keyhole effects created by VDU-based interfaces creates the potential for operator performance 

to become data-limited, i.e., crews may miss important information because it is not contained on 

the presently available displays. 

The operators reported that the system mimic displays were also valuable, but we did not observe 

them using such displays often during the emergency scenarios. The likely reason is that, 

according to the operators, system mimics are mostly used to support specific tasks such as tests 

or inspections that require a detailed view of a particular system (e.g., a particular valve 

maneuver). It would be necessary to observe operators during normal operations in the plant to 

get a better assessment of how the operators use those types of displays (cf. Vicente, Mumaw, 

Roth, and Burns, 1996). 

5.5 Confidence in Software 

Operators did not appear to be concerned with following the software logic on which the HSI 

systems were based. Operators assumed that the software logic would be correct. They did not 

feel a need to double-check system logic in real time. For example, in the case of the CBP, 

operators did not feel a need to go through every procedure sub-step to assess for themselves 

whether a step was satisfied or violated. They relied on the CBP to resolve the logic correctly. 

Rather than retreading the same path as the HSI systems, operators preferred to double-check 

55 
5 DISCUSSION 

system conclusions using independent methods. For example, operators assessed plant state 

independent from the CBP based on independent sources of information such as alarms, graphic 

displays and board indicators. They typically did not check the specific parameters and logic 

used by the CBP. 

The operators' approach to monitoring the performance of the computerized system is consistent 

with results in other studies of supervisory control of automated systems both within and outside 

the nuclear domain (Roth, Malin, and Schreckenghost, 1997; Tenney, Rogers, and Pew, 1995; 

Vicente, Mumaw, and Roth, 1997). The results have implications for the design of software 

systems. First, the reliance of operators on the accuracy of the software logic places additional 

emphasis on ensuring the accuracy of the underlying software. While operators don't want to 

have to 'shadow' or 'second guess' the computer system to make sure they are on track, it is 

important that they have confidence that the systems will generally give correct information. 

Second, the results emphasize again the importance of including multiple, independent and 

redundant sources of information in the control room to allow operators to detect problems. 

5.6 Impact on Operations and Crew Structure 

The introduction of new HSI systems, particularly CBPs, can affect the structure and dynamics of 

the team as a whole. This in turn can affect the quality of decision making. We observed that 

the introduction of the CBP had broad effect on the cognitive performance of individual crew 

members as well as on the functioning of the crew as a team. It affected the scope of 

responsibility of the different crew members, the communication pattern among crew members, 

and the situation awareness of the different crew members. The change in crew member 

responsibility and crew structure have implications for training as well as human performance 

and reliability. 

The shift supervisor and board operators were able to work more in parallel. The shift supervisor 

concentrated on working through the procedures and the board operators concentrated on 

monitoring the AAS, graphics display, and control board HSIs. The change in crew structure 

appeared to have two consequences. On the one hand, the shift supervisor and the board 

operators, individually reported improved situation awareness and greater confidence in the 

accuracy and speed of their performance within their own locus of responsibility. Shift 

supervisors reported that they felt they could work through the procedure faster and more 

accurately. Board operators reported that they felt they were able to maintain better situation 

awareness of the plant state and to take appropriate control actions. On the other hand, the ability 

to maintain awareness of each other's situation assessment and activities appeared to require 

more conscious attentional effort than before. 

'The cognitive workload associated with following the procedures appeared to have been reduced. 

'The shift supervisor had less need to rely on the operators to obtain information about plant state. 

56 
5 DISCUSSION 

As a result, shift supervisors reported that they felt more in control and that they experienced 

improvements in situation awareness. While the computerized procedure reduced the shift 

supervisor’s workload in some respects, it introduced a new element of workload. The new 

elements included the need to keep the crew informed of his assessment of the situation and the 

status and direction of the procedural path as he worked through the procedure. Shift supervisors 

reported a need to consciously remember to inform the crew of their status through the procedure 

and to consciously formulate what to communicate. The new communication requirement is a 

substantial cognitive task that appeared to improve with training and experience. Shift 

supervisors varied in the extent and content of their communication to the crew, and crews 

reported that communication improved over the course of the training week. The variance 

among people and across time suggests that this is a skill that can be improved with training and 

experience. 

Similarly, the introduction of the CBP changed the cognitive load, task responsibilities, and 

communication requirements of the board operators. Because the board operators no longer 

needed to provide detailed plant parameter information to the shift supervisor, they had more 

attentional resources to devote to monitoring the AAS, graphic displays and control board. From 

this perspective, they felt they had opportunity for greater understanding of plant state and thus 

better situation awareness. 

At the same time, the board operators were working more independently from the shift 

supervisor. They had more responsibility and less direct supervision. This has several 

consequences. First, as reported by the operators, the fact that the operators were now working 

more independently meant that their knowledge and skill became more important. Second, while 

operators reported that they felt their understanding and situation awareness of plant state was 

better, their ability to maintain awareness of the progress through the procedure required more 

cognitive effort. Procedure awareness depended on the skill of the shft supervisor in 

communicating to operators progress through the procedure. It also depended on the knowledge 

Operators had of the structure and objectives of the procedures. Further, because the board 

operators’ focus was more on monitoring the plant, they had to consciously attend to keeping 

track of the shift supervisor’s progress through the procedures. The need to pay attention to the 

shift supervisor to recognize when he is communicating something important to them becomes a 

secondary task. One crew in particular mentioned that the shift supervisor had to occasionally 

repeat a request to the board operators because he had failed to get their attention the first time. 

(Generally, the board operators indicated that they were able to maintain awareness of progress 

through the procedure, particularly as the week progressed. The main point is that with the new 

systems, maintaining awareness of the shift supervisor’s progress through the procedure becomes 

;a separate task that requires knowledge, skill and devotion of attentional resources on the part of 

lboth the shift supervisor and the board operators. This activity needs to be recognized and 

explicitly trained. 

57 
5 DISCUSSION 

There is also a need for board operators to keep the shift operator informed of key findings they 

observe during monitoring and important control actions that they take. While generally board 

operators did so, there was one case where key information was never communicated to the shift 

supervisor, and he was not able to pick it up from the CBP. This case highlights the importance 

of communication, the potential for communication breakdown, and the need to explicitly train 

for effective communication. 

To summarize, a major finding of the study is that the new HSIs resulted in a change in the 

information available to the different members of the crew and the communication pattern among 

crew members. Because the shift supervisor and board operators are attending to separate 

sources of information, the need to keep each other informed to insure a common frame of 

reference is potentially more important, but more difficult. More research is needed to 

understand the factors that contribute to a shared frame of reference among crew inembers and to 

establish the impact on crew performance and reliability of the degree to which crews share a 

common frame of reference as opposed to maintaining different perspectives. 

5.7 Implications for Training 

The study raised two issues with respect to training. One deals with training on the use of the 

new HSI systems in handling emergencies. The focus of the training was on handling events. 

There was little focus of attention on training how to most effectively utilize the HSI systems in 

handling the events. An open question is whether operator performance in emergencies would be 

improved if there was explicit training on effective use of the available HSI systems. 

The second training issue raised by the study deals with training crew communication skills 

aimed at developing shared situation awareness among crew members. The study reinforced the 

principle that in a control room environment, key information is distributed among crew 

members. The quality of decision-making depends on the ability of everyone of the team to be 

aware of important control actions about to be taken and to evaluate the appropriateness of the 

actions based on their knowledge and perspective. We observed a number of critical incidents 

where inappropriate control actions suggested by the shift supervisor based on the CBP were 

caught by the board operators or shift engineer who had different sources of information 

available to them. The study further highlighted that advanced HSI systems increase the level of 

knowledge and understanding of individual crew members through increased availability of 

information. On the other hand, they can increase the compartmentalization of knowledge, so 

that different crew members are more likely to be in possession of different information. This 

:places an increased premium on effective communication among crew members. 

'We observed wide variability among crews in the extent and content of their communication. 

'The impact of the differences in communication style on individual and crew situation awareness 

and quality of team decision making is not well understood. More research is required on this 

58 
5 DISCUSSION 

topic in order to provide better guidance for development and evaluation of training programs on 

crew decision making. 

Training on Use of the HSI Systems 

While operators were introduced to the features of the HSI systems, there was no effort within 

this timeframe to focus on how to best exploit the systems in different situations, either normal, 

abnormal, or emergency. The focus of simulator training was on handling emergencies properly. 

There was little discussion on how the HSIs were used during a scenario, or how they could have 

been used better. It was up to the individual crew members to decide how to use the HSIs. As a 

consequence, there appeared to be lots of individual variability within and between crews with 

respect to extent to which they used the different systems (e.g., the graphic display system) and 

how they used them. 

The lack of explicit training in the use of the HSI systems reflects variously (a) an acceptance 

and/or expectation that operators will develop familiarity with the various features of the systems 

informally on their own; that there is no formal need to teach this type of knowledge and skill, 

and (b) an acceptance and/or expectation that different individuals will tailor the systems to their 

own personal style (e.g., that different operators will choose to put up different graphic displays 

or different collections of windows on their VDUs). A similar finding was observed at other 

plants where new computer-based HSI systems were introduced (Mumaw, Vicente, Roth, and 

Burns, 1995). In that study it was found that operators had developed clear strategies for 

managing the HSI that evolved over time and were disseminated across operators and crews, but 

it was all transmitted informally. 

While it is important to guard against over-regimentation and to allow operators flexibility in 

tailoring their VDU workspace to their particular needs, there may also be value in formally 

introducing operators to effective ways to use the computerized system. Further, such flexibility 

increases the demands on operators to engage in tasks not directly related to process control. As 

noted above, operators are often reluctant to perform such tasks in high-workload situations. 

Training and Guidance for Team Situation Awareness 

'The implications of the above discussion for team training include: 

It is important to train the shift supervisor to keep the crew informed as he moves 

through the procedures. 

It is important for operators to train as a cohesive (intact) crew, since it is important 

for team members to learn the specific communication styles of their team-mates 

59 
5 DISCUSSION 

A useful technique may be to cross-train operators so that each is aware of the 

communication needs and burdens of the other (e.g., have the board operators' 

experience using the CBP). 

It is not clear whether the natural demands for interaction that emerge from using the 

procedure (e.g., the need for the shift supervisor to ask the board operators to take 

action) are sufficient to provide a common frame of reference, or whether more 

guidance on communication should be provided. 

There may be differences in training requirements between "new" crews and 

"refresher" training. Current crews have had the opportunity to learn the structure and 

logic of the paper procedures that may help them transfer to computerized procedures. 

The same level of expertise may have to be provided for new crews. 

It is important to train on paper procedures and on the transition from HSI systems to 

paper procedures. 

60 
6 CONCLUSIONS AND RECOMMENDATIONS 

The introduction of computer-based HSI systems significantly affects crew structure and crew 

communication. These changes have implications for human performance and reliability. The 

fact that the crew is using multiple independent sources of information and multiple independent 

perspectives may increase the crew reliability by increasing the probability of detecting and 

correcting errors in situation assessment, thus reducing the potential for errors of intention. 

However, the improvement in performance and reliability depends on the ability of the crew to 

effectively communicate and maintain a shared situation awareness. Thus, there is need for 

research to better understand the implications of the types of changes in crew structure observed 

in this study on communication and individual and team situation awareness. The knowledge 

gained in such research can support the development of guidance on effective crew structures and 

styles of crew communication for high-reliability team performance. 

Based upon the findings and issues discussed in this report, the following recommendations are 

made regarding future research: 

1. This study examined the initial introduction of new HSIs to operators for use in handling 

events in a dynamic, full-mission simulation. A longitudinal investigation of how new 

HSI systems are implemented over time to observe full transition and operator acceptance 

in the plant and in training exercises is recommended. 

2. The research can be extended to more challenging conditions. For example, the use of 

the systems for handling more complex events should be investigated. Such complex 

events can include situations where there are potentially misleading indicators (e.g., Roth, 

Mumaw, and Lewis, 1994). Another situation can be failure of the new computerized 

HSIs at a more cognitively-demanding time, such as when the crew is far into an event 

and there are multiple aspects of the EOPs being followed, with numerous steps of 

continuous applicability being monitored, and an extensive procedure history to transfer 

to paper. 

3. Our study utilized naturalistic observational and interview methodologies. A more 

formal experimental investigation of the effects of paper vs. computerized procedures 

would provide the opportunity to systematically quantify the magnitude of the effects and 

issues discussed above. An experiment could address: 

Quantitative differences in performance: 

- Time to complete critical steps 

- Errors of execution, such as skipping steps, missing transitions, and missing 

cautions 

61 
6 CONCLUSIONS AND RECOMMENDATIONS 

0 Measure the situation awareness of crew members, such as board operators, shift 

supervisors and shift engineers 

Measure the effects on communication to determine: 

- What the shift supervisor should say as he reads the procedure 

- Where the acceptable limits are for a board operator taking action without alerting 

the shift supervisor and getting his approval 

- Where the acceptable limits are for a board operator to decide not to inform the 

shift supervisor that an action was taken 

- If the communication that takes place when the shift supervisor has to ask the 

board operator to take an action is enough to keep the shift supervisor and board 

operators on the same page 

- If and when there should be hold points to make sure everyone is on the same 

Page 

Measure the ability of the crew to detect and recover when the procedure goes astray 

Measure the ability of the crew to transition to paper 

4. As noted above, research addressing a better model of what constitutes good crew 

performance is needed to better understand the implications to crew reliability of the 

types of effects that were observed in this study. Understanding of the effect on crew 

roles and changes in communication style caused by new technology will be supported by 

such a model. In addition, research on effective crew structure and how best to train it 

will be important to providing a basis from which to develop guidance in this area. 

5. Evaluation of the types of graphic displays a new HSI provides and their use in normal 

operations for "routine" monitoring is recommended. The types of scenarios observed in 

our study involved plant transitions and disturbances. As such, some of the displays that 

may be used in typical monitoring were not used often. An important role of operators in 

contributing to overall plant reliability and safety is the ability to detect and handle 

equipment malfunctions early, before they lead to a plant trip situation. As a result, it is 

important to understand the impact of the new HSIs on the types of routine monitoring 

that occur during normal operations. 

62 
7 REFERENCES 

Billings, C., "Aviation Automation: The Search for a Human-Centered Approach," Erlbaum, 

Hillsdale, NJ, 1996. 

Brown, W., O'Hara, J., and Higgins, J., "Advanced Alarm Systems: Revision of Guidance and its 

Technical Basis," NUREG/CR-6684, U.S. Nuclear Regulatory Commission, Washington, DC, 

2000. 

Carrera, J. P. et al., "Simulator Testing of the Westinghouse AWARE Alarm Management 

System," IAEA Specialists Meeting on Experience and Improvements in Advanced Alarm 

Annunciation Systems in Nuclear Power Plants, Chalk River, Ontario, Canada, September 1996. 

Easter, J. R. and Lot, L., "Back-Fitting a Fully Computerized Alarm System into a Westinghouse 

PWR: A Progress Report," IEEE Fifth Conference on Human Factors and Power Plants, 

Monterey, CA, June 7-1 3, 199 1. 

Easter, J.R. and Watson, C.D., "Display Design for Real-Time Process Control: The Engineering 

Beneath the Art," IEEE Sixth Conference on Human Factors and Power Plants, Orlando, FL, 

June 8-12, 1997. 

Elm, W.C. and Woods, D.D., "Getting Lost: a Case Study in Interface Design," in Proceedings 

of the Human Factors Society, 2gth Annual Meeting, Human Factors Society, Santa Monica, CA, 

pp. 927-931, 1985. 

Kauffman, J.V., Lanik, G.F., Trager, A. and Spence, R.A., "Operating Experience Feedback 

Report - Human Performance in Operating Events," NUREG-1275, Office for Analysis and 

Evaluation of Operational Data, U.S. Nuclear Regulatory Commission, Washington, DC, 1992. 

Layton, C., Smith, P.J. and McCoy, E., "Design of a Cooperative Problem-Solving System for 

En-Route Flight Planning: An Empirical Evaluation," Human Factors, Vol. 36, pp. 94-1 19, 

1994. 

Levett, M. J. et al., "AWARE, an Advanced Alarm Management System - Recent Developments 

and Operational Experience," Institute of Nuclear Engineers (INE), Second International 

(Conference on Control and Instrumentation in Nuclear Installations, Churchill College, 

'University of Cambridge, U.K, 1995. 

ILipner, M.H., "Computerized Procedures System for Emergency Procedure Execution 

Monitoring," International Atomic Energy Agency Technical Committee Meeting on Advanced 

Control and Instrumentation Systems in Nuclear Power Plants: Design, Verification and 

'Validation, Helsinki, Finland, June 20-23, 1994. 

63 
7 REFERENCES 

Lipner, M.H. and Kerch, S.P., "Operational Benefits of an Advanced Computerized Procedures 

System," Nuclear Science Symposium and Medical Imaging Conference, Norfolk, VA, October 

30 - November 5,1994. 

Lipner, M.H. and Orendi, R.G., "Issues Involved with Computerizing Emergency Operating 

Procedures," American Nuclear Society AI91 : Frontiers in Innovative Computing for the 

Nuclear Industry, Jackson, WY, September 15-18, 1992. 

Mosier, K.L., "Myths of Expert Decision Making and Automated Decision Aids," in 

Zsambok, C.E. and Klein, G. (Eds.), Naturalistic Decision Making, Erlbaum, Hillsdale, NJ, 

1997. 

Mosier, K.L., Palmer, E.A., and Degani, A., "Electronic Checklists: Implications for Decision 

Making," in Proceedings of the Human Factors Society 36th Annual Meeting, Human Factors 

Society, Santa Monica, CA, pp. 7- 11, 1992. 

Mumaw, R.J., Vicente, K.J., Roth, E.M., and Burns, C.M., "A Study of Operator Monitoring 

During Normal Operations," Final Report AECB, Atomic Energy Control Board, Ottawa, 

Canada, 1995. 

O'Hara, J., Brown, W., Higgins, J., and Stubler, W., "Human Factors Engineering Guidelines for 

the Review of Advanced Alarm Systems," NUREG/CR-6105, U.S. Nuclear Regulatory 

Commission Washington, DC, 1994. 

O'Hara, J., Brown, W., Hallbert, B., Skiking, G., Persensky, J., and Wachtel, J., The Effects of 

Alarm Display, Processing, and Availability on Crew Performance," NUREG/CR-669 1 , 

U.S. Nuclear Regulatory Commission, Washington, DC, 2000. 

O'Hara, J., Brown, W., Stubler, W., Wachtel, J., and Persensky, J., "Human-System Interface 

Design Review Guideline," NUREG-0700, Rev. 1 , U.S. Nuclear Regulatory Commission 

Washington, DC, 1996. 

O'Hara, J., Stubler, W. and Nasta, K., "Human-System Interface Management: Effects on 

Operator Performance and Issue Identification," BNL Draft Letter Report W6546- 1- 1-7/97, 

Brookhaven National Laboratory, Upton, NY, 1997. 

O'Hara, J., Stubler, W., and Higgins, J., "Hybrid Human System Interfaces: Human Factors 

Considerations," BNL Report J6012-T1-4/96, Brookhaven National Laboratory, Upton, NY, 

1996. 

64 
7 REFERENCES 

Qrasanu, J., "Shared Problem Models And Flight Crew Performance." in N. Johnston, 

N. McDonald, and R. Fuller (Eds.) Aviation Psychology In Practice, pp. 255-285, 

Ashgate Publishing, Aldershot, UK, 1994. 

Orasanu, J., and Fischer, U., "Team Cognition in The Cockpit: Linguistic Control of Shared 

Problem Solving," in Proceedings of the Fourteenth Annual Conference of the Cognitive Science 

Society, pp. 189-194, Erlbaum, Hillsdale, NJ, 1992. 

Rasmussen, J., "Information Processing And Human-Machine Interaction: An Approach to 

Cognitive Engineering," North Holland, NY, 1986. 

Roth, E.M., Bennett, K. and Woods, D.D., "Human Interaction with an 'Intelligent' Machine," 

International Journal of Man-Machine Studies, Vol. 27, pp. 479-525, 1987, reprinted in 

G. Mancini, D.D. Woods, and E. Hollnagel (Eds.), Cognitive Engineering in Complex Dynamic 

Worlds, Academic Press, 1988. 

Roth, E.M., Brockhoff, C.S., Rusnica, LA., Kenney, S., Kerch, S.P., and Thomas, V.M., 

"Rapid Prototyping and Simulator Evaluation of a Wall Panel Overview Display," in Proceedings 

of the 1997 EEE Sixth Conference on Human Factors and Power Plants), pp. 18-19, Institute of 

Electrical and Electronic Engineers, Orlando, FLY June 8-13, 1997. 

Roth, E.M., Malin, J.T., and Schreckenghost, D.L., "Paradigms For Intelligent Interface Design," 

in Helander, M.G., Laundauer, T.K. and Prabhu, P.V. (Eds.), Handbook of Human-Computer 

Interaction, Second Edition, Elsevier Science, North-Holland, Amsterdam, pp. 1 177- 1201 , 1997. 

Roth, E.M., Mumaw, R.J., and Lewis, P.M., "An Empirical Investigation of Operator 

Performance in Cognitively Demanding Simulated Emergencies," NUREGKR-6208, 

U.S. Nuclear Regulatory Commission, Washington, DC, 1994. 

Roth, E.M., Mumaw, R.J., and Stubler, W.F., "Human Factors Evaluation Issues for Advanced 

Control Rooms: A Research Agenda," in Proceedings of the 1992 EEE Fifth Conference on 

Human Factors and Power Plants, Monterey, CA, June 7-1 1 , 1992. 

Roth, E.M., Mumaw, R.J., Vicente, K.J. and Burns, C.M., "Operator Monitoring During Normal 

Operations: Vigilance or Problem Solving?" in Proceedings of the 41st Annual Meeting of the 

Human Factors and Ergonomics Society, Albuquerque, NM, September 22-26, 1997. 

Mas, E., Cannon-Bowers, J.A. and Johnston, J.H., "How Can You Turn a Team of Experts into 

#an Expert Team?: Emerging Training Strategies," in Zsambok, C.E. and Klein, G. (Eds.), 

:Naturalistic Decision Making, Erlbaum, Hillsdale, NJ, pp. 359-370, 1997. 

65 
7 REFERENCES 

Stabler, W., Higgins, J., and O'Hara, J., "Evaluation of the Potential Safety Significance of 

Hybrid Human-System Interface Topics," BNL Technical Report J60 12-T2-6/96, Brookhaven 

National Laboratory, Upton, NY, 1996. 

Tenney, Y.J., Rogers, W.H. and Pew, R.W., "Pilot Opinions on High Level Flight Deck 

Automation Issues: Toward the Development of a Design Philosophy," NASA Contractor Report 

4669, NASA Langley Research Center, Huntington, VA, 1995. 

Vicente, K.J., Mumaw, R.J., Roth, E.M., and Burns, C.M., "A Field Study of Operator 

Monitoring: Strategies for Adapting Control Room Technology," in Proceedings of Cognitive 

Systems Engineering in Process Control '96, Kyoto University, Kyoto, Japan, pp. 222-229, 1996. 

Vicente, K.J., Mumaw, R.J. and Roth, E.M., "Cognitive Functioning of Control Room Operators 

- Final Phase," Final Report AECB, Atomic Energy Control Board, Ottawa, Canada, 1997. 

Vicente, K. J. and Rasmussen, J., "Ecological Interface Design: Theoretical Foundations," in 

lEEE Transactions of SMC, Vol. 22, pp. 589-607, 1992. 

Woods, D.D. and Watts, J.C., "How not to have to Navigate Through too Many Displays," in 

Helander, M. G., Laundauer, T. K. and Prabhu, P. V. (Eds.), Handbook of Human-Computer 

Interaction, Second Edition, Elsevier Science, North-Holland, Amsterdam, pp. 6 17-650, 1997. 

Wreathall, J., Reason, J. and Dougherty, Jr., E.M., "Latent Failures and Human Performance in 

Significant Operating Events," Draft report, U.S. Nuclear Regulatory Cornmission, 

Washington, DC, 1993. 

66 
APPENDIX A 

e 

e 

2 

0 

General Information Topics Addressed Through Discussions with 

Utility, Design, Operations, Maintenance, and Training Personnel 

GENERAL QUESTIONS 

Background 

Why did the utility decide to initiate this design modification program? 

- What factors led to the decision to upgrade? 

Scope of the upgrade? 

- What are the current control room human-system interfaces (HSIs) like? 

- What control room functions are being addressed by the new systems? 

- Are any of the new systems "safety grade" or do they interface with "safety grade" 

systems? 

Are the new systems an enhancement or a replacement of the current control room HSIs? 

- Will the old and new continue to coexist after the upgrade? 

- Will the new system replace existing HSIs? 

- What additional capabilities are provided? 

Design 

What are the design goals of the new system? 

- Was there a philosophy of use for the system (e.g., who uses it, when is it used, how 

is it used in conjunction with the existing control room systems)? 

What was the design basis of the new system (e.g., existing product lines, customer 

requirements) ? 

Generally, how were the new systems designed and their components integrated? 

How was the software development process handled? 

- What was the requirement for software reliability? 

- How will software problems be handled once the systems have been installed? 

A- 1 
APPENDIX A 

a 

a 

a 

e 

What was the involvement of utility personnel: management, engineering operations, 

training, and maintenance personnel in the design process? 

What were the considerations for integration with the rest of HSI? 

- Are any changes required in the control room HSIs due to the new system? 

From a designer's standpoint, what werehe the major benefits? 

- Are the potential benefits fully realized (and if not, what is needed to fully realize the 

benefits of the new system)? 

From a designer's standpoint, what werehe the significant challenges in this type of 

program? 

- Were there any surprises? 

From a designer's standpoint, what are some of the "lessons learned" from the project? 

Design Evaluation 

What was the verification and validation process employed? 

- What types of evaluations were performed of the new system? 

- How was the acceptability of new technology demonstrated (e.g., computer-based 

procedure system)? 

- How was software V&V performed? 

From an evaluator's standpoint, what werehe the significant challenges in this type of 

program? 

- Were there any surprises? 

From an evaluator's standpoint, what are some of the "lessons learned" from the project? 

A-2 
APPENDIX A 

4 

0 

5 

0 

Implementation and Transition 

How wadis the upgrade implemented? 

- Were all HSI components changed at once or were they changed in phases? 

- Will the old system be left in place during the transition? 

What activities were/will be performed to facilitate use of the upgrade by operations, 

maintenance, and engineering personnel? 

Operator Training 

How will operators be trained to use the new system? 

- Are there any differences in the "philosophy of use" or training requirements for using 

this system in comparison to the existing control room systems? 

How will their regular training be changed in the long term? 

If the upgrade is phased, how will operators be trained for interim control room 

configurations between the current control room and the final one after the new system is 

completely installed? 

Is operator acceptance an issue? 

From a trainer's standpoint, what werehe the major benefits? 

- Are the potential benefits fully realized (and if not, what is needed to fully realize the 

benefits of the new system)? 

From a trainer's standpoint, what werehe the significant challenges in this type of 

program? 

- Were there any surprises? 

From a trainer's standpoint, what are some of the "lessons learned" from the project? 

A-3 
APPENDIX A 

Impact on Crew Operations, Staffing, Qualifications, Communication, and 

Coordination 

How do the new systems impact what tasks operators must perform and in the way events 

are handled as a team? 

In general, what are the effects of the new systems on task performance, situation 

assessment, and workload? 

Are there any differences in the overall decision-making process? Are there any changes 

in the way individual operators contribute to the decision-making process? 

What is the impact on crew communication and coordination from the way operations 

were handled before? 

Were there any differences in the way you kept track of the progress on the event and 

understood the situation? 

Do the new systems introduce new possibilities for errors to occur? 

Are there any changes in staffing envisioned due to the upgrade? 

Do the new systems require any modifications to operator qualifications? 

From an operations standpoint, what werehe the major benefits? 

- Are the potential benefits fully realized (and if not, what is needed to fully realize the 

benefits of the new system) 

From an operations standpoint, what werehe the significant challenges in this type of 

program? 

- Were there any surprises? 

From an operations standpoint, what are some of the "lessons learned" from the project? 

Configuration, Maintenance, and Upgrade 

Are there configuration modes for the new system and how are they accomplished and 

controlled? 

A-4 
APPENDIX A 

0 How is the system maintained? 

- Who is responsible for software maintenance? 

- How is it different from current maintenance activities? 

- Are the qualifications for maintenance personnel affected by the new systems? 

0 How are upgrades, such as software revisions, accomplished and controlled? 

8 Regulatory Review Considerations 

0 Is the new system being addressed by the regulatory authorities? 

- Which technical areas are involved (eg, I&C, HFE)? 

- Has the system received regulatory approval? 

- Are there fwther activities required before final approval is obtained? 

e If a regulatory review has been performed, what has been the technical approach? 

9 Follow-up Evaluations 

0 Is there a plan to conduct follow-up evaluations or to collect lessons-learned as the 

systems are implemented and operational experience is gained over the next year or so? 

I1 DESIGN-SPECIFIC QUESTIONS 

2 Computer-Based Procedure (CBP) System 

a What do you see as its major impact on operator performance in comparison to paper 

procedures? 

- Response times, situation awareness, workload, communication, error opportunity 

- Has the system had differential impact on the shift supervisor vs. board operator? 

0 How is situation assessment impacted by CBPs as compared with paper procedures? 

- Ability to view multiple procedures and associated plant data simultaneously 

0 Was the viewing area provided by VDUs sufficient? 

- How does reading procedural text from the VDU compare with paper procedures? 

A-5 
APPENDIX A 

2 

a 

* 

e 

* 

How is navigation (within one procedure, or among multiple procedures and related 

information) within the CBP system handled? 

How do operators compare information provided by the CBP system with other sources 

of information in the control room? 

Is the CBP system and its HSI consistent with the rest of the HSI? 

How will the transfer between paper procedures and CBPs that are used only in 

emergency conditions be handled? 

How will the transfer from CBPs to paper procedures upon loss of CBPs be handled? 

What would you like to see in the next generation of this system? 

Alarm System 

What do you see as its major impact on operator performance? 

- Response times, situation awareness, workload, communication, error opportunity 

- Has the system had differential impact on the shift supervisor vs. board operator? 

Are definitions as to what constitutes an alarm clear and reasonable? 

Are the methods used for alarm reduction appropriate? 

What is the basis for alarm priority and can important alarms be easily found? 

- Since the system does not display all alarms at one time, are hidden alams a concern? 

How are alarm management functions handled? 

- The new alarm system provides for greater interaction with the alarm system 

Was the viewing area provided by VDUs sufficient? 

- How is navigation accomplished? 

How are the different alarm systems used? 

How is the transition between the different types of alarm displays handled? 

A-6 
APPENDIX A 

* 

* 

How is the transition between the alarm displays and other plant information systems 

handled? 

What would you like to see in the next generation of this system? 

Display System 

What is the role of the new display system and how does it relate to the existing displays? 

What is the scope of the display system? 

- Types of displays included 

- Number of displays 

When do operators use the VDU displays and who uses them? 

- Overview displays 

- Physical displays 

- Functional displays 

What do you see as its major impact on operator performance? 

- Response times, situation awareness, workload, communication, error opportunity 

- Has the system had differential impact on the shift supervisor vs. board operator? 

Are the display formats and page arrangements to operator tasks? 

Was the viewing area provided by VDUs sufficient? 

- How is navigation accomplished? 

0 What would you like to see in the next generation of this system? 

4 S ystems-Integration 

0 How well are the CBP, alarm, and display systems integrated? 

- Is it easy to transition between them? 

How well are the existing control room HSIs and the new system integrated? 

A-7 
APPENDIX A 

- Is it easy to transition between them? 

0 Was it possible to compare information provided by the new systems (CBP, AAS, and the 

Graphic Display System) with the conventional instrumentation (such as indicators, 

instruments, and pen recorders) on the main boards in order to check for any 

discrepancies? 

A-8 
APPENDIX B 

Question Used in Operator Interviews 

Opinions on the New Control Room Systems 

e Now that you have experienced all three new systems [CBP, MS, Graphic Display 

System], overall what features do you think will make a difference in how you do your 

job? 

- Are there any specific features that especially caught your attention? 

e CBP 

- What do you see as the major impact of the computer-based procedure system on 

operator performance (in comparison to paper procedures)? 

- How does procedure use compare with use of paper? 

e MS 

- What do you see as the major impact of the new alarm system on operator 

performance? 

- You will now have three separate alarm systems in the control room, what do you see 

as the role for each? 

e Graphic Display System 

- What do you see as the major impact of the new display system on operator 

performance? 

- When do operators use the VDU displays and who uses them? 

e How well are the CBP, alarm, and display systems integrated with each other and the 

existing controls and displays? 

e Much of the information provided by the new systems was presented on VDUs. Was the 

viewing area provided by VDUs sufficient? 

e Information presented by the new systems involves software processing. Is it easy to 

understand how the software processes data and what the displayed information means? 

How confident are you in the accuracy of the information provided by the new systems? 

B-1 
APPENDIX B 

e Was it possible to compare information provided by the new systems (CBP, AAS and the 

Graphic Display System) with the conventional instrumentation (such as indicators, 

instruments, and pen recorders) on the main boards in order to check for any 

discrepancies? 

e If you were designing the next version of this system, what improvements would you 

recommend? 

e In what situations do you think this system is most useful (e.g., normal plant conditions, 

minor abnormalities, plant faults that result in a trip, complex multiple fault situations)? 

e Do you have any concerns about using this new system? Things that you worry about? 

Impact on OperationsKrew Performance 

e Do you think that CBP, AAS and the Graphic Display System change how you work as a 

team to respond to incidents? In what ways? Do you think these changes make you better 

able to handle an incident? Do you have any concerns? 

e You ran one simulated scenario where none of the computerized systems were available 

(simulating a computer system failure). The following questions relate to the differences 

between the handling the event with and without the new systems. 

- Did you notice any differences in the way you handled the event as a team? 

- Was there an impact on the coordination of individual crew members and on 

communication between the group members? 

- Were there any differences in the way you kept track of the progress of the event and 

understood the situation? 

- Was there any difference in the overall decision making process? Were there any 

changes in the way individual operators contribute to the decision-making process? 

e We are interested in how operators work as a crew to detect and correct errors. In the 

scenarios that you ran this week can you think of any cases where you felt confused or led 

astray by the new computerized systems? In those cases how did you get back on track? 

Training 

0 Do you think based on the training you have gotten so far, that you would feel confident 

using these new systems in an actual incident? 

B-2 
APPENDIX B 

0 What additional activities or training do you think is needed before you will feel 

confident in using these new systems in an actual incident? 

Operator Input to Design and Transition 

We are trying to study how control rooms are being updated (how operator input is 

obtained and used, the transition process for introducing the new systems) in order to 

provide recommendations for future upgrades to insure that operator concerns are taken 

into account. What are some things you think have been good about the upgrade process 

for your control room that you would recommend to other plants? Are there things you 

would recommend be done differently? 

B-3 
NRC FORM 335 US. NUCLEAR REGULATORY COMMJSSION 1. REPORT NUMBER 

(2-89) (Assigned by NRC, Add Vol., Supp., Rev., 

and Addendum Numbers, if any.) NRCM 1102, 

3201,3202 

- NUREG/CR-6749 

2. TITLE AND SUBTITLE 

BIBLIOGRAPHIC DATA SHEET 

(See instructions on the reverse) 

 

BNL-NUREG-52638 

Integrating Digital and Conventional Human-System Interfaces: Lessons Learned from a 

Control Room Modernization Program 3. DATE REPORT PUBLISHED 

MONTH YEAR 

September 2002 

4. FIN OR GRANT NUMBER 

JCN W6546 

5. AUTHOR(S) 6. TYPE OF REPORT 

Emilie Roth (Roth Cognitive Engineering) and John O'Hara (Brookhaven National Laboratory) Technical 

7. PERIOD COVERED (Inclusive Dates) 

- I 

NRC FORM 335 (2-89) This form was electronically produced by Elite Federal Forms, Ini 

I 

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (If NRC, provide Division, Office or Region. U.S. Nuclear Regulatory Commission, and mailing address; if contractor, 

provide name and mailing address.) 

12. KEY WORDSIDESCRIPTORS (List words or phrases that will assist researchers in locating the report.) 

Human factors engineering, human-system interface, control room, alarm systems, procedures, plant 

modernization, digital systems 

Brookhaven National Laboratory Roth Cognitive Engineering 

13. AVAILABILITY STATEMENT 

unlimited 

14. SECURITY CLASSIFICATION 

(This Page) 

unclassified 

Energy Sciences and Technology Dept. 

Upton, NY 1 1973-5000 

89 Rawson Road 

Brookline, MA 02445-4509 

(This Report) 

I 16. PRICE 
UNITED STATES 

NUCLEAR REGULATORY COMMISSION 

WASHINGTON, DC 20555-0001 

OFFICIAL BUSINESS 

PENALTY FOR PRIVATE USE, $300 
Federal Recycling Program I' 
UNITED STATES 

WCLEAR REGuiAiORY COMMISSION 

WASHINGTON, DC 20555-0001 

OFFICIAL BUSINESS 

PENALTY FOR PRIVATE USE, $300 

i