NUREG/CR-6634 

BNL-NUJREG-52564

Computer-Based 

Procedure Systems: Technical

Basis and Human Factors 

Review Guidance 

Brookhaven National Laboratory 

U.S. Nuclear Regulatory Commission 

Office of Nuclear Regulatory Research 

Washington, DC 20555-0001
AVAILABILITY NOTICE 

Availability of Reference Materials Cited in NRC Publications

NRC publications in the NUREG series, NRC regu

lations, and Title 10, Energy, of the Code of Federal 

Regulations, may be purchased from one of the fol

lowing sources: 

1. The Superintendent of Documents 

U.S. Government Printing Office 

PO. Box 37082 

Washington, DC 20402-9328 

< http://www.access.gpo.gov/sudocs > 

202-512-1800 

2. The National Technical Information Service 

Springfield, VA 22161-0002 

<http://www.ntis.gov> 

1 -800-553-6847 or locally 703-605-6000 

The NUREG series comprises (1) brochures 

(NUREG/BR-XXXX), (2) proceedings of confer

ences (NUREG/CP-XXXX), (3) reports resulting from international agreements (NUREG/IA-XXXX), 

(4) technical and administrative reports and books 

[(NUREG-XX00) or (NUREG/CR-XXXX)], and (5) 

compilations of legal decisions and orders of the 

Commission and Atomic and Safety Licensing 

Boards and of Office Directors' decisions under 

Section 2.206 of NRC's regulations (NUREG

xxxx). 

A single copy of each NRC draft report for com

ment is available free, to the extent of supply, upon 

written request as follows: 

Address: Office of the Chief Information Officer 

Reproduction and Distribution 

Services Section 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

E-mail: <DISTRIBUTION@nrc.gov> 

Facsimile: 301-415-2289 

A portion of NRC regulatory and technical informa

tion is available at NRC's World Wide Web site:

<http://www.nrc.gov> 

After January 1,2000, the public may electronically 

access NUREG-series publications and other NRC 

records in NRC's Agencywide Document Access 

and Management System (ADAMS), through the 

Public Electronic Reading Room (PERR), link 

<http://www.nrc.gov/NRC/ADAMS/index.html>. 

Publicly released documents include, to name a 

few, NUREG-series reports; Federal Register no

tices; applicant, licensee, and vendor documents 

and correspondence; NRC correspondence and 

internal memoranda; bulletins and information no

tices; inspection and investigation reports; licens

ee event reports; and Commission papers and 

their attachments. 

Documents available from public and special tech

nical libraries include all open literature items, such 

as books, journal articles, and transactions, Feder

al Register notices, Federal and State legislation, 

and congressional reports. Such documents as 

theses, dissertations, foreign reports and transla

tions, and non-NRC conference proceedings may 

be purchased from their sponsoring organization. 

Copies of industry codes and standards used in a 

substantive manner in the NRC regulatory process 

are maintained at the NRC Library, Two White Flint 

North, 11545 Rockville Pike, Rockville, MD 

20852-2738. These standards are available in the 

library for reference use by the public. Codes and 

standards are usually copyrighted and may be 

purchased from the originating organization or, if 

they are American National Standards, from

American National Standards Institute 

11 West 42nd Street 

New York, NY 10036-8002 

<http://www.ansi.org> 

212-642-4900

DISCLAIMER

This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United 

States Government nor any agency thereof, nor any of their em ployees, makes any warranty, expressed or implied, or assumes

any legal liability or responsibility for any third party's use, or the 

results of such use, of any information, apparatus, product, or process disclosed in this report, or represents that its use by 

such third party would not infringe privately owned rights.
NUREG/CR-6634 

BNL-NUREG-52564 

Computer-Based 

Procedure Systems: Technical 

Basis and Human Factors 

Review Guidance 

Manuscript Completed: January 2000 

Date Published: March 2000 

Prepared by 

J. M. O'Hara, J. C. Higgins, W. F. Stubler/BNL 

J. Kramer/NRC 

Brookhaven National Laboratory 

Upton, NY 11973 

J. Kramer, NRC Project Manager 

Prepared for 

Division of Systems Analysis and Regulatory Effectiveness 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

NRC Job Code J6012
ABSTRACT 

Plant procedures are instructions to guide operators in monitoring, decision making, and controlling nuclear power 

plants. While plant procedures historically have been paper-based, computer-based procedures (CBPs) are being 

developed to support procedure use. CBPs have a range of capabilities that may support operators and reduce 

demands associated with paper procedures. The objective of this study was to establish human factors review 

guidance for CBP systems based on a technically valid methodology. While the study mainly addressed emergency 

operating procedures, much of the guidance developed applies to other types of procedures. First, a CBP 

characterization was developed for describing their key design features including both procedure representation 

and functionality. Then, the research on CBPs and related areas was reviewed. This information provided the 

technical basis on which the guidelines for design review were developed. The review guidelines address both the 

design process and the implementation of CBP systems. For some aspects of CBPs the technical basis was 

insufficient to develop guidance; these aspects were identified as issues to be addressed in future research.

iii NUREG/CR-6634
CONTENTS 

PagAe 

ABSTRACT. . ..................................................................... iii 

LIST O F FIGURES ... ................................................................... ix 

LIST OF TABLES.... ................................................................ xi 

EXECUTIVE SUMMARY................................................................. xiii 

PREFACE D....... ................................................................. xvii 

ACKNOW LEDGM ENTS ... .............................................................. xix 

ACRONYMS......................................................................... xxi 

PART 1: Guidance Development and Technical Basis 

1 INTRODU CTION ............................................................... 1-1 

1.1 B ackground .............................................................. 1-1 

1.2 Earlier NRC Work on Computer-Based Procedures .............................. 1-2 

1.3 Organization of the Report .................................................. 1-3 

2 O BJEC TIV E .................................................................... 2-1 

3 M ETHODOLOGY ............................................................... 3-1 

3.1 O verview ............................................................... 3-1 

3.2 Characterization of CBP Systems ............................................. 3-2 

3.3 Development of Technical Basis ............................................ 3-2 

3.4 Development and Documentation of Guidance .................................. 3-4 

3.5 Identification of Issues ..................................................... 3-4 

3.6 Peer Review ............................................................. 3-4 

4 CHARACTERIZATION OF CBP SYSTEMS ......................................... 4-1 

4.1 Representation of Procedures ................................................ 4-2 

4.2 Functionality of Procedures ................................................. 4-4 

4.3 Management and Support of Procedures ....................................... 4-6 

4.4 CBP Hardware ........................................................... 4-6 

4.5 Backup System for Procedures ............................................... 4-6 

4.6 Integration with Other HSI Components ....................................... 4-7 

5 TECHNICAL BASIS FOR DEVELOPING CBP GUIDELINES ........................... 5-1 

5.1 Cognitive Tasks Associated With Operating the Plant ............................. 5-1 

5.2 Issues With Paper-Based Procedures and Implications for Computerization ........... 5-9 

5.3 Existing Guidance for CBP Systems .......................................... 5-18

V NUREG/CR-6634
CONTENTS (Continued) 

Pane 

5.4 Research on Computer-Based Procedure Systems ................................ 5-22 

5.4.1 Empirical Evaluations of CBPs Based on Personnel Performance ............ 5-22 

5.4.2 Analytical Evaluations of CBPs ....................................... 5-32 

5.4.2.1 Performance-Model Analyses ................................. 5-32 

5.4.2.2 Risk-Informed Analyses ..................................... 5-33 

5.4.3 Expert Opinion .................................................... 5-36 

5.5 Other Related Research on Computerization of Task Aids ......................... 5-39 

5.5.1 General Comparisons of Hard-Copy and Computer-Based Task Performance ... 5-39 

5.5.2 Computerized Operator Support Systems (COSSs) ........................ 5-42 

5.6 Summary and Discussion ................................................... 5-46 

5.6.1 Supervisory Control and Procedure Use ................................ 5-46 

5.6.2 The Effects of CBPs on Crew Performance .............................. 5-46 

5.6.3 C B P Issues ....................................................... 5-49 

6 DEVELOPMENT OF GUIDANCE .................................................. 6-1 

6.1 CBP Design Process Review Guidance ........................................ 6-1 

6.2 CBP HFE Design Review Guidelines ......................................... 6-2 

7 SU M M A RY ................................................................. 7-1 

8 REFERENCES ............................................................... 8-1 

PART 2: CBP Guidelines For Design Review 

9 GUIDANCE FOR CBP DESIGN PROCESS REVIEW .................................. 9-1 

9.1 HFE Program M anagement ................................................. 9-1 

9.2 Operating Experience Review ............................................... 9-2 

9.3 Functional A nalysis ....................................................... 9-2 

9.4 Task Analysis ......................................................... 9-3 

9.5 Staffing ............................................................. 94 

9.6 Human Reliability Analysis ........ 9-4 

9.7 Human-System Interface Design .............................................. 9-5 

9.8 Procedure Development .................................................... 9-6 

9.8.1 Scope of Procedures ................................................ 9-6 

9.8.2 Bases of Procedures ................................................ 9-6 

9.8.3 Technical Information ............................................. 9-8 

9.8.4 M aintenance of Procedures .......................................... 9-9

NUREG/CR-6634 vi
CONTENTS (Continued) 

Page 

9.9 Training Program Development .............................................. 9-10 

9.10 Human Factors Verification and Validation ..................................... 9-11 

10 GUIDELINES FOR CBP HFE DESIGN REVIEW ...................................... 10-1 

10.1 Representation of Procedures ................................................ 10-1 

10.1.1 Identification of Procedures .......................................... 10-1 

10.1.2 Basic Steps ....................................................... 10-2 

10.1.3 Warnings, Cautions, Notes, and Supplementary Information ................ 10-4 

10.1.4 L ists ............................................................ 10-5 

10.1.5 Organization of Procedures ........................................... 10-5 

10.1.6 Formatting and Screen Layout ........................................ 10-6 

10.2 Functionality of Procedures ................................................. 10-7 

10.2.1 Supervision and Control of Procedures ................................. 10-7 

10.2.2 Monitoring and Assessment of Procedures .............................. 10-8 

10.2.3 M onitoring Users' Actions ........................................... 10-12 

10.2.4 Planning and Implementation ......................................... 10-12 

10.3 Management and Support of Procedures ....................................... 10-13 

10.3.1 Path M onitoring ................................................... 10-13 

10.3.2 N avigation ....................................................... 10-14 

10.3.3 H elp ............................................................ 10-14 

10.4 CBP Hardware ........................................................... 10-15 

10.5 Backup for CBP Procedures ................................................. 10-15 

10.6 Integration of CBPs with Other HSI Components ............................... 10-16 

APPENDIX A: Human Factors Engineering Program Review Model (NUREG-071 1) 

Element 8: Procedure Development .......................................... A-1 

APPENDIX B: High-Level Design Review Principles from NUREG-0700, Rev. I ................... B-I

vii NUREG/CR-6634
LIST OF FIGURES 

Paae 

3.1 Major Steps in the Development of NUREG-0700 Guidance .............................. 3-1 

3.2 Technical Basis and Guidance Development Process .................................... 3-3 

5.1 Hierarchical Influence of Human Activity on Plant Performance ........................... 5-2 

5.2 Generic Primary Tasks of a Supervisory Controller ...................................... 5-2

ix NUREG/CR-6634
LIST OF TABLES 

Page 

4.1 Levels of Automation of Procedure Functions .......................................... 4-5 

5.1 Deficiencies in Paper-Based Procedures .............................................. 5-12 

5.2 Attentional Demands of PBP and CBP Systems ........................................ 5-33 

9.1 Examples of Deficiencies in Paper-Based Procedures .................................... 9-3

xi NUREG/CR-6634
EXECUTIVE SUMMARY 

The Human-System Interface Design Review Guideline, NUREG-0700, Rev. 1 (O'Hara et al., 1996), was 

developed to provide guidance on human factors engineering (HFE) for the U.S. Nuclear Regulatory Commission 

(NRC). The NRC staff uses NUREG-0700 for (1) reviewing submittals of human-system interface (HSI) designs 

prepared by licensees or applicants for a license or design certification of a commercial nuclear power plant (NPP), 

and (2) undertaking HSI reviews that could be included in an inspection or other types of regulatory review of HSI 

designs, or incidents involving human performance. It describes those aspects of the HSI design review process 

that are important to identifying and resolving human engineering discrepancies that could adversely affect plant 

safety. NUREG-0700 also has detailed HFE guidelines for assessing the implementation of HSI designs. 

In generating NUREG-0700, Rev. 1, several topics were identified as "gaps" because there was an insufficient 

technical basis upon which to formulate guidance. One such topic is the integration of advanced HSI technology 

into conventional NPPs. The NRC is currently sponsoring research at Brookhaven National Laboratory (BNL) to 

(1) better define the effects of changes in HSIs brought about by incorporating digital technology on personnel 

performance and plant safety, and (2) develop HFE guidance to support safety reviews, should a review of plant 

modifications or HSIs be necessary. 

Based upon the literature, interviews, and site visits, O'Hara et al. (1996) identified changes in HSI technology and 

their potential effects on personnel performance. The topics were then evaluated for their potential safety 

significance (Stubler et al., 1996); computer-based procedures (CBPs) was one HSI technology that was found to be 

potentially safety significant. (The safety analysis is described in more detail in Section 5.4.2.2 of this report.) 

Plant procedures provide instructions to guide operators in monitoring, decision making, and controlling the plant. 

Historically, plant procedures have been paper-based and were not considered part of the HSI. Following the 

accident at Three Mile Island, the nuclear power industry recognized the importance of having technologically 

sound and easy-to-use procedures to handle major plant disturbances. For emergency operations, symptom-based 

procedures were established that enabled operating crews to restore and maintain the plant's safety functions 

without having to diagnose events or the specific causes of process disturbances. 

Paper-based procedures (PBPs) have characteristics that limit how information can be presented to the operators. 

These limitations include presenting information in sequential form, requiring numerous iterations through steps, 

and cautions or warnings that may not be applicable for all system states (Wourms and Rankin, 1994; Mampaey et 

al., 1988). PBPs also impose tasks on the operator that are not directly related to controlling the plant. To make 

transitions between procedure steps and documents, and maintain awareness of the status of procedures that are in 

progress, operators must handle, arrange, scan, and read PBPs in parallel with monitoring and control tasks. 

CBPs are being developed to support procedure management. CBPs have a range of capabilities that may support 

operators in controlling the plant and reduce the demands associated with PBPs. In their simplest form, CBPs 

show the same information via computer-driven video display units (VDUs). More advanced CBPs may include 

features to support managing procedures (e.g., making transitions between steps and documents, and maintaining 

awareness of procedures in progress), detecting and monitoring the plant's state and parameters, interpreting its 

status, and selecting actions and executing them. 

The objective of this study was to develop HFE review guidance for CBP systems based on a technically valid 

methodology. To support this objective, the following tasks were undertaken: 

* Development of a framework for characterizing key design features of CBP systems 

Development of a technical basis using research and analyses on human performance relevant to CBPs

xiii NUREG/CR-6634
EXECUTIVE SUMMARY

* Development of HFE review guidelines for CBPs in a format that is consistent with NUREG-0700, Rev. 1, and 

NUREG-0711 

* Identification of remaining CBP issues for which research was insufficient to support our development of NRC 

review guidance 

The status of each will be briefly addressed below. 

CBP System Characterization Framework 

For this study, CBP systems were narrowly defined to encompass computer systems that support procedure 

presentation and use. The focus was on the HFE aspects of CBPs, and not the I&C or software aspects (although 

the latter are important as well, and are described in other NRC regulatory and research programs). CBPs were 

characterized along the following dimensions: 

* Representation of Procedure Elements 

* Procedure Functionality 

* Interface Management and Support 

* CBP Hardware 

* Backup Systems for Procedures 

* Integration of CBP System with the HSI 

Development of the Technical Basis 

The effects of CBPs on crew performance were determined by examining three types of research: (1) empirical 

studies of CBPs where data on personnel performance were collected, (2) analyses of personnel performance using 

models, and (3) expert opinion about their postulated effects on personnel performance. 

The human performance research was organized into three categories: comparisons of CBP and PBP systems, 

observations of operators' use of CBPs, and comparisons of design characteristics of procedures. Several 

conclusions were made from comparing CBPs with PBPs: 

* Operators perform tasks more quickly. 

* Operators' overall cognitive workload is reduced. 

* Operators may make fewer errors in transitioning through procedures. 

* Operators may accept CBPs readily and find them easier to use. 

However, much of the human performance research had insufficient detail to evaluate its generalizability. Studies 

that were sufficiently documented had potential methodological weaknesses which limited their conclusiveness and 

generalization.

NUREG/CR-6634 xiv
EXECUTIVE SUMMARY

Personnel performance was analyzed with two classes of techniques: performance models and risk models. The 

performance models showed no clear advantage of CBPs over PBPs. Instead, they illustrated the importance of 

performance tradeoffs in assessing different procedure systems. In general, complexity and attentional demands 

were higher, while data retrieval was easier and task completion time was less for CBPs. Similarly, mixed results 

were obtained from the risk analyses. They illustrated the potential for these systems to improve performance by 

supporting such procedure-related activities as process monitoring, logic analysis, navigation, and place keeping. 

However, when poorly implemented, CBPs can reduce human reliability. 

Finally, the SME review of CBPs identified many positive aspects of their use on the crew's performance. 

However, they also identified a wide range of issues to be resolved in developing CBPs. The review highlighted 

the importance of considering HFE activities in CBP development, e.g., the integration of the CBP system with the 

other HSIs and with the overall operational philosophy of the plant. Thorough V&V programs were also 

emphasized. In general, these findings were consistent with the information discussed earlier. 

When considering all the results, we concluded that there is evidence that CBPs can support and enhance operator 

performance. However, important issues remain to be addressed both in research and in the development of 

individual systems. Thus, we repeat the advice of researchers and developers: CBP systems should be developed in 

such a way that their benefits and drawbacks can be fully evaluated for each specific system. CBPs have important 

impacts on NPP operations, some of which extend beyond those the designers intended. 

Reflecting this approach, we offer some general considerations for near-term approaches to CBP systems: 

* Support cognitive functions that may be distracting and error prone, such as 

- process monitoring 

- logic analysis (cautiously so not to underspecify the analysis and undermine operator's judgement) 

* Support procedure management, e.g., step completion, place keeping, transitioning between procedures 

Provide PBP backup systems and ensure similarity of CBPs and PBPs in order to (1) ensure confidence in 

near-term CBP applications, (2) enable operating experience to be gained, (3) minimize the impact on 

function allocation, (4) ease the training burdens associated with both systems, and (5) ensure successful crew 

performance when transitions to and from backups are necessary (minimize the potential for negative transfer 

or difficulties in performance) 

HFE Review Guidelines 

Guidance for the review of CBPs was developed to address the CBP design process and HFE design. Both types of 

guidance are needed for a design review. That is, while there was a sufficient technical basis to develop detailed 

guidance for design-implementation review, as is typical in NUREG-0700, several limitations in the technical basis 

were identified. Many issues (listed below) remain for which typical NUREG-0700 guidance could not be 

developed. Therefore, until the additional guidance is developed, these issues should be addressed for specific CBP 

systems using CBP design process guidance.

xv NUREG/CR-6634
EXECUTIVE SUMMARY

CBP Issues 

As noted above, several human performance issues associated with CBPs were identified. They represent topics for 

which research is necessary before developing guidance. From a regulatory review perspective, many of them can 

be dealt with on a case-by-case basis during the design process review. Briefly, the issues included the following: 

Methodological and Criterion Requirements for Evaluating CBP Effects - Most of the studies reviewed had 

methodological weaknesses which limited their conclusiveness and generalizability. This issue addresses the need 

to evaluate CBPs and their effects on crew performance comprehensively, to better understand them under a wide 

range of scenarios and complex situations, using varied measures of personnel and system performance. 

Role of Plant Personnel in Procedure Management - This issue addresses the need to determine how to design and 

review CBP systems (1) to allow operators to maintain an independent perspective and to recognize the 

procedure's contribution to higher-level safety goals, (2) to automate distracting and lower-level error-prone tasks, 

and (3) to monitor the crew's performance, especially when the crew and CBPs disagree. 

Team Performance - This issue addresses the requirement to explore the effect of CBPs on crew member's roles, teamwork, and communication. How CBPs can be designed to effectively promote both is considered as well. 

Situation Awareness, Response Planning, and Operator Error - This issue addresses the need to assess the effect of 

CBPs on situation awareness including: 

* Procedure management, such as status of procedure steps, how procedures are structured, and the current 

location within a procedure or between a set of procedures 

* The appropriateness of procedures for achieving high-level procedure goals 

* The plant's status 

Level ofAutomation of Procedure Functions - This issue addresses the need to evaluate the tradeoffs between 

automating procedure functions, e.g., the analysis of procedure step logic, and the operator's involvement, 

independence, and supervisory control. 

Keyhole Effects and Use of Multiple CBP Procedures - This issue concerns the requirement to evaluate the 

significance of the keyhole effect in situations where operators are required to be in multiple procedures and must 

access information in parallel. 

CBP Failure in Complex Situations - This issue involves the need to evaluate operator's management of the 

transition from CBPs to PBPs and back to CBPs under complex conditions, e.g., in a situation where operators are 

deep into the procedures, multiple procedures are open, many steps are completed, many are continuously 

applicable, and time and parameter steps are being monitored by the CBPs. 

Hybrid Procedure Systems - This issue addresses the need to evaluate any differential effects of having all plant procedures presented in a CBP system versus a hybrid system, e.g., EOPs presented using CBPs and all other 

procedures are paper-based. 

Specific CBP Design Features - This issue addresses the need to evaluate the relative effects of specific CBP 

design features on performance.

NUREG/CR-6634 xvi
PREFACE 

This report was prepared by Brookhaven National Laboratory for the Division of Systems Technology of the 

U.S. Nuclear Regulatory Commission's (NRC's) Office of Nuclear Regulatory Research. It is submitted as part of 

the requirements of the project Human Factors Topics Associated with Hybrid Human System Interfaces (NRC 

JCN J6012), specifically, as part of Task 3, "Develop Review Guidance." The NRC Project Manager is Joel 

Kramer and the BNL Principal Investigator is John O'Hara.

NUREG/CR-6634 xvii
ACKNOWLEDGMENTS 

The authors wish to express their sincere gratitude to our colleagues Bill Brown, Lew Hanes, Mike Fineberg and 

his review team at the Crew Systems Ergonomics Information Analysis Center (CSERIAC), and Bill Ruland, Greg 

Galletti, Jim Bongarra, Clare Goodman, Jim Stewart, and Rich Correia of the U.S. NRC for their review of the 

reports. These reviewers provided insightful comments and perspectives on the issues addressed in the report and 

their knowledge and understanding significantly contributed to the study. 

We also wish to thank Barbara Roland, Mary Anne Corwin, and Avril Woodhead for their preparation and careful 

technical editing of the report.

NUREG/CR-6634 xix
ACRONYMS 

ADS Automatic depressurization system 

AECB Atomic Energy Control Board (Canada) 

AECL Atomic Energy Canada, Limited 

ALWR Advanced Light Water Reactor 

ANS American Nuclear Society 

BNL Brookhaven National Laboratory 

BWR Boiling water reactor 

CALS Continuous acquisition and life-cycle support 

CANDU Canadian Deuterium Uranium Reactor 

CBP Computer-based procedure 

CFR Code of Federal Regulations (U.S.) 

COMPRO Westinghouse Computerized Procedure 

COPMA Computerized Operation Manual 

COPRO Computerized Procedure 

COSS Computerized operator support system 

CR Control room 

CSF Critical safety function 

DDD Detection-diagnosis-decision making 

DOD Department of Defense (U.S.) 

DOE Department of Energy (U.S.) 

DSIN Nuclear Installations Safety Directorate (France) 

EdF Electricite de France 

EOP Emergency operating procedure 

EOPTS Emergency Operating Procedure Tracking System 

EPRI Electric Power Research Institute 

GE General Electric 

GOMS Goals, operators, methods, and selection 

GTG General technical guidance 

HFE Human factors engineering 

HRA Human reliability analysis 

HSI Human-system interface 

I&C Instrumentation and control 

IAEA International Atomic Energy Association 

IPSN Institute for Nuclear Safety and Protection (France) 

IETM Interactive Electronic Technical Manuals 

ISLOCA Interfacing systems loss-of-coolant accident 

KBS Knowledge-based system 

LOCA Loss-of-coolant accident 

LWR Light water reactor

xxi NUREG/CR-6634
ACRONYMS 

M-MIS 

MCOSS 

MIDAS 

MMI 

NAS 

NASA-TLX 

NPP 

NRC 

P&ID 

PBP 

PEAM 

PRA 

PSN 

PWR 

RAI 

RCS 

RO 

SDT 

SME 

SPDS 

SRO 

TMI 

URD 

V&V 

VDU

Man-machine information system 

Mitsubishi Computerized Operator Support System 

Man-Machine Integrated Design and Analysis System 

Man-machine interface 

National Academy of Sciences 

National Aeronautics and Space Administration - Task Load Index 

Nuclear power plant 

Nuclear Regulatory Commission (U.S.) 

Piping and instrumentation diagram 

Paper-based procedures 

Portable Electronic Aid for Maintenance 

Probabilistic risk assessment 

Institute for Nuclear Safety and Protection (France) 

Pressurized water reactor 

Request for additional information 

Reactor coolant system 

Reactor operator 

Signal detection theory 

Subject-matter expert 

Safety parameter display system 

Senior reactor operator 

Three Mile Island (nuclear power plant) 

Utility Requirements Document 

Verification and validation 

Video display unit

NUREG/CR-6634 xxii
PART 1 

Guidance Development and Technical Basis

NUREG/CR-6634
1 INTRODUCTION

1.1 Background 

The Human-System Interface Design Review Guideline, NUREG-0700, Rev. 1, (O'Hara et al., 1996) was 

developed to provide guidance on human factors engineering (HFE) for the U.S. Nuclear Regulatory Commission 

(NRC). The NRC staff uses NUREG-0700 for (1) reviewing submittals of human-system interface (HSI) designs 

prepared by licensees or applicants for a license or design certification of a commercial nuclear power plant (NPP), 

and (2) undertaking HSI reviews that could be included in an inspection or other types of regulatory review of HSI 

designs, or incidents involving human performance. It describes those aspects of the HSI design review process 

that are important to identifying and resolving human engineering discrepancies that could adversely affect plant 

safety. NUREG-0700 also has detailed HFE guidelines for assessing the implementation of HSI designs. 

In generating NUREG-0700, Rev. 1, several topics were identified as "gaps" because there was an insufficient 

technical basis upon which to formulate guidance. One such topic is the integration of advanced HSI technology 

into conventional NPPs. The NRC is currently sponsoring research at Brookhaven National Laboratory (BNL) to 

(1) better define the effects of changes in HSIs brought about by incorporating digital technology on personnel 

performance and plant safety, and (2) develop HFE guidance to support safety reviews, should a review of plant 

modifications or HSIs be necessary. This guidance will be integrated into NUREG-0700 and provide the NRC's 

staff with the technical basis to help ensure that HSI designs or plant modifications do not compromise safety. 

The results of this project are expected to contribute to satisfying the NRC's goals of (1) maintaining safety, 

(2) increasing public confidence, (3) increasing regulatory efficiency and effectiveness, and (4) reducing 

unnecessary burden. 

Based upon the literature, interviews, and site visits, O'Hara et al. (1996) identified changes in HSI technology and 

their potential effects on personnel performance. The topics were then evaluated for their potential safety 

significance (Stubler et al., 1996); computer-based procedures (CBPs) was one HSI technology that was found to be 

potentially safety significant. (The safety analysis is described in more detail in Section 5.4.2.2 of this report.) 

Plant procedures provide instructions to guide operators in monitoring, decision making, and controlling the plant. 

Historically, plant procedures have been paper-based and were not considered part of the HSI. Following the 

accident at the Three Mile Island NPP, the nuclear power industry recognized the importance of having 

technologically sound and easy-to-use procedures to handle major plant disturbances. For emergency operations, 

symptom-based procedures were established that enabled operating crews to restore and maintain the plant's safety 

functions without having to diagnose events or the specific causes of process disturbances. The NRC and industry 

put a great deal of effort into the design and review of emergency operating procedures (EOPs) (American Nuclear 

Society, 1981; Barnes et al., 1989; Galletti and Sutthoff, 1992; NRC, 1982). More recently, studies of other 

operating procedures (e.g., normal and abnormal procedures) also have demonstrated the importance of these 

categories of procedures to plant safety (Grant et al., 1989). 

Paper-based procedures (PBPs) have characteristics that limit how information can be presented to the operators. 

These limitations include presenting information in sequential form, requiring numerous iterations through steps, 

and cautions or warnings that may not be applicable for all system states (Wourms and Rankin, 1994; Mampaey et 

al., 1988). PBPs also impose tasks on the operator that are not directly related to controlling the plant. To make 

transitions between procedure steps and documents, and maintain awareness of the status of procedures that are in 

progress, operators must handle, arrange, scan, and read PBPs in parallel with monitoring and control tasks.

1-1 NUREG/CR-6634
1 INTRODUCTION

CBPs are being developed to support procedure management. CBPs have a range of capabilities that may support 

operators in controlling the plant and reduce the demands associated with PBPs. In their simplest form, CBPs 

show the same information via computer-driven video display units (VDUs). More advanced CBPs may include 

features to support managing procedures (e.g., making transitions between steps and documents, and maintaining 

awareness of procedures in progress), detecting and monitoring the plant's state and parameters, interpreting its 

status, and selecting actions and executing them. 

CBPs are being developed for new plants, e.g., the Westinghouse AP600, and as upgrades for existing plants, e.g., the Beznau plant in Switzerland. Although CBP systems are being developed, the general consensus is that 

guidance for their design is lacking (Chignell and Zuberec, 1993; Converse, 1992; EPRI, 1993a). 

The introduction of advanced HSI technology, such as CBP systems, is generally considered to enhance 

performance, but there also is the potential to lower human performance, spawn new types of human errors, and 

reduce human reliability (O'Hara, 1994). Therefore, it is important to consider the potential effects of these 

technologies on human performance. Like other advanced HSI technologies, CBPs have many characteristics that 

can enhance a crew's performance, but other characteristics may impair their responses. In addition, CBP failures 

may place special demands on operators, e.g., transitions between CBPs and PBPs may introduce problems 

associated with their different presentation media and requirements for operation. 

1.2 Earlier NRC Work on Computer-Based Procedures 

As part of their review of advanced reactors, the NRC's staff evaluated the Electric Power Research Institute's 

(EPRI) Advanced Light Water Reactor Utility Requirements Document, hereafter referred to as the URD (see 

EPRI, 1993a). EPRI specified CBPs as a requirement in the URD, but gave limited guidance for their development 

and implementation. The NRC (1994) concluded that CBPs were a "desirable goal" whose appropriate 

implementation must be demonstrated. 

In 1994, the NRC staff in the Office of Nuclear Reactor Regulation published the Human Factors Engineering 

Program Review Model (NUREG-07 11), giving an approach to reviewing the HFE aspects of advanced reactor 

designs (O'Hara, Higgins, Stubler, Goodman, Eckenrode, Bongarra, and Galletti, 1994). Criterion 7 of Element 8, 

Procedure Development, of the NUREG-07 11 states the following: 

An analysis should be conducted to determine the impact of providing computer-based procedures, CBPs, (either partial or complete), and to specify where such an approach would improve procedure utilization and reduce 

operating crew errors related to procedure use. The justification for use of CBPs over paper procedures should be 

documented. An analysis of alternatives in the event of loss of CBPs should be performed and documented. 

In supporting NUREG-07 11, preliminary review guidance was developed for CBPs based upon the considering 

current issues and practices in using PBPs (Barnes et al., 1996). 

From a research perspective, CBPs were identified in 1994 as a technology being developed in the nuclear power 

industry for which little human factors knowledge and guidance existed (O'Hara, 1994). After this identification 

of the CBPs issue, the literature was reviewed for the NRC (Wourms and Rankin, 1994) and a workshop for 

subject-matter experts (SMEs) was held in San Diego, California, to identify the state-of-the-art in CBP research 

and design. The workshop generated an excellent overview of the systems under development and some of the 

human factors challenges to their use and evaluation (see Section 5.4.3, Expert Opinion, of this report). Also in 

1994, the NRC started a study of CBP systems and their potential impact on human and plant reliability (Orvis and

NUREG/CR-6634 1-2
1 INTRODUCTION

Spurgin, 1996). It also sponsored a comparison of operators' performance with CBPs and PBPs in a simulator 

study (Converse, 1995). 

As noted above, the potential human performance issues and safety significance of CBPs recently were evaluated 

the early phases of this project (O'Hara, Stubler, and Higgins, 1996; Stubler, Higgins, and O'Hara, 1996). Also, 

observations were made on introducing advanced HSI technology, including CBPs, into a conventional NPP (Roth 

and O'Hara, 1998). 

This work is discussed in greater detail in Section 5 of this report; all of it contributed to the CBP review guidance 

developed. 

1.3 Organization of the Report 

The report is divided into two parts. Part 1 describes the methodology for developing guidance and its technical 

basis. The objective of the study is described in Section 2, and the guidance development methodology in 

Section 3. Section 4 characterizes CBP systems, and Section 5 discusses the literature and information that served 

as the technical basis for the review guidance. The actual way we used the technical information is described in 

Section 6. Our CBP research is summarized in Section 7. Section 8 lists the references to the published literature. 

Part 2 of the document contains the results of the guidance development, presented in two sections. Section 9 

identifies the design-process considerations for CBP review, and Section 10 contains the HFE design guidelines for 

reviewing an implemented CBP design.

1-3 NUREG/CR-6634
2 OBJECTIVE

The objective of this study was to develop HFE review guidance for CBP systems based on a technically valid 

methodology. While the primary focus of the guidance was on EOPs, many of the principles identified apply to 

other types of plant procedures. 

To support this objective, several tasks were performed: 

"* Development of a framework for characterizing key design features of CBP systems 

"* Development of a technical basis using research and analyses on human performance relevant to CBPs 

"* Development of HFE review guidelines for CBPs in a format that is consistent with NUREG-0700, Rev. 1, and 

NUREG-0711 

"* Identification of remaining CBP issues for which research was insufficient to support the development of NRC 

review guidance

2-1 NUREG/CR-6634
3 METHODOLOGY 

3.1 Overview 

Figure 3.1 shows the overall methodology used for developing NUREG-0700 guidance. The process is discussed 

in detail elsewhere (O'Hara, Brown, and Nasta, 1996; Stubler and O'Hara, 1996). The portion of the methodology 

applicable to this report and project is boxed in the figure. This section of the report describes the general rationale 

behind guidance development.

Figure 3.1 Major Steps in the Development of NUREG-0700 Guidance

The methodology was guided by the following objectives: 

• Establish a process that will result in valid, technically defensible, review criteria 

* Establish a generalizable process applicable to any aspect of HSI technology needing review guidance 

* Establish a process that optimally uses available resources; i.e., develop a cost-effective methodology 

The methodology places a high priority on establishing the validity of the guidelines. Validity is defined along two 

dimensions: internal and external. Internal validity is the degree to which the individual guidelines are based on 

an auditable technical basis. The technical basis is the information upon which the guideline is established and 

justified, and varies for individual guidelines. Some guidelines may be based on technical conclusions from a 

preponderance of empirical research evidence, some on a consensus of existing standards, while others are based 

on judgement that a guideline represents good practices based on the information reviewed. Maintaining an audit 

trail from each guideline to its technical basis serves several purposes: 

* Evaluation of the technical merit of the guideline by others 

* A more informed application of the guideline since its basis is available to users 

* Evaluation of deviations or exceptions to the guideline 

External validity is the degree to which the guidelines are subjected to independent peer review. Peer review is a 

good method for screening guidelines for conformance to accepted HFE practices, and for comparing guidelines to 

the practical operational experience of HSIs in real systems.

3 -1 NUREG/CR-6634
3 METHODOLOGY

For individual guidelines, these forms of validity can be inherited from the source documents that form their 

technical basis. Some HFE standards and guidance documents, for example, already have good internal and 

external validity. However, if validity is not inherited, it is established as part of the process of guidance 

development. The NUREG-0700 methodology was established to ensure validity, both inherited from its technical 

basis, and through the development and evaluation of guidance. 

Figure 3.2 depicts the process used to develop the technical basis and guidance; it emphasizes information sources 

with the highest degree of internal and external validity. Thus, primary and secondary source documents were 

sought for guidance first, followed by tertiary source documents, basic literature, and industry experience, and from 

them design principles and lessons from industry experience were identified. Using this technical basis as a 

foundation, the guidance was developed. For specific aspects of the topic in which there was an inadequate 

technical basis to develop guidance, unresolved research issues were defined. Thus, the analysis of information led 

to the formulation of both guidance and issues. The resulting guidance documentation includes HFE guidelines, 

technical basis, the development methodology, and unresolved research issues. 

Each step in this research - characterizing the topics, developing the technical basis, developing and documenting 

guidance, identifying issues, and peer review - is discussed in greater detail in the sections that follow. 

3.2 Characterization of CBP Systems 

The first step in the development process was to identify the areas for which guidance was needed. Existing CBP 

systems were reviewed to identify the features and functions along which CBP systems can be defined. 

Characterization was important because it provided a structure with which to organize the design review guidance. 

The characterization will also provide a reviewer with a framework for requesting information about a CBP 

system. Section 4 describes the characterization of CBP systems. 

3.3 Development of Technical Basis 

The development of detailed review guidelines began by collecting technical information on which they would be 

based (see Figure 3.2); the process was designed to develop valid guidance cost effectively. First, primary source 

documents were sought. These were HFE standards and guidance documents with internal and external validity; 

that is, these documents generally had their own research bases, and the developers of these documents had 

considered the available research and operational experience, along with their own expertise, to establish HFE 

guidelines. These primary source documents were extensively peer reviewed. They were developed by experts who 

consider research in terms of its applicability and generalizability to real systems, include knowledge and expertise 

gained through operational experience and the application of guidance, and modify the guidance based on 

extensive peer review. Such documents provided a technically valuable starting place. 

Since little primary source information was available, the technical basis for CBPs considered the other sources 

identified in Figure 3.2. Secondary sources were documents for which either internal or external validity had been 

established. They were preferred over tertiary source documents for which neither was established. 

In addition to these sources, the results from basic literature were analyzed (articles from technical journals, reports 

from research organizations, and papers from technical conferences). When guidance was based on basic 

literature, engineering judgement was required to generalize from the individual experiments to actual applications 

in the workplace because individual experiments had unique constraints limiting their generality (such as their 

unique participants, types of tasks performed, and types of equipment used). For example, most scientific

NUREG/CR-6634 3-2
3 METHODOLOGY

Basic Literature ¥[ .• Development Guidance and of 

Sufficient Yes Documentation 

e.g., scientific, technical, and HFE guidelines 

trade journals Technical basis •Development methodology 

Industry Experience _ 

Sufficient Yes 

e.g., interviews with operators, 

designers, and researchers 

Originalufcin Research 

e.g., studies conducted 

specifically to develop guidance 

Identification of 

Unresolved 

Issues 

Figure 3.2 Technical Basis and Guidance Development Process 

experiments do not involve tasks of the complexity of NPP operations, nor do they examine tasks under the same 

performance shaping factors (such as rotating shifts, stress, and fatigue) as exist in a work environment. While 

information from research is a valuable part of guidance development, it usually cannot be uncritically adopted. 

Thus, the results must be interpreted in the context of real-world tasks and systems, based on professional and 

operational experience.

NUREG/CR-6634

Development of Technical Basis 

Primary and Secondary 

Source Documents Yes 

Sufficient 

e.g., HFE standards 

and guidelines 

No 

Tertiary Source 

Documents Yes 

Sufficient 

e.g., HFE handbooks 

No

3-3
3 METHODOLOGY

Industry experience also was used, such as published case studies, surveys, and interviews with knowledgeable 

experts. Although such information may lack a rigorous experimental basis (and thus, a measure of validity), it is 

highly relevant. 

Finally, some issues were evaluated by original research. This approach has the advantage of being focused on 

specific issues of interest, and has both high relevance and a sound basis from which to establish validity. The 

study (Roth and O'Hara, 1998) is described in Section 5.4.1, Empirical Evaluation of CBPs Based on Personnel 

Performance. 

3.4 Development and Documentation of Guidance 

Once the technical information was assembled, a draft set of guidelines was developed from it. The guidelines 

were organized and specified in a standard format (discussed in Section 6). They are identified in Part 2 of this 

document. 

3.5 Identification of Issues 

Where there was insufficient information to provide a technical basis upon which to develop valid design review 

guidance, an issue was defined; these issues are described in Section 7. 

From a research standpoint, issues reflect aspects of CBP design and use that require additional investigation to 

resolve. From a design review standpoint, these issues will have to be addressed case-by-case. For example, an 

issue can be dealt with as part of design-specific tests and evaluations. 

3.6 Peer Review 

The resulting technical basis and guidance was submitted for review by knowledgeable experts. These included 

reviews by personnel from the U.S. NRC with expertise in human factors engineering and engineering fields 

directly related to the topic. Additional reviews were conducted by human factors specialists outside the NRC who 

have expertise in human performance in complex systems, such as nuclear power plants and aviation. These 

external reviews included evaluations of the topic characterization along the following criteria: clarity, accuracy, 

and completeness; and of the review guidance along the following criteria: organization, necessity, sufficiency, 

resolution, and technical basis. Comments from the peer reviews were incorporated into the present version of this 

document.

NUREG/CR-6634 3-4
4 CHARACTERIZATION OF CBP SYSTEMS 

In the nuclear power industry, a procedure has been generally defined as a written document (including both text 

and graphics) that presents a series of decision and action steps to be performed by plant personnel (e.g., operators, 

technicians) to accomplish a goal safely and efficiently. NPP personnel use procedures for a wide variety of tasks, 

from administration to testing, and plant operation. This project is focused on procedures that prescribe 

interactions between personnel and the plant systems and components. The purpose of NPP procedures is to guide 

human actions when performing a task to increase the likelihood that the actions will safely achieve the task's 

goal. In contrast to decision aids, procedures define decisions to be made and actions to be taken where the task 

goals are unambiguous and the correct or desired course of action is generally known. 

In recent years, many efforts have been started in NPPs to assist personnel through the computerization of 

procedure information. Several CBP systems have been, or are being, installed in operating plants or in their 

training simulators. The following are examples of some of the more mature systems: 

* Westinghouse Computerized Procedure (COMPRO) System at Beznau, Switzerland, and Temelin, Czech 

Republic 

* EdF Computerized Control Room (CR) for N4 Reactors at Chooze and Civaux, France 

• EPRI Boiling Water Reactor (BWR) Emergency Operating Procedure Tracking System (EOPTS) at Kuosheng, 

Taiwan 

* Tokyo Electric's BWR Computerized EOPs, France 

(For a general description of specific CBP systems, see Moieni and Spurgin, 1993a; Spurgin, Wachtel, and Moieni, 

1993.) 

For this document, CBPs are defined narrowly to include computer systems whose purpose is supporting the 

presentation and use of procedures; systems whose functions include diagnosis or disturbance analysis are not 

within its scope. The focus of the effort is on the HFE aspects of CBPs, not the software aspects (for a discussion of 

general software development, testing, and management, see NRC Regulatory Guides 1.168 through 1.173; NRC, 

1997 a-f). 

The characterization and guidance focuses on EOPs. However, it is recognized that normal and abnormal 

operating procedures have been identified as important contributors to many significant events (Trager, 1988), and 

play a significant role in the plant's safety (Grant et al., 1989). Much of the guidance may apply to such 

procedures, and to test, surveillance, troubleshooting, and maintenance procedures when they are delivered by CBP 

systems. 

The design review of CBP systems requires two types of guidance: procedure guidance and HSI guidance. The 

first addresses the human factors aspects of procedure design and is intended to ensure that technically correct and 

usable procedures are developed. There is considerable guidance on procedure design, such as NUREG-0899, but 

because it was developed for PBPs, modifications may be necessitated by computerization of CBPs. Sections 9 and 

10 of this report have guidance specific to CBPs. 

For HSI guidance, CBPs will share many of the HSI resources and characteristics as other plant information 

systems. That is, information will be presented on VDUs, and operators will interact with the CBP information 

using the computer's dialogue and navigation facilities, accessed with input devices, such as keyboards and mice.

4-1 NUREG/CR-6634
4 CHARACTERIZATION OF CBP SYSTEMS

Many human factors guidelines currently exist, such as NUREG-0700, Rev. 1, covering these general 

characteristics of HSI design. What still is needed is the specific application of HSI principles to the 

computerization of procedure functions, such as monitoring steps that are continuously applicable. 

In this section, a characterization framework for CBP systems is discussed based on our examination of many 

CBP-system implementations. A system characterization is the identification of important design features and 

functions that can be used to describe it. The characterization provides a framework for NRC's reviewers to collect 

information about the system for reviewing its design. It also forms an organizational structure for the guidelines 

used to review the system. 

The CBP characterization framework discussed includes the following: 

* Representation of Procedures (Section 4.1) 

* Functionality of Procedures (Section 4.2) 

• Management and Support of Procedures (Section 4.3) 

• CBP Hardware (Section 4.4) 

* Backup System for Procedures (Section 4.5) 

* Integration with Other HSI Components (Section 4.6) 

4.1 Representation of Procedures 

In their basic form, procedures have a number of elements for which considerable guidance already exists. These 

same elements must be represented in the CBP system. 

Identification Information for Procedures 

Procedures are identifiable to the operators and maintainers through their title, procedure number, revision 

number, and date. Procedures also contain statements of the high-level objective and its applicability, including 

their category, e.g., emergency or abnormal. 

Basic Steps 

Steps are the basic unit of the procedure. A basic action step is composed of a verb and a direct object. In general, 

the rules of English grammar are followed and the syntax reflects concise language that is simply stated, explicit, 

and consistent. Decision steps give instructions for evaluating conditions and for then choosing the appropriate 

action(s) from a predefined set. The decisions may involve conditional logic, i.e., where the actions are to be 

performed only if a specified set of conditions exists. Action steps identify actions to be taken, i.e., instructions to 

perform physical steps (e.g., "Depress") and mental ones (e.g., "Verify"); they also describe the objectives of those 

actions. Some procedure steps (e.g., in EOPs) have a dual nature, with an action to be accomplished in one 

column, and a second action if the first is not successful. Some steps may require calculations. 

Implementation of procedures has a temporal flow, i.e., some steps are carried out when encountered, others are 

continuous (steps of continuous applicability), while time or process criteria determine when others are undertaken.

NUREG/CR-6634 4-2
4 CHARACTERIZATION OF CBP SYSTEMS

Performance of a procedure step may be supported by information, such as cautions and notes, that qualifies the 

required actions and decisions. 

Warnings, Cautions, Notes, and Supplementary Information 

Warnings alert operators to potential hazards of their actions that may cause death or injury to workers or the 

public; cautions alert operators to potential hazards for machinery or equipment. Notes call attention to important 

supplemental information that may enhance an operator's understanding and performance of the procedure. 

Procedure steps may cite supporting supplementary material helping the operator implement the step; this material 

may be tables, figures, lists, text, or numeric information. 

Lists 

A list is a display containing alphanumeric strings arranged in a single column. Procedures frequently use list 

formats to present groups of items, such as actions, conditions, components, criteria, and systems. When lists are 

used in CBPs, additional considerations relate to the grouping of items, provision of checkoff capability, and 

operator alerts to potentially overlooked items. 

Organization of Procedures 

NPP procedures are not simple checklists where the operator starts at the top and linearly proceeds step-by-step to 

the end. Based on the plant's conditions, the operator may be required to branch from one part of a procedure to 

another, or from one procedure to another. Thus, the way procedures are organized is important. 

Format and Screen Layout 

PBPs generally present the basic steps in text or flowchart formats. The CBP systems that have been designed also 

follow these principles, and may use either format. Thus, the Westinghouse COMPRO CBP is text based and is 

consistent with the two-column format developed by the Westinghouse Owner's Group for their EOPs. The EdF 

N4 CBP uses a flowchart format. 

Unlike PBPs, CBPs are viewed through the limited display area of one or more VDUs. Thus, whether the format is 

text or flowchart, the designer must decide whether the procedure will be presented in a continuous, scrollable 

display or be divided into discrete pages. 

The overall layout of the screen for showing elements of the procedure refers to the determination on what 

information should be continuously presented, and the manner in which individual elements are presented. 

For example, the procedure's title and identification information may be displayed continuously at the top of the 

CBP screen, while the steps are shown on a scrollable window. Cautions may be represented in a separate window. 

Supporting features, such as bookmarks, checklists, and operators' comments may also be displayed. 

Presentation formats, such as text and flowcharts, can be enhanced by the coding capabilities of computer-based 

displays, e.g., color, flashing, animation, and auditory cuing, which enhance the salience of important information. 

CBPs use coding for conditions such as: 

Whether procedure step logic is satisfied or not

4-3 NUREG/CR-6634
4 CHARACTERIZATION OF CBP SYSTEMS

* Whether information is static or dynamic with the plant's state 

* When a caution is in effect 

* When a change occurs in the status of a continuously monitored step 

CBPs can be designed to allow operators to choose the level of detail shown. For example, operators may choose to 

have less detail presented when a procedure step is satisfied. Alternatively, an operator may choose to see all of the 

individual evaluations that led to the conclusion that the step is satisfied. 

4.2 Functionality of Procedures 

A significant difference between PBPs and CBPs is in the functionality provided by the latter. Procedure functions 

can be organized into four cognitive categories: monitoring and detection, situation assessment, response planning, and response implementation (see Section 5.1 for an in-depth discussion of these cognitive functions). In 

monitoring and detection, operators must monitor the process parameters referenced by procedures, and also their 

own actions in response. 

Situation assessment is frequently required by procedures. While EOPs enable operators to act without diagnosing the disturbance, operators must assess whether EOP entry conditions exist. Within the procedure, operators assess 

each decision step by comparing actual values to the reference values, evaluating whether cautions are applicable, 

assessing the completeness of each step, and tracking and remembering their path through the procedure (the 

procedure history); at the same time, they must evaluate steps of continuous applicability and steps that are time or 

parameter dependent. Operators also must assess the applicability of individual steps because PBPs are generic and not context sensitive (context sensitivity is the selection of procedural information based on the plant's state). 

Finally, operators must evaluate the success of the current procedure in achieving the high-level procedure goals, 

and the procedure's termination conditions. 

Procedures were originally designed to support response planning. For example, EOPs assist operators in 

responding to events by setting out the steps necessary to achieve safety goals. They relieve the operator of the 

burden of formulating response plans in real time. Instead, the actions necessary to restore and maintain critical 

safety functions were analyzed in advance and developed into a set of detailed procedures. However, operators 

must still evaluate whether transitions to other parts of the procedure or other procedures are warranted. Rarely, 

they may have to modify procedures when current conditions render the existing procedure inapplicable (see 

Section 5.1). 

With respect to response implementation, the operator's responses involve acting upon the procedures themselves, 

such as making the transition from one step to the next, to other parts of the procedure, or to other procedures. 

Responses also include controlling equipment based on procedural guidance. 

While PBPs support response planning, they give little active support for monitoring, situation assessment, and 

responses. On the other hand, CBPs may support these cognitive functions as well; the extent to which they do so 

depends on their design. Examining the role of the operator is very significant in defining how cognitive functions 

are supported by CBP design (see Section 5). 

Table 4.1 provides an overall scheme within which the level of automation of CBPs can be organized. It illustrates 

the wide levels of automation and functionality that CBPs may possess. The table also can be used to catalog the 

functionality of a particular CBP system.

NUREG/CR-6634 4-4
4 CHARACTERIZATION OF CBP SYSTEMS

In the rows, the general cognitive functions (as described above) are identified, along with the associated 

procedure-related activities. In the columns, four levels of automation are identified; manual, advisory, shared, 

and automated: 

* Manual - The function is performed by the operators with no assistance from the CBP. 

* Advisory - The CBP provides advice only. For example, it may advise the operator that Pump A should be 

started, but does not start it. 

Shared - The CBP and the operators both perform the function. For example, a CBP system may perform 

process monitoring but may not monitor all information about the system, such as a valve's position, because it 

lacks the instrumentation. When this type of information needs to be monitored, the operator provides it. 

Table 4.1 Levels of Automation of Procedure Functions

Transition to other parts of procedure or to other 

procedures 

Control of plant equipment

NUREG/CR-6634

Procedure Functions Level of Automation 

Manual Advisory Shared Automatic 

Monitoring and Detection 

Process parameter values NA 

Operator actions NA 

Situation Assessment 

Procedure entry conditions NA 

Resolution of procedure step logic NA 

Step status (incomplete or completed) NA 

Procedure history NA 

Context sensitive step presentation NA 

Assessment of continuous, time, and parameter steps NA 

Assessment of cautions NA 

High-level goal attainment and procedure exit NA 

conditions 

Response Planning 

Selection of next step or procedure 

Procedure modification based on current situation

Response Implementation

Transition from one step to the next

Note: NA means "not applicable." For a given CBP system, the advisory level of automation may not be applicable or 

an entire function may not be applicable.

4-5
4 CHARACTERIZATION OF CBP SYSTEMS

• Automated - The CBP performs the function automatically without the operator's direct intervention; the 

operators may or may not be notified of the actions taken. 

A given level of automation does not necessarily apply to all functions. For example, for process monitoring, it is 

not meaningful to have advisory automation. The CBP system will either have monitoring capability or not. 

These are indicated by NA (not applicable) in the table. 

A given procedure system may not provide an entire function. For example, a particular CBP may not address the 

control of equipment in any capacity, not even manual; equipment would be operated from other HSIs. 

Individual CBP systems differ in terms of their levels of automation (i.e., the extent to which they provide features 

beyond those identified above as the basic procedure elements). For example, to allow manual control of 

components, the CBP must include a control, e.g., a soft control, for that equipment. 

4.3 Management and Support of Procedures 

CBP systems have design features that support operators' interaction with the system, procedure maintenance, and 

configuration control. Therefore, interface management features (such navigation aids)' are part of the 

characterization of CBP systems. Procedure-specific management support includes HSIs to transition between 

procedure steps and between different procedures. The use of procedures can be supported by facilities to monitor 

and record the operator's actions and to provide help. 

Maintenance of procedures and configuration control are important for CBPs, as they are for PBPs. However, their 

mechanisms are likely to differ, such as how procedures are entered into the computer system, how their quality is 

verified (e.g., no typos or omissions), how errors in the CBPs are identified, tracked and corrected, how changes 

are incorporated, and how configuration is controlled. Guidance on these aspects is not part of this project. 

Many general interface management design features are addressed in NUREG-0700. More specific guidance to 

address soft controls (Stubler, O'Hara, and Kramer, 2000) and interface management (O'Hara, Stubler, and Nasta, 

1997) is being developed. 

4.4 CBP Hardware 

CBPs utilize CR devices such as VDUs, printers, and computer input devices, such as alphanumeric keyboards, 

trackballs, mice, and touch screens that are part of the CBP characterization. NUREG-0700, Rev. 1 has guidance 

for their review. 

4.5 Backup System for Procedures 

CBPs can fail or malfunction. When important operations cannot be suspended or put off while the system is 

repaired, backup to the CBP is needed. For EOPs, a delay in operations during a failure is unacceptable; therefore, 

some form of procedure backup is warranted. 

SGui

for interface management review is currently being developed in a separate NRC research 

project.

NUREG/CR-6634 4-6
4 CHARACTERIZATION OF CBP SYSTEMS

4.6 Integration with Other HSI Components 

Integration of the CBP with other CR HSIs must be considered. Depending on the level of automation (see 

Table 4.1), CBP systems require varying types of interconnection with the remainder of the CR HSI. Their 

consistency and compatibility with other HSI components can affect operators' performance. Thus, important 

considerations in reviewing CBPs include the degree to which (1) the display of plant variables in the CBP is 

compatible with normal monitoring displays, (2) coding schemes are compatible, and (3) control modes of the CBP 

are consistent with the rest of the HSI (e.g., with the modes of automated control systems).

4-7 NUREG/CR-6634
5 TECHNICAL BASIS FOR DEVELOPING CBP GUIDELINES 

The purpose of this section is to establish a technical basis for developing CBP guidelines and to identify human 

performance issues. The review considers human performance research that contributes to understanding CBP 

design and operational use. The research included CBP use in the NPP industry and several related areas: issues 

associated with PBPs, computerization of tasks traditionally using paper performance aids, and computerized 

operator support systems (COSSs). In addition, we consider the general cognitive functions associated with the 

supervisory control tasks which procedures support. 

5.1 Cognitive Tasks Associated with Operating the Plant 

Operators contribute to the plant's defense-in-depth approach to safety, serving a vital function in ensuring its safe 

operation. However, they may impact safety by making errors. Basically, an error occurs when personnel do not 

perform a safety-related action within the time required (sometimes called an error of omission). An error also 

may occur because personnel have an incorrect understanding of conditions and take the wrong action (an error of 

commission). Many attempts were made over the past 20 years to identify the causes of error. The main 

conclusion is that few errors represent random events; instead, most can be explained by human cognitive 

mechanisms (Reason, 1988; Rasmussen, 1986). Therefore, it is important to understand how operators process 

information and how this relates to HSI design and human error. 

The operator's role in an NPP is that of a supervisory controller, i.e., the plant's performance results from the 

interaction of human and automatic control. Reason (1990) called this a complex multiple-dynamic configuration, 

which is difficult for personnel to handle when things go wrong. In addition to process failures, automatic control 

systems and the HSI also can fail. Thus, personnel must respond to plant failures and to the interfaces that 

communicate their occurrence. One significant aspect of the HSI in responding to process failures is the procedure 

system. In complex systems using a defense-in-depth philosophy operations are analyzed in advance to provide 

procedural support for both normal and abnormal events. However, even when procedures are used, operators 

must still engage in higher-level cognitive functioning (Dien, Montmayeul, and Beltranda, 1991; Roth, Mumaw, 

and Lewis, 1994). 

The operators' impact on the plant's functions, processes, systems, and components is mediated by a causal chain 

from their physiological and cognitive processes, to task performance, and ultimately, to the plant's performance 

through the operators' manipulation of the HSI (see Figure 5.1). HSI design, including its procedures, affects the 

plant's performance through personnel tasks that support operations. Conceptually, the role of personnel can 

involve two types of tasks. Primary tasks are those the operator performs as part of the functional role of 

supervising the plant. Operators may be required to act in support of the plant's performance of a higher-level 

function. Even when they are not required to take an explicit action, they must monitor the performance of 

automatic systems and intervene when the systems fail or perform at unacceptable levels. 

Primary tasks involve several generic cognitive tasks; i.e., situation assessment, monitoring and detection, response 

planning, and response implementation (see Figure 5.2). For primary tasks, these generic cognitive tasks are 

discussed, rather than the detailed specific tasks, such as monitoring steam flow, starting pumps, and aligning 

valves. Secondary tasks are those the operator must perform when interacting with the HSIs or job performance 

aids, but which are not directed to the primary task (O'Hara, Stubler, and Nasta, 1997). They include navigating 

through an information system and manipulating windows on a VDU. To adequately perform both primary and 

secondary tasks, operators use their information processing resources, such as attention, reasoning, and memory.

5-1 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

PLANT

PERSONNEL

Functions 

Critical safety function violations 

Systems 

Safety system actuations 

Components 

Major parameter values 

HSI 

Control, displays, procedures 

Personnel Tasks 

Primary 

Secondary 

Cognitive Factors 

Attention 

Cognitive workload 

Physiological Factors 

Vision & reach 

Physical workload

Figure 5.1 Hierarchical Influence of Human 

Activity on Plant Performance

Figure 5.2 Generic Primary Tasks of a Supervisory Controller

NUREG/CR-6634 5-2
5 TECHNICAL BASIS DEVELOPMENT

In their traditional paper form, plant procedures mainly support response planning. On the other hand, CBPs 

potentially can affect all the generic cognitive primary and secondary tasks that operators undertake: 

Monitoring and detection, especially monitoring parameters used to evaluate procedure steps, monitoring steps 

of continuous applicability, and detecting violations of the conditions specified in these steps 

Situation assessment, especially assessing the plant's state with respect to the steps' logic, steps of continuous 

applicability, and cautions 

• Response planning, the main function of procedures 

* Response implementation, for manual or automatic control of (1) procedure flow (transitions within and 

between procedures), or (2) plant equipment 

Situation assessment and response planning are discussed next because they are the most important and 

complicated cognitive functions involved in using procedures. Monitoring and response implementation are 

described afterwards. 

Situation Assessment 

When operators observe indications of an abnormal occurrence, they try to construct a coherent, logical 

explanation for them. This cognitive activity may be called situation assessment and involves two related concepts: 

the situation model, and the mental model. Operators develop and update a mental representation, or so-called 

situation model, of factors known or hypothesized to be affecting the plant's state at a point in time. The situation 

model is the person's understanding of the specific situation, and the model is constantly updated as new 

information is received. To construct a situation model, operators use their general knowledge and understanding 

-about the plant and its operation to interpret information and understand its implications. Limitations in 

knowledge may result in incomplete or inaccurate situation models and response plans. 

The general knowledge governing the performance of highly experienced individuals may be referred to as a 

mental model which constitutes the operator's internal representation of the physical and functional characteristics 

of the system and its operation. Mental models may not always be accurate or complete. The mental model is built 

up through formal education, system-specific training, and operational experience; it resides in the knowledge 

bases of long-term memory. An accurate mental model is a defining characteristic of expert performance (e.g., 

Wickens, 1984; Bainbridge, 1986; Moray, 1986; Rasmussen, 1983; Sheridan, 1976) and is extremely important to 

many aspects of information processing. It is thought to drive skill-based processing, control rule-based activity 

through the mediation of the operator's conscious effort in working memory, and provide the substantive capability 

to reason and predict future plant states required of knowledge-based processing (Rasmussen, 1983). 

The distinctions between the mental and situation models reflect their cognitive basis in long-term and working 

memory, respectively. The mental model is relatively permanent. By contrast, an operator's situation model is the 

current interpretation of the plant's status and, therefore, changeable. 

When the operator's situation model accurately reflects the plant's state, the operator has good situation awareness. 

Thus, the accuracy of situation awareness depends on the correlation between the operator's situation model and 

the actual conditions. An operator can have a good mental model (e.g., knowledge of how the plant functions) but 

poor situation awareness because the situation model does not match the current conditions. Endsley (1988)

5-3 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

identified situation assessment as the single most important factor in improving a crew's effectiveness in complex 

systems. Endsley (1995) distinguishes three levels of situation awareness (discussed below). 

An experienced, well-trained operator easily develops an accurate situation model when the HSI provides information that readily maps to knowledge in the operator's mental model. If no easy match can be made between 

them, then situation assessment will require more working memory and attention, and cognitive workload will be 

high (Endsley, 1993, 1995; Fraker, 1988). However, in addition to supporting situation assessment, working 

memory must support other activities, such as selecting and taking actions. Accordingly, if other tasks place high 

demands on working memory, situation awareness may suffer. 

Situation awareness and cognitive workload may vary inversely under complex, ambiguous situations. For 

example, under unfamiliar or otherwise difficult conditions, a high cognitive workload may entail decreased 

situation awareness, possibly due to a lack of available attentional resources for analyzing the situation. However, 

Endsley (1993) points out that situation awareness and cognitive workload, while interrelated, may vary independently. For example, a task may be intensive, but readily recognizable. Situation awareness demands 

cognitive resources that contribute to workload, but is not the only cognitive activity requiring such resources. 

Thus, mental models enable operators to engage in situation assessment and to establish situation models. Good 

situation models include a knowledge of the important elements of the current situation, and a comprehension of 

how they interrelate to reflect the overall situation. These two aspects of good situation models correspond to 

Endsley's (1995) Level 1 (Perception of Elements) and Level 2 (Comprehension of Situation) situation awareness. 

Mental models enable operators to make predictions and form expectations; projection of future states corresponds 

to Endsley's (1995) Level 3 situation awareness. These expectations guide monitoring and affect how information 

is interpreted. This is a general characteristic of information processing; it is a synthesis of "bottom-up" 

processing (what an operator perceives from the environment) and "top-down" processing (what an operator expects) (Neisser, 1967). An example of bottom-up processing occurs during a disturbance when an operator 

monitors the HSI and processes data from the interface to determine what is wrong. Simultaneously, these data are 

used to formulate hypotheses or expectations about the plant's status that structure the perceptual process and data

gathering at lower levels. This is top-down processing. Both contribute to the operator's interpretation of the 

situation. 

The ability to predict from a mental model based on the current situation model facilitates "open-loop" performance (Moray, 1986). "Open-loop" in this context means that behavior becomes less driven by feedback and 

more governed by the operator's prediction of future system behavior and the desired goal. An NPP mental model 

includes such knowledge as the physical interconnections among plant systems to predict flow paths (e.g., 

considering piping and valve interconnections to figure out how water from one system could get into another), and knowledge of mass and energy changes in one system to predict the effect on a second system (e.g., predicting 

the effect that changes in the secondary side steam generator levels and temperatures will have on cooldown of the 

primary system). While mental models provide the principles upon which predictions are made, the situation 

model provides the starting point and is the basis for developing expectations about events that should be 

happening at the same time, how they should evolve over time, and any future effects. 

The operator's expectations of the near-term future of the plant guide the sampling of indicators to confirm the 

inference (Bainbridge, 1974). Expectations are used to search for evidence to confirm the current situation model, 

and to explain observed symptoms. If a new symptom is consistent with the operator's expectations, a ready 

explanation for it will be developed, yielding greater confidence in the situation model.

NUREG/CR-6634 5-4
5 TECHNICAL BASIS DEVELOPMENT

While the mental model allows prediction and expectancy to guide control responses, expectancy may confound the 

detection of subtle system failures (Wickens and Kessel, 1981). When a new symptom is inconsistent with an 

operator's expectations, it may be discounted or misinterpreted in a way to make it consistent with the expectations 

of the current situation model. For example, an operator may fail to detect key signals, or detect them but 

misinterpret or discount them, because of an inappropriate understanding of the situation and the derived 

expectations. Operators tend to ignore or discount symptoms that are not consistent with their situation model. 

However, if the new symptom is recognized as an unexpected behavior, the need to revise the situation model may 

become apparent. In that case, the symptom may trigger situation assessment activity to better explain current 

observations. In turn, situation assessment may involve developing a hypothesis for what might be occurring, and 

then searching for confirmatory evidence. Thus, situation assessment can result in the detection of abnormal plant 

behavior and of symptoms and alarms that otherwise might not have been observed or might have been missed, 

and the identification of problems such as sensor failures or plant malfunctions. 

The situation model is constantly updated as new information is received and a person's understanding of a 

situation changes. In NPPs, maintaining and updating a situation model entails tracking the changing factors 

influencing plant processes, including faults, operators' actions, and automatic system responses. 

The importance of mental and situation models, and the expectations generated, cannot be overemphasized. They 

not only govern situation assessment, but play an important role in guiding monitoring, using procedures and 

formulating response plans, and implementing responses. 

Response Planning 

Response planning refers to deciding upon a course of action to address an event. Response planning can be as 

simple as selecting an alarm response or EOP, or it may involve thoroughly developing a plan when existing 

procedures have proved incomplete or ineffective. 

In general, response planning involves operators using their situation model to identify goal states and the 

transformations required to achieve them. The goal state may be varied, such as to identify the proper procedure, 

assess the status of back-up systems, or diagnose a problem (Rasmussen, 1981). To achieve the goals, operators 

generate alternative response plans, evaluate them, and select the one most appropriate to the current situation 

model. 

This is the basic sequence of cognitive activities in response planning; one or more of these steps may be skipped 

or modified based on the operator's assessment in a particular situation. When available procedures are judged 

appropriate to the current situation, the need to generate a response plan in real-time may be largely eliminated. 

However, even then, some aspects of response planning will be undertaken. For example, operators still need to 

(1) identify goals based on their own situation assessment, (2) select the appropriate procedure, (3) evaluate 

whether the procedure-defined actions are sufficient to achieve those goals, and (4) adapt the procedure to the 

situation, if necessary. 

The decision making involved in situation assessment and response planning, especially in ambiguous situations 

when available procedures do not suffice, can be a large cognitive burden and draw heavily upon working memory, 

long-term memory, and attentional resources. Information then is consciously manipulated in working memory, 

and the ability to do so is a direct function of attentional resources available. Working memory has very limited 

capacity, and without sustained attentional resources (or transfer of the information to long-term memory), 

information decays rapidly. Information can be lost due to (1) insufficient attentional resources to keep it active, 

(2) overload of the working-memory capacity, and (3) interference from other information in working memory. To

5-5 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

increase the capacity of working memory, operators use memory heuristics, such as chunking - aggregating and 

organizing information into higher-level, meaningful units. A heuristic, as used here, means a shortcut for 

information processing developed through experience and trial-and-error, rather than systematic, formal analysis. Once this is accomplished, the higher-level units, not the individual elements, are stored in working memory. 

Dien et al. (1991) discussed the importance of higher-level cognitive functions when operators use procedures. 

Operators must compensate for inadequacies, fill in gaps, and resolve conflicts between the control objectives specified in the procedures and those established by the operators assessing the situation. Operators sometimes 

must implement more practical strategies than those in the procedures. They must also consider whether operating 

actions should be anticipated, or whether automatic devices should be left to operate. 

Roth et al. (1994) demonstrated the need to maintain a supervisory role, even when responses are largely dictated 

by EOPs. They investigated how operators handle cognitively demanding emergencies, their objective being to 

examine the role of situation assessment and response planning on guiding a crew's performance when EOPs were 

being utilized. NPP operators from two different utilities performed interfacing systems loss-of-coolant accident 

(ISLOCA) and loss-of-heat-sink scenarios on training simulators where complexities made it difficult to simply 

follow the appropriate procedure. The results illustrated the importance of high-level cognitive functions during 

use of EOPs. The operators developed an understanding of the plant's state and confirmed their situation 

assessment, and also attempted to understand plant performance that was unexpected based on their current 

situation model. These cognitive activities enabled them to evaluate the appropriateness of the EOP to achieve the 

high-level goal dictated by the situation assessment within the context of current conditions. Roth et al. noted the 

importance of the crew's interactions and communications to these high-level cognitive functions, due partly to the 

need to obtain information from many HSIs in different locations. In addition, communication helped operators 

overcome the fact that EOPs do not cover all the important information on the current plant state. When a specific 

procedure seemed to fail to meet the high-level goal, operators would alter the procedure path to better address the 

current situation. 

Thus, Roth et al. (1994) demonstrated the importance of understanding the basis of the procedure and its intended 

higher-level goals. The need to formulate modifications to procedure pathways also means that operators may not 

simply proceed linearly through a procedure. They may need to consider future steps, reexamine previous ones, 

and refer to other procedures to verify that their current activities are correct and will meet the high-level goals. 

Roth (1994) considered the implications of the Roth et al. (1994) study for designing operator support systems. First, the requirement of situation assessment and response planning independent from procedures suggests that 

operators must maintain awareness of abnormal plant symptoms, determine what malfunctions could produce 

them, and know the manual and system actions that are being undertaken and their effects. 'Second, since crews 

must anticipate the consequences of their actions, operator support systems could help in identifying their 

consequences and side effects: Third, operators must understand the assumptions and logic behind the procedures, 

i.e., their intent, their overall strategies, and the transition logic between them. Since operators may not move 

linearly between procedures, CBP navigation systems will be important to the success of CBPs in complex 

emergencies. 

Monitoring and Detection 

Monitoring and detection refer to the activities involved in extracting information from the environment. 

Monitoring is checking the state of the plant to determine whether the systems are operating correctly; it can 

include checking parameters indicated on the CR panels, monitoring those displayed by the process computer, 

obtaining verbal reports from operators in the plant areas, and sending them to other areas to check equipment.

NUREG/CR-6634 5-6
5 TECHNICAL BASIS DEVELOPMENT

Detection is the operator's recognition that something is operating abnormally. Procedures guide monitoring and 

detection by identifying the parameters to be monitored. However, operators must also monitor the crew's actions 

taken in response to disturbances. 

Monitoring and detection are influenced by two factors: (1) the characteristics of the environment, and (2) the 

operator's knowledge and expectations. These factors lead to two types of monitoring: (1) data-driven, and 

(2) model-driven. Monitoring driven by environmental characteristics often is referred to as data-driven 

monitoring. Data-driven monitoring is affected by the salience of the information's presentation (e.g., size, color, 

and loudness). Thus, alarm systems are basically automated monitors designed to influence data-driven 

monitoring by using physical salience to attract attention. Auditory alerts, flashing, and color coding are physical 

characteristics that enable operators to quickly identify an important new alarm. Data-driven monitoring also is 

influenced by the behavior of the information, such as the bandwidth and rate of change of the information signal; 

observers more frequently monitor a rapidly changing signal. 

Operators may initiate monitoring based on their knowledge and expectations (model-driven) about the most 

important information; this typically is called knowledge- or model-driven monitoring. Model-driven monitoring 

can be viewed as active monitoring, in that the operator is not merely responding to environmental characteristics 

that "shout out" like an alarm system does, but is deliberately directing attention to areas expected to provide 

specific information. 

Model-driven monitoring may be initiated by several factors. First, it may be guided by operating procedures or 

standard practice (e.g., control panel walk-downs at shift turnovers). Second, it can be triggered by situation 

assessment or response planning activities and is, therefore, strongly influenced by a person's current situation 

model. The situation model allows the operator to direct attention and focus monitoring effectively. However, 

model-driven monitoring can lead operators to miss important information. For example, an incorrect situation 

model may focus operators' attention in the wrong place, cause them to fail to observe a critical finding, or to 

misinterpret or discount an indication. 

An operator is faced with an information environment containing more variables than can be realistically 

monitored. The real challenge comes from the fact that there are many potentially relevant things to attend to at 

any time, and the operator must determine what information is worth pursuing within a constantly changing 

environment (Vicente, Mumaw, and Roth, 1997). Then, the operator must decide what to monitor and when to 

shift attention elsewhere. These decisions are strongly influenced by an operator's current situation model, which 

guides the allocation of attentional resources to sampling data from the environment based on its statistical 

properties; i.e., expected probability and correlation. The operator's ability to develop and effectively use 

knowledge to guide monitoring relies on the ability to understand the current state of the process. As cognitive 

workload increases, monitoring strategies become less thorough, and the capability to detect particular failures 

decreases (Ephrath and Young, 1981). 

Under normal conditions, situation assessment is attained by mapping the information obtained from monitoring to 

elements in the situation model. For experienced operators, this comparison is relatively effortless. During 

unfamiliar conditions the process is considerably more complex. The first step in realizing that the current plant 

conditions are inconsistent with the situation model is detecting a discrepancy between information representing 

the current situation and that derived from monitoring. This process is facilitated by the alarm system, which 

directs the operator's attention to an off-normal situation. 

When determining whether a signal is significant and warrants further investigation, operators examine it in the 

context of their current situation model. They must judge whether the anomaly indicates a real abnormality or an

5-7 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

instrumentation failure. They then will assess the likely cause of the abnormality, and evaluate the importance of 

the signal in determining their next action. 

Monitoring has been described in terms of signal detection theory (SDT) (Green and Swets, 1988). Process control 

operators are in a monitoring environment described in SDT terms as an alerted-monitor system (Sorkin et al., 1985; Sorkin et al., 1988). Such a system is composed of an automated monitor and a human monitor. The 

automated monitor in an NPP is the alarm system which detects off-normal conditions. When a parameter exceeds 

the criterion of the automated monitor, the human monitor is alerted and then must detect, analyze, and interpret the signal as a false alarm or a true indication of an upset. The human monitor also can assess plant parameters independently of the automated monitor (the alarm system). Both monitors have their own specific detection 

parameter values for sensitivity (d') and response criterion. The latter refers to the amount of evidence needed 

before an operator will conclude that a signaled event is actually present; this is called response bias since it 

describes an operator's conservatism. Sensitivity refers to the resolution of the system, i.e., the ease with which 

signals (represented as a statistical distribution) can be distinguished from signals plus noise (similarly 

represented). 

SDT research has many implications for understanding how operators process information during a disturbance. 

First, the response criterion is affected by expectancy; i.e., the expected probability that an event will occur and the 

payoff structure (rewards and penalties for making correct and incorrect detections, respectively). While alarms 

can occur frequently, significant off-normal events in NPPs typically have a low probability. Therefore, operators have low expectancy about their actual occurrence which creates a conflict between the cost to productivity for 

falsely taking an action that shuts down the reactor versus the cost for failing to take a warranted action. In 

actuality, because disturbances have a low probability, operators rely on redundant, supplemental information to 

confirm the alarm. Having verified several confirmatory indicators, the operator can accept the alarm information 

as indicating an actual off-normal condition. 

There are two types of anomalies: (1) deviations from desired system functions, called abnormal findings, and 

(2) deviations from the operator's situation model, or unexpected findings. The two anomalies lead to different 

follow-up reasoning and monitoring behavior: 

* Abnormal findings lead to information processing about how to cope with the disturbance (response planning) and to monitoring behavior to see if the expected coping responses have occurred and are having the desired 

effect. 

Unexpected findings or process behavior lead to situation assessment activity and model-driven monitoring to 

explain the finding. 

Failures in monitoring can include failing to observe parameters, misunderstanding the significance of parameters, 

or failing to obtain needed reports from plant areas. Failures in detection can include failure to recognize an 

abnormality despite proper monitoring. An error in monitoring or detection can cause the operator's failure to 

respond to the event, or at least, to respond within the required time. 

Response Implementation 

Response implementation is the performance of the actions identified in response planning. This can be as simple 

as an individual operator selecting and operating a control, or it can involve communications and coordination 

with teams of operators in different parts of the plant, who each then select and operate equipment controls in a

NUREG/CR-6634 5-8
5 TECHNICAL BASIS DEVELOPMENT

centrally coordinated manner. The actions may be discrete (e.g., flipping a switch) or may involve continuous 

control (e.g., controlling steam generator level). 

The results of actions are monitored through feedback loops. Two aspects of NPPs jeopardize the implementation 

of responses: time and indirect observation. Time and feedback delays disrupt response implementation because 

they make it difficult to determine whether control actions are having their intended effect. Consequently, the 

operator's ability to predict future states from mental models can be more important in controlling responses than 

feedback. Further, since plant processes cannot be directly observed, their status is inferred through indications; 

thus, errors in cognition can impede performance. 

Failures in implementing a response can lead to the operation of the wrong equipment, or the incorrect operation 

or control of particular components. 

Summary 

The role of the operating crew in an NPP is that of a supervisory controller that must engage in situation 

assessment, monitoring and detection, and response planning and implementation. These cognitive functions are 

applied to tasks for which the crew has primary responsibility, as well as to automated systems and systems 

designed to support crew tasks. Procedures fall into the latter category. Historically, procedures were designed to 

support response planning by providing operators with strategies that were based on previous detailed analyses of 

normal and abnormal plant states. However, when these preplanned strategies are applied to the unique 

circumstances of a particular disturbance, unforseen or unanticipated situations may render an aspect of a 

procedure inappropriate or ineffective. Thus, confronted with the complexities of real-world process disturbances, 

operators must monitor the performance of the procedure to verify its conformity to the higher-level goals that it 

was designed to achieve. Under such circumstances, it is important for operators to assess the effectiveness of the 

response plan even when it is described by established procedures, evaluate the consequences of particular 

procedure actions, and evaluate the appropriateness of the procedure path for achieving identified goals. This 

assessment enables operators to detect when procedures are not achieving the goals, when they may contain errors, 

or when errors are made in carrying out procedure steps. Another cognitive activity is adapting the response plan. 

This includes filling in gaps in a procedure, modifying it to fit the specific situation, redirecting its path, and using 

additional or alternative procedures. 

Thus, rather than assuming the role of rote, verbatim "procedure-followers," it is important that the operators 

maintain the role of supervisory controllers and monitor the performance of the procedures as well as the process. 

With the development of CBPs, the support of procedure systems extends beyond response planning and includes 

aspects of situation assessment, monitoring and detection, and response implementations. This support may be 

applied to the operator's primary tasks as well as secondary tasks. 

5.2 Issues with Paper-Based Procedures and Implications for 

Computerization 

As we discussed, plant procedures provide instructions to guide operators in monitoring, deciding on appropriate 

actions, and controlling the plant. The design of procedures was identified as a major cause of human error 

because PBPs have characteristics limiting the manner in which information can be presented, and impose tasks 

upon operators that are not directly related to controlling the plant. Properly following the procedures sometimes

5-9 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

is complicated by the necessity to track several EOPs or EOP branches simultaneously. Current symptom-based 

procedures appear to place significant workloads on operators. 

The purpose of examining the human performance issues associated with PBPs is to (1) ensure that these design deficiencies are addressed by CBPs where possible, and (2) identify aspects of procedure use that can be supported by computerization where human performance issues are not a simple function of the paper medium. 

In this section, the factors associated with difficulties with PBPs are identified, and their implications for CBP 

design are discussed. 

Issues with PBPs 

Several studies have addressed the problems associated with PBPs, and identified a broad range of deficiencies that 

fall into the following categories: design process, implementation, training, and maintenance. The deficiencies 

identified in these studies came from various sources including NRC procedure inspections, operator interviews, 

and literature reviews. Table 5.1 compiles the overall results. Four studies whose findings are included in the 

table are briefly discussed next. 

For non-EOP procedures, the types of deficiencies identified include (1) an excessive number of procedures and 

poor classification schemes for their use, (2) technical inaccuracies, (3) lack of clearly specified goals and criteria 

for determining that the intent of the procedure was attained, and (4) vaguely written procedures that do not 

specifically describe the necessary actions (Morgenstern et al., 1987; Barnes and Radford, 1987). 

These procedure limitations were associated with numerous problems in performance. Flow control and transitions 

between procedures can be associated with potential safety-significant errors (Chignell and Zuberec, 1993) when 

operators do the following: 

* Skip a step in the procedure 

* Follow out of sequence 

* Inadvertently use the wrong step 

• Follow an out-of-date, erroneous, imprecise, or ambiguous procedure 

• Follow the wrong procedure due to incomplete procedure references 

* Miss a procedure transition and continue in the current one 

* Become lost or confused when a transition is identified in a caution rather than as an action step 

Teamwork and communication are also important. Hoecker et al. (1994) and Hoecker and Roth (1996) identified 

errors in communication during the acquisition of procedure-specified information as problematic, including delays, suspended tasks, and difficulty identifying the correct display from other displays. These limitations 

increase workload and the likelihood of procedural errors.

NUREG/CR-6634 5- 10
5 TECHNICAL BASIS DEVELOPMENT

Implications for CBPs 

The deficiencies identified in PBPs are associated with implementing the procedures as well as with the procedure 

design process, training, and maintenance (illustrated in Table 5.1). Teigen and Ness (1994) identified the 

following CBP features as important in addressing these limitations: 

* A structured, consistent format 

• Ease of transition between procedures, and recording transitions 

* Place keeping in procedures when operators are in several simultaneously or when they access support 

information, such as tables and charts 

* Clear, consistent logic statements 

* Monitoring of process control parameters 

* Simplification of flowcharts by allowing operators access to varying levels of detail 

* Maintenance of procedures 

Similarly, Lipner and Rusnica (1996) identified some of the CBPs' features that can reduce the mental load and 

time demands of working with PBPs. They included monitoring plant parameters, centralizing all procedural 

information in one place, creating detailed record keeping on procedure implementation, and facilitating the 

maintenance of procedures. 

CBPs can directly affect many, but not all such deficiencies. In fact, a significant consideration in evaluating CBPs 

may be the extent to which they solve these problems. Next, we describe each of the problems associated with the 

paper medium and their possible resolutions by CBPs. For those problems not directly impacted by 

computerization, many of the same factors that contribute to PBP problems can also undermine CBPs. 

We now consider the major areas of procedure use that can be impacted by computerization. Several categories 

were previously identified (Barnes et al., 1996). While they mainly cover implementation of procedures, an initial 

category is identified which includes the more general areas of design process and support: 

Design Process and Support 

* General Cognitive Workload 

* Level of Detail 

* Context Sensitivity 

* Sequence Control and Navigation 

* Management of Multiple Procedures 

* Maintenance of Technical Accuracy of Procedures 

Integration of Procedure Tasks and Other Tasks

5-11 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Table 5.1 Deficiencies in Paper-Based Procedures 

Issue Deficiency Note 

Design • Inadequate participation of operations and training personnel in developing (1) 

Process procedures 

. Technically incorrect EOPs (1) 

. Address standard situations, but are less supportive in unusual situations (1) 

. Incomplete procedures (3) 

. Inadequate -consideration of the time required to complete procedural actions (1) 

• Insufficient verification and validation (V&V) of procedures (I) 

Imple- • Non-specific entry and exit conditions for support procedures (1) 

mentation * Fixed and inflexible procedures (2) 

. Incorrect sequencing of action steps (1) 

. Inadequate consistency across procedure (1) 

. Inconsistencies in formatting and use of terminology (1) 

. Incorrect identification of equipment (1) 

. Inadequate ability to provide varying level of detail (4) 

. Non-sequential presentation of information (4) 

. Navigation to related information (4) 

. Management of multiple procedures (4) 

. Integration of procedure tasks and other tasks (4) 

. Problems in labeling and headings (3) 

. Notes and cautions in improper places (3) 

• Lack of context-dependent highlighting and navigation (3) 

. Need to use multiple procedures simultaneously and move between sections (3) 

. Lack of flowcharts to guide use of procedure (3) 

. Inadequate support and reference material (1) 

• Bulkiness (3) 

• Physical handling of procedures near control panels (2) 

• Separation from other information sources, such as the safety parameter (3) 

display system (SPDS) 

. Inconsistency with other HSIs in referring to plant equipment (1) 

Training • Poor training of operators in use of procedures (1) 

Maintenance • Maintaining technical accuracy of procedures (3,4) 

Notes: (1) Lapinsky, 1989; Galletti and Sutthoff, 1992 

(2) Niwa, Hollnagel, and Green, 1996 

(3) Chignell and Zuberec, 1993 

(4) Barnes, Desmond, Moore, and O'Hara, 1996

" NUREG/CR-6634 5-12
5 TECHNICAL BASIS DEVELOPMENT

Design Process and Support 

The NRC's analysis of EOPs (Lapinsky, 1989) concluded that deficiencies in the design of EOPs were likely-to be 

found when any of the following were lacking: 

* A multidisciplinary team 

• An independent review to assure technical accuracy and usability 

* A systematic process to ensure EOPs do not degrade over time 

• Management's commitment to the EOP design process 

These factors can also negatively impact the development and use of CBPs. A development program should 

address these issues and those in the non-implementation categories in Table 5.1 to ensure that the final CBP 

system will adequately support safe operations. 

General Cognitive Workload 

Many problems with PBPs result from the high demands on cognitive activities, especially monitoring (e.g., of 

parameters needed to use procedure logic), decision making (e.g., analyzing procedure step logic), and memory 

(e.g., to perform steps of continuous applicability). These problems are amplified by the stress created by complex 

process disturbances. Mumaw (1994) found that stress lowers human performance by (1) narrowing and shifting 

attentional focus, (2) reducing working memory capacity, and (3) impairing the crew's communication patterns. 

Reducing the demands on cognitive processes can support the operator in managing stress and maintaining 

performance. By supporting cognitive functions such as obtaining parameter values (monitoring), comparing them 

to reference values, and monitoring steps of continuous applicability (discussed further below), CBPs may reduce 

the demands on attentional resources and working memory, enabling operators to focus on evaluating higher-level 

procedure goals. 

Level of Detail 

Space for explanatory information is limited in PBPs and the level of detail in procedure steps is fixed. 

Determining the appropriate level of detail in presenting procedure steps, and deciding upon the type and level of 

detail in supporting information are inexact processes that may be facilitated by computerization. Substantial 

interaction with trainers and operators is needed to decide upon the level of detail, which increases costs and still 

has an uncertain outcome. Linkages between training programs and the procedures can lessen over time, and so 

the operators' knowledge bases can change. In addition, even though all operators have reached the expertise 

required for licensing, differential experience may lead to differences in familiarity with the components, systems, 

and processes defined in the procedures. Those more familiar with a task may become impatient if the information 

is too detailed and may inadvertently skip steps to avoid wading through unneeded information. Operators who are 

less familiar with the task may be unable to perform the procedure correctly with the amount of information 

provided. 

An advantage of CBPs is that they can provide varying levels of detail (Fischer et al., 1991; Jenkinson et al., 1991), 

and computerization has been used elsewhere to resolve these problems. For example, in the early 1980s, the Navy 

developed computer-based troubleshooting aids for maintenance that allowed the user to choose between two

5- 13 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

different levels of detail. In the more detailed option, general steps (e.g., "start the pump") were broken up into 

specific substeps (e.g., "depress the start button on Pump XYZ") and more detailed graphical displays were shown. 

Hypertext applications are under development within the Department of Defense (DOD) as part of the Continuous 

Acquisition and Life-Cycle Support (CALS) initiative that also allow the user to determine the level of detail of 

information. Some systems allow the user not only to set a preferred level of detail, but also to browse through supplementary information, and the information that was used to develop the technical content of the procedure 

steps. Access to this latter information may be particularly useful if the procedure cannot be performed under 

existing plant conditions and must be modified. 

NPPs and vendors of nuclear steam supply systems have developed procedure-basis documents that give significant details and insights into the reasons for procedure steps and the logic for choosing parameters, operations, and step order; such information does not normally appear in procedures to avoid clutter. In CBPs this information can be 

made available when necessary. 

Improper implementation of a variable level of detail potentially may impair operators' performance if operators do 

not understand the level appropriate to their use of the procedure. 

Context Sensitivity 

Irrelevant information about conditions that do not exist during a specific implementation of a procedure must 

continuously be shown for decision steps in PBPs. Because nearly all procedures involve decisions, operators may 

have to read several pages of irrelevancies to find the appropriate action steps. This can cause operators to lose 

track of their place in the procedures, to miss important information, and to delay their performance. 

Several techniques are used in PBPs to present decision steps, some of which require less space than others. In 

text-format procedures, decision steps typically are represented by Boolean logic terminology (e.g., "if the 

following conditions exist, then perform these actions"). In flowchart-format procedures, the antecedent conditions 

are typically presented in decision diamonds, and consequent actions in rectangular symbols that are linked by flow 

lines to the decision diamonds. Other formats are possible, such as the two-column, "response not obtained" 

format used in EOPs for pressurized water reactors. 

These techniques also may be used in CBP systems, but may create similar paging problems on a VDU. However, 

CBPs can display only the relevant action steps for existing conditions. For example, once the operator has 

evaluated the existing conditions and chosen an action, only the information relevant to that action would be 

displayed. Alternatively, a CBP system could be designed to evaluate the existing conditions and to choose the 

action for the user. Then, both the full listing of possible antecedent conditions and the action steps for non

existent conditions could be "hidden" from the user. Although there may be some value in having the decision 

criteria and the "paths not taken" continuously available for review, a system may be designed to reduce the 

amount of information displayed to reduce errors and improve the efficiency of task performance. 

Sequence Control and Navigation 

In PBPs, information is presented sequentially. However, as Roth (1994) indicated, non-sequential access to other 

procedure information and support materials may be necessary for operators to adequately assess the procedure. In 

addition, even in current procedures, some steps are not performed sequentially. Two examples may clarify these 

problems and show how computerization could resolve them. Steps of continuous applicability are performed at 

any point in a procedure at which certain conditions are met (e.g., pressurizer pressure exceeds a given level).

NUREG/CR-6634 5-14
5 TECHNICAL BASIS DEVELOPMENT

Therefore, because the triggering conditions may be met at any time, the steps of continuous applicability always 

should be immediately available to the user. However, space on paper procedure pages is at a minimum (as it is on 

a VDU), so that continuously repeating the steps on each page increases the amount of skimming through pages 

that the operator must do. Further, because the steps of continuous applicability are not part of the direct sequence 

of actions means that they can be easily overlooked, even if they are invariably shown in a dedicated box on the 

procedure page or on a facing page. If a CBP system was designed to detect the triggering conditions for these 

steps, it could insert the appropriate step exactly when it is needed, so that the operator could immediately attend to 

it. Thus, space on the VDU screen is not wasted with a continuous display. 

Time-dependent steps are similarly problematic because they are only performed after some specified period (e.g., 

some NRC notification requirements in EOPs). Because the time taken to progress through a procedure may vary 

under different circumstances, it is difficult to show a time-dependent step exactly when in the sequence of steps it 

must be undertaken. Presenting the time-dependent step at the point in the sequence that "starts the clock" may 

mean that operators forget to perform it after the designated amount of time has passed. Continuously displayed 

reminders have the same limitations as repeating steps of continuous applicability in PBPs. However, because 

timekeeping is easily automated, a CBP system could have the timekeeping function (where either the operator or 

the system "starts the clock") and then display the action step when it must be performed. 

In paper procedures, cross-referencing between steps and procedures introduces errors and delays in task 

performance. Navigating through such cross-linked steps and procedures is a significant problem for NPP 

operators with PBPs because cross-references interrupt the user's sequential performance of action steps. For 

example, unconditional branches instruct the user to leave the current procedure and begin again in another 

procedure or in another section of the same one. References direct the user to another procedure for supplementary, 

information or for a series of action steps, after which the user is redirected to the original procedure and continues 

then to follow it. These non-sequential movements through PBPs cause the operators to lose track of their place in 

the original procedure, or to waste time trying to locate the procedure to which they are referred. 

CBP systems could be designed to assist operators in following cross-references or to eliminate the need for them. 

A CBP system can simplify the user's search task, for example, with a menu of procedures allowing the user to 

choose the cross-referenced procedure, rather than having to physically locate it in a paper manual. Operators can 

select the step to begin performing actions in the cross-referenced procedure, rather than scanning a document to 

locate the desired steps. Windowing or some other technique can support the function of place keeping in the 

original procedure so that the user can later return to it. Checkoff and place-keeping functions can be automated, 

so that operators can easily determine where they have been, what steps were completed, and where they left off in 

various procedures. 

More sophisticated systems are under development in the DOD's CALS initiative. Interactive electronic technical 

manuals (IETMs) for maintenance tasks are being developed in which procedure steps and other elements are 

stored in a database. At the beginning of a task, the user specifies the task and the circumstances under which it 

will be performed. The sequence of action steps and associated supplementary information (i.e., a complete 

procedure) then are generated by the system from the database, so that no cross-referencing is required. 

Management of Multiple Procedures 

Physical management of multiple procedures and place keeping when concurrently carrying them out are awkward 

with PBPs; the EOPs currently used in many boiling water reactors (BWRs) are an example. Operators who must 

use EOPs based on the General Electric (GE) Owners' Group technical guidelines, in some emergency scenarios 

may have to manage the concurrent performance of actions in up to 14 different procedures. The bulk of the action

5-15 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

steps are shown in flowcharts the size of engineering drawings; finding laydown space for these procedures in a 

typical CR is difficult, and the reader must physically move around them to track procedure steps. In addition, 

operators may be required to carry out abnormal operating procedures and some system operating procedures at the 

same time as the actions in the EOPs. Prioritizing the performance of steps in the different procedures is left to the 

unit supervisor, based on the staff available and the extent of degradation of particular parameters. The 

opportunities are clear for the CR crew to overlook steps in the procedures and to commit other types of errors. 

At a simple level, computer management of the progress and place keeping for multiple procedures may facilitate 

their use. At a more sophisticated level, CBP systems can be developed to prioritize and sequence the actions for 

each anticipated emergency scenario, and thereby substantially reduce the operators' workload. However, 

designing such a CBP system would seem to necessitate a significant change in the underlying mitigation strategy 

adopted by the GE Owners' Group. Because the Westinghouse Owners' Group's Emergency Response Guidelines 

already prioritize and sequence actions for pressurized water reactor (PWR) crews and involve fewer instances in 

which multiple procedures must be performed, EOPs for PWRs may be easier to translate into CBPs. Additionally, 

physical management of PBP manuals at remote locations, such as the remote shutdown panel, can be difficult. 

CBPs offer a simple solution to this problem, as they can provide the operators with the same interface normally 

used in the main CR. 

Maintaining Technical Accuracy of Procedures 

It can be difficult to maintain the accuracy of procedures, due to procedure modifications or to changes in other 

plant operations (e.g., regulatory requirements, equipment modifications). Maintaining technical accuracy is 

particularly difficult on paper. Thus, a design change in a single component can invalidate every procedure that 

refers to that component. Similarly, a procedure revision that changes the step numbers in one procedure can 

invalidate every step in other procedures cross-referencing the changed procedure. Some licensees already have 

developed elaborate configuration control software to solve this problem. 

Procedure-generation systems, such as the IETMs described above, can overcome some of these difficulties. Where 

procedure actions and the objects of those actions are stored in a database as objects, a change in a step or a part of 

a step in the database will ensure that the step is correct whenever it is used in any future procedure generated by 

the system. 

Integration of Procedures and Other Tasks 

The tasks associated with handling and reading a paper procedure may be incompatible with other tasks the 

operator has to perform. CBP systems for tasks performed at workstations, where the control actions can be 

performed at the same workstation at which the user obtains procedures information, can decrease the delays and 

potential errors associated with PBPs. Indeed, many licensees assign the task of reading the procedure to a 

different person to avoid delays and potential errors; CBP systems can eliminate this requirement. 

Summary 

Numerous limitations of PBPs have been identified and associated with delayed performance and human errors. 

CBPs offer the opportunity to rectify these problems. The following are PBP issues and the types of CBP support 

that may address them: 

. Design Process and Support

NUREG/CR-6634 5- 16
5 TECHNICAL BASIS DEVELOPMENT

- Weaknesses in the design process and management support have led to problems in PBPs and should be 

addressed when developing CBPs. 

General Cognitive Workload 

- CBPs may reduce the demands on attentional resources and working memory, and enable the operator to 

focus more on evaluating higher-level procedure goals. 

- CBPs can support cognitive functions, such as obtaining parameter values (monitoring), comparing actual 

values to reference values (resolution of procedure step logic), and monitoring steps of continuous 

applicability. 

° Level of Detail 

- CBPs can allow adjustment of the level of detail for operators with varying familiarity with the tasks, 

components, systems, and processes defined in the procedures may enable them to use procedural 

guidance more efficiently. 

• Context Sensitivity 

- CBPs can display only the relevant procedure steps for existing conditions, so operators are not distracted 

by irrelevant information. 

* Sequence Control and Navigation 

- CBPs can take advantage of non-sequential access to information using computer navigation functions. 

- CBPs can automatically detect the triggers for the steps of continuous applicability and time-dependent 

steps and insert the action step to be performed exactly when it is needed. 

- CBPs can perform cross-referencing, place keeping, and checkoffs. 

* Management of Multiple Procedures 

- CBPs can manage the progress of, and place keeping for, multiple procedures. 

- CBPs can prioritize and sequence the needed actions. 

- CBPs can eliminate some physical problems and coordinate many procedure manuals, especially when 

laydown space is unavailable. 

* Maintaining Technical Accuracy of Procedures 

- Procedure-generation systems using procedure databases can enhance accuracy. 

Integration of Procedures and Other Tasks

5-17 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

- Control actions for executing the procedure can be performed at the same workstation as the CBP, and can 

decrease the delays and the potential for errors associated with using PBPs. 

We note that some of these concepts have not yet been fully implemented or evaluated. Others may require 

substantial changes in the manner in which technical information is managed in NPPs. 

5.3 Existing Guidance for CBP Systems 

In this section, we discuss the available guidance for CBP systems. Three principal conclusions emerge. First, 

guidance for CBP systems is extremely limited. Wourms and Rankin (1994) noted that no comprehensive 

standards or guidelines are available for designing or evaluating CBPs. EPRI (1991) indicated that "... guidelines 

for such soft procedures are not well established and will have to be developed by the M-MIS designer." 

Second, there is some uncertainty over generalizing the principles and guidelines for PBPs to CBPs. Converse 

(1992) stated that "... There is no evidence that guidelines for the design of traditional hard-copy procedures can be 

successfully applied to computerized procedures, and few guidelines specifically address the design of 

computerized procedures" (p. 170). Similarly, Tolbert et al. (1991) concluded while much is known about the 

design of PBPs, the applicability of the information to CBPs is unknown. They also believe that problems may 

result from combining the use of PBPs and CBPs due to the need to train operators on both systems, to changes in "allocation of function" due to the CBP, consistency differences, and issues of procedure maintenance. 

Based on the generally acknowledged lack of HFE CBP guidance, the third conclusion is that the development of 

CBP systems for operational use should proceed in a way such that the benefits and drawbacks of CBP systems can 

be fully evaluated for each specific system. From reviewing the literature on CBPs, Chignell and Zuberec (1993) 

determined that "... a cautious approach should be taken [to computerization of procedures]. Relatively little is 

known about how operating procedures should be used in practice, and there is a possibility that problems with 

existing hard copy procedures may be compounded when they are computerized" (p. 1). Similarly, after reviewing 

several CBP systems, Spurgin, Wachtel, and Moieni (1993) concluded that "... more work needs to be done before 

the industry can make a safe transition from traditional paper and pencil procedures to computerized systems" 

(p. 1017). 

In the remainder of this section, the existing sources for high-level CBP design principles and guidance are 

discussed. 

EdF CBP Design Principles 

Based on their experience with developing CBPs for the N4 design, EdF (Dien, Montmayeul, and Beltranda, 1991) 

offered the following general high-level guidance: 

* The CBP should leave the operators in-the-loop. Therefore, it has no advisory role and leaves final decisions 

to the operators. 

* The CBP display screens should associate the control objectives, the current process solution, and the required 

actions. 

* Operators should be able to navigate freely within the procedure to make up for its insufficiencies.

NUREG/CR-6634 5- 18
5 TECHNICAL BASIS DEVELOPMENT

* The procedures should be represented at different levels to accommodate various operator skill levels. The 

EdF system has an action level, which covers the detailed tasks, and an "objectives-task" level showing the 

chronology and links between different objectives. 

It is noteworthy that several of these guidelines relate to the postulated ways in which CBPs can improve on PBP 

limitations discussed previously. 

EPRI Utility Requirements Document 

High-level CBP guidance was developed also by EPRI in the URD (EPRI, 1993b). It was based on EdF's CBP 

experience. EPRI required CBP systems in advanced light water reactors (paragraph 3.4.2.2). However, due to the 

lack of industry experience with them, the URD suggests that simulations are needed to develop detailed guidance 

and to validate the systems. The URD guidance is summarized below: 

• Procedures shall be in the form of logic or flowcharts. 

* The procedures shall normally provide, on the same display, the parameters necessary for the operator to make 

each decision. 

• Plant parameters and status in the procedures should be continuously updated. 

* The operator will be able to access the control needed to carry out the tasks directly from the procedure. 

* The procedures should have software to verify the operators' decisions. The operator shall retain control and 

be the final authority as to whether or not to proceed. Disagreements should be automatically logged. 

• Where appropriate, the procedures shall provide software which retraces certain sequences of steps to assure 

that proper status of systems or components is maintained. These steps shall not include actions taken by the 

operators to control components. 

* For control stations where CBPs are impractical, and to supplement CBPs, hard-copy procedures should be 

available. 

The format and content of hard-copy procedures should be consistent with the CBPs. Their practices shall 

also consider using PBPs when the normal CBPs are not available. In their rationale for this guideline, EPRI 

noted that the correspondence of CBPs and PBPs is important in minimizing the training burden and the 

potential for errors and misunderstanding. Further, this consideration is especially important when hard-copy 

procedures are used as a backup. 

• The M-MIS design process shall include validation of each operating procedure using the plant's simulator "and performance model. 

The NRC's review of the URD raised questions about the basis for the last URD requirement (see RAI 620.13, in 

NRC, 1991, pp. 6-7). As noted above, EPRI (1991) indicated that CBP guidance is lacking and that it will have to 

be developed by the designer, using simulation. The response noted that "... Since both the 'soft' and 'hard' 

procedures are subject to the test of active simulation, there will inherently be a direct comparison between the 'soft' 

and the 'hard' procedures as part of the design process. Differences in operator performance with the computer

presented procedures compared to the conventional printed procedures should be evident from these evaluations"

5-19 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

(p. 31). Further, EPRI indicated that "... If the soft procedures are not concluded to represent an improvement 

when active simulation is attempted, there is a clear fall-back to hard copy procedures" (p. 30). This is consistent 

with the URD requirements for an unproven HSI technology. The URD defines proven technology as one which 

has at least three years of documented satisfactory service in a light water reactor (LWR) or similar application. 

When these criteria cannot be met, a testing and V&V program must be conducted. 

In considering the EPRI URD and the subsequent response to the RAI, the staff noted the following: 

...the development of electronically displayed procedures is a desirable goal for the overall 

integration of operator information needs. The staff position is that the M-MIS designer should 

consider the use of electronically displayed procedures early in the design process to resolve any 

issues concerning their development, operability, maintainability, and reliability. If electronically 

displayed procedures are determined to be an improvement over hard-copy procedures and the M

MIS designer has integrated electronically displayed procedures into the overall M-MIS design, they 

should be provided as part of the design. (NRC, 1994, p. 10.B-17) 

Barnes, Desmond, Moore, and O'Hara 

Barnes et al. (1996) developed a set of principles representing a logical extension of PBP guidance to CBPs based 

on their experience with PBP design, operational experience, and with issues on computer-based systems. 

Recognizing concerns about generalizing PBP principles to CBP applications, it was done carefully, in a limited 

fashion. 

The guidance was divided into two primary sections. The first, "Design Development," covered aspects of the 

design process in procedure development. The second section, "Implementation," addressed the detailed design of 

the procedures in the HSI. Each section was divided into several subsections: 

* Design Development 

- Concept of Operations 

- Procedure Bases 

- Design Process Considerations 

- Maintainability of Computer-Based Procedures 

- Training Specifications 

• Implementation 

- General Considerations 

- Detailed Considerations 

- Detailed Interface Design for Constructing Basic Steps 

- Steps Containing Conditional Relationships

NUREG/CR-6634 5 -20
5 TECHNICAL BASIS DEVELOPMENT

Warnings, Cautions, and Notes 

Level of Detail 

Organization 

These guidelines were evaluated and used in the context of the current review. 

Niwa, Hollnagel, and Green 

Niwa, Hollnagel, and Green (1996) identified some high-level requirements for CBP systems. In general, they 

stated that computerization should make procedure tasks easier without imposing additional tasks, and that CBP 

systems should provide the following to improve the use of EOPs: 

"* Formatting - The CBPs can help to structure the various procedure components such as steps, conditions, 

comments, and advice. Graphical techniques may also help operators to understand the logical relationships 

(conditionals, conjunctive, and disjunctive) defined in procedure steps. 

"* Process linking - Integrating parameter values into procedure steps will facilitate their usage. 

"* Navigation Support - CBPs can assist operators in moving between procedures and support information. 

"* Progress monitoring - CBPs can track what steps were completed. Check boxes can be used, either manual or 

automatic, depending on whether the CBP has the specific criteria and information to determine whether a 

step was completed. Completion also can be time-stamped to facilitate post-hoc incident analysis. 

"* Help and explanation - Information can be provided to help operators carry out procedure steps. For example, 

the help facility could describe how a control action should be carried out. The rationale for procedure steps 

could also be explained. CBP systems could also permit variations in the level of detail based on operators' 

experience and input. 

"* Procedure adaptation - CBPs may facilitate changing a procedure to better meet the current situation. 

Niwa et al. (1996) stated that their guidelines are good for general aspects of procedures, but lower-level details 

should be developed with operations and engineering personnel. They stressed the need for consistency between 

the CBPs and the rest of the HSIs in characteristics such as colors, typography, interaction methods, and input 

devices. Such lack of integration is a potential source of risk, and reduced reliability in performance. They noted 

that lack of consistency may be a problem with third-party CBP systems. 

Summary 

The principal conclusions of this section are, first, that while some guidance for CBP systems exists, it is limited. 

Second, there is uncertainty over the generalization of PBP principles and guidelines to CBPs. Finally, CBP 

systems for operational use should be developed in such a way that the benefits and drawbacks for each specific 

system can be fully evaluated.

5 -21 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

5.4 Research on Computer-Based Procedure Systems 

Human performance concerns related to CBPs have been raised. For example, Wourms and Rankin (1994) 

indicated that CBPs may exceed the processing and attentional limitations of operators. Chignell and Zuberec 

(1993) suggested that operators may become disoriented and lost, may suffer from keyhole effects, and may lose the 

location of information in windows displays. In this section, the effects of CBPs on performance are considered 

from three perspectives: empirical evaluations of performance, analytical evaluation of CBPs, and expert opinion. 

5.4.1 Empirical Evaluations of CBPs Based on Personnel Performance 

Empirical evaluations of CBP systems and characteristics provide the best information upon which to develop 

guidance for design reviews. Several empirical investigations of CBPs have been reported, but before discussing 

them, it is important to identify the criteria by which such studies were evaluated. A most important consideration 

is that the CBP studies provide a basis from which conclusions can be generalized beyond the specific individual 

study. Therefore, the CBP studies were evaluated within the context of validation reviews where generalization 

(external validity) is a primary consideration. 

O'Hara, Stubler, Brown, and Higgins (1997) discussed the detailed methodological considerations for validating 

complex human-machine systems, and developed a conceptual approach that identified important principles and 

their relationships. The general concepts are concerned with (1) establishing the requirements for making a 

logical, defensible inference from validation tests to predicted integrated system performance under actual 

operating conditions, and (2) identifying aspects of validation methodology that are important to the inference 

process. The technical basis for inference is based upon four general forms of validity: system representation, 

performance representation, test design, and statistical conclusion. 

Validity of system representation refers to the degree to which the tests include aspects of the integrated system 

that are important to real-world conditions. Specifically, this validity is based on the representativeness of the 

system model, human-system interface, personnel, and operational events. Inference is supported to the extent that 

important aspects of the integrated system are represented with high fidelity, and to the extent to which important 

contributors to potential variability in system performance were adequately sampled. It is especially important in 

evaluating CBP studies that they were conducted in situations - test scenarios - that reveal the complexity of 

procedure use, and that the procedures were used by professional operators trained in their use. 

Validity of performance representation refers to the completeness and representativeness of the performance 

measures. A comprehensive, hierarchal approach to evaluation guided by supervisory control theory may be used 

to specify important aspects of performance, ranging from the operators' cognitive processes to system functions. 

The effects of CBPs on performance can stem from both the technology itself, and its interaction with the other CR 

technologies. In general, the effects can be related to (1)personnel role - a change in functions and responsibilities 

of personnel, (2) primary tasks - a change in the way that personnel perform their primary tasks, such as process 

monitoring, situation assessment, response planning, and response implementation and control, (3) secondary 

tasks - a change in the tasks the operator must perform when interacting with the CBP, such as navigating through 

displays and searching for data, (4) cognitive factors - e.g., a change in cognitive workload, and (5) personnel 

factors - a change in the required qualifications or training of plant personnel. The performance measures used 

must address these effects. Failure to include measures of all important performance variables, poor measurement 

properties, and poorly specified criteria weaken this validity and the ability to generalize the results. 

Validity of test design addresses the procedures used for conducting the tests. Inappropriate test procedures can 

bias the relationship between the observations of performance and the integrated system, and thus undermine their

NUREG/CR-6634 5 -22
5 TECHNICAL BASIS DEVELOPMENT

causal linkage. When factors introduced by the test methodology weaken the ability to interpret this correlation, 

validity is compromised. 

Finally, validity of statistical conclusions addresses the relationship between the observed data and established 

performance criteria. 

While these types of validity and their associated methodologies were evaluated in the studies reviewed in this 

section, the analysis was severely limited by the extent to which the studies are documented. With this caveat in 

mind, the studies are discussed below. 

The research is organized into three sections: (1) Comparisons of CBP and PBP Systems, (2) Observations of 

Operators' CBP Use, and (3) Comparisons of CBP Design Characteristics. 

Comparisons of CBP and PBP Systems 

Spurgin et al. (1990) compared the BWR Emergency Operating Procedure Tracking System (EOPTS) with PBPs 

in flowchart format. The study was conducted with professional operators at the training simulator of the 

Kuosheng plant in Taiwan. The operators did not have much experience with either form of EOPs. EOPTS is 

used by the shift supervisor. It automatically engages when an entry condition is specified; otherwise, the EOPTS 

display screens are blank. The plant is controlled through the normal HSI. 

The study was conducted in two phases. In phase 1, six crews performed four scenarios; three crews used the 

CBPs, and three used the PBPs. The study was undertaken as the crews were being trained on EOPTS. In phase 2, 

there were 12 crews, six for each condition. The performance measures included: 

* Time - specific waypoints were defined for each scenario, measured from first cue to when the appropriate 

response was made. 

Errors - the number of deviations from EOPTS-specified actions (this measure also was applied to the PBP 

groups because, the EOPTS was running although the crews could not use it). 

* System measures - specific measures of system performance were defined for each scenario. 

The results showed that the CBP compared favorably to paper flowcharts. The time measures for human 

interactions were not completely reported. For those that were, the median response times for the CBP crews were 

faster in 16 of the 18 HSIs analyzed (an overall time reduction of about 75 percent). In addition, the response 

times of CBP crews were less variable than those of the PBP crews. There were scenario effects, as well. For two 

of the six scenarios, the response time was slightly increased in the CBP group. 

Overall, the operators made about twice as many errors with the flowchart procedures; unrecovered errors were 

65 percent with flowcharts, and 27 percent using EOPTS. When using flowcharts, the most likely source of error 

was misinterpretation of a procedure statement. With EOPTS, it was communication with the control board 

operators. 

Several possible flaws in this study make interpretation of the findings difficult. First, since it was a between

groups design and a small number of crews was used, their differences may have been confounded with differences 

in the presentation of procedures.

5 -23 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Measurement was another issue. Errors were identified as the number of deviations from the specified path through the procedure, as defined by the CBP. However, as discussed earlier, such a strategy of following 

procedures by rote may not be desirable (Barnes et al., 1996; Roth et al., 1994) and may reflect a blind, 

unverifiable approach. Further, the instruction given to the CBP group was to follow the CBP verbatim, but, no 

equivalent instructions were given to the PBP group. Thus, a confound was created favoring the CBP group who 

were given the criterion for performance while the PBP group was not. This also affects the interpretation of the "error" recovery measure. Another issue was that since the crew's responses differed across crews, the appropriate 

procedural response differed within the same scenarios. 

System measures were not reported in any organized detail. In fact, they were only reported for the "LOCA with 

Dry Well H2 Control" scenario. The maximum drywell hydrogen concentration averaged 5.9 percent for the CBP 

group, and 8.8 percent for the PBP group. Further, the cumulative time below "top of active fuel" was 92.5 and 

325 seconds for the CBP and PBP groups, respectively. 

Without system measures for other scenarios, it is difficult to assess whether the differences between the crews' 

performance on the two procedure systems are meaningful. 

Several interesting observations were reported on situation awareness and crew communication and coordination. 

Spurgin et al. noted that SROs using EOPTS were likely to use it as their primary way of following a transient (i.e., 

not using other HSIs), which may have hampered their awareness of the overall condition of the plant. Crew 

members in the EOPTS condition who were not using the CBPs expressed concern about being aware of the EOP's 

status. 

Time differences also were important. One SRO called for ADS initiation twice because he did not think it had 

occurred after his first request. The misunderstanding was due to the delay in the CBP's updating of ADS status. 

It is interesting to examine the establishment of EOPTS at Kuosheng. The system was gradually introduced so that 

all plant personnel could become familiar with it. It was first introduced into the training simulator (1) as an aid to 

instructors to track the operators' responses to accidents, (2) as a training tool for the crews to examine accident 

response strategies, and (3) as a tool to be used by the crews during accident response. During this time, in 

addition to training, the correctness of EOPTS was examined. Spurgin et al. (1990) point out that several errors in 

the PBPs then were discovered as well. They noted that crews in the CBP conditions operated much more in the 

skill- and rule-based mode, while crews in the PBP conditions operated more in the knowledge-based mode. This 

could certainly be another artifact of the instructions to follow the procedures verbatim. However, it is not 

necessarily a positive outcome (see discussion in Section 5.1 on the need for crews to maintain an independent 

perspective on the procedures). Spurgin et al. considered that the major benefit of EOPTS was that it helped 

operators to follow the procedures correctly, and to interpret the logical statements that are a part of the procedure 

steps. 

Another factor that may have affected the results was the crews' inexperience with either form of procedure. A 

comparison of performance with the flowcharts between the first and second phases of the study indicated that 

there was a considerable improvement. In fact, Spurgin et al. noted that one crew studied the flowcharts before the 

scenarios. They used the process computer to display important variables referenced in the EOPs, and their 

response time was comparable to that of the EOPTS crews. Since the CBP group was following the CBP verbatim, 

their task may have been easier only because they did not have to know how to use the flowchart. If so, 

performance differences between the two groups may be considerably less if they received additional training, or if 

they gained more experience with the PBPs.

NUREG/CR-6634 5 -24
5 TECHNICAL BASIS DEVELOPMENT

The observations of differences between crews and scenarios led Orvis and Spurgin (1996) to recommend that 

CBPs should be thoroughly validated using several crews and scenarios. CBP systems should be validated by 

operator-in-the-loop evaluations to ensure that they achieve their objectives, and that a smooth transition between 

CBPs and PBPs can occur when necessary. 

In general, while this study illustrates some potential benefits of CBPs, the results are limited by (1) 

methodological confounds and procedural limitations, (2) incomplete reporting of data, (3) questionable measures 

of performance, and (4) underspecification of performance measures, i.e., important aspects of performance were 

not measured, such as situation awareness and workload. 

Nelson et al. (1990) compared another procedure system, Halden's Computerized Operation Manual (COPMA), to 

performance using PBPs. COPMA was an earlier version of COPMA II. Fourteen Halden reactor operators 

participated in the study during simulated process disturbances. The dependent variables included time to access 

and complete the procedure, number of errors, and process parameters reflecting the operator's effectiveness in 

handling the disturbance. In general, it took longer to access the correct procedure with COPMA than with PBPs, 

a difference attributed to processing time. Further, COPMA did not reduce the time needed to perform procedure 

activities, and sometimes the PBP condition was significantly faster. The COPMA group made slightly more 

errors than the PBP group; however, the differences were not significant. No significant differences were observed 

for the process variables. 

On a methodological note, Folleso, Meyer, and Volden (1993) and Hallbert and Meyer (1995a, 1995b) indicated 

that there were large differences between the COPMA and PBP groups as measured in a pretest, and concluded 

that the results were confounded by them. When the pretest measures were used to adjust the performance 

measures, the differences between the two groups lessened (the CBP group performed a little better than the PBP 

group). However, the assumption of confounding based on pretest differences may not be warranted when 

participants are randomly assigned to conditions. Thus, such an approach for correcting the data may not be 

justified in a randomized design, and is more appropriately used in a quasi-experimental design where non

equivalent groups exist prior to the study. Thus, the results for the unadjusted data are reported above. 

Based on the evaluation of COPMA, the CBP was revised to produce COPMA-II. Some of the changes were 

increased functionality, including support for procedure search, improved instructions on procedure steps, and 

more explicit references to procedure branches. 

Crews' performance using COPMA-II was compared to PBPs in another study (Converse, 1994, 1995). Sixteen 

licensed operators managed a change in power, small-break LOCA, and a steam generator tube rupture on the 

Scaled Pressurized Water Reactor Facility at North Carolina State University. The operators worked in teams of 

two, with an SRO managing the procedure, and a reactor operator (RO) assisting in data collection and control. 

The dependent measures were procedure initiation time, completion time, subjective estimate of workload using 

the National Aeronautics and Space Administration Task Load Index (NASA-TLX), and number of errors (defined 

as deviations from the "optimal" sequence of procedure actions). 

The operators responded faster in accident scenarios with PBPs, but their response-completion time showed no 

significant differences. Measurements of the operators' accuracy revealed an interaction between the type of 

procedure and the accident scenario. The error rate for PBPs was four times higher than COPMA-II for the LOCA 

emergency event, while there was no significant difference for the tube rupture. As with the Spurgin et al. study, 

the meaning of the error data may be open to alternative interpretation. Defining error as a deviation from an 

optimal sequence may be overly restrictive. If an operator looks ahead at upcoming steps, an error is recorded, but

5 -25 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

these types of activities are not necessarily undesirable (Roth et al., 1994). There were no differences in workload 

between the two procedures. 

Converse concluded that future evaluations of CBPs should systematically vary the type of scenario because of the 

different CBP effects on the two types. 

Like the EOPTS study, the results of this study are difficult to interpret. The most significant result was related to 

error data and, like the EOPTS study, the definition of error is questionable. No differences in completion time nor 

workload were found. Again, performance measures were underspecified, i.e., situation awareness and plant 

performance were not measured. 

Collier (1996) developed the following lessons learned from Halden's CBP evaluations: 

* Operators must maintain an appropriate degree of control. The CBP system should not overly structure the 

operator's movement through the procedure, but should offer flexibility to skip steps or skim over them quickly 

when appropriate. 

* New HSI systems must offer an advantage over other HSI resources to be used. 

* To be effective, automation needs the operator's trust. One reason offered for the slower performance with 

COPMA compared with PBPs was that operators spent time double-checking COPMA information because 

they may not have developed confidence in it. 

Three other studies comparing CBP and PBPs were reviewed; however, they did not provide sufficient information 

to analyze them in detail. Kang (1997) described an intelligent HSI being developed in Korea by the Korea Power 

Engineering Company, Inc., that included a CBP for EOPs. Each step is composed of an observation, judgement, 

and control. Observations are performed automatically. The system makes two types of judgements. Quantitative 

judgements are easily made. Qualitative judgements, for example, "If RCS average temperature is greater than 

292°C and increasing, then..." are evaluated by fuzzy logic.2 The level of automation is varied depending on the 

complexity of the required decisions. For simple skill- and rule-based tasks, the system is automated. For 

knowledge-based tasks, control is manual. The related piping and instrumentation diagram (P&ID) and summary 

information are automatically presented. The system was tested in a steam generator tube rupture scenario with a 

full-scope simulator. The response time for completing the required actions were compared with hard-copy EOP 

performance. The operators took 37 percent more time with the PBP system than with the CBP. 

The Emergency Operator Support System was developed in Japan to support the transition from event- to function

based emergency procedures, and for using of the EOPs (Yamamoto and Ito, 1993). The system automatically 

displays the highest priority procedure in a flowchart form consistent with that of the PBPs. The system extends 

beyond procedures and includes supporting displays for diagnosing event and plant status. The system was 

validated in tests involving ten crews; errors were reduced approximately 50 percent with the CBP. 

Mavko et al. (1995) state that classical paper-based EOPs are not suitable for use in CRs with digital process 

information systems. They developed the Computerized Procedure (COPRO) system. COPRO is function oriented 

and enables operators to restore and maintain critical safety functions. It includes the same information as the 

symptom-based PBPs, to provide operators with necessary information, record their actions, and perform tasks 

2 This type of step is qualitative because its specification is imprecise in that judgement is used to determine 

the rate of increase that is minimally necessary for operators to conclude the rate is actually increasing.

NUREG/CR-6634 5 -26
5 TECHNICAL BASIS DEVELOPMENT

automatically - such as monitoring of critical safety function (CSF) status trees ahd comparing referenced and 

actual values. The procedure continued automatically until stopped. The system was tested using a small-break 

LOCA event. An operator completed the procedure more quickly with the CBP than with the PBP and believed 

that fewer errors would be made. 

Observations of Operators' Use of CBPs 

The N4 CBP system includes all the N4 procedures, not only the EOPs. However, while EdF spent significant 

effort on designing and evaluating the CBPs, we are not aware of any papers with detailed results of the 

evaluations. Therefore, we reviewed the results of several papers that discussed various lessons learned from the 

evaluation. 

Bozec et al. (1990) investigated an early version of the N4 CBP system. Six crews of operators participated in the 

tests with the N4 simulator. Their evaluation was mainly qualitative, but from the deficiencies revealed, they made 

recommendations for improving the system. They found that the objectives of the procedure needed to be better 

emphasized to increase the operators' awareness of the high-level goals. They suggested that providing too much 

detail should be avoided, except when there is a problem. The operators did not want the procedure to 

automatically reset or return to a previous step when the status of the process changed. However, they wanted 

automatic monitoring of previous steps and indications of a change in their status. The operator should be able to 

override a course of action that is recommended by a CBP system, as when the operator has access to information 

that is not available to the CBP, the CBP's guidance is too strict, or the CBP is using old information. 

Pirus and Chambon (1997) offered additional lessons learned from EdF's CBP evaluations. Handling multiple 

procedures is easier when the relevant information in each is highlighted, so that when operators transition from 

one to another, the highlighted information directs them to the appropriate location. Also, automatic monitoring 

of process parameters helps the operators. Finally, the quality of operations is improved when operators are alerted 

to deviations from the specified procedure path, because they then can decide if that is what they want to do. 

Jeffroy and Charron (1997) discussed the safety assessment of the EdF CBP system performed by France's Institute 

for Nuclear Safety and Protection (French acronym IPSN) for the Nuclear Installations Safety Directorate (DSIN). 

The evaluation was a simulator exercise which revealed several problems. While these have been resolved in the 

N4 system, they are important considerations for other CBP systems: 

Overall View of the Process - Early in the design, operators worked through the flowchart and responded yes or no 

to each step. Their responses were monitored by a "path monitoring function" and deviations from the computer 

were highlighted. This enabled operators to catch "local" errors, but made it easy to lose the overall view because 

of the step-by-step attentional demands. By presenting procedures as a series of pages, computerization makes it 

more difficult to view the path taken, to apply hindsight, and to anticipate the consequences of an answer. 

Conflicts - While a high degree of guidance can be delivered, not all steps can be specified in equal detail. While 

the CBP is designed with steps in a certain sequence, operators sometimes need to alter it. Also, sometimes 

operators may disagree with the CBP's recommendations. In both situations, operators may find it hard to disagree 

with the procedures, especially when the level of detail is high. CBPs also make it difficult to view the path taken, 

and this can hamper independence from the procedure. Operators would sometimes reset the procedure to get a 

better sense of how they got to a particular place.

5 -27 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Limited view - While the CBP monitors the plant through instrumentation and control (I&C), the operators must 

supply some information; thus, the CBP may consider specific components to be available when they are not. If 

such communications fail, the CBP may make incorrect assessments and give incorrect guidance. 

These findings illustrate the importance of operators being aware of the CBP's constraints. However, some 

situations made it particularly difficult to recognize them. For example, after the crew negates a procedure 

decision, their awareness of the basis for the procedure decisions becomes less clear, and the operators' and CBP's 

"situation awareness" begins to diverge. Operators then may not understand the information provided nor the 

effects of their actions on the computer's interpretation of steps. 

Jeffroy and Charron (1997) concluded that operation of procedures is a dynamic process involving interpretation of 

plant data, actions to be carried out by the plant, and interactions between crew members. Knowledge-based 

understanding is needed to properly follow procedures and to evaluate the correctness of recommendations. 

Procedure steps often require the operators' input and cannot be resolved independently. 

Roth and O'Hara (1998) studied the integration of advanced HSIs into an NPP. During computer replacement, the 

plant's CR was upgraded to include a CBP system, an advanced alarm system, and a graphic-based plant display 

system. The authors observed crews during their initial training with the new systems on a full-scope simulator, 

and interviewed operators and other utility and vendor personnel. The training included full-scope simulations of 

plant disturbances. 

This study was one of the first to evaluate a text-based CBP system (previous studies were of flowchart CBPs) and 

one of the first to look at the transition to PBP upon a CBP failure. It also was one of the first to examine 

performance with a combination of computerized HSIs. The results are summarized below. 

(1) The general effect of the CBP on performance was good. The SROs could go through procedures more 

quickly, and felt that their cognitive workload was reduced because information on plant parameters was 

immediately available (the SROs did not have to ask for it, and operators did not have to run around to get 

it), and the SROs did not have to resolve step logic. In general, procedures were followed more efficiently 

because the operator was less likely to miss a transition step and did not have to track location within the 

procedure, steps of continuous applicability, applicable cautions, or applicable foldout page criteria. The 

CBP was easy to learn, and the operators' acceptance was high. 

(2) The CBP had an important effect on the crew's roles and communication; the extent of the change was 

greater than anticipated for board operators. Since the SRO could handle the procedure mainly alone, the 

need for communication between the SRO and ROs was reduced. The operators identified the importance 

of communication in maintaining effective teamwork. The ROs expressed a need to be aware of status of 

EOPs. Because the ROs no longer needed to support the SROs in following the EOPs (by providing 

parameter values called out in the EOPs), they had more time and attentional resources available to 

monitor the plant, giving them an additional independent overview of its state. The ROs felt they became 

more independent and, thus, had more responsibility. Therefore, their individual skills become more 

important. 

(3) The operators' trust in the CBP was high. They generally assumed that the software logic was correct and 

did not feel a need to double-check it by reading every substep. Instead, they double-checked the system's 

conclusions from independent sources (e.g., the alarms and the board indicators). Because they generally 

trusted the CBP, they sometimes felt there was too much information, preferring detailed information only 

when a procedure step was not satisfied.

NUREG/CR-6634 5 -28
5 TECHNICAL BASIS DEVELOPMENT

(4) Operators occasionally wound up in the wrong place when using the CBPs, such as when step-logic 

resolution was oversimplified, but usually recovered quickly. Determining whether a step was satisfied 

was sometimes more complicated than at first it appeared (e.g., interpreting apparently simple statements 

such as "If pressure is decreasing..." can involve judgement that is difficult to reduce to a simple 

calculation). Mostly, crews could detect when CBP information was inappropriate. The operators were 

generally tolerant of these situations, and felt that similar ones occur with PBPs. 

One question was whether the ability to identify such errors would be the same with crews that were not 

initially trained with PBPs. Walking through paper-based EOPs enabled operators to identify the goals 

and logic behind them which they could transfer in using the CBPs. CBPs may exacerbate the tendency to 

follow EOPs verbatim, without sufficiently reflecting on the appropriateness of procedure steps to high

level goals. 

(5) Operators expressed initial concern over lowered situation awareness with CBPs, but it diminished with 

practice. 

(6) Operators did not have a problem when transitioning to PBPs upon CBP failure, although such failures 

were simple ones and happened early in the EOP. 

The results should be interpreted within the context of the study's constraints: (1) the observations were made 

during the first training period using the new CBP systems, (2) the CBP systems were not completely debugged, 

and (3) the scenarios were limited to relatively simple events. 

During an evaluation of CBPs for a low-pressure injection system, Blackman and Nelson (1988) noted the 

following: 

(1) Operators tended to believe the computerized procedure even if it was wrong; they should be trained to 

question it. 

(2) If selecting procedures is automated, the operators' involvement was reduced, and they reported that they 

thought less, and acted as switch-turners. Operators should be trained on the decision process used by the 

procedure system and to verify its recommendations. 

(3) Operators continued attempts to implement a computer's recommendation even when failures prevented it. 

Operators need to be trained to take over if the computer fails or is in error. 

(4) Computers do not have common sense functions employed by operators. The computer cannot consider 

what operators are doing nor other important information. 

The major conclusion was that operators need to understand the overall purpose of the procedures and stay 

cognitively involved in their progress. They should be trained to question any recommended steps that appear 

inconsistent with the overall goals. Similarly, CBPs should be designed to maintain information on anything the 

crew is doing that is relevant to implementing the procedure. 

Comparisons of CBP Design Characteristics 

In this section, we discuss research on three characteristics of procedure design: presentation format, salience 

coding, and integration of indicators.

5 -29 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Presentation Format 

PBPs generally present procedural information in text or as flowcharts. A concern associated with CBPs is the 

appropriateness of the presentation format, given the operator's task requirements and the characteristics of the 

display system. 

Wourms and Rankin (1994) commented that text is a sequential format that often requires users to read 

information that is not relevant to existing conditions. Sorting through this information to identify the correct 

course of action is time-consuming and confusing. An important consideration in using text is to establish the 

appropriate level of detail. Operators rely more on their memories than on the actual procedures because the 

narrative style uses too many conditional statements, which slows their response time. Some systems overcome 

this problem by providing information at more than one level of detail for each step. For example, an extended 

text version of a particular step may be used by less experienced operators, while an abbreviated version may be 

used for familiar procedures and steps. 

In general, a flowchart format is useful because of its ability to specify the sequence of, and relationships between, 

procedures (Krohn, 1983; Wourms and Rankin, 1994). Desaulniers, Gillian, and Rudisill (1988) compared 

flowchart formats to text and extended text formats. Each was displayed in a six-line window. Participants were 

asked to diagnosis a malfunction in a Space Shuttle system. Performance was most accurate with the flowchart 

format. Overall completion times did not differ between formats, but individual steps were completed faster with 

the flowchart format. 

In a second experiment, an interaction was revealed between format and window size (6- vs. 12-line). As the 

window's size increased, performance degraded with the flowchart format, but improved with the text formats. An 

examination of the errors that occurred suggested that participants lost their place on the flowchart with the larger 

display. For text procedures, the increased window size helped users to better understand the context of procedure 

steps. The effect of screen size upon accuracy may have important consequences when converting PBPs featuring 

flowcharts to a computer-based medium. Currently, paper-based EOPs based on the GE Owners' Group technical 

guidelines have flowcharts that are the size of engineering drawings (Barnes et al., 1996). 

Salience Coding 

CBPs are intended to guide the operators' performance during plant upsets that may be associated with time 

pressure and stress. Also, multiple procedures may be simultaneously in use. Salience coding can visually 

enhance presentation formats, such as text and flowcharts. Color, flashing, and animation may be used to enhance 

the salience of important information. These techniques can lower workload by helping to organize information 

and guide the operators' attention to that which is most important. 

However, because salience coding can affect the operators' behavior, care must be given to avoiding coding 

schemes that are distracting, confusing or misleading. For example, Mosier, Palmer, and Degani (1992) state: 

The logical conclusion from the results of research on salience effects on decision making has been that, in a 

diagnostic situation, the brightest flashing light, or the gauge that is largest or most focally located will bias the 

operator toward processing its diagnostic information content over that of other stimuli. Time pressure, stress, 

or information overload can cause a "perceptual tunneling" and exacerbate this tendency to focus on central or 

salient cues. (p. 10)

NUREG/CR-6634 5 -30
5 TECHNICAL BASIS DEVELOPMENT

Integration of Indicators 

Incorporating plant indications into CBPs poses both potential benefits and obstacles to human performance. 

Errors associated with monitoring the wrong display can be avoided by providing the operator with specific 

indications; Galletti (1996, event 4) describes the actuation of an engineered safety feature because the operator 

was monitoring a wide-range instrument rather than a narrow-range one. However, while PBPs force the operator 

to monitor plant indications, incorporating them into CBPs may increase errors by becoming a substitute for good 

monitoring practices, or by competing with other information sources in the CR for the operator's attention. For 

example, an assessment of electronic checklists (Mosier, Palmer, and Degani, 1992) concluded that those 

encouraging crews to rely on the system's state, as indicated by the checklist, rather than as indicated by the system 

itself, can discourage information gathering, and may lead to dangerous errors. In this aircraft simulation, the 

mean number of informational items discussed among crew members decreased as the checklist became more 

automated. Pilots who used paper-based procedures were less likely to shut down one of the aircraft's engines 

unnecessarily. 

Thus, while using CBPs, operators may not feel the need to look at other sources of information in the CR and, 

thus, may miss important indications that are not present in the CBP system. This need for other information is 

particularly important where the system's designer did not fully understand the plant's behavior, or where the CBP 

system fails in a manner that is not immediately obvious to the operator. For example, in events 2 and 3 described 

by Galletti (1996), lockups of the plant's alarm systems were only discovered after other information sources were 

compared to the data provided by the failed alarm systems. 

Summary 

In the beginning of this section, we discussed criteria for assessing studies. However, only two studies were 

described in sufficient detail to evaluate their generalizability (Spurgin et al., 1990; Converse, 1994, 1995). Both 

had potential methodological weaknesses which limit the conclusiveness and generalizability of their results. Most 

other studies were not reported in sufficient detail to make this evaluation, or contained only qualitative 

observations. 

Even considering the weaknesses in the design or in the reporting of methodology and results, some tentative 

conclusions can be drawn based on human performance data, observations, and interviews. On the positive side, 

when using CBPs 

"* operators can perform procedures more quickly, 

"* operators' cognitive workload seems to be reduced, and 

* operators may make fewer errors in transitioning through procedures. 

In addition, the CBP systems seem to be relatively easy to use and are accepted by operators. 

However, there remain several important, unresolved questions needing additional empirical research: 

* What is the effect of CBPs on team performance and reliability? 

* What is the effect of CBPs on the operators' high-level situation awareness of the status of the overall 

procedure goal and the plant?

5-31 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

* Do operators become over reliant on, and unquestioning of CBPs, or can they maintain the independence and 

objectivity to evaluate the adequacy of the procedure to achieve high-level goals? 

* What is the overall effect on operators' errors of CBP systems (especially where errors are not defined in terms 

of verbatim compliance)? 

* What is the effect of CBPs on performance in complex disturbances involving many procedures or branches? 

* How well do operators manage complex CBP failures, such as when multiple procedures are being used, many 

steps have been completed, and many steps of continuous applicability are being monitored? 

* What are the relative effects of specific design features on performance (most studies were overall system 

comparisons, e.g., CBP vs. PBP, not systematic evaluations of individual characteristics, such as the 

appropriate level of automation)? 

5.4.2 Analytical Evaluations of CBPs 

In this section, two classes of analytical techniques, performance models and risk models, are described that were 

used to evaluate CBP design. 

5.4.2.1 Performance-Model Analyses 

CBPs were evaluated using a variety of performance-analysis models including the Goals, Operators, Methods, and 

Selection (GOMS) model, MicroSaint Task Network Modeling, the Man-Machine Integrated Design and Analysis 

System (MIDAS) model, and classical task analysis. 

GOMS 

Endestad and Meyer (1993) compared COPMA and COPMA-II using the GOMS model of HSI developed by Card, 

Moran, and Newel (1983). As noted earlier, changes in the CBP included increased functionality for searching 

procedures; improvement in instructions for procedure steps, and more explicit references to procedure branches. 

Their results indicated that the modifications resulting in COPMA-II require additional learning and make the 

system more complex; however, COPMA-II can be used more rapidly. They did not identify the net effect of this 

tradeoff on human performance. 

MicroSaint Task Network Modeling 

COMPA-II was evaluated using MicroSaint Task Network Modeling (Laughery and Persensky, 1994). The 

operator's performance was compared with experimental data collected on a simulator (Converse, 1995). The 

comparisons were described as encouraging; the model's predictions of performance differences were consistent 

with the data in five out of six conditions. 

MIDAS 

Hoecker et al. (1994) and Hoecker and Roth (1996) used the MIDAS to evaluate CBPs (Westinghouse's 

COMPRO) against PBPs. We note that the primary objective of the study was to demonstrate the application of 

MIDAS to HSI evaluation. This comparison provided a test case.

NUREG/CR-6634 5 -32
5 TECHNICAL BASIS DEVELOPMENT

The results indicated that the effect of CBPs on workload depends on the situation. For example, the demands of 

using procedures can fall when the delays associated with waiting for a response from a board operator are 

eliminated. However, when the operators need to access information in parallel, the CBP system can increase load. 

Task Analysis 

Niwa, Hollnagel, and Green (1996) evaluated CBP systems as part of a CBP development study for the Institute of 

Nuclear Safety Systems in Japan. They identified several reasons for their slow development. First, there has been 

a tendency to keep procedures separate from the HSI so they constitute a "fallback for when all else fails." Second, 

procedures are not easily automated because they contain imprecise elements and depend on information about 

conditions that are not easily instrumented. 

Niwa et al. (1996) made a subjective comparison of the attentional demands of PBPs and CBPs based on task 

analysis. The basic tasks for using EOPs involve identifying which EOP to use, proceeding step-by-step through 

the procedure, carrying out actions specified in steps, checking-off completed steps, and retrieving additional 

information (from other documentation). The results are summarized in Table 5.2 (adapted from Table 1 of Niwa 

et al., 1996). CBP ratings were based on a "well human-factored solution" although this was not clearly defined. 

In general, they determined that interactions with a CBP are more complicated than interactions with a PBP. 

CBPs may increase attentional demands in selecting the required display (turning procedure pages) and check

marking step completions. However, other aspects of procedure use are easier with CBPs, such as retrieving data. 

Table 5.2 Attentional Demands of PBP and CBP Systems' 

[Activity PBP CBP 

Go through steps medium medium or high 

Turn pages (select display) very small small or medium 

Check-mark completion very small small or medium 

Retrieve additional information very high small or medium 

Access required source medium or small small 

Find information medium small or medium 

'Information is based on the findings of Niwa et al. (1996) 

In summary, these evaluations show no clear advantage of CBPs over PBPs. Instead, they illustrate the importance 

of performance tradeoffs in assessing different procedure systems. In general, cognitive load, complexity, and 

attentional demands were higher for CBPs, while data retrieval was easier and task-completion time was less. 

5.4.2.2 Risk-Informed Analyses 

There have been several risk-informed analyses of CBPs. In one qualitative study of the anticipated impacts on 

human performance introducing digital technology in NPP designs, Wilhelmsen et al. (1992) identified several 

concerns. By impacts, the study referred to potential changes in generic failure rates associated with a crew's 

performance in traditional systems. One of the "most pressing issues" identified was the availability of on-line

5 -33 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

procedures. The study also indicated that these systems might improve performance if they list procedure steps, 

logic flow, and allow simultaneous access to multiple procedures. 

Two other studies examined different aspects of performance: one evaluated whether CBPs represent a potential 

safety-significant issue (Stubler et al. 1996), and the second examined their potential effects on components of 

human error (Orvis and Spurgin, 1996). 

Stubler et al. based their safety evaluation methodology on an adaptation of EPRI's approach in Guideline on 

Licensing Digital Upgrades (EPRI, 1993b), which was endorsed by the NRC in Generic Letter 95-02 (NRC, 1995). 

The following aspects of CBPs were associated with potentially negative effects on human performance: 

* Level of automation - The appropriate level of automation of CBP systems for managing information is not 

well understood. 

Design errors - CBPs that assess plant conditions and then present corresponding procedure steps may have 

design errors that stem from the system designer's incomplete understanding of the plant's behavior. These 

errors may result in inappropriate analysis of information or incorrect guidance to the operator. 

Situation awareness - Because only a portion of the procedure can be observed at one time, operators may lose 

a sense of where they are within the total set of active procedures. The display space may be inadequate to 

allow simultaneous viewing of multiple procedures and associated plant data. 

* Overreliance on CBP information - PBPs require operators to monitor plant indications. If these are present in 

the CBP, the operator may not feel the need to look at other sources of information in the CR and, thus, may 

miss important indications that are lacking in the CBP. 

* Navigation - Navigation within one procedure, or among multiple ones and related supporting information, 

can be time consuming and error prone. 

Computer-based text characteristics - In general, comparisons of task performance for information presented 

via VDUs and hard copy indicate that there are significant differences between them. Reading is generally 

slower and more fatiguing using VDUs, and they have been associated with poorer performance and lower 

usability ratings. 

Salience coding - Presentation formats, such as text, flowcharts, and hypertext, can be visually enhanced by 

the graphical capabilities of computer-based displays. For example, color, flashing, and animation can 

enhance the salience of important information. However, improper coding can have negative effects on 

operators' behavior by de-emphasizing or drawing attention away from important information. Thus, the 

design of coding schemes is critical for successfully implementing CBPs. 

Consistency with the HSI - Any inconsistency of the CBP with the rest of the HSI can lower performance and 

increase the likelihood of errors. Some important aspects include the degree to which the display of plant 

variables and units of measurement used in the CBP are the same as in the normal monitoring displays, the 

same coding schemes are used, and navigation mechanisms are compatible with those of other display devices 

in the HSI. 

Transfer between CBPs and PBPs - Under some circumstances, the operators may be required to switch 

between PBPs and CBPs. For example, if the normal operating procedures are given in paper-based form and

NUREG/CR-6634 5 -34
5 TECHNICAL BASIS DEVELOPMENT

the EOPs in computer-based form, then the operator must switch to CBPs when the EOPs are to be used. If 

the CBP system fails, then the operators may be required to use PBPs. Transfers between them may be 

difficult, especially if their formats (e.g., flowchart versus text format) and mechanisms for managing the use 

of multiple procedures (e.g., "place holders" and navigation features) differ. The operator's burden is likely to 

increase with such switching when other demands are high (e.g., as a result of the condition that required 

using EOPs). 

Orvis and Spurgin (1996) evaluated CBPs from a perspective of a cognitive reliability model. We note the analysis 

assumed that CBPs had positive effects on performance and, therefore, it was aimed at where improvements in 

crew reliability can be expected. For example, Moieni and Spurgin (1993b) have noted that 

...computers can make up for some of the human limitations, such as short term memory and limited 

working memory capacity, and together with the human operator can be more effective and reliable 

than either acting separately. Thus, computers can help the user find his way through the procedures 

and help ensure that steps in the procedures are taken in the correct sequence. More importantly, 

they can support the crew in taking into account the correct set of symptoms, and help ensure that 

key elements are not ignored. In some systems, the computer can take control if the crew fails to 

follow the procedures as prescribed. 

The cognitive model had two separate phases: the detection-diagnosis-decision making (DDD) phase, and the 

implementation phase. There are three pathways to failure to provide the correct response within the required . 

time. First, the crew may fail to detect the need to take action, or may make a misdiagnosis (P,). Second is the 

failure to complete the DDD phase within the required time (P2). The third path is the crew's failure to complete 

all required actions (P,). The total probability of human error for a given human interaction is 

PHumTot ý P1 + P2 + P 3 

Orvis and Spurgin felt that the CBP should reduce the probability of all failure pathways. Since the CBP 

automatically detects parameters and matches them to EOP conditions, P, could be essentially eliminated. When 

CBPs monitor whether an action was taken and notify the crew if it is still needed, P3 can be essentially eliminated 

as well. Thus, 

PHum.Tot (CBP) < Hum.,Tot (PBP) 

Orvis and Spurgin (1996) have determined that the following features of CBPs will affect crew reliability: 

* Quality of display 

* Number of windows concurrently open 

* Coupling with plant parameters 

* Coupling with alarms 

* Display of control status 

Display of plant mimics with component status

5 -35 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

* Automatic EOP selection 

* Easy navigation 

* Similarity of operation for normal and abnormal procedure use 

* Automatic place keeping in EOPs 

* Limited amount of user configuration 

• No lockup on erroneous use 

The analysis by Orvis and Spurgin was to be an assessment of the potential benefits of CBPs. Potentially negative 

factors, such as those examined by Stubler, Higgins, and O'Hara (1996), were not examined. They are careful to 

point out that improvements in reliability have to be made using the results from simulations. 

While the evaluations of performance models showed no clear advantage of CBPs over PBPs, the risk-oriented 

analyses show that while CBPs have the potential to increase the reliability of human performance, when poorly 

implemented, they can reduce it. 

5.4.3 Expert Opinion 

This section reviews literature that discusses CBPs where the findings are based on the opinion of subject-matter 

experts (SMEs), rather than specific data collection or analyses. It includes the NRC-sponsored review of CBP 

systems by Spurgin et al., the NRC CBP workshop, and an IAEA working group on computerization of CRs that 

covered CBPs. 

NRC-Sponsored Review of CBP Systems 

Spurgin, Wachtel, and Moieni (1993) reviewed several CBP systems based on a literature review, a questionnaire, 

and interviews. Their findings indicated that CBPs have important impacts on NPP operations, some of which 

extend beyond those intended by the designers. The change from PBPs to CBPs may affect the crew's structure, 

human reliability, training, and selection criteria. They identified the following general findings: 

* CBPs can perform many tasks typically undertaken by multiple crew members; these include monitoring 

functions, selecting a procedure, selecting procedure steps based on the plant's state, and providing the 

rationale for the choices. Thus, CBP use will require a single crew member. 

* EOPs are used differently in different countries; therefore, their design will reflect these differences. 

* Recovery from human error is faster with CBPs than with PBPs. 

* CBPs appear to be beneficial during multiple failures. 

* CBPs may introduce new types of errors related to software errors and those due to the designer's incomplete 

understanding of the plant. 

Thorough V&V is a crucial requirement.

NUREG/CR-6634 5 -36
5 TECHNICAL BASIS DEVELOPMENT

The extent of system automation and allocation of functions between the CBP and crew are important, but may 

not be sufficiently considered by designers. 

• CBPs will significantly affect the administrative control and configuration control of plant procedures. 

The way in which CBPs are introduced into plants and the training that operators receive are very important. 

For example, it is unknown whether CBPs should be introduced into training simulators first, or whether they 

should be given to novice crews only. 

It was concluded that "... more work needs to be done before the industry can make a safe transition from 

traditional paper and pencil procedures to computerized systems" (p. 1017). 

Spurgin (1995) discussed the effect of computerizing EOPs on the operator's role. Two classes of CBPs were 

identified. The first presents EOP information to the operators, and they decide on the subsequent actions. The 

second class recommends how to proceed, and the operator confirms. The latter minimizes the operator's role and, 

according to Spurgin, may be a deterrent to taking action. 

NRC CBP Workshop 3 

The NRC conducted a major workshop on CBPs in 1994 to identify the key issues that need to be resolved to 

support HFE guidance for reviewing CBPs. Fifteen participants were identified as SMEs by an international 

selection process. Individual presentations on the current status of CBP systems development and research were 

discussed. The SMEs were divided into two working groups to identify the issues. The results are summarized 

below. 

CBP Taxonomy 

A taxonomy to describe CBP features and functions is needed to support regulatory evaluations and regulators in 

exploring differences in systems. 

Automation and Task Allocation 

While it is important for operators to be in control of the CBP, they may become more complacent or dependent on 

the CBP and fail to consider whether it is malfunctioning. The extent of the crew's monitoring should be 

considered. The CBP should inform operators of disagreements, but should allow them to take actions if they want 

to override. The crew's actions should be logged. 

The degree to which specific CBP features should by automated is unknown and should be assessed. 

Crew Performance and Coordination 

The impact of CBPs on task performance of crew members needs careful assessment; CBPs should support 

cooperation, interaction, and decision making. Beyond these generalities, several specific issues were identified: 

3

The findings from the NRC CBP Workshop are being documented in a report that is currently in draft form (Kancler, 

D., Schopper, A., and Wachtel, J. Findings of a workshop on computer-based procedures in nuclear power plant control 

rooms. Washington, D.C.: U.S. Nuclear Regulatory Commission.) For the purposes of our research, any findings from the 

workshop that contributed to CBP design review guidance development are included in the summary provided in this section.

5 -37 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

An operator's situation awareness should not be adversely affected by CBPs. Two aspects are important: 

awareness of the plant's status (information should be available, and interacting with CBPs should not 

interfere with situation assessment), and procedural awareness (awareness of procedure goals, how they are 

structured, and knowledge of the location within a procedure or between a set). Situation awareness should be 

measured in evaluating the systems. 

• High workload may be a concern. 

* Use of CBPs should be consistent with normal, daily operations to be effective. 

• CBPs should support cross-checking. 

Training 

The SMEs noted that training was critical to the success of CBPs, but that significant changes may be required. 

VDUs may introduce effects such as glare and eyestrain. They also offer features not found in PBPs. Training 

should address the procedures' structure, conventions, and rules of use. 

Training also should focus on limitations of the system and establishing the operators' trust. However, operators 

should be trained to minimize overdependence or reliance on CBPs, and also on the proper means of handling 

disagreements between the crew and the CBP system about appropriate actions. In addition, operators need to be 

trained to detect failures and on both CBPs and their backup systems. The evaluation of training should ensure 

that any effects of negative transfer are minimized. 

Human-System Interface 

Several aspects of HSI design were discussed. CBPs should represent procedure attributes, such as steps of 

continuous applicability and transitions, and should provide navigational means to access different parts of the 

procedure, different procedures, and additional information. Another issue identified was the appropriate 

application of the computer's capabilities (color coding, animated graphics, and video) to procedures. 

Desirable characteristics of CBPs included the capability to adjust the level of detail and to annotate the CBP. The 

degree of flexibility in the CBP HSI was identified as a concern; such flexibility should not affect procedure 

information. Consistency with other HSIs also was identified as an important requirement. 

An issue was raised as to whether the computer medium may affect how operators interpret procedures. For 

example, some procedural details intentionally are left abstract. While computerization may allow an increase in 

detail, the result could be an inadvertent change in the procedure's context and its interpretation. 

CBPs should be able to support improvements in procedures. Gaps occur in procedures because they do not cover 

all possible situations and actions; CBPs allow operators to log their occurrence in an on-line database. The 

database could be accessed to identify aspects of the procedure that need improvement. 

System Reliability 

Failure modes (bugs, logic failures, and bad input) need careful evaluation along with how operators can detect 

those that were not corrected during design.

NUREG/CR-6634 5 -38
5 TECHNICAL BASIS DEVELOPMENT

Verification and Validation 

CBP V&V evaluations should provide evidence that operators can perform their tasks in real time. They should 

involve procedure guidance (such as NUREG-0899), usability testing, findings of prior research, and realistic 

scenarios. V&V should address CBP failure, the transition between PBPs and CBPs, the introduction of CBPs into 

a PBP CR, and using PBPs if the CBP fails. Finally, V&V for software should be clarified. 

IAEA Working Group on CR Computerization 

Similar to Spurgin (1995), the IAEA working group on CR computerization concluded that CBPs potentially can 

minimize the operator's role but may deter people from taking action (IAEA, 1995). They considered CBPs to be a 

future trend. Some of the advantages noted were that (1) information will be integrated, (2) events will be 

confirmed (e.g., CBP can indicate if a procedure step is satisfied), and (3) information will be context sensitive 

(procedures can inform operators based on the current state, e.g., they will not display the step, "turn on Pump A," 

if Pump A is on already). CBPs should guide systematic, rapid implementation of procedures. However, the IAEA 

believed that current usage of CBPs is "minimal and in its infancy" and, therefore, recommended that CBPs 

"should be developed as research projects and prototypes, and feasibility tested on suitable full-scale application 

where these may be possible" (p. 62). 

Summary 

The SME examination of CBPs identified many positive aspects of CBPs' use on crew performance. However, 

SMEs also identified a wide range of unresolved issues which partly have led to the conclusion that CBPs should 

be introduced carefully into operational plants. The issues identified should be considered in developing CBPs. A 

noteworthy observation is that CBP development must consider related HFE activities, such as training, and 

integrating the CBP system with the other HSIs and the overall operational philosophy of the plant. V&V 

programs are again emphasized. In general, these findings are fully consistent with the other sources of 

information discussed earlier. 

5.5 Other Related Research on Computerization of Task Aids 

Two areas of research will be discussed that provide insights generalizable to procedures, while not specifically 

addressing their computerization. The first research includes general comparisons of task performance using 

computerized versions of what historically were hard-copy support materials. The second area is the topic of 

computerized operator support systems (COSSs). 

5.5.1 General Comparisons of Hard-Copy and Computer-Based Task Performance 

In this section, we briefly review the general research literature comparing hard-copy and computer-based 

presentations of the same information. While there have been many such comparisons for reading, there have been 

few studies of the effects on job performance (of which reading is only a component), such as maintaining 

equipment. 

Reading is an important part of any task in which information is presented to users on a computer, and an 

important aspect of procedure use. Therefore, a great deal of research was devoted to comparing reading 

performance on computers vs. typical hard copies. Generally, reading is slower and more fatiguing with VDUs 

than with a hard copy of the same material (Gould et al.,1987).

5 -39 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Examining the computerization of technical manuals, Shneiderman (1987) identified several potential 

disadvantages: 

* Computer screens are not as readable as printed material. 

* Computer screens provide less information than paper, and also the rate of paging is slower. 

• The need to use a computer interaction technique, such as command language, and navigation, requires more 

mental effort and may interfere with the primary tasks. 

If the display screen is used for other work, users may have to switch back and forth between the computerized 

manual and other information. 

To examine these potential issues, Shneiderman reviewed several studies of computerized manuals; he concluded 

that performance may not improve and may actually degrade. Thus, Shneiderman concluded that "At this stage of 

technology, paper manuals are still preferred" (p. 382). However, this conclusion was based on 1980's data. 

We reviewed several more recent studies which compared task performance using computer-based and hard-copy aids. Consistent with Shneiderman's conclusions, they generally found that computer-based presentations are 

associated with slower, poorer task performance (e.g., Reaux and Williges, 1988; Fox, 1992) and the use of 

different task strategies (e.g., Ogawa and Yonemura, 1992). 

Nelson and Smith (1990) set up repair manuals, including text and graphics for mining equipment, in HyperCard 

on an Apple Macintosh computer. Subjects performed tasks using either the computer-based or the hard-copy 

manuals. The first task required subjects to complete written statements by searching the manual and filling in the 

exact information that was found in it. The second task required subjects to answer multiple choice or true-false 

questions on nine realistic maintenance situations. They then were asked about their personal preferences to assess 

their acceptance of the modes of information presentation, and how it compared to other manuals they had used. 

Subjects using the computer-based manuals were significantly slower and finished fewer of the tasks, but 

performed much more accurately on the parts they completed. While those using the on-line manual considered it 

harder to use, subjective evaluations were positive ("quick response, good illustrations, compact; fun to use, finding 

general subjects area and Word Find are very helpful, and no greasy, dirty,, tom pages"). The problems identified 

included annoyance due to its brightness, eyestrain, and headaches. 

Federico (1991) tested Navy subjects' identification of Soviet and U.S. planes using either computer-based or hard

copy presentations. The two modes did not differ in accuracy or internal consistency; however, the subjects' 

confidence in their recognition was greater when using the hard-copy presentation. As the experimenters 

hypothesized, "...the longer exposures intrinsic to the paper-based method seemed to have facilitated subjects' 

recognition scores. They performed significantly better on the paper-based test than the computer-based test." 

Krauss, Middendorf, and Willits (1991) compared one group of subjects who learned to use a software product 

through an on-line tutorial with another group learning the same tasks using a hard-copy tutorial. Subjects were 

given a sample application task and a main application task. In the first, they were led explicitly through the 

actions necessary to complete the task in a cookbook fashion. Immediately afterwards, they began the main task, 

which required them to develop an application that allowed entry of information about employees (such as names, 

social security numbers, and job classifications). To accomplish this task, they had to specify tables, define 

records, and create screens, and were expected to refer back to their respective tutorials. Subjects working on-line 

were slower and found it more difficult than did those with the hard copy. This was due to navigational problems

NUREG/CR-6634 5 -40
5 TECHNICAL BASIS DEVELOPMENT

associated with their confusion with manipulating windows and finding information on hidden screens; they 

reported a "lost" feeling. The authors hypothesized that providing an outline of the entire document (e.g., in the 

comer of the screen) and highlighting the user's location in it might mitigate this problem. 

Weldon, Koved, and Shneiderman (1985) compared two types of information structure: linear (usually found in 

books) and tree (browsing through specific titles and finding details elsewhere). Subjects read from four different 

versions (one for each of the experimental conditions: online-linear, online-tree, hard copy-linear, and hard copy

tree) of a simulated electronic intercom-maintenance manual, written for the experiment. Each version was 

identical in content, but organized differently. The subjects were asked to determine the correct settings for two 

sets of eight dip-switches soldered to a prototyping card. The problems required different combinations of on and 

off switch settings. The dependent variables were the time to solve the problems, the number of errors, the number 

of pages viewed, and the subjective evaluations. It was found that the information's structure did not affect 

performance. Instead, the important variable was whether the subject had read from the on-line manual or from 

the hard-copy manual; subjects using hard copies were faster. Within the online condition itself, there was a 

significant difference in the number of pages viewed; subjects given tree-structured information looked at more 

pages than people given linear information. There were no significant differences in the number of errors in 

switch-setting combinations among the experimental conditions. In the subjective evaluations, subjects preferred 

the on-line mode over the paper mode, but there were no significant differences in type of information structure 

preferred. The experimenters hypothesized that structure may be more important in studies which used larger 

manuals, but this suggestion was not tested. 

Kincaid, Schurman, and Hays (1990) compared a paper maintenance manual with a computer-based manual, the 

Portable Electronic Aid for Maintenance (PEAM), observing technicians in a tank-maintenance task. The results 

indicated that use of the electronic system resulted in only about 1/3 of the errors of the paper manuals; however, 

the time to perform the task was slightly longer, which was attributed to computer delays in presenting 

information. Based on the PEAM results, Inaba (1990) identified several lessons learned: 

• HFE principles for paper presentations also apply to electronic presentations. 

* The major advantage of the electronic manual was its ability to display, store, and retrieve large amounts of 

data. 

* The major hardware limitation of a restricted screen area can be overcome by applying HFE principles. 

One study found a positive effect of computerization. Andre and Pouraghabagher (1995) compared computer 

checklists to paper checklists for missile Launch Control Center tasks. The computer-based formats reduced the 

response time of expert operators by 10 percent, and substantially reduced their error rates (by 58 percent) 

compared to the paper-based system. The effects were not as marked with non-experts. 

Summary 

It can be concluded from the general literature that task performance differs when information is computer based 

versus when it is presented as hard copy. Reading from a VDU is generally slower and more fatiguing. VDU

based complex task performance also was associated with poorer performance and problems in usability. 

Contributing to these differences are difficulties in maintaining a sense of location (knowing where you are in a 

document), navigation (moving from one place in a document to another), and fatigue. Chignell and Zuberec 

(1993) noted similar potential difficulties with use of CBPs: visual fatigue, glare, and resolution.

5 -41 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

5.5.2 Computerized Operator Support Systems (COSSs) 

Numerous COSSs based upon knowledge-based systems (KBSs), such as expert systems, assist in cognitive tasks 

such as evaluating plant conditions, diagnosing faults, and selecting response strategies. Intelligent aids may 

include (1) automatic checks which track operators' actions and compare them to actions expected from plant 

procedures or other models, (2) automatic warnings based on current conditions, predicted consequences or side 

effects, and (3) smart interlocks that block control actions that conflict with the plant's current configuration. 

The nuclear industry has developed a wide range of KBS applications for off-line analysis and on-line cognitive 

support to plant personnel (IAEA, 1993, 1995): 

* Fault detection and diagnosis 

* Safety function monitoring (e.g., severity of challenges to critical safety functions) 

* Plant-performance monitoring (e.g., efficiency of main pumps, turbine, and generator) 

* Core monitoring 

* Advising on unforeseen maintenance problems 

* Interpretation of complex procedures or regulations 

* Support for controlling the plant 

Several off-line systems have been applied to areas related to safety. The overall trend in the nuclear power 

industry appears to be a move from conventional off-line applications toward on-line systems. The principal area 

of application appears to be fault diagnosis, which requires a monitoring capability. A variety of computer-based 

aids that analyze plant conditions and then make recommendations to personnel (e.g., for improving plant 

performance, diagnosing failures, and identifying success paths) are discussed in the literature; many of them are 

research prototype systems. For example, expert systems, based on artificial neural network technology, were 

developed for the following NPP applications: diagnosing faults, analyzing core vibrations, monitoring loose parts, 

modeling thermodynamics, estimating thermal margins, and identifying transients (Uhrig, 1994). The commercial 

uses of computer-based aids include: 

• Emergency response projection code - Software for projecting doses that would be received by the areas 

surrounding a nuclear generating station in an accident involving airborne release of radioactive materials. 

[The Pickering Emergency Response Projection computer code is described in AECB (1994).] 

• Fuel loading expert system - A computer-assisted system for fuel reloading while at power was designed for 

CANDU NPPs (Gertman et al., 1994). 

AECL has several aids under development (O'Hara, Stubler, and Higgins, 1996): 

Advanced Process and Analysis Control System - This is a rule-based computer system that assists operators 

and maintenance personnel with on-line diagnosis of process and equipment faults. The prototype system was 

applied to the CANDU Bruce B feedwater system.

NUREG/CR-6634 5 -42
5 TECHNICAL BASIS DEVELOPMENT

* Feedwater Corrosion-Monitoring and Prediction Analysis System - This system supports the detection, 

monitoring, diagnosis, and prediction of corrosion problems in the secondary side of a CANDU plant, based 

on chemical analyses. Neural nets are used in the diagnosis portion of the system. 

• A signal-analysis system for calibrating trip channel signals. 

* A virtual-reality-based system for visualizing the interior of a reactor fuel channel to support the removal of 

stuck fuel bundles. This system is envisioned as a training aid. 

AECB described an operator decision aid currently under development in Canada that simulates plant performance 

using an ideal model. It continuously compares actual plant values to simulated values to identify plant systems 

that may be degrading or failing. 

Japan Atomic Power Company's Tsurugan NPP (Unit 2) includes a Mitsubishi Computerized Operator Support 

System (MCOSS). Its objective is to aid operator's decision making by detecting abnormal operating conditions 

before they become serious and to advise the operator of appropriate actions. If its early warning capability does 

not prevent a plant trip, the MCOSS assists the operator in reaching safe shutdown. The utility was concerned that 

measurement noise would impair the system's diagnostic capability; however, this was not the case. The system's 

response time is approximately 6 to 7 seconds, which was judged acceptable, although shorter times were 

preferred. 

This system will be further developed in the Mitsubishi Advanced PWR. Mitsubishi is developing a KBS for use 

during accident conditions and when operators are under high stress. The system develops hypotheses about plant 

conditions based upon the available symptoms, and then tests each one. It uses a windows interface with dialogue 

control at the bottom of the screen. The operator can request an explanation of the system's hypothesis, procedural 

guidance, and evaluation of alternative solutions before actions are taken. It is anticipated that use of the system in 

a real CR would be a full-time job, probably for a senior operator. 

Despite the development of many COSSs, there is not much experience with operational aspects of their use. 

Several experimental evaluations of the value of expert systems to operators were inconclusive (Bernard and 

Washio, 1989). Furthermore, there is a trend for expert systems to be abandoned after prototype testing and brief, 

in-plant trials. The transition from a prototype system to a production-grade product requires a significantly 

greater effort than initial prototyping (Cain and David, 1989). 

As discussed, the predominant role of COSSs has been as decision aids. In this role, perhaps the most significant 

factors are intelligibility and communication (IAEA, 1995; Malin et al., 1991a, 1991b; Land et al., 1995; Rook and 

Donnell, 1993). It is essential to the operators' acceptance and use of COSSs that the reasoning process is fully 

understood by those using them. Personnel must be able to communicate both ways with the KBS COSSs, i.e., the 

degree to which the bases for its results are given and to which operators can query the system when its results are 

not understood is important. 

Rook and Donnell (1993) experimentally manipulated the interface of an expert system which was designed to 

support fault diagnosis during simulated space station problem-solving situations to make its reasoning more or 

less intelligible to the subjects. They found that users of the system had to have a good mental model of its 

reasoning process to use it effectively. Since understanding the system was critical to its use, they predicted that 

the design of the display would be a significant factor in using the system.

5 -43 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

Malin et al. (1991a) discussed case studies of the design of 15 intelligent systems developed for a variety of 

aerospace projects. The systems mainly were real-time fault management systems. The interfaces between the 

human operator and intelligent systems were found to be problematic. Some specific concerns identified were 

quite similar to those identified above: providing visibility into the system's reasoning, understanding its 

reasoning, the system's response in the context in which a question is asked, distinguishing hypotheses from facts, 

determining the credibility and validity of information, handling interruptions, handling changes in planned 

activity sequences, distinguishing between modes of operation, gaining control over the system's actions, and 

identifying system errors. The systems also had many problems related to their general HFE design. Operators 

often did not get the information they needed; it was presented in confusing formats not well suited to their task 

requirements, and excessive detail was given. This made it difficult for operators to "visualize the intelligent 

system's situation assessment and recommendations in relation to the flow of events in the monitored process." 

Similar results were obtained by Dien and Montmayeul (1995), who surveyed operating experience with COSSs 

placed into existing CRs. They concluded that while much effort went into their design, the focus was on 

technology, and feedback showed that operations were not improved by their implementation. In many cases, the 

approach led to failure. COSSs often provide guidance for situations that operators already are equipped to handle. 

That is, they are designed for situations which were previously analyzed with which designers are familiar. Such 

aids are of little help to operators, except for confirmation. These systems poorly address unforseen circumstances 

and may not then provide appropriate guidance. 

Another problem observed was that aids were "acontextual." That is, their guidance had little reference to the 

current situation. Also, guidance was given without appropriately communicating what led to its issuance, what 

parameters were analyzed, or what sequence of reasoning was followed. When the reasoning process is shown, it 

may conflict with that of the operators, i.e., it may be based on the designer's theoretical understanding and not on 

the operator's practical experience. 

The new aids are often poorly integrated with other HSI systems, and their design characteristics, such as dialog 

principles and coding, are often different. Reed, Hogg, and Hallbert (1995) found that concerns about the interface 

design and system implementation limited the usefulness of a KBS system, the Process Operations and 

Management System, which was installed in a conventional British Nuclear Fuels plant to provide on-line early 

warning and fault diagnosis. These limitations led operators to prefer the conventional systems. 

Roth, Bennet, and Woods (1987) indicated that the interface provided to a KBS must enable a cooperative dialogue 

so the operator can better understand and utilize the system. In general, these aids tend to be technology driven 

and do not address the needs of the operator; that is, they are developed by finding an application for a given 

technology, rather than being designed to meet users' needs. An inadequate analysis of users' requirements 

usually leads to problems with information content. The system should provide accurate information that is 

needed, and not force extraneous material on the user. Expert systems should support the operators' cognitive 

processes and reinforce their existing approach to plant operation developed through training and experience. The 

KBS should not require operators to conform to the machine's method of analysis (Bernard and Washio, 1989). 

KBSs also have inherent limitations (Terry, 1989). They cannot reason broadly over a field of expertise, and are 

limited to narrow tasks. They cannot reason from axioms, analogies, or general theories. In addition, they lack 

common sense and often do not make simplifying assumptions. They are limited by their programming and cannot 

learn. Their performance deteriorates rapidly when applied to large problems. Such limitations of the expert 

system should be made obvious, so operators are not required to decide between their own judgment and the 

machine-generated advice (Bernard and Washio, 1989). The system's security should be controlled so that 

inappropriate changes cannot be made.

NUREG/CR-6634 5-44
5 TECHNICAL BASIS DEVELOPMENT

The IAEA (1994) identified several criteria that should be considered when licensing authorities evaluate COSSs: 

(1) Compatibility with the operations - To work effectively, operators may require more than occasional 

simulator training to become familiar enough with COSSs to use them. They may need day-to-day 

experience. 

The operator's effectiveness in using the COSS requires that the system is used not only in very 

specific conditions for which it was designed, but also in normal operation. For maximum 

compatibility with the global MMI, the data produced by the COSS must be integrated into the 

procedures used by the operators for normal operations, as well as in the specific abnormal or 

emergency conditions for which the COSS was designed. This may be an issue for CBP systems 

that are designed for emergency systems only. (p. 31). 

(2) Consistency with the HSI design - The detailed design of the COSS, e.g., labeling and dialog conventions, 

should be fully consistent with the rest of the HSI. This may be a particular problem for off-the-shelf 

systems. 

(3) Cognitive support - The COSS must enhance performance, or, at a minimum, not degrade it. 

(4) Team performance - The user of the COSS must be clearly specified. However, NPP control is a team 

operation. COSSs may change task allocation and the type and quantity of information to be 

communicated between crew members. These effects must be evaluated and it should be demonstrated 

that the team's performance is not degraded. 

Summary 

The following human performance concerns are associated with COSSs: 

(1) The design of computer-based systems commonly fails to account for user needs. This includes the need 

for information in the context of current tasks, goals, and objectives for operations, maintenance, crew 

configuration, and feedback from control actions. 

(2) COSSs and other computer-based systems add to the plant's complexity. Operators must have a good 

mental model or understanding of the computer-based system for monitoring, supervision, and 

maintenance of the plant. Failure to account for this leads to poor situation awareness and a sense of 

being out-of-the-loop. 

(3) COSSs often are not designed so that their logic is sufficiently observable. That is, they do not make clear 

their reasoning basis or enable operators to adequately query or otherwise verify system performance. 

(4) Compatibility with day-to-day operations needs to be considered. Systems with very limited use in normal 

operations that are only used in infrequent special circumstances may have little success. 

(5) Integration of a COSS with other CR HSIs is important. 

(6) Training and team performance are significant considerations when introducing new technology.

5 -45 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

5.6 Summary and Discussion 

This section summarizes and discusses the implications for CBPs of the material reviewed in Section 5. 

5.6.1 Supervisory Control and Procedure Use 

The role of the operating crew in an NPP is that of a supervisory controller that must engage in situation 

assessment, monitoring and detection, and response planning and implementation. These cognitive functions are 

applied to tasks for which the crew has primary manual responsibility as well as to automated systems, and systems 

which support the tasks. Procedures fall into the latter category. 

Historically, procedures were designed to support response planning by providing operators with strategies based 

on "off-line" detailed analyses of both normal and abnormal states. However, when these preplanned strategies are 

applied to the unique circumstances of a particular process disturbance, unforseen or unanticipated situations may 

render parts of a procedure inappropriate or ineffective. Thus, confronted with complex, real-world process 

disturbances, operators must monitor the performance of the procedure to verify its correspondence to the higher

level goals that it was designed to achieve. It is important for operators then to assess the effectiveness of the 

response plan even when described by established procedures, the consequences of particular actions, and the 

appropriateness of the path for achieving identified goals. This enables operators to detect when procedures are 

not achieving the goals, when procedures are erroneous, or when errors are made in carrying out procedure steps. 

Another cognitive activity is adapting response plans. Adapting plans to the current situation is necessary because 

steps may be vague and have to be interpreted by the operators, or their judgement is necessary to evaluate the 

procedure. In addition, procedures do not have all the information about the plant that the operators do. Operators 

must fill in the gaps in a procedure, modify it to fit the specific situation, and direct the procedure path. Thus, 

rather than assuming the role of rote, verbatim procedure followers, it is important that operators maintain the role 

of supervisory controllers and monitor the performance of the procedures as well as the process. They need to stay 

cognitively involved in the procedure's progress. Operators need to understand the intent of procedures, their 

overall strategies, assumptions and underlying principles, and the transition logic between procedures. They should question procedure steps that appear inconsistent with the overall goals of the procedure for the situation at 

hand. 

With the development of CBPs, procedure systems have the potential to support not only response planning, but 

also aspects of situation assessment, monitoring and detection, and response implementation. This support may be 

applied to the operators' primary tasks, such as monitoring parameters, and to secondary tasks, such as navigating 

from one portion of a procedure to another. 

5.6.2 The Effects of CBPs on Crew Performance 

There are limitations to PBPs that CBPs potentially can address: cognitive workload associated with process 

monitoring and analysis of the logic in procedure steps; attention required for assessment of procedure steps that 

are continuously applicable, time dependent, and process dependent; the need for varying levels of detail in 

procedure information; the lack of context sensitivity; management and place keeping in multiple procedures; and, 

sequence control and navigation. The limitations of PBPs have been associated with delayed task performance and 

human errors in existing plants.

NUREG/CR-6634 5 -46
5 TECHNICAL BASIS DEVELOPMENT

CBPs may address these issues; however, they must maintain acceptable performance on the main tasks for which 

procedures are used while not introducing unintended negative effects. The latter is an important consideration. 

Our general review of the literature indicated that comparisons of task performance for information presented 

either on a VDU or in hard copy revealed significant differences between them. Reading on a VDU is generally 

slower and more fatiguing. VDU-based task performance also is associated with slower and poorer performance 

and concerns about usability. In addition, different task strategies are used. Contributing to these differences are 

keyhole effects, difficulties associated with maintaining a sense of location (knowing where you are in a 

document), navigation (moving from one place in a document to another), and fatigue. Some of these same 

concerns were raised regarding computerization of NPP procedures (Chignell and Zuberec, 1993). 

In general, the computerization of other types of support systems, e.g., COSSs, has had limited success (Dien and 

Montmayeul, 1995; IAEA, 1994; Malin et al., 199 la; Roth, Bennet, and Woods, 1987). The problems included 

the failure to account for users' needs and therefore, incompatibility with day-to-day operations; added complexity 

of the HSI; obscurity of the reasoning basis; inadequate communication facilities preventing operators from asking 

questions; poor integration with other HSIs; and personnel concerns, such as training and team performance. 

Thus, while there are PBP deficiencies that may be resolved by computerization, it is essential to carefully examine 

the effects on personnel performance. We did this by reviewing three types of research: (1) empirical studies of 

CBPs where performance data were collected, (2) analyses of personnel performance using models, and (3) expert 

opinion on postulated effects on performance. Each is briefly summarized below. 

The human performance research was organized into three categories: comparisons of CBP and PBP systems, 

observations of operators' CBP use, and comparisons of design characteristics of procedures. Several conclusions 

were drawn about using CBPs compared to PBPs: 

* Operators may perform procedure tasks more quickly. 

* Operators' cognitive workload may be reduced. 

* Operators may make fewer errors in transitioning through procedures. 

* Operators may accept CBPs more readily and find them easier to use. 

However, there remain several important unresolved questions including (but not limited to) the following ones: 

* What is the effect of CBPs on team performance and reliability? 

* What is the effect of CBPs on operators' high-level situation awareness of the status of the overall procedure 

goal and the plant? 

* Do operators become overreliant and unquestioning of CBPs, or can they maintain independence and 

objectivity to evaluate the adequacy of the procedure to achieve its goals? 

• What is the effect of CBPs on performance in complex disturbances that may involve many procedures or 

branches? 

* How well do operators manage complex CBP failures, such as when multiple procedures are being used, many 

steps have been completed, and many steps of continuous applicability are being monitored?

5 -47 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

• What are the relative effects of specific design features on performance? 

These and other CBP issues are discussed in Section 5.6.3. 

Another problem with the human performance research reviewed was that many studies were not discussed in 

sufficient detail to evaluate their generalizability. Those studies that were sufficiently documented had potential 

methodological weaknesses which limited their conclusions and generalizability. 

Personnel performance also was analyzed using two classes of analytical techniques: performance models and risk 

models. CBPs were evaluated by a variety of performance analysis models including the GOMS model, 

MicroSaint Task Network Modeling, the MIDAS model, and classical task analysis. The performance models 

showed CBPs had no clear advantage over PBPs. Instead, they illustrated the importance of performance tradeoffs 

in assessing different procedure systems. In general, complexity and attentional demands were higher for CBPs 

while data retrieval was easier and task completion time was less. 

It is interesting that the use of performance models for evaluating procedures has had some success. Their 

continued development may focus the testing of specific design issues that must be addressed in CBP design; this 

view is consistent with the conclusions of a National Academy of Science (NAS) assessment of applying 

quantitative models of human performance to complex systems. The NAS (Baron et al., 1990) concluded that "In 

all, there are compelling reasons to believe that systematic human performance modeling efforts should be 

regularly advocated and used along with expert judgement and manned part- and full-task simulation, as a regular 

part of the design process for large-scale human-machine systems" (p. 86). 

Several risk-informed analyses of CBPs have been made, each looking at risk somewhat differently: examining the 

potential to change generic failure rates, the potential effects of CBPs on components of human error probabilities, 

and whether CBPs may represent a potentially safety significant issue. 

Like the studies with performance model analyses, the findings were mixed. They illustrated the potential for these 

systems to improve performance by supporting such procedure-related work as process monitoring, logic analysis, 

navigation, and place keeping. However, they indicated that poorly implemented CBPs can reduce human 

reliability. 

Finally, SME opinion on the postulated effects on personnel performance was reviewed, including an NRC

sponsored review of CBP systems, an NRC CBP workshop, and an IAEA working group on computerization of 

CRs that addressed CBPs. The SME review of CBPs identified many positive effects of their use on the crew's 

performance; however, a wide range of issues was identified to be resolved in developing CBPs. Also noteworthy 

was the observation that CBP development must consider related HFE activities, such as training, and integrating the CBP system with the other HSIs and with the plant's overall operational philosophy. V&V programs were 

emphasized. In general, these findings were consistent with those from other sources discussed earlier. 

Considering all the results, we concluded that CBPs have the potential to support operators' performance and there 

is evidence to support this claim. As the NRC indicated in its review of the URD, "...the development of 

electronically displayed procedures is a desirable goal for the overall integration of operator information needs" 

(NRC, 1994). 

However, there are also important issues to be considered both in research and in the development of individual 

systems. Thus, the advice of several researchers and developers of CBP systems is repeated: the development of

NUREG/CR-6634 5 -48
5 TECHNICAL BASIS DEVELOPMENT

CBP systems for operational use should proceed in a way that the benefits and drawbacks of CBP systems can be 

fully evaluated for each specific system. CBPs have important impacts on NPP operations, some of which extend 

beyond those intended by the designers (Spurgin, Wachtel, and Moieni, 1993). 

The following are some general considerations for near-term approaches to CBP systems: 

* Support cognitive functions which may be distracting and error prone, such as 

- process monitoring 

- logic analysis (cautiously, so as not to underspecify the analysis and undermine the operator's judgement) 

'Support procedure management, e.g., step completion, place keeping, transitioning between procedures 

Provide PBP backup systems and ensure the similarity of CBPs and PBPs to (1) ensure confidence in near

term CBP applications, (2) enable operating experience to be gained, (3) minimize impacts on function 

allocation, (4) reduce burdens in training operators to use both systems, and (5) ensure successful crew 

performance when using backups (minimize the potential for negative transfer or difficulties in performance 

arising from disuse). 

5.6.3 CBP Issues 

This section summarizes the human performance issues associated with CBPs identified from the literature review. 

These issues represent topics for which research is needed before developing additional guidance. From a 

regulatory review perspective, these issues may be addressed on a case-by-case basis, as part of the design process 

review discussed in Part 2 of this document. 

The issues are not mutually exclusive; they overlap and some are more general than others; some may be 

considered secondary to others. Interdependencies are unavoidable, as they all pertain to the interactions within an 

integrated human-machine system. 

Methodological and Criterion Requirements for Evaluating CBP Effects 

A more definitive conclusion about the value of CBP systems was hampered by the lack of operational experience 

with their use, and lack of quality experimental evaluations. The detailed methodological considerations validating 

complex human-machine systems and a conceptual approach to it were discussed. The methodology focused on 

(1) establishing the requirements for making a logical, defensible inference from validation tests to predicted 

integrated system performance under actual operating conditions, and (2) identifying aspects of validation 

methodology that are important to the inference process. The technical basis for inference in validation is based 

upon four general forms of validity: system representation, performance representation, test design, and statistical 

conclusion. 

The studies examined generally had not undertaken well-controlled, comprehensive evaluations that would supply 

valuable data to better understand the impact of CBP effects under a wide range of scenarios and complex 

situations, using varied personnel and system measures. Most of them had methodological weaknesses which 

limited their conclusiveness and generalizability. Thus, important questions remain (many are discussed in more 

detail below). A good comprehensive evaluation of CBPs and their effects on crew performance has yet to be 

made.

5 -49 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

One question that needs to be addressed from both research and regulatory review perspectives is, "What are the 

criteria for CBP acceptance?" While some authors specified that such systems should improve performance, others 

indicated that performance should not be degraded (implying that equivalent performance with PBPs and CBPs is 

acceptable). This is an extremely important distinction because of the impact on performance that would be 

necessary if CBPs were required to improve it. 

Role of Plant Personnel in Managing Procedures 

Procedures are guidance to operators for achieving high-level objectives. While they are correct most of the time, 

for analyzed situations, adaptation sometimes may be necessary. Thus, operators must remain as independent 

supervisors who manage procedure implementation and independently assess its appropriateness. Operators must 

understand the overall purpose of the procedures, stay cognitively involved with their progress, and question any 

steps inconsistent with the overall high-level safety goals. However, CBPs potentially might work against this 

independence, minimizing the operator's role. They may increase the tendency to follow procedures without a 

critical independent perspective, and may even deter the operator's action. Resolving these concerns affects both 

design and training. 

Thus, one pressing issue is how to design and review CBP systems that enable the operators to maintain this 

independent perspective, but at the same time, reduce the operator's workload, automate distracting and lower

level error-prone tasks, and monitor the crew's performance, especially when the crew and CBPs disagree. Equally 

important is how to train operators in handling this role while using CBPs. The knowledge required to manage a 

CBP system may differ from that required to handle PBPs. For example, the CBP system may use different 

analyses to resolve procedure logic steps than operators do. 

Team Performance 

Research showed that CBPs may significantly affect crew member's roles, teamwork, and communication. 

Teamwork is an important element of defense-in-depth. Operators work as a team to support situation assessment, 

error detection and recovery. These roles and communication may be changed more than anticipated. Since SROs 

using CBPs can handle a procedure almost completely on their own, communication between the SRO and ROs 

may be reduced (Roth and O'Hara, 1998). While this is not, in itself, good or bad, its impact on team performance 

needs assessment. Board operators identified the importance of communication in maintaining effective teamwork 

when the SRO is using a CBP and expressed a need to be'aware of the status of EOPs. Thus, the potential for 

isolating the CBP user from the other operators, and changing operators' roles and responsibilities may undermine 

team performance in emergencies. Such effects on team performance were noted for many aspects of computer

based HSI technology (Stubler and O'Hara, 1996). 

The function of supporting coordination of the crew's work centers on the need for operators to be aware of the 

activities of other crew members. The CR is the context within which personnel convey, directly and indirectly, 

their intentions and actions to others. Advanced CRs, especially those with individual workstations, may isolate 

operators, making an individual's information and control actions less visible to others, thus reducing team 

effectiveness. 

Salas et al. (1992) define a team as "...a distinguished set of two or more people who interact, dynamically, 

interdependently, and adaptively toward a common and valued goal/objective/mission, who have each been 

assigned specific roles or functions to perform" (p. 4). In a CR setting, operators must share information and 

coordinate their tasks to satisfy specific goals or mission requirements. This requires a common understanding of

NUREG/CR-6634 5 -50
5 TECHNICAL BASIS DEVELOPMENT

the status of the system and of each others' actions and intentions. Identification and resolution of errors, 

coordinated information exchange, and team reinforcement were identified as important to team performance 

(Oser et al., 1989). Successful teams actively located errors, questioned improper procedures, and monitored the 

status of others. In a study of ship navigation, Hutchins (1990) discussed team performance in terms of facilitating 

error checking by others, allowing others to assist, and supporting training in the work setting. 

Hutchins found that longstanding work environments with traditional technologies have characteristics that 

contribute to team performance: horizon of observation, openness of tools, and openness of interaction. However, 

when computer-based technologies are introduced, these positive characteristics may be compromised. 

Horizon of Observation - This refers to the portion of the team task that can be seen or heard by each 

individual. It results from the arrangement of the work environment (e.g., proximity of team members) and is 

influenced by the openness of tools and interactions. By making portions of a task more observable, team 

members can monitor errors of intent and implementation, and determine when assistance might be helpful. 

* Openness of Tools - This is the degree to which an observer is able to infer information about another's 

ongoing tasks through observation of a tool's use. Open tools show characteristics of the problem that give an 

observer the context for understanding what has been done and the possible implications. 

Openness of Interaction - This is the degree to which the interactions between team members provide an 

opportunity for others with relevant information to contribute. Openness of interaction depends on the type of 

communication (e.g., discussing actions or decisions in the presence of others) and the style of interaction 

(e.g., the extent to which unsolicited input is accepted). Openness of interaction is also influenced by 

characteristics of the work environment (e.g., openness of tools, horizon of observation) that allow other team 

members to see and hear the interaction. 

Conventional CR designs typically have a broad horizon of observation facilitating the observation of team 

activities. In addition, they may be "open tools" in the sense that an observer can infer information about control 

actions (e.g., which plant system was involved, which control was operated, and what action was taken) by 

observing the operator's location at a control panel and the action performed. Interactions may be considered 

"open" because most involve speech that can be heard from across the CR. 

Advanced HSI technologies, such as CBPs, may impair these good characteristics. For example, using an 

individual computer-based workstation with an individual view of the plant may reduce the horizon of observation 

because that view cannot be readily seen by others and may lead to less open styles of communication. Also, the 

openness of tools may be impaired by having methods of user-system interaction that convey less task-related 

information to observers. 

Situation Assessment, Response Planning, and Operator Error 

The effect of CBPs on the operator's situation awareness has not been carefully evaluated. Operators need to 

maintain several levels of situation awareness when using procedures, including assessment of: 

• procedure steps, how procedures are structured, one's location within a procedure or between a set of 

procedures, 

* the appropriateness of procedures to achieve high-level procedure goals, and

5-51 NUREG/CR-6634
5 TECHNICAL BASIS DEVELOPMENT

* the overall situation in the plant 

Some concern over lowered situation awareness with CBPs was noted (Roth and O'Hara, 1998). PBPs require 

operators to monitor plant indications. However, if CBPs are present, the operator may not feel the need to look at 

other sources of information and may miss important indications that are not present in the CBP. The situation 

awareness of other operators is affected as well. Spurgin et al. (1990) noted that SROs use CBPs as their primary 

way of following the overall plant condition rather than relying on information from crew members. 

Consequently, the other crew members expressed concern about being aware of the EOP status. 

Jeffroy and Charron (1997) described another aspect of situation awareness - the joint awareness of the operators 

and the CBP. Such combined awareness may be separated when operators depart from the recommendations of the 

CBP, creating a situation that makes it difficult for them to recognize the CBP's constraints. They may not 

understand how their actions affect the procedure's ability to analyze individual steps. 

Research is needed to clarify the effects of CBPs on these different levels of situation awareness, the crew's ability 

to detect errors, and adaptation of the response plan in the face of procedure failures. In addition, knowledge is 

needed on the effect of CBPs on the number and types of operator errors (especially where errors are not defined in 

terms of verbatim compliance). 

Two related issues affect situation awareness: complexity and level of abstraction. Research on COSSs 

emphasized that computerized support systems add to complexity. Operators need a good mental model or 

understanding of the computer-based system to properly monitor and supervise the CBP. Failure to account for 

this leads to poor situation awareness and a sense of being out-of-the-loop. 

Roth and O'Hara (1998) observed that too little information presented at each procedure step can cause operators to 

lose a sense of where they are, while too much may distract them. The level of abstraction at which the results of 

procedure steps are presented will affect the operators' situation assessment. 

Level of Automation of Procedure Functions 

The human performance issues associated with automation have been well documented; see O'Hara, Stubler, and 

Higgins (1996) for a discussion of general automation. Table 4.1 listed procedure-related functions in terms of 

several levels of automation. The choices of levels of automation and their implementation will impact operator's 

performance, situation awareness, workload, and errors. Blackman and Nelson (1988) found that when the 

procedures were selected automatically, operators' involvement was reduced; they reported that they thought less 

and acted as switch-turners. A better understanding is needed of the tradeoffs between automatic procedure 

functions and operators' involvement, independence, and supervisory control. 

One area of procedure automation is especially noteworthy. An important capability of CBP systems is the analysis 

of procedure step logic; that is, comparing actual parameter values to the reference value in procedures using the 

logical relationships described in the step. When the step logic or the data analysis required to evaluate the step 

logic is underspecified, both the procedure and the operator can misjudge the situation. Therefore, procedures, 

especially EOPs, must be carefully designed and evaluated to guard against such underspecification. Where the 

operator's judgement is involved, such analyses are better kept manual.

NUREG/CR-6634 5 -52
5 TECHNICAL BASIS DEVELOPMENT

Keyhole Effects and Use of Multiple CBPs 

Viewing information through the limited area provided by VDUs is referred to as the "keyhole effect" (Woods et 

al., 1990); its consequence is that, at any time, most information is hidden. Therefore, operators must know what 

information and controls are available in the computer system, where they are, and how to navigate and retrieve 

them. 

The keyhole effect was identified as a root cause of many challenges to performance (O'Hara, Stubler, and Nasta, 

1997). If the viewing area is insufficient for operators to perform their tasks, they may have to navigate repeatedly. 

A problem with the keyhole effect is that access to controls and displays tends to be serial, e.g., only a few controls 

can be accessed at once, in contrast to the parallel presentation of controls and displays in conventional CRs. The 

sheer burden of navigating and retrieving many displays can interfere with the operators' ability to obtain an 

overview of the plant's situation. If workload is already high, operators decide not to retrieve all the information 

they need so they can invest their mental resources in their current task. 

The issues may become significant for CBPs when operators are required to be in multiple procedures. Hoecker 

et al. (1994) and Hoecker and Roth (1996) noted that when the operators are required to obtain information in 

parallel, the CBP system can increase workload. This lack of parallel access is a limitation of the keyhole effect. 

Because only a portion of the procedure can be observed in the display's space at one time, operators may lose a 

sense of where they are within the total set of active procedures. 

CBP Failure in Complex Situations 

Ensuring the transfer from CBPs to PBPs was recognized as an important consideration in designing and 

evaluating CBPs. This transition may be easy when the procedure context is simple, such as when operators are in 

its first few steps. However, the transition may be quite complex if operators are deep into the procedures, or when 

there are multiple procedures open, many steps completed, many steps of continuously applicability, time

dependent steps, and parameter-dependent steps being monitored by the CBPs. How operators will manage 

failures in such complex situations is unknown. 

Hybrid Procedure Systems 

Some CBP systems computerize all plant procedures (e.g., in EdF N4 CBP) while others contain only the EOPs 

(e.g., EOPTS). The ability to use CBPs effectively when they are designed only for emergencies may be 

problematic. 

Several studies recommended that for COSSs to be effective, they must be well integrated into everyday operations. 

Further, operators may require more than occasional simulator training to become familiar enough with COSSs to 

use them, as the IAEA (1994) stated: 

Operator effectiveness in using the COSS requires that the system be used not only in very specific 

conditions for which it was designed but also in normal operation. For maximum compatibility with 

the global MMI, it is necessary to integrate the data produced by the COSS into the procedures used 

by the operators for normal operations, as well as in the specific abnormal or emergency conditions 

for which the COSS may have been designed. Note that this may be an issue for CBP systems that 

are designed for emergency systems only. (p. 31)

5 -53 NUREG/CR-6634
Others commented on the need for CBPs to be consistent with normal, daily operations (NRC CBP Workshop). 

While EOPs are not used in daily operations, their computerization and the use and functionality of the system may 

raise difficulties if the operators' interactions with them are unlike those with other systems in the CR. 

Specific CBP Design Features 

Most studies we reviewed did not discuss the relative effects of specific CBP design features on performance. They 

were overall system comparisons, e.g., CBP vs. PBP, not systematic evaluations of individual characteristics. In 

addition, concern over the generalization of PBP guidance to CBPs was expressed. 

Thus, traditional procedure formats may require modifications when implemented on a computer. Two primary 

formats are used for procedures: text and flowcharts (Section 4). While both are successful in paper form, 

Chignell and Zuberec (1993) questioned whether flowchart presentations are acceptable in computer media where 

the limited screen and need for scrolling may make them less effective. Similarly, reading extended text from 

VDUs was found to be visually fatiguing. The proper implementation of CBPs in text and flowchart formats may 

require more guidance than that available for the paper forms. The effects of HSI techniques (such as outline 

views, navigational aids, and highlighting) on text and flowchart use, requires exploration.

NUREG/CR-6634 5 -54
6 DEVELOPMENT OF GUIDANCE 

Section 5.6.2 identified an approach to near-term CBPs, based on a consideration of lessons learned and of the 

remaining CBP issues. General considerations for near-term approaches to CBP systems include (1) supporting 

cognitive functions, such as process monitoring and logic analysis; (2) supporting procedure management, such as 

step completion, place keeping, and transitioning between procedures; and (3) providing PBP backup systems and 

ensuring similarity of CBPs to them. The review reflects these considerations. 

As an emerging technology, the technical basis for CBP guidance is limited, and there remain unresolved issues 

that cannot currently be reviewed with HFE guidelines. Thus, CBP reviews will require review of the CBP design 

processes as well as the design implementation. The latter conclusion is fully consistent with the NUREG-0700 

approach to reviewing HSI technology for which HFE design review guidance is limited, and also with the URD 

approach to an unproven technology. The development of guidance, therefore, took two forms: HFE design 

review guidance (typically found in NUREG-0700), and design process guidance. Guidance for products and 

processes is described in Part 2 of this report. 

As part of the design process, CBP systems will have to be evaluated using simulations and comparisons to 

reference systems, i.e., PBPs. This will confirm their acceptability and will support the development of more 

detailed guidance for specific systems. In the staffs URD review, the NRC (1994) stated that the: 

... designer should consider the use of electronically displayed procedures early in the design process to resolve 

any issues concerning their development, operability, maintainability, and reliability. If electronically displayed 

procedures are determined to be an improvement over hard-copy procedures and the M-MIS designer has 

integrated electronically displayed procedures into the overall M-MIS design, they should be provided as part 

of the design. (p. 10.B-17) 

NUREG-07 11 gives the NRC's high-level design process criteria for reviewing overall HFE programmatic goals 

and objectives but not for detailed reviews of final HSI designs, such as displays, control, or procedures. Rather it 

cites NUREG-0700 for guidance on detailed reviews of plant HSIs. In Part 1 of NUREG-0700, Rev. 1, the design 

process is covered under the review of plant HSIs for which the general framework of NUREG-07 11 was used. 

However, Part 1 also addresses general aspects of reviewing HSI designs, i.e., it does not identify the specific 

considerations that may be important for an individual technology, such as CBPs. The only detailed HSI 

technology-specific guidelines are in Part 2 of NUREG-0700; they cover the detailed form and functional 

characteristics for HSI implementations, but not design process considerations. 

Both types of guidance are necessary for a design review of CBPs. That is, while there is sufficient technical basis 

to develop detailed design-implementation guidance for many characteristics of CBPs, as is typical in 

NUREG-0700, several limitations in the technical basis have been identified, and so issues remain for which 

typical NUREG-0700 guidelines could not be developed. However, until more guidance becomes available, these 

issues can be resolved for specific CBP systems. To support resolution of issues for specific systems, guidance for 

the CBP design process review was established. 

6.1 CBP Design Process Review Guidance 

Guidelines were formulated for the design process review to address important points raised in the literature, and 

to provide a place where CBP human performance issues could be explored during a design review on a case-by

case basis. The format of the guidelines corresponds to the NRC's general guidance in NUREG-07 11. They are 

organized into the following sections:

6-1 NUREG/CR-6634
6 GUIDANCE DEVELOPMENT

* HFE Program Management 

* Operating Experience Review 

* Functional and Task Analysis 

• Staffing 

* Human Reliability Analysis 

* Human-System Interface Design 

* Procedure Development 

- Scope 

- Bases 

- Technical Information 

- Maintenance 

* Training Program Development 

* Human Factors Verification and Validation 

These guidelines are discussed in Section 9, Part 2 of this report. 

6.2 CBP HFE Design Review Guidelines 

A draft set of guidelines was developed from the findings and source materials that we surveyed, along with the 

high-level design review principles from NUREG-0700, Rev. 1. These principles were developed from research 

and industrial experience on integrating personnel into complex systems. They reflect the important design goals 

of (1) maximizing primary task performance (i.e., process monitoring, decision making, and control), (2) 

minimizing secondary task demands unrelated to the primary task (e.g., the distraction of tasks such as configuring 

a workstation), and (3) minimizing human errors and making systems more tolerant of them. These principles are 

also set out in Appendix B. 

An example of these guidelines written in the standard format of NUREG-0700, Rev. 1 is presented below: 

10.2.2-2 Automatic Monitoring of Plant Parameters and Equipment Status 

The CBP should automatically provide accurate and valid information on the values of parameters and 

status of equipment, when they are available to the system. 

ADDITIONAL INFORMATION: It should be clear to operators what specific information is used as the 

source of these actual values and states. 

Discussion: Supporting cognitive functions, such as obtaining parameter values (monitoring) may reduce 

the demands on attentional resources and working memory and enable the operator to focus more on 

evaluating higher-level procedure goals. It may also help solve PBP issues. This capability was identified

NUREG/CR-6634 6-2
6 GUIDANCE DEVELOPMENT

as being beneficial to the crew's reliability (Orvis and Spurgin, 1996; Pirus and Chambon, 1997; Niwa et 

al., 1996). Further, presenting plant parameters and status in procedure steps is a URD requirement 

(EPRI, 1993a). This guideline is an application of the High-Level Design Review Principles of Situation 

Awareness and Cognitive Workload (see Appendix B). 

Each guideline has the following components: 

* Guideline Number - Within each section, individual guidelines are numbered consecutively. The number 

includes its section and subsection location, followed by a dash and its unique number. 

• Guideline Title - Each guideline has a brief, unique, descriptive title. 

* Review Criterion - Each guideline contains a statement of an HSI characteristic so that the reviewer may judge 

the HSI's acceptability. The criterion is not a requirement, and discrepant characteristics may be judged 

acceptable based on the procedures in the review process. 

• Additional Information - For many guidelines, there is additional information including clarifications, 

examples, exceptions, details on measurement, figures, and tables to support the reviewer's interpretation or 

application of the guideline. 

* Discussion - The discussion summarizes the technical basis of the guideline. It may identify the primary 

source documents, the technical literature, such as journal articles, or the general principles from which the 

guideline was derived. This section will be removed when the guidance is integrated into NUREG-0700, 

Rev. 2. 

In place of the Discussion will be a Source field: 

* Source - The source field identifies the NUREG or NUREG/CR (or other document) containing the technical 

basis and development methodology for the guideline. As is the standard practice for NUREG-0700, the source 

field will give a reference to this document. 

The guidelines, contained in Section 10, are organized into the following sections: 

* Procedure Representation 

- Identification 

- Basic Steps 

- Warnings, Cautions, Notes, and Reference Materials 

- Lists 

- Organization 

- Formatting and Screen Layout

6-3 NUREG/CR-6634
6 GUIDANCE DEVELOPMENT

* Procedure Functionality 

- Procedure Supervision and Control 

- Procedure Monitoring and Assessment 

- Monitoring Operator Actions 

- Planning and Implementation 

• Procedure Management and Support 

Path Monitoring 

- Navigation 

- Communication and Help 

* CBP Hardware 

* CBP Procedure Backup 

CBP Integration with Other HSI Components.

NUREG/CR-6634 6-4
7 SUMMARY

The objective of this study was to develop HFE review guidance for CBP systems based on a technically valid 

methodology. To support this objective, the following tasks were undertaken: 

* Development of a framework for characterizing key design features of CBP systems 

* Development of a technical basis using research and analyses on human performance relevant to CBPs 

* Development of HFE review guidelines for CBPs in a format that is consistent with NUREG-0700, Rev. I, and 

NUREG-0711 

* Identification of remaining CBP issues for which research was insufficient to support our development of NRC 

review guidance 

The status of each will be briefly addressed below. 

CBP System Characterization Framework 

For this study, CBP systems were narrowly defined to encompass computer systems that support procedure 

presentation and use. The focus was on the HFE aspects of CBPs, and not the I&C or software aspects (although 

the latter are important as well, and are described in other NRC regulatory and research programs). CBPs were 

characterized along the following the dimensions: 

* Representation of Procedure Elements 

* Procedure Functionality 

* Interface Management and Support 

* CBP Hardware 

* Backup Systems for Procedures 

* Integration of CBP System with the HSI 

Development of the Technical Basis 

The effects of CBPs on crew performance were determined by examining three types of research: (1) empirical 

studies of CBPs where data on personnel performance were collected, (2) analyses of personnel performance using 

models, and (3) expert opinion about their postulated effects on personnel performance. 

The human performance research was organized into three categories: comparisons of CBP and PBP systems, 

observations of operators' use of CBPs, and comparisons of design characteristics of procedures. Several 

conclusions were made from comparing CBPs with PBPs: 

* Operators perform tasks more quickly.

7-1 NUREG/CR-6634
7 SUMMARY

• Operators' overall cognitive workload is reduced. 

* Operators may make fewer errors in transitioning through procedures. 

* Operators may accept CBPs readily and find them easier to use. 

However, much of the human performance research had insufficient detail to evaluate its generalizability. Studies 

that were sufficiently documented had potential methodological weaknesses which limited their conclusiveness and 

generalization. 

Personnel performance was analyzed with two classes of techniques: performance models and risk models. The 

performance models showed no clear advantage of CBPs over PBPs. Instead, they illustrated the importance of 

performance tradeoffs in assessing different procedure systems. In general, complexity and attentional demands 

were higher, while data retrieval was easier and task completion time was less for CBPs. Similarly, mixed results 

were obtained from the risk analyses. They illustrated the potential for these systems to improve performance by 

supporting such procedure-related activities as process monitoring, logic analysis, navigation, and place keeping. 

However, when poorly implemented, CBPs can reduce human reliability. 

Finally, the SME review of CBPs identified many positive aspects of their use on the crew's performance. 

However, they also identified a wide range of issues to be resolved in developing CBPs. The review highlighted 

the importance of considering HFE activities in CBP development, e.g., the integration of the CBP system with the 

other HSIs and with the overall operational philosophy of the plant. Thorough V&V programs were also 

emphasized. In general, these findings were consistent with the information discussed earlier. 

When considering all the results, we concluded that there is evidence that CBPs can support and enhance operator 

performance. However, important issues remain to be addressed both in research and in the development of 

individual systems. Thus, we repeat the advice of researchers and developers: CBP systems should be developed in 

such a way that their benefits and drawbacks can be fully evaluated for each specific system. CBPs have important 

impacts on NPP operations, some of which extend beyond those the designers intended. 

Reflecting this approach, we offer some general considerations for near-term approaches to CBP systems: 

* Support cognitive functions that may be distracting and error prone, such as 

- process monitoring 

- logic analysis (cautiously so not to underspecify the analysis and undermine operator's judgement) 

* Support procedure management, e.g., step completion, place keeping, transitioning between procedures 

Provide PBP backup systems and ensure similarity of CBPs and PBPs in order to (1) ensure confidence in 

near-term CBP applications, (2) enable operating experience to be gained, (3) minimize the impact on 

function allocation, (4) ease the training burdens associated with both systems, and (5) ensure successful crew 

performance when transitions to and from backups are necessary (minimize the potential for negative transfer 

or difficulties in performance).

NUREG/CR-6634 7-2
7 SUMMARY

HFE Review Guidelines 

Guidance for the review of CBPs was developed to address the CBP design process and HFE design. Both types of 

guidance are needed for a design review. That is, while there was a sufficient technical basis to develop detailed 

guidance for design-implementation review, as is typical in NUREG-0700, several limitations in the technical basis 

were identified. Many issues (listed below) remain for which typical NUREG-0700 guidance could not be 

developed. Therefore, until the additional guidance is developed, these issues should be addressed for specific CBP 

systems using CBP design process guidance. 

CBP Issues 

As noted above, several human performance issues associated with CBPs were identified. They represent topics for 

which research is necessary before developing guidance. From a regulatory review perspective, many of them can 

be dealt with on a case-by-case basis during the design process review. Briefly, the issues included the following: 

Methodological and Criterion Requirements for Evaluating CBP Effects - Most of the studies reviewed had 

methodological weaknesses which limited their conclusiveness and generalizability. This issue addresses the need 

to evaluate CBPs and their effects on crew performance comprehensively, to better understand them under a wide 

range of scenarios and complex situations, using varied measures of personnel and system performance. 

Role of Plant Personnel in Procedure Management - This issue addresses the need to determine how to design and 

review CBP systems (1) to allow operators to maintain an independent perspective and to recognize the 

procedure's contribution to higher-level safety goals, (2) to automate distracting and lower-level error-prone tasks, 

and (3) to monitor the crew's performance, especially when the crew and CBPs disagree. 

Team Performance - This issue addresses the requirement to explore the effect of CBPs on crew member's roles, 

teamwork, and communication. How CBPs can be designed to effectively promote both is considered as well. 

Situation Awareness, Response Planning, and Operator Error - This issue addresses the need to assess the effect of 

CBPs on situation awareness including: 

"• procedure management, such as status of procedure steps, how procedures are structured, and the current 

location within a procedure or between a set of procedures, 

"* the appropriateness of procedures for achieving high-level procedure goals, and 

"* the plant's status. 

Level ofAutomation of Procedure Functions - This issue addresses the need to evaluate the tradeoffs between 

automating procedure functions, e.g., the analysis of procedure step logic, and the operator's involvement, 

independence, and supervisory control. 

Keyhole Effects and Use of Multiple CBP Procedures - This issue concerns the requirement to evaluate the 

significance of the keyhole effect in situations where operators are required to be in multiple procedures and must 

access information in parallel.

7-3 NUREG/CR-6634
7 SUMMARY

CBP Failure in Complex Situations - This issue involves the need to evaluate operator's management of the 

transition from CBPs to PBPs and back to CBPs under complex conditions, e.g., in a situation where operators are 

deep into the procedures, multiple procedures are open, many steps are completed, many are continuously 

applicable, and time and parameter steps are being monitored by the CBPs. 

Hybrid Procedure Systems - This issue addresses the need to evaluate any differential effects of having all plant 

procedures presented in a CBP system versus a hybrid system, e.g., EOPs presented using CBPs and all other 

procedures are paper-based. 

Specific CBP Design Features - This issue addresses the need to evaluate the relative effects of specific CBP design 

features on performance.

NUREG/CR-6634 7-4
8 REFERENCES 

AECB (1994). Errors in software for emergency offsite dose estimation (Significant Event Report 9492). Ottawa, 

Canada: Atomic Energy Control Board. 

American Nuclear Society (1981). ANSI/ANS 18.7. LaGrange Park, IL: American Nuclear Society. 

Andre, A. and Pouraghabagher, C. (1995). Evaluation of computer-based progress indicators in the missile launch 

control center. In Proceedings of the Human Factors Society 39th Annual Meeting. Santa Monica, CA: Human 

Factors Society. 

Bainbridge, L. (1974). Analysis of verbal protocol from a process control task. In E. Edwards and F. Lees (Eds.), 

The human operator in process control. London: Taylor and Francis. 

Bainbridge, L. (1986). What should a good model of the nuclear power plant operator contain? In Proceedings of 

the International Topical Meeting on Advances in Human Factors in Nuclear Power Systems. LaGrange Park, IL: 

American Nuclear Society. 

Barnes, V., Moore, C., Wieringa, D., Isakson, C., Kono, B., and Gruel, R. (1989). Techniques for preparing 

flowchart format emergency operating procedures (NUREG/CR-5228, Volumes I and 2). Washington, DC: U.S. 

Nuclear Regulatory Commission. 

Barnes, V., Desmond, P., Moore, C., and O'Hara, J. (1996). Preliminary review criteria for evaluating computer

based procedures (BNL Technical Report E2090-T4-2-9/96). Upton, NY: Brookhaven National Laboratory. 

Barnes, V. and Radford, L. (1987). Evaluation of nuclear power plant operating procedure classification and 

interfaces: Problems and techniques for improvement (NUREG/CR-4613). Washington, DC: U.S. Nuclear 

Regulatory Commission. 

Baron, S., Kruser, D., and Huey, B. (Eds.) (1990). Quantitative modeling of human performance in complex, 

dynamic systems. Washington, DC: National Academy Press. 

Bernard, J. and Washio, T. (1989). The utilization of expert systems within the nuclear industry. In 1989 

American Control Conference. American Automatic Control Council. 

Blackman, H. and Nelson, W. (1988). Unexpected effects of computer presented procedures. In 1988 Fourth 

Conference on Human Factors and Power Plants. New York: Institute of Electrical and Electronics Engineers. 

Blakey, A. (1992). Object database technology: What is it, what are its advantages, and who is using it? CALS 

Journal, 1 (2), 66-70. 

Bozec, J., Dien, Y., LaMarre, J., and Meauzoone, L. (1990). Operations in all situations with on CRTprocedures: 

Main results of the tests performed in 1989 on the $3C computerized control room simulator for 1400 mw (N4) 

PWR plants (Tech. Report HT-54/90-57A). Paris: Electricite de France. 

Cain, and David, G. (1989). Artificial intelligence applications in accident management. Nuclear Engineering 

and Design, 113,251-257.

8-1 NUREG/CR-6634
8 REFERENCES

Card, S., Moran, T., and Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: 

Erlbaum. 

Carter, R. and Uhrig, R. (1990). Human factors issues associated with advanced instrumentation and controls 

technologies in nuclear power plants (NUREG/CR-5439). Washington, DC: U.S. Nuclear Regulatory 

Commission. 

Chignell, M. and Zuberec, S. (1993). Computerization of operating procedures. Toronto: University of Toronto. 

Collier, S. (1996). Summary of lessons learned at the OECD Halden Reactor Project on advanced control rooms, 

automation and allocation offunction (HWR-46 1). Halden, Norway: OECD Halden Reactor Project. 

Converse, S. (1992). Computerized procedures for nuclear plants: Evaluation of the computerized procedures 

manual (COPMA II). In Proceedings of the 1997 IEEE Fifth Conference on Human Factors and Power Plants. 

Washington DC: IEEE. 

Converse, S. (1994). Operating procedures: Do they reduce operator error? In Proceedings of the Human Factors 

and Ergonomics Society 38th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. 

Converse, S. (1995). Evaluation of the Computerized Procedure Manual II (COPMA H1) (NUREG/CR-6398). 

Washington, DC: U. S. Nuclear Regulatory Commission. 

Converse, S., Perez, P., Clay, M., and Meyer, S. (1992). Computerized procedures for nuclear power plants: 

Evaluation of the Computerized Procedure Manual (COMPA-II). In Proceedings of the 1992 IEEE Fifth 

Conference on Human Factors and Power Plants. New York: IEEE. 

Degani, A. and Wiener, E. (1993). Cockpit checklists: Concepts, design, and use. Human Factors, 35, 345-360. 

Desaulniers, D. (1997). Stress in the control room: Effects and solutions. In Proceedings of the 1997 IEEE Sixth 

Conference on Human Factors and Power Plants. Washington DC: IEEE. 

Desaulniers, D., Gillan, D., and Rudisill, M. (1988). The effects of format in computer-based procedure displays. 

In Proceedings of the Human Factors Society 32nd Annual Meeting. Santa Monica, CA: Human Factors Society. 

Dien, Y. and Montmayeul, R. (1995). Taking account of human factors in control-room design. In Proceedings of 

the Topical Meeting on Computer-Based Human Support Systems: Technology, Methods and Future. LaGrange 

Park, IL: American Nuclear Society. 

Dien, Y., Montmayeul, R., and Beltranda, G. (1991). Allowing for human factors in computerized procedure 

design. In Proceedings of the Human Factors Society 35nd Annual Meeting. Santa Monica, CA: Human Factors 

Society. 

DOE (1992). Writer's guide for technical procedures (DOE STD-1029-92). Washington, DC: U.S. Department of 

Energy. 

Endestad, T. and Meyer, P. (1993). GOMS analysis as an evaluation tool in process control: An evaluation of the 

ISACS-1 Prototype and the COPMA System (HWR-349). Halden, Norway: OECD Halden Reactor Project.

NUREG/CR-6634 8-2
8 REFERENCES

Endsley, M. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human 

Factors 32ndAnnual Meeting. Santa Monica, CA: Human Factors Society. 

Endsley, M. (1993). Situation awareness and workload: Flip sides of the same coin. In Proceedings of the 7th 

International Symposium on Aviation Psychology. 

Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32-64. 

Ephrath, A. and Young, L. (1981). Monitoring vs. man-in-the-loop detection of aircraft control failures. In J. 

Rasmussen and W. Rouse (Eds.), Human detection and diagnosis of system failures. New York: Plenum Press. 

EPRI (1991). Project No. 669 - Request for additional information on EPRI advanced light water reactor 

requirements document for passive plant designs - Human Factors Branch, TAC NO. 77871 (Letter from Kintner 

to NRC, 6/13/9 1). Washington, DC: U.S. Nuclear Regulatory Commission. 

EPRI (1 993a). Advanced light water reactor utility requirements document, Volume III, AL WR Passive Plant, 

Chapter 10. Man-Machine Interface Systems (Revisions 5 and 6). Palo Alto, CA: Electric Power Research 

Institute. 

EPRI (1993b). Guideline on licensing digital upgrades (EPRI TR-102348). Palo Alto, CA: Electric Power 

Research Institute. 

Federico, P. (199 1). Recognition measurement: Computer-based and paper-based methods. In Proceedings of the 

Human Factors Society 35th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Fischer, H., Hofmann, H., and Roth-Seefrid, H. (1991). Advanced functions and systems for operator support and 

plant management. Reliability Engineering and System Safety, 33, 341-363. 

Folleso, K., Meyer, P., and Volden, F. (1993). Source material for lessons learnedfrom test and evaluation 

activities performed at the OECD Halden Reactor Project (HWR-337). Halden, Norway: OECD Halden Reactor 

Project. 

Forzano, P. and Castagna, P. (1997). Procedures, quality, standards, and the role of human factors and 

computerized tools. In Proceedings of the 1997 IEEE Sixth Conference on Human Factors and Power Plants. 

New York: IEEE. 

Forzano, P. and Perini, C. (1988). TURBOSTART: An expert system as operator guide. In Proceedings of the 

1988 IEEE Fourth Conference on Human Factors and Power Plants. New York: IEEE. 

Fox, J. (1992). The effects of using a hypertext tool for selecting design guidelines. In Proceedings of the Human 

Factors Society 36th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Fraker, M. (1988). A theory of situation awareness: Implications for measuring situation awareness. In 

Proceedings of the Human Factors Society 32nd Annual Meeting. Santa Monica, CA: Human Factors Society. 

Frey, P. and Garris, R. (1992). Big graphics and little screens: Designing graphical displays for maintenance 

tasks. In IEEE Transactions on Systems, Man, and Cybernetics, 22, 10-19.

8-3 NUREG/CR-6634
8 REFERENCES

Galletti, G. (1996). Human factors issues in digital system design and implementation. In Proceedings of the 

1996 American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation, Control, and 

Human-Machine Interface Technologies. La Grange Park, IL: American Nuclear Society. 

Galletti, G. and Sutthoff, A. (1992). Lessons learned from the special inspection program for emergency 

operating procedures, Supplement I (NUREG-1358). Washington, DC: U.S. Nuclear Regulatory Commission. 

Gertman, D., Ostrum, L., Wilhelmsen, C., and Romero, H. (1994). Methodologies for assessing the risk impact of 

new technologies. Draft Rev. 2 (Tech. Report EGG-2740). Idaho Falls, ID: Idaho National Engineering 

Laboratory. 

Glushko, J. (1992). Seven ways to make a hypertext project fail. Technical Communication, 39, 226-230. 

Gould, J., Alfaro, L., Barnes, V., Finn, R., Grischowsky, N., and Minuto, A. (1987). Reading is slower from CRT 

displays than from paper: Attempts to isolate a single-variable explanation. Human Factors, 29, 269-299. 

Graham, P. (1989). Emergency operating procedure upgrade program and audit results. Transactions of the 

American Nuclear Society, Supplement, 59, 48-49. 

Grant, T., Harris, M., Barnes, V., Larson, L., Thurman, A., and Weakley, S. (1989). Value-impact assessment for 

a candidate operating procedure program (NUREG/ CR-5458) Washington, DC: U.S. Nuclear Regulatory 

Commission. 

Green, D. and Swets, J. (1988). Signal detection theory andpsychophysics. Los Altos, CA: Peninsula Publishing. 

Hallbert, B., and Meyer, P. (1995a). Summary of lessons learned at the OECD Halden Reactor Project for the 

design and evaluation of human-machine systems. In Transactions of the American Nuclear Society, Supplement, 

59, 48-49. 

Hallbert, B. and Meyer, P. (1 995b). Summary of lessons learned at the OECD Halden Reactor Project for the 

design and evaluation of human-machine systems (HWR-376). Halden, Norway: OECD Halden Reactor Project. 

Harpster, J., Shulman, G., and Liebowitz, H. (1989). Visual performance on CRT screens and hard-copy displays. 

Human Factors, 31, 247-257. 

Hoecker, D., Corker, K., Roth, E., Lipner, M., and Bunzo, M. (1994). Man-machine Design and Analysis System 

(MIDAS) applied to a computer-based procedure-aiding system. In Proceedings of the Human Factors and 

Ergonomics Society 38th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. 

Hoecker, D. and Roth, E. (1996). Using models of operator performance to support MMIS design: An example in 

the use of procedural aids. In Proceedings of the 1996 American Nuclear Society International Topical Meeting 

on Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies. La Grange Park, IL: 

American Nuclear Society. 

Holt, R., Boehm-Davis, D., and Schultz, A. (1989). Multilevel structured documentation. Human Factors, 31, 

215-228.

NUREG/CR-6634 8-4
8 REFERENCES

Hutchins, E. (1990). The technology of team navigation. In J. Galegher, R. Kraut, and C. Egido (Eds.), 

Intellectual teamwork: Social and technologicalfoundations of cooperative work. Hillsdale, NJ: Erlbaum. 

IAEA (1993). The potential of knowledge based systems in nuclear installations (IAEA-TECDOC-700). Vienna: 

International Atomic Energy Agency. 

IAEA (1994). Development and implementation of computerized operator support systems in nuclear installations 

(Technical Reports Series No. 372). Vienna: International Atomic Energy Agency. 

IAEA (1995). Computerization of operation and maintenance for nuclear power plants (IAEA-TECDOC-808). 

Vienna: International Atomic Energy Agency. 

Inaba, K. (1990). Some useful lessons learned about electronic presentation ofjob performance aids. Advances in 

Human Factors Research on Man/Computer Interactions. LaGrange Park, IL: American Nuclear Society. 

James, G. (1985). Document databases. New York: Van Nostrand Reinhold Company. 

Jeffroy, F. and Charron, S. (1997). Safety assessment to research in the domain of human factors: The case of 

operation with computerized procedures. In Proceedings of the 1997 IEEE Sixth Conference on Human Factors 

and Power Plants. Washington DC: IEEE. 

Jenkinson, J., Shaw, R., and Andow, P. (1991). Operator support systems and artificial intelligence. Reliability 

Engineering and System Safety, 33, 419-437. 

Kammann, R. (1975). The comprehensibility of printed instructions and the flowchart alternative. Human 

Factors, 17, 183-191. 

Kang, K. (1997). Development strategies of an intelligent human-machine interface for next generation nuclear 

power plants. In 1997 IEEE Sixth Conference on Human Factors and Power Plants. Washington DC: IEEE. 

Kincaid, J., Schurman, D., and Hays, R. (1990). Field test of a portable paperless technical manual for job aiding. 

Advances in Human Factors Research on Man/Computer Interactions. LaGrange Park, IL: American Nuclear 

Society. 

Krauss, F., Middendorf, K., and Willits, L. (1991). A comparative investigation of hardcopy vs. online 

documentation. In Proceedings of the Human Factors Society 35th Annual Meeting. Santa Monica, CA: Human 

Factors Society. 

Krohn, G. (1983). Flowcharts used for procedural instructions. Human Factors, 25, 573-581. 

Land, S., Malin, J., Thronesberry, C., and Schreckenghost, D. (1995). Making intelligent systems team players: A 

guide to developing intelligent monitoring systems (NASA Technical Memorandum 104807). Houston, TX: 

National Aeronautics and Space Administration. 

Lapinsky, G. (1989). Lessons learned from the special inspection program for emergency operating procedures 

(NUREG-1358). Washington, DC: U.S. Nuclear Regulatory Commission.

8-5 NUREG/CR-6634
8 REFERENCES

Laughery, R. and Persensky, J. (1994). Network modeling of nuclear operator procedure. In Proceedings of the 

Human Factors and Ergonomics Society 38th Annual Meeting. Santa Monica, CA: Human Factors and 

Ergonomics Society. 

Link, W., Von Holle, J., and Madison, D. (1987). Integrated Maintenance Information System (IMIS): A 

maintenance information delivery concepts (AFHRL Technical Paper 87-27). Wright-Patterson Air Force Base, 

OH: U.S. Air Force 

Lipner, M., Pitcairn, F., and Bastien, R. (1992). Computerized system for procedure implementation monitoring. 

In IAEA/1WG-NPPCI Special Meeting on Operating Procedures for Nuclear Power Plants and Their Presentation. 

Vienna, Austria: International Atomic Energy Agency. 

Lipner, M. and Rusnica, L. (1996). Computerized systems for procedures implementation monitoring. In 

Proceedings of the 1996 American Nuclear Society International Topical Meeting on Nuclear Plant 

Instrumentation, Control, and Human-Machine Interface Technologies. La Grange, IL: American Nuclear Society. 

Malin, J., Schreckenghost, D., Woods, D., Potter, S., Johannesen, L., Holloway, M., and Forbus, K. (1991a). 

Making intelligent systems team players: Case studies and design issues. Volume 1: Human-computer interaction 

design (NASA Technical Memorandum 104738). Houston, TX: National Aeronautics and Space Administration. 

Malin, J., Schreckenghost, D., Woods, D., Potter, S., Johannesen, L., Holloway, M. (1991b). Making intelligent 

systems team players: Case studies and design issues. Volume 2: Fault management system cases (NASA 

Technical Memorandum 104738). Houston, TX: National Aeronautics and Space Administration. 

Mampaey, L., Moeyaert, P., Bastenaire, F., Casier, F., and Chi, N. (1988). Operator advisor: An expert system to 

reduce human error in emergency response. In Proceedings of the International ENS/ANS Conference on Thermal 

Reactor Safety, NUCSAFE 88. Avignon, France: Societe Francaise d'Energie Nucleaire. 

Matthews, M. and Mertins, K. (1989). Visual performance and subjective discomfort in prolonged viewing of 

chromatic displays. Human Factors, 31, 259-271. 

Mavko, B., Stritar, A., and Salamun, I. (1995). Computer managed tool for advanced nuclear power plant control 

room. In Topical Meeting on Computer-Based Human Support Systems: Technology, Methods, and Future. 

LaGrange Park, IL: American Nuclear Society. 

Mohageg, M. (1992). The influence of hypertext linking structures on the efficiency of information retrieval. 

Human Factors, 34, 351-368. 

Moieni, P. and Spurgin, A. (1993a). Computerized emergency operating procedures research. San Diego, CA: 

Accident Prevention Group. 

Moieni, P. and Spurgin, A. (1993b). Evaluation of computerized emergency operating procedure systems as an 

operator support in nuclear power plants. In Proceedings of PSAM-I1. San Diego, CA: University of California. 

Moray, N. (1986). Monitoring behavior and supervisory control. In K. Boff, L. Kaufman, and J. Thomas (Eds.), 

Handbook of Human Perception and Performance. New York: Wiley.

NUREG/CR-6634 8-6
8 REFERENCES

Morgenstern, M., Barnes, V., McGuire, M., Radford, L., and Wheeler, W. (1987). Study of operating procedure 

in nuclear power plants: Practices andproblems (NUREG/CR-3968). Washington, DC: U.S. Nuclear Regulatory 

Commission. 

Mosier, K., Palmer, E., and Degani, A. (1992). Electronic checklists: Implications for decision making. In 

Proceedings of the Human Factors Society 36th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Mumaw, R. (1994). The effects of stress on nuclear power plant operational decision making and training 

approaches to reduce stress effects (NUREG/CR-6127). Washington, DC: U.S. Nuclear Regulatory Commission. 

Neisser, U. (1967). Cognitive psychology. New York, NY: Appleton-Century Crofts. 

Nelson, B. and Smith, T. (1990). User interaction with maintenance information: A performance analysis of 

hypertext versus hard copy formats. In Proceedings of the Human Factors Society 34th Annual Meeting. Santa 

Monica, CA: Human Factors Society. 

Nelson, W., Fordestrommen, N., Holmstrom, C., Krogsaeter, M., Karstad, T., and Tunold, 0. (1990). 

Experimental evaluation of the computerized procedure system COPMA (HWR-277). Halden, Norway: OECD 

Halden Reactor Project. 

Nielson, J. (1990) Hypertext and hypermedia. Boston, MA: Harcourt Brace Jovanovich, Publishers. 

Niwa, Y., Hollnagel, E., and Green, M. (1996). Guidelines for computerized presentation of emergency operating 

procedures. Nuclear Engineering and Design, 167, 113-127. 

NRC (1997a). Verification, validation, reviews, and audits for digital computer software used in safety systems of 

nuclear power plants (Regulatory Guide 1.168). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1997b). Configuration management plans for digital computer software used in safety systems of nuclear 

power plants (Regulatory Guide 1.169). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1 997c). Software test documentation for digital computer software used in safety systems of nuclear power 

plants (Regulatory Guide 1.170). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1997d). Software unit testing for digital computer software used in safety systems of nuclear power plants 

(Regulatory Guide 1.171). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1997e). Software requirements specifications for digital computer software used in safety systems of nuclear 

power plants (Regulatory Guide 1.172). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1997f). Developing software life cycle processes for digital computer software used in safety systems of 

nuclear power plants (Regulatory Guide 1.173). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1978). Quality assurance program requirements (Regulatory Guide 1.33, Revision 2). Washington, DC: 

U.S. Nuclear Regulatory Commission.

8-7 NUREG/CR-6634
8 REFERENCES

NRC (1982). Guidelines for the preparation of emergency operating procedures (NUREG-0899). Washington, 

DC: U.S. Nuclear Regulatory Commission. 

NRC (1984). Standard review plan (NUREG-0800, Rev. 1). Washington, DC: U.S. Nuclear Regulatory 

Commission. 

NRC (1991). Request for additional information on EPRI Advanced Light Water Reactor Requirements Document 

for Passive Plant Designs - Human Factors Branch (Letter from Wilson to Kintner, TAC NO. 77871, 4/17/9 1). 

Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1994). NRC review of Electric Power Research Institute's Advanced Light Water Requirements Document 

(NUREG-1242-V3-P2). Washington, DC: U.S. Nuclear Regulatory Commission. 

NRC (1995). Use of NUMARC/EPRI Report TR-102348, guideline on licensing digital upgrades, in determining 

the acceptability ofperforming analog-to-digital replacements under 10 CFR 50.59 (NRC Generic Letter 95-02). 

Washington, DC: U. S. Nuclear Regulatory Commission. 

Ogawa, K. and Yonemura, S. (1992). Usability analysis of design guideline database in human-computer interface 

design. In Proceedings of the Human Factors Society 36th Annual Meeting. Santa Monica, CA: Human Factors 

Society. 

O'Hara, J. (1994). Advanced human system interface design review guideline: General evaluation model, 

technical development, andguideline description (NUREG/CR-5908, Volume 1). Washington, DC: U.S. Nuclear 

Regulatory Commission. 

O'Hara, J., Brown, W., and Nasta, K. (1996). Development of NUREG, 0700, Revision I (BNL Report L-1317-2

12/96). Upton, NY: Brookhaven National Laboratory. 

O'Hara, J., Brown, W., Stubler, W., Wachtel. J., and Persensky, J. (1996). Human-system interface design review 

guideline (NUREG-0700, Rev. 1). Washington, DC: U.S. Nuclear Regulatory Commission. 

O'Hara, J., Higgins, J., Stubler, W., Goodman, C., Eckenrode, R., Bongarra, J., and Galletti, G. (1994). Human 

factors engineering program review model (NUREG-071 1). Washington, DC: U.S. Nuclear Regulatory 

Commission. 

O'Hara, J., Stubler, W., Brown, W., and Higgins, J. (1997). Integrated system validation: Methodology and review 

criteria (NUREG/CR-6393). Washington, DC: U.S. Nuclear Regulatory Commission. 

O'Hara, J., Stubler, W., and Higgins, J. (1996). Hybrid human system interfaces: Human factors considerations 

(BNL Report J6012-T1-4/96). Upton, NY: Brookhaven National Laboratory. 

O'Hara, J., Stubler, W., and Nasta, K. (1997) Human-system interface management: Effects on operator 

performance and issue identification (BNL Report W6546-1-1-7/97). Upton, NY: Brookhaven National 

Laboratory.

NUREG/CR-6634 8-8
8 REFERENCES

Orende, R. (1996). Control room I&C upgrades, innovations, and HMI considerations for the Temelin Nuclear 

Plant. In Proceedings of the 1996 American Nuclear Society International Topical Meeting on Nuclear Plant 

Instrumentation, Control, and Human-Machine Interface Technologies. La Grange Park, IL: American Nuclear 

Society. 

Orvis, D. and Spurgin, A. (1996). Research in computerized emergency procedures systems to enhance reliability 

of nuclear power plant operating crews (APG Report No. 35). San Diego, CA: Accident Prevention Group. 

Oser, R., McCallum, G., Salas, E., and Morgan, B. (1989). Toward a definition of team work: An analysis of 

critical team behaviors (Tech. Report 89-004). Orlando, FL: Naval Training Systems Center, Human Factors 

Division. 

Pirus, D. and Chambon, Y. (1997). The computerized procedures for the French N4 series. In Proceedings of the 

1997 IEEE Sixth Conference on Human Factors and Power Plants. Washington, DC: IEEE. 

Rasmussen, J. (1981). Models of mental strategies in process plant diagnosis. In J. Rasmussen and W. Rouse 

(Eds.), Human detection and diagnosis of system failure. New York: Plenum Press. 

Rasmussen, J. (1983). Skills, rules, knowledge: Signals, signs, and symbols and other distinctions in human 

performance models. IEEE Transactions on Systems, Man, and Cybernetics, 13, 257-267. 

Rasmussen, J. (1986). Information processing and human-machine interaction: an approach to cognitive 

engineering. New York: North-Holland. 

Reason, J. (1988). Modeling the basic error tendencies of human operators. Reliability Engineering and System 

Safety, 22, 137-153. 

Reaux, R. and Williges, R. (1988). Effects of level of abstraction and presentation media on usability of user

system interface guidelines. In Proceedings of the Human Factors Society 32nd Annual Meeting. Santa Monica, 

CA: Human Factors Society. 

Reed, J., Hogg, D. and Hallbert, B. (1995). An evaluation of an on-line expert system in nuclear process control. 

Proceedings of the Topical Meeting on Computer-Based Human Support Systems: Technology, Methods, and 

Future. La Grange Park, IL: American Nuclear Society. 

Rook, F. and Donnell, M. (1993). Human cognition and the expert system interface: Mental models and inference 

explanations. IEEE Transactions on Systems, Man, and Cybernetics, 23, 1649-1661. 

Roth, E. (1994). Operator performance in cognitively complex simulated emergencies: Implications for computer

based support systems. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting. 

Santa Monica, CA: Human Factors and Ergonomics Society. 

Roth, E., Bennett, K., and Woods, D. (1987). Human interaction with an 'intelligent' machine. International 

Journal of Man-Machine Studies, 27, 479-525.

8-9 NUREG/CR-6634
8 REFERENCES

Roth, E., Mumaw, R., and Lewis, P. (1994). An empirical investigation of operator performance in cognitively 

demanding simulated emergencies (NUREG/CR-6208). Washington, DC: U.S. Nuclear Regulatory Commission. 

Roth, E. and O'Hara, J. (1998). Integrating digital and conventional HSIS: Lessons learned from a control room 

modernization program (BNL Report J6012-3-4-5/98). Upton, NY: Brookhaven National Laboratory. 

Rubinsky, Y. (1993). Technical documentation in the world of STEP. CALS Journal, 2, 63-66. 

Salas, E., Dickinson, T., Converse, S., and Tannenbaum, S. (1992). Toward an understanding of team 

performance and training. In R.W. Swezey and E. Salas (Eds.), Teams: Their Training and Performance. 

Norwood, NJ: Ablex. 

Sheridan, T. (1976). Toward a general model of supervisory control. In T. Sheridan and G. Johannsen (Eds.), 

Monitoring behavior and supervisory control. New York: Plenum Press. 

Sheridan, T. (1987). Supervisory control. In G. Salvendy (Ed.), Handbook of human factors. New York: Wiley. 

Shneiderman, B. (1987). Designing the user interface: Strategies for effective human-computer interaction. New 

York: Addison-Wesley. 

Sorkin, R. and Woods, D. (1985). System with human monitors: A signal detection analysis. Human Computer 

Interaction, 1, 49-75. 

Sorkin, R., Kantowitz, B., and Kantowitz, S. (1988). Likelihood alarm displays. Human Factors, 30, 445-459. 

Spurgin, A. (1995). Impact of computer-based support systems on control room operations. In Topical Meeting on 

Computer-Based Human Support Systems: Technology, Methods, and Future. LaGrange Park, IL: American 

Nuclear Society. 

Spurgin, A., Orvis, D., Spurgin, J. and Luna, C. (1990). The BWR Emergency Operating Procedures Tracking 

System (EOPTS): Evaluation by control-room operating crews (EPRI NP-6846). Palo Alto, CA: Electric Power 

Research Institute. 

Spurgin, A., Wachtel, J., and Moieni, P. (1993). The state of practice of computerized operating procedures in the 

commercial nuclear power industry. In Proceedings of the Human Factors and Ergonomics Society 37th Annual 

Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. 

Stubler, W., Higgins, J., and Kramer, J. (2000). Maintainability of digital systems: Technical basis and human 

factors review guidance (NUREG/CR-6636). Washington, DC: U.S. Nuclear Regulatory Commission. 

Stubler, W., O'Hara, J., and Kramer, J. (2000). Soft controls: Technical basis and human factors review guidance 

(NUREG/CR-6635). Washington, DC: U.S. Nuclear Regulatory Commission. 

Stubler, W., Higgins, J., and O'Hara, J. (1996). Evaluation of the potential safety-significance of hybrid human

system interface topics (BNL Report J6012-T2-6/96). Upton, NY: Brookhaven National Laboratory.

NUREG/CR-6634 .8-10
8 REFERENCES

Stubler, W. and O'Hara, J. (1996). Group-view displays: Functional characteristics and review criteria (BNL 

Technical Report E2090-T4-4-4/95, Rev. 1). Upton, NY: Brookhaven National Laboratory. 

Tai, I., Naito, N., and Makino, M. (1991). Advanced control complex for BWR nuclear power plant. In 

Proceedings of the Human Factors Society 35th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Teigen, J. and Ness, E. (1994). Computerized support in the preparation, implementation, and maintenance of 

operating procedures. In IFA C Workshop on Computer Software Structures Integrating AI/KBS Systems in 

Process Control. Lund, Sweden. 

Terry, P. (1989). A layman's guide to expert systems. Power Engineering, September, 52-55. 

Tolbert, C., Moore, C., and Wieringa, D. (1991). Emerging issues for procedures in the nuclear industry. In 

Proceedings of the Human Factors Society 35th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Trager, E. (1988). Significant events involving procedures (AEOD Report S801). Washington, DC: U.S. Nuclear 

Regulatory Commission. 

Uhrig, R. (1994). Artificial neural networks in nuclear power plants. Nuclear News, 37 (9), 38-40. 

U.S. Code of Federal Regulations, Part 50, Domestic Licensing of Production and Utilization Facilities, Title 10, 

Energy, U.S. Government Printing Office, Washington, DC, revised periodically. 

U.S. Code of Federal Regulations, Part 50.59, Changes, Tests, and Experiments, Title 10, Energy, U.S. 

Government Printing Office, Washington, DC, revised periodically. 

Vicente, K., Mumaw, R. and Roth, E. (1997). Cognitive functioning of control room operators -final phase 

(Report prepared for the Atomic Energy Control Board). Toronto, Canada: University of Toronto. 

Weldon, L., Koved, L., and Shneiderman, B. (1985). The structure of information in online and paper technical 

manuals. In Proceedings of the Human Factors Society 29th Annual Meeting. Santa Monica, CA: Human Factors 

Society. 

Wickens, C. (1984). Engineering psychology and human performance. Columbus, OH: Merrill Publishing 

Company. 

Wickens, C. and Kessel, C. (1981). The detection of dynamic system failures. In J. Rasmussen and W. Rouse 

(Eds.), Human detection and diagnosis of system failures. New York, NY, Plenum Press. 

Wieringa, D., Moore, C., and Barnes, V. (1992). Procedure writing: Principles and practices. Columbus, OH: 

Battelle Press. 

Wilhelmsen, C., Gertman, D., Ostrom, L., Nelson, W., Galyean, W., and Byers, J. (1992). Reviewing the impact 

of advanced control room technology. In Proceedings of the Human Factors Society 36th Annual Meeting. Santa 

Monica, CA: Human Factors Society.

8-11 NUREG/CR-6634
8 REFERENCES

Woods, D. and Elm, W. (1985). Getting lost: A case study in interface design. In Proceedings of the Human 

Factors Society 29th Annual Meeting. Santa Monica, CA: Human Factors Society. 

Woods, D., Roth, E., Stubler, W., and Mumaw, R. (1990). Navigating through large display networks in dynamic 

control applications. In Proceedings of the Human Factors Society 34th Annual Meeting. Santa Monica, CA: 

Human Factors and Ergonomics Society. 

Wourms, D. and Rankin, W. (1994). Computer-basedprocedures (Tech Report CSERIADC-RA-94-002). 

Wright-Patterson Air Force Base, OH: Crew Systems Ergonomics Information Analysis Center. 

Wright, P. and Reid, F. (1973). Written information: Some alternatives to prose for expressing the outcome of 

complex contingencies. Journal of Applied Psychology, 57, 160-166. 

Yamamoto, Y. and Ito, K. (1993). Development of computerized supporting system for PWR plant emergency 

response guidelines of Japan. In Proceedings of the Specialist Meeting on Operator Aids for Severe Accidents 

Management and Training. Halden, Norway: OECD Halden Reactor Project.

NUREG/CR-6634 8-12
PART 2 

CBP Guidelines for Design Review

NUREG/CR-6634
9 GUIDANCE FOR CBP DESIGN PROCESS REVIEW 

As discussed in Section 6. 1, the design process guidelines were developed to address important aspects of the 

process and to provide a means whereby human performance issues may be assessed during a design review. The 

guidelines were formatted to correspond to the NRC's general guidance for design process review in NUREG-0711. 

They are organized into the following sections: 

* HFE Program Management 

* Operating Experience Review 

* Functional Analysis 

* Task Analysis 

* Staffing 

* Human Reliability Analysis 

* Human-System Interface Design 

* Procedure Development 

• Training Program Development 

* Human Factors Verification and Validation 

Guidelines may specify that some identified aspect of CBPs needs to be "evaluated." NUREG-0700 defines general 

methods for evaluation and for identifying criteria. Since the guidance in this section will eventually be 

incorporated into NUREG-0700, those methods and criteria are not repeated below. 

9.1 HFE Program Management 

(I) CBP design and evaluation should be performed with a multidisciplinary team. 

Discussion: The NRC's analysis of EOPs (Lapinsky 1989) noted that the lack of a multidisciplinary team 

was associated with procedure deficiencies which can negatively impact the development and use of 

CBPs; therefore, a CBP development program should address this issue. The exact skills needed on the 

multidisciplinary team will vary, depending on the scope of the CBP systems. Appendix A, NUREG-0711 

gives a range of the broadest possible skills required; typical ones would be those described in Subsections 

2, 4, 6, 7, 9, and 10; namely, Systems Engineering, I&C Engineering, Human Factors Engineering, Plant 

Operations, Plant Procedure Development, and Personnel Training. 

(2) An implementation plan should be developed to deal with CBP design, maintenance, training and 

evaluation. 

Discussion: The NRC's analysis of EOPs (Lapinsky, 1989) noted that the lack of a systematic process was 

associated with procedure deficiencies, and can negatively impact the development and use of CBPs; 

therefore, a CBP development program should include this issue.

9-1 NUREG/CR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

(3) The CBP's design constraints or assumptions should be documented and their implications for safety 

should be evaluated to ensure they do not compromise the CBP system's goals. 

Discussion: Identifying design constraints and assumptions is important to HFE design in general (O'Hara 

et al., 1994), and for procedures systems in particular (Barnes et al., 1996). Reviewers should evaluate the 

constraints and assumptions applied by the designers of the system and specify their implications for 

safety. These might include limitations in the capabilities of the hardware or software, assumptions about 

operators' knowledge and skills, uncertainties about particular content areas that are not fully developed, 

or decisions to leave the design of some system aspects of the CBP to other individuals. The reviewer 

should evaluate the acceptability of the designers' assumptions and assure that any constraints do not 

compromise the system's goals. 

(4) The CBP development program should be fully documented, including design goals and assumptions, use 

of operating experience, design analyses, establishment of system requirements, tests and evaluations, 

detailed description of the design, and verification and validation. 

Discussion: Reviewers should be able to follow the designers' development process from the analysis of 

requirements to the final design and testing. In addition, reviewers should ascertain that the information 

used to develop the procedures will be retained in a form accessible to the licensees who will implement 

the CBPs at a new or existing plant. Such records are essential for assuring that the procedures can be 

kept current (Barnes et al., 1996; Lapinsky, 1989). 

9.2 Operating Experience Review 

(1) The CBP design should eliminate or minimize PBP problems where practical. Experience with paper 

procedures should be reviewed to take advantage of lessons learned in their operational use, maintenance, 

and configuration control as well as to help ensure that problems in implementing PBPs are resolved. 

Table 9.1 is a partial list of identified PBP problems. 

Discussion: Studies of PBPs at NPPs, the experiences of assisting licensees in developing procedures, and 

lessons learned through inspections suggest that some problematic aspects of PBPs may be rectified by 

computerization (Barnes et al., 1996). 

(2) Operating experience with CBP systems should be reviewed to take advantage of lessons learned in using 

the systems, as well as to ensure that any problems in implementing CBPs are dealt with. 

Discussion: The review should consider the use of CBPs in the nuclear industry and related industries. 

(3) Human performance issues, such as visual fatigue, arising from the computerization of documents and 

manuals should be addressed. 

Discussion: Many HFE issues were identified that limited personnel performance when support aids are 

transferred from paper to computers. Familiarity with this literature may help to minimize these issues. 

9.3 Functional Analysis 

An overall concept should be developed of the operators' role in managing and supervising plant procedures. 

Discussion: Operators must be able to supervise the conduct of procedure operations, evaluate their success at 

achieving safety goals, and formulate response plans when those goals are not being met. Research shows that 

CBPs can lessen operator independence. A clear statement of the operators' roles and responsibilities will help the 

design and training aspects of CBP development. The design decisions (e.g., scope and content, integration, and 

function allocation) should flow clearly from the designers' overall concept of the operators' role in managing plant

NUREG/CR-6634 9-2
9 CBP DESIGN PROCESS REVIEW GUIDANCE

systems. Design documents should clearly articulate this concept and its rationale, and describe how the design 

carries out the concept (Barnes et al., 1996). 

9.4 Task Analysis 

(1) The effect of the CBP on the tasks of individual members of the crew should be analyzed, considering any 

potential changes that may result from the combined use of CBPs and PBPs, and also the effect on 

communications. 

Table 9.1 Examples of Deficiencies in Paper-Based Procedures 

Issue Deficiency 

Design Process * Inadequate participation of operations and training personnel in developing 

procedures 

* Technically incorrect EOPs 

* Suitable under standard situations, but less support in unusual situations 

* Incomplete procedures 

* Inadequate consideration of the time required to complete procedural actions 

* Insufficient verification and validation (V&V) of procedures 

Implementation 0 Nonspecific entry and exit conditions for support procedures 

* Procedures are fixed and inflexible 

* Incorrect sequencing of action steps 

* Inadequate consistency across procedures 

• Inconsistencies in formatting and use of terminology 

• Incorrect identification of plant equipment 

* Inadequate provision for varying level of detail 

* Non-sequential presentation of information 

* Difficulties in navigating to related information 

* Inadequate management of multiple procedures 

• Unsatisfactory integration of procedure tasks and other tasks 

• Problems in labeling and headings 

* Notes and cautions in improper places 

* Lack of context-dependent highlighting and navigation 

* Requirements to use multiple procedures simultaneously and move between 

sections 

• Lack of flowcharts to guide procedure use 

* Inadequate support and reference material 

* Bulkiness 

* Physical handling of procedures near control panels 

* Separation from other information sources, such as SPDS 

* Inconsistency with other HSIs in terms of references to plant equipment 

Training 0 Operators poorly trained in using procedures

9-3 NUREG/CR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

Maintenance I * Maintaining technical accuracy of procedures lacking 

Discussion: CBP may have an important effect on crew member's roles and communication. The extent 

to which they are changed relative to PBP use may impact the crew's situation awareness and plant safety 

(Roth and O'Hara, 1998). 

(2) CBP tasks should be analyzed and used as an input to its design. 

Discussion: To ensure that the design of CBPs is acceptable and usable, the results of task analyses 

should be incorporated. Following the decisions on function allocation, the operators' tasks should be 

defined at increasing levels of detail to specify their actions and information requirements. 

(3) Tasks associated with CBP failure and back-up should be identified to define the requirement for 

indicating malfunctions. The task of smoothly transitioning from CBPs to a back-up method (such as 

PBPs) also should be addressed. 

Discussion: By identifying this capability as a task, the designer will then include it in the HSI, 

procedures, and training. A failure of the CBP System may be total or a more insidious partial one that is 

not obvious to the operators. Besides designing-in indicators of failure, a means should be identified to 

smoothly move to the back-up system; the content of the CBP and PBP (or other back-up) should be 

compatible. 

9.5 Staffing 

(1) The demands of operating and maintaining the CBP should be assessed for their implications for 

personnel skills and qualifications. 

Discussion: CBPs may impose demands on plant personnel that are unlike other systems, for example, 

maintaining a large database. Human error in that particular task was identified as a major cause of 

events involving these systems (O'Hara, Stubler, and Higgins, 1996). 

9.6 Human Reliability Analysis 

(1) Any effects on performance caused by computerization of procedures should be analyzed for their 

implications for those human actions modeled in a PRA. 

Discussion: PRAs may reflect analyses of human actions based on paper procedures. CBPs have broad 

effects on performance, both from team and individual perspectives. Accessing EOPs through a computer 

system may create keyhole effects and may increase interface management demands. Some tasks may be 

eliminated, such as monitoring procedure-specified parameters, or analyzing procedure logic. Also, 

human errors in maintaining digital systems is a major cause of events (O'Hara, Stubler, and Higgins, 

1996). All of these potential effects should be considered in evaluating impacts on reliability assumptions 

and analyses. Further, since operating experience with CBPs is limited, assessing the impact of CBPs on 

human performance and reliability should utilize, in part, the results of tests obtained during CBP design, 

evaluation, verification, and validation (Converse, 1995; EPRI, 1993a; Orvis and Spurgin, 1996). 

(2) The analysis should consider the effects on human reliability of loss of CBPs and transfer to PBPs. 

Discussion: Using PBPs places different demands on the crew and can change their interaction and roles; 

risk analyses should consider the implications of these changes.

NUREG/CR-6634 9-4
9 CBP DESIGN PROCESS REVIEW GUIDANCE

9.7 Human-System Interface Design 

(1) The HSI design should consider methods by which procedure elements are represented in the CBP and the 

extent to which usability principles for PBPs generalize to CBP systems. 

Discussion: There are many guidelines for designing PBPs. However, how far they are applicable for 

implementing a CBP system must be assessed. For example, representing procedure format, e.g., in a 

flowchart or text, may not reflect a simple application of the PBP guidance. 

(2) The procedure functions to be provided by the CBP system should be carefully analyzed to ensure that the 

system is consistent with the utilities' general approach to procedure-based operations, and that the 

operator's inputs and judgements are included, where appropriate. 

Discussion: The CBP system should provide operators with capabilities and functions to support their 

roles as system supervisors and their performance of tasks. 

(3) The following aspects of CBP design should be carefully evaluated to ensure that the use of procedures is 

not jeopardized and that task requirements are adequately supported: 

* Number of VDUs 

• Interface management and navigation functions 

* Flexibility of CBP display and operations 

Discussion: The keyhole effect that results from the limited view of plant information afforded by VDUs, 

interface management tasks, and computer system flexibility can significantly degrade performance. 

These aspects of the design should be evaluated as part of the design review. 

(4) The potential interactive effects between procedure use and the hardware and software used to implement 

them should be evaluated. 

Discussion: NUREG-0700, Rev. 1 has guidance on hardware aspects of interacting with CBPs, such as 

VDUs and input devices. However, there may be other such interactions, including those with software, 

that are not addressed. Since there is no technical basis for guidance, these interactions should be 

examined during the design process. 

(5) The means by which CBPs can support crew cooperation, communication, and decision making should be 

evaluated. 

Discussion: The NRC CBP workshop and several investigations (e.g., Roth and O'Hara, 1998) 

highlighted the need to address crew interactions during the design and implementation of CBPs. 

(6) Operators should be involved in developing and evaluating prototypes to ensure that their final design is 

usable. 

Discussion: Lacking guidance, CBP designers should have access to system prototypes, control room 

mockups or simulations, and representatives of the users to refine the design of the CBPs. Having users 

carry out procedure steps at the worksite (or a simulation) provides important information about step

sequencing, implementation times, access to the displays and controls, and other physical characteristics 

of the work environment, and an opportunity to collect their feedback on initial designs (Barnes et al., 

1996).

9-5 NUREG/CR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

9.8 Procedure Development 

9.8.1 Scope of Procedures 

(1) The purpose and scope of the CBP system should be clearly defined. 

Discussion: If CBPs encompass only some operator tasks, justification should be given for excluding 

others. The implications for operator performance of using CBPs and PBPs for different tasks should be 

considered. 

9.8.2 Bases of Procedures 

Procedure bases refer to the background information used to develop the CBPs. Procedures are critical 

management tools because they are among the more important means of guiding human interactions with the plant 

systems. The procedures must not only prescribe technically correct actions, but must also implement licensee's 

and the NRC's expectations for the conduct of operations. Consequently, their content should be consistent with 

the technical, regulatory, and management bases of plant operations, no matter what medium is used to present 

them. 

(1) The technical bases for procedures should be documented. Where the documented bases for paper 

procedures are unchanged by computerization, the existing document may be used. This should include 

the sources of technical information, as well as the process by which the information was used to define 

the desired operator actions and supplemental information, such as cautions and warnings, figures, and 

tables. 

Discussion: The technical bases for procedures are the information used to define the plant's operational 

characteristics and may be beyond the scope of a human factors review. However, an HFE reviewer 

should evaluate whether and how the CBP designers used this information to define the operators' actions 

and supplemental information, such as cautions and warnings, figures, and tables. Technical bases for 

procedures should include the following: 

* Results of operational "lessons learned" 

* Technical guidelines from owners' group 

* Plant-specific technical guidelines 

* Deviation documentation 

* Results of safety analyses and accident analyses 

* Probabilistic risk assessments (PRAs) 

* Engineering documents 

* Engineering standards applied to the design of the plant 

* Design criteria for the plant's components and systems

NUREG/CR-6634 9-6
9 CBP DESIGN PROCESS REVIEW GUIDANCE

* Drawings and the specifications applied to designing and constructing the plant 

* Records of the basis for, and development of, methods and calculations 

* Results of design verification, qualification tests and functional tests 

* Operational safety limits and technical specifications 

* Expected configuration of plant systems when the procedure (or specific action step) is performed 

* Other anticipated conditions of performance 

* Documentation of setpoints 

* Information on equipment and component labeling 

* Information on location of equipment and components 

(2) The regulatory bases for procedures should be specified, and the manner in which they were applied in 

developing the CBPs should be documented. 

Discussion: The regulatory bases for the procedures are the requirements and guidelines that affect, 

constrain or are implemented by the CBPs, including: 

* NRC Rules, such as 10 CFR 50.54(m) pertaining to shift-staffing requirements, and 10 CFR 50.47(b) 

pertaining to emergency plans 

* NRC Regulatory Guides and Standards, such as ANS/ANSI 18.7 (ANS, 1981) endorsed in Reg. 

Guide 1.33, on plant procedures 

* NRC guidance documents, e.g., NUREG-0800 (NRC, 1984); NUREG-0711 (O'Hara, Higgins, 

Stubler, Goodman, Eckenrode, Bongarra, and Galletti, 1994); NUREG-0899 (NRC, 1982), NUREG

1358 and Supplement I to NUREG-1358 (Galletti and Sutthoff, 1992); generic communications; and 

NUREG/CRs 

* Any commitments made by the licensee to the NRC that affect the procedures 

(3) The management bases for procedures should be documented. 

Discussion: The management bases for procedures are plant or site specific: 

* The licensee's operational philosophy 

* Roles, responsibilities, and authorities assigned to procedure users 

* Policies, programs, and plans for managing plant operations that may affect the content or 

performance of the procedures, such as quality assurance or emergency response 

* Requirements for adhering to procedures

9-7 NUREGICR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

Requirements for independent verification of step completion and accuracy 

(4) If the CBPs are to be implemented in an operating plant using PBPs, their impact on existing 

management bases should be evaluated. 

Discussion: If the CBPs are to be installed in an existing plant, designers should obtain information on 

the management bases from plant personnel. Introducing the CBP system may impact existing licensee 

programs and procedures, such as operator-licensing training programs, emergency response plans, or the 

role of senior operations personnel in managing outages. The licensee's management, rather than the 

CBP designers, should be responsible for determining the impact of CBP design and whether the changes introduced by the CBPs are acceptable or should be revised to conform with existing practices. Those 

policies, practices, programs, and procedures affected should be revised before implementing the system. 

(5) If the CBPs are developed for a generic plant design or for new designs, plans and methods should be 

specified for incorporating the licensee-specific management bases. Since the specific characteristics of 

the intended users and their work environments may not be known, the methods by which the CBPs can 

be tailored for them should be identified. 

Discussion: For CBP designs for advanced control rooms, complete bases for the procedures may be 

unavailable. For example, the plant's Technical Specifications and plant-specific design information may 

not exist until an advanced reactor is built in the United States or a current licensee decides to install 

CBPs in an existing control room. A final review of the incorporation of the bases in the CBPs cannot 

occur until plant- or site-specific information is available. 

The management bases for CBPs for a new plant will not be available in a generic design. Therefore, 

plans and methods for incorporating the management bases for CBPs should be developed by their 

designers. In addition, when CBPs are being developed for generic designs, the specific characteristics of 

the intended users and their work environments may not be known; accordingly, designers should include 

the following provisions for tailoring a CBP design for site-specific applications: 

• Any unique aspects of a plant's design (such as characteristics of heat sinks) 

* The attributes of the worksite (e.g., ambient noise levels, physical location of required displays and 

controls) 

* The intended users (e.g., operator language, experience levels and types, training, crew size, and 

roles) 

9.8.3 Technical Information 

(1) The selection of parameters and indicators of plant state to be monitored at each procedure step should be 

reviewed. 

Discussion: How the CBPs use parameters and plant states will affect the evaluation of procedure steps 

and use by operators, as well as the system's design. Parameters and indicators of equipment states 

should be appropriate. 

(2) The means by which any the CBPs make the following types of assessments should be completely 

documented and reviewed by a multidisciplinary team, including plant operators: 

Conditions for entering procedures

NUREG/CR-6634 9-8
9 CBP DESIGN PROCESS REVIEW GUIDANCE

* Analysis of step logic 

* Assessment of cautions and notes 

* Performance of calculations 

* Assessment of exit conditions from procedures 

* Assessment of high-level procedural goals 

Discussion: Underspecifying procedure logic can cause misunderstandings and potential errors in their 

use. The appropriateness of the analyses must be assured, and the role of operators' judgement accounted 

for. The exact skills needed on the multidisciplinary team will vary with the scope of the CBP systems. 

Appendix A of NUREG-07 11 lists the range of the broadest skills required; typical ones are those in 

Subsections 2, 4, 6, 7, 9, and 10; namely, Systems Engineering, I&C Engineering, Human Factors 

Engineering, Plant Operations, Plant Procedure Development, and Personnel Training. 

(3) Procedures should be specifically tailored to the intended users, their physical work environment, and the 

organization in which the tasks are performed. 

Discussion: Some of the information necessary to prepare a procedure can be developed generically by a 

design organization (e.g., Owners' Group Technical Guidelines for EOPs in current plants). However, 

lessons learned in the nuclear power and other industries showed that procedures must be specially 

tailored to fulfill their functions of supporting users' accurate performance, their physical work 

environment, and the organization in which the tasks are performed. If there is a mismatch, procedures 

may not be followed, or they may be used in unintended ways. Because procedures also are management 

tools, mismatches between licensees' management philosophies and the processes defined in procedures 

can introduce unintended organizational changes or break down existing structures and processes (Barnes 

et al., 1996). 

9.8.4 Maintenance of Procedures 

(1) Methods should be specified for assuring that procedure revisions do not introduce technical inaccuracies, 

or inconsistencies in how the CBPs are presented. 

Discussion: CBP designers should provide for maintaining the integrity of the CBPs and their supporting 

documentation. Because characteristics of users, systems, regulatory requirements, and operational and 

management practices change over time, methods must be devised to control revisions to the CBPs and 

any documentation and databases on which they depend. Methods should be specified for assuring that 

revisions do not lead to technical inaccuracies, or to inconsistencies in how the CBPs are presented. For 

example, a CBP system may depend upon a database to maintain a list of required setpoints for different 

conditions and automatically generate setpoint information included in procedure steps. To maintain the, 

integrity of the CBPs if the database is revised, it is critical that the implications of changing any value 

can be traced and controlled whenever that value appears in procedure steps (Barnes et al., 1996). 

(2) Provisions should be made for temporarily changing procedures. Administrative procedures for 

introducing and handling procedure changes should identify how to properly implement the changes in 

the CBP system. These changes should be clearly identified in the CBP's interface.

9-9 NUREG/CR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

Discussion: It should be very clear to personnel what temporary changes were made and whether an 

aspect of the procedure being used is a temporary one. Thus, the HSI should support such discriminations 

(Barnes et al., 1996). 

9.9 Training Program Development 

(1) The training program should address the role of the operators to assure that they remain in control of the 

CBP system and independently supervise it. 

Discussion: Operators need to understand the overall purpose of a procedure, and should stay cognitively 

involved with its progress. They should be trained to be in control and to question recommended steps 

apparently inconsistent with the overall procedure goals (Blackman and Nelson, 1988). While operators need to trust the CBP (Collier, 1996), overreliance on its information can be a concern. For example, while operators using PBPs monitor a variety of plant indications, operators using CBPs may not feel the 

need to look at other sources of information in the CR and, thus, may miss important indications that are 

not present in the CBP (O'Hara, Stubler, and Higgins, 1996). 

(2) The knowledge, skills, and abilities that users will require to interact successfully with the CBP should be 

specified by the designers. 

Discussion: The demands of CBPs on personnel may be different than those of PBPs (Barnes et al., 

1996); designers need to fully analyze personnel requirements so training can address them. 

(3) The training requirements for using CBPs should be specified and incorporated into a training program which should cover both initial and ongoing training. Training should consider the design features, 

functions, and limitations of CBPs (such as the potential for incorrect assessments). 

Discussion: Training was identified as critical to CBP use and may require significant changes (NRC CBP 

Workshop). 

(4) The training program should inform operators about limited and complete failures of the CBP. Operators 

should be trained to determine when to override CBP evaluations and advice. They should be able to 

manage the transition to PBPs when CBPs are lost and move back to them when system function is 

restored. 

Discussion: Research showed that operators may be reluctant to override the CBP's advice, and may 

believe the computerized procedure even when it is wrong (Blackman and Nelson, 1988). Operators 

should be trained on making such judgements, and on what to do when they disagree with the CBP 

(Jeffroy and Charron, 1997). 

(5) The training program should address the importance of teamwork and communication when the CBP is 

being used. 

Discussion: The NRC CBP Workshop and several investigations of CBPs (e.g., Roth and O'Hara, 1998) 

correlated the importance of the crew's communications and interactions to their reliability; this should be 

addressed in CBP training. 

(6) For CBP systems used for EOPs only, the compatibility with day-to-day operations needs to be evaluated 

to ensure that the system can be easily understood and used. 

Discussion: IAEA (1994) noted that the CBP's compatibility with day-to-day operations needs to be 

considered. Systems that have very limited use in normal operations and are only used under infrequent 

special circumstances may have limited success. Thus, they noted that "...this may be an issue for CBP 

systems that are designed for emergency systems only" (p. 31).

NUREG/CR-6634 9- 10
9 CBP DESIGN PROCESS REVIEW GUIDANCE

(7) The means by which the CBP will be introduced and implemented in an operating plant should be 

specified. 

Discussion: Spurgin et al. (1990) discussed the gradual introduction of EOPTS at Kuosheng. It was first 

introduced into the training simulator (1) as an aid to instructors to track the operators' responses to 

accidents, (2) as a training tool for crews to examine accident-response strategies, and (3) as a tool to be 

used by crews in responding to accidents. During this time, the correctness of EOPTS was examined, and 

several errors in the PBPs were discovered. Roth and O'Hara (1998) indicated the importance of the 

method of implementing the system to the operators' subsequent confidence in it. 

9.10 Human Factors Verification and Validation 

(1) A verification and validation (V&V) plan should be established. 

Discussion: The complexity and formalization of the plan will depend on the scope of the CBP systems. 

In some cases, it may be part of the overall CBP implementation plan discussed in Section 9.1 (2) above. 

In others, it should be more formal and extensive. NUREG-0700 and NUREG-0711 give more detailed 

guidance on V&V plans. 

(2) V&V of procedures should ensure that the CBPs are technically correct and usable. Three types of design 

considerations must be addressed when evaluating their usability: (1) HFE design standards and 

guidelines for human-computer interaction, (2) HFE design guidelines for the format of text instructions 

and graphics used in presenting procedural information, and (3) the unique influence of site-specific 

characteristics and users. The acceptability of a CBP system cannot be determined without documentation 

that all three types of considerations were acceptably addressed. 

Discussion: V&V refers to methods of ensuring that the CBPs are technically correct and usable. The 

scope, methods, timing, and composition of the V&V team are important to the success of the system. For 

CBP systems that select the task instructions to be displayed, V&V will be a more complex process than 

for paper procedures or CBPs that are not integrated with the plant's information display and control 

systems. 

(3) An independent review team should conduct V&V. 

Discussion: The NRC's analysis of EOPs (Lapinsky, 1989) noted that the lack of an independent review 

to assure technical accuracy and usability was associated with procedure deficiencies that may negatively 

impact the development and use of CBPs; therefore, a CBP development program should provide for 

independent review. The exact skills needed on the review team will depend on the scope of the CBP 

systems. Appendix A of NUREG-07 11 gives the broadest range of skills likely to be required; typical ones 

are those in Subsections 2, 4, 6, 7, 9, and 10 - namely, Systems Engineering, I&C Engineering, Human 

Factors Engineering, Plant Operations, Plant Procedure Development, and Personnel Training. 

(4) CBP evaluations should use several crews and scenarios. They also should use operator-in-the-loop 

evaluations to ensure that the system's objectives are achieved and that any transitions between CBP and 

PBP are accomplished. 

Discussion: For details, see Converse, 1995; EPRI, 1993a; Orvis and Spurgin, 1996. 

(5) Each CBP EOP procedure should be evaluated in the plant's simulator. 

Discussion: Use of simulation to evaluate CBP systems was identified as an important component of 

determining their acceptability (EPRI, 1993a). 

(6) Operators should be able to detect CBP errors and failures.

9-11 NUREG/CR-6634
9 CBP DESIGN PROCESS REVIEW GUIDANCE

Discussion: The NRC CBP Workshop and several investigations of CBPs (e.g., Roth and O'Hara, 1998) 

identified the importance of crews being able to detect errors and failures of the CBP system. 

(7) The V&V should establish that crew performance is not degraded as compared with that reached using 

PBPs. 

Discussion: Teamwork is essential to a defense-in-depth approach to safely operating the plant and any 

failings can compromise it. 

(8) The criteria for accepting the CBP should be specified. 

Discussion: The specific criteria by which the CBP will be accepted should reflect considerations of task

performance criteria, such as task time and error rate, determined by analysis, and of criteria based on a 

comparison to performance with PBP systems.

NUREG/CR-6634 9-12
10 GUIDELINES FOR CBP HFE DESIGN REVIEW 

The guidelines in this section follow the characterization of CBP systems discussed in Section 4. They also reflect 

the findings from our literature review of the effects of CBPs on crew performance, specifically the identification of 

functions discussed in Section 5.6.2 as reflecting near-term approaches to implementing CBPs. According to the 

HSI design review procedure described in Part 1 of NUREG-0700, Rev. 1, the first step in a design review is to 

select a subset of guidelines relevant to the unique aspects of the particular design. There is a wide range of CBP 

designs, and some may not include all of the characteristics and functions in these guidelines; the reviewer will 

have to determine, case-by-case, the importance of CBP features that are included in the guidelines but not part of 

the system being reviewed. This determination should be based on considerations of the specific purposes and 

goals of that CBP system. 

As described in Section 6.2, guidelines were developed from the findings and source materials reviewed in 

Section 5. These guidelines were constructed in the standard format adopted in NUREG-0700, Rev. 1 (see 

Section 6.2 of this report), and organized into the following sections: 

* Representation of Procedures 

* Functionality of Procedures 

* Management and Support of Procedures 

* CBP Hardware 

* CBP Procedure Backup 

* Integration of CBPs with Other HSI Components 

These new guidelines will be integrated into NUREG-0700, Rev. I. 

Guidelines may specify that some identified aspect of CBPs needs to be "evaluated." NUREG-0700 defines general 

approaches to methods of evaluation and for identifying criteria. Since the guidance in this section will be 

incorporated into NUREG-0700, those methods and criteria are not repeated below. 

10.1 Representation of Procedures 

10.1.1 Identification of Procedures 

10.1.1-1 Procedure Title and Identification Information 

Each procedure should contain identifying information including title, procedure number, revision 

number, date, and organizational approval. 

ADDITIONAL INFORMATION: This information helps the user establish the appropriate context for 

using the procedure. 

Discussion: This guideline was developed for application to CBPs as an extension of HFE guidance for 

paper-based procedures, e.g., NUREG-0899 (NRC, 1982), and from lessons learned discussed in Barnes, 

Desmond, Moore, and O'Hara (1996).

10- 1 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

10.1.1-2 High-Level Goals 

Each procedure should state its high-level goals and applicability, including its procedure category, 

e.g., emergency or abnormal. 

ADDITIONAL INFORMATION: Information should be given allowing the user to understand the 

purpose or goal of a series of steps and supporting the user's assessment of the success of the procedure in 

achieving its safety goal. 

Discussion: Procedure objectives need to be emphasized to increase operator's awareness of the high-level 

goals (Bozec et al., 1990; Wieringa, Moore, and Barnes, 1992). This guideline is an application of the 

High-Level Design Review Principle of Situation Awareness (see Appendix B). 

10.1.2 Basic Steps 

10.1.2-1 Concise Steps 

Procedure steps should be concise. 

ADDITIONAL INFORMATION: Steps should be designed to communicate information clearly and 

unambiguously so that they can be easily understood and interpreted without error. 

Discussion: This guideline was developed for application to CBPs as an extension of HFE guidance for 

paper-based procedures, e.g., NUREG-0899 (NRC, 1982), and from lessons learned discussed in Barnes, 

Desmond, Moore, and O'Hara (1996). This guideline is an application of the High-Level Design Review 

Principle of Simplicity of Design (see Appendix B). 

10.1.2-2 Short Sentences 

Procedure steps should be written as short sentences. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-3 Active Voice 

Procedure steps should be written in active voice. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-4 Positive Commands 

Procedure steps should be written as positive commands. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-5 Simple Wording 

Short, simple words from standard American English should be used. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-6 Standard Punctuation 

Punctuation should conform to standard American English usage. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps.

NUREG/CR-6634 10-2
10 CBP HFE DESIGN REVIEW GUIDELINES

10.1.2-7 Consistent Word References 

Words, phrases, and equipment names and numbers should be used consistently within and among 

procedures, drawings, other HSIs, and equipment labels. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Consistency (see Appendix B). 

10.1.2-8 Abbreviations and Acronyms 

Abbreviations and acronyms should be used consistently and limited to those well known to the users. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of User Model Compatibility and Consistency (see Appendix B). 

10.1.2-9 Units of Measures 

Numerical information should include units of measure. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-10 Numerical Precision 

Numbers should be specified at the appropriate precision. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Task Compatibility (see Appendix B). 

10.1.2-11 Number Ranges 

Ranges of numbers should be specified, rather than error bands. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Situation Awareness, Task Compatibility, and Cognitive 

Workload (see Appendix B). 

10.1.2-12 Use Arabic Numerals 

Arabic numerals should be used. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-13 Spelled Numbers 

Numbers that are spelled out should be consistently spelled under the same conditions. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.2-14 Presentation of Conditional Steps 

Conditional steps should be shown in traditional text formats following the guidance in Appendix B of 

NUREG-0899. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps.

10-3 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

10.1.2-15 Specification of Preconditions for Steps 

The procedure should specify any conditions that must be met before an action can be undertaken. 

ADDITIONAL INFORMATION: Information about preconditions in the procedure should be located so 

that users read the information before acting. Information given in other locations may be overlooked, or 

require additional actions to retrieve it, which may be distracting and time consuming. Further, if 

conditions are implied, users may easily miss or misinterpret them. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Situation Awareness, Task Compatibility, Timeliness, Feedback, 

and Response Workload (see Appendix B). 

10.1.3 Warnings, Cautions, Notes, and Supplementary Information 

10.1.3-1 Parallel Display with Procedure Step 

The warnings and cautions applicable to a single step (or to a series of steps) should be displayed when 

the step(s) is on the screen. 

ADDITIONAL INFORMATION: Displaying warnings and cautions at the same time as their associated 

procedure steps will help ensure that users read the information when they evaluate the step. Information 

provided elsewhere may be overlooked, or may require retrieval by distracting and time-consuming 

actions. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Situation Awareness, Timeliness, and Response Workload (see 

Appendix B). 

10.1.3-2 Position Before Action Steps 

Warnings, cautions, and notes should be presented so that they will be read before the applicable action 

steps. 

ADDITIONAL INFORMATION: Displaying warnings, cautions, and notes before action steps will help 

ensure that users will read the information before taking action. Information provided in other places may 

be overlooked or may be distracting and time consuming to retrieve. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Situation Awareness, Task Compatibility, Feedback, and 

Timeliness (see Appendix B). 

10.1.3-3 Action References 

Warnings, cautions, and notes should not include implied or actual action steps. 

ADDITIONAL INFORMATION: Actions should be specified in procedure steps only. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Task Compatibility (see Appendix B). 

10.1.3-4 Distinction from Other Procedure Elements 

Warnings, cautions, and notes should be uniquely presented, so that they are easily distinguished from 

each other and from other display elements. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Task Compatibility and Organization of HSI Elements (see 

Appendix B).

NUREG/CR-6634 10-4
10 CBP HFE DESIGN REVIEW GUIDELINES

10.1.3-5 Supplementary Information 

All supplementary information (such as tables and figures) required for a procedure step and available to 

the CBP should be shown on the screen concurrently with the step, or on another easily viewed display. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Situation Awareness, Task Compatibility, and Response 

Workload (see Appendix B). 

10.1.4 Lists 

10.1.4-1 Appropriate Application of Lists 

Groups of three or more related items. (e.g., actions, conditions, components, criteria, systems) should be 

presented as a list. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Organization of HSI Elements (see Appendix B). 

10.1.4-2 Distinction from Other Procedure Elements 

Formatting should be used to differentiate items in a list from other procedure elements. 

ADDITIONAL INFORMATION: See additional information in Guideline 10.1.2-1, Concise Steps. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. 

10.1.4-3 Identification of Precedence 

The presence or absence of precedence among items in lists should be indicated. 

ADDITIONAL INFORMATION: It should be clear to users whether some items take precedence over 

others. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Situation Awareness (see Appendix B). 

10.1.4-4 List Overviews 

Overviews should introduce each list. 

ADDITIONAL INFORMATION: An example of an overview is "Ensure that all of the following tests 

were completed:" 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Situation Awareness (see Appendix B). 

10.1.4-5 Assuring Users' Attention 

The method for assuring that each item in a list has received the users' attention should be consistent. 

ADDITIONAL INFORMATION: For example, an electronic checklist may be provided so that operators 

can check off items they have attended to. If operators proceed before all items are checked off, the CBP 

may alert them to the unchecked items. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Task Compatibility (see Appendix B). 

10.1.5 Organization of Procedures 

10.1.5-1 Hierarchical, Logical Organization 

The procedures should be organized in a hierarchical, logical, consistent manner.

10-5 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

ADDITIONAL INFORMATION: Organization will make it easier for users to see the relationships 

among procedures. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Logical/Explicit Structure and Consistency (see Appendix B). 

10.1.5-2 Organization of Procedure Steps 

Each procedure should be organized into sections of related steps. 

Discussion: See discussion for Guideline 10.1.2-I, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Logical/Explicit Structure and Consistency (see Appendix B). 

10.1.6 Formatting and Screen Layout 

10.1.6-1 Organization Format of Procedures 

The procedure's format should reflect its organization. 

ADDITIONAL INFORMATION: Formatting methods to indicate the organization of a procedure may include the use of headings or colors to distinguish parts of the procedure. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Logical/Explicit Structure and Consistency (see Appendix B). 

10.1.6-2 Format of Procedures 

A consistent format should be used to display procedures. 

ADDITIONAL INFORMATION: Whether procedures are presented in text, flowchart, or otherwise, a 

consistent approach across procedures will facilitate using and moving between multiple procedures. 

Discussion: There is insufficient research to specify one format over another for presenting CBPs. 

Further, it is important that CBPs are consistent with paper procedures. However, whatever format is 

used, consistency supports the rapid use of information when moving within and between procedures, and 

enables operators to form expectancies which can reduce the workload of finding information. This will 

also speed procedure use and reduce errors. This guideline is an application of the High-Level Design 

Review Principles of Logical/Explicit Structure and Consistency (see Appendix B). 

10.1.6-3 Partitioning Procedures 

A consistent approach to partitioning procedures should be used. 

ADDITIONAL INFORMATION: Partitioning refers to how a procedure is organized to be displayed on 

the VDU screen. For example, it may be divided into distinct pages, and users would navigate from one 

to the next. Alternatively, it may be presented as one continuous display that the user scrolls. 

Discussion: Unlike PBPs, CBPs are viewed through the limited display area of one or more VDUs. Thus, regardless of format, the designer must decide whether the procedure will appear as a continuous 

scrollable display or be divided into discrete pages. This guideline is an application of the High-Level 

Design Review Principles of Task Compatibility, Logical/Explicit Structure, and Consistency (see 

Appendix B). 

10.1.6-4 Organization of Display Screen 

Each display screen should locate information and HSI features consistently. 

ADDITIONAL INFORMATION: When the information and features, such as procedure steps, controls, 

and navigation aids are consistently located, users' performance improves because expectations can guide the search for information, and reduce the time and workload associated with finding it. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principles of Logical/Explicit Structure and Consistency (see Appendix B).

NUREG/CR-6634 10-6
10 CBP HFE DESIGN REVIEW GUIDELINES

10.1.6-5 Continuously Presented Procedure Information 

The procedure's title and identification should be continuously presented. 

ADDITIONAL INFORMATION: This information helps set the context for the overall procedure within 

which its steps are interpreted. It is especially important when more than one procedure can be open at 

one time. 

Discussion: One concern identified with CBP systems is the loss of awareness of the context in which 

procedures are used, e.g., high-level safety goals and plant status (NRC CBP Workshop; Roth and O'Hara, 

1998; Spurgin et al., 1990). The identifying information maintains focus on the way in which individual 

steps are interpreted, especially when multiple procedures are in use. This guideline is an application of 

the High-Level Design Review Principle of Situation Awareness (see Appendix B). 

10.1.6-6 Continuously Presented Status of High-Level Goals 

The status of high-level procedure goals should be continuously presented. 

ADDITIONAL INFORMATION: This information helps set the overall context in which procedure steps 

are interpreted. Continuous presentation of high-level goal status, such as status of critical safety 

functions, will facilitate users' awareness of them, particularly when more than one procedure is open 

simultaneously. 

Discussion: The loss of awareness of the context in which procedures are used, e.g., high-level safety 

goals and plant status is a concern with CBPs (NRC CBP Workshop; Roth and O'Hara, 1998; Spurgin, et 

al. 1990). Awareness of high-level goals is important to interpreting individual steps and for determining 

which procedure is appropriate. Roth and O'Hara (1998) also observed that the most significant series of 

information needed by operators on loss of the CBP was critical safety function status. This guideline is 

an application of the High-Level Design Review Principles of Situation Awareness and Timeliness (see 

Appendix B). 

10.2 Functionality of Procedures 

10.2.1 Supervision and Control of Procedures 

10.2.1-1 Users' Control of Procedure Path 

Users should be in control of the sequence of steps that are followed. 

ADDITIONAL INFORMATION: Most procedures have specifically defined steps that have to be 

performed sequentially, and others that can be varied at the operator's discretion; CBPs should identify 

which one is applicable. However, operators should have the flexibility to move around within the 

procedure, so that they can check and make verifications. 

Discussion: The CBP guidance level should leave the operators in the loop so they retain control and are 

the final authority (EPRI, 1993a; Dien et al., 1991). This guideline is an application of the High-Level 

Design Review Principles of Cognitive Compatibility and Situation Awareness (see Appendix B). 

10.2.1-2 Users' Control of Pace of Procedures 

Users should be in control of the pace at which procedure steps are followed. 

ADDITIONAL INFORMATION: Operators need to maintain situation awareness of procedure-related 

decisions. To accomplish this, they must be in control of the pace at which steps are followed. 

Discussion: The operator should retain control and should be the final authority (EPRI, 1993a; Dien et 

al., 1991). This guideline is an application of the High-Level Design Review Principles of Cognitive 

Compatibility and Situation Awareness (see Appendix B). 

10-7 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

10.2.1-3 Understandability of Analysis of Procedure Steps 

The methods by which CBPs analyze procedure steps should be consistent with the methods by which 

users analyze steps in procedure logic steps, so that the results are understandable. 

ADDITIONAL INFORMATION: Users must be able to judge the acceptability of the CBP's advice and 

recommendations. 

Discussion: To maintain their role of system supervisors, operators need to be able to understand and 

evaluate the appropriateness of procedure analyses. The CBP should not require the operator to conform 

to its method of analysis (Bernard, 1989). This guideline is an application of the High-Level Design 

Review Principles of Task Compatibility and Situation Awareness (see Appendix B). 

10.2.1-4 Users' Verification of CBP Information 

The users should be able to verify the system's assessment of plant status. 

ADDITIONAL INFORMATION: This verification includes process parameters, equipment status, 

analysis of procedure step logic, and evaluation of cautions. Any analysis done by the CBP should be 

accessible to users for review. 

Discussion: To maintain their role of system supervisors, operators need to be able to access information 

enabling them to determine the appropriateness of procedure information. This guideline is an 

application of the High-Level Design Review Principles of Task Compatibility and Situation Awareness 

(see Appendix B). 

10.2.1-5 Users' Override of CBP 

Users should be able to override any CBP information, calculation, evaluation, or assessment. 

Discussion: Operators should be able to override a course of action suggested or recommended by a CBP 

system. This is necessary for situations in which the operator has access to information that is not 

available to the CBP, the CBP's guidance is too strict, or when the CBP uses out-of-date in formation 

(Bozec et al., 1990). This guideline is an application of the High-Level Design Review Principles of 

Cognitive Compatibility and Situation Awareness (see Appendix B). 

10.2.2 Monitoring and Assessment of Procedures 

10.2.2-1 Automatic Identification of Procedures 

The CBP should alert users when entry conditions to a procedure are satisfied. 

ADDITIONAL INFORMATION: This capability will help users determine the appropriate procedures for 

the existing plant situation. 

Discussion: This capability was identified as being beneficial to crew reliability (Orvis and Spurgin, 

1996). This guideline is an application of the High-Level Design Review Principles of Situation 

Awareness and Cognitive Workload (see Appendix B). 

10.2.2-2 Automatic Monitoring of Plant Parameters and Equipment Status 

The CBP should automatically provide accurate and valid information on the values of parameters and 

status of equipment, when they are available to the system. 

ADDITIONAL INFORMATION: It should be clear to operators what specific information is used as the 

source of these actual values and states. 

Discussion: Supporting cognitive functions, such as obtaining parameter values (monitoring) may reduce 

the demands on attentional resources and working memory and enable the operator to focus more on 

evaluating higher-level procedure goals. It may also help solve PBP issues. This capability was identified 

as being beneficial to the crew's reliability (Orvis and Spurgin, 1996; Pirus and Chambon, 1997; Niwa et 

al., 1996). Further, presenting plant parameters and status in procedure steps is a URD requirement

NUREG/CR-6634 10-8
10 CBP HFE DESIGN REVIEW GUIDELINES

(EPRI, 1993a). This guideline is an application of the High-Level Design Review Principles of Situation 

Awareness and Cognitive Workload (see Appendix B). 

10.2.2-3 Frequent Monitoring 

The CBP should frequently monitor procedure-defined parameters. 

ADDITIONAL INFORMATION: Frequent monitoring, such as twice a second, promptly notifies users of 

status changes. 

Discussion: The continuous updating of plant parameters and status is identified as a URD requirement 

(EPRI, 1993a). This guideline is an application of the High-Level Design Review Principles of Situation 

Awareness, Cognitive Workload, and Timeliness (see Appendix B). 

10.2.2-4 Automatic Calculation of Procedure-Referenced Values 

The system should undertake calculations, such as subcooling margin, that are required when using 

procedures. 

Discussion: The capability to perform calculations was identified as an important feature of CBPs (Roth 

and O'Hara, 1998; Barnes et al., 1996). This guideline is an application of the High-Level Design Review 

Principle of Cognitive Workload (see Appendix B). 

10.2.2-5 Analysis of Step Logic 

The CBP should evaluate the logic of each procedure step and show the results to the user. 

ADDITIONAL INFORMATION: Procedure steps often contain logical relationships; for example, 

actions are to be performed if an identified set of conditions exists. The analysis of these logical 

relationships must be carefully verified to avoid underspecification. This occurs when the logic used to 

resolve a procedure step is too simplified, and does not address all of the considerations that operators do 

when evaluating the step. 

Discussion: Supporting cognitive functions, such as comparing actual parameter values to reference 

values (resolution of procedure step logic) may reduce the demands on attentional resources and working 

memory, and enable the operator to focus on evaluating high-level procedure goals. This CBP capability 

was identified as a major benefit and one which helped operators to follow the procedures correctly, and to 

interpret the logical statements that are a part of the procedure steps (Spurgin et al., 1990). It also was 

thought to improve the crew's reliability (Orvis and Spurgin, 1996; Moieni and Spurgin, 1993b). This 

guideline is an application of the High-Level Design Review Principles of Situation Awareness and 

Cognitive Workload (see Appendix B). However, while this is a potentially powerful feature, it must be 

used cautiously. Some procedural details are intentionally left relatively abstract because they require the 

operator's judgement on the basis of local knowledge; e.g., knowledge of equipment availability and status 

at the current time. While computerization can increase the detail, this could inadvertently change the 

procedure's context and the operator's interpretation of it. Thus, underspecification of the logic can be an 

issue (O'Hara, Stubler, and Higgins, 1996; Roth and O'Hara, 1998). Further, the CBP is not fully aware 

of what operators are doing nor of their intentions (Blackman and Nelson, 1988). 

10.2.2-6 Continuous Analysis of Non-Current Step Logic 

Steps of continuous applicability, time-dependent steps, and process-dependent steps should be monitored 

by the CBP and the user should be alerted when conditions in those steps become effective. 

ADDITIONAL INFORMATION: The analysis must be carefully verified to avoid underspecifying its 

logic. The alert should not automatically remove the user's current display. Instead, it should be 

presented as a supplemental display or as an alert. 

Discussion: See discussion of the previous guideline. In addition, operators prefer that procedures not 

automatically reset or return to a previous step when there is a change in process status; instead automatic

10- 9 NUREGICR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

monitoring of previous steps and indications of a change in their status is preferred (Bozec et al., 1990). 

This guideline is an application of the High-Level Design Review Principles of Situation Awareness and 

Cognitive Workload (see Appendix B). 

10.2.2-7 Coding of Logical Analysis 

When procedure's step logic indicates a violation of the step, the information should be coded to make 

that step more salient to users. 

Discussion: Handling of multiple procedures is easier when the relevant information in each is 

highlighted. When operators transition from one to another, the highlighted information directs them to 

the appropriate location (Pirus and Chambon, 1997). This guideline is an application of the High-Level 

Design Review Principle of Situation Awareness (see Appendix B). 

10.2.2-8 Analysis of Cautions 

The conditions described in cautions should be automatically monitored by the CBP system, and the user 

should be alerted when the caution is in effect. 

ADDITIONAL INFORMATION: Evaluating cautions and alerting users to their applicability will ensure 

that users will read the information at the appropriate time, and reduce the chance that it may be 

overlooked. The conditions for cautions must be established with care such that the logic is not 

underspecified. 

Discussion: Supporting cognitive functions, such as monitoring caution conditions and comparing their 

reference values, may reduce the demands on attentional resources and working memory and enable the 

operator to attend more to the higher-level procedure goals. Alerting operators to applicable cautions will 

help ensure that they are not overlooked. This guideline is an application of the High-Level Design 

Review Principles of Situation Awareness and Cognitive Workload (see Appendix B). 

10.2.2-9 Coding Applicable Cautions 

CBPs should use coding to indicate when a caution is in effect. 

ADDITIONAL INFORMATION: Coding techniques, such as color coding, may be used to enhance the 

salience of important information. 

Discussion: This guideline is an application of the High-Level Design Review Principle of Situation 

Awareness (see Appendix B). 

10.2.2-10 Users' Acknowledgment of Procedure Analyses 

Users should make some form of acknowledgment of procedure steps and recommendations for 

terminations and transitions. 

ADDITIONAL INFORMATION: As an example, operators may acknowledge that a step is satisfied by 

depressing the "Return" key, or clicking on an onscreen acceptance button. Such acknowledgment helps 

the operators to maintain awareness of the procedure's status. 

Discussion: The CBP guidance level should leave the operator in the loop, so they retain control and are 

the final authority (EPRI, 1993 a; Dien et al., 1991). This guideline is an application of the High-Level 

Design Review Principles of Cognitive Compatibility and Situation Awareness (see Appendix B). 

10.2.2-11 Identification of User Input Requirements 

The CBP should provide users with clear, timely indications when they need to input any information not 

available to it. 

ADDITIONAL INFORMATION: CBPs may rely on users to for process parameter values, equipment 

status (such as whether a valve is open or closed), analyses of logic steps where users' judgement is 

involved, or to assess any conditions not within the capability of the CBP.

NUREG/CR-6634 10- 10
10 CBP HFE DESIGN REVIEW GUIDELINES

Discussion: While the CBP monitors the system through the I&C, operators must provide some 

information. Failures to do so can lead to incorrect assessments and guidance from the CBP (Jeffroy and 

Charron, 1997). This guideline is an application of the High-Level Design Review Principles of 

Logical/Explicit Structure and Timeliness (see Appendix B). 

10.2.2-12 Adjustable Level of Detail 

Users should be able to choose the level of detail with which procedures are presented. 

ADDITIONAL INFORMATION: While plant practices on using procedures may be specified by 

management, there may be flexibility in the level of detail that can be provided. For example, users may 

want less detail when a procedure step is satisfied. Alternatively, a user may choose to see all of the 

individual evaluations leading to the conclusion that the step was satisfied. This must be done with care 

so that it does not affect the interpretation of procedure information. Also, users should be trained as to 

how and when to vary levels of detail. 

Discussion: Procedural guidance can be used more efficiently when CBPs can adjust the level of detail for 

operators with varying familiarity with the tasks, components, systems, and processes defined in the 

procedures. This may also help address a deficiency of PBPs. It was identified as a desirable feature of 

CBPs by many studies (NRC CBP Workshop; Dien et al., 1991). However, providing too much detail 

should be avoided (Bozec et al., 1990; Roth and O'Hara, 1998), especially for experienced operators (Niwa 

et al., 1996). This guideline is an application of the High-Level Design Review Principles of Flexibility 

(see Appendix B). 

10.2.2-13 Context-Specific Guidance 

Procedure guidance should be context sensitive where possible. 

ADDITIONAL INFORMATION: For example, the CBP system should not indicate an action to start a 

pump when it can determine that the pump is already running. 

Discussion: A general problem observed with COSSs is that the information is "acontextual," i.e., their 

guidance had little reference to the current situation (Dien and Montmayeul, 1995). For CBPs, this 

problem can be corrected by supporting procedure sensitivity to the current situation (Niwa et al., 1996). 

Removing information inappropriate to the current situation and which is, therefore, potentially 

distracting and uses up valuable time, will help operators to concentrate on important information. This 

guideline is an application of the High-Level Design Review Principles of Cognitive Compatibility and 

Situation Awareness (see Appendix B). 

10.2.2-14 Assessment of High-Level Goal Status 

The CBP should continuously assess and present the status of higher-level safety goals, such as critical 

safety functions, and alert the user to any challenges. 

Discussion: Supporting cognitive functions, such as comparing parameter values to goal-reference values, 

may reduce the demands on attentional resources and working memory and enable the operator to better 

attend to determining the success of the procedure in achieving the higher-level goals. Alerting operators 

to possible challenges will help ensure that they will not be overlooked. The availability of safety-goal 

status is important to operators' overall assessment of the procedure (Roth and O'Hara, 1998). This 

guideline is an application of the High-Level Design Review Principles of Situation Awareness and 

Cognitive Workload (see Appendix B). 

10.2.2-15 Assessment of Conditions Terminating a Procedure 

The CBP should automatically identify when conditions are met for transitioning or exiting from a 

procedure.

10- 11 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

ADDITIONAL INFORMATION: This capability will help users determine when procedures they are 

using are no longer appropriate for the existing situation. 

Discussion: By helping users determine when procedures become inappropriate for the existing situation, 

the chances of operators delaying actions identified in the appropriate procedure are reduced. This 

guideline is an application of the High-Level Design Review Principles of Situation Awareness and 

Cognitive Workload (see Appendix B). 

10.2.3 Monitoring Users' Actions 

10.2.3-1 Monitoring Users 

User responses to procedures should be monitored and recorded by the CBP. 

ADDITIONAL INFORMATION: Monitoring information on users' input to information requested by the 

procedure and their subsequent actions is necessary if the CBP is to properly assess appropriate procedural 

pathways. 

Discussion: CBPs should be designed to maintain information on what the crew is doing that is relevant 

to implementing the procedure (Blackman and Nelson, 1988). To evaluate procedure steps the operators 

must be aware of the information being analyzed by the CBP. To the extent that the CBP system has 

information on users' actions, it can perform this task more effectively. 

10.2.3-2 Alert Users to Deviations in Procedure 

Users should be alerted if their input is incorrect, or when their actions are not consistent with CBP 

evaluations. 

ADDITIONAL INFORMATION: The alert should be advisory and not discourage the user's actions. 

This feature must be supported with training, so users are not reluctant to go against the CBP's 

evaluations. 

Discussion: EPRI suggested that CBPs should have software to verify the operators' decisions. While the 

operator should retain control and authority as to how to proceed, disagreements should be logged 

automatically (EPRI, 1993a). Alerting crews to possible unintentional deviations from the procedure was 

identified as a potential improvement to the crew's reliability by enabling them to recover from mistakes 

(Orvis and Spurgin ,1996; Moieni and Spurgin, 1993b) and to catch "local" errors (Jeffroy and Charron, 

1997). Other studies also identified this as a desirable CBP feature (NRC CBP Workshop). By alerting 

operators, they can decide if that is what they want to do (Pirus and Chambon, 1997). This guideline is an 

application of the High-Level Design Review Principles of Error Tolerance in Control, Feedback, and 

Situation Awareness (see Appendix B). However, care must be taken to assure that operators are not 

reluctant to deviate from the CBP. As Jeffroy and Charron (1997) noted, there are situations where 

operators may disagree with the CBP's recommendations and may find it hard to disagree with the 

procedures, especially when the level of detail in the CBP is high. 

10.2.4 Planning and Implementation 

10.2.4-1 Display of Action Status 

The status of procedure-related actions should be displayed by the CBP. 

Discussion: This feature is a potential improvement to crew reliability (Orvis and Spurgin, 1996). This 

guideline is an application of the High-Level Design Review Principles of Situation Awareness and 

Feedback (see Appendix B).

NUREG/CR-6634 10- 12
10 CBP HFE DESIGN REVIEW GUIDELINES

10.2.4-2 Timing of Procedures 

The CBP's timing, such as status update rates, screen changes, and navigation features, should be 

consistent with the time demands of the task. 

Discussion: The timing of CBP responses affects operators' performance. Spurgin et al. (1990) indicated 

that an SRO requested ADS initiation twice thinking it had not been presented after the first request; the 

misunderstanding was due to the delay in the CBPs update of ADS status (Spurgin et al., 1990). This 

guideline is an application of the High-Level Design Review Principles of Timeliness and Feedback (see 

Appendix B). 

10.3 Management and Support of Procedures 

10.3.1 Path Monitoring 

10.3.1-1 Monitoring Step Status 

There should be an indication of whether or not a step was completed. 

ADDITIONAL INFORMATION: The indication can be manual or automatic, depending on whether the 

CBP has the specific criteria and information to determine this. 

Discussion: CBPs can keep track of what steps have been completed, using check boxes. This can be 

manual or automatic, depending on whether the CBP has the specific criteria and information to 

determine whether a step was completed. Completion also can be time stamped to facilitate post-hoc 

incident analysis (Niwa, Hollnagel, and Green, 1996). This guideline is an application of the High-Level 

Design Review Principles of Cognitive Workload and Situation Awareness (see Appendix B). 

10.3.1-2 Alert User to Incomplete Procedure Steps 

Users should be alerted to incomplete procedure steps. 

ADDITIONAL INFORMATION: The alert should be advisory and not discourage the crew's actions. 

Discussion: CBPs should monitor whether procedure steps were not fully completed and notify the crew if 

further action is needed (Orvis and Spurgin, 1996; Moieni and Spurgin, 1993b). This guideline is an 

application of the High-Level Design Review Principles of Error Tolerance in Control, Feedback, and 

Situation Awareness (see Appendix B). 

10.3.1-3 Coding Current Location 

The current procedure step(s) should be indicated. 

Discussion: Automatic place keeping is a CBP feature that can improve the crew's reliability (Orvis and 

Spurgin, 1996), especially when using multiple procedures. This guideline is an application of the High

Level Design Review Principle of Cognitive Workload (see Appendix B). 

10.3.1-4 Automatic Path Monitoring 

The pathway taken through procedures should be stored and made available to users. 

ADDITIONAL INFORMATION: A history should be maintained and available for display on request. 

Step completion can be time stamped to facilitate post-hoc incident analysis (Niwa, Hollnagel, and Green, 

1996). 

Discussion: CBPs can keep track of what steps have been completed; this can be manual or automatic 

depending on whether the CBP has the specific criteria and information to make this determination 

(Niwa, Hollnagel, and Green, 1996). This guideline is an application of the High-Level Design Review 

Principle of Cognitive Workload (see Appendix B).

10- 13 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

10.3.1-5 Indication of Multiple Active Procedures 

The user should be informed when multiple procedures or multiple procedure steps are to be followed 

concurrently. A list of all currently active procedures should be available. 

ADDITIONAL INFORMATION: It may be helpful for the list of active procedures to include start and 

stop times for the procedures in use. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps. This guideline is an application of the 

High-Level Design Review Principle of Cognitive Workload (see Appendix B). 

10.3.2 Navigation 

10.3.2-1 Flexible Navigation 

Navigation support should allow users to freely and easily move between procedure steps, to other parts of 

the same procedure, and to other procedures. 

ADDITIONAL INFORMATION: Users should not be forced to access procedures in a fixed sequence of 

the procedure nor should their access to supporting information be limited. (See also the additional 

information on Guideline 10.2.1-1.) 

Discussion: Navigation within one procedure, or among multiple procedures and related supporting 

information, can be time consuming and error prone (O'Hara, Stubler, and Higgins, 1996). Collier (1996) 

noted that the CBP system should not overly structure the operator's movement through the procedure but 

should offer flexibility for operators to skip steps or skim over them quickly. Operators need to move 

easily between procedures and support information (Niwa, Hollnagel, and Green, 1996), in part to make 

up for the insufficiencies of procedures (Dien et al., 1991). In addition, flexibility improves the crew's 

reliability (Orvis and Spurgin, 1996). This guideline is an application of the High-Level Design Review 

Principles of Response Workload and Flexibility (see Appendix B). 

10.3.2-2 Support Parallel Access to Information 

The CBP should have the ability to access more than one piece of information at once. 

Discussion: Hoecker et al. (1994) and Hoecker and Roth (1996) found that the workload associated with 

CBPs can increase when the operators cannot access needed information in parallel. Similarly, CBPs can 

significantly improve the crew's performance in comparison with PBPs if they allow simultaneous access 

to multiple procedures (Wilhelmsen et al., 1992). This guideline is an application of the High-Level 

Design Review Principles of Cognitive Workload and Response Workload (see Appendix B). 

10.3.2-3 Navigational Links to Related Information 

Navigational links to cross-referenced information and to notes, cautions, warnings, reference material, 

and communication and help facilities should be provided. 

ADDITIONAL INFORMATION: Techniques such as hyperlinks can expedite navigation to information 

material cross-referenced in a procedure or its supporting material. 

Discussion: Navigation to and from cross-referenced material can be time consuming, distracting, and 

error prone. Computer support for these transitions can reduce the workload associated with these tasks. 

This guideline is an application of the High-Level Design Review Principles of Response Workload and 

Flexibility (see Appendix B). 

10.3.2-4 Access to Contingency Actions 

Users should be able to easily access appropriate contingency actions. 

Discussion: See discussion for Guideline 10.1.2-1, Concise Steps.

NUREG/CR-6634 10- 14
10 CBP HFE DESIGN REVIEW GUIDELINES

10.3.3 Help 

10.3.3-1 Explanation Facilities 

CBPs should have facilities to enable the user to determine how CBP functions are performed. 

ADDITIONAL INFORMATION: When CBPs support users' decision making, such as offering advice on 

how to select procedures, analyze step logic or follow procedure paths, users should be able to query the 

basis for the advice. Cooperative dialogue enables the user to better understand and utilize the system. 

Discussion: In general, COSSs often are not designed to be sufficiently observable. That is, they do not 

clarify their reasoning basis, nor have adequate communication facilities to enable operators to question 

and verify system performance. Guidance may be given without sufficient communication about what led 

to its issuance, what parameters were analyzed, and what sequence of reasoning was followed. When the 

reasoning process is shown, it may conflict with that of the operators, i.e., it may be based on the 

designer's theoretical understanding and not the operators' practical experience (Dien and Montmayeul, 

1995; IAEA, 1994; Malin et al., 1991a; Roth, Bennett, and Woods, 1987). Explanations of the rationale 

for procedure steps have been identified as a necessary CBP feature (Niwa et al., 1996). This guideline is 

an application of the High-Level Design Review Principle of User Guidance and Support (see 

Appendix B). 

10.3.3-2 Help Facilities 

Help for performing procedure specified activities should be provided. 

Discussion: Information should be given to help operators carry out procedure steps. For example, a help 

facility could provide information as to how a control action should be carried out (Niwa, Hollnagel, and 

Green, 1996). This guideline is an application of the High-Level Design Review Principle of User 

Guidance and Support (see Appendix B). 

10.3.3-3 Note Taking 

There should be a way for users to record their notes and comments in the CBP. 

Discussion: Procedures have gaps because they do not cover all possible situations and actions. CBPs can 

help eliminate'them by allowing operators to log omissions in an on-line database which then could be 

accessed to identify improvements to the procedure (NRC CBP Workshop). This guideline is an 

application of the High-Level Design Review Principle of Response Workload (see Appendix B). 

10.4 CBP Hardware 

Guidance for CBP hardware, including VDUs, printers, computer input devices, is part of the CBP review, and is 

available in NUREG-0700. An additional consideration is discussed below. 

10.4-1 Number of VDUs 

The number of VDUs on which CBP information is displayed should be sufficient to provide all the 

procedure-related information needed for a procedure step, including cautions and reference material. 

Discussion: VDUs can create a keyhole effect and the requirement for potentially distracting interface 

management tasks. This guideline is an application of the High-Level Design Review Principles of 

Situation Awareness and Task Compatibility (see Appendix B).

10- 15 NUREG/CR-6634
10 CBP HFE DESIGN REVIEW GUIDELINES

10.5 Backup for CBP Procedures 

10.5-1 Paper-Based Procedure Availability 

PBPs should be available in the event of CBP failure. 

Discussion: PBPs will enable operators to perform safety-related tasks in situations where the CBP system 

is malfunctioning or has failed. This guideline is an application of the High-Level Design Review 

Principle of Error Tolerance and Control (see Appendix B). 

10.5-2 Consistency of PBPs and CBPs 

The content and presentation of procedure information in PBPs and CBPs should be consistent. 

ADDITIONAL INFORMATION: Smooth transfer between CBPs and PBPs and vice versa will be 

facilitated by the degree to which their formatting is consistent; this also will facilitate training in 

procedure use. 

Discussion: The hard-copy procedures should be consistent in format and content with the CBPs. EPRI 

noted that their consistency will minimize the training burden and lower the potential for errors and 

misunderstandings. This consideration is especially important when the hard-copy procedures have to be 

used as a backup (EPRI, 1993a). This guideline is an application of the High-Level Design Review 

Principles of Consistency and Error Tolerance and Control (see Appendix B). 

10.5-3 Support for Transfer to PBPs 

Upon transfer to PBPs, a means should be provided to support the user's determination of currently open procedures, location in the procedures, completed and not completed steps, and currently monitored steps. 

ADDITIONAL INFORMATION: When the CBP is lost, it may be difficult for operators to reconstruct 

this information from memory. Therefore, the operator should be supported in making a safe, easy 

transition. For example, a CBP system might automatically print out a status sheet with this information 

once every minute so that if it fails, the operator can retrieve the latest sheet and use it to establish the 

crew's tasks for using PBPs. 

Discussion: Operators may be in multiple procedures when the CBP fails. For each, the CBP may have 

been monitoring progress, monitoring and evaluating steps of continuous applicability and other steps in 

the background. Providing these supports is one of the CBP benefits that reduce the operators' cognitive 

workload in remembering this information. When the CBP is lost, it may be difficult for operators to 

reconstruct this information from memory. Therefore, some means should be provided to support this 

transfer. This guideline is an application of the High-Level Design Review Principles of Cognitive 

Workload and Error Tolerance and Control (see Appendix B). 

10.6 Integration of CBPs with Other HSI Components 

10.6-1 Consistency with Other HSI Conventions 

The detailed CBP design should be fully consistent with the rest of the HSI. 

ADDITIONAL INFORMATION: HSI features for format and functionality (such as labeling, acronyms, 

dialog conventions, use of colors, and input devices) should be consistent between the CBP and other HSI 

components. Consistency may be a special consideration when reviewing "off-the-shelf" systems. 

Discussion: Lack of consistency between CBPs and the other HSI resources was identified as an 

important consideration (NRC CBP Workshop). Any such inconsistency can degrade the operator's 

performance and increase the likelihood of errors. Thus, inconsistency was identified as a potential source 

of risk and reduced performance reliability (Niwa et al., 1996). This guideline is an application of the 

High-Level Design Review Principle of Consistency (see Appendix B).

NUREG/CR-6634 10- 16
APPENDIX A

Human Factors Engineering 

Program Review Model (NUREG-071 1) 

Element 8: Procedure Development 

(This appendix reflects the Element 8 changes that 

will be made in Revision 1 to NUREG-0711)

NUREG/CR-6634
ELEMENT 8: PROCEDURE DEVELOPMENT 

A.1 Background 

In the nuclear industry, the development of procedures historically was considered the responsibility of individual 

utilities, but the rationale for including a procedure development element in NUREG-07 11 is that procedures are 

an essential component of the HSI design, and should be derived from the same design processes and analyses as 

other HSIs (e.g., displays, controls, operator aids) and evaluated in the same way. Technically detailed, emergency 

operating procedures (EOPs) were an improvement instituted after the accident at Three Mile Island (TMI) to 

support safe operations. First, the NPP owners groups developed generic technical guidance (GTG) and utilities 

then produced EOPs based on the GTG. Thus, procedure development programs were conducted by the individual 

utilities and were not part of HSI design activities. However, since procedures were developed after the design of 

the plant HSI (e.g., control room), they were essentially retrofitted to suit the existing interface. Further, since they 

were established by individual utilities, their development and final implementation varied greatly. As a result, 

human factors problems existed, and the identification, access, interpretation, and validation of procedures 

remained troublesome for years in several plants, as shown by the NRC EOP inspection series (Lapinsky, 1989; 

Galletti and Sutthoff, 1992). In addition, inconsistencies between procedures and the HSI have been a source of 

difficulty for operators. 

For new plant designs and advanced reactors, these problems should clearly be addressed and solved during the 

design process. To accomplish this objective, GTG and, if possible, procedures should be developed as part of the 

same design process as that for other components of the HSI to ensure their full integration into the HSI. The same 

human factors analyses, such as task analysis, should be used to guide the design of the control panel, as well as 

procedure development. The same human factors principles should be applied to both aspects of the interface to 

ensure complete integration and consistency. Further, procedures should be evaluated in conjunction with the HSI; 

procedures are a significant aspect of system verification and validation (Element 10). 

A.2 Objective 

The objective of this review is to ensure that the applicant's procedure development program will result in 

procedures that guide human interactions with plant systems and control plant-related events. Human engineering 

principles and criteria should be applied, along with all other design requirements, to develop technically accurate, 

comprehensive, explicit, easy to utilize, and validated procedures. 

A.3 Applicant Submittals 

The applicant should provide the following documents for staff review: implementation plan, analysis-results 

report, and HFE design team evaluation report. Section 1.4.4 (of NUREG-071 I) describes these submittals. 

In addition, GTG and draft procedures should be available for review. 

A.4 Review Criteria 

(1) The following procedures are within the scope of the element: 

. GTG for EOPs

A-1I NUREG/CR-6634
APPENDIX A

* Plant and system operations (including startup, power, and shutdown operations) 

• Abnormal and emergency operations 

• Preoperational, startup, and surveillance tests 

* Alarm response 

(2) The basis for procedure development should include: 

• Plant design bases 

* System-based technical requirements and specifications 

• Results of task analyses 

* Risk-important human actions identified in the HRA/PRA 

* Initiating events to be considered in the EOPs, including those events in the design bases 

* GTG for EOPs 

(3) A writer's guide should be developed to establish the process for developing technical procedures that are 

complete, accurate, consistent, and easy to understand and follow. The guide should contain sufficiently 

objective criteria so that resulting procedures are consistent in their organization, style, and content. The 

guide should be used for all procedures within the scope of this element. It should provide instructions on 

the procedures' content and format, including writing action steps and specifying acceptable acronym lists 

and terms. 

(4) The content of the procedures should incorporate the following elements: 

* Title 

* Statement of applicability 

* References 

• Prerequisites 

* Precautions (including warnings, cautions, and notes) 

* Limitations and actions 

* Required human actions 

• Acceptance criteria 

* Checkoff lists

NUREG/CR-6634 A-2
APPENDIX A

(5) In addition to the general procedure elements identified in Criterion 4 above, GTG should be symptom 

based with clearly specified entry conditions. 

(6) All procedures should be verified and validated; a review should ensure they are correct and can be carried 

out. They should be finally validated in a simulation of the integrated system, as part of the verification 

and validation activities described in Element 10. 

(7) An analysis should determine the impact of providing computer-based procedures, CBPs, (either partial or 

complete) and specify where such an approach would improve the use of procedures and reduce related 

errors by the operating crew. Justification for using CBPs rather than paper procedures should be given. 

An analysis should be made and documented of alternatives in the event of loss of CBPs. 

(8) A plan for maintaining procedures and controlling updates should be developed. 

(9) The physical means by which operators access and use procedures, especially during operational events, 

should be evaluated as part of the HFE design process. This criterion generally applies to both hard-copy 

and computer-based procedures, although the types of issues differ somewhat for them. For example, the 

process should address the storage of procedures, ease of operator access to the correct procedures, and 

laydown of hard-copy procedures for use in the control room, remote shutdown facility, and local control 

stations. 

(10) The following documents may be used as guidance (per Section 1.4.4): 

NUREG-0800, Rev. 1: Standard Review Plan, 1984 (NRC). 

NUREG-0899: Guidelines for the Preparation of Emergency Operating Procedures, 1982 (NRC). 

NUREG-1358: Lessons Learned From the Special Inspection Program for Emergency Operating 

Procedures, 1989 (NRC). 

NUREG- 1358: Lessons Learned From the Special Inspection Program for Emergency Operating 

Procedures, Supplement 1, 1989 (NRC). 

NUREG/CR-5228: Techniques for Preparing Flowchart Format Emergency Operating Procedures, 

Volumes 1 and 2, 1989 (NRC - Barnes et al.). 

NRC Regulatory Guide 1.33, Rev. 2: Quality Assurance Program Requirements, 1978 (NRC). 

ANS 3.2-1994: Administrative Controls and Quality Assurance for the Operational Phase of NPPs, 1994 

(American Nuclear Society). 

BNL TR E2090-T4-2-9/96: Preliminary Review Criteria for Evaluating Computer-Based Procedures, 

1996 (Barnes et al.)

A-3 NUREG/CR-6634
APPENDIX B

High-Level Design Review Principles from 

NUREG-0700, Rev. 1

NUREG/CR-6634
HIGH-LEVEL DESIGN REVIEW PRINCIPLES FROM NUREG-0700 

The design of human-system interfaces (HSIs) should support the operating personnel's primary task of monitoring 

and controlling the plant, without imposing an excessive workload associated with using the HSI (manipulating 

windows, selecting displays, and navigating, for example). The HSI also should support the recognition, tolerance, 

and recovery from any human errors. Guidelines for reviewing human factors engineering designs help to ensure 

that these goals are achieved. As part of the guidance development for NUREG-0700, Rev. 1, a set of "high-level" 

design review principles was developed representing the generic HSI characteristics necessary to support personnel 

performance. They were used to develop many detailed review guidelines in Part 2 NUREG-0700 (O'Hara, Brown, 

and Nasta, 1996 discuss their use). The high-level principles also were used in formulating guidelines for 

computer-based procedures. 

The 18 principles are divided into four categories: general principles, primary task design, secondary task control, 

and task support. The categories and the principles that underlie them are described below. 

B.1 General Principles 

These principles ensure that the HSI design supports personnel safety, and is compatible with their general 

cognitive and physiological capabilities. 

• Personnel Safety - The design should minimize the potential for injury and exposure to harmful materials. 

* Cognitive Compatibility - The operators' roles should consist of purposeful, meaningful tasks that enable them 

to maintain familiarity with the plant and maintain a level of workload that is not so high as to lower 

performance, but sufficient to maintain vigilance. 

* Physiological Compatibility - The design of the interface should reflect consideration of human physiological 

characteristics, including visual/auditory perception, biomechanics (reach and motion), characteristics of 

motor control, and anthropometry. 

* Simplicity of Design - The HSI should represent the simplest design consistent with function and task 

requirements. 

* Consistency - There should be a high degree of consistency between the HSI, the procedures, and the training 

systems. At the HSI, the way the system functions and appears to the operating crew always should be 

consistent, reflect a high degree of standardization, and be fully consistent with procedures and training. 

B.2 Primary Task Design 

These principles support the operator's primary tasks of monitoring and detection, situation assessment, response 

planning, and response implementation. 

* Situation Awareness - The information presented to the users by the HSI should be correct, rapidly recognized, 

and easily understood (e.g., "direct perception" or "status at a glance" displays) and support the higher-level 

goal of their awareness of the system's status. 

* Task Compatibility - The system should meet the requirements of users in performing their tasks (including 

operation, safe shutdown, inspection, maintenance, and repair). Data should be presented in forms and

B- 1 NUREG/CR-6634
APPENDIX B

formats appropriate to the task (including the need to access confirmatory data or raw data in the case of 

higher-level displays), and control options should encompass the range of potential actions. There should be 

no unnecessary information or control options. 

User Model Compatibility - All aspects of the system should be consistent with the users' mental models 

(understanding and expectations about how the system behaves from training, use of procedures, and 

experience). All aspects of the system also should be consistent with established conventions (i.e., expressed 

in customary, commonplace, useful and functional terms, rather than abstract, unusual or arbitrary forms, or in 

forms requiring interpretation). 

Organization of HSI Elements - The organization of all aspects of the HSI (from the elements in individual 

displays, to individual workstations, to the entire control room) should be based on the users' requirements and 

should reflect the general principles of organization by importance, frequency, and order of use. Critical safety 

function information should be available to the entire operating crew in dedicated locations to ensure its 

recognition, and to minimize data search and response. 

Logical/Explicit Structure - All aspects of the system (formats, terminology, sequencing, grouping, and 

operator's decision-support aids) should reflect an obvious logic based on task requirements or some other non

arbitrary rationale. The relationship of each display, control, and data-processing aid to the overall task and 

function should be clear. The structure of the interface and its associated navigation aids should make it easy 

for users to recognize where they are in the data space, and should enable them to rapidly access data not 

currently visible (e.g., on other display pages). The way the system works, and is structured, should be clear to 

the user. 

Timeliness - The system's design should take into account users' cognitive processing capabilities as well as 

process-related time constraints to ensure that tasks can be performed within the required time. Information 

flow rates and control performance requirements that are too fast or too slow could diminish performance. 

Controls/Displays Compatibility - Displays should be compatible with the requirements for data entry and 

control. 

° Feedback - The system should provide useful information on its status, permissible operations, errors and 

error recovery, dangerous operations, and validity of data. 

B.3 Secondary Task Control 

These principles minimize secondary tasks, i.e., tasks personnel must perform when interfacing with the system that are not directed to the primary one. Examples include managing the interface, such as navigation through 

displays, manipulating windows, and accessing data. Performing secondary tasks detracts from the crew's primary 

tasks, so the demands of secondary tasks must be controlled. 

Cognitive Workload- The information presented by the system should be rapidly recognized and understood; 

therefore, the system should minimize requirements for making mental calculations or transformations and 

using recall memory (recalling lengthy lists of codes, complex command strings, information from one display 

to another, or lengthy action sequences). Raw data should be processed and presented in directly usable form 

(although raw data should be accessible for confirmation).

NUREG/CR-6634 B-2
APPENDIX B

Response Workload- The system should require a minimum number of steps to accomplish an action; e.g., 

single versus command keying, menu selection versus multiple command entry, single input mode (keyboard, 

mouse) versus mixed mode. In addition, the system should not require redundant data to be entered, nor the 

re-entry of information already in the system, or information the system can generate from already resident 

data. 

B.4 Task Support 

These principles address the characteristics of the HSI that support its use by personnel, such as providing (1) HSI 

flexibility so tasks can be accomplished in more than one way, (2) guidance for users, and (3) mitigation of errors. 

Flexibility - The system should give the user multiple means to carry out actions (and verify automatic 

actions) and permit displays and controls to be formatted in a configuration most convenient for the task. 

However, flexibility should be limited to situations where it is advantageous for task performance (such as to 

accommodate different levels of experience of the users); it should not be provided for its own sake because 

there is a tradeoff between flexibility and the increase in interface management workload (which detracts from 

monitoring and operations). 

* User Guidance and Support - The system should provide an effective "help" function. Informative, easy-to

use, and relevant guidance should be given on-line and off-line to help the user understand and operate the 

system. 

Error Tolerance and Control - A fail-safe design should be provided wherever failure can damage equipment, 

injure personnel, or inadvertently operate critical equipment. Therefore, the system should generally be 

designed such that a user's error will not have serious consequences. The negative effects of errors should be 

controlled and minimized. The system should offer simple, comprehensible notification of the error, and 

simple, effective methods for recovery.

B- 3 NUREG/CR-6634
NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER 

(2.89) (Assigned by NRC. Add Vol., Supp.. Rev.. 

NRCM 1102, and Addendum Numbers, if any.) 

3201. 3202 

BIBLIOGRAPHIC DATA SHEET NUREG/CR-6634 

(See instuctions on the reverse) BNL-NUREG-52564 

2. TITLE AND SUBTITLE 

Computer-Based Procedure Systems: Technical Basis and Human Factors Review 3. DATE REPORT PUBLISHED 

Guidance MONTH YEAR 

March 2000 

4. FIN OR GRANT NUMBER 

J-6012 

5. AUTHOR(S) 6. TYPE OF REPORT 

John M. O'Hara, James C. Higgins, William F. Stubler, and Joel Kramer Technical 

7. PERIOD COVERED (inclusive Dates) 

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (if NRC, provide Division, Office or Region, U.S. Nuclear Regulatory Commission, and mailing address, if contractor; provide 

name and mailing address.) 

Department of Advanced Technology 

Brookhaven National Laboratory 

Upton, NY 11973-5000 

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (if NRC, type Same as above'; if contractor, provide NRC Division, Office or Region, U.S. Nuclear Regulatory Commission, 

and mailing address.) 

Division of Systems Analysis and Regulatory Effectiveness 

Office of Nuclear Regulatory Research 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555-0001 

10. SUPPLEMENTARY NOTES 

J. Kramer, NRC Project Manager

11. ABSTRACT (200 words or less) 

Plant procedures are instructions to guide operators in monitoring, decision making, and controlling nuclear power plants. While 

plant procedures historically have been paper-based, computer-based procedures (CBPs) are being developed to support procedure 

use. CBPs have a range of capabilities that may support operators and reduce demands associated with paper procedures. The 

objective of this study was to establish human factors review guidance for CBP systems based on a technically valid methodology. 

While the study mainly addressed emergency operating procedures, much of the guidance developed applies to other types of 

procedures. First, a CBP characterization was developed for describing their key design features including both procedure 

representation and functionality. Then, the research on CBPs and related areas was reviewed. This information provided the 

technical basis on which the guidelines for design review were developed. The review guidelines address both the design process 

and the implementation of CBP systems. For some aspects of CBPs the technical basis was insufficient to develop guidance; these 

aspects were identified as issues to be addressed in future research.

12 KEY WORDSIDESCRIPTORS (List words or phrases that will assist researchers in tocating the report.) 13. AVAILABILITY STATEMENT 

Control rooms, human factors engineering, human-system interface, man-machine systems, Unlimited 

reactor safety, reactor operators, test and evaluation, human-factors review criteria 14. SECURITY CLASSIFICATION 

(This Page) Unclassified 

(This Report) Unclassified 

15. NUMBER OF PAGES 

16. PRICE

NRC FORM 335 (2-49)
Federal Recycling Program
NUREG/CR-6634 MARCH 2000

UNITED STATES 

NUCLEAR REGULATORY COMMISSION 

WASHINGTON, D.C. 20555-0001

SPECIAL STANDARD MAIL 

POSTAGE AND FEES PAID 

USNRC 

PERMIT NO. G-67

COMPUTER-BASED PROCEDURE SYSTEMS: TECHNICAL BASIS AND 

HUMAN FACTORS REVIEW GUIDANCE