Nuclear Safety
NEA/SEN/SIN/WGHOF(2012)1
March 2012www.oecd-nea.org

Summary of Survey andWorkshop Results on Areasof Research in Human Factors
for the Design and Operationof New Nuclear Plant
Technology
Final Report

N U C L E A R

E N E R G Y

A G E N C Y

Unclassified

NEA/SEN/SIN/WGHOF(2012)1

Organisation de Coopération et de Développement Économiques
Organisation for Economic Co-operation and Development

08-Mar-2012________________________________________________________________________________________________________English text only
NUCLEAR ENERGY AGENCY

NEA/SEN/SIN/WGHOF(2012)1
Unclassified

Committee on the Safety of Nuclear Installations
Working Group on Human and Organisational Factors

SUMMARY OF SURVEY AND WORKSHOP RESULTS ON AREAS OF RESEARCH IN HUMAN
FACTORS FOR THE DESIGN AND OPERATION OF NEW NUCLEAR PLANT TECHNOLOGYFINAL REPORT

JT03317313

English text only

Complete document available on OLIS in its original format
This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation ofinternational frontiers and boundaries and to the name of any territory, city or area.

NEA/SEN/SIN/WGHOF(2012)1

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENTThe OECD is a unique forum where the governments of 34 democracies work together to address the economic, socialand environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand and to help
governments respond to new developments and concerns, such as corporate governance, the information economy and thechallenges of an ageing population. The Organisation provides a setting where governments can compare policy
experiences, seek answers to common problems, identify good practice and work to co-ordinate domestic and international
policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, Chile, the Czech Republic, Denmark, Estonia,
Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Luxembourg, Mexico, the Netherlands,
New Zealand, Norway, Poland, Portugal, the Republic of Korea, the Slovak Republic, Slovenia, Spain, Sweden,
Switzerland, Turkey, the United Kingdom and the United States. The European Commission takes part in the work of theOECD.
OECD Publishing disseminates widely the results of the Organisation‟s statistics gathering and research on economic,
social and environmental issues, as well as the conventions, guidelines and standards agreed by its members.

This work is published on the responsibility of the OECD Secretary-General.
The opinions expressed and arguments employed herein do not necessarily reflect the official
views of the Organisation or of the governments of its member countries.

NUCLEAR ENERGY AGENCYThe OECD Nuclear Energy Agency (NEA) was established on 1 February 1958. Current NEA membership consists of
30 OECD member countries: Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, France,
Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Luxembourg, Mexico, the Netherlands, Norway, Poland,
Portugal, the Republic of Korea, the Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Turkey, the United Kingdom
and the United States. The European Commission also takes part in the work of the Agency.
The mission of the NEA is:– to assist its member countries in maintaining and further developing, through international co-operation, thescientific, technological and legal bases required for a safe, environmentally friendly and economical use of
nuclear energy for peaceful purposes, as well as– to provide authoritative assessments and to forge common understandings on key issues, as input to government
decisions on nuclear energy policy and to broader OECD policy analyses in areas such as energy and sustainable
development.
Specific areas of competence of the NEA include the safety and regulation of nuclear activities, radioactive waste
management, radiological protection, nuclear science, economic and technical analyses of the nuclear fuel cycle, nuclear law
and liability, and public information.
The NEA Data Bank provides nuclear data and computer program services for participating countries. In these andrelated tasks, the NEA works in close collaboration with the International Atomic Energy Agency in Vienna, with which it
has a Co-operation Agreement, as well as with other international organisations in the nuclear field.

This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation ofinternational frontiers and boundaries and to the name of any territory, city or area.
Corrigenda to OECD publications may be found online at: www.oecd.org/publishing/corrigenda.
© OECD 2012
You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia
products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgment of the OECD as source
and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to rights@oecd.org. Requests for
permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at
info@copyright.com or the Centre français d'exploitation du droit de copie (CFC) contact@cfcopies.com.

2

NEA/SEN/SIN/WGHOF(2012)1

COMMITTEE ON THE SAFETY OF NUCLEAR INSTALLATIONSThe Committee on the Safety of Nuclear Installations (CSNI) of the OECD Nuclear Energy Agency(NEA) is an international committee made up of senior scientists and engineers. It was set up in 1973 todevelop, and co-ordinate the activities of the Nuclear Energy Agency concerning the technical aspects ofthe design, construction and operation of nuclear installations insofar as they affect the safety of such
installations. The Committee’s purpose is to foster international co-operation in nuclear safety among theOECD member countries.
The CSNI constitutes a forum for the exchange of technical information and for collaboration
between organisations, which can contribute, from their respective backgrounds in research, development,
engineering or regulation, to these activities and to the definition of the programme of work. It also reviews
the state of knowledge on selected topics on nuclear safety technology and safety assessment, including
operating experience. It initiates and conducts programmes identified by these reviews and assessments inorder to overcome discrepancies, develop improvements and reach international consensus on technicalissues of common interest. It promotes the co-ordination of work in different member countries including
the establishment of co-operative research projects and assists in the feedback of the results to participatingorganisations. Full use is also made of traditional methods of co-operation, such as information exchanges,
establishment of working groups, and organisation of conferences and specialist meetings.
The greater part of the CSNI current programme is concerned with the technology of water reactors.
The principal areas covered are operating experience and the human factor, reactor coolant systembehaviour, various aspects of reactor component integrity, the phenomenology of radioactive releases inreactor accidents and their confinement, containment performance, risk assessment, and severe accidents.
The Committee also studies the safety of the nuclear fuel cycle, conducts periodic surveys of the reactor
safety research programmes and operates an international mechanism for exchanging reports on safety
related nuclear power plant accidents.
In implementing its programme, the CSNI establishes co-operative mechanisms with NEA Committee
on Nuclear Regulatory Activities (CNRA), responsible for the activities of the Agency concerning theregulation, licensing and inspection of nuclear installations with regard to safety. It also co-operates withNEA Committee on Radiation Protection and Public Health and NEA Radioactive Waste Management
Committee on matters of common interest.
************The opinions expressed and the arguments employed in this document are the responsibility of theauthors and do not necessarily represent those of the OECD.
Requests for additional copies of this report should be addressed to:
Nuclear Safety Division
OECD Nuclear Energy Agency
Le Seine St-Germain
12 boulevard des Iles
92130 Issy-les-MoulineauxFrance3

NEA/SEN/SIN/WGHOF(2012)1

WORKING GROUP ON HUMAN AND ORGANISATIONAL FACTORS (WGHOF)

The main mission of the NEA Working Group on Human and Organisational Factors (WGHOF) is toimprove the understanding and treatment of human and organisational factors within the nuclear industry
in order to support the continued safety performance of nuclear installations and improve the effectivenessof regulatory practices in member countries.
The working group shall report to the Committee on the Safety of Nuclear Installations (CSNI) and assistthat committee with its work. The working group shall prepare an integrated plan for its activities
consistent with the mandate as well as proposed CSNI safety issues and shall update this at regularintervals. The group will also collaborate with, and respond to requests from, the Committee for Nuclear
Regulatory Activities (CNRA) and other CSNI working groups. The WGHOF programme of work will beapproved by CSNI.
In delivering its mission, the WGHOF will seek to address the challenges identified in the joint CSNI/CNRAStrategic Plan and to implement the CSNI Operating Plan. The WGHOF will:Constitute a forum for exchange of information and experience about safety-relevant human andorganisational issues in member countries, thereby promoting co-operation and maintaining aneffective and efficient network of experts.

Identify and prioritise current and emerging human and organisational safety issues.

Identify human and organisational factors methodologies and practices where further work andresearch are needed.

Identify those issues which appear most suitable to be addressed by the WGHOF in a co-ordinated
way across the international community.

Facilitate international convergence on safety issues related to human and organisational factors and,
where practicable, seek to develop a shared understanding and common positions on important issues.
Compare, and where possible, benchmark, practices and methodologies currently applied by member
countries in the assessment of safety-relevant human and organisational issues.
Prepare technical reviews of human and organisational factors work where such reports are needed forfurther development and to assist the application of human and organisational factor methods inmember countries.
Collaborate with, and support cross-cutting initiatives proposed by, other CSNI/CNRA groups. Ensurethat CSNI, CNRA and other organisations are consulted as appropriate when potential cross-cutting
work on human and organisational factors is proposed by the WGHOF.
Sponsor specialist meetings, workshops and other means of fostering international collaboration with
nuclear and other industries, where appropriate, to further its objectives.

4

NEA/SEN/SIN/WGHOF(2012)1

ACKNOWLEDGEMENT

The contents of this Working Group on Human and Organisational Factors (WGHOF) technical noteare principally the product of a workshop hosted by the US NRC in Washington DC in March 2010.
WGHOF members would like to acknowledge the following individuals without whose dedicated effort,
this technical note would not have been possible. These individuals were responsible for conducting thesurvey, organizing the workshop and preparing this technical note:
Idaho National Laboratory, USA
Julius J Persensky
Jeffrey Joe
Robert E Richards
WGHOF members would also like to thank the following individuals who contributed to the delivery ofthis technical note:
US Nuclear Regulatory Commission
Valerie Barnes
Michelle Gonzalez
Finally, WGHOF members would like to thank all those who presented and participated in the workshop in
Washington.

5

NEA/SEN/SIN/WGHOF(2012)1

6

NEA/SEN/SIN/WGHOF(2012)1

RESEARCH ON HUMAN FACTORS FOR THE DESIGN AND
OPERATION OF NEW NUCLEAR PLANT TECHNOLOGY:
FINAL REPORT

Executive Summary

The nuclear community is currently at a stage where existing reactor control stations are undergoing
various forms of modernization, new reactors are being built in many countries with computer-based
control rooms, and advanced reactors are being designed through international cooperation to support
power generation for decades to come. With the introduction of advanced plants, we will see new reactor
and system designs, new tools to support plant personnel, and changes to nuclear power plant (NPP)
staffing configurations. The concepts of operation and maintenance for this new generation of plants arelikely to be quite different from those employed in today‟s plants. It is important that the potential impactof these developments is evaluated and understood by prospective operators and regulators responsible fordetermining the acceptability of new designs to support human performance in maintaining plant safety.
The introduction of new technology is viewed as having promise for improving the safe and efficient
operation of NPPs. To ensure the appropriate application of technology to support human performance andplant safety, it is important to evaluate the technological advances in terms of both potential negative andpositive effects. Research described in this paper can provide the technical basis to help ensure that thebenefits of new technology are realized and that the potential negative effects are minimized.
The impetus for the current effort grew out of a Nuclear Energy Agency, Committee on the Safety of
Nuclear Installations, Working Group on Human and Organizational Factors, Technical Opinion Paper(TOP) titled, “Research on Human Factors in New Nuclear Plant Technology” [NEA/CSNI/R(2009)7],
which identified eight broad topic areas that warrant further research:
1. Operating Experience (OpEx) from New and Modernized Plants.
2. Evolving Concepts for the Operation of Nuclear Power Plants.
3. The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems.
4. Management of Unplanned, Unanticipated Events.
5. Human System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions.
6. Complexity Issues in Advanced Systems.
7. Organizational Factors – Safety Culture.
8. Human Factors Engineering (HFE) Methods and Tools.
The WGHOF undertook two efforts to respond to recommendations in the TOP. The first was thedevelopment and implementation of a survey to identify the level of interest in performing research projects
related to the eight research topic areas and to assess the level of interest in collaborating in such research.
The second effort was implementation of a workshop to discuss how the use of new human-systemtechnology in the operation of NPPs may affect reliable human performance and plant safety.

7

NEA/SEN/SIN/WGHOF(2012)1

8

NEA/SEN/SIN/WGHOF(2012)1

SURVEY RESULTS

The general results from the survey indicated that every research and development topic surveyed hassome level of ongoing or planned activity. Some research topics showed a significant amount of ongoingor planned activity, e.g., safety culture and capturing and using operating experience (OpEx) for
application to safety culture issues; activities being planned are in the areas of effects of new technologies,
new concepts of operation, and multi-unit reactors. Respondents also described the facilities they are using
to perform their research. Numerous published documents were identified as were specific researchers.

9

NEA/SEN/SIN/WGHOF(2012)1

10

NEA/SEN/SIN/WGHOF(2012)1

WORKSHOP OUTCOMES

Proposed research projects for each topic area discussed at the workshop included, by topic area.
Operating Experience from New and Modernized Plants
The proposals from the operating experience topic area focus on gathering and analyzing OpEx fromoperating plants, new plant construction, and plant modernizations to identify further lessons learned thatare unique to the implementation of new technologies. The proposals also suggest the need to collectinformation on the application of new technologies from other industries. Limitations relative to
information availability and personnel qualifications were noted.
Evolving Concepts for the Operation of Nuclear Power Plants
The focus of the concepts of operation topic was on identifying effective tools to design and evaluate
alternative concepts of operation to determine when to encourage shared control between humans andautomation and when to assign final authority to the humans by studying of operator roles that separate themonitoring of plant status and supervision of automation.
The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
The concept of agents as part of the operational context is introduced in the topic of automation. An
agent can be a person, a piece of equipment, soft- or firm-ware, or a computer operated surveillance system.
Suggested research topics include: appropriate allocation of responsibilities to each agent in the system,change of responsibilities, use of modeling to simulate teamwork, trust in automation, reliability of theautomated system, management of automation failures and degraded conditions, and drivers of the movetoward automation.
Management of Unplanned, Unanticipated Events
Research into whether the ability to handle the unexpected is a personality factor or a function of
better mental models that support projections, diagnosis, and anomaly detection was suggested. Proposals
included establishing technical bases for: design and review guidelines for systems to support detection,state awareness, and diagnosis of unexpected or low probability events; developing appropriate procedures;developing training programs to aid in response to unanticipated events; and establishing a databasefocused on unanticipated events and event precursors in nuclear and other industries.
Human System Interface Design Principles for Supporting Operator Cognitive Functions
The focus of research proposals for this topic area was on developing a better understanding of what
creates cognitive burden in NPP operations and how to best deploy computer technologies to facilitateproceduralized tasks, communications, and teamwork. A specific proposal was to develop detailed guidance
and acceptance criteria for addressing integrated system validation.
Complexity Issues in Advanced Systems
Research was proposed to define complexity with the goal of developing a tool to measure system andperceived complexity so that a better understanding of the effects of different types of complexity can be11

NEA/SEN/SIN/WGHOF(2012)1
achieved. This would include determining the impact of complexity on human performance and ways tominimize the perceived complexity through new HSI design. Part of this research could determine how tobalance increased technical reliability (and complexity) with better human reliability (which may be
negatively impacted by this complexity),
Organizational Factors – Safety Culture
The research proposals for the organizational factors topic area include: creating a compendium ofmajor safety culture initiatives, including case studies of high reliability organizations; determining if
effective safety culture looks different at different stages of an organization‟s life cycle; looking forstrengths and weaknesses in various organizational cultures and what approaches have been successful;research on effective assessment tools with more clearly defined standards.
Human Factors Engineering Methods and Tools
Several related methods and tools (M&T) projects were proposed to address the strengths andlimitations of new HFE M&T and criteria that should be applied to evaluating the acceptability of the
M&T for use in advanced reactor control station designs. The research should include: Who should use
what tools, when and for what purpose? What are the merits to joint development and application? And
how might the M&T be deemed acceptable? This could lead to validation of human performance models
for the nuclear arena, where criteria for validation and regulatory acceptance criteria are established alongwith integration into a risk assessment framework.
Workshop Chair’s Summary
The Chair of the workshop summarized the outcome of the meetings as a series of six cross-cutting
themes: Drivers for change; Scope of change; the Design process; Design principles; Test, evaluation, andreview; and Implementation. The Chair went on to conclude that research projects can be pursued in all of the
topic areas, that the state of the art is constantly advancing, that there was a consensus among the participants
regarding the appropriate approaches to research, and that a lack of human factors expertise in the nuclear
domain is a limiting factor. She went on to recommend: enhancement for reporting, disseminating, and using
operating experience; that a model of an integrated design process be developed, including the use of
projective tools; areas where clear acceptance criteria for using methods/tools and evaluation results can be
identified and established; and that work be continued in the area of safety culture.

12

NEA/SEN/SIN/WGHOF(2012)1

CONCLUSIONS AND RECOMMENDATIONS

Based on the results of the survey and workshop and combined with the information in the TOP, thereport has a number of recommendations, both for future WGHOF activities and for those organizations
that participated in the survey and workshop. The first recommendation relates to prioritization.
Information considered from this report suggests that individual organizations have their own needs anddrivers for the implementation of advanced HSI systems. Therefore, WGHOF will forward the TOP andthis report to all participating organizations for their use in establishing research priorities.
A second recommendation aimed at participating organizations is that participating organizations
make use of the information provided in the report to seek out and establish collaborative arrangements to
conduct research of common interest. To the extent allowed by its charter, WGHOF may facilitatecommunications among parties.
WGHOF should consider a proposal to collaborate with WGOE and WGRNR to form a task force tostudy the enhanced collection of operating experience information in HOF areas related to new nuclear
technologies. This effort could be initiated through a jointly sponsored activity proposal.
WGHOF will encourage researchers, designers, and regulators of member nations, and otherworkshop participants, to determine the availability or development of test, evaluation, and review
methods and tools that provide quantitative information, for which clear acceptance criteria can beestablished. For the human factors of advanced control systems there is interest in alarm systems,
procedures (especially computerized procedures), HSIs, automation measures and acceptance criteria for
integrated system validation. Further, there is interest in how these issues affect concepts of operation, andapplications to control room modernization, new control stations for new reactors, and the impact of small
modular reactor designs on human performance. WGHOF proposes to follow developments in these areasand consider workshops that focus on specific topics as the state of knowledge advances and new issues
are identified.
Finally, WGHOF plans to closely follow the EURATOM, Man-Machine-Organization throughInnovative Orientations for Nuclear (MMOTION) project because it is addressing so many efforts ofcommon interest. WGHOF notes that the TOP on which this workshop was based, and the workshop
results, were used as inputs to the research agenda developed by the MMOTION project. The WGHOF
encourages funding authorities to implement the MMOTION research agenda and share research results.

13

NEA/SEN/SIN/WGHOF(2012)1

14

NEA/SEN/SIN/WGHOF(2012)1

INTRODUCTION

The nuclear community is currently at a stage where existing reactor control stations are undergoing
various forms of modernization, new reactors are being built in many countries with computer-based control
rooms, and advanced reactors are being designed through international cooperation to support power
generation for decades to come. With the introduction of advanced plants, we will see new reactor and systemdesigns, new tools to support plant personnel, and changes to nuclear power plant (NPP) staffing
configurations. The concepts of operation and maintenance for this new generation of plants are likely to bequite different from those employed in today‟s plants. It is important that the potential impact of thesedevelopments is evaluated and understood by prospective operators and regulators responsible for determining
the acceptability of new designs to support human performance in maintaining plant safety.
Research and development (R&D) for advanced plants is having an impact on existing plants as well. For
example, the technology developed to provide advanced plants with state-of-the-art digital instrumentation andcontrol (I&C) systems and computer-based control rooms is also becoming prominent in the modernization ofexisting plants and in new builds. One difference between plant modernizations and new builds is that building
a new plant creates an opportunity for an entirely new design, while modernizing an operating plant hasconstraints imposed by the existing facilities and concepts of operation. In addition, modernization projects
often require implementation of new systems over the course of several outages, thus creating interim states
between the old and new design that may themselves pose human performance challenges.
The introduction of new technology is viewed as having promise to improve the safe operation of NPPs.
In order to ensure the appropriate application of technology to support human performance and plant safety, it
is important to evaluate the technological advances in terms of their potential negative as well as positiveeffects. Research described in this paper can provide the technical basis to help ensure that the benefits of
new technology are realized and that the potential negative effects are minimized.
The impetus for the current effort grew out of a workshop hosted by the Halden Reactor Project (HRP) on“Future Control Station Designs and Human Performance Issues in Nuclear Power Plants” held in Halden,
Norway in May 2006 [NEA/CSNI/R (2007)8], and organised in cooperation with the Nuclear Energy Agency(NEA), Committee on the Safety of Nuclear Installations (CSNI), Special Experts‟ Group on Human andOrganizational Factors (SEGHOF), The SEGHOF was the predecessor of CSNI‟s Working Group on Humanand Organizational Factors (WGHOF), One output from that workshop was a Technical Opinion Paper (TOP)titled, “Research on Human Factors in New Nuclear Plant Technology” [NEA/CSNI/R (2009)7], which
identified eight broad topic areas that warrant further research:
1.
2.
3.
4.
5.
6.
7.
8.

Operating Experience (OpEx) from New and Modernized Plants.
Evolving Concepts for the Operation of Nuclear Power Plants.
The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems.
Management of Unplanned, Unanticipated Events.
Human System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions.
Complexity Issues in Advanced Systems.
Organizational Factors – Safety Culture.
Human Factors Engineering (HFE) Methods and Tools.

A recommendation in the TOP was “that the international community should pursue a collaborative andcoordinated approach to addressing these important research areas by identifying different nations‟ ongoingand planned research related to these program topics, and also by identifying topics that are currently not underinvestigation or that could benefit from greater attention.” The current work is the initial step in fulfilling thisrecommendation.
15

NEA/SEN/SIN/WGHOF(2012)1
The WGHOF undertook two efforts to respond to this recommendation in the TOP. The first was thedevelopment and implementation of a survey. The purpose of the survey was to identify the level of interestin performing research projects related to the eight research topic areas to enhance the state of knowledge inthe human and organizational factor (HOF) aspects of control station modernizations in existing plants, new
NPPs and advanced reactors. A secondary purpose was to assess the level of interest in collaborating in suchresearch. The survey results are described in Section 2 of this report. The second effort to respond to the TOP
was a workshop on human performance and the operation of new nuclear power plant technology. The
objective of the workshop was to discuss how the use of new human-system technology in the operation of
NPPs may affect reliable human performance and plant safety. The outcome of the workshop is discussed in
Section 3 of this report. An overall summary of the survey and workshop is in Section 4. The attachments are:a copy of the survey instrument, Attachment 1; a list of references collected from the survey, Attachment 2; alist of contacts who have been working in the various topic areas and who may be interested in futurecollaboration, Attachment 3; and a copy of the workshop agenda, Attachment 4.

16

NEA/SEN/SIN/WGHOF(2012)1

SURVEY RESULTS

Introduction
A survey was prepared to support of WGHOF‟s task to study human performance in new nuclear
plant technology. This survey, issued to the NEA Member Countries‟ representatives and other selectedsubject matter experts (SMEs), helped identify research activities that have been conducted, are ongoing,or are planned in the topic areas discussed in TOP.
The purpose of this survey was to identify the level of interest in performing research projects related
to the listed research topic areas that will enhance the state of knowledge in the HOF aspects of control
station modernizations in existing plants, NPPs, and advanced reactors. A secondary purpose was to assess
the level of interest in collaborating in such research.
For the main part of this survey, two primary questions were asked. They were: Extent of Involvement: Is your organization performing, planning to perform, or has it performed
research or development activities in each of these topic areas?
 Interest in Collaboration: Are you interested in collaboration with others on these research ordevelopment areas?
There were also a number of open-ended questions. Those questions were: What research facilities do respondents have, or could have access to, to perform HOF research?
 What measures do respondents use to assess human performance and how do the measures relate
to plant performance?
 At the time of the survey, did the respondent have any cooperative research agreements in placewith other countries or organizations (including industry)?
 What other research or development issues were not covered in the survey that may be of generalinterest?
 What documentation exists on these research or development activities, which is publicly
available?
 Who are the key personnel involved in these research or development activities?
MethodThe survey was issued through WGHOF to all member countries. The respondents included consultants,
regulators, researchers, utilities, and vendors. For the purposes of this survey, “new nuclear plant technology”
included the introduction of computerization of the human-system interfaces (HSIs) as well as other
innovations in the design of the plants themselves (e.g., multi-modular plants, passive systems)
A commercially available web-based survey tool was used to distribute the invitations to the
respondents, present the questions, and collect the responses. A copy of the survey instrument can be foundin Attachment 1. Survey participation was distributed and tracked by the WGHOF Secretariat and thesurvey steering team members. The survey steering team encouraged participation by contacting those towhom the survey was sent to ask them either to complete the survey or forward it to someone else in theorganization that might be more appropriate or have time to complete it. This mode worked well inensuring the invitation was widely distributed. As the deadline for completion approached, the WGHOF
Secretariat sent a reminder to those who had been invited and had not responded. As a result of thereminder, the response rate approximately doubled.

17

NEA/SEN/SIN/WGHOF(2012)1
The data from the survey were downloaded into a database and then into text documents and pivot
tables, which were used to tabulate and visualize the results.
Results
Overall, the survey fulfilled its intended purpose of identifying the level of interest among WGHOF
members in performing HOF research. Sixty different surveys were returned. Table 1 lists all of theorganizations to which survey respondents belonged. However, the answers generally came from anindividual or were a collective response from a small group within the organization, rather than a formalorganizational response. Further, the results of the survey have been made anonymous to mitigate the naturaltendency to infer that a specific response to the survey represents the views or the entirety of research
activities occurring at a given organization.
Table 1. Listing of Organizations to which Survey Respondents Belonged
Organization....................................................................................................
Alion Science & Technology ............................................................................
AREVA NP GmbH ...........................................................................................
AREVA NP Inc. ...............................................................................................
AREVA TA ......................................................................................................
Atomic Energy of Canada Limited (AECL) .....................................................
BackPacker Jack, Inc./ IEEE ............................................................................
Beville Engineering, Inc. ..................................................................................
Brookhaven National Laboratory (BNL) ..........................................................
Center for Operator Performance (COP) ..........................................................
CEZ Group........................................................................................................
Comisión Nacional de Seguridad Nuclear y Salvaguardias (CNSNS) .............
Commissariat à l‟Energie Atomique (CEA) .....................................................
Det Norske Veritas Inc. (DNV) USA ...............................................................
Electric Power Researcher Institute (EPRI) ......................................................
Électricité de France (EDF) R&D .....................................................................
Électricité de France (EDF) SEPTEN ...............................................................
Forsmark Vattanfall (FKA) ..............................................................................
Fortum, Power Division ....................................................................................
George Mason University (GMU) ....................................................................
Gesellschaft für Reaktorsicherheit (GRS) .........................................................
Idaho National Laboratory (INL) ......................................................................
Institut de radioprotection et de sûreté nucléaire (IRSN) ..................................
Institute for Safety and Reliability, GmbH (ISaR) ............................................
Institute of Nuclear Safety System, Inc. (INSSI) ..............................................
Institutt for energiteknikk/ OECD Halden Reactor Project ..............................
Japan Nuclear Energy Safety Organization (JNES) ..........................................
John Wreathall & Co./The WreathWood Group ...............................................
Korea Atomic Energy Researcher Institute (KAERI) .......................................
Korea Institute of Nuclear Safety (KINS) .........................................................
Krueger Ergonomics .........................................................................................
Longenecker and Associates .............................................................................
Mitsubishi Electric Corp./ Advanced Technology R&D Center .......................
Mitsubishi Nuclear Energy Systems, Inc. .........................................................
Nuclear Installations Inspectorate (NII) ............................................................
Nuclear Research Institute Rez Plc. ..................................................................
Okayama University .........................................................................................
Oskarshamn NPP ..............................................................................................

18

Country
United States
FranceFranceFranceCanadaUnited States
United States
United States
United States
Czech Republic
MexicoFranceUnited States
United States
FranceFranceSwedenFinlandUnited States
Germany
United States
FranceGermanyJapan
NorwayJapan
United States
South Korea
South Korea
United States
United States
Japan
United States
United Kingdom
Czech Republic
Japan
SwedenNEA/SEN/SIN/WGHOF(2012)1
Table 1. Listing of Organizations to which Survey Respondents Belonged (Cont‟d)Organization....................................................................................................
Radiation and Nuclear Safety Authority (STUK) .............................................
Roth Cognitive Engineering .............................................................................
SA Technologies ...............................................................................................
Sandia National Laboratories (SNL).................................................................
State Office for Nuclear Safety (SUJB) ............................................................
Swedish Radiation Safety Authority .................................................................
Tecnatom, S. A. ................................................................................................
The Institute of Applied Energy (IAE) .............................................................
The Ohio State University (OSU) .....................................................................
Tokyo Electric Power Company, Human Factors Group .................................
U.S. Nuclear Regulatory Commission (USNRC) .............................................
UJD SR-NRA of the Slovak Republic ..............................................................
University of Central Florida (UCF) .................................................................
Vattenfall Ringhals AB .....................................................................................
VTT Technical Research Centre of Finland .....................................................
VUJE, Inc. ........................................................................................................
Westinghouse Electric Company ......................................................................

Country
FinlandUnited States
United States
United States
Czech Republic
SwedenSpain
Japan
United States
Japan
United States
Slovak Republic
United States
SwedenFinland
Slovak Republic
United States

Extent of Involvement
Figure 1 is a high level summary of how involved all “types” of respondents (e.g., regulators,
researchers, vendors, utilities, and consultants) were in each of the eight broad topic areas. As the figure
shows, the HSI topic area had the highest overall level of involvement and the topic of complexity had thelowest. The topics OpEx, HSI, Org Safety Culture, and HFE all had high levels of ongoing activity, but
respondents indicated that the amount of research they plan to do in the future will be less. The topics
Evolving Concepts, Automation and Personnel, Unanticipated Events, and Complexity all showed relatively
lower levels of Performed and Ongoing research activity, but that the amount of research that is being
planned for the future is increasing. Overall, the trends for involvement are positive from the standpoint that: HOF topics that are currently being researched are projected to continue to be highly researched
topics.
 HOF topics that previously were not being researched extensively are being, or will be, researchedmore.
To further analyze the extent of involvement responses, a numerical score was assigned to eachsurvey respondent‟s answer according to the categorization scheme below.
No response = 0

Performed = 1

Ongoing = 5

Planned = 3

Once each individual response was assigned a score, they were summed for each topic area. The
maximum score for any question was used to represent the entire area to show the strongest levels of
involvement. The higher the number, the greater the extent of involvement.
Once each individual response was assigned a score, they were summed for each topic area. The
maximum score for any question was used to represent the entire area to show the strongest levels of
involvement. The higher the number, the greater the extent of involvement.
However, this analysis approach also generates a unique and non-overlapping set of summed numericalscores, which allows one to interpret the results more precisely. Each specific summed score is an indicationof any and all combinations of the respondent‟s involvement in a given topic area. For example, a score of 9means that the respondent indicated that they have performed (1), are performing (5), and are planning tocontinue performing research in the given topic area (i.e., 1 + 5 + 3 = 9), A score of 1 means the respondent
19

NEA/SEN/SIN/WGHOF(2012)1
only indicated that they previously performed research in the topic area. Table 2 below shows how thesummed scores were generated based on all possible combinations of survey responses.
Figure 1. Summary of Involvement in HOF Research Activities

Table 2. Legend for the Matrices on Involvement
If the survey respondents answered:
Performed (1)Ongoing (5)Planned (3)
X
X
X
X
X
X
X
X
X
X
X
X

Then their resulting summed score was:
1
3
4
5
6
8
9

The level of involvement in each of the eight topic areas is shown as a function of respondent type(e.g., regulator, researcher, vendor, utility, and consultant) in Table 3. Also note that no response to aquestion was given a score of 0. Respondents were instructed to not provide a response if they had not,were not performing, or were not planning to perform research in the topic area. To help distinguish
responses indicating some level of involvement from no involvement, the “no response” scores (0) in thetables below were left blank.
The results for the questions on involvement in HOF research and development activities indicate that
there were a number of organizations that are involved in HOF research activities. Research has been done,
is currently on going, or is planned in all eight topic areas by a variety of organizations. For each research
topic, more than one organization was currently performing or planning to perform research, whichindicates some consensus that research is needed to help establish the technical bases for addressing HOF
issues in new and existing nuclear power plants.

20

NEA/SEN/SIN/WGHOF(2012)1
Table 3. Respondents’ Involvement in HOF Research Activities

Interest in Collaboration
The analysis of the responses to the question on interest in collaboration was straightforward. For
each of the topic areas, the respondent was asked to indicate his or her level of interest (high, medium, orlow) in collaborating on research in the particular topic area.
It is important to note, however, that if the respondent indicated they were not involved in HOF
research for a given topic area, the survey was designed such that they were not given the opportunity to21

NEA/SEN/SIN/WGHOF(2012)1
indicate if they would be interested in collaborating with others on that topic. The desire to streamline thesurvey and reduce the amount of time it took a respondent to complete it created this oversight.
Consequently, the interest in collaboration results should be interpreted with the disclaimer that any
reported interest in collaboration is dependent on the respondent already having some level of involvementin that research topic.
In Figure 2, the level of interest in collaborating is shown across all respondent types. Generally
speaking, the checkered and solid bars indicate interest in collaborating (at some level) while the crosshatched bars indicate that the respondent either did not have an opportunity to state whether they wereinterested in collaborating or was not interested. A notable finding from this analysis is that many of the
respondents indicated “High Interest” in collaborating across all eight topic areas.
Figure 2. Summary of Interest in Collaboration

Again, it is not possible to ascertain whether those who reported no involvement in the topic areas were,or were not, truly interested in collaborating. As such, there is no direct way to ascertain the percentage of
respondents who were interested in collaborating, at some level, relative to those who were not. Rather, if one
wanted to perform a comparative analysis of the responses (i.e., bars) in Figure 2, a valid interpretation would
be the following: with the exception of Evolving Concepts and Complexity, there was more interest incollaborating, at some level, with others on these HOF research activities than there was lack of involvementin those topic areas. In the cases of OpEx, HSI, and HFE, there was considerably more interest incollaborating than there was lack of involvement in these topic areas.
A more detailed summary of the interest in collaboration responses is provided in Table 4. This table
presents the results at a finer level of detail. It shows the level of interest results at the level equivalent toan individual respondent, and organizes the results by respondent type and topic area.
A general conclusion for the questions on interest in collaborating on HOF research is that there is
generally a high amount of interest in collaboration on all of the eight topics for all respondent types.

22

NEA/SEN/SIN/WGHOF(2012)1
Table 4. Interest in Collaboration (blank is no response given, 1= low, 2=moderate, 3=high)

Additional Analyses of Extent of Involvement and Interest in Collaboration
The results of the analysis of questions on involvement and interest were plotted against each other,by respondent type and HOF topic area to better understand the relationship between the two. The resulting
scatter plots provide additional insights into questions such as: which respondents were most interested incollaborating and on what topics they were interested in collaborating. Similarly, respondents that were notinterested in collaborating and topics that did not have high collaboration potential can be seen. Figure 3
shows the resulting scatter plots. The larger size dots indicate multiple respondents with the same score.
The best fit trend lines in the scatter plots in Figure 3 show there is no obvious consistency to therelationship between involvement in HOF research and interest in collaboration. No attempt was made totry to analyze each relationship by respondent type and HOF topic area, though Figure 3 does provide that
opportunity to anyone interested in trying to understand a particular relationship. However, some ancillarycomparisons were performed on the available involvement and collaboration data.
23

NEA/SEN/SIN/WGHOF(2012)1
Figure 3. Scatter plots of involvement and interest by respondent type and HOF topic area

There were 22 cases where a respondent was currently performing and planning to do more research
on an HOF research topic (i.e., an involvement score of 8 or 9) and expressed moderate or high interest incollaboration (i.e., an interest score of 2 or 3), In general, this is a positive indication that those involved in
HOF research activities were also interested in collaboration. There was only one exception, which was
one researcher who was currently performing and planning to do more research on Evolving Concepts, but
expressed low interest in collaboration.
There were 27 cases where a respondent was currently performing research on an HOF topic (i.e., an
involvement score of 5) and expressed high interest in collaboration (i.e., an interest score of 3), However,there were eight cases where a respondent was currently performing research on a topic area, but expressed
low interest in collaboration. Taken together, this is a more positive than negative indication that those whoare currently performing HOF research are more interested in collaboration than not.
There were 22 cases in which a respondent indicated they were planning to perform research on a
topic area (i.e., involvement score of 3 = Planned) and expressed high interest in collaboration (i.e., aninterest score of 3), However, there were 12 cases where a respondent was planning to perform research on
a topic area (i.e., involvement score of 3 = Planned) but expressed low interest in collaboration (i.e., ascore of 1), Taken together, this is a more positive than negative indication that those who are planning toperform HOF research are interested in collaboration.
In general, the relationship between the actual reported involvement and expressed interest incollaboration would appear to be one where at least some respondents were highly interested incollaboration, regardless of their level of involvement, with the exception of two cases where there wasonly a moderate level of interest in collaboration (i.e., Regulators researching Evolving Concepts, andVendors researching Complexity), There were only a few cases where respondents who had some level ofinvolvement in any of the topic areas expressed only low interest in collaboration.
In addition, while some obvious questions to ask next might be: “Why are some respondents interested
in collaboration, and others not?” and, “Why do some HOF topic areas have high collaboration interest, andothers do not?” it is not possible to determine from the data available the motivation for the respondent‟s
input to the survey. The questions in the survey asked the respondents to report their extent of involvement(response options to these questions were: performed, on-going, planned) and their level of interest incollaboration (response options to these questions were: low, medium, high), One cannot reasonably attribute
24

NEA/SEN/SIN/WGHOF(2012)1
the motivation of the respondents, given the responses options provided. An analysis of the few open-ended
comments voluntarily provided by some respondents did not lend any reliable or conclusive insights into themotivations of all respondents.
General Observations of Involvement and Interest in Collaboration
The broader message that emerges when combining the results on the questions of involvement in
HOF research and development and interest in collaboration is that there is a considerable amount ofresearch activity occurring, or planned, and those who are performing this research have, in general, aninterest in collaborating. However, there are additional factors to consider beyond opportunity and interest.
These include, for example, whether there are facilities available to perform research, whether there issufficient overlap in the kinds of human performance measures different researchers use to support
collaboration, and whether there are cooperative agreements in place. Responses these survey questions are
summarized in the following sections.
Research Facilities
To analyze this open-ended question, a coder reviewed the responses and categorized them by theme.
The responses were further categorized by respondent type. Results are shown in Table 5.
Table 5. Research Facilities to Perform HOF R&D

Regulator
Researcher
VendorUtilityConsultant

Full-scoperesearch simulator
HRP*
Other
2
2
1
2

Training/
development
simulator
2
5
62

Part-task
simulator

2

1
3
2
2

Research
ReactorOther
application
simulator

Modeling
tools

2

2

1
1

1

* Halden Reactor ProjectMeasures of Human Performance
Respondents were also asked to describe the measures they use to assess human performance and how
the measures relate to plant performance. Their responses were analyzed using the same method describedabove, and the results are summarized in Table 6.

1

1
2

1
4
2

2

1

1
5

1

25

Psycho Physiological

Safety Culture andOrganizational Health
Measures

Observation

3
1

Questionnaires

6
2
1
2

EventsRisk-informed Measures

Regulator
Researcher
VendorUtilityConsultant

Standard Human
Performance Measures(e.g., workload, situation
awareness, accuracy,
time, errors)

Table 6. Measures to Assess Human Performance

2
4
1
2

NEA/SEN/SIN/WGHOF(2012)1
Cooperative Research Agreements
Respondents were asked to describe any cooperative research agreements they had in place tofacilitate collaboration. Their responses were analyzed using the same method described above, and theresults are summarized in Table 7.

2

2
1

1

1
1

1
1

2

1

University

VendorMMOTION

SAFIR
1

2

Other
Application

4
5
2
4
1

National Lab

Regulator
Researcher
VendorUtilityConsultant

Utility

HRP

Table 7. Cooperative Research Agreements

1
1
1

General Observations
The results of the questions on research facilities, measures, and cooperative research agreements showthat there are a number of facilities (mostly simulators) that are being used, and that a number oforganizations already have cooperative research agreements in place to facilitate collaboration. It isinteresting to note that the Halden Reactor Project (HRP) appears to be centrally involved in many of the
HOF research activities, which should not be surprising. HRP is a membership research organization throughwhich many organizations pursue research of mutual interest and benefit, and it turned out that many of the
respondents to the survey were affiliated with one of many organizations that is a member of HRP. It is alsointeresting to note that there are many different kinds of human performance measures in use, and that there
is considerable variability in terms of the extent to which the measures relate to plant performance.
Other Research Topics that may be of General Interest
The survey also asked respondents to suggest other HOF research topics of interest that were notincluded in the TOP. To analyze this open-ended question, a coder reviewed the responses and categorized
them by theme. The following list summarizes the responses:
 Research on multi-modular operator control.
 Research on alarms, large screen displays, and operator aids for supervision of plant status.
 Studies of the impact of new forms of HSIs (e.g., computer-based procedures) on teamwork andshared situation awareness.
 Studies on the impact of new HSI resources on changes in philosophy regarding function
allocation and interaction across crew members.
 Research into applying a comprehensive, integrated HFE process to new plant design andcomprehensive modernization projects.
 Development of guidance for Integrated System Validation (ISV), Development of guidance for preparing working procedures.
 Development of guidance for data collection for Human Reliability Analysis (HRA) for
Probabilistic Safety Assessment (PSA),
 Development of resilience measures for application in HRA.
 Development of methods to assess resilience and vulnerability of organizations.
 Development of guidance for the application of (quantitative) risk management to human factors.
 Assessment of the relative risks from maintenance in passive systems vs. the roles of operators.
26

NEA/SEN/SIN/WGHOF(2012)1






Means for quantifying violations.
Knowledge transfer.
Need for a simulator for research and development and for certification and human error research.
Studies of all the human and organizational factors aspects concerning outage, radiation protection,
occupational safety, field operation and maintenance, and impact of external factors on nuclear
safety.
Study on how regulations and/or regulatory approaches affect human performance at utilities.

Publicly Available Documents and Key Personnel
For each of the eight topic areas of the survey, respondents were asked to list the reference citation forany publicly available documents that describe the research they have performed in the topic area, and tolist key personnel that others could contact if they so desired. The responses to these questions areprovided in Attachments 2 and 3.
Survey Summary
Considering the voluntary nature of the survey and the relatively informal mechanisms used to
distribute the invitations to participate, as well as the volume of the data being requested, an overall
response rate cannot be calculated, but the information is useful as both a snapshot of the current state of
the research in the eight topic areas and as a listing of key research institutions focusing some of their workon the topic areas. The survey returned a list of points of contact and over four dozen pages of references topublicly available information. A majority of workshop participants had participated directly in the survey.
In this regard, the process of thinking through the survey undoubtedly served as preparation for the intense
face-to-face workshop discussed in Section 3. Points of importance are:
 WGHOF member countries were well represented in the survey responses.
 All categories of SMEs, including regulators, researchers, utilities, vendors, and consultants,
were represented.
 Research has been done, is currently on-going, or is planned in all eight topic areas identified in theTOP. No research area has only been studied in the past or entirely projected but unstudied atpresent.
 Involvement data can be used to identify research topics of current and future attention.
 There is a high level of interest in collaborating with others on the research topics(s) respondents
are studying or planning to study.
 There are hundreds of relevant research reports, papers, and presentations that have been publishedon the research topics.
Given the points of contact from each of the participating organizations, and stated interest levels incollaborating, the path forward can be facilitated based on the survey.

27

NEA/SEN/SIN/WGHOF(2012)1

28

NEA/SEN/SIN/WGHOF(2012)1

WORKSHOP OUTCOMES

Introduction“The Workshop on Human Performance and the Operation of New Nuclear Power Plant Technology”
was convened in Rockville, MD, USA, March 1-3, 2010. The workshop was sponsored by the Nuclear
Energy Agency (NEA), Committee on the Safety of Nuclear Installations (CSNI), Working Group on
Human and Organizational Factors (WGHOF), hosted by the US Nuclear Regulatory Commission(USNRC), and organized by the Idaho National Laboratory (INL) under contract to the USNRC. As statedin Section 1, this workshop implemented a recommendation in TOP, NEA/CSNI/R(2009)7. The objectiveof the workshop was to discuss research on the use of new human-system technology in the operation of
NPPs to identify how it may affect reliable human performance and plant safety.
The workshop was structured to maximize discussion and interaction among participants. The
morning of the first day began with a welcome, a presentation of the results of the survey discussed in
Section 2 and keynote speeches that were intended to show that there are organizations from otherapplication domains who are also interested in the workshop topics. The next four, half-day sessions werestructured such that a challenge speaker for each of two topics made presentations to all workshop
participants that were intended to set the stage for the working sessions that followed. There were fourparallel working sessions (two for each topic) in which the participants were asked to identify one or two
specific research projects for the topic under consideration. They were encouraged to describe the projectin as much detail as time allowed, e.g., facilities needed and measures to be used. Each working group thenpresented the results in a plenary session for comment by the assembled participants. The final afternoon
included the Chair‟s Summary and open discussion (see Agenda, Attachment 4),
Welcome and Introduction
The workshop started on March 1, 2010, with a welcome address and introductory talks.
Approximately 50 SMEs were welcomed to the workshop by Mr. James Lyons, the Deputy Director of the
USNRC Office of Nuclear Regulatory Research, Dr. Valerie Barnes, USNRC, Chair of the workshop, and
Mr. Radomir Rehacek, Secretariat of the WGHOF, from the NEA. Mr. Rehacek talked about the NEA
history, mission, committees, the background of the WGHOF, the workshop organizing committee, and theorganization of the workshop.
Survey Results
Mr. Jeffrey Joe, from the INL, stated that the results of the survey indicated that every research anddevelopment topic has some level of ongoing or planned activity. Ongoing activities are focused on
assessment of safety culture and capturing and using OpEx for application to safety culture issues;
activities being planned are in the areas of effects of new technologies, new concepts of operation, andmulti-unit reactors. Facilities currently being used to perform research and numerous published documents
were identified as were specific researchers.
Keynote Presentations
There were two keynote presentations. The first was provided by Mr. David Strobhar, who described thework of the Center for Operator Performance (COP), which is collaboration among oil and chemical
processing industry companies and academia. The key issues of the COP are alarm processing, advanced
29

NEA/SEN/SIN/WGHOF(2012)1
controls, graphics, and workstations. There are several sub-issues as well, such as procedures and knowledge
management. Personnel in this industry face operational issues that are very similar to those faced by nuclear
industry personnel, e.g., alarm handling, procedures, human-system interface; even though their concepts ofoperation are somewhat different than that of the nuclear power industry, e.g., single function control stations
rather than centralized control rooms and lack of strict adherence to procedures.
The second speaker was François Dionis, who discussed the EURATOM, Man-Machine-Organization
through Innovative Orientations for Nuclear (MMOTION) project. The objectives of the MMOTION
project are to analyze the current state-of-the-art and what is anticipated for the future concerning ManMachine-Organization and safety for present and next generation NNPs, to define and propose a researchroadmap to help guide future Euratom research programs, and to comprehensively address the capability of
the research community to address the recognized issues for the period 2010-2015. The areas of interest
identified to date are: automation; human-system interface; organization and management; humanperformance, teamwork and training; and assessment and evaluation, with some sub-issues also identified.
Some of the sub-issues include: Integrated human-centered design of socio-technical systems, safety
culture, and assessment and evaluation of the human role in operation.
Both representatives of these organizations expressed interest in potential collaboration with WGHOF
members and others on issues of common interest.
Challenge Presentations and Working Group Results
Operating Experience from New and Modernized Plants
The challenge speaker for the operating experience topic was Thomas Gunnarsson, a control room
specialist who described experience from the modernization of the Oskarshamn NPP in Sweden. The focus
was on lessons learned from the control room modernization process of two units with some suggested areasof research identified as a result. The two control room modernization projects were sequential, so the utility
was able to use the lessons learned from the first to improve the process of modernizing the second. The
primary lesson from the first was to do the modernization in a stepwise fashion over time rather than try to do
the entire project at one time because of the scope of the effort and impact on resources. Additionally, alarm
handling during outages was found to be problematic because of the number of nuisance alarms and newalarms. The lessons learned taken from the two projects were:Both the contractors and management had a limited understanding of the need for the integrationof a structured HFE process within the design process.
There was a large amount of effort and a significant challenge to establish a common understanding
among designers and management of the role of HFE in both projects.
– HFE training and education of project personnel, including management, could have been
improved.
– HFE analyses came too late in the design process and often with no coordination between
I&C and process design and HFE design.
– Plans for HFE and HF verification and validation were often developed after I&C and design
processes were complete.
The contractor should have plant knowledge (e.g., systems, structures, processes, and concept ofoperations) to perform HFE work.

Mr. Gunnarsson also proposed some problem areas which should be considered for future
investigation and which could be helpful in other control room modernization projects. These include:

Human resources/competence, i.e., limited number of HFE experts in the nuclear industry,
limited availability of HFE education, insufficient knowledge and skills with the new technology
by the owners, insufficient understanding of plant behavior among the contractors
30

NEA/SEN/SIN/WGHOF(2012)1






Operator interface issues, i.e., guidelines and standards, aging of software applications, display
resolution capabilities
Alarm presentation: potential increase in number of alarms, failure to support pattern recognition
Effectiveness of an overview presentation
Improved presentation of information during outages
Validation guidelines, methods, acceptance criteria, and timing of integrated system validation

Both of the working groups that discussed OpEx recommended collection of more operating experience
information from a broad spectrum of sources, e.g., operating plants, new plant construction, and plantmodernizations, and from other industries to identify lessons learned that are unique to the implementation of
new technologies. Such a large scope effort could be hindered by:






Lack of a standardized approach to collecting OpEx information
Confidentiality issues
Current OpEx feedback systems generally collect limited human and organizational information
There is a general lack of communication and transfer of knowledge related to OpEx both within
an organization and among organizations
Implementation of lessons learned is not prevalent in the industry

An approach proposed by the groups was to collect OpEx and develop lessons learned on goodpractices as well as incidents. The scope of the data collection should include experiences from outages;
how OpEx is collected and used internally; how experience from other nuclear utilities, operators, andvendors is used; and how experience external to the nuclear industry is incorporated in their OpEx
programs. This research effort would require developing an example of an effective OpEx program fromdevelopment through implementation, performance measures, and consideration of the importance of alearning culture to ensure implementation of lessons learned.
Working group 2 also suggested a specific focus for the OpEx data collection which would include:
alarm display, pattern recognition and alarm handling; OpEx data extraction and analysis; computerized
systems with improved precautions; control room modernization; navigation from high level to detailed
level on displays; and validation of digital control systems via OpEx.
The plenary discussion that followed focused on the problems and issues associated with thecollection of OpEx. Some of the problems were common to several participating countries (e.g., level ofdetail regarding human performance, knowledge of human performance) and some were country-specific(e.g., privacy rights and fear of disclosure), Further, the importance of common terminology and datacollection formats was discussed.
Evolving Concepts for the Operation of Nuclear Power Plants
The challenge speaker for the concepts of operations topic was Gyrd Skraaning, Jr. of the HRP. He
described the context of concepts of operation in terms of automation, staffing, and user interface, where: Automation consisted of:– Cooperative and intelligent “autopilots”.
– Shared control between humans and automatic systems.
– Transparency of automation.
 Staffing consisted of:– Remote operation.
– Multi-unit stations.
– Distributed teamwork.
– Reduced staffing levels.
– New operator roles.
31

NEA/SEN/SIN/WGHOF(2012)1
User interface consisted of:– Natural and intuitive interaction with the plant.
– Complete process overview.
– Fluent teamwork.

On the topic of Automation, Mr. Skraaning presented several research issues, including: Designing automation through development of more intelligent automation algorithms andadapting automation to each individual‟s work style.
 Determining when to encourage shared control between humans and automation and when toassign final authority to the humans.
 Enhancing process transparency by presenting automation goals and activities to operators andthe use of visualization technologies.
 The need to study operator roles that separate the monitoring of plant status and supervision of
automation.
On the topic of Staffing, Mr. Skraaning discussed a study performed at HRP related to multi-unit
operations. As part of this study, he described work that the HRP has been doing for the Norwegian offshore oil industry and the development of integrated operations among off-shore facilities and an onshore
control room. He also related the work to some Canadian plants where several nuclear units are controlledfrom a single control facility.
Finally, he described a futuristic control interface that represents the plant in a direct and intuitive
manner similar to the analogue interfaces in most current plants. His idea was that command line andgraphical user interface approaches present an abstracted understanding of the plant to the operator. He
then showed an example of what he termed a “Natural User Interface.” He presented the example toencourage the working groups to look to the future for research projects and not be bound by current
approaches to operational concepts and interfaces.
The two working groups approached the assignment somewhat differently. The first group started byrecommending a survey to determine what different concepts of operation are in use or contemplated for boththe nuclear and other industries. One goal of the survey would be to determine what drives the decision tofollow one concept or another, e.g., reducing human error, operations and maintenance costs, reduced
staffing, or the introduction of radically different process technology. The results can then help to guide anexperimental program. They also proposed a project to identify approaches to design and evaluation ofalternative concepts of operation. The assumptions were that the impacts of automation, organizational
factors, staffing, distributed personnel and remote experts are not well known. Further, the group suggested
that there is a need to develop tools to assess the results, for example: simulation (models and simulators),
rapid prototyping, and visualization. Resources and facilities needed to perform the research are: part- andfull-task simulators, computer models, human factors and operations subject matter experts, and test subjects.
Measures that could be used include: situation awareness; workload; and operator performance, including
errors and successes, using normal and abnormal scenarios.
Working group 2 started out by identifying several key topics under concepts of operation. These
included: Operations and system mapping (see below), Operator skills and roles.
 Information technology (IT) security and reliability assurance.
 Regulatory acceptance criteria.
 Intelligent components vs. intelligent I&C.
 Multi-modular reactors: “Right-sized” reactors.
 Stepwise testing and implementation of digital systems.
32

NEA/SEN/SIN/WGHOF(2012)1
The group expanded on four of these topics. Within the operations and systems mapping topic, they
indicated that there is a need for a thorough analysis and dynamic linking of the goals, functions, activities,
tasks, processes, etc. for any new concepts of operation. This effort could possibly result in identifying thedrivers for higher levels of automation. However, it may be found that increased automation could haveunintended consequences, such as the effects of degraded computerized systems. For the stepwise testing andimplementation topic, the group suggested that technology is not implemented overnight. For example, as thenuclear industry transitions from analog to digital I&C, the industry is moving to hybrid control room
upgrades rather than making wholesale changes. Within IT security and reliability assurance, the groupsuggested that as the nuclear industry increases the use of advanced I&C and automation, the infrastructure
must be in place and verified to ensure its secure, reliable, and safe performance. When discussing operator
roles several questions were posed: As we increase automation, is the same skill set required of plant staff?
 How can the nuclear industry ensure that operators will be engaged with the interface?
 To what extent should the system be allowed to make key decisions?
 How might the nuclear industry leverage skilled operators from other industries?
There was considerable discussion following the working group presentations on defining anddescribing concepts of operation and how to characterize them for research. The relationship of concepts ofoperation to other topics, such as, staffing and automation, play a role in any such research. Further, it isimportant to identify meaningful measures and acceptance criteria for assessing alternative concepts ofoperation in experimental and field research settings.
The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
The challenge speaker for the automation topic was Dr. John O‟Hara from Brookhaven National
Laboratory. He began his presentation with an historical perspective on how the use of automation byindustry has progressed from the industrial revolution to the present. Given this background, he listed anumber of human performance issues. These were: Change in the overall role of operations and maintenance personnel.
 Understanding the role of automation in operations.
 Transition in workload for the operator when automation degrades or fails.
 Monitoring of displays, vigilance or situation awareness, and complacency.
 Out-of-the-loop unfamiliarity with underlying processes.
 Degradation and loss of operational and maintenance skills.
The current trends in automation were described as: New and advanced plants will be more highly automated.
 Expansion of the applications of automation.
– Greater use of automation for process control.
– Operator aids and decision support.
– HSIs that adjust without operator input.
 Greater range of the ways automation is implemented.
– Automation that is more interactive and cooperative.
– Shared control, breakpoint control (thresholds for changing functions among agents), dynamic
allocation of functions among agents.
Dr. O‟Hara postulated that human factors specialists now consider that automation is part of a multiagent system where human and machine agents work cooperatively to accomplish plant safety andproduction goals. The primary operational tasks are still monitoring and detection, situation assessment,
response planning, and response implementation, but with increased use of digital systems and automation
33

NEA/SEN/SIN/WGHOF(2012)1
new human performance concerns can be introduced. These include different types of and higher levels ofworkload and the introduction of new error modes, e.g., mode error and errors related to miscalibrated trustin the automation. He also showed a three-dimensional model of automation with flexibility, functions, andlevels on the three axes.
He then listed several challenges for discussion, examples include: What is the appropriate allocation of responsibilities to each agent in the system and how/whenshould the responsibilities change?
 What are appropriate models of teamwork for multi-agent systems?
 How can we foster properly calibrated trust in automation to minimize misuse and disuse?
 How should automation‟s reliability be quantified and represented in HSIs?
 How can workload be managed with interactive automation?
 What are the requirements for HSIs to support the diverse implementation of automation?
 How can automation failures and degraded conditions be managed?
Each of these challenges was accompanied by research ideas.
One working group chose to address the assignment by first presenting a list of topics related toautomation. These included: establishing an appropriate group organizational model; using humanbehavior modeling; failure and degradation of automation; automated vs. human decision making;challenges to the traditional concept of teamwork; and transparency of automation and situation awareness.
The projects proposed by the group included: review other fields to determine what drove them to use
more automated systems; study how to prepare for degradation or failure of automation; determine goodpractices for information presentation and information selection; and use of OpEx from nuclear and otherindustry experience with current digital systems.
The second working group focused on the three-dimensional structure of automation that Dr. O‟Hara
presented and proposed that it could be used as a basis to guide future research. They went on to propose two
specific projects: one on the use of visualization and one on teamwork with multi-agent teams. The purposeof the visualization project would be to better understand how to best represent processes, automation, andtheir interaction through surveying other industry applications, laboratory studies and use of simulation toassess safety, cost, accuracy, workload, situation awareness, and process measures. For multi-agent teamresearch an initial step would be to determine what models of teamwork can be used to guide humanautomation interactions; then testing them through surveys, laboratory studies, and simulations.
The plenary discussion reiterated many of the topics raised by the challenge speaker and the working
groups and expanded on some topics. The number of possible research projects that would come out of thethree-dimensional model of automation proposed by Dr. O‟Hara is extensive and would need to betempered by practical application to those combinations that are of most potential utility. Measures andacceptance criteria would be important in understanding the results of research in this topic area. Much of
the research would require the use of a full-scope, reconfigurable simulator and experienced operators.
Management of Unplanned, Unanticipated Events
The challenge speaker for the management of events topic was Dr. Mica Endsley of SA Technologies.
Dr. Endsley started her presentation by challenging the notion that there truly are “unanticipated (orunforeseeable) events.” She postulated that unusual events are infrequent but foreseeable. She gave examples
of some events that many people had not foreseen, but on reflection and when reviewing the record, those
events could have or may have been predicted (e.g., New Orleans levees in hurricane Katrina and the space
shuttle Columbia), There is often a gradual slipping of “acceptable risk” and “denial of the low probability
event” present in our society.
34

NEA/SEN/SIN/WGHOF(2012)1
She then described a five-step template for how to deal with the unexpected when it does happen
based on the concept of situation awareness (SA): Perception – detect that there is a problem.
 Comprehension – diagnose to understand what is going on.
 Projection – project the outcome of current status and trends.
 Decision – determine corrective action.
 Performance/Action – carry out corrective action.
From a control room design perspective she suggested that the question then becomes one of systemdesign to support these steps. For detection, designers need to remember that situation awareness required tosupport normal operations is different from that for supporting off-nominal conditions. Anomalies should bemade clearly visible to the operator and salient, e.g., not buried in the midst of numerous nuisance alarms.
Interfaces should be designed so that personnel easily detect that an off-nominal condition exists.
Dr. Endsley discussed diagnosis in terms of research, which has determined how experts handle
unexpected situations. The first aspect is anticipation, where experts are ready for events before they happen
and are proactive rather than reactive. The experts achieve this by training for infrequent but critical events,
through which they can develop mental models to diagnose and respond to unusual situations.
A challenge for the diagnosis aspect of SA is whether research can lead to training for NPP personnel
that helps establish mental models/schema to support projections, diagnosis, and anomaly detection, so that
operators can better deal with the unexpected. Further, can research develop a basis for designing systems
that support detection and diagnosing the unexpected or low probability event? From a research
perspective, one might study whether automation will: 1) catch low-probability anomalies that people
might not catch, 2) cause more situations where operators do not understand the status of plant systems andprocesses, and 3) result in skill and knowledge degradation? Measures such as SA, workload, andperformance effects should be employed in research and evaluation using tools such as modeling andsimulation for both normal and off-nominal conditions.
The first working group focused on the development of a database of unanticipated events andprecursor events (near misses), The goal would be to help anticipate the unanticipated through a better
understanding of what led to the events and near-events. The primary tool for the research would be asurvey of facilities and existing event databases. It would be desirable to do a literature search and where
possible observe simulator training where unusual events are presented to operating teams to observe their
behavior and use of HSIs. The goal would be to develop a lessons learned report that could help plant
personnel have a better sense of impending events through control room design and training.
The second working group developed four potential research efforts. Their basic assumptions were
that: new technology will be present in control rooms, the new technology will present signals to operatorsin new ways, and this new technology will pose new questions for designers and operators. The potential
research topics were framed in terms of the following questions: How do symptom-based procedures address these new sources of information?
 What are the individual characteristics (cognitive, personality styles, and workload-handlingcapabilities) and organizational factors that can be optimized to enable recognition and handlingof unanticipated events?
 How can we conceptualize and understand the unique conditions that are needed to aid personnel
in recognizing that they are in an unanticipated situation?
 When in an unanticipated situation, how can new HSI technologies support the identification ofpotential mitigating and aggravating factors of that situation and of recovery pathways?

35

NEA/SEN/SIN/WGHOF(2012)1
The plenary discussions following the working group summaries focused on whether or not events are
ever truly unanticipated and on the training associated with maintaining SA in unexpected situations.
Mental models, individual characteristics, and symptom-based procedures were the focus of the
discussion.
Human-System Interface Design Principles for Supporting Operator Cognitive Functions
The challenge speaker for the operator cognitive functions topic was Dr. Emilie Roth of Roth Cognitive
Engineering. She began by discussing trends in cognitive engineering for supporting cognitive andcollaborative work. These trends include: methods for uncovering cognitive and collaborative demands,
design approaches for supporting cognitive work, and evaluation methods for assessing effectiveness of
support. As an example, she described a situation involving decision support where the challenge wasdynamic re-planning. The approach was the use of visualizations and the consideration of joint cognitive
systems, where teams are trained to think using a common framework based on visualization displays. The
example demonstrated the importance of: analysis of the work domain demands; creating visualizations thatenable users to directly perceive work goals, affordances and constraints; and deploying machine intelligenceto synthesize information, flag problems, and suggest solutions in the context of work visualizations. In thesevisualizations the automation process is transparent and can be managed.
Dr. Roth then presented some specific challenges for new nuclear plant technologies. These were: How to best take advantage of digital technology for more effective cognitive and collaborative
support?
 How to more effectively support the distributed cognitive team across all modes of plant
operations?
 How to best deploy computer technologies to facilitate proceduralized tasks?
 How to foster resilient performance under unanticipated conditions?
 How to develop more detailed guidance and consensus on addressing control room validation
challenges?
The first working group proposed four projects. The first project was a lessons learned effort that would
focus on identifying where and when operating crews need support, e.g., outage management and severe
accident management, and determining where designers may have gone too far, for example, keeping track of
hospital tasks on a white-board vs. using a computer-based system where the information needed is hard tofind. The second project was to determine what aspects of communication and teamwork have potential
negative impacts on SA and planning. Based on this research develop and test a communication protocol.
Further, perform research to determine how best to use HSIs to facilitate communication and teamwork, e.g.,visualization. An effort on merging procedures, plant data and controls was the third proposed project. This
project would focus on how to balance SA considerations while managing workload that results frominteracting with the HSIs. This project may also assess trust in the automated system. The fourth project
would be to develop guidelines for verification and validation of integrated systems, including consideration
for sampling, use of non-traditional statistics, and pass-fail or diagnostic thresholds.
The second working group explored how to achieve a better understanding of the design solutions thatmight optimize cognitive burden. This would be accomplished through: Developing or identifying a cognitive model to better understand what creates a cognitive burden in
NPP operations.
 Benchmarking to better understand best practices.
– Experience of existing plants with various design solutions.
– Experience of other industries and how they resolved similar problems.
 Further testing of design solutions.
 Developing acceptance criteria for judging a design solution.
36

NEA/SEN/SIN/WGHOF(2012)1
To perform this research, access to plants for interviews and observations, access to operators fromother arenas for benchmarking, and testing facilities (simulators) for experimental research on design
options would be needed.
The plenary discussion focused on the understanding of mental/cognitive models of operators and how
HSIs and computerized systems can be best designed to accommodate those models both individually andacross teams. The use of simulation and visualization for research and application were discussed along with
the need for guidance for validation of integrated systems. The potential use of resilience engineering was
suggested as a way to address design of systems to help operators deal with unexpected conditions.
Complexity Issues in Advanced Systems
The challenge speaker for the topic of complexity was Mr. Bernard Papin of CEA Cardarche. He
initiated his presentation by decomposing the topic into three questions: What are advanced systems?
 What is complexity?
 What are the issues?
He described advanced systems in three categories: 1) advanced concepts of operation, e.g., remote
operation; 2) advanced I&C, e.g., automation and computer aides; and 3) advanced reactor features, e.g.,
passive systems. These in turn relate to plant staff and organization, HSIs and the systems that drive them,and plant processes, respectively.
Complexity was described graphically in terms of objective complexity and subjective complexity.
Objective complexity includes structural, functional, and operational complexity. Whereas subjective
complexity overlaps with operational complexity, it also includes aspects of the operator and the operator‟s
perceptions of complexity.
The issues he presented were: Technical systems can become more complex and difficult to manage by their users (e.g., increased
physical and functional interactions, making it difficult to understand in depth their behaviour,
particularly during complex incidental or accidental sequences.
 These difficulties threaten operational efficiency and safety.
 Designers and users do not have the same viewpoint on the system complexity.
 When designers search for better systems reliability, they often increase their (operational)
complexity.
 Advanced HSIs and automation may increase the complexity perceived by the operators.
Mr. Papin went on to suggest some specific research topics: What is the real impact of complexity on human performance?
 How can we assess and minimize the complexity of a new design?
 How can we balance increased technical reliability (and complexity) with better human reliability(negatively impacted by this complexity)?
 How can we design HSIs actually limiting perceived complexity?
 At what phase of the design should we act on system complexity?
The first working group proposed one research project to develop tools to measure system complexity.
This would be done to: 1) form a better understanding of the relationship between system complexity andperceived complexity, 2) identify contributors and mitigating factors of the perceived complexity, and 3)
identify contributors specific to advanced systems. The group proposed to accomplish the research by using
37

NEA/SEN/SIN/WGHOF(2012)1
the framework for complexity suggested by Mr. Papin and adding human performance measures, such as
workload, reaction time, situation awareness, errors of omission, and errors of commission, when performing
simulator and laboratory research. The product would inform design and inform regulatory reviews.
The second working group suggested several research topics, including: How do we define complexity?
 How do we filter less important information while making it available when needed?
– What is the role of the HSI in mediating complexity?
– How should a complex system be best represented?
 How do we match designer and operator mental models?
 What happens when complexity management fails?
The group distilled these topics into two projects. The first would be to define complexity by first asking
if complexity matters in the nuclear arena. Next, determine if complexity is related to other concepts, such as
usability, by reviewing existing notions of complexity and its role in advanced systems. To perform theresearch, a user/task analysis should be done to see what systems are affected by complexity, metrics tomeasure complexity should be developed, and the concept should be validated on existing approaches tosimplified design such as ecological interface design. The second project would involve: determining whatorganizations currently focus on complexity and if the concept applies to continuous processes; focus on how
information is displayed to operators in these organizations; analyze activities and tasks and how they affectend users and how they differ from the nuclear arena; determine the consequences if complexity management
fails. The group concluded by stating that complexity is complex and recommended more work to create theproper research framework.
The plenary discussion continued the struggle to define complexity and how the concept can beaddressed through HSI design. Metrics to assess complexity and responses to it were also discussed. The
understanding of mental models plays a role in both research and design of systems to reduce the impact ofcomplexity on operator performance.
Organizational Factors – Safety Culture/Safety Management
The challenge speaker for organizational factors was Dr. Sonja Haber of Human Performance Analysis
Corp. She focused on four aspects of culture – defining, assessing, changing, and maintaining a desirable
safety culture. In the area of defining, she indicated that there are multiple definitions and suggested that thedifferences and similarities be determined. She suggested that there may be specific differences in thedefinition of culture for the nuclear field and asked why the differences exist. One question of particularinterest is how safety and security culture are complementary and what aspects of each could negatively
impact the other? She went further into describing the various levels of defining culture: Cultural differences – what impact do national or regional cultural differences have on thedefinition of culture?
 Industry differences – what can we learn from each other?
 Subcultures within organizations – how do we use these subcultures when defining the overall
culture?
 Subcontractor cultures – how are subcultures integrated into the parent organization‟s culture?
With regard to assessing safety culture she proposed research to answer questions such as whether astandardized assessment methodology can work. Also, at what level of construct can the instruments bevalid, and what types of data can be correlated with assessment results to demonstrate leading indicators ofsafety culture? Further, can organizations effectively self-assess their own safety culture? What are someproblems with self-assessment and how can the problems be addressed? Many regulatory organizations

38

NEA/SEN/SIN/WGHOF(2012)1
require independent third party assessment, but the process of conducting satisfactory third party
assessment remains unclear. Given the narrow band of variability among High Reliability Organizations(HRO), can HRO be meaningfully researched?
Dr. Haber suggested that research in the area of changing safety culture could include: Does effective safety culture look different at different stages of an organization‟s cycle: design,
construction, operation, accident, recovery?
 Can an effective safety culture just be thought of as an effective leadership style? How so? What
are the implications?
 Standards in safety culture are not clearly defined; what to change to; how to „benchmark‟?
 The focus has been on process and outcomes, not behavior; will new efforts require a paradigm
shift?
Finally, for the area of maintaining safety culture, research into measures of safety performance that
correlate with measures of safety culture need to be established. Approaches to balancing all stakeholder
needs with maintaining an effective safety culture, e.g., production vs. safety, need to be developed.
Research into how organizations can move past a compliance mentality and minimally acceptable criteria
to setting higher goals for themselves would also be useful.
The first working group took a broad approach to identifying a research effort by suggesting a single
topic – Identify where to look for strengths and weaknesses in various organizational cultures. Theirapproach was to explore other industries via interviews, surveys, etc., and perform a literature review onsafety culture in the nuclear and other industries. They proposed that this would identify and address gaps
in knowledge, such as the lack of a benchmark for an effective safety culture, the qualifications of people
needed to perform safety culture assessments, and identification or development of effective assessment
tools. To accomplish this they would need access to power plants and other industry facilities.
Working group 2 proposed a similar exploratory effort but focused on new builds in the nuclear
industry. This would include looking at contractors, multi-national vendors, new organizational systems andmodels, new human-system interfaces and concepts of operation, e.g., networked organizations. This could
be accomplished using workshops and previous IAEA work. The group also suggested several other research
projects:
 Transfer of expertise from contractors to plant staff
 Defining the boundaries of safety culture Focus on what we can do to improve by adopting successful approaches
 Create a compendium of major safety culture initiatives/case studies/definitions across industries
 Determine how HROs and safety culture relate
The plenary discussions focused on safety culture, its components, and its measurement. Further, theimportance of research to establish the relationship between safety culture and operational performance
was stressed. Lessons learned from research on HROs was suggested to better understand the role ofsafety culture. Application of what we know about safety culture in operational settings to new builds was
also stressed.
Human Factors Engineering (HFE) Methods and Tools
The challenge speaker for this topic was Mr. Chris Plott, an SME in human performance modelingand simulation at Alion Science and Technology. In his talk he described how human performance models
are being used in other industries, such as the Department of Defense and in the transportation industry.

39

NEA/SEN/SIN/WGHOF(2012)1
He also talked about the measures of human performance that can be used in the models, such as: Time and accuracy.
 Workload and situation awareness.
 Staffing and personnel characteristics.
 Algorithmic assessment of performance shaping factors including:– Fatigue.
– Training/Skill Level.
– Stressors such as heat, cold, and vibration.
– Personnel Protective Equipment.
Further, he described how the Federal Railroad Administration (FRA) has developed, with Alion, asimulator for its own use that can be applied: As a tool for safety assessments.
 For technology assessments related to human performance.
 To identify roles and functions for automation.
 To demonstrate technology innovations.
 As a methodology for assessing efficiency, capacity, and operational performance.
Based on this experience he challenged the nuclear regulators to consider having a tool similar to the
FRA‟s for their own use.
His final challenge to the working groups was to address the question: What are the strengths andlimitations of new HFE methods and tools and what criteria should be applied to evaluating theiracceptability for use in advanced reactor control station designs? He included in this question three parts:
 Who should use what tools, when and for what purpose?
 Are there merits to joint development and application?
 What does it take to make a tool “acceptable?”
The two working groups approached the task differently. The first group suggested two projects. The
first project was to clarify and validate an integrated approach to HFE design and evaluation, where humanfactors specialists are included early in the design stage. Topics such as: the timing of inclusion of HFE inthe design process; the use of a graded approach to HFE evaluation; the skills of practitioners; integrationof users into development; the appropriate sharing of analyses, data, and models among designers, humanfactors specialists, and management; and the need for a suite of tools to perform needed tasks. The second
project suggested developing and implementing a process for validation of the various HFE tools andmethods. Such validation would provide a more consistent and transparent process, ensuring that HFE
information and considerations are applied in the appropriate places in the design process.
The second group did not propose a specific project but suggested areas for HFE tool (e.g., humanperformance models, visualization) improvement. These included: linking tools to a risk assessment
framework, improving tools for the development and capture of both quantitative and qualitative data,
reconsidering tools that may not have been previously considered ready for implementation in the nuclear
arena (e.g., human behavioural modeling or virtual reality), and better distribution and frequent update of
the NRC/BNL report on methods and tools (BNL-90424-2009, “Trends in HFE Methods and Tools andTheir Applicability to Safety Reviews”), including tools outside the traditional HFE tool box.
The use of HFE tools was discussed from both the research and design standpoint. The application of
more advanced tools such as human performance modeling and simulation is better known in the researchand design communities than in regulatory and nuclear industry communities. Thus, the focus of discussion
was on how to enhance the acceptance of these tools to these latter communities. Verification and validationof the tools and their use in capturing both quantitative and qualitative data was stressed.
40

NEA/SEN/SIN/WGHOF(2012)1
Chair’s Summary
Dr. Valerie Barnes, the Chair of the workshop, presented her analysis of what she saw as six crosscutting themes that emerged over the course of the workshop. The first theme was “Drivers for Change” orunderstanding what drives the decision to adopt new technologies. These drivers included: obsolescence(equipment/workforce), more precise control/other production goals, reduced costs (primarily staffing),
improved safety, regulatory requirements, or simply because new technology is available.
The second theme was “Scope of Change” or the ripple effect that adoption of new technology has on
the entire organization. Dr. Barnes stated that the workshop discussions pointed out that a focus on operator
performance is too narrow and that we have to look at the entire organization and plant system. Adoption of
new technology can significantly change skill sets and staffing needs, particularly for maintenance personnel
and others (trainers, engineers, field operators), and this widespread impact may have unplanned,
unanticipated consequences.
The third theme was that the “Design Process” can be improved. The workshop discussions indicatedthat most design processes still generally occur serially, with a lack of early and on-going coordination andcommunication among designers, human factors personnel and users. The mismatches with regard togoals, methods, and processes among designers, users, and managers are still prevalent. Integration could
be improved through the use of tools such as rapid prototyping, modeling, and visualizations.
Theme four, “Design Principles,” related to the design process, but focused more on concepts such as
designing for both cognition and collaboration, where there may be a much larger potential for data overload.
With advanced technology there may be life-cycle challenges, where users‟ needs and behaviours changewith experience. Also design should account for a wide scope of design scenarios that include outages,
transitions (e.g., degraded I&C/HSI), severe accidents, and interactions outside the main control room.
“Test, Evaluation, Review” was the fifth theme and focused on measures. Dr. Barnes stated that the
speakers and discussions demonstrated that there are mature methods and tools available for use in research,
design and regulatory reviews. Further, there are numerous dependent (or intermediate) measures available
that are primarily defined by task goals and typically include situation awareness, workload, accuracy,
teamwork, cost/efficiency, and perhaps perceived complexity. However, consensus on acceptance criteria is
still lacking and there is an outstanding question on the usefulness of the data which are collected for
application to HRA.
The final theme was “Implementation.” Dr. Barnes focused on two areas where the workshop
demonstrated that there are still weaknesses in our ability to implement the desirable research on the eight
topics. The first was operating experience, which exists and is constantly being generated, but for which
there are many barriers, e.g., dissemination, level of detail, negativity, applicability, and prejudices.
Further, OpEx may not be used effectively by the industry or regulators. The other topic was safety culture.
The focus of safety culture efforts has been on operating facilities and we now see safety culture problems
occurring in new builds. There is also the ongoing problem that there is no international agreement on thedefinition of safety culture or which assessment methods are desirable.
Dr. Barnes concluded that: Viable research can be performed in each of the WGHOF TOP topic areas.
 The state-of-the-art in human factors continues to advance.
 There is consensus on the appropriate approaches for conducting research in each topic area.
 Human factors expertise in the nuclear domain is a limiting factor.

41

NEA/SEN/SIN/WGHOF(2012)1
She also summarized the discussions by recommending that the HOF community:
 Enhance methods for reporting, disseminating, and using operating experience.
 Pursue development of a “grand unified design process”.
 Identify areas where clear criteria can be established regarding which methods and tools are
acceptable for use, the situations in which they are most useful, and criteria can be developed forevaluating results derived from applying those tools.
 Continue work on safety culture.

42

NEA/SEN/SIN/WGHOF(2012)1

SUMMARY

Survey Results
The general results from the survey indicated that every research and development topic surveyed hassome level of ongoing or planned activity. That is, no research area is completely resolved (i.e., only
studied in the past), Some research topics showed a significant amount of ongoing or planned activity, andcertain areas seem to be getting the most current attention, e.g., ongoing activities are focused on
assessment of safety culture and capturing and using operating experience (OpEx) for application to safety
culture issues; activities being planned are in the areas of effects of new technologies, new concepts ofoperation, and multi-unit reactors. Respondents also described the facilities they are using to perform theirresearch. Numerous published documents were identified as were specific researchers.
Workshop Results
Proposed research projects for each topic area are briefly summarized below. Because of the inherent
overlap and interdependence in the topic areas, e.g., concepts of operation and automation, some of the
projects may appear to be duplicative, and in some cases are, but with a different focus.
Operating Experience from New and Modernized Plants
The proposals from the operating experience topic area focus on gathering and analyzing OpEx fromoperating plants, new plant construction, and plant modernizations to identify further lessons learned that are
unique to the implementation of new technologies. The proposals also suggest the need to collect informationon the application of new technologies from other industries that have more experience in applying the newtechnologies than does the nuclear industry. From this work, lessons learned can be derived and used todevelop a good practices characterization of an effective OpEx program from development throughimplementation. One purpose of this work is to develop validated guidelines and standards for HSI issues
such as alarms, use of overview displays, and information presentation during outages. Discussion on this
topic also brought to light the importance of properly qualified HFE practitioners, the limited availability of
HFE education specific to the nuclear domain, insufficient knowledge and skills with the new technology by
the owners, and insufficient understanding of plant behavior among the contractors.
Evolving Concepts for the Operation of Nuclear Power Plants
The concepts of operation topic area was closely tied to the topic area of automation when the groups
were developing projects because automation is a major factor in new concepts of operation. The focus of the
projects developed was on determining when to encourage shared control between humans and automation
and when to assign final authority to the humans. This leads to a study of operator roles that separates themonitoring of plant status and supervision of automation. The study would identify effective tools to designand evaluate alternative concepts of operation through an analysis and dynamic linking of the goals,
functions, activities, tasks, and processes. This in turn can lead to guidance on how to design automation
through development of more intelligent automation algorithms and adapting automation to human work
style, which includes transparency in the process by presenting automation goals and activities to operatorsand the use of visualization technologies. Elements of the design tools could be used by the regulator toassess proposed concepts of operation.

43

NEA/SEN/SIN/WGHOF(2012)1
The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
As stated above, there is a close interconnection between concepts of operation and automation. In the
topic of automation the concept of agents as part of the operational context is introduced, where an agent
can be a person, a piece of equipment, soft- or firm-ware, or a computer operated surveillance system.
Therefore, a leading question to pursue in this topic area is: What is the appropriate allocation of
responsibilities to each agent in the system and how/when should the responsibilities change? This in turnleads to several potential projects on use of modeling to simulate teamwork, trust in automation, reliabilityof the automated system, and management of automation failures and degraded conditions. Perhaps before
much of this research is pursued one might ask what is driving the move toward automation and how mightthat lead to identifying high-payoff opportunities for the application of automation?
Management of Unplanned, Unanticipated Events
Proposals for this topic suggested research into whether the ability to handle the unexpected is apersonality factor or a function of better mental models that support projections, diagnosis, and anomaly
detection. Regardless there were proposals for establishing technical bases for: design and review guidelines
for systems to support detection, state awareness, and diagnosis of unexpected or low probability events,
including a plant state indicator; developing appropriate procedures; and developing training programs to aid
in response to unanticipated events. As part of these efforts, establishing a database focused on unanticipated
events and event precursors in nuclear and other industries was seen as a useful step. This database can thenbe analyzed to determine if there are any effects of individual and organizational differences and training on
avoiding and handling/mitigating unanticipated events.
Human System Interface Design Principles for Supporting Operator Cognitive Functions
The focus of research proposals for this topic area was on developing a better understanding of what
creates cognitive burden in NPP operations and how to best deploy computer technologies to facilitateproceduralized tasks, communications, and teamwork. This might include benchmarking best practices
with various design solutions and experience of other industries. Research was also suggested that would
consider the potential for fostering resilient performance under unanticipated conditions through merging
procedures, plant data, and controls. A specific proposal was to develop detailed guidance and acceptancecriteria for addressing integrated system validation.
Complexity Issues in Advanced Systems
As a starting point for the complexity topic area, research proposals are to define complexity with thegoal of developing a tool to measure system and perceived complexity so that a better understanding ofdifferent types of complexity can be achieved. This can then lead to determining the impact of complexity
on human performance and to ways to minimize the perceived complexity through new HSI design byfiltering less important information while making information available when needed and by matching
design and operator mental models. Part of this research could determine how to balance increased
technical reliability (and complexity) with better human reliability (which may be negatively impacted bythis complexity),
Organizational Factors: -- Safety Culture
The research proposals for the organizational factors topic area start with creating a compendium ofmajor safety culture initiatives including case studies of high reliability organizations across industries.
One purpose for this is to determine if effective safety culture looks different at different stages of anorganization‟s life cycle: design, construction, operation, accident, and recovery. This effort could lead to
identifying where to look for strengths and weaknesses in various organizational cultures and what
44

NEA/SEN/SIN/WGHOF(2012)1
approaches have been successful. In the past the focus of safety culture assessment has been on process
and outcomes. Research to change the focus to assessing behavior may lead to more effective assessment
tools and more clearly defined standards.
Human Factors Engineering Methods and Tools
In the methods and tools topic area, several related projects were proposed that addressed determining
the strengths and limitations of new HFE methods and tools and the criteria that should be applied toevaluating their acceptability for use in advanced reactor control station designs. As part of that
determination the researchers might ask: Who should use what tools, when and for what purpose? What are
the merits to joint development and application? And how might tools be deemed acceptable? This could
result in validation of existing human performance models for the nuclear arena, where criteria forvalidation are established along with regulatory acceptance criteria and potential integration with a risk
assessment framework.
Workshop Chair’s Summary
The Chair of the workshop summarized the outcome of the meetings as a series of six cross-cutting
themes. These were: Drivers for change or what drives the decision to adopt new technologies; Scope ofchange or the ripple effect that adoption of new technology has on the entire organization; the Design
process can still be improved; Design principles focused on designing for concepts such as cognition andcollaboration; Test, evaluation, and review or the lack of consensus on acceptance criteria; andImplementation where there are still weaknesses in implementing existing knowledge in the collection anduse of operating experience and assessment of safety culture. The Chair went on to conclude that research
projects can be pursued in all of the topic areas, that the state of the art is constantly advancing, that there
was a consensus among the participants regarding the approaches to research, and that a lack of humanfactors expertise in the nuclear domain is a limiting factor. She went on to recommend: enhancement forreporting, disseminating, and using operating experience needs to be developed; that an integrated design
process be developed, including the use of projective tools; areas where clear acceptance criteria for using
methods/tools and evaluation results can be identified and established; and that work be continued in thearea of safety culture.

45

NEA/SEN/SIN/WGHOF(2012)1

46

NEA/SEN/SIN/WGHOF(2012)1

CONCLUSIONS AND RECOMMENDATIONS

Based on the results of the survey and workshop, including group discussions, and combined with theinformation in the TOP, this report has a number of recommendations, both for future WGHOF activities
and for participating organizations. These recommendations will be considered by WGHOF in theirplanning of future working group activities.
An important conclusion from the information considered from this report is that individualorganizations have their own needs and drivers for the implementation of advanced HSI systems. Therefore,
it is not worthwhile for WGHOF to attempt to reach consensus on which research areas should receive
attention first. Rather, WGHOF will forward the TOP and this report to all participating organizations fortheir use in establishing research priorities.
A second recommendation aimed at participating organizations is that participating organizations
make use of the information provided in the report to seek out and establish collaborative arrangements to
conduct research of common interest. To the extent allowed by its charter, WGHOF may facilitatecommunications among parties.
One topic that cut across all of the discussions at the workshop and was highlighted in Section 3.5.1 andby the Workshop Chair was the need for enhanced operating experience information. Therefore, WGHOF
proposes to collaborate with WGOE and WGNR to form a task force to study the issue of enhanced
collection of operating experience information in HOF areas of interest and the development of a joint
information repository. This effort could be initiated through a jointly sponsored CAPS and workshop.
Another topic that was raised in several group discussions, and reflected in the Workshop Chair‟ssummary as a cross-cutting theme, was the need for test, evaluation, and review methods and tools thatprovide quantitative information, for which clear acceptance criteria can be established. This is an area inwhich WGHOF may take the lead to encourage researchers, designers, and regulators to identify orrecommend development of such methods and tools and make the information available to members.
The results of the workshop support WGHOF continuing to work on organizational factors/safety
culture and human factors of advanced control systems. In the organizational factors/safety culture area,
the WGHOF proposes to continue its efforts to identify means to analyze safety culture. As part of thiseffort they might investigate research into high reliability organizations, as suggested by the challenge
speaker for the organizational factors topic, and the potential application of resilience engineering, assuggested in some of the group discussions. The safety culture of new builds should also be a focus of
WGHOF efforts.
For the human factors of advanced control systems, several studies were proposed at the workshop as
well as those identified through the survey and the TOP. There is interest in and ongoing efforts related toalarm systems, procedures (especially computerized procedures), HSIs, and automation. There is also aneed for more research into measures and acceptance criteria for research and integrated system validation.
Further, there is interest in how these issues affect concepts of operation, and applications to control room
modernization, new control stations for new reactors, and the impact of small modular reactors. WGHOF
proposes to follow developments in these areas and plan workshops that focus on specific topics, e.g.,concepts of operation, automation, human factors engineering tools, performance measures and acceptancecriteria, the potential application of resilience engineering, and the impact of HSIs on response tounexpected events and cognitive models.
47

NEA/SEN/SIN/WGHOF(2012)1
Finally, WGHOF plans to closely follow the EURATOM, Man-Machine-Organization throughInnovative Orientations for Nuclear (MMOTION) project because it is addressing so many efforts ofcommon interest. WGHOF notes that the TOP on which this workshop was based, and the workshop
results, were used as inputs to the research agenda developed by the MMOTION project. The WGHOF
encourages funding authorities to implement the MMOTION research agenda and share research results.

48

NEA/SEN/SIN/WGHOF(2012)1
Attachment 1
Survey Instrument

WGHOF Survey
Page 1 – Heading
Thank you for your participation.
The purpose of this survey is to identify research and development (R&D) activities that may enhance thestate of knowledge related to the human and organisational factors (HOF) aspects of control station
modernisations in existing plants, new nuclear power plants (NPPs) and advanced reactors.
The eight broad topic areas are included in this survey are:
1. Operating Experience from New and Modernised Plants.
2. Evolving Concepts for the Operation of Nuclear Power Plants.
3. The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems.
4. Management of Unplanned, Unanticipated Events.
5. Human System Interface Design Principles for Supporting Operator Cognitive Functions.
6. Complexity Issues in Advanced Systems.
7. Organisational Factors – Safety Culture/Safety Management.
8. Human Factors Engineering (HFE) Methods and Tools.
Generally, the survey gathers the following data:a) Whether your organization is performing, planning to perform, or has performed R&D in this area.
b) Names of your key personnel involved.
c) Documents you are willing to share.
d) Your interest in collaborating.
The last two pages ask some miscellaneous questions and then probe your interest in participating in a March
Workshop.
Page 1 – Question 1 – Open Ended – One or More Lines with Prompt
Please enter the following contact information:








Country:
Organization/Affiliation:
Contact person(s)Mailing address:e-mail address(es):Phone number(s)Page 1 – Question 2 – Yes or No

1/8: Operating Experience from New and Modernized Plants
Considering the number of new plants and modernized control rooms around the world little operating
experience is available. Information on the impacts of such changes on human performance is very important to
the development of future designs and as an input to the development of regulatory approaches to the safety
review of new technology.
Is your organization performing, planning to perform, or has it performed research or development activities on
Operating Experience from New and Modernized Plants?
○ Yes○ No [If No, skip to Question 27]

49

NEA/SEN/SIN/WGHOF(2012)1
Page 2 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 2 – Heading
1/8 (Cont’d): Operating Experience from New and Modernized Plants
Is your organization performing, planning to perform, or has it performed research or development
activities involving: ..............................................................................................................................................
Page 2 – Question 3 – Choice – Multiple Answers (Bullets)
Developing a methodology for gathering and analyzing human performance and human factors engineering(HFE) data from operating experience from current Generation III and III+ plants?
□ Ongoing□ Planned□ Performed

Page 2 – Question 4 – Choice – Multiple Answers (Bullets)
Identifying lessons learned from operating experience related to human performance and human factors
engineering (HFE), This applies to both your country or on a broader basis, e.g., EU-wide?
□ Ongoing□ Planned□ Performed

Page 2 – Question 5 – Choice – Multiple Answers (Bullets)
Seeking operating experience related to human performance and human factors engineering (HFE) throughdirect contacts with organizations (vendors, utilities and regulatory authorities)?
□ Ongoing□ Planned□ Performed

Page 2 – Question 6 – Choice – Multiple Answers (Bullets)
Evaluating the information collected to derive lessons learned that may be generally applicable to designing
future reactors designs and regulatory reviews?
□ Ongoing□ Planned□ Performed
Page 2 – Heading

Which of the following types of operating experience are you collecting, planning to collect, or have youcollected related to impacts on human performance and human factors engineering (HFE)?
Page 2 – Question 7 – Choice – Multiple Answers (Bullets)
Automation□ Ongoing□ Planned□ Performed

Page 2 – Question 8 – Choice – Multiple Answers (Bullets)
Characteristics of the user interfaces to the automation□ Ongoing□ Planned□ Performed

50

NEA/SEN/SIN/WGHOF(2012)1
Page 2 – Question 9 – Choice – Multiple Answers (Bullets)
Characteristics of the user interfaces for plant monitoring□ Ongoing□ Planned□ Performed
Page 2 – Question 10 – Choice – Multiple Answers (Bullets)
Characteristics of disturbance management systems and interfaces□ Ongoing□ Planned□ Performed
Page 2 – Question 11 – Choice – Multiple Answers (Bullets)
Computer-based procedures□ Ongoing□ Planned□ Performed

Page 2 – Question 12 – Choice – Multiple Answers (Bullets)
Computerized operator aids
□ Ongoing□ Planned□ Performed

Page 2 – Question 13 – Choice – Multiple Answers (Bullets)
Design of computer-based tools for performing tasks such as maintenance, equipment tagout, and testing
systems, structures and components□ Ongoing□ Planned□ Performed
Page 2 – Question 14 – Choice – Multiple Answers (Bullets)
Software upgrades and modifications□ Ongoing□ Planned□ Performed

Page 2 – Question 15 – Choice – Multiple Answers (Bullets)
Operator modifiable features such as set-point adjustment, temporary alarms, and temporary displays□ Ongoing□ Planned□ Performed
Page 2 – Question 16 – Choice – Multiple Answers (Bullets)
Large-screen overview displays□ Ongoing□ Planned□ Performed

Page 2 – Question 17 – Choice – Multiple Answers (Bullets)
Advisory systems□ Ongoing□ Planned□ Performed

Page 2 – Question 18 – Choice – Multiple Answers (Bullets)
Visualization technologies□ Ongoing□ Planned□ Performed

Page 2 – Question 19 – Choice – Multiple Answers (Bullets)
Digital safety systems□ Ongoing□ Planned□ Performed

51

NEA/SEN/SIN/WGHOF(2012)1
Page 2 – Question 20 – Choice – Multiple Answers (Bullets)
New types of training technology, such as virtual reality□ Ongoing□ Planned□ Performed
Page 2 – Question 21 – Choice – Multiple Answers (Bullets)
New technologies for assuring safety system security□ Ongoing□ Planned□ Performed
Page 2 – Question 22 – Choice – Multiple Answers (Bullets)
How operating experience analysts identify and treat risk-important personnel actions□ Ongoing□ Planned□ Performed
Page 2 – Question 23 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Operating Experience from New and Modernized Plants.

Page 3 – Heading
1/8 (Cont’d): Operating Experience from New and Modernized Plants
Page 3 – Question 24 – Open Ended – Comments Box
Please list the key personnel involved in your research activity or activities.

Page 3 – Question 25 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 3 – Question 26 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Operation Experience from New
and Modernized Plants.
□ High□ Medium□ Low
Page 4 – Question 27 – Yes or No
2/8: Evolving Concepts for the Operations of Nuclear Power Plants
The concepts of operations for some new reactor designs are different from the concept of operations
implemented at conventional NPPs, e.g., with partial decentralisation of operation away from the plant or theoperation of multiple modular reactors from a common control room. New concepts of operations may
significantly change the nature of tasks and therefore the human-system interfaces required, staffing
requirements, etc.
Is your organization performing, planning to perform, or has it performed research on: the Effects of New
Concepts of Operations on Safe and Efficient Operations?
○ Yes○ No [If No, skip to Question 45]

52

NEA/SEN/SIN/WGHOF(2012)1
Page 5 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 5 – Heading
2/8 (Cont’d): Evolving Concepts for the Operations of Nuclear Power Plants
Is your organization performing, planning to perform, or has it performed research involving:
Page 5 – Question 28 – Choice – Multiple Answers (Bullets)
The relative roles of personnel and automation in a more automated environment?
□ Ongoing□ Planned□ Performed
Page 5 – Question 29 – Choice – Multiple Answers (Bullets)
Staffing levels (number of personnel) under a new concept of operations?
□ Ongoing□ Planned□ Performed
Page 5 – Question 30 – Choice – Multiple Answers (Bullets)
Training requirements in a new concept of operations?
□ Ongoing□ Planned□ Performed
Page 5 – Question 31 – Choice – Multiple Answers (Bullets)
The role and task responsibilities of personnel under normal operating conditions?
□ Ongoing□ Planned□ Performed
Page 5 – Question 32 – Choice – Multiple Answers (Bullets)
Human performance under abnormal and emergency operating conditions?
□ Ongoing□ Planned□ Performed
Page 5 – Question 33 – Choice – Multiple Answers (Bullets)
Maintenance and change management systems and processes?
□ Ongoing□ Planned□ Performed
Page 5 – Question 34 – Choice – Multiple Answers (Bullets)
The qualifications (e.g., education or experience) that operators and other plant personnel must have?
□ Ongoing□ Planned□ Performed
Page 5 – Question 35 – Choice – Multiple Answers (Bullets)
Effects of functional decentralization on human performance?
□ Ongoing□ Planned□ Performed
Page 5 – Question 36 – Choice – Multiple Answers (Bullets)
Effects of process differences in new concept of operation on human performance?
□ Ongoing□ Planned□ Performed

53

NEA/SEN/SIN/WGHOF(2012)1
Page 5 – Question 37 – Choice – Multiple Answers (Bullets)
Effects of new reactor technologies on human performance?
□ Ongoing□ Planned□ Performed
Page 5 – Question 38 – Choice – Multiple Answers (Bullets)
The effects of operating several (modular) units from the same Control Room at facilities in other industries
where similar changes have already be implemented?
□ Ongoing□ Planned□ Performed
Page 5 – Question 39 – Choice – Multiple Answers (Bullets)
Simulator or case (e.g., field) studies to better understand the implications of different concepts of operations for
personnel performance?
□ Ongoing□ Planned□ Performed
Page 5 – Question 40 – Choice – Multiple Answers (Bullets)
Developing and testing a new regulatory review approach and guidance to address safety-important aspects ofdifferent concepts of operations?
□ Ongoing□ Planned□ Performed
Page 5 – Question 41 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Evolving Concepts for the Operations of Nuclear Power Plants.

Page 6 – Heading
2/8 (Cont’d): Evolving Concepts for the Operations of Nuclear Power Plants
Page 6 – Question 42 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

Page 6 – Question 43 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 6 – Question 44 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Evolving Concepts for the
Operations of Nuclear Power Plants.
□ High□ Medium□ Low

54

NEA/SEN/SIN/WGHOF(2012)1
Page 7 – Question 45 – Yes or No
3/8: The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
The operation of a nuclear power plant (NPP) depends on the coordinated activities of multi-person teams. With
new technology, there is a growing recognition that the automation is becoming a de facto member of control
room crews. There is a need to consider how automation is designed and implemented with respect to its
potential effects on team performance.
Is your organization performing, planning to perform, or has it performed research on: The Role of Automation
and Personnel: New Concepts of Teamwork in Advanced Systems○ Yes○ No [If No, skip to Question 56]
Page 8 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 8 – Heading
3/8 (Cont’d): The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
Is your organization performing, planning to perform, or has it performed research involving:
Page 8 – Question 46 – Choice – Multiple Answers (Bullets)
Reviewing literature on teamwork in high-technology systems and its application to nuclear systems?
□ Ongoing□ Planned□ Performed
Page 8 – Question 47 – Choice – Multiple Answers (Bullets)
Reviewing and analyzing the literature regarding the effects of teamwork on advances in automation?
□ Ongoing□ Planned□ Performed
Page 8 – Question 48 – Choice – Multiple Answers (Bullets)
Reviewing and analyzing the literature regarding the effects of teamwork on function allocation?
□ Ongoing□ Planned□ Performed
Page 8 – Question 49 – Choice – Multiple Answers (Bullets)
Reviewing and analyzing the literature regarding the effects of teamwork on human-system interface designs to
support interaction?
□ Ongoing□ Planned□ Performed
Page 8 – Question 50 – Choice – Multiple Answers (Bullets)
Identifying principles for best supporting teamwork within human-automation systems?
□ Ongoing□ Planned□ Performed
Page 8 – Question 51 – Choice – Multiple Answers (Bullets)
Testing the principles identified above using laboratory, simulator, or field studies?
□ Ongoing□ Planned□ Performed

55

NEA/SEN/SIN/WGHOF(2012)1
Page 8 – Question 52 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with the Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems.

Page 9 – Heading
3/8 (Cont’d): The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems
Page 9 – Question 53 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

Page 9 – Question 54 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 9 – Question 55 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of The Role of Automation and
Personnel: New Concepts of Teamwork in Advanced Systems.
□ High□ Medium□ Low
Page 10 – Question 56 – Yes or No
4/8: Management of Unplanned, Unanticipated Events
New designs and technologies may increase the challenge for crews to respond to beyond design basis events.
Understanding and responding to such events involves many aspects of plant design, that may be brought
together to ensure that operators are able to manage novel events without becoming dependent on the systems orcomplacent in their roles.
Is your organization performing, planning to perform, or has it performed research on: Management of
Unplanned, Unanticipated Events○ Yes○ No [If No, skip to Question 67]
Page 11 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 11 – Heading
4/8 (Cont’d): Management of Unplanned, Unanticipated Events
Is your organization performing, planning to perform, or has it performed research involving:56

NEA/SEN/SIN/WGHOF(2012)1
Page 11 – Question 57 – Choice – Multiple Answers (Bullets)
Analyzing literature related to operators‟ ability to respond to beyond-design-basis events which addresses
Resilient and flexible systems and processes?
□ Ongoing□ Planned□ Performed
Page 11 – Question 58 – Choice – Multiple Answers (Bullets)
Analyzing literature related to operators‟ ability to respond to beyond-design-basis events which addresses
Systems to aid the handling of unexpected events?
□ Ongoing□ Planned□ Performed
Page 11 – Question 59 – Choice – Multiple Answers (Bullets)
Analyzing literature related to operators‟ ability to respond to beyond-design-basis events which addresses
Systems to support situation awareness?
□ Ongoing□ Planned□ Performed
Page 11 – Question 60 – Choice – Multiple Answers (Bullets)
Analyzing literature related to operators‟ ability to respond to beyond-design-basis events which addresses
Complacency and over-dependence on automation?
□ Ongoing□ Planned□ Performed
Page 11 – Question 61 – Choice – Multiple Answers (Bullets)
Developing a set of principles for the design of human-system interfaces, procedures, and training for novel
events?
□ Ongoing□ Planned□ Performed
Page 11 – Question 62 – Choice – Multiple Answers (Bullets)
Conducting laboratory, simulator, or field studies to evaluate the principles developed?
□ Ongoing□ Planned□ Performed
Page 11 – Question 63 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Management of Unplanned, Unanticipated Events.

Page 12 – Heading
4/8 (Cont’d): Management of Unplanned, Unanticipated Events
Page 12 – Question 64 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

57

NEA/SEN/SIN/WGHOF(2012)1
Page 12 – Question 65 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 12 – Question 66 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Management of Unplanned,
Unanticipated Events.
□ High□ Medium□ Low
Page 13 – Question 67 – Yes or No
5/8: Human-System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions
Computer-based procedures (CBPs) and other computer-based operator support systems (COSSs) are
increasingly used in existing and new plants. Research indicates that CPBs and COSSs fundamentally alter how
personnel process and share information, and develop and maintain situational awareness. It may be useful toestablish standard practices for the design of CBPs and COSSs, e.g., using diverse computer-based or paperbased procedures as a back-up to system failure.
Is your organization performing, planning to perform, or has it performed research on: Human-System Interface(HSI) Design Principles for Supporting Operator Cognitive Functions○ Yes○ No [If No, skip to Question 76]
Page 14 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 14 – Heading
5/8 (Cont’d): Human-System Interface Design Principles for Supporting Operator Cognitive Functions
Is your organization performing, planning to perform, or has it performed research involving:
Page 14 – Question 68 – Choice – Multiple Answers (Bullets)
Identifying features and functions of the HSIs currently used that can support human performance?
□ Ongoing□ Planned□ Performed
Page 14 – Question 69 – Choice – Multiple Answers (Bullets)
Identifying trends in Human-System Interface (HSI) design and their effects on human performance?
□ Ongoing□ Planned□ Performed
Page 14 – Question 70 – Choice – Multiple Answers (Bullets)
Characterizing features of Human-System Interfaces (HSIs) which can be functionally integrated to support
human performance?
□ Ongoing□ Planned□ Performed

58

NEA/SEN/SIN/WGHOF(2012)1
Page 14 – Question 71 – Choice – Multiple Answers (Bullets)
Conducting laboratory, simulator, or field studies to better understand the impact of design features andfunctions on crew performance?
□ Ongoing□ Planned□ Performed
Page 14 – Question 72 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Human-System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions.

Page 15 – Heading
5/8 (Cont‟d): Human-System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions
Page 15 – Question 73 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

Page 15 – Question 74 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 15 – Question 75 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Human-System Interface (HSI)
Design Principles for Supporting Operator Cognitive Functions.
□ High□ Medium□ Low
Page 16 – Question 76 – Yes or No
6/8: Complexity Issues in Advanced Systems
One of the recurrent themes identified in recent human and organizational factors (HOF) literature is the issue ofcomplexity of computer-based control systems. Increased complexity may result, for example, from the physicalproperties of plant systems or increased amount of information that is processed by the automated systems
before it is presented to operators, as well as the sheer amount of information available to operators through the
HSIs.
Is your organization performing, planning to perform, or has it performed research on: Complexity Issues inAdvanced Systems○ Yes○ No [If No, skip to Question 87]
Page 17 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.

59

NEA/SEN/SIN/WGHOF(2012)1
Page 17 – Heading
6/8 (Cont’d): Complexity Issues in Advanced Systems
Is your organization performing, planning to perform, or has it performed research involving:
Page 17 – Question 77 – Choice – Multiple Answers (Bullets)
Reviewing and analyzing the literature on complexity in computer-based control systems from related industries
and academic domains to determine its applicability to NPP design and operations?
□ Ongoing□ Planned□ Performed
Page 17 – Question 78 – Choice – Multiple Answers (Bullets)
Obtaining subject matter expert (SME) input on the most current approaches and models of complexity?
□ Ongoing□ Planned□ Performed
Page 17 – Question 79 – Choice – Multiple Answers (Bullets)
Developing models of plant system and HSI complexity for application to NPPs?
□ Ongoing□ Planned□ Performed
Page 17 – Question 80 – Choice – Multiple Answers (Bullets)
Analysing the impact of different physical behaviours of the process (example: shorter grace delays for action,
non-linear effects, irreversible effect of systems, etc.) on the human performance?
□ Ongoing□ Planned□ Performed
Page 17 – Question 81 – Choice – Multiple Answers (Bullets)
Analysing the impacts of requirements to manage new hazards (ex: chemical hazard due to the use of sodium,high temperature hazards, etc.) on the activity of the operation teams?
□ Ongoing□ Planned□ Performed
Page 17 – Question 82 – Choice – Multiple Answers (Bullets)
Conducting laboratory or simulator research to validate complexity models and measures used?
□ Ongoing□ Planned□ Performed
Page 17 – Question 83 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Complexity Issues in Advanced Systems.

Page 18 – Heading
6/8 (Cont’d): Complexity Issues in Advanced Systems
Page 18 – Question 84 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

60

NEA/SEN/SIN/WGHOF(2012)1
Page 18 – Question 85 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 18 – Question 86 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Complexity Issues in Advanced
Systems.
□ High□ Medium□ Low
Page 19 – Question 87 – Yes or No
7/8: Organizational Factors – Safety Culture/Safety Management
Assuring that work groups on which nuclear utilities rely share a strong nuclear safety culture and meet safety
management expectations has sometimes been an issue in the current generation of plants. Further, both
operators and regulators have had difficulty in assessing plant safety culture and corrective actions implemented
by licensees.
Effective methods for transmitting and maintaining a shared, strong nuclear safety culture and assuring theimplementation of rigorous safety management practices are needed.
Is your organization performing, planning to perform, or has it performed research or development activities on:
Organizational factors-Safety Culture/Safety Management○ Yes○ No [If No, skip to Question 101]
Page 20 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 20 – Heading
7/8 (Cont’d): Organizational Factors – Safety Culture/Safety Management
Is your organization performing, planning to perform, or has it performed research or development activities
involving:
Page 20 – Question 88 – Choice – Multiple Answers (Bullets)
Reviewing literature from all industries addressing principles and methods for successful organizational culturechange?
□ Ongoing□ Planned□ Performed
Page 20 – Question 89 – Choice – Multiple Answers (Bullets)
Evaluating operating experience regarding methods that have been used successfully in the nuclear and otherindustries to assess (measure) safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 90 – Choice – Multiple Answers (Bullets)
Evaluating operating experience regarding methods that have been used successfully in the nuclear and otherindustries to transmit safety culture expectations?
□ Ongoing□ Planned□ Performed

61

NEA/SEN/SIN/WGHOF(2012)1
Page 20 – Question 91 – Choice – Multiple Answers (Bullets)
Evaluating operating experience regarding methods that have been used successfully in the nuclear and otherindustries to reinforce a positive safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 92 – Choice – Multiple Answers (Bullets)
Seeking operating experience related to safety culture and safety management through direct contacts withorganizations (vendors, utilities and regulatory authorities)?
□ Ongoing□ Planned□ Performed
Page 20 – Question 93 – Choice – Multiple Answers (Bullets)
Developing methods and techniques for assessing nuclear safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 94 – Choice – Multiple Answers (Bullets)
Developing methods and techniques for quantifying safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 95 – Choice – Multiple Answers (Bullets)
Developing methods and techniques for transmitting nuclear safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 96 – Choice – Multiple Answers (Bullets)
Developing methods and techniques for reinforcing a positive safety culture?
□ Ongoing□ Planned□ Performed
Page 20 – Question 97 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Organizational Factors--Safety Culture/Safety Management.

Page 21 – Heading
7/8 (Cont’d): Organizational Factors – Safety Culture/Safety Management
Page 21 – Question 98 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

Page 21 – Question 99 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

62

NEA/SEN/SIN/WGHOF(2012)1
Page 21 – Question 100 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Organizational Factors – Safety
Culture/Safety Management.
□ High□ Medium□ Low
Page 22 – Question 101 – Yes or No
8/8: Human Factors Engineering (HFE) Methods and Tools
Advances in the methods and tools used by human factors engineering (HFE) professionals are revolutionizing
the ways in which the professionals accomplish their tasks. As a result, both operators and regulatory authorities
will be faced with evaluating the methods and tools used for new designs as an important aspect of developmentof a safety case and for safety reviews.
Is your organization performing, planning to perform, or has it performed research on: Human Factors
Engineering (HFE) Methods and Tools○ Yes○ No [If No, skip to Question 112]
Page 23 – Heading
Note: For the items below, click all (Ongoing, Planned or Performed) that apply. If your organization is NOT
performing, NOT planning to perform, or has NOT performed research or development activities on agiven topic, then do not provide any response to these questions. We will interpret a null response fromyou on a given question as being equivalent to you saying “No” or “Not applicable”.
Page 23 – Heading
8/8 (Cont’d): Human Factors Engineering (HFE) Methods and Tools
Is your organization performing, planning to perform, or has it performed research involving:
Page 23 – Question 102 – Choice – Multiple Answers (Bullets)
Identifying trends in HFE methods and tools that are significantly impacting the nuclear and other industries?
□ Ongoing□ Planned□ Performed
Page 23 – Question 103 – Choice – Multiple Answers (Bullets)
Identifying or evaluating HFE methods and tools that could successfully be applied in the nuclear industry by
Designers?
(Examples include tools to assess plant staffing profiles and tools to examine changes to risk-important humanactions)□ Ongoing□ Planned□ Performed
Page 23 – Question 104 – Choice – Multiple Answers (Bullets)
Identifying or evaluating HFE methods and tools that could successfully be applied in the nuclear industry byRegulators?
(Examples include tools to assess plant staffing profiles and tools to examine changes to risk-important humanactions)□ Ongoing□ Planned□ Performed
Page 23 – Question 105 – Choice – Multiple Answers (Bullets)
Developing tools to support reviews of the HFE aspects of NPP design, operations, and maintenance?
□ Ongoing□ Planned□ Performed

63

NEA/SEN/SIN/WGHOF(2012)1
Page 23 – Question 106 – Choice – Multiple Answers (Bullets)
Developing methods to validate HFE methods and tools for use in the nuclear industry?
□ Ongoing□ Planned□ Performed
Page 23 – Question 107 – Choice – Multiple Answers (Bullets)
Developing regulatory review guidance to ensure that HFE methods and tools are being appropriately applied?
□ Ongoing□ Planned□ Performed
Page 23 – Question 108 – Open Ended – Comments Box
Please describe any other research activities you are performing, planning to perform, or have performed
associated with Human Factors Engineering (HFE) Methods and Tools.

Page 24 – Heading
8/8 (Cont‟d): Human Factors Engineering (HFE) Methods and Tools
Page 24 – Question 109 – Open Ended – Comments Box
Please list the key personnel involved in the research activity or activities.

Page 24 – Question 110 – Open Ended – Comments Box
Please list any publicly available documentation that describes the research you have performed in this area.

Page 24 – Question 111 – Choice – One Answer (Bullets)
Please select your level of interest in collaborating on research in the area of Human Factors Engineering (HFE)
Methods and Tools.
□ High□ Medium□ Low
Page 25 – Heading
Next to Last Page--Other General R&D Information
Page 25 – Question 112 – Open Ended – Comments Box
Please describe any research facilities that you have, to which you may have access, or may be planning to build
to perform any of the types of research described above, e.g., laboratories, research simulators, training
simulators, data collection tools, behavioural modelling tools, event data bases, etc.

64NEA/SEN/SIN/WGHOF(2012)1
Page 25 – Question 113 – Open Ended – Comments Box
Please describe any measures that you use to assess human performance and how the measures relate to plant
performance, e.g., how do the human performance measures relate to plant performance indicators or
WANO/INPO ratings?
WANO = World Association of Nuclear Operators
INPO = Institute of Nuclear Power Operations

Page 25 – Question 114 – Open Ended – Comments Box
Please describe any cooperative research agreements you currently have in place with other countries ororganizations (including industry) to perform research related to the topics described above.

Page 25 – Question 115 – Open Ended – Comments Box
Please describe any research or development issues of which you are aware that are not described above thatyou think may be of general interest.

Page 26 – Heading
Final Page: Interest in Attending WGHOF Workshop
Page 26 – Heading
Please express your level of interest in a workshop to exchange information about current and planned research,
determine the state-of-the-knowledge on the research topics above, identify areas of potential overlap anddiscuss the desirability and mechanisms for coordination, and means of collaborating.
Page 26 – Question 116 – Yes or No
Would you be interested in participating in such a workshop held in the Washington, DC area?
○ Yes
○ No
Page 26 – Question 117 – Yes or No
Would you be likely to participate: In person
○ Yes
○ No
Page 26 – Question 118 – Yes or No
Would you be likely to participate: By video conference
○ Yes
○ No
Page 26 – Question 119 – Yes or No
Would you be likely to participate: By teleconference
○ Yes
○ No

65

NEA/SEN/SIN/WGHOF(2012)1
Thank You Page
Thank you for taking the time to fill out this survey.
If you have any comments or questions about the contents of this survey,
please contact Julius Persensky in the USA (Julius.Persensky@inl.gov, +1-301-840-1829)
If you have any comments or questions about WGHOF, or the upcoming WGHOF workshop
on March 1-3, 2010 in the Washington, DC area, please contact Radomir Rehacek in France(Radomir.Rehacek@oecd.org, +33 145 24 10 58)
If you have any comments or technical questions about the survey software,please contact Jeffrey Joe in the USA (Jeffrey.Joe@inl.gov, +1-208-526-4297),
End of survey. You may now close your browser window.

66

NEA/SEN/SIN/WGHOF(2012)1

Attachment 2
References

1.

The respondents submitted the references included in this Attachment through the survey. The
references have been reformatted, but in some cases the data was scrambled in the languagetranslation, or lost through the survey entry, download, and/or results assembly process. Please contact
the respondent who submitted the reference if you have any questions.

2.

The order of the references is by: TOP topic area, the organization providing the information, and thenthe order in which the survey respondent listed the references.

3.

HRP reports older than 5 years are generally available, while reports that are less than 5 years old are
restricted to HRP members. The HRP has graciously provided short abstracts for several restricteddistribution reports. If a short abstract is included beneath an entry, it is because that report isrestricted to HRP members only.

1.

Operational Experience from New and Modernized Plants

AREVA TA
Two Articles RGN (Revue Générale du Nucléaire) sur OSCAR* Participation ŕ 2 ouvrages collectifs : Guidede la Sûreté de Fonctionnement, Les Facteurs Humains de la Fiabilité articles et communications ŕ
congrčsBNLO‟Hara, J., Gunther, W. & Martinez-Guridi, G. (2009), The Effects of Degraded Digital Instrumentation andControl Systems on Operator Performance (BNL Tech Report No. 6526-T2-8-09, Rev. 1), Upton, NY:
Brookhaven National Laboratory. O‟Hara, J., Higgins,
CEA
Papin, B. (2000), Plant Computerized Human Machine Interface: Promises, Concerns And Realities, NPIC& HMIT WASHINGTON (USA)
DNVeritas
Nelson, W.R. and Green, R.E. (2007), From Risk to Reality, 8th IEEE Conference on Human Factors inPower Plants, Monterey, CA.
Nelson, W.R.. (2008), Next Generation PSA: Insight from Industry Responses to Catastrophic Accidents,
American Nuclear Society Spring Meeting, Anaheim, CA.
Nelson, W.R. and Van Scyoc, K. (2008), Focus on Mission Success: Process Safety for the Atychiphobist,
Mary Kay O‟Connor Process Safety Center 2008 International Symposium, College Station, TX.
Nelson, W.R.. (2009), Integrating Risk Management and Safety Culture in a Framework for Risk Informed
Decision Making, Canadian Nuclear Society Annual Meeting, Calgary, Alberta.
Nelson, W.R. and Van Scyoc, K. (2009), Focus on Mission Success: Process Safety for the Atychiphobist,
Journal of Loss Prevention in the Process Industries, 22, 764-768.

67

NEA/SEN/SIN/WGHOF(2012)1
EDF R&D
Labarthe, J.P. (2004), “French EPR project: a preliminary test for assessing the feasibility of the designprinciples under consideration for computerized operation”, (Fourth American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Controls and Human- MachineInterface Technologies (NPIC&HMIT 2004), Columbus, Ohio.
Labarthe, J-P, De la Garza C. (2007) ,”The human factors evaluation program of a control room: the French
EPR approach”, Article soumis en septembre 2009 ŕ la revue: Human Factors and Ergonomics in
Manufacturing (special issue on control room design)”A new approach to express new functionalities
for operation aids. The “Wiki operation” project.” Paper presented to the EHPG-meeting, Halden.
Dionis, F., De la Garza, C., Pirus D., Grassaud A. (2008), “MERMOS: an extended second generation HRA
method”, P. Meyer, P. Le Bot, H. Pesme, présentation au congrès IEEE / HPRCT, 26-31, Monterey CA,
USA.”The impact of procedure guidance on operators work. The case of the French computerized
EOP”.
Filippi, G. (2006), “Little stories to explain Human reliability Assessment : a practical approach of the
MERMOS method”, Fifth American Nuclear Society International Topical Meeting on Nuclear Plant
Instrumentation, Controls and Human- Machine Interface Technologies (NPIC&HMIT 2006),
Albuquerque , New Mexico.
Pesme, H., Le Bot, P., Meyer, P. (2008), Présentation au congrès IEEE / HPRCT, 26-31/08/07, Monterey
CA, USA, “Taking into account the socio-organisational aspects oh the remote monitoring system in
Radiation protection”, I Fucks S. Guyot. Communication au congrès européen ISOE, ŕ Turku en
Finlande, juin 2008. “Conception d‟une nouvelle organisation du travail et coopération entre ergonomes
et sociologues”. 43ième Congrès de la Société d‟Ergonomie de Langue Française, Ajaccio, France.
Guillaume, O., JP Labarthe, “Emergency operation in nuclear power plants: proposition of analysis protocol
to highlight collective team performance in simulation situations”.
De la Garza C., Le Bot P. (2008), International conference on Probabilistic Safety Assessment andManagement, Proceedings of the PSAM9, Hong Kong, China, “Performance ahead: innovative
information and communication technologies for safer and more efficient day to day operations in
nuclear power plants”.
Coudert L. Scherrer B. Vignol N, Filippi G (2009), Sixth American Nuclear Society International Topical
Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies
EDF SEPTEN
Why and How a Functional Information System Improves Computerized Operations. ANS NPIC
Albuquerque
How a Functional Approach Allow to Set an On-Line Flexible Level of Automation For NPP‟s
Computerized Operation. ANS NPIC Columbus
EPRI
Work in EPRI reports. In general they are not available at this time except to EPRI members.
FortumKerneis, C. Methodology and principle for emergency operation computerization, ICONE16-48929, AREVANP SAS.
Longenecker
All Longenecker reports are proprietary68

NEA/SEN/SIN/WGHOF(2012)1
Mitsubishi
Non-proprietary versions of APWR HFE Technical Reports and the HFE Topical Report are available. These
include a summary of the operating experience review.
NRI RezHolý, J., (2004), Some insights from recent applications of HRA methods in PSA effort and plant operation
feedback in Czech Republic, Reliability Engineering and System Safety, Vol.83, No.2.
Oskarshamn
Most of research in this area was made together with IFE HRP and college in the Nordic countries in the
HAMBO group, which have a lot of documentation in this area. In Oskarshamn documented validation
activities are available.
SRSA
Ongoing research concerning validation methods, not yet publishedTokyo ElectricIwaki, K. (1990), Control Room Design and Automation in Advanced BWR (ABWR), IAEA Symposium,
Munich, Germany.
Ross. M. A., Iwaki, K., Makino, M., Miyake, M. (1990), Control Room Design and Automation in the
ABWR, IEEE Nuclear Power Systems Symposium, Arlington, VA, USA.
Shirakawa, T. (1992), Current Status and Perspective on Simulator for Nuclear Power Plant at TEPCO, ANS
Conference, Orlando, FL, USA.
Iwaki, K. (1992), Progress of I&C System and Control Room Design in TEPCO Nuclear Power Plants, IEEE
Human Factor Conference, Monterey, CA, USA.
Iwaki, K., Shirakawa T. (1992), Development of the ABWR Type Control Room Panels, IAEA Conference,
Tokyo, Japan.
Kawano, R., Fujiie, M., Ohtsuka, T. (1996), Evaluation of Human-Machine Interface of the ABWR Type
Control Panel based on Operator‟s Behaviors and Subjective Data, 1996·
Makino, S. (1999), Further Improvement of Human-Machine Interface for ABWR Main Control Room,ICONE-10.
Masuda, N., Imai, K., Hirata, K. (2005), Guidelines for human-machine interface designs for thecomputerized main control rooms of nuclear power plants, The 6th International Topical Meeting on
Nuclear Reactor Thermal Hydraulics, Operation and Safety, 2004.·JEAG 4617.
Guide for Development and Design of Computerized Human-Machine Interfaces in the Main Control Roomof Nuclear Power Plants, Japan Electric.
VTT
http://www.vtt.fi/publications/from this website under the key persons names
VUJE
Loen (2008), The conference hybrid control room, HRP, EHPG Proceedings of the Man-TechnologyOrganization Sessions.
WEC
There are no reports that are available to public.
69

NEA/SEN/SIN/WGHOF(2012)1
2.

Evolving Concepts for the Operation of Nuclear Power Plants

AREVA TA
2 Articles RGN (Revue Générale du Nucléaire) sur OSCAR* Participation ŕ 2 ouvrages collectifs : Guidede la Sûreté de fonctionnement, Les facteurs humains de la fiabilité articles et communications ŕ
congrèsBNLO‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues
Associated with Emerging Nuclear Plant Technology, Nuclear Engineering and Technology, 41, 225236.
EDF SEPTEN
Gol, Computerized operation using formal Plant functional breakdown HRP EHPG meeting, Norway
HRP
Hallbert, B.P., Sebok, A., Morisseau, D.S., & Persensky, J.J. (1997), The Effects of Advanced Plant
Design Features and Control Room Staffing on Operator and Plant Performance. In Proceedings ofthe IEEE Sixth Conference on Human Factors and Power Plants, (pp. 5-7 – 5-12), Orlando, FL, USA.
Hallbert, B.P., Sebok, A., Morisseau, D.S. (2000), A Study of Control Room Staffing Levels for Advanced
Reactors. NUREG/IA-0137 U.S. Nuclear Regulatory Commission, Washington, D.C., USA.
Nystad, E., Drøivoldsmo, A., and Sebok, A. (2002), Use of Radiation Maps in a Virtual Training
Environment for NPP Field Operators. OECD Halden Reactor Project. Halden, Norway: Halden
Work Report HWR-681.
Skjerve, A.B.M., Strand, S., Skraaning, G., Nihlwing, C., Helgar, S., Olsen, A., Kvilesjø, H.Ø., Meyer, G.,
Drøivoldsmo, A., & Svengren, H. (2005), The Extended Teamwork 2004/2005 Exploratory Study.
Study Plan. (HWR-791), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Strand, S., Skraaning, G., & Nihlwing, C. (2005), The Extended Teamwork 2004/2005
Exploratory Study. Preliminary Results (HWR-812), Halden, Norway: OECD Halden Reactor Project.
Hollnagel, E., Miberg, A.B. (1999), Human-Centred Automation: An Explorative Study (HWR-595),
Halden, Norway: OECD Halden Reactor Project.
Strand, S. (2001), Trust and Automation: The Influence of Automation Malfunctions and SystemFeedback
on Operator Trust (HWR-643), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Andresen, G., Saarni, R., Skraaning, G. Jr. (2001), The Influence of Automation
Malfunctions on Operator Performance, Study Plan for the Human-Centred Automation 2000
Experiment (HWR-659), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Andresen, G., Skraaning, G. Jr, Saarni, R., Brevig, L. (2001), Human-Centred Automation
2000 Experiment, Preliminary Results (HWR-660), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Strand, S., Saarni, R., Skraaning, G. Jr. (2002), The influence of Automation
Malfunctions and Interface Design on Operator Performance, The HCA-2001 Experiment (HWR686), Halden, Norway: OECD Halden Reactor Project.
Massaiu, S., Skjerve, A.B.M., Skraaning, G., Jr., Strand, S., Wćrø, I. (2004), Studying Human-Automation
Interactions: Methodological Lessons Learned from the Human-Centered Automation Experiments
1997-2001 (HWR-760), Halden, Norway: OECD Halden Reactor Project.
Skraaning Jr., G. and Skjerve, A.B.M. (2003), The Man without a Face, and other Stories about HumanCentered Automation in Nuclear Process Control, Presented at the HCI International 2003.

70

NEA/SEN/SIN/WGHOF(2012)1
Holst, B., Nystad, E., (2007), Oil & Gas offshore/onshore Integrated Operations – introducing the Brage
2010+ project. Joint 8th IEEE HFPP/13th HPRCT
Drøivoldsmo, A., Kvamme, J.L., Nystad, E., Lunde-Hanssen, L.S., Larsen, R, Berge-Leversen, T. (2007),
Integrated Operations and Insights on Functional Analysis Techniques, Joint 8th IEEE HFPP/13th HPRCT
Drøivoldsmo, A., Nystad, E. and Heimdal, J.O. (2009), Function (re)allocation in Integrated Operations –Man Technology Organisation analysis, Expert Workshop on Allocation of Functions. Zurich,
Switzerland, (ALLFNEX‟09),
Thunem, A., Drøivoldsmo, A., Nystad, E. (2009), A method for function analysis and allocation in IO-driven
MTO systems: Challenges and prospects. Sixth American Nuclear Society International Topical
Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies
Reigstad, M., Olsen, A., Drøivoldsmo, A., Helgar, S. (2004), Distributed decision making and wearable
computing, HWR-732
Strand, O.M., Johansen, P., Drøivoldsmo, A., Reigstad, M., Olsen, A., Helgar, S. (2004), Interaction andcontrol in wearable computing, HWR-733.
Sebok., A., Nystad, E. (2004), Training in Virtual Reality: A Comparison of Technology Types, HWR-734
Sebok, A., Nystad, E. (2005), Training in Virtual Reality: Qualitative Results from a Comparison ofTechnology Types, HWR-768.
Short abstract: HWR-734 describes an experiment performed to compare different types of VR
display technologies and their effects on learning. This report provides additional insights into theresults by analysing the subjective data collected during that study.
Sebok, A., Nystad, E. (2005), Radiation visualization in virtual reality: A comparison of flat andtopographic map types, presented on four different display technologies, HWR-772.
Short abstract: HWR-734 describes an experiment performed to compare different types of VR display
technologies and their effects on learning. In the study, two different ways of presenting radiation
information were compared. One was a flat radiation map with different colours for different levels ofradiation. The other was a topographic map, where radiation levels were distinguished both by colour andby the elevation of the map. The efficiency of the maps for learning radiation information, and subjective
preferences was assessed. The results indicated that the maps were each suited for different kinds of use.
It is recommended to follow up this study with further investigation of radiation map efficiency.
Louka, M. (2005), The Third Halden Reactor Project Workshop on Virtual Reality: Session Summaries
from the March 2005 Meeting, HWR-804.
Short abstract: A workshop was held in Halden 2nd-3rd March 2005 to discuss “VR in the Future
Industrial Workplace: Working Together – Regardless of Distance.” The workshop sessions anddiscussions focused on design, operations and maintenance, training, and engineering virtual reality
systems, and provided useful insights into the current state of the art of research and development in thefields of virtual and augmented reality.
Nystad, E., Strand, S. (2006), Collaboration in the virtual process plant: Supporting communication
between control room and field operators, HWR-798.
Short abstract: Virtual reality (VR) has been used in a study of a hypothetical future operational concept
for a nuclear power plant. VR made it possible to include field operators in the study by letting them
operate in a virtual plant. VR was also used for supporting communication between the team of one
control room operator and two field operators. The goals of the study were to investigate the fidelity of
the VR simulation of the plant, to investigate how the VR technology influenced collaboration andcommunication between the operators, and to find out what problems and advantages the operators
experienced in interacting with the virtual plant. These issues were investigated by quantitativetechniques (analysis of questionnaires) and qualitative techniques (interviews and observations),
Implications for design of VR systems are discussed.

71

NEA/SEN/SIN/WGHOF(2012)1
Førdestrømmen, N., Skjerve, A.B. (2007), Advanced Reactors Control Centre Design and Review –Potential Research Topics and Research Plan for 2007-08, HWR-864.
Short abstract: Work is currently initiated worldwide on several future advanced reactor designs. The
different advanced reactor designs can roughly be divided into 2 groups with respect to the anticipatedtime of deployment: Near-term reactor designs, so-called Generation III+ reactors, expected to be built
from about 2015 and onwards Generation IV reactors, expected to be built from about 2025 andonwards Generation IV designs are presently less specific with respect to control centre design than thenear-term Generation III+ reactors. On this basis, the HRP 2006-2008 programme will concentrate on
Generation III+ control centre design and review issues. The main objectives of the Halden Project
research on advanced reactors are to address issues of importance to human performance, within theframework of new operation philosophies and control centre designs. The goal for the work in 2006 is tobring forward a research plan pointing to specific topics of research for the Halden Project within thearea of advanced reactors. For 2007-08, the goal is to implement parts of the research plan.
Nystad, E. (2008), Virtual collaborative training of maintenance operation and risk awareness, HWR-860.
Short abstract: A collaborative virtual reality system for practicing a maintenance operation and learning
risk awareness has been tested with prospective users of the system. Two teams of maintenancepersonnel participated in the study, in total five participants. The participants found the system to be
useful for familiarizing with the job. They thought it would be useful for personnel new to the job forgetting an introduction to the job, or for refreshing people who have done the job before. The
participants thought virtual procedures in general could be an advantage over text procedures when
preparing for an operation. Risk visualization was used to show when the users had made a mistake. The
participants noticed the warnings, but sometimes did not understand fully what had happened. However,
the warnings seemed to work well as learning aids, because the users were careful not to make the same
mistakes the second time they worked through the procedure. A way to ensure better understanding of
the errors could be to use the error incidents as the basis for a discussion about what happened and how
it could have been avoided.
Nystad, E. (2009), Follow-up study of virtual collaborative training for maintenance operations and hazard
awareness, HWR-913.
Short abstract: As a follow-up from the collaborative training study reported by Nystad (2009), a
study has been performed to test some changes to the VR training system. The changes were expected
to lead to improvements in usability, collaboration support, simulation realism and hazard awareness.
The study indicates that all of the improvements were successful, but particularly the changes made tonavigation showed marked improvement.
Nystad, E. (2009), Lessons learned from HRP VR-based training studies, HWR-914.
Short abstract: The Halden Reactor Project (HRP) has been investigating the use of virtual reality (VR)for training nuclear power plant personnel in a series of studies. The topics for these studies haveincluded training of radiation patterns, route training, maintenance procedure training, and training offield operators. The purpose of this report is to summarize and collect the insights gained from the HRP
VR training studies.The VR training studies have found that VR training can be used as a supplement toreal training to prepare personnel for performing maintenance activities, to familiarize personnel with awork area or hazards related to work activities. Designing training sessions that utilize the interactive
nature of VR systems has been found to be important for training to be effective. Challenges facing
researchers include how to make VR training systems more usable, and how to make it easier and lesstime-consuming to build simulation environmentsSkråning, S. (2010), Coping with Automation in Future Plants: Results from the 2009 HAMMLAB
Experiment, HWR-937.
Eitrheim, M. H. R. (2010), Staffing Strategies in Future Plants: Results from the 2009 HAMMLAB
Experiment, HWR-938.
Jokstad, H. (2010), Experience of Handheld Computing from the HBWR 2008 experiment. HWR-942.

72

NEA/SEN/SIN/WGHOF(2012)1
Rindahl, G. (2010), A Study of Outage Planning with the Halden Planner Software. HWR-943.
INL
Hallbert, B. P., Persensky, P.P., Smidts, Carol, Aldemir, Tunc, Naser, Joseph. (2009), “Light Water
Sustainability Workshop on Advanced Instrumentation, Information, and Control Systems andHuman-System Interface Technologies” posted to the OSU-ACE website. Follow the link below foreasy access.http://www.nuclear.osu.edu/files/nuclear/Workshop_Final_Report.pdf
Longenecker
All Longenecker reports are proprietary
Mitsubishi
Non-proprietary versions of APWR HFE Technical Reports that cover simulator studies are available. Also
Conference proceedings papers:
Hall, R. E. et al. (2009) US-APWR Human Systems Interface System Verification and Validation Results:
Impact on Digital I&C Design, Proceedings of the 17th International Conference on Nuclear
Engineering, ICONE 17, July 12-16, 2009, Brussels, Belgium.
Hall, R.E. et al. (2009), US-APWR Human Systems Interface System Verification & Validation Results:
Application of the Mitsubishi Advanced Design to the US Market. In Proceedings of ICAPP „09,
Tokyo, Japan, May 10-14, 2009.
Mashio, K. et al. (2009), US-APWR Human System Interface System Verification and Validation. ANS
Winter Meeting, 2009
VTT
http://www.vtt.fi/publications/search from the website under the names of key persons given
3.

The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems

Alion
Luck, J., McDermott, P., Allender, L., and Fisher, A. (2006), “Advantages of Co-Location for Effective
Human To Human Communication of Information Provided by an Unmanned Vehicle.” Proceedingsof The Human Factors and Ergonomics Society 50th Annual Meeting pg 550-554.
Allender, L., McDermott, P., Luck, J., and Fisher, A. (2006), “Team Communication With And Without
Aids For Transmitting Remote Information.” Proceedings of The Human Factors and Ergonomics
Society 50th Annual Meeting 2006 pg 492-496.
McDermott, P., Fisher, A., and Allender, L. (2008), “The Transmission of Spatial Route Information inDistributed Unmanned Vehicle Teams.” Proceedings of The Human Factors and Ergonomics Society
52nd Annual Meeting 2008 pg 257-261.
BNL
O‟Hara, J, & Higgins, J. (In Press), Human-System Interfaces to Automatic Systems: Review Guidance
and Technical Basis. Upton, NY: Brookhaven National Laboratory.
O‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues
Associated with Emerging Nuclear Plant Technology. Nuclear Engineering and Technology, 41, 225236.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R., Persensky, J., Lewis, P. & Kramer, J. (2008), Human
Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants (NUREG/CR73

NEA/SEN/SIN/WGHOF(2012)1
6947), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R. (2008), Human Factors Considerations with Respect toEmerging Technology in Nuclear Power Plants: Detailed Analyses (BNL Technical Report No:
79947-2008), Upton, NY: Brookhaven National Laboratory.
O‟Hara, J. et al. (2006), Implications of Changing Concepts of Operations for Advanced Plants forRegulatory and Safety Evaluations. In Proceedings of the Conference on Future Control Station
Designs and Human Performance Issues In Nuclear Power Plants, Halden, Norway: Institute forEnergy Technology.
O‟Hara, J., et al. (2005), Insights into the Role of the Operator in Advanced Reactors. In ANS Transactions.
La Grange Park, Illinois: American Nuclear Society, Inc.
Persensky, J and O‟Hara, J. (2005), Insights into the Role of the Operator in Advanced Reactor Systems(NRC Technical Report), Washington, D.C.: U. S. Nuclear.
CEA
Papin, B., (1996), “Towards the optimum automation level in NPP operation and control: recent
achievements in the CEA‟s ESCRIME project”, EHPGM,‟LOEN (NOR)
Papin, B. (1998), “The role of a multi-agent organization in the management of NPP operational
objectives”, EHPGM, LILLEHAMMER (Nor)DNVeritas
Nelson, W. R., and Jenkins, J. P. (1985), “Experimental Applications of an Expert System to Operator
Problem Solving in Process Control”, Chemical Engineering Progress.
Nelson, W. R., and Blackman, H. S. (1987), “Experimental Evaluation of Expert Systems for Nuclear
Reactor Operators: Human Factors Considerations”, International Journal of Industrial Ergonomics,
91-100.
Blackman, H. S., and Nelson, W. R., “Techniques for Incorporating Operator Expertise Into Intelligent
Decision Aids and Training”, Reliability Engineering and System Safety, Vol. 22, Nos. 1-4.
Nelson, W. R., (1982), “REACTOR: An Expert System for Diagnosis and Treatment of Nuclear Reactor
Accidents”, National Conference on Artificial Intelligence, AAAI-82, Pittsburgh, PA.
Nelson, W. R., Sebo, D. E. and Haney, L. N. (1987), “Development of an Accident Management Expert
System for Containment Assessment”, ANS Topical Meeting on Artificial Intelligence and Other
Innovative Computer Interactions in the Nuclear Industry, Snowbird, UT.
Nelson, W. R., Sebo, D. E., and Haney, L. N. (1987), “Development of an Accident Management Expert
System for Containment Assessment”, ANS Topical Meeting on Artificial Intelligence and Other
Innovative Computer Interactions in the Nuclear Industry, Snowbird, UT.
Nelson, W. R., Byers, J.C., Haney, L.N., Ostrom, L.T., and Reece, W.J. (1995) “Lessons Learned from theIntroduction of Cockpit Automation in Advanced Technology Aircraft”, ANS Topical Meeting on
Computer-Based Human Support Systems: Technology, Methods, and Future, Philadelphia.
Nelson, W. R., Byers, J.C., Haney, L.N., Ostrom, L.T., and Reece, W.J. (1992), “Human Error Analysis of
ASRS Reports: Altitude Deviations in Advanced Technology Aircraft”, NASA Contractor Report.
Nelson, W. R., Lango, T.L., Haney, L.N., and Reece, W.J. (1999), “Human-System Safety for Advanced
Air Transportation Technologies”, NASA Contractor Report INEEL/EXT-99-00114.
Nelson, W. R., and Novack, S.D. (2003), “Real-Time Risk and Fault Management in the Mission
Evaluation Room for the International Space Station”, NASA Contractor Report INEEL/EXT-0300661.

74

NEA/SEN/SIN/WGHOF(2012)1
EDF R&D
See references listed under Operational Experience (OpEx)
GMU
Miller & Parasuraman, Human Factors, 2007
Parasuraman, Sheridan, & Wickens, 2000
HRP
Braseth, A.O., Nihlwing, C.; Svengren, H.; Veland, O.; Hurlen, L.; Kvalem, J. (2009), Lessons learned
from Halden Project research on human system interfaces. Nuclear engineering and technology. –Vol. 41, no. 3 (2009), 215-224
Braseth, A.O., Nurmilaukas, V., and Laarni, J. (2009), Realizing the information rich design for theLoviisa Nuclear Power Plant. Sixth American Nuclear Society International Topical Meeting on
Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies. NPIC&HMIT
2009, Knoxville, Tennessee, April 5-9, 2009, on CD-ROM, American Nuclear Society, LaGrange
Park, IL (2009)
Kvalem, J., Braseth, A.O., Hurlen, L., Karlsson, T., Nihlwing, C., (2009), What constitutes a good humansystem interface? Lessons learned from Halden project research. Sixth American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-MachineInterface Technologies. NPIC&HMIT 2009, Knoxville, Tennessee, April 5-9, 2009, on CD-ROM,
American Nuclear Society, LaGrange Park, IL (2009)
Laarni, J., Koskinen, H., Salo, L., Norros, L., Braseth, A.O., Nurmilaukas, V., (2009), Evaluation of theFortum IRD pilot. Sixth American Nuclear Society International Topical Meeting on Nuclear Plant
Instrumentation, Control, and Human-Machine Interface Technologies. NPIC&HMIT 2009,
Knoxville, Tennessee, April 5-9, 2009, on CD-ROM, American Nuclear Society, LaGrange Park, IL(2009)
Andresen, G., Friberg, M., Teigen, A., and Pirus, D. (2005) “Function-Oriented Display System: First
Usability Test”, HWR-789, (+ HWR-793)
Førdestrømmen, N., TASK BASED DISPLAYS – RATIONALE, DESIGN, USER TEST AND
ASSESSMENT, HWR-701.
Saarni, R., Andresen, G., and Nystad, E. (2002), “INTEGRATED TASK-ORIENTED DISPLAYSYSTEM: FIRST USER TEST”, HWR- 701:
Veland, Ø., Eikĺs, M., (2007), “STATE-OF-THE ART REPORT ON ECOLOGICAL INTERFACEDESIGN” A Novel Design for an Ultra-Large Screen Display for Industrial Process Control. HCIInternational 2007, Beijing, China HWR-750
Hollnagel, E., Miberg, A.B. (1999), Human-Centred Automation: An Explorative Study (HWR-595),
Halden, Norway: OECD Halden Reactor Project.
Strand, S. (2001), Trust and Automation: The Influence of Automation Malfunctions and System Feedback
on Operator Trust (HWR-643), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Andresen, G., Saarni, R., Skraaning, G. Jr. (2001), The Influence of Automation
Malfunctions on Operator Performance. Study Plan for the Human-Centred Automation
2000Experiment (HWR-659), Halden, Norway: OECD Halden Reactor Project.
Skjerve, A.B.M., Andresen, G., Skraaning, G. Jr, Saarni, R., Brevig, L. (2001), Human-Centred
Automation 2000 Experiment, Preliminary Results (HWR-660), Halden, Norway: OECD Halden
Reactor Project.Skjerve,
Strand, S., Saarni, R., Skraaning, G.Jr. (2002), The influence of Automation Malfunctions and Interface
Design on Operator Performance. The HCA-2001 Experiment (HWR-686), Halden, Norway: OECD75

NEA/SEN/SIN/WGHOF(2012)1
Halden Reactor Project.
Massaiu, S., Skjerve, A.B.M., Skraaning, G., Jr., Strand, S., Wćrø, I. (2004), Studying Human-Automation
Interactions: Methodological Lessons Learned from the Human-Centred Automation Experiments
1997-2001 (HWR-760), Halden, Norway: OECD Halden Reactor Project.
Skraaning Jr., G. and Skjerve, A.B.M. (2003), The Man without a Face, and other Stories about HumanCentred Automation in Nuclear Process
Torralba, B. (2010), Work Practices and New Technologies: A Review of Research and Practical
Experience. HWR-964.
Short abstract: A task of the work practices project is to perform a literature review of relevant
research in the area. The main objective is to identify how computer-based systems may affect work
practices in control rooms in high reliability organizations (particularly in the nuclear domain, but also
in other human-machine systems), where aspects such as communication, coordination, situation
awareness and workload are described.
The basic literature included is mainly based on papers from scientific research journals and technicalconferences proceedings, and provides a theoretical basis for understanding work practices in highreliability organizations.
All reports were reviewed with regard to eleven characteristics (domain of origin, type of study,
purpose of the study, theoretical underpinning, model of team performance, issues of work practices,
methods and measures, main results, recommendations, further research, and additional references),and each of the studies were presented in a summary table. Several ways to read the state of the art
report are possible, as the review can be used with regard to domain of origin, type of study,
theoretical underpinnings, methodological issues, and empirical results. Most of the reviewed studies
included recommendations and ideas for further research. As a main output of the report, the results ofthe studies indicate that much more needs to be done in this area.
Strand, S., Kaarstad, M. (2010), Work Practices 2009 HAMMLAB Study: Transparent Teamwork in Near
Future Computer-Based NPP Control Rooms. HWR-952.
Short abstract: The purpose of this HAMMLAB study was to investigate whether or not teamtransparency is an important issue by 1) obtaining user input on a set of team transparency design
initiatives 2) obtaining data on team communication, and 3) obtaining data indicating potential
performance impact. The emphasis was on obtaining data in relation to user input andcommunication. Six crews, each consisting of one shift supervisor, one reactor operator and one
turbine operator, participated in four different scenarios. In two of the scenarios, the crews operated ina control room environment designed to support team transparency through different parts of thehuman system interface and the workstation layout. In the other two scenarios, the crews operated in abaseline control room environment that was not explicitly designed for increasing team transparency.
Kaarstad, M. (2010), Work Practices: Field Study of Challenges and Opportunities in a Computer-Based
Nuclear Power Plant Control Room. HWR-953.
Short abstract: The study presented here was performed as one study within the project “Safe Work
Practices in near future control rooms”. The objectives of this project are to investigate how
computer-based systems may affect work practices in nuclear power plant control rooms and what
solutions could be suggested to overcome possible challenges. Of particular interest in this project arewhat type of information should be observable for the entire team, and how this information might bepresented to the respective team member. The aim of the field study presented here was to observe
two teams of operators in their daily work with computer-based systems, to interview operators
regarding their views and experiences on the effects of control room modernisations and digital
control room technology on operator work, and to discuss and present current ideas and get input to
new ideas.
A short summary of related empirical findings is given in the report, and advantages anddisadvantages with the field study approach are discussed. Also, some conceptual frameworks, which
76

NEA/SEN/SIN/WGHOF(2012)1
the field study is based on, are presented. These conceptual frameworks are cognitive ergonomics,
common ground, and distributed cognition.
The field study was found to be a useful method. We were able to observe and interview operators
about sources for information monitoring, what makes monitoring easy and difficult, how the teamcommunicated, the roles and responsibilities of the team members, and we were able to discuss anumber of different issues regarding the operators‟ experiences with a computer-based control room.
The findings in the field study was compared with findings from related studies, and a generalrecommendation, based on all reported studies, was that it is essential to put effort into design ofinterfaces and alarm systems and that operators should be trained prior to implementation. Some of
the findings and ideas this study generated, will be further tested and explored in a laboratory setting
at later stages in the Work practices project.
Kaarstad, M., Strand., S., Nihlwing, C. (2008), Work practices in computer-based control rooms – Insightsfrom workshop with operators. –HWR-892.
Short abstract: The objectves of the “Work practices” project are to identify how computer-based
systems may affect work practices in upgraded nuclear power plant control rooms, and to suggest
solutions on what type of information that should be observable for the entire team, and how thisinformation should be presented on the displays in a computerised control room. In order to getpractical and useful input to the project, two workshops with nuclear power plant operators were heldin the Man-Technology-Organisation (MTO) HAlden Man-Machine Laboratory (HAMMLAB)
facility. One workshop was with operators normally working in a panel-based control room. The otherworkshop was with operators from a modernized control room. The purpose of the workshops was toget input from the operators with regard to how the introduction of new technology may affect work
practices; more specifically communication, operator roles/ tasks and need for information, in nuclear
control rooms. Several advantages and challenges with the introduction of computer-based systems incontrol rooms were identified. During the workshops, also possible ways to handle the challenges that
could occur was discussed. Further literature reviews and field and laboratory studies will provide
additional input to this research program regarding work practices and computer-based control rooms.
Skjerve, A.B., Nihlwing, C., Nystad, E. (2008), Lessons Learned from the Extended Teamwork Study. –HWR-867.
Short abstract: The overall purpose of the Extended Teamwork study was to explore how familiarity
with operation in a new operational environment would affect teamwork. As compared to mostpresent plants, the operational environment used in the study was characterize by holding: 1. Higher
automation levels: Four automatic agents, who performed a large subset of the tasks that are usually
allocated to a turbine operator, were introduced. 2. Changed operator roles: Three operator roles wereintroduced, i.e., Control-room operator, Coordinator, and Technician. All operators worked fromdifferent physical locations during the study. 3. New types of collaboration technology: Thecollaboration technology included a display which allowed the Control-room operator to observe thelocation of operators in the plant, displays that allowed all operators to access the “view” ofcolleagues doing field work, and a video display allowing operators outside the control room, tomonitor the activity in the control room. The operational environment used in the study allowed arange of lessons learned to be derived on issues related to interface design and teamwork. The lessons
learned are organized under four headings: The overall effects of familiarization, co-operation acrossdistances via collaboration technology, co-operation with advanced automatic agents, and operator
roles and work practices.
Kaarstad, M., Strand, S. (2007), Work Practices – Findings from Previous HRP Studies. – HWR-848.
Short abstract: The main purpose of the Work Practices programme is to provide knowledge in order toimprove work practices (the way of operating the plant) in upgraded and computer-based control rooms.
By carrying out this programme, the intention is also to call attention to human performance challenges
when introducing computer-based systems in control rooms in order to overcome potential problems

77

NEA/SEN/SIN/WGHOF(2012)1
and to be better prepared for the future. In the initial phase of the project, work practices will be studied
by investigating possible changes in communication, operator tasks, information needs and transparency
of information sharing between crew members in computer-based control rooms. The present HWR is areview of previous research programmes performed within the Halden Reactor Project (HRP), which are
relevant for operator work practices in computer-based and upgraded control rooms. These research
programmes are Hybrid Control Rooms, Staffing, Human-Centered Automation, Procedure
Automation, Teamwork and Task Management, Operational Culture and Extended Teamwork. The
review findings confirm some of the initial ideas of the Work Practices programme of what could beinteresting and important aspects to focus on. These aspects are communication, operator tasks,
information needs and transparency of information sharing between crew members. In addition, thefindings provide some input to new ideas. The programme will not cover all identified needs for further
work in this area, but will focus on providing practical and relevant input for the nuclear industry withregard to the introduction of computer-based systems in the near future nuclear control rooms.
Skjerve, A.B., Strand, S., Skraaning G.Jr., Nihlwing, C. (2005), The Extended Teamwork 2004/2005
Exploratory Study. Preliminary Result. – HWR-812. (Reference not available until the end of 2010).
Short abstract: The report documents the preliminary outcomes of the Extended Teamwork 2004/2005
exploratory study. The purpose of the study was to contribute with empirical knowledge on the effectof a new operational concept, implying increased automation levels, changed operator roles, redefinedcompetence requirements to the operators, and new technologies to support co-operation, on
teamwork. The study was performed in the MTO Labs. The operational environment contained in the
study comprised a subset of the elements assumed to be of key importance for teamwork within theabove sketched new operational concept: High-automation levels in the form of four automatic agentsthat were able to perform a large subset of the tasks that have traditionally been allocated to a turbine
operator, and the need for operators to co-operate across distances using co-operation technology. The
study covered the early transition phase, i.e., from the time the operators were introduced to thepossible future operational environment, to the time they had completed the twelve scenarios
comprised by the study. It assessed how familiarity with operation in the new operational environment
might affect the extent and quality of teamwork. The report briefly summarizes the background andthe main characteristics of the study – but otherwise refers to the Study Plan (HWR-791) for details
on the theoretical and methodological approach. The preliminary outcomes of the study are reported.
Interesting issues were found related to how fast the operators‟ adapted to operation in new
operational setting, and to the implications of different communication patterns in term of task
distribution and support. The outcomes are preliminary in the sense that they almost exclusively cover
the results of the quantitative measurements, with the exception of initial analyses of semi-structured
interviews. A discussion of the results obtained suggests a range of issues, which it would be relevant
to address at later stages in the analysis process. Based on these suggestions, one set of additional
analyses is reported here. Finally, the initial potential practical implications of the study are outlined
and discussed with reference to all of the results that are presently available. It should be clearlystressed that the limited scope of the preliminary analyses implies that the research question only canbe partly addressed, and that the final conclusions of the study cannot be made at this point. The
intention is that the present report should be replaced by a new HWR that provides a complete
documentation and discussion of the empirical findings of the Extended Teamwork 2004/2005
exploratory study. This report is planned to be completed during year 2006.
Skjerve, A.B., Strand, S., Skraaning, G. Jr., Nihlwing, C., Helgar, S., Olsen, A., Kvilesjø, H.Ø., Meyer, G.,
Drøivoldsmo, A., Svengren, H. (2005), The Extended Teamwork 2004/2005 Exploratory Study.
Study Plan. – HWR-791. (Reference not available until the end of 2010)
Short abstract: The report documents the study plan for the Extended Teamwork 2004/2005 exploratory
study, which is performed within the Extended Teamwork HRP research program. The purpose of theresearch program is to generate ideas on how teamwork in nuclear power plants may be affected by the
introduction of new operational concepts. The Extended Teamwork 2004/2005 exploratory study
78

NEA/SEN/SIN/WGHOF(2012)1
contributes with empirical knowledge on the effect of a new operational concept, implying increased
automation levels, changed operator roles, redefined competence requirements to the operators, and newtechnologies to support co-operation, on teamwork. The Extended Teamwork 2004/2005 exploratory
study covered occurrences during the early transition phase, i.e., from the time the operators are
introduced to the possible future operational environment, to the time they have completed the twelve
scenarios comprised by the study. The study assessed how familiarity with operation in the possible
future operational environment may affect the extent and quality of co-operation. The report accounts
for the motivation for performing the exploratory study, and explains the research question. It describes
the theoretical approach, which is based on Co-operation Theory, the human-centered automation
approach, and theories on co-operation across distances, and introduces the concept extended teamwork.
It also describes the method applied: it provides a detailed description of the possible future operational
environment, including requirements with respect to autonomy and authority – both for humans and forautomatic agents, and describes the technology applied to support co-operation in the control-room
team. In addition, all measurement techniques applied in the study are accounted for (system logs,
questionnaires, interviews, etc.)
Andresen, G., Svengren, H., Heimdal, J.O. Nilsen, S. Hulsund, J.E., Bisio, R., Debroise, X. (2004),
Procedure Automation: The Effect of Automated Procedure Execution on Situation Awareness andHuman Performance, HWR-759
Skjerve, A.B. M. (2002), The Halden Cooperation Scale. Human-Automation Cooperation in Control Room
Setting, HWR-685
Braarud, P.Ø., Ludvigsen, J.T. (2002), Experimental study of the effect of Task Priority and Coordination
Strategy on Crew Performance, HWR-704
Longenecker
All Longenecker reports are proprietary
Mitsubishi
Previously listed conference proceedings papers, In addition, Emilie Roth has been doing some research on
principles for design of interfaces for planning tasks for a military command and control applications:
Scott, R., Roth, E. M., Truxler, R., Ostwald, J., Wampler, J. (2009), Techniques for effective collaborative
automation for air mission replanning. In Proceedings of the Human Factors and Ergonomics Society
53rd Annual Meeting. (pp. 202-206), Santa Monica, CA: Human Factors and Ergonomics Society.
Okayama U
Gofuku, A., Ozaki, Y., Ohi, T., Ito, K. (2003), Development of a Dynamic Operation Permission System
for CRT-based Operation Interfaces, Human-Computer Interaction: Theory and Practice (Part II)(Volume 2 of the Proceedings of HCI International 2003), pp. 1198-1202
Gofuku, A., Sato, T., (2009) Dynamic Operation Permission System for Oil Refinery Plants, The
International Journal of Intelligent Control and Systems, Vol. 14, No. 2, pp. 149- 157
VTT
http://www.vtt.fi/publications/WreathWood
Relative Risk of Workload Transitions in Positive Train Control, DOT/FRA/ORD-07/12, March 2007,
FRA.

79

NEA/SEN/SIN/WGHOF(2012)1
4.

Management of Unplanned, Unanticipated Events

BNL
O‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues Associatedwith Emerging Nuclear Plant Technology, Nuclear Engineering and Technology, 41, 225-236.
O‟Hara, J., Gunther, W. & Martinez-Guridi, G. (2009), The Effects of Degraded Digital Instrumentation
and Control Systems on Operator Performance (BNL Tech Report No. 6526-T2-8-09, Rev. 1), Upton,
NY: Brookhaven National Laboratory.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R., Persensky, J., Lewis, P. & Kramer, J. (2008), Human
Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants (NUREG/CR6947), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R. (2008), Human Factors Considerations with Respect toEmerging Technology in Nuclear Power Plants: Detailed Analyses (BNL Technical Report No:
79947-2008), Upton, NY: Brookhaven National Laboratory.
O‟Hara, J., Higgins, J., and Kramer, J. (2000), Advanced information systems: Technical basis and humanfactors review guidance (NUREG/CR-6633), Washington, D.C.: U.S. Nuclear Regulatory Commission.
CEA
Papin, B. et al. (1988), Display of accident operating conditions procedures based on the generalized state
approach for computerized control rooms – NUCSAFE Avignon, France.
Papin, B. et al. (1990), Operator assistance for emergency operation in computerized control rooms, ENC 90,
Lyon, France.
Papin, B., (2010), “Operator guidance in plant operation; characterization and orientations for better teamperformance and socio-technical system resilience”, EHPGM‟, Storefjell, Noway.
DNVeritas
Hanson, D. J., Blackman, H. S., Nelson, W. R., Wright, R. E., Leonard, M. T., and DiSalvo, R. (1988),“Analysis of Containment Venting at the Peach Bottom Atomic Power Station”, Nuclear Engineeringand Design, 108.
Nelson, W. R., and Blackman, H. S. (1986), “Human Reliability Analysis for Venting a BWR Mark I
During a Severe Accident”, ANS/ENS Topical Meeting on Operability of Nuclear Power Systems in
Normal and Adverse Environments, Albuquerque, NM.
Nelson, W. R., Sebo, D. E., and Haney, L. N. (1987), “Development of an Accident Management Expert
System for Containment Assessment”, ANS Topical Meeting on Artificial Intelligence and Other
Innovative Computer Interactions in the Nuclear Industry, Snowbird, UT.
Nelson, W. R., Hanson, D. J., and Solberg, D. E. (1988), “Identification of the Operating Crew‟s Information
Needs for Accident Management”, American Nuclear Society Meeting, Washington, D. C., USA.
Blackman, H.S., Hahn, H.A., and Nelson, W.R. (1992), “Complex Human Performance Measurement inan Aviation Environment”, Human Performance, Vol. 5, No. 4.
Nelson, W.R. and Novack, S.D. (2003), “Real-Time Risk and Fault Management in the Mission Evaluation
Room for the International Space Station”, NASA Contractor Report INEEL/EXT-03-00661.
HRP
Kvalem, J., Braseth, A.O., Hurlen, L., Karlsson, T., Nihlwing, C., (2009), What constitutes a good humansystem interface? Lessons learned from halden project research. Sixth American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine80

NEA/SEN/SIN/WGHOF(2012)1
Interface Technologies. NPIC&HMIT 2009, Knoxville, Tennessee, April 5-9, 2009, on CD-ROM,
American Nuclear Society, LaGrange Park, IL (2009)
Førdestrømmen, N. (date unknown) Task Based Displays – Rationale, Design, User Test and Assessment,
HWR-701
Saarni, R., Andresen, G. and Nystad, E. (2002) (newer research on task-based displays is not yet publicavailable) “Integrated Task-Oriented Display System: First User Test.” HWR- 701
Okayama U
Wakabayashi, J., Tashima, S., Gofuku, A. (1985), Two Identification Techniques for DisturbanceDiagnosis in Nuclear Power Plants, Nuclear Technology, Vol. 70, pp. 343-353
Gofuku, A., Yoshikawa, H., Hayashi, S., Shimizu, K., Wakabayashi, J. (1988), Diagnostic Techniques of a
Small-Break Loss-of-Coolant Accident at a Pressurized Water Reactor Plant, Nuclear Technology,
Vol. 81, pp. 313-332
Gofuku, A., Hirata, T., Wakabayashi, J. (1990), Disturbance Identification Method of Pressurized Water
Reactor Plant Operated with Automatic Frequency Control by Applying Projection Operator
Technique, Journal of Nuclear Science and Technology, Vol. 27, No. 5, pp. 431-449.
Ringhals
Boring, R.L. (2009), HRA for Design, IFE/HR/F-2009/1422
Furniss, D., and Hildebrandt, M. (2009), Resilience Engineering for NPPs. IFE/HR/F-2009/1241.
Oxstrand, J., and Boring, R.L. (2009), Human reliability for design applications at a Swedish nuclear
power plant: Preliminary findings and principles from a user-needs analysis. Proceedings of the 2nd
International Symposium on Resilient Control Systems, New York: Institute of Electrical andElectronics Engineers. (pp. 5-10),
Oxstrand, J., and Boring, R.L. (2009), Human Reliability Guidance – How to Increase the Synergies
Between Human Reliability, Human Factors, and System Design & Engineering, Phase 1: The Nordic
Point of View – A User Needs Analysis. NKS-R-77 Interim Report.
Boring, R.L., J. Oxstrand, and M. Hildebrandt,(2009), “Human reliability analysis for control room
upgrades”, Proceedings of the Human Factor and Ergonomics Society 53rd Annual Meeting, pp.
1584-1588.
SNL
Because of identified risk factors, this information is considered sensitive and is not publicly available.
USNRC
See BNL‟s response.
VTT
http://www.vtt.fi/publications/Wreath Wood
A Technique for Human Error Analysis (ATHEANA), NUREG/CR-6350.

81

NEA/SEN/SIN/WGHOF(2012)1
5.

Human System Interface (HSI) Design Principles for Supporting Operator Cognitive Functions

Alion
Wickens, C., Sebok, A., Bagnall, T., and Kamienski, J. , (2007) “Modeling of Situation AwarenessSupported by Advanced Flight Deck Displays.” Proceedings of The Human Factors and Ergonomics
Society 51st Annual Meeting, 794-798.
Wickens, C., McCarley, J., Steelman-Allen, K., Sebok, A., Bzostek, J. and Sarter, N., (2009) “NT-SEEV:
A model of attention capture and noticing on the Flight Deck.” Proceedings of The Human Factors
and Ergonomics Society 53rd Annual Meeting pg 769-773.
Sebok, A., Wickens, C., Quesada, S., Socash, C., and Sarter, N. (2009), “A Design Advisor for Supporting
Cognitive Aspects of Human Automation Interaction”, Sixth American Nuclear Society International
Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface
Technologies.
Wickens, C., Hooey, B., Gore, B., Sebok, A., Koenecke, C., and Salud, E. (2009), “Predicting Pilot
Performance in Off-Nominal Conditions: a Meta-Analysis and Model Validation”, Proceedings of the
Human Factors and Ergonomics Society 53rd Annual Meeting pg 86-90.
Steelman-Allen, K., McCarley, J., Wickens, C., Sebok, A., and Bzostek, J. (2009), “N-SEEV: A
Computational Model of Attention and Noticing”, Proceedings of the Human Factors and Ergonomics
Society 53rd Annual Meeting pg 774-778.
AREVA TA
Presentation at the Halden workshop on computerized procedures in Trnava, 2009BNL
O‟Hara, J, & Higgins, J. (In Press), Human-System Interfaces to Automatic Systems: Review Guidance
and Technical Basis. Upton, NY: Brookhaven National Laboratory.
O‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues
Associated with Emerging Nuclear Plant Technology. Nuclear Engineering and Technology, 41, 225236.
O‟Hara, J., Gunther, W. & Martinez-Guridi, G. (2009), The Effects of Degraded Digital Instrumentation
and Control Systems on Operator Performance (BNL Tech Report No. 6526-T2-8-09, Rev. 1), Upton,
NY: Brookhaven National Laboratory.
O‟Hara, J & Persensky, J. (2009), Human Performance and Plant Safety Performance: Establishing a
Technical Basis and Framework for Evaluating New Human-System Interfaces (Chapter 4), In
Skjerve, A., Bye, A., Øwre, F., Haugset, K., Skraaning G. Jr., Boring, R. & Neeb, J. (Eds) The
HAMMLAB History – 25 Years of Experimental Research and Development, Halden, Norway:
Institute of Energy Technology.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R., Persensky, J., Lewis, P. & Kramer, J. (2008), Human
Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants (NUREG/CR6947), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R. (2008), Human Factors Considerations with Respect toEmerging Technology in Nuclear Power Plants: Detailed Analyses (BNL Technical Report No:
79947-2008), Upton, NY: Brookhaven National Laboratory.
O‟Hara, J., Persensky, J. & Szabo, A. (2006), Development of Human Factors Engineering Guidance for
Safety Evaluations of Advanced Reactors. In the Proceedings of the ANS International Meeting on
Instrumentation, Control, and Human Machine Interface Technology. La Grange Park, Illinois:
American Nuclear Society, Inc.
82

NEA/SEN/SIN/WGHOF(2012)1
O‟Hara, J. et al. (2006), The Role of Human-System Interface Guidelines in the Design and Review of
Innovative Technology. In Proceedings of Workshop on Human System Interfaces – Design and
Evaluation. Halden, Norway: Institute for Energy Technology.
O‟Hara, J. & Roth, E. (2005), Operational Concepts, Teamwork, and Technology in Commercial Nuclear
Power Stations. In C. Bowers, E. Salas, & F. Jentsch (Eds.) Creating High-Tech Teams: Practical
Guidance on Work Performance and Technology. Washington, DC: American Psychological
Association. (Chapter 7, pp 139-159),
O‟Hara, J. et al. (2004), Regulatory Review of Advanced and Innovative Human-System Interface
Technology. In the Proceedings of the ANS International Meeting on Instrumentation, Control, andHuman Machine Interface Technology. La Grange Park, Illinois: American Nuclear Society, Inc.
O‟Hara, J. et al. (2004), Human Factors Engineering Program Review Model (NUREG-0711, Rev. 2),
Washington, D.C.: U.S. Nuclear Regulatory Commission.
O‟Hara, J., Pirus, D., Nilsen, S., Bisio, R., Hulsund, J-E, and Zhang, W. (2003), Computerization ofprocedures: Lessons learned and future perspectives (HPR-355), Halden, Norway: Institute forEnergiteknikk – OECD Halden Reactor Project.
O‟Hara, J. & Brown, W. (2002), The effects of interface management tasks on crew performance andsafety in complex, computer-based systems. (NUREG/CR-6690), Washington, D.C.: U.S. Nuclear
Regulatory Commission.
O‟Hara, J., et al. (2002), Human system interface design review guideline (NUREG-0700, Rev 2),
Washington, D.C.: U.S. Nuclear Regulatory Commission.
Roth, E. & O‟Hara, J. (2002), Integrating digital
CEA
Papin, B., et al. (2003), “Design and first evaluation of a prototype of function-based NPP display dedicatedto the shift supervisors” – Workshop on innovative HSI and their evaluation, Halden, Norway.
DNVeritas
Nelson, W. R. and Jenkins, J. P. (1985), “Experimental Applications of an Expert System to Operator
Problem Solving in Process Control”, Chemical Engineering Progress.
Nelson, W. R. and Blackman, H. S. (1987), “Experimental Evaluation of Expert Systems for Nuclear
Reactor Operators: Human Factors Considerations”, International Journal of Industrial Ergonomics, 2,
91-100.
Blackman, H. S. and Nelson, W. R. “Techniques for Incorporating Operator Expertise Into Intelligent
Decision Aids and Training”, Reliability Engineering and System Safety, Vol. 22, Nos. 1-4.
Nelson, W. R. (1980), “Response Trees for Emergency Operator Action at the LOFT Facility, “ANS/ENS
Topical Meeting on Thermal Reactor Safety, Knoxville, TN.
Nelson, W. R. (1982), “REACTOR: An Expert System for Diagnosis and Treatment of Nuclear Reactor
Accidents”, National Conference on Artificial Intelligence, AAAI-82, Pittsburgh, PA.
Nelson, W. R. (1984), “Experimental Evaluation of an Expert System for Nuclear Reactor Operators”,
International Conference on Artificial Intelligence: Potential for Applications, Villeneuve-lesAvignon, France.
Nelson, W.R., Byers, J.C., Haney, L.N., Ostrom, L.T., and Reece, W.J. (1993), “Lessons Learned from
Pilot Errors Using Automated Systems in Advanced Technology Aircraft”, ANS Topical Meeting on
Nuclear Plant Instrumentation, Control and Man-Machine Interface Technologies, Oak Ridge TN.
Nelson, W.R., Byers, J.C., Haney, L.N., Ostrom, L.T. and Reece, W.J. (1995), “Lessons Learned from theIntroduction of Cockpit Automation in Advanced Technology Aircraft”, ANS Topical Meeting on
Computer-Based Human Support Systems: Technology, Methods, and Future, Philadelphia.
83

NEA/SEN/SIN/WGHOF(2012)1
Nelson, W. R. and Blackman, H. S. (1985), “Response Tree Evaluation: Experimental Assessment of anExpert System for Nuclear Reactor Operators”, NUREG/CR-4272.
EDF R&D
See references listed under Operational Experience (OpEx)
EDF SEPTEN
Development and Evaluation of a Function-Oriented Display System: Background and Initial Results.
ISOFIC
HRP(most references already mentioned)
Braseth, A.O., Nihlwing, C.; Svengren, H.; Veland, O.; Hurlen, L.; Kvalem, J. (2009), Lessons learned
from Halden Project research on human system interfaces. Nuclear engineering and technology. –Vol. 41, no. 3 (2009), 215-224
Braseth, A.O., Nurmilaukas, V., and Laarni, J. (2009), Realizing the information rich design for theLoviisa Nuclear Power Plant. Sixth American Nuclear Society International Topical Meeting on
Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies. NPIC&HMIT
2009, Knoxville, Tennessee, April 5-9, 2009, on CD-ROM, American Nuclear Society, LaGrange
Park, IL (2009)
Kvalem, J., Braseth, A.O., Hurlen, L., Karlsson, T., Nihlwing, C. (2009), What constitutes a good humansystem interface? Lessons learned from halden project research. Sixth American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-MachineInterface Technologies. NPIC&HMIT 2009, Knoxville, Tennessee, April 5-9, 2009, on CD-ROM,
American Nuclear Society, LaGrange Park, IL (2009)
Laarni, J., Koskinen, H., Salo, L., Norros, L., Braseth, A.O., Nurmilaukas, V., (2009), Evaluation of theFortum IRD pilot. Sixth American Nuclear Society International Topical Meeting on Nuclear Plant
Instrumentation, Control, and Human-Machine Interface Technologies. NPIC&HMIT 2009,
Knoxville, Tennessee, April 5-9, 2009, on CD-ROM, American Nuclear Society, LaGrange Park, IL.
Andresen, G., Friberg, M., Teigen, A. and Pirus, D. (2005), Function-Oriented Display System: First
Usability Test. HWR-789, (+ HWR-793)
Saarni, R., Andresen, G. and Nystad, E. (2002), Integrated Task-Oriented Display System: First User Test.
HWR- 701.
Veland, Ø. (2004), “State-Of-The Art Report On Ecological Interface Design.” HWR-750.
Bisio, R., Hulsund, J.E., Nilsen, S. (2001), COPMA-III: A tool for developing effective computerizedprocedure systems.
Kaarstad, M., Strand, S., Nihlwing, C. (2009), Introduction of Computer-Based Systems in Near Future
Nuclear Power Plant Control Rooms. Experiences from a Small-Scale Empirical Study. 17th IEA
conference, Beijing. HWR-644
Kaarstad, M., Nystad, E., Strand, S. (2007) (accepted for publication), Work Practices and Cooperation ina Near Future and Far Future Operational Environment, BOOK CHAPTER 18. In: A.B. Skjerve andA. Bye (Eds.), Simulator-Based Human Factors Studies Across 25-Years. The History of the Halden
Man-Machine Laboratory, Springer.
Veland, Ø., Eikås, M. (2007), “STATE-OF-THE ART REPORT ON ECOLOGICAL INTERFACEDESIGN” A Novel Design for an Ultra-Large Screen Display for Industrial Process Control. HCIInternational 2007, Beijing, China HWR-750
Førdestrømmen, N. (2004), Task Based Displays – Rationale, Design, User Test and Assessment, HWR84

NEA/SEN/SIN/WGHOF(2012)1
751.
Andresen, G., Friberg, M., Teigen, A., Pirus, D. (2004), Function-Oriented Display System: Backgroundand First Prototypes. – HWR-752
Veland, Ø. (2004), Workshop Meeting on “Innovative Human-System Interfaces and their Evaluation”,
1st-2nd September 2003. HWR-753
Bisio, R., Hulsund, J.E., Nilsen, S. (2005), Experience from Using COPMA-III as an Integrated Procedure
Maintenance Support Tool. – HWR-782.
Short abstract: The maintenance of operating procedures in a NPP is a complicated task. It is
important that everything relevant to a procedure is revised and updated in case there are any changes
to the process. Several aspects of the procedure must be considered in a revision process. Pertinentdetails and attributes of the procedure must be checked. An organizational structure must be created
and responsibilities allotted for drafting, revising, reviewing and publishing procedures. Available
powerful computer technology provides solutions within document management and computerisationof procedures. These solutions can also support the maintenance of procedures. Not all parts of theprocedure life cycle are equally amenable to computerised support.
Andresen, G., Friberg, M., Teigen, A., Pirus, D. (2005), Function-oriented display system: first usability
test. – HWR-789. (Reference not available until the end of 2010)
Short abstract: This report describes the results of the first usability test of a function-oriented Human
System Interface (HSI) prototype in HAMMLAB. The prototype was implemented on the FRESH
Pressurized Water Reactor (PWR) simulator, covering the feedwater system and the steam generators.
Operating displays, computerised procedures, monitoring displays and an alarm system were
implemented. The test was a formative usability test; i.e., the main purpose was for the development
team to get feedback on the usability of the prototype. Three turbine operators from the Swedish
Ringhals PWRs participated. Each operator went through six scenarios involving both normal andabnormal plant conditions. Several data-collection techniques were used: observations, videorecordings, questionnaires and interviews. The results indicate that the overall usability of the
protototype was good. The operators particularly liked the overview display, navigation hierarchy andcomputerised procedures. There were also display elements the operators experienced problems withusing. For example, the operators experienced some difficulties using the disturbance procedures.
Detailed analysis of interaction breakdowns revealed that the structure of the procedure flowchart andlacking information about what procedure steps had been executed, contributed to these difficulties.
The report discusses the main findings and makes recommendations for the continuation of thedevelopment work.
Welch, R., Friberg, M., Nystad, E., Teigen, A., Veland, Ø. (2005), EID – Prototype design & User test
2004. – HWR-790.
Short abstract: The purpose of the Ecological Interface Design (EID) project in the HRP 2003-2005
programme is to gain insight into how this methodology can contribute to the design of operator
displays in the nuclear industry. To do this, it was decided to design a limited number of displays on
the FRESH simulator and conduct a user test to examine whether operators would be able to use andaccept this type of design. The FRESH EID displays intend to show information and relationships in agraphical form that would require substantially more mental resources to utilize if using theconventional displays. This HWR presents the background for EID, the analysis process, the displaysthat have been designed, the user test and the outcome of the user test. This first attempt at developing
and evaluating an EID has provided both valuable practical lessons learned and promising results forfurther work.
Svengren, H., Strand, S. (2005), Design of the task-based display prototype and the first user-test, HWR-792.
Short abstract: This report provides a description of the Task-based displays prototype for BWRs, anddocuments the results of the first user-test. Three different kinds of Task-based displays were
designed: 1) displays for performing first checks and obtaining overview in emergency situations, 2)85

NEA/SEN/SIN/WGHOF(2012)1
displays for performing emergency procedures, and 3) displays that facilitate understanding of theprocess and logic. The procedures included an expert-help function to support the operators in
performing the procedures. The design was only implemented on the reactor side of the plant. The
user-test was conducted using the HAMBO simulator in HAMMLAB. 6 crews from Forsmark 3 andOskarshamn 3 participated in the test. The crews participated in 4 scenarios, which all wereconsidered as highly complex scenarios that included actuation of several reactor protection signals.
The data collection techniques (questionnaires and interviews) intended to cover usability-related
issues and mainly involved the reactor operators. The results demonstrated that the Task-based design
approach was very well received by the operators, and that the work initiated by the design of this first
prototype should be continued. The operators generally found the procedure-implementation and thedisplays useful and easy to understand. They considered the amount of information contained in theprocedures and the displays as good, and associated with low levels of frustration. The expert-help
provided in the current test was very premature. This resulted in somewhat poorer usability ratingscompared to the procedures and the displays. However, the operators explicitly stated the usefulnessof providing expert-help in the combination with computerized procedures. The operators rated thecomputer-based procedures as slightly easier, less time-consuming, and slightly less mentally
demanding than paper-based procedures. All the operators considered navigation within and between
the procedures as easy, although some recommendations were made to improve the navigation
further. The operators gave several detailed comments with regard to the procedures, the displays andthe expert-help. Some implications for further design and improvement are pointed out.
Andresen, G., Teigen, A., Broberg, H., Friberg, M. (2005), Function-Oriented Display System: Lessons
Learned From the Development Process. – HWR-793.
Short abstract: The function-oriented displays and alarm project has developed a series of display
prototypes to investigate strengths and weaknesses of a design philosophy called Function Oriented
Design (FOD), The purpose of this report is to summarise the lessons learned so far from thedevelopment work. After presenting the main characteristics of the design concept, the lesson learned
from four prototypes are described. Next, the main phases of the development process are outlined:
goals-means analysis, hierarchical decomposition, function specification, procedure specification anddisplay implementation. This is followed by the development team‟s interpretation andimplementation of two central characteristics of the design concept: the function area and the inservice signal. Finally, some of the challenges associated with developing displays for research
purposes are discussed.
Collier, S., (2005), Human factors guidelines for large-screen displays. – HWR-796.
Short abstract: Any control-room project (including upgrades or evolutionary improvements toexisting control-rooms) is well advised at the outset first to gather and update related backgroundmaterial for the design. This information-gathering exercise should also take into account experiencefrom similar projects and operating experience. For these reasons, we decided to use our research, andexperience in large-screen display design with several clients to update human factors guidance forlarge-screen displays, to take into account new ergonomics guidelines, operating experience, andwork from similar projects. To write the updated guidelines, we drew on much of our experience
across several departments at IFE, including research funded by the HRP programme, and experience
with individual clients. Guidance here is accordingly focused mainly on recent areas of technical andhuman innovations in the man-machine interface. One particular area of focus was on the increasinguse of large-screen display systems in modern control-rooms, and on how guidelines could be adaptedand supplemented for their design. Guidance or reference to recommended sources is also given forcontrol suite arrangement and layout, control-room layout, workstation layout, design of displays andcontrols, and design of the work environment, especially insofar as these ergonomic issues interact
with the effectiveness of modern displays, in particular large screen displays. The work shows that
there can be synergy between HRP research and bilateral activities: the one side offers a capability todevelop tools and guidelines, while the other side gives an opportunity to test and refine these in86

NEA/SEN/SIN/WGHOF(2012)1
practice, to the benefit of both parties.
Skraaning, G. Jr, Lau, N., Welch, R., Nihlwing, C., Andresen, G., Brevig, L.H., Veland, Ø., Jamieson,
G.A., Burns, C., Kwok, J. (2007), The Ecological Interface Design Experiment (2005), – HWR-833.
Short abstract: The objective of the Ecological Interface Design experiment (2005) was todemonstrate the presumed benefits of ecological displays during unanticipated NPP events. The study
was performed in HAMMLAB on the HAMBO BWR simulator with six participating crews. Eachcrew consisted of one reactor operator and one turbine operator. We compared ecological displays to
traditional computerized displays in the detection and mitigation phase of within design basis, andbeyond design basis scenarios. The ecological displays were implemented only on the turbine side of
the process. Therefore, the scenarios and the data analysis focused on the turbine operator in eachcrew. Even though the ecological displays provided process information according to EcologicalInterface Design (EID) principles, a traditional process mimic was integrated and used forintervention and control of the system. The experiment concentrated on how the display types affected
the operators‟ Situation Awareness (SA), A model of SA for nuclear process control was developed,
extracting three dimensions of operator cognition in the control room: (a) Process overview, (b)Scenario understanding, and (c) Metacognitive accuracy (degree of realistic self-assessment), The
hypothesis was that ecological displays would support the Situation Awareness of NPP operators inbeyond design basis events, and during the detection phase of the scenarios. The findings suggest thatthe ecological displays supported Situation Awareness in the detection phase of beyond design basis
scenarios. If the operators were given more training, and the ecological design elements fully
supported intervention with the process, it is possible that the benefits of EID would extend to themitigation phase of beyond design basis scenarios as well.
Strand, S., Svengren, H., Nihlwing, C., Kristiansen, L.I., Andresen, G., Meyer, B.D. (2007), Task-based
displays – prototype extensions and the second user test. – HWR-841(Reference not available until
the end of 2010)
Short abstract: The overall goal of the Task-based Display (TBD) project is to make it easier for theoperators to complete procedures within a reasonable amount of time, and to minimize the risks forerroneous operations. The TBD concept developed within the frames of this project aims to integrate
procedures with ordinary process displays, and thereby to replace paper based procedures incomputerised control rooms. On the basis of the feedback obtained in a first user test conducted in
2004/2005, design improvements have been made, and the concept have been broadened to include
the turbine operator and, to some extent, the shift supervisor. Currently, three different kinds ofdisplays are developed for the BWR-simulator in HAMMLAB. The three displays complement eachother, and together they constitute the “Task-based display (TBD) concept” as applied in this project:
The Procedure selection and Overview Display (PSOD), the Procedure Performance Display (PPD),
and the Eventdependent Assistance Display (EdA), The procedures are selected in the PSOD, and thecorresponding PPD and EdA displays then appear. The PPD is applied for performing the selected
procedure. The EdA display contains information about the most important parameters andcomponents relevant for the actual situation and event, and the information presented on this display
thus depends on the selected procedure and the overall situation. All displays are continuously
updated on the basis of actuated safety systems and procedure status. A second user test was
conducted in January 2006. Three crews participated in four scenarios. The results showed that theoperators were generally very positive toward the TBD approach, both in relation to the overall TBD
concept and the respective displays. Valuable input to further refinement of the displays was provided
by the participants.
Welch, R., Braseth, A.O., Nihlwing, C., Skraaning, G. jr., Teigen, A., Veland, Ø., Lau, N., Jamieson, G.A.,
Burns, C.M., Kwok, J. (2007), The 2005 ecological interface design process and the resulting
Displays. – HWR-847.
Short abstract: The design principles called Ecological Interface Design (EID) provide a theoreticalbasis and design guidance for developing novel displays for monitoring and controlling complex
87

NEA/SEN/SIN/WGHOF(2012)1
systems such as industrial processes. EID is a central research topic in the Halden Reactor Project(HRP) research programme on innovative Human System Interfaces and this HWR describes theanalysis phase, the display design phase, the completed display designs and operator feedback for the
HAMBO BWR simulator EID displays. The work domain analyses and the display design wereconducted simultaneously in the three work groups, the HRP, the University of Toronto and the
University of Waterloo. This allowed us to investigate how different types of teams perform Work
Domain Analyses or design displays that are supposed to function together, and what challenges canappear when teams working geographically distributed are trying to create compatible Abstraction
Hierarchies and displays.
Veland, Ø., (2007), Halden Reactor Project Workshop on Human System Interfaces (HSI) Design and
Evaluation. – HWR-865.
Short abstract: A workshop was held at Park Hotel in Halden 4-5 May 2006 with a total of 34 external
participantsand Halden staff. It featured presentations of experiences, findings and results from
Human System Interface (HSI) work of the Halden Project and other organizations, as well as otherrelevant input to the Project‟s work in this area. While new computer-based information displays areclearly improvements over the control boards of the past, their similarity in both information content
and overall display logic is apparent. The power of digital technology to improve the organization andpresentation of information to plant personnel is yet to be fully utilized, and the papers anddiscussions at this workshop were focused on breaking out of this pattern in order to provide
approaches to information presentation that better meet the needs of plant personnel. In the last halfday session the external participants were invited to join Halden staff in exploring strategic issues anddirections for further work and thus take part in shaping the new HRP strategy for HSI research in the
future (2007-2012),
Veland, Ø. (2007), Human System Interface Research Strategy for the Halden Project. –HWR-866. Shortabstract: This document describes the strategy plan developed in 2006 to direct the Project‟s researchand development work within the Human System Interface (HSI) area in the period 2007 to 2012. The
background and motivation for the strategy is explained, as well as the process by which it has beendeveloped. The approach defined by the strategy is to focus future efforts in this area into two
substantial projects with clearly defined goals: 1) Development of a near term unified HSI concept
aimed to meet the needs of upcoming control room modernization projects in the nuclear industry,and 2) Establishing a more far-reaching HSI concept for use with future plant designs and operational
concepts.
Skraaning, G. Jr., Nihlwing, C. (2008), The impact of graphically enhanced mimic-displays on operator
Task Performance and Situation Awareness, HWR-871.
Short abstract: Computer graphics offer fantastic possibilities with respect to how process information canbe displayed to operators, but this potential is not taken advantage of in nuclear process control.
Instead, computerized interfaces are more or less direct translations of conventional analog control
panels. We conducted a simulator experiment in HAMMLAB to evaluate the effect of a graphicallyenhanced computerized interface on human performance. An example of a “graphical enhancement”
is the integration of small trend diagrams for key system parameters into traditional process mimics. It
was expected that such design elements would improve operator Task Performance and Situation
Awareness. The graphically enhanced interface was compared experimentally to a traditional
interface without graphical enhancements in rule-based and knowledge-based scenarios.
Lau, N., Skraaning G. jr., Jamieson, G.A., Burns, C.M. (2008), The Impact of Ecological Displays on
Operator Task Performance and Workload, HWR-888.
Short abstract: Laboratory studies have shown that ecological interfaces can enhance operator performancein process control. However, the nuclear industry needs empirical evaluation in representative settingsin order to justify the adoption of the Ecological Interface Design (EID) framework. This reportpresents an empirical study as a first step towards the validation of EID in the nuclear domain. The
88

NEA/SEN/SIN/WGHOF(2012)1
empirical study compared the operator task and workload performance of ecological displays against
mimic-based displays for the turbine side of a boiling water reactor plant. The results suggest that
ecological displays have a marked advantage in supporting operator performance during monitoringfor unanticipated events as compared to mimic-based displays. This study provides supporting or
validation evidence that EID is effective at a scale and level of complexity that is representative of
nuclear power plant operations.
Veland, Ø. (2010), Theoretical Background for Design Patterns, HWR-932.
Hurlen, L., Andresen, G. (2010), Design Patterns for Large Screen Displays, HWR-933.
Braseth, A.O. (2010), IRD-based Large Screen Displays for NPPs, HWR-934
Hurlen, L. (2010), Transparent Automation Displays – Design and Evaluation, HWR-935.
Hurlen, L., Koskinen, H. (2010), Touch Interfaces for NPP Operation: Emerging Technology andApplication Concepts, HWR-940.
Nilsen, S.R., Hulsund, J.E. (2010), Standardization of Procedure Content Definitions, HWR-962.
Krueger
Not so available publically, mostly involved in examination of equipment operators working extensively
long work shifts and having experienced sleep deprivation, lapses of attention, etc. and fatigue.
Mitsubishi
Previously listed proceedings papers.
Okayama U
Gofuku, A., Ishiga, Y. “An Experimental Study to Evaluate the Applicability of Displaying Plant
Condition Based on MFM Model by Measuring Eye Fixation Points”, CD-ROM Proc. International
Symposium on Symbiotic Nuclear Power Systems in the 21st Century, pp.298SATech
Endsley, M. R., Bolte, B., & Jones, D. G. (2003), Designing for situation awareness: An approach tohuman-centered design. (London: Taylor & Francis),
SRSA
Thunberg, A. (2008), Utformning av alarm system i svenska kärnkraftverk, SKI rapport:43 (in Swedish): A
cognitive approach to design of alarm system for nuclear power plant control room, Chalmers
University of Technology.
USNRC
See BNL‟s response. Other documents are under development.
VTT
http://www.vtt.fi/publications/
VUJE
See publ. on The Conference Hybrid Control Room89

NEA/SEN/SIN/WGHOF(2012)1
6.

Complexity Issues in Advanced Systems

BNL
O‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues Associatedwith Emerging Nuclear Plant Technology, Nuclear Engineering and Technology, 41, 225-236.
O‟Hara, J., Gunther, W. & Martinez-Guridi, G. (2009), The Effects of Degraded Digital Instrumentation
and Control Systems on Operator Performance (BNL Tech Report No. 6526-T2-8-09, Rev. 1), Upton,
NY: Brookhaven National Laboratory.
O „Hara, J., Higgins, J., Brown, W. & Fink, R., Persensky, J., Lewis, P. & Kramer, J. (2008), Human
Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants (NUREG/CR6947), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R. (2008), Human Factors Considerations with Respect toEmerging Technology in Nuclear Power Plants: Detailed Analyses (BNL Technical Report No:
79947-2008), Upton, NY: Brookhaven National Laboratory.

CEA
Papin, B., et al. (2006), The operational complexity index: a new method for the global assessment of thehuman factor impact on the safety of advanced reactors concepts, Nuclear Engineering and Design, 236
DNVeritas
Blackman, H.S., Hahn, H.A., and Nelson, W.R. (1994) “Complex Human Performance Measurement in an
Aviation Environment”, Human Performance, Vol. 5, No. 4, 1992.W.R. Nelson, “Application of
Functional Models to System Design, Operation, and Performance Assessment”, 1994 International
Conference on Systems, Man, and Cybernetics, San Antonio.
Nelson, W.R. (2000) “Critical Function Models for Operation of the International Space Station”, ANS
Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface
Technologies, Washington DC.
Nelson, W.R., Lango, T.L., Haney, L.N. and Reece, W.J. (1999), “Human-System Safety for Advanced
Air Transportation Technologies”, NASA Contractor Report INEEL/EXT-99-00114.
Nelson, W.R. and Novack, S.D. (2003), “Real-Time Risk and Fault Management in the Mission Evaluation
Room for the International Space Station”, NASA Contractor Report INEEL/EXT-03-00661.
Nelson, W.R. (2008), “Next Generation PSA: Insight from Industry Responses to Catastrophic Accidents”,
American Nuclear Society Spring Meeting, Anaheim, CA.
Nelson, W.R. and Van Scyoc, K. (2008), “Focus on Mission Success: Process Safety for the
Atychiphobist”, Mary Kay O‟Connor Process Safety Center 2008 International Symposium, College
Station, TX.
HRP
Lauman, K., Braarud, P.Ø., Svengren, H. (2005), The Task Complexity Experiment. Halden reactor
Project, HWR-758
Braarud, P. Ø. (2000) Development Of A Diagnostic Complexity Questionnaire. Halden reactor Project,
HWR-536.
Braarud, P.Ø. (2000), Subjective Task Complexity In The Control Room, Halden reactor Project, HWR621.
Andresen, G., Strand, S. (2007), Usability Questionnaires and Human System Interface Evaluations:
Review of Standardised Questionnaires and Lessons Learned from HAMMLAB. – HWR-856.
Short abstract: This report investigates usability questionnaires, a method frequently used in evaluations of90

NEA/SEN/SIN/WGHOF(2012)1
Human System Interfaces (HSI), A review was conducted, covering four standardised usability
questionnaires, and questionnaires used by the innovative design projects in HAMMLAB. The
questionnaires are reviewed on five topics: a) background, b) content and format, c) how it is used, d)
how it was developed and e) to what extent it has been validated. Challenges involved in developing
and using usability questionnaires for evaluations of HSIs are discussed. For the nuclear industry, we
claim that a standardised usability questionnaire is most useful for system validations. If thequestionnaire should assist the validation team to identify usability problems, it is necessary todevelop a questionnaire dedicated to the nuclear domain, if diagnosis is not required, an existingdomain-independent questionnaire will be sufficient. Creating norms (i.e., a reference database) for an
HSI usability questionnaire is primarily a practical challenge, requiring more than 300 data points.
However, this could be achieved if there is a “field” version of the questionnaire, making it possible tocomplete the questionnaire at the operators‟ home plant.
Braarud, P. Ø. (2010), Team Cognition in a Complex Accident Scenario. HWR-955.
OSU
Metzroth K., Denning R., Smidts C., Aldemir T., (2008), “Incorporation of a Human Reliability Model
into the ADAPT PRA Methodology”, PSAM 9: International Conference on Probabilistic Safety
Assessment & Management, CD-ROM, International Association for Probabilistic Safety Assessment
and Management, California.
SATechEndsley, M. R., Bolte, B., & Jones, D. G. (2003), Designing for situation awareness: An approach tohuman-centered design. (London: Taylor & Francis)
7.

Organizational Factors – Safety Culture/Safety Management

DNVeritas
Nelson, W.R., and Green, R.E. (2008), From Risk to Reality, 8th IEEE Conference on Human Factors in Power
Plants, Monterey, CA, August 26-31, 2007.W.R. Nelson, “Next Generation PSA: Insight from Industry
Responses to Catastrophic Accidents”, American Nuclear Society Spring Meeting, Anaheim, CA.
Nelson, W.R. and Van Scyoc, K. (2008), “Focus on Mission Success: Process Safety for the
Atychiphobist”, Mary Kay O‟Connor Process Safety Center 2008 International Symposium, College
Station, TX.
Nelson, W.R. (2009), “Integrating Risk Management and Safety Culture in a Framework for Risk
Informed Decision Making”, Canadian Nuclear Society Annual Meeting, Calgary, Alberta.
Nelson, W.R. and Van Scyoc, K. (2009), “Focus on Mission Success: Process Safety for the
Atychiphobist”, Journal of Loss Prevention in the Process Industries, 22, 764-768.
EDF R&D
Dien, Y., Llory, M. and Montmayeul, R. (2004) Investigation of Organisational Accidents: Methodology andLessons Learned Article publié dans la revue Journal of HAZARDOUS MATERIALS” Volume 111.
Dien, Y., Llory, M. (2004) “Effects of the Columbia Space Shuttle Accident on High Risk Industries or:
Can We Learn Lessons from Other Industries?” Keynote Paper de la conférence HAZARDS XVIII –Manchester.
Schram, J. (2005), “Le questionnaire de perception de la sûreté” invitation du Centre d‟étude de l‟énergie
nucléaire –Université de Liège Belgique Atelier “La politique de Sűreté dans les organisations” 22 avril.
Fucks, I. (2005), “La culture sûreté selon une démarche compréhensive” invitation du Centre d‟étude del‟énergie nucléaire et de l‟Université de Liège (Belgique) dans le cadre du “Program of investigation
91

NEA/SEN/SIN/WGHOF(2012)1
of Social Aspects into Nuclear research”
Voicu, A., Le-Bot, P. (2005), “Exploratory study of the impact of organisational factors on safety using
Probabilistic Safety Assessments” Congrès PSA‟05 San Francisco.
Mbaye, S. et al (2009), “L‟explication naďve et la perception des risques comme des voies pour améliorer
les pratiques de REX : des études dans l‟industrie chimique et l‟industrie nucléaire”. Numéro 2009-08
des Cahiers de la Sécurité Industrielle, Institut pour une Culture de Sécurité Industrielle, Toulouse,
France (ISSN 2100-3874), Disponible ŕ l‟URL http://www.icsi-eu.org/
Pierlot, S., Dien, Y., Llory. M. (2006), “Risks Management between safety requests and production
pressures, Présentation au Congrčs ESREL 2006, Estoril septembre
Dien, Y., Llory, M., Pierlot, S. (2006), “Sécurité et performance : antagonisme ou harmonie ? Ce que nous
apprennent les accidents industriels”, Présentation au Congrčs lm 15.
Pierlot, S., Dien, Y., Llory, M. (2007), From Organizational Factors to an Organizational Diagnosis of the
Safety Congrčs ESREL, Stavanger (Norway),
Voirin, M., Pierlot, S., Llory, M. (2007), Avalability Organisational Analysis : Is it a Hazard for Safety?
Séminaire ESReDA “Future challenges of accident investigation”, Ispra Italy.
Dien, Y., Pierlot, S., Voirin, M. (2008), “Internal availability organisational analysis: what are the lessons
for a high risk industrial company?” ESREL Yves
Couix S., De la Garza C., Delgoulet C. (2009), “Impacts de l‟organisation sur la “performance collective” 2études de cas en salle de commande de centrale nucléaire. In “Ergonomie et organisation du travail”,
actes de la SELF, Toulouse.
Pierlot, S., Llory, M., Dien, Y. (2009), “Resilience and vulnerability: a complementary approach in order
to diagnose the level of safety of an organisation”, ESREL 2009, Prague
Llory, M., Dien, Y., Pierlot, S., Dechy, N. “Are Lessons Learned from Accidents Really Learned and are
Potential Improvements Implemented?”, 36ième Séminaire ESReDA “Lessons learned from accident
investigations”
Magne, L. & Vasseur, D. Lavoisier éditions, Dien, Y. (2006), “Les Facteurs Organisationnels des accidents
industriels “in Risques industriels : complexité, incertitude et décision, sous la direction de : Coll.
EDF R&D.
Hollnagel, E. & al, Ashgate. Le Bot, P. (2007), “Analysis of the Scottish case”. Contribution ŕ un ouvragecollectif sur la Résilience des Organisations “Remaining Sensitive to the Possibility of Failure.
HRP
Skjerve, A.B. (2008), The use of Mindful Safety Practices at Norwegian Installations. Safety Science, 46,
1002-1015(Change and transition project for Eurocontrol – not published)
Nilsen, S., Bisio, R., Ludvigsen, J.T. (2004), Knowledge Management in the NPP Domain. – HWR-747
Heimdal, J.O. (2008), Operational culture workshop. – HWR-885.
Short abstract: This document describes the Operational Culture Workshop arranged by the OECD
Halden Reactor Project (HRP) in Halden 24th and 25th September 2007. The aim of the workshop
was to identify cultural challenges and issues, and the need for research and development of guidancematerial. The 17 participants came from seven nations and seven different organisations. Invited
speakers and participants gave presentations, and there were facilitated group sessions. The main
findings from the workshop were that the term operational culture can and should be introduced at thegroup level of an organisation, and that further research is needed in order to fully understand andcontrol operational culture. Further, that the consequences of differences in national culture should be
investigated regarding generalizability of research findings as well as its contribution to work
practices and operational culture. The results from the workshop will be used in the Operational
Culture Project.
92

NEA/SEN/SIN/WGHOF(2012)1
Heimdal, J.O. (2008), Operational Culture Literature Review. – HWR-901.
Short abstract: This literature review is one of the activities of the Operational Culture Project. The
purpose of the literature review is to explore and identify cultural aspects of work practices relevant in
the nuclear industry. The focus has been on identifying different models, theories, and methods of how
culture influences work behaviour. Relevant content for the review is material on the generalizability of
research findings with respect to national culture, and a bottom-up perspective to work practices,
including team behaviour and processes. The document is organised according to culture levels. Firstly ageneral discussion on work-related attitudes and values is presented, followed by an introduction of theconcept of culture, and then narrowing down towards national, organisational, and operational cultures
respectively. In addition, a chapter on team and group aspects of work behaviour is included. A separate
chapter describes the status of culture within the nuclear domain. Overall a number of different models,theories, and methods of how culture influences work behaviour has been identified and described. Thisknowledge will be used as input to the Operational Culture project‟s study plan, and as a theoreticalbasis for the study.
Heimdal, J.O., Skråning, G. (2010), National and Organisational Culture in an Operational Nuclear
Environment – Analyses of Cultural Measures and HAMMLAB 2009 Experimental Data. HWR-956.
IAE
No English papers available
INSSI
Fukui, H. & Sugiman, T. (2010), Organizational learning for nurturing safety culture in a nuclear power
plant. In E. Hollnagel (Ed), Safer complex industrial environments -A human factors approach-. CRC
Press. pp.171-187.
Sakuda, H., Fukui, H., Yoshida, M., Yoshiyama, N. (2005), Safety climate at nuclear power plants inJapan: Recent Findings. In N. Itoigawa, B. Wilport, B. Fahlbruch (Eds), Emerging demands for thesafety on nuclear power organizations -Challenge and response-. CRC Press. pp.45-56.
Shibaike, H. & Fukui, H. (2010), Workplace safety climate, human performance, and safety. In E.
Hollnagel (Ed), Safer complex industrial environments -A human factors approach-. CRC
JNESIshii, Y., & Makino, M. (2009), “A comprehensive evaluation of licensees‟ efforts on safety culture.” In
E.Hollnagel (Edited), Safer complex industrial environments – A human factors approach – (pp.219236), New York: CRC Press Taylor & Francis Group.
Makino, M., Hata, T. & Ogasawara, M. (2009), “A regulatory perspective on analysis practices and trends
in causes.” In E.Hollnagel (Edited), Safer complex industrial environments – A human factors
approach – (pp.133-154), New York: CRC Press
Taylor & Francis Group. NISA (2007), “Guideline for the regulatory body to evaluate the licensee‟s efforts
to prevent degradation of safety culture and organizational climate.”, NISA-166c-07-10. Tokyo:
NISA
JNES-SS report (2006), “The safety culture assessment method (The implementation manual), “, JNESSS-0616. Tokyo: JNES.JNES-SS report (2005), “Concept of challenge and view point ofunderstanding for prevention of deterioration of organizational cultute.”, JNES-SS-0514. Tokyo:
JNES.
Makino, M., Sakaue, T. & Inoue, S. (2005), “Toward safety culture evaluation tool” In Itoigawa, N.,
Wilpert, B. & Fahlbruch, B. (Eds.), Emerging demands for the safety of nuclear power operations(pp.73-84), London: CRC Press.

93

NEA/SEN/SIN/WGHOF(2012)1
KruegerNational Academy of Sciences-Transportation Research Board (TRB) Commercial Truck and Bus Safety
Synthesis Program (CTBSSP) Syntheses (five of them) most notably: Synthesis No. 15 on Health andWellness Programs for Commercial Drivers (May 2007) available at www.trb.org. Click on
publications and then on CTBSSP series.
SRSA
Reiman T. & Oedewald P. (2009), Evaluating safety critical organizations – emphasis on the Nuclear
Industry, 12 Swedish Radiation Safety Authority (SSM)
Reiman, T. & Pietikäinen, E. “Indicators of safety culture – selection and utilization of leading safety
performance indicators”, (in publish as a research report of Swedish Radiation Safety Authority)
Lowe, A. and Hayward, B. (1997), Safety Culture Enhancement Project Final Report A Field Study on
Approaches to Enhancement of Safety Culture, SKI Report 1997:26
VTT
http://www.vtt.fi/publications/WreathWood
Wreathall, J. (2000), Leading Indicators of Human Performance. ASQ Energy and Environmental Division– Annual Conference, Tucson, AZ, American Society for Quality.
Jones, A. and J. Wreathall (2000), Leading Indicators of Human Performance – The Story So Far. 6th
Annual Human Performance/Root Cause/Trending Conference, Philadelphia, PA.
8.

Human Factors Engineering (HFE) Methods and Tools

Alion
See descriptions of several of our tools at http://www.maad.com/index.pl/products
BNLO‟Hara, J., et al. (2009), Trends in HFE Methods and Tools and Their Applicability to Safety Reviews(BNL Technical Report BNL-90424-2009), Upton, NY: Brookhaven National Laboratory.
O‟Hara, J. (2009), Applying Human Performance Models to Designing and Evaluating Nuclear Power
Plants: Review Guidance and Technical Basis (BNL Technical Report BNL-90676-2009), Upton,
NY: Brookhaven National Laboratory.
O‟Hara, J, Higgins, J. & Brown, W. (2009), Identification and Evaluation of Human Factors Issues Associatedwith Emerging Nuclear Plant Technology. Nuclear Engineering and Technology, 41, 225-236.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R., Persensky, J., Lewis, P. & Kramer, J. (2008), Human
Factors Considerations with Respect to Emerging Technology in Nuclear Power Plants (NUREG/CR6947), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J., Higgins, J., Brown, W. & Fink, R. (2008), Human Factors Considerations with Respect toEmerging Technology in Nuclear Power Plants: Detailed Analyses (BNL Technical Report No:
79947-2008), Upton, NY: Brookhaven National Laboratory.
Higgins, J., O‟Hara, J. et al. (2007), Guidance for the Review of Changes to Human Actions: Final Report(NUREG-1764), Washington, D.C.: U. S. Nuclear Regulatory Commission.
O‟Hara, J. et al. (2006), Advances in HFE Methods and Their Implications for Regulatory Reviews.
Transactions of the American Nuclear Society. La Grange Park, Illinois: American Nuclear Society, Inc.
94

NEA/SEN/SIN/WGHOF(2012)1
O‟Hara, J. and Brown, W. (2004), Incorporation of Human Factors Engineering Analyses and Tools into
the Design Process for Digital Control Room Upgrades. In the Proceedings of the ANS International
Meeting on Instrumentation, Control, and Human Machine Interface Technology. La Grange Park,
Illinois: American Nuclear Society, Inc.
O‟Hara, J. and Higgins, J. (2004), Human Factors Engineering Plan For Reviewing Nuclear Plant
Modernization Programs. Stockholm: Sweden, SKI (Swedish Nuclear Power Inspectorate).
O‟Hara, J. et al. (2004), Human Factors Engineering Program Review Model (NUREG-0711, Rev. 2),
Washington, D.C.: U.S. Nuclear Regulatory Commission.
O‟Hara, J. (2002), Perspectives on Validating Complex Human-Machine System Performance. In
Proceedings of the Human Factors and Ergonomics Society – 46th Annual Meeting. Santa Monica,
CA: Human Factors and Ergonomics Society.
Higgins, J., O‟Hara, J, & Almeida, P. (2002), Improving Control Room Design and Operations Based on
Human Factors Analyses. The IEEE Conference on Human Factors and Power Plants. Washington
D.C.: IEEE.
Higgins, J., O‟Hara, J., Lewis, P., Persensky, J. & Bongarra, J. (2002), Development of a Risk Screening
Method for Credited Operator Actions The IEEE Conference on Human Factors and Power Plants.
Washington D.C.: IEEE.
O‟Hara, J., Higgins, J., & Stubler, W. (2002), Human factors evaluation of hybrid human-system interfaces
in nuclear power plants. In T. O‟Brien and S. Charlton (Eds.) Handbook of Human Factors Testingand Evaluation (2nd Edition), Hillsdale, New Jersey: Lawrence Erlbaum, Associates, Inc.
O‟Hara, J. (1999), A quasi-experimental model of complex human-machine system validation.
International Journal of Cognition, Technology, and Work, 1, 37-46.
O‟Hara, J , et al. (1997), The use of simulation in the development of human factors guidelines for alarmsystems. The IEEE Conference on Human Factors and Power Plants. Washington D.C.
Center for Operator Performancesee http://www.operatorperformance.org
DNVeritas
Nelson, W. R. and Blackman, H. S. (1987) “Experimental Evaluation of Expert Systems for Nuclear
Reactor Operators: Human Factors Considerations”, International Journal of Industrial Ergonomics,
291-100.
Nelson, W. R. and Blackman, H. S. (1998) “Techniques for Incorporating Operator Expertise Into Intelligent
Decision Aids and Training”, Reliability Engineering and System Safety, Vol. 22, Nos. 1-4.
Blackman, H.S., Hahn, H.A., and Nelson, W.R. (1992), “Complex Human Performance Measurement inan Aviation Environment”, Human Performance, Vol. 5, No. 4.
Nelson, W.R., Haney, L.N., Ostrom, L.T. and Richards, R.E. (1998), “Structured Methods for Identifying andCorrecting Potential Human Errors in Space Operations”, Acta Astronautica, Vol. 43:3-6, p. 211-222.
Nelson, W. R., Hanson, D. J. and Solberg, D. E. (1988), “Identification of the Operating Crew‟s Information
Needs for Accident Management”, American Nuclear Society Meeting, Washington, DC, Oct. 31-Nov. 4.
Nelson, W.R. (1994), “Application of Functional Models to System Design, Operation, and Performance
Assessment”, 1994 International Conference on Systems, Man, and Cybernetics, San Antonio.
Nelson, W.R. (2000), “Critical Function Models for Operation of the International Space Station”, ANS
Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface
Technologies, Washington DC.
Nelson, W.R. and Novack, S.D. (2003), “Real-Time Risk and Fault Management in the Mission Evaluation
Room for the International Space Station”, NASA Contractor Report INEEL/EXT-03-00661.
95

NEA/SEN/SIN/WGHOF(2012)1
EDF R&D
Le-Guilcher, B. (2004), “Analyse critique des normes traitant de la prise en compte des facteurs humains
pour la conception des salles de commande des centrales nucléaires” (Séminaire “Conception,
Facteurs Humains et sécurité” organisé par l‟Institut pour la Maîtrise des Risques et la Sűreté deFonctionnement (IMdR-SdF) au Ministčre de la Recherche
Le Guilcher, B. (2008), “La prise en compte des dimensions Socio-Organisationnelle et Humaine dans lesprojets d‟évolution du Parc Nucléaire” Communication ŕ la Table ronde “Prendre en compte l‟activité
dans la conception des systèmes industriels ŕ risques” au 43ième Congrès de la SELF 17-19 septembre:
Ergonomie et conception “concevoir pour l‟activité humaine”
Labarthe, J.P. (2004), “French EPR project: a preliminary test for assessing the feasibility of the designprinciples under consideration for computerized operation” (Forth American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Controls and Human- MachineInterface Technologies (NPIC&HMIT 2004), Columbus, Ohio.
Labarthe, J.P., Guyard, E. (2005), “Le REX FH dans un projet de conception : le cas du projet EPR”
congrès Société française d‟énergie nucléaire.
Labarthe J-P, De la Garza C. (2009), “Les Facteurs Humains et la Sûreté”, The human factors evaluation
program of a control room: the French EPR approach. Article soumis en septembre ŕ la revue: Human
Factors and Ergonomics in Manufacturing (special issue on control room design Organisation d‟une
session special.
De la Garza C., Fadier E. (2008), “L‟utilisateur final dans la conception: profil, rôle et impact dans lesdécisions du projet”.
De la Garza C. avec J.-P. Labarthe, (2008), présentation : “Exemple d‟un projet dans le nucléaire:
intégration des utilisateurs finaux dans la conception du réacteur EPR”, SELF, Ajaccio.
HRP
Heimdal, J.O., Skraaning, G. Jr., and Braarud, P.R. (2004), Integrated System Validation: Status And
Research Needs – HWR-754.
Sebok, A., Helgar, S., Nystad, E. (2002), Navigation in Desktop Virtual Environments: An Evaluation andLessons Learned During Experimental and Usability Studies – HWR-720
Henriksdottir, S., Fredriksen, R. (201), Modernisation of digital I&C – a field survey on requirements
engineering. HWR-945.
Lau, N. (2010), Situation Awareness in Monitoring Nuclear Power Plants – The Process Overview
Concept and Measure. HWR-954.
IAE
No English papers available
INSSI
Sone, F., & Maeda, N. (2009), Development of “Analysis Tool for Organizational and direct causes ofProblems (ATOP)”. INSS Journal.
Mitsubishi
The previously listed proceedings papers and the Non-proprietary versions of APWR HFE Technical
Reports and the HFE Topical Report describe our current approach to advanced control room V&V.

96

NEA/SEN/SIN/WGHOF(2012)1
OSU
Smidts, C., Shen, S.H., and Mosleh, A., (1997), “A Cognitive Model for Nuclear Power Plant Operation
under Accident Conditions: IDA”, Reliability Engineering and System Safety, 55, 51-71.
Shen, S.H., Smidts, C., and Mosleh, A., (1997), “A Methodology for Collection and Analysis of Human
Error Data Based on a Cognitive Model: IDA”, Nuclear Engineering & Design, 172, 157-186.
Ringhals
Oxstrand, J., and Boring, R.L. (2009), Human reliability for design applications at a Swedish nuclear power
plant: Preliminary findings and principles from a user-needs analysis. Proceedings of the 2nd
International Symposium on Resilient Control Systems, New York: Institute of Electrical andElectronics Engineers. (pp. 5-10),
Boring, R.L., J. Oxstrand, and M. Hildebrandt, (2009), “Human reliability analysis for control room
upgrades”, Proceedings of the Human Factor and Ergonomics Society 53rd Annual Meeting, pp.
1584-1588.
SATechEndsley, M. R., Bolte, B. & Jones, D. G. (2003), Designing for situation awareness: An approach tohuman-centered design. (London: Taylor & Francis),
SNL
Forester, J., Kolaczkowski, A., Cooper, S., Bley, D., Lois, E. (2007) ATHEANA User‟s Guide, HRA
Guidance NUREG-1880, U.S. Nuclear Regulatory Commission, Washington, D.C.
Boring. R.L. (2009), International HRA Empirical Study – Phase 1 Report (NUREG/IA-0216, Volume 1)
SRSA
Chockie, A. (2005) Safety-Related Contractor Activities at Nuclear Power Plants New Challenges forRegulatory Oversight, Alan Chockie, SKI Report 2005:63 SKI Regulatory Guidebook Maintenance(not published),
O‟Hara, John and James Higgins, (2005) Human Factors Engineering Plan for Reviewing Nuclear Plant
Modernization Programs, SKI Report2005:15
TecnatomPaper in last two ASME conferences (ICONE 16 & 17) and in Spanish Nuclear Society
Tokyo Electric
Furuhama, Y. (2009), “Beyond Procedures: Development and Use of the SAFER Method” in E.Hollnagel
eds. “Safer Complex Industrial Environments a Human Factors Approach” CRC Press.
Furuhama, Y. et al. (2004), Development of an “Organizational Safety Measurement Questionnaire”,
Proceedings CSEPC2004 Conference, Sendai Japan.
USNRC
See BNL‟s response
VTT
http://www.vtt.fi/publications/

97

NEA/SEN/SIN/WGHOF(2012)1
VUJE
see The Conference Hybrid Control Room98

NEA/SEN/SIN/WGHOF(2012)1

Attachment 3
Points of Contact

AECL

Survey POC and
KeyContact
Scott Malcolm

AECL

Thomas A. Moir

Organization

Chris Plott, Angie
Sebok
AREVA NP Robert Starkey
AREVA NP Douglas Hill
AREVA TA Gullermain Hubert
Alion

BackPacker

Jack Martin

BevilleDavid A. Strobhar

BNLJohn O‟Hara

CEA

Bernard Papin

CEZ

Miroslav Trnka

CNSNS

Antonio Hernandez
Maldonado

COP

Bob Slough

DNVeritas

Bill Nelson

Laine Patrick andDionis Francois
EDF SEPTEN Dominique Pirus
EDF R&D

EPRI

Joseph Naser

FKA

Anders Viklund /Jan Lövgren

FortumMikko Martinsuo

GMU

Raja Parasuraman

GRS

Jürgen Hartung

HRP

Andreas Bye and
Jon Kvalem

IAE

Yukiharu Ohga

IAE

Hiroshi Ujita

Full Name of
Email
Organization
AECL
malcolms@aecl.ca
Atomic Energy Of
moirt@aecl.ca
Canada, Ltd
Alion Science &cplott@alionscience.com,
Technology
asebok@alionscience.com
AREVA NP Inc. robert.starkey@areva.com
AREVA NP GmbH douglas.hill@areva.com
AREVA TAhubert.guillermain@areva.com
BackPacker Jack,jack.martin@itexas.net
Inc./ IEEE
Bevilledstrobhar@beville.com
Engineering, Inc
Brookhavenohara@bnl.gov
National Laboratory
Commissariat à
bernard.papin@cea.frl‟Energie Atomique
CEZ
miroslav.trnka@cez.cz
Comisión Nacionalde Seguridad
ahernandez@cnsns.gob.mx
Nuclear y
Salvaguardias
Center for Operator
sloughrj@yahoo.com
Performance
Det Norske Veritas
bill.nelson@dnv.com
Inc.

Type of
Organization
Canada Vendor

EDF R&D

Country

Canada Vendor
USA

Consultant

USA
VendorGermany Vendor
France Vendor
USA

Consultant

USA

Consultant

USA

Research

FranceResearch

Czech

Utility

Mexico Regulator

USA

Research

USA

Consultant

francois.dionis@edf.fr

FranceUtility

EDF SEPTENElectric Power
Research Institutedominique.pirus@edf.fr

FranceUtilityjnaser@epri.com

USA

Researcher

FKA

anv@forsmark.vattenfall.se

Sweden Utility

Fortum, Power
Division
George Mason
University
Gesellschaft für
Reaktorsicherheit
Institutt forenergiteknikk/
OECD Halden
Reactor Project
The Institute ofApplied Energy
The Institute ofApplied Energy

mikko.martinsuo@fortum.com Finland Utility
rparasur@gmu.edu

USAjuergen.hartung@grs.de

Germany Research

Andreas.Bye@hrp.no /Jon.Kvalem@hrp.no

Norway Research

ohga@iae.or.jp

Japan

Researchujita@iae.or.jp

Japan

Research

99

Research

NEA/SEN/SIN/WGHOF(2012)1
Full Name ofType of
Email
Country
Organization
Organization
Department of
INL
Bruce Hallbert
Energy – Idaho
bruce.hallbert@inl.gov
USA
Research
National Laboratory
Institute of Nuclear
INSSI
Hiroshi Sakuda
sakuda@inss.co.jp
Japan
Research
Safety System, Inc.
IRSN
Hélène Faye
IRSN
helene.faye@irsn.fr
France Research
ISaR
Anselm Schaefer
ISaR Instituteasc@isar.tum.deGermany Research
JNES
Ryuji Kubota
JNES
kubota-ryuji@jnes.go.jp
Japan
Regulator
Japan Nuclear
JNES
Maomi Makino
Energy Safety
makino-maomi@jnes.go.jp
Japan
Regulator
Organization
Korea Atomic
Wondea Jung
Energy Research wdjung@kaeri.re.kr
Korea
Researcher
Institute
Korea Institute of
KINS
Yougsik Yoon
Ysyoon@kins.re.kr
Korea
Regulator
Nuclear Safety
Korea institute of
KINS
DongHoon Lee
dhlee@kins.re.kr
Korea
Regulator
Nuclear Safety
Krueger
Krueger
Gerald P. Krueger
Ergonomics
JerryKrueg@aol.com
USA
Consultant
ConsultantsLongenecker andLongenecker Edward Quinn
tedquinn@cox.net
USA
Consultant
Associates
Mitsubishi Electric
Corp./ Advanced Oi.Tadashi@
Mitsubishi
Tadashi Oi
Japan
VendorTechnology R&D
bx.MitsubishiElectric.co.jp
CenterMitsubishi Nuclear
Energy Systems,
Kenji Mashio andkenji_mashio@mnes-us.com
Mitsubishi
Inc. and RothJapan
VendorEmilie Rothemroth@mindspring.com
Cognitive
Engineering
Nuclear
NII
Jane Bowie
Installations
jane.bowie@hse.gsi.gov.uk
UK
Regulator
Inspectorate
Nucelar Research
NRI Rez
Jaroslav Holy
hoj@ujv.cz
Czech ResearchInstitute Rez, Rez
OkayamaOkayama U Akio Gofuku
fukuchan@sys.okayama-u.ac.jp Japan
Research
University
Oskarshamn Thomas Gunnarsson Oskarshamn NPP thomas.gunnarsson@okg.eon.se Sweden Utility
The Ohio State
OSU
Carol Smidts
smidts.1@osu.edu
USAResearch
University
Vattenfall Ringhals johanna.oxstrand@
Ringhals
Johanna Oxstrand
Finland UtilityABvattenfall.com
SATechMica Endsley
SA Technologies mica@satechnologies.com
USA
Consultant
Sandia Nationalrlborin@sandia.gov /SNL
Ronald Laurids BoringUSA
Research
Laboratories
ron@boringfamily.info
Yvonne Liljeholm
Swedish Radiation yvonne.liljeholm.johansson@ss
SRSA
Johansson
Sweden Regulator
Safety Authority
m.se ; perolof.sanden@ssm.se
Per-Olof Sandén
Radiation and
STUKMilka Holopainen
Nuclear Safety
Milka.Holopainen@stuk.fi
Finland Regulator
Authority
Organization

Survey POC and
KeyContact100NEA/SEN/SIN/WGHOF(2012)1

Organization
SUJB

Survey POC and
KeyContact
Karel Matejka

Full Name ofOrganization
State Office for
Nuclear Safety

Emailkarel.matejka@sujb.cz

CountryType of
Organization

Czech

Regulator

Luis Fernandez/Pedro
lillobre@tecnatom.es/ptrueba@t
Tecnatom, S. A.
Spain
VendorTruebaecnatom.es
TokyoTokyo EletricFuruhama Yutaka
mishima.takaki@tepco.co.jp
Japan
Utility
Electric
Company
Human Factors
TokyoGroup, Tokyo
Takaki Mishima
furuhama.yutaka@tepco.co.jp Japan
Utility
Electric
Electric Power
Company
Lauren ReinermanUniversity of
UCF
lreinerm@ist.ucf.edu
USAResearch
Jones
Central Florida
UJD SR-NRA ofUJD SR-NRA Juraj Rovny
juraj.rovny@ujd.gov.sk
Slovakia Regulator
the Slovak Republic
U.S. Nuclear
Stephen Fleger &USNRCRegulatory
Stephen.Fleger@nrc.gov
USA
Regulator
David DesaulniersCommission
VTT Technical
VTT
Leena Norros
Research Centre of Leena.Norros@vtt.fi
Finland Research
FinlandVTTJari Laarni
VTTjari.laarni@vtt.fi
Finland Research
VUJE
Adam Gieci
VUJE, Inc.
gieci@vuje.sk
Slovakia Vendor
Westinghouse
WEC
Ruiqi Ma
mar@westinghouse.com
USA
VendorElectric Company
Westinghouse Julie Reed
Westinghouse
reedji@westinghouse.com
USA
VendorJohn Wreathall &WreathWood John Wreathall
Co/Thejohn@wreathall.com
USA
Consultant
WreathWood group
Tecnatom101

NEA/SEN/SIN/WGHOF(2012)1

102

NEA/SEN/SIN/WGHOF(2012)1

Attachment 4
Workshop Agenda

AGENCE DE L’OCDE POUR L’ÉNERGIE NUCLÉAIRE
OECD NUCLEAR ENERGY AGENCYOECD Nuclear Energy Agency (NEA)
Committee on the Safety of Nuclear Installations (CSNI)
Working Group on Human and Organisational Factors (WGHOF)

OECD/NEA Workshop
Human Performance and the Operation of New Nuclear Plant Technology

Hosted by the US Nuclear Regulatory Commission
Rockville Hilton Hotel and Executive Meeting Center
1750 Rockville Pike
Rockville, Maryland, USA, 20852-1699
March 1 – 3, 2010
Workshop Programme
On-line information is available
www.nea.fr/html/nsd/calendar.html

103

NEA/SEN/SIN/WGHOF(2012)1
Monday 1 March 2010
07.30

Registration of the participants

OPENING SESSION – Plaza III
Chair:
Valerie BARNES – Workshop Chair (US NRC, USA)
Co-Chair: Daniel TASSET – CSNI/WGHOF Chair (ASN, France)

8:00

Opening and Welcome
James LYONS, Deputy Director, Office of Nuclear Regulatory Research(NRC, USA)Workshop Organisation Remarks
Radomir REHACEK, (NEA, France)
Julius J. PERSENSKY (INL, USA)

09.00

WGHOF Survey on HOF R&D Activities
Jeffrey JOE(INL, USA)

10.30

A View from the Chemical Industry
David STROBHAR (COP, USA)

11:15

The European Commission – MMOTION Study
François DIONIS (EDF, France)

Session A Operating Experience and Concept of Operation
Chair:

Magnhild KAARSTAD – (HRP, Norway) – Plaza III

13.05

Operating Experience from New and Modernized Plants(Challenge paper Topic 1) – Plaza III
Thomas GUNNARSSON (OKG, Sweden)

13.25

Evolving Concepts for the Operations of Nuclear Power Plants(Challenge paper Topic 2) – Plaza III
Gyrd SKRAANING (HRP, Norway)

13.45

Break Out Discussion of Topic 1 (Jackson and Monroe Rooms) and2 (Wilson and Truman Rooms) IN THE GROUPS

15.30

Report Out Topic 1 – Plaza III

16.15

Report Out Topic 2 – Plaza III

17.00

Session a Summary Discussion – Plaza III

17.30

End of the First Day

104

NEA/SEN/SIN/WGHOF(2012)1
Tuesday 2 March 2010
Session B
Role of Automation and Management Unplanned Events
Chair:

Bernard PAPIN – (CEA, France) – Plaza III

08.05

The Role of Automation and Personnel: New Concepts of Teamwork in Advanced Systems(Challenge paper Topic 3) – Plaza III
John O‟HARA (BNL, USA)

08.25

Management of Unplanned, Unanticipated Events(Challenge paper Topic 4) – Plaza III
Mica ENDSLEY (SA Technologies, USA)

09.00

Break Out Discussion of Topic 3 (Jackson and Monroe Rooms) and4 (Wilson and Truman Rooms) in the Groups

10.45

Report Out Topic 3 – Plaza III

11.30

Report Out Topic 4 – Plaza III

Session C
HSI to Support Cognition and Complexity Issue
Chair:

Julius J. PERSENSKY – (INL, USA) – Plaza III

13.35

Human-System Interface (HSI) Design Principles for Supporting Operator Cognitive
Functions(Challenge paper Topic 5) – Plaza III
Emilie ROTH (Roth Cognitive Engineering, USA)

13.55

Complexity Issues in Advanced Systems(Challenge paper Topic 6) – Plaza III
Bernard PAPIN (CEA, France)

14.15

Break Out Discussion of Topic 5 (Jackson and Monroe Rooms) and6 (Wilson and Truman Rooms) in the Groups

16.15

Report Out Topic 5 – Plaza III

17.00

Report Out Topic 6 – Plaza III

17.45

Session B and C Summary Discussion – Plaza III

18.30

End of the Second Day

105

NEA/SEN/SIN/WGHOF(2012)1
Wednesday 3 March 2010
Session D
Organisational Factors and HFE Tools
Chair:

Jeffrey JOE – (INL, USA) – Plaza III

08.05

Organizational Factors — Safety Culture/Safety Management(Challenge paper Topic 7) – Plaza III
Sonja B. HABER (Human Performance Analysis, Corp., USA

08.25

Human Factors Engineering (HFE) Methods and Tools(Challenge paper Topic 8) – Plaza III
Chris PLOTT (Alion Science and Technology, USA)

09.00

Break Out Discussion of Topic 3 (Jackson and Monroe Rooms) and4 (Wilson and Truman Rooms) in the Groups

10.45

Report Out Topic 7 – Plaza III

11.30

Report Out Topic 8 – Plaza III

Summary Session
Plaza III
Chair:
Valerie BARNES – Workshop Chair (US NRC, USA)
Co-Chair: Daniel TASSET – CSNI/WGHOF Chair (ASN, France)

Radomir REHACEK (NEA, France)
13.45

Discussion, Conclusions and Recommendations

17:00

Closure of the Workshop

106