REACTOR OVERSIGHT PROCESS

INITIAL IMPLEMENTATION EVALUATION PANEL

FINAL REPORT

MAY 10, 2001

ADAMS ML011290025
TABLE OF CONTENTS

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Introduction ................................................................1

Approach and Objectives .....................................................1

Panel Conclusions and Recommendations ........................................2

Overall Reactor Oversight Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Significance Determination Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Assessment and Enforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

ROP Self-Assessment Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

IIEP Charter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 1

IIEP Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 2

Bibliography of Panel Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 3

Sources of Panel Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 4
i

EXECUTIVE SUMMARY

On May 17, 2000, the Commission directed the NRC staff to convene a panel under the

Federal Advisory Committee Act to evaluate the first yearís implementation of the Reactor

Oversight Process (ROP). The Initial Implementation Evaluation Panel (IIEP) was established

on October 17, 2000. The panel met six times between November 2000 and April 2001.

The IIEP has concluded that the ROP is a notable improvement over the previous licensee

performance assessment program and should be continued. The reactor oversight process

has made progress toward achieving the Agencyís four performance goals: 1) maintain safety,

2) increase public confidence, 3) increase regulatory effectiveness and efficiency, and 4)

reduce unnecessary regulatory burden. In addition, the process provides a more objective,

risk-informed, predictable, and understandable approach to the oversight of commercial nuclear

reactor facilities. 

The NRC and the nuclear industry expended substantial time and effort communicating with

their staff members and public stakeholders about the process changes and paradigm shifts

embodied in the ROP. Although the change-management tools used to communicate and

carry out the changes were generally successful, the panel identified continuing tension as a

result of three changes in regulatory philosophy: maintaining safety rather than improving

safety, applying risk-informed regulation rather than deterministic regulation, and using

indicative measures of performance rather than predictive measures of performance. To a

large extent, stakeholder concern with these regulatory changes are common denominators for

the various issues identified by the panel. The tension created by these underlying changes

has affected the degree to which the performance goals of the ROP can ever be fully achieved

according to various stakeholdersí perspectives about what constitutes a successful oversight

process. On the other hand, it is this tension that helps to ensure that the NRC, the industry,

and public stakeholders will continue to engage in creative dialogue focused on the safe

operations of nuclear power reactors. The panel considers the interaction among stakeholders

throughout the development process, and going forward, to be an important feature of the

ROP. Continued management attention is needed to sustain genuine and substantive

interaction among stakeholders, mindful of the challenges and creative potential inherent in the

tensions of the complex nuclear regulatory environment.

An ancillary common concern raised by the panel members in evaluating the specific issues

discussed in the report is the continuing need for sufficient resources to maintain the formal

processes and infrastructure for the ROP. NRC resources are needed to evaluate, pilot,

communicate, and implement future enhancements and ensure regional consistency. Although

the staff has learned many lessons from the first yearís implementation and has already made

numerous changes, there are many issues to be resolved and still other issues to be

discovered as the ROP continues to evolve. The many issues regarding the significance

determination process reflect the degree of change in the use of risk insights in the ROP and

the substantial work that remains to complete development of the staffís suite of tools. 

The panel considered the following recommendations as high priority:

! Establish a formal program and assign sufficient resources to enhance communications

necessary for improving the ROP. The program should accumulate lessons learned,
ii

provide multiple and diverse opportunities for comment to all internal and external

stakeholders, respond to stakeholdersí comments, and have a process for making

timely process changes.

! Revise the ROP communication plan to include outreach activities designed to inform

the public about the process and its relationship to the Agencyís mission of protecting

the public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the

information on the ROP Web page to improve and simplify public access to the

information. Identify methods, using stakeholder input, to improve public outreach

efforts.

! Establish a structured ongoing program to evaluate long-term ROP effectiveness and to

test ROP assumptions. As a minimum, this includes integrating the insights of the ROP

self-assessment program and the overall assessment of industry performance. The

staff should also consider periodically engaging internal and external stakeholders to

independently assess the ROP.

! Continue the efforts of the crosscutting issues task force and clarify the ROP guidance

on the identification and disposition of crosscutting issues.

! Evaluate lessons learned from initial implementation to achieve parity in the treatment of

risk-significant inspection findings and crossed performance indicator thresholds. The

evaluation should verify that the outcomes from the performance indicators and

inspection findings accurately reflect the significance of the issues. The staff should

consider addressing this issue by adjusting the Green/White thresholds or modifying the

action matrix.

! Ensure that the staffís ROP self-assessment program identifies and evaluates any

unintended consequences or unnecessary regulatory burden caused by the

performance indicators and that changes are made where appropriate.

! Expedite the efforts to resolve the concerns about the safety system unavailability

performance indicators and implement any needed revisions to NEI 99-02, ìRegulatory

Assessment Performance Indicator Guideline,î to achieve consistency with other

applications.

! Evaluate the inspection approach for the physical protection cornerstone and revise the

inspection program as necessary after the pending safeguards performance

assessment pilot program and the physical security rulemaking are completed.

! Validate and issue the revised reactor safety significance determination process Phase

2 worksheets.

! Continue efforts to obtain improved and standardized risk analysis tools for the reactor

analysts.
iii

! Review lessons learned from use of the fire protection significance determination

process, and improve the risk characterization tool to make it more meaningful,

effective, and efficient.

! Continue development of an improved physical protection significance determination

process.

! Evaluate the need for other significance determination tools. The staff should carefully

evaluate any potential changes against the Agencyís goals. For example, a new

significance determination process that increases regulatory burden should have a

corresponding benefit in maintaining safety or increasing the Agencyís effectiveness.

! Evaluate lessons learned from initial implementation and revise the as-low-asreasonably-achievable (ALARA) significance determination process as necessary.

! Evaluate a graded approach for resetting non-Green inspection findings as entry

conditions into the action matrix.

! Evaluate and clarify the guidance on the designation, definition, and use of what are

presently called no-color findings, and find a more appropriate term for these findings.

As part of its evaluation, the panel reviewed the staffís self-assessment program and the

performance measures for the ROP. The panel concluded that the self-assessment program

has the necessary elements to evaluate the ROP against the Agencyís performance goals;

however, the panel could not evaluate the effectiveness of the program given that the first

yearís assessment data are not yet available. Nevertheless, the panel found that, for the most

part, the staff had identified the same issues the panel identified.

The IIEP had the benefit of information obtained during the first full year of nationwide

implementation of the ROP. During the year, the oversight process elements were extensively

exercised and many experiences that exemplified concerns and issues were available for the

panelís consideration. The panel observed that the staff actively solicited stakeholder input

throughout initial implementation, as it did during development and piloting of the ROP. The

panel believes that the level of stakeholder involvement has been unprecedented for an NRC

process change and is reflected in the quality of the process. Public workshops, public

meetings near all facilities, surveys, and formal internal feedback processes were critical to the

staffís efforts to further refine the ROP.
1

FINAL REPORT OF THE REACTOR OVERSIGHT PROCESS

INITIAL IMPLEMENTATION EVALUATION PANEL

Introduction

The NRC implemented, nationwide, a revised Reactor Oversight Process (ROP) for commercial

nuclear power plant licensees on April 2, 2000. Background information on the development of

the ROP and the results of the pilot program is given in Commission papers SECY-99-007,

ìRecommendations for Reactor Oversight Process Improvements,î dated January 8, 1999,

SECY-99-007A, ìRecommendations for Reactor Oversight Process Improvements (Follow-up

to SECY-99-007),î dated March 22, 1999, and SECY-00-049, ìResults of the Revised Reactor

Oversight Process Pilot Program,î dated February 24, 2000. These Commission papers

described the scope and content of performance indicator reporting, a new risk-informed

baseline inspection program, a new assessment process, and revisions to the enforcement

policy. Commission paper SECY-00-049 also described the results from the Pilot Program

Evaluation Panel (PPEP), including a recommendation to proceed with initial implementation of

the ROP at all power reactor facilities. On March 28, 2000, the Commission approved initial

implementation of the ROP. 

On May 17, 2000, in a staff requirements memorandum in response to SECY-00-049, the

Commission directed the NRC staff to convene another evaluation panel under the Federal

Advisory Committee Act to evaluate the first yearís implementation of the ROP. The Initial

Implementation Evaluation Panel (IIEP) was established on October 17, 2000. The purpose of

the cross-disciplinary oversight panel was to independently monitor and evaluate the results of

the first yearís implementation of the ROP and provide advice and recommendations to the

Director of the Office of Nuclear Reactor Regulation on reforming and revising the ROP (see

Attachment 1). The panel, like the PPEP, was made up of a cross-section of stakeholders. An

NRC senior resident inspector and a senior reactor analyst were added to the panel at the

Commissionís request. The NRC selected the panel members to represent the views of diverse

groups that had an expressed interest in the changes to the ROP (see Attachment 2).

Approach and Objectives

The IIEP conducted six meetings during the first yearís implementation of the ROP. All

meetings were open to the public and all meeting material was placed in the NRCís public

document room. Additionally, the meeting notices, summaries, and transcripts were placed on

the NRCís ROP Web page. Attachment 3 is a bibliography of the significant documents on the

panelís activities. 

The IIEP worked as a management-level cross-disciplinary oversight group of experts to

evaluate whether the new regulatory oversight process can be effectively carried out and

whether it has achieved its overall objectives. The panel solicited and obtained additional

views, to supplement the membersí personal insights, from representatives of four States (New

Jersey, Pennsylvania, Illinois, and Vermont), the Union of Concerned Scientists, the Nuclear

Energy Institute (NEI), NRC resident inspectors, NRC senior reactor analysts, the NRC Office

of Public Affairs, and McGraw-Hill (see Attachment 4). The NRC staff members directly

involved in the process development reported on the status of the initial implementation and

responded to questions and comments.
2

During the IIEP meetings, the panel discussed and generally agreed to the following objectives: 

(1) Determine whether the ROP is achieving the following goals: 

ï Maintaining safety 

ï Increasing public confidence 

ï Increasing effectiveness and efficiency 

ï Reducing unnecessary regulatory burden 

ï Objective

ï Risk-informed

ï Predictable

ï Understandable

(2) Determine whether the more significant problem areas of the ROP have been identified. 

(3) Determine whether the NRC has developed a sound self-assessment program for the ROP 

 and, if so, whether it includes mechanisms for self-correction. 

To accomplish these objectives, each panel member was requested to provide a list of issues

regarding implementation of the ROP. The panel memberís issues were compiled, categorized,

and integrated with other issues presented to the panel during its meetings. The issues were

sorted according to the ROP elements they affected: performance indicators (P), inspection (I),

significance determination process (S), and assessment and enforcement (A). Those issues

that were identified as being pertinent to more than one element of the ROP were placed into

an overall (O) category.

With a facilitatorís assistance, the panel collectively evaluated the list of issues and developed

group consensus on the description of each issue, its priority, the primary ROP performance

goals affected, and the panelís recommendation for addressing the issue. The panel defined

consensus in its bylaws as no one panel member dissenting from the position taken. In

practice, this meant all panel members ìcould live withî the group position taken on the issue.

 

Each issue was assigned one of two priorities. The panel defined Priority 1 issues as those

issues that should receive high priority. Priority 2 issues were defined as issues for the staffís

consideration. Although the panel prioritized the various issues, it did not identify a timetable

within which the staff should resolve the issues. The panel recognized the complexity of some

issues and the need for additional time and information to evaluate solutions and the impacts of

any proposed changes. Furthermore, the panel did not lower the priority of an issue because

the staff was already resolving the issue. In developing its recommendations, the panel did not

specify how to resolve the issues identified in this report. The staff resolved some of the

issues the panel identified before the panel completed its activities. The panel did not include

these issues in the report.

Panel Conclusions and Recommendations

The IIEP concluded that the ROP is a notable improvement over the previous licensee

performance assessment program and should be continued. The reactor oversight process

has made progress toward achieving the Agencyís four performance goals: 1) maintain safety,

2) increase public confidence, 3) increase regulatory effectiveness and efficiency, and 4)
3

reduce unnecessary regulatory burden. In addition, the process provides a more objective,

risk-informed, predictable, and understandable approach to the oversight of commercial nuclear

reactor facilities.

The NRC and the nuclear industry expended substantial time and effort communicating with

their staff members and public stakeholders about the process changes and paradigm shifts

embodied in the ROP. Although the change-management tools used to communicate and

carry out the changes were generally successful, the panel identified continuing tension as a

result of three changes in regulatory philosophy: maintaining safety rather than improving

safety, applying risk-informed regulation rather than deterministic regulation, and using

indicative measures of performance rather than predictive measures of performance. To a

large extent, stakeholder concern with these regulatory changes are common denominators for

the various issues identified by the panel. The tension created by these underlying changes

has affected the degree to which the performance goals of the ROP can ever be fully achieved

according to various stakeholdersí perspectives about what constitutes a successful oversight

process. On the other hand, it is this tension that helps to ensure that the NRC, the industry,

and public stakeholders will continue to engage in creative dialogue focused on the safe

operations of nuclear power reactors. The panel considers the interaction among stakeholders

throughout the development process, and going forward, to be an important feature of the

ROP. Continued management attention is needed to sustain genuine and substantive

interaction among stakeholders, mindful of the challenges and creative potential inherent in the

tensions of the complex nuclear regulatory environment.

! Maintaining safety rather than improving safety: One premise of the NRCís strategic

plan is that the nuclear power industryís performance has improved substantially over

the past 10 years and nuclear reactors, collectively, are operating above acceptable

safety levels consistent with the Agencyís Safety Goal Policy. The staff designed the

ROP to maintain the current level of safety. The process is designed to improve safety

performance before it falls below acceptable levels, not to continually improve the safety

margins that currently exist. Some public stakeholders do not believe that current

nuclear industry performance is sufficient to assure public health and safety without

continual improvement. This could limit the publicís confidence in the process.

! Risk-informed regulation rather than deterministic regulation: In many ways the ROP is

ahead of the other regulatory processes in using risk insights. It is difficult to implement

a risk-informed oversight process while in a deterministic regulatory framework. For

example, the integration of the significance determination process with the NRCís

assessment program puts both the inspector and licensee in a conflicting situation. The

ROP focuses on risk-significant issues, but licensees must still comply with regulatory

requirements that are not risk-informed. This conflict has contributed to many of the

issues discussed in this report. An additional concern of public stakeholders is the

perceived overreliance on existing risk analysis tools for regulatory decision making. 

Over the long term, the staff must continue risk-informing the regulations to close the

gap between the regulatory framework and the oversight process.

! Indicative measures of performance rather than predictive measures of performance:

The ROP, using performance indicator thresholds and the significance determination

process, is an indicative process, whereas the previous performance assessment
4

process attempted to be predictive by using performance issues of low safety

significance to identify declining performance trends. A premise of the ROP is that the

licenseeís corrective action program best handles low-level performance trends and that

a regulatory response is not required until a threshold is crossed. A related assumption

is that a licensee will not normally pass directly from the licensee response column to

the unacceptable performance column of the action matrix, giving the NRC time to

respond before plant performance becomes unacceptable. Many of the concerns about

crosscutting issues and inspection report thresholds come from skepticism about this

assumption.

Though the panel focused on areas needing improvement, it noted many positive attributes and

outcomes. We note some of these in the introductory comments on each ROP element.

The panelís recommendations are as follows:

! Establish a formal program and assign sufficient resources to enhance communications

necessary for improving the ROP. The program should accumulate lessons learned,

provide multiple and diverse opportunities for comment to all internal and external

stakeholders, respond to stakeholdersí comments, and have a process for making

timely process changes. (O-1)

! Revise the ROP communication plan to include outreach activities designed to inform

the public about the process and its relationship to the Agencyís mission of protecting

the public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the

information on the ROP Web page to improve and simplify public access to the

information. Identify methods, using stakeholder input, to improve public outreach

efforts. (O-2)

! Establish a structured ongoing program to evaluate long-term ROP effectiveness and to

test ROP assumptions. As a minimum, this includes integrating the insights of the ROP

self-assessment program and the overall assessment of industry performance. The

staff should also consider periodically engaging internal and external stakeholders to

independently assess the ROP. (O-3)

! Continue the efforts of the crosscutting issues task force and clarify the ROP guidance

on the identification and disposition of crosscutting issues. (O-4)

! Evaluate lessons learned from initial implementation to achieve parity in the treatment of

risk-significant inspection findings and crossed performance indicator thresholds. The

evaluation should verify that the outcomes from the performance indicators and

inspection findings accurately reflect the significance of the issues. The staff should

consider addressing this issue by adjusting the Green/White thresholds or modifying the

action matrix. (O-5)

! Ensure that the staffís ROP self-assessment program identifies and evaluates any

unintended consequences or unnecessary regulatory burden caused by the

performance indicators and that changes are made where appropriate. (P-1)
5

! Expedite the efforts to resolve the concerns about the safety system unavailability

performance indicators and implement any needed revisions to NEI 99-02, ìRegulatory

Assessment Performance Indicator Guideline,î to achieve consistency with other

applications. (P-2)

! Continue the efforts to identify and evaluate improvements to performance indicators. 

The staff should thoroughly evaluate any significant changes to the performance

indicators, following the structured process in Inspection Manual Chapter 0608,

ìPerformance Indicator Program.î Further, the staff should evaluate the effect of any

significant changes in the scope of information provided by performance indicators on

the action matrix and the baseline inspection program, and should identify any additional

costs and benefits. (P-3)

! Continue efforts to incorporate the answers to frequently asked questions into the

performance indicator guidance document, NEI 99-02, and to make the answers more

generic where possible. (P-4)

! Evaluate the inspection approach for the physical protection cornerstone and revise the

inspection program as necessary after the pending safeguards performance

assessment pilot program and the physical security rulemaking are completed. (I-1)

! Evaluate and revise guidance to inspectors as necessary to clarify and promote

consistency in documenting inspections. Continue conducting periodic audits of

inspection reports to identify and correct inconsistencies. The new documentation

thresholds for issues that have a defined level of safety or regulatory significance are

appropriate. Inspection observations and insights that do not reach the threshold should

continue to be communicated verbally to licensees for their consideration. (I-2)

! Evaluate inspection findings and performance indicator results for the first yearís

implementation and determine the appropriate level of effort to adequately assess risksignificant areas in the baseline inspections. Modify the process as appropriate. (I-3)

! After further experience with the ROP, review the results and consider whether to waive

certain parts of the baseline team inspections and let licensees assess themselves

under defined circumstances. (I-4)

! Validate and issue the revised reactor safety significance determination process Phase

2 worksheets. (S-1)

! Continue efforts to obtain improved and standardized risk analysis tools for the reactor

analysts. (S-2)

! Review lessons learned from use of the fire protection significance determination

process, and improve the risk characterization tool to make it more meaningful,

effective, and efficient. (S-3)
6

! Continue development of an improved physical protection significance determination

process. (S-4)

! Evaluate the need for other significance determination tools. The staff should carefully

evaluate any potential changes against the Agencyís goals. For example, a new

significance determination process that increases regulatory burden should have a

corresponding benefit in maintaining safety or increasing the Agencyís effectiveness. 

(S-5)

! Evaluate lessons learned from initial implementation and revise the ALARA significance

determination process as necessary. (S-6)

! Use lessons learned from initial implementation to make the risk characterization

process expedient, scrutable, and understandable to all stakeholders. (S-7)

! Use lessons learned from initial implementation to clarify the definition of a performance

deficiency. (S-8)

! Evaluate a graded approach for resetting non-Green inspection findings as entry

conditions into the action matrix. (A-1)

! Evaluate and clarify the guidance on the designation, definition, and use of what are

presently called no color findings, and find a more appropriate term for these findings.

(A-2)

! Clarify the guidance on the objectives and structure of regulatory conferences and

communicate this guidance to the external and internal stakeholders. (A-3)

! Develop clear guidance on how to handle multiple related inspection findings, and

communicate the guidance to all stakeholders. (A-4)

The following five sections provide the consensus of the panel with respect to the description of

the issues, the priority, the primary performance goals affected, and the panelís

recommendations. Where appropriate, the panel has referred to the recommendations of the

Pilot Program Evaluation Panel when the two panels identified similar issues or made similar

recommendations.

Panel members expressed two minority views:

(1) The initial implementation does not appear to demonstrate that the ROP has improved

identification of design basis issues or validated recent initiatives to scope design basis

issues. NUREG-1275, Volume 14, ìCauses and Significance of Design-Basis Issues at

U.S. Nuclear Power Plants,î draws a clear connection between the number of

engineering (and design) inspection hours expended and the number of design basis

issues discovered. However, the number of engineering inspection hours in the first

yearís implementation has not yet been compiled, indicating to some observers that the

staff is not focused on design basis issues under the new process. Public confidence

will not be enhanced unless there is assurance that plants are properly designed, built
7

as designed, modified only with proper analysis, and properly maintained including

sufficient attention to "aging" phenomena. Uncertainties regarding design basis issues

serve to undermine confidence in Probabilistic Risk Assessments and the concept of

"maintaining" safety. In an October, 17, 1996 All Employees NRC meeting, Chairman

Shirley Jackson opined that one reason for the events at Millstone was that ". . . we

stopped doing design basis inspections too early, and relied on industry . . . without

maintaining an appropriate regulatory focus to assess whether in fact they were dealing

with the issue in a timely manner." 

(2) The IIEP make-up was weighted with regulators and industry to the extent that common

interest in moving the ROP forward, and other commonalties, may have limited the

panel's perspective. Given the common working culture, it is not surprising that the IIEP

critique was quite similar to the staff review group critique. Future panels might benefit

from inclusion of some additional individuals from outside of the NRC-licensee set, for

example, attorneys with nuclear specialties or academics. Panel builders might also

add to the quality of deliberation by a more even gender mix. While it enriches dialogue

and broadens perspective to add members from public interest groups, adding only one

or two with views likely to be quite alien to the remainder of the group on some issues is

problematic in terms of free and meaningful consensus building.
8

Overall Reactor Oversight Process

The ROP is a notable improvement over the previous licensee performance assessment

program and should be continued. The reactor oversight process has made progress toward

achieving the Agencyís four performance goals: 1) maintain safety, 2) increase public

confidence, 3) increase regulatory effectiveness and efficiency, and 4) reduce unnecessary

regulatory burden. In addition, the process provides a more objective, risk-informed,

predictable, and understandable approach to the oversight of commercial nuclear reactor

facilities. However, the panel identified the following issues that the staff should address:

Issue O-1: Process improvements and stakeholder feedback 

Priority 1

Primary performance goals affected: Public confidence/effectiveness and efficiency

Issue description: As with any regulatory process, it is important that a formal infrastructure

be established to allow for stakeholder comments and questions. The infrastructure should

promptly review feedback and implement process improvements. The frequently asked

questions process used in the performance indicator program is a good mechanism for raising

and resolving licensee and inspector issues. This process promotes the open exchange of

information and establishment of uniform and consistent guidance. Other elements of the

ROP, such as the significance determination process, would benefit from a similar approach. 

This process should also include a mechanism for the public to retrieve information on past

questions and answers and ensure that lessons learned and feedback information are

communicated to the other regions. 

The PPEP recommended continued feedback from inspectors and the ongoing modification of

procedures during the industrywide implementation to assure that the procedures are clear and

appropriately address the cornerstones.

Panel recommendation: Establish a formal program and assign sufficient resources to

enhance communications necessary for improving the ROP. The program should accumulate

lessons learned, provide multiple and diverse opportunities for comment to all internal and

external stakeholders, respond to stakeholdersí comments, and have a process for making

timely process changes.

Issue O-2: Public access to ROP information

Priority 1

Primary performance goals affected: Public confidence/understandable

Issue description: It is important that the public have confidence that the ROP provides the

regulator a means for accurately assessing the safety of plants and taking action where

necessary and that the process and actions be effectively communicated to the public. 

Likewise, it is essential that the public have clear, unfettered access to accurate and meaningful

information to be able to reach its own conclusions.

The staff made significant improvements to the publicís access to plant performance

information during the initial implementation period. The staff established a Web site that

displayed information about the ROP. The publicís response to the Web site was generally
9

very positive. The panel observed that the staff actively solicited stakeholder input throughout

initial implementation, as it did during development and piloting of the ROP. Public workshops,

public meetings near all facilities, and surveys provided opportunities for valuable input to the

staff as they sought to further refine the ROP.

However, much remains to be done to make the ROP understandable and accessible to public

stakeholders. At first, the public and media thought the new process relied solely on

performance indicators because the indicators were highlighted on the Web page. The

highlighting caused a few public stakeholders to believe that the NRC had abandoned the

resident inspector program. Many stakeholders did not understand that insights from both

performance indicators and inspection findings were used to assess overall licensee

performance. 

The staff has improved the structure and format of the Web pages to correct this perception. 

However, the Web pages need additional improvements. The ROP Web pages should start

with overview information and provide layered access to more detailed information (i.e., links to

the site-related documents). Site-specific pages could include a bulletin board with the status of

enforcement items and inspections. This would enable the public to understand, without

conducting exhaustive research, the status of important issues at the plant in their locale. A

high-level summary would also provide a means to differentiate the performance of one plant

from another. The timely posting of information is important to enhancing public confidence.

Panel recommendation: Revise the ROP communication plan to include outreach activities

designed to inform the public about the process and its relationship to the Agencyís mission of

protecting the public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the information on the

ROP Web page to improve and simplify public access to the information. Identify methods,

using stakeholder input, to improve public outreach efforts. 

Issue O-3: Long-term process effectiveness

Priority 1

Primary performance goals affected: Maintain safety/effectiveness and efficiency/public

confidence

Issue description: The panel recognizes and agrees with the concern of members and

stakeholders that there are limits to what may be learned from a 1-year test of the ROP. The

staff should evaluate the long-term effectiveness of the process to determine whether the

performance indicators and inspection findings identify poorly performing plants. 

The significance determination process tools are a key element in ensuring the effectiveness of

the ROP. However, the significance determination process Phase 2 worksheets were not

available for much of initial implementation (see S-1). 

The ROP is based on certain assumptions (e.g., that licensee corrective action programs are

mature and support the basis for the licensee response band concept, that degraded

performance will reveal itself by ever increasing significant issues and crossed performance

indicator thresholds, and that all violations of NRC regulations do not require followup by the

NRC). As sufficient information and experience is obtained, assumptions either will be
10

confirmed or refuted. Whatever the result, there must be a validation process. In addition,

there may be unintended consequences of the ROP elements such as with some performance

indicators (see P-1). The final oversight process must focus on identifying issues of safety

significance, eliminating any underestimation of risk characterization determinations (false

negatives), and minimizing overestimation of issues (false positives).

The PPEP recommended that the staff continue to monitor industrywide implementation to

ensure that when a risk-significant event occurs, the event-specific response requires

reevaluation of the performance indicators and inspection results to address whether they

missed a crosscutting or common-mode failure issue. The PPEP also concluded those process

assumptions had not been tested sufficiently.

Panel recommendation: Establish a structured ongoing program to evaluate long-term ROP

effectiveness and to test ROP assumptions. As a minimum, this includes integrating the

insights of the ROP self-assessment program and the overall assessment of industry

performance. The staff should also consider periodically engaging internal and external

stakeholders to independently assess the ROP.

Issue O-4: Crosscutting issues

Priority 1

Primary performance goals affected: Maintain safety/public confidence

Issue description: During the development of the ROP, and initial implementation, some

inspectors were concerned about the identification and disposition of crosscutting issues. The

concern was that licensee performance in the crosscutting areas of human performance,

safety-conscious work environment, and problem identification and resolution could become

degraded without being detected by the baseline inspection program and performance

indicators. The ROP addresses crosscutting issues by highlighting them in inspection reports

when they are notable contributors to inspection findings or if an appreciable trend or pattern

has emerged. The staff further amplifies these concerns in assessment letters to the licensee

when the concerns constitute a substantive issue. 

The current process does not have sufficient criteria, thresholds, and definitions of crosscutting

issues to ensure consistency in handling these issues. In addition, there is no predefined NRC

action if the inspection program identifies a substantive crosscutting issue such as a deficient

corrective action program. The ROP does not provide for additional NRC engagement on

crosscutting issues unless they are contributing causes to performance indicators or inspection

findings that have been characterized as White or greater. Some inspectors are also

concerned about the lack of a process to handle low-level human performance trends when it

appears that NRC actions could prevent the occurrence of a significant performance issue. 

The industry believes the ROP should focus on performance outcomes, of which crosscutting

issues are but one possible cause. 

One premise of the ROP is that either performance indicators or inspection findings will detect

degradation in the crosscutting areas in time to allow for Agency action to protect the public

health and safety. Early data obtained from initial implementation suggests that there is a

correlation between crosscutting issues and crossed thresholds consistent with the premise of
11

the process (i.e., the number of crosscutting findings per plant appears to increase as you

move to the right in the action matrix).

The PPEP final report also highlighted the divergent views on the identification and disposition

of crosscutting issues.

Panel recommendation: Continue the efforts of the crosscutting issues task force and clarify

the ROP guidance on the identification and disposition of crosscutting issues.

Issue O-5: Basis of Green/White thresholds

Priority 1

Primary performance goals affected: Public confidence/understandable

Issue description: The bases for the performance indicator Green/White thresholds are not

risk-informed. The thresholds were selected to identify the 95-percent performance level (i.e.,

industry outliers). Since NRC action is the same for both White performance indicators and

White inspection findings, which are risk-informed, several problems have resulted. First, the

NRC and the licensee have a different perception of the impact and importance of White

issues. Second, it is difficult to communicate to public stakeholders that a White performance

indicator may not be risk-significant when the NRC increases its regulatory response according

to the action matrix. This could impact public confidence in the NRC.

Panel recommendation: Evaluate lessons learned from initial implementation to achieve parity

in the treatment of risk-significant inspection findings and crossed performance indicator

thresholds. The evaluation should verify that the outcomes from the performance indicators

and inspection findings accurately reflect the significance of the issues. The staff should

consider addressing this issue by adjusting the Green/White thresholds or modifying the action

matrix.
12

Performance Indicators

The integration of performance indicators into the ROP has provided objective measures for

assessing licensee performance. Additionally, licensees can accurately report performance

indicators without an excessive burden, and the public can easily understand the performance

data. The initial implementation period has verified that the performance indicators can focus

both licensee and NRC attention on issues that are either risk significant or relevant to

promoting desired performance. However, the panel identified the following issues that the

staff should address:

Issue P-1: Unintended negative consequences of performance indicators

Priority 1

Primary performance goals affected: Maintain safety/unnecessary regulatory burden

Issue description: The use of performance indicators may have unintended negative

consequences when they measure both desirable actions and performance issues. This could

lead to nonconservative decisions by licensees. In addition, NRC may impose unnecessary

regulatory burden when it takes actions based, in part, on licensee actions that are desirable

and appropriate.

The staff has noted that some licensees have altered normal operating and maintenance

practices solely to avoid conditions that may contribute to crossing a performance indicator

threshold. For example, the Unplanned Power Change performance indicator may cause a

licensee to delay needed equipment repairs for 72 hours to avoid counting a power reduction. 

In contrast, a plant that appropriately conducts equipment repairs in a well-planned manner

within 72 hours may be considered a poor performer. Another example is the Safety System

Unavailability performance indicator, which includes unavailability time for planned preventive

maintenance as well as unplanned corrective maintenance and equipment failures. A licensee

may consider delaying discretionary maintenance if it is near the performance indicator

threshold. Despite these concerns, there have not been any known instances to date of unsafe

actions by a licensee because of the performance indicators.

Panel recommendation: Ensure that the staffís ROP self-assessment program identifies and

evaluates any unintended consequences or unnecessary regulatory burden caused by the

performance indicators and that changes are made where appropriate.

Issue P-2: Safety System Unavailability performance indicators

Priority 1

Primary performance goals affected: Maintain safety/risk-informed/understandable

Issue description: Many of the performance indicator frequently asked questions (FAQs)

during initial implementation involved the safety system unavailability performance indicators 

definitions and guidance. The performance indicator definition of equipment unavailability is

different from that used by other NRC and industry programs that monitor or consider

unavailability of safety equipment (e.g., maintenance rule program and licenseeís probabilistic

risk assessments). The major hurdle in resolving this problem is determining whether the

unavailability is to be measured against the design basis or the risk analyses (i.e., operable
13

versus functional). For example, consideration of operator recovery actions are limited by this

indicator but are allowed in other programs. 

Other issues are how to treat fault exposure hours and what allowances should be made for

planned overhaul maintenance when a quantitative risk assessment has been previously

performed and approved by the NRC. The large number of generic and site-specific exceptions

to what equipment unavailability is counted in these performance indicators has made them

difficult to understand and may erode public confidence. Finally, these indicators also measure

appropriate actions by the licensee, such as planned preventive maintenance, so there is a

potential for unintended consequences (see P-1). 

Panel recommendation: Expedite the efforts to resolve the concerns about the safety system

unavailability performance indicators and implement any needed revisions to NEI 99-02,

ìRegulatory Assessment Performance Indicator Guideline,î to achieve consistency with other

applications.

Issue P-3: New performance indicators

Priority 2

Primary performance goals affected: Maintain safety/risk-informed

Issue description: Some current performance indicators and associated thresholds do not

directly correlate with risk. In addition, the panel found at least one unintended consequence of

performance indicators, specifically the misunderstanding that can occur because the

Green/White threshold for performance indicators is not related to risk, as with inspection

findings (see O-5). This misunderstanding causes the application of the action matrix to

sometimes appear inconsistent and calls into question the value of some performance

indicators as an input to performance assessment. For example, some emergency

preparedness and physical protection performance indicators do not directly correlate to risk to

the public health and safety, but are rather intended to identify weaknesses in licensee

programs. The staff specifically avoids the identification of such weaknesses in the more riskfocused cornerstone areas.

Panel recommendation: Continue the efforts to identify and evaluate improvements to

performance indicators. The staff should thoroughly evaluate any significant changes to the

performance indicators following the structured process in Inspection Manual Chapter 0608,

ìPerformance Indicator Program.î Further, the staff should evaluate the effect of any significant

changes in the scope of information provided by performance indicators on the action matrix

and the baseline inspection program, and should identify any additional costs and benefits.

Note: The panel cautions the staff not to eliminate a performance indicator solely because it

does not provide risk-informed information. The performance indicator may provide information

useful for other purposes (e.g., increasing public confidence, identifying programmatic issues). 

In this case, the staff may need to adjust the threshold.
14

Issue P-4: Frequently asked questions

Priority 2

Primary performance goals affected: Understandable/effectiveness and efficiency

Issue description: During initial implementation of the ROP, licensee and NRC staff members

asked many questions regarding the performance indicator guidance. These questions were

documented, answered, and posted on the NRCís ROP Web site. Although this was a useful

approach for clarifying and interpreting the guidance in NEI 99-02, the large number of

questions made it difficult for all stakeholders to understand the guidance on the performance

indicators. In addition, the inspectors noted that some licensees used site-specific answers to

the questions out of context when applying it to their specific situation.

Panel recommendation: Continue efforts to incorporate the answers to frequently asked

questions into the performance indicator guidance document, NEI 99-02, and to make the

answers more generic where possible.
15

Inspection

The new inspection process has been effective in focusing the NRCís inspection efforts on

areas more important to risk and safety. The combination of baseline and supplemental

inspections provides sufficient coverage of the attributes of the seven safety cornerstones. The

planning for both the overall inspection effort at a given site and for the selection of individual

procedural samples has been more risk-informed. The improvements in procedural guidance

have focused NRC resources on reviewing and assessing performance as opposed to

providing subjective views of various licensee processes and programs. The new procedural

guidance, being more objective and risk-informed, has also improved the consistency among

the regions and the individual inspectors. Since the outcomes of the new inspection program

are more risk-informed, the dialogue between the licensee and NRC is now primarily focused

on safety. However, the panel identified the following issues that the staff should address:

Issue I-1: Physical protection cornerstone inspections

Priority 1

Primary performance goals affected: Objective/unnecessary regulatory burden

Issue description: Licensees have expressed concern regarding the NRCís approach to

inspecting the licenseeís response to contingency events (i.e., force-on-force drills) and

applying the physical protection significance determination process (see S-4). Before initial

implementation, the staff removed the evaluation of force-on-force exercises from the baseline

inspection procedure (71130.03 ìResponse to Contingency Events (Protective Strategy and

Implementation of Protective Strategyî) and continued the Operational Safeguards Response

Evaluation (OSRE) inspections under inspection procedure 81110. The industry has proposed

a pilot program for industry-conducted drills and exercises (SECY-01-0060). The industry has

requested that self-assessment initiatives be considered as part of the inspection process

similar to how the staff handles the evaluation of emergency drills in the emergency

preparedness cornerstone.

Panel recommendation: Evaluate the inspection approach for the physical protection

cornerstone and revise the inspection program as necessary after the pending safeguards

performance assessment pilot program and the physical security rulemaking are completed. 

Issue I-2: Inspection report documentation threshold

Priority 2

Primary performance goals affected: Public confidence/effectiveness and efficiency

Issue description: The staff significantly revised the guidance for documenting inspections

under the ROP to provide a more structured approach for deciding the issues to be

documented. NRC Inspection Manual Chapter 0610*, ìPower Reactor Inspection Reports,î

changed the documentation threshold by eliminating discussions of positive performance

attributes, minor violations, licensee-identified findings, and nonregulatory concerns (such as

general weaknesses in programs or inspector observations) from the reports. The staff

implemented these documentation changes, in part, to address industry concerns that

subjective performance observations and inspector opinions in past reports were not based on

regulatory requirements and did not correspond to any observable performance problems. In

parallel with development of the ROP, the Office of Enforcement also developed improved
16

guidance to clarify the threshold for minor violations. This guidance was incorporated into MC

0610*. With the higher threshold for issues discussed in the inspection reports, some public

stakeholders are concerned that they may receive less performance information than under the

previous program. On the other hand, some stakeholders have noted that the more focused,

albeit reduced, information in inspection reports allows for a more effective identification of

significant regulatory and performance issues. Some inspectors are also concerned that they

may miss low-level performance trends if they do not include them in the inspection reports. 

Stakeholders have also expressed a concern that the staff is not consistently implementing the

new report documentation thresholds. 

NRC Inspection Manual Chapter 2515, ìLight Water Reactor Inspection Program - Operations

Phase,î endorses action by inspectors to provide licensees well-considered insights beyond

those to be documented in the inspection report. Although the documentation threshold has

changed, most licensees request that inspectors continue to share all of their observations at

exit meetings. Since these observations do not meet the report documentation threshold,

inspectors do not provide them to the public. This practice creates the appearance that

relevant performance information is being intentionally withheld from the public. Some public

stakeholders have suggested that all exit meetings should be open to the public to address this

concern.

Panel recommendation: Evaluate and revise guidance to inspectors as necessary to clarify

and promote consistency in documenting inspections. Continue conducting periodic audits of

inspection reports to identify and correct inconsistencies. The new documentation thresholds for

issues that have a defined level of safety or regulatory significance are appropriate. Inspection

observations and insights that do not reach the threshold should continue to be communicated

verbally to licensees for their consideration. 

Issue I-3: Appropriate level of baseline inspection

Priority 2

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: The inspection resource expenditures are about the same under the ROP

as under the previous inspection program. The staff has noted wide ranges in actual resource

expenditures, however, for certain procedures during the first year of implementation. 

Licensees have commented that specific cornerstones, such as occupational radiation

exposure, appear to have too many resources applied when licensee performance trends and

the previous inspection program are considered objectively (i.e., inspection resources have

increased although overall occupational exposure has decreased).

The PPEP recommended that the resource levels required to plan and implement the baseline

inspection program be evaluated during industry wide implementation, but cautioned that

process effectiveness not be measured solely by increases or decreases in resource utilization. 

They also recommended that the appropriateness of the inspection frequency and scope

continue to be assessed during industry wide implementation.
17

Panel recommendation: Evaluate inspection findings and performance indicator results for the

first yearís implementation and determine the appropriate level of effort to adequately assess

risk-significant areas in the baseline inspections. Modify the process as appropriate.

Issue I-4: Use of licensee self-assessment information

Priority 2

Primary performance goals affected: Unnecessary regulatory burden/effectiveness and

efficiency/public confidence

Issue description: In the previous inspection program, there were cases where the NRC did

not conduct portions of specific team inspections if the licensee had conducted a rigorous selfassessment of the same area and placed the self-assessment in the public domain. The staff

outlined this process in Inspection Procedure 40501, ìLicensee Self-Assessments Related to

Team Inspections.î When NRC management approved the use of this procedure as an

alternative to independent NRC inspection, the staff reviewed the scope and results of the

licenseeís self-assessment and the qualification of team members, and monitored ongoing

portions of the licenseeís review. The ROP has not provided this flexibility for reducing

unnecessary regulatory burden. While this approach did not save a significant amount of NRC

inspection resources, it reduced regulatory burden on the licensees, and the licensee staff

gained a better understanding of their systems. However, it could reduce public confidence in

the NRC as an independent regulator. 

Panel recommendation: After further experience with the ROP, review the results and

consider whether to waive certain parts of the baseline team inspections and let licensees

assess themselves under defined circumstances.
18

Significance Determination Process

The significance determination process (SDP) has shown that risk information can be used in a

systematic, practical, and repeatable manner. The SDP has given NRC inspectors an objective

process for consistently characterizing inspection findings, and it has provided the NRC with

improved tools for prioritizing emergent issues and selecting individual inspection samples. The

SDP has focused both NRC and licensee attention on the risk associated with identified issues,

as opposed to focusing attention and resources on the compliance implications. However, the

panel identified the following issues that the staff should address:

Issue S-1: SDP Phase 2 Worksheets

Priority 1

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: The primary tools to be used by field inspectors in determining the risk

significance of reactor safety cornerstone inspection findings, the Phase 2 worksheets, were

not available to the inspectors during initial implementation. The draft Phase 2 worksheets did

not accurately reflect the current site probabilistic risk assessments and equipment

configurations and needed to be revised. It was necessary for the regional risk analysts to

perform resource-intensive Phase 3 analyses of all potential non-Green reactor safety issues to

determine the risk significance of findings. The lack of adequate Phase 2 worksheets

negatively affected the effectiveness and efficiency of the ROP during the first year.

The PPEP highlighted the importance of having plant-specific SDP worksheets before

industrywide implementation.

Panel recommendation: Validate and issue the revised reactor safety SDP Phase 2

worksheets.

Issue S-2: Quality of NRC PRA tools

Priority 1

Primary performance goals affected: Maintain safety/public confidence

Issue description: The ROP relies on the quality and consistency of the probabilistic tools

used by the NRC risk analysts and inspectors for their risk characterizations and decisions. 

Currently, the NRC relies heavily on the individual plant probabilistic risk assessments

developed by the licensees, but the quality of these tools varies. The lack of validated Phase 2

worksheets magnified this concern during initial implementation (see S-1). For some findings

that reached the Phase 3 analysis stage, the licensees with state-of-the-art tools felt penalized

when the staff used their less sophisticated results to determine the risk significance of an

issue. Their concern was predictability, since the calculated risk significance may have been

greater with a less sophisticated tool. The lack of defined standards for methods and models

also hampers the staffís ability to obtain timely and consistent results when evaluating findings. 

Panel recommendation: Continue efforts to obtain improved and standardized risk analysis

tools for the reactor analysts. 
19

Issue S-3: Fire Protection SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/understandable

Issue description: The application of the fire protection SDP during the first year showed that

it was excessively complex and subjective. This has limited its usefulness as a tool in

evaluating some fire protection findings. Besides the complexity of the fire protection SDP, the

resulting risk characterization of the findings did not seem consistent with findings in other

cornerstone significance determinations.

Panel recommendation: Review lessons learned from use of the fire protection SDP, and

improve the risk characterization tool to make it more meaningful, effective, and efficient.

Issue S-4: Physical Protection SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/risk-informed

Issue description: The physical protection SDP was initially aligned to the reactor safety SDP. 

The staff found that the process was problematic in several cases during initial implementation. 

The SDP results seemed inconsistent with the actual risk significance. The staff made interim

revisions to the physical protection SDP to incorporate direction contained in Staff

Requirements Memorandum COMSECY-00-0036 dated January 25, 2001. 

Panel recommendation: Continue development of an improved physical protection SDP.

Issue S-5: Development of SDPs for other areas

Priority 1

Primary performance goals affected: Effectiveness and efficiency/predictable

Issue description: Substantial work is needed to complete the suite of SDP tools. During the

first yearís implementation, the established SDP did not provide an effective tool for evaluating

all inspector findings in certain areas (e.g., the staff identified the need for effective significance

determination tools concerning shutdown, containment, and external events). The staff cannot

easily assess other process-oriented inspection findings, such as those involving inadequate

application of the maintenance rule, with the existing SDP unless there is a measurable impact

on plant equipment.

Panel recommendation: Evaluate the need for other significance determination tools. The

staff should carefully evaluate any potential changes against the Agencyís goals. For example,

a new SDP that increases regulatory burden should have a corresponding benefit in

maintaining safety or increasing the Agencyís effectiveness.
20

Issue S-6: ALARA SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: During initial implementation, the staff experienced problems when using

the ALARA SDP. The SDP screened out all issues identified at plants that had a 3-year

average collective dose equal to or below the screening criteria based on the median industry

performance. The unintended consequence of this SDP structure is that the inspectors cannot

document the occurrence of an ALARA failure at these better performers in the inspection

report, but they document the identical finding at a plant above the screening criteria. In

addition, the staff designed the SDP structure to evaluate the licenseeís performance in ALARA

on a per job basis, but did not define a job. Another potential unintended consequence is that

some licensees may estimate exposure in a very conservative manner during ALARA planning. 

Some stakeholders have commented that the SDP screening criteria are an inappropriate de

facto definition of ALARA for occupational doses at nuclear power plants.

Panel recommendation: Evaluate lessons learned from initial implementation and revise the

ALARA SDP as necessary.

Issue S-7: Process for evaluating and communicating SDP conclusions

Priority 2

Primary performance goals affected: Effectiveness and efficiency/understandable/public

confidence

Issue description: Using an SDP to place all inspection insights into a risk-informed context

has been beneficial, but it has resulted in challenges. The breadth of potential issues and the

uniqueness of each plantís design and associated risk profile are leading to a highly complex

and time-consuming process that is challenging public confidence. The experience with

implementation of the SDP during the first year has been that the final risk characterizations are

often untimely and the process is not always transparent to the licensees and external

stakeholders. 

Inherent to the SDP is the fact that regional risk analysts and inspectors discuss technical

information and risk analysis assumptions with the licenseeís technical staff. These discussions

are used to ensure the NRCís risk analysis is technically sound, but they have given the

perception to many stakeholders that the finding is being debated out of the public view. 

Several public stakeholders have expressed concern that ìnegotiationsî occur between the

NRC staff and licensees during the risk characterization process. Public stakeholders have

also observed that the communication of the basis for the final risk significance determination is

not clear in all inspection reports, and does not always provide sufficient information for any

interested party to independently reconstruct the analysis. Some stakeholders have suggested

that all of the information used in the SDP, including licensee probabilistic risk assessments,

should be docketed. 

The time and resources committed to process individual potential non-Green issues have been

higher than expected, and many final determinations have not met the Agencyís timeliness

goals. The guidance concerning Agency decisions emphasizes risk-based criteria as opposed
21

to risk-informed. This could encourage protracted ìPRA battlesî that will hold up NRC actions

and could reduce the publicís confidence in the new processís effectiveness and efficiency. 

Stakeholders have observed that excessive time is also spent resolving disagreements

regarding Green inspection findings, which appears contrary to being risk-informed and

efficient. 

The PPEP noted that the staff needs to shorten the turnaround time for Phase 3 evaluations. 

The PPEP also recommended that the process for interactions between the NRC and licensee

during SDP evaluations be better defined, and that attention be focused on explaining the basis

for color assignments.

Panel recommendation: Use lessons learned from initial implementation to make the risk

characterization process expedient, scrutable, and understandable to all stakeholders. 

Issue S-8: Definition of a performance deficiency

Priority 2

Primary performance goals affected: Public confidence/maintain safety

Issue description: Early in the implementation of the ROP, the staff developed guidance in

NRC Inspection Manual Chapter 0609, ìSignificance Determination Process,î that required

inspectors to demonstrate a licensee performance deficiency before entering the SDP. This

policy caused concern among some inspectors because the NRC might appear to be

overlooking risk-significant issues simply because it had not established a clear performance

deficiency. In addition, the staff had difficulty developing a licensee performance deficiency if

the licensee could not establish a root cause for an equipment failure. There is a potential to

erode public confidence by giving the appearance that the NRC is not taking consistent actions

on risk-significant issues.

Panel recommendation: Use lessons learned from initial implementation to clarify the

definition of a performance deficiency.
22

Assessment and Enforcement

The assessment process associated with the ROP has effectively reduced the subjectivity of

the previous assessment process. The use of a predetermined action matrix, which places

objective performance indicators and inspection findings in a risk-informed context, has made

the determination and communication of NRC actions more streamlined and predictable. This

has improved public confidence by providing a clear road map for understanding Agencylicensee interactions and regulatory decisions. Enforcement actions are now performancebased, with an emphasis on placing the significance of the noncompliance into a safety context

and ensuring that the licensee is taking corrective actions. The response of the Agency is more

timely under the new assessment scheme than under the previous process. However, the

panel identified the following issues that the staff should address:

Issue A-1: Length of time inspection finding is included in action matrix

Priority 1

Primary performance goals affected: Unnecessary regulatory burden/understandable

Issue description: According to NRC Inspection Manual Chapter 0305, ìOperating Reactor

Assessment Program,î a non-Green inspection finding is normally carried forward in the

assessment program (i.e., action matrix) for a total of four calendar quarters. Performance

indicators are recalculated quarterly. Licensees have proposed that there be a graded

approach for how long findings remain active rather than the fixed 1 year. Considering the risk

significance of the various findings, it may be beneficial to establish a graded approach for

resetting inspection findings in the action matrix.

Panel recommendation: Evaluate a graded approach for resetting non-Green inspection

findings as entry conditions into the action matrix. 

Issue A-2: No-color inspection findings

Priority 1

Primary performance goals affected: Understandable/public confidence

Issue description: The ROP has proceduralized the use of no-color findings. The role of nocolor findings, however, is not clear and has contributed to process inconsistencies and

confusion among many stakeholders. No-color findings are associated with specific

extenuating circumstances listed in NRC Inspection Manual Chapter 0610*. These findings

typically address regulatory issues that are more than minor violations, but do not meet the

threshold for entry into the existing cornerstone significance determination process. These

issues do not receive a severity level or color and, therefore, the NRC does not characterize

their importance. Early in the process, the staff guidance was nonspecific and the result was

the appearance of a new finding classification. 

Licensees are concerned that these findings may be inappropriately used to artificially inflate

the significance of individual issues; likewise, they believe that many of the issues do not

warrant inclusion in inspection reports. External stakeholders have noted that the staff has

established a new undefined category of findings. Furthermore, no-color findings are colored

blue on the NRCís Web page, and their role in the process is not understandable.
23

Panel recommendation: Evaluate and clarify the guidance on the designation, definition, and

use of what are presently called no-color findings, and find a more appropriate term for these

findings.

Issue A-3: Purpose of the Regulatory Conference

Priority 2

Primary performance goals affected: Public confidence/effectiveness and

efficiency/understandable

Issue description: The purpose of a regulatory conference is to gain a complete

understanding of the risk significance of an inspection finding and to obtain information

pertinent to understanding any apparent violations. During initial implementation of the ROP,

stakeholders noted that the objectives of the regulatory conferences were not clear. The

structure of the regulatory conferences was very similar to that of the enforcement conferences

in the previous program. The regulatory conference discussions sometimes focused more on

enforcement and corrective actions rather than on the determination of the risk significance of

the issue. Additionally, licensee and NRC managers have expressed discomfort with holding

public meetings concerning a potentially risk-significant issue and not including all potential

decision-makers. 

Panel recommendation: Clarify the guidance on the objectives and structure of regulatory

conferences and communicate this guidance to the external and internal stakeholders.

Issue A-4: Multiple related inspection findings

Priority 2

Primary performance goals affected: Unnecessary regulatory burden/effectiveness and

efficiency

Issue description: It is not clear how the NRC should disposition multiple inspection findings

that are related to the same technical problem or root cause. For example, would it be

appropriate to characterize an inspection finding involving five related issues (3 Green, 1 White,

1 Yellow) as one finding or five separate findings? If they are handled as separate findings, the

NRC response, as determined by the action matrix, will be different from if they are considered

one finding. Experience has shown that significant events and conditions are often caused by

multiple performance failures.

Panel recommendation: Develop clear guidance on how to handle multiple related inspection

findings, and communicate the guidance to all stakeholders.
24

ROP Self-Assessment Program

Through briefings to the IIEP by the NRC staff and review of the preliminary self-assessment

metrics, the panel evaluated the ROP self-assessment program to determine whether the

developed program was sound and whether it included mechanisms for self-correction. The

staff developed the self-assessment program to determine whether the ROP is meeting its

objectives (including the Agencyís performance goals), to gather information about overall

industry performance, and to develop information to support possible improvements. The selfassessment program includes more than 75 metrics for measuring the success of the overall

ROP and how effectively it supports the four ROP elements (performance indicators, inspection

program, significance determination process, assessment program). The metrics are aligned to

the ROP objectives (risk-informed, predictable, understandable, objective) and the Agencyís

performance goals (maintain safety, increase public confidence, increase effectiveness and

efficiency, reduce unnecessary regulatory burden). The staff evaluates the metrics on a

periodic basis using information from various sources, including the inspection program,

performance indicators, periodic audits, stakeholder surveys, and public comments.

Based on the timing of the panelís activities and the parallel collection and assessment of selfassessment data, limited data and results were available for the panel to review. Although the

panel acknowledges the significant efforts of the staff in developing the self-assessment

program, it is best described as a noteworthy work in progress, which may require further

enhancements and refinements based upon the evaluation of the data collected. Although

individual panel members provided comments to the staff for consideration in further defining

and revising the individual metrics, the panel, as a whole, took no consensus position on the

overall adequacy and acceptability of the metrics. They are, even now, continuing to be refined

by the staff and the data for assessing their overall efficacy were not available. Nevertheless,

the panel concluded that the self-assessment program has the necessary elements to evaluate

the ROP against the Agencyís performance goals. The panel, however, could not evaluate the

effectiveness of the program given that the first yearís assessment data was not yet available. 

The panel was encouraged by the fact that, for the most part, the staff had identified the same

issues the panel identified.
Attachment 1

IIEP CHARTER

UNITED STATES NUCLEAR REGULATORY COMMISSION CHARTER

 REACTOR OVERSIGHT PROCESS INITIAL IMPLEMENTATION EVALUATION PANEL

1. The Committeeís official designation:

 Reactor Oversight Process Initial Implementation Evaluation Panel (IIEP)

2. The Committees objectives and the scope of its activity:

The NRC has implemented a revised reactor oversight process (ROP) for commercial

nuclear power plant licensees. The ROP is described in NRC Inspection Manual

Chapter 2515. Background information on the development of the ROP is contained in

Commission papers SECY-99-007, ìRecommendations For Reactor Oversight Process

Improvements,î dated January 8, 1999, SECY-99-007A, ìRecommendations For

Reactor Oversight Process Improvements (Follow-up to SECY-99-007),î dated March

22, 1999, and SECY-00-049, ìResults of the Revised Reactor Oversight Process Pilot

Program,î dated February 24, 2000. These Commission papers describe the scope and

content of performance indicator reporting, a new risk-informed baseline inspection

program, a new assessment process, and revisions to the enforcement policy. 

Commission paper SECY-00-049 also describes the results from the Pilot Program

Evaluation Panel (a previous Federal Advisory Committees Act (FACA) panel), including

a recommendation from the panel to proceed with initial implementation of the ROP at

all power reactor facilities. On March 28, 2000, the Commission approved initial

implementation of the ROP, and on May 17, 2000, the Commission directed the NRC

staff to convene another evaluation panel under FACA to evaluate the first year of

implementation of the ROP. The staff has established this IIEP in response to the

Commissionís directions.

The IIEP will function as a cross-disciplinary oversight group to independently monitor

and evaluate the results of the first year of initial implementation of the ROP and provide

advice and recommendations to the Director of the Office of Nuclear Reactor Regulation

on reforming and revising the ROP. IIEP meetings will be announced to the public in

advance and (unless closed according to the provisions of FACA) open to the public. 

Subject to the Freedom of Information Act, all material made available to or prepared for

or by the IIEP will be made available to the public. Meetings will be transcribed and

meeting summaries will be prepared following each meeting to document the results of

the meetings. The transcripts and meeting summaries will be publicly available.

The IIEP will evaluate the ROP results against performance measures. The IIEP will

provide a written report containing an overall evaluation of the ROP to the Director of the

Office of Nuclear Reactor Regulation. This report will include the consensus views of

the panel, or the majority and minority views when panel consensus cannot be achieved. 

The NRC staff will use the IIEP evaluation as a major input to its deliberative process to

determine what modifications, if any, are needed to the ROP following initial

implementation.
3. The period of time necessary for the Committee to carry out its purpose:

Nine months

4. The NRC official to whom this Committee will report:

Committee Chairman

Loren Plisco

Director, Division of Reactor Projects

Region II

5. The NRC office and individual responsible for providing support for the Committee:

Office of Nuclear Reactor Regulation

Designated Federal Official

John Monninger

Technical Assistant, Associate Director for Inspection & Programs

Office of Nuclear Reactor Regulation

6. A description of the duties for which the Committee is responsible, and if such duties are

not solely advisory, a specification of the authority for such functions:

The panel will provide advice and recommendations only.

7. The estimated annual operating costs, in dollars* and staff years, for the Committee:

a. $120,000 (*Includes travel and per diem)

b. 0.50 FTE

8. The estimated number and frequency of the Committee meetings:

Approximately 3 meetings will be held, on an as needed basis, during the period the

panel is in existence.

9. Organization - The IIEP will create any subcommittees which may be necessary to fulfill

the IIEPís mission. In addition, NRC and IIEP will establish such operating procedures

as are required to support the group, consistent with the Federal Advisory Committee

Act, as amended.

10. The Committeeís termination date, if less than two years from the date of establishment:

 July 31, 2001

11. Charter Filing Date:

October 17, 2000

/RA/

___________________________

Andrew L. Bates

Advisory Committee Management Officer

Office of the Secretary of the Commission
Attachment 2

IIEP MEMBERS

In selecting members for the Initial Implementation Evaluation Panel, the NRC considered

interested persons and groups with professional, technical, or personal qualifications or

experience that could contribute to the functions and tasks of the panel. The NRC considered

several factors in appointing IIEP members, including (1) the requirements in 10 CFR Part 7,

ìAdvisory Committees,î directing balance in advisory committee membership in terms of the

points of views represented and the functions to be performed, (2) Commission direction in the

staff requirements memorandum for SECY-00-0049 for a panel with a cross-section of

stakeholders similar to those who participated in the original evaluation panel, and with the

addition of at least one resident inspector and one senior reactor analyst, (3) a desire for

independence from the NRC office responsible for development and oversight of the ROP

(Office of Nuclear Reactor Regulation) and a focus on those stakeholders most affected by the

ROP, and (4) a desire to provide both continuity and new perspectives in terms of the individual

panel members selected. 

Panel Members

Loren Plisco, NRC, Region II (Chairman)

Randolph Blough, NRC, Region I

Kenneth Brockman, NRC, Region IV

Richard Borchardt, NRC, Office of Enforcement*

Mary Ferdig, Benedictine University; Ferdig, Inc.**

Steve Floyd, Nuclear Energy Institute 

David Garchow, PSEG Nuclear LLC

Richard Hill, Southern Nuclear Operating Company

Rod Krich, Exelon Corporation

Robert Laurie, California Energy Commission

David Lochbaum, Union of Concerned Scientists**

James Moorman, NRC, Region IV (Senior Resident Inspector)

Steven Reynolds, NRC, Region III 

Edward Scherer, Southern California Edison

James Setser, Georgia Department of Natural Resources

Raymond Shadis, New England Coalition on Nuclear Pollution**

James Trapp, NRC, Region I (Senior Reactor Analyst)

Support to the Panel

Facilitator: Francis X. Cameron - NRC, Office of the General Counsel

Designated Federal Official: John D. Monninger - NRC, Office of Nuclear Reactor Regulation

* Richard Borchardt was originally appointed to the panel and participated in all panel activities

through the fourth meeting of February 26-27, 2001. Subsequently, he was appointed to the

position of NRR Associate Director for Inspection and Programs and assumed those duties as

of March 19, 2001. In recognition of the desire for independence in panel membership from

NRR, Mr. Borchardt decided to recuse himself from panel activities effective April 2, 2001.

** David Lochbaum was originally appointed to the panel and participated in the first panel

meeting on November 1-2, 2000. He resigned from the panel on November 6, 2000. On

December 4, 2000, Mary Ferdig and Ray Shadis were appointed to the panel.
Attachment 3

BIBLIOGRAPHY OF PANEL ACTIVITIES

1. Results of the Revised Reactor Oversight Process Pilot Program, SECY-00-049,

February 24, 2000, ADAMS ML003683227

2. Staff Requirements Memorandum on SECY-00-0049, ìResults of the Revised Reactor

Oversight Process Pilot Program (Part 2),î May 17, 2000, ADAMS ML003715823

3. IIEP Charter, October 17, 2000, ADAMS ML003760300

4. Letter to General Services Administration, October 17, 2000, ADAMS ML003760307

5. Letter to Library of Congress, October 17, 2000, ADAMS ML003760327

6. Letters to Congressional Oversight Committees, October 17, 2000, ADAMS

ML003763800

7. First IIEP Meeting Summary and Transcript, November 1-2, 2000, ADAMS

ML003774521

8. Second IIEP Meeting Summary and Transcript, December 11-12, 2000, ADAMS

ML010090359 & ML010530300

9. Third IIEP Meeting Summary and Transcript, January 22-23, 2001, ADAMS

ML010530104

10. Fourth IIEP Meeting Summary and Transcript, February 26-27, 2001, ADAMS

ML010880350

11. Fifth IIEP Meeting Summary and Transcript, April 2-3, 2001, ADAMS ML011140513

12. Sixth IIEP Meeting Summary and Transcript, April 25, 2001, ADAMS ML011280333

13. Final IIEP Report, May 10, 2001, ADAMS ML011290025
Attachment 4

SOURCES OF PANEL INFORMATION

December 11-12, 2000

Bill Sherman, Vermont Department of Public Service

Gary Wright, Illinois Department of Nuclear Safety

January 22-23, 2001

Joseph Brady, NRC Senior Resident Inspector

Sonia Burgess, NRC Senior Reactor Analyst

Stephen Campbell, NRC Senior Resident Inspector

Jeffrey Clark, NRC Senior Resident Inspector

Steven Jones, NRC Senior Resident Inspector

William Jones, NRC Senior Reactor Analyst

Jill Lipoti, New Jersey Department of Environmental Protection

James Trapp, NRC Senior Reactor Analyst

Dennis Zannoni, New Jersey Department of Environmental Protection

February 26-27, 2001

Victor Dricks, NRC Office of Public Affairs

Steve Floyd, Nuclear Energy Institute

Rich Janati, Pennsylvania Department of Environmental Protection

Judith Johnsrud, ECNP, Sierra Club

Steve Kerekes, Nuclear Energy Institute

David Lochbaum, Union of Concerned Scientists

Jenny Weil, McGraw Hillís Inside NRC