Reactor Oversight Process 

Independent Assessment 

2013

Enclosure
ii 

Reactor Oversight Process Independent Assessment 

Working Group Members: 

Eric Benner 

Daniel Collins 

Stuart Easson 

Michael Jones 

Brian McDermott 

David Pelton 

Acknowledgment 

The Reactor Oversight Process (ROP) Independent Assessment Working Group has gathered 

insights from internal and external stakeholders as inputs in developing an informed 

independent view of the ROP and its implementation. The working group deliberated on its 

recommendations in a collegial manner and collaborated to produce the final report text. The 

working group was comprised of individuals with more than 135 years of collective experience in 

the nuclear field who also had no recent duties associated with ROP implementation or 

substantive ties to the original ROP development effort. The diversity of the group members’ 

experience and views provided a strong measure of independence to this review. I thank the 

working group members for their active engagement in the review and professional commitment 

to the project. 

Brian J. McDermott 

 
iii 

EXECUTIVE SUMMARY 

The U.S. Nuclear Regulatory Commission’s (NRC’s) Reactor Oversight Process (ROP) 

integrates the agency’s inspection, assessment, and enforcement activities for operating 

commercial nuclear power reactors. The Operating Reactor Assessment Program evaluates 

the overall performance of the facilities using the ROP results and communicates this 

information to licensee management, members of the public, and other stakeholders. 

The ROP was initially implemented in 1999 and has undergone several evolutionary changes in 

its implementation. Self assessments are conducted by the Office of Nuclear Reactor 

Regulation (NRR) on an annual basis and have identified opportunities to enhance and refine 

the oversight process. While some changes in the implementation of the process have 

enhanced how specific aspects of plant performance are evaluated, there have been no 

significant changes in the ROP’s conceptual framework since the inception of the program. 

In October 2012, the Commission directed agency staff to provide a fresh review of the 

practices and approaches the NRC has developed for the ROP over the course of years. 

Specifically, the Commission requested an independent review of the program’s objectives and 

implementation. The ROP Independent Assessment Working Group developed this report in 

response to the Commission’s request. 

Review Objectives and Approach 

A six-person working group comprising of NRC staff with no current duties associated with 

implementation of the ROP and no substantive ties to the original ROP development effort 

performed the 2013 independent assessment of the ROP. The working group members were 

selected on the basis of their diversity of nuclear and regulatory experience. To provide an 

independent assessment, the working group sought information about the ROP implementation 

from a variety of internal and external sources, identified focus areas, and then made its 

assessment of the ROP outcomes in comparison to the program objectives. The charter for the 

working group is provided as Attachment 1. 

The ROP derived its original objectives from the NRC Strategic Plan and included maintaining 

safety; increasing openness; making NRC activities and decisions more effective, efficient, and 

realistic; and reducing unnecessary regulatory burden. As the program has matured, the 

objective of reducing unnecessary regulatory burden has become an integral part of the second 

objective, “making NRC activities and decisions more effective, efficient, and realistic.” The 

working group used multiple means to collect information and stakeholder perspectives 

regarding the effectiveness of the ROP in meeting these objectives. Prior assessments of the 

ROP and its effectiveness completed between 2001 and 2012 were reviewed to identify 

common themes. These included program office self assessments, external reviews and 

audits, and collections of stakeholder feedback. To obtain current perspectives, the working 

group interviewed a range of internal and external stakeholders including industry groups and 

corporate leaders, non-governmental organizations, and NRC staff and senior leadership. 

The working group formulated its assessment of the ROP and its implementation by comparing 

the collection of past and present feedback as well as program outcomes with the original 

objectives of the ROP. Themes identified during the working group’s initial review were 

discussed at a public meeting on June 18, 2013, and stakeholder input on the issues was 

considered in the final assessment. The 2013 ROP Independent Assessment Report 
iv 

documents the working group’s review, recommendations, and suggestions. A list of the 

documents reviewed by the working group and additional information regarding the public 

meeting to solicit stakeholder views are provided as Attachments 2 and 3, respectively. 

Conclusions, Recommendations, and Suggestions 

The working group found that the ROP has been effective in accomplishing its objectives of 

maintaining safety, increasing openness, and making NRC activities and decisions more 

effective, efficient, and realistic. The ROP framework has enabled the NRC to effectively and 

predictably allocate oversight resources based on the safety and security performance of the 

operating reactor fleet. The working group’s assessment of views provided by internal and 

external stakeholders in interviews and written evaluations found no broad issues of concern 

regarding the ROP framework. The insights provided as well as the working group’s 

assessment indicate that the ROP’s systematic approach to oversight provides sufficient 

information to reach reliable regulatory conclusions. While implementation of the ROP 

framework for the next generation of nuclear power plants would need to account for substantial 

differences in the baseline risk of new reactor designs, the fundamental ROP framework will 

continue to serve the agency well. 

The ROP benefits from an extensive self assessment process that includes consideration of 

program metrics, outcomes achieved, and input from external stakeholders. These self 

assessments have resulted in evolutionary changes in the program implementation such as the 

adoption of the Mitigating System Performance Indicator. The working group considered the 

focus on self assessment an important element in the success of the ROP. 

The working group’s overall assessment is that the ROP is functioning well. Opportunities to 

further enhance the ROP structure, program implementation, and the conduct of self 

assessments were noted, and each issue was prioritized by designation as either a 

“recommendation” or a “suggestion.” A recommendation identifies an issue that the working 

group believes would enhance the ROP in terms of meeting its overall objectives and warrants 

staff evaluation. A suggestion relates to an option for potential improvement at an operational 

level that the working group believes should be considered by staff as future changes to the 

ROP are evaluated. 

While each recommendation or suggestion has its individual merits, two recommendations 

suggest that a thorough reexamination of the current ROP approach would be beneficial. 

Specifically, the threshold for the Degraded Cornerstone column of the ROP Action Matrix 

(Recommendation 5) and the use of Substantive Cross-Cutting Issues/Safety Culture Attributes 

in the ROP process (Recommendation 6). These two issues garnered the most commentary 

from both the regulators and the regulated community. While functional in their current state, 

these aspects of the ROP present the greatest opportunity to enhance the efficiency of the 

oversight process. 

A summary table of all recommendations and suggestions identified through this ROP 

Independent Assessment is provided as Attachment 4 to this report. The details of the 

recommendations and suggestions are provided in the body of the report. The following are the 

recommendations of the ROP Independent Assessment working group organized by the ROP 

goals: 
v 

Maintaining Safety

Issue: Licensee requests for an extended period of time to conduct cause evaluations and plan 

corrective actions in advance of Supplemental Inspections for Column 4 of the ROP Action 

Matrix could potentially delay NRC reviews to verify the adequacy of the licensee’s corrective 

action program while operations continue. 

Recommendation 1: Clarify expectations for the timing of supplemental inspections for 

Column 4 of the ROP Action Matrix, or portions thereof, to ensure that the NRC’s assessment of 

continued operation and consideration of additional regulatory actions are completed in a timely 

manner. (Section 4.1) 

Issue: The ROP assessment process does not require a licensee to demonstrate improved 

performance (i.e., adequate and lasting corrective actions) before a plant is moved from a 

“higher” Action Matrix column back to the Licensee Response column. This can lead to abrupt 

swings in the Action Matrix characterization of a plant’s performance, might appear disjointed to 

external stakeholders, and can result in unnecessary resource burdens for the NRC and 

licensees. 

Recommendation 2: Consider including additional measures in the ROP to minimize abrupt 

changes in the Action Matrix characterization of plant performance caused by mechanistic 

relaxation of oversight based on the passage of time and completion of NRC inspections. 

(Section 4.2) 

Issue: Generic issues are often considered “resolved” by licensee commitments to 

programmatic changes and future actions. After initial NRC reviews, the inspection program 

does not generally include explicit activities to confirm continued licensee implementation over 

time. 

Recommendation 3: Include a risk-informed periodic review of licensee programs or actions 

implemented to address generic issues to enhance the agency’s assurance that these 

measures continue to be effectively implemented. (Section 4.3) 

Increasing Openness

Issue: Current NRC message presentation used for routine ROP communication and outreach 

with stakeholders can lead to confusion regarding the safety significance of licensee 

performance issues or the meaning of regulatory actions. The heavy reliance on ROP 

terminology in NRC messages contributes to heightened stakeholder concern over performance 

issues that are within the anticipated range of licensee performance and do not impact safe 

operations. 

Recommendation 4: Consider enhancements to improve the effectiveness of NRC messages 

through more extensive use of plain language, a focus on the desired effect of the 

communication on stakeholder perceptions, and the use of wording that coveys the significance 

of issues to the broadest possible audience. (Section 5.1) 

Making NRC Activities and Decisions More Effective, Efficient, and Realistic

Issue: Internal and external stakeholders agree that the resources expended to disposition a 

finding at the Green/White threshold can be excessive. Licensees are willing to expend a great 
vi 

deal of resources to challenge a White finding because, in part, the increase in regulatory 

oversight for two White findings in a cornerstone (transition to Column 3 of the NRC Action 

Matrix) is not viewed as proportionate to the risk associated with the White findings. 

Consequently, the NRC also expends significant resources to finalize the characterization of 

White findings. 

Recommendation 5: The NRC should review the criteria for transition to Column 3 of the NRC 

Action Matrix against the original ROP program goals to ensure that the significance of White 

inspection findings is not being overemphasized and to ensure that agency resources used to 

process White inspection findings are commensurate with findings that, by definition, are of low 

to moderate safety significance. (Section 6.3.3) 

Issue: A rigorous comparison of the benefits of considering cross-cutting issues and safety 

culture in the ROP on licensee safety performance and an assessment of the NRC resources 

expended to achieve those benefits have not been performed. Additionally, the working group 

found differences between various NRC policy, program, and communication documents when 

describing the purposes and uses of cross-cutting issues and safety culture in the ROP. 

Recommendation 6: The NRC should perform a comprehensive analysis to determine whether 

the use of cross-cutting issues and safety culture, as currently incorporated in the ROP, 

provides regulatory value in terms of licensee safety performance for the resources expended. 

To support that determination, NRC staff should clarify and document the goals, purposes, 

uses, and desired outcomes associated with the inclusion of cross-cutting issues and safety 

culture in the ROP. If program changes are needed, the staff should determine whether 

Commission approval is required for implementation. (Section 6.4.3) 

Issue: The identification of performance issues applicable to multiple licensees through 

inspections or through the Task Interface Agreement process is an anticipated outcome of the 

NRC’s oversight process; however, some issues could be more efficiently and effectively 

addressed by NRC and industry by using other NRC processes to better ensure a consistent 

generic response. 

Recommendation 7: Clarify ROP program expectations for when performance issues that are 

common to multiple facilities should be considered for resolution through a generic issues 

process in order to improve the use of NRC inspection resources and ensure a thorough and 

consistent industry response. (Section 6.5) 

Issue: The ROP self assessment process focuses on identifying and addressing operational 

issues impacting program implementation and does not explicitly evaluate the effectiveness of 

the ROP in achieving expected outcomes. 

Recommendation 8: Consider revising the ROP self assessment process to better solicit and 

assess both tactical and strategic feedback. Reexamine how internal and external feedback is 

collected, analyzed, and used to improve oversight approaches to and the implementation of the 

ROP. (Section 6.6) 
vii 

CONTENTS 

1.0 INTRODUCTION .................................................................................................................... 1

2.0 INDEPENDENT REVIEW ....................................................................................................... 2

2.1 ROP Independent Assessment Working Group ............................................................. 2

2.2 Summary of Review Activities ........................................................................................ 2

3.0 ROP BACKGROUND AND BASES ....................................................................................... 4

4.0 ROP GOAL - MAINTAINING SAFETY .................................................................................. 7

4.1 Timing of Supplemental Inspections for Column 4 ......................................................... 7

4.2 Demonstration of Improved Plant Performance Before Returning to Column 1 ............. 9

4.3 Assessment of Licensee Corrective Actions in Response to Generic Issues ............... 11

5.0 ROP GOAL - INCREASING OPENNESS ............................................................................ 13

5.1 Impacts of Message Presentation on Stakeholder Perceptions ................................... 13

5.2 Meaning of “All Green” Performance Indicators (PIs) ................................................... 14

6.0 ROP GOAL - EFFECTIVE AND EFFICIENT DECISIONMAKING ....................................... 16

6.1 ROP Assessment Process ........................................................................................... 16

6.1.1 Consideration of Additional Performance Inputs ................................................... 16

6.1.2 Timing of Supplemental Inspection Activities for Column 2 and Column 3 ........... 19

6.1.3 Discussion of Licensee Performance at ROP Management Meetings .................. 20

6.1.4 Action Matrix Deviations ........................................................................................ 22

6.2 ROP Inspection Process .............................................................................................. 23

6.2.1 Flexibility Within the ROP Inspection Program ...................................................... 23

6.3 The Significance Determination Process (SDP) ........................................................... 25

6.3.1 SERPs and Uncertainties Associated with Risk Assessment Tools ...................... 26

6.3.2 Documentation of SDP Decisionmaking ............................................................... 27

6.3.3 Impact of White Findings on Resources ................................................................ 29

6.4 Use of Cross-Cutting Issues and Safety Culture in the ROP........................................ 31

6.4.1 Background and Discussion .................................................................................. 31

6.4.2 Stakeholder Feedback .......................................................................................... 35

6.4.3 Assessing the Value of Using Cross-Cutting Issues and Safety Culture ............... 37

6.4.4 Effective Use of Substantive Cross-Cutting Issues (SCCIs) ................................. 37

6.5 Generic Treatment of Common Plant Performance Issues .......................................... 39

6.6 Scope of ROP Self Assessments ................................................................................. 40

7.0 TOPICS OF INTEREST ........................................................................................................ 43

7.1 ROP Applicability and Implementation Strategy for New Reactors .............................. 43
viii 

ATTACHMENT 1: CHARTER (3 PAGES) 

ATTACHMENT 2: LIST OF DOCUMENTS REVIEWED 

ATTACHMENT 3: PUBLIC MEETING TO SOLICIT STAKEHOLDER VIEWS 

ATTACHMENT 4: TABLE OF RECOMMENDATIONS AND SUGGESTIONS 
1.0 INTRODUCTION 

In the staff requirements memorandum (SRM) for SECY-12-0081, “Risk-Informed Regulatory 

Framework for New Reactors” (Agencywide Documents Access and Management System 

(ADAMS) Accession No. ML12296A158), the Commission approved two staff recommendations 

associated with the review and analysis for new reactor designs and disapproved a staff 

recommendation related to the Reactor Oversight Process (ROP) that would have resulted in 

continued staff use of existing risk-informed ROP tools and augmentation of qualitative tools 

with deterministic backstops to ensure an appropriate regulatory response for new reactor 

designs. In its disapproval of the staff’s recommendation related to the ROP, the Commission 

specified that the staff should give additional consideration to the use of relative risk metrics or 

other options that would provide a more risk-informed approach to the determination of the 

significance of inspection findings for new reactors. In addition, the Commission requested an 

independent review of the practices and approaches the U.S. Nuclear Regulatory Commission 

(NRC) has developed for the ROP over the course of years. In its tasking, the Commission 

stated the following: 

The Commission would benefit from a fresh review of the practices and 

approaches the NRC has developed for the [ROP] over the course of years. The 

staff should pursue an independent review of the program’s objectives and 

implementation including the relative roles of headquarters and regional staff, our 

interactions with industry over performance indicator assessments, and the 

effectiveness of NRC’s assessment of substantive cross-cutting issues [(SCCIs)]. 

Such an assessment would provide a reinforced foundation upon which the 

agency can plan for the operational review of new nuclear power plants based on 

Generation III+ reactor technology. 

This paper responds to the Commission’s specific tasking for the conduct of an independent 

assessment of the practices and approaches the NRC has developed for the ROP. The staff’s 

response to the Commission tasking regarding relative risk metrics is being addressed by a 

separate staff working group and is not contained in this assessment report. 

 
2 

2.0 INDEPENDENT REVIEW 

2.1 ROP Independent Assessment Working Group

On February 26, 2013, the Deputy Executive Director for Reactor and Preparedness Programs 

established a ROP Independent Assessment Working Group to conduct the independent review 

directed by the Commission in the SRM for SECY-12-0081. The working group created a 

charter to define the objective, scope, coordination and communication, expected products, 

schedule, and staffing. A copy of the working group charter and a list of the documents 

reviewed by the working group are provided as Attachments 1 and 2 to this report. 

To provide the independence directed by the Commission, the selection of working group 

members was limited to individuals who were not substantially involved in the ROP 

development and have not recently been involved in ROP implementation. The charter for this 

independent review specified that review be conducted by NRC staff, independent of the 

present stewards of the ROP in the Office of Nuclear Reactor Regulation (NRR) and the NRC 

staff members that initially developed the program. 

The independent review was conducted as a collaborative assessment by the working group 

members. The working group sought to identify programmatic issues based on its assessment 

of a wide variety of inputs against the original ROP goals and objectives. The inputs included 

the collection of existing self assessments, external reviews, and stakeholder feedback 

amassed since the initial implementation of the ROP in 2001. The working group also obtained 

current views on the effectiveness of the program relative to its goals by conducting structured 

interviews with current ROP practitioners and stakeholders and by observing a sample of 

ROP-related processes. Additionally, themes identified during the working group’s initial review 

were also discussed at a public meeting on June 18, 2013, and stakeholder input on the issues 

was considered in the final assessment. Information regarding this public meeting to solicit 

stakeholder views is provided as Attachment 3 to this report. 

The working group was cognizant of efforts underway in NRR to enhance the ROP including an 

ROP enhancement initiative and a business process improvement project for the significance 

determination process (SDP). The independent review focused on strategic-level achievement 

of objectives and outcomes rather than on ROP implementation issues, thereby minimizing the 

opportunity for duplication of efforts by the working group and NRR staff. 

2.2 Summary of Review Activities

The working group reviewed 22 previously documented internal and external assessments of 

the ROP along with any reports of related follow-up actions. These documents along with the 

ROP’s bases documents helped the group gain an appreciation for changes to the ROP since 

its inception. It also provided an opportunity for the working group to identify recurring themes 

and potential areas for improvement. 

Working group members observed a number of routine ROP activities in progress including 

several Monthly ROP meetings with industry, the 2012 ROP End-of-Cycle (EOC) review 

meeting, an escalated enforcement review panel, and the Agency Action Review Meeting. The 

working group also observed the regional session at the 2013 Regulatory Information 

Conference where a panel of NRC and Industry leaders discussed a variety of ROP issues. 
3 

The observations provided the group with insights on the issues of concern to individuals 

actively engaged in implementing the program. 

Interviews were conducted with a representative sample of internal and external stakeholders 

with the goal of obtaining their insights on the effectiveness of the ROP and areas for 

improvement. A combination of individual and group interviews were conducted to obtain 

candid views, as well as to benefit from the synergy offered by interviewing a group of ROP 

practitioners. The interviewees included NRC management and staff, industry and corporate 

leaders, and representatives from non-governmental organizations. 

A summary table of all recommendations and suggestions identified through this ROP 

independent assessment is provided as Attachment 4 to this report. 

 
4 

3.0 ROP BACKGROUND AND BASES 

The ROP integrates the NRC’s inspection, assessment, and enforcement programs. The 

Operating Reactor Assessment Program evaluates the overall performance of operating 

commercial nuclear reactors and communicates this information to licensee management, 

members of the public, and other stakeholders. 

The Operating Reactor Assessment Program collects information from NRC inspections and 

licensee-reported performance indicators (PIs) to enable the NRC to develop objective 

conclusions about a licensee’s safety performance. Based on this assessment information, the 

NRC determines the appropriate level of its response, such as performing supplemental 

inspections, conducting meetings with NRC and licensee management, or issuing orders to shut 

down plants. The assessment information and NRC response are then communicated to the 

public, except for certain security-related information associated with the Security Cornerstone 

that the Commission has determined to withhold from public disclosure. The NRC conducts 

follow-up actions, as applicable, to ensure that the corrective actions designed to address 

performance issues were effective. 

The ROP was also developed to meet the following four agency strategic performance goals: 

• maintain safety; 

• increase openness; 

• make NRC activities and decisions more effective, efficient, and realistic; and 

reduce unnecessary regulatory burden.1

The ROP applies to all operating commercial nuclear reactors except those sites that are under 

Inspection Manual Chapter (IMC) 0350, “Oversight of Reactor Facilities in a Shutdown 

Condition Due to Significant Performance and/or Operational Concerns” (ADAMS Accession 

No. ML063400076). While this process outlines the NRC’s general approach to reactor 

oversight, the structure and guidance of the ROP do not restrict the NRC from taking any 

necessary actions to fulfill its responsibilities under the Atomic Energy Act of 1954, as amended. 

Some of the key tenets of the ROP and the drivers in its development were (1) to improve the 

objectivity of the oversight processes so that subjective decisionmaking is minimized, (2) to 

improve the scrutability and predictability of NRC actions so that regulatory response has a 

clear tie to licensee performance, and (3) to risk inform the processes so that NRC and licensee 

resources are focused on performance issues with the greatest impact on safe plant operation. 

In ways consistent with Regulatory Guide (RG) 1.174, “An Approach for Using Probabilistic Risk 

Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis” 

(ADAMS Accession No. ML100910006), the ROP’s risk-informed processes integrate risk 

insights with more traditional deterministic factors (such as defense in depth and maintaining 

safety margins) to guide regulatory decisionmaking. 

 

1

 In 2004, the NRC implemented a revised Strategic Plan in which an Effectiveness goal was established to 

“Ensure that NRC actions are effective, efficient, realistic, and timely;” this replaced the previous goal to 

“Make NRC activities and decisions more effective, efficient, and realistic.” At the same time, the prior goal 

to “Reduce unnecessary regulatory burden” was eliminated. 
5 

The regulatory framework for reactor oversight consists of the following three key strategic 

performance areas: reactor safety, radiation safety, and safeguards. Within each strategic 

performance area are cornerstones that reflect the essential safety aspects of facility operation. 

These seven cornerstones include initiating events, mitigating systems, barrier integrity, 

emergency preparedness, public radiation safety, occupational radiation safety, and security. 

Satisfactory licensee performance in the cornerstones provides reasonable assurance that the 

licensee’s facilities are being operated safely and that the NRC’s safety mission is being 

accomplished. Each cornerstone contains inspection procedures and licensee-reported PIs to 

ensure that their objectives are being met. Both inspection findings and licensee-reported PIs 

are evaluated and given a color designation based on their safety significance, and this 

designation feeds the ROP Action Matrix to determine a predictable regulatory response. This 

framework is shown in the following figure. 

SDP implementation guidance is contained in IMC 0609 (ADAMS Accession 

No. ML101400479). IMC 0609, Appendix A, “The Significance Determination Process (SDP) for 

Findings at Power” (ADAMS Accession No. ML101400574), is used to determine the risk 

significance of performance deficiencies in the initiating events, mitigating systems, and barrier 

integrity cornerstones. Risk thresholds are a function of changes in core damage frequency 

(CDF) and large early release frequency (LERF) against a plant’s baseline risk. For those 

relatively infrequent cases in which sufficient probabilistic risk assessment (PRA) methods and 

tools are not available or appropriate to provide reasonable and timely estimates of safety 

significance, the staff uses IMC 0609, Appendix M, “Significance Determination Process Using 

Qualitative Criteria” (ADAMS Accession No. ML101550365), and considers the best available 

information and factors such as defense in depth, safety margins, and the potential for plant 

wide impacts attributable to the performance deficiency to determine the safety significance in 

those cases. Several additional SDPs are more subjective to determine an equivalent 

regulatory response (i.e., emergency preparedness, radiation safety, and security). 

Aspects of licensee performance such as human performance, the establishment of a safety 

conscious work environment (SCWE), and the effectiveness of licensee problem identification 

and resolution programs, although not identified as specific cornerstones, are still important to 

meeting the agency’s safety mission. While determining how best to account for these aspects 

of performance, which can affect multiple cornerstones, the staff concluded that these items 

generally manifest themselves as the root causes of performance problems. Adequate licensee 
6 

performance in these “cross-cutting” areas is assessed either explicitly in each cornerstone area 

or is inferred through cornerstone performance results from both NRC inspections and 

licensee-reported PI data. 

The ROP was developed with the presumption that plants that have significant licensee 

performance issues with cross-cutting areas would be revealed through the existence of safety 

significant PIs or inspection findings. The NRC identifies an SCCI to inform the licensee that the 

NRC has a concern with the licensee’s performance in the cross-cutting area and to encourage 

the licensee to take appropriate actions before more significant performance issues emerge. 

Implementation guidance for the PI program including but not limited to the mitigating systems 

performance index (MSPI) is contained in IMC 0608, “Performance Indicator Program” (ADAMS 

Accession No. ML12219A374). More detailed guidance on the data collection and PI 

calculations are contained in Nuclear Energy Institute (NEI) 99-02, “Regulatory Assessment 

Performance Indicator Guideline” (ADAMS Accession No. ML13261A116), which is jointly 

produced and maintained by the NEI and the NRC. The MSPI covers five systems important to 

safety; it tracks the unavailability of monitored trains and the unreliability of monitored 

components. The MSPI calculation reflects the deviation of a specific unit’s performance from 

an industry baseline, converted to a simplified delta (change in) CDF (ΔCDF). A performance 

limit, or deterministic backstop, is also used for determining degraded performance. 

The ROP Action Matrix identifies the range of NRC and licensee actions and the appropriate 

level of communication for different levels of licensee performance. The Action Matrix describes 

a graded approach for addressing performance issues and was developed with the philosophy 

that within a certain level of safety performance (i.e., the licensee response band), licensees 

would address their performance issues without additional NRC engagement beyond the 

baseline inspection program. NRC actions are graded in such a way that the NRC becomes 

more engaged as licensee performance declines. If licensee actions are not being taken to 

address performance issues, the NRC may consider expanding the scope of the applicable 

supplemental inspection to appropriately address the area(s) of concern. 

The implementation guidance for NRC’s response to events is contained in MD 8.3, “NRC 

Incident Investigation Program” (ADAMS Accession No. ML031250592), and supplemented by 

IMC 0309, “Reactive Inspection Decision Basis for Reactors” (ADAMS Accession 

No. ML111801157). Deterministic criteria are used for initial event screening and a range of risk 

thresholds are subsequently applied to determine whether a reactive inspection will be 

launched. The risk-informed reactive inspection thresholds are a function of conditional core 

damage probability (CCDP) and conditional large early release probability. An overlap of 

options provides flexibility based on uncertainty and deterministic insights, and additional 

deterministic criteria are reviewed and documented as the basis for staff decisions on the 

appropriate regulatory response. While these ranges offer some flexibility in determining the 

level of event response, they still involve thresholds based on risk values. 

 
7 

4.0 ROP GOAL - MAINTAINING SAFETY 

The NRC is committed to maintaining safety, and the ROP framework, as discussed in 

Section 3.0, has as its highest level the NRC’s overall mission to ensure that commercial 

nuclear power plants are operated in a manner that provides adequate protection of public 

health and safety. The staff has identified those aspects of licensee performance that are 

important to the mission and, therefore, merit regulatory oversight and identified them as the 

strategic performance areas of reactor safety, radiation safety, and safeguards which form the 

second level of the regulatory oversight framework. 

With a risk-informed perspective, the staff has also identified the most important elements in 

each of these strategic performance areas that form the foundation for meeting the overall 

agency mission. These elements were identified as the cornerstones of safety in the third level 

of the regulatory oversight framework structure. These cornerstones are the fundamental 

building blocks for the ROP, and acceptable licensee performance in these cornerstones should 

provide reasonable assurance that the overall mission of adequate protection of public health 

and safety is met. 

The working group found that the ROP has been effective in accomplishing its objective of 

maintaining safety. In a way consistent with the spirit of continuous improvement demonstrated 

in the ROP self assessments, the working group’s independent review identified opportunities to 

further enhance the ROP’s ability to maintain safety as discussed below. 

4.1 Timing of Supplemental Inspections for Column 4

When licensee performance is in Column 4 (the Multiple/Repetitive Degraded Cornerstone

column) of the ROP Action Matrix, the licensee is expected to place the identified deficiencies in 

its corrective action program and perform an evaluation of the root and contributing causes for 

both the individual and the collective issues. However, from an oversight perspective, the 

significance of these performance issues may indicate that the licensee’s corrective action has 

pervasive or systemic deficiencies, and the NRC needs to assess whether additional regulatory 

action is necessary. 

 

The NRC performs Inspection Procedure (IP) 95003, “Supplemental Inspection for Repetitive 

Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs or One Red 

Input” (ADAMS Accession No. ML102020551), to review the breadth and depth of the 

performance deficiencies, assess the licensee’s evaluation of its safety culture, and 

independently perform a graded assessment of the licensee’s safety culture. This procedure 

was written with the assumption that supplemental inspections (either IP 95001, “Supplemental 

Inspection for One or Two White Inputs in a Strategic Performance Area” (ADAMS Accession 

No. ML102020522), or IP 95002, “Supplemental Inspection for One Degraded Cornerstone or 

Any Three White Inputs in a Strategic Performance Area” (ADAMS Accession 

No. ML102020532)) have been conducted to evaluate the licensee’s root-cause, 

extent-of-cause, and extent-of-condition evaluations and associated corrective actions for 

“White” or greater performance indicators or inspection findings. If those supplemental 

inspections have not been conducted, the scope of this inspection should include inspection of 

the licensee’s evaluation of those issues. 

 
8 

IP 95003 includes the following inspection objectives: 

• to provide the NRC with additional information to be used in deciding whether the 

continued operation of the facility is acceptable and whether additional regulatory actions 

are necessary to arrest declining plant performance, and 

• to provide an independent assessment of the extent of risk significant issues to aid in the 

determination of whether an unacceptable margin of safety or security exists. 

The procedure requires that some sample reviews be performed for all key attributes of the 

affected strategic performance areas because “additional NRC assurance is required to ensure 

public health and safety, and security”. 

As indicated in IMC 0305, “Operating Reactor Assessment Program” (ADAMS Accession 

No. ML13178A032), following the completion of IP 95003, the NRC’s Office of the Executive 

Director for Operations (EDO) or their designee in conjunction with the appropriate Regional 

Administrator and the Director of NRR will decide whether additional NRC actions are 

warranted. At a minimum, the regional office issues a Confirmatory Action Letter (CAL) to 

document the licensee’s commitments (as discussed in its performance improvement plan) and 

any other written or verbal commitments. Other actions are also considered including 

performing additional supplemental inspections, issuing a demand for information, or issuing an 

order up to and including a plant shutdown. 

Stakeholders provided feedback that flexibility in the timing for the conduct of IP 95003 to 

accommodate licensee actions can result in significant delays in performance of the 

supplemental inspection. Feedback indicated that such delay can impact NRC management’s 

ability to make timely decisions regarding plant performance. Other stakeholders commented 

that they believe the ROP needs to provide a better mechanism to prompt 

performance-challenged plants to correct their deficiencies in a timely manner and that allowing 

delays in the conduct of supplemental inspections runs counter to that belief. 

Once a licensee has entered Column 4 of the ROP Action Matrix, the ROP indicates that 

supplemental inspection is necessary. The inspection is, in part, to determine whether other 

regulatory actions up to and including plant shutdown are necessary. However, a specific time 

frame for initiating the inspection is not specified. IP 95003 indicates that, in most cases, the 

licensee will complete its root-cause, extent-of-cause, and extent-of-condition investigations of 

the performance deficiencies and will complete an independent third-party assessment of their 

safety culture before the NRC begins the inspection. In some cases, NRC inspection of these 

activities has been deferred to accommodate a longer time required for the licensee to complete 

these actions. 

In the latest ROP self assessment metric report, the staff has noted that the average time 

elapsed before the supplemental inspections were performed in CY 2012 was higher than the 

yearly average of all prior years. The delays in performing the supplemental inspections were 

often caused by the licensee not being ready for the inspection; the staff intends to add 

guidance to the next revision of IMC 0305 to emphasize that supplemental inspections should 

be completed in a timely manner. The staff acknowledges that licensees play a significant role 

in the timely completion of supplemental inspections, which are not initiated until the licensee 

indicates its readiness. The working group was informed that as part of its ROP Enhancement 

effort, the NRR staff plans to examine additional ways to encourage the timely completion of 

supplemental inspections. 
9 

Conceptually, a licensee in Column 4 could be required to shut down as a result of performance 

issues identified during conduct of an IP 95003 inspection. However, if the conduct of the 

inspection were delayed for an extended period of time, the NRC's ability to make the 

necessary regulatory decision would also be delayed. The working group acknowledges that 

the NRC could make such regulatory decisions in the absence of the inspection. Additionally, a 

significant delay could impact whether the licensee was placed in the appropriate column of the 

ROP Action Matrix because of delayed identification of performance deficiencies. 

The working group believes that either (1) setting specific timeframes for when IP 95003 should 

be performed or (2) more explicitly segmenting the procedure to conduct some sample reviews 

of key attributes of the affected strategic performance area before the licensee’s completion of 

its evaluation would better provide NRC decisionmakers with timely information to decide 

whether additional regulatory actions are necessary. 

Issue: Licensee requests for an extended period of time to conduct cause evaluations and plan 

corrective actions in advance of Supplemental Inspections for Column 4 of the Reactor 

Oversight Process (ROP) Action Matrix could potentially delay NRC reviews to verify the 

adequacy of the licensee’s corrective action program while operations continue. 

Recommendation 1: Clarify expectations for the timing of supplemental inspections for 

Column 4 of the ROP Action Matrix, or portions thereof, to ensure that the NRC’s assessment of 

continued operation and consideration of additional regulatory actions are completed in a timely 

manner. 

4.2 Demonstration of Improved Plant Performance Before Returning to Column 1

The ROP was developed with the philosophy that within a certain level of safety performance 

(i.e., the licensee response band), licensees would address their performance issues without 

additional NRC engagement beyond the baseline inspection program. If a licensee’s 

performance declines to such a degree that the assessment results in the plant moving to 

higher columns in the Action Matrix, the NRC assesses the licensee’s root-cause evaluation and 

corrective actions through baseline and supplemental IPs 95001 (Regulatory Response; 

Column 2), 95002 (Degraded Cornerstone; Column 3), and 95003 (Multiple/Repetitive 

Degraded Cornerstone; Column 4). Although the 95003 procedure contains provisions for 

ensuring that licensee corrective actions are effective, the 95001 and 95002 inspections allow 

credit for licensees having planned, but not necessarily implemented, corrective actions in order 

to successfully complete the supplemental inspection. 

Because the 95001 and 95002 procedures allow inspectors to credit actions planned by the 

licensee, the inspectors do not assess the licensee’s effectiveness in improving performance 

before completion of the inspection that would move the licensee to the left in the Action Matrix 

and that would reduce NRC oversight. The procedures only require a verification that 

appropriate corrective actions are identified and scheduled and that measures of success are 

put in place; the inspection focuses on plans, not actual performance. Consequently, poorly 

performing licensees can successfully complete the inspection and return to the Licensee 

Response column (Column 1) before the implementation of corrective actions thereby resulting 

in a situation in which the plant assessment may not match actual plant performance. 

Additionally, once the inspection is completed, licensees could adjust/modify their corrective 

action plans without regulatory involvement. This can result in a situation in which corrective 
10 

actions are deferred, thereby raising the possibility of a return to higher columns of the Action 

Matrix. 

The feedback received indicated that this process does not ensure effective oversight or 

improved licensee performance in all cases. Licensees that have difficulty in identifying and 

implementing adequate and lasting corrective actions might wind up back in the Regulatory 

Response or Degraded Cornerstone columns in a short period of time – which “cycles” both 

NRC and licensee resources. Specific feedback provided included, but was not limited to, the 

following: 

• Plants are closing more issues to “trending” rather than fixing the issues. 

• Room is needed for “wisdom” in moving between columns (i.e., movement should be 

3 → 2 → 1 rather than jumping from 3 →1) when findings clear off of the board. In other 

words, sustained performance should be observed before a licensee is put back in 

Column 1. 

• There is not a lot of rigor behind licensee follow-up actions to address SCCIs. 

• IP 95001 is adequate to look at a specific issue that caused a White finding, but it 

doesn’t really look at the “bigger picture” with respect to licensee performance. 

• Guidance is needed on how to determine when a licensee’s corrective action program is 

not working. The ROP is based on the licensee’s corrective action program working, but 

we don’t have criteria for determining/validating that the corrective action program is 

working properly Should there be a periodic validation/“recertification” of the corrective 

acton program? 

The working group attempted to validate the anecdotal feedback received on this matter. 

Review of historical plant assessment information revealed that there have been 37 instances 

since the inception of the ROP in which a plant moved from an Action Matrix column higher than 

Column 1 to a lower column and then subsequently moved back to a higher column in fewer 

than four quarters. Since 2009, this has occurred 10 times. In several of these cases, the 

underlying issues that contributed to the subsequent move to the right were identical, or related, 

to the issues that contributed to initial move to the higher column. In the most extreme example, 

one plant moved into a higher column, returned to a lower column, and then re-returned to the 

higher column five times in a 6-year period. 

The working group concludes that the effectiveness of the ROP could be enhanced by requiring 

licensees to demonstrate improved performance before returning to the Licensee Response

column or by otherwise providing a graduated process for returning to the Licensee Response

column. Recognizing that the ROP is intended to be performance based, the group also 

believes that this recommended enhancement could be structured in a manner that would allow 

those licensees that have a consistent track record for effectively managing their corrective 

action process to be given credit for planned corrective actions; whereas licensees that have not 

demonstrated effective implementation of corrective actions would have to implement and 

demonstrate effectiveness of corrective actions for the root causes of the greater-than-Green 

inputs to the assessment process. 
11 

Issue: The Reactor Oversight Process (ROP) assessment process does not require a licensee 

to demonstrate improved performance (i.e., adequate and lasting corrective actions) before a 

plant is moved from a “higher” Action Matrix column back to the Licensee Response Column. 

This can lead to abrupt swings in the Action Matrix characterization of a plant’s performance, 

might appear disjointed to external stakeholders, and can result in unnecessary resource 

burdens for the NRC and licensees. 

Recommendation 2: Consider including additional measures in the ROP to minimize abrupt 

changes in the Action Matrix characterization of plant performance caused by mechanistic 

relaxation of oversight based on the passage of time and completion of NRC inspections. 

4.3 Assessment of Licensee Corrective Actions in Response to Generic Issues

To fulfill its responsibility to protect public health and safety, the NRC performs the following five 

principal regulatory functions: 

• developing regulations and guidance; 

• licensing the operation of nuclear power plants; 

• inspecting and assessing licensees to ensure that licensees comply with NRC 

requirements and taking appropriate follow-up or enforcement actions when necessary 

(accomplished through the ROP for reactor licensees); 

• evaluating operational experience; and 

• conducting research, holding hearings, and obtaining independent reviews to support 

regulatory decisions. 

As identified in Item 4 of Figure 1 from the NRC 2013-2014 Information Digest (NUREG-1350, 

Volume 25), reproduced below as Figure 4-1, generic issues are occasionally identified which 

require regulatory action by the NRC and licensees. When the NRC identifies a generic issue, it 

may impose new requirements on licensees by issuing orders or revising regulations. The NRC 

also clarifies the application of existing requirements to newly-identified circumstances through 

various generic communications such as bulletins, generic letters, and regulatory information 

summaries. 
12 

Figure 4-1 How We Regulate 

If new requirements are imposed or existing requirements are clarified, the NRC often performs 

near-term oversight of reactor licensees to assess the adequacy of any actions that they have 

taken. This near-term oversight is often performed through the issuance of a Temporary 

Instruction which provides guidance to inspectors on how to inspect for the particular issue. 

Feedback indicates that this process is generally effective for ensuring that licensees have 

taken adequate short-term corrective actions to address the particular issue. However, 

feedback and observation indicate (1) that the ROP is less effective at ensuring that longer-term 

corrective actions are adequately implemented and (2) that, as corrective actions are migrated 

into routine operational programs, they continue to be linked to the instigating generic issue for 

context and continue to remain effective in mitigating the instigating generic issue. The group 

believes that the program could be enhanced to provide more effective oversight of long-term 

implementation of licensee corrective actions to resolve generic issues by developing 

risk-informed mechanisms to identify attributes of generic issue resolution that could be 

incorporated into or linked to the baseline inspection program. 

Issue: Generic issues are often considered “resolved” by licensee commitments to 

programmatic changes and future actions. After initial NRC reviews, the inspection program 

does not generally include explicit activities to confirm continued licensee implementation over 

time. 

Recommendation 3: Include a risk-informed, periodic review of licensee programs or actions 

implemented to address generic issues to enhance the agency’s assurance that these 

measures continue to be effectively implemented. 

 
13 

5.0 ROP GOAL - INCREASING OPENNESS 

The NRC is committed to improving its communications with stakeholders and has directed the 

staff to use plain understandable language whenever possible in documents and at public 

meetings. Several years of ROP periodic self assessment reports included feedback from a 

variety of stakeholders which highlighted that the assessment process could be enhanced by 

modifying the way NRC communicates technical information to the public. Specifically, the 

feedback indicated that staff should look for ways to clarify for industry, and the public, the 

meaning of Green and White performance indicators within the ROP, The feedback claimed 

that these indicators are too complex and cumbersome and at times do not appear consistent 

across all ROP cornerstones. Commenters on the ROP echoed what was found in the self 

assessments (primarily that communications are not nearly as effective for the public as they 

are for the industry). Press releases and other forms of public information do not always clearly 

communicate the significance of Green and White findings which causes confusion around the 

risk to the public. 

5.1 Impacts of Message Presentation on Stakeholder Perceptions

The working group’s review of documentation and interviews of internal and external 

stakeholders were used as an opportunity to gain feedback related to NRC message 

presentation. The working group reviewed the feedback to assess how NRC’s messages 

related to the ROP and how findings influence the public’s perceptions regarding plant safety. 

Specific feedback provided included but was not limited to the following: 

• The public has difficulty with the color schemes of the Action Matrix. 

• Getting the agency to use plain language continues to be a struggle. 

• Annual meetings for Columns 1 and 2 of the Action Matrix are of lesser value, and 

meetings for Green or White findings cause confusion. 

• The NRC’s strong messaging and degree of public outreach regarding White findings 

result in escalated stakeholder concerns while the overall safety of a facility is strong. 

• Action Matrix uses versus resource allocation – Does the NRC clearly communicate the 

meaning of Action Matrix outcomes? How does resource allocation match up with 

overall performance by licensees? 

• Use of Action Matrix Column 5 versus IMC 0350, IMC 0351, CALs, etc. – How well does 

the NRC communicate the differences between actions associated with IMCs 0350 and 

0351 or issuance of a CAL versus being in Column 5 of the Action Matrix (unacceptable 

performance)? 

• Usefulness of annual public meetings and the messages that are conveyed – Annual 

meetings for Columns 1 and 2 of the Action Matrix are of lesser value. Annual meetings 

for Green or White findings cause confusion. 

Most messages associated with the ROP consist of program language with a heavy focus on 

the specific details of a particular issue or process and less focus on the larger message being 

given to stakeholders. Currently, there is neither a comprehensive strategy to develop message 
14 

presentation that conveys the intended significance to the broadest possible audience, nor is 

there a structured review to assess how effectively the NRC’s intended messages are conveyed 

to the general public. The working group recognizes the challenge in retaining sufficient 

program detail to ensure clear messages for industry and well-informed stakeholders while at 

the same time providing a clear safety message for the general public. 

Issue: Current NRC message presentation used for routine Reactor Oversight Process (ROP) 

communication and outreach with stakeholders can lead to confusion regarding the safety 

significance of licensee performance issues or the meaning of regulatory actions. The heavy 

reliance on ROP terminology in NRC messages contributes to heightened stakeholder concern 

over performance issues that are within the anticipated range of licensee performance and do 

not impact safe operations. 

Recommendation 4: Consider enhancements to improve the effectiveness of NRC messages 

through more extensive use of plain language, a focus on the desired effect of the 

communication on stakeholder perceptions, and the use of wording that coveys the significance 

of issues to the broadest possible audience. 

5.2 Meaning of “All Green” Performance Indicators (PIs)

Within the ROP Regulatory Framework, the NRC evaluates plant performance by analyzing the 

following two distinct inputs: inspection findings resulting from NRC's inspection program and 

PIs reported by the licensees. Several years of ROP periodic self assessment reports, in 

addition to feedback from a variety of stakeholders, highlight the need to clarify the message 

delivered to stakeholders when PI’s are “always Green.” 

In SRM-M080604, “Briefing on Results of the Agency Action Review Meeting (AARM), 

9:00 A.M., Wednesday, June 4, 2008, Commissioners’ Conference Room, One White Flint 

North, Rockville, Maryland (Open to Public Attendance)” dated June 30, 2008 (ADAMS 

Accession No. ML081820771), the Commission directed the staff to look for ways to clarify to 

industry and the public the meaning and use of Green PIs within the ROP. Although the staff 

endeavored to clarify the meaning and role of PIs in the assessment process, stakeholder 

feedback continued to indicate an ongoing concern that PIs should better distinguish between 

levels of licensee performance. In response, staff revised slides used in public meetings to 

emphasize that Green PI performance represents performance in which cornerstone objectives 

are fully met. NRC staff changed the NRC ROP Web site to better explain how inputs to the 

ROP assessment were used. In 2008, the NRC discontinued the Personnel Screening Program 

and the Fitness-for-Duty/Personnel Reliability PIs because they provided minimal input and 

because reasonable confidence existed through the security baseline inspection program. 

The NRC has developed and implemented a robust operational experience feedback program 

that has a well organized and structured inspection program with clear goals and publicly 

available procedures to ensure plant safety. The working group interview respondents generally 

indicated that the ROP program’s use of PIs was effective. There were a few areas in which the 

respondents questioned whether the NRC could improve its communication and perhaps the 

breadth of issues covered by PIs. Representative questions included the following: 

• Are PIs that are typically Green providing a reasonable assessment tool? 
15 

• If licensees are managing their processes and equipment efficiently, did the PI have its 

appropriate outcome? 

• Should there be a PI for other areas such as quality of licensing (might indicate 

weaknesses in engineering, licensing skills, or other aspects) or financial aspects? 

The working group’s takeaways from its review of documentation and interviews during this 

review indicate that stakeholders do not understand the value that “always Green” PIs can 

provide or the consideration of other indicators that have been contemplated. As the staff 

periodically evaluates existing PIs or considers alternative PIs, the stakeholders would benefit 

from presentation of strategic messages to ensure that the value of indicators used, or 

considered, is effectively communicated to the public.

Issue: Many stakeholders question the value of Performance Indicators (PIs) that are “always 

Green.” 

Suggestion 1: Review current PIs to evaluate whether they are providing meaningful 

information on licensee performance. If the PIs are validated as being appropriate and not 

needing adjustment, develop messages to enhance stakeholder understanding of how the PIs 

continue to contribute to the NRC’s assessment of plant safety and licensee performance. 

 
16 

6.0 ROP GOAL - EFFECTIVE AND EFFICIENT DECISIONMAKING 

The goal of making NRC activities and decisions more effective and efficient was a priority of 

the NRC strategic plan at the time the ROP was originally implemented. This goal was intended 

to ensure adequate protection of public health and safety, as well as the environment, through a 

focus on maintaining the quality of the technical base for our decisions and by optimizing our 

regulatory activities, while maintaining safety and increasing public confidence. In working 

toward this performance goal, the NRC applied its Principles of Good Regulation which include 

improved efficiency, clarity, and reliability. 

In the late 1990s, feedback from stakeholders, self assessments, international experience, and 

research results suggested that the NRC should capitalize on advances in technology, 

implement efficiencies to improve our internal processes, and improve the quality and bases for 

decisionmaking. Feedback and the NRC’s own analyses suggested that the NRC should 

improve the consistency and predictability of regulatory decisions by evolving to a more 

risk-informed and performance-based approach. 

The NRC implemented the following strategies to make NRC activities and decisions more 

effective, efficient, and realistic: 

• used risk information to improve the effectiveness and efficiency of NRC activities and 

decisions; 

• based agency decisions on technically sound and realistic information; 

• anticipated challenges posed by the introduction of new technologies and changing 

regulatory demands; and 

• identified, prioritized, and modified processes based on effectiveness reviews in order to 

maximize opportunities to improve those processes. 

The working group conducted reviews of ROP-related documents (such as self assessments), 

conducted interviews with internal and external stakeholders, and observed some routine ROP 

activities in progress. Based on its assessment of these inputs, the working group concluded 

that the ROP continues to contribute to improved NRC decision quality, timeliness, and 

consistency with regard to licensee oversight and plant safety. The program includes a blend of 

technical and organizational reviews and feedback processes that contribute to effective 

risk-informed, performance-based decisionmaking. During the conduct of this independent 

assessment, the working group identified opportunities for improvement regarding the 

assessment, inspection, and SDP processes. 

6.1 ROP Assessment Process

6.1.1 Consideration of Additional Performance Inputs 

IMC 0308, “Reactor Oversight Process (ROP) Basis Document” (ADAMS Accession 

No. ML071860181), indicates that the original architects of the ROP gave consideration to use 

of traditional enforcement information in the licensee assessment process. The Commission at 

that time, however, gave the staff specific guidance in the SRM for SECY-98-045, “Status of the 

Integrated Review of the NRC Assessment Process for Operating Commercial Nuclear 
17 

Reactors” (ADAMS Accession No. ML003752969), stating, “While the enforcement program is a 

valuable regulatory tool, the Commission does not desire that enforcement be used as a ‘driving 

force’ of assessment activities.” 

After several years of ROP implementation, the periodic self assessment process (which 

included feedback from a variety of stakeholders) highlighted that the assessment process 

might be enhanced by including consideration of a broader set of inputs than just inspection 

findings and licensee-reported performance indicator data. In 2009, IMC 0305 was revised to 

include consideration of traditional enforcement and SCCIs in the process of assessing licensee 

performance. 

While the revised guidance provides important flexibility in the assessment process, the 

feedback received from stakeholders interviewed for this independent assessment indicates that 

the revised guidance is either not fully understood or does not completely explain when/how to 

consider “other” information in the assessment process. Specifically, the feedback provided 

included, but was not limited to, the following: 

• The public/licensees don’t always understand differences between ROP and traditional 

enforcement and/or how they work together. 

• More could be done to integrate the ROP with enforcement. “Why doesn’t traditional 

enforcement work its way into the Action Matrix?” 

• IMC 0305 and IMC 2515 (“Light Water Reactor Inspection Program – Operations 

Phase,” (ADAMS Accession No. ML13176A336) could be clearer on how to handle 

emergent issues. 

• SCCIs are arbitrary. Are we driving resources to areas where they really aren’t 

warranted? There are too many different bins in which to put issues; and there is a need 

to simplify the process. There is confusion regarding the role of cross-cutting aspects. 

• The process needs better structure for following up on SCCIs (particularly for long 

standing issues). IMC 0305 doesn’t really have guidance on what to do after a licensee 

has had four or five cycles of SCCIs. 

• SCCIs seem to be poor predictors of licensees’ improving or declining performance. 

• ROP has trained licensees to focus on only what the ROP covers. 

• Health physics and emergency preparedness inspectors are not looking at cross-cutting 

issues as hard as they should. They don’t seem to recognize how effective this tool can 

be. 

• How does ROP enforcement history get considered in traditional enforcement space 

(when a plant is getting traditional enforcement for an issue)? 

An extension of the consideration of traditional enforcement and SCCI information, which are 

discussed in IMC 0305, would be to consider “other” potentially relevant information that is not 

explicitly discussed in IMC 0305. Examples of potentially relevant information include petitions 

under Title 10 of the Code of Federal Regulations (10 CFR) 2.206, “Requests for Action under 
18 

this Subpart;” licensing issues; and financial issues that are within the scope of NRC 

regulations. 

The working group discussions included consideration that a licensee assessment should not 

be unduly influenced by a simplistic measure such as the volume of unverified or 

unsubstantiated 2.206 petitions or allegations. However, information regarding substantiated

2.206 petitions or allegations, information related to the quality and timeliness of licensing action 

submittals (which in many cases is subject to the same quality assurance (QA) program as the 

licensee’s maintenance program), and financial issues that are within the scope of NRC 

regulations (such as decommissioning funding assurance and adequacy of underlying assets) 

could potentially provide meaningful insights into broader licensee performance on matters that 

are ultimately non-trivial. Examples of feedback received included the following: 

• Enforce 10 CFR 50.9, “Completeness of Accuracy of Information,” in the licensing 

space. 

• Force strict adherence to the design and licensing basis in all aspects of the ROP. 

• Independent spent fuel storage installations could be better incorporated into the ROP 

(the same licensee and the same QA program that are covered by the ROP ... but 

treated differently when it comes to dry fuel storage). 

• The process needs more focus on licensed operator requalification. 

• Commercial-grade dedications aren’t getting looked at. 

• Aging management is not fully addressed in ROP space. 

• Completeness and accuracy of licensing action submittals can vary widely. Incomplete 

or inaccurate submittals require additional staff time and effort to review and can 

unnecessarily challenge timeliness goals. 

• Observation: Significant weakness in licensing technical support performance (resulting 

in application rejection and/or the identification of significant quality issues) can identify 

weaknesses in licensee capability. 

Based on its assessment of the feedback from stakeholders and review of the program 

documents, the working group concluded that there is potential value in considering, in a limited 

way, a wider variety of performance information (such as substantiated 2.206 petitions, 

substantiated allegations, licensing issues, and financial issues that are within the scope of NRC 

regulations) in the assessment process. Additionally, this could include consideration of 

licensing-related factors (such as technical adequacy of engineering support or quality 

assurance). In order to effectively use information other than inspection findings and 

licensee-reported PI data, the process for including “other” considerations in NRC performance 

assessments needs to be well defined and transparent in order to achieve consistent results. 
19 

Issue: Although traditional enforcement, allegations, substantive cross-cutting issues, and 

safety culture are discussed in Reactor Oversight Process program documents as contributors 

to NRC’s assessment process, current guidance documents do not clearly explain how or when 

to factor consideration of these issues into the assessment process. 

Suggestion 2: Enhance the Operating Reactor Assessment Program to ensure that plant 

performance assessment decisions fully and consistently consider “other” relevant performance 

perspectives discussed in Inspection Manual Chapter (IMC 0305), including traditional 

enforcement, allegations, substantive cross-cutting issues, and safety culture in addition to the 

Action Matrix outcomes. 

Issue: The population of “other” types of information (such as traditional enforcement) that may 

be considered in the assessment process is narrowly defined in Reactor Oversight Process 

program documents. Additional indicators of licensee performance may be available to support 

NRC performance assessments but are not included in the set of “other” information currently 

specified in Inspection Manual Chapter (IMC) 0305. 

Suggestion 3: Consider expanding the list of relevant indicators of licensee performance in the 

Operating Reactor Assessment Program description in IMC 0305 to include significant 

performance concerns that may come to light through 10 CFR 2.206 petitions, licensing issues, 

or financial issues that are within the scope of NRC regulations. 

6.1.2 Timing of Supplemental Inspection Activities for Column 2 and Column 3 

When plant performance is determined to be in Column 2 (the Regulatory Response column) or 

Column 3 (the Degraded Cornerstone column) of the ROP Action Matrix, the licensee is 

expected to place the identified deficiencies in its corrective action program and perform an 

evaluation of the root and contributing causes for both the individual and the collective 

performance issues. 

Subsequently, the NRC performs IP 95001 for Column 2 or IP 95002 for Column 3. These 

supplemental inspections assess the licensee’s actions to evaluate and correct identified issues. 

The 95002 procedure also assesses whether safety culture components caused or significantly 

contributed to the risk significant performance issues. As indicated in IMC 0305 and MD 8.14, 

“Agency Action Review Meeting” (AARM, ADAMS Accession No. ML060940095), a licensee 

that has been in Column 3 of the Action Matrix for 3 years will be discussed at the AARM and 

may be invited to meet with the Commission. The Commission will then evaluate the need for 

subsequent briefings by the licensee with senior agency management after the AARM. 

The 95001 and 95002 procedures state the following: 

A reasonable amount of time (generally within 30-60 days) should be allowed for 

the licensee staff to complete their evaluation; however, all corrective actions 

may not be fully completed upon commencement of this supplemental 

inspection. The inspection should not be scheduled until the licensee has 

completed its problem identification, evaluation, and corrective action plan. In 

the event that the licensee has not defined their corrective action plan within a 

reasonable time, regional management should prompt the licensee to provide the 

basis, including risk insights, for the delay. 
20 

Such flexibility in the procedure can allow significant delays in performance of supplemental 

inspections. Feedback from internal stakeholders indicated that untimely or inadequate 

licensee preparations can add unnecessary delay to reaching a regulatory decision. 

Additionally, a significant delay in conduct of a supplemental inspection could impact whether 

the licensee is subsequently placed in the appropriate column of the ROP Action Matrix 

because of delayed identification of performance deficiencies. Specific feedback provided 

included, but was not limited to, the following: 

• Why is it okay for it to take 2 plus years to complete a 95002 or 95003 inspection? 

– NRC should wait a reasonable amount of time and then inspect whether the 

licensee is ready or not. 

– The extended time for licensees to get ready cheats the Commission out of an 

opportunity to get engaged. 

– These time frames were never envisioned by the program. 

• Licensees fight White findings in PRA space which takes a lot of staff time when the 

supplemental inspection would only be 40 hours. 

The working group concluded that specific timeframes should be established for performance of 

supplemental inspection procedures 95001 and 95002, or allowance added to these procedures 

for management discretion to move forward with the supplemental inspection activities for cases 

in which a licensee is not making sufficient progress in preparing for these inspection activities. 

Issue: Completion of supplemental inspections can be significantly delayed by either untimely 

or inadequate licensee preparations. Such delays create inefficiencies in the oversight process 

and add unnecessary delay to regulatory decisions. 

Suggestion 4: Consider including additional guidance in Inspection Manual Chapter (IMC) 0305 

to enable the use of management discretion in determining whether to accelerate the timing of 

supplemental inspection activities particularly for cases in which it is determined that a licensee 

is not making reasonable progress in preparing for these inspection activities. 

6.1.3 Discussion of Licensee Performance at ROP Management Meetings 

In order to facilitate open, cross disciplinary discussions of reactor plant performance, the ROP 

assessment process includes several planned meetings, including Mid-Cycle Review Meetings, 

EOC Review Meetings, EOC Summary Meetings, and an annual AARM. Each of these 

meetings is described in detail in IMC 0305, IMC 0308, and MD 8.14. These meetings also 

provide opportunities for headquarters and regional office participants to share any significant 

insights into licensee performance over the course of the annual assessment period, provide an 

independent validation of the regional office’s assessment of licensee performance (from 

another office’s perspective), provide clarifying or ancillary remarks regarding ongoing or current 

issues under their cognizance, and make decisions regarding regional inspection resource 

allocation. 

During this assessment, the working group conducted interviews with senior management and 

inspection staff based in the regions and at NRC headquarters, attended the 2013 EOC 
21 

Summary Meeting, and attended the 2013 AARM. Feedback provided to the working group 

regarding the quality, level of discussion, and collaborative decisionmaking during the Mid-Cycle 

and EOC Review Meetings was very positive. Most of the feedback on, and group observations 

of, the 2013 EOC Summary Meeting and AARM were equally positive. In particular, feedback 

frequently indicated that the AARM was a significant enhancement from the original Senior 

Management Meeting process based on the perspectives of transparency, consistency, and 

predictability of senior management decisionmaking. However, some feedback from 

participants of these later two meetings indicated that it seemed as though meeting efficiency 

and a rather high threshold for plant performance discussions made the discussions conducted 

during these meetings seem too “mechanistic.” Additional similar feedback included the 

following: 

• At times, the scope of the AARM appears to preclude an open dialogue of lower level 

plant performance. Mid-Cycle and EOC Review Meeting discussions seem to include a 

number of problem site discussions. 

• There seems to be a large dichotomy in perspective available to senior leaders 

depending on which meeting they participate in. Why couldn’t “near miss” plants (those 

whose performance appears to be at risk of degrading to Column 3 or 4 of the NRC 

Action Matrix) also be discussed at the EOC Summary Meeting and AARM? 

• ROP decisionmaking continues to appear to be very mechanical and dependent on 

process criteria. There is not much open discussion on how well the process is working. 

Discussions have become too scripted particularly at the AARM. 

• Plant performance discussions often take on a different tone the higher you go. The 

NRC seldom has an open dialogue regarding whether or not it “did the right thing.”

• The discussion of individual plant performance issues during the 2013 EOC Summary 

Meeting, at times, resulted in senior management asking, “Does it make sense that the 

plant is in Column 1 or 2 of the NRC Action Matrix given the other issues at the site 

(i.e., numbers of allegations, [SCCIs], and unique site phenomena such as flooding)?” 

Such questions and plant performance conclusions are not always fully explored at this 

meeting. 

As reflected in the above feedback; discussed in IMC 0305, IMC 0308, and MD 8.14; and 

demonstrated during the conduct of Mid-Cycle and EOC Review Meetings; the thresholds 

established for discussing plant performance at these meetings provide ample opportunities for 

participants to discuss a broad range of performance issues. However, the working group noted 

that the bar rises very quickly during the EOC Summary Meeting and AARM. Ultimately, the 

AARM process, while still collegial, tends to limit plant performance discussions to those plants 

whose performance falls in Column 4 or 5 of the Action Matrix (noting that no plants have 

historically made their way into Column 5). While these discussions are often very good and 

provide ample opportunities for senior management to vet performance conclusions and 

planned actions, the plants discussed are only those that have been determined to cross a 

threshold. In borderline cases, discussion among senior agency officials may be beneficial to 

validate the performance assessment, application of ROP thresholds, and characterization of a 

plant’s performance within the Action Matrix.
22 

Issue: Plant performance meetings conducted within the Reactor Oversight Process continue to 

provide an efficient and consistent means of vetting important plant performance information 

with multiple levels of NRC management. Based on the working group review and feedback 

from stakeholders, the meetings and the decisionmaking that occurs as a result of these 

meetings would benefit from some additional flexibility regarding the thresholds established in 

Inspection Manual Chapter (IMC) 0305, IMC 0308, and Management Directive (MD) 8.14 for 

discussing plant performance issues. 

Suggestion 5: The criteria for discussion of licensee performance issues during the Agency 

Action Review Meeting and End-of-Cycle Summary Meeting should allow senior management 

the opportunity to discuss plants with performance issues considered to be at the threshold for 

additional regulatory action particularly those considered to be at risk of moving into Column 3 

or 4 of the NRC Action Matrix. 

6.1.4 Action Matrix Deviations 

IMC 0305 provides the definition of Action Matrix deviations as well as guidance on how to 

process deviations. According to IMC 0305, it might be necessary on rare occasions to deviate 

from the regulatory responses associated with each column of the NRC Action Matrix. Under 

such circumstances, regional senior management may use the deviation process to request 

approval from the EDO to increase or decrease the inspection activities, degree of regulatory 

outreach, and/or expected licensee actions outlined in the Action Matrix. The ROP assessment 

process provides for an extensive vetting of deviations and ensures that approved changes are 

communicated to affected licensees. 

During the 2012 EOC Summary Meeting, the working group observed a number of discussions 

among senior managers regarding the use the deviation process and whether the process was 

always the most efficient means for ensuring that adequate regulatory oversight was brought to 

bear for significant plant specific issues. This prompted the working group to perform a review 

of deviations, both active and previously closed, to gain an understanding of how the process 

has traditionally been used. 

The working group reviewed each of the 22 deviations that have been issued since the 

inception of the ROP and identified the following: 

• Five deviations (approximately 23 percent) have been issued documenting 

recommended reductions in the inspection activities associated with a particular column 

of the NRC Action Matrix. 

• Regardless of performance, eight deviations (approximately 36 percent) have been 

issued recommending that additional oversight be provided at plants with unique, 

ongoing, technical challenges. 

• Four deviations (approximately 18 percent) have been issued recommending that 

additional oversight be provided at plants with ongoing safety culture or SCCIs. 

• Five deviations (approximately 23 percent) have been issued recommending that 

additional oversight be provided at plants to allow appropriate follow-up for long standing 

plant performance issues. 
23 

Feedback provided during interviews with regional and headquarters senior management 

indicates that being able to use deviations from the Action Matrix to do more than simple 

accounting is a useful flexibility of the program. The working group acknowledges the value in a 

process that allows regional management to exercise discretion in order to ensure that 

adequate resources are brought to bear and to provide adequate oversight for significant 

plant-specific issues. The working group also believes that, in some cases, an alternative 

process could be established to provide a more efficient means for regional management to 

make resource decisions. 

Since the inclusion of the deviation process in IMC 0305, NRR has taken actions to improve the 

process including stipulating certain inspection follow-up activities that regional management 

can prescribe at plants with significant performance issues without seeking formal approval from 

the EDO. Additionally, examples of approved deviations can now be found on the NRC’s public 

“ROP Action Matrix Deviations” Web site. The working group believes that an opportunity exists 

to gain additional efficiency by expanding the set of activities stipulated in IMC 0305 for which 

regional management can make informed resource decisions without the added administrative 

burden of the deviation process. Doing so would also help support minimizing deviations from 

the Action Matrix, as originally mandated by the Commission in the SRM for SECY-00-0049, 

“Results of the Revised Reactor Oversight Process Pilot Program (Part 2),” dated May 17, 2000 

(ADAMS Accession No. ML003715823). As reflected in the historical review above, such 

activities might include providing additional oversight at plants with unique, ongoing technical 

challenges and at plants with ongoing safety culture or SCCIs. 

Issue: Reactor Oversight Process (ROP) Action Matrix deviations have been used as a 

mechanism to gain internal NRC alignment on resource and other issues not specifically 

addressed in the ROP. While a functional process, the administrative burden of processing an 

ROP deviation might not be necessary to gain internal alignment for management decisions. 

Suggestion 6: The list of activities that do not constitute Action Matrix deviations provided in 

Inspection Manual Chapter (IMC) 0305 could be expanded to allow more efficient management 

decisions on resources needed to address oversight at plants with unique, ongoing, technical 

challenges or with ongoing safety culture or substantive cross-cutting issues. 

6.2 ROP Inspection Process

6.2.1 Flexibility Within the ROP Inspection Program 

As discussed in Attachment 2, “Technical Basis for the Inspection Program,” to IMC 0308 

(ADAMS Accession No. ML062890421), the agency strove to develop an inspection process 

ensuring that the minimum level of inspection required for a plant is performed while also 

ensuring that inspectors are afforded a sufficient degree of flexibility in applying their inspection 

focus to areas that they feel need more or less inspection effort based on their overall 

knowledge of a specific plant. To that end, the baseline inspection program provides inspectors 

with a wide variety of individual inspection procedures that can be adjusted, within an 

established range of hours and samples, in such a way that inspection focus areas can be 

explored without compromising the overall effectiveness of the program. 

During interviews with NRC region-based management and inspection staff, the working group 

received feedback covering a spectrum of views regarding the degree of flexibility afforded by 

the ROP inspection program. While many inspectors and managers stated that they believed 
24 

that the ROP afforded an appropriate degree of flexibility, others maintained that it did not. 

Specific feedback indicating a lack of flexibility in the program included statements such as the 

following: 

• The ROP structure is inhibiting “follow your nose” activities. 

• ROP doesn’t allow for “collection of pebbles.” 

• Sample selection is not flexible enough. 

• Inspections tend to be hour driven rather than scope driven. 

• There seems to be an overreliance on ROP program “mechanics.” 

As discussed in IMC 0308, similar concerns with program flexibility were raised by regional 

management and inspectors during the early years of ROP implementation. To address these 

earlier concerns, program changes were made, in consultation with regional management and 

inspectors, that provided inspection procedures with a range of hours and samples to support 

the ability of inspectors to identify and explore areas of degrading licensee performance. 

Despite attempts to address concerns with ROP inspection flexibility, negative views such as 

those listed above continue to persist. 

Additionally, some inspectors stated that they viewed indirect inspection activities (i.e., activities 

that support inspections such as preparation, documentation, and gathering plant status 

information through the conduct of plant walkdowns and by observing various licensee work 

planning and plant status meetings) as inherently “administrative” in nature and that they felt 

that such activities, at times, detracted from an inspector’s focus on safety. Specifically, 

feedback received from inspectors included the following: 

• Resident Inspectors seem to be more focused on counting numbers of samples and 

inspection hours than identifying performance themes and “pulling the string” on issues. 

• New inspectors are overwhelmed. The focus on administrative issues detracts from 

safety. 

• Spending time on administrative matters causes inspectors to miss out on “walk-around” 

time. 

• Inspectors are often more focused on accounting for completion of the program than 

they are on working toward having an integrated picture of plant performance. 

Interestingly, many other inspectors stated that indirect inspection activities were necessary and 

that the impact on the ability to conduct direct inspection activities could be minimized through 

teamwork and the use of computer-based tools such as the Inspection Planning and Inspection 

Reporting aspects of the Reactor Program System. During its review of ROP basis documents, 

and through discussions with staff from NRR’s Division of Inspection and Regional Support 

(DIRS), the working group noted that there are several web sites, such as the Reactor 

Inspection and Assessment Network (http://nrr10.nrc.gov/rop-digital-city/index.html), that 

provide (or link to) training and best practices regarding the management of indirect inspection 

activities. 
25 

The working group observed that the performance of “direct inspection” activities (i.e., activities 

associated with individual NRC inspection procedures) associated with the Baseline Inspection 

Program is a critical element of the ROP, independently verifying that the plant is operated in a 

manner that supports public health and safety. An equally important element for the 

effectiveness of the ROP is the performance of “indirect inspection” activities (defined above). It 

is through inspection activities that the NRC demonstrates to key stakeholders, such as 

Congress and members of the public, that the minimum level of inspection required for a given 

plant (regardless of performance) is being performed to give the NRC sufficient information to 

determine whether plant performance is acceptable. The indirect inspection efforts help to 

ensure the overall effectiveness and uniformity of the implementation of the inspection program. 

A broad spectrum of views was shared by regional management and inspectors regarding ROP 

inspection flexibility and the impact of indirect inspection activities. While there were no 

indications that the majority of individuals felt constrained in their ability to identify safety issues, 

the fact that some individuals believe the program is overly restrictive led the working group to 

conclude that there is an opportunity to facilitate a better understanding among management 

and inspectors regarding the flexibility inherent in the ROP inspection program and the 

importance of indirect inspection activities. Inspector counterpart seminars, training, and 

mentoring provide opportunities to reinforce best practices for applying ROP program flexibility 

to assess areas of degrading licensee performance and to manage inspection activities in a way 

that supports, rather than detracts from, the performance of direct inspection activities. 

Issue: Some individuals implementing the Reactor Oversight Process (ROP), while not in the 

majority, do not have an appreciation for the flexibilities within the ROP that allow inspectors to 

focus additional effort in assessing areas of potentially degrading licensee performance. 

Suggestion 7: Periodic inspector counterpart seminars, training, and mentoring should be used 

as opportunities to ensure that inspectors and managers have a common understanding of the 

inherent flexibilities in the ROP inspection program. Best practices in using the ROP flexibilities 

should continue to be highlighted, and a similar approach should be used to communicate the 

importance of indirect inspection activities and how to effectively and efficiently manage these 

activities. 

6.3 The Significance Determination Process (SDP)

IMC 0308, Attachment 3, “Significance Determination Process Basis Document” (ADAMS 

Accession No. ML062890430), describes the SDP as a process that includes all associated 

provisions designed to meet ROP objectives, such as formal opportunities for licensee input 

(i.e., Regulatory Conferences), NRC management review for any significance characterization 

greater than Green (i.e., the Significance and Enforcement Review Panel (SERP)), and licensee 

appeal options. The SDP is implemented using various cornerstone specific SDP tools also 

described in IMC 0308. 

While conducting this assessment, the working group was briefed by staff and management 

from NRR’s DIRS regarding the efforts of an ongoing SDP Business Process Improvement 

(BPI) project. As stated in the SDP BPI charter, the intent of this project is to “... balance the 

goal of having reliable significance determination process outcomes against the need for 

efficient and timely regulatory decisionmaking.” To minimize the duplication of effort and 

maintain a more strategic approach to the working group’s efforts, the working group shared 
26 

much of the timeliness feedback and recommendations provided by stakeholders with the SDP 

BPI project lead and focused its efforts on the objectivity, scrutability, and resource attributes of 

the SDP. 

Based on feedback received from internal and external stakeholders, reviews of ROP and SDP 

implementation and basis documents, and discussions with NRR/DIRS management and staff 

as described above; the working group identified the following areas where it believes that 

opportunities exist to enhance the efficiency, effectiveness, and communication of agency 

decisionmaking:

• SERP members’ understanding of uncertainties associated with the NRC’s PRA tools, 

• documentation of the qualitative considerations applied during the final characterization 

of greater-than-Green inspection findings, and 

• resources expended by the NRC and industry to characterize inspection findings that fall 

on the Green/White threshold. 

Further discussion of the background, program requirements, and specific stakeholder feedback 

is included in the following sections. 

6.3.1 SERPs and Uncertainties Associated with Risk Assessment Tools 

IMC 0308 states, in part, that each SDP tool should attempt to provide a decision logic or a 

decision framework that remains constant across applicable findings. This enhances objectivity 

by minimizing the likelihood that SDP results will be influenced by different value judgments held 

by different individuals. IMC 0308 also states that probabilistic risk models, no matter how 

detailed, should not automatically be accepted without understanding their influential 

assumptions, limitations, and uncertainties. 

During interviews with the working group, internal and external stakeholders shared many 

positive experiences about the use and outcomes of the various components of the SDP 

(specifically, Regulatory Conferences, SERPs, and the licensee inspection finding appeal 

process). However, the working group also received feedback indicating that not all NRC 

participants on SERPs had a common understanding of the uncertainties associated with the 

NRC’s PRA tools. The feedback also indicated that, at times, SERP decisionmakers appeared 

to make final decisions solely on the outcomes of PRA tools (i.e., a quantitative assessment) 

rather than a balanced combination of both quantitative and qualitative factors. Examples of the 

feedback received include the following: 

• There appear to be many SERP members that do not have a basic understanding of 

PRA or its limitations. 

• The findings of SERPs often fall in the “gray area” between colors. The process should 

support the use of qualitative factors under such circumstances. 

• Several SERP members lack technical understanding about the nature of PRA, 

probabilities, and uncertainties. This impacts decisionmaking based on risk-analysis 

results, which is especially crucial when differing opinions on the risk-analysis results 

exist and the SERP must resolve these differences before making a final decision. 
27 

• The PRA process does not always account for the propagation of uncertainties. Should 

consider including sensitivity studies in the PRA process. 

• Uncertainty of data and results is largely ignored. Standardized Plant Analysis Risk 

(SPAR) models and Systems Analysis Programs for Hands-on Integrated Reliability 

Evaluations (SAPHIRE) software have the capability to quantify the uncertainty, and it 

should be factored into the decision. 

• Managers and staff have a tendency to be interested in hard thresholds of color 

assessments in lieu of using a balanced risk-informed approach. 

• Defense-in-Depth and PRA are not always factored into agency decisionmaking in a 

balanced manner. There is too much reliance on PRA. For example, flooding analysis 

is often fraught with uncertainties but treated as an “absolute” by decisionmakers when 

evaluating findings and violations. 

• SERP members do not have a minimum understanding of the role of PRA and place too 

much emphasis on the SDP outcomes. 

Developing and maintaining a common understanding among SERP members regarding the 

uncertainties associated with the NRC’s PRA tools and the use of qualitative and deterministic 

aspects of findings in reaching a decision is important for consistency and efficiency of the 

SERP decisionmaking. This common understanding would also help ensure that the process 

remains objective and “risk-informed.” As discussed in IMC 0308, the SDP is considered to be 

a risk-informed process so long as SDP results are sufficiently understood by those best 

positioned to examine the probabilistic and technical assumptions as well as by the 

management decisionmakers who ultimately make the decisions. If the SERP’s decisions are 

made without understanding the uncertainties in the PRA analysis, the panel’s conclusions 

could be considered, or perceived to be, risk-based decisions. Periodic training of SERP 

participants (i.e., briefings by agency PRA practitioners) could aid in building a common 

understanding among SERP members regarding uncertainties inherent in the NRC’s PRA tools 

and the role PRA outputs should play in the panel’s decisionmaking. 

Issue: Not all NRC participants on Significance and Enforcement Review Panels (SERPs) have 

a common understanding of uncertainties associated with the NRC’s probabilistic risk 

assessment (PRA) tools. Feedback indicated that, at times, SERP decisionmakers appeared to 

make final decisions solely on the outcomes of PRA tools (i.e., a quantitative assessment) 

rather than a consideration of both quantitative and qualitative factors. 

Suggestion 8: SERP members should be provided with periodic training or briefings regarding 

the uncertainties inherent in the agency’s PRA tool outputs and the use of PRA quantitative 

results in SERP decisionmaking. 

6.3.2 Documentation of SDP Decisionmaking 

As discussed in IMC 0308, the SDP was developed in a manner that provides a clear 

(i.e., scrutable) framework to facilitate communication of each significance determination and its 

basis among technically knowledgeable stakeholders (both internal and external). The objective 

of such communication is to achieve a common understanding, to the extent desired by any 
28 

interested stakeholder, of SDP decision bases. This allows for broad and independent 

verification of the staff’s objectivity and most directly enhances NRC public credibility. IMC 0308 

also discusses the need to clearly document quantitative insights used to inform decisionmaking 

(whether solely based quantitative insights or a combination of qualitative and quantitative 

insights) in a manner that allows stakeholders the means to independently assess the SDP 

result. Such documentation should include influential assumptions considered and the basis of 

these assumptions as well as the limitations and uncertainties of the risk model and how these 

were considered by the staff in arriving at a final result. 

The results of SERP meetings and SDPs are to be documented in accordance with IMC 0609, 

Attachment 1, “Significance and Enforcement Review Panel Process” (ADAMS Accession 

No. ML101400488); for example, in inspection reports, Notices of Violation, Preliminary/Final 

Significance Determination Letters to licensees, etc. During interviews conducted with internal 

and external stakeholders, the working group received feedback, largely from external 

stakeholders, indicating that the results of SERP meetings and SDPs are not always 

documented in a manner that supports scrutability (i.e., openness) and an understanding of 

bases for decisionmaking about inspection findings. Examples of feedback received included 

the following: 

• Public citizens’ groups and State agencies indicated that the SDP is too complex and 

inaccessible for public scrutiny. 

• Many stakeholders indicated that the SDP is very technical and is inconsistently 

documented. 

• It seems as though the final characterization of inspection findings was based on factors 

not included (plant-specific assumptions, political pressure, or management discretion) 

in the SERP documentation or in the inspection report. 

• The NRC needs a better way of communicating the results of SERPs and SDPs PRAs to 

the public. There is limited information provided in Choice Letters and the information 

provided is often very complex. 

• SERP results lack transparency. Consideration should be given to issuing SERP 

summaries and making them publicly available, as appropriate. 

Clearly, the SDP was developed with scrutability of decisionmaking in mind. IMC 0308 

discusses at length measures in place to ensure that the results of SERPs and PRAs are clearly 

documented and include the basis for results. However, the stakeholder feedback suggests 

that the results of SDP decisionmaking are not always communicated with all audiences in 

mind. The working group recognizes that it is often challenging to document all aspects of 

decisionmaking in a way that all stakeholders easily understand, particularly when inspection 

findings involve complex PRAs, challenging technical issues, and (at times) sensitive security 

information. However, the group also believes that scrutability of decisionmaking is an 

important fundamental attribute of the SDP and that steps should be taken to more effectively 

communicate SDP results to the broad audience of stakeholders. 
29 

Issue: A range of stakeholders and audiences continue to have difficulties understanding the 

NRC’s communication of significance determination process (SDP) proceedings and results. 

Suggestion 9: To the extent practicable, steps should be taken to better understand 

stakeholder concerns with the documentation of SDP results and to enhance the 

communication of SDP results to improve the transparency of the process for the broad 

audience of stakeholders. 

6.3.3 Impact of White Findings on Resources 

Supplemental inspection activities have varying degrees of regulatory impact which can be 

generally characterized by their associated resources. Each supplemental IP provides 

guidance/estimates of the resources required to conduct the direct inspection activities (not 

including indirect resources for preparation and documentation). These are as follows: 

• IP 95001: 40 hours for each White issue (max of 80 hours), 

• IP 95002: 200 hours, and 

• IP 95003: 3000 hours. 

Although the impact on a licensee from an IP 95001 supplemental inspection is fairly modest, 

the escalating “costs” associated with additional White (or greater) findings are non-trivial if 

those findings lead to the NRC conducting a 95002 or 95003 inspection. During feedback 

sessions with stakeholders, the working group heard a wide variety of perspectives regarding 

the impacts of White findings which can be decisive inputs to the assessment process and to 

the determination of regulatory follow-up. Internal stakeholders tended to offer a perspective 

that many licensees challenge the validity of all White findings regardless of the potential Action 

Matrix outcome. The anecdotes included cases in which licensees invested far more staff time 

and financial resources in protest of a White finding, and required far more NRC staff time to 

reach an SDP conclusion, than would have been required to conduct the associated 

supplemental inspection. In those cases, the lengthy decisionmaking process was not only 

costly in terms of NRC and licensee resources, but it also added unnecessary delays to the 

regulatory decisionmaking process. 

Feedback from external stakeholders presented an alternative perspective that, in some cases, 

suggested the NRC assessment of issues is biased to yield White inputs to the Action Matrix. 

This was voiced principally as a concern regarding entry into Column 3 of the Action Matrix but 

could conceivably influence entry into other Action Matrix columns. Specific feedback from 

licensees included the following: 

• It was anticipated that plants would have White findings as a matter of course. Now, the 

NRC treats White findings as though safety has been compromised at the plant. 

• A 95002 inspection for a White finding doesn’t seem to add value for cases in which the 

Senior Resident Inspector/Resident Inspector could do the follow-up. 

• Seventy-five Green findings at one facility are seemingly okay, but one White finding 

garners far more “heat” from the NRC and from other stakeholders. 
30 

• Too much time and effort goes into the cross-cutting area process with seemingly no 

“substantive” safety gain. NRC subjectivity and inconsistencies from region to region 

only make matters worse. 

• The NRC needs to reduce what licensees believe is unnecessary focus on White 

findings. 

While the external and internal stakeholders don’t always agree on whether the SDP yields 

“correct” assessments (i.e., Green versus White) of issues, both groups do agree that the 

determination process for issues that are near a threshold (particularly Green/White) is 

frequently lengthy and has a large resource impact. Accordingly, the working group suggests 

that a review of the criteria for transition to Column 3 of the NRC Action Matrix be conducted 

against the original ROP program goals to ensure that the significance of White inspection 

findings is not being over-emphasized and to ensure that agency resources are not being 

unnecessarily expended to deal with issues that, by definition, are of low to moderate safety 

significance. 

The working group also received a great deal of feedback suggesting that licensees frequently 

expend a great deal of resources to “push back” during Pre-decisional Enforcement 

Conferences regarding the significance of potential White inspection findings. Additionally, 

some SERP participants observed that NRC staff invests a substantial amount of time and effort 

preparing for this anticipated licensee pushback particularly if the licensee already has an 

existing White inspection finding or performance indicator associated with the same ROP 

cornerstone or strategic performance area. 

When the working group asked for more elaboration, some stakeholders suggested that the 

current threshold established in IMC 0305 for Column 3, “Degraded Cornerstone Column,” of 

the NRC Action Matrix might contribute to the extent of licensee pushback on potential White 

inspection findings. Specifically, concerns were shared that simply increasing the number of 

White inspection findings at a facility from one to two in any cornerstone (i.e., the IMC 0305 

Action Matrix threshold for entry into Column 3) results in a disproportionate level of additional 

regulatory oversight (i.e., an IP 95002 supplemental inspection, a licensee root cause analysis, 

and potentially an independent safety culture assessment or an inspection under IP 71152, 

“Problem Identification and Resolution” (ADAMS Accession No. ML13179A365)). One way to 

quantify this increase is to note that the supplemental inspection hours alone will increase from 

80 hours to 400 hours as a result of having two White findings in once cornerstone. Other 

specific feedback included the following: 

• Resources expended by the agency and industry supporting PRAs/SERPs are “huge.” 

Efforts can sometimes involve 20 to 40 people, large amounts of time spent debating 

SDP results, senior resident analysts (SRAs) reviewing licensee procedures and 

traveling to sites to walk down systems, licensees even developing mockups costing 

millions, etc. 

• We have never understood how multiple [two] White findings [can] become a Yellow 

finding. Safety does not seem to be best served using this approach. 

• The NRC needs to reduce what we [external stakeholders] believe is unnecessary focus 

on White findings. 
31 

• In 2007, two licensees appealed the final determination of two separate findings 

characterized as White. The Regional Administrator upheld the original decisions to 

maintain the characterization of the performance deficiencies as White; however, both 

licensees petitioned for a second appeal through the Office of the EDO even though no 

such process exists. 

• Licensee efforts to ensure that findings are characterized to have as low a safety 

significance as possible results in delays in NRC management assigning a color and 

determining placement within the NRC’s Action Matrix essentially delaying the increased 

oversight called for if the finding is determined to be greater than Green. 

• Because most inspection findings are Green, one White finding at a plant can place it in 

the “bottom quartile” of plant performance. Therefore, licensees try to avoid this 

placement and will expend a great deal effort and resources to ensure that the risk level 

for a finding is appropriate. 

IMC 0308 describes how plant performance bands, based on a combination of CDF and LERF, 

were used in developing the five columns of the Action Matrix. However, the manual chapter 

does not provide a specific basis for why a licensee with as few as two White inspection findings 

should be subjected to the same degree of regulatory oversight as a licensee with one Yellow 

inspection finding (finding of substantial safety significance). 

Based on the working group’s review of the ROP implementation relative to the Green/White 

threshold issues and stakeholder feedback, the working group believes there would be value in 

further exploring ways to address licensee and NRC staff concerns with White inspection 

findings. Specifically, the staff should consider whether the Action Matrix threshold for entering 

Column 3 (based on two White inspection findings) and the prescribed increase in regulatory 

oversight are commensurate with overall impact such findings have on the continued safe 

operation of the facility. 

Issue: Internal and external stakeholders agree that the resources expended to disposition a 

finding at the Green/White threshold can be excessive. Licensees are willing to expend a great 

deal of resources to challenge a White finding because, in part, the increase in regulatory 

oversight for two White findings in a cornerstone (transition to Column 3 of the NRC Action 

Matrix) is not viewed as proportionate to the risk associated with the White findings. 

Consequently, the NRC also expends significant resources to finalize the characterization of 

White findings. 

Recommendation 5: The NRC should review the criteria for transition to Column 3 of the NRC 

Action Matrix against the original Reactor Oversight Process program goals to ensure that the 

significance of White inspection findings is not being overemphasized and to ensure that agency 

resources used to process White inspection findings are commensurate with findings that, by 

definition, are of low to moderate safety significance. 

6.4 Use of Cross-Cutting Issues and Safety Culture in the ROP

6.4.1 Background and Discussion 

The accidents at Three Mile Island in 1979 and at the Chernobyl nuclear plant in 1986 brought 

attention to the importance of sound organizational practices and on the impact that 
32 

weaknesses in safety culture can have on safety performance. During early ROP development, 

the use of cross-cutting areas related to human performance, problem identification and 

resolution, and SCWE was established. Following the Davis-Besse degraded reactor head 

event in 2002, the NRC Davis-Besse Lessons Learned Task Force identified root causes that 

would be characterized today as a weak safety culture (ADAMS Accession No. ML022760172). 

In 2003, the Commission directed the staff to incorporate enhanced SCWE processes in the 

ROP. The Commission also directed the staff to “... monitor [safety culture] developments 

abroad so as to ensure that the Commission remains informed about these efforts and their 

effectiveness. In particular, because subjectivity is a principal objection to the direct regulation 

of safety culture, the staff should also monitor efforts to develop objective measures that serve 

as indicators of possible problems with safety culture” (“Policy Options and Recommendations 

for Revising the NRC’s Process for Handling Discrimination Issues,” SRM-SECY-02-0166, 

dated March 26, 2003, ADAMS Accession No. ML030850783). Following a series of 

interactions with the Commission, the staff revised the ROP and issued the approved changes 

on July 1, 2006. On July 31, 2006, the staff issued NRC Regulatory Issue Summary 

(RIS) 2006-13, “Information on the Changes Made to the Reactor Oversight Process to More 

Fully Address Safety Culture” (ADAMS Accession No. ML061880341). 

6.4.1.1 Are Cross-Cutting Issues and Safety Culture the Same Thing? 

The working group found that within the NRC the terms “cross-cutting issues” and “safety 

culture” are frequently used interchangeably. 

Within NRC headquarters, feedback to the working group indicates there is a difference of 

opinion on whether the use of cross-cutting issues and safety culture comprise co-dependent or 

independent activities. Staff safety culture experts interviewed by the working group believe 

that safety culture envelops cross-cutting issues. Staff ROP practitioners interviewed by the 

working group believe that safety culture traits and attributes should be addressed separately 

from the cross-cutting areas, components, and aspects. 

This ideological difference is borne out in the NRC’s descriptive literature and IMCs. For 

example, the “Reactor Oversight Process” brochure (NUREG/BR-508, March 2013, ADAMS 

Accession No. ML13064A291) doesn’t include any discussion of safety culture. IMC 0310, 

“Components Within the Cross-Cutting Areas” (ADAMS Accession No. ML091480473), 

describes Other Safety Culture Components to be considered only in the supplemental 

inspection program. IMC 0310 also describes cross-cutting components as “the components of 

safety culture directly related to one of the cross-cutting areas.” 

The NRC Safety Culture Policy Statement brochure (SCPS, NUREG/BR-0500, Revision 1, 

December 2012, ADAMS Accession No. ML12355A122) makes no mention of the ROP or 

cross-cutting areas, but the Federal Register Notice that announced publication of the final 

SCPS (“Issuance of Final Safety Culture Policy Statement,” 76 FR 34773, June 14, 2011) 

mentioned that the policy was based on “... a variety of sources including the 13 safety culture 

components used in the ROP.” IMC 0310 does list a total of 13 components, but 9 of the 13 are 

described as “cross-cutting components” while the remaining 4 are described as “other safety 

culture components.” 
33 

6.4.1.2 Correlation Between Cross-Cutting Issues, Safety Culture, and Licensee Safety 

Performance 

NRC subject matter experts have assessed general industry studies of safety culture and 

applied their findings within the nuclear industry context where applicable. In a technical paper 

published in 2012 (“Independent Evaluation of INPO’s Nuclear Safety Culture Survey and 

Construct Validation Study,” S. Morrow and V. Barnes, ADAMS Accession No. ML12172A093), 

the NRC performed an analysis on the results of the Institute of Nuclear Power Operations’ 

(INPO’s) 2010 safety culture survey of domestic nuclear power plants. The primary purposes of 

the paper “... were to investigate the factors that comprise the concept of safety culture in the 

nuclear power industry, assess the extent to which they match the traits identified in the NRC’s 

SCPS, and evaluate the relationships between the safety culture factors identified from the 

survey and other measures of organizational and safety performance.” 

The analysis results indicated that, in the aggregate, there were statistically significant 

(i.e., non-random) correlations between the survey response data and the NRC’s plant safety 

performance data for 2010. Examples from the analysis include correlations between: 

management responsibility for safety and the number of inspection findings with cross-cutting 

aspects at a plant per year (with a correlation value of -0.44), a questioning attitude and the 

number of allegations reported to the NRC from plant personnel per year (-0.41), supervisor 

responsibility for safety and the rate of human performance errors per year (-0.40), and safety 

communications and the number of unplanned scrams at a plant per year (-0.35) (Regulatory 

Information Conference 2013 Presentation TH36, “Domestic and International Cooperation to 

Advance Nuclear Safety Culture Research,” S. Morrow (NRC) and G.K. Koves (INPO)). Such 

negative correlation values indicate an inverse relationship between the factors (i.e., better 

safety culture means fewer human performance errors). 

The NRC’s independent evaluation confirmed “... that the INPO safety culture survey is 

multidimensional, consists of factors similar to the traits in the NRC’s SCPS, and demonstrates 

statistically significant relationships with some, but not all, measures of safety performance in 

the expected directions.” The report results indicate that there were correlations between 

aspects of safety culture and plant safety performance, but further study is needed to determine 

whether causal relationships between safety culture and safety performance exist. 

These analysis results conceptually demonstrate the potential value of including safety culture 

in the ROP. However, based on stakeholder feedback, the practical application of safety culture 

as an indicator of licensee safety performance in an operational environment is problematic as 

currently implemented. 

6.4.1.3 The Current Use of Cross-Cutting Issues and Safety Culture in the ROP 

According to stakeholder feedback received by the working group, the current implementation of 

cross-cutting issues and safety culture in the ROP is viewed as subjective and subject to 

manipulation. Stakeholders interviewed during the independent assessment described 

examples of tagging inspection findings with various cross-cutting aspects either to reach or to 

avoid reaching identification of an SCCI. Regional differences were described by stakeholders 

in program implementation of cross-cutting issues and safety culture. Depending on the 

stakeholder’s perspective, this practice could be characterized as programmatic flexibility or as 

inconsistent implementation. 
34 

The working group found that the graded approach for escalated oversight of licensee safety 

culture (as referenced in IMC 0310) is well described in IMC 0305, IP 95001 (Action Matrix 

Column 2), IP 95002 (Action Matrix Column 3), IP 95003 (Action Matrix Column 4), and 

IP 40100 (“Independent Safety Culture Assessment Followup,” ADAMS Accession 

No. ML080040273). However, the NRC’s bases (reasons) used to develop the definition of a 

cross-cutting “theme” as described in IMC 0305 are not provided in IMC 0308. Industry 

feedback questioned the number of inspection findings tagged with the same cross-cutting 

aspect needed to establish a cross-cutting theme and, therefore, to be considered for 

classification as an SCCI. 

SCCI closure criteria are left to be developed by the regions for inclusion in licensee 

assessment letters. Significantly, absent other operational performance findings, there is no 

apparent regulatory action required by or available to the NRC should the licensee fail to 

remedy a long standing SCCI. 

6.4.1.4 How Does the NRC Safety Culture Policy Statement Affect the ROP? 

The SRM for SECY-11-0005, “Proposed Final Safety Culture Policy Statement” (Accession 

No. ML110660547), states, “Since the policy statement is not a regulation or requirement, staff 

activities beyond communication and education should not be pursued without further specific 

Commission approval.” The Commission also directed, “Before staff implements new initiatives 

based on the policy statement, it should seek Commission policy review of its plan including 

short-term and long-term goals for implementation and the vision for each program office’s 

oversight.” 

The staff provided the proposed SCPS implementation plans for the Office of Federal and State 

Materials and Environmental Management Programs, NRR, the Office of New Reactors, and the 

Office of Nuclear Material Safety and Safeguards, with supporting roles for the Office of 

Enforcement, the Office of Nuclear Regulatory Research, and the Office of Nuclear Security and 

Incident Response, to the Commission for approval in SECY-12-0008, “Implementation Plan for 

the Safety Culture Policy Statement” (ADAMS Accession No. ML11334A073 for the entire 

package). The plans presented included communication and education initiatives as directed in 

the Commission’s staff guidance. But additionally, in the SECY “Background” section, the EDO 

noted, “At present, the staff has not engaged in any new actions that go beyond outreach and 

education or are not under previous Commission direction” (emphasis added). This language 

appears to have been included by the staff to bridge the current uses of safety culture/ 

cross-cutting issues in the ROP that can be interpreted as going beyond the Commission’s 

approved scope of communication and education activities in the SCPS. 

Additionally, the SCPS explicitly states, “... the NRC does not monitor or trend the traits in the 

Policy Statement.” Further, the SCPS notes, “... these traits are not necessarily inspectable 

and were not developed for that purpose.” However, one of the traits listed in the SCPS is 

Problem Identification and Resolution (also one of the three cross-cutting areas currently used 

in the ROP). 

In the SRM for SECY-12-0008 (ADAMS Accession No. ML120620120), the Commission 

approved the proposed implementation plan submitted in SECY-12-0008, reaffirming the 

direction, “Staff activities beyond communication and education should not be pursued without 

further specific Commission approval.” The Commission made no comment regarding the prior 

usage of safety culture concepts in the existing ROP and subsequently approved safety culture 
35 

policy documents have not addressed this apparent difference between the general SCPS 

direction and existing ROP implementation. 

6.4.1.5 Future Changes to the ROP for Incorporation of the Nuclear Safety Culture Common 

Language (NSCCL) 

The working group noted that the NRC has recently reached consensus with industry on the 

NSCCL. It will be up to the NRC ROP practitioners to meld this new safety culture “dictionary” 

with the existing ROP guidance documents, training, and inspection procedures. The working 

group independently cross-mapped existing cross-cutting areas, components, and aspects to 

the traits and attributes of the NSCCL. Based on its review, the working group believes that the 

newly developed common language traits and attributes do envelop the cross-cutting areas, 

components, and aspects currently used. 

The development of the NSCCL is intended to provide a common base of understanding of 

safety culture terms and definitions for all stakeholders. Regardless of the particular safety 

culture traits and definitions established in the common language, the working group was not 

able to identify any plans to address the downstream process activities within the ROP that 

would change how cross-cutting aspects are handled. Therefore, the working group did not find 

a reason to expect the broader treatment of cross-cutting issues to improve solely because of 

the incorporation of the new common language. 

During its assessment, the working group was informed of preliminary plans for implementation 

of the common language that would maintain the three existing cross-cutting areas (Human 

Performance, Problem Identification and Resolution, and SCWE) and parse the new common 

language traits and aspects under these areas. There are a total of ten common language traits 

and forty common language attributes included in the common language. The working group 

believes that simply incorporating the new traits and attributes into the existing cross-cutting 

area taxonomy is likely to cause additional confusion for ROP practitioners and stakeholders 

regarding the intended application of cross cutting and safety culture concepts. 

During the fact checking phase in development of this report, NRR/DIRS provided the working 

group with a draft revision of IMC-0310 incorporating the NSCCL. The working group noted that 

the existing cross-cutting taxonomy has been revised to eliminate the classification level of 

“cross-cutting components.” However, the draft language persists in using the “cross-cutting” 

terminology rather than “safety culture” terminology, continuing an unclear distinction of usage. 

6.4.2 Stakeholder Feedback 

Most stakeholders agree that including cross-cutting issues and safety culture in the ROP as a 

program element is important. However, the working group found a diverse set of opinions on 

the best way to incorporate these program elements in the ROP and the expected outcomes. 

During focus group and individual interviews, the working group received a significant amount of 

feedback regarding the use of cross-cutting issues and safety culture. Some representative 

comments included the following: 

• For human performance and cross-cutting issues, as we define them, the measures are 

imperfect but workable. Better (less subjective) than the term “safety culture.” There is 

some direct connection between SCCI and safety performance even if the linkage is not 

well identified. 
36 

• Use of cross-cutting as a surrogate for safety culture – What is the benefit to safety? 

The basis for closeout of these issues is not always clear even though the closure 

criteria are supposed to be set at the beginning. 

• The increased attention to safety culture is appropriate and strongly supported by the 

industry. 

• Cross-cutting issues rolling up into substantial cross-cutting issue. Good first effort, but 

both sides seem to agree that it needs refinement. 

• Continue work on honing the safety culture process. Consider “common language” work 

with INPO. 

• The cross-cutting issue matrix seems to be in need of work. I’m not sure it’s working as 

well as it should. Also, with growing regulation, the NRC gets a greater number of 

findings so consideration should be given to increasing the number of findings in a single 

cross-cutting area before a SCCI exists. 

• The NRC should be revisiting cross-cutting issues now that we have the safety culture 

process. The ROP is set up to be objective and the cross-cutting areas are subjective. 

• Cross-cutting aspects do not always seem to “cut across” multiple processes. Many 

times these aspects are isolated to a particular process. 

• There is(are) no clear action(s) established to manage long standing SCCIs. 

• Cross-cutting aspects are often subjective and appear to have regional office bias 

regarding when to assign a cross-cutting aspect to a finding, which bin it falls under, and 

what actions are appropriate as a result. This can create the perception that there are 

varying “shades of Green.” 

• Too much time and effort goes into cross-cutting area process with seemingly no 

“substantive” safety gain. The NRC subjectivity and inconsistencies from region to 

region only make matters worse. 

• Has the NRC ever measured the success of the cross-cutting area process? 

• The cross-cutting area process is almost always a burden with little perceived benefit. 

• SCCIs have become so processed – What are we trying to accomplish? We have some 

that have lingered for years and become almost meaningless. They do have some 

value, helping to focus on certain areas, but what do you do with the ultimate issue? It 

kind of hangs. 

The working group’s assessment of the feedback received during annual ROP self assessments 

and during interviews conducted by the working group is that the current academic state of the 

art and practical application of safety culture in the nuclear industry is still evolving. 
37 

6.4.3 Assessing the Value of Using Cross-Cutting Issues and Safety Culture 

The inclusion of cross-cutting issues and safety culture considerations in the ROP was initially 

directed by the Commission in 2003. The staff has consulted with the Commission on its efforts 

to incorporate and refine safety culture considerations in the ROP since that time. While there 

is a general presumption regarding the benefits of incorporating cross-cutting issues and safety 

culture in the ROP, the working group did not find evidence of a rigorous assessment to 

systematically measure or quantify the benefits of incorporating these concepts in the oversight 

program. Further complicating this issue is the absence of explicit goals and outcomes 

expected to result from the incorporation of these concepts in the ROP. The working group also 

determined through interviews and document research that the NRC resources expended to 

obtain the expected benefits have not been tracked over time. The working group believes that 

to fully assess the value of incorporating cross-cutting issues or safety culture in the ROP, 

additional rigor must be applied to define expected outcomes and assess whether the resources 

invested in this aspect of the ROP are being used most effectively. 

The NRC faces a significant strategic challenge in determining the best uses of cross-cutting 

issues and safety culture. Additional study is needed to identify definitive causal relationships 

between cross-cutting issues, safety culture, and sustained licensee safety performance. In 

parallel, the NRC needs to decide how best to apply the information to the regulatory 

environment. For example, how should the agency’s role as a regulator govern its actions if it 

determines that specific safety culture measures correlate directly and strongly with specific 

plant safety measures? Should the agency act to preemptively intervene in some way to drive 

licensee safety culture performance improvements? Or should the agency rely on licensee 

improvement efforts knowing that a potential safety performance issue could arise if the 

improvement efforts were not effective or timely? Should the safety culture framework remain 

as part of the ROP, or should it be separate? 

Issue: A rigorous comparison of the benefits of considering cross-cutting issues and safety 

culture in the Reactor Oversight Process (ROP) on licensee safety performance and an 

assessment of the NRC resources expended to achieve those benefits have not been 

performed. Additionally, the working group found differences between various NRC policy, 

program, and communication documents when describing the purposes and uses of crosscutting issues and safety culture in the ROP. 

Recommendation 6: The NRC should perform a comprehensive analysis to determine whether 

the use of cross-cutting issues and safety culture, as currently incorporated in the ROP, 

provides regulatory value in terms of licensee safety performance for the resources expended. 

To support that determination, NRC staff should clarify and document the goals, purposes, 

uses, and desired outcomes associated with the inclusion of cross-cutting issues and safety 

culture in the ROP. If program changes are needed, the staff should determine whether 

Commission approval is required for implementation. 

6.4.4 Effective Use of Substantive Cross-Cutting Issues (SCCIs) 

In the Staff Requirements Memo (SRM) for SECY-12-0081, the Commission directed the staff to 

pursue an independent review of the program’s objectives and implementation including “... the 

effectiveness of NRC’s assessment of [SCCIs].” 
38 

The working group reviewed existing ROP documentation and discussed with NRC 

stakeholders whether a similar review had been conducted in the past. The group determined 

that an internal review of SCCI effectiveness had not been conducted previously, but a current 

review has been commissioned within NRR’s DIRS. 

The working group suggests that effectiveness could be measured in several ways using 

existing program data. Potential measures of effectiveness might include the following: 

• the number of sites/plants that have had SCCIs identified, corrected the SCCIs, and 

subsequently had additional SCCIs; 

• the number of assessment cycles required for a site/plant to clear an SCCI; and 

• the degree of alignment between SCCIs identified by the NRC and subsequent findings 

of site’s independent or third party safety culture assessments (if conducted). 

The working group notes that the term “effectiveness” is subject to interpretation and that any 

discussion of the term should include a definition of the term’s use. For example, 

“effectiveness” could be one input used for determining the value of using SCCIs in the ROP as 

discussed above. 

Issue: There are ideological differences between NRC’s safety culture subject matter experts 

and Reactor Oversight Process (ROP) practitioners on how best to apply cross-cutting issues 

and safety culture in the ROP. Based on feedback received by the working group, there is 

stakeholder disagreement on the purposes, application, and disposition of substantive 

cross-cutting issues.

Suggestion 10: Consider replacing the use of substantive cross-cutting issues and the current 

cross-cutting aspects, components, and areas with a process that uses the recently developed 

Nuclear Safety Culture Common Language (NSCCL) traits and attributes in a graded regulatory 

response aligned with the overall ROP philosophy. Further, a more graded regulatory response 

could be established to allow licensees to address safety culture issues when performance is in 

the Licensee Response column and would escalate the NRC’s level of engagement as the 

significance of findings increase. 

The following figure reflects the suggested approach: 
39 

6.5 Generic Treatment of Common Plant Performance Issues

As discussed in Section 4.3 of this report, “Assessment of Licensee Corrective Actions in 

Response to Generic Issues,” generic issues are occasionally identified which require regulatory 

action by the NRC and licensees. However, it is often not clear whether an issue is generic 

when it is first identified. The ROP is focused on assessing performance of individual facilities, 

but findings identified at one facility might be present at others. This has borne out in findings at 

a facility resulting in subsequent, similar, duplicate findings at other facilities. 

Identification of performance issues applicable to multiple licensees through inspections or 

through the Task Interface Agreement (TIA) process is an expected outcome of the oversight 

process. However, some issues with generic applicability are not recognized as such and are, 

therefore, not placed into NRC processes that would facilitate consistent treatment by NRC or 

licensees. The TIA process is the means by which the regions can ask NRR technical and 

regulatory questions which might ultimately result in the region identifying a finding relative to a 

single licensee. The process for TIAs is detailed in NRR Office Instruction COM-106, “Control 

of Task Interface Agreements” (ADAMS Accession No. ML073440014). 

During the working group’s review, a number of stakeholders raised concerns about how 

findings identified at one plant, or regulatory concerns identified in the resolution of a plantspecific TIA, can often result in findings being identified at other licensees. The stakeholders 

indicated that such issues are often not widely communicated to the industry using transparent 

and retrievable communications means (even though there are sometimes verbally 

communicated during NRC/industry interface meetings) and could be better addressed through 

established regulatory tools for generic issues. 

IMC 0970, “Potentially Generic Items Identified by Regional Offices,” (ADAMS Accession 

No. ML101600326) outlined the responsibility of regional offices to identify potentially generic 

safety questions to NRR for evaluation and determination of whether generic action is 

warranted. The working group noted that IMC 0970 was issued on December 11, 2000, and 

does not reflect the current NRR operating experience (OpE) program and appears to be 

replaced by other guidance documents (but is still available on the NRC website). The 

responsibility in IMC 0970 described above appears to remain appropriate but does not appear 

to be captured in other documents. Providing generic treatment of common issues would 

improve the NRC’s regulatory effectiveness and reduce the burden associated with identification 

and resolution of duplicate findings at multiple facilities. Depending on the safety significance of 

an issue, a mechanism that would allow industry to enter issues directly into licensee corrective 

action processes for resolution might be the most effective and efficient means for resolving 

generic issues. 

IMC 2523, “NRC Application of the Reactor Operating Experience Program in NRC Oversight 

Processes” (ADAMS Accession No. ML12332A099), describes the interface between the 

reactor OpE program (as described in MD 8.7, “Reactor Operating Experience Program,” 

ADAMS Accession No. ML122750292), and the ROP. IMC 2523 discusses the attributes of the 

OpE program including internal communication avenues available for inspectors to share and 

become aware of OpE issues and the outcomes of the OpE evaluation process which can 

impact the inspection program; for example, through the issuance of a generic communication. 

Of particular note is the OpE Smart Sample program. The Smart Sample program is designed 

to provide NRC inspection staff with a detailed synopsis of select issues that have safety 

significance and can be reviewed as part of baseline inspection activities. The Smart Sample 

program supports the ROP by informing inspection scope and sample selection. The 
40 

information and trends identified from Smart Sample inspections might provide further indication 

that a specific issue warrants additional agency action, such as a technical instruction or generic 

communication. 

The Smart Sample reports are made publicly available. This program appears to be an 

appropriate mechanism for informing baseline inspection scope and sample selection, but it is 

not clear how well industry is informed of new Smart Samples when they are issued. There also 

does not appear to be a “latency period” afforded to industry when new issues are identified that 

would allow licensees to self initiate corrective actions before the NRC inspects the Smart 

Sample. Adding features to the OpE program that make new information on potentially generic 

issues more readily available to industry and that enable licensees to use their corrective action 

programs to address these issues could improve the effectiveness of the OpE program and 

reduce industry concerns regarding duplicate findings. 

Issue: The identification of performance issues applicable to multiple licensees through 

inspections or through the Task Interface Agreement process is an anticipated outcome of the 

NRC’s oversight process; however, some issues could be more efficiently and effectively 

addressed by NRC and industry by using other NRC processes to better ensure a consistent 

generic response. 

Recommendation 7: Clarify Reactor Oversight Process program expectations for when 

performance issues that are common to multiple facilities should be considered for resolution 

through a generic issues process in order to improve the use of NRC inspection resources and 

ensure a thorough and consistent industry response. 

6.6 Scope of ROP Self Assessments

IMC 0307, “Reactor Oversight Process Self Assessment Program” (ADAMS Accession 

No. ML090300565), describes the ROP self assessment program. The self assessments are 

intended to objectively evaluate the effectiveness of the ROP in achieving the goals of being 

objective, risk informed, understandable, and predictable as well as the applicable agency 

performance goals listed in the NRC’s Strategic Plan. The reviews are to inform the 

Commission, NRC senior management, and the public of the results of the program including 

any conclusions and resulting improvement actions. 

IMC 0307 identifies several types of periodic ROP self assessments: 

• Periodic Reports: Metric data is collected and analyzed on a quarterly basis and 

periodic reports may be issued as deemed necessary to address particular issues of 

concern resulting from the quarterly data and analysis. 

• Annual Performance Metric Report and Commission Paper: NRR develops an annual 

ROP performance metric report after the conclusion of each calendar year and presents 

the results in a Commission paper. The overall summary report must discuss any 

metrics that did not meet their pre-established criteria, the staff’s analysis of the reasons 

for not meeting the criteria, and any actions taken or planned to change the program or 

improve its implementation. The Commission paper is written to support the AARM and 

the Commission briefing on AARM results that follows the review meeting. NRR also 

prepares a consolidated response to the external survey used in the self assessment 

process. 
41 

• Customized Audits of the ROP: After each annual ROP cycle, NRR may use the 

insights gained from the self assessment to develop topics for audits that delve more 

deeply into those aspects of the ROP that show indications of weaknesses or areas for 

future development. 

At the initiation of the ROP, the staff and Commission acknowledged the need for continuous 

self assessment of program effectiveness after the program’s implementation and developed 

IMC 0307. Stakeholder feedback and group review of self assessments show that the current 

self assessment process appears to provide a robust mechanism for identifying operational 

enhancements to the ROP. However, stakeholders provide a variety of feedback on potential 

improvements including issues such as the following: 

• the need for the agency to assess its own performance after poorly performing facilities 

are identified and to perform effectiveness reviews after supplemental and reactive 

inspections to determine whether the baseline is adequate; 

• the need for occasional benchmarking of outcomes from the ROP against expert 

elicitation to see whether the program needs revision as well as evaluation of the results 

of decisionmaking tools versus the final outcomes to identify process enhancements; 

• better assessment of the cost effectiveness of the ROP in general and of the self 

assessment program in particular; and 

• increased focus on independent strategic evaluation of achievement of program goals. 

This feedback was separate from the Commission direction to perform this independent 

assessment but was similar in nature. 

The working group felt that a common element of this feedback was a need for a more strategic 

(versus operational) periodic look at the program similar to this independent assessment. To 

achieve this, the group believes that the current self assessment process could be reconfigured 

to be more effective with no increase in resources expended by developing the following: 

• a streamlined version of the current process to continue to identify operational 

enhancements with reduced expenditure of resources and 

• a periodic broader assessment of accomplishment of the ROP strategic goals 

incorporating independent staff similar to those who participated in this independent 

assessment and also perhaps incorporating a panel of more senior experts to assess 

the program and stakeholder feedback. 

As part of the independent assessment, the working group examined annual ROP self 

assessment reports from 2001 through 2012 with specific regard to cross-cutting issues and 

safety culture. There were several observations identified during the reviews. 

• First, the multiple revisions to the survey format make it difficult to reliably trend 

(compare) the perception of the safety culture enhancements over time. The criterion to 

be met by the metric “Perceived Effectiveness of Safety Culture Enhancements to ROP” 

(AS-11 in 2006, AS-8 in 2007 through 2012, and O-10 in 2013) is “expect stable or 

increasingly positive perception over time.” The internal and external survey formats 
42 

have evolved since 2006 with a significant expansion of the number of internal survey 

questions (from 3 to 11) on the topic in 2010. 

• Second, the limited survey population and the binary response format (agree/disagree) 

for individual survey questions mean that the results of the tabulated surveys are a 

coarse indicator. Minor positive or negative changes in survey percentages of 

agreement over time are likely not discernible using this format. 

• Third, the survey questions and formats used for internal and external stakeholder 

surveys are not the same, making it difficult to compare internal and external responses 

directly. 

• Finally, there appears to be a consistent and persistent difference in stakeholder 

perception between internal and external stakeholders with regard to the value of using 

cross-cutting issues and safety culture in the ROP. 

These observations support the need to strengthen the ROP self assessment process (from a 

data collection and analytical perspective) and to periodically review the program (from a 

strategic perspective). 

Issue: The Reactor Oversight Process (ROP) self assessment process focuses on identifying 

and addressing operational issues impacting program implementation and does not explicitly 

evaluate the effectiveness of the ROP in achieving expected outcomes. 

Recommendation 8: Consider revising the ROP self assessment process to better solicit and 

assess both tactical and strategic feedback. Reexamine how internal and external feedback is 

collected, analyzed, and used to improve the oversight approaches to and the implementation of 

the ROP. 

 
43 

7.0 TOPICS OF INTEREST 

The independent assessment of the ROP practices and approaches was intended to compare, 

at a high level, the program’s original objectives and with its current implementation. In 

SRM-SECY-12-0081, the Commission also highlighted the following three implementation 

topics of particular interest: the relative roles of headquarters and regional staff, interactions 

with industry over performance indicator assessments, and the effectiveness of NRC’s 

assessment of SCCIs. Lastly, the Commission noted that the independent assessment would 

be helpful as the agency plans for the operational review of new nuclear power plants based on 

Generation III+ reactor technology. 

 

In conducting its review, the working group included consideration of these specific Commission 

topics of interest as context for the potential focus areas of this report. The working group 

looked for indicators related to these topics when evaluating the historical documents assessing 

the effectiveness of the ROP, during interviews with current ROP practitioners and 

stakeholders, and during observation of ROP-related process interactions between NRC staff 

and industry. The working group’s assessment of ROP objectives and implementation, in 

general, was used as the basis for the team’s view on potential applicability of the ROP to new 

nuclear power plants. 

 

The topic of interest, “effectiveness of NRC’s assessment of [SCCIs],” resonated with internal 

and stakeholders and is extensively discussed in Section 6.4 of this report. The remaining two 

topics of interest, “the relative role of headquarters and regional staff,” and “interactions with 

industry over performance indicator assessments,” did not resonate the same way with the 

stakeholders or internal focus groups areas of significant concern. Stakeholders interviewed did 

not express, and the team did not identify, concerns regarding roles and responsibilities of the 

headquarters or regional staff related to the ROP. Staff interactions with industry related to 

performance indicators were observed as part of the working group’s assessment which is 

generally discussed in Section 5.2. Stakeholder feedback and the team’s observations revealed 

no significant issues related to NRC and industry interactions with respect to implementation of 

the ROP PI program. 

The full applicability of the ROP relative to new reactors is a topic that goes beyond the scope of 

this report. However, since the Commission noted the potential relevance of the independent 

assessment to future planning for the operational review of new nuclear power plants, the 

working group considered whether any information identified during the course of its review 

might impact the applicability of the ROP framework to new reactors. The working group’s view 

of this issue is documented in Section 7.1 below. 

7.1 ROP Applicability and Implementation Strategy for New Reactors

The working group reviewed the current ROP structure to determine how well the existing 

framework would adapt for use with new large light water reactors and with new light water 

small modular reactors. While implementation of the ROP framework for the next generation of 

nuclear power plants would need to account for substantial differences in the baseline risk of 

new reactor designs, the working group concluded that the fundamental ROP framework should 

continue to serve the agency well. 
44 

The working group noted that there are current efforts by staff to adapt the ROP to the next 

generations of power reactors. These efforts include the following: 

• adaptation of the ROP to a construction environment for new reactors in a program 

known as the Construction ROP (cROP) and 

• preparation of staff recommendations for risk informing the ROP for new reactors. 

The most recent annual self assessment for the cROP process is found in SECY-13-0042, 

“Construction Reactor Oversight Process Self Assessment for Calendar Year 2012,” dated 

April 15, 2013 (ADAMS Accession No. ML13045A493 for the entire package). The staff 

determined in the self assessment report that the cROP met the agency’s organizational 

excellence objectives (i.e., openness and effectiveness) from the NRC’s Strategic Plan for 

Fiscal Years 2008 through 2013 and the strategic goals of ensuring safety and security through 

objective, risk-informed, understandable, and predictable oversight. 

Staff recommendations for risk informing the ROP for new reactors were provided to the 

Advisory Committee on Reactor Safeguards (ACRS) for review and comment in June 2013 and 

the ACRS provided feedback to the staff in September 2013. The staff responded to the ACRS 

comments in December 2013 (ADAMS Accession No. ML13283A029 for the entire package). 

The staff issued its final recommendations on this subject to the Commission in December 2013 

as SECY-13-0137 (ADAMS Accession No. ML13263A351 for the entire package). 
CHARTER (3 PAGES) 

Attachment 1 
2 
3 
LIST OF DOCUMENTS REVIEWED

1. “Risk-Informed Regulatory Framework for New Reactors,” SECY-12-0081, June 6, 2012, 

Agencywide Documents Access and Management System (ADAMS) Accession 

No. ML12117A012. 

2. “Risk-Informed Regulatory Framework for New Reactors,” Staff Requirements 

Memorandum (SRM) for SECY-12-0081, October 22, 2012, ADAMS Accession 

No. ML12296A158. 

3. Transmittal of Draft Commission Paper on “Recommendations for Risk Informing the 

Reactor Oversight Process for New Reactors,” package of draft SECY paper and staff 

memorandum to the Advisory Committee on Reactor Safeguards (ACRS), 

June 24, 2013, ADAMS Accession No. ML13169A408. 

4. “Proposed Final Safety Culture Policy Statement,” SECY-11-0005, January 5, 2011, 

ADAMS Accession No. ML103200087. 

5. “Proposed Final Safety Culture Policy Statement,” SRM-SECY-11-0005, March 7, 2011, 

ADAMS Accession No. ML110660547, and the associated Commission Voting Record 

(VR) for SECY-11-0005, March 7, 2011, ADAMS Accession No. ML110660554. 

6. “Implementation Plan for the Safety Culture Policy Statement,” SECY-12-0008 and 

enclosures, January 19, 2012, ADAMS Accession No. ML11334A073 (package). 

7. “Implementation Plan for the Safety Culture Policy Statement,” SRM-SECY-12-0008, 

March 1, 2012, ADAMS Accession No. ML120620120. 

8. “Construction Reactor Oversight Process Self Assessment for Calendar Year 2012,” 

SECY-13-0042, April 15, 2013, ADAMS Accession No. ML13045A493. 

9. NRC “Information on the Changes Made to the Reactor Oversight Process to More Fully 

Address Safety Culture,” Regulatory Issue Summary (RIS) 2006-13, July 31, 2006, 

ADAMS Accession No. ML061880341. 

10. “Policy Options and Recommendations for Revising the NRC’s Process for Handling 

Discrimination Issues,” SRM-SECY-02-0166, March 26, 2003, ADAMS Accession 

No. ML030850783. 

11. “Recommended Staff Actions Regarding Agency Guidance in the Areas of Safety 

Conscious Work Environment and Safety Culture,” SECY-04-0111, July 1, 2004, 

ADAMS Accession No. ML041750238. 

Attachment 2 
2 

12. “Recommended Staff Actions Regarding Agency Guidance in the Areas of Safety 

Conscious Work Environment and Safety Culture,” SRM-SECY-04-0111, 

August 30, 2004, ADAMS Accession No. ML102500658. 

13. “Recommended Staff Actions Regarding Agency Guidance in the Areas of Safety 

Conscious Work Environment and Safety Culture,” VR-SECY-04-0111, August 30, 2004, 

ADAMS Accession No. ML042440149. 

14. “Status of Safety Culture Initiatives and Schedule for Near-Term Deliverables,” 

SECY-05-0187 and attachment, October 19, 2005, ADAMS Accession 

No. ML052850012 (package). 

15. “Status of Safety Culture Initiatives and Schedule for Near-Term Deliverables,” 

SRM-SECY-05-0187, December 21, 2005, ADAMS Accession No. ML053550519. 

16. “Status of Safety Culture Initiatives and Schedule for Near-Term Deliverables,” 

VR-SECY-05-0187, December 21, 2005, ADAMS Accession No. ML053560116. 

17. “Safety Culture Initiative Activities to Enhance the Reactor Oversight Process and 

Outcomes of the Initiatives,” SECY-06-0122, May 24, 2006, ADAMS Accession 

No. ML061320282. 

18. “Gap Analysis of the Reactor Oversight Process,” April 8, 2011, ADAMS Accession 

No. ML110810078 (package). 

19. Union of Concerned Scientists, “The NRC’s Reactor Oversight Process: An Assessment 

of the First Decade,” Issue Brief, Cambridge, MA, January 27, 2011, 

http://www.ucsusa.org/assets/documents/nuclear_power/

20110127-ucs-brief-rop-first-decade.pdf (accessed October 30, 2013). 

20. Union of Concerned Scientists, “The NRC and Nuclear Power Plant Safety in 2012: 

Tolerating the Intolerable,” Report, Cambridge, MA, March 2013, 

http://www.ucsusa.org/assets/documents/nuclear_power/

NRC-nuclear-safety-2012-report.pdf (accessed October 30, 2013). 

21. “NRC Staff White Paper: A Comparison of International and US Nuclear Industry 

Performance Indicators to the Current ROP Performance Indicator Program,” 

December 29, 2009, ADAMS Accession No. ML093340455. 

22. Casto, C., “Double Down, Pivot and Blindside: The Dynamic Future of the ROP,” 

presentation for “Regulatory Information Conference (RIC) 2010: The Future of the 

ROP,” March 11, 2010, http://www.nrc.gov/public-involve/conference-symposia/ric/

past/2010/slides/th38castocpv.pdf (accessed October 31, 2013). 

23. Morrow, S., and V. Barnes, Office of Nuclear Regulatory Research, U.S. Nuclear 

Regulatory Commission, “Independent Evaluation of INPO’s Nuclear Safety Culture 
3 

Survey and Construct Validation Study,” Technical Paper, June 2012, ADAMS 

Accession No. ML12172A093. 

24. “The NRC’s Safety Culture Policy Statement - Domestic and International Initiatives,” 

RIC 2013 Presentation Track TH-36, slide presentations, March 14, 2013, 

http://www.nrc.gov/public-involve/conference-symposia/ric/past/2013/docs/abstracts/

sessionabstract-36.html (all pages and linked files accessed October 31, 2013): 

Shoop, U. (NRC), “Common Language for a Common Understanding,” 

http://www.nrc.gov/public-involve/conference-symposia/ric/past/2013/docs/abstracts/

shoopu-rev1-hv-th36.pdf, Morrow, S. (NRC), and G.K. Koves (Institute of Nuclear Power 

Operations, INPO), “Domestic and International Cooperation to Advance Nuclear Safety 

Culture Research,” http://www.nrc.gov/public-involve/conferencesymposia/ric/past/2013/docs/abstracts/morros-kovesk-rev1-hv-th36.pdf, Elsasser, R. 

(Idaho National Laboratory), “Creating the Foundation and Organizational Climate for a 

Strong Nuclear Safety Culture,” http://www.nrc.gov/public-involve/conferencesymposia/ric/past/2013/docs/abstracts/elsasserr-hv-th36.pdf. 

25. “Nuclear Safety Culture Common Language, 4th Public Workshop, 

January 29–31, 2013,” slide presentation, ADAMS Accession No. ML13028A508. 

26. International Atomic Energy Agency (IAEA), Final “IAEA 2010 Integrated Regulatory 

Review Service (IRRS) Report to the United States of America,” March 1, 2011, ADAMS 

Accession No. ML110630400. 

27. “Operating Reactor Assessment Program,” NRC Inspection Manual Chapter (IMC) 0305, 

October 18, 2013, ADAMS Accession No. ML13178A032. 

28. “Reactor Oversight Process Self Assessment Program,” IMC 0307, March 23, 2009, 

ADAMS Accession No. ML090300565. 

29. “Reactor Oversight Process (ROP) Basis Document,” IMC 0308, November 8, 2007, 

ADAMS Accession No. ML071860181. 

30. “Components Within the Cross-Cutting Areas,” IMC 0310, October 28, 2011, ADAMS 

Accession No. ML091480473. 

31. “Significance Determination Process,” IMC 0609, June 2, 2011, ADAMS Accession 

No. ML101400479. 

32. “Power Reactor Inspection Reports,” IMC 0612, January 24, 2013, ADAMS Accession 

No. ML12244A483. 
4 

33. “Safety Culture Assessor Training and Qualification Journal,” Appendix C-12 to 

“Qualification Program for Operating Reactor Programs,” IMC 1245, 

September 26, 2012, ADAMS Accession No. ML12166A543. 

34. “Light Water Reactor Inspection Program – Operations Phase,” IMC 2515, 

June 19, 2013, ADAMS Accession No. ML13176A336. 

35. “Independent Safety Culture Assessment Followup,” NRC Inspection Manual Inspection 

Procedure (IP) 40100, April 5, 2011, ADAMS Accession No. ML080040273. 

36. “Problem Identification and Resolution,” IP 71152, August 13, 2013, ADAMS Accession 

No. ML13179A365. 

37. “Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area,” 

IP 95001, February 9, 2011, ADAMS Accession No. ML102020522. 

38. “Supplemental Inspection for One Degraded Cornerstone or Any Three White Inputs in a 

Strategic Performance Area,” IP 95002, February 9, 2011, ADAMS Accession 

No. ML102020532. 

39. “Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded 

Cornerstones, Multiple Yellow Inputs or One Red Input,” IP 95003, February 9, 2011, 

ADAMS Accession No. ML102020551. 

40. “Degradation of the Davis-Besse Nuclear Power Station Reactor Pressure Vessel Head 

Lessons-Learned Report,” September 30, 2002, ADAMS Accession No. ML022760172. 

41. “Reactor Oversight Process,” NUREG/BR--0508 (brochure), March 2013, ADAMS 

Accession No. ML13064A291. 

42. “Safety Culture Policy Statement,” NUREG/BR-0500 (brochure), Revision 1, 

December 2012, ADAMS Accession No. ML12355A122. 

43. “Davis-Besse Reactor Pressure Vessel Head Degradation: Overview, Lessons Learned, 

and NRC Actions Based on Lessons Learned,” NUREG/BR-0353 (brochure), Revision 1, 

August 2008, ADAMS Accession No. ML082490129. 

44. “Browns Ferry Nuclear Plant - NRC Supplemental 95003 Inspection Report 

05000259/2013011, 05000260/2013011, and 05000296/2013011,” August 22, 2013, 

ADAMS Accession No. ML13234A539. 

45. “Confirmatory Action Letter - Browns Ferry Nuclear Plant, Units 1, 2, and 3, 

Commitments Related to the Integrated Improvement Plan,” EA-13-185, 

August 22, 2013, ADAMS Accession No. ML13232A105. 
5 

46. “Annual Assessment Letter for Susquehanna Steam Electric Station, Units 1 and 2 

(Report 05000387/2012001 and 05000388/2012001),” March 4, 2013, ADAMS 

Accession No. ML13059A425. 

47. “Annual Assessment Letter for Wolf Creek Generating Station 

(Report 05000482/2012801),” March 4, 2013, ADAMS Accession No. ML13060A436. 

48. “Annual Assessment Letter for Browns Ferry Nuclear Plant Units 1, 2, and 3 (NRC 

Inspection Report 05000259, 260, 296/2012001),” March 4, 2013, ADAMS Accession 

No. ML13063A461. 

49. “Annual Assessment Letter for Cooper Nuclear Station (Report 05000298/2012801),” 

March 4, 2013, ADAMS Accession No. ML13063A076. 

50. “Annual Assessment Letter for Prairie Island Nuclear Generating Plant, Units 1 and 2 

(Report 05000282/2012001 and 05000306/2012001),” March 4, 2013, ADAMS 

Accession No. ML13060A115. 

51. Annual “End-Of-Cycle Assessment Letter for Palisades Nuclear Plant 

(Report 05000255/2012001),” March 4, 2013, ADAMS Accession No. ML13063A141. 

52. “Reactor Oversight Process Metric Report for Calendar Year 2012,” April 3, 2013, 

ADAMS Accession No. ML13084A194 (package). 

53. “Solicitation of Public Comments on the Reactor Oversight Process,” November 29, 

2011, ADAMS Accession No. ML112030316 (package). 

54. “Consolidated Response to the 2011 Reactor Oversight Process External Survey,” 

October 18, 2012, ADAMS Accession No. ML12297A327 (package). 

55. “Reactor Oversight Process Annual Performance Metric Report for Calendar Year 2010,” 

April 13, 2011, ADAMS Accession No. ML110740073 (package). 

56. “Solicitation of Public Comments on the Implementation of the Reactor Oversight 

Process,” September 17, 2009, ADAMS Accession No. ML092590482. 

57. “Reactor Oversight Process Annual Performance Metric Report for Calendar Year 2009,” 

April 5, 2010, ADAMS Accession No. ML100540037. 

58. “Reactor Oversight Process Self Assessment for Calendar Year 2008,” SECY-09-0054, 

April 6, 2009, ADAMS Accession No. ML090540513 (package). 
6 

59. Reactor Oversight Process Annual Self Assessment Performance Metrics – “Metric 

Report for Calendar Year 2008,” April 2, 2009, ADAMS Accession No. ML090690616 

(package). 

60. Metric Report – “Reactor Oversight Process Annual Self Assessment Performance 

Metrics [for Calendar Year (CY) 2007],” March 28, 2008, ADAMS Accession 

No. ML080350368. 

61. “Reactor Oversight Process Annual Self Assessment Performance Metrics [for 

CY 2006],” March 26, 2007, ADAMS Accession No. ML070720085. 

62. “Reactor Oversight Process Annual Self Assessment Performance Metrics [for 

CY 2005],” March 30, 2006, ADAMS Accession No. ML060590135. 

63. “Reactor Oversight Process Annual Self Assessment for Calendar Year 2004,” 

SECY-05-0070, March 25, 2005, ADAMS Accession No. ML050630297 (package). 

64. “Reactor Oversight Process Annual Self Assessment for Calendar Year 2003,” 

SECY-04-0053, April 6, 2004, ADAMS Accession No. ML040620267 (package). 

65. Bonaca, Mario V., ACRS, letter to Richard A. Meserve, NRC, “Reactor Oversight 

Process,” March 13, 2003, ADAMS Accession No. ML030730366. 

66. Travers, William D., NRC, letter to Mario V. Bonaca, ACRS, “Reactor Oversight 

Process,” April 29, 2003, ADAMS Accession No. ML030980473. 

67. “Reactor Oversight Process Self Assessment for Calendar Year 2002,” SECY-03-0062, 

April 21, 2003, ADAMS Accession No. ML030640004 (package). 

68. Carpenter, C., memorandum to Deputy Regional Administrators, “Changes Resulting 

from the 2002 Reactor Oversight Process Internal Survey,” January 1, 2003, ADAMS 

Accession No. ML030870883. 

69. “Calendar Year 2001 Reactor Oversight Process Self Assessment,” SECY-02-0062, 

April 3, 2002, ADAMS Accession No. ML020710781 (package). 
PUBLIC MEETING TO SOLICIT STAKEHOLDER VIEWS 

Themes identified during the working group’s initial review were also discussed at a public 

meeting on June 18, 2013, and stakeholder input on the issues was considered in the final 

assessment. Refer to the following referenced documents for additional information. 

• Public meeting notice and agenda, Agencywide Documents Access and Management 

System (ADAMS) Accession No. ML13157A384 

• Public meeting slides, ADAMS Accession No. ML13196A385 

• Public meeting summary, ADAMS Accession No. ML13196A372 

List of External Stakeholders Requesting Public Meeting Agenda and Participation Information 

Name Affiliation 

J. Bergman Scientech 

B. Carroll Duke Energy 

S. Dolley Platts’ Inside NRC

R. Ritzman First Energy Corporation 

J. Slider Nuclear Energy Institute 

J. Tomkins Certrec 

T. Tramm Certrec 

P. Wilson TVA 

T. Zimmerman Duke Energy 

Attachment 3
TABLE OF RECOMMENDATIONS AND SUGGESTIONS 

ROP Goal: Maintaining Safety (Report Section 4.0) 

Recommendation 1 Clarify expectations for the timing of supplemental inspections for 

Column 4 of the Reactor Oversight Process (ROP) Action Matrix, or 

portions thereof, to ensure that the U.S. Nuclear Regulatory 

Commission’s (NRC’s) assessment of continued operation and 

consideration of additional regulatory actions are completed in a timely 

manner. (Section 4.1) 

Recommendation 2 Consider including additional measures in the ROP to minimize abrupt 

changes in the Action Matrix characterization of plant performance 

caused by mechanistic relaxation of oversight based on the passage of 

time and completion of NRC inspections. (Section 4.2) 

Recommendation 3 Include a risk-informed periodic review of licensee programs or actions 

implemented to address generic issues in order to enhance the 

agency’s assurance that these measures continue to be effectively 

implemented. (Section 4.3) 

ROP Goal: Increasing Openness (Report Section 5.0)

Recommendation 4 Consider enhancements to improve the effectiveness of NRC messages 

through more extensive use of plain language, a focus on the desired 

effect of the communication on stakeholder perceptions, and the use of 

wording that coveys the significance of issues to the broadest possible 

audience. (Section 5.1) 

Suggestion 1 Review current Performance Indicators (PIs) to evaluate whether they 

are providing meaningful information on licensee performance. If the 

PIs are validated as being appropriate and not needing adjustment, 

develop messages to enhance stakeholder understanding of how the 

PIs continue to contribute to the NRC’s assessment of plant safety and 

licensee performance. (Section 5.2) 

ROP Goal: Effective and Efficient Decisionmaking (Report Section 6.0)

Recommendation 5 The NRC should review the criteria for transition to Column 3 of the 

NRC Action Matrix against the original ROP program goals to ensure 

that the significance of White inspection findings is not being 

overemphasized and to ensure that agency resources used to process 

White inspection findings are commensurate with findings that, by 

definition, are of low to moderate safety significance. (Section 6.3.3) 

Attachment 4
2 

ROP Goal: Effective and Efficient Decisionmaking (Report Section 6.0) (continued)

Recommendation 6 The NRC should perform a comprehensive analysis to determine 

whether the use of cross-cutting issues and safety culture, as currently 

incorporated in the ROP, provides regulatory value in terms of licensee 

safety performance for the resources expended. To support that 

determination, NRC staff should clarify and document the goals, 

purposes, uses, and desired outcomes associated with the inclusion of 

cross-cutting issues and safety culture in the ROP. If program changes 

are needed, the staff should determine whether Commission approval is 

required for implementation. (Section 6.4.3) 

Recommendation 7 Clarify ROP program expectations for when performance issues that are 

common to multiple facilities should be considered for resolution 

through a generic issues process in order to improve the use of NRC 

inspection resources and ensure a thorough and consistent industry 

response. (Section 6.5) 

Recommendation 8 Consider revising the ROP self assessment process to better solicit and 

assess both tactical and strategic feedback. Reexamine how internal 

and external feedback is collected, analyzed, and used to improve 

oversight approaches to and the implementation of the ROP. 

(Section 6.6) 

Suggestion 2 Enhance the Operating Reactor Assessment Program to ensure that 

plant performance assessment decisions fully and consistently consider 

“other” relevant performance perspectives discussed in Inspection 

Manual Chapter (IMC) 0305 including traditional enforcement, 

allegations, substantive cross-cutting issues, and safety culture in 

addition to the Action Matrix outcomes. (Section 6.1.1) 

Suggestion 3 Consider expanding the list of relevant indicators of licensee 

performance in the Operating Reactor Assessment Program description 

in IMC 0305 to include significant performance concerns that may come 

to light through 10 CFR 2.206 petitions, licensing issues, or financial 

issues that are within the scope of NRC regulations. (Section 6.1.1) 

Suggestion 4 Consider including additional guidance in IMC 0305 to enable the use of 

management discretion in determining whether to accelerate the timing 

of supplemental inspection activities particularly for cases in which it is 

determined that a licensee is not making reasonable progress in 

preparing for these inspection activities. (Section 6.1.2) 

Suggestion 5 The criteria for discussion of licensee performance issues during the 

Agency Action Review Meeting and End-of-Cycle Summary Meeting 

should allow senior management the opportunity to discuss plants with 

performance issues considered at the threshold for additional regulatory 

action particularly those considered to be at risk of moving into 

Column 3 or 4 of the NRC Action Matrix. (Section 6.1.3) 

 
3 

ROP Goal: Effective and Efficient Decisionmaking (Report Section 6.0) (continued)

Suggestion 6 The list of activities that do not constitute Action Matrix deviations 

provided in IMC 0305 could be expanded to allow more efficient 

management decisions on resources needed to address oversight at 

plants with unique, ongoing technical challenges or with ongoing safety 

culture or substantive cross-cutting issues. (Section 6.1.4) 

Suggestion 7 Periodic inspector counterpart seminars, training, and mentoring should 

be used as opportunities to ensure that inspectors and managers have 

a common understanding of the inherent flexibilities in the ROP 

inspection program. Best practices in using the ROP flexibilities should 

continue to be highlighted, and a similar approach should be used to 

communicate the importance of indirect inspection activities and how to 

effectively and efficiently manage these activities. (Section 6.2.1) 

Suggestion 8 Significance and Enforcement Review Panel (SERP) members should 

be provided with periodic training or briefings regarding the uncertainties 

inherent in the agency’s probabilistic risk assessment (PRA) tool 

outputs and the use of PRA quantitative results in SERP 

decisionmaking. (Section 6.3.1) 

Suggestion 9 To the extent practicable, steps should be taken to better understand 

stakeholder concerns with the documentation of significance 

determination process (SDP) results and to enhance the communication 

of SDP results to improve the transparency of the process for the broad 

audience of stakeholders. (Section 6.3.2) 

Suggestion 10 Suggestion 10: Consider replacing the use of substantive cross-cutting 

issues and the current cross-cutting aspects, components, and areas 

with a process that uses the recently developed Nuclear Safety Culture 

Common Language (NSCCL) traits and attributes in a graded regulatory 

response aligned with the overall ROP philosophy. Further, a more 

graded regulatory response could be established to allow licensees to 

address safety culture issues when performance is in the Licensee 

Response column and would escalate the NRC’s level of engagement 

as the significance of findings increase. (Section 6.4.4) 

The following figure reflects the suggested approach: