Consolidated Response to the 

2011 Reactor Oversight Process External Survey 

October 18, 2012 
CONTENTS 

INTRODUCTION ......................................................................................................................... 2 

SURVEY QUESTIONS AND RESPONSES ................................................................................. 3 

(1) The performance indicator (PI) program provides useful insights, particularly when 

combined with the inspection program, to help ensure plant safety and/or security. ........ 3 

(2) Appropriate overlap exists between the PI and the inspection programs to provide for a 

comprehensive indication of licensee performance. ......................................................... 6 

(3) Nuclear Energy Institute (NEI) 99-02, "Regulatory Assessment Performance Indicator 

Guideline," provides clear guidance regarding performance indicators. ........................... 9 

(4) PI program effectively contributes to the identification of performance outliers based on 

risk-informed, objective, and predictable indicators. ....................................................... 10 

(5) Information contained in inspection reports is relevant, useful, and written in plain 

English. ........................................................................................................................... 13 

(6) The inspection program adequately covers areas that are important to plant safety 

and/or security and is effective in identifying and ensuring the prompt correction of 

performance deficiencies. ............................................................................................... 15 

(7) The Significance Determination Process results in an appropriate regulatory response to 

performance issues. ........................................................................................................ 17

(8) The U.S. Nuclear Regulatory Commission (NRC) takes appropriate actions to address 

performance issues for those plants outside the Licensee Response Column of the 

Action Matrix. .................................................................................................................. 22 

(9) Information contained in assessment reports is relevant, useful, and written in plain 

English. ........................................................................................................................... 24 

(10) The Reactor Oversite Process (ROP) safety culture enhancements help in identifying 

licensee safety culture weaknesses and focusing licensee and the NRC attention 

appropriately. .................................................................................................................. 26 

(11) ROP oversight activities are predictable (i.e., controlled by the process) and reasonably 

objective (i.e., based on supported facts, rather than relying on subjective judgment). .. 31 

(12) The ROP is risk-informed, in that actions and outcomes are appropriately graduated on 

the basis of increased significance. ................................................................................ 33 

(13) The ROP is understandable and the processes, procedures, and products are clear and 

written in plain English. ................................................................................................... 34 

(14) The ROP provides adequate assurance, when combined with other NRC regulatory 

processes, that plants are operated and maintained safely and securely....................... 35 

(15) NRC actions related to the ROP are high quality, efficient, realistic, and timely. ............ 37 

(16) The ROP ensures openness in the regulatory process. ................................................. 40 

(17) There are sufficient opportunities for the public to participate in the process. ................ 41 

(18) NRC is responsive to public's comments and inputs on the ROP. .................................. 43 

(19) The ROP has been implemented as defined by program documents. ............................ 44 

(20) The ROP does not result in unintended consequences. ................................................. 44 

ACRONYMS ............................................................................................................................... 49 

REFERENCES ........................................................................................................................... 50 
2 

INTRODUCTION 

In accordance with Inspection Manual Chapter 0307, “Reactor Oversight Process (ROP) SelfAssessment Program,” the U.S. Nuclear Regulatory Commission (NRC) staff publishes a survey 

in the Federal Register biennially to obtain external stakeholder feedback on ROP effectiveness. 

The staff requests stakeholders to provide comments specific to ROP performance metrics. 

Stakeholders can also provide general comments on the ROP. The staff maximizes awareness 

of the survey’s availability by mailing hundreds of surveys directly to stakeholders, emailing to 

stakeholders who signed up to receive notification of the survey, placing a direct link to the 

survey information on the public Webpages, and issuing a press release. The staff’s responses 

to survey respondents’ comments are available on the “ROP Program Evaluations and 

Stakeholder Feedback” Web page of the NRC’s external Web site. 

On November 29, 2011, the staff published in the Federal Register a solicitation for comments 

on ROP implementation (Agencywide Documents Access and Management System (ADAMS) 

accession number ML1120301660). The comment period expired on January 13. 2012. The 

NRC received 15 responses to the survey; 7 were from utility representatives, 4 were from State 

or local officials, 2 were from the public, and 2 were from NRC employees. These responses 

are publicly available (reference ADAMS package accession number ML12033A103). 

The staff appreciates the insightful comments from respondents who participated in the 2011 

ROP self-assessment survey. The self-assessment is used to determine whether the ROP was 

effective in meeting program goals and achieving its intended outcomes, as well as to identify 

areas for potential improvement. This document contains the NRC staff’s response to the 

survey respondents’ comments. The staff consolidated respondents’ comments and organized 

the NRC’s response to those comments by survey question. The staff attempted to document 

stakeholder comments exactly as they were received; therefore, the staff did not change the 

spelling, grammar, punctuation, or content of the comments. 

 
3 

SURVEY QUESTIONS AND RESPONSES 

(1) The performance indicator (PI) program provides useful insights, particularly 

when combined with the inspection program, to help ensure plant safety and/or 

security. 

Respondent Comments: 

Public 

• The PI program should be used to Maximize plant safety and security, not just help 

ensure it. 

• The answer to this is both "Yes" and "No". NRC Staff have found that the Performance 

Indicator program does not provide meaningful insight into plant performance and does 

not predict declining performance. We [think] that is partially true. The mitigating 

systems PIs appear to anticipate problems. However, the remaining indicators do not 

sufficiently discriminate given today's high level of safety performance. They are too 

lagging and too old. They should be modified or replaced. 

Industry 

• Include industry quartile values so plants can assess how they are doing with respect to 

the rest of the industry. 

• There is currently some disagreement between the industry and the U.S. Nuclear 

Regulatory Commission (NRC) regarding interpretation of reporting requirements related 

to the Safety System Functional Failures PI. NRC should ensure that the any changes 

to the reporting guidance are accompanied by an analysis of effect on the performance 

band thresholds for this PI. 

• The PI program provides an effective way for NRC and industry to communicate with 

one another and with the public about the state of performance at U.S. nuclear power 

plants. This helps maintain public confidence in the safety of US plants and the NRC’s 

independent oversight of the U.S. fleet. 

State/Local Government Official 

• Yes - but, the inspection program findings are too soft on licensees. There could be 

hundreds of green findings and the assessments would be that all's fine because it's 

green. I know that there is no mathematical connection to the # of green's to equal a 

yellow, etc. Even the cross-cutting issues don't add up to indicate that there are 

significant problems until it's too late. 

NRC Employee 

• The PI system is stale and can be managed to make them look good. While this may be 

fine for Emergency Preparedness (EP) PIs, this can be done at the expense of doing 

timely maintenance on mitigating systems. There have also been way too many 

examples of the Office of Nuclear Reactor Regulation (NRR) approving licensee 

frequently asked questions (FAQs) that undercut the validity of the PI reporting. 

Licensees have also been deliberately slow submitting FAQs to allow older PI hits to 
4 

drop off before a new one would go on the record, thereby avoiding a white PI. Fort 

Calhoun and Wolf Creek have done this repeatedly over the past 4 years. The system is 

full of loopholes. The different thresholds between Boiling Water Reactors (BWRs) and 

Pressurized Water Reactors (PWRs) in indefensible and should be corrected. Some PIs 

have such high thresholds that they have never had a white one in any plant. This 

should be corrected or the PI eliminated as a waste of resources. The scrams PI has 

too high a threshold. You would have to have 8 times the national average before you 

go white? Also, the Mitigating System Performance Index (MSPI) is a black-box 

calculator that cannot be checked, cannot be understood or interpreted, and when it is in 

danger of going white, licensees change their risk models to make the performance look 

good. MSPI needs to get replaced with more simple indicators. See the Region IV 

memo by Mike Runyan and George Replogle on this subject. 

• If we want improvement then the thresholds should be lowered so that more plants cross 

into the regulatory response portion. 

NRC Staff Response: 

The ROP survey respondents generally indicated that the PI program, in conjunction with the 

inspection program, provides useful insights to ensure plant safety and security. One comment 

indicated that the PI program is an effective way to communicate licensee performance. Some 

comments suggested that the PIs are not predictive of performance, have too high of 

thresholds, and can be easily manipulated. One respondent suggested that the PI program 

should maximize safety. 

When the ROP, including the PI Program, was developed in the late 1990s, the ROP designers 

acknowledged that the industry was mature and had demonstrated a good performance record. 

The ROP designers decided that one of the goals of the ROP was to maintain the level of 

reactor safety achieved at that time, not to improve reactor safety. Therefore, the staff 

recommended, and the Commission agreed, that the industry’s performance would be 

measured based on the level reflected in the 1995-to-1997 timeframe. The ROP was not 

intended to further improve this level of performance; nor was it intended to predict 

performance. Rather, the ROP was designed to effect appropriate regulatory responses to 

demonstrated (not anticipated) performance. As noted in an industry comment in Question 4, 

the Institute for Nuclear Power Operations (INPO) is the organization that fosters and promotes 

excellence and improvement in the industry. The role of the NRC, on the other hand, is to 

ensure licensees are operating the plants in a safe manner. 

PIs were developed to provide an objective indication of licensee performance, and they were 

not intended to be used alone to assess licensee safety performance. A risk-informed baseline 

inspection program was developed to inspect those aspects of licensee performance not 

adequately covered by a PI and to independently verify the PIs. PIs, in conjunction with 

inspection results, provide useful trending information and input to licensee performance 

assessment. PIs having very low (i.e., green) safety significance confirm that performance in 

the attribute covered by the PI (not necessarily overall performance) is acceptable and within 

the expected range based on industry norms and regulatory requirements. Therefore, no 

additional NRC oversight is required for green PIs. PIs have contributed to a number of greaterthan-green Action Matrix inputs that affected the level of agency response. 
5 

If a licensee does not report PI data in accordance with Nuclear Energy Institute (NEI) 99-02, 

does not correct PI reporting errors in a timely manner, or delays maintenance at the expense of 

safety to avoid a PI input that may result in a greater-than-green PI, the NRC pursues inspection 

findings and/or enforcement action in accordance with Inspection Manual Chapter (IMC) 0612, 

“Power Reactor Inspection Reports,” and the NRC Enforcement Policy. If specific feedback 

(e.g., real examples) is provided on how the current PI reporting guidance has had a negative 

impact on maintenance practices, the staff could evaluate and determine if changes need to be 

made to the guidance. Specific allegations of potential wrongdoing (e.g., a licensee or 

individual intentionally delaying reporting PI data to avoid Action Matrix movement) can be 

reported to the NRC and are handled via the NRC’s Allegations Program. The following Web 

site has additional information on reporting safety concerns to the NRC: 

http://www.nrc.gov/about-nrc/regulatory/allegations/safety-concern.html. 

There are some situations for which the guidance is not clear, possibly because the situations 

were not anticipated. Lessons learned and operating experience may result in the need to 

clarify or change the guidance. The FAQ process can result in changes to or clarifications of 

NEI 99-02. The staff strives to ensure that changes to the PI guidance serve to support the 

ROP goals. Tentative FAQ responses are publicly available prior to being finalized (with the 

exception of those involving plant-specific security information), which provides stakeholders an 

opportunity to express concerns about the response. If a stakeholder believes that the NRC 

response to an FAQ is incorrect, the stakeholder can provide those concerns to NRC staff or 

during the public meetings of the ROP Working Group. An NRC staff contact for the ROP 

Working Group public meetings is listed on the meeting notices. 

The safety system functional failure (SSFF) PI has different green-to-white thresholds for PWRs 

(five SSFFs over a 4-quarter period) and BWRs (six SSFFs over a 4-quarter period). The SSFF 

PI threshold was determined using the industry mean plus one standard deviation based on 

data from July 1995 through June 1997. BWRs continue to have a higher number of SSFFs 

than PWRs, possibly because BWRs have more single-train systems than PWRs. In the year 

2000, NRC and industry decided to raise the green-to-white threshold for BWRs from 5 to 6 

based on a review of historic PI data. For more information on thresholds of all PIs, please refer 

to Attachment 1 to IMC 0308, “Technical Basis for Performance Indicators.” 

Some commenters stated that the MSPI is complicated, and NRC staff agrees with the 

statement. However, the MSPI has resulted in licensees implementing modifications to make 

plants safer. In addition, NRC staff and industry are exploring changes to NEI 99-02 to improve 

MSPI guidance (e.g., for maintaining Probabilistic Risk Assessment (PRA) quality and improving 

demand estimates) in response to the memorandum referenced in the comment. The staff is 

working with Senior Reactor Analysts in Regions II and III to develop and provide additional 

MSPI training to further ensure a common understanding of this complex indicator. 

There is one suggestion to include industry quartile data so plants can self-assess and 

benchmark their performance with peers. The NRC’s role is to ensure adequate public health 

and safety and not to facilitate industry benchmarking in a spirit of achieving excellence. 

Nevertheless, significant data is publicly available in Agencywide Documents Access and 

Management System (ADAMS) and through the ROP Website, and can be utilized by interested 

stakeholders to self-assess and benchmark their performance with peers, as desired. 

Additionally, the NRC has, in response to President Barack Obama’s Open Government 

Initiative, developed a high-value dataset titled, “Operating Reactor Performance Indicators,” 
6 

which displays raw PI data from the past two years in a text format. This information can be 

downloaded and exported into a spreadsheet for trending and comparison purposes. This 

information can be found on the following Web site: http://www.nrc.gov/public-involve/open.html. 

The NRC also collects and analyzes industry-wide data to monitor the overall safety 

performance of operating plants. The staff most recently reported these results to the 

Commission in the Office of the Secretary (SECY)-12-0056, “Fiscal Year 2011 Results of the 

Industry Trends Program for Operating Power Reactors,” and during the annual Commission 

meeting on the results of the Agency Action Review Meeting. 

One respondent commented that there was some disagreement regarding interpretation of 

reporting guidance for the SSFF PI, and that any changes to the reporting guidance be 

appropriately analyzed for impact on the PI. The pending modifications to NUREG-1022 do not 

represent a change in the reporting requirements that would affect the SSFF PI, but instead 

clarify the existing guidance to ensure a more consistent application. Nevertheless, further 

discussion on potential impacts of the upcoming revision of NUREG-1022 on the PI could occur 

during future ROP Working Group public meetings, as necessary. Additional discussion on 

potential issues with SSFF PI guidance and impact of NUREG-1022 are addressed in the 

response to Question 4. 

One respondent commented on how the NRC handles green findings in the inspection program. 

Green findings are of very low safety significance (risk) and as such only negligibly affect overall 

public health and safety, even when aggregated. Also once identified, those green findings are 

entered into the licensee’s corrective action program. Cross-cutting aspects are assigned to 

findings within specific categories to potentially reveal cross-cutting issues. If a cross-cutting 

theme has been identified as part of the assessment process, and the Agency has a concern 

with the licensee’s scope of efforts or progress in addressing the cross-cutting theme, a 

Substantive Cross-Cutting Issue will be initiated to document and communicate the concern. 

Additional discussion on the inspection program and cross-cutting issues are discussed in the 

response to Questions 6 and 10 respectively. 

(2) Appropriate overlap exists between the PI and the inspection programs to provide 

for a comprehensive indication of licensee performance. 

Respondent Comments: 

Public 

• Too many loopholes exist that render both programs deficient because of irrational 

concessions and exceptions granted to licensees because of the plant’s age or other 

factors. The routine waiving of safety rules, to allow plants to stay on-line, or because of 

past flawed NRC decisions that remain flawed without any recourse. 

• Data to support a conclusion about this are lacking. Too many plants get "Greater than 

Green" findings year after year, just as Davis Besse did before the event there. At least 

there have not been any equivalent events since so presumably the strength of the 

combined PI+Inspections procedure is working better than in the past. 

 
7 

Industry 

• In the Initiating Events and Mitigating Systems cornerstones, the inspection overlap can 

be excessive. This is especially noticeable in the Problem Identification and Resolution 

(PI&R) inspections and large team inspections such as the Component Design Bases 

Inspections (CDBI), where substantial inspection effort is focused on events and issues 

reported under the performance indicator program. In some cases, the overlap can be 

excessive. This is especially noticeable in findings of low safety significance that also 

affect PIs and safety culture monitoring. An example involves the many recent findings 

regarding degraded voltage relay settings raised during Component Design Bases 

Inspections. In some cases, these findings have also resulted in impacts to PIs (e.g., 

safety system functional failures), safety culture aspects (e.g., problem identification and 

conservative assumptions), and traditional enforcement, even though the findings have 

raised legitimate questions about potential backfit issues. NRC may want to consider 

adjusting outcomes based on the aggregate impact to licensees in unusual cases such 

as these. 

• PIs look at the areas where clear performance thresholds have been developed and 

tested. As envisioned in the development of the ROP, this allows the inspection program 

to look at cornerstone attributes not covered by the PIs, and to spend more time looking 

at areas that require more evaluation and investigation. The process is well integrated 

and some overlap does exist—in some cornerstones more than others. 

In the Initiating Events and Mitigating Systems cornerstones, the inspection overlap can 

be excessive. This is especially noticeable in the PI&R inspections and large team 

inspections such as the Component Design Bases Inspections CDBI, where substantial 

inspection effort is focused on events and issues reported under the PI program. In 

addition, CDBIs rarely yield more than a few Green findings. This suggests the 

considerable amount of NRC and licensee resources put into these inspections could be 

better spent in other areas. 

State/Local Government Official 

• See #1and #6. 

NRC Employee 

• The inspections done to check PI reporting accuracy are basically prevented from 

enforcement by declaring everything minor unless you find the one reporting error that 

would have pushed a plant across a threshold. This is unacceptable. 

• A method that allows ongoing reviews of maintenance activities. 

NRC Staff Response: 

Some commented on the excessive overlap in the mitigating systems and initiating events 

cornerstones, particularly for the PI&R and CDBI inspections, and suggested that the NRC 

adjust outcomes based on the aggregate impact to licensees. The ROP does consider the 

aggregate impact of PIs and inspection findings with the same underlying cause. If a safetysignificant PI issue or occurrence has the same underlying cause as a safety-significant finding, 

the PI occurrence is subtracted from the overall PI value for consideration in the Action Matrix. 
8 

In addition, traditional enforcement (TE) violations and cross-cutting aspects do not cause 

Action Matrix movement; however, the NRC has determined that these issues warrant oversight 

and monitoring because of their safety and regulatory impacts. 

The PI and inspection programs work together to monitor plant safety and licensee 

performance. PI occurrences may or may not be related to licensee performance deficiencies 

(PDs) as defined in IMC 0612, and it may take several of these occurrences to cause a PI to 

cross a threshold. It is possible that that a PI occurrence related to a safety-significant PD 

would not cause a PI to cross the green-to-white threshold. Therefore, the inspection program 

appropriately allows for further review of these PI occurrences to determine if any PDs and 

associated TE or cross-cutting issues were associated with the PI occurrence and if the 

corrective actions in response to the issue causing the PI occurrence were adequate. 

PI&R inspections are resource intensive because they confirm one of the major foundations of 

the ROP: the NRC’s confidence in a licensee’s corrective action program. Licensees receive 

enforcement discretion in the form of non-cited violations for violations having very low safety 

significance if, in part, the licensees enter the issues into their corrective action programs. The 

NRC ensures through inspection that its confidence in a licensee’s corrective action program, 

and the enforcement discretion allowed by participation in the ROP, are justified. 

Regarding the comment about PI reporting inaccuracies and their relationship to enforcement, 

the NRC’s Enforcement Policy typically treats violations involving the failure to report complete 

or accurate PI information as minor. However, if the reporting error would have caused the PI to 

cross a threshold if reported correctly or the licensee does not correct the error, a PI reporting 

violation can be treated as more than minor within the NRC’s enforcement policy. 

Regarding the comment about maintenance activities, the ROP does provide for ongoing 

reviews of maintenance activities. Currently, the inspection program, via routine baseline 

procedures IP 71111.12, “Maintenance Effectiveness,” and IP 71111.13, “Maintenance Risk 

Assessments and Emergent Work Control,” covers the key safety attributes associated with 

maintenance within the Mitigating Systems Cornerstone. In addition, the staff has previously 

considered suggestions to incorporate new PIs or modifications to PIs to further incorporate 

maintenance activities. Most recently, the staff performed an analysis of potential gaps in the 

ROP in 2010 (see ADAMS Accession No. ML110810078) and concluded that the inspection 

program adequately addresses maintenance activities at reactor facilities, and a PI in this area 

is not needed. 

One respondent commented that too many plants consistently have greater-than-green findings, 

so the appropriateness of overlap between the PI and inspection programs cannot be 

concluded. The NRC assesses plant performance continuously and determines its regulatory 

response in accordance with an Action Matrix that provides for a range of actions 

commensurate with the significance of the PI and inspection results. The Action Matrix is 

intended to provide consistent, predictable, and understandable agency responses to licensee 

performance. The responses are graded such that the NRC becomes more engaged as 

licensee performance declines. In implementing the baseline program, the NRC can make 

adjustments to the inspection plan based on plant performance trends. For example, if a PI is 

trending toward the green/white threshold, the NRC can focus inspection effort in that area. For 

plants that have greater-than-green PIs or inspection findings, the NRC will perform 
9 

supplemental inspections beyond the baseline program and initiate other actions commensurate 

with the safety significance of the issues. 

Regarding the comment about too many loopholes rendering both programs deficient, 

insufficient detail was provided for effective use of the feedback. The NRC welcomes detailed 

descriptions of actual occurrences that could be evaluated for potential program improvements. 

(3) NEI 99-02, "Regulatory Assessment Performance Indicator Guideline," provides 

clear guidance regarding performance indicators. 

Respondent Comments: 

Public 

• See #2 

• There continues to be problems in figuring out how to integrate safety culture 

measurements with the ROP. We could find little evidence that NRC is strongly 

engaged in collecting data and assessing patterns in plant problems when it comes to 

safety culture. 

Industry 

• MSPI is complex and FAQ process is marginally effective but I can't come up with ways 

to improve it. 

• Discussions with the staff about NEI 99-02 in the past two years have sometimes led to 

disagreements about the meaning or intent of certain passages in the document. These 

disagreements have turned on differing understandings of what the authors intended the 

passages to mean. These discussions have highlighted gaps in the transfer of 

knowledge from NRC and industry personnel who worked on previous versions of NEI 

99-02 to those maintaining and using the document today. Past contributors to NEI 99-

02 wrote the document with a presumption that readers would share their understanding 

of intent and con-text. Examples such as the discussion of the wording on Unplanned 

Scrams with Complications in the Wolf Creek and Palo Verde FAQ cases in 2010 

highlight what was mutually understood by earlier NRC staff and Reactor Oversight 

Process Task Force (ROPTF) members, but not made explicit in the wording of NEI 99-

02. It would be helpful to capture as much of this implicit knowledge as practical while at 

least some of those individuals remain accessible to NRC staff and ROPTF members. 

We believe the guidance can be made clearer and more accessible to readers who do 

not have the original authors’ shared background and assumptions. We expect to work 

with the NRC staff to effect such improvements in future updates to NEI 99-02. 

The formal process for resolving questions on the guidance (identified in NEI 99-02 as 

the FAQ Process) appears to be working well. Questions are discussed at monthly 

public meetings of the ROPTF and the inspection and assessment branches of the NRC. 

Questions appear to be discussed and resolved in a timely and effective manner. There 

have been some challenges in meeting expectations for timely posting of approved (i.e., 

re-solved/completed) FAQs, but the ROPTF and NRC staff are taking steps to improve 

the process. 
10 

• The structured FAQ and FAQ appeal process continues to improve consistency in 

application of performance indicator guidance and capture the rationale for the decisions 

made in a durable way. The Shared Transformation Acquisition Regulation System 

(STARS) Alliance supports and looks forward to assisting in the continuing efforts to 

further develop and improve the ROP. 

State/Local Government Official 

• Yes, but still concerned over maintenance problems that don't meet the matrix time line. 

The oversight matrix needs to be adjusted to address those nuclear power plants that 

have routine maintenance failures that tend to fall just outside what is required. 

NRC Employee 

• The basic guidance has been totally overcome by the FAQ process creating so many 

special cases and loopholes. It has become like the tax code - too many deductions are 

available in the fine print. 

NRC Staff Response: 

Respondents generally have positive comments on NEI 99-02 guidance, including the FAQ 

process. However, some respondents indicated that the FAQ process might undermine the 

basic guidance. In addition, some comments suggested that MSPI is complex. The staff 

addressed these comments in its response to Question 1. 

One respondent suggested documenting the bases for decisions for knowledge management. 

The NRC agrees and will continue to work with industry to improve the dissemination of FAQ 

information, and to ensure that decision bases are clearly documented for consistent 

understanding. 

The comment about integrating safety culture measurements with the ROP is addressed in the 

response to Question 10. 

One respondent expressed concern that equipment failures caused by poor maintenance may 

not be addressed by the Action Matrix. To the extent maintenance issues reflect current 

licensee performance, they can be revealed through both the PI and inspection programs. If 

they are of greater-than-green significance, they will impact the Action Matrix. Additional 

discussion regarding the NRC’s oversight of maintenance is addressed in the staff’s response to 

Question 2 and others. 

(4) PI program effectively contributes to the identification of performance outliers 

based on risk-informed, objective, and predictable indicators. 

Respondent Comments: 

Public 

• Commonsense is often overruled by risk-based criteria alone, which then becomes the 

basis for determining safety significance and safety consequences. 
11 

• I am not sure how to answer this question, see comments to question 1. On the one 

hand the ROP seems to working. On the other hand, other than the Mitigating Systems 

PIs, it is not clear to which the other PIs effectively discriminate at today's high level of 

safety performance. Most seem to be too lagging, and the NRC has acknowledged the 

need to find more "leading" indicators that detect potential problems early. Given that 

some indicators have never resulted in a finding other than green, they are probably not 

very useful. The failure to create safety culture PIs is glaring given that NRC itself 

recognizes that safety culture is a critical risk component. 

Industry 

• Only MSPI hardware includes a risked informed element. The PIs are historically base 

and there is no predictive element. 

• The recent increase in reporting of safety system functional failures has been driven 

partially by changes in NRC interpretation of reporting guidance. It is not clear that this 

increase is due to a decline in the performance of the industry or of any individual plant. 

• The PI Program, in conjunction with the Inspection Program, effectively identifies 

performance outliers based on risk-informed, objective and predictable measures. 

In recent years, some in NRC have questioned the value of performance indicators that 

are “too green”. The implication is that PIs are meaningful only if they easily and often 

change color (i.e., cross thresholds). This perspective appears based on a 

misunderstanding of the purpose and basis for the PIs. 

The PI thresholds are based on analysis in SECY 99-007 demonstrating that the overall 

performance of industry had dramatically improved in the 1990s and that, with the 

occasional exception, operating performance was safe enough. The Commission 

agreed with this conclusion. Thresholds then were set at levels that would recognize 

outliers against the overall acceptable safety levels. Since then, performance has 

continued to improve in almost all the indicators. This reflects the influence of operating 

experience, the industry’s pursuit of excellence through the INPO, business needs and 

other forces that have prompted licensees to improve plant performance and reduce risk 

since the PI program began. For example, in 2010 the ROPTF identified ways in which 

the MSPI has helped spread awareness of risk-significant operations and design 

features. This increased awareness has fostered significant improvements in plant 

design and procedures that contribute to better plant performance, greater safety 

margins and as a result, more “green” PIs. 

The PIs were designed to provide a timely indication of meaningful changes in the state 

of performance, appropriate for supporting NRC decisions about allocation of oversight 

resources. They were not de-signed to provide continuous indication of the smallest 

variations in performance. (Such indicators are used by plant management to control 

performance, and would be too “noisy” for NRC to use in over-sight.) In this sense, the 

NRC PIs are analogous to the “Check Engine” light on a car’s dashboard. Green 

indicators provide useful confirmation that conditions are nominal and that operations 

and baseline inspections are adequate to assure public safety. The PIs were designed 
12 

to respond to meaningful trends in performance, rather than flutter with minor variations 

in performance. Thus, the frequency of updates and other features of the PI program 

were thoughtfully designed to serve the oversight and public communications roles of 

the PI program. 

The current NRC staff initiative to update the guidance in NUREG-1022 has the potential 

to change a PI by changing the SSFF reporting criteria. 

(See also response to Survey Question 20.) The documentation of historical SSFF in a 

Licensee Event Report may be counterproductive for both the NRC and licensees as the 

result could lead to a unit entering Column 2 of the NRC Action Matrix and require 

supplemental inspection activity for issues that are not reflective of current licensee 

performance. 

• The documentation of "historical" SSFFs in a licensee event report (LER) may be 

counter-productive for both the licensee and for the NRC, since it could unnecessarily 

trigger a supplemental inspection that is not warranted based on licensee current 

performance. 

NRC Employee 

• This statement is true for EP and radiation safety, but wrong for the others. Inspections 

actually identify outliers that are having performance issues. Most PIs can be 

"managed" to show green. 

• Thresholds are too high. 

NRC Staff Response: 

One commenter suggested that the PI program is risk-based. The PI program, as well as other 

aspects of the ROP, is risk-informed, not risk-based. In accordance with Commission policy, 

NRC staff seeks to include risk-informed insights into the ROP, where practicable. 

The comments about PIs being lagging and not predictive, PIs having high thresholds, and PIs 

being manipulated easily are addressed in the response to Question 1. 

One respondent commented on the safety culture in PIs. The Commission directed the staff to 

incorporate safety culture into the ROP, in part, through the cross-cutting areas. Because PIs 

are reported to the NRC 21 days after the end of a quarter, the current construct of assigning 

cross-cutting aspects to findings may provide for an earlier indication of any potential areas that 

a licensee should review further. As such, the inspection program provides indications of crosscutting organizational issues, including safety culture. 

One respondent commented that the current NRC staff initiative to update the guidance in 

NUREG-1022 has the potential to change the SSFF reporting criteria. NEI 99-02 guidance 

currently states that the report date of an LER should be used for reporting SSFF PI data, and 

NUREG-1022, “Event Reporting Guidelines: 10 CFR 50.72 and 50.73,” establishes a 

three-year window-of-interest for event reporting. Therefore, any SSFF that occurred within 

three years would be reported (consistent with NUREG-1022) and count toward the PI in the 

quarter in which it was reported (consistent with NEI 99-02). The staff intends to ensure that 

inspectors and the industry correctly apply SSFF PI guidance and the guidance in Revision 2 of 
13 

NUREG-1022 (ML003762595) because the reporting requirements of Title 10 of the Code of 

Federal Regulations, Part 50, Section 73 (10 CFR 50.73), “Licensee Event Report System,” 

affect inputs to the SSFF PI. As noted under the staff response to Question 1, further 

discussion on potential impacts of the upcoming revision of NUREG-1022 on the SSFF PI could 

occur during future ROP Working Group public meetings, as necessary. 

(5) Information contained in inspection reports is relevant, useful, and written in plain 

English. 

Respondent Comments: 

Public 

• Though what is written in reports is relevant and factual, there are often many findings of 

safety significance that are not documented in the reports that are soon forgotten since 

they are left to licensee discretion to take action or disposition. 

• We found it very hard to obtain all inspection reports. The only way to review safety 

culture findings for instance is to go to each site assessment letter. The NRC can do a 

better job of providing a simple, searchable relational data base of the findings that could 

provide more summary data. 

Industry 

• In some cases, the NRC’s process for refining or revising findings between the plant exit 

meeting and the issuance of the inspection report needs improvement. During the 

inspection efforts, the inspection program appropriately allows for licensee input in 

characterizing a finding and any related determinations, such as safety culture aspects. 

The same principle should apply when the NRC is considering changes in finding 

characterization following the exit meeting with the licensee. In many cases, the NRC 

does communicate with licensees in these situations; however, this practice is 

sometimes not followed, and the NRC should consider reinforcing this expectation. 

State/Local Government Official 

• Information in the inspection reports only reflect the narrow scope of the planned 

inspection. Inspection reports do not show the "other" things that inspectors see during 

their inspections. By keeping the inspections narrowly focused, and the reports the 

same, the other findings go unreported. These issues may be passed on to the 

licensees, but lack the openness that the NRC says it has. 

NRC Employee 

• Only for an "informed" and knowledgeable citizen. 

NRC Staff Response: 

Respondents commented that inspection reports generally contain relevant and useful 

information. 
14 

A respondent commented that NRC does not document findings of safety significance. All 

findings, except those that are licensee-identified green findings (which are of very low safety 

significance), are documented in inspection reports. The threshold for establishing and 

documenting findings can be found in IMC 0612, “Power Reactor Inspection Reports,” and in its 

Appendix B, “Issue Screening.” In addition, all findings having greater than very low safety 

significance (i.e., White, Yellow, or Red) will receive further inspection under an applicable 

supplemental inspection procedure based on the plant’s position in the ROP Action Matrix (i.e., 

IP 95001, 95002, or 95003) and IP 71152, “Problem Identification and Resolution.” 

Licensee-identified findings have to be entered into the corrective action program (CAP) to 

qualify for non-cited violation (NCV) treatment in section 4OA7 of the inspection report. If they 

are not entered into the CAP, they would be documented as notices of violation (NOVs). 

A respondent commented that the NRC does not always inform the licensee of changes in 

finding characterizations made between the inspection exit meeting and the issuance of the 

inspection report. The staff notes that Section 12.01 of IMC 2515, “Light-Water Reactor 

Inspection Program – Operations Phase,” states that changes to the characterization of issues 

after the initial exit meeting will be communicated to the licensee prior to the issuance of the 

inspection report. IMC 0612 also reinforces that inspection report content should not conflict 

with information presented at the exit meeting. These expectations are routinely reinforced 

during counterpart meetings and training. If a licensee becomes aware of NRC staff not 

following these expectations, the licensee should inform regional NRC management. 

There is a comment that inspection reports only focuses on narrow scopes. Inspection reports 

are used to document completed inspection activities, associated enforcement actions and 

findings entered into the Plant Issues Matrix in support of licensee assessment and regulatory 

response. Inspection reports use a consistent format and have a defined threshold for 

documentation which omits most minor issues and general observations by design. This 

approach limits the content of the report to only that information that is considered germane to 

safety performance. Additional documentation beyond this is not an effective use of NRC 

resources and would obfuscate more important information. 

One respondent indicated that the NRC can do a better job of providing a simple, searchable 

relational data base of the findings that could provide more summary data. The staff 

appreciates this feedback and will consider this comment for improvement opportunity as time 

and resources permit. 

One commenter indicated that the inspection report is written for informed and knowledgeable 

citizens. Inspection reports are written to effectively communicate NRC inspection results with 

the licensee and other informed stakeholders. Nevertheless, the staff uses a consistent format 

and utilizes plain English to the extent practicable to explain potentially complex technical 

issues. As indicated in the 2012 Metric Report, survey respondents generally agreed that 

inspection reports are relevant, useful, and written in plain language. That said, the NRC 

continues to review and modify IMC 0612, “Power Reactor Inspection Reports,” as necessary to 

improve the readability and consistency of the reports. 
15 

(6) The inspection program adequately covers areas that are important to plant safety 

and/or security and is effective in identifying and ensuring the prompt correction 

of performance deficiencies. 

Respondent Comments: 

Public 

• Many loopholes exist that provide too many exceptions to too many plants. In addition, 

non-safety modifications that are not part of the licensing and design bases, are 

considered untouchable by NRC, even though they could jeopardize the integrity of 

safety systems through inadequate and flawed designs by placing undue challenges on 

the safety system. 

• As noted above, it is very hard to document how safety culture assessments are carried 

out. Safety culture should be examined as part of the Quality Assurance Program 

requirement. 

Industry 

• The resident inspectors are usually effective in ensuring areas important to safety are 

appropriately addressed through the baseline inspection program. The inspection 

program and the ROP assessment methodology are effective in ensuring identified 

performance deficiencies are promptly corrected. However, the larger team inspections 

(such as the CDBI) have a tendency to inspect the same systems and re-inspect issues 

that have already been inspected, come up with very little useful information, and should 

be reviewed for improvement or elimination.

State/Local Government Official 

• Yes, but your definition of prompt is way too slow. The NRC needs to reinvent itself in 

this area. Many times there are items that are cited as "licensee identified". These 

items were only identified because the NRC was planning to inspect the area -where the 

deficiency(ies) were found and the licensee goes ahead to clean up and find all they can 

before the NRC inspectors show up. By allowing these last minute findings to go without 

consequence does not reflect the true condition of the plants. If the NRC had not 

announced the inspection then the items would likely have been found by the NRC's 

inspectors or gone on uncorrected. NRC inspections should be unannounced thereby 

providing a truer picture of what plant conditions really are. When a licensee is allowed 

to pre-condition the inspection results it should be treated no differently than if they had 

pre-conditioned a surveillance test. 

NRC Employee 

• I agree with the statement in general. However, it is time to add some flexibility into the 

baseline inspection program. In particular, the regions should have some flexibility to 

shift samples between resident IPs to focus on declining performance areas. For 

example, if a plant has a great fire protection program but is doing poor operability 

evaluations, we should be able do less Fire Protection (FP) walkdowns and more op 

eval reviews. 
16 

• We need to review more maintenance activities. The ROP currently fails to address 

aging management issues effectively. 

NRC Staff Response: 

A respondent commented that non-safety modifications are untouchable by the NRC. Under 

the ROP, the NRC can inspect any system or activity that impacts safety, regardless of whether 

or not it is safety-related. The NRC staff is not aware of any non-safety modifications that could 

jeopardize the integrity of safety systems. In addition, the NRC will consider a backfit (per 10 

CFR 50.109) if it identifies non-safety modifications that can affect the integrity of safety 

systems. The backfitting process is further discussed in the response to Question 14. 

One respondent suggested that larger team inspections, such as the CDBI, are not effective 

because they re-inspect the same systems and issues. CDBIs have identified a significant 

number of risk significant issues with an above average number of inspection findings per 

inspection hour. Some inspections may examine previously sampled areas based on our riskinformed sampling, updated information, and operating experience when warranted. It should 

also be noted the CDBI procedure has been recently revised to improve sample selection 

guidance. 

A respondent suggested that safety culture should be examined as part of the Quality 

Assurance Program. Safety culture is considered during all inspection activities, and every 

inspection finding is analyzed for potential cross-cutting aspects related to a licensee’s safety 

performance. Safety culture is also assessed during the mid- and end-of-cycle assessment 

meetings, which consider assigned cross-cutting aspects in developing potential substantive 

cross-cutting issues. Additionally, safety culture assessments may be performed by the 

licensee in response to a long-standing substantive cross cutting issue or a 95002 supplemental 

inspection, and these would be followed up by the NRC using IP 40100, “Independent Safety 

Culture Assessment Follow-up,” and documented in an inspection report. The NRC may also 

specifically examine the licensee’s safety culture during a 95003 supplemental inspection by 

either performing or evaluating an independent safety culture evaluation. 

A respondent suggested that the NRC inspections should be unannounced to provide a more 

accurate picture of plant conditions. The NRC resident inspectors routinely conduct 

unannounced inspections, including inspections during evening hours and on weekends. These 

inspections allow the NRC to inspect typical plant conditions and licensee performance. 

Generally, team inspectors request large amounts of current information from the licensee and 

this imposes a significant burden on licensee resources. In accordance with the Paperwork 

Reduction Act, all large information requests must be approved by the Office of Management 

and Budget as part of a strategy to limit the burden of Government regulation. Inspectors often 

review this information prior to arriving on site to increase the effectiveness of their inspections 

and to minimize their travel costs. For these reasons, team inspections are generally 

announced. 

Regarding the pre-conditioning of inspection results, the licensee may conduct a selfassessment or pre-inspection prior to a periodic NRC team inspection and may identify a 

number of potential findings that may not otherwise have been identified. Identifying and 

correcting problems is always good for safety. Licensee-identified findings of very low safety 

significance are not generally (or are only briefly) documented in the inspection report and are 
17 

not considered in the assessment process. The licensee is given credit for self-identifying and 

correcting these findings of very low safety significance to encourage an aggressive problem 

identification and resolution process. This credit is applied irrespective of the licensee’s 

motivation for finding areas of improvement in their performance. Findings that are of greater 

than very low safety significance always receive appropriate treatment under the significance 

determination process and Action Matrix regardless of who identifies them. 

One respondent suggested that the NRC add flexibility in its baseline inspection program. 

Baseline inspections are considered to be the minimum sampling required to assess safety and 

security performance across the seven cornerstones. Adding flexibility to reduce sampling 

below the minimum sampling required for each inspection procedure could result in inadequate 

sampling in a cornerstone area. It should be noted that inspectors currently may choose to 

perform the minimum number samples allowed by one inspection procedure and more than the 

minimum number samples in others to achieve some level of flexibility. 

One respondent suggested that the NRC needs to improve the ROP to address maintenance 

and aging management issues. The staff recognized the opportunity to improve the inspection 

program to better address aging management issues and recently added related guidance to 

several baseline inspection procedures, including IP 71152, IP 71111.06, “Flood Protection 

Measures,” and IP 71111.21, “Component Design Bases Inspection.” Also, a significant 

number of baseline inspection hours are allocated to the review of maintenance activities where 

age management performance issues may be identified. Nevertheless, the staff plans to 

consider additional opportunities to improve inspection focus on aging management issues in its 

upcoming holistic review of the inspection program as part of the biennial ROP realignment 

effort. 

(7) The SDP results in an appropriate regulatory response to performance issues. 

Respondent Comments: 

Public 

• A complicated system that is easily manipulated and subject to interpretation such that 

real issues often get thrown-out by SDP. 

• The NRC acknowledges that problems in safety culture are the root cause of declines in 

plant performance. If that is the case, identified problems in safety culture should be put 

into the Significant Determination Process. 

Industry 

• It appears that the Security SDP is producing a disproportionate number of greater than 

green findings. A rough count shows that 24 of 85 greater than green findings identified 

in the ROP from 2008 to present have been in the physical security cornerstone. This 

appears to be related to extremely low thresholds for violation areas such as safeguards 

control. 

• The SDP does an adequate job of assigning risk significance to findings and violations, 

but is not as transparent or efficient as it should be. We believe that improvements in 
18 

the NRC guidance (e.g., the RASP Handbook) could provide greater transparency and 

efficiency. Our observations and recommendations are presented below. 

1. Application of human recovery credit in the SDP continues to be an issue. In many 

cases, the preliminary SDP performed by the Senior Reactor Analysts failed to credit 

recovery proposed by the licensee or conservatively assessed the human error 

probability (HEP). However, for a substantial number of those cases, the final SDP 

applied additional recovery credit. In some cases, this resulted in the final risk 

significance being lowered by one color in the final SDP. Two specific issues related to 

human recovery credit continue to be a problem: 

a. The NRC typically rejects credit for non-proceduralized recovery actions, 

including ad hoc recovery actions that may be developed by the Emergency 

Response Organization. This was specifically cited as a basis for rejecting 

licensee proposed recovery credit in at least three reviewed SDPs. In one case, 

the proposed recovery is credited in the baseline PRA and is in plant procedures, 

but was not in plant procedures for the dominant risk contributor. Improved 

guidance for providing partial credit for these types of recovery actions would 

result in more realistic SDPs. 

b. The NRC typically rejects additional recovery credit for equipment using 

different success criteria than specified in the baseline PRA, even if supported by 

analysis. Given the iterative nature of PRA model development, use of realistic 

success criteria for determining the risk significance of events should be an 

acceptable option. 

2. Common cause failure (CCF) modeling in SDPs is the source of many disagreements 

between the NRC staff and licensees. The criteria for determining a failure to be 

independent in the SDP process are unduly stringent. The result is that licensee 

assessments of appropriate CCFs often differ significantly from the NRC’s assessment. 

3. As a result of unclear guidance in the Risk Assessment Standardization Project 

(RASP) Handbook, the NRC continues to be somewhat inconsistent in application of 

initiating event frequencies for performance deficiencies resulting in an actual plant trip. 

In most cases, a conditional core damage probability (CCDP) is used with the initiating 

event set to a probability of 1.0. However, in some cases, an initiating event frequency 

of 1.0/exposure time is used to represent the initiating event frequency. Further 

guidance in this area would yield consistent treatment of initiating event frequencies. 

4. The guidance in the RASP Handbook Volume 2 (External Events) is at a much higher 

level than the guidance in Volume 1 (Internal Events). While this may be due to the fact 

that the state-of-the-art for external events is somewhat in flux, it may result in providing 

inadequate guidance to the intended audience for the more detailed analyses needed to 

support a Phase 3 SDP. 

5. Recent SDPs seem to be using NUREG/CR-6850 guidance rather than the 

information presented in Section 2.0 of Volume 2 of the RASP Handbook for the 

evaluation of internal fire risks. This is problematic for two reasons. First, this creates a 

disconnect between the referenced methodologies and those actually used in the 
19 

process. Second, Fire PRA methodologies have evolved appreciably since the issuance 

of NUREG/CR-6850, and use of the data and methods in that document may not yield 

accurate or appropriate assessments. The methodologies included in the RASP 

Handbook should be updated to include reference to recently-issued the Electric Power 

Research Institute (EPRI) documents on state-of-the-art Fire PRA methods to ensure that 

the process is supported by the best resources available. 

6. The internal flood guidance provided in Section 3.0 of Volume 2 of the RASP 

Handbook does not use the pipe failure frequencies developed in EPRI TR-10131419 or 

EPRI TR-102108610 based up-on more recent industry experience than the 1991 and 

1993 vintage documents referenced in the handbook. In addition, the methodologies do 

not represent the state-of-the-art internal flooding analysis guidance presented in EPRI 

TR-1019194.11. The guidance in Section 3.0 should be updated to incorporate 

information from these more current sources. 

7. The guidance provided for assessment of seismic risks in Section 4.0 of Volume 2 of 

the RASP Handbook provides a relatively complete treatment of the analysis elements 

involved in seismic PRA. However, in many areas the guidance is simplified compared 

to the current state-of-the-art. For example, the number of seismic bins used and use of 

generic fragility data are not consistent with current state-of-the-art methodologies. In 

addition, more guidance is needed related to use of the seismic equipment list to identify 

impacts resulting from failure of equipment not modeled in the internal events PRA, 

screening of equipment, modeling of seismic influences on modeled Human Factor 

Engineering (HFEs), and treatment of seismically induced internal flooding. 

8. Section 5.0 of the Volume 2 of the RASP Handbook does not provide sufficiently 

detailed guidance for evaluating other external events (e.g., river flooding and high winds 

hazards) to ensure consistent treatment. The section should be revised to ensure that 

all three elements of external hazards PRA (i.e., hazard definition, fragility analysis, and 

plant response) are discussed at a sufficient level of detail to support the analysis of 

external hazards other than seismic. 

9. More sophisticated modeling methods for various external events will become 

available over the next few years. As they do, the affected portions of the RASP 

Handbook should be updated accordingly, to avoid major discrepancies between the 

licensee and NRC assessment methodologies. 

State/Local Government Official 

• No, the creation of the SDP has the appearance that industry had a major influence on 

its development. SDP needs to be notched up to reflect a truer picture of safety issues 

within the plants and industry. 

NRC Employee 

• The phase 3 SDP process is under-staffed (SRAs) and overly burdensome (mostly to 

management). The resources needed to process a phase 3 and a subsequent 

regulatory conference with all the attendant work is not adequately budgeted. 

 
20 

NRC Staff Response: 

One respondent commented that the SDP was complicated and subject to interpretation. The 

staff understands the perception of subjectivity in the decision-making process, and 

acknowledges that, in a risk-informed process, subjectivity can be minimized but it cannot be 

eliminated. Risk-informed SDP determinations involve a level of uncertainty inherent in the 

assessment. As a result, the final SDP determination could be viewed as a distribution that 

spans both sides of a threshold with the average falling on one side of the threshold (e.g., high 

White). The decision makers consider all the available information to make an informed and 

well-reasoned decision in a process that is largely (but not completely) objective. Another way 

to look at this scenario would be to consider the final decision as the most objective decision 

made given the available, inherently uncertain information in a risk- informed decision-making 

process. In addition, to maintain consistency, the guidance in IMC 0609, “Significance 

Determination Process,” Attachment 1, “Significance and Enforcement Review Panel (SERP) 

Process,” provides a framework to ensure that SERP panel members follow common guidelines 

when determining the preliminary and final significance of an inspection finding. In the 2012 

Metric Report, the SDP met the metrics of being predictable, repeatable and risk-informed. 

One respondent suggested that the safety culture-related issues should be entered in the SDP. 

All findings are processed by the SDP and are screened for potential cross-cutting aspects. 

The assessment process is used to evaluate these inputs to determine whether there is a 

substantive cross-cutting issue. 

One respondent commented on the large number of greater-than-green security findings, 

compared to other cornerstones. The ROP consists of seven cornerstones within three 

strategic performance areas: reactor safety, radiation safety, and safeguards. Since the 

Security cornerstone is the only cornerstone in the Safeguards strategic performance area, it is 

not surprising that it accounts for a significant portion of the total number of greater-than-green 

findings. However, if the number of greater-than-green findings were categorized by strategic 

performance area, the ratios would be different. For example: in the Reactor Safety strategic 

performance area the cornerstones are Initiating Events, Mitigating Systems, Barrier Integrity, 

and Emergency Preparedness. The total number of greater-than-green findings in the Reactor 

Safety strategic performance area makes up a large portion of the greater-than-green findings 

from 2008 to the present. Nevertheless, the security SDPs are being reviewed and revisions 

thereto are under consideration. 

One respondent commented that the industry had a major influence on the development of the 

SDP and that the SDP needs to be improved. During the construction of the ROP, the NRC 

held several public meetings and workshops to build the structure of the PI Program, the 

Inspection Program, the Assessment Program, and the SDP. These programs were 

established to be objective, risk-informed, understandable, and predictable. Industry and other 

interested stakeholders were active participants in these meetings and workshops and continue 

to be involved in informing potential ROP improvements. The risk evaluations used to support 

the SDP reflect a best estimate of the risk impact after taking into account both model and 

information uncertainties. Since the implementation of the ROP, the SDP has continuously 

improved, as have the associated guidance and risk tools (e.g., computer software) used by 

senior reactor analysts and risk analysts to perform detailed risk evaluations. 
21 

One respondent commented on the lack of resources to perform phase 3 of the SDP. 

Estimating the risk associated with a degraded condition or programmatic weakness resulting 

from an inspection finding can be an extremely complex process. As a result, the NRC staff at 

times can dedicate many hours in attempting to accurately estimate the risk significance of a 

finding. However, as described in the SDP guidance (i.e., IMC 0609 and its associated 

appendices and attachments), the staff should use the best available information to make a riskinformed decision in accordance with the prescribed timeliness goal. In addition, ROP 

performance metric SDP-5 from IMC 0307 Appendix A measures whether SDP expenditures 

are appropriate and do not exceed 10 percent of the total regional direct inspection effort (DIE) 

with a stable or declining trend. This metric has been met each year since ROP inception with 

the highest percentage remaining well below the threshold at 7 percent. Nevertheless, the staff 

assembled a project team to conduct a review of the SDP, from a resource and timeliness 

perspective, to critically evaluate established practices and guidance to identify opportunities for 

improvement to the SDP and other impacted programs. 

The comment regarding the lack of budgeted resources for phase 3 of the SDP is more globally 

addressed under the response to Question 8. 

One respondent commented on human recovery credit in the SDP and the RASP Handbook. 

Credit for non-proceduralized actions is normally not considered since there is not a sufficient 

level of confidence in the reliability of those human actions. As a result, non-proceduralized 

human actions do not receive much credit in the risk analysis. Credit for equipment using 

different success criteria can be considered if there is a technically justifiable basis for using 

different success criteria. However, if there is not a strong basis for using different success 

criteria, the optimal decision is to use the initial success criteria in the risk analysis. If the staff 

does not use proposed assumptions for the SDP result, the staff considers the suggested 

insights via sensitivity analyses. In addition, the staff documents the manner in which 

information provided by the licensee was evaluated and considered in the final significance 

determination letter. 

Regarding CCF modeling in the SDP, criteria for determining a failure to be independent will 

continue to be stringent in the revised RASP Handbook guidance. However, methods are 

presented in the revised guidance for treating special cases of failures based on the state of 

another component outside the common cause component group (e.g., pre-initiator human 

errors, environmental stress, etc). In addition, a draft NUREG presents opportunities to 

enhance the state-of-the-art in CCF modeling that would apportion CCF dependencies at the 

causal level, which could potentially remove unduly conservative assessments. RASP 

handbook guidance will be included in Section 5, "CCF Treatment," in Revision 2 of Voume.1. 

The staff is currently working on a revision to the RASP Handbook to specifically clarify the 

application of initiating event frequencies that result in a plant trip. The changes will be 

documented in RASP handbook guidance to include Section 8, "Initiating Event Analysis," in 

Revision 2 of Volume 1. 

The external event guidance in Volume 2 of the RASP Handbook is at a higher level of detail 

because of the current state-of-the-art for external event models. As the external event model 

fidelity increases and becomes a more standardized product, the detailed risk analyses that are 

driven by both internal and external events should become less variable. 
22 

The EPRI documents noted above are not publicly available. As such, the staff cannot cite 

information from those documents in a public document (e.g., RASP Handbooks). However, the 

information from EPRI documents can be used as inputs in the overall risk evaluation to support 

the SDP as long as the proprietary information is not disclosed. 

The tools and information in the area of external events are generally less developed than they 

are for internal event analyses. Fire, seismic, and external flooding PRAs in particular are not 

available for all plants, and are much less detailed than internal event PRAs, which are 

continually improving in quality and detail. The purpose of the SDP is to determine if further 

NRC response to inspection findings is warranted; it is not intended to advance the state of the 

art in external event PRAs. As a result, the SDP uses “best available” information. As NRC and 

industry develop more detailed external event PRA methods and tools, the staff expects the 

SDP tools to follow accordingly. 

(8) The NRC takes appropriate actions to address performance issues for those 

plants outside the Licensee Response Column of the Action Matrix. 

Respondent Comments: 

Industry 

• While it appears that, in general, NRC takes appropriate actions, the decision-making 

process to get to those actions is not always transparent. We would urge the NRC to 

ensure that each performance deficiency that is formally documented in an inspection 

report is consistent with the deficiency as it was presented at the inspection exit meeting. 

When performance deficiencies are poorly specified, or are substantially modified after 

being presented to the licensee as “fully formed”, the NRC undermines confidence in the 

inspection and performance assessment process.

State/Local Government Official 

• No, ever since the NRC stopped issuing fines for violations (again under the pressure of 

the nuclear industry) the "hammer" that the NRC has used has been unused or too softly 

applied. There needs to be a means to grab the licensee's attention where they feel it 

most. Fines often reflected poorly of their performance in a way that every rating agency 

and the public could clearly see. When the fines were done away with the NRC said that 

they were too small to really mean anything. Rather than increase the amount of the 

penalties the NRC went the other way to eliminate them entirely. Some of the reasoning 

was that is they hurt them financially, thru penalties or ratings it would carry over to the 

safety of the plant. If this were true than the NRC would have had to take even further 

action, which the industry lobbied heavily against. 

NRC Employee 

• NRC does not adequately budget for the management workload needed for plants 

outside Column 1. 

NRC Staff Response: 

One respondent commented on the potential inconsistency in the characterization of a finding 

between the exit meeting and the inspection report. A finding’s characterization discussed at an 
23 

exit meeting should be consistent with the characterization documented in the inspection 

report. However, all matters discussed during an exit meeting are pre-decisional and subject to 

management review; therefore, they can change. In accordance with NRC policy, if a finding’s 

characterization changes after the exit meeting, the NRC would hold another exit meeting to 

communicate the characterization that will be documented in the inspection report. If the 

licensee does not agree with the content of the inspection report they are encouraged to 

respond on the docket or official record. The NRC would consider the matter and would issue 

an amended inspection report if warranted. This comment was also addressed in the staff’s 

response to Question 5. 

A respondent commented that the NRC doesn’t take appropriate action after the agency 

stopped issuing violations. The NRC can and does issue civil penalties in accordance with 

Section 2.3.4 of the NRC Enforcement Policy. Violations assessed under the SDP are not 

normally considered for civil penalties. However, civil penalties are considered for violations 

associated with inspection findings that involve actual consequences, involve wrongdoing, or 

impede the regulatory process. The NRC may also exercise discretion and issue a separate 

violation and attendant civil penalty up to the statutory limit for each day the violation continues. 

The NRC may exercise this discretion when a licensee was aware of a violation, or if the 

licensee had a clear opportunity to identify and correct the violation but failed to do so. 

As outlined in SECY 99-007A, “Recommendations for Reactor Oversight Process 

Improvements,” the staff revised to the Enforcement Policy when developing the ROP to 

complement the assessment program. For violations associated with findings that are assessed 

using the SDP, the Action Matrix is used to formulate the agency’s response and to emphasize 

the need to improve performance for safety-significant violations. Use of the Action Matrix with 

its escalating responses, (e.g., increased inspection, regulatory attention, and regulatory 

actions) provides appropriate incentives to operate the plant safely and deters licensees from 

moving to the right in the Action Matrix. Some violations at operating power reactors cannot be 

addressed only through the ROP. These violations are typically dispositioned using the 

traditional enforcement process (as defined in Section 7.0 of the Enforcement Policy) and can 

involve civil penalties when warranted. 

Regarding the comment that the NRC does not adequately budget for the management 

workload needed for plants outside Column 1, as well as similar comments regarding 

inadequate budgets for the SDP and participation and travel for public meetings, the NRC’s 

oversight budget is defined at a summary level with considerations for the specific activities 

discussed in the comments based upon historical usage. This model results in stable budget 

and staffing levels that are predictable from year to year, even if the resource usage for specific 

activities varies. Historical data have shown that the resources allocated have been adequate 

at the summary budget level. In addition, resources are specifically targeted to certain activities, 

(including SDP, supplemental inspections, and meeting attendance and preparation), and hours 

expended are tracked and do not vary significantly from the projected resources. NRC staff 

performs an annual review of the ROP resources as part of the annual ROP self-assessment 

process. As noted in the most recent self-assessment in SECY-12-0055, overall staff effort to 

implement the ROP in CY 2011, as reflected in expended hours, increased by 1 percent 

compared with CY 2010. Fluctuations were noted in the baseline, plant-specific, and generic 

safety issues inspections, as well as in the performance assessment and other ROP support 

activities, which is consistent with the typical level of variation from year to year. The specific 

comments related to the budget will be considered as input to internal NRC annual budget 
24 

development exercise. If appropriate, resources in the oversight budget may be reallocated 

from lower to higher priority activities in accordance with the NRC’s Planning, Budgeting, and 

Performance Management process.

(9) Information contained in assessment reports is relevant, useful, and written in 

plain English. 

Respondent Comments: 

Public 

• Isn’t this the same statement as Item 5? 

• Please see comments to question 5 

Industry 

• The information contained in assessment reports is, for the most part, relevant, useful, 

and well written. Inspection schedules in particular are good to have in advance even if 

they are not fully refined. When significant changes are made to inspection schedules, 

revised schedules should be made publicly available. 

An element of the assessment letters that could be improved is the discussion about 

substantive cross cutting issues. Greater consistency in the language and the detailed 

discussion used across regions would be appropriate. Historically, the criteria for 

opening and closing SCCIs have not been clear; therefore it is not surprising that 

assessment letters have done a poor job of explaining why SCCIs have been opened or 

closed. We are waiting to see whether relatively recent revisions in the NRC guidance 

on SCCIs will make a noticeable difference in the treatment SCCIs get in assessment 

reports. At the same time, we are optimistic that implementation of the industry’s safety 

culture initiative (see response to Question 10 below) will demonstrate that it provides a 

far more effective means of gauging and managing safety culture than do SCCIs.

State/Local Government Official 

• No, they are too textbook and wordy. Get to the point and lay out the facts. 

NRC Employee 

• Assessment reports have very little useful information to licensees. The assessment 

process takes place behind closed doors, the information reviewed is never identified, 

and the conclusions are [only] discussed at a very high level, provided the plant 

exceeded a very high threshold. A huge fraction of the discussion and the output is 

focused on cross-cutting aspects, which is the least important part of the inspection 

program. We continue to send the wrong message about what is important by doing 

this. 

• For people somewhat familiar with our industry and have a knowledge of how NRC 

regulates. 

 
25 

NRC Staff Response: 

A respondent suggested that the NRC be consistent in the language used for discussion of 

substantive cross-cutting issues (SCCIs). The staff agrees with this suggestion. The 

Commission’s Final Safety Culture Policy Statement was issued in June of 2011 (76 FR 34773; 

June 14, 2011). This policy statement outlines the Commission’s expectations that all licensees 

maintain a positive safety culture at their facilities. After publication of the policy statement, 

NRR re-started an initiative to develop safety culture common language and held a joint public 

workshop on December 13-14, 2011, with a panel consisting of members from the NRC, 

Nuclear Energy Institute (NEI), the Institute for Nuclear Power Operations (INPO), and members 

of the public. A second public workshop was held on April 17-18, 2012, with a similar panel and 

participants to further develop the common language. The purpose of these workshops was to 

develop, to the extent possible, common safety culture terminology for use by NRC, INPO and 

International Atomic Energy Agency (IAEA). This language could be adopted for use in NRC’s 

ROP and INPO’s assessment process as well as other safety culture applications. A 

subsequent public workshop is being planned for the fall to work towards finalizing the common 

language between the NRC and industry/INPO, and the anticipated result is a common 

language document (e.g., a NUREG or other controlled NRC document). 

As noted in a comment from industry, NEI, in partnership with INPO, has tested a broad 

initiative to monitor and improve the industry’s nuclear safety culture through an industry pilot 

program. The NRC staff has observed these pilot applications to become familiar with the 

initiative and to evaluate associated tools that could possibly be leveraged to gain efficiencies in 

the ROP. The NRC communicated its observations of the process in a letter to NEI (reference 

ADAMS Accession No. ML11061A007). Safety culture and the SCCI process are further 

discussed in the staff’s response to Question 10. 

Some respondents commented that the assessment reports are too wordy and are written for 

people who understand how the NRC regulates. Assessment reports/letters are intended to 

clearly and concisely describe a licensee’s performance during the assessment period. Plain 

language is used to the maximum extent possible. The assessment letters were designed to 

communicate the Action Matrix column of performance, any identified substantive cross-cutting 

issues, and what, if any, NRC oversight actions are pending. This language becomes more 

standard if licensee performance is governed by the licensee response column and more 

descriptive as needed to address performance issues as NRC oversight increases and plants 

move to the right in the Action Matrix. The staff created templates for the regions to use to 

achieve the goals mentioned above, and the regions provide the pertinent information based on 

actual plant performance and assessment. The regions modify the template as necessary to 

address specific issues or questions unique to a particular licensee or group of external 

stakeholders. 

A respondent commented that the assessment process is not open, and too much emphasis is 

placed on SCCIs during assessment discussions and in the assessment letters. Section 7 of 

IMC 0305, “Operating Reactor Assessment Program,” outlines the different performance 

reviews the NRC performs, the information discussed at these reviews, and the output 

generated from these reviews. The purpose of the mid-cycle and end-of-cycle assessment 

reviews is to perform an overall review and assessment of the performance of all plants, Agency 

actions taken in response to crossed thresholds, and the effectiveness of licensee corrective 
26 

actions to address identified performance deficiencies. The annual assessment reviews provide 

senior regional and Headquarters managers an opportunity to review those plants with 

significant performance deficiencies and the Agency’s planned and completed actions in 

response to licensee performance issues. IMC 0305 outlines in detail the items and various 

topics to be discussed during assessment reviews. With the exception of proprietary, predecisional, security-related information and other non-publicly available information (such as 

INPO assessment results, allegations, investigations, etc.), most of the information discussed is 

publicly available. For example, operating data, previous assessment results, inspection 

results, performance indicator results and previous substantive cross-cutting issues are all 

publicly available. 

The decision to determine if a cross-cutting theme is an SCCI is not entirely objective because 

the regional staff needs to qualitatively determine (i.e., have confidence in) the extent to which 

the licensee understands the issue and is correcting it. IMC 0305 provides guidance to assist 

with this determination by listing various degrees of a licensee’s recognition of, and corrective 

actions for, the SCCI. Differences exist among the regions in making SCCI decisions, 

establishing exit criteria, and documenting SCCI decisions because each licensee, situation, 

and regional concern is unique. As a result, the assessment letters need to clearly articulate the 

basis for decisions to open, close, or not open an SCCI even though a theme exists. Therefore, 

assessment letters containing SCCI discussion may be longer in length and contain significant 

discussion regarding the SCCI determination so all stakeholders understand the Agency’s 

decision for opening, sustaining or closing SCCIs. 

And finally, some respondents referred to Question 5, which specifically applies to information 

contained in inspection reports as opposed to assessment reports/letters. Please refer to the 

staff’s response to that question.

(10) The ROP safety culture enhancements help in identifying licensee safety culture 

weaknesses and focusing licensee and the NRC attention appropriately. 

Respondent Comments: 

Public 

• Sometimes, depending on the relationship that exists between the NRC and the 

licensee/NEI, and on political pressures. 

• It is difficult to answer this question given the way it is asked. What do you mean by 

"help?" Safety culture should be fully incorporated into the ROP by making it a 4th 

Strategic Area. We think the ROP ought to look like this: 
27 

Industry 

• For the most part, but a significant number of issues can be binned in not following 

procedure or inadequate procedure, when other "causes" exist. 

• The safety culture enhancements do not focus licensee resources appropriately. Since 

licensees apply significant resources to correcting safety culture issues identified by the 

NRC, it is important that the NRC process for identifying these issues reflects an 

integrated picture of a licensee’s safety culture. Basing conclusions about safety culture 

at a plant on the relatively small number of safety culture crosscutting aspects that are 

assigned to findings over a period of time does not provide an accurate assessment of 

safety culture. The “greater than three findings” threshold for a substantive cross-cutting 

issue seems to have no basis. There has been enough run-time on the program to reevaluate the threshold. In addition, the cross-cutting aspect definitions are broad 

enough that deficiencies within an aspect may be unrelated and not constitute a valid 

trend in a particular area; however a substantive crosscutting issue could be considered. 

Differences across regions and plants in the number of inspection hours and findings 

naturally produce variations in the number of safety culture aspects assigned and a 

corresponding wide variation in the number of substantive cross-cutting issues identified 

by the NRC. 
28 

• The ROP safety culture changes to the ROP consist of identifying cross-cutting aspects 

of performance deficiencies, and accumulating them into substantive cross-cutting 

issues (SCCI). 

The identification of cross-cutting aspects associated with inspection findings does 

provide value to the licensee to consider in the assessment of safety culture. Note, 

however, that the association by NRC is done without conducting a formal root or 

apparent cause, and may very well be mistaken. Licensees should incorporate the 

NRC’s association into their ongoing assessment of safety culture, using all available 

site data (for example, safety culture assessments, employee concerns issues, site PI 

data, self-assessments, audits, benchmarking, industry evaluations, operating 

experience, etc.). More accurate conclusions can be reached by integrating all of the 

information available on the site safety culture through the NEI 09-0713 process that 

was implemented October 1, 2011. 

Industry does not believe that the practice of accumulating aspects into SCCIs is 

appropriate or effective. First, the number (usually four in a year) is arbitrary and not 

based on research, and its appropriateness has not been assessed (benchmarked 

against actual safety performance) since the changes were implemented. For example, 

it does not appear reasonable that four procedure adherence issues (usually all green, 

or of very low safety significance) over a year’s period represents a cultural problem. 

(The thousand people at a station likely perform more than one procedure per person 

per day, for 365 days a year, which would be hundreds of thousands of opportunities, 

with only four failures.) Furthermore, the number four is not normalized based on the 

inspection hours or the number of units on site, and therefore can create a false 

impression of cultural weakness merely because there were more opportunities to 

identify violations which are assigned an aspect. Second, many of the aspects are not 

safety culture issues per se, but rather process errors (for example an error in a 

procedure step, or a deficiency in the corrective action program). A more thorough 

examination of multiple process errors is needed to determine whether there was a 

common cultural aspect that deserves corrective action beyond just fixing the process 

error. Third, much time and effort is expended discussing which aspects apply, 

particularly as one approaches the number of four. Fourth, it is not at all clear what the 

objective criteria are for determining whether the licensee is taking appropriate action to 

address the supposed substantive issue, or what needs to be done to clear the issue if it 

in fact exists. Fifth, the use of two different languages to discuss safety culture (the 

NRC’s and the industry’s) can lead to confusion in identifying and resolving cultural 

issues. (The recent NRC workshop to develop common language is an important step in 

the right direction.) In summary, industry believes that the SCCI process results in 

excessive use of NRC and licensee management resources, and it diverts resources to 

address perceived problems from correcting actual safety issues, including safety culture 

issues. 

The industry wants to be proactive in ensuring plants have a strong nuclear safety 

culture. Licensees are responsible for the safe operation and safety culture of their 

plants; the NRC is responsible for providing effective oversight. Therefore industry is 

working with the NRC and other stakeholders to: (1) develop a common language of 

safety culture to be used by the regulator and the licensee; (2) implement an integrated 

approach for licensees to assess their safety culture on an ongoing and proactive basis 
29 

with the NRC providing effective, transparent oversight, and (3) develop a common 

methodology for conducting self, independent and third-party safety culture 

assessments. 

State/Local Government Official 

• No, the ROP helps, but there needs to be a "feel" for what is happening. The ROP helps 

the NRC with resource utilization to an extent, however, this can get in the way of using 

resources in more important areas. As mentioned elsewhere in these responses, the 

NRC has a vast knowledge base that can help to focus the licensee and NRC attention 

more appropriately. 

NRC Employee 

• This statement is only true for plants that get up into Column 3 or 4. We have no leg to 

stand on below this. In fact, we get lots of extra allegations because nuclear workers 

have many false ideas about what SCWE, safety culture, and chilled environment really 

mean. We need to better educate the workers. 

NRC Staff Response: 

A respondent suggested creating a fourth strategic area in the ROP for safety culture. The staff 

believes that safety culture is more appropriately addressed through safety culture assessments 

by appropriately qualified staff. In addition, cross-cutting issues related to safety culture are 

addressed through cross-cutting areas that affect all cornerstones consistent with the original 

construct of the ROP. During the 1980s and 1990s, the NRC investigated working 

environments and aspects of organizational culture through means such as diagnostic 

evaluations and special inspections at NRC licensed facilities. The creation and implementation 

of the ROP in April of 2000 transformed the NRC inspection process from a deterministic 

approach to inspection and enforcement to one that is more objective, risk informed, 

understandable, and predictable. The ROP framework consists of 7 cornerstones, including 

initiating events; mitigating systems; barrier integrity; EP; public radiation safety; occupational 

radiation safety; and security. Satisfactory licensee performance in the cornerstones provides 

reasonable assurance of safe facility operation. In addition, the ROP has 3 cross-cutting areas 

– areas that “cut across” all of the ROP cornerstones: safety conscious work environment, 

problem identification and resolution, and human performance. As described below, identifying 

cross-cutting aspects (CCAs) and substantive SCCIs within these cross-cutting areas is 

intended to reveal potential cross-cutting issues that affect licensee performance, and provide 

the licensee with the opportunity to address the issues before they result in a more significant 

safety concern. 

Some respondents commented that cross-cutting issues do not serve as good assessments of 

a licensee’s safety culture. The staff agrees and, as noted above, believes that safety culture is 

more appropriately addressed through safety culture assessments by appropriately qualified 

staff. As noted in previous consolidated responses, the intent of identifying CCAs and SCCIs is 

not to provide an overall assessment of a licensee’s safety culture. Rather, the intent is to 

reveal potential cross-cutting issues that affect licensee performance and provide the licensee 

with the opportunity to address the issues before they result in a more significant safety 

concern. SCCIs that remain open for 18 months may prompt the NRC to request a licensee to 

perform a safety culture assessment. The NRC draws conclusions about safety culture based 

on the effectiveness and the results of these assessments, as well as safety culture 
30 

assessments conducted in accordance with IP 95002 and IP 95003 as plants move to the right 

in the ROP Action Matrix, not based on the presence of SCCIs. 

One respondent commented that licensee and NRC resources are being inappropriately used to 

address CCAs. As noted in previous consolidated responses, because CCAs correlate to the 

causes of findings, the staff expects licensees to spend resources on addressing these causes 

to ensure that potential underlying organizational issues are appropriately attended to. 

Regarding the statement about safety culture only being adequately addressed for plants that 

are in Column 3 or 4, the assigning of CCAs and determination of SCCIs allow for insights into 

cross-cutting areas of performance to be attained continually for all licensees. Additionally, in 

the baseline inspection program, the NRC uses IP 71152 to inspect licensee performance in the 

cross-cutting areas. This inspection is performed every two years at each licensed plant. The 

IP 71152 inspection procedure also has annual and semi-annual samples that can be used to 

follow up on SCCIs 

Regarding comments concerning consistent implementation of ROP safety culture, as noted in 

the response to Question 9, the NRC, NEI and INPO are working together on developing a 

common language for safety culture that should improve communication and reduce confusion 

for the inspectors and for industry. The staff is also working on enhancing inspector training to 

ensure consistent implementation of the safety culture aspects of the ROP. 

 

Some respondents questioned the criteria to have an SCCI. The SCCI threshold consists of the 

two criteria described in Section 14 of IMC 0305. The number of CCAs identified at a site (e.g., 

greater than three for PI&R and human performance issues) is only one consideration of the 

SCCI decision-making process. Another important consideration (the second SCCI criterion) is 

the NRC’s confidence in the licensee’s scope of efforts or progress in addressing the crosscutting theme(s). Although a licensee may have multiple findings with the same CCA, the NRC 

may not identify an SCCI if the licensee is taking adequate corrective actions to address the 

theme. As such, the staff’s identification of an SCCI is in large part a function of its confidence 

in the licensee’s corrective actions. The NRC’s confidence will be based on the four 

considerations described in Section 14.02 of IMC 0305, which are informed, in part, by the 

results of PI&R inspections. IP 71152 instructs inspectors to review a licensee’s corrective 

actions for cross-cutting themes and SCCIs. This IP lists general attributes that inspectors 

consider when determining the effectiveness of licensees’ corrective actions. Nevertheless, as 

noted below, the staff will consider adjustments to the ROP if there is evidence that the industry 

initiatives are effective and that the effectiveness is sustained. 

Some respondents commented that the NRC should adopt the NEI’s proposed safety culture 

oversight process and eliminate the use of SCCIs. The staff has communicated that the NRC is 

supportive of industry initiatives to address safety culture and strongly encourages the industry 

to monitor and promote a culture that strives to enhance safe plant operation. If the NEI 09-07 

process proves to be effective, the number of SCCIs would likely decline throughout the 

industry. A licensee’s safety culture initiative, if properly implemented, can provide the NRC 

with sufficient confidence that a process is in place to monitor and correct cross-cutting themes, 

which could lead to fewer SCCIs. The staff will consider adjustments to the ROP if there is 

evidence that the industry initiatives are effective and that the effectiveness is sustained. The 

staff will continue to interact with NEI on safety culture initiatives. 
31 

(11) ROP oversight activities are predictable (i.e., controlled by the process) and 

reasonably objective (i.e., based on supported facts, rather than relying on 

subjective judgment). 

Respondent Comments: 

Public 

• The interpretation of Rules, Regulations, and activities are very subjective, and often 

depend on the whims of the Resident Inspectors, the Inspection Team Leaders, and 

their management. Furthermore, ROP interpretations vary widely from Region to Region. 

• We could not find adequate evidence to support this finding, and because safety culture 

has not been truly integrated into the ROP it is hard to see how this could be a true 

statement given that the NRC acknowledges safety culture as a leading risk. 

Industry 

• In general, yes. However, NRC should formally evaluate the estimated number of 

inspection hours for large team inspections (Triennial FP, Component Design Bases 

Inspection, and PI&R) against the actual inspection hours experienced in the last few 

years for these inspections. The Regional Utilities Group (RUG) IV plants have noted 

that the actual inspection hours quite frequently exceed the estimated hours, sometimes 

by a factor of two, even allowing for pre-inspection prep time. This lack of predictability 

in inspection hours makes it difficult for licensees to budget and plan resources to 

support inspections. Additionally, the RUG IV plants have noted a recent tendency for 

last minute changes in the published inspection schedule. We realize that some of 

these changes may be unavoidable; however, we request that NRC evaluate ways to 

minimize the need for schedule changes, especially for near-term inspections. 

• Overall, the ROP is predictable and reasonably objective. Some opportunities for 

improvement include the following. 

1. The assumptions used by the NRC in the Significance Determination Process are at 

times subjective and arbitrary (refer to comment on Question 7). This has in some cases 

resulted in delays in finalizing the final results of an SDP. Industry encourages the use of 

licensees’ PRA models which accurately reflect the as-built, as-operated plant. We also 

encourage additional work in the areas of common cause and human performance. 

2. The process is not objective in the area of fire protection/Alternate Safe Shutdown 

(ASSD) capability, and this issue will be exacerbated due to NRC expectations for 

conservative fire PRA assumptions as the basis for NFPA 805 implementation. These 

models do not provide results consistent with operating experience or internal events 

PRA models, and this bias will need to be accommodated in the SDP process, because 

insights and experience from the piloting of transitioning to NFPA 805 have not been 

incorporated into the ROP. 

3. The availability definitions have been somewhat confusing (in that there are several) 

and are continually a topic of discussion. Recent difficulty regarding the definition of 

availability centers around how much credit can be taken for simple actions that restore 
32 

equipment and make it usable; for example, a manual action that has been determined 

to be feasible to allow the equipment to be ready to perform its risk significant function 

(Note that this does not involve trying to take credit for the actions to avoid counting a 

failure; only to restore availability). Since differences exist, and regulatory interpretations 

are not consistent, issues regarding availability and the definition of availability have 

become distractive. 

4. When MSPI was developed the intent was to align the definitions of availability 

between the PI manual, NEI 99-02, and the Maintenance Rule definition of unavailability 

in NUMARC 93-01. More work is needed in this area (understanding of the definitions of 

availability and alignment.) NEI submitted proposed revisions to NUMARC 93-01 to 

address this inconsistency. 

 

5. There is room for improvement in the closure of unresolved items (URIs) identified in 

inspection reports. There are numerous cases of URIs remaining open for extended 

periods. When they are eventually closed, they often result in findings or violations that 

are no longer reflective of current performance. When dealt with in the aggregate, they 

may result in increased inspection activity via supplemental inspections or substantive 

cross cutting issue closure. To support the ROP principle of predictability, the NRC staff 

should have a goal on the order of six months for closure of URIs. 

• There is room for improvement in the closure of unresolved or open issues identified in 

inspection reports. Licensees typically maintain metrics associated with the length of 

time unresolved issues (URIs) and LERs are open. It would be helpful if the NRC also 

did this with a goal of closure in 6 months as this would promote increased predictability 

in the ROP oversight activities. 

State/Local Government Official 

• Yes, but that's not necessarily the best for the situations at hand. Subjective judgment 

can still offer insights that the predictive indicators fail to offer. The NRC has many years 

of experience that could contribute much more. 

NRC Staff Response: 

Some respondents commented on the subjectivity of ROP. Similar to the staff response to SDP 

(Question 7), the ROP has inherent subjectivity because inspectors are required to use 

reasonable judgment in applying established program requirements and guidance to each plant. 

The NRC conducts inspector training and continues enhancement to ROP requirements and 

guidance to minimize wide variation. 

One respondent indicated that safety culture is not truly integrated into the ROP. The staff 

response to Question 10 addresses how safety culture is integrated into the ROP. 

One respondent suggested that the NRC evaluate estimate versus actual inspection hours. The 

NRC formally evaluates the estimated number of inspection hours every two years during the 

ROP Realignment along with other considerations to align the historically expended hours with 

the inspection scope defined in each inspection procedure. Some variations in actual inspection 
33 

time may vary based on the time it takes an inspection team to complete the required inspection 

objectives at a particular site. The NRC is also sensitive to rescheduling team inspections and 

attempts to minimize the need for schedule changes, recognizing the potential for unnecessary 

regulatory burden. 

One respondent indicated that the ROP is predictable and reasonably objective; and provided 

several improvement opportunities for the ROP. The staff addressed the assumptions used by 

the NRC in the SDP in its response to Question 7. 

Regarding the comment on findings related to fire protection/ASSD capability, as part of the 

overall process for determining the risk significance of an inspection finding, after the 

preliminary risk determinations is made, the licensee is given the opportunity to provide any 

additional information to support the risk analysis. If the licensee perceives conservative biases, 

then those can be considered by the staff prior to the final determination. 

One respondent commented on the definition of availability. The staff notes that the MSPI and 

10 CFR 50.65, “Requirements for monitoring the effectiveness of maintenance at nuclear power 

plants,” differ in purpose. As a result, the concept of unavailability is applied differently in these 

different contexts. NEI 99-02 also acknowledges that differences in definitions and guidance, in 

most instances, are deliberate and necessary. This issue has also been addressed in previous 

consolidated responses to the external survey. The staff and industry have discussed this issue 

and determined that no additional revisions to either the MSPI or maintenance rule guidance are 

necessary or planned. 

Some respondents commented on the closure of unresolved issues. The agency manages 

URIs and attempts to close these issues in a timely manner. The imposition of an arbitrary 

timeliness goal over active management could have a negative impact on safety or security if 

URIs were closed to meet timeliness goals. Sometimes additional information is required to 

determine if a performance deficiency or violation exists, or if the performance deficiency is 

more than minor and this information may not be obtainable within an imposed timeliness goal. 

For example, certain areas of the plant may be inaccessible to inspectors during plant 

operation. Nevertheless, the staff acknowledges that this may be an area for improvement and 

continues to look for ways to improve the management and prioritization of inspection resources 

to close out URIs commensurate with their importance to safety and security. 

(12) The ROP is risk-informed, in that actions and outcomes are appropriately 

graduated on the basis of increased significance. 

Respondent Comments: 

Public 

• Risk informed must at times give way to common sense no matter what the numbers 

appear to portray. 

• We could not find adequate evidence to support this finding, and because safety culture 

has not been truly integrated into the ROP it is hard to see how this could be a true 

statement given that the NRC acknowledges safety culture as a leading risk. 
34 

Industry 

• See the comments in response to Question 7 regarding the Security SDP. 

State/Local Government Official 

• No, the grading needs to be much stricter. 

NRC Employee 

• Whenever possible - some items do not fit into our risk models and require subjective 

decisions 

NRC Staff Response: 

One respondent suggested that common sense should be placed in the ROP. In a riskinformed framework for decision making, quantitative risk insights can provide useful 

information. However, the uncertainties and sensitivities to the underlying assumptions need to 

be articulated and understood when evaluating the results of the risk analyses. When the 

uncertainties are large or the critical assumptions, when varied, result in significant changes to 

the overall risk evaluation, the staff considers qualitative information to arrive at a final decision, 

consistent with the risk-informed (not risk-based) regulatory approach described in Regulatory 

Guide (RG) 1.174, “An Approach for Using Probabilistic Risk Assessment in Risk-Informed 

Decisions on Plant-Specific Changes to the Licensing Basis.” 

The staff addressed the comment on safety culture in its response to Question 10. 

One respondent suggested that the grading needs to be stricter. The staff believes the 

comment refers to thresholds. Please see response to Question 1 regarding the principles of 

the ROP. 

One respondent indicated that some items require subjective decision making. The staff does 

not disagree with this comment. In addition to the staff’s response to Question 7 regarding 

subjectivity in SDP, there are inspection findings that are the proximate cause of a degraded 

condition or programmatic weakness, the significance of which cannot be determined using the 

existing SDP tools (i.e., appendices of IMC 0609). When these infrequent situations arise, the 

staff uses IMC 0609 Appendix M, “Significance Determination Process Using Qualitative 

Criteria,” to qualitatively assess the significance of the finding. 

(13) The ROP is understandable and the processes, procedures, and products are 

clear and written in plain English. 

Respondent Comments: 

Public 

• Repetitive successful plant operation dulls ROP interpretation. What should be cause for 

grave concern and immediate action is often delegated as a routine issue of no 

consequence. 

 
35 

Industry 

• MSPI is complex and FAQ process is marginally effective but I can't come up with ways 

to improve it. 

• SDP procedures can be cumbersome and difficult to follow. 

NRC Staff Response: 

The staff disagrees with the notion that issues of grave concern are often delegated as a routine 

issue. The ROP is designed to focus NRC and licensee resources on the most safety 

significant issues and to ensure prompt and effective corrective actions. Plant safety is our 

highest priority. Specific examples where this objective is not being or has not been met would 

be welcomed and greatly appreciated. 

The staff addressed comments regarding MSPI and the FAQ process in its response to 

Question 1. 

One respondent indicated that SDP procedures can be cumbersome. The staff understands 

that some SDP appendices are complex and difficult to follow. The staff continues to refine and 

clarify the SDP guidance as necessary to make the guidance as understandable as possible 

and facilitate usability. Currently, the staff is in the process of revising Appendix F (fire 

protection) and Appendix G (shutdown operations) to IMC 0609 to improve clarity and make the 

appendices more user-friendly. 

(14) The ROP provides adequate assurance, when combined with other NRC 

regulatory processes, that plants are operated and maintained safely and 

securely. 

Respondent Comments: 

Public 

• See Items #2, 4, 6, 11, 12, 13. Furthermore, some SERs written when existing plants 

were licensed are technically flawed, yet NRC remains reluctant to admit, challenge, and 

correct these flawed SERs that clearly affect safety. Many older plants should not be 

exempt from meeting the most basic of safety regulations and industry standards; this 

NRC practice is then perpetuated even when these plants are granted life extensions. 

Something seems very wrong. 

• On the one hand the ROP seems to working. On the other hand, other than the 

Mitigating Systems PIs, it is not clear to which the other PIs effectively discriminate at 

today's high level of safety performance. Most seem to be too lagging, and the NRC has 

acknowledged the need to find more "leading" indicators that detect potential problems 

early. Given that some indicators have never resulted in a finding other than green, they 

are probably not very useful. The failure to create safety culture PIs is glaring given that 

NRC itself recognizes that safety culture is a critical risk component. 

 
36 

State/Local Government Official 

• No, as mentioned above, issues such as what the inspectors see that never make it to a 

report, the lack of subjectivity, etc. and the difficulty for a licensee to get anything more 

than a "green" finding make the ROP appear to be an industry created program. Look 

back at the original launch of the ROP. The industry comments and direction that it tried 

to sway the NRC will be obvious that they wanted a program that would not get them 

into trouble in the public's eye which would then affect their ratings, earnings, and stature 

amongst peers. 

• I don't believe so. A plant that has routine maintenance failures can get by as long as it 

does not meet the PI matrix time line. In other words the time line needed to meet a 

particular indicator can mask repeated maintenance failures. 

NRC Employee 

• This assurance is relative to having all indications of safety being lagging indicators. 

The other regulatory processes are going to have to improve following the events at 

Fukushima. 

NRC Staff Response: 

One comment indicated that the NRC has acknowledged the need to find more "leading" 

indicators that detect potential problems early. This is not the case. The staff repeats that the 

ROP was designed to ensure appropriate and predictable regulatory responses to 

demonstrated performance; it was not designed to be predictive, and leading indicators of 

performance have for decades eluded the NRC and industry alike. Comments on PIs being 

lagging indicators and that they have high thresholds are further addressed by the staff in its 

response to Question 1. 

A respondent commented that the time line needed to meet a particular indicator can mask 

repeated maintenance failures. Most PIs consider plant data over a certain time interval. For 

example, a PI threshold is exceeded when a plant has certain number of SSFFs over a 4-

quarter interval. The inspection program is used to follow-up on individual PI occurrences or 

maintenance-related failures that involve performance deficiencies. Those performance 

deficiencies are then assessed for their impact on safety. Even though individual PI 

occurrences may be minor or not associated with performance deficiencies, the PIs can reveal 

trends if underlying performance issues are not corrected. The PIs encourage licensees to 

maintain good safety performance so that the thresholds are not crossed; however, the 

inspection program is used to follow-up on individual PI occurrences to evaluate them for 

performance issues and their impact on safety. 

Regarding the comment that the ROP appears to be an industry created program, the staff 

recognizes that the industry (and its improved safety performance in the late 1990’s) was 

instrumental in the focus of the ROP development efforts. Nevertheless, the development of the 

ROP was open to all stakeholders, and it was designed and established in an open, 

collaborative manner. The staff solicits input and feedback from all stakeholders and strives to 

continue to make improvements to the ROP. The other aspects of that comment were 

addressed by various responses to other survey questions. 
37 

The comment regarding flawed SERs is not within the scope of the ROP. Nevertheless, a 

cornerstone of the NRC’s philosophy is that current operating plants maintain the required level 

of safety. Over the plant’s life, this level of safety is enhanced through maintenance of the plant 

and its licensing basis. A plant’s licensing basis is an evolving set of requirements and 

commitments. The licensing basis for plants has changed over time to address safety issues. 

Examples include changes to licensing basis of plants to address station blackout, anticipated 

transient without scram, aircraft impact assessment, and beyond-design-basis fires and 

explosions. A recent example includes the imposition of Orders to enhance spent fuel pool 

instrumentation at all operating power plants as a result of lessons learned from the Fukushima 

Dai-ichi Nuclear Power Plant accident. 

All plants are required to meet basic safety standards. As those standards have changed over 

time, licensing bases are modified to reflect new requirements. The backfitting process allows 

the NRC to issue new or revised requirements or staff positions to licensees of nuclear power 

reactor facilities, when warranted. Backfitting is expected to occur and is an inherent part of the 

regulatory process. However, it is to be done only after formal, systematic review to ensure that 

changes are properly justified and suitably defined. Requirements for proper justification of 

backfits and information requests are provided by two NRC rules, Title 10 of the Code of 

Federal Regulations, Sections 50.109 and 50.54(f). 

(15) NRC actions related to the ROP are high quality, efficient, realistic, and timely. 

Respondent Comments: 

Public 

• See Item #11, 14 

• It takes way too long to post annual assessment letters in publicly available data bases. 

Industry 

• For the most part, the ROP is effective, efficient, realistic, and timely. A continuing area 

for improvement is the SDP. The SDP is a fundamental process for the ROP, as it is 

exercised frequently and is used to determine the safety significance of findings. As 

such, it is in both industry and NRC interests that the process be efficient, transparent, 

and objective. Current concerns with the SDP include timeliness of completion and 

subjectivity in the determination of outcomes. While timeliness has improved somewhat 

in the past several years, the timeliness and subjectivity concerns are linked; often 

licensees spend much time challenging SDP determinations that appear to involve 

subjective elements in the use of risk tools, thus delaying SDP completion. 

• For the most part, the ROP is effective, efficient, realistic, and timely. However, some 

areas could be improved, specifically timeliness in finalizing the characterization of a 

finding using the SDP. One reason for the delay is the use of subjective assumptions by 

the NRC. Industry encourages the use of licensees’ PRA models which have been 

evaluated and peer reviewed against consensus PRA standards and NRC Regulatory 

Guide 1.200 to support the SDP process as they become available and believes that by 

doing so, improvements would be made in timely application of the SDP. 
38 

Continuing efforts to make the Mitigating Systems Performance Index more elegant and 

theoretically pure, but which result in miniscule changes in results, are diverting 

resources that could be applied more productively for other improvements. It must be 

remembered that the MSPI is an indicator of performance which is “risk-informed.” The 

resources being applied by NRC and industry to “perfect” MSPI are not available to 

make plants safer and will not improve the allocation of NRC inspection resources. 

State/Local Government Official 

• No, the actions are way too soft. 

• For the past three years, I have been attempting to assist Larry Criscione in addressing 

an incident which occurred at Ameren’s Callaway Plant. Attached to this email is an 

article about the incident which Larry wrote for the Professional Reactor Operator 

Society. 

According to Mr. Criscione, on October 21, 2003 Callaway Plant was shutting down for a 

forced outage when the NRC licensed operators lost control of reactor temperature due 

to a build-up of a radioactive gas called Xenon-135. The falling temperature caused the 

plant’s filtering system to automatically isolate. During the confusion of responding to the 

falling temperature and the loss of the filtration system, the operators failed to notice the 

reactor passively shutting down. 

At our November 8, 2011 meeting, representatives from the NRC (Tony Vegel and Dave 

Dumbacher) confirmed that the NRC’s agrees with Mr. Criscione’s assessment that from 

10:18 to 11:25 am on October 21, 2003 the NRC licensed operators at Callaway Plant 

were not aware that the reactor they were supposed to be monitoring had passively shut 

down. 

At 11:25 am an alarm annunciated on the reactor plant’s Main Control Board, indicating 

to the personnel in the control room that reactor power had lowered “into the Source 

Range”. At this point, according to Mr. Criscione, any competent operator should have 

known that his/her duty was to: (1) promptly insert the reactor’s control rods to ensure it 

stayed shutdown once the xenon decayed away, (2) notify the plant’s upper 

management that the reactor was no longer operating, and (3) document in the plant’s 

Corrective Action Process that the crew had failed to notice the reactor passively 

shutting down. None of these items were done (the control rods were not inserted for 

another 40 minutes). It is Mr. Criscione’s belief (which he has alleged to the US NRC on 

multiple occasions) that the crew failed to perform these actions because of dishonesty – 

that is, they knew their obligations yet failed to comply because they wished to cover up 

their mistakes from the site’s upper management. It is also Mr. Criscione’s contention 

that the Plant Director (Dave Neterer, who at the time was the Operations Manager) was 

in the Main Control Room at 11:25 am and that Neterer was involved in covering up the 

incident. 

Mr. Criscione claims that the NRC has refused to adequately investigate his concerns 

because, under the Reactor Oversight Process, incidents for which the risk of a reactor 

accident were low are not pursued. I have a concern with this. As a member of the public 

and as an elected representative of the public, it is troubling to me that the uppermost 
39 

management of at a nuclear plant would allow the insertion of the reactor’s control rods 

to be delayed for 40 minutes in order to keep his operators from looking bad – if that is 

indeed what happened here. I have looked at the incident from many angles, and the 

only conclusions that make sense to me are dishonesty and/or incompetence. 

Regardless of what the risk of a reactor accident was, I expect an allegation such as this 

to be thoroughly investigated. A year ago, Mr. Criscione submitted a 10CFR2.206 

petition requesting the NRC obtain answers from Ameren regarding why it took them 40 

minutes to insert the control rods after recognizing the reactor was in the source range. 

The NRC rejected Mr. Criscione’s petition. In my opinion, with regard to the October 21, 

2003 passive reactor shutdown at Callaway Plant the NRC has failed to: “give the public 

timely and understandable assessments of plant performance.” 

Mr. Criscione’s latest petition (concerning Ameren’s practice of disabling one of its safety 

systems in violation of its operating license) is an example of yet another important issue 

with little direct risk of a reactor accident. It is evident from the email trail which Mr. 

Criscione provided in his petition that Ameren had ample opportunity to recognize the 

need to amend their operating license but through a combination of mismanagement 

and disregard for professional engineering opinions failed to take the necessary steps to 

operate the plant in accordance with their license. Yet, because there was no significant 

risk of an accident, the NRC is unlikely to address the matter. However, as a member of 

the public and as an elected representative of the public, it troubles me greatly that the 

operators of a nuclear plant would fail to live up to the commitments they made to the 

US Nuclear Regulatory Commission when they obtained a license to operate their plant. 

NRC Employee 

• Processing greater than green findings is inefficient, burdensome, and inherently 

untimely. Average time from onsite inspection to issuing the final color is about 9 

months. The huge increase in workload falls on the senior inspectors and management. 

• Some timeliness goals related to completing the SDP are unrealistic - the information 

can require a long time to acquire and the Phase 3 evaluations can be quite involved. 

NRC Staff Response: 

One comment indicated that the posting of assessment letters is untimely. It is the NRC’s 

practice to post all publicly available documents in ADAMS within 48 hours after the document 

becomes official. The assessment letters are also available on the NRC’s public Website at: 

http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/listofasmrpt.html. The annual assessment 

letters are accessible through the “Q4” hyperlinks, and the mid-cycle assessment letters are 

accessible through the “Q2” hyperlinks. And we are looking for ways to improve the usability of 

these Web sites. 

The NRC is aware of the alleged incident at Callaway. As noted in the letter to Representative 

Oxford dated April 20, 2012 (ADAMS Accession No. ML12167A508), the NRC conducted an 

independent inspection and investigation regarding the incident and concluded that the 

shutdown did not endanger public health and safety. Furthermore, no evidence was obtained 

during the investigation and inspection that would indicate that there was wrongdoing, such as a 

cover-up, involved in the plant shutdown. The staff did not identify any new information from 
40 

these comments that we have not previously considered; therefore the NRC does not plan to 

pursue this issue further. 

Some respondents commented on the timeliness of SDP. The staff notes that 100 percent of 

inspection items finalized as having greater-than-green safety significance met the SDP 

timeliness goals, as documented in the CY 2011 ROP metric report (ADAMS Accession No. 

ML12037A174). SDP timeliness and the information needed for an evaluation are further 

addressed under the staff’s response to Question 7. 

(16) The ROP ensures openness in the regulatory process. 

Respondent Comments: 

Public 

• Not from the Publics’ perspective. Comments which challenge the technical basis of 

NRC decisions are frequently ignored and remain unacknowledged. 

Industry 

• The ROP process, with its many public meetings and opportunities for involvement, 

promotes openness not available in the previous process. However, improvements could 

be made in soliciting stakeholder feedback when revising or developing regulatory 

documents such as Inspection Procedures, Manual Chapter guidance, or Regulatory 

Issue Summaries (RIS). At times, the staff appears unduly reluctant to solicit or consider 

industry input for ROP documents. We appreciate all opportunities to contribute the 

wealth of industry experience to the betterment of the ROP and its implementation 

documents. 

State/Local Government Official 

• No, the dog and pony shows that the NRC annually puts on in front of the public are a 

mockery of the system. The NRC has already met with the licensees and then attempt to 

fool the public that the news the licensees are receiving is new to their ears. Sorry NRC, 

you are not fooling anyone and only making fools of yourself. If you truly want to be open 

then be open. Don't play games in public because it doesn't work. 

NRC Staff Response: 

Regarding the comment that public comments are frequently ignored, the NRC strives to 

implement the ROP in an open, predictable, and collaborative manner. The NRC encourages 

public participation and feedback on ROP implementation, including comments which challenge 

the technical basis of ROP decisions. Our goal is to be responsive to the feedback, and we’d 

welcome specific examples where stakeholders felt their comments were ignored or not 

adequately addressed. 

A respondent commented that improvements could be made in soliciting stakeholder feedback 

when revising or developing regulatory documents, and that the staff appears unduly reluctant 

to solicit or consider input for ROP documents. As noted in the staff’s consolidated responses 

to previous external surveys, the NRC holds monthly and other meetings with external 

stakeholders to provide information about changes to various regulatory documents (e.g., 
41 

inspection manual chapters, inspection procedures, temporary instructions, and regulatory issue 

summaries) that may be of interest to external stakeholders. The staff has not solicited 

comments on changes to some of these documents because the changes remained within the 

ROP regulatory framework. Some level of independence from external influences on what is 

inspected and how inspections are conducted is warranted and needs to exist. However, the 

staff has been sensitive to revisions of regulatory documents that might involve changes to 

policy or regulatory positions because these changes could potentially result in an unnecessary 

increase in regulatory burden on licensees with no commensurate improvement to safety. The 

staff also needs to periodically clarify NRC documents to provide additional guidance to both 

inspectors and the industry. In these situations (e.g., the inspection of 10CFR50.69 activities), 

the staff will communicate and work with external stakeholders to resolve these issues when 

they are brought to the NRC’s attention. 

Some respondents commented negatively on the effectiveness of the annual public meetings. 

The staff has revised IMC 0305 to make the annual assessment public meetings more effective. 

The revision will provides the regional staff more flexibility to conduct public outreach events for 

plants that were in Column 1 or 2. The amount of public interest in plant performance varies 

widely from plant to plant. To maximize stakeholder participation, events such as open houses, 

poster sessions, virtual meetings, or other similar activities will be considered for public 

outreach. Participating in an event sponsored by another organization, such as a county fair, 

will also be considered if such an event would maximize public engagement. 

The NRC strives to hold public meetings with licensees that are conducive to public 

participation. With the exception of meetings during which non-publicly available information 

(such as proprietary or security-related information) will be discussed, most, if not all, 

assessment meetings with the licensees are open to the public. The NRC conducts public 

meetings in many locations across the country in an effort to engage a variety of stakeholders in 

the assessment program. The agency is now taking steps to improve participation at these 

meetings, including broadening participation through the use of Web conferencing, 

implementing a new meeting facilitation program, and using Web technology to interact with 

participants before, during, and after the meetings. 

The comment that the public meetings are a mockery and that the NRC has previously met with 

licensees in private is simply untrue. It may appear that the licensee is not surprised by the 

assessment being delivered at the meeting, but this is because the ROP is designed to be 

predictable; information being discussed has already been communicated in publicly available 

documents, such as inspection reports and assessment letters, to the extent allowed by 

Commission policy. 

(17) There are sufficient opportunities for the public to participate in the process. 

Respondent Comments: 

Public 

• See Item 16. NRC appears unwilling to consider viewpoints that are contrary established 

NRC positions and regulations. NRC competence in these technical areas becomes 

questionable. 
42 

• NRC provides a great deal of opportunity for public participation. However, given the 

complexity of the issues involved, to have truly informed public participation, NRC should 

consider funding public advocates for their time and to enable them to get access to the 

technical expertise they need to be truly effective advocates. 

Industry 

• The public has been afforded adequate opportunity to participate in most of the ROP 

and to provide inputs and comments by way of the public monthly ROP meetings, ROP 

feedback surveys, and the annual assessment public meetings. This is not the case 

however in the area of Physical Protection. The Physical Protection area of the ROP is 

not very open to the public, which may be appropriate in most cases; however, program 

and process changes should go through a change management process (similar to the 

ROP). 

• The STARS Alliance recognizes the NRC's efforts to continuously improve the ROP by 

way of the routine ROP public meetings and the periodic solicitation of public feedback. 

These interactions have assisted the ROP in effectively meeting the intended objectives, 

i.e., to provide tools for inspecting and assessing licensee performance in a manner that 

is more risk-informed, objective, predictable, and transparent. We support initiatives that 

continue to move the ROP to be more risk-informed and would oppose changes that 

would erode the risk-informed principles that the ROP was originally built upon. 

• FPL and NextEra endorse the continued monthly interactions between NRC and industry 

through the ROP Task Force as they are critical to continued improvement of the ROP 

and ensuring the process is as predictable and efficient as possible. 

State/Local Government Official 

• No, there should be many more opportunities. Even the 2.206 process is too difficult to 

get issues onto the table for greater evaluation. Almost all 2.206 petitions result in 

rejection. You do the math - then take a good hard look at the issues to see that there is 

a public that cares about nuclear safety. The 2.206 process should also be amended. As 

it stands now the process requires a team of lawyers to get past square one. There 

needs to be a better way for the common person to get concerns about safety issues 

into the NRC. 

NRC Employee 

• Probably too many. The NRC does not budget for all the public meetings and attendant 

travel, and at most sites, the public participation is almost non-existent. 

NRC Staff Response: 

Some respondents commented that the ROP public meetings are effective and the staff agrees. 

Many of the other comments regarding the effectiveness of the public meetings are addressed 

in the staff’s response to Question 16. 

A respondent suggested that the NRC fund the public advocates and grant them access to 

information. The NRC does not fund public advocates. However, the NRC makes information 

publicly available to the extent allowed by Commission policy. The public is welcome to 
43 

participate in the regulatory process through various venues, including public meetings. The 

staff will continue to provide publicly available information so that individuals, including public 

advocates, can formulate their own views on matters of importance to them. 

One respondent commented on the lack of publicly available information on Physical Protection, 

and that program and process changes should go through a change management process 

similar to other ROP cornerstones. IMC 0040, “Preparing, Revising and Issuing Documents for 

the NRC Inspection Manual,” describes the change management process for ROP guidance 

documents, including those related to physical protection. However, in accordance with 

Commission policy, there are provisions to ensure that individuals cannot obtain and use 

sensitive, security-related information about a nuclear facility’s design, operation, and protective 

capabilities for malevolent purposes, and this security-related information is withheld from public 

disclosure. 

The comment regarding the lack of NRC-budgeted resources for public meeting participation 

and travel is more globally addressed under the response to Question 8. 

The comment regarding the effectiveness of the 2.206 petition process is not within the scope of 

the ROP but has been forwarded on to NRR’s Division of Policy and Rulemaking for 

consideration. 

(18) NRC is responsive to public's comments and inputs on the ROP. 

Respondent Comments: 

Public 

• See Items 16 and 17. In addition, based on past comments provided on the ROP 

process and other safety issues, NRC has appeared unwilling to admit any opinions that 

differs from pre-established NRC positions 

Industry 

• The NRC for the most part has been responsive to public inputs and comments on the 

ROP. 

State/Local Government Official 

• Yes, but in one case when matrix time line was identified as the reason a plant does not 

get the critical rating there didn't seem to be a recommendation on how it could be 

changed. 

• No, the NRC has an answer to every question asked showing that the NRC is never 

wrong or in need of improvement. 

NRC Staff Response: 

The NRC encourages public participation, and actively solicits feedback from the public 

regarding the ROP through external surveys, public meetings, workshops, and other venues. 

Feedback from all stakeholders is evaluated during the annual ROP self-assessment process 
44 

and has resulted in a number of ROP improvements over the years. In addition, the NRC 

publishes a consolidated response to the survey comments after each external survey. 

Although the NRC does not always agree with or implement the suggestions or comments, the 

staff strives to explain the reasons why in the consolidated response. 

(19) The ROP has been implemented as defined by program documents. 

Respondent Comments: 

Public 

• Program documents are easily subject to misinterpretation, thus weakening the ROP. 

Industry 

• For the most part, the ROP is implemented as defined by program documents. On 

occasion, the staff has put a lot of weight on whether there was a performance 

deficiency when determining if a condition should count in the performance indicators. 

The existence of a performance deficiency is not a criterion in NEI 99-02 for determining 

whether a condition should count as a PI hit, and, in fact, many PI counts are not 

performance deficiencies. 

Industry is also concerned about apparent inconsistencies in number of findings, 

violations, and safety culture cross-cutting aspects issued across the four regions. We 

encourage NRC to continue efforts to ensure that the ROP is consistently implemented 

across the regions. 

NRC Staff Response: 

Regarding the comment that program documents are easily subject to misinterpretation, the 

staff strives to develop and implement ROP program guidance that is clear and concise to 

ensure that the ROP remains predictable, reliable, and understandable. The staff welcomes 

specific examples of these concerns so that it can evaluate them and determine if refinements 

or clarifications need to be made to the guidance. 

Regarding perceived inconsistencies in ROP implementation, the staff recognizes that there 

may be some variance between regions, and as noted in SECY-12-0055, the staff continued to 

implement the ROP reliability initiatives in 2011. The Deputy Regional Administrators initiated 

these activities to improve ROP implementation through sharing inspection resources, 

conducting Branch Chief benchmarking visits to other NRC Regions, discussing reliability 

topics, and assessing inspection report quality. 

Regarding the comment about determining whether a performance deficiency exists as applied 

to the PI program, the staff agrees that the existence of a performance deficiency is not a 

criterion for determining whether a condition should count in the PI program. The staff does not 

recall any specific instances where that point was debated, but welcomes further discussion of 

the concern if it persists. 

 
45 

(20) The ROP does NOT result in unintended consequences. 

Respondent Comments: 

Public 

• Accidents and near misses caused by a series of chain events, each relatively low-risk 

and harmless by itself, but catastrophic when linked, are not part of ROP philosophy or 

thinking. 

• Please define what is meant by "unintended" consequences. 

Industry 

• A potential concern for unintended consequences is raised by the NRC's planned reintegration of physical security into the action matrix. Due to the relatively high number 

of greater than green findings in the physical security cornerstone, re-integration poses a 

potential for an increased number of plants in the multiple/repetitive degraded 

cornerstone column. NRC’s current policy allows for flexibility in determining if a 

licensee should be placed in this column. We encourage the NRC to retain this flexibility 

with the re-integration of security into the action matrix. 

• SECY 99-007, “Recommendations for Reactor Oversight Process Improvements,” 

outlines the key objectives for the ROP as: 

a. Improve the objectivity of the oversight processes so that subjective decisions and 

judgment are not central process features. 

b. Improve the scrutability of these processes so that NRC actions have a clear tie to 

licensee performance. 

c. Risk-inform the processes so that NRC and licensee resources are focused on those 

aspects of performance having the greatest impact on safe plant operation. 

Unintended consequences result whenever actions taken by the NRC or licensees are 

not in full alignment with these objectives. In general, the ROP has been a success and 

has avoided unintended consequences. Several areas for improvement are listed below. 

1. Significant NRC and licensee resources are spent characterizing the significance of 

findings. The majority of these resources are focused on findings that have minimal risk 

significance. This result is inconsistent with the ROP objective to “focus resources on 

aspects of performance having the greatest impact on safe plant operation.” 

2. Responses to several questions above have discussed the need for improvement in 

the safety culture approach of the ROP. Removing the subjectivity of the SCCIs and 

replacing it with an integrated industry approach with robust NRC oversight (i.e., the 

industry’s safety culture initiative) should help. 

3. Problems exist in the interpretation of safety system functional failures. Modifications 

of NUREG-1022 presented in Draft Revision 3 would alter what constitutes reportable 
46 

SSFFs in a way that would undermine the basis for the current SSFF performance 

indicator. 

4. A desire to make the MSPI risk-based rather than risk-informed has unintended 

consequences resulting in wasted resources for little or no gain. NEI 99-02 Revision 6 

states (emphasis added): 

“Mitigating System Performance Index (MSPI) is the sum of changes in a simplified 

core damage frequency evaluation resulting from differences in unavailability and 

unreliability relative to industry standard baseline values.” 

“The MSPI is an approximation using information from a plant’s PRA and is intended as 

an indicator of system performance. More accurate calculations using plant-specific 

PRAs or SPAR models cannot be used to question the outcome of the PIs computed in 

accordance with this guideline.” 

At times the staff has lost sight of the fact that MSPI was designed to be simple and 

understandable. The paragraph that discusses licensees not being able to use their 

plant-specific PRA model and the NRC not being able to use the plant-specific SPAR 

model to challenge the output of the MSPI calculation was added specifically because it 

was recognized that we were calculating an approximation of the change in core 

damage frequency. 

SECY-99-007, “Recommendations for Reactor Oversight Process Improvements,” states 

the following: 

(Page 5): “An efficient oversight process is one that applies agency resources in a riskinformed manner.” 

(Page 6): “Risk-inform the processes so that NRC and licensee resources are focused 

on those aspects of performance having the greatest impact on safe plant operation.” 

Unfortunately, NRC and industry resources continue to be used on efforts attempting to 

improve the perceived accuracy of MSPI beyond what is practical with the current state 

of the PRA modeling. This diverts resources from investigating real improvements to the 

index. 

Monthly interactions between the NRC and industry through the ROP Working Group 

are critical to continued improvement of the ROP. The willingness to devote resources to 

these meetings is a clear indication of the NRC’s commitment to making the process as 

predictable and efficient as possible. 

Consideration should be given to updating bases documents for all of the performance 

indicators, similar to what was developed when the Scrams with Complications indicator 

was revised. Mastery of the body of knowledge on which the ROP is based is at risk 

because of attrition and turnover in the industry and the NRC. NEI and the industry are 

prepared to work with the NRC to address this challenge in 2012. 

 
47 

State/Local Government Official 

• Rather than answer this final question to this slanted survey, I offer this parting 

comment. The NRC likes to claim that they are independent. In many areas they are and 

would most likely argue that they are always independent, however, for example, in the 

decommissioning arena the industry has won a major victory over the NRC RIS 2001- 

07 Rev. 1. This RIS was seeking to get at the truth about the co-mingling of radiological 

decommissioning funds with those required for states portion for non-radiological 

decommissioning and site cleanup and restoration. The NRC had a great document 

prepared, but made the submission of data voluntary. In the Backfit Discussion it states 

that "Any response to this request for additional information is strictly voluntary." 

Because this RIS was voluntary critical information was not provided. If licensees would 

have provided this data the NRC would have clearly seen that not all of the funds were 

available for the radiological decommissioning, and that the actual funds would be 

proven deficient. The NRC would then have had to compel many licensees to contribute 

additional funds to the decommissioning accounts. Should the NRC wish to verify any of 

this they should merely look to see how many licensees actually provided the requested 

information. If the NRC has true regulatory authority over the nuclear industry, and is 

truly independent, it would have made submission of this information mandatory. It was 

not burdensome, could have been done in a simple letter and the truth would have been 

evident. 

NRC Staff Response: 

There is no official or standard definition of “unintended” consequences, but a basic explanation 

could be undesirable results that were not planned or anticipated and may adversely affect 

reactor safety or the overall effectiveness of the ROP. As a result of this feedback, the staff 

plans to add this explanation/example to the metric definition that governs this question in the 

upcoming revision to Appendix A, “Reactor Oversight Process Self-Assessment Metrics,” to IMC 

0307. 

One respondent commented on the potential for higher number plants in multiple/repetitive 

degraded cornerstone column when the security cornerstone is reintegrated back into the ROP 

Action Matrix. Prior to 2010, only one plant had entered the Degraded Cornerstone Column 

(Column 3) of the Security Action Matrix. However, in CY 2010, three plants transitioned to this 

column. If safety performance issues at these plants had simultaneously placed them in the 

Column 3 of the ROP Action Matrix, they would not have received the more comprehensive 

inspection under IP 95003, “Supplemental Inspection for Repetitive Degraded Cornerstones, 

Multiple Degraded Cornerstones, Multiple Yellow Inputs or One Red Input.” As such, the NRC’s 

current structure would require a deviation to ensure that the NRC is responding appropriately 

to licensee performance. Looking at historical data none of these plants had additional 

degraded cornerstones so the NRC’s regulatory response would not have changed if the 

security cornerstone was reintegrated. Furthermore, now that the security cornerstone is 

reintegrated, the NRC retains the flexibility to deviate from the Action Matrix if a licensee’s 

performance does not warrant the regulatory response dictated by the Action Matrix, regardless 

of column or cornerstone. 

One respondent commented on the significant resources spent on characterizing the 

significance of findings. The staff agrees that neither NRC nor licensee resources should be 

focused on findings that have minimal risk significance, and that they should instead be focused 
48 

on aspects of performance having the greatest impact on safety. All inspection findings are 

processed by the SDP, and the majority of those findings “screen to green” such that minimal 

effort should be expended to reach the final characterization. However, some detailed risk 

evaluations result in a risk characterization that is greater than green, and it is these issues that 

the NRC expects additional resources to be expended on. The staff recognizes that there can 

be instances where the licensee and/or NRC expend potentially unnecessary resources when 

characterizing the final significance of a finding, and the staff strives to use the best available 

information within the SDP timeliness guidelines to reach an informed and predictable 

determination. Nevertheless, as noted in the response to Question 7, the staff assembled a 

project team to conduct a review of the SDP, from a resource and timeliness perspective, to 

critically evaluate established practices and guidance, and to identify opportunities for 

improvement to the SDP and other impacted programs. 

The staff addressed the comments regarding SSFF and MSPI in its responses to Questions 1 

and 4. 

The comments regarding the need to improve safety culture and the subjectivity of SCCIs were 

addressed in the staff’s responses to Questions 9 and 10. 

The staff acknowledges the concern regarding updating the bases documents for the 

performance indicators for knowledge transfer purposes as noted in its response to Question 3. 

The comment regarding decommissioning funds is not within the scope of the ROP but has 

been forwarded on to NRR’s Financial Analysis and International Projects Branch for 

consideration. 
49 

ACRONYMS 

ADAMS Agencywide Documents Access and Management System 

ASSD Alternate Safe Shut Down 

CCA Cross-Cutting Aspect 

CDBI Component Design Bases Inspection 

CFR Code of Federal Regulations 

CY Calendar Year 

EPRI Electric Power Research Institute 

FAQ Frequently Asked Questions 

IMC Inspection Manual Chapter 

IP Inspection Procedure 

MSPI Mitigating Systems Performance Index 

NCV Non-Cited Violation 

NEI Nuclear Energy Institute 

NFPA National Fire Protection Association 

NRC U.S. Nuclear Regulatory Commission 

NSIR Office of Nuclear Security and Incident Response 

PI Performance Indicator 

PI&R Problem Identification & Resolution 

PRA Probabilistic Risk Assessment 

RASP Risk Assessment Standardization Project 

RG Regulatory Guide 

ROP Reactor Oversight Process 

SCCI Substantive Cross-Cutting Issue 

SCWE Safety Conscious Work Environment 

SDP Significance Determination Process 

SPAR Standardized Plant Analysis Risk 

SRA Senior Reactor Analyst 

SRM Staff Requirements Memorandum 

SSFF Safety System Functional Failure 

SSU Safety System Unavailability 
50 

REFERENCES 

Available from http://www.nrc.gov/reading-rm/doc-collections/: 

• IMC 0102, “Oversight and Objectivity of Inspectors and Examiners at Reactor Facilities” 

• IMC 0305, “Operating Reactor Assessment Program” 

• IMC 0307, “Reactor Oversight Process Self-Assessment Program” 

• IMC 0308, “Reactor Oversight Process (ROP) Basis Document” 

• IMC 0310, “Components within the Cross-Cutting Areas” 

• IMC 0608, “Performance Indicator Program” 

• IMC 0609, “Significance Determination Process” 

• IMC 0609, Attachment 1, “Significance and Enforcement Review Panel Process” 

• IMC 0612, “Power Reactor Inspection Reports” 

• IMC 0612, Appendix B, “Issue Screening” 

• IMC 0612, Appendix E, “Examples of Minor Issues” 

• IMC 2515, “Light-Water Reactor Inspection Program – Operations Phase” 

• IP 71111.06, “Flood Protection Measures” 

• IP 71111.21, “Component Design Bases Inspection” 

• IP 71152, “Problem Identification and Resolution” 

• IP 95002, “Inspection for One Degraded Cornerstone or Any Three White Inputs in a 

Strategic Performance Area” 

• IP 95003, “Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple 

Degraded Cornerstones, Multiple Yellow Inputs or One Red Input” 

• NUREG/CR-6850, “EPRI/NRC-RES Fire PRA Methodology for Nuclear Power Facilities” 

• NUREG-1022, “Event Reporting Guidelines: 10 CFR 50.72 and 50.73” 

• RG 1.200, Revision 2, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities” 

• SECY-09-0143, “Status of the Accident Sequence Precursor Program and the Standardized 

Plant Analysis Risk Models” 

• SECY-98-144, “White Paper on Risk-Informed and Performance-Based Regulation” 

• SECY-99-007, “Recommendations for Reactor Oversight Process Improvements” 

Available from http://www.nrc.gov/reactors/operating/oversight/program-documents.html: 

• RASP Handbook 

Not Publicly Available: 

• IMC 0609, Appendix E, Part I, “Baseline Security Significance Determination Process for 

Power Reactors” 

• IMC 0609, Appendix E, Part II, “Force-on-Force Physical Protection Significance 

Determination Process for Power Reactors” 

• SPAR Model Quality Assurance Plan 

• TI 2515/171, “Verification of Site Specific Implementation of B.5.b Phase 2 & 3 Mitigating 

Strategies”