Consolidated Response to the 

2009 Reactor Oversight Process External Survey 

Dated May 7, 2010 
CONTENTS 

INTRODUCTION ........................................................................................................................ 2

SURVEY QUESTIONS AND RESPONSES ............................................................................... 2

(1) Does the performance indicator (PI) program provide useful insights, particularly when 

combined with the inspection program, to help ensure plant safety and/or security? ....... 2

(2) Does appropriate overlap exist between the PI and the inspection programs to provide 

for a comprehensive indication of licensee performance? ............................................... 5

(4) Does the PI program effectively contribute to the identification of performance outliers 

based on risk-informed, objective, and predictable measures? ..................................... 10

(5) Does the inspection program adequately cover areas that are important to plant safety 

and/or security, and is it effective in identifying and ensuring the prompt correction of 

performance deficiencies? ............................................................................................ 13

(6) Is the information contained in NRC inspection reports relevant, useful, and written in 

plain English? ............................................................................................................... 15

(7) Does the SDP result in an appropriate regulatory response to performance issues? .... 16

(8) Does the NRC take appropriate actions to address performance issues for those plants 

outside the Licensee Response Column of the Action Matrix? ...................................... 20

(9) Is the information contained in NRC assessment letters relevant, useful, and written in 

plain English? ............................................................................................................... 21

(10) Do the ROP safety culture enhancements help in identifying licensee safety culture 

weaknesses and focusing licensee and NRC attention appropriately? .......................... 22

(11) Are the ROP oversight activities predictable (i.e., controlled by the process) and 

reasonably objective (i.e., based on supported facts rather than subjective judgment)? 27

(12) Is the ROP risk-informed in that the NRC's actions are appropriately graduated on the 

basis of increased significance? .................................................................................... 29

(13) Is the ROP understandable, and are the processes, procedures, and products clear and 

written in plain English? ................................................................................................ 31

(14) Does the ROP provide adequate assurance, when combined with other NRC regulatory 

processes, that plants are operated and maintained safely and securely? .................... 32

(15) Are NRC actions related to the ROP effective (e.g., are NRC actions of high quality, 

efficient, timely, and realistic to enable the safe use of radioactive materials)? ............. 33

(16) Does the ROP ensure openness in the regulatory process (e.g., does the NRC 

appropriately inform stakeholders in the regulatory process)? ....................................... 34

(17) Has the public been afforded adequate opportunity to participate in the ROP and to 

provide inputs and comments (e.g., does the NRC appropriately involve stakeholders in 

the regulatory process)? ............................................................................................... 36

(18) Has the NRC been responsive to public inputs and comments on the ROP? ................ 37

(19) Has the NRC implemented the ROP as defined by program documents? ..................... 38

(20) Does the ROP result in unintended consequences? ..................................................... 40

(21) Please provide any additional information or comments related to the ROP. ................ 43

ACRONYMS ............................................................................................................................. 47

REFERENCES ......................................................................................................................... 48
2 

INTRODUCTION 

In accordance with Inspection Manual Chapter (IMC) 0307, “Reactor Oversight Process [ROP] 

Self-Assessment Program,” the U.S. Nuclear Regulatory Commission (NRC) staff publishes a 

survey in the Federal Register biennially to obtain external stakeholder feedback on ROP 

effectiveness. The staff requests stakeholders to provide comments specific to ROP 

performance metrics. Stakeholders can also provide general comments on the ROP. The staff 

maximizes awareness of the survey’s availability by mailing hundreds of surveys directly to 

stakeholders, placing a direct link to the survey information on the ROP and “Documents for 

Comment” Web pages of the NRC’s external Web site, and issuing press releases. The staff’s 

responses to survey respondents’ comments are available on the “ROP Program Evaluations 

and Stakeholder Feedback” Web page of the NRC’s external Web site. 

On September 25, 2009, the staff published in the Federal Register a solicitation for comments 

on ROP implementation (Agencywide Documents Access and Management System (ADAMS) 

accession number ML092590469). The comment period expired on November 6, 2009. The 

staff received five responses from the nuclear power industry and its representatives; however, 

the staff did not receive any responses from state or local agencies, public interest groups, or 

members of the public. The following list contains the survey respondents and the ADAMS 

accession numbers of the official record copy of the comments. 

• Southern Nuclear Operating Company (ML093140305) 

• Nuclear Energy Institute (NEI) (ML093140556) 

• Regional Utility Group IV (ML093140557) 

• Strategic Teaming and Resource Sharing (ML093140558) 

• Dennis Mosebey, Wolf Creek Generating Station (ML093290157) 

This document contains the NRC staff’s response to the survey respondents’ comments. The 

staff consolidated respondents’ comments and organized the NRC’s response to those 

comments by survey question. The staff attempted to document stakeholder comments exactly 

as they were received; therefore, the staff did not change the spelling, grammar, punctuation, or 

content of the comments. The staff’s responses are located after the survey respondents’ 

comments. 

SURVEY QUESTIONS AND RESPONSES 

(1) Does the performance indicator (PI) program provide useful insights, particularly when 

combined with the inspection program, to help ensure plant safety and/or security? 

Respondent Comments: 

Southern Nuclear Operating Company 

The thresholds for the performance indicators should be re-evaluated given the amount of time 

the ROP has been in place and determine if the thresholds are where they should be. 

The rolling 36 month period for the failure to count against the indicator is too long and should 

be changed to a rolling 12 month period to be consistent with the inspection findings. 
3 

Nuclear Energy Institute

The Performance Indicators (PIs) are performance standards that the industry strives to exceed. 

Since the PIs are based on NRC defined acceptable limits, they reinforce industry and Licensee 

safety performance. Industry has striven to improve its performance in PIs, as evidenced by PI 

results. Performance outside the licensee response band (“green”), given the outstanding 

performance of industry in PIs, provides a very useful insight. It is important to keep in mind that 

the PIs are used in conjunction with inspection findings in ensuring plant safety and security. 

Program enhancements have been made to the PIs in the Initiating Events Cornerstone to 

better define Scrams that challenge the operators and the Mitigating Systems Cornerstones to 

help maintain a risk informed focus on equipment and plant operations that are important to 

plant safety. The Mitigating Systems Performance Index (MSPI) PIs, while more risk informed 

than many of the other NRC PIs, tends to be more complex. Efforts should be made to maintain 

the indicators simple to understand and manage but at the same time provide meaningful 

indication. Proposed changes to the MSPI that add complexity in order to convert the indicator 

into a precise measurement, and which do not appreciably change the result, should be 

avoided. 

Regional Utility Group IV

In general, the performance indicator (PI) program provides useful insights to ensure plant 

safety and security. Since the last NRC request for comments in 2007, the revision to the PI for 

Unplanned Scrams with Complications (replacing the Scrams with Loss of Normal Heat 

Removal PI) has resulted in a more-risked informed indicator. 

Licensees focus attention and resources on the underlying issues that result in good 

performance in the PI program. Thus, continued improvement in the industry-wide PI results 

should be viewed as a successful outcome of the reactor oversight process (ROP). The 

industry supports changes to the PI program that result in fundamental improvement in the way 

licensee safety performance is monitored. However, the NRC should limit attempts to change 

PI program guidance or interpretation if those changes are mainly intended to produce 

additional instances of performance outside the green band. For example, recent discussions in 

ROP public meetings have included the potential for changes to the guidance for the Safety 

System Functional Failures PI and the Mitigating Systems Performance Index (MSPI) PIs. Care 

should be taken to ensure that these potential changes be evaluated against the goal of 

fundamental improvement in the process for measuring performance. 

Strategic Teaming and Resource Sharing 

The Performance Indicators (PIs) have developed into performance standards that the industry 

strives to exceed. Since the Pis are based on NRC defined acceptable limits, they reinforce 

industry and licensee safety performance. Program enhancements have been made to the PIs 

in the Initiating Events Cornerstone to better define Scrams that challenge the operators and the 

Mitigating Systems Cornerstones to help maintain risk-informed focus on equipment and plant 

operations that are important to plant safety. The Mitigating Systems Performance Index (MSPI) 

PIs, while more risk-informed than many of the other NRC Pts, tend to be more complex. Efforts 

should be made to make the indicator simpler and easier to understand and manage but, at the 

same time, maintain meaningful indication. Changes to the indicators that add complexity in 

order to convert the indicator into a precise measurement should be avoided. With the possible 

exception of the MSPI, we believe the PIs are straightforward and, therefore, can be understood 

by the general public. 
4 

Dennis Mosebey, Wolf Creek Generating Station 

Generally yes but the MSPI indicator is way too labor intensive for the value gained. Consider 

deleting it and if a related reliability indicator is needed, all one needs to do is divide 1 by the 

unplanned unavailability which used to be reported. 

Regarding the SSFI indicator. It is stated the staff feels this is a strong precursor indicator, but in 

many instances it can duplicate information in the MSPI. For instance if a plant has to shut down 

due to the unavailability of both Safety Injection pumps, then this is an entry into Tech Spec 

3.0.3 and hence an LER which is then also counted in the SSFI. The SSFI needs to be 

eliminated as the crosscutting issues of human performance and or corrective action can also 

be used to pick up this facet. In other words it is double jeopardy and redundant. 

NRC Staff Response: 

Most respondents commented that the PI program provides useful insights to ensure plant 

safety and security. Respondents acknowledged improvements made to the PI program. The 

staff agrees that the PI program is generally successful and provides useful insights into plant 

performance. 

Some respondents were supportive of reevaluating PI thresholds, but others cautioned that the 

NRC should not change PI program guidance or interpretation in order to produce additional 

instances of PIs crossing safety thresholds. The staff assesses the PI program to ensure it 

continues to provide useful insights into licensee performance. The staff discusses proposed PI 

changes with the industry during the ROP monthly public meetings and is interested in hearing 

specific proposals related to the adequacy of PI thresholds. The staff is also conducting public 

meetings to discuss a framework for potential new PIs. 

Some respondents recommended changes to the mitigating systems performance index 

(MSPI). A respondent suggested replacing the MSPI with a modification of the safety system 

unavailability (SSU) PI, which was withdrawn from the ROP. The staff will not implement this 

suggestion because the MSPI was created based on the industry and staff's desire to address 

deficiencies with the SSU PI and to better risk-inform the ROP. The SSU PI was not riskinformed and did not reflect the actual availability of the monitored system. A respondent 

suggested reducing the MSPI rolling three-year period for counting failures to a rolling one-year 

period. The staff notes that the MSPI provides performance information by averaging 

unavailability and unreliability over a three-year period. This period of time is necessary to gain 

insights into unreliability because shorter monitoring periods would not enable some plant 

components to yield sufficient performance data to provide valid and reliable indication. 

Some respondents stated that changes to the MSPI should make it easier to understand, 

manage, and maintain meaningful indication rather than increase its complexity without a 

corresponding increase in safety. The staff understands that tabulating and tracking the MSPI 

is a considerable effort; however, this effort yields a significant safety benefit by identifying 

declining performance. Changes to the MSPI should result in an improvement in safety. The 

recent MSPI changes resulted in data interpretations that produce more accurate safety 

characterizations. The staff will continue to discuss any future efforts for MSPI improvement in 

the ROP monthly public meetings. 
5 

A respondent indicated that the safety system functional failures (SSFF) PI and event reporting 

requirements duplicate the MSPI and some of the cross-cutting areas. The staff does not agree 

that the SSFF PI is redundant to the MSPI. These PIs are different in purpose and design and 

complement each other. The SSFF PI is deterministic in nature, includes a wider scope for 

monitoring plant systems, and is based on the loss of a defined safety function. The SSFF PI 

may be a leading indicator of declining plant performance, as noted in SECY-99-007, 

“Recommendations for Reactor Oversight Process Improvements,” because a trend in failures 

could be indicative of larger problems. The staff also does not agree that the SSFF PI is 

redundant to the cross-cutting areas. Although the SSFF PI and cross-cutting areas can both 

indicate problems that are not yet risk-significant, cross-cutting aspects (CCAs) are not 

equipment failures or performance deficiencies; rather, as discussed in IMC 0305, “Operating 

Reactor Assessment Program,” CCAs are performance characteristics that are the most 

significant contributors to performance deficiencies that resulted in findings. CCAs provide 

insights into performance characteristics that could affect multiple ROP cornerstones of safety 

and can only apply to PI-related equipment failures if the failures had associated inspection 

findings. 

(2) Does appropriate overlap exist between the PI and the inspection programs to provide 

for a comprehensive indication of licensee performance? 

Respondent Comments: 

Southern Nuclear Operating Company

Shifting columns to the right in the action matrix as a result of white findings that are of low to 

moderate safety significance can result in re-allocation of resources away from more safety 

significant priorities. 

Nuclear Energy Institute

PIs look at the areas where clear performance thresholds have been developed. As envisioned 

in the development of the ROP (see SECY 99-007), this allows the inspection program to look 

at cornerstone attributes not covered by the PIs, and to spend more time looking at those areas 

that require more evaluation and investigation. The process is well integrated and some overlap 

does exist – in some cornerstones more than others. 

In some cases, specifically in the Initiating Events and Mitigating Systems cornerstones, the 

inspection overlap can be excessive. This is especially noticeable in the Problem Identification 

and Resolution (PI&R) inspections and large team inspections such as the Component Design 

Bases Inspections (CDBI) where an inordinate amount of inspection effort is focused on events 

and issues reported under the performance indicator program. In addition, CDBIs very rarely 

have anything other than a few Green findings. This is not worth the considerable resources 

necessary for the NRC and licensees to participate in these inspections. 

At times, there is too much overlap. On several occasions, the staff has put a lot of weight on 

whether there was a performance deficiency when determining if a condition should count as an 

MSPI failure. The existence of a performance deficiency is not listed as a criterion in NEI 99-02 

in determining if a condition is an MSPI failure. (And, in fact, many PI counts are not 

performance deficiencies.) 

Regional Utility Group IV
6 

The process is well integrated and appropriate overlap does exist – in some cornerstones more 

than others. By their nature, PIs are useful for areas that measure licensee performance 

quantitatively, rather than by evaluating individual events. In theory, this would allow the 

inspection program to spend more time looking at those areas that require more evaluation and 

investigation. In the FAQ process, the NRC has sometimes provided interpretations of the PI 

guidance that hinge on the existence of licensee performance deficiencies during individual 

events. The industry believes that the PI program guidance should not be based on 

interpretations of licensee performance during individual events. The inspection program is the 

fundamental process for evaluating the significance of licensee performance on individual 

events and issues. 

Strategic Teaming and Resource Sharing

PIs look at the areas where clear performance thresholds have been developed. This allows the 

inspection program to spend more time, looking at those areas that require more evaluation and 

investigation. The process is well integrated and overlap does exist - in some cornerstones 

more than others as would be expected. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes I think so. 

NRC Staff Response: 

Most respondents commented that the PI and inspection programs are well-integrated and they 

expect the existing overlap in the programs. 

Some respondents commented that licensee performance during events should not be an input 

to PI program guidance. The staff notes that the criteria and definitions for PI inputs are 

provided in NEI 99-02, “Regulatory Assessment Performance Indicator Guideline.” Timely 

determinations of operability, functionality, and availability of plant equipment are important 

performance aspects of reporting accurate PI data. To verify accurate reporting of PI data, NRC 

inspectors need to understand licensees’ evaluations of equipment problems and decisions for 

(not) reporting a PI. If the industry believes the NRC is incorrectly requiring licensee 

performance deficiencies as PI inputs, the staff welcomes specific comments and observations 

and will implement the frequently asked question (FAQ) process described in NEI 99-02 and 

IMC 0608, “Performance Indicator Program,” if applicable, to address such issues. 

A respondent commented that the large team inspections, such as the problem identification 

and resolution (PI&R) inspections and CDBIs, focus too much on events and issues reported 

under the PI program. The staff notes that the PI and inspection programs work together to 

monitor plant safety and licensee performance, and the inspection program allows inspectors to 

assess licensee performance associated with PI inputs. The CDBI and PI&R inspection 

procedures (IPs) provide guidance for reviewing licensee corrective action program documents, 

which may be associated with PI-related issues. 

A respondent indicated that because the component design bases inspections (CDBIs) rarely 

produce findings having greater-than-green safety significance, the considerable resources 

allocated to participating in these inspections are inappropriate. The staff believes that these 

inspections are important, and resources are well spent, because of the insights provided into 

licensee performance, as evidenced by engineering-related findings resulting from these 
7 

inspections. The staff is enhancing the CDBI to improve its effectiveness by focusing on 

improved component sample selection by reviewing various licensee programs and using 

operating experience. The staff plans to conduct an initial inspection during calendar year 

(CY) 2010, with full implementation of the new engineering inspection in CY 2011 if it is 

determined to be effective. 

A respondent commented that shifting columns to the right in the Action Matrix as a result of 

findings having low to moderate (i.e., white) safety significance can result in allocation of 

resources away from more safety-significant priorities. The NRC will take action beyond the 

baseline inspection program, such as increased applied resources, inspection scope, and 

management oversight, if licensee performance declines, as demonstrated by the quantity and 

significance of inspection findings and crossed PI thresholds. White findings, which are safetysignificant, warrant increased agency and licensee resources to analyze and correct. 

(3) Does NEI 99-02 provide clear guidance regarding PIs? 

Respondent Comments: 

Southern Nuclear Operating Company

The guidance is relatively clear, but there needs to be some dialogue with the industry regarding 

the criteria for crossing thresholds and the time in which an indicator must remain despite 

having all corrective actions completed and excellent plant operation. However, the definitions 

for the boundary of a monitored component are not clear in all cases. 

Nuclear Energy Institute

While questions on the guidance do arise, a formal Frequently Asked Question (FAQ) process 

is available to the industry and NRC inspectors to resolve questions with the guidance. An 

industry task force (the Reactor Oversight Process Task Force or ROPTF) meets monthly with 

the inspection and assessment branches of the NRC to discuss and resolve the questions that 

arise with the guidance. When resolution is not achieved at the monthly Reactor Oversight 

Process (ROP) meetings, an FAQ appeal process is available and has been used to drive 

issues to resolution. The FAQ process and appeal process have proven to be effective and 

should be maintained. 

Because the MSPI basis document may not consider all situations and available paths, coupled 

with the appropriate rigidness of the MSPI reporting requirements, plants may, on occasion, 

achieve an MSPI result that is not consistent with actual performance parameters and is thus 

not “risk informed,” nor “objective.” 

When an indicator change is the result of a methodology, treatment selection, or bookkeeping 

error there should be an avenue to identify and address this consideration in terms of reporting 

other than moving a plant from column 1 to column 2 or 3 in the Action Matrix. 

Finally, the guidance is appropriately very prescriptive in terms of what data elements feed into 

the performance indicator algorithms. If data is not available to produce a performance indicator 

result, no result should be posted to the NRC website unless the guidance specifically 

addresses an approved method to produce such an indicator. This did not appear to be the 

case when Browns Ferry was returned to service after being shut down for several years. Early 
8 

discussion of these types of situations should be conducted to avoid violating the ROP principle 

of predictability. 

Regional Utility Group IV

While questions on the guidance do arise, a formal FAQ process is available to the industry and 

NRC inspectors to resolve issues. When resolution is not achieved at the monthly ROP 

meetings, an FAQ appeal process is available and has been used to drive issues to resolution. 

The FAQ process and appeal process have proven to be effective and should be maintained. 

The NRC should resist changes that increase the complexity of the PI guidance. For example, 

recent and pending changes to the MSPI guidance have the potential to significantly increase 

the complexity of an already-difficult indicator. Additionally, recent public meetings have 

discussed the potential for changes to the Safety System Functional Failures PI to decouple this 

guidance from the guidance for determining safety system functional failures contained in 

NUREG-1022. These changes should only be implemented if they will result in a clear 

improvement in the ability to measure licensee safety performance. 

Strategic Teaming and Resource Sharing

While questions on the guidance do arise, a formal Frequently Asked Question (FAQ) process 

is available to the industry and NRC inspectors to resolve questions with the guidance. An 

industry task force (the Reactor Oversight Process Task Force) meets monthly with the 

Inspection and Assessment Branch of the NRC to discuss and resolve the questions that arise 

with the guidance. When resolution is not achieved at the monthly ROP meetings, an FAQ 

appeal process is available and has been used to drive issues to resolution. The FAQ process 

and appeal process have proven to be effective and should be maintained. As appropriate, 

outcomes from these processes are used to improve NEI 99-02. 

There are areas of the guidance that need refinement and more clarity, particularly in the 

Initiating Events Cornerstone. The difficulties center around differences in how plants are 

designed to react to automatic power reductions and trips. One possible way to improve the 

indicators under the Initiating Events Cornerstone would be to better integrate unique plant 

design considerations into the Pis such that a plant's mitigating capabilities are credited or 

better accounted for. Another possible way to improve the indicators under Initiating Events 

Cornerstone would be to integrate risk insights into the indicator. The current indicators count 

events with little consideration for actual realized core damage risk associated with the events. 

Finally, the guidance is very prescriptive in terms of what data elements feed into the PI 

algorithms. For new plants and plants returning to service after being shut down for several 

years the existing PI guidance is not reflective of a plant's performance (see response to 

question 18 for additional information). 

Dennis Mosebey, Wolf Creek Generating Station 

For the most part yes, though the MSPI section is somewhat spread out with part in Main Text 

and then Appendices? It would be nice to have it all in one place when referencing it. 

NRC Staff Response: 

Respondents generally commented that NEI 99-02 guidance is clear and the FAQ process and 

ROP monthly meetings are effective in resolving issues with the guidance. The staff agrees 

with these comments. 
9 

A respondent commented that the NRC needs to engage the industry regarding the criteria for 

crossing PI thresholds, the amount of time PI data remains in effect, and the definitions of 

boundaries of monitored components. The staff and industry address issues such as these 

during the ROP monthly public meetings. The staff welcomes specific observations and 

suggestions for improvement. A respondent also commented that the MSPI guidance located 

throughout NEI 99-02 should be consolidated into one location. The staff intends to forward the 

comment to NEI for consideration. 

A respondent commented that an MSPI result may not be consistent with actual performance 

parameters because of the content of the MSPI basis document. The staff encourages 

licensees to notify NRC staff during the ROP monthly public meetings of these discrepancies so 

that the staff and industry can initiate the process for modifying the indicator, if applicable. 

During the January and March 2010 ROP monthly public meetings, the staff discussed with the 

industry the need to routinely update MSPI basis documents to reflect plant changes. The staff 

and industry are developing proposals for revisions to NEI 99-02 to address this issue. 

Some respondents commented the NRC should resist changes to the MSPI and SSFF PI 

guidance that increase its complexity. The staff may need to change the MSPI to ensure it is 

effectively identifying declining performance; however, the staff does not intend to increase its 

complexity. The staff intends to ensure that inspectors and the industry correctly apply SSFF PI 

guidance and the guidance in Revision 2 of NUREG-1022, “Event Reporting Guidelines: 

10 CFR 50.72 and 50.73,” (ML003762595), because the reporting requirements of Title 10 of 

the Code of Federal Regulations, Part 50, Section 73 (10 CFR 50.73), “Licensee event report 

system,” affect inputs to the SSFF PI. The staff has discussed this effort with the industry 

during ROP monthly public meetings and plans to host public meetings to develop a revision to 

the event reporting guidance in NUREG-1022. 

A respondent commented that incorporating risk insights and plant-specific design information 

into the PI and bases documents would improve PIs. The staff notes that integrating unique 

plant designs and risk insights into the Initiating Events Cornerstone PIs might add value for 

accurately capturing individual plant performance. The staff is interested in discussing these 

risk-informed options and reviewing proposals of how this effort could be implemented with the 

industry during the ROP monthly public meetings. 

Some respondents commented that the effect of new plants and plants returning to service from 

extended shutdowns should be incorporated into PI algorithms. The staff agrees that 

discussions are necessary regarding new plants and plants returning to service after extended 

shutdowns and is interested in discussing this topic during the ROP monthly meetings or other 

public meetings. 

A respondent commented that when a crossed PI threshold results from a methodology, 

treatment selection, or bookkeeping error, there should be an avenue to identify and address 

these considerations in terms of reporting rather than changing a plant’s Action Matrix column 

designation. The staff notes that if Action Matrix movement resulted from a licensee’s 

bookkeeping error, actions can be taken to correct the licensee’s Action Matrix placement. If 

the movement resulted from some other unique situation, the agency may take no action, or it 

can consider deviating from the Action Matrix to ensure an appropriate level of regulatory 

response. 
10 

(4) Does the PI program effectively contribute to the identification of performance outliers 

based on risk-informed, objective, and predictable measures? 

Respondent Comments: 

Southern Nuclear Operating Company

The indicators at times are too rigid and do not allow enough flexibility to take credit for 

compensatory actions that do not increase risk. 

Nuclear Energy Institute

The Performance Indicator Program effectively identifies, in conjunction with the Inspection 

Program, performance outliers based on risk-informed, objective and predictable measures. 

There has been significant discussion within the NRC questioning the efficacy of the 

performance indicator program because the results are “too green” and are no longer providing 

value. PI thresholds were initially set based on a careful analysis in SECY 99-007 

demonstrating that the overall performance of industry had dramatically improved in the 1990s 

and that, with the occasional exception, plants were safe enough. The Commission agreed with 

this conclusion. The thresholds therefore were set at levels that reflected outliers to the overall 

acceptable safety levels. Since the inception of the ROP, performance in the PIs has continued 

to improve in almost all of the indicators such that most of the indicators are now green. This 

improvement has been driven by at least three causes: 

1. When the NRC set performance expectations (green/white thresholds) the industry 

responded by placing emphasis on activities to improve their performance. (The aphorism, 

“what gets measured, gets improved,” applies.) 

2. The industry set internal higher standards and developed trending to ensure that they would 

not, if at all possible, exceed the thresholds. Thus the existing PI thresholds have resulted in 

even higher standards by the licensees. 

3. Licensees have taken action to understand the performance indicators and make 

modifications and improvements in training, programs, procedures, calculations, and risk 

analyses at the plants to improve plant performance and reduce risk. Some plants have 

specific programs that focus site attention in areas with specific goals to improve equipment 

availability and reliability and improve plant performance. Examples of actions taken include: 

• The addition of station blackout EDGs. This reduced a plant’s baseline cdf by over 50%. 

• Platforms were installed to provide access and a procedure was prepared to provide for 

manual manipulation of some valves. Actions and modifications improved MSPI margin 

for cooling water 

• Implemented Modification adding small generators to maintain battery power and heat 

SF6 gas allowing realignment of the plant to the grid following a station blackout event 

lasting more than 4 hours. Modification will improve MSPI margin in the area of EAC. 

• Created procedure to allow manual manipulation of service water valves supporting the 

Component Cooling Heat Exchanges. Procedure improved MSPI margin for Cooling 
11 

Water Systems. 

• Modifications to change out solenoid operated valves with more reliable motor operated 

valves in the auxiliary feedwater system. 

• For challenges with river grassing, procedures were revised and screen modifications 

are being implemented to reduce the risk of river grassing as an initiating event. Actions 

and modifications have resulted in improvement to the Initiating Events Cornerstone. 

• Performed an analysis and developed a procedure to support cooling of the Control 

Room Ventilation during a loss of cooling. This change reduces overall plant risk and 

improved MSPI margin in Service Water 

• “Just in Time” training provided to operators to reduce and mitigate the effects of known 

vibration problems during startup with new mono-block main turbine rotors. 

• Procedure changes made to anticipate and mitigate adverse environmental effects on 

plant operations. 

• A plant performed a single point vulnerability study and as a result, implemented 

numerous modifications to eliminate conditions where failure of a single component 

could lead to a unit trip. This impacts the Unplanned Scrams indicator. 

• The INPO/Industry initiative “Zero by Ten” (Zero Fuel Defects by 2010) is a great 

example of the industry working together to improve performance. This improvement 

would be reflected in the RCS activity indicator. 

• Several plants have modified their sirens to increase reliability. This impacts the ANS 

indicator. 

These modifications and improvements help drive good plant and equipment performance 

which is a desirable effect and tends to keep performance indicators green. 

Finally, perhaps a reason some at the NRC are concerned that the performance indicators are 

“too green,” is that the performance indicators don’t seem to be able to distinguish the good 

from the poor performers. There are two answers to that concern. First, over time, everyone’s 

performance has improved so that the variability between plants has shrunk. The delta between 

a top performer and an average or below average performer is no longer significant in the PI 

area. (This by the way is also true in the inspection finding area as well.) Second, NRC’s role is 

to ensure that the regulations are being met and that plants are safe. If that is the case, it does 

not matter that plants are all green; in fact it should be viewed as a measure of NRC success. 

That being said, we, NRC and industry, need to continue to be vigilant to identify declining 

performance. New tools to access performance should be explored and if appropriate, be 

implemented. It does not mean that the current process is deficient. 

Regional Utility Group IV

In general, the PI program effectively identifies performance outliers. The RUG IV member 

plants support continued improvements that further increase the objectivity and risk-informed 

nature of the PIs. As stated in the response to question 1 above, the improvements in industry 

performance that have resulted in improved PI results should be viewed as a success of the PI 

program. 

Strategic Teaming and Resource Sharing
12 

The PI Program does effectively contribute to the identification of performance outliers. Some 

PIs produce more visible trends than others. Visible trends are more clearly seen in the Initiating 

Events, Emergency Preparedness and Occupational Radiation Safety cornerstones and tend to 

be objective and predictable. However, not all PI results produce visible trends and not all are 

risk-informed. The MSPI is the first risk-based indicator and identifies conditions based on risk 

implications but does not typically produce a visible trend nor is it typically predictable because 

of the significant influence equipment failures have on the indicators. Most of the other PIs have 

limited risk insights and while they tend to be more predictable and indicate performance trends 

they do not provide quantifiable indication of risk to the public. 

Since the inception of the ROP, licensees have taken action to understand the PIs and make 

modifications and improvements in training, programs, procedures, calculations and risk 

analyses at the plants to improve plant performance and reduce risk. Typically plants have 

programs that focus site attention in areas with specific goals to improve equipment availability 

and reliability and improve plant performance. Examples of actions taken include: 

* Modifications made to increase the reliability of valves installed in the auxiliary feedwater 

system. 

* "Just in Time" training provided to operators to mitigate the effects of known vibration during 

startup with new mono-block main turbine rotors. 

* Procedure changes made to anticipate and mitigate adverse environmental effects on plant 

operations. 

These modifications and improvements help drive good plant and equipment performance 

which in turn tend to keep PIs green. Behavior that improves plant performance and results in 

better PI performance are a positive result of the PI program and should continue to be 

supported and encouraged by the program. 

Dennis Mosebey, Wolf Creek Generating Station 

For the most part Yes, but again I question the value of SSFI since it is truly not a risk informed 

item, it is just straight LER report count per quarter. Thus it really does not fit in a risk informed 

process. Also in the MSPI while the competition against one’s own CDF has value, the 

tabulation and labor required does not yield a significant benefit at all. 

NRC Staff Response: 

Most respondents commented that the PI program effectively contributes to the identification of 

performance outliers. Respondents commented that PI thresholds should not be changed in 

order to produce additional instances of crossed thresholds. The staff addressed this comment 

in its response to Question 1. The staff also addressed the complexity and safety benefit of the 

MSPI in its responses to Questions 1 and 3. 

Respondents commented that some PIs need to be more objective and risk-informed and 

consider credit for compensatory actions. A respondent commented that the SSFF PI was an 

example of a PI that is not risk-informed and doesn’t provide value. The staff agrees that new 

tools to assess performance should be explored and implemented, if appropriate. However, the 

current PIs are not deficient. The staff notes that PIs having very low (i.e., green) safety 

significance represent performance in which cornerstone objectives are fully met and additional 

NRC oversight is not required. PIs provide useful trending information and input to licensee 

performance assessment. As mentioned in its response to Question 3, the staff is willing to 

discuss the possibility of risk-informing certain PIs for more effective identification of declining 
13 

performance. The SSFF PI was used prior to the initial implementation of the ROP and is 

based on a regulation (i.e., 10 CFR 50.73) that is not risk-informed. However, the SSFF PI 

does reflect the failures of various safety systems and whether those systems can perform their 

intended safety functions, which adds valuable insight from the NRC staff’s perspective. 

(5) Does the inspection program adequately cover areas that are important to plant safety 

and/or security, and is it effective in identifying and ensuring the prompt correction of 

performance deficiencies? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes. The Inspection Program along with the Corrective Action Program ensures performance 

deficiencies are addressed promptly and thoroughly. However, at times it appears regulations 

are promulgated by inspection findings (forces licensees to take actions not necessarily required 

by existing regulation). 

Nuclear Energy Institute

The resident inspectors are usually effective in ensuring areas important to safety are 

appropriately addressed through the baseline inspection program. The inspection program and 

the ROP assessment methodology are effective in ensuring identified performance deficiencies 

are promptly corrected. However, the larger team inspections (such as the CDBI) have a 

tendency to inspect the same systems and re-inspect issues that have already been inspected, 

come up with very little useful information, and should be reviewed for improvement or 

elimination. 

Also, the inspection threshold used for identifying performance deficiencies is low and 

subjective. This diverts licensee resources from addressing actual causes and concerns that 

contribute to risk. 

One area that could be improved is resident inspector review of data that licensees submit to 

the NRC in the ROP. For example, there have been two different efforts this year where NRC 

consultants have identified several instances where they felt that licensees were reporting 

Safety System Functional Failures and the types (i.e Start, Load/Run, Run) of MSPI failures 

incorrectly. These efforts occurred long after the events or failures occurred, and have little 

relevance to current performance (ROP principle of “timely” assessment). This does not ensure 

prompt correction of performance deficiencies. The resident inspectors have the ability to look at 

the data reporting real time, and in fact have inspection hours assigned to look at PI data. The 

NRC should consider whether or not additional training is needed in this area. 

(Note that we are not opposing the need to conduct efforts to ensure data element definitions 

are clear and that data is being accurately reported.) 

Regional Utility Group IV

In general, the inspection program is effective in ensuring areas important to safety are 

appropriately addressed. However, the component design bases inspections, while initially 

providing useful insights into system design and performance, have tended to continue to focus 

on the same set of plant systems and may no longer be providing fresh insights at the same 

level of importance as in the initial rounds of these inspections. Also, while improvements have 
14 

occurred in the classification of radiation protection findings, the number of inspection hours in 

this area appears to be high compared to the number and significance of findings in this area. 

Strategic Teaming and Resource Sharing

The inspection program is effective in thoroughly inspecting areas for compliance and 

adequately covers the areas needed to address plant safety. The system of baseline and 

supplemental inspections appropriately increase the level of scrutiny provided based on the 

performance level of the licensee. Further refinement of larger team inspections (e.g., 

Component Design Basis Inspections) should be continuously evaluated for value added as 

they are a significant impact on both the licensees and the NRC. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes I feel it does. 

NRC Staff Response: 

All respondents provided positive comments regarding the overall effectiveness of the 

inspection program in identifying and ensuring the prompt correction of performance 

deficiencies. Several respondents questioned the value added from inspections performed in 

accordance with IP 71111.21, “Component Design Bases Inspection.” The staff addressed 

these comments in its response to Question 2. 

A respondent commented that the number of inspection hours spent during radiation protection 

inspections appears to be high compared to the number and significance of findings. The staff 

notes that as a result of the ROP realignment process, the staff revised all radiation safety IPs 

to provide a more performance-based inspection for each of the functional areas of a radiation 

safety program. The revisions may result in a reduction of inspection hours because the scope 

and completion requirements were clarified. The revisions also direct inspectors to focus 

attention on more risk-significant areas. The revisions took effect January 1, 2010, and the staff 

will continue to monitor the effectiveness of the procedures. 

A respondent commented that inspectors could use additional training regarding reviewing data 

that licensees submit to the NRC. The staff agrees that training related to SSFF reporting and 

10 CFR 50.73(a)(2)(v) compliance would be useful and provided such training to resident 

inspectors in CY 2009. The staff is currently reviewing MSPI emergency diesel generator failure 

mode definitions and discussing the issue with the industry during the ROP monthly public 

meetings. 

A respondent stated that it appears that regulations are promulgated by inspection findings and 

licensees are forced to take actions not necessarily required by existing regulations. The staff 

notes that in accordance with IMC 0102, “Oversight and Objectivity of Inspectors and Examiners 

at Reactor Facilities,” regional management assesses inspectors’ adherence to agency-wide 

regulatory positions and policies and avoidance of personal interpretations and opinions. 

However, inspectors may identify safety issues that the agency determines require backfitting in 

accordance with 10 CFR 50.109, “Backfitting.” IMC 0612, “Power Reactor Inspection Reports,” 

instructs staff to cite the regulatory requirement against which a violation is written. If a licensee 

believes the regulatory requirement is not part of its current licensing basis, the licensee should 

discuss this with inspectors and provide information that would alert them to a potential backfit. 

NRC inspection report cover letters also have provisions for licensees to contest violations. 
15 

A respondent commented that the threshold for identifying performance deficiencies is low, 

subjective, and diverts licensee resources from addressing issues that contribute to risk. In 

response to Question 7, some respondents also indicated that the process for screening 

performance deficiencies for more-than-minor significance is subjective and inspectors’ 

assumptions used in this process are not clearly documented. The staff notes that a measure 

of subjectivity in issue screening is anticipated and accepted as no completely objective or 

mechanistic process has been identified that can satisfy the objectives of the ROP. However, 

the performance deficiency determination is only the first step in a process that establishes 

multiple barriers against inappropriate diversion of licensee resources. These process steps, 

barriers, and their bases are articulated in IMC 0308, “Reactor Oversight Process (ROP) Basis 

Document,” IMC 0612 and its Appendix B, “Issue Screening,” and IMC 0609, “Significance 

Determination Process [SDP].” In addition, inspection findings are evaluated by regional 

management to determine whether they warrant inclusion in inspection reports and to improve 

reliability in the decision-making process. Findings are also sampled and reviewed to gauge 

conformity with IMC 0612 guidance. The staff has identified areas for improvement in guidance 

and inspector understanding and has embarked upon a series of activities to enhance ROP 

guidance. Those activities, which include revising IMC 0612 and its Appendix E, “Examples of 

Minor Issues,” are ongoing. The staff revised Appendix B of IMC 0612 to improve its clarity and 

the screening process and conducted regional training in December 2009 to improve 

consistency in screening issues of concern. Stakeholders can provide recommendations and 

comments to the staff during the monthly ROP public meetings, and the staff will consider these 

recommendations for improving ROP processes. Specific examples can be very helpful in 

illustrating areas for improvement. 

(6) Is the information contained in NRC inspection reports relevant, useful, and written in 

plain English? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes. The reports are clear and relevant. 

Nuclear Energy Institute

The NRC inspection reports are relevant, useful and well written. 

Regional Utility Group IV

The NRC inspection reports are relevant, useful, and well written. 

In some cases, the NRC’s process for refining or revising findings between the plant exit 

meeting and the issuance of the inspection report needs improvement. During the inspection 

efforts, the inspection program appropriately allows for licensee input in characterizing a finding 

and any related determinations, such as safety culture aspects. The same principle should 

apply when the NRC is considering changes in finding characterization following the exit 

meeting with the licensee. In many cases, the NRC does communicate with licensees in these 

situations; however, this practice is sometimes not followed, and the NRC should consider 

reinforcing this expectation. 

Strategic Teaming and Resource Sharing
16 

The NRC inspection reports are relevant, useful and well written. 

Dennis Mosebey, Wolf Creek Generating Station 

Generally yes. 

NRC Staff Response: 

All respondents commented that inspection reports contained relevant, useful, and well-written 

content. A respondent commented that the NRC does not always inform the licensee of 

changes in finding characterizations made between the inspection exit meeting and the 

issuance of the inspection report. The staff notes that Section 12.01 of IMC 2515, “Light-Water 

Reactor Inspection Program – Operations Phase,” states that changes to the characterization of 

issues after the initial exit meeting will be communicated to the licensee prior to the issuance of 

the inspection report. IMC 0612 also reinforces that inspection report content should not conflict 

with information presented at the exit meeting. If a licensee becomes aware of NRC staff not 

following these requirements, the staff welcomes specific licensee’s observations and 

comments so that the staff can address the concerns. 

(7) Does the SDP result in an appropriate regulatory response to performance issues? 

Respondent Comments: 

Southern Nuclear Operating Company

The change in the columns in which a licensee finds themselves can result in low to moderate 

safety findings moving a licensee to Column III in the Action Matrix. 

Very low level NCVs that are assigned cross cutting aspects seem to be severe. There is no 

credit given to the Corrective Action Process to resolve the issue and the 3 aspect categories 

don't fully model all elements of safety culture. 

Nuclear Energy Institute

Not consistently. The assumptions used by the NRC in the Significance Determination Process 

(SDP) are at times subjective and arbitrary. Examples include the amount of penalty a licensee 

must take for common cause and the amount of credit allowed for human performance 

(operator actions). 

Another area where the SDP process is overly restrictive is the area of fire protection/safe 

shutdown. This problem will be exacerbated by conservative PRA methods being required by 

NRC for plants implementing NFPA 805. These methods will provide exaggerated fire risk 

values that are not comparable to existing internal events results, and a method to account for 

this inherent bias will need to be developed. Otherwise, erroneous conclusions are likely to be 

reached. The SDP guidance must be updated and regional staff training/understanding must be 

improved in this area to ensure accurate and consistent results are achieved. 

Industry encourages the use of licensees’ PRA models that have been evaluated to R.G. 1.200 

to support the SDP process as they become available. The NRC could continue maintain a 

degree of independence by using a verification process modeled after the PI verification 

process. The current practices lack transparency and scrutability and in some cases result in 

assigning higher significance to an issue than is warranted. 
17 

Regional Utility Group IV

The current Significance Determination Process (SDP) process lacks transparency and 

objectivity, and in some cases results in assigning higher significance to an issue than is 

warranted. This is largely due to the use of NRC-developed probabilistic risk assessment (PRA) 

models that are not as sophisticated as licensee models, and also due to subjective 

determinations regarding specific assumptions, such as common cause and human reliability 

factors. The RUG IV member plants encourage the use of licensee models for determination of 

the risk associated with findings. Licensee PRA models that achieve compliance with 

Regulatory Guide 1.200, Revision 2 will be the most complete and accurate tools for assessing 

the significance of events. The NRC could allow use of licensee PRA models for SDPs and 

continue to maintain a degree of independence by using a verification process modeled after 

the PI verification process. 

Strategic Teaming and Resource Sharing

There has been some improvement in the NRC guidance for determining minor findings (MC 

0612, Appendix E), however the Screen for More than Minor (MC0612, Appendix B) can be 

subjective depending on how the questions are answered. In particular, MC 0612 Appendix B 

Section 1-3, items c.2 and c.4 are cited predominantly as the basis for a more than minor 

violation. These criteria are quite subjective and often are applied very generally. For example, 

under item c.4 the justification that the cornerstone objective has been adversely affected is 

rarely provided. In addition, there is not a common understanding between licensees and 

inspectors on the application of the phrase "the cause was reasonably within the licensees 

ability to foresee and correct" from the MC 0612 definition of performance deficiency. 

For the most part, the Significance Determination Process (SDP) does result in the appropriate 

regulatory response to performance issues. In some cases, the assumptions used in the 

process are subjective and/or arbitrary or at least are not well supported technically. The SDP 

for Security continues to be more subjective and unevenly applied and because of the extensive 

changes made in the New Part 73 rule, which has resulted in the NRC developing many more 

attributes, there is a high likelihood that findings will be considered more significant (higher 

points) just because there are so many more attributes. The STARS alliance encourages good 

dialogue between the NRC senior reactor analysts and the licensees PRA staff to gain a 

thorough understanding of the bases for outcomes from the SDP. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes except recently my plant got a green noncited violation for several valves with dry boric acid 

in minimal amounts on them. These were manual valves and the safety significance was low, 

yet it was classified Green Non Cited vice minor. I feel inspectors still do not play fair with this 

and I also feel that there is still heavy emphasis on determinism in the NRC vice a true 

acceptance of risk informed processes. In the field in real life I just do not see it from the 

Residents and also sometimes from outside inspection teams, especially on CDBI’s. 

NRC Staff Response: 

The respondents indicated that overall, the SDP generally resulted in the appropriate regulatory 

response, although they suggested areas for improvement. Some respondents commented on 

screening performance deficiencies for more-than-minor significance. The staff addressed 

these comments in its response to Question 5. 
18 

A respondent encouraged good dialogue between the NRC senior reactor analysts (SRAs) and 

the licensees’ probabilistic risk assessment (PRA) staff to gain a thorough understanding of the 

bases for outcomes from the SDP. The staff agrees with this recommendation and notes that it 

is already incorporated in ROP guidance. For potentially greater-than-green SDP outcomes, 

IMC 0609, Attachment 1, “Significance and Enforcement Review Panel Process,” instructs the 

staff to offer the licensee an opportunity to submit a written response or request a regulatory 

conference. The NRC provides sufficient information to allow a licensee to reasonably 

understand the basis for the staff’s preliminary significance determination with the objective of 

promoting a common understanding of the significance of the finding. A regulatory conference 

gives a licensee the opportunity to provide the staff with additional information, which may affect 

the outcome of the final significance determination. 

Some respondents commented that the assumptions used in the SDP are occasionally 

subjective, arbitrary, and not well-supported, as evidenced by determining common cause 

probabilities and human performance-related credit. The staff understands the respondents’ 

comments; however, the staff does not agree that common cause failure and human error 

probabilities introduce subjective and arbitrary assumptions into the standardized plant analysis 

risk (SPAR) models. The staff believes that failure to consider these probabilities would provide 

results that are non-conservative, subjective, and unsupported. The staff continues to expand 

technology in these areas. As discussed in SECY-09-0143, “Status of the Accident Sequence 

Precursor Program and the Standardized Plan Analysis Risk Models,” the staff is reviewing the 

quality of SPAR models in accordance with the SPAR Model Quality Assurance Plan, which 

governs SPAR model development activities. In addition to internal quality assurance efforts, 

the staff works with industry representatives to update and improve risk models and assessment 

techniques. The NRC staff strives to ensure that the SDP is transparent, objective, scrutable, 

and consistent. The Risk Assessment Standardization Project (RASP) Handbook, which is 

available on the “ROP Program Documents” Web page of the NRC’s external Web site, was 

developed to ensure these attributes and the regulatory applications to reduce conservatism to 

the extent possible given the best available information are maintained. Guidance in the RASP 

Handbook is built on existing NRC-accepted methods for risk analysis of operational events and 

licensee performance deficiencies. Best available information provided by the licensee and 

NRC are used to support the bases of realistic assumptions used in the SDP. 

Some respondents encouraged the NRC to use licensee PRA models that are compliant with 

Revision 2 of Regulatory Guide (RG) 1.200, “An Approach for Determining the Technical 

Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities,” for determining 

the safety significance of findings and events. These respondents stated that the NRC could 

continue to maintain a degree of independence by using a verification process modeled after 

the PI verification process. The staff reviewed the SDP tools in CY 2009 to identify potential 

enhancements. In cooperation with industry experts, the staff peer-reviewed typical boiling 

water and pressurized water reactor SPAR models in accordance with Revision 2 of RG 1.200 

and American National Standards Institute/American Society for Mechanical Engineers 

Standard RA-S-2002, “Standard for Probabilistic Risk Assessment for Nuclear Power Plant 

Applications.” The review team concluded that the SPAR models were valid PRAs and 

appropriate for their intended use. As discussed in SECY-09-0143, the Electric Power 

Research Institute (EPRI) and the staff are also conducting research to resolve technical issues 

that account for differences between NRC’s SPAR models and licensees’ PRAs. 
19 

A respondent commented that the SDP guidance related to implementation of National Fire 

Protection Association (NFPA) Standard 805, “Performance-Based Standard for Fire Protection 

for Light Water Reactor Electric Generating Plants,” must be updated and regional staff training 

and understanding must be improved to ensure accurate and consistent results are achieved. 

The NRC is working with industry to develop fire PRAs that are as realistic as possible for plants 

transitioning to NFPA-805. The staff cooperated with EPRI to develop a detailed state-of-the-art 

fire PRA methodology, which is published in NUREG/CR-6850 (EPRI TR 1011989), “Fire PRA 

Methodology for Nuclear Power Facilities,” (ML052580075 and ML052580118). The staff is 

planning to update SDP guidance governing fire protection and train SRAs to improve 

understanding and ensure results are reasonably accurate and reliable in this area. 

A respondent commented that the security-related SDP continues to be more subjective and 

unevenly applied and that there is a high likelihood that findings will be considered more 

significant because of the SDP’s many attributes. The staff worked with stakeholders to 

address the new requirements of 10 CFR 26, “Fitness for Duty Programs,” and 10 CFR 73, 

“Physical Protection of Plants and Materials,” and to revise and issue IMC 0609, Appendix E, 

Part I, “Baseline Security Significance Determination Process for Power Reactors.” These 

enhancements were made to increase the security SDP’s predictability and repeatability. The 

staff is also working with stakeholders to improve the effectiveness of IMC 0609, Appendix E, 

Part II, “Force-on-Force Physical Protection Significance Determination Process for Power 

Reactors.” 

A respondent questioned why a green non-cited violation (NCV) would be documented if it had 

low safety significance. Performance deficiencies need to have more than minor significance 

before they can be documented in inspection reports and assigned a safety significance and/or 

severity level. Issuance of a green NCV means that the staff determined that the issue was a 

performance deficiency involving a violation having more than minor safety significance. A 

green finding is a finding having very low safety significance. Violations associated with findings 

that the SDP evaluated as having very low safety significance will normally be described in 

inspection reports as NCVs. The staff believes these green NCVs should be documented in 

NRC inspection reports because they are indicative of licensee performance. 

A respondent indicated that findings having white safety significance should not result in a 

licensee entering the Degraded Cornerstone Column (i.e., Column 3) of the Action Matrix. The 

staff addressed a similar comment in its response to Question 2. Multiple white inputs in one 

cornerstone warrant increased NRC oversight because they represent a more substantial 

degradation of a particular aspect of licensee performance. 

A respondent commented that inspectors emphasize a deterministic rather than a risk-informed 

approach to regulation. The staff notes that the Commission’s staff requirements 

memorandum (SRM) to SECY-98-144, “White Paper on Risk-Informed and Performance-Based 

Regulation,” states that the use of PRA technology should be increased in all regulatory matters 

to the extent supported by state-of-the-art in PRA methods and data and in a manner that 

complements the NRC’s deterministic approach and supports the NRC’s traditional defense-indepth philosophy. The SRM stated that the transition to a risk-informed regulatory framework 

was expected to be incremental. Many regulations are based on deterministic and prescriptive 

requirements that cannot be quickly replaced. In addition, the Commission’s policy has 

consistently promoted risk-informed, rather than risk-based, regulation and oversight. As such, 

deterministic requirements remain in effect while risk-informed and/or performance-based 
20 

regulations are being developed and implemented. A risk-informed regulatory approach is 

intended to enhance, rather than replace, the deterministic approach. 

A respondent indicated that findings should have greater than green safety significance in order 

to be assigned a CCA. The staff notes that the NRC identifies CCAs to licensees so that 

licensees can correct performance problems before they can result in more safety-significant 

issues. Limiting insights into the cross-cutting areas to only safety-significant findings and 

waiting for these findings to occur before providing feedback would not be sufficiently proactive 

to ensure adequate protection of public health, safety, and security. 

A respondent stated that the NRC does not credit a licensee’s corrective action process when 

resolving cross-cutting issues. The staff notes that IMC 0305 requires the staff to consider the 

licensee’s progress in addressing CCAs in the substantive cross-cutting issue (SCCI) 

determination process. In many cases, the staff refrains from identifying an SCCI because a 

licensee’s corrective actions are proactive and effective. In these cases, the staff does credit 

the licensee’s corrective action process. 

A respondent commented that the three cross-cutting areas do not fully model all aspects of 

safety culture. The staff agrees. The ROP identifies 13 components of safety culture, of which 

nine are cross-cutting and considered during all inspections. The staff considers the four safety 

culture components that are not cross-cutting only during supplemental inspections. The staff 

further addresses this and similar comments in its response to Question 10. 

(8) Does the NRC take appropriate actions to address performance issues for those plants 

outside the Licensee Response Column of the Action Matrix? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes 

Nuclear Energy Institute

The action taken by the NRC to address performance issues at plants outside the Licensee 

Response Column of the action matrix has recently been more predictable and appropriate. 

However industry has a concern about comments occasionally made at public meetings that 

deviations from the action matrix are permitted on a case by case basis and should be 

considered an option. This should be rare and for exceptional circumstances in order to support 

the ROP principle of “predictability.” Program changes that can be made to avoid routine 

deviations from the action matrix should be evaluated and incorporated. 

Regional Utility Group IV

NRC action to address performance issues at plants outside the Licensee Response Column of 

the action matrix is generally appropriate. The content and scope of supplemental inspections 

has recently been more predictable and consistent with the safety significance of the 

performance issues that led to the supplemental inspections. 

Strategic Teaming and Resource Sharing

The action taken by the NRC to address performance issues at plants outside the Licensee 

Response Column of the action matrix has recently been more predictable and appropriate. 
21 

However, the STARS alliance has a concern about comments occasionally made at the public 

monthly ROP meetings that deviations from the action matrix are permitted on a case by case 

basis and should be considered an option. Deviations from the action matrix should be rare. 

Program changes that can be made to avoid routine deviations from the action matrix should be 

evaluated and incorporated. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the NRC’s actions to address performance issues at plants 

outside the Licensee Response Column of the Action Matrix are generally appropriate. Several 

respondents also noted improvements in the NRC’s supplemental inspection program. Some 

respondents commented on the use of deviations from the Action Matrix. The staff notes that 

since the inception of the ROP, the NRC has approved only 17 deviations. Deviations from the 

Action Matrix are permitted and are intended for rare instances in which the regulatory actions 

dictated by the Action Matrix may not be appropriate, consistent with IMC 0305. IMC 0305 and 

Management Directive 8.14, “Agency Action Review Meeting,” require the staff to discuss 

deviations at the agency action review meeting and subsequent Commission meeting to 

determine if modification to the ROP is warranted. Similarly, IMC 0307 requires the staff to 

evaluate the causes for deviations and identify programmatic changes to the ROP, if necessary. 

(9) Is the information contained in NRC assessment letters relevant, useful, and written in 

plain English? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes 

Nuclear Energy Institute

The information contained in assessment reports is, for the most part, relevant, useful, and well 

written. Inspection schedules in particular are good to have in advance even if they are not fully 

refined. When significant changes are made to inspection schedules, revised schedules should 

be made publicly available. 

An element of the assessment letters that could be improved is the discussion about 

substantive cross cutting issues. Greater consistency in the language and the detailed 

discussion used across regions would be appropriate. The criteria for opening and closing 

SCCIs are not clear; therefore it is not surprising that the assessment letters do a poor job of 

explaining why SCCIs have been opened or closed. This does not support the ROP principles of 

“predictability,” and “objectivity.” 

Regional Utility Group IV

The information contained in assessment reports is, for the most part, relevant, useful, and well 

written. Inspection schedules in particular are good to have in advance even if they are not fully 

refined. When significant changes are made to inspection schedules, however, revised 

schedules should be made publicly available. 
22 

Strategic Teaming and Resource Sharing

The information contained in assessment reports is, for the most part, relevant, useful, and well 

written. Inspection schedules in particular are good to have in advance even if they are not fully 

refined. When significant changes are made to inspection schedules however, revised 

schedules should be made publicly available. One element of the assessment letters that could 

be improved is the discussion about substantive cross-cutting issues. The amount of discussion 

about safety culture issues can add confusion to the assessment letter. Much of the discussion 

in the assessment letters about safety culture is boilerplate and could be simplified using a table 

with notes. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the information contained in NRC assessment letters is 

generally relevant, useful, and well-written. Some survey respondents commented that the 

SCCI-related sections of the assessment letters can be improved and the criteria for opening 

and closing SCCIs are not clear. The staff addresses these comments in its response to 

Question 10. 

Some respondents commented that significantly revised inspection schedules should be made 

publicly available. The staff notes that IMC 0305 requires the staff to include the inspection plan 

with the publicly available semi-annual assessment letters. Assessment letters contain a 

discussion of findings being evaluated by the SDP that may affect the inspection plan. 

IMC 0305 also states that if there are significant changes in the inspection plan for a plant in the 

Multiple/Repetitive Degraded Cornerstone Column of the Action Matrix, the regions should issue 

a separate assessment follow-up letter to ensure the licensee is aware of these changes. The 

staff regularly informs licensees (i.e., the primary stakeholders affected by schedule changes) of 

inspection schedule changes and therefore considers the current process for making the 

schedules publicly available to be adequate. 

(10) Do the ROP safety culture enhancements help in identifying licensee safety culture 

weaknesses and focusing licensee and NRC attention appropriately? 

Respondent Comments: 

Southern Nuclear Operating Company

No. The aspects are too broad and vague. The threshold of greater than 3 in any given aspect 

is too low and does not reflect a long term look. It should also be weighted based on the number 

of major inspections per period (e.g. PI&R, CDBI, and FP Triennial inspections may all occur in 

the same year). In addition, cross cutting issues and area aspects are by themselves a poor 

measure of safety culture. The proposed NEI process for measuring safety culture is much 

more comprehensive and inclusive and serves as a better measure of safety culture. SNC 

believes that the NRC should adopt this proposal. 

Nuclear Energy Institute
23 

The ROP safety culture changes to the ROP consist of identifying cross-cutting aspects of 

performance deficiencies, and accumulating them into substantive cross-cutting issues (SCCI). 

The identification of cross-cutting aspects associated with inspection findings does provide 

value to the licensee to consider in the assessment of safety culture. Note, however, that the 

association by NRC is done without conducting a formal root or apparent cause, and may very 

well be mistaken. Licensees should incorporate the NRC’s association into its ongoing 

assessment of safety culture, using all available site data (for example, safety culture 

assessments, employee concerns issues, site PI data, self assessments, audits, benchmarking, 

industry evaluations, operating experience, etc.) More accurate conclusions can be reached by 

integrating all of the information available on the site safety culture. 

Industry does not believe that the practice of accumulating aspects into SCCIs is appropriate or 

effective. First, the number (usually four in a year) is arbitrary and not based on research, and 

its appropriateness has not been assessed (benchmarked against actual safety performance) 

since the changes were implemented. For example, it does not appear reasonable that four 

procedure adherence issues (usually all green, or of very low safety significance) over a year’s 

period represents a cultural problem. (The thousand people at a station likely perform more than 

one procedure per person per day, for 365 days a year, which would be hundreds of thousands 

of opportunities, with only four failures.) Furthermore, the number four is not normalized based 

on the inspection hours or the number of units on site, and therefore can create a false 

impression of cultural weakness merely because there were more opportunities to identify 

violations which are assigned an aspect. Second, many of the aspects are not safety culture 

issues per se, but rather process errors (for example an error in a procedure step, or a 

deficiency in the corrective action program). A more thorough examination of multiple process 

errors is needed to determine whether there was a common cultural aspect that deserves 

corrective action beyond just fixing the process error. Third, much time and effort is expended 

discussing which aspects apply, particularly as one approaches the number of four. Fourth, it is 

not at all clear what the objective criteria are for determining whether the licensee is taking 

appropriate action to address the supposed substantive issue, or what needs to be done to 

clear the issue if it in fact exists. Fifth, the use of two different languages to discuss safety 

culture (the NRC’s and the industry’s) can lead to confusion in identifying and resolving cultural 

issues. In summary, industry believes that the SCCI process results in excessive use of NRC 

and licensee management resources, and it diverts resources to address perceived problems 

from correcting actual safety issues, including safety culture issues. 

The industry wants to be proactive in ensuring our plants have a strong nuclear safety culture: 

• Licensees are responsible for the safe operation and safety culture of their plants. 

• NRC is responsible for providing effective oversight. 

Therefore industry has proposed three actions and is working with the NRC and other 

stakeholders to achieve them: 

1. Develop a common language of safety culture to be used by the regulator and the 

licensee. 

2. Develop an integrated approach for licensees to assess their safety culture on an 

ongoing and proactive basis with NRC providing effective, transparent oversight, and 

3. Develop a common methodology for conducting self, independent and third party safety 

culture assessments. 

Regional Utility Group IV
24 

The safety culture enhancements do not focus licensee resources appropriately. Since 

licensees apply significant resources to correcting safety culture issues identified by the NRC, it 

is important that the NRC process for identifying these issues reflects an integrated picture of a 

licensee’s safety culture. Basing conclusions about safety culture at a plant on the relatively 

small number of safety culture crosscutting aspects that are assigned to findings over a period 

of time does not provide an accurate assessment of safety culture. The “greater than three 

findings” threshold for a substantive cross-cutting issue seems to have no basis. There has 

been enough run-time on the program to re-evaluate the threshold. In addition, the crosscutting aspect definitions are broad enough that deficiencies within an aspect may be unrelated 

and not constitute a valid trend in a particular area; however a substantive crosscutting issue 

could be considered. Differences across regions and plants in the number of inspection hours 

and findings naturally produce variations in the number of safety culture aspects assigned and a 

corresponding wide variation in the number of substantive cross-cutting issues identified by the 

NRC. 

Strategic Teaming and Resource Sharing

The use of cross-cutting aspects (CCAs) that are assigned to a relatively small number of 

issues over a period of time does not provide an accurate assessment of safety culture. With 

CCAs being assigned to virtually all findings, the CCAs are almost becoming more of a focus 

than the finding itself due to the low threshold for a substantive cross-cutting issue. In addition, 

there is significant inconsistency in applying CCAs. Some inspectors use input from the 

licensee's causal analysis for the finding to determine the appropriate CCA. Others ,assign the 

CCA they believe is appropriate and will not consider the licensees causal analysis even when 

the results are in conflict with the assigned CCA. Finally, there are increasingly more cases 

where inspectors are applying the CCAs contrary to the definitions of the CCAs in MC 0305. 

Assigning safety culture CCAs to findings from the more intensive team (e.g., PI&R, Triennial 

Fire Protection, CDBI, etc) or supplemental inspections (e.g., 95001 and 95002 inspections), 

introduces inconsistency into the assessment process because with the higher number of 

inspection hours, more opportunity exists to identify findings and more CCAs will be counted. 

The higher number of CCAs is a factor driven by the higher number of opportunities or 

inspection hours - not necessarily declining safety culture. A similar effect occurs for sites with 

multiple units due to the higher number of inspection hours. There has been enough run-time on 

the program that we would suggest the CCA threshold of three findings be reevaluated. 

The industry is piloting a process for assessing the facility's Nuclear Safety Culture that is based 

on more process input data. It is expected that this process will be a more valid approach for 

evaluating Nuclear Safety Culture than the current process that was integrated into the ROP. 

Dennis Mosebey, Wolf Creek Generating Station 

There are many examples of where the questions asked by Residents at my plant actually divert 

focus from safety. Some examples are, “Why are not all the cotter pins on the A Diesel injectors 

not bent the same amount?” This came up while working on restoration from a 7 day TESO and 

then the inspector demanded an answer within an hour later about B DG. The cotter pins are a 

secondary fastener on the Diesel Generator and any reasonable person understands that any 

cotter pin even bent a slight amount will do its function. Yet this caused a lot of diversion of 

focus from critical evolutions. In another case we were issued a violation for preconditioning 

involving cycling our rods prior to an actual surveillance test even though the NRC’s own 

document states that if one is following vendor recommendations it is not to be considered 
25 

preconditioning. The Resident stuck to the violation and my utility chose not to protest it. 

Obviously a utility cannot protest every such violation. This illustrates that Residents at least at 

my plant pursue their own agenda and beliefs outside NRC’s own guidance with no apparent 

accountability to Regional Headquarters. In another case the Resident was going to give two 

violations for not documenting risk assessments properly. These were refuted by the utility and 

with the help of interface from Headquarters were eventually retracted but it took much time and 

effort on our part. More examples can be given, but while Residents can and must ask 

questions, there is a strong tendency at least at my plant to pursue the issue way beyond the 

safety significance once the answer has been given. It is not clear to me why this is so. 

NRC Staff Response: 

Although a respondent stated that the identification of CCAs associated with inspection findings 

does provide value to the licensee to consider in the assessment of safety culture, all 

respondents indicated that the ROP safety culture enhancements do not help identify licensee 

safety culture weaknesses nor focus licensee and NRC attention appropriately. 

Some respondents commented that cross-cutting issues do not serve as good assessments of 

a licensee’s safety culture. The staff notes that the intent of identifying of CCAs and SCCIs is 

not to provide an overall assessment of a licensee’s safety culture. Rather, the intent is to 

identify the NRC’s concerns about licensee performance specific to an aspect of safety culture 

and provide the licensee with the opportunity to address the issue before it could result in a 

more significant safety concern. SCCIs that remain open for 18 months may prompt the NRC to 

request a licensee to perform a safety culture assessment. 

Some respondents commented that the CCAs are not well-defined or focused and that they do 

not reflect safety culture issues. The staff notes that the NRC is engaging stakeholders, 

including nuclear power reactor stakeholders, to collaboratively develop a common safety 

culture terminology, which will include a definition and high-level descriptions of safety culture. 

The product of this effort will inform the Commission’s final safety culture policy statement. 

Upon the issuance of the Commission’s final safety culture policy statement, the NRC staff will 

determine if any changes to the ROP safety culture component and aspects are necessary to 

conform to the policy statement. 

Some respondents commented that the NRC staff is not consistently or correctly applying ROP 

guidance for identifying CCAs. The staff notes that IMC 0612 was revised, and 0310, 

“Components Within the Cross-Cutting Areas” was developed, to consolidate and streamline the 

guidance for identifying CCAs. Section 14.06 of IMC 0612 provides guidance for information 

that becomes available after the inspection report is issued and that results in a change to the 

CCA. If a licensee disagrees with an assigned CCA, the staff welcomes new information that 

may better inform its CCA characterization, provided that the new information is shared with the 

NRC in a timely manner. 

Some respondents commented that the SCCI opening and closure criteria are not well-defined, 

understood, or documented in assessment letters and the SCCI threshold is inadequate. The 

survey respondents also suggested that the SCCI threshold should be weighted based on the 

number of major inspections performed or inspection hours spent each year. The staff notes 

that the SCCI threshold consists of the two criteria described in Section 13 of IMC 0305. The 

number of CCAs identified at a site is only one consideration of the SCCI decision-making 
26 

process. Another important consideration (the second SCCI criterion) is the NRC’s confidence 

in the licensee’s scope of efforts or progress in addressing the cross-cutting theme(s). Although 

a licensee may have multiple findings with the same CCA, the NRC may not identify an SCCI if 

the licensee is taking adequate corrective actions to address the theme. As such, the staff’s 

identification of an SCCI is in large part a function of its confidence in the licensee’s corrective 

actions. The NRC’s confidence will be based on the four considerations described in 

Section 13.02 of IMC 0305, which are informed, in part, by the results of PI&R inspections. 

IP 71152, “Problem Identification and Resolution,” instructs inspectors to review a licensee’s 

corrective actions for cross-cutting themes and SCCIs. This IP lists general attributes that 

inspectors consider when determining the effectiveness of licensees’ corrective actions. 

Section 13.03 of IMC 0305 provides examples of SCCI closure criteria, examples of which 

include the number or significance of findings with the same aspect and the NRC’s concerns 

about the adequacy of corrective actions. IMC 0305 provides flexibility when choosing the 

closure criteria to allow for site-specific considerations and to ensure the staff’s concerns are 

addressed. In 2008 and 2010, the staff reviewed SCCI decisions documented in assessment 

letters and determined that the decisions were reliable because (1) they conformed to ROP 

guidance, and (2) the same conclusions would have been reached by staff from different 

regions. The staff is currently revising IMC 0305 assessment letter guidance and will consider 

potential improvements to documenting SCCI-related decisions. 

Some respondents commented that licensee and NRC resources are being inappropriately 

used to address CCAs. Because CCAs correlate to the causes of findings, the staff expects 

licensees to spend resources on addressing these causes to ensure that potential underlying 

organizational issues are appropriately attended to. 

Some respondents commented that the NRC should adopt the NEI’s proposed safety culture 

oversight process. The staff has communicated to NEI that the NRC is supportive of industry 

initiatives to address safety culture and strongly encourages NEI and the industry to monitor and 

promote a culture that strives for enhancing safe plant operation. If the NEI 09-07 process 

proves to be effective, the number of SCCIs should decline throughout the industry. A 

licensee’s safety culture initiative, if properly implemented, can provide the NRC with sufficient 

confidence that a process is in place to correct the cross-cutting themes. The staff will consider 

adjustments to the ROP if there is evidence of a declining trend in the number of SCCIs. At the 

request of the NEI, the staff agreed to observe three key elements of the safety culture 

initiatives underway at the pilot plants. The staff conducted a public meeting with NEI, the pilot 

plants, and regional staff on February 24, 2010, to provide staff observations on the strengths, 

weaknesses, and areas for improvement. The staff shared its significant reservations about the 

reliability and validity of the survey and interview instruments. The staff will continue to interact 

with NEI on safety culture initiatives. 

A respondent commented on the regulatory impact resulting from inspectors’ questions and 

dispositions of issues. The staff notes that the NRC staff developed a process for obtaining 

continual feedback from licensees and annually reports the feedback to the Commission. The 

process requires regional management to solicit and record informal feedback from licensees 

during routine visits to reactor sites. The staff evaluates the feedback and takes any necessary 

corrective actions. This process has given licensees frequent opportunities to comment on the 

NRC’s regulatory impact. During the last fiscal year, over 95 percent of the comments received 

were favorable. The staff reviewed the negative feedback for trends and found that it related 
27 

only to isolated incidents or differences in professional opinions. The NRC management 

continues to emphasize to the staff the importance of professional conduct. Senior NRC 

managers reinforce these expectations in inspector counterpart meetings, workshops, and 

training courses, as well as during site visits conducted in accordance with IMC 0102. The staff 

will continue to closely monitor the regulatory impact of inspector performance. 

(11) Are the ROP oversight activities predictable (i.e., controlled by the process) and 

reasonably objective (i.e., based on supported facts rather than subjective judgment)? 

Respondent Comments: 

Southern Nuclear Operating Company

The findings are reasonably objective, but the final significance determination seems to rely 

more on the NRC SPAR model risk assessment rather than the licensees superior risk model. 

The result is subjective judgment and a bias toward consistency rather than the real risk of the 

situation. 

Nuclear Energy Institute

Overall, the ROP is predictable and reasonably objective. Some opportunities for improvement 

include the following. 

1. The assumptions used by the NRC in the Significance Determination Process are at times 

subjective and arbitrary (ref comment on question 7). This has in some cases resulted in 

delays in finalizing the final results of an SDP. An example of this was a recent security 

finding at plant where the characterization (color) changed several times because of 

changing subjective assumptions. Industry encourages the use of Licensees’ PRA models 

which have been evaluated and peer reviewed against consensus PRA standards and NRC 

Regulatory Guide 1.200 to support the SDP process as they become available. We also 

encourage additional work in the areas of common cause and human performance. 

2. The process is not objective in the area of fire protection/Alternate Safe Shutdown (ASSD) 

capability, and this issue will be exacerbated due to NRC expectations for conservative fire 

PRA assumptions as the basis for NFPA 805 implementation. These models do not provide 

results consistent with operating experience or internal events PRA models, and this bias 

will need to be accommodated in the SDP process, because insights and experience from 

the piloting of transitioning to NFPA 805 have not been incorporated into the ROP. 

3. Also, please refer to our comments in question 10 regarding the predictability and objectivity 

of the NRC’s safety culture approach. 

4. The availability definitions have been somewhat confusing (in that there are several) and are 

continually a topic of discussion. 

 Recent difficulty regarding the definition of availability centers around how much credit can 

be taken for simple actions that restore equipment and make it usable; for example, a 

manual action that has been determined to be feasible to allow the equipment to be ready to 

perform its risk significant function (Note that this does not involve trying to take credit for the 

actions to avoid counting a failure; only to restore availability). Since differences exist, and 
28 

regulatory interpretations are not consistent, issues regarding availability and the definition 

of availability have become distractive. 

5. When MSPI was developed the intent was to align the definitions of availability between the 

PI manual, NEI 99-02, and the Maintenance Rule definition of unavailability in NUMARC 93-

01. More work is needed in this area (understanding of the definitions of availability and 

alignment.) NEI has submitted proposed revisions to NUMARC 93-01 to address this 

inconsistency. 

Regional Utility Group IV

In general, the ROP provides a predictable and objective framework for NRC oversight and is a 

significant improvement over the previous reactor oversight process. Refinements in the ROP 

have generally improved the process further. The biggest opportunity for reduction in 

subjectivity is in the safety culture portion of the ROP. As discussed in the response to item 10, 

this process has produced unintended consequences in driving licensee response based on an 

incomplete assessment of safety culture. 

Strategic Teaming and Resource Sharing

For the most part the ROP is structured such that the outcomes should be predictable and 

objectively based on facts. However, there are some areas as noted in the response to previous 

questions where there is enough subjectivity that just about any outcome could be reached. 

Additionally, the ROP becomes much less predictable when the elements of the process are not 

implemented as written which does occur in some cases. For example, as mentioned earlier 

CCA definitions are not followed in their appropriate context when assigning CCAs to findings' 

Dennis Mosebey, Wolf Creek Generating Station 

Nope, Residents still apply their own interpretations and then it is up to the utility to deal with it 

on the FAQ process which is very time consuming. I think this ties back to ineffective buy in of 

risk informed aspects all the way back to headquarters. The “ratchet” process is also still used 

even though this new process was supposed to limit it. See also comments to safety culture 

question above. 

NRC Staff Response: 

Most respondents commented that the ROP is reasonably objective and predictable, and a 

respondent acknowledged that previous refinements have improved the ROP. The 

respondents’ SDP-related comments are addressed in the staff’s response to Question 7, and 

the safety culture-related comments are addressed in the staff’s response to Question 10. 

A respondent commented that multiple definitions of availability cause confusion and that 

NUMARC 93-01, “Industry Guideline for Monitoring the Effectiveness of Maintenance at Nuclear 

Power Plants,” (ML101020415 and ML101020466) should be revised to address these 

inconsistencies. The staff notes that the MSPI and 10 CFR 50.65, “Requirements for monitoring 

the effectiveness of maintenance at nuclear power plants,” differ in purpose. As a result, the 

concept of unavailability is applied differently in these different contexts. NEI 99-02 also 

acknowledges that differences in definitions and guidance in most instances are deliberate and 

necessary. The industry’s current proposal to align the definition of unavailability in 

NUMARC 93-01 with the MSPI definition may impact the requirement to track unavailability of 

systems during plant shutdowns. Specifically, some licensees might accrue significant amounts 
29 

of shutdown unavailability that would not be counted and therefore result in a less realistic 

evaluation of system health and risk. The staff agrees that options to clearly define terms 

should be considered; however, attempting to derive a single definition may not be necessary. 

A respondent expressed dissatisfaction with resident inspectors’ interpretations of ROP 

guidance. The staff understands that stakeholders may perceive inspection staff applying 

portions of ROP guidance inconsistently. These situations may occur because ROP guidance 

allows flexibility for plant-specific conditions. Differences in licensee performance and unique 

circumstances may contribute to the perceived inconsistencies among regions. ROP guidance 

achieves consistency to the extent practicable, and the staff strives to ensure that agency 

decisions are reliable and conform to governing procedures and policies. Licensees and the 

industry can provide NRC staff with specific examples of significant inconsistencies in 

application of ROP guidance through a number of forums, including the ROP monthly public 

meetings and regional management site visits. The staff will continue work with stakeholders to 

ensure that guidance is clarified and understood in an effort to maintain reliability and objectivity 

when implementing the ROP. In addition, as discussed in the staff’s response to Question 19, 

the staff has developed initiatives to improve the reliability of the ROP. 

(12) Is the ROP risk-informed in that the NRC's actions are appropriately graduated on the 

basis of increased significance? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes. 

Nuclear Energy Institute

In general, the NRC’s Action Matrix provides appropriate graduation on the basis of increased 

risk for NRC actions to be taken. However, over the survey period, the NRC has made or 

considered making changes to the ROP that were contrary to the risk informed philosophy 

initially established as a goal for the ROP. Two examples are the actions to integrate NRC 

assessment of safety culture and consideration of including traditional enforcement into the 

ROP. Safety culture assessments and traditional enforcement issues are based on deterministic 

concepts not risk-informed concepts. While these elements are important and need to have 

NRC oversight, they should not be integrated into the ROP if the ROP (in particular the action 

matrix) is going to remain risk informed and objective. 

We have discussed safety culture previously. Regarding including traditional enforcement as 

input into the Action Matrix, the NRC sponsored open and frank discussion in public meetings to 

consider alternatives. As a result, process changes were made which reflected the NRC’s need 

to follow up on traditional enforcement violations without compromising the risk-informed 

approach of the action matrix. 

Regional Utility Group IV

The NRC’s action matrix, if properly implemented, provides appropriate graduation on the basis 

of increased risk as the basis for NRC action to be taken. However, recent changes to the ROP 

are eroding the risk-informed elements that were initially established as goals for the process. 

Two examples are attempts to integrate NRC assessment of safety culture and traditional 

enforcement into the ROP. Safety culture assessments and traditional enforcement issues are 
30 

based on deterministic concepts. While these elements are important and need to have NRC 

oversight, they should not be integrated into the ROP if the process is going to remain riskinformed. 

Strategic Teaming and Resource Sharing

The NRC's action matrix, if properly implemented, provides appropriate graduation on the basis 

of increased risk as the basis for NRC action to be taken. However, changes have been made 

to the ROP that are eroding the risk-informed elements initially established as goals for the 

process. Two examples of these changes are the integration of the NRC's assessments of 

safety culture and traditional enforcement into the ROP. Safety culture assessments and 

traditional enforcement issues are based on deterministic concepts not risk-informed concepts. 

While these elements are important and need to have NRC oversight, they should not be 

integrated into the ROP if the ROP is going to remain risk-informed. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes for the most part although in the case I mention above on the boric acid valves the 

classification of Green Non Cited vice Minor was way overboard. 

NRC Staff Response: 

All respondents commented that the NRC’s actions are generally appropriately graduated on 

the basis of increased significance. A respondent disagreed with the staff’s determination that a 

specific finding had more than minor significance. The staff encourages licensees to discuss 

such disagreements with regional staff and addresses a similar comment in its response to 

Question 5. 

Some respondents commented that the integration of traditional enforcement into the ROP 

erodes the risk-informed nature of the ROP. However, a respondent acknowledged that the 

NRC sponsored open and frank discussions in public meetings to consider alternatives, and as 

a result, process changes were made that reflected the NRC’s need to follow up on traditional 

enforcement violations without compromising the risk-informed approach of the Action Matrix. 

The staff notes that the integration of traditional enforcement into the ROP was achieved by not 

considering traditional violations as Action Matrix input. Appendix B of IMC 0612 was revised to 

clarify that any underlying performance deficiency associated with traditional enforcement 

violations can be evaluated using the ROP screening and significance determination processes 

and therefore considered as an Action Matrix input. 

Some respondents commented that the integration of safety culture into the ROP erodes the 

risk-informed nature of the ROP. The staff notes that safety culture is a factor that influences 

human reliability. While safety culture per se is not explicitly modeled in PRAs, it does affect 

performance shaping factors such as procedure compliance and team dynamics, which are 

considered in human reliability analyses; therefore, a nexus to risk exists. In addition, the 

identification of CCAs and SCCIs does not result in Action Matrix movement or supplemental 

inspections. Performance deficiencies have to be more than minor (i.e., findings) to be 

considered for CCAs. In addition, CCAs can only be considered for NRC-identified, selfrevealing, or greater-than-green licensee-identified findings. The NRC will request a licensee to 

perform a safety culture assessment only when (1) that licensee has not corrected a repetitive 

SCCI, (2) the NRC identifies issues with the licensee’s assessment of safety culture aspects 

during a supplemental inspection performed in accordance with IP 95002, “Inspection for One 
31 

Degraded Cornerstone or Any Three White Inputs in a Strategic Performance Area,” or (3) the 

licensee enters the Multiple/Repetitive Degraded Cornerstone Column of the Action Matrix. 

Therefore, the staff believes that its safety culture oversight is risk-informed in that a 

deterministic regulatory approach is enhanced by risk insights. 

(13) Is the ROP understandable, and are the processes, procedures, and products clear and 

written in plain English? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes. 

Nuclear Energy Institute

The ROP procedures and products are generally clear and understandable. Changes to Manual 

Chapter 0305 regarding the definition of Multiple/Repetitive Cornerstone Column in the Action 

Matrix and additional guidance to prevent double counting an inspection finding and 

performance indicators with the same underlying cause added clarity to the manual chapter. 

Regional Utility Group IV

The ROP procedures and products are generally clear and understandable. Changes to 

Manual Chapter 0305 regarding the definition of the multiple/repetitive degraded cornerstone 

column in the action matrix and additional guidance to prevent double counting an inspection 

finding and PIs with the same underlying cause added clarity to Manual Chapter 0305. 

However, it does not appear that the process for characterizing performance deficiencies is 

applied uniformly across all regions and plants. This can be seen in the difference in 

performance deficiencies found between regions and plants. Some deficiencies are found at 

particular plants that do not appear to be at issue in others. 

Strategic Teaming and Resource Sharing

The ROP procedures and products are generally clear and understandable. Changes to Manual 

Chapter 0305 regarding the definition of "Multiple/Repetitive Cornerstone Column in the Action 

Matrix" and additional guidance to prevent double counting an inspection finding and Pis with 

the same underlying cause added clarity to the Manual Chapter. However, it does not appear 

that the process for characterizing performance deficiencies is clear in that it does not seem to 

be applied uniformly across all regions and plants. This can be seen in the difference in 

performance deficiencies found between regions and plants. Some deficiencies are found at 

particular plants and the same deficiency does not appear to be an issue in others. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that ROP procedures, products, and processes are generally clear 

and understandable. Several respondents also commented that changes to IMC 0305 clarified 

the guidance. Some respondents commented that the process for characterizing performance 
32 

deficiencies may not be clear because of the lack of consistency in regional application. The 

staff addressed these comments in its responses to Questions 5 and 11. 

(14) Does the ROP provide adequate assurance, when combined with other NRC regulatory 

processes, that plants are operated and maintained safely and securely? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes. 

Nuclear Energy Institute

The NRC’s internal analysis of industry performance data demonstrates that plant operations 

and safety margins have improved greatly since the adoption of ROP. For example, improving 

trends in unplanned plant scram rates and in the decreasing frequency of significant operating 

events provide objective evidence of the effectiveness of the ROP. 

The ROP provides adequate assurance, when combined with other regulatory processes, that 

plants are being operated and maintained safely and securely. As discussed in the response to 

question 4 above, the current set of PIs has contributed to plant performance improvements 

made by licensees and has in some cases resulted in more PIs being green. When the ROP 

was initially implemented, performance standards were established that provided assurance 

that the NRC’s mission to protect the health and safety of the public was met. Industry 

organizations such as INPO exist to promote excellence. The ROP should continue to be 

focused on protecting the health and safety of the public. The current performance indicators 

and inspections are effective in ensuring safety and regulatory compliance. The ROP should 

continue to assess performance to maintain safety and security. 

Regional Utility Group IV

The ROP, when combined with other regulatory processes, provides adequate assurance that 

plants are being operated and maintained safely. By almost every measure, industry safety 

performance continues to improve. Changes in the ROP have contributed to this trend, as 

shown by industry improvement following introduction of new performance indicators and 

inspection modules. Since the industry has other processes in place to encourage excellence in 

operations, future changes to the ROP should be considered based on the need to correct 

identified deficiencies in the NRC’s ability to provide an adequate level of safety assurance. 

Strategic Teaming and Resource Sharing

The STARS alliance strongly agrees that the ROP provides adequate assurance, when 

combined with other regulatory processes, that plants are being operated and maintained 

safely. As discussed in comment number 4 above, the current set of PIs has contributed to plant 

performance improvements made by licensees and has in some cases resulted in more PIs 

being green. When the ROP was initially implemented, performance standards were established 

that provided assurance that the NRC's mission to protect the health and safety of the public 

was met. Industry organizations exist to promote excellence. The>ROP should continue to be 

focused on protecting the health and safety of the public. The current PIs and inspections are 

effective in ensuring regulatory compliance and the standard for future changes to the ROP 

should be the NRC's ability to provide an adequate level of safety assurance. 
33 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the ROP provides adequate assurance, when combined with 

other NRC regulatory processes, that plants are operated and maintained safely and securely. 

Some respondents stated that changes in the ROP have contributed to improving industry 

safety performance, the current set of PIs is adequate, and because industry organizations exist 

to promote excellence, any changes to the ROP should only focus on the NRC’s ability to 

provide an adequate level of safety assurance. The staff addressed these comments in its 

responses to Questions 1, 4 and 11. 

(15) Are NRC actions related to the ROP effective (e.g., are NRC actions of high quality, 

efficient, timely, and realistic to enable the safe use of radioactive materials)? 

Respondent Comments: 

Southern Nuclear Operating Company

No comments. 

Nuclear Energy Institute

For the most part, the ROP is effective, efficient, realistic, and timely. However, some areas 

could be improved, specifically timeliness in finalizing the characterization of a finding using the 

SDP. One reason for the delay is the use of subjective assumptions by the NRC. Industry 

encourages the use of licensees’ PRA models which have been evaluated and peer reviewed 

against consensus PRA standards and NRC Regulatory Guide 1.200 to support the SDP 

process as they become available and believes that by doing so, improvements would be made 

in timely application of the SDP. 

Regarding the extensive resources that are being expended by NRC and industry on changes 

to the MSPI performance indicator: Recent efforts to make the indicator more “elegant” and 

theoretically pure, but which result in miniscule changes in results, are wasting resources 

needed for other improvements. It must be remembered that the MSPI is an indicator of 

performance which is “risk-informed.” The resources being applied by NRC and industry are not 

going to make plants safer, will not effectively allocate NRC inspection resources and are not 

appropriate. 

Regional Utility Group IV

For the most part, the ROP is effective, efficient, realistic, and timely. A continuing area for 

improvement is the SDP. The SDP is a fundamental process for the ROP, as it is exercised 

frequently and is used to determine the safety significance of findings. As such, it is in both 

industry and NRC interests that the process be efficient, transparent, and objective. Current 

concerns with the SDP include timeliness of completion and subjectivity in the determination of 

outcomes. While timeliness has improved somewhat in the past several years, the timeliness 

and subjectivity concerns are linked; often licensees spend much time challenging SDP 

determinations that appear to involve subjective elements in the use of risk tools, thus delaying 

SDP completion. The industry and NRC should continue to work together to improve the SDP, 
34 

including consideration of the proposal to use licensee PRA models as discussed in the 

response to question 7 above. 

Strategic Teaming and Resource Sharing

The ROP is an effective means for ensuring the safe use of radioactive materials. 

Dennis Mosebey, Wolf Creek Generating Station 

Realism and efficiency are NOT key aspects of the new process. Especially realism is lacking in 

the new inexperienced residents being sent to the field. Often we at the utilities end up training 

them through countless hours of questions and indeed responding and contesting violations. 

Painful for us, painful for them. I do not think the NRC has a very good training program and I 

doubt it could pass an INPO style accreditation process like the utilities have to undergo. 

NRC Staff Response: 

The majority of respondents commented that the ROP is usually effective, efficient, realistic, and 

timely. 

Some respondents commented on the timeliness and subjectivity of SDP decisions. The staff 

notes that 100 percent of inspection items finalized as having greater-than-green safety 

significance met the SDP timeliness goals, as documented in the CY 2009 ROP performance 

metric report (ML100540037). The staff addressed the additional SDP-related comments in its 

response to Question 7. 

A respondent commented that the resources expended by the NRC and industry on MSPI 

changes did not result in an increase in safety. The staff agrees that improved safety should be 

the primary consideration when expending resources to change PIs. The staff addressed 

similar comments in its response to Question 1. 

A respondent commented that new inexperienced resident inspectors lack realism, the NRC 

does not have a very good training program, and the NRC’s training program would probably 

not pass an INPO-style accreditation process. The staff notes that while the NRC training 

program does not use an INPO-style accreditation process, power-reactor inspectors are 

trained in accordance with IMC 1245, “Qualification Program for Operating Reactor Programs.” 

This competency-based training program requires inspectors to complete individual, classroom, 

and on-the-job training and pass a rigorous qualification interview before being certified as a 

fully-qualified inspector. As noted in the staff’s response to Question 10, the staff reviewed the 

feedback from licensee managers and found that the rare cases of negative feedback related 

only to isolated incidents or a difference in professional opinion. Because licensee managers 

routinely complimented the effectiveness of NRC inspections and noted the high quality of 

inspections and the effective and professional working relationship between the NRC and its 

licensees, the staff concludes that the respondent’s comment was isolated and will continue to 

closely monitor the effectiveness of the training program. 

(16) Does the ROP ensure openness in the regulatory process (e.g., does the NRC 

appropriately inform stakeholders in the regulatory process)? 

Respondent Comments: 
35 

Southern Nuclear Operating Company

Yes. 

Nuclear Energy Institute

The ROP process, with its many public meetings and opportunities for involvement, does 

ensure openness not available in the previous process. However, improvements could be made 

in soliciting stakeholder feedback when revising or developing regulatory documents such as 

Inspection Procedures, Manual Chapter guidance, or Regulatory Issue Summaries (RIS). As the 

agent for the industry, NEI routinely requests the opportunity to review draft documents and 

provide feedback in a public venue. However, the NRC is sometimes reluctant to share draft 

information, particularly changes to inspection procedures and processes, and the 

inspection/enforcement process being addressed by the Office of Nuclear Security and Incident 

Response (NSIR). 

Press releases for special inspections and reactive inspections are released indicating that the 

NRC is going to look at a significant problem at a plant. However, seldom is the outcome 

publicized unless the outcome is negative. Providing balanced information to the public is not 

promoting the industry, but rather providing pertinent information. 

Regional Utility Group IV

The ROP process, with its many public meetings and opportunities for involvement, does 

ensure openness that was not available in the previous process. However, improvements could 

be made in soliciting stakeholder feedback when revising or developing regulatory documents 

such as Inspection Procedures, Manual Chapter guidance, or Regulatory Issue Summaries. As 

the agent for the industry, NEI routinely requests the opportunity to review draft documents and 

provide feedback in a public venue. However, the NRC is sometimes reluctant to share draft 

information, particularly changes affecting PIs and the inspection and enforcement process 

being addressed by the Office of Nuclear Security and Incident Response (NSIR). 

Strategic Teaming and Resource Sharing

The ROP process, with its many public meetings and opportunities for involvement, does 

ensure openness not available in the previous process. However, improvements could be made 

in soliciting stakeholder feedback when revising or developing regulatory documents such as 

Inspection Procedures, Manual Chapter guidance, or Regulatory Issue Summaries (RIS). As the 

agent for the, industry, NEI routinely requests the opportunity to review draft documents and 

provide feedback in a public venue. However, the NRC is sometimes reluctant to share draft 

information, particularly changes affecting PIs and the inspection and enforcement process 

being addressed by the Office of Nuclear Security and Incident Response (NSIR). 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the ROP does ensure openness in the regulatory process and 

noted the numerous public meetings and opportunities for stakeholder involvement. 

Several respondents expressed a desire for the staff to solicit stakeholder feedback when 

revising or developing regulatory documents. As noted in the staff’s consolidated response to 
36 

the CY 2007 external survey, the NRC holds monthly and other meetings with external 

stakeholders to provide information about changes to various inspection-related regulatory 

documents that may be of interest to external stakeholders. The staff has not solicited 

comments on changes to some IPs because the changes remained within the ROP regulatory 

framework. Some level of independence from external influences on what is inspected and how 

inspections are conducted is warranted and needs to exist. However, the staff has been 

sensitive to revisions of regulatory documents that might involve changes to policy or regulatory 

positions because these changes could potentially result in an unnecessary increase in 

regulatory burden on licensees with no commensurate improvement to safety. The staff also 

needs to periodically clarify NRC documents to provide additional guidance to both inspectors 

and the industry. In these situations, the staff will communicate and work with external 

stakeholders to resolve these issues when they are brought to the NRC’s attention. 

Several respondents commented that the NRC is sometimes reluctant to share draft information 

related to the security-related PI and inspection program. Staff from the NRC’s Office of 

Nuclear Security and Incident Response (NSIR) meets with the Nuclear Security Working 

Group, which consists of representation from NEI and senior industry security personnel, to 

discuss security-related information. NSIR staff is also engaging the industry in public meetings 

to enhance the Force-on-Force SDP. As discussed during the March 18, 2010, ROP monthly 

public meeting, the staff intends to follow IMC 0608 to implement any changes to the securityrelated PI. 

A respondent commented that the outcome of special and reactive inspections is not publicized 

in NRC’s press releases unless the outcome is negative. The staff notes that the NRC’s 

inspection results are made publicly available and the NRC involves public stakeholders in the 

discussion of the results of the NRC’s annual assessment of a licensee’s performance. The 

NRC communicates inspection results when journalists or other members of the public express 

interest in the results. When the NRC is challenged by reporters and members of the public on 

safety issues they may believe are not being addressed by the licensee or NRC, the NRC refers 

to inspection reports in which the staff documented its evaluation of an issue and any concerns 

with the licensee’s performance. The NRC’s Office of Public Affairs makes press release 

decisions based on the newsworthiness of a particular occurrence. A licensee may also contact 

the media if it wishes to further inform the public of NRC inspection results. 

(17) Has the public been afforded adequate opportunity to participate in the ROP and to 

provide inputs and comments (e.g., does the NRC appropriately involve stakeholders in 

the regulatory process)? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes 

Nuclear Energy Institute

The public has been afforded adequate opportunity to participate in most of the ROP and to 

provide inputs and comments by way of the public monthly ROP meetings, ROP feedback 

surveys, and the annual assessment public meetings. This is not the case however in the area 

of Physical Protection. The Physical Protection area of the ROP is not very open to the public, 
37 

which may be appropriate in most cases; however, program and process changes should go 

through a change management process (similar to the ROP). 

Regional Utility Group IV

The public has been afforded adequate opportunity to participate in the ROP and to provide 

inputs and comments by way of the public monthly ROP meetings, ROP feedback surveys, and 

the public plant performance assessment meetings. 

Strategic Teaming and Resource Sharing

The public has been afforded adequate opportunity to participate in most of the ROP and to 

provide inputs and comments by way of the public monthly ROP meetings, ROP feedback 

surveys, and the public plant performance assessment meetings. This is not the case however 

in the area of Physical Protection. The Physical Protection area of the ROP is not very open to 

the public, which may be appropriate in some cases. Program and process changes should go 

through a change management process (similar to ROP). Security-related information could be 

addressed by limited membership attendance as appropriate. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the NRC has afforded the public adequate opportunity to 

participate in the ROP. Several respondents noted the public participation opportunities 

provided by the ROP monthly meetings, the ROP feedback surveys, and the annual plant 

performance assessment meetings. However, respondents indicated that the public does not 

have the same participation opportunities if the subject matter involves security, which may be 

appropriate in some cases. The staff notes that security has a separate but parallel safety 

oversight process to the ROP and follows the same change management process of the ROP. 

The staff also addressed this comment in its response to Question 16. 

(18) Has the NRC been responsive to public inputs and comments on the ROP? 

Respondent Comments: 

Southern Nuclear Operating Company

Yes 

Nuclear Energy Institute

The NRC for the most part has been responsive to public inputs and comments on the ROP. 

The NRC published a response to the 2007 ROP survey in which comments received were 

dispositioned. Industry strongly encourages the NRC to publish a response for these and any 

future ROP survey comments submitted. 

There was one example where the NRC’s responsiveness to comments made in the 2007 

public ROP comment survey was disappointing regarding the treatment of a plant that had been 

shutdown for many years and was returning to service. The NRC interpreted the performance 

indicator guidance in such a way that data elements to produce an indicator were 

inappropriately extrapolated from a limited set of actual plant performance data. In addition, the 
38 

PI threshold values continued to be based on data derived from a mature fleet of operating 

reactors rather than one in essentially a new plant startup condition. Industry recommends that 

performance for plants returning to service after being shut down for several years, and new 

plants that are beginning initial operation be assessed using data and thresholds that better 

account for challenging issues associated with new plant operations. 

Regional Utility Group IV

The NRC for the most part has been responsive to public inputs and comments on the ROP. 

The NRC published a response to the 2007 ROP survey in which comments received were 

dispositioned. 

Strategic Teaming and Resource Sharing

The NRC for the most part has been responsive to public inputs and comments on the ROP. 

The NRC published a response to the 2007 ROP survey in which comments received were 

dispositioned. The STARS alliance supports the NRC publishing a response and encourages a 

published response by the NRC for these and any other comments submitted in response to a 

solicitation published in the Federal Register. 

There is one example in which the NRC's response to comments made in the 2007 public ROP 

comment survey was disappointing regarding the treatment of a plant that had been shutdown 

for many years and was returning to service. The NRC interpreted performance indicator 

guidance in such a way that the data elements used to produce an indicator were extrapolated 

from a limited amount of actual plant performance data. This led to increased NRC 

supplemental inspection activity because the plant crossed a PI threshold that was based on 

operational performance data from a mature fleet of operating reactors. The STARS alliance 

recommends that performance for plants returning to service after being shut down for several 

years and new plants that are beginning initial operation be assessed using indicators that 

better account for challenging issues associated with new plant operations. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

All respondents commented that the NRC has been responsive to public inputs and comments 

on the ROP. A majority of the respondents commended the NRC for publishing a response to 

the ROP survey comments received from stakeholders. Some respondents expressed 

disappointment in the NRC’s response to comments provided to the CY 2007 ROP survey 

regarding the PI guidance for plants returning to service from an extended shut down. The staff 

addressed this comment in its response to Question 3. 

(19) Has the NRC implemented the ROP as defined by program documents? 

Respondent Comments: 

Southern Nuclear Operating Company

To a degree. There remain some subjective areas that seem to default to regional consistency 

rather than determinations made for a particular licensee. 
39 

Nuclear Energy Institute

For the most part, the ROP is implemented as defined by program documents. However, the 

staff continues to consider whether a performance deficiency occurred as an input into 

determining whether or not an event should count against a Performance Indicator. There is no 

criterion in the performance indicator manual, NEI 99-02, related to performance deficiencies 

and their relationship with PI data. This is an example of inappropriate overlap between PIs and 

inspection findings. 

Industry is also concerned about the inconsistencies that are apparent in number of findings, 

violations, and safety culture cross-cutting aspects issued across the four regions. We 

encourage NRC to continue efforts to ensure that the ROP is consistently implemented across 

the regions. 

Regional Utility Group IV

For the most part, the ROP is implemented as defined by program documents. However, RUG 

IV member plants are concerned about inconsistencies in the number of findings, violations, and 

safety culture cross-cutting aspects issued across the four regions. RUG IV member plants 

recommend that continued efforts be made to ensure that the NRC programs are consistently 

implemented. 

Strategic Teaming and Resource Sharing

For the most part, the ROP is implemented as defined by program documents. However, the 

STARS alliance is concerned about the inconsistencies that are apparent in the number of 

findings, violations, and safety culture CCAs issued across the four regions. The STARS 

alliance also has observed that inspectors will sometimes focus on areas involving minimal risk 

significance which undermines the optimum implementation of the ROP. The STARS alliance 

recommends that continued efforts be made to ensure that the NRC programs are consistently 

implemented across regions. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes. 

NRC Staff Response: 

The respondents commented that the NRC usually implements the ROP as defined by program 

documents. A respondent commented that consideration of a performance deficiency into PI 

data was an example of inappropriate overlap between the inspection and PI programs. The 

staff addressed a similar comment in its response to Question 2. 

Most respondents commented on perceived inconsistencies among the regional offices’ 

inspection results; however, one respondent indicated that NRC decisions should be specific to 

a licensee’s performance rather than be made to maintain regional consistency. As discussed 

in its response to Question 11, the staff understands that stakeholders may perceive inspection 

staff to be applying ROP guidance inconsistently. These situations may occur because ROP 

guidance allows flexibility for plant-specific considerations. The staff agrees that NRC decisions 

should be specific to a licensee’s performance rather than be made to maintain regional 

consistency, and NRC decisions should be reliable in that they have understandable bases and 

conform to ROP guidance. The NRC’s regional offices developed the following four ROP 

reliability initiatives: (1) enhanced inspection resource sharing among regions, (2) branch chief 
40 

benchmarking visits to other regions, (3) periodic discussion of reliability topics, and (4) ROP 

self-assessments of inspection report quality. The staff is leveraging these ongoing efforts to 

improve the reliability of ROP implementation, including the SCCI process. 

(20) Does the ROP result in unintended consequences? 

Respondent Comments: 

Southern Nuclear Operating Company

Occasionally it could. Low to moderate safety significant items could divert resources from more 

important ones. 

Nuclear Energy Institute

SECY 99-007, “Recommendations for Reactor Oversight Process Improvements” outlines the 

key objectives for the ROP as: 

a. Improve the objectivity of the oversight processes so that subjective decisions and 

judgment are not central process features. 

b. Improve the scrutability of these processes so that NRC actions have a clear tie to 

licensee performance. 

c. Risk-inform the processes so that NRC and licensee resources are focused on those 

aspects of performance having the greatest impact on safe plant operation. 

Unintended consequences result whenever actions taken by NRC or licensees are not in full 

alignment with these objectives. In general, the ROP has been a success and has avoided 

unintended consequences. Several areas for improvement are listed below. 

1. Significant NRC and licensee resources are spent characterizing the significance of 

findings. The majority of these resources are focused on findings that have minimal risk 

significance. This result is inconsistent with the ROP objective to “focus resources on 

aspects of performance having the greatest impact on safe plant operation.” 

2. Responses to several questions above have discussed the need for improvement in the 

safety culture approach of the ROP. Removing the subjectivity of the SCCIs and 

replacing it with an integrated industry approach with robust NRC oversight is the goal of 

the industry safety culture pilot beginning this year. 

3. Problems exist in the interpretation of safety system functional failures and how to report 

corrections to the data. We believe NRC and industry need to resolve these concerns, 

both in the interpretations of NUREG 1022 as to what constitute SSFFs, and in the FAQ 

process, on how to report the effective date of a revised LER. 

4. A desire to make the MSPI risk-based rather than risk-informed has unintended 

consequences resulting in wasted resources for little or no gain. 

NEI 99-02 Revision 6 states (emphasis added): 
41 

“Mitigating System Performance Index (MSPI) is the sum of changes in a simplified core 

damage frequency evaluation resulting from differences in unavailability and unreliability 

relative to industry standard baseline values.” 

“The MSPI is an approximation using information from a plant’s PRA and is intended as 

an indicator of system performance. More accurate calculations using plant-specific 

PRAs or SPAR models cannot be used to question the outcome of the PIs computed in 

accordance with this guideline.” 

At times the staff has lost sight of the fact that MSPI was designed to be simple and 

understandable. The paragraph that discusses licensees not being able to use their 

plant-specific PRA model and the NRC not being able to use the plant-specific SPAR 

model to challenge the output of the MSPI calculation was added specifically because it 

was recognized that we were calculating an approximation of the change in core 

damage frequency. 

SECY-99-007 “ Recommendations For Reactor Oversight Process Improvements states 

the following: 

From page 5 – “An efficient oversight process is one that applies agency resources in a 

risk-informed manner.” 

One of the objectives of the ROP listed on page 6 of the SECY – “Risk-inform the 

processes so that NRC and licensee resources are focused on those aspects of 

performance having the greatest impact on safe plant operation.” 

Unfortunately, NRC and industry resources continue to be used on prolonged research 

projects in an attempt to improve the perceived accuracy of MSPI beyond what the 

current state of the PRA modeling is capable of. This diverts resources from 

investigating real improvements to the index. 

Examples of recent activities that are consuming large amounts of resources and are not 

risk-informed, will not appreciably affect outcomes (PI values), will not affect NRC 

oversight and allocation of inspection resources, most importantly, will have no impact 

on safe plant operation include: 

1. MSPI rounding calculations 

2. Fuel Oil Transfer Pumps/Mixed Priors 

3. Monitoring trains (i.e., Birnbaum values << 1E-09) that will never impact the 

MSPI. 

More specific details of these and other issues are discussed in the public monthly meetings of 

the ROP working group. We look forward to discussing these concerns in detail in those 

meetings. 

Regional Utility Group IV

In some cases, the ROP as implemented has resulted in unintended consequences. As 

discussed in the response to question 11 above, the safety culture enhancements have resulted 
42 

in unintended consequences regarding the amount of attention that licenses apply to incomplete 

data regarding safety culture. 

Regarding the PI program, when examples of unintended consequences become evident, the 

FAQ process has generally been effective in correcting them. A recent example that has not yet 

been addressed through the FAQ process is in the way corrections are made to the Safety 

System Functional Failure (SSFF) PI. The current guidance specifies that an SSFF should be 

counted for PI purposes when the licensee event report (LER) is submitted to the NRC. When a 

change to the SSFF indicator is required due to an error in reporting or an incorrect initial 

reportability determination, recent NRC interpretation of the PI guidance requires the SSFF PI to 

be reported in the quarter that the revised LER is submitted. The unintended consequence is 

that this interpretation of the PI guidance does not accurately reflect the timing of occurrences of 

failures. While the RUG IV member plants support the use of the FAQ process for this type of 

an issue, in general, efforts should be made to preserve the original intent of the indicator when 

interpretations are made. 

Strategic Teaming and Resource Sharing

In some cases, the ROP as implemented has resulted in unintended consequences. When 

examples of unintended consequences associated with Pis become evident, the FAQ process 

has generally been effective in correcting them. A recent example that has not yet been 

addressed through the FAQ process is in the way corrections are made to the Safety System 

Functional Failure (SSFF) Pl. The current guidance specifies that a SSFF should be counted for 

PI purposes when the Licensee Event Report (LER) is submitted to the NRC. When a change to 

the SSFF indicator is required due to an error in reporting or an incorrect initial reportability 

determination, recent NRC interpretation of the PI guidance requires the SSFF PI to be reported 

in the quarter that the revised LER is submitted. The unintended consequence is that the PI now 

is becoming an indicator of licensee performance to determine reportability instead of an 

indicator of a plant system's ability to perform its intended safety function. While the STARS 

alliance is confident that this example eventually will be resolved using the FAQ process, efforts 

should be made to preserve the original intent of the indicator when interpretations are made of 

the performance indicator guidance. 

Dennis Mosebey, Wolf Creek Generating Station 

Yes the recent change in how the NRC looks at the SSFI and report data I am afraid will result 

in personnel, for the gray areas choosing not to report. In the past if one was uncertain it was 

best to report, now with this SSFI in effect it can have a chilling effect. A plant may choose to 

wait to report pending more investigation vice being timely. Very bad message at least to me. 

NRC Staff Response: 

Most respondents indicated that in general, the ROP has not resulted in unintended 

consequences and that the FAQ process has generally been effective in correcting unintended 

consequences. A respondent commented that items having low to moderate safety significance 

could divert resources from more important ones. The staff addressed a similar comment in its 

response to Question 2. Some respondents commented that the safety culture enhancements 

resulted in unintended consequences. The staff addressed similar comments in its responses 

to Questions 7, 10 and 12. A respondent commented that the desire to make the MSPI riskbased rather than risk-informed had unintended consequences resulting in wasted resources. 

The staff addressed a similar comment in its responses to Questions 1 and 15. 
43 

A respondent commented that significant NRC and licensee resources are spent on 

characterizing findings that have minimal risk significance. The staff notes that the NRC was 

successful in meeting its metric goals for SDP timeliness and resource expenditures, as 

discussed in the CY 2009 ROP performance metric report (ML100540037). The staff described 

its efforts for improving the SDP in its response to Question 7. The agency’s regulatory 

response is based on the significance of Action Matrix inputs; therefore, additional resources 

are justified to ensure accurate significance determinations. The staff envisions the efficiency 

and effectiveness of significance determinations will increase as SDP tools improve. 

Most respondents commented that the NRC’s interpretation of SSFF PI guidance has resulted 

in unintended consequences. A respondent commented that the SSFF reporting and PI 

requirements may result in licensees choosing not to report or failing to submit reports in a 

timely manner. The staff addressed similar comments related to the SSFF PI in its responses to 

Questions 1 and 3. The staff notes, however, that reporting SSFFs is a regulatory requirement, 

and SSFF PI data submitted to the NRC must be complete and accurate in all material aspects. 

(21) Please provide any additional information or comments related to the ROP. 

Respondent Comments: 

Southern Nuclear Operating Company

In years when major inspections take place (e.g. CDBI, Triennial Fire Inspection, PI&R, etc.) 

there is an increased likelihood for findings/violations with cross cutting aspects. Depending 

upon the timing of these inspections, thresholds could be crossed, which may not be indicative 

of declining performance but rather more indicative of inspection hours. Therefore, there should 

be a weighting factor assigned based upon the inspection hours. 

The Reactor Oversight process is an improvement over previous assessment processes. It 

assists licensees to focus on the areas important to safety in a consistent manner throughout 

the industry. However, the use of cross cutting aspects as an indicator of safety culture is 

subjective in nature. The NEI approach to safety culture monitoring is a better alternative to 

address safety culture. 

Nuclear Energy Institute

Monthly interactions between NRC and industry through the ROP Working Group are critical to 

continued improvement of the ROP. The willingness to devote resources to these meetings is a 

clear indication of NRC’s commitment to making the process as predictable and efficient as 

possible. 

Consideration should be given to updating bases documents for all of the performance 

indicators, similar to the bases that was developed when the Scrams with Complications was 

revised. Understanding and maintaining the original bases for some of the indicators and even 

some of the basic fundamentals on which the ROP was founded are becoming obscure, partly 

because of the high attrition in the industry and the NRC associated with career moves into new 

plant work and aging workforce retirements. 

Regional Utility Group IV
44 

Consideration should be given to develop bases documents for all of the PIs, similar to the 

bases that were developed when the PI for Unplanned Scrams with Complications was revised. 

The original bases for some of the indicators are becoming obscure, partly because of the high 

turnover in the industry and the NRC. 

The NRC’s request for ROP comments is made in the fall during the time when a high 

percentage of power plants schedule refueling outages. This makes it difficult to collect and 

coordinate a response. RUG IV recommends that the future requests provide a longer 

comment period. 

As the industry moves toward operation of new plants, a potential concern is the early use of PI 

data for newly-operating plants that may not provide a true picture of the performance of these 

plants. RUG IV recommends that the NRC plan ahead to assess performance at new plants 

using indicators that account for the challenges associated with new plant operations. 

Strategic Teaming and Resource Sharing

PIs are "indicators" not precise measurements. With regard to the MSPI, too many resources 

are being expended to improve the precision of the indicators at the expense of keeping the 

indicators simple. As an example, several months of debate that took place at the monthly ROP 

meetings to resolve the question of whether the MSPI should be reported to two or three 

significant digits - in reality, the MSPI begins to lose significance after one significant digit. 

Consideration should be given to develop bases documents for all of the PIs, similar to the basis 

that was developed when the Scrams with Complications was revised. Understanding and 

maintaining the original bases for some of the indicators and even some of the basic 

fundamentals on which the ROP was founded on are being lost, partly because of the high 

attrition in the industry and the NRC associated with career moves into new plant work and 

aging workforce retirements. 

The scheduling of significant baseline or supplemental inspections should consider whether 

overlapping the inspection with significant planned station activities is prudent and in the best 

interests of safety. An example would be scheduling a 95003 inspection during a planned steam 

generator replacement outage may unnecessarily challenge the licensee's focus on safe 

conduct of the outage and their ability to provide adequate support to the inspection team. 

The ROP external survey typically is taken in the fall during the time when a high percentage of 

power plants schedule their refueling outage. Collecting and coordinating a submittal during 

refueling outage season is very difficult. The STARS alliance recommends that the future ROP 

external survey either be moved to a time of the year that is outside the typical refueling outage 

season or a longer comment period be provided. 

Dennis Mosebey, Wolf Creek Generating Station 

Overall it is not bad, but I do not see the buy- in to its true premises from Residents and 

Inspection Teams. I think the old absolute determinism camp is still very alive and well and is 

still trying to impose their will to retard the positive benefits to NRC, Public, and the Utilities of 

the risk informed approach. With the departure of Diaz, McGaffigan, and Merrifeld from the 

Commission the risk informed approach lost true allies. I am glad to see Apostolakis from MIT is 

now being considered for a Commission position. This will help. But the gains made are being 

slowly eroded away. The present Commissioners are not in my view committed to risk informed 
45 

thought processes. This is too bad because great strides were being made in this very important 

area. 

NRC Staff Response: 

The NRC staff acknowledges the respondent’s comments that the ROP monthly public 

meetings are a clear indication of NRC’s commitment to making the process as predictable and 

efficient as possible. Some of the respondents’ comments have been addressed in the staff’s 

response to other questions. The staff addressed the safety culture-related comments in its 

response to Question 10. The staff addressed comments related to PIs for new plants in its 

response to Question 3. The staff addressed comments related to MSPI-related resources 

expenditures in its responses to Questions 1 and 15. 

A respondent indicated that NRC’s Commissioners are not committed to risk-informed 

regulation. The staff addressed comments related to risk-informed and deterministic regulatory 

approaches in its responses to Questions 7 and 12. The Commission’s policy on risk-informed 

regulation is still in effect. The staff notes that the Commissioners have frequently advocated 

risk-informed regulatory approaches in recent Commission speeches, which can be found on 

the NRC’s public website, www.nrc.gov. The current NRC Commission staff includes Chairman 

Jaczko, Commissioner Svinicki, Commissioner Magwood, Commissioner Ostendorff, and 

Commissioner Apostolakis. 

Some respondents suggested that basis documents should be developed for all PIs and ROP 

fundamentals. The staff agrees that understanding and maintaining ROP basis documents is 

extremely important. As both the industry and NRC have experienced increased turnover rates, 

knowledge management and transfer is paramount to ensure that the ROP fundamentals are 

preserved. The staff also agrees that consideration should be given to the possibility of 

developing additional basis documents for some or all of the PIs in a manner similar to the 

Unplanned Scrams with Complications PI. The staff notes that ROP basis documents are 

contained in IMC 0308, “Reactor Oversight Process (ROP) Basis Document,” and its 

attachments, which are revised, as necessary, to reflect the basis for any significant 

programmatic changes. 

A respondent commented that the scheduling of inspections should consider whether 

overlapping the inspection with significant planned station activities is in the best interests of 

safety. The staff notes that the NRC attempts to schedule inspections in a manner that will not 

overburden licensees. Licensees can request regional management to reschedule inspections 

to accommodate significant activities. However, some inspections, such as supplemental 

inspections that result from licensee performance problems, will be resource intensive and 

intrusive for licensees because of the need for reasonable assurance that a licensee’s 

performance does not pose a threat to public health, safety, and security. For example, the 

results of an IP 95003, “Supplemental Inspection for Repetitive Degraded Cornerstones, 

Multiple Degraded Cornerstones, Multiple Yellow Inputs or One Red Input,” inspection may 

inform the Commission’s decision to allow a plant to continue operating or to modify, suspend, 

or revoke an operating license. Because of the staff’s responsibility to the public to ensure that 

plants are safely operated, the regional staff should not delay inspections in response to 

degrading licensee performance in order to accommodate other licensee activities. 
46 

Some respondents commented that responding to the external ROP survey is difficult because 

the survey is only available when a high percentage of licensees schedule refueling outages. 

The staff plans to reconsider the content and frequency of the ROP surveys or potentially 

explore alternate venues to obtain stakeholder feedback as a result of the declining number and 

breadth of survey participants. The staff will also consider either moving any future ROP 

external surveys to a time of the year that is outside the typical refueling outage season or 

potentially providing a longer comment period to encourage participation and feedback. 
47 

ACRONYMS 

ADAMS Agencywide Documents Access and Management System 

CCA Cross-Cutting Aspect 

CDBI Component Design Bases Inspection 

CFR Code of Federal Regulations 

CY Calendar Year 

EPRI Electric Power Research Institute 

FAQ Frequently Asked Questions 

IMC Inspection Manual Chapter 

IP Inspection Procedure 

MSPI Mitigating Systems Performance Index 

NCV Non-Cited Violation 

NEI Nuclear Energy Institute 

NFPA National Fire Protection Association 

NRC U.S. Nuclear Regulatory Commission 

NSIR Office of Nuclear Security and Incident Response 

PI Performance Indicator 

PI&R Problem Identification & Resolution 

PRA Probabilistic Risk Assessment 

RASP Risk Assessment Standardization Project 

RG Regulatory Guide 

ROP Reactor Oversight Process 

SCCI Substantive Cross-Cutting Issue 

SCWE Safety Conscious Work Environment 

SDP Significance Determination Process 

SPAR Standardized Plant Analysis Risk 

SRA Senior Reactor Analyst 

SRM Staff Requirements Memorandum 

SSFF Safety System Functional Failure 

SSU Safety System Unavailability 
48 

REFERENCES 

Available from http://www.nrc.gov/reading-rm/doc-collections/: 

• IMC 0102, “Oversight and Objectivity of Inspectors and Examiners at Reactor Facilities” 

• IMC 0305, “Operating Reactor Assessment Program” 

• IMC 0307, “Reactor Oversight Process Self-Assessment Program” 

• IMC 0308, “Reactor Oversight Process (ROP) Basis Document” 

• IMC 0310, “Components within the Cross-Cutting Areas” 

• IMC 0608, “Performance Indicator Program” 

• IMC 0609, “Significance Determination Process” 

• IMC 0609, Attachment 1, “Significance and Enforcement Review Panel Process” 

• IMC 0612, “Power Reactor Inspection Reports” 

• IMC 0612, Appendix B, “Issue Screening” 

• IMC 0612, Appendix E, “Examples of Minor Issues” 

• IMC 2515, “Light-Water Reactor Inspection Program – Operations Phase” 

• IP 71111.21, “Component Design Bases Inspection” 

• IP 71152, “Problem Identification and Resolution” 

• IP 95002, “Inspection for One Degraded Cornerstone or Any Three White Inputs in a 

Strategic Performance Area” 

• IP 95003, “Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple 

Degraded Cornerstones, Multiple Yellow Inputs or One Red Input” 

• NUREG/CR-6850, “EPRI/NRC-RES Fire PRA Methodology for Nuclear Power Facilities” 

• NUREG-1022, “Event Reporting Guidelines: 10 CFR 50.72 and 50.73” 

• RG 1.200, Revision 2, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities” 

• SECY-09-0143, “Status of the Accident Sequence Precursor Program and the Standardized 

Plant Analysis Risk Models” 

• SECY-98-144, “White Paper on Risk-Informed and Performance-Based Regulation” 

• SECY-99-007, “Recommendations for Reactor Oversight Process Improvements” 

Available from http://www.nrc.gov/reactors/operating/oversight/program-documents.html: 

• RASP Handbook 

Not Publicly Available: 

• IMC 0609, Appendix E, Part I, “Baseline Security Significance Determination Process for 

Power Reactors” 

• IMC 0609, Appendix E, Part II, “Force-on-Force Physical Protection Significance 

Determination Process for Power Reactors” 

• SPAR Model Quality Assurance Plan 

• TI 2515/171, “Verification of Site Specific Implementation of B.5.b Phase 2 & 3 Mitigating 

Strategies”