CONSOLIDATED RESPONSE TO THE 

2007 REACTOR OVERSIGHT PROCESS EXTERNAL SURVEY

This document contains the consolidated results of the Federal Register notice (FRN) that 

solicited external stakeholder comment and feedback on the Reactor Oversight Process (ROP). 

The FRN, entitled ASolicitation of Public Comments on the Implementation of the Reactor 

Oversight Process,@ was published on October 11, 2007 (Agencywide Documents Access and 

Management System (ADAMS) Accession Number ML072700055). This notice was part of an 

ongoing effort by the Nuclear Regulatory Commission (NRC) staff to obtain external stakeholder 

input regarding the efficacy of the ROP. The comment period expired on December 7, 2007. 

In an effort to actively solicit feedback on the implementation of the ROP, the NRC staff: (1) 

mailed approximately 700 surveys directly to stakeholders, (2) placed a direct link to the survey 

information on both the ROP Web page and the ADocuments for Comment@ page of the NRC=s 

external Web site, and (3) issued a press release and posted it on the NRC=s external Web site. 

The staff has prepared a consolidated response each year beginning with the calendar year 

(CY) 2004 external survey, and formalized the process with its revision to Inspection Manual 

Chapter (IMC) 0307, AReactor Oversight Process Self-Assessment Program.@ These responses 

are all available on the ROP Program Evaluation and Stakeholder Feedback webpage. 

In CY 2007, the staff continued to focus on stakeholder involvement and open communication 

regarding the ROP. The staff used a variety of communication methods, in this case, the 

external survey, to ensure that all stakeholders were able to access ROP information and were 

able to participate in the process and provide feedback. The staff seeks and implements 

improvements to the ROP based on feedback and insights from all stakeholders. 

Following the 2007 ROP external survey, the NRC staff has again developed a response to the 

comments received. This includes consolidating the comments by survey question and 

providing a response. The respondent comments for each question are listed in chronological 

order as received and the NRC responses are provided in no particular order. 

The questions used in the questionnaire were developed for the NRC staff to gain specific 

feedback regarding the ROP=s performance metrics as described in IMC 0307. This allows the 

NRC staff to assess whether the metrics are meeting the required criteria. A table showing the 

relation of each survey question to its specified performance metric can be found in section 2 on 

pages 7 through 9. 

For those respondents who made general comments that were not directed to a specific 

question, the comments were listed as a response to question #21 (additional information or 

comments on other program areas related to the ROP). In order to provide a more concise 

response to comments in this category, those comments were grouped and addressed by the 

appropriate ROP program areas of performance indicators, inspection, significance 

determination process, and assessment. Comments that were outside of these areas are 

addressed under the category of other. 

The NRC staff attempted to present stakeholder comments in this document exactly as they 

were received; therefore, no corrections were made for spelling or grammatical errors on the 

FRN responses (but corrections have been included in parenthesis for clarification purposes as 

needed). The accession numbers from ADAMS after each respondent has also been provided 

for access to the official record copy of the specific FRN response. 
- 2 -

Comments were received from the following respondents (listed in chronological order as 

received). 

J. Foster (Public) (ML072960472) 

Union of Concerned Scientists (Public) (ML073600850) 

Entergy (Industry) (ML073600853) 

Department of Health, State of Ohio (State) (ML073600856) 

Nuclear Energy Institute (Industry) (ML073600803) 

Strategic Teaming and Resource Sharing (Industry) (ML073600835) 

Region IV Utility Group (Industry) (ML073600840) 

Four of the seven responses came from the Nuclear Energy Institute (NEI) or utilities endorsing 

the NEI response, one response came from State or local agencies, and two responses came 

from public interest groups or members of the public. The overall number of responses 

decreased from last year. However, the number of responses received this year was similar to 

the number of responses that included comments in last year=s survey. The most noticeable 

decrease was in State, and local agency responses. 

Each question number includes all comments received from stakeholders, followed by the 

NRC=s response to those comments. The Table of Contents on the following pages (pages 3-6) 

can be used to find the comments and responses to specific survey questions. To the extent 

practicable, the relevant portions of the annual staff paper to the Commission (SECY-08-0046, 

ML080460148) and the annual ROP metric report (ADAMS ML080350368) are referenced to 

demonstrate how the NRC staff addressed the comments. 

In some cases the NRC staff plans to consider the specific comments and suggested 

improvements in future revisions to program guidance. Accordingly, some issues will be entered 

into the ROP issue tracking system as feedback forms in accordance with IMC 0801, AReactor 

Oversight Process Feedback Program,@ to ensure these issues are considered and tracked to 

resolution. 

This consolidated response, along with the Commission paper and the annual ROP 

performance metric report, will be posted to the ROP Web page and sent to each respondent to 

the survey. 
- 3 - 

Section 1 Table of Contents

1. Does the Performance Indicator Program provide useful insights to help ensure plant 

safety? 

Respondent Comments: Page 10 

NRC Response: Page 11 

2. Does appropriate overlap exist between the Performance Indicator Program and the 

Inspection Program? 

Respondent Comments: Page 12 

NRC Response: Page 14 

3. Does NEI 99-02, “Regulatory Assessment Performance Indicator Guideline” provide 

clear guidance regarding Performance Indicators? 

Respondent Comments: Page 15 

NRC Response: Page 16 

4. Can the Performance Indicator Program, including the Mitigating Systems Performance 

Index, effectively identify performance outliers based on risk-informed, objective, and 

predictable indicators? 

 Respondent Comments: Page 17 

NRC Response: Page 18 

5. Does the Inspection Program adequately cover areas important to safety, and is it 

effective in identifying and ensuring the prompt correction of performance deficiencies? 

Respondent Comments: Page 19 

NRC Response: Page 21 

6. Is the information contained in inspection reports relevant, useful, and written in plain 

English? 

Respondent Comments: Page 23 

NRC Response: Page 24 

7. Does the Significance Determination Process result in an objective and understandable 

regulatory response to performance issues? 

Respondent Comments: Page 25 

NRC Response: Page 26 

8. Does the NRC take appropriate actions to address performance issues for those plants 

with identified performance deficiencies? 

Respondent Comments: Page 28 

NRC Response: Page 29 

9. Is the information contained in assessment reports relevant, useful and written in plain 

English? 

Respondent Comments: Page 31 

NRC Response: Page 32 
- 4 - 

10. Are the ROP oversight activities predictable (i.e., controlled by the process) and 

reasonably objective (i.e., based on supported facts, rather than relying on subjective 

judgment)? 

Respondent Comments: Page 33 

NRC Response: Page 36 

11. Is the ROP risk-informed, in that the NRC’s actions are appropriately graduated on the 

basis of increased significance? 

Respondent Comments: Page 37 

NRC Response: Page 38 

12. Is the ROP understandable and are the processes, procedures and products clear and 

written in plain English? 

Respondent Comments: Page 40 

NRC Response: Page 41 

13. Does the ROP provide adequate regulatory assurance, when combined with other NRC 

regulatory processes, that plants are being operated and maintained safely? 

Respondent Comments: Page 42 

NRC Response: Page 43 

14. Is the ROP effective, efficient, realistic, and timely?? 

 Respondent Comments: Page 44 

NRC Response: Page 46 

15. Does the ROP ensure openness in the regulatory process? 

Respondent Comments: Page 48 

NRC Response: Page 50 

16. Has the public been afforded adequate opportunity to participate in the ROP and to 

provide inputs and comments? 

Respondent Comments: Page 51 

NRC Response: Page 52 

17. Has the NRC been responsive to public inputs and comments on the ROP? 

Respondent Comments: Page 53 

NRC Response: Page 54 

18. Has the NRC implemented the ROP as defined by program documents? 

Respondent Comments: Page 54 

NRC Response: Page 55 

19. Does the ROP result in unintended consequences? 

Respondent Comments: Page 57 

NRC Response: Page 57 
- 5 - 

20a. Does Do the ROP inspection and assessment safety culture enhancements help to focus 

licensee and NRC attention on performance issues associated with aspects of safety 

culture? 

Respondent Comments: Page 58 

NRC Response: Page 59 

20b. Would Do the baseline Identification and Resolution of Problems inspection procedure 

(71152) and the special inspection procedures (93800 and 93812 respectively) provide 

an appropriate level of guidance on safety culture aspects and on the consideration of 

causal factors related to safety culture? 

 Respondent Comments: Page 60 

 NRC Response: Page 61 

20c. Do the supplemental inspection procedures (Inspection for One or Two White Inputs in a 

Strategic Performance Area (95001), Inspection for One Degraded Cornerstone or any 

Three White Inputs in a Strategic Performance Area (95002)) respectively provide an 

appropriate level of guidance to evaluate whether safety culture components have been 

adequately considered as part of the licensees’ root cause, extent of condition, and 

extent of cause evaluations and to independently determine if safety culture components 

caused or significantly contributed to the risk significant performance issues? 

 Respondent Comments: Page 62 

NRC Response: Page 63 

20d. Does the procedure for a Supplemental Inspection for Repetitive Degraded 

Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs, or One Red Input 

(95003) provide an appropriate level of guidance to independently assess the licensee’s 

safety culture and evaluate the licensee’s assessment of their safety culture? 

 Respondent Comments: Page 63 

NRC Response: Page 64 

20e. Do the ROP inspection reports clearly describe inspection finding cross-cutting aspects? 

 Respondent Comments: Page 65 

NRC Response: Page 66 

20f. Do the Operating Reactor Assessment Program (0305) cross-cutting components and 

cross-cutting aspects provide an adequate coverage of the cross-cutting areas? 

 Respondent Comments: Page 66 

NRC Response: Page 68 

21. Please provide any additional information or comments related to the Reactor Oversight 

Process. 

Performance Indicator Section

Respondent Comments: Page 68 

NRC Response: Page 68 

Inspection Section

Respondent Comments: Page 70 

NRC Response: Page 70 
- 6 - 

Significance Determination Process Section

Respondent Comments: Page 71 

NRC Response: Page 71 

Assessment Section

Respondent Comments: Page 72 

NRC Response: Page 72 

Other Section

Respondent Comments: Page 74 

NRC Response: Page 74 
- 7 - 

Section 2 Survey Questions and Corresponding Metrics 

2007 External Survey Question 

ROP 

Performance 

Metric 

ROP Metric Des

1. Does the Performance Indicator Program provide useful insights to help ensure plant safety? PI-4 PI Program Provides Insights t

Safety 

2. Does appropriate overlap exist between the Performance Indicator Program and the Inspection Program? 

PI-6 Stakeholders Perceive Approp

the PI Program and Inspection 

3. Does NEI 99-02, ARegulatory Assessment Performance Indicator 

Guideline@ provide clear guidance regarding Performance Indicators? 

PI-7 Clarity of Performance Indicato

4. Can Performance Indicator Program, including the Mitigating Systems Performance x, effectively identify performance outliers based on risk-informed, objective, and predictable indicators? 

PI-8 PI Program Identifies Performa

Objective and Predictable Man

5. Does the Inspection Program adequately cover areas important to safety, and is it effective in identifying and ensuring the prompt correction of any performance deficiencies? 

IP-8 Inspection Program Effectiven

Covering Areas Important to S

6. Is the information contained in inspection reports relevant, useful, and written in plain English? 

IP-7 Inspection Reports Are Releva

in Plain Language 

7. Does the Significance Determination result in an objective and understandable regulatory response to performance issues? 

SDP-4 The SDP Provides an Objectiv

Regulatory Response to Perfo

8. Does the NRC take appropriate actions to address performance issues for those plants with identified performance deficiencies? 

AS-5 NRC Takes Appropriate Action

Performance Issues 

9. Is the information contained in assessment reports relevant, useful and written in plain English? 

AS-6 Assessment Reports are Relev

Written in Plain Language 

10. Are the ROP oversight activities predictable (i.e., controlled by the process) and reasonably objective (i.e., based on supported facts, rather than relying on subjective judgment)? 

O-1 Stakeholders Perceive the RO

and Objective 

11. Is the ROP risk-informed, in that the NRC=s actions and 

outcomes are appropriately graduated on the basis of increased significance? 

O-2 Stakeholders Perceive the RO

12. Is the ROP understandable and are the processes, procedures and products clear and written in plain English? 

O-3 Stakeholders Perceive the RO

Understandable 
- 8 - 

2007 External Survey Question 

ROP 

Performance 

Metric 

ROP Metric Des

13. Does the ROP provide adequate regulatory assurance when combined with other NRC regulatory processes that plants are being operated and maintained safely? 

O-4 Stakeholders Perceive that the 

Adequate Regulatory AssurancOperated and Maintained Safe

14. Is the ROP effective, efficient, realistic, and timely? O-5 Stakeholders Perceive the RO

Efficient, Realistic, and Timely 

15. Does the ROP ensure openness in the regulatory process? O-6 Stakeholders Perceive that the 

Openness 

16. Has the public been afforded adequate opportunity to participate in the ROP and to provide inputs and comments? 

O-7 Opportunities for Public Partici

17. Has the NRC been responsive to public inputs and comments on the ROP? 

O-8 Stakeholders Perceive the NR

its Inputs and Comments 

18. Has the NRC implemented the ROP as defined by program documents? 

O-9 Stakeholders Perceive that the 

as Defined 

19. Does the ROP result in unintended consequences? O-10 Stakeholders Perceive that the 

in Unintended Consequences 

20a. Do the ROP inspection and assessment safety culture enhancements help to focus licensee and NRC attention on performance issues associated with aspects of safety culture? 

AS-8 Perceived Effectiveness of Saf

Enhancements 

20b. Do the baseline Identification and Resolution of Problems inspection procedure (71152) and the special inspection procedures (93800 and 93812 respectively) provide an appropriate level of guidance on safety culture aspects and on the consideration of causal factors related to safety culture? 

AS-8 Perceived Effectiveness of Saf

Enhancements 

20c. Do the supplemental inspection procedures (Inspection for One or Two White Inputs in a Strategic Performance Area (95001), Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area (95002)) respectively provide an appropriate level of guidance to evaluate whether safety culture components have been adequately considered as part of the licensees’ root cause, extent of condition, and extent of cause evaluations and to independently determine if safety culture components caused or significantly contributed to the risk significant performance issues? 

AS-8 Perceived Effectiveness of Saf

Enhancements 
- 9 - 

2007 External Survey Question 

ROP 

Performance 

Metric 

ROP Metric Des

20d. Does the procedure for a Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs, or One Red Input (95003) provide an appropriate level of guidance to independently assess the licensee’s safety culture and evaluate the licensee’s assessment of their safety culture? 

AS-8 Perceived Effectiveness of S

Enhancements 

20e. Do the ROP inspection reports clearly describe inspection finding cross-cutting aspects? 20f. Do the Operating Reactor Assessment Program (0305) cross-cutting components and cross-cutting aspects provide an adequate coverage of the cross-cutting areas? 21. Please provide any additional information or comments related to the Reactor Oversight Process 
- 10 - 

Section 3: Survey Questions and Responses

1. Does the Performance Indicator Program provide useful insights to help ensure 

plant safety? 

Respondent Comments:

Nuclear Energy Institute

The Performance Indicator (PI) Program, in conjunction with the Inspection Program, continues 

to support the NRC’s overall mission to ensure that commercial nuclear power plants are 

operated in a manner that provides adequate protection of public health and safety. Useful 

insights are provided by the performance indicators. The level of insight provided is improved 

when a clear nexus can be made between performance indicator results and plant safety. The 

Mitigating Systems Performance Index (MSPI) provides very useful insights by providing a riskinformed measure of performance for key systems and components. Efforts should continue to 

better risk-inform the performance indicators and improve the level of insight they provide. 

Strategic Teaming and Resource Sharing

The Performance Indicators (PIs) have developed into performance standards that the industry 

strives to exceed. Since the PIs are based on NRC defined acceptable limits, they reinforce 

industry and Licensee safety performance. The program enhancements made to the PIs in the 

Initiating Events and Mitigating Systems cornerstones (i.e. Scrams with Complications and 

Mitigating Systems Performance Index PIs) help maintain risk informed focus on equipment and 

plant operations most important to plant safety. While more time is needed to fully assess the 

effectiveness of these program enhancements, initial indications are that the changes have 

improved the program. In addition, the technical basis documents that have been developed to 

support the changes help provide a better understanding of the safety basis for the program. 

Union of Concerned Scientists

No. As the September 2006 report by the U.S. Government Accountability Office (GAO-06-1029, 

“Nuclear Regulatory Commission: Oversight of Nuclear Power Plant Safety Has Improved, but 

Refinements Are Needed”) documented, the numbers of greater-than-green performance 

indicators (PIs) in 2003, 2004, and 2005 were 41, 33, and 23 respectively. Yet the numbers of 

reactors receiving heightened NRC attention over those same years were 36, 37, and 32. 

Clearly, the performance improvement suggested by the significant reduction in greater-thangreen PIs was bogus; otherwise, the number of reactors needed heightened NRC attention 

would also have declined. The “greenwashing” of the PIs reflects the efforts of the nuclear 

industry to pervert the system and render it incapable of detecting performance drops. They 

have been successful in their perversion. It’s sad. 

The above comments may or may not apply to the physical protection PIs. Since August 2004, 

the Commission has wrongfully and shortsightedly withheld this information from the American 

public and therefore no one, except the industry, can comment on the useful insights from the 

physical protection PIs. 
- 11 - 

Foster

The Performance Indicator Program provides only some insights to plant safety, as the DavisBesse head corrosion event shows. It may be difficult to show that some Indicators are actually 

related to plant safety. 

State of Ohio

Yes, it appears that performance indicators are based on the cornerstones of operating safety. 

But it’s unclear whether the indicators are “re-set” periodically to reflect changes in observed 

occurrences and current expectations. 

Region IV Utility Group

The Performance Indicators (PIs) have developed into performance standards that the industry 

strives to meet. Since the PIs are based on NRC defined acceptable limits, they reinforce 

industry and licensee safety performance. Implementation of MSPI is considered an 

enhancement by adding a more risk informed performance indicator to the PI program. 

The new Unplanned SCRAMS with Complications (USwC) PI appears to be better focused with 

more clear guidance. This new PI takes into account the risk informed safety significance of 

specific SCRAMS. However, more run time will be necessary to draw meaningful conclusions. 

NRC Response:

As noted in the ROP self-assessment and metric report for PI-4, “PI Program Provides Insights 

to Help Ensure Plant Safety,” responses from the public did not believe that the PI Program 

helps ensure plant safety. The State respondent agreed that PIs are based on operating safety 

but questioned whether the indicators in the PI program are periodically “re-set.” Industry 

responses were generally in agreement that the PI program does provide useful insights to 

ensure plant safety but noted that efforts should continue to better risk-inform the PIs. As a 

result, this metric did not meet its criteria because it received significant feedback that the PIs do 

not provide an adequate indication of declining safety performance. The staff recognizes the 

need to improve the PI Program to provide more meaningful indications of declining plant 

performance. 

With regards to the “re-set” of indicators, the staff believes that one goal of the PI program is to 

identify the level of performance that is acceptable without additional NRC oversight and to 

respond to potential declining licensee performance if it reaches criterion thresholds. To change 

these thresholds without a specific safety-related purpose would be ratcheting licensees into 

higher levels of performance. Since the staff believes the current thresholds are indicative of an 

adequate level of performance, “re-set” of the thresholds would be unjustified. 

There was a comment from a public interest group that there have been a declining number of 

greater-than-green PIs but not a proportional decrease in the number of reactors receiving 

heightened NRC attention. As noted in Enclosure 1 to SECY-08-0046, the staff continued to 

improve the PI program in CY 2007 to provide more meaningful indication of declining plant 

performance and to identify outliers. For example, the NRC replaced the Unplanned Scrams 

with Loss of Normal Heat Removal (USwLONHR) PI with the Unplanned Scrams with 

Complications (USwC) PI in the third quarter of 2007 as a result of a joint industry and NRC staff 

effort. The MSPI provided a significant input to the ROP Action Matrix; of the 16 new greaterthan-green PIs in CY 2007, 10 were from MSPI. The staff also recognizes the need to further 

improve the PI program to provide more timely and meaningful indications of plant performance 
- 12 - 

and will continue to work with the industry to revise and/or introduce other PIs to improve the 

program’s effectiveness in contributing to the identification of declining performance. 

Regarding the comment that PIs only provide some insight to plant safety, referring to the Davis 

Besse plant; and that it was difficult to show that some PIs are actually related to plant safety, it 

is important to remember that the PIs were designed to be a part of the ROP along with the 

baseline inspection program. The PIs, in conjunction with direct inspection, provide a 

comprehensive review of licensee performance. It would be inappropriate to rely on PIs alone to 

reveal declining performance. 

The staff continues to evaluate ways to identify declining performance. Performance 

weaknesses may be surfacing in areas not easily identifiable by numerical indicators. One way 

the staff is attempting to address this issue is through enhancements to the ROP safety culture 

program. Revisions to various inspection procedures will take place in 2008. 

With respect to the comment on the physical protection PIs, the objective of this cornerstone is 

to provide assurance that the licensees' security system and material control and accounting 

program use a defense-in-depth approach and can protect against (1) the design basis threat if 

radiological sabotage from external and internal threats, and (2) the theft of loss of radiological 

materials. Although the NRC is actively overseeing the security cornerstone, the Commission 

has decided that certain security related inspection and assessment information will not be 

publicly available to ensure that potentially useful information is not provided to a possible 

adversary. Therefore, only the cover letters to security inspection reports may be viewed at this 

point. However, the staff is seeking ways to increase the openness and transparency in the 

security inspection programs. The staff will hold a series of public meetings in 2008 on security 

openness and explore such topics as making the results of security inspections public, 

reinstating public reporting of PI results, and making NRC security inspection procedures and 

end-of-cycle and mid-cycle security cornerstone assessment letters public. Any significant 

information and useful feedback would be reported to the Commission to request approval to 

implement an option to increase transparency and public availability of security inspection 

program information. 

2. Does appropriate overlap exist between the Performance Indicator Program and the 

Inspection Program?

Respondent Comments:

Nuclear Energy Institute

Performance Indicators are most effective in areas where clear performance thresholds can be 

developed. This then allows the inspection program resources to be more appropriately 

allocated to areas that are best addressed through direct evaluation and inspection. Some 

overlap between performance indicator and inspection is inevitable and appropriate. However, it 

takes vigilance on the part of NRC to ensure the objectives of the ROP are not compromised by 

double counting inspection findings and performance indicator results that arise from the same 

event. 

One area where overlap exists is in the Mitigating Systems cornerstone. There have been recent 

instances where inspection findings and greater-then-green PIs have resulted from the same 
- 13 - 

event/conditions. Because the risk-significance of component failures is directly measured in the 

Mitigating Systems Performance Index (MSPI), the performance of an SDP evaluation for the same 

component failures should be limited to circumstances where contributing factors, beyond the 

measure of MSPI, are identified. In instances where an SDP evaluation and MSPI result address the 

same event, it is important to ensure that the ROP recognizes the overlap and avoids “double 

counting” of PI and SDP results in the determination of appropriate regulatory actions. 

Strategic Teaming and Resource Sharing

Performance Indicators look at the areas where clear performance thresholds have been developed. 

This allows the inspection program to spend more time looking at those areas that require more 

evaluation and investigation. The process is well integrated and, while overlap exists, the overlap 

seems appropriate in most cornerstones. 

Entergy

In general, the appropriate overlap exists between the Performance Indicator Program and the 

Inspection Program. In the area of performance issues that result in both a Finding in the 

inspection program and have a negative impact on a performance indicator the potential exists 

for an inappropriate impact of the event on the Action Matrix considering the safety significance 

of the event. 

Manual Chapter 0305 addresses double counting of performance indicators and inspection 

findings. The applicable guidance states that issues with the same underlying causes should 

not be double-counted. Furthermore, an example is provided where inoperability of a support 

system that causes a White inspection finding, as well as several performance indicators to 

cross the green/white threshold, should only be counted as a single white input because the 

finding pertains to the same underlying issue." 

Appropriate guidance needs to be developed to ensure the Action Matrix does not double count 

issues associated with a greater than Green Finding by considering their impact on Performance 

Indicators. Double counting these issues would result in an incorrect representation of licensee 

performance. 

Union of Concerned Scientists

No. As detailed in the response to Question 1, the PIs have been rendered as useless as an 

appendix in a mannequin as far as providing indications of licensee performance. 

It is impossible to comment on the overlap between the Performance Indicator Program and the 

Inspection Program for physical protection since the Commission wrongfully and shortsightedly 

withheld this information from the American public since August 2004. 

Foster

This is a difficult question; again, the easy answer is yes, but Davis-Besse head corrosion slipped by 

since the agency had paid little, if any attention to head corrosion issues (what other issues are not 

addressed?). 

State of Ohio

It appears that the PI and Inspection Program results are intended to work together in determining a 

comprehensive evaluation of licensee performance, but we are not knowledgeable enough to 

ascertain this information. 
- 14 - 

Region IV Utility Group

Performance indicators look at the areas where clear performance thresholds have been developed. 

This allows the inspection program to spend more time looking at those areas that require evaluation 

and investigation. The process is well integrated and, while overlap exists, the overlap seems 

appropriate. 

NRC Response:

As noted in the ROP self-assessment and metric report for PI-6, “Stakeholders Perceive 

Appropriate Overlap between the PI Program and Inspection Program,” public response varied 

regarding proper overlap between the PI and inspection programs. One public response again 

noted the ineffectiveness of the PI program. Another public response agreed that appropriate 

overlap exists but gave an example of not enough overlap to catch some safety issues. The 

State response indicated that there appears to be appropriate overlap but was not 

knowledgeable enough to ascertain the information. Industry comments noted appropriate 

overlap overall, but some stated concerns with possible double counting issues that arise from 

the same event. Overall, this metric did meet its criteria because there was a decline in the 

number of negative comments. 

Industry comments noted appropriate overlap overall, but some stated concerns with possible 

double counting of inspection findings and performance indicator results that arise from the 

same event. The staff has recognized that double counting issues may exist between MSPI, 

other PIs, and inspection findings. This has been resolved and revised guidance will be issued 

with the next revision of IMC 0305 that will outline specific steps required to determine if these 

double counting issues exist and what actions may need to be taken. Licensees may be 

required to perform calculations that extract the common event (i.e., failure, scram) from the PIs 

to determine if the new PI value would remain greater-than-green. If the new value is not greater 

than green, then the PI is not an input into the Action Matrix. If the PI remains greater than 

green, then it is a valid input into the Action Matrix and both the PI and finding are included and 

this is not considered double counting. 

As noted in Enclosure 1 to SECY-08-0046, the staff is currently making significant efforts, “to 

assess the effectiveness of the MSPI since it was implemented nearly 2 years ago in 2006. The 

most significant effort is the MSPI lessons learned review being conducted by the Office of 

Nuclear Regulatory Research (RES). This review, which will use data collected over the past 24 

months, will focus on detecting identifiable trends and outliers in performance, aspects of the 

MSPI guidance that could be improved, and areas of the MSPI that are not providing benefit in 

assessing performance in either unavailability or unreliability. The staff will periodically update 

the industry on its progress during this review and will share its findings during the monthly ROP 

public meetings. The industry is also conducting a review of the MSPI, and the staff and 

industry will integrate both results in the future.” 

With respect to the comments on the security PIs and Davis Besse plant, the staff addressed 

similar comments in its response to Question 1 of this document. 
- 15 - 

3. Does NEI 99-02, ARegulatory Assessment Performance Indicator Guideline@ provide 

clear guidance regarding Performance Indicators?

Respondent Comments:

Nuclear Energy Institute

Although NEI 99-02 provides sufficient guidance for most situations, questions do arise. The 

FAQ process and the monthly ROP meetings are the mechanisms to resolve questions with the 

guidance. The ROP meetings are held monthly with the NRC to discuss and resolve questions. 

When resolution is not achieved, an FAQ appeal process is available and has proved to be an 

effective means to resolve plant specific issues. 

Strategic Teaming and Resource Sharing

While questions on the guidance do arise, a formal Frequently Asked Question (FAQ) process is 

available to the industry and NRC inspectors to resolve questions with the guidance. An industry 

task force (the Reactor Oversight Process Task Force or ROPTF) meets monthly with the 

inspection and assessment branch of the NRC to discuss and resolve the questions that arise 

with the guidance. When resolution is not achieved at the monthly Reactor Oversight Process 

(ROP) meetings, an FAQ appeal process is available and has been used to drive issues to 

resolution. The FAQ process and appeal process have proven to be effective and should be 

maintained. 

Union of Concerned Scientists

Sure. How hard can “all green all the time” be? 

Foster

Combined with the FAQs, yes. 

State of Ohio

The revised document is guidance for the licensee to help reduce unnecessary reporting burden, 

provide clearer guidance, and aid the accurate and consistent reporting of PI information. We have 

not applied the guidance in this manner, therefore, this question is more suited to the licensees. 

Region IV Utility Group

While questions on the guidance do arise, the FAQ process is responsive to those questions. 

The guidance is then updated periodically based on the FAQs to enhance the guidance in an 

ongoing process. The Reactor Oversight Process Task Force (ROPTF) monthly meeting, in 

concert with its NRC counterpart meeting, is an effective means to drive these issues to 

resolution. The current FAQ appeal process is an effective tool and should be maintained. 

The introduction of the Mitigating Systems Performance Index (MSPI) as a replacement 

performance indicator for the Safety System Unavailability performance indicator introduced the 

concept of using component reliability along with train unavailability as a means of estimating the 

total probability that a system would be unable to perform its risk significant function when called 

upon. 

An issue has emerged with regard to determination of MSPI unavailability (UA). Current 

guidance is subject to interpretation regarding treatment of a newly discovered condition when a 

subsequent investigation determines the condition may have existed for some period of time 

prior to its discovery. This has led to discussions over what is the correct “time of discovery” to 
- 16 - 

be used as the starting point for counting UA. Clarification of this “failed discovery” or “time of 

discovery” question is necessary to ensure the appropriate UA is applied to each MSPI input. 

RUG IV believes the UA that is already accounted for through the reliability part of the MSPI 

must be also be considered. The inclusion of a failure of a component in the index calculation is 

equivalent to a given amount of unavailability. Counting both the unavailability and the failure 

would result in “double counting” the risk impact of the condition. Recent FAQs addressing this 

concern have not resulted in resolution of this difference in interpretation. RUG IV is encouraged 

that discussions have also recognized that MSPI does account for, but may not precisely 

capture the effects of latent defects such as errors that are identified through design analysis. In 

these cases, the ROP significance determination process may be the more appropriate tool for 

addressing performance issues associated with failed discovery. The event response aspects of 

the ROP may also require specific inspection activities to evaluate issues that are also within the 

scope of MSPI. The ROP does include provisions for addressing double counting of assessment 

inputs, should there be both an inspection and PI input to consider for assessment. RUG IV 

encourages continued efforts to ensure consistent implementation of the NEI 99-02 guidance, 

with appropriate clarifications as issues are resolved. 

NRC Response: 

As noted in the ROP self-assessment and metric report for PI-7, “Clarity of Performance 

Indicator Guidance,” external stakeholders generally agreed that the guidance was clear. 

Industry stakeholders also gave positive feedback on the FAQ process and the appeal process. 

Overall, this metric met its criteria. 

One public response stated that combined with the FAQ process, the guidance was clear. 

The State stakeholder generally felt that the guidance was clear but that it would be more 

appropriate for the licensees to provide comments on effectiveness of the PI guidance. 

Industry respondents commented that the PI guidance is clear and that the FAQ process 

and the appeal process have proven to be responsive and effective in addressing questions 

and resolving issues. 

An industry group raised an issue regarding MSPI double counting. The staff has 

addressed the same comment in its response to Question 2 of this document. 

With regards to the MSPI, the NRC continues to address specific changes via the ROP 

Monthly Meetings and the FAQ process. NEI and NRC have reached general agreement 

on modifications to the “time of discovery” issue, and the changes will be approved in future 

ROP Monthly Meetings. In addition, several other clarifications/modifications to the MSPI 

are under discussion between the NRC and the industry. Further discussions and 

adjustments will be considered, as well as dialogue during an NEI-sponsored MSPI 

workshop to be held in August 2008. 

As the staff identifies areas of weakness or areas needing additional clarification in NEI 99-

02, proposed changes are presented to the ROP Task Force for consideration. 
- 17 - 

4. Can Performance Indicator Program, including the Mitigating Systems 

Performance Index, effectively identify performance outliers based on riskinformed, objective, and predictable indicators?

Respondent Comments:

Nuclear Energy Institute

The Performance Indicator Program is a useful and effective tool for identifying adverse 

performance trends in those areas where measures have been established. By design, not all 

areas of the ROP are monitored by the PI program. The PI program is supplemented in all areas 

by the Inspection Program. While Performance Indicators are effective in providing an indication 

of performance trends, there remains a need for licensees and NRC to investigate performance 

trends to determine if the indications are valid. 

One measure of the effectiveness of performance indicators in identifying declining performance 

is the number of instances where greater-than-green performance indicators occur in advance of 

any greater-than-green inspection findings in the associated area. 

Strategic Teaming and Resource Sharing

Some PIs produce more visible trends than others. Visible trends are more clearly seen in the 

Initiating Events, Emergency Preparedness and Occupational Radiation Safety cornerstones 

and tend to be objective and predictable. However, not all PI results produce visible trends and 

not all are risk informed. The MSPI is the first risk-based indicator and identifies conditions 

based on risk implications but does not typically produce a visible trend. Most other PIs have 

limited risk insights and may inaccurately identify risk significant conditions. A current example is 

the Yellow Alert and Notification System (ANS) PI at Cook Units 1 and 2. The ANS design and 

implementation is approved by FEMA with recognized limitations that are mitigated through the 

use of diverse media and actions specified under the emergency plan. A Yellow band for this PI 

is overly conservative. 

The industry and NRC staff should continue efforts to better risk inform all of the PIs. 

Union of Concerned Scientists

It can. It did initially. It doesn’t now. The original PIs had all those admirable attributes and 

served very well to distinguish between risk-informed performance differences. But those 

licensees identified as falling short objected in a predictable manner and undertook to blur the 

line. Through devices like the infamous FAQ [frequently asked question] process, the nuclear 

industry has been able to maintain the PI thresholds as-is but re-define being on either side of 

that line as “green.” The PIs are now useless measures not worth the time or attention of 

kindergarten students. 

The PIs for physical protection may or may not identify declining performance. It’s impossible to tell 

since the Commission wrongfully and shortsightedly withheld such information from the American 

public since August 2004. 

Foster

The best answer is “maybe.” This depends on the type of performance involved and the indicator 

being discussed. 
- 18 - 

State of Ohio

Yes; this program provides for the ability to track and observe trends, but again we don’t know if the 

values are “normalized” to reflect long term observations or current concerns. 

Region IV Utility Group 

The MSPI is a good risk-informed indicator and does identify conditions based on risk 

implications for the systems monitored. Other indicators may falsely indicate conditions as risk 

significant when they are not, because of the limited risk insights in the design of the PI. The 

industry and NRC staff should continue to develop more risk informed elements for other 

existing indicators. Risk informed PIs also tend to permit early identification of declines in 

performance, which RUG IV believes is our mutual goal. 

NRC Response:

As noted in the ROP self-assessment and metric report for PI-8, “PI Program Identifies 

Performance Outliers in an Objective and Predictable Manner,” comments from external 

stakeholders were generally favorable but with suggested areas for improvement. As a result of 

the decline in the number of negative comments, this metric met its criteria. A public respondent 

alleges the FAQ process has decreased the value of the PI program. The State stakeholder 

noted that the PI program thresholds may need periodic reexamination. Industry responses 

encouraged continued development of risk informed PIs. The staff is continually reviewing and 

revising PIs to provide meaningful indications of plant performance and to better identify 

performance outliers. As questions on PI guidance are identified, the staff seeks to modify the 

guidance to eliminate the unclear language. 

An industry group made a comment regarding a yellow ANS PI and that the yellow band is overly 

conservative. The significance of emergency preparedness issues in the ROP is based upon 

the potential consequences if an aspect of the emergency plan is not successfully implemented 

when necessary. Public notification is considered a risk-significant area in emergency 

preparedness, and as such, the significance of deficiencies is increased. The 90% threshold for 

a yellow ANS PI was established based on FEMA’s 90% requirement for siren reliability 

submitted in each State’s Annual Letter of Certification. At less than 90% reliability, FEMA can 

call into question the standard of reasonable assurance to protect the public. If FEMA withdraws 

reasonable assurance and corrective actions aren’t taken to improve performance, the 

Commission may take action against the licensee. FEMA’s 90% threshold is based on a 

January to December calendar year, where NRC’s PI is measured over the past four quarters. 

The staff, therefore, believes that 90% is not overly conservative, given the potential 

consequences. 

With respect to the comments on the security and normalizing thresholds, the staff addressed 

similar comments in its response to Questions 1 and 2 of this document. 
- 19 - 

5. Does the Inspection Program adequately cover areas important to safety and is it 

effective in identifying and ensuring the prompt correction of performance 

deficiencies? 

Respondent Comments:

Nuclear Energy Institute

It is important that NRC continually monitor the inspection program to identify inspections that 

are inadequate or highly inefficient in identifying meaningful results. The recent ROP 

Realignment review conducted under IMC 0307, Appendix B is an example of how NRC can 

perform such monitoring. However, large-scale efforts such as the realignment review are only 

one method for monitoring and may not be the most efficient. Obtaining, reviewing and acting on 

continuous feedback from licensees and inspection personnel can be more timely and efficient. 

Maintaining a self-critical, questioning position regarding the value of inspection activities is a 

key to keeping the inspection program vital and robust. 

Strategic Teaming and Resource Sharing

The resident inspectors are effective in ensuring areas important to safety are appropriately 

addressed. However, enhancements could be made to the inspection program to make better 

use of the NRC’s generic communications program – especially for inspection issues that have 

generic implications such as technical questions identified during inspections that involve 

development of new regulatory positions. Two examples of inspection issues with generic 

implications are manual actions for response to fires and assessment of post-fire safe shut down 

equipment. A process is needed to ensure early stakeholder involvement in the identification and 

resolution of inspection issues that potentially have generic implications. The current FAQ 

process used for PIs could serve as a model for such a process. Enhanced use of generic 

communications would also promote consistency between the NRC regions. 

Union of Concerned Scientists

The Inspection Program is pretty good at identifying problems. The inspection program applied 

at the Palo Verde plant identified a string of good catches that had been overlooked and/or misdiagnosed by plant workers. NRC Region IV deserves recognition for its commendable efforts at 

Palo Verde. 

The Inspection Program’s largest front-end problem involves the problem identification and 

resolution (PI&R) aspects. Corrective action program deficiencies are common threads among 

serious performance declines at nuclear plants. Yet the PI&R inspections chronically fail to 

notice degradation until conditions make the situation obvious to all. 

UCS believes that the fundamental flaw with the NRC’s approach to corrective action program 

assessments is in the agency’s belief that PI&R inspections provide the necessary insights to 

evaluate program health. History repeatedly demonstrates it simply isn’t true. UCS recommends 

that NRC re-direct all the resources wasted on the futile PI&R inspections into more fruitful 

applications. For example, instead of NRC inspectors examining a small handful of corrective 

action reports and trying to guess the overall health of a corrective action program handling 

more than 2,000 reports annually, better insights would be obtained by having those NRC 

inspectors instead followup on findings from the NRC’s Inspection Program. After all, in theory 

NRC inspectors should find nothing – the licensees own programs should find and fix all safety 
- 20 - 

problems. When NRC inspectors find broken widgets, leaking gaskets, and calculation errors, 

they have also implicitly identified failures of the corrective action programs. 

Far more insightful PI&R assessments would result from NRC inspectors pulling the strings on 

such NRC inspection findings to determine why the corrective action programs failed. Such NRC 

followup would answer vital questions like: 

Were the scope and frequency of the licensees’ testing and inspection programs adequate? 

Are the licensees’ testing and inspection methodologies adequate? 

Were opportunities to identify the problems missed in the past? 

Were prior attempts to remedy the problems adequate? 

The key to success for a limited-scope audit is in the smart selection of the samples examined. 

Under the current PI&R process, the NRC inspectors select the samples almost randomly, 

almost like drawing names from a very large hat. By using NRC inspection findings to define the 

sample, the audit automatically begins with evidence of corrective action program problems. The 

NRC’s PI&R task then condenses to determining whether those problems are isolated in nature 

or reflective of a broader performance deficiency. 

Foster

Yes, but inspectors need more freedom to inspect rather than follow prescribed quotas of items. The 

second answer to this two-part question is no. The program does not ensure prompt corrective 

action; indeed, it does not ensure corrective action, only that an item be entered into the corrective 

action program. 

State of Ohio

Yes, the few inspections we have been able to observe have been very comprehensive and detailed 

and have resulted in attention given to areas of concern which does lead to correction of 

performance deficiencies. 

Region IV Utility Group

Yes. 

The NRC should consider enhancing the use of generic communications when inspection trends 

become evident. RUG IV believes development of a process that parallels the ROP PI & 

Security FAQ processes would be beneficial for resolution of inspection or enforcement issues 

that have potential generic aspects. Issues could be considered to have potential generic 

aspects if they are potentially applicable to more that one licensee or NRC guidance documents 

are affected. The process would be consistent with the principles of good regulation, 

enforcement policy goals and continued improvement of ROP programs and processes. 

Examples of prior inspection issues that could have been evaluated by such a process include; 

manual actions for response to fires, assessment of post-fire safe shut down equipment, and 

technical questions identified during inspections that involve development of new regulatory 

positions. Enhanced use of generic communications would also promote consistency between 

the NRC regions. 

RUG IV believes such a process could be implemented that maintains NRC timeliness goals. RUG 

IV intends to prepare draft documents to facilitate further discussion and establish the feasibility of 

such a process.
- 21 - 

NRC Response:

As noted in the ROP self-assessment and metric report for IP-8, AInspection Program 

Effectiveness and Adequacy in Covering Areas Important to Safety,@ the external survey 

responses to whether the inspection program adequately covers areas that are important to 

safety and is effective in identifying and ensuring the prompt correction of performance 

deficiencies were positive with comments for making improvements. For the most part, the 

external survey responses were positive to the question of whether there was adequate 

inspection coverage of licensee’s programs to ensure prompt identification and correction of 

performance deficiencies. 

Regarding an industry group’s comment on “realignment review,” the Inspection Program Branch 

receives, responds to and makes changes, when appropriate, to the inspection program documents 

based on numerous feedback and reviews from variety of sources throughout the year. For example, 

inspectors in the regions submit about 100 recommendations a year for either clarification or 

improvements to the inspection program using the Feedback Program (IMC 0801). The Inspection 

Program Branch receives feedback from various sources, such as NRC task forces, NRC internal 

working groups and internal meetings (periodic ROP teleconferences and regional division director 

meetings) as ways to improve the inspection program. Additionally, feedback from external 

stakeholders on the inspection program is received during external meetings such as Regional and 

National Utility Group meetings, Licensing Forums and monthly meetings with public stakeholders. 

Also, the Inspection Program Branch periodically conducts a more focused review on inspection data 

to improve the effectiveness of the inspection program. Recently, the Inspection Program Branch 

and the regions completed the second ROP realignment review (IMC 0307 Appendix B) during CY 

2007 to determine the most effective distribution of inspection resources in the baseline inspection 

program. Based on this review, changes were made to about 60% of the baseline inspection 

procedures, including restructuring of the engineering inspections. These improvements to the 

baseline inspection program were implemented starting in CY 2008. 

Regarding an industry group’s comment on generic communication program, the NRC has several 

diverse means of communicating generic issues to the industry and other external public 

stakeholders. One on-going method that has been in place since the conception of the ROP is the 

monthly ROP Public Meeting, in which, generic topics of interest, such as technical questions 

identified during inspections that involve development of new regulatory positions, are presented and 

discussed. In regards to the two examples cited, namely manual actions for response to fires and 

assessment of post-fire safe shut down equipment, the Agency issued an NRC Regulatory Issue 

Summary (RIS) 2006-10 (dated June 30, 2006) to address the regulatory expectations with Appendix 

R, Paragraph III.G.2 Manual Operator Actions. This is one form of Generic Communication used by 

the Agency to communicate matters of regulatory interest with external stakeholders. In addition, 

Enforcement Guidance Memorandum (EGM) 07-004 granting enforcement discretion for Post-Fire 

manual actions used as compensatory measure for fire induced circuit failures was issued on June 

30, 2007. The EGM was published in the Federal Register for public comment. The staff believes 

that the ROP monthly meeting is the most effective way of handling these types of issues. 

Regarding a public interest group’s comment and an industry group’s comment on corrective action 

program and PI&R, IP71152 currently examines a spectrum of licensee corrective action program 

aspects to identify whether the licensee's corrective action program is detecting and preventing 

problems in an appropriate manner. The procedure already specifies that the resolution of non-cited 

violations and programmatic weaknesses identified by supplemental inspections are mandatory 
- 22 - 

samples for the IP71152 biennial inspections. NRC identified issues from other inspections are also 

considered for inspection. The staff disagrees with the characterization that inspection samples are 

selected on an almost random basis. IP71152 provides a number of explicit factors, including the 

risk significance of the issue, for inspectors to consider when selecting PI&R inspection samples. 

Inspection Procedure 71152 Section 03.01 describes how most baseline inspection procedures also 

contain a requirement to inspect PI&R within the baseline inspection area. Inspectors routinely 

perform PI&R reviews for approximately 10 to 15 percent of the baseline inspection resource hours. 

One of the inspection performance attributes (03.01.b.1) is to review the licensee’s identification of 

problems in a timely manner. Implicitly, if the NRC, instead of the licensee, identified a problem, this 

attribute should lead inspectors to query the licensee why its processes were not effective. 

On a broader basis, the implementation of IP71152 has resulted in the identification of a substantial 

number of findings (a total of 115 findings in FY2007 including 2 White findings). In addition, 

IP71152 allows inspectors to identify PI&R related observations and assessments which resulted in 

the licensee taking actions to resolve the identified concerns before they result in NRC findings. The 

staff disagrees with the characterization that the IP71152 inspection resources are being wasted. 

The staff recently completed a best practice review of IP71152 that validated the overall 

effectiveness of IP71152. The review also identified areas of IP71152 that can be further enhanced. 

The staff does not believe it would be prudent to shift all the IP71152 inspection resources into 

examining what went wrong in the licensee corrective action program that resulted in NRC identified 

findings. The staff believes that IP71152 provides an appropriate balance of items inspectors should 

examine. In addition, NRC supplemental inspection procedures currently contain inspection 

guidance (section 03.01.a of IP95001 and IP95002) that “If the NRC identified the White [or 

performance] issue, the [licensee] evaluation should address why licensee processes such as peer 

review, supervisory oversight, inspection testing, self assessments, or quality activities did not identify 

the problem.” This guidance prompts inspectors to examine, for risk significant performance issues, 

why the NRC identified the issue instead of the licensee’s processes. 

The staff will evaluate the merits to augment the IP71152 inspection guidance to include the following 

suggested questions for situations where the NRC has identified the problem: 1) Were opportunities 

to identify the problems by the licensee missed in the past, and 2) were prior attempts by the licensee 

to remedy the problems adequate. 

The inspection program is constructed to ensure that all reactor cornerstone areas receive adequate 

level of inspection. This is accomplished through inspection of key licensee’s programs in each of 

the reactor cornerstone areas using inspection procedures that define the breadth and scope of 

review. Satisfactory completion of each of the inspection procedure is accomplished through 

completion of applicable inspection requirements contained in each of the inspection procedure and 

through completion of the minimum number of samples specified in the inspection procedure. 

Because the regions have been budgeted sufficient inspection resources to complete the nominal 

number of inspection samples, inspectors can shift some inspection resources by completing the 

minimum rather than the maximum number of inspection samples, if conditions warrant. Although 

completion of nominal number of inspection samples at a given site can be challenging depending on 

conditions at the plant, the specifics of inspection requirements and inspection samples in the 

baseline program ensure consistency in application of the baseline inspection program and adequate 

review of licensee’s key programs at all sites. Additionally, the staff has allocated substantial amount 

of inspection resources to resident inspectors. The resources allow resident inspectors to become 
- 23 - 

knowledgeable of plant activities and to perform follow-ups on any potentially risk-significant 

conditions. 

Licensees are required to restore compliance to identified violations of NRC requirements within a 

reasonable time after a violation was identified. Because most identified performance deficiencies 

are determined to be greater-than-minor in significance, they become inspection findings that result 

in either a cited or non-cited violation. Most identified performance deficiencies are corrected within a 

reasonable time period as required. 

6. Is the information contained in inspection reports relevant, useful, and written in plain 

English? 

Respondent Comments:

Nuclear Energy Institute

Inspection reports generally provide sufficient detail for knowledgeable persons to understand 

the areas that were inspected and the conclusions that were reached. 

When documenting findings or violations, often more effort is spent documenting the details of 

the issue then in providing insights as to why the issue is an issue of concern (finding, violation) 

and the nature and true magnitude of the risk significance of the issue. Specifically, it is often 

difficult for a reader to determine why a finding is considered to be of greater then minor 

significance based solely on the language in the inspection report. On many occasions the 

justification is a simple assertion that one of the criteria of IMC 0612, Appendix B, Section 3 is 

met with no further elucidation (e.g., “the finding is associated with one of the cornerstone 

attributes” or “finding affects the associated cornerstone objective”). 

When cross-cutting aspects are assigned to findings/violations, the inspection report 

documentation sometimes lacks a clear explanation as to how the cross-cutting aspect is a 

significant contributor to the cause of the issue. Again, in many cases, the documentation only 

asserts that the cross-cutting aspect is related to (or affects) the cause of the issue, but fails to 

explain how. The link between the cross-cutting aspect and the cause of the finding should be 

clear to the reader of the inspection report. 

Strategic Teaming and Resource Sharing

Generally the reports are relevant, useful and well written. The recent revision to the ROP 

guidance which provides a numbering scheme for the cross-cutting aspects identified in 

inspection reports is a welcomed improvement. 

Union of Concerned Scientists

Yes. 

Foster

No, current inspection reports are so concise that it is often difficult to obtain useful information; 

it is even more difficult to read a past years report and figure out exactly what the inspector 

looked at. 
- 24 - 

State of Ohio

Yes. 

Region IV Utility Group 

Yes. 

Generally, the reports are relevant, useful and written in plain English. 

The recent revision to ROP guidance with the addition of the numbering scheme in MC 0305 for 

crosscutting aspects is an improvement. However, improvements can be made in the 

arrangement of the inspection report. Often it is necessary to review multiple sections of the 

report to identify all of the analysis elements associated with an Inspection Finding. For example, 

to understand the performance deficiency, why the issue is greater than minor, why the issue is 

non-cited or not, and the associated cross-cutting aspect, it may be necessary to review the 

Summary of Findings, the Report Details, and the Supplemental Information sections of the 

inspection report. Enhanced guidance may be necessary to provide consistency in documenting 

cross-cutting aspects. 

The recent revision to MC 2515 regarding Inspection Exit Meetings will help to promote a 

mutual understanding of the issues identified during the inspection and should promote 

enhanced clarity of the inspection report. 

NRC Response: 

As noted in the ROP self-assessment and metric report for IP-7, "Inspection Reports Are 

Relevant, Useful, and Written in Plain Language," external stakeholders responded that the 

inspection reports were clearly written and useful. There were, however, comments for 

improvement in this area. 

An industry interest group made comments concerning finding characterization. Although there 

already is a requirement in ROP program document, Inspection Reports (IMC 0612), to describe 

the set of conditions that make the finding greater than minor, there are inconsistencies on how 

well inspectors document the bases of why findings should be considered greater than minor 

significance in inspection reports. Similarly, IMC 0612 also contains a requirement which states 

that “for every finding that has a cross-cutting aspect, inspectors should document the reasons 

why that cross-cutting aspect is a significant contributor to the finding, using language that 

parallels the descriptions of cross-cutting aspects as discussed in section 06.07.c of IMC 0305 

(see IMC 0612, Appendix F for examples).” However, the staff also has noted inconsistencies 

on how inspectors document cross-cutting aspects of their inspection findings in the inspection 

reports. During CY 2008, The Inspection Program Branch is ensuring that various working 

groups (e.g., IMC 0612 working group; CDBI Component Design Basis Inspections working 

group; HP working group; PI&R working group) are working together to address these 

concerns. The goal is to provide improved guidance and training to inspectors by late 2008. 

The staff expects to have clearer program guidance documents which will result in improved 

discussion of the two aspects of inspection findings in inspection reports. 

As part of the review of the inspection report guidance, the staff is also reviewing the format of 

the inspection report. This effort should improve the overall quality of the report and provide 

more detailed information. 
- 25 - 

7. Does the Significance Determination Process result in an objective and 

understandable regulatory response to performance issues?

Respondent Comments:

Nuclear Energy Institute

The Significant Determination Process (SDP), while generally risk informed, contains significant 

subjective elements. For reactor SDPs supported by SPAR [standardized plant analysis risk] 

modeling, the methodology appears to minimize realistic factors such as operator actions and 

tends to utilize hypothetical worst case assumptions. The incorporation of a bounding, “worstcase” bias in SDP evaluations is contrary to standard PRA practices and is inconsistent with the 

thresholds against which SDP results are judged. As a result, while the steps in an SDP decision 

are well outlined, the bases and rationale for these steps are often neither objective nor 

predictable and can be inconsistent with industry norms for PRA decisions. 

A long recognized weakness in the significance determination process is the level of consistency 

across ROP cornerstones. Specifically, issues and events in the areas of Physical Protection, Fire 

Protection, Emergency Preparedness and Public Radiation Safety and Occupational Radiation 

Safety are evaluated using processes that are more subjective or qualitative in nature and may result 

in exaggeration of actual risk. In some cases, the SDP conclusions appear to be reached based on 

predetermined outcomes rather then allowing risk based measures to drive the characterization. 

Strategic Teaming and Resource Sharing

There are too many SDPs not based on risk or actual effect thresholds. Specifically, the 

Radiation Protection, Security, and Emergency Preparedness SDPs are subjective and 

deterministic. Many of these produce inconsistent results because of the dependence on the 

subjective views of the individuals applying the SDP guidance, especially in the case of the 

Security SDP. The industry and NRC staff should strive to improve these SDPs by including 

more risk-based elements, thus limiting the subjectivity and promoting more consistent 

significance determinations between cornerstone areas. Also, the SPAR models used by the 

NRC in determining risk are of limited value. The STARS alliance encourages the use of 

Licensee’s Regulatory Guide 1.200 compliant PRA models to support the SDP process as they 

become available. The NRC could maintain a degree of independence by using a verification 

process modeled after the PI verification process. 

Union of Concerned Scientists

No. The SDP remains too slow, too vague, and too subjective. It’s the third-worst part of the 

ROP, behind PIs and the alleged enforcement program. 

It is impossible to comment on the SDP for physical protection findings since the Commission 

wrongfully and shortsightedly withheld such information from the American public since August 

2004. 

Foster

Yes; however, one must be an expert on the SDP process, and few individuals are experts, 

either in the NRC or in industry. 

State of Ohio

Yes, it is useful tool to determine a plant’s status in specific oversight areas quickly. 
- 26 - 

Region IV Utility Group

Comment 1: 

There are too many SDPs that are not based on risk or actual effect thresholds. The Radiation 

Protection, Security, and Emergency Preparedness, and other deterministically based SDPs, are 

very subjective (and commonly aggregate multiple non-significant findings into a single 

significant finding). The industry and NRC staff should continue to improve these SDPs by 

including more risk-based elements thus helping to limit the process subjectivity. 

Comment 2: 

The SPAR models used by the NRC in determining risk are outdated. As an alternative, RUG IV 

encourages the use of the licensee Regulatory Guide 1.200 compliant PRA models to support 

the SDP process as they become available. Use of this approach would be analogous to the 

current structure for reporting of Performance Indicators. The data is collected and submitted 

utilizing industry developed guidance which has been endorsed for use by the NRC via a 

Regulatory Issue Summary (RIS). The reliability of the Performance Indicator data submitted by 

licensees is confirmed through PI Verification inspections as part of the baseline inspection 

program. 

Comment 3: 

Also of concern is the subjectivity in the application of human reliability analysis as an input to a 

given significance determination evaluation. The success criteria for personnel actions to 

mitigate an event are not clear. This area needs additional consideration as the industry and 

NRC continue to improve their risk modeling tools, as well as continuing their expanded use. 

Comment 4: 

An additional concern is the aggregation of individual findings to increase the significance of the 

finding. As an example, the Security Physical Protection SDP was applied by scoring the 

individual violations and then totaling the points to total the score for the finding. This conflicts 

with the Enforcement Manual guidance which requires the overall significance of the aggregated 

finding is established by the most significant finding when evaluated separately.

NRC Response: 

As noted in the ROP self-assessment and metric report for SDP-4, "SDP Provides an Objective 

and Understandable Regulatory Response to Performance Issues," external stakeholders’ 

perceptions of the SDP remain negative. Respondents cited lack of SDP yielding consistent 

regulatory response across all ROP cornerstones, complexity and subjectivity issues, and some 

cornerstones not being risk-based. As a result of the stable negative perception of the SDP, the 

metric did not meet its criteria. 

Generally, stakeholder’s perception of the SDP process continued to be mixed, but tended to be 

more negative. Respondents perceived the SDP to be a useful tool to quickly determine a 

plant’s status in specific oversight areas and that it is generally risk-informed; however, it 

remains complex requiring one to be an expert on the SDP process. More than half of the 

industry respondents stated that the Radiation Protection, Security, and Emergency 

Preparedness SDPs are subjective and deterministic, not based on risk or actual effect 

thresholds and may result in exaggeration of actual risk. 
- 27 - 

These comments tend to be recurring year to year, although the staff has taken action to 

address many of these concerns. For example, in early 2008, the staff issued the revised Public 

Radiation Safety SDP as directed by the Commission in the Staff Requirement Memorandum 

(SRM) for SECY-07-0112, “Staff Evaluation and Proposed Revision to the Public Radiation 

Safety Significance Determination Process to Address Radioactive Liquid Spills and Leaks,” 

dated July 6, 2007. The staff worked with internal and external stakeholders and received 

feedback on various aspects of the SDP to improve its effectiveness and efficiency. The scope 

of the review consisted of an evaluation of (1) the current criteria for a white finding to ensure 

consistency with risk-informed goals of the ROP, (2) the entry conditions into the radioactive 

effluent release program branch of the SDP flowchart for spills and leaks, and (3) the SDP to 

ensure that it reflects the NRC Strategic Plan goal of openness. The staff also made two other 

changes to the Public Radiation Safety SDP - removing a yellow characterization from the 

transportation branch of the SDP and a white characterization for the aggregation of findings in 

the radioactive material control branch of the SDP. These changes were necessary because 

the level of the characterization of findings is not in keeping with the risk-informed nature of the 

ROP. 

The staff continuously reviews the various SDPs to make improvements. The staff is currently 

interacting with the industry and other stakeholders to revise the Occupational Radiation Safety 

SDP. The significance criteria in this SDP are based on radiation dose received, the potential 

for receiving dose in excess of the limits, or those program deficiencies that impede the NRC’s 

regulatory processes. Although these are clear and objective criteria, it was recognized that the 

proposed changes are needed to bring the significance outcomes of the SDP more inline with 

the NRC Enforcement policy. In addition, the staff is reviewing all security related SDPs to 

identify if improvements are warranted. 

For reactor SDPs supported by SPAR modeling, the industry representatives believe the 

methodology minimizes realistic factors such as operator actions and tends to utilize hypothetical 

worst case assumptions. While the steps in an SDP decision are well outlined, they feel that the 

bases and rationale for these steps are neither objective nor predictable and can be inconsistent 

with industry norms for probabilistic risk assessment (PRA) decisions. They further believe that 

the NRC should abandon the use of SPAR and use the Licensee’s Regulatory Guide 1.200 

compliant PRA models to support the SDP process as they become available. While industry 

expressed the need for the NRC to maintain a degree of independence, their consensus is that 

this could be accomplished by verifying licensee results in a verification process similar to that 

used for the performance indicator verification. 

During 2007, the staff met with representatives from NEI, industry, and other stakeholders in a 

series of public meetings to discuss the industry proposal to use industry probabilistic risk 

assessment analyses in lieu of NRC risk assessment tools for assessing the significance of 

findings. The NRC reviewed the industry proposal and concluded that the ROP required the 

NRC to maintain independence by evaluating the significance of findings and not just reviewing 

the results of the licensee’s assessment. At present, the industry has not uniformly implemented 

a standardized approach to performing risk analysis that would ensure uniform application 

across the spectrum of industry probabilistic risk assessment models. In this regard, the NRC’s 

use of standardized plant analysis risk (SPAR) models, together with the ongoing development 

of guidance on conducting Phase 3 risk assessments, commonly referred to as the risk 

assessment standardization project (RASP), ensures greater uniformity in the agency’s 
- 28 - 

regulatory assessments. Summaries of the public meetings (ML071490069 and ML070640567) 

and the final NRC response to NEI (ML072490566) can be accessed in NRC ADAMS. 

To more accurately model plant operation and configuration and to identify the significant 

differences between the licensees’ PRA and SPAR logic, cut-set level reviews are currently 

being accomplished. To date, 41 models have detailed cut-set level reviews completed. It was 

implemented in September 2006. Limited scope validation and verification is accomplished by 

comparison to licensee PRA/IPEEE models (as available), or comparisons to NRC NUREGs 

and analyses. Limited scope peer reviews include internal QA review by contractors, by NRC 

Headquarters staff, and by Region Senior Reactor Analysts (as available). 

A stakeholder expressed concern of subjectivity in the application of human reliability analysis as 

an input to a given significance determination evaluation and that the success criteria for 

personnel actions to mitigate an event are not clear. Improved techniques to assess human 

performance are currently being developed. It is the staff intention to provide additional 

consideration as the industry and NRC continue to improve these risk modeling tools as we 

continue to expand their use. 

To support the consistent implementation of PRA techniques in the ROP, the NRC initiated the 

“Risk Assessment of Operating Events” Handbook (hereafter referred to as the RASP 

handbook) to establish procedures and improve the methods of risk assessment in various riskinformed regulatory applications. One specific purpose of this project was to develop guidelines and 

methods that the NRC staff could use to achieve more consistent results when performing risk 

assessments of operational events and licensee performance issues. 

The RASP Handbook was divided into three volumes designed to address internal events 

(Volume 1), external events (Volume 2), and SPAR model reviews (Volume 3). Volumes 1 and 2 

updated staff guidance that was provided for trial use in 2005 and 2006, respectively. Volume 3 

provides analysts and SPAR model developers with additional guidance to ensure that the SPAR 

models used in the risk analysis of operational events represent the as-built, as-operated plant to the 

extent needed to support the analyses. The information in the RASP Handbook has been 

beneficial to the risk analyst staff and is referenced in the SDP program guidance. The staff also 

recently made the RASP Handbook publicly available on the ROP Web page and in ADAMS. 

8. Does the NRC take appropriate actions to address performance issues for those plants 

with identified performance deficiencies? 

Respondent Comments:

Nuclear Energy Institute

Actions taken by the NRC to address performance issues follow the process and are consistent 

and predictable. 

Strategic Teaming and Resource Sharing

NRC action in accordance with the Action Matrix is clear and consistent for single white findings 

and single white performance indicators; but is less clear for more complex issues. Also less 

clear is the assessment and disposition of substantive cross-cutting issues. For Licensees with 

identified substantive cross-cutting issues in Region IV, the rate of findings identified with cross-
- 29 - 

cutting aspects was essentially 100 percent, where as for the other three Regions, the rate was 

closer to 75 percent. There appears to be a degree of inconsistency between regions with the 

assignment of cross-cutting aspects. 

Union of Concerned Scientists

Yes, and no. Yes, when it comes to plants moving towards the right in the Action Matrix and 

therefore dropping away from the licensee response level. Yes, when it comes to responding to 

indications of safety culture problems and cross-cutting issues. No, when it comes to 

enforcement actions involving civil penalties. 

It is impossible to comment on the appropriateness of NRC actions taken for security 

performance deficiencies since the Commission wrongfully and shortsightedly withheld such 

information from the American public since August 2004. 

The NRC’s alleged enforcement program is the second-worst part of the ROP, a close second to 

the totally useless PIs. The absurdity of the NRC’s enforcement program was demonstrated – 

again – earlier this year at the Indian Point nuclear plant. To comply with federal law, the NRC 

ordered – not urged, not requested, but ordered – the Indian Point licensee to install backup 

power supplies for the emergency sirens around the facility by a specified date. 

The NRC relaxed the deadline, yet the licensee still failed to comply with the NRC’s order. The 

NRC determined that the failure was within the licensee’s control; in other words, the agency 

established that the violation involved an avoidable performance failure. NRC’s regulations 

provide for a $130,000 civil penalty to be levied for each day of a violation. NRC’s enforcement 

policy clearly states that daily penalties are reserved for ongoing violations. Yet NRC imposed a 

one-time civil penalty of $130,000 even though the licensee was still in non-compliance at the 

time of the sanction and remains so today. The NRC’s enforcement program needs significant 

repair. It ain’t bent, it’s broken. The NRC’s enforcement program is pitiful. 

State of Ohio

Yes. 

Region IV Utility Group

Yes. 

The NRC action in accordance with the Action Matrix is clear and consistent for single White 

findings, but appears less consistent for more complex issues. There is evidence that process is 

not always followed (or may be deviated from) when circumstances should result in moving a 

licensee to a lower action state. Once a deviation from the process has occurred, it becomes 

unclear how to exit from the overall process. 

For licensees with identified Substantive Cross-Cutting Issues, the rate of findings identified with 

cross-cutting aspects is essentially 100 percent for NRC Region IV licensees, as compared to 

approximately 75 percent for licensees in the remaining three regions. The data suggests a 

need for clarified guidance to provide consistency and predictability.

NRC Response: 

As noted in the ROP self-assessment and metric report for AS-5, "NRC Takes Appropriate 

Actions to Address Performance Issues," the public, State, and industry respondents generally 

agreed that the NRC takes appropriate actions to address plants with identified performance 
- 30 - 

deficiencies. The overall level of external stakeholder satisfaction in this area was generally 

favorable and similar to previous years. There were however, concerns related to enforcement 

and cross-cutting aspects. 

A comment from the industry stated that the Agency’s action with the Action Matrix is consistent 

for single white findings but less consistent when dealing with multiple issues as well as for cross 

cutting issues. The staff believes that the Action Matrix inspection finding and performance 

indicator inputs are clearly and consistently applied by the regions. With respect to the 

assignment of cross-cutting aspects the staff acknowledges some variability as the percentage 

of findings with assigned cross-cutting aspects has ranged between approximately 60 to 80 

percent between the regions. However, for inspection findings issued in Fiscal Year 2007, 

Regions I and IV both assigned cross-cutting aspects to approximately 80 percent of their 

respective findings. The staff has also noted a trend that the percentage of findings assigned a 

cross-cutting aspect by the regional offices is converging. 

The staff also notes that the ROP assessment for substantive cross-cutting issues (SCCIs) 

takes into account not only the number of findings with the same cross-cutting aspect, but also 

takes into consideration whether there is a concern with the licensee’s scope of efforts or 

progress in addressing the cross-cutting theme. Regardless of the number of findings identified, 

or the percentage assigned a cross-cutting aspect, the actions being taken by licensees in 

response to cross-cutting themes are an integral input into the NRC decision-making process to 

 identify an SCCI. 

In 2007 the staff also performed a review of the regional implementation practices for assigning 

cross-cutting aspects and identifying SCCIs. In general, the review found that each region was 

appropriately applying IMC 0305, “Operating Reactor Assessment Program” guidance and 

criteria for assessments. Overall, the review found the regions are properly assigning crosscutting aspects to findings in accordance with IMC 0612. However the staff recognizes that the 

ROP safety culture guidance could be further enhanced. The staff discussed proposed draft 

changes to IMC 0612, “Power Reactor Inspection Reports” with external stakeholders at a public 

meeting on May 14, 2008. The intent of the draft changes in part is to improve the guidance to 

inspectors on whether to assign cross-cutting aspects to findings. 

Regarding a comment from the public that security performance deficiency information is 

withheld and is not available for public input, the staff is seeking ways to increase the openness 

and transparency in the security inspection programs. The staff will hold a series of public 

meetings in 2008 on security openness and explore such topics as making the results of security 

inspections public, reinstating public reporting of PI results, and making NRC security inspection 

procedures and end-of-cycle and mid-cycle security cornerstone assessment letters public. Any 

significant information and useful feedback would be reported to the Commission to request 

approval to implement an option to increase transparency and public availability of security 

inspection program information. 

With respect to the comment on enforcement, the NRC considers many factors when deciding 

whether or not to issue a civil penalty to a licensee and the appropriate amount of the civil 

penalty. As documented in the NRC Enforcement Policy, these factors include; the licensees’ 

enforcement history, whether or not the licensee identified the violation, and what corrective 

actions the licensee has taken. In the case of the Indian Point Emergency Notification System 

(ENS), the NRC issued civil penalties totaling $780,000 for failure to implement Orders issued by 

the NRC. The staff determined that this issue warranted Severity Level III violations. The staff 
- 31 - 

also determined that it was appropriate to exercise discretion to increase the civil penalty from 

the base amount of $65,000 for a Severity Level III violation to emphasize the importance of 

compliance with NRC Orders. The staff considered that Indian Point failed to comply with the 

Orders for an extended duration due, in large part, to circumstances within the licensee’s control. 

The staff also considered that the existing ENS is capable of alerting the general public in the 

vicinity of the Indian Point station if an emergency condition occurs. 

9. Is the information contained in assessment reports relevant, useful and written in plain 

English? 

Respondent Comments:

Nuclear Energy Institute

The information contained in assessment reports is relevant, useful and clearly worded. Recent 

improvements have been noted in Assessment Report documentation of the basis for 

assignment and non-assignment of substantive cross-cutting issues. 

Strategic Teaming and Resource Sharing

The information contained in assessment reports is, for the most part, relevant, useful, and well 

written. Inspection schedules in particular are good to have in advance even if they are not fully 

refined. One area that could be improved deals with cross-cutting aspects that change from the 

time they are initially identified in an inspection report. The current assessment guidance permits 

cross-cutting aspects to be changed if additional insights are available following publication of 

the associated inspection report. Given the regulatory principles that guided the development of 

the ROP (that overall assessments of Licensee performance remain transparent, 

understandable, objective, predictable, risk-informed, and performance-based), any change in 

the assigned aspect should be readily available to the Licensee as well as other stakeholders. 

When an assigned cross-cutting aspect is changed, the issue should be re-exited and the 

inspection report updated before the assessment report is issued. 

Union of Concerned Scientists

Yes, in terms of being in plain English. No, in terms of being either relevant or useful. The NRC is far 

too enamored by assessment report boilerplates to the point that the major differences between the 

assessment report for the best performer and worst performer are the addresses of the plants and 

the names of the NRC individuals signing the letters. There may be an occasional “not” tossed into 

the assessment letters for the poorer performers (e.g., “your performance is not acceptable” vice 

“your performance is acceptable”), but there’s really very little relevant or useful information among 

the plain English. 

Foster

I have not read recent assessment reports (SALP was somewhat useful, at least), but circa 

2001, the wording needed improvement. 

State of Ohio

Yes. 

Region IV Utility Group

Yes. 
- 32 - 

The recent work by the NRC staff to clarify the exit process for a Substantive Cross-Cutting 

Issue was very effective. The documented analysis of cross-cutting aspect inputs to the 

assessment process has improved, primarily as a result of the NRC’s revised guidance to their 

inspectors. Given the regulatory principles that guided the development of the ROP (that overall 

assessments of licensee performance remain transparent, understandable, objective, 

predictable, risk-informed, and performance-based), any change in the assigned cross-cutting 

aspect should be re-exited and the inspection report updated. 

NRC Response: 

As noted in the ROP self-assessment and metric report for AS-6, AAssessment Reports Are 

Relevant, Useful, and Written in Plain Language,@ external stakeholders generally agreed that 

the information contained in assessment reports is relevant, useful, and written in plain English. 

One public interest group stated that the assessment reports contain too much boilerplate 

information preventing substantive insights about performance at individual sites. The overall 

level of external stakeholder satisfaction in this area was generally favorable and similar to 

previous years. 

Regarding a comment from the public that there is too much boilerplate information in its 

assessment report, the staff already addressed a similar comment as noted in the 2006 

Consolidated Response [ML072070140]. The assessment letters were designed to 

communicate the Action Matrix column of performance, any identified substantive cross-cutting 

issues (and more recently, whether two of the three criteria were met), and what if any NRC 

oversight actions were pending. This language becomes more standard if licensee performance 

is governed by the licensee response column and more descriptive for licensees in the 

multiple/repetitive degraded cornerstone column. As noted in SECY-05-0070, program 

guidance IMC 0305, AOperating Reactor Assessment Program,@ directs that the level of detail in 

the assessment letters increases as the plant performance decreases. For example, IMC 0305 

requires additional detail for those plants outside of the licensee response column of the Action 

Matrix and/or those plants that have substantive cross-cutting issues. SECY-05-0070 further 

noted that, “A review by the staff of the most recent mid-cycle and end-of-cycle assessment 

letters confirmed that the scope and depth of the letters were significantly more for plants with 

performance issues (i.e., outside the Licensee Response Column of the Action Matrix) than for 

those plants which had no performance indicators and inspection findings that were greater than 

green. 

An industry group made a comment that the current assessment guidance permits the cross 

cutting aspect to be changed if additional insights are available following publication of the 

associated inspection report. When new information is discovered, the NRC can change the 

characterization of the inspection finding, a re-exit is subsequently conducted. A similar 

comment was addressed in the staff’s response to Question 9 of the 2006 Consolidated 

Response [ML072070140]. Referring to IMC 0305, it stated that, "In order to support the 

evaluation of findings with cross-cutting aspects, the inspectors should provide sufficient detail in 

the PIM and provide periodic updates as new information becomes available in accordance with 

IMC 0306 and IMC 0612." The intent is to keep the process transparent so that changes in the 

cross-cutting aspect characterization are readily available to the licensee and other 

stakeholders. 
- 33 - 

10. Are the ROP oversight activities predictable (i.e., controlled by the process) and 

reasonably objective (i.e., based on supported facts, rather than relying on 

subjective judgment)? 

Respondent Comments:

Nuclear Energy Institute

ROP oversight activities are generally predictable and reasonably objective. However, the 

implementation of the SDP for Initiating Event and Mitigating System cornerstones by NRC staff 

appears to involve use of prescribed guidance instead of relying upon “risk-informed” methods. 

This guidance, particularly in the areas of common cause and human factors, prescribes use of 

conservative values and appears to constrain the analyst’s capability to incorporate the specifics 

and circumstances of the event being analyzed. The methods, which have not been made 

publicly available, have been noted to require the assumption that common cause factors 

contribute to the event being analyzed. This can readily lead to an order of magnitude impact on 

assessment results. 

The reduction of subjectivity in methods must be achieved in order to ensure that the objective of 

obtaining a risk-informed assessment based upon supported facts is achieved. 

There remains a high degree of subjectivity in guidance addressing safety culture, most notably 

in guidance for determining whether or not a cross cutting aspect is ascribed to a finding. 

In addition, the predictability and objectivity of the results of the ROP process are reduced in some 

cornerstones (i.e., EP, Radiation Protection, and Physical Protection) that are not closely tied to risk 

analysis. 

Strategic Teaming and Resource Sharing

The STARS alliance has some concerns that applicable ROP guidance is not being followed as 

written or intended particularly in the areas of double counting and determination that a finding is 

greater than minor. 

Relative to double counting, the STARS alliance has the following concern: 

On page 8 of the NRC consolidated response to the 2006 Reactor Oversight Process (ROP) 

external survey (ADAMS Accession Number ML072070140) the response states: 

“Additionally, with the development and implementation of the relatively new MSPI, given that 

this is the first set of PIs that are risk-informed, there will be instances where MSPI inputs and 

inspection findings on the same system will both be counted in the Action Matrix, because the 

two processes are fundamentally different in concept, thus have different meanings and each 

should stand on their own merit.” 

The STARS alliance views this last sentence to be inconsistent with the stated basic tenet of the 

ROP to not “double-count” events/findings in the Action Matrix. Contrary to the NRC staff 

statement made above, MSPI results are risk-informed and share more in common with riskinformed Significant Determination Process (SDP) results than any other performance indicator. 

Relative to the determination that a finding is greater than minor, the STARS alliance has the 

following concern: 
- 34 - 

Manual Chapter 0612, Appendix E contains examples of findings that should be characterized 

as minor. Some potential findings bypass the examples in Appendix E and default to the 

rationale of “affecting a cornerstone objective” as the reason for the issue being greater than 

minor. The decision point in the examples in Appendix E appears to be focused on 

consequences – if there were no consequences, the potential finding is generally characterized 

as minor. When the screening defaults to the question of affecting the cornerstone objective, the 

focus on consequences is eliminated. As such, some minor issues are being classified as 

greater than minor. 

As an example, Licensees routinely establish dose rate limits in addition to integrated dose limits 

for a given work activity. Both types of limits are typically incorporated into radiation work permit 

requirements and associated with electronic dosimeter alarm setpoints. However, except in 

extreme cases, dose rates are not clearly tied (if at all) to quantifiable risk to the worker’s health 

and safety – provided that the integrated dose limits established for the worker are not 

exceeded. Dose rate limitations are generally used to increase the sensitivity of the worker and 

RP staff to worker practices as well as potential changes in area dose rate conditions. As such, 

dose rate limits established by the Licensee are essentially administrative limits more 

conservative than any regulatory or industry standards. Accordingly, the intent of the 

administrative limit examples in Appendix E should apply to deficiencies associated with dose 

rate limitations. However, the NRC has established the interpretation that an administrative limit 

is a self-imposed limit that has a corresponding regulatory limit and since there is no regulatory 

requirement for limiting dose rates during work activities, there cannot be an associated 

“administrative limit.” Based on this position, the NRC has not applied Appendix E examples in 

the screening process for performance deficiencies associated with dose rate alarms. Clearly, 

there is a level of performance deficiency related to dose rate limits that can be considered 

minor - where the intended integrated dose is not exceeded or there is no substantial potential to 

challenge the worker's health and safety. 

Manual Chapter 0612 Appendix E should be a living document. More examples are needed to 

provide better guidance on characterizing minor issues. 

Union of Concerned Scientists

If ‘yes’ equals 10 and ‘no’ equal 1, I’d say 4 to 41⁄2. In the past year, I attended more than one of 

the monthly ROP meetings between the nuclear industry and the NRC. I heard Jim Anderson 

report the results of secret meetings held at NRC on ROP decisions, such as FAQs for PIs and 

on SDPs. Since the NRC doesn’t bother to provide a public, written account for these decisions, 

it’s impossible to determine the reasons why Mr. Dyer keeps downgrading identified problems. I 

can see the problems and the outcomes, but the NRC fails to document how it weighed factors 

in obtaining those outcomes. Absent that public documentation, I assume that the decisions are 

arbitrary and capricious. 

Foster

Yes, and that may be the biggest problem with the process; some subjectivity is beneficial. The 

activities are predictable to a large extent, but only by those who are very familiar with the 

program and the SDP process (not the public). 

State of Ohio

Yes, the inspection procedures we have reviewed provide consistency and objectivity. 
- 35 - 

Region IV Utility Group. 

Comment 1: 

In terms of predictability, RUG IV has concerns with whether the applicable guidance is being 

followed. A number of performance deficiencies default to "affects the cornerstone objective" or 

“could become more significant if left uncorrected” as the reason for the issue being greater than 

minor. Current guidance to inspectors for whether a potential inspection finding is greater than 

minor is contained in MC 0612 Appendix B, Issue Screening and Appendix E, Examples of 

Minor Issues. Much of the current criteria are subjective in nature. MC 0612 Appendices B & E 

should be considered living documents. Insufficient examples are currently available to establish 

a consistent philosophy. Use of a process similar to the PI FAQ process could be used to 

develop additional examples and provide enhanced guidance for answering screening 

questions. 

Comment 2: 

In the NRC consolidated response to the 2006 Reactor Oversight Process (ROP) external 

survey (ADAMS Accession Number ML072070140), the NRC staff response on page 8 stated: 

“Additionally, with the development and implementation of the relatively new MSPI, given that 

this is the first set of PIs that are risk-informed, there will be instances where MSPI inputs and 

inspection findings on the same system will both be counted in the Action Matrix, because the 

two processes are fundamentally different in concept, thus have different meanings and each 

should stand on their own merit.” 

NEI addressed this response in a letter to the NRC dated September 24, 2007, and noted the 

following; 

“We view this last sentence to be inconsistent with the stated basic tenet of the ROP to not 

“double-count” events/findings in the Action Matrix. Contrary to the NRC staff statement made 

above, MSPI results are risk-informed and share more in common with risk-informed Significant 

Determination Process (SDP) results than any other performance indicator. 

The above statement on “double-counting” of MSPI/SDP results is particularly troubling. We are 

aware that NRC is currently performing a review of events at a plant where a failure was 

common to both the white performance indicator result and a white SDP finding. Review is 

underway to determine whether or not the circumstances associated with this issue constitute 

“double-counting” of a performance indicator and an inspection finding. Current ROP guidance 

states that issues with the same underlying causes should not be “double-counted” in the 

assessment program. The failure cited above caused the MSPI indicator to go from green to 

white. This same failure was evaluated under the SDP to be white. We see no basis for a 

conclusion that supports “double-counting” in this circumstance. As such, we would view a 

decision to count the white MSPI result and the white SDP result in the Action Matrix as a 

deviation, subject to the requirements contained in IMC 0305, Section 06.06.f. 

Regarding the statement cited above that indicates that MSPI results and SDP results “have 

different meanings and each should stand on their own merit,” we request a response that 

clearly identifies the basis for the statement or, preferably, a retraction.” 

RUG IV agrees that MSPI results are risk-informed and are subject to the ROP guidance for 

double-counting in the assessment process. Therefore, RUG IV fully supports the position set 

forth in the NEI letter. 
- 36 - 

NRC Response:

As noted in the ROP self-assessment and metric report for O-1, AStakeholders Perceive the 

ROP to Be Predictable and Objective,@ the State respondent noted that ROP oversight activities 

did provide consistency and objectivity. Public responses were contrary. One public respondent 

stated that the objectivity of the ROP was one of the biggest problems and that subjectivity is 

more beneficial. However, another public response stated that due to the lack of public 

documentation of the basis for decisions, they perceived the ROP activities and outcomes to be 

arbitrary. The responses from industry were generally in favor of the predictability and objectivity 

aspects in the ROP. Comments were similar to previous years, and as a result, this metric did 

meet its criteria. 

An industry group commented that the use of prescribed guidance, particularly in the area of 

common-cause failure (CCF) and human reliability analysis (HRA), affects the predictability and 

reasonable outcome. These two areas have inherent modeling and analysis limitations that need 

to be clearly understood. The NRC developed the SPAR HRA (SPAR-H) Method to provide a 

traceable, easy-to-use HRA method. The method uses defined performance shaping factors 

(PSFs) and human error probabilities (HEP) to estimate the outcome on a straightforward 

worksheet. The method is explained in NUREG/CR-6883, SPAR-H Reliability Analysis Method. 

Likewise, the staff has issued NUREG/CR-6268, Rev. 1, Common-Cause Failure Database and 

Analysis System which presents an overview of CCF analysis methods for use in NPPs. 

Improvements to assessing CCF are being incorporated into the Saphire code used to perform 

SDP Phase 3 assessments and a new section to the RASP Handbook will be issued shortly. 

Regarding the comment that there is a high degree of subjectivity in guidance addressing safety 

culture, the staff has performed a review of the regional implementation practices for assigning 

cross-cutting aspects. Overall, the review found the regions are properly assigning cross-cutting 

aspects to findings in accordance with IMC 0612. The staff also discussed proposed draft 

changes to IMC 0612, “Power Reactor Inspection Reports” with external stakeholders at a public 

meeting on May 14, 2008. The intent of the draft changes in part is to improve the guidance to 

inspectors on whether to assign cross-cutting aspects to findings. For example, the staff is 

proposing additional guidance to assist inspectors in making a determination whether the crosscutting aspect is reflective of current performance (if it is not reflective of current performance, a 

cross-cutting aspect is not assigned to the finding). 

A comment from an industry group discussed a concern about determining findings as more 

than minor as described in IMC 0612 Appendix E. Manual Chapter 0612 Appendix E is used in 

assisting the inspector to determine if a finding is of minor significance. The inspector is to 

compare the finding to the Appendix E examples to determine it is similar to a minor example. If 

the finding is similar to the samples listed as being minor, then the finding should not be 

documented in the inspection report; however, if the finding is similar to the samples as being 

greater than minor, then the inspector will describe the set of conditions that make the finding 

greater than minor (e.g., the associated cornerstone attribute and how the objective was 

affected). Since Appendix E is limited in examples, if no example is found, then the inspector 

will answer the minor questions found in Appendix B, Section 3. Appendix E is a living 

document which is updated frequently. The NRC is working with its external stakeholder to 

revise Manual Chapter 0612 and Appendix B to provide better guidance on characterizing minor 

issues. 
- 37 - 

Regarding the comment that some decisions made during ROP FAQ meetings are not made 

available to the public, the staff recognizes this concern and will continue to consider 

enhancements to improve public involvement in the area. FAQ appeal meetings are also public 

meetings. Decisions made as a result of the FAQ appeal process are rendered by the DIRS 

(Division of Inspection and Regional Support) Division Director after the meeting. These 

decisions are made public and are documented in the following monthly ROP public meeting. 

The comment on ‘double counting” was addressed in response to Question 2 of this document. 

The staff has recognized that double counting issues may exist between MSPI, other PIs, and 

inspection findings. This has been resolved and revised guidance will be issued with the next 

revision of IMC 0305 that will outline specific steps required to determine if these double 

counting issues exist and what actions may need to be taken. Licensees may be required to 

perform calculations that extract the common event (i.e., failure, scram) from the PIs to 

determine if the new PI value would remain greater-than-green. If the new value is not greater 

than green, then the PI is not an input into the Action Matrix. If the PI remains greater than 

green, then it is a valid input into the Action Matrix and both the PI and finding are included and 

this is not considered double counting. 

The comment on the lack of objectivity and predictability in the Emergency Preparedness (EP) 

and Radiation Protection (RP) cornerstones was addressed in Question 7 of this document. 

11. Is the ROP risk-informed, in that the NRC=s actions are graduated on the basis of 

increased significance? 

Respondent Comments:

Nuclear Energy Institute

The Initiating Events, Mitigating Systems and Barrier Integrity cornerstones are risk-informed. For 

other cornerstones, while their thresholds are graduated on the basis of increased significance, the 

risk basis for the thresholds is not, in all cases, clear. This results in more subjective and less 

predictable outcomes in these cornerstones. 

Strategic Teaming and Resource Sharing

The NRC’s Action Matrix, if properly implemented, provides appropriate graduation on the basis 

of increased risk as the basis for NRC action to be taken. However, the STARS alliance believes 

that NRC management should continuously monitor the implementation of this process. The 

implementation of the process has at times resulted in a perception by some licensees that the 

action taken may not have been merited based on their understanding of the issue and the 

application of the Action Matrix to it. 

Entergy

The Occupational Radiation Safety SDP does not provide sufficient mechanisms to distinguish 

between “minor” violations and non-cited violations. Therefore, issues that should be 

characterized as minor become non-cited violations. As a result, issues of no or minor 

significance are inappropriately raised in importance. 
- 38 - 

Using the flowchart in MC 0609, Appendix C, a Green non-cited violation can still be issued 

when all of the following questions are answered “NO”: 

1) ALARA [As Low As Reasonably Achievable] Planning or Work Controls? 

2) Was it an overexposure? 

3) Was there a substantial potential? 

4) Was the ability to assess dose compromised? 

In addition, the minor examples in Appendix E of MC 0609 only contain two examples in the 

ALARA Planning or Work Controls section for review by the inspectors. Additional guidance, 

similar to guidance contained in the other cornerstones, should be developed to allow inspectors 

to determine the true “risk” associated with the finding. 

This comment also applies to the Public Radiation Safety Cornerstone. 

Union of Concerned Scientists

As bizarre as it sounds, the ROP is actually overly risk-informed. “Risk-informed” has been 

redefined by the nuclear industry to something akin to “green” if the underlying problem was 

more than 30 seconds away from reactor meltdown and “dark green” otherwise. The nuclear 

industry’s cognitive dissonance is exposed. Industry representatives repeatedly proclaim that the 

number of OHSA [OSHA] reportable events provides telling insights about performance. OSHA 

reportable events have little or no direct nexus to reactor core damage frequency. But when 

NRC inspectors identify a totally deficient access control program or electrical engineering 

calculation control program or instrument setpoint calibration process, the nuclear industry 

rushes to point out that the reactor core damage frequency of the finding is less than one times 

ten to the minus five and magically transforms unacceptable to acceptable. And the NRC plays 

along. 

The NRC cannot continue to downplay performance deficiencies simply because they didn’t carry us 

to the brink of nuclear disaster. The original ROP was far better in this regard. It focused NRC 

attention to areas of higher risk significance and had NRC actions graduated based on the depths of 

the identified performance shortfalls. Over years, the nuclear industry has compelled changes such 

that the current ROP risk-informs where NRC looks and how NRC grades its findings. The result 

nearly collapses the ROP gradations into two bins: meltdown bad, no meltdown good. An ROP isn’t 

needed for that coarse binning. The NRC really needs to turn back the clock and recapture the ROP 

value lost over the years. 

Foster

Yes. This is again perhaps a program weakness. Risk-informed means you are using 

probability as your guide, and you risk a misjudgment (did I mention Davis-Besse?). 

State of Ohio

Yes, in the inspections we have observed this has been the case. 

Region IV Utility Group

Yes. 

See also responses to Questions 7 and 10 as applicable to this item.
- 39 - 

NRC Response:

As noted in the ROP self-assessment and metric report for O-2, AStakeholders Perceive the 

ROP To Be Risk-informed,@ overall, respondents believe the ROP is generally risk informed. 

However, one public response, thought this was a program weakness, stating it was overly riskinformed. Industry responses were favorable but suggested areas for improvement as 

discussed below. 

A comment from the industry group stated that some cornerstones, such as Initiating 

Events, Mitigating Systems and Barrier Integrity, are risk-informed, but other cornerstones, 

the risk basis for the thresholds is not clear. The staff addressed a similar question as 

noted in its response to Question 12 of the 2006 Consolidated Response [ML072070140]. 

The staff stated that other cornerstones, such as EP and RP SDPs are deterministic, and 

that the industry and NRC staff should strive to improve the deterministic SDPs by including 

more risk-based elements (thus helping to limit the subjectivity in the process). In 

establishing SDPs in these areas, it is difficult to relate deficient performance in these areas 

to quantitative risk measures such as core damage frequency or large early release 

frequency. The staff has worked to define SDPs in the EP and RP areas that result in the 

agency response that is considered appropriate for a range of performance problems. 

Because of the inherent differences between the various cornerstones, the staff does not 

presently envision being able to define a single risk-informed SDP approach that would 

address all cornerstones. As also noted in the staff’s response to Question 7 of this 

document, the staff has made changes in this area to improve the risk-informed aspect. 

In response to a licensee’s comment on Occupational Radiation Safety SDP, the issue of 

minor findings vs. more than minor findings is currently undergoing an NRC re-evaluation. 

Two public meetings have been held to discuss the issue. 

On December 5, 2007, a public meeting was held between the NRC, NEI and the 

nuclear power industry at the NRC Headquarters in Rockville, MD. The purpose of 

this meeting was to discuss improvements in the Revised Oversight Process related 

to consistency of Health Physics inspection findings. 

On May 15, 2008, a public meeting was held between the NRC, NEI and the nuclear 

power industry. The purpose of the meeting was to offer NEI and industry 

representatives an opportunity to provide input to NRC staff regarding the ROP’s 

Health Physics Programs. The following documents were provided for discussion 

during the meeting: (1) Public Meeting Agenda (ADAMS No. ML081200115), (2) 

IMC 0612, Power Reactor Inspection Reports (ADAMS No. ML070720191), (3) IMC 

0612, Appendix B, Issue Screening (ADAMS No. ML071720417), (4) IMC 0612, 

Appendix E, Examples of Minor Issues (ML070720202), and (5) NEI Examples to 

Support Discussion of the IMC 0612 Process for Determining Whether An Issue is 

Minor or More than Minor (ML081510501). A revision of IMC 0612, Appendix B and 

Appendix E of IMC 0612 has been planned to make any needed improvements. 

Regarding the comment that the NRC downplays performance deficiencies simply because 

they didn’t carry us to the brink of nuclear disaster, the staff believes that it takes 

appropriate graded actions based on the severity of the issue or event. This philosophy is 

defined in IMC 0308. 
- 40 - 

12. Is the ROP understandable and are the processes, procedures and products clear 

and written in plain English? 

Respondent Comments: 

Nuclear Energy Institute

In general the ROP is understandable and the processes, procedures and products are clear. 

However, there is some disagreement about being written in “plain English.” It was noted that 

some documents can be difficult to follow without the appropriate technical background. In 

addition, areas that lack objective guidance, as discussed in comments provided for question 10, 

call into question that adequacy of the “plain English”. 

Strategic Teaming and Resource Sharing

The ROP procedures and products are generally clear and understandable. The ROP process 

can be complex and does require significant Licensee resources to maintain a working level 

understanding. An example of complexity is the section of Manual Chapter 0305 that discusses 

repetitive degraded cornerstone status. (Refer to page 18 section b.4, “Multiple/Repetitive 

Degraded Cornerstone Column”): 

“Assessment inputs result in a repetitive degraded cornerstone (2 white or 1 yellow input for five 

or more consecutive quarters), multiple degraded cornerstones, multiple yellow inputs or a red 

input. Regarding repetitive degraded cornerstone, if the only greater than green findings in the 

fifth quarter have been held open greater than four quarters, the repetitive degraded cornerstone 

does not apply. If, however, one of the greater than green findings is still within the original four 

quarters and one or more findings has been held open greater than four quarters, the repetitive 

degraded cornerstone does apply. In this instance, the plant would stay in the Multiple/Repetitive 

Degraded Cornerstone column until there was only one greater than green finding, regardless of 

the length of time the findings have been opened.” 

This discussion is complex. While earlier sections of Manual Chapter 0305 define inputs to the 

Action Matrix as findings and PIs the discussion in this paragraph after the first sentence limits 

the inputs to findings. 

The Emergency Preparedness SDP is not clear and lends itself to multiple interpretations, which 

makes its use unpredictable. The STARS alliance suggests reworking the SDP to assure uniform 

interpretation between the NRC and the Licensees is achieved. 

Union of Concerned Scientists

The fact that the Commission adopted the Mitigating Systems Performance Indicator (MSPI) 

strongly suggests that the agency no longer cares about this aspect. MSPI is an abomination 

that never should have been foisted on the American public and should be withdrawn as quickly 

as possible. 

Foster

Yes. 

State of Ohio

Yes. 
- 41 - 

Region IV Utility Group

Comment 1: 

Yes. 

The ROP products are generally clear and understandable. A process for addressing potential 

generic inspection issues would be helpful instead of identifying the issues plant by plant. Please 

see our discussion in response to Item 5. 

Comment 2: 

Regulatory Issue Summary, RIS 2007-21 “Adherence to Licensed Power Limits”, was issued 

August 23, 2007 to reinforce adherence to the maximum power level specified in individual plant 

licenses. The RIS also retracted long-standing enforcement guidance that has been used by 

both inspectors and licensees to determine whether normal and expected fluctuations in power 

meet plant license requirements. It should be noted this guidance has been retained in the ROP 

inspection program. 

RUG IV fully supports the primary message of the RIS. NRC licensees should not intentionally 

operate above 100% steady state rated thermal power (RTP), and they should take corrective 

action to reduce thermal power whenever they find it above the operational limit specified in the 

plant-specific operating license. However, some degree of fluctuation in thermal power is a 

normal part of plant operation, and is neither a license violation nor outside the design basis. By 

retraction of the long standing guidance regarding normal and expected fluctuations in power, an 

unintended consequence was created by not replacing the best available guidance, leaving no 

practical definition of steady-state operation at RTP. 

RUG IV fully endorses the NEI letter to the NRC dated September 24, 2007 which addresses 

this concern. RUG IV agrees unintended consequences such as this could be substantially 

reduced if the generic communication were made available for a public comment period prior to 

being issued. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-3, AStakeholders Perceive the 

ROP to Be Understandable,@ most of the stakeholders stated that the ROP is understandable 

and that products are written in clear and plain English. However, some stakeholders noted 

concerns. 

In general, industry, State and public respondents stated that the ROP is understandable 

and that products are written in clear and plain English. One response from the public 

disagreed, citing the MSPI as a specific example. Licensees expressed difficulty in 

following some of the complex documents without the appropriate technical background. 

Similar to previous years, the SDP is recognized as the most complex portion of the ROP 

and the NRC staff is continuously looking to make this aspect of the program more public 

friendly. 

Regarding the comment that Emergency Preparedness SDP is not clear and lends itself to 

multiple interpretations, the staff agrees on the comment which suggests reworking the 

Emergency Preparedness SDP to assure uniform interpretation between the NRC and the 

Licensees is achieved. This comment will be taken under consideration. The Appendices 

to IMC 0609, SDP are reviewed and updated on periodic bases. 
- 42 - 

Regarding the comment from an industry group that a process for addressing potential 

generic inspection issues would be helpful instead of identifying the issues plant by plant, 

the staff addressed this issue in its response to Question 5. 

Regarding the comment concerning RIS 2007-21, “Adherence to Licensed Power Limits”, the RIS 

was issued to remind licensees that they must adhere to the maximum power limit specified in their 

license and that in no way did the NRC authorize or condone exceeding that limit. The RIS clearly 

stated that the longstanding enforcement guidance (1980 Jordan Memo) was provided for NRC 

Inspectors and not intended for operational guidance. The RIS also states that the NRC 

recognizes that slight changes in thermal power may occur due to expected variances in plant 

parameters. With that stated, the NRC has been working with NEI to develop industry guidance 

that is consistent with the RIS guidance and ensures that licensees meet their license condition for 

maximum thermal power level. Additionally, Inspector guidance will be enhanced to ensure 

consistent treatment for conditions where the license condition is exceeded. This process is still in 

progress at this time. 

With regards to issuing the RIS without a public comment period, this was addressed in the RIS, 

which states “A notice of opportunity for public comment on this RIS was not published in the 

Federal Register because this RIS is informational, and does not represent a departure from 

current regulatory requirements.” 

Regarding a comment about the complexity of MSPI, the staff addressed a similar question as 

noted in its response to Question 12 of the 2006 Consolidated Response [ML072070140]. The 

staff agrees that MSPI is a complex algorithm, is difficult to understand, and its underlying data 

input is not made public due to the security nature of its reliance on licensee PRA information. 

The staff regrets that much of the MSPI information has been withheld, and the output cannot be 

traced based on the limited information on the website, but this is offset by the fact that MSPI is 

a true risk-informed PI and is the first PI to accurately indicate an integrated performance 

summed from unavailability and reliability. 

The staff is currently reviewing the wording and definition for the repetitive, degraded 

cornerstone column. It is unclear at this time what courses of action the NRC will take on this 

issue. 

The ROP was generally not designed to predict and/or prevent all failures, but was more 

realistically designed to detect declining performance and focus NRC and licensee attention 

and resources on the most significant issues. This design recognizes the defense-in-depth 

approach to NRC regulation. The staff believes that the ROP has been successful in 

assuring that plants are being operated and maintained safely. 

13. Does the ROP provide adequate regulatory assurance when combined with other 

NRC regulatory processes that plants are being operated and maintained safely? 

Respondent Comments:

Nuclear Energy Institute

The ROP, in combination with other NRC regulatory processes, does provide adequate regulatory 

assurance that plants are being operated and maintained safely. 
- 43 - 

Strategic Teaming and Resource Sharing

The STARS alliance strongly agrees that the ROP provides adequate assurance, when 

combined with other regulatory processes, that plants are being operated and maintained safely. 

Union of Concerned Scientists

Not even close. Until the NRC consistently enforces its regulations, nuclear plants will be 

operated at higher risk than necessary. The NRC can post on its ROP webpage a zillion green 

PIs and a bazillion green inspection findings for the Shearon Harris nuclear plant and that won’t 

change the fact this plant doesn’t not comply with the NRC’s fire protection regulations. 

Likewise, all the greens in the world won’t change the fact that pressurized water reactors have 

operated for decades with containment sump screens that would likely clog with debris in event 

of an accident (e.g., the GSI-191 issue). 

In order to provide adequate assurance, the NRC would have to consistently enforce its safety 

regulations, quickly resolve generic safety issues with appropriate compensatory measures in the 

interim, administer a non-greenwashed ROP, and acquire (perhaps through eBay) an enforcement 

program. 

Foster

No (did I mention Davis-Besse?). 

State of Ohio

Yes, the ROP provides a substantial framework for ensuring that safety remains a primary focus for 

the licensees. 

Region IV Utility Group

Yes. 

NRC Response: 

As noted in the ROP self-assessment and metric report for O-4, AStakeholders Perceive That the 

ROP Provides Adequate Regulatory Assurance That Plants Are Operated and Maintained 

Safely,@ industry and State stakeholders believe the ROP combined with other regulatory 

processes assures that the plants are operated and maintained safely. The State response was 

more positive than in previous years. The two public respondents did not believe the ROP 

(combined with other regulatory processes) assured safe plant operations. There has been an 

increasingly positive industry and State perception of the ROP maintaining safety. Public 

response was similar to previous years. 

As previously noted in last year=s consolidated response, the staff believes that the ROP is 

successful in assuring that plants are being operated and maintained safely. A basic tenant 

of the ROP is the assumption that the nuclear industry is mature and operating safely, and 

that old design issues are mostly identified and dispositioned. Although the ROP is not 

designed to predict and/or prevent all failures, it does very effectively detect declining 

performance and focuses NRC and licensee attention on the most significant performance 

issues. The staff recognizes that it assumes plants are built as designed, and PRAs reflect 

this assumption. But, the staff also believes that design flaws and most generic issues only 

minimally impact the ROP’s ability to provide effective oversight. This is primarily due to the 
- 44 - 

understanding that the defense-in-depth philosophy the Commission has maintained 

mitigates most of the risk from any shortcomings in the design and operation of the plant. 

As to the Shearon Harris fire comment, the licensee is a pilot plant in the NFPA-805 riskinformed fire standard per 10CFR50.48(c). This new standard allows a plant to use risk to 

access plant conditions with respect to the fire regulations. Because the existing fire regulations 

were not developed until after many of the nuclear power plants had been built, the Agency 

allows enforcement discretion for those regulations not in full compliance where compensatory 

measures have been put in place. These compensatory measures have been inspected by 

certified inspectors and determined that they meet the minimum safety standard. The Shearon 

Harris plant has been under the scrutiny of both regional and headquarters staff during this 

transition period. 

Regarding the GSI-191 issue, the NRC believes that appropriate actions have been and are 

being taken to ensure that containment sump screens would function acceptably in the unlikely 

event of a loss-of-coolant accident. Licensees have taken many actions to enhance plant 

safety with regard to this issue. These actions have included installation of much larger sump 

screens at all plants, and removal from many plants of insulation materials that could lead to 

clogging. 

The staff agrees that Generic Safety Issue 191, PWR Sump Performance, has taken a long time 

to fully resolve. This issue has been particularly challenging because of the complex and 

difficult-to-predict processes involved. As the staff has delved deeper into the issue, it raised 

questions that have necessitated additional evaluation (e.g., chemical effects). The challenge 

for licensees is to demonstrate with appropriate and conservative tests and evaluations that the 

sump screen clogging issue has been fully addressed. The staff is currently performing 

comprehensive reviews of February 2008 licensee submittals intended to make this 

demonstration. The staff expects the reviews to conclude and the issue to be resolved in the 

first half of 2009. 

14. Is the ROP effective, efficient, realistic, and timely? 

Respondent Comments: 

Nuclear Energy Institute

The ROP is generally timely and measures have been established by NRC to identify areas for 

continued improvement. Performance measures have not been established for effectiveness, 

efficiency and realism. Changes to the significance determination processes are needed to 

improve efficiency and reduce the level of subjectivity. Early and increased interaction with the 

licensees on SDP evaluations can assist the process in these two areas. 

Current ROP appeal practices could be improved by assuring an appropriate level of 

independence at the various stages of the appeal process and through greater licensee 

involvement. 

Efficiencies can also be achieved by ensuring direct communication with the licensee whenever 

possible. The current process appears to limit direct communication between the licensee and 
- 45 - 

NRR PRA personnel; requiring indirect communication via regional personnel. This introduces 

inefficiencies and delays and increases the potential for miscommunication. 

Strategic Teaming and Resource Sharing

For the most part, the ROP is effective, efficient, realistic, and timely. However, some areas 

could be improved. The current Component Design Basis Inspections (CDBI) are consuming 

substantial Licensee resources. There appears to be a significant opportunity to improve the 

efficiency of this process by applying more discipline to maintaining the schedule. The number 

and significance of the findings to date do not support the level of resources the inspection 

requires. The STARS alliance is encouraged by recent ROP realignment efforts and concurs 

with the recommendation to extend the frequency of these inspections to three years, from the 

current two year frequency. The STARS alliance remains concerned the scope of these 

inspections is not commensurate with Licensee performance and is overly burdensome. 

Additional efficiencies could be realized by forming permanent inspection teams to conduct the 

inspections. 

Occasionally, inspection exit meetings are significantly delayed in time from close of inspection 

activities onsite, resulting in additional inefficiencies in the process. 

Consideration should also be given to extending the assessment frequency for Licensees with 

good performance to annual and retain the six month assessment period for Licensees with 

substantive cross-cutting issues or who are in a degraded cornerstone. In some cases, the midcycle assessments consume resources unnecessarily for both the NRC and Licensees. 

The ROP is not realistic when it comes to incorporating plants that were mothballed years ago and 

are now being brought on line (or new plants) into the ROP. These plants will experience a 

“shakedown” period when they are first brought on line. The current ROP thresholds were based on a 

mature fleet of plants with several years of operating experience. A realistic process needs to be 

developed for bringing these plants into the ROP. 

Union of Concerned Scientists

Nope, nope, nope, and nope. 

Foster

Perhaps, yes, likely, and no. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Comment 1: 

Overall we are in general agreement. However, the current cycle of CDBI inspections have 

committed substantial NRC and licensee resources. CDBI inspections are identifying a 

disproportionately low number of findings for this effort. Additionally, only one “greater than 

Green” finding has been identified. A significant opportunity exists to improve the efficiency of 

this process by applying more discipline to schedule adherence of the inspection. Additionally, 

NRC inspector exit meetings are occasionally significantly delayed in time from the close of 

inspection activities onsite, resulting in additional process inefficiencies. 

We are encouraged by current discussions to extend the frequency of these inspections to three 

years, from the current two year frequency. However, we remain concerned the scope of these 
- 46 - 

inspections is not commensurate with licensee performance and is overly burdensome. 

Licensees are not typically making significant design changes to plants that would warrant 

inspection activities this extensive. Additional efficiencies could also be realized by forming 

permanent inspection teams to conduct the inspections. 

Comment 2: 

The current occupational radiation safety inspections are also an area where the level of 

inspection activity reduction may be appropriate. Industry performance in the area of personnel 

radiation collective dose continues to improve. The current inspections are scheduled 2 or 3 

times per year, and could be more effective if scheduled to align with the licensees operating 

cycle which is typically 18 to 24 months. MC 0308, App. C, notes, “Reactor licensees currently 

have mature ALARA programs..”. Consideration should be given to extending the frequency and 

reducing the inspection hours allocated for the Occupational Radiation Safety cornerstone 

consistent with current licensee performance. 

Comment 3: 

We are also encouraged by proposed changes under consideration for the SDP process. The 

proposed changes would permit in progress licensee efforts, to be considered in the final 

significance determination decision. This would make the best available information available 

when dealing with complex issues that can occur when potential “greater than green” findings 

are identified and ensure the most appropriate significance is established for the finding. 

Comment 4: 

The current ROP assessment process requires an evaluation of licensee performance every six 

months. In some cases, the mid-cycle assessments consume resources unnecessarily both for 

the NRC and Licensees. For plants with good performance; i.e. Licensee or Regulatory 

response Column of the NRC Action Matrix, it may be appropriate to formally assess licensee 

performance annually. Licensees with performance in the Degraded Cornerstone column of the 

Action Matrix or an identified Substantive Cross-Cutting Issue should continue to receive an 

assessment every six months. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-5, AStakeholders Perceive the 

ROP to Be Effective, Efficient, Realistic, and Timely,@ external stakeholders generally believe 

that the ROP is effective, efficient, realistic and timely in comparison to previous programs. 

Some specific comments regarding areas of improvement are addressed below. 

An industry group made a comment indicated that changes to the SDP are needed to 

improve efficiency and reduce the level of subjectivity. As noted in SECY-08-0046, the staff 

developed several significant enhancements to the SDP guidance, including revamping the 

initial screening and characterization of findings process, improving the SDP appeal 

process, and revising the Public Radiation Safety SDP to improve its objectivity. The SDP 

continues to mature and remains an effective tool for determining the safety significance of 

identified performance issues. Oversight of the process has continued to focus on the 

timeliness of SDP reviews and on improvements to the process based on feedback from 

internal and external stakeholders. Most notably, the SDP met the timeliness goal of 

90 days for a second consecutive year. The staff acknowledged that changes to the SDP 

are needed and will continue to make changes to improve efficiency and reduce the level of 

subjectivity. 
- 47 - 

In response to the comment concerning inspection exit meetings, the staff strives to adhere 

to timeliness concerns by holding exit meetings at the end of the onsite inspection period. 

However, in some cases, new or additional information surfaces at the end of the active 

inspection period that has bearing on the inspection and needs to be reviewed. 

Sometimes, the staff has to gather additional inspection information before it holds a final 

exit meeting. Although these instances might give the appearance of inefficiencies in the 

ROP, they do serve to ensure that the ROP is effective and realistic. 

An industry group made a comment and recommended that there should be direct 

communication between NRR PRA staff and the licensee. The assessment of licensee 

performance deficiencies through the SDP process is controlled by the regions. The regional 

senior reactor analyst (SRA) is encouraged to communicate with the licensee throughout the 

development of the finding and assessment process. The NRR PRA staff performs peer review 

and quality checks on findings with potentially greater than green significance. Any questions 

raise by the NRR staff are directed to the regional SRA to resolve. 

Although the NRR staff agrees that there may be benefits to be gained from direct 

communication at technical level, there are also potential drawbacks. The NRR staff is 

limited in what information can be shared with the licensee since the region owns the finding 

(i.e., pre-decisional issues). Therefore, if the licensee participates in internal, deliberative, 

and pre-decisional issues, then the NRC staff would be obligated to allow the public and 

special interest groups similar access. 

Regarding the comment concerning current occupational radiation safety, the staff does not 

agree with the commenter that the inspections are not aligned to the licensees operating 

cycles. The level of effort established for the three procedure attachments are generally 

aligned with licensee operating cycles. The majority of the inspection effort in the 

occupational radiation safety area (Attachments 02 and 03) is completed on a biennial (24 

month) basis. Attachment 01 that covers the licensee’s control of access to radiologically 

significant areas is an annual inspection since it can be assessed whether the plant is in an 

outage or not. Attachment 02 that covers ALARA planning and job controls is structured to 

recognize the mature nature of the licensee’s ALARA programs. This inspection procedure 

has a variable level of inspection hours depending on the particular licensee’s relative 

performance against the other plants of its type in the industry as measured by the three 

year average annual collective occupational dose. More ALARA inspection hours (to a 

maximum of 80 hours biennially) are scheduled for those licensees with a more significant 

radiological challenge (those in the highest quartile of collective dose). For those licensees 

with a lesser radiological challenge (as indicated by their being in the lowest quartile of 

collective dose) the level of effort to complete Attachment 02 is reduced to a minimum of 64 

hours biennially. 

An industry group suggested in its comment that mid-cycle assessments should be eliminated 

for plants with good performance. But, the staff believes that mid-cycle assessments should be 

retained because not only does this include the quarterly Action Matrix and PI results, it also 

allows the staff to gain valuable insights into safety culture by looking at the cross-cutting 

aspects of performance issues associated with green inspection findings. These safety culture 

insights are, in part, determined through the presence of substantive cross-cutting issues 

(SCCIs), which can materialize even though the licensee is still in the licensee response column 
- 48 - 

of the Action Matrix. SCCIs are only assessed bi-annually, so the mid-cycle assessment is the 

vehicle in which this is performed. 

An industry group commented on bringing mothballed plants into the ROP, the staff has 

been working with its external stakeholders to define how the PI program is implemented for 

plants starting up from long outage periods. The staff expects to complete this effort in 

2008. 

Comments from industry groups showed a concern that resources were spent on a 

relatively lower number of findings from CDBI inspections. As previously noted in last year’s 

consolidated response, CDBIs were developed to improve the effectiveness of NRC 

design/engineering inspections based on lessons learned from past inspections and events. 

 The intent of these inspections is to focus on risk significant, low-margin components and 

operator actions that could potentially affect risk significant structures, systems, and 

components. During the 2006/2007 ROP cycle, CDBIs were completed at all licensed 

power plants. Starting January 2008, the CDBI frequency was revised from biennial to 

triennial so that CDBIs can better focus on risk-significant, low margin components and 

operator actions that have not been previously inspected in CDBIs. Information Notice 

2008-02 dated March 19, 2008, "Findings Identified During Component Design Bases 

Inspections" documented ten generic areas of CDBI findings that were identified at 40 plant 

sites during the 2006/2007 ROP cycle. Overall, the staff believes that these inspections are 

important and that the resources are well spent. 

15. Does the ROP ensure openness in the regulatory process?

Respondent Comments: 

Nuclear Energy Institute

The ROP is acknowledged as an open regulatory process. Most process descriptions and 

procedures are available on the NRC Public Website. The ranking of any plant in the Action 

Matrix is also readily available on the NRC public website. Annual and Mid-cycle assessment 

letters are publicly available and their issuance is announced by a NRC Press Release. The 

Annual Assessment meeting for each site is accompanied by a public meeting that is announced 

prior to the meeting. Individual inspections reports are made available on the NRC ADAMS 

system. Monthly public meetings are held to discuss ROP issues in which members of the public 

are invited to participate in person or by teleconference. During these meetings, significant 

dialogue between the industry and the NRC is held with meaningful discussions and both sides 

are typically able to come to reasonable resolutions. However, some documents that the NRC 

staff utilize to reach decisions, such as the Risk Assessment Standardization Project (RASP) 

Notebooks, are not publicly available. 

Strategic Teaming and Resource Sharing

The ROP process, with its many public meetings and opportunities for involvement, does ensure 

openness not available in the previous process. However, improvements could be made in 

soliciting stakeholder feedback when revising or developing regulatory documents such as 

Inspection Procedures, Manual Chapter guidance, or Regulatory Issue Summaries (RIS). As the 

agent for the industry, NEI routinely requests the opportunity to review draft documents and 

provide feedback in a public venue. However, the NRC is reluctant to share draft information, 
- 49 - 

particularly in the areas of Inspection Procedures and changes to Manual Chapter guidance. 

RIS 2007-21 “Adherence to Licensed Power Limits”, was recently issued without stakeholder 

feedback. The RIS resulted in unintended consequences – the RIS removed guidance that the 

industry used for maintaining average full power operation without providing or specifying 

alternative guidance. If stakeholder feedback had been solicited before the RIS was issued, the 

unintended consequences may have been avoided. 

Another area that is not open but sometimes used for ROP issues is the Task Interface 

Agreement (TIA) process (NRR Office Instruction COM 106). This process is really designed for 

internal use by the NRC. When the TIA process is used to resolve questions that an inspector 

may have as a result of an inspection issue, the process is not open to Licensee input and as 

such only the inspector’s question is considered. Licensees in many cases are unaware that the 

TIA process is being used and are not able to ensure that the question being asked by the 

inspector contains all relevant facts and information. In the interest of promoting consistency and 

reducing unintended consequences, the TIA process should be more open to stakeholder input 

and feedback. 

Union of Concerned Scientists

Nope. As indicated in the response to Question 10, many key ROP decisions are made by Mr. 

Dyer in secret with no publicly available documentation of the factors considered in the decisionmaking. The outcomes alone are made public. There won’t be openness in the ROP until the 

bases for NRC’s decisions are made publicly available. 

It is galling for the NRC to even pose this question after the Commission wrongfully and 

shortsightedly withheld information on the physical protection components of the ROP since 

August 2004. 

Foster

Largely, the indicators are made public, perhaps more insight into an industry that most. 

State of Ohio

It promotes openness, but we are not sure that it ensures openness. For example: asking for 

public comments at a public meeting can be very intimidating for members of the public. While 

the NRC would appreciate all comments it cannot ensure that anyone will comment. 

Region IV Utility Group

Yes. 

Comment 1: 

However, as discussed in the response to Question 12, unintended consequences may occur 

when stakeholder input is not considered whenever appropriate. As the agent for the industry, 

NEI routinely requests the opportunity to review draft documents and provide feedback in a 

public venue. However, the NRC is reluctant to share draft information, particularly in the areas 

of inspection procedures and changes to Manual Chapter guidance. RUG IV is interested in 

working with the NRC to identify methods that may be available to exchange information as 

documents are developed to avoid unintended consequences, while maintaining the NRC’s 

required independence. 

Comment 2: 

Another area that is not open but sometimes used for ROP issue resolution is the Task Interface 

Agreement (TIA) process (NRR Office Instruction COM 106). This process is designed for 
- 50 - 

internal use by the NRC. When the TIA process is used to resolve questions that an inspector 

may have as a result of an inspection issue, the process is not open for Licensee input and as 

such only the inspector’s input may be considered. In many cases, Licensees are unaware the 

TIA process is being used and are not able to ensure that the question being asked by the 

inspector is accompanied by all relevant facts and information. In the interest of promoting 

consistency and reducing unintended consequences, the TIA process should be more open to 

stakeholder input and feedback. 

RUG IV believes the key elements of the TIA process could be incorporated into a process 

similar to the ROP PI FAQ process which could provide a venue for appropriate stakeholder 

inputs prior to NRC reaching a decision. Additionally, this process could provide a mechanism 

for documenting the resolution and updating applicable guidance documents, thereby promoting 

consistency. See also response to Question 18. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-6, AStakeholders Perceive That the 

ROP Ensures Openness,@ external stakeholders generally acknowledged that the ROP ensures 

openness in the regulatory process, but both public and utility stakeholders expressed some 

concerns and noted that further improvements could be made as discussed below. 

An industry group made a comment that documents the staff used in Risk Assessment 

Standardization Project (RASP) were not made publicly available. As noted in Encl. 1 of SECY08-0046, the staff recently made the RASP Handbook publicly available on the ROP Web page 

and in ADAMS. 

Regarding the comment that draft documents of Inspection Procedures, Manual Chapters 

and RIS were not available for public comments before they are issued, the Inspection 

Program Branch holds monthly and other meetings with external stakeholders to provide 

information about changes to various inspection-related regulatory documents which may 

be of interest to external stakeholders. The staff has not solicited comments on changes to 

some inspection procedures because these changes have stayed within the regulatory 

framework established by the Reactor Oversight Process (ROP). The staff believes that 

there needs to be some level of independence and freedom from external influences on 

what is inspected and how inspections should be conducted. On the other hand, the staff 

has been sensitive to changes to documents such as Inspection Manual Chapters that 

might involve policy changes or changes to NRC's regulatory positions taken in the past 

because these changes could potentially result in an unnecessary increase in regulatory 

burden to the licensee with no commensurate improvement to safety. Additionally, the staff 

anticipates that it periodically needs to clarify NRC documents to provide additional 

guidance to both inspectors and the industry. In these types of situations, the staff will 

communicate and work with external stakeholders when issues, such as licensee’s 

adherence to licensed power limits, are brought to the NRC’s attention. When they are 

identified, the staff will solicit external feedback to successfully resolve these issues. 

Two industry groups commented on the Task Interface Agreement (TIA) process (NRR Office 

Instruction COM 106). This suggestion assumes that a response to a TIA is, in effect, a generic 

regulatory staff position. A TIA is a formal request for technical assistance from an NRC region 

or another NRC office containing questions involving regulatory or policy interpretations, specific 

plant events, inspection findings, or allegation-related issues. A TIA is a process tool that allows 
- 51 - 

the regional staff to interact with technical experts from NRR to ensure that the proposed staff 

position is consistent with the applicable regulatory staff position for that licensee. The 

inspection finding, itself, provides the result of the TIA process. And, as with all inspection 

issues, the licensee has ample opportunity to interact with the NRC staff in the inspection and 

enforcement processes. The NRC makes its office Instruction (COM 106) and its TIA responses 

publicly available to help stakeholders better understand the process and its results. 

Although licensee specific, some TIAs may have implications that are generic. To address this 

issue, the staff has been making time available at ROP monthly meetings to discuss recent TIAs 

that have been issued. In addition, the staff has been working with NEI and other stakeholders 

on the Regulatory Issue Screening Process that may provide additional means to promote 

consistency and reduce unintended consequences from plant specific issues with generic 

implications. 

A public interest group made a comment questioning openness of the ROP, the staff addressed 

similar comments in its response to Question 10 and Question 21 (Other Section) of this 

document. 

Regarding the comment that it could be intimidating for the public to make comments in a public 

meeting, the Agency has made public meetings more accommodating by making most subject 

documents publicly available on the web and in ADAMS. In addition, the Agency also provides 

the public opportunities to provide comments on its website and though email. 

16. Has the public been afforded adequate opportunity to participate in the ROP and to 

provide inputs and comments? 

Respondent Comments:

Nuclear Energy Institute

The public has been afforded adequate opportunity to participate in the ROP and to provide 

inputs and comments. These opportunities include the monthly public meetings, the annual 

assessment meetings at each site, and the ROP survey. Members of the public are frequently 

present at these meetings. 

Strategic Teaming and Resource Sharing

The public has been afforded adequate opportunity to participate in most of the ROP and to provide 

inputs and comments by way of the public monthly ROP meetings, annual ROP feedback surveys, 

and the annual assessment public meetings. This is not the case however in the area of Physical 

Protection. The Physical Protection area of the ROP is not open to the public, which is appropriate in 

most cases. However, program and process changes should go through a change management 

process (similar to ROP). Security-related information could be addressed by limited membership 

attendance as appropriate. 

Union of Concerned Scientists

Nope. I have attended more than one ROP monthly meeting between the nuclear industry and 

the NRC this past year. Until recently, the agendas for the public meeting notices did not explain 

what was to be discussed during the meeting, although such details had clearly been 

determined in advance between the nuclear industry and the NRC. These meetings are 
- 52 - 

conducted as Category 2 public meetings, meaning I and other members of the public cannot 

ask questions or make comments except during designated times during the meetings (usually 

coinciding with the lunch breaks and preadjournments). 

NRC staff typically arrives and departs these meetings for their specific issues. When an opportunity 

for public comment arrives (i.e., the lunch hour arrives), the NRC staff who discussed a topic 

between 9am and 10am earlier that meeting are often long gone. Thus, the public really doesn’t have 

adequate opportunity to participate and provide inputs. The monthly ROP public meetings should be 

conducted as Category 3 meetings and have public meeting notices that provide non-vague 

agendas. 

Foster

No. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Yes. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-7, "Opportunities for Public 

Participation in the Process," public respondents do not believe there is sufficient opportunity for 

public participation. Industry stakeholder responses were very positive. Industry acknowledged 

the ample opportunities for public participation such as monthly public meetings at NRC 

headquarters, annual public meetings conducted in the reactor communities, annual solicitation 

of public comments, annual ROP survey, and the NRC staff’s consolidated response. The 

specific comments are addressed below. 

Regarding a comment from the public that security performance deficiency information is 

withheld and is not available for public input, the staff is seeking ways to increase the openness 

and transparency in the security inspection programs. The staff will hold a series of public 

meetings in 2008 on security openness and explore such topics as making the results of security 

inspections public, reinstating public reporting of PI results, and making NRC security inspection 

procedures and end-of-cycle and mid-cycle security cornerstone assessment letters public. Any 

significant information and useful feedback would be reported to the Commission to request 

approval to implement an option to increase transparency and public availability of security 

inspection program information. The staff addressed a similar comment in its response to 

Question 8 of this document. 

A public interest group made a comment that more details in meeting agendas should be made 

available to the public, and suggested that the monthly ROP meeting be made to Category 3 

instead of Category 2. The staff believes that the ROP monthly meeting is appropriately 

conducted as a Category 2 public meeting, since it is a meeting between the NRC staff and the 

Nuclear Energy Institute ROP Task Force. Category 2 meetings are typically held with a group 

of industry representatives, licensees, vendors or non-governmental organizations. At this type 

of meeting, NRC anticipates that the public would obtain factual information and provide the 

agency with feedback on the analysis of the issues, alternatives and/or decisions. The public is 

invited to discuss regulatory issues with the agency at designated points identified on the 
- 53 - 

agenda. In order to improve the quality of public interactions in the ROP monthly meetings in the 

future, at the end of each agenda heading the staff will allow a period for public comment, so 

that any subject matter experts who are present for that discussion are available for questions 

and comments. 

As noted in Encl 3 of SECY-08-0046, based on feedback from external stakeholders, the staff 

began adding more detail to the agenda in the meeting notices so that potentially interested 

stakeholders could determine beforehand whether the NRC planned to discuss topics of 

interest. As noted, the meeting agendas have previously been enhanced to provide more 

information on the topics of discussion. In most cases, the details and context of items in the 

agendas are accessible on the Agency website or from ADAMS. 

17. Has the NRC been responsive to public inputs and comments on the ROP? 

Respondent Comments:

Nuclear Energy Institute

We believe that NRC has generally been responsive to both industry and public inputs and 

comments on the ROP. 

Strategic Teaming and Resource Sharing

The NRC has been responsive to public inputs and comments on the ROP. The NRC published 

a response to the 2006 ROP survey in which comments received were dispositioned. The 

STARS alliance supports the NRC publishing a response and encourages a published response 

by the NRC for these and any future ROP survey comments submitted. 

Union of Concerned Scientists

No. The NRC has been responsive to the comments from UCS, but has largely ignored and 

discouraged comments from other public interest groups and individual members of the public. 

The responses to this public comment solicitation are evidence. Many public interest groups and 

individuals submitted comments in past years, but that number has declined to nearly none. In 

discussions I have had with public colleagues, the Number One answer I get to the question of 

why haven’t you submitted ROP comments is, “it’s a waste of time.” The perception among the 

public is that the NRC does not want and will not consider input from the public. 

UCS shares that belief – but we’re going to submit comments any way even if we have to wrap 

them around a brick and throw them through windows out at NRC-land. 

Foster

No information; you should know this answer.... 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Yes. 

RUG IV supports continued published responses to public feedback. 
- 54 - 

RUG IV also supports the proposed changes to the SDP for the Public Radiation Safety 

Cornerstone that eliminates the aggregation of findings when the individual inputs are below the 

detectable threshold. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-8, "Stakeholders Perceive the 

NRC to Be Responsive to its Inputs and Comments," overall, external stakeholder satisfaction 

was generally favorable and consistent. Industry stakeholders believe that the NRC is 

responsive to inputs and comments, noting the published response to the 2006 ROP survey and 

encouraged continued responses for future surveys. Public stakeholders feel that the NRC is 

responsive to input and feedback but with only select public interest groups. The NRC 

disagrees with these perceptions. The NRC actively seeks feedback from the public regarding 

the ROP through external surveys, public meetings, and other venues. The NRC will continue to 

publish its consolidated response to the external surveys and encourage public input and 

feedback. In addition, the Agency received the same number of written comments in this survey 

as in 2006. 

18. Has the NRC implemented the ROP as defined by program documents?

Respondent Comments:

Nuclear Energy Institute

The ROP is generally implemented as defined by program documents 

Strategic Teaming and Resource Sharing

For the most part, the ROP is implemented as defined by program documents. However, the 

STARS alliance is concerned that in isolated cases, regional inspectors may be receiving 

guidance from individuals at NRR via informal communication channels. The primary concern is 

this approach does not result in durable documented guidance which in turn promotes regulatory 

inconsistency. 

Entergy

Inspectors tend to use MC 0609, Appendix E, “Examples of Minor Issues”, very narrowly. ROP 

findings should be clarified as minor if the finding is “similar” to the examples provided in the 

Appendix. Additional guidance should be given to ensure that failures to implement 

requirements that have insignificant safety or regulatory impact or findings that have no more 

than minimal risk are categorized as minor. 

Union of Concerned Scientists

No. For example, the NRC’s Office of the Inspector General documented in OIG-07-A-15, “Audit 

of NRC’s License Renewal Program,” dated September 6, 2007, that “Although expected to, 

audit team members do not consistently review or independently verify licensee-supplied 

operating experience information because program managers have not established 

requirements and controls to standardize the conduct and depth of such reviews” and also that 

“Post-renewal inspections are considered vital to ensure that licensees adhered to commitments 

made. However, the agency has only recently focused its attention on developing and 

overseeing details associated with these inspections.” So, it appears that NRC inspectors are 
- 55 - 

walking through the plants as governed by the ROP documents, but are not meeting basic 

expectations established for such inspections. Given that the NRC has been granting license 

renewals for years and two of the key lessons from the Davis-Besse debacle involved operating 

experience and commitment adherence, it’s inexplicable that NRC inspectors in 2007 would 

essentially just be going through the motions in these areas. Very little credit should be given just 

for showing up. 

Foster

I believe so. 

State of Ohio

In as far as completing inspections in a diligent manner, our experience in this area is positive. 

Region IV Utility Group

Yes. 

See our comments on Questions 5 and 12 above. 

RUG IV has a concern that regional inspectors may receive guidance from NRR staff regarding 

what they believe to be the intent of the guidance through informal communication channels. 

The primary concern is this approach may not be consistent with an in an agency position, is 

typically not well documented and may promote continued inconsistency. 

NRC Response: 

As noted in the ROP self-assessment and metric report for O-9, "Stakeholders Perceive That the 

ROP Is Implemented as Defined," most external stakeholders including utilities, State and public 

respondents believe the ROP is being implemented as defined. There were comments for areas 

of improvement as discussed below. 

Regarding the comment concerning IMC 0609 Appendix E, the staff believes that the 

respondent meant to comment on IMC 0612 Appendix E, “Examples of Minor Issues”. The 

same comment was addressed in the staff’s response to Question 10. 

Regarding the concern from an industry group that, in rare occasions, inspectors get 

guidance through informal channels from NRR, the staff agrees that it happens infrequently. 

 IMC 0305, that describes the Operating Reactor Assessment Program, and NEI 99-02, 

Rev. 5, that describes the PI Program, are two very important guidance documents 

concerning how to assess plant performance. As part of its responsibilities, NRR provides 

interpretation guidance to the regions covering these documents. However, issues 

involving PI interpretations, especially where the licensee and the region (and site resident) 

see the issue differently, are normally handled through the FAQ process. This process is 

conducted by NRC and NEI ROP Working Group, but will involve the region and licensee 

when questions arise concerning accuracy of facts, and opinion. This process appears to 

be working as intended, so the staff is not aware of any significant concerns or issues in this 

area. 

Regarding the comment from a public interest group on license renewal, the Office of the 

Inspector General (OIG) found that NRC has developed a comprehensive license renewal 

process to evaluate applications for extended periods of operation. However, OIG also 

identified areas where improvements would enhance program operations, such as 
- 56 - 

consistent evaluation of operating experience would improve NRC reviews, and more 

attention is needed in planning for post-renewal inspections as discussed below. 

1) Operating Experience: 

In the development of the Generic Aging Lessons Learned (GALL) Report, the staff 

reviewed domestic and international nuclear power plant operating experience. The 

GALL Report, Revision 0, covers operating experience up to 1998, through the 

review of all licensee event reports and the GALL Report, Revision 1, includes a 

review of license event reports and available international operating experience 

associated with these revised aging management programs (AMPs) up to 2004. 

The NRC staff reviews the operating experience program element of an applicants' 

AMP description based on guidance provided in the Standard Review Plan for 

Review of License Renewal Applications for Nuclear Power Plants (NUREG-1800). 

This includes review of any enhancements or modifications to the program based on 

site specific or industry operating experience. The staff asks questions during the 

onsite audit and/or as part of the Request for Additional Information process. In 

addition, IP 71002, “License Renewal Inspection,” directs the inspectors to verify 

operating experience. The regional inspectors review the plant’s corrective action 

database to ensure that relevant plant-specific operating experience has been 

captured. Findings from this review are documented in the inspection report 

generated by the region. 

2) Commitments: 

The NRC had already identified the need to revise IP 71003, “Post-Approval Site 

Inspection for License Renewal,” and started in summer 2006. Following 

consideration of comments from internal and external stakeholders, the staff issued 

the revised IP 71003 on February 15, 2008, in time for the regions to be prepared for 

the IP 71003 inspections for the first plants to go into the period of extended 

operation in 2009. 

An objective of IP 71003 is to verify that license conditions added as part of the 

renewed license, license renewal commitments, selected aging management 

programs, and license renewal commitments revised after the renewed license was 

granted, are implemented in accordance with Title 10 of the Code of Federal 

Regulations (CFR) Part 54, “Requirements for the Renewal of Operating Licenses 

for Nuclear Power Plants.” 

19. Does the ROP result in unintended consequences? 

Respondent Comments:

Nuclear Energy Institute

SECY 99-007, “Recommendations for Reactor Oversight Process Improvements” outlines the 

key objectives for the ROP as: 

1. Improve the objectivity of the oversight processes so that subjective decisions and judgment 

are not central process features. 
- 57 - 

2. Improve the scrutability of these processes so that NRC actions have a clear tie to licensee 

performance. 

3. Risk-inform the processes so that NRC and licensee resources are focused on those aspects 

of performance having the greatest impact on safe plant operation. 

Unintended consequences result whenever actions taken by NRC or licensees are not in full 

alignment with these objectives. For example, significant NRC and licensee resources are spent 

characterizing the significance of findings. The majority of these resources are focused on 

findings that have minimal risk significance. This result is inconsistent with the ROP objective to 

“focus resources on aspects of performance having the greatest impact on safe plant operation.” 

Strategic Teaming and Resource Sharing

In some cases, the ROP as implemented has resulted in unintended consequences. One 

example is the issuance of RIS 2007-21. This RIS was intended to address intentional abuses of 

average full power operation but in some cases resulted in plants de-rating the power level at 

which they were licensed to operate because of lack of appropriate guidance. 

Union of Concerned Scientists

Nope. 

Foster

Yes. I believe that sometimes licensees put too much emphasis on Indicators to the suffering of other 

items. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Current licensee experience finds that on occasion, significant time is expended on minor 

issues. Continued discipline in this area is needed to reduce the unnecessary regulatory burden 

that can occur when efforts are not well focused. 

RUG IV understands that the staff is open to further discussion on this issue at future ROP 

working group meetings. RUG IV supports continued discussion in this area and will support the 

efforts of the ROP Working Group. Please see Question 10 for additional comments. 

NRC Response:

As noted in the ROP self-assessment and metric report for O-10, "Stakeholders Perceive That 

the ROP Does Not Result in Unintended Consequences,” most external stakeholders believe 

the ROP is being implemented as defined. This metric met its criteria with mostly positive 

comments and a stable positive perception over time. There were a few concerns in specific 

areas as discussed below. 

A comment from an industry group noted that occasionally significant time is expended on minor 

issues. Based on recent interactions with the industry group, the un-cited example might be 

tracking unavailability on safety related equipment during shutdown via the Maintenance Rule. 

Another might be conducting rigorous (and expensive) assessments to justify when a 

component was unavailable. This example impacts MSPI, so it can have an impact on the 

Action Matrix if that particular PI were to cross a performance threshold as a result. The staff is 
- 58 - 

aware of these types of issues, and continues to improve the ROP based on stakeholder 

feedback. 

Another industry group made a comment on RIS-2007-21, “Adherence to Licensed Power 

Limits.” When the NRC issued the RIS, the main point was to remind licensees that they must 

adhere to their maximum thermal power licensed limit. Additionally, the RIS superseded the 

guidance issued to NRC Inspectors via the Jordan Memo, since the ROP process addressed 

conditions where the licensee exceeded their thermal power licensed limit. The NRC does not 

provide guidance to licensees on how to operate their plant but does ensure that the licensees 

operate within their licensed conditions. The RIS did not change any guidance but re-iterated 

that licensees must adhere to their license conditions. 

The NRC is currently working with NEI to establish additional industry guidance to ensure that 

licensees do not intentionally exceed their licensed thermal power limit. The NRC recognizes 

that the inspector guidance could be enhanced so that conditions that result in licensees 

exceeding their licensed limit would be evaluated and documented in a consistent manner. In 

the interim, licensees that initially de-rated after the RIS was issued interacted with the NRC and 

reached an understanding that it was not the intention of the NRC to require these de-rates. 

Note the staff addressed a similar comment in its response to Question 12 of this document. 

20a. Do the ROP inspection and assessment safety culture enhancements help to 

focus licensee and NRC attention on performance issues associated with aspects 

of safety culture? 

Respondent Comments:

Nuclear Energy Institute

The safety culture enhancements have only been in place for two assessment periods. No clear 

evidence exists to indicate that the additional attention brought to performance issues due to 

issuance of Substantive Cross-Cutting Issues (SCCI) have brought any appreciable increase in 

ability to determine deteriorating performance or performance issues that would have otherwise 

gone undetected. The NRC’s 18-month assessment of this process may provide some useful 

insight. 

Strategic Teaming and Resource Sharing

More time is needed to fully assess the effectiveness of the safety culture enhancements. 

However, an initial observation is only four or five aspect bins are being used to any great 

degree. If the entire set of Safety Culture bins were used it would better focus Licensees and 

NRC attention on performance issues associated with aspects of safety culture. Additionally, 

interpretation of implementing procedures (i.e. assignment of cross-cutting aspects) has varied 

across the regions. This variance along with the practice of assigning of cross-cutting aspects 

without the benefit of a completed evaluation by the Licensee has resulted in inconsistencies or 

changes in assignments. It has also resulted in the assignment of cross-cutting aspects for 

issues that are really minor in nature which in turn results in a disproportionate number of plants 

with substantive cross-cutting issues. 

The threshold of four aspects is inappropriate to assign a substantive cross-cutting issue. This 

threshold should be used to perform a more in-depth evaluation in that area to determine if a 
- 59 - 

substantive cross-cutting issue exists. The existence of four issues out of a population of 

thousands of issues identified in a typical corrective action program is not statistically significant. 

The low threshold for assigning a substantive cross-cutting issue coupled with unclear criteria for 

clearing a substantive cross-cutting issue creates significant uncertainty. 

Union of Concerned Scientists

I don’t know. 

Foster

Safety culture is difficult to define, difficult to address, and changes constantly. I do not believe 

that the NRC will ever be truly successful in this area. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Yes. 

The safety culture enhancements were implemented in 2006. Additional run time is needed to 

draw any conclusions with regard to whether the desired focus on performance issues 

associated with safety culture has been realized. It is notable implementation has not been 

consistent across all regions. RUG IV believes the overall goal to be able to identify declining 

performance prior to the identification of a significant safety concern is appropriate, but 

continued monitoring for improvements will be necessary to determine the desired outcomes 

have been achieved. 

NRC Response: 

As noted in the ROP self-assessment and metric report for AS-8, APerceived Effectiveness of 

Safety Culture Enhancements to ROP,@ this was only the second time that the external survey 

has included a safety culture question. Several stakeholders commented that it was too early in 

the implementation phase of the related program documents to answer whether the changes are 

helping to identify safety culture weaknesses and focusing licensee and NRC resources 

appropriately. 

In response to the comment that clear evidence does not exist to correlates the identification of 

Substantive Cross-Cutting Issues (SCCIs) with an increased ability to identify deteriorating plant 

performance, the staff agrees that it is difficult to determine a close relationship. The staff 

continues to believe that the identification of SCCIs is beneficial since it raises licensee 

awareness of issues related to safety culture aspects. Licensee actions in response to SCCIs 

can serve to prevent further deterioration in plant performance.

In response to comments that only four or five cross-cutting aspects are being used to any great 

degree, the staff has periodically examined inspection finding data on cross-cutting aspects. For 

all inspection findings issued in a typical 12 month period (from 10/1/2006 through 9/30/2007) 

the staff noted that nine cross-cutting aspects were each assigned to 30 or more inspection 

findings. Also, an additional 5 cross-cutting aspects were assigned to between 10 and 20 

findings. While situations did not arise that warranted the assignment of all cross-cutting 

aspects, the staff believes that a reasonable distribution of cross-cutting aspects resulted over 

the course of the fiscal year. In addition, the staff has embarked on an effort to modify some of 

the safety culture components and aspects and will take into consideration the appropriateness 
- 60 - 

to split, or combine, the components and aspects. In addition, the staff is evaluating the 

threshold for potential substantive cross-cutting issues. 

In response to the comment that cross-cutting aspects are assigned without the benefit of a 

completed licensee evaluation, the staff notes that inspectors use their best judgment to assign 

a cross-cutting aspect (if appropriate) at the time when they hold their exit meeting. The 

inspectors use available licensee information and information gained during their inspection to 

make the cross-cutting aspect assignment. If a subsequent licensee evaluation indicates the 

potential to assign a different aspect, the licensee may present that information to the region for 

consideration. Also, if the staff determines later through additional assessment not related to 

licensee evaluation that another cross-cutting aspect should have been assigned than the one 

originally assigned, the cross-cutting aspect can be revised. The staff agrees that it is important 

to retain transparency in the change of assignment of cross-cutting aspects. The staff presented 

draft IMC 0612, “Power Reactor Inspection Reports” guidance at a public meeting on May 14, 

2008 that describes how any NRC changes in cross-cutting aspect assignment could be 

communicated and documented. 

In response to the concern that cross-cutting aspects are assigned to issues that are really 

minor in nature, the guidance in IMC 0612 for making the decision on whether a finding is Green 

was not changed by the safety culture initiative. Outside of the safety culture initiative, the staff 

is evaluating the IMC 0612 guidance to identify greater than minor findings and plans to discuss 

the guidance at an ROP monthly meeting with the industry. 

In response to the concern that implementation of the safety culture enhancements has not 

been consistent across all regions, the staff has performed a review of the regional 

implementation practices for assigning cross-cutting aspects and identifying SCCIs. In general, 

the review found that each region was appropriately applying IMC 0305, “Operating Reactor 

Assessment Program” guidance and criteria for assessments. Overall, the review also found the 

regions are properly assigning cross-cutting aspects to findings in accordance with IMC 0612. 

However, the staff will continue to monitor regional consistency relative to the ROP safety culture 

enhancements. 

20b. Do the baseline Identification and Resolution of Problems inspection procedure 

(71152) and the special inspection procedures (93800 and 93812 respectively) 

provide an appropriate level of guidance on safety culture aspects and on the 

consideration of causal factors related to safety culture? 

Respondent Comments:

Nuclear Energy Institute

The procedures appear to direct inspectors to be aware of potential impacts of safety culture 

components on any identified issues. That level of guidance appears appropriate for inspectors. 

Strategic Teaming and Resource Sharing

The elements and structure of the safety culture guidance found in these inspection procedures 

and guidance incorporated by reference provides an adequate level of guidance for evaluating 

causal factors related to safety culture. 
- 61 - 

Union of Concerned Scientists

What does 71152 mean? Is that the score for the phrase ‘Identification and Resolution of 

Problems’ in Scrabble? 

Foster

This is obviously not a question for the public, but for licensees, which will probably be the 

principal responders to this questionnaire. I would have to read the procedures (again), but 

believe, as stated above, that the NRC will never have complete success in this area. The 

guidance is probably as good as it is going to get. Keep in mind that you get a “snapshot” that is 

likely to change, perhaps rapidly. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Yes. 

NRC Response:

The respondents generally indicated that the cited inspection procedures have an 

appropriate level of detail for guidance on safety culture aspects and consideration of safety 

culture causal factors. 

In response to the question about the meaning of IP71152, the “Identification and 

Resolution of Problems” inspection procedure is numbered IP71152. 

As part of the lessons learned process, the staff has developed draft safety culture changes 

for IP 71152, “Problem Identification and Resolution.” Those draft changes were shared 

with external stakeholders at a public meeting on May 14, 2008 for comment. The staff 

notes that ongoing NRC feedback processes are in place to provide opportunities for 

internal and external stakeholders to identify suggestions to enhance inspection guidance 

related to the safety culture area or any other part of the ROP. 

20c. Do the supplemental inspection procedures (Inspection for One or Two White 

Inputs in a Strategic Performance Area (95001), Inspection for One Degraded 

Cornerstone or any Three White Inputs in a Strategic Performance Area (95002)) 

respectively provide an appropriate level of guidance to evaluate whether safety 

culture components have been adequately considered as part of the licensees’ 

root cause, extent of condition, and extent of cause evaluations and to 

independently determine if safety culture components caused or significantly 

contributed to the risk significant performance issues? 

Respondent Comments:

Nuclear Energy Institute

The guidance directs a level of involvement that escalates based on the potential indication of 

degrading plant performance. At the level of one or two white inputs, it is appropriate that the 

inspector check that the licensee is appropriately considering the possibility of safety culture 
- 62 - 

components being a significant contributor to the cause(s) of the issues that resulted in the 

inputs. 

Strategic Teaming and Resource Sharing

There has been insufficient experience with 95001 and 95002 inspections and the application of the 

safety culture components to provide a meaningful answer. There is, however, a disconnect between 

corrective action programs and the 95001 and 95002 procedures. Corrective action program 

requirements do not include assessment of the safety culture components in the assessment of root 

and contributing causes as a normal matter of course. In many cases the root cause investigation 

may be complete before the determination of significance is completed. In these cases the licensee 

either has to go back and assess the root cause to apply safety culture components or allow the NRC 

to do that as part of the inspection process. For example, root cause investigators are not typically 

trained to assess whether safety conscious work environment was a contributor to an event. 

Union of Concerned Scientists

Rather than answering this question, I’d like to rephrase Question 9 and ask “Is the information 

contained in ROP solicitation questions relevant, useful, and written in plain English”? The 

answer to that question for Question 20c would be NO! 

Foster

Is someone practicing run-on questions or just making the survey invalid? I do not have any relevant 

information, but believe that the answer should be “no.” 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

Yes. 

However, RUG IV is concerned that recent 95001 and particularly 95002 supplemental 

inspections are not being successfully completed by licensees. RUG IV believes further 

interaction with the industry would be appropriate to discuss the challenges and proposed 

remedies for successful completion of the supplemental inspection activities. These insights 

could result in overall improved industry performance. The desired interactions could be 

provided with some combination of industry meetings/workshops and generic communications 

(update to RIS 2006-013). RUG IV would welcome the opportunity to assist in facilitation of a 

workshop for Region IV licensees. 

NRC Response:

In response to the comment that there is a disconnect between the inspection procedures 

IP95001 and IP95002 with the licensee corrective action programs, the staff notes that 

licensees are not required to incorporate the safety culture components into their corrective 

action program evaluations. The staff is aware that some licensees voluntarily revised their 

corrective action procedures to formally integrate the consideration of the safety culture 

components into their corrective action program evaluations. In addition, some licensees 

have performed cross-walks to demonstrate to inspectors that the safety culture 

components have been considered in their processes. Licensees are also aware that the 

inspection staff will perform their own evaluations, if necessary, to ascertain whether the 

safety culture components were appropriately considered by the licensee. The staff 
- 63 - 

believes that it is important for licensees to consider the safety culture component 

contribution to the risk significant issues. In those cases where the staff concludes the 

safety culture components were not appropriately considered during the implementation of 

IP95002 and the safety culture component caused or significantly contributed to the issue, 

the NRC inspection and assessment program described in IMC 0305 includes actions the 

staff can take including a request for the licensee to perform an independent safety culture 

assessment. 

In response to concerns about recent IP95001 and IP95002 inspections not being 

successfully completed, the staff has met with the industry at both Region III and IV Utility 

Group meetings to discuss this concern. No specific items of concern were brought to the 

staff’s attention during the Region III meeting. During the Region IV meeting a concern 

was brought to the staff’s attention as several licensees made general comments that the 

inspection guidance for determining the adequacy of the licensee’s actions could be 

improved to ensure greater consistency in implementation. The staff notes that ongoing 

NRC feedback processes are in place to provide opportunities for the staff to identify 

suggestions to enhance inspection guidance for areas related to safety culture as well as 

other parts of the ROP. In addition, specific industry concerns can be brought up at monthly 

ROP public meetings. The NEI ROP working group has proposed this topic for a future 

ROP monthly meeting. 

In response to the input that this question was poorly worded, for follow-on external surveys 

the staff will re-examine the safety culture related questions and re-write them as needed to 

improve their clarity. 

20d. Does the procedure for a Supplemental Inspection for Repetitive Degraded 

Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs, or One 

Red Input (95003) provide an appropriate level of guidance to independently 

assess the licensee’s safety culture and evaluate the licensee’s assessment of 

their safety culture? 

Respondent Comments:

Nuclear Energy Institute

It is too early to determine if the guidance in 95003 is adequate. It should be evaluated after the 

Palo Verde inspection has been completed. However, the procedure appears to overlyemphasize the possible contribution of safety culture. The procedure appears to be built with an 

inherent presumption that a failure to address safety culture attributes is a significant contributor 

to the issues identified in the findings. The procedure may more appropriately directed to first 

determine if a predominant cause of the findings was a licensee failure to implement one or 

more safety culture attributes. 

Strategic Teaming and Resource Sharing

For years the industry has been performing safety culture assessments using accepted survey 

techniques. In the 95003 process as modified to incorporate the safety culture assessment, the 

acceptability of these long-accepted techniques is now being questioned. For example, survey 

techniques that have been relied upon previously are now being questioned for their statistical 
- 64 - 

validity. An inspection procedure is not the appropriate place to establish standards for survey 

instruments. 

Union of Concerned Scientists

I don’t know. 

Foster

No information. 

Region IV Utility Group

Please see the response to Question 20c. Currently only one 95003 supplemental inspection 

has been performed utilizing this revised guidance and is still in progress. Therefore, it is too 

soon to draw any conclusions. 

It will be important to evaluate the results of the inspection to ensure the focus did achieve the 

desired results with regard to 1) establishing the remaining safety margin and 2) determining the 

extent the licensee’s safety culture contributed to the decline in performance. Because the 

inspection is appropriately more diagnostic in nature, significant resources are required by both 

the licensee and NRC to complete the inspection. Care must be taken this level of effort does 

not have unintended consequences and do more harm than good. 

NRC Response:

In response to the comment that the IP95003 guidance should be evaluated following the 

conduct of the Palo Verde inspection, the staff has obtained recommendations from the 

Palo Verde inspection team on potential enhancements for IP95003. Based upon the 

recommendations from the Palo Verde team along with all of the other lessons learned 

input, the staff has prepared a draft revision to IP95003 that was shared with external 

stakeholders at a public meeting on April 17, 2008. A number of draft changes were 

proposed relative to having a graded NRC safety culture assessment that is based on an 

NRC evaluation of the licensee’s safety culture assessment efforts. External stakeholders 

were given the opportunity to comment upon the staff’s proposed changes to IP95003. 

In response to the comment that IP95003 guidance is calling into question industry survey 

techniques based upon statistical validity, the staff believes that for plants that are expected 

to perform a third-party safety culture assessment (since they are in the ROP Action Matrix 

Multiple Repetitive Degraded Cornerstone column) that the survey tool statistical validity is 

an important aspect. The staff notes that generic guidance for conducting surveys (e.g., the 

American Association for Public Opinion Research: Best Practices for Survey and Public 

Opinion Research, numerous introductory textbooks on attitude measurement) outlines the 

importance of applying statistical methods to assess survey instrument reliability and validity 

to ensure that the conclusions drawn from surveys are accurate and meaningful. The 

Nuclear Energy Institute recently discussed a new initiative to develop an industry guidance 

document for safety culture assessments for licensee facilities. The staff plans to interact 

with industry as the guidance is developed to ensure that appropriate safety culture survey 

and other safety culture assessment tools’ (e.g., personal interviews, focus groups, 

observations, document reviews) expectations are described in the guidance. 

In response to the comment that the level of effort of the IP95003 may have unintended 

consequences, the staff has proposed a draft change of IP95003 to conduct a graded NRC 
- 65 - 

safety culture assessment that should result in the application of a more appropriate level of 

NRC resources. 

20e. Do the ROP inspection reports clearly describe inspection finding cross-cutting 

aspects? 

Respondent Comments:

Nuclear Energy Institute

The inspection reports clearly describe the cross-cutting aspects associated with the findings. 

However, improvements could be achieved in explaining in the inspection report how the 

inspector concluded that the cross-cutting aspect significantly contributed to the cause of the 

finding. Often the explanation is a mere assertion that the aspect was a contributor to the cause 

without a discussion of how the inspector determined that the aspect was a significant 

contributor to the cause(s). 

Strategic Teaming and Resource Sharing

The inspection reports identify cross-cutting aspects and tie the aspect to the associated finding 

through its relationship to the cause of the performance deficiency. Additionally, an alphanumeric numbering scheme is used to clearly identify the cross-cutting aspect that has been 

assigned. Aside from clearly identifying and describing the assigned cross-cutting aspect, there 

are times when a safety culture aspect is assigned without the benefit of the Licensee’s 

evaluation. In some cases this results in inconsistencies or changes in assignments, usually 

during the assessment process, that are not apparent to the Public (and sometimes the 

Licensee) because the inspection report or some other form of communication is not issued to 

document the change. When an assigned cross-cutting aspect is changed, the issue should be 

re-exited and the inspection report updated before the assessment report is issued. 

Union of Concerned Scientists

I don’t know. 

Foster

No, not well. 

State of Ohio

Yes, in our experience, the ROP inspection reports ensure linkage between observations and 

events from multiple areas of concern, looking for causes in one area that impacted results in 

other areas. 

Region IV Utility Group

Yes 

Please refer to the response to Question 6. 

NRC Response:

In response to the comment that NRC inspection report documentation should more clearly 

describe how the assigned cross-cutting aspect was a significant contributor to the finding, 

the staff discussed proposed draft changes to IMC 0612, “Power Reactor Inspection 
- 66 - 

Reports” with external stakeholders at a public meeting on May 14, 2008. The intent of the 

draft changes is to improve the guidance to inspectors on how to document the crosscutting aspects. 

In response to the comment that clearer communication is necessary whenever the staff 

changes the assigned cross-cutting aspect, the staff discussed proposed draft changes to 

IMC 0612 with external stakeholders at public meetings on May 14, and June 18, 2008. 

The intent of the draft changes is to increase the transparency of the NRC decisions to 

assign a different cross-cutting aspect and to communicate the change with the licensee 

and to document in an inspection report. 

20f. Do the Operating Reactor Assessment Program (0305) cross-cutting components 

and cross-cutting aspects provide an adequate coverage of the cross-cutting 

areas? 

Respondent Comments:

Nuclear Energy Institute

Although the components and aspects provide an adequate coverage of the cross-cutting areas 

a review of all findings associated with cross-cutting aspects indicates that most fall into a few 

aspects. Specifically, cross-cutting aspects H.4(b) (Human Performance, Work Practices, 

Procedural Compliance) and P.1(c) (Problem Identification and Resolutions, Corrective Action 

Program, Evaluation of Causes) are too broad in comparison with other aspects and are 

therefore assigned a disproportionate number of findings. 

Strategic Teaming and Resource Sharing

Cross-cutting components and cross-cutting aspects provide adequate coverage of the crosscutting area. However, the majority of the assigned findings with cross-cutting aspects generally 

use four or five bins. The original process was detailed and appeared to be designed for a more 

complex analysis of the safety culture for issues that rose to a significance level that warranted 

such analysis. Focus should be on the cause of the performance deficiency and the most 

appropriate cross-cutting aspect that was a significant contributor to the finding. If there was no 

aspect that was a significant contributor or the performance deficiency was not indicative of 

current performance, then no aspect should be assigned. Implementation of Manual Chapter 

0612 Appendix E is not consistent – finding classification of issues that resulted in no 

consequences are not consistently being classified a minor and are being tagged with an aspect 

- usually in one of the four or five most used bins. This practice is diluting the effectiveness and 

consistency of the assessment program. 

Entergy

The criteria for determining when a Substantive Cross-cutting Issue is considered should be reevaluated. Currently, a substantive cross-cutting issue is considered when four or more green 

or safety significant inspection findings are documented in the areas of human performance or 

problem identification and resolution. At the time the guidance was developed, there was a high 

threshold for the identification of cross-cutting aspects associated. During the last few years, 

additional guidance to inspectors on the identification of finding with cross-cutting aspects has 

been provided lowering this threshold. Subsequently, there has been a significant increase in 

the number of findings with cross-cutting aspects identified. As result, the potential for false 
- 67 - 

positive indications of a cross-cutting issue has greatly increased. This threshold should be reconsidered. 

The NRC should consider one annual assessment review for cross-cutting issues versus a MidCycle and an End of Cycle review. In the current process, the timing of the cross cutting issues 

can influence if a substantive cross cutting issue is identified. If the cross cutting aspects are 

identified late in the assessment period, there is little time for the licensee to provide information 

demonstrate improved performance or to improve performance. While this would also be true 

for a one-year assessment period, a one-year period would allow more time to implement 

actions and realize performance improvement during the assessment period for finding identified 

early in the period. With the current six-month assessment period, there is little time to address 

these issues. 

The NRC should re-assess the current cross-cutting themes. Some of the current themes are so 

broad that they are issued for a large percentage of the findings. One example is H.4.b, “defines and 

effectively communicates expectations regarding procedural compliance and personnel follow 

procedures”. There may be a need to re-evaluate the themes to determine if they should be split or 

more closely defined 

Union of Concerned Scientists

I don’t know. 

Foster

My experience was that cross-cutting items were not well handled by the process. 

State of Ohio

Not enough experience to comment. 

Region IV Utility Group

The design of the cross-cutting aspects does provide broad coverage of the cross-cutting areas. 

However, in actual practice only a few of the available cross-cutting aspects are assigned to 

findings. While the process is relatively new, there may be a need to clarify or redefine the 

individual cross-cutting aspect definitions. 

NRC Response:

In response to the comments about the shortcomings of the cross-cutting component and 

aspect characterizations, the staff is currently working to modify some of the IMC 0305, 

“Operating Reactor Assessment Program” safety culture components to sharpen their focus 

and remove redundancy. As part of this effort, the staff examined data on the frequency 

that cross-cutting aspects were assigned to findings and is assessing the need to either 

combine or split the cross-cutting aspect characterizations. The draft modifications of the 

safety culture components and aspects will be presented to external stakeholders for 

comment. 

In response to the comment that the staff should re-consider the threshold for a substantive 

cross-cutting issue, this evaluation will be done as part of the modifications of the safety 

culture components and aspects. 
- 68 - 

In response to the comment that the staff should conduct annual assessments to identify substantive 

cross-cutting issues, the staff does not believe a change is warranted at this time from the mid- and 

end-of-cycle assessment process. Within any given assessment time period, it is hypothetically 

possible for four or more findings with the same cross-cutting aspect to be identified late in the 

assessment period. However, in accordance with IMC 0305, the staff does not automatically identify 

a SCCI based on the existence of four or more findings with the same cross-cutting aspect. Instead, 

the staff considers available information about the licensee’s scope of efforts and progress to 

address the potential cross-cutting issue as part of the decision on whether to identify an SCCI. 

21. Please provide any additional information related to the Reactor Oversight Process 

*Note to better address the comments: The NRC staff evaluated these comments according to the 

ROP program areas (performance indicators, inspection, significance determination process, and 

assessment) that they were related to. The responses were then accordingly grouped, assessed, 

and addressed by the appropriate ROP program area leads within their respective areas of expertise. 

Comments received that were outside of the ROP program areas are addressed under the category 

of other. 

Performance Indicator Section

Respondent Comments:

None. 

NRC Response: 

N/A 
- 69 - 

Inspection Section

Respondent Comments:

None. 

NRC Response: 

N/A 
- 70 - 

Significance Determination Process Section

Respondent Comments:

None. 

NRC Response: 

N/A 
- 71 - 

Assessment Section

Respondent Comments: 

Strategic Teaming and Resource Sharing

Some Licensees have remained in column 3 or 4 of the Action Matrix for a long time (more than 

three years) and the Commission recently expressed the need to evaluate the reasons why. The 

ROP program does not provide a early release path from column 3 or 4 and in effect aggravates 

the condition by mandating the number of quarters that greater than green findings stay active in 

the Action Matrix and forcing three-year PIs to stay active with no method to reset the Action 

Matrix inputs (findings and PIs) after a reasonable amount of time if effective corrective actions 

have been implemented and the performance issues have been addressed. If the 

Commissioners are concerned with the amount of time a Licensee remains in column 3 or 4 of 

the Action Matrix, the ROP should provide clear exit pathways from these columns and a method 

to reset the long term indicators and findings if the issues have been resolved. 

When the Safety Culture program was conceived, greater than three aspects in a common bin 

were considered significant enough to identify a plant as an outlier. Since the program has been 

implemented, the percentage of findings with cross-cutting aspects has increased to nearly 100 

percent. As a result, the greater than three threshold is no longer valid in identifying statistical 

outliers. Originally the new program would conceptually spread the binning of aspects out; 

instead more aspects are being identified and binned in four or five commonly used bins. 

Region IV Utility Group

The MC 0305 assessment process requires greater than three findings with a common theme be 

evaluated to determine whether a substantive cross-cutting issue may exist. This threshold was 

established prior to the implementation of the safety culture initiative in 2006 as described in RIS 

2006-013. When this threshold was established a limited number if inspection findings were assigned 

cross-cutting aspects. However, with this recent initiative, the majority of inspection findings have 

assigned cross-cutting aspects. RUG IV recommends the threshold be adjusted to reflect the current 

practice of assigning cross-cutting aspects to all findings.

NRC Response: 

Regarding the industry group’s comment that some licensees have remained in column 3 or 4 

for too long (more than three years) and the ROP program does not provide an early release 

path for when licensee and staff actions have been satisfactory completed, the staff is aware of 

these concerns. However, licensees who remain in Column 3 of the Action Matrix are there 

because of performance issues, some of which may be attributed to the industry-backed MSPI, 

which uses a rolling 12 quarter window. In theory, these 5 PIs can remain greater than green if 

the performance issues that resulted in crossing threshold all occur in a short period of time. 

However, this situation is expected to be rare. Additionally, the Emergency AC System, typically 

a high risk significant system, comprises many of these cases seen so far since MSPI was 

implemented in April of 2006. Once the industry has more experience with MSPI and makes 

adjustments through plant modifications, PRA upgrades, and other procedural improvements, it 

is expected that these occurrences will be significantly reduced. In this regard, the staff believes 

that it is not necessary to alter the framework of the ROP at this time. Additionally, plants that 

enter Column 4 have significant performance issues that typically cannot be addressed in a 

short period of time. The licensee will be requested to perform an independent assessment of 

safety culture, and the staff will conduct its own assessment of the licensee’s review. These two 
- 72 - 

activities take time. Additionally, the staff usually implements a Confirmatory Action Letter and 

the licensee is held in Column 4 until they show they have addressed the performance issues 

satisfactorily and have shown for a period of time that they can perform at an appropriate level. 

Given this, the staff believes that an early exit criterion for either a Column 3 or 4 plant is not 

appropriate. 

Regarding the industry group’s comment that threshold to establish substantive cross-cutting 

issue theme was set prior to the implementation of the safety culture initiative in 2006, and the 

concern that, with the implementation of safety culture initiative, the number of substantive 

cross-cutting issues called have been increased significantly, the staff is currently working to 

modify some of the IMC 0305, “Operating Reactor Assessment Program” safety culture 

components to sharpen their focus and remove redundancy. A re-evaluation of the threshold 

for a potential substantive cross-cutting issue will be done as part of the modifications of the 

safety culture components and aspects. The staff plans to discuss these proposed changes 

with external stakeholders at a public meeting. 
- 73 - 

Other Section

Respondent Comments:

Nuclear Energy Institute

Monthly interactions between NRC and Industry through the ROP Working Group are critical to 

continued improvement of the ROP. The willingness to devote resources to these meetings is a 

clear indication of NRC’s commitment to making the process as predictable and efficient as 

possible. 

Union of Concerned Scientists

UCS has a long history of engagement on the ROP, including having been appointed to and 

having served on the Federal Advisory Committee Act panel chartered by NRC to assess the 

ROP’s pilot program and commented every year in response to NRC’s public comment 

solicitations (with many unsolicited comments between annual solicitations). 

UCS devoted these resources because we firmly believe that the absolute best protection that 

members of the public can have from the inherent hazards of the nuclear power reactors 

operating in their communities is an effective ROP. Sadly, the ROP is becoming less and less 

effective with each passing year since its inception. The ROP was a significant improvement 

over its predecessor, which relied almost heavily on the Systematic Assessment of Licensee 

Performance (SALP) reporting system. Whereas SALP was largely subjective, the new ROP 

was largely objective. Whereas SALP evaluated performance in broad categories, the new ROP 

assessed performance in discrete areas. Whereas SALP issued assessments every now and 

then, the new ROP issued assessments for each reactor each quarter. Whereas SALP defined 

few mandated NRC responses to declining performance signs, the new ROP clearly established 

the NRC reaction to each non-routine assessment outcome. 

The new ROP, for the first time in the NRC’s history, established both crisp, clear expectations 

and mandated regulatory responses when licensees fell short. For the first time, the NRC drew a 

line and had measures in place to ensure licensees stayed on the proper side of that line. 

The nuclear industry apparently abhors such accountability, for they have expended 

considerable effort – and achieved success – in chipping away at the new ROP until the current 

scheme is ROP by name only. The original objectivity is largely gone. The discrete areas remain, 

but have been rendered nearly pointless by the “all green all the time” colorization. And after 

nearly every mandated NRC response to a performance decline, the industry coerces the NRC 

into changing the ROP such that the next performance decline won’t trigger that response. The 

current ROP is a mockery of the program unveiled in 2000. 

State of Ohio

None at this time. 

NRC Response:

Regarding the comment that the ROP has been rendered pointless and has changed such that 

the next performance decline won’t trigger a response, as noted in SECY-08-0046, the selfassessment results for CY 2007 indicate that the ROP provided effective safety oversight, as 

demonstrated by meeting the seven program goals and achieving its intended outcomes. The 

ROP was successful in being objective, risk informed, understandable, predictable, and in 
- 74 - 

ensuring safety, openness, and effectiveness. Although the staff continues to experience 

challenges in certain areas, the staff will continue to improve various aspects of the ROP as a 

result of stakeholder participation, feedback, and lessons learned. As noted in Enclosure 1 to 

SECY-08-0046, “During CY 2007, the staff identified a possible declining trend within industry 

performance, as evidenced by an increase in the number of sites in columns 3 and 4 of the ROP 

Action Matrix. Approximately 5–7 sites (7–10 units) were in columns 3 and 4 between CYs 2003 

and 2006; however, during CY 2007, the number increased to 11 sites (17 units). Although a 

similar decline was not evident in the current industry trends program (ITP) results, the staff is 

evaluating this data, as well as other indicators, to determine whether this is an early indication 

of declining industry performance. The staff plans to discuss this potential concern during the 

2008 AARM, and any conclusions or insights gained during the AARM discussions will be 

shared with the Commission during the Commission briefing on the AARM results.” 

END