POLICY ISSUE 

Information 

December 11, 2015 SECY-15-0156

FOR: The Commissioners 

FROM: Victor M. McCree 

Executive Director for Operations 

SUBJECT: IMPROVEMENTS TO THE REACTOR OVERSIGHT PROCESS 

SELF-ASSESSMENT PROGRAM 

PURPOSE: 

The purpose of this paper is to inform the Commission of the U.S. Nuclear Regulatory 

Commission (NRC) staff’s revised approach to and implementation plans for the annual 

self-assessment of the Reactor Oversight Process (ROP) for calendar year (CY) 2015 and 

beyond. This paper does not address any new commitments or resource implications. 

BACKGROUND: 

The ROP was designed and implemented in 2000 to provide an objective, risk-informed, 

performance-based, transparent, and predictable approach to the regulatory oversight of 

nuclear power plant performance. The ROP is a mature and effective oversight process that 

has continued to evolve based on feedback and lessons learned. A contributor to its ongoing 

success has been the opportunity for, and inclusion of, continuous feedback and ongoing 

improvements via the staff’s ROP self-assessment program. The ROP self-assessment 

program is governed by Inspection Manual Chapter (IMC) 0307, “Reactor Oversight Process 

Self-Assessment Program.” Every year since 2000, the staff has conducted an ROP 

self-assessment, issued an ROP self-assessment Commission paper, and briefed the 

Commission on the results of the self-assessment following the Agency Action Review Meeting 

CONTACT: Ronald K. Frahm, Jr., NRR/DIRS 

 301-415-2986 
The Commissioners - 2 - 

(AARM). In the most recent paper, SECY-14-0047, “Reactor Oversight Process 

Self-Assessment for Calendar Year 2013” (Agencywide Documents Access and Management 

System (ADAMS) Accession No. ML14066A365), the staff noted that it had initiated an effort to 

improve the ROP self-assessment process and explore more objective performance metrics for 

assessing ROP effectiveness. 

In COMSECY-14-0030, “Proposed Suspension of the Reactor Oversight Process 

Self-Assessment for Calendar Year 2014” (ADAMS Accession No. ML14168A532), the staff 

requested Commission approval to suspend the ROP self-assessment for one year to modify 

the review methodology. Specifically, the staff noted its intent to: (1) develop a more effective 

self-assessment process with more meaningful metrics for use in 2015 and beyond; and (2) 

address ROP improvement recommendations from the multiple independent and focused 

ROP-related assessments performed in CY 2013 and CY 2014. In its staff requirements 

memorandum (SRM) to COMSECY-14-0030 (ADAMS Accession No. ML14262A078), the 

Commission approved the staff’s suspension of the annual ROP self-assessment for CY 2014 

and noted that the staff should inform the Commission of the status of ROP enhancements in 

the CY 2015 ROP self-assessment. 

As a result of early staff discussions on potential program improvements and efficiencies, the 

staff developed COMSECY-15-0014, “Proposed Elimination of Annual Reporting Requirements 

for Specific Evaluations within the Reactor Oversight Process Self-Assessment Process” 

(ADAMS Accession No. ML15072A202). In this COMSECY, the staff recommended eliminating 

three evaluations that had been enclosures to the annual ROP self-assessment: the regulatory 

impact summary, the resident inspector demographic analysis, and the ROP resource 

expenditure analysis. The staff noted that these detailed evaluations had been shown to offer 

only limited insights, were redundant to other processes, and did not appear to add the level of 

value as they did when they were initiated by Commission direction. The staff further noted its 

intent to incorporate certain objective aspects of these three evaluations into the revised ROP 

self-assessment process performance metrics. In its SRM to COMSECY-15-0014 (ADAMS 

Accession No. ML15169B131), the Commission approved the staff's request to eliminate annual 

reporting of these three evaluations from the existing ROP self-assessment process. 

DISCUSSION: 

In 2015, the NRC staff redesigned the ROP self-assessment process to better assess the 

effectiveness of a mature program by focusing on the efficacy of recent changes to the program, 

performing in-depth reviews of specific areas of interest, and verifying agency adherence to 

program governance. The new self-assessment approach will ensure that the ROP is being 

implemented reliably and predictably across all four NRC regional offices and Headquarters. In 

addition, the new approach will ensure that the staff appropriately invests resources in 

addressing value-added insights that improve the efficiency and effectiveness of the program. 

Through a series of meetings and interactions, a multidisciplinary team that included members 

from the Office of Nuclear Reactor Regulation, Office of Nuclear Security and Incident 

Response, and each of the four regions, developed the new approach and revised the 

associated governance documents. Other offices provided support as well as feedback on the 

draft governance documents. The staff discussed the proposed approach with senior NRC 

management at the AARM in April 2015 and at the Commission meeting on the results of the 
The Commissioners - 3 - 

AARM in May 2015. This topic was also discussed during a Commissioner Assistants’ briefing 

on ROP enhancements in July 2015, the Operating Reactor Business Line Commission briefing 

in August 2015, and an Advisory Committee for Reactor Safeguards briefing in September 

2015. 

The redesigned self-assessment process is governed by the revisions to IMC 0307 and its 

appendices (ADAMS Accession No. ML15307A023). IMC 0307 describes the process and 

revised approach, Appendix A to IMC 0307 describes the self-assessment metrics, and 

Appendix B describes the process for performing the baseline inspection procedure reviews. 

NRC staff updated each of these three governance documents to reflect the revised 

self-assessment approach. 

The revised self-assessment approach consists of three distinct elements: 

Element 1: Measure the effectiveness of and adherence to the current program, 

using objective metrics; 

Element 2: Monitor ROP revisions and assess recent program changes for 

effectiveness; and 

Element 3: Perform focused assessments of specific program areas and peer 

reviews of regional offices. 

This approach also addresses Recommendation 8 from the Commission-directed ROP 

Independent Assessment, “Reactor Oversight Process Independent Assessment 2013” 

(ADAMS Accession No. ML14035A571), which recommended revising the ROP 

self-assessment process to better solicit and assess both tactical and strategic feedback. 

Specifically, the first element provides for a tactical review of how the ROP is currently operating 

(from a data collection and analytical perspective). The second and third elements provide for a 

more strategic review and assessment of the efficacy of recent program changes and specific 

areas of management focus. 

Under Element 1 of the new self-assessment process, the staff will measure the effectiveness of 

and adherence to the current program using objective, measurable metrics based on readily 

available data. The metrics align with the Principles of Good Regulation, and a graded 

approach is used to measure metric adherence. For example, one of the efficiency metrics 

measures whether ROP feedback forms are completed within the applicable timeliness goals. 

The metric will be considered green if more than 90% of the feedback forms are timely 

(expected performance), yellow if less than 90% but greater than 80% are timely, and red (or 

failed) if less than 80% are timely. Yellow metrics warrant staff attention to address the 

declining trend, while red metrics require further evaluation and likely staff action to address the 

cause(s) for the failed metric. 

Using the objective metrics and other relevant feedback, the staff will continue to evaluate the 

effectiveness of each of the four major program areas of the ROP (i.e., the performance 

indicator program, the inspection program, the significance determination process, and the 

assessment program). The program area evaluations will also summarize changes to the 

program, current and/or future focus areas, and potential recommendations for improvement. 
The Commissioners - 4 - 

These evaluations will continue to serve as the planned program reviews for the ROP as 

stipulated in Appendix C to NRC’s Strategic Plan for Fiscal Years 2014–2018. 

Under Element 2 of the new self-assessment process, the staff will monitor ongoing ROP 

enhancement initiatives and summarize changes to the ROP that were completed during the 

year. In addition, the staff will select recent significant program changes and perform 

effectiveness reviews to ensure that the intended results of the implemented changes have 

been realized and to assess any unintended consequences. The selected topics for the 

effectiveness reviews will typically be identified early in the calendar year based on the extent of 

recent changes and whether sufficient implementation time has elapsed to adequately evaluate 

the effectiveness of the change. 

Under Element 3 of the new self-assessment process, the staff will select one or more topics for 

a focused assessment that delves more deeply into those aspects of the ROP. The 

assessments will typically involve focused surveys or interviews to gather feedback and 

perspectives from affected stakeholders. From these activities, recommendations will be 

provided to Senior NRC leadership to determine which program area(s) to pursue for the 

focused assessments as part of the AARM. 

Another key aspect of Element 3 is the conduct of peer reviews to ensure accountability to 

program governance as well as predictable and reliable program implementation across the 

regions. The staff will perform a peer review of a selected region each year on a rotating basis. 

Every fifth year, in lieu of a peer review, the staff will perform a comprehensive independent 

assessment similar to the ROP Independent Assessment performed in 2013. Independent 

assessments performed by other entities (e.g., the Office of the Inspector General, Government 

Accountability Office) may also serve as the comprehensive independent assessment. 

The staff will document the results of the metric analyses, program evaluations, effectiveness 

reviews, focused assessments, and peer reviews in the annual ROP self-assessment report. 

Staff will present the self-assessment results and any planned improvement actions at the 

AARM and subsequent Commission meeting. Complete details of the revised self-assessment 

process are documented in IMC 0307. 

For CY 2015, the staff plans to perform and document a limited self-assessment when it 

submits its annual report to the Commission in April 2016. The CY 2015 self-assessment will 

include only the metrics and program evaluations from Element 1 of the revised process and the 

status of ongoing and recently completed ROP enhancements from Element 2. The more 

detailed Element 2 and Element 3 assessments require more time to be effectively implemented 

than was possible before this new self-assessment process was finalized hence will not be 

included in the CY 2015 self-assessment. For CY 2016 and beyond, the staff will implement all 

three elements of the revised self-assessment program. 

CONCLUSIONS: 

The revised self-assessment process provides for a tactical and strategic review of the ROP 

and will increase opportunities to identify and implement meaningful program improvements. 
The Commissioners - 5 - 

COORDINATION: 

The Office of the General Counsel has reviewed this Commission paper and has no legal 

objection. The Office of the Chief Financial Officer has reviewed this Commission paper and 

determined that there is no resource implications. 

 /RA/ 

Victor M. McCree 

Executive Director 

 for Operations
The Commissioners - 5 - 

COORDINATION: 

The Office of the General Counsel has reviewed this Commission paper and has no legal 

objection. The Office of the Chief Financial Officer has reviewed this Commission paper and 

determined that there is no resource implications. 

 

/RA/ 

Victor M. McCree 

Executive Director 

 for Operations 

ADAMS Accession No.: ML15310A086 *Concurrence via e-mail

OFFICE NRR/DIRS/IPAB Tech Editor NRR/DIRS/IPAB NRR/DIRS/IRIB NRR/DIRS:DD 

NAME RFrahm CHsu NSanfilippo SRose SMorris 

DATE 11/06/15 11/06/15 11/06/15 11/06/15 11/12/15 

OFFICE RGN I* RGN II* RGN III* RGN IV* NSIR* 

NAME MScott KODonohue AMStone TVegel BMcDermott 

DATE 11/18/15 11/13/15 11/22/15 11/17/15 11/19/15 

OFFICE OGC OCFO* NRR EDO 

NAME MYoung JOjeda WDean VMcCree 

DATE 11/20/15 11/16/15 11/25/15 12/11/15 

OFFICIAL RECORD COPY