**NRC INSPECTION MANUAL**	IRAB

INSPECTION MANUAL CHAPTER 0307

# REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM

# Effective Date:  05/03/2022

# 0307-01	PURPOSE

The Reactor Oversight Process (ROP) self-assessment program evaluates whether the ROP meets its pre-established goals and intended outcomes by appraising the uniformity and effectiveness of regional and program office ROP implementation, evaluating effectiveness of significant ROP changes, and performing comprehensive reviews of selected ROP program areas to verify adherence to ROP program governance documents. The program uses a mix of data-driven monitoring efforts and traditional team-based retrospective reviews to accomplish the above purpose. This self-assessment approach ensures that the staff maintains multiple avenues for internal and external stakeholder feedback for continuous ROP improvement.

# 0307-02	OBJECTIVES

02.01	To establish the processes for collecting information and data to support the ROP selfassessment program.

02.02	To establish a process for objectively evaluating the effectiveness of the ROP in adhering to the Principles of Good Regulation and achieving the ROP program goals and intended outcomes.

02.03	To provide timely, objective information to inform program planning and to develop recommended improvements to the ROP.

02.04	To leverage ROP program execution data in all aspects of ROP self-assessment to support data-driven decision-making.

02.05	To appraise NRC regional program performance and program office performance in terms of effectiveness and uniformity in implementing the ROP.

02.06	To assess the effectiveness of significant changes to the ROP.

02.07	To ensure that complex ROP feedback from supplemental and reactive inspections is considered as appropriate for improvements to the ROP.

02.08	To perform focused in-depth assessments of specific program areas of interest.

02.09	To monitor trends in the implementation of the baseline inspection program and to conduct periodic baseline inspection program comprehensive reviews.

02.10	To provide mechanisms to solicit and assess feedback from both internal and external stakeholders.

02.11	To support the agency’s strategic goals as described in the current NRC Strategic Plan, NUREG-1614.

02.12	To inform the Commission, NRC senior management, and the public of the results of the ROP self-assessment program, including any conclusions and resulting ROP enhancements.

# 0307-03	APPLICABILITY

The self-assessment process described in this Inspection Manual Chapter (IMC) is designed to assess the uniformity and effectiveness of the implementation of the ROP, which outlines the oversight process for operating reactors. This self-assessment applies to all seven cornerstones of the ROP, and all processes and procedures in place that are utilized to implement the ROP. Reactors that are under construction or are in decommissioning are not within the scope of this process, nor are materials licensees or small modular or research and test reactors.

# 0307-04 	RESPONSIBILITIES AND AUTHORITIES

## 04.01	Director, Division of Reactor Oversight (DRO), Office of Nuclear Reactor Regulation (NRR)

a. Oversees the implementation of the ROP self-assessment program.

b. Develops policies and procedures for the ROP self-assessment program.

c. Reviews, approves, and ensures issuance of the annual ROP self-assessment SECY and other supporting reports.

d. Identifies, with assistance from regional division directors, the topics for focused assessments.

e. Confirms, as indicated by IMC 0040 Document Issuing Forms, any significant ROP changes that require effectiveness reviews.

f. Ensures ROP program area leads and ROP baseline inspection procedure (IP) leads are assigned appropriately.

## 04.02	Directors, Division of Security Operations (DSO) and Division for Preparedness and Response (DPR), Office of Nuclear Security and Incident Response (NSIR)

a. Assist in identifying topics for focused assessments.

b. Support NSIR office-level review and concurrence on the annual ROP self-assessment SECY.

c. Support the ROP implementation audits, baseline inspection program reviews, and focused assessments, as applicable.

d. Ensure data is collected and submitted to facilitate analysis of the ROP performance metrics.

## 04.03	Regional Directors, Divisions of Reactor Safety, Reactor Projects, Operating Reactor Safety, Radiological Safety and Security, and Nuclear Materials Safety (as applicable)

a. Assist in identifying topics for focused assessments.

b. Review the annual ROP self-assessment SECY, as coordinated by the regional Technical Support and Assessment Team Lead or Branch Chief.

c. Support the ROP implementation audits, baseline inspection program reviews, and focused assessments, as applicable.

d. Ensure data is collected and submitted to facilitate analysis of the ROP performance metrics.

## 04.04	Chief, Reactor Assessment Branch (IRAB)

a. Develops and updates governance documents for the ROP self-assessment program.

b. Ensures data from all sources are collected and consolidated to facilitate analysis of the ROP performance metrics.

c. Recommends and implements improvements to the ROP self-assessment program.

d. Ensures coordination and execution of planned ROP self-assessment activities.

e. Monitors the effectiveness of corrective actions and improvements to the ROP that are developed in response to recommendations from self-assessment activities.

f. Develops the annual ROP self-assessment SECY.

## 04.05	Chief, Reactor Inspection Branch (IRIB)

a. Ensures data are collected and consolidated to facilitate inspection program analyses.

b. Reviews results of baseline IP monitoring, effectiveness reviews, and other analysis conducted by IP leads and other IRIB staff.

c. Provides support or staffing support to the baseline inspection program review, ROP implementation audits, or focused assessments, as applicable.

## 04.06	ROP Program Area Leads

a. Monitor implementation and effectiveness of assigned program areas of the performance indicator (PI) program, the inspection program, the significance determination process (SDP), and the assessment program.

b. Collect and analyze self-assessment data for the previous year and develop the annual program evaluation for assigned program area for input into the annual ROP SECY.

## 04.07	ROP Baseline Inspection Procedure Leads

a. Monitor ROP program execution data as it relates to all assigned baseline IPs as described in IMC 0307, Appendix B, “Reactor Oversight Process Self-Assessment Baseline Inspection Program Monitoring and Comprehensive Review,” and act as appropriate to address any identified issues.

b. Participate and/or provide support or subject matter expertise and input to the baseline inspection program review every 5th year.

## 04.08	ROP Inspection Manual Chapter/Inspection Procedure Leads[footnoteRef:2] [2:  This includes ROP Baseline IP leads.] 

a. Perform the actions as described in Section 06.02 of this IMC regarding the effectiveness reviews of significant ROP changes.

b. Monitor ROP program execution data as it relates to all assigned IMCs and IPs and act as appropriate to address any identified issues, in accordance with the memorandum “Staff Expectations for Inspection Procedure and Inspection Manual Leads of Reactor Oversight Process Governance Documents,” (ML19219A225, nonpublic) and in accordance with IMC 0040, IMC 0308, IMC 0801, and Management Directive (MD) 8.13.

# 0307-05	SELF-ASSESSMENT BACKGROUND AND BASIS

The ROP is the NRC's primary means of ensuring that commercial nuclear power plants are operated safely, securely, and in accordance with applicable regulations. It is important that the ROP be periodically evaluated and improved when necessary to ensure continued achievement of its specified goals and intended outcomes. Additional information on the history and basis of the ROP self-assessment program can be found in Section 05.07 of IMC 0308.

The ROP self-assessment process has been an integral part of the staff’s implementation of the ROP since its inception in April 2000. The ROP development model presented in SECY99007, “Recommendations for Reactor Oversight Process Improvements,” dated January 8, 1999, included a steady-state process evaluation, or self-assessment process, that utilize measured objectives and predetermined success criteria to monitor the performance of the ROP. The results of these initial self-assessment efforts helped to determine the effectiveness of the new ROP, and in SECY-00-0049, “Results of the Revised Reactor Oversight Process Pilot Program,” dated February 24, 2000, the staff stated that it would continue to perform ROP self-assessments to collect additional lessons learned and gain insights from the new oversight process. In SECY-01-0114, “Results of the Initial Implementation of the New Reactor Oversight Process,” dated June 25, 2001, the staff reaffirmed “that the ROP will continue to require close scrutiny and oversight and established a self-assessment program that will identify additional areas for improvement.” Additionally, in the same SECY, the staff committed to, “as part of the Agency Action Review process... continue to report to the Commission on an annual basis the results of its self-assessment and any significant changes to the ROP.” These staff commitments form the purpose and the reporting requirements of the ROP self-assessment program. In the SRMs to SECY-00-0049 (Part 1 and Part 2), dated March 28, 2000, and May 17, 2000, the Commission approved the implementation of the new ROP program, including implementation of the associated self-assessment program.

## 05.01	ROP Goals

As noted in IMC 0308, “Reactor Oversight Process Basis Document,” the goals of the ROP include the four specific program goals of being objective, risk-informed, understandable, and predictable. Each of these ROP goals support the NRC’s mission and characterize the manner in which the agency intends to achieve its strategic goals of safety and security: to ensure adequate protection of public health and safety and the environment, and to ensure adequate protection in the secure use and management of radioactive materials.

The four ROP goals are summarized below:

Objective - Decisions are based on factual information and uninfluenced by emotion, surmise, or personal prejudice.

Risk-informed - Risk insights are considered along with other factors (such as engineering judgment, safety limits, redundancy, and diversity) to better focus licensee and regulatory attention on issues commensurate with their importance to health and safety.

Understandable - The process and its results are clear and written in plain language.

Predictable - More than one individual can follow the same defined process and arrive at the same conclusion in a consistent manner (i.e., repeatable).

## 05.02	Principles of Good Regulation

As noted in the Strategic Plan, the NRC maintains its regulatory competence, conveys that competence to stakeholders, and promotes trust in the agency by adhering to the longstanding Principles of Good Regulation and its organizational values. These principles focus on ensuring safety and security while appropriately balancing the interests of the NRC's stakeholders, including the public and licensees. The five Principles of Good Regulation are summarized below:

Independence - Ethical performance and professionalism should influence regulation. Final decisions must be based on objective, unbiased assessments of all information, and must be documented with reasons explicitly stated.

Openness - The public must be informed about and have the opportunity to participate in the regulatory processes as required by law.

Efficiency - Regulatory decisions should be made without undue delay. Regulatory activities should be consistent with the degree of risk reduction they achieve.

Clarity - Regulations should be coherent, logical, and practical. Agency positions should be readily understood and easily applied.

Reliability - Regulatory actions should always be fully consistent with written regulations and should be promptly, fairly, and decisively administered.

The goals of the ROP are consistent with the Principles of Good Regulation.

## 05.03	ROP Intended Outcomes

The ROP self-assessment process utilizes program evaluations and performance metrics to determine its success in meeting the goals and intended outcomes of the ROP. The intended outcomes of the ROP, which support its basis and are incorporated into various ROP processes, include the following:

Monitor and assess licensee performance

Identify performance issues through NRC inspection and licensee PIs

Determine the significance of identified performance issues

Adjust resources to focus on significant performance issues

Evaluate the adequacy of corrective actions for performance issues

Take necessary regulatory actions for significant performance issues

Communicate inspection and assessment results to stakeholders

Make program improvements based on evaluation of stakeholder feedback and lessons learned

Ensure reliable and predictable program implementation

## 05.04	Relation to NRC Strategic Plan, NUREG-1614

The ROP self-assessment program fulfills the ROP planned program review described in Appendix C of the current NRC Strategic Plan, Fiscal Years 20182022. In that way, the ROP self-assessment program is one aspect of ensuring the agency meets its goals as identified in the Strategic Plan.

## 05.05	Self-Assessment Approach

The ROP self-assessment approach has evolved since the staff conducted the first ROP self-assessment in 2001 after the first year of ROP implementation. In 2015, the self-assessment program was redesigned as a 3-element program to measure effectiveness of and adherence to the ROP, to evaluate effectiveness of recent ROP changes, and to perform in-depth reviews of specific areas of interest. The 3element program also addresses and aligns with Recommendation 8 from the Commissiondirected independent assessment, “Reactor Oversight Process Independent Assessment 2013” (ADAMS Accession No. ML14035A571), to revise the ROP self-assessment process to better solicit and assess both tactical and strategic feedback.

In 2020, the staff improved the self-assessment process, maintaining the 3-element approach while realigning the periodicity, scope, and type of reviews with a mature ROP (SECY-20-0039, “Revisions to the ROP Self-Assessment Program,” dated April 30, 2020). The enhanced self-assessment approach ensures that the ROP is being implemented reliably (consistently and as designed) across all regional and headquarters offices. Additionally, the new approach ensures that the staff appropriately invests resources to streamlined reviews and assessments that reveal high-value improvements in ROP program efficiency and effectiveness. Finally, the revised program ensures that to the maximum extent possible, self-assessment activities leverage ROP program data monitoring and analytics to evaluate ROP effectiveness.

The self-assessment approach consists of three distinct elements as described in this manual chapter:

1. Measure regional and headquarters program effectiveness and uniformity in implementing the ROP,

2. Assess effectiveness of recent ROP changes and evaluate the NRC’s response to significant licensee events or declining licensee performance, and

3. Perform focused assessments of specific ROP program areas, including the baseline inspection program.

# 0307-06	FORMAT AND STRUCTURE FOR SELF-ASSESSMENT

The self-assessment program consists of three distinct elements as noted in section 05.05 above. This section provides additional detail for each aspect of the ROP self-assessment program, organized by the three program elements.

## 06.01	Element 1 – Measure Regional and Headquarters Program Effectiveness and Uniformity in Implementing the ROP

a. Objective Performance Metrics

A set of performance metrics will be monitored and assessed as an integral part of each annual ROP self-assessment. These performance metrics align with the Principles of Good Regulation and are consistent with the goals and intended outcomes of the ROP. The performance metrics are designed to be objective and measurable based on readily available ROP program execution data, and to maximize the use of existing databases. Metric data is collected and analyzed by region and agency-wide for comparison purposes and to ensure reliable and uniform program implementation. A detailed description of these performance metrics is contained in Appendix A.

Each metric in Appendix A includes a definition and basis, the criteria to determine whether it is met, the organization responsible for gathering the data, program area(s) affected, and cross-references to the principle(s) of good regulation, ROP goals, and ROP intended outcomes that each metric is intended to support. A graded approach is used for performance criteria:

1. Green - A metric is considered Green if it meets or exceeds the specified criterion that represents expected performance and does not warrant further evaluation;

2. Yellow - A metric is considered Yellow if it falls within the specified range that warrants further evaluation and potential staff action to correct before the acceptance criterion has been exceeded; and

3. Red - A metric is considered Red if it meets the criterion that represents unexpected performance and necessitates further evaluation and likely staff action to address the cause(s) for the failed metric.

Separate and distinct from the ROP performance metrics, the NRC utilizes a planning, budgeting, and performance management process to ensure that the performance goals of the NRC Strategic Plan are properly assessed, and that key performance measures and program goals are met. The Agency performance management framework, which includes performance indicators and targets in the Annual Performance Plan and Congressional budget justification, are the primary means of determining whether the strategic performance goals are being met.

The ROP self-assessment program is not intended to replicate or replace this activity; however, many of the ROP self-assessment program metrics are the same as or similar to performance indicators found in the operating reactor business line annual performance plan. When applicable the same data should be provided, and the same guidance should be followed for both the ROP self-assessment program metrics and the performance indicators that are part of the annual performance plan to simplify data collection and to ensure clear and consistent data reporting.

The ROP performance metrics will be reviewed as part of the annual ROP selfassessment process to evaluate their effectiveness in providing a useful assessment of the ROP. Metrics will be added, deleted, or modified as necessary to provide a meaningful assessment of ROP implementation and effectiveness.

b. ROP Data Trending Focus Areas

The NRR/DRO staff monitors ROP program execution data related to an established set of data trending focus areas on a monthly basis. The purpose of the ROP data trending program is to supplement the ROP performance metrics in identifying positive or negative trends (as compared to historical averages or expected trends) in selected focus areas preferably spanning all four ROP program areas (inspection, PIs, SDP, and assessment). Early identification of these trends will guide staff toward areas where weakness or strength may exist, and/or where staff action may be required to improve program performance. Appendix A includes a list of the data trending focus areas, including a description, program area(s) affected, and the principles of good regulation each data trending focus area is intended to support. To the maximum extent possible, this data trending will be accomplished via automated data analysis and visualization software in a dashboard-style display.

Should any significant trends or insights be identified, the data will be provided to the appropriate program area lead for further analysis and action, including input to the annual metrics report and ROP self-assessment paper. Additionally, significant observations from the data trending focus areas will inform future ROP self-assessment efforts, including topics for change effectiveness reviews, focused assessments, and regional ROP implementation audits.

The ROP data trending focus areas will be reviewed as part of the annual ROP selfassessment process to evaluate their effectiveness as a tool in evaluating effectiveness and uniformity in ROP implementation. Data trending focus areas will be added, deleted, or modified as necessary to provide a meaningful assessment of ROP implementation and effectiveness.

c. ROP Program Area Evaluations

ROP program area leads evaluate each of the four ROP program areas (the PI program, the inspection program, the SDP, and the assessment program) for their effectiveness, potential improvements, and potential future focus areas on an annual basis. Specifically, the ROP program area leads conduct their respective program area evaluations based on ROP performance metrics data and analysis, ROP data trending insights, other ROP program execution data, internal (e.g., regional staff and resident inspectors) and external (e.g., ROP monthly public meeting discussions with Nuclear Energy Institute (NEI) staff or members of the public, Regulatory Information Conference discussions, or insights from participation in other industry events) feedback, and other relevant information. The evaluations for each ROP program area also include summaries of significant changes over the past year. The ROP program area evaluations align directly with and fulfill the intent and scope of the ROP planned program review as stipulated in Appendix C to the current volume of the NRC’s Strategic Plan, Fiscal Years 20182022, NUREG-1614.

d. ROP Implementation Audits

NRR will lead an annual ROP implementation audit of one region each year on a rotating basis. The objective of the audit is to appraise regional program performance in terms of effectiveness and uniformity of implementation of the ROP. The audit will be conducted in two parts: (1) a set of standardized, data-driven audit items covering the four ROP program areas that will objectively evaluate the region’s implementation of definitive program requirements (meets/does not meet), and (2) one or two focus areas where the audit team will perform a deeper-dive review. IMC 0307, Appendix C, “Reactor Oversight Process Self-Assessment ROP Implementation Audit,” contains the guidance for this self-assessment element. Should there be any programmatic recommendations resulting from an ROP implementation audit, these recommendations will be entered into the ROP Lessons Learned Tracker for disposition.

## 06.02	Element 2 – Assess Effectiveness of Recent ROP Changes and Evaluate the NRC’s Response to Significant Licensee Events or Declining Licensee Performance

a. Assess Effectiveness of Recent, Significant ROP Changes

Significant changes to the ROP that require Commission approval prior to implementation, consistent with MD 8.13, “Reactor Oversight Process,” will undergo effectiveness reviews. Additionally, an effectiveness review shall be performed of the status of all of the ROP recommendations from any of the activities described in Section 06.02.b. within 4 years of the report issuance. NRR/DRO management has discretion to approve any additional ROP changes that may warrant effectiveness reviews, as proposed by NSIR, regional offices, and NRR/DRO staff. As part of the IMC/IP revision process for these significant ROP changes, the responsible IMC/IP lead will identify on the Document Issuing Form (see IMC 0040) the specific data streams that they will monitor, as well as the duration of monitoring needed to determine change effectiveness and any unintended consequences. To the maximum extent possible, IMC/IP leads will use existing, automated ROP program execution data streams. As necessary, the effectiveness review lead will engage regional representatives or other ROP program or IP/IMC leads to shed light on the effectiveness review or any effects of the change on their programs.

Once the monitoring period is complete, the resulting data analysis, along with any additional insights, will form the basis for the staff’s assessment of change effectiveness. At a minimum, each effectiveness review will include a review of the basis of the change, verification of the intended outcomes of the specific change, an assessment of unintended consequences of both individual and collective changes (e.g., cumulative impacts) made to the ROP during the period of time under consideration, an explanation of how effectiveness was measured, and verification that the changes remain consistent with the ROP goals, Principles of Good Regulation, and ROP intended outcomes as specified in Sections 05.01 through 05.03 of this manual chapter. If the review concludes that a change has been ineffective or warrants further improvement, adjustments will be considered as needed to more fully address the issue or concern.

An effectiveness review report memo template is available for use at ML19274B459 (non-public) for the formatting and minimum requirements of the effectiveness review. As appropriate, recommendations from the effectiveness review requiring revisions to ROP program documents shall be tracked through the ROP feedback form process governed by IMC 0801, “Inspection Program Feedback Process.” The staff will include a summary of the results of the effectiveness review(s) completed each year in the annual ROP self-assessment SECY.

b. Evaluate NRC Response to Significant Licensee Events and Declining Licensee Performance

If an incident investigation team (IIT) is warranted in response to a significant licensee event (see MD 8.3), part of the scope of the investigation will consider whether NRC activities preceding and during the event were timely and adequate. If a licensee’s performance has declined such that the licensee is in Column 4 of the ROP Action Matrix, and a supplemental IP 95003 is performed, a portion of that inspection (see IP 95003, Sections 02.11 and 03.11) includes a review of the NRC’s assessment and inspection process (including the PI program, the inspection program, the SDP, and the assessment program) at the subject facility. If a licensee’s performance has declined such that the licensee has entered the IMC 0350 process, the IMC 0350 oversight panel will collect lessons learned from the IMC 0350 process and any subsequent recommended changes to the ROP and provide them to NRR/DRO (see IMC 0350, Sections 0350-07.01.f. and 0350-09.n.).

The overall purpose of these assessments is to determine the effectiveness of NRC response to significant licensee events and declining licensee performance. These inward-looking evaluations are an important aspect of ROP self-assessment, in that they can provide complex feedback regarding NRC oversight and incident response capabilities, guidance, and execution. Additionally, in the case of supplemental inspections, the evaluations provide a direct assessment of whether the ROP (as implemented) adequately detected declining licensee performance, and where the ROP might be improved in that regard.

As described below in Section 06.02.c., the complex ROP feedback from the response to significant licensee events and declining licensee performance is tracked to completion using the ROP lessons learned tracker. As the feedback from one of these activities (IIT, IP 95003, IMC 0350) is being incorporated into the ROP, an effectiveness review shall be performed of the status of all of the recommendations from that effort, in accordance with Section 06.02.a. of this IMC.

c. ROP Lessons Learned Tracker

NRR/DRO has the overall responsibility to gather feedback, lessons learned, and recommendations for ROP program improvement and to ensure they are adequately addressed. The staff maintains a non-public SharePoint database (https://usnrc.sharepoint.com/teams/NRR-ROP-Lessons-Learned-Tracker/Lists/ROP%20Tracker/) of the lessons learned from the evaluations as described in Section 06.02.b. to track the status of complex, longer-term program changes, as they often involve multiple internal and external stakeholders to evaluate and resolve and may require Commission approval to revise the policy and implement the changes, as required by MD 8.13. This lessons learned tracker is also used to track other complex ROP feedback as appropriate, such as complex ROP action items from Government Accountability Office or the NRC Office of the Inspector General audits or any other complex ROP feedback from other ROP self-assessment activities. As applicable, the staff will enter less complex ROP recommendations pertaining to only one IMC or IP into the ROP feedback form process governed by IMC 0801. The ROP feedback form process and ROP lessons learned database ensure that ROP recommendations are gathered, assessed, and tracked to completion. The staff will compile and provide status updates of ROP changes periodically to NRC management and will provide an overall summary of the staff’s efforts in this area in the annual ROP self-assessment SECY.

## 06.03	Element 3 – Perform Focused Assessments of Specific ROP Program Areas, Including the Baseline Inspection Program

a. Focused Assessments

Every 3 years, NRR/DRO, in consultation with the regions and NSIR, will designate one topic for a focused assessment that delves more deeply into a programmatic area of the ROP that shows indications of potential weakness or areas for improvement. The topic may be prompted by any of the data sources described in section 07.01, or any other insights, and consideration should be given to ensure that focused assessments are performed across a number of the four ROP program areas (i.e., PI program, inspection program, SDP, and assessment program) at least once within a tenyear period. For the inspection program, the baseline inspection program review described in Section 06.03.b below is considered to be a focused assessment and may be counted as such if it aligns with the focused assessment periodicity. The selected focused assessment topic will be presented to senior agency management at the Agency Action Review Meeting (AARM). The Office Director of NRR will make the final determination of the selected focused assessment topic.

NRR/DRO will assemble a working group and develop a charter that tailors the review to the subject area, specifically defines the scope of the focused assessment, identifies the attributes to be verified and relevant standards or guidance, and intended outcomes. The assessments can verify consistency of ROP program area implementation among the regions, verify an aspect of the program over all four regions, or focus on specific areas within one or two regions. The assessments will utilize ROP program execution data to the maximum extent possible in analyzing program performance or gaps, while also typically including focused surveys and/or interviews to gather feedback and perspectives from affected stakeholders. The assessments are generally led by NRR/DRO staff, with working group members including staff with relevant experience from NRR, stakeholders from other program offices, and the regions.

The three-year periodicity of the focused assessments is set such that the team/working group has sufficient time to complete the assessment, develop recommendations, communicate appropriately, and implement the resulting program changes, if any. Because of the comprehensive nature of the focused assessments, the recommendations resulting from these efforts often require Commission communication in the form of Information or Notation Vote SECY papers, or Commissioner’s Assistant Notes, as appropriate. The focused assessment teams/working groups should consider these requirements when developing their project plans. Changes resulting from these focused assessments will also frequently meet the threshold for conducting an effectiveness review as described in Section 06.02.a above, and special attention shall be paid to monitor appropriate data streams for both effectiveness and unintended consequences.

b. Baseline Inspection Program Comprehensive Review and Routine Monitoring

Every fifth year after all regions have received an ROP implementation audit, the staff will conduct a baseline inspection program review, with engagement from NRR/DRO and regional Division management, and participation from a cross-section of baseline IP leads and regional representatives.

The lead for each baseline IP will conduct data-driven monitoring of assigned baseline IPs. Based on the results of the monitoring, baseline IP leads will act as necessary to maintain the effectiveness of each baseline IP. The data-driven approach described in this aspect of the self-assessment process does not preclude each IP lead’s responsibility to maintain relationships and avenues for feedback from resident inspectors and regional staff, including periodically observing or participating in inspections to verify baseline IP adequacy and clarity firsthand.

Guidance for the baseline inspection program comprehensive review and baseline IP monitoring is provided in Appendix B to this manual chapter.

The results of evaluations conducted under the three elements of the ROP selfassessment program will be documented or referenced in the annual ROP selfassessment report. The results will also be presented to senior NRC management at the AARM. The staff will enter lessons learned and recommended program improvements from ROP self-assessment activities into the ROP feedback form process or ROP lessons-learned tracker, as appropriate.

# 0307-07	DATA COLLECTION AND STAKEHOLDER FEEDBACK

## 07.01	Data Sources and Collection

NRR/DRO has the overall responsibility for data collection, with support from the regional offices, NSIR, and other NRC organizations, as necessary. A variety of methods and sources are used to collect data regarding the performance of the ROP. These methods include compiling data and information from the Reactor Program System (RPS) or other existing databases, internal and external stakeholder surveys and interviews, independent audits, responses to *Federal Register* notices, industry-level indicators and operating experience, agency performance indicator databases, program evaluations and document reviews, and other stakeholder interactions.

To the extent possible, data collection is from agency databases and the need for ad hoc, manually developed data is minimized, particularly for the ROP performance metrics, ROP data trending focus areas, data-driven portions of the ROP implementation audit, and baseline IP monitoring efforts. Since the self-assessment program heavily relies on the quality of the data contained in the RPS database, it is imperative that the regions and other internal stakeholders ensure the accuracy and timeliness of the RPS data. NRR/DRO staff will periodically evaluate the need to modify existing systems such as RPS (or add new automated systems) to ensure efficiency and consistency in obtaining necessary ROP program execution data.

Data to support the ROP performance metrics is typically collected quarterly. Data collection and reporting is typically completed within 45 calendar days of the end of the quarter under review. In cases where the NRC performance plan performance indicators match the ROP performance metrics, the regions are not required to double-report that data on a quarterly basis as it can be pulled from on the agency’s performance management SharePoint site. However, all regions and NSIR will compile and submit ROP performance metrics data at the end of each calendar year. Due to the short timeframe required to develop the annual ROP self-assessment SECY, this end-of-calendar-year reporting is requested to be completed within 15 calendar days of the end of the calendar year. Offices and regions shall take care to ensure that data is collected and reported accurately.

## 07.02	Soliciting and Evaluating Stakeholder Feedback

The staff emphasizes stakeholder involvement and open communication regarding the ROP. The staff uses a variety of communication methods to ensure that all stakeholders can access ROP information and have an opportunity to participate in the process and provide feedback. The staff actively seeks feedback and implements improvements to the ROP based on evaluation of feedback and insights from all stakeholders.

a. External Stakeholder Feedback Opportunities

1. The staff conducts periodic public working-level meetings with NEI, the industry, and interested stakeholders to discuss the status of ongoing refinements to the ROP. The staff provides a meeting notice with an agenda so that interested stakeholders can determine beforehand whether the NRC plans to discuss topics of interest. The staff also offers the opportunity for public comment at the end of each topical area to enhance the public's ability to engage relevant staff members on topics discussed during these meetings. Additional topic-specific public meetings are held, as warranted.

During these public meetings, the staff provides and collects completed public meeting feedback forms, which are used to evaluate the effectiveness of public messaging and use of plain language. These public meeting feedback forms will be used to inform external communication initiatives to ensure that external content and publications relative to the ROP are clear and understandable.

2. The staff conducts public meetings or other engagement activities in the vicinity of each operating reactor site to discuss the results of the NRC’s assessment of the licensee’s performance. These annual engagements provide an opportunity for interested stakeholders to engage NRC staff on its role in ensuring safe and secure plant operations, either by asking questions during the meeting or engaging NRC staff. Public meeting feedback forms (described above) are collected during these meetings as well.

3. The staff maintains numerous external Web pages to communicate current ROPrelated information and results. The staff monitors the ROP Web pages to ensure information accuracy.

Additionally, the staff maintains an ROP “Contact Us” form on the external NRC Website. External stakeholders can use this web-based form to provide feedback or ask questions regarding ROP implementation. The staff will provide an initial response acknowledging receipt of the feedback generally within 5 working days and will provide a more complete technical response generally within 45 days. Feedback received through this mechanism that is outside the scope of the ROP will be forwarded to the Office of Public Affairs and/or other appropriate organizations for their consideration. All feedback received through with the ROP Contact Us form will be maintained per the public ADAMS records guidelines and requirements.

4. The staff typically sponsors a breakout session at the annual Regulatory Information Conference focused on ROP-related issues and topics of interest, or sponsors an ROP poster session, so that interested stakeholders can gather information, ask questions, or provide feedback about the ROP. Additional ROP topics may be discussed during the regional breakout sessions.

5. The staff conducts external stakeholder surveys and/or interviews as needed to perform focused assessments per Element 3 of the ROP self-assessment process.

6. Independent assessments of ROP-related program areas from external agencies such as the Government Accountability Office provides valuable external feedback on the efficacy and consistency of the ROP.

b. Internal Stakeholder Feedback Opportunities

1. NRR and NSIR staff and management conduct periodic conference calls with regional management and staff to discuss current issues associated with the ROP. The staff also meets periodically with regional managers to discuss more complex ROP topics and issues.

2. NRR and NSIR staff participate in each region’s inspector counterpart meeting and/or end-of-cycle meetings so that regional staff and management can provide feedback on ROP implementation.

3. The ROP feedback process described in IMC 0801 allows NRC staff to identify concerns or issues and recommend improvements related to ROP governance or guidance documents. Responsible staff will respond to and address the feedback in accordance with the requirements and expectations of IMC 0801.

4. As described under Element 2 of the self-assessment program, the staff performs lessons-learned evaluations and recommends ROP program improvements upon completion of significant NRC reactive or supplemental inspection activities.

5. The staff maintains an internal Contact Us Form on the ROP Digital City Website. Internal stakeholders can provide feedback or ask questions regarding ROP implementation across a wide array of topic areas, including but not limited to, resident inspector support concerns. Similar to the external ROP Contact Us form, responsible staff will provide an initial response acknowledging receipt of the feedback generally within 5 days and will provide a more complete response generally within 45 days. Feedback received through this mechanism that is more technical in nature will follow the formal ROP feedback process described in IMC 0801.

6. The staff conducts focused surveys and/or interviews as needed to perform the focused assessments per Element 3 of the ROP self-assessment process.

7. While no longer an explicit aspect of the ROP self-assessment program, the staff has previously completed internal independent evaluations of the ROP, such as the “Reactor Oversight Process Independent Assessment 2013” (ADAMS Accession No. ML14035A571). The staff maintains the ability to perform independent assessments of the ROP and may choose to do so when deemed appropriate.

The staff notes that there are a number of ROP communications efforts that promote regional uniformity and effectiveness in implementing the ROP, but do not necessarily provide an avenue for explicit feedback. For instance, NRR staff issues the inspector newsletter on a quarterly basis to share value-added findings, best practices, inspection guidance, and regulatory issues that are timely and have wide application and interest to inspectors and staff implementing the ROP. NRR and each of the four regions has a newsletter editorial board member who identifies potential topics and authors for newsletter articles and provides technical review of the content. Other efforts that promote uniformity and effectiveness include operating experience communications and the NRR staff’s maintenance and frequent updates to the ROP dynamic website.

# 0307-08	ROP SELF-ASSESSMENT DATA ANALYSIS

NRR/DRO has the overall responsibility for setting appropriate thresholds for evaluating ROP implementation and effectiveness, analyzing ROP program execution data and other self-assessment results, developing plans to address any potential areas of weakness, and implementing appropriate ROP improvements. NRR/DRO staff, with regional input, determine the performance thresholds for acceptable ROP implementation and effectiveness, which are provided for each ROP performance metric in Appendix A. There are no performance thresholds for the data trending focus areas listed in Appendix A, however, NRR/DRO staff use data analytics and graphics tools to determine any significant positive or negative trends in these areas.

Based on the staff’s program-wide analysis of each calendar year’s self-assessment results, a positive overall favorable comparison of results to performance criteria would indicate that the ROP met its program goals and objectives for that calendar year. However, for any instance where an aspect of the ROP program may exhibit signs of weakness in terms of performance (e.g., a “Red” performance metric), the staff, as noted above, will develop a plan to further analyze and address the issue, determine causal factors, and develop recommended process improvements. A situation such as a Red performance metric may result in programmatic improvements and/or other action (such as clarifying program documents or training) to improve performance in the measured area. If no changes or other actions are being proposed or planned based on a Red metric, the staff will justify the reason(s) for the anomaly and why the anomaly is not indicative of poor performance or a program issue.

Due to their direct experience with the inspection and oversight programs gained through their implementation of the procedures, the regions and NSIR will be consulted during the data analysis and recommendation development process to ensure the regional insights are incorporated into the evaluation and change process.

# 0307-09	ROP SELF-ASSESSMENT REPORTING

There are several periodic ROP self-assessment reports that serve different purposes as described below:

## 09.01	Annual ROP Performance Metric Report

NRR/DRO develops an annual ROP performance metric report after the conclusion of each calendar year. The overall summary report must discuss any metrics that did not meet their pre-established criteria, the staff’s analysis of the reasons for not meeting the criteria, and any actions taken or planned to change the program or improve its implementation. The report will briefly discuss any other significant lessons from the analyses of the metrics, even if the lesson is related to a metric that did meet its criteria. The report will also identify any metrics not counted during the previous year with appropriate justification. Lastly, the report will also include any significant positive or negative trends or insights gained from the staff’s monitoring and analysis of the ROP data trending focus areas, including any staff actions to address identified issues. The annual ROP performance metric report may be issued separately from or as an enclosure to the annual Commission paper discussed in Section 09.02.

## 09.02	Annual Commission Paper

The staff presents the results of its annual self-assessment of the ROP in a SECY Information paper. The annual ROP self-assessment SECY also supports the AARM and the Commission briefing on the results of the AARM. This paper typically includes the results of each element of the ROP self-assessment program for the preceding calendar year including lessons learned, planned ROP self-assessment activities for the next calendar year, and any other significant ROP changes or updates.

The paper also presents the staff’s overall conclusions as to whether the ROP has been successful in meeting the goals and intended outcomes of the ROP and the Principles of Good Regulation. The paper typically includes the ROP program area evaluations (Element 1 of the ROP self-assessment program) as an enclosure.

## 09.03	Reports Documenting ROP Self-Assessment Activities

The results of the staff’s effectiveness reviews, ROP implementation audits, and focused assessments may be documented in individual reports or may be incorporated into the annual ROP Commission paper discussed above. Recommendations made by internal and external stakeholders resulting from these self-assessment activities will be tracked and addressed under Element 2 of the ROP self-assessment program.

# 0307-10	REFERENCES

This list of references encompasses the entire ROP self-assessment process, including the Appendices to this Chapter.

IMC 0040, “Preparing, Revising and Issuing Documents for the NRC Inspection Manual”

IMC 0102, “Oversight and Objectivity of Inspectors and Examiners at Reactor Facilities”

IMC 0305, “Operating Reactor Assessment Program”

IMC 0306, “Planning, Scheduling, Tracking, and Reporting of the Reactor Oversight Process (ROP)”

IMC 0308, “Reactor Oversight Process Basis Document”

IMC 0350, “Oversight of Reactor Facilities in a Shutdown Condition Due to Significant Performance and/or Operational Concerns”

IMC 0608, “Performance Indicator Program”

IMC 0609, “Significance Determination Process”

IMC 0611, “Power Reactor Inspection Reports”

IMC 0801, “Inspection Program Feedback Process”

IMC 1245, “Qualification Program for Reactor Inspectors”

IMC 2515, “Light-Water Reactor Inspection Program Operations Phase”

IMC 2523, “NRC Application of the Reactor Operating Experience Program in NRC Oversight Processes”

IP 95003, “Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs or One Red Input”

MD 3.5, “Attendance at NRC Staff-Sponsored Meetings”

MD 8.13, “Reactor Oversight Process”

MD 8.3, “NRC Incident Investigation Program”

NEI 99-02, “Regulatory Assessment Performance Indicator Guideline”

NRR Office Instruction OVRST-102, “NRR Procedures for Processing Inspection Manual Documents” (nonpublic)

NUREG-1614, “NRC Strategic Plan”

SECY-99-007, “Recommendations for Reactor Oversight Process Improvements,” dated January 8, 1999

SECY-00-0049, “Results of the Revised Reactor Oversight Process Pilot Program,” dated February 24, 2000

SECY-01-0114, “Results of the Initial Implementation of the New Reactor Oversight Process,” dated June 25, 2001

SECY-20-0039, “Revisions to the ROP Self-Assessment Program,” dated April 30, 2020

SRMs for SECY-00-0049 “Results of the Revised Reactor Oversight Process Pilot Program (Part 1 and Part 2)," March 28, 2000, and May 17, 2000

Staff report, “Process Improvement Review of the Significance Determination Process,” dated November 17, 2014

Staff report, “Davis-Besse Reactor Vessel Head Degradation Lessons-Learned Task Force Report,” dated September 30, 2002

# END

Appendices:

Appendix A, Reactor Oversight Process Self-Assessment Metrics and Data Trending

Appendix B, Reactor Oversight Process Self-Assessment Baseline Inspection Program Monitoring and Comprehensive Reviews

Appendix C, Reactor Oversight Process (ROP) Self-Assessment ROP Implementation Audit

Appendix D, Power Reactor Resident Inspector Retention and Recruitment Program Monitoring and Assessment

Attachment:

Attachment 1, Revision History for IMC 0307

Issue Date:  05/03/22	2	0307

Attachment 1: Revision History for IMC 0307

2

Issue Date:  05/03/22	Att1-1	0307