April 5, 2010 

MEMORANDUM TO: Frederick D. Brown, Director 

 Division of Inspection and Regional Support 

 Office of Nuclear Reactor Regulation 

FROM: Rani L. Franovich, Chief /RA/ 

 Program Assessment Branch 

 Division of Inspection and Regional Support 

 Office of Nuclear Reactor Regulation 

SUBJECT: REACTOR OVERSIGHT PROCESS ANNUAL PERFORMANCE 

METRIC REPORT FOR CALENDAR YEAR 2009 

The Reactor Oversight Process (ROP) self-assessment program evaluates the effectiveness of 

the ROP through its success in meeting pre-established goals and intended outcomes. The 

staff evaluates performance metrics to determine the success of the ROP in meeting these 

goals and outcomes. The staff performed the calendar year (CY) 2009 performance metric 

analysis in accordance with Inspection Manual Chapter (IMC) 0307, “Reactor Oversight Process 

Self-Assessment Program.” 

IMC 0307 describes performance metrics associated with each of the four ROP program areas, 

which are the performance indicator (PI) program, inspection program (IP), significance 

determination process (SDP), and assessment (AS) program. The staff designates the 

program-specific metrics as the PI, IP, SDP, and AS metrics, respectively. The staff also 

monitors and analyzes metrics of a more general nature, which are designated as the 

O metrics, to assess the overall performance of the ROP. The staff used the metric analyses as 

input to the annual ROP self-assessment Commission paper. 

The staff relies on information from various sources, including public comments and surveys of 

stakeholders, to evaluate the performance metrics. The staff solicited comments on ROP 

implementation from external stakeholders in a Federal Register notice (74 FR 49043) 

published on September 25, 2009. All five survey respondents were from the nuclear power 

industry and its representatives. The staff did not conduct an internal survey in CY 2009, 

consistent with its biennial frequency defined by IMC 0307. The staff will conduct the next 

survey to internal stakeholders in CY 2010. 

The staff’s CY 2009 metric analyses are enclosed. All but one of the 45 metrics met the 

established criteria in IMC 0307. Metric AS-8, “Perceived Effectiveness of Safety Culture 

Enhancements to ROP,” was not met as a result of negative external stakeholder feedback 

provided in the responses to the ROP survey. The staff is aware of the industry’s concern with 

CONTACT: Audrey Klett, NRR/DIRS/IPAB 

 301-415-0489
F. Brown - 2 - 

the process for identifying substantive cross-cutting issues and will continue to consider industry 

proposals. The staff will directly address the survey respondents’ comments, including 

comments on safety culture, in its consolidated response to external ROP survey comments. 

Enclosure: 

As stated 
F. Brown - 2 - 

the process for identifying substantive cross-cutting issues and will continue to consider industry 

proposals. The staff will directly address the survey respondents’ comments, including 

comments on safety culture, in its consolidated response to external ROP survey comments. 

Enclosure: 

As stated 

DISTRIBUTION: 

RidsNrrDirsIpab RidsNrrOd Nrr_Distribution Ipab 

Nrr_Distribution Irib RidsOgcMailCenter RidsNrrAdro 

RidsAcrsAcnwMailCenter RidsOPAMail RidsRgnIMailCenter 

RidsRgnIIMailCenter RidsRgnIIIMailCenter RidsRgnIVMailCenter 

JMchale FBrown SWeerakkody 

MCheok RFranovich TKobetz 

DLew JClifford DRoberts 

PWilson LWert JMunday 

KKennedy HChristensen SWest 

GShear ABoland KObrien 

DChamberlain AVegel RCaniano 

TPruett RPowell DMerzke 

JLara CPaulk RCostello 

TBarry MAshley UShoop 

ADAMS ACCESSION NUMBER: ML100540037 Concurrence by email* 

OFFICE NRR/DIRS NRR/DIRS NRR/DIRS NRR/DIRS NRR/DIRS NSIR/DSO* NRR/DIRS NRR/DIRS 

NAME AKlett RFrahm UShoop JMcHale TKobetz RCostello RFranovich FBrown 

DATE 03/4 /2010 03/10 /2010 03/11 /2010 03/ 4 /2010 03/ 10/2010 03/31 /2010 04/ 5 /2010 04/5 /2010 

OFFICIAL RECORD COPY
ENCLOSURE 

Calendar Year 2009 Analysis of the 

Reactor Oversight Process Self-Assessment Metrics 
TABLE OF CONTENTS 

I. PERFORMANCE INDICATOR PROGRAM METRICS ............................................... - 1 -

II. INSPECTION PROGRAM METRICS .......................................................................... - 9 -

III. SIGNIFICANCE DETERMINATION PROCESS METRICS ....................................... - 16 -

IV. ASSESSMENT PROGRAM METRICS ..................................................................... - 22 -

V. OVERALL ROP METRICS ....................................................................................... - 30 -
- 1 -

I. PERFORMANCE INDICATOR PROGRAM METRICS 

PI-1 Consistent Results Given Same Guidance 

Definition: Independently verify performance indicators (PIs) using Inspection 

Procedure 71151, API Verification.@ Count all PIs that either (a) result in a 

crossed threshold based on a data correction by the licensee (as noted in the 

resultant inspection report) or (b) have been determined to be discrepant by the 

staff in accordance with Inspection Procedure 71150, ADiscrepant or Unreported 

Performance Indicator Data.@

Criterion: Expect few occurrences, with a stable or declining (i.e., improving) trend. 

Goals Supported: Objective, Predictable 

Analysis: The following chart presents the number of PI discrepancies, as described in the 

metric definition, per quarter. During this assessment period, no PIs crossed a 

threshold based on data correction identified by the U.S. Nuclear Regulatory 

Commission (NRC) staff. The chart data shows an improving trend since 

calendar year (CY) 2006. Because no PI discrepancies existed in CY 2009 with 

an improving trend, the metric criterion is met. 

Performance Indicator Discrepancies

0

1

2

3

4

5

1Q/052Q/053Q/054Q/051Q/062Q/063Q/064Q/061Q/072Q/073Q/074Q/071Q/082Q/083Q/084Q/081Q/092Q/093Q/094Q/09

Calendar Quarter

Number of Occurrences

Metric Criterion Met: Yes 
- 2 -

PI-2 Questions Regarding Interpretation of PI Guidance

Definition: Quarterly, count the number of frequently asked questions (FAQs). 

Criterion: Expect low numbers, with a stable or declining (i.e., improving) trend. 

Goals Supported: Understandable, Risk-Informed, Predictable 

Analysis: The chart below presents the total number of new FAQs introduced during the 

ROP monthly public meetings held during the respective quarter. The number of 

FAQs in CY 2009 is at its lowest value over the past five years with an improving 

trend. Therefore, the metric criterion is met. 

Frequently Asked Questions

0

5

10

15

1Q/052Q/053Q/054Q/051Q/062Q/063Q/064Q/061Q/072Q/073Q/074Q/071Q/082Q/083Q/084Q/081Q/092Q/093Q/094Q/09

Calendar Quarter

Number of FAQs

Metric Criterion Met: Yes 
- 3 -

PI-3 Timely Indication of Declining Plant Performance

Definition: Quarterly, track PIs that cross multiple thresholds (e.g., green to yellow or white 

to red). Evaluate and characterize these results to allow timely indication of 

declining performance. 

Criterion: Expect few occurrences, with a stable or declining (i.e., improving) trend. 

Goals Supported: Risk-Informed, Effective 

Analysis: The chart below presents the number of PIs that crossed multiple thresholds per 

calendar quarter. No PIs crossed multiple thresholds during CY 2009. The chart 

below shows an improving trend since CY 2007. Therefore, the metric criterion 

is met. 

PIs that Cross Multiple Thresholds

0

1

2

3

4

1Q/052Q/053Q/054Q/051Q/062Q/063Q/064Q/061Q/072Q/073Q/074Q/071Q/082Q/083Q/084Q/081Q/092Q/093Q/094Q/09

Calendar Quarter

Number of Occurrences

Metric Criterion Met: Yes 
- 4 -

PI-4 PI Program Provides Insights to Help Ensure Plant Safety and/or Security

Definition: Survey external and internal stakeholders asking whether the PI Program 

provides useful insights, particularly when combined with the inspection program, 

to help ensure plant safety and/or security. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Risk-Informed, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307. 

Analysis: The external ROP survey respondents generally indicated that the PI program in 

conjunction with the inspection program provides useful insights to ensure plant 

safety and security. The responses were positive compared to the previous ROP 

survey to external stakeholders. Therefore, the metric criterion is met. 

Several survey respondents provided comments on and recommended changes 

to the PI program. One respondent stated that the thresholds for the PIs should 

be re-evaluated given the amount of time the ROP has been in place. Several 

respondents asserted that the mitigating system performance index is too 

complex, labor intensive, and difficult to understand. The respondents generally 

supported changes to the PI program that would result in a fundamental 

improvement to the manner in which safety performance is monitored. The staff 

will address these comments in the consolidated response to the ROP external 

survey comments. 

Metric Criterion Met: Yes 
- 5 -

PI-5 Timely PI Data Reporting and Dissemination

Definition: Within five weeks of the end of each calendar quarter, track (count) late PI 

postings on NRC=s external Web site. Also note the number of late submittals 

from licensees that did not meet the 21-day timeliness goal. 

Criterion: Expect few occurrences, with a stable or declining (i.e., improving) trend. 

Goals Supported: Effective, Open, Predictable 

Analysis: There have been no late PI data postings on the NRC's external Web site since 

the inception of the ROP. Therefore, the metric criterion is met. There were 

three late PI data submittals in CY 2009 (two in the second quarter and one in 

the third quarter). Each submittal was only one day late and did significantly 

impact the NRC’s ability to properly process the PI data. 

Metric Criterion Met: Yes 
- 6 -

PI-6 Stakeholders Perceive Appropriate Overlap between the PI Program and 

Inspection Program 

Definition: Survey external and internal stakeholders asking if appropriate overlap exists 

between the PI program and the inspection program. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents generally indicated there is an appropriate overlap 

between the PI and inspection programs to provide a comprehensive indication 

of licensee performance. The staff noted a positive perception compared to 

responses to previous external surveys. Therefore, the metric criterion is met. 

Several respondents indicated that the overlap can be extensive, particularly in 

the initiating events and mitigating systems cornerstones. Some respondents 

stated that the PI program implementation at times has unnecessarily been 

linked with performance deficiencies. These respondents commented that the PI 

program guidance has occasionally been misapplied to licensee performance 

deficiencies concerning individual events and mitigating systems performance 

index failures. One respondent asserted that a significant amount of effort is 

expended on evaluating events and issues reported under the PI program during 

problem identification and resolution and component design basis inspections. 

The staff will address these comments in the consolidated response to the ROP 

external survey comments. 

Metric Criterion Met: Yes 
- 7 -

PI-7 Clarity of Performance Indicator Guidance

Definition: Survey external and internal stakeholders asking if Nuclear Energy Institute 

(NEI) 99-02, ARegulatory Assessment Performance Indicator Guideline,@ provides 

clear guidance regarding PIs. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Understandable, Open, Objective 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The respondents generally agreed that NEI 99-02 provides clear guidance 

regarding the PI program. The majority of the respondents commented that the 

FAQ process has proven to be effective and should be maintained. The staff 

noted a stable trend in perception compared to comments to previous surveys. 

Therefore, the metric criterion is met. 

One respondent recommended accounting for unique plant designs when 

considering PIs to ensure that the specific plant’s mitigating capabilities are 

accurately credited. Another respondent suggested integrating risk insights into 

the initiating events indicators. Several respondents commented that any future 

modifications to the guidance should not add complexity. The staff will address 

these comments in the consolidated response to the ROP external survey 

comments. 

Metric Criterion Met: Yes 
- 8 -

PI-8 PI Program Contributes to the Identification of Performance Outliers in an 

Objective and Predictable Manner

Definition: Survey external and internal stakeholders asking if the PI program effectively 

contributes to the identification of performance outliers based on risk-informed, 

objective, and predictable indicators. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Risk-Informed, Objective, Predictable, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The respondents generally indicated that the PI program effectively contributes to 

the identification of performance outliers based on risk-informed, objective, and 

predictable measures. The staff noted a stable trend in perception compared to 

comments from previous surveys. Therefore, the metric criterion is met. 

The majority of the respondents commented that an increasing number of green 

PIs should be construed as an overall improvement in industry performance. 

One respondent noted that some PIs are more predictable and indicate 

performance trends compared to other PIs and there is a disparity in the degree 

to which each PI is risk-informed. One respondent also stated that the safety 

system functional failure indicator is not risk-informed. The staff will address 

these comments in the consolidated response to the ROP external survey 

comments. 

Metric Criterion Met: Yes 
- 9 -

II. INSPECTION PROGRAM METRICS 

IP-1 Inspection Findings Documented in Accordance with Requirements

Definition: Audit inspection reports in relation to program requirements (IMC 0612, APower 

Reactor Inspection Reports@) for documenting findings and violations. Report the 

percentage of findings that meet the program requirements. 

Criterion: Expect a stable or improving trend in the percentage of findings documented in 

accordance with program requirements. 

Goals Supported: Objective, Risk-Informed, Predictable 

Analysis: The chart below presents the percentage of audited inspection findings that were 

documented in accordance with IMC 0612 requirements. In CY 2009, the staff 

audited 41 non-security related inspection reports issued by the regional offices. 

The staff found that 98% of sampled findings were documented in accordance 

with IMC 0612 requirements. The data confirm that a stable trend has been 

maintained since CY 2005. Therefore, the metric criterion is met. 

Findings Documented in accordance with Requirements

70

75

80

85

90

95

100

2005 2006 2007 2008 2009

Calendar Year

% Findings that Meet All Requirements

Metric Criterion Met: Yes 
- 10 -

IP-2 Completion of Baseline Inspection Program

Definition: Annual completion of baseline inspection program. 

Criterion: Defined in IMC 2515, ALight-Water Reactor Inspection Program - Operations 

Phase.@

Goals Supported: Predictable, Effective 

Analysis: The inspection program independently verified that licensees operated plants 

safely and securely in CY 2009 and identified and corrected performance issues 

in a timely manner in accordance with IMC 2515, “Light-Water Reactor 

Inspection Program—Operations Phase,” and IMC 2201, “Security and 

Safeguards Inspection Program for Commercial Nuclear Power Reactors.” Each 

region documented its CY 2009 completion of the baseline inspection program in 

a memorandum available in the Agencywide Documents Access and 

Management System (ADAMS) at Accession Nos. ML100390084 for Region I, 

ML100550802 for Region II, ML100560313 for Region III, and ML100601032 for 

Region IV. Additionally, the Office of Nuclear Security and Incident Response 

(NSIR) completed all security baseline inspections in CY 2009, as documented in 

a non-publicly available memorandum (ADAMS Accession No. ML100640428). 

All regions completed their baseline inspections in CY 2009 within the allocated 

resources. 

Metric Criterion Met: Yes 
- 11 -

IP-3 Inspection Reports are Timely

Definition: Obtain Reactor Program System data on the total number of reports issued and 

the number of reports issued within the timeliness goals stipulated in IMC 0612, 

APower Reactor Inspection Reports.@

Criterion: Expect 90% of inspection reports to be issued within the program's timeliness 

goals. 

 Note: For inspections not conducted by a resident inspector, inspection 

completion is normally defined as the day of the exit meeting. For integrated 

inspection reports, inspection completion is normally defined as the last day 

covered by the inspection report. 

Goals Supported: Effective, Open, Predictable 

Analysis: The chart below presents the percentage of inspection reports that were issued 

on time. During CY 2009, the NRC issued 621 inspection reports. The regions 

met or exceeded the inspection report timeliness goal of 90% in each quarter 

throughout the year. In CY 2009, 619 out of 621 (99.7%) inspection reports met 

the timeliness requirements in IMC 0612. Therefore, the metric criterion is met. 

Inspection Report Timeliness

90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

Q1/05Q2/05Q3/05Q4/05Q1/06Q2/06Q3/06Q4/06Q1/07Q2/07Q3/07Q4/07Q1/08Q2/08Q3/08Q4/08Q1/09Q2/09Q3/09Q4/09

Calendar Quarter

Percentage of Total Reports

Metric Criterion Met: Yes 
- 12 -

IP-4 Temporary Instructions (TIs) are Completed Timely

Definition: Audit the time to complete TIs by region or office. Compare the completion 

status in RPS to TI requirements. Report by region or office the number of TIs 

closed within goals. 

Criterion: Expect all TIs to be completed within TI requirements. 

Goals Supported: Effective, Predictable 

Analysis: In CY 2009, the staff completed TI 2515/150, “RPV Head and VHP Nozzle,” and 

TI 2515/174, “Hydrogen Igniter Back-up Power.” The staff completed these TIs 

within established deadlines; therefore, the metric criterion is met. 

Metric Criterion Met: Yes 
- 13 -

IP-5 Inspection Reports are Relevant, Useful, and Written in Plain Language

Definition: Survey external and internal stakeholders asking whether the information 

contained in inspection reports is relevant, useful, and written in plain English. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Understandable, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: All five external stakeholder responses contained favorable comments regarding 

this metric. The respondents commented that the reports are useful and clearly 

written. The staff noted a stable trend compared to the previous external ROP 

survey. Therefore, the metric criterion is met. 

One respondent commented that the NRC staff should consider the licensee’s 

views and positions when revising or refining an inspection finding after the initial 

inspection exit meeting with the licensee. The staff will address this comment in 

the consolidated response to the ROP external survey comments. 

Metric Criterion Met: Yes 
- 14 -

IP-6 Inspection Program Effectiveness and Adequacy in Covering Areas 

Important to Plant Safety and/or Security

Definition: Survey external and internal stakeholders asking whether the inspection program 

adequately covers areas that are important to plant safety and/or security and is 

effective in identifying and ensuring the prompt correction of performance 

deficiencies. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Risk-Informed, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The responses to the external ROP survey were generally positive. The 

respondents commented that the inspection program was generally effective in 

ensuring areas important to safety are appropriately addressed. The staff noted 

a stable trend compared to the previous ROP external survey. Therefore, the 

metric criterion is met. 

Some respondents commented on the need to improve the component design 

bases inspection to ensure that the same systems are not re-inspected. 

Respondents also suggested improving the resident inspector’s timely review of 

safety system functional failures and the types of mitigating systems performance 

index failures before the licensee submits the information to the NRC. One 

respondent also expressed a concern that the inspection threshold used for 

identifying performance deficiencies is low and subjective. The staff will address 

these comments in the consolidated response to the ROP external survey 

comments. 

Metric Criterion Met: Yes 
- 15 -

IP-7 Analysis of Baseline Inspection Procedures

Definition: Annually, review each baseline inspection procedure to determine its 

effectiveness and contribution to the overall effectiveness of the baseline 

inspection program. The objectives of the review are: (1) to determine if 

changes in scope, frequency, or level of effort are needed based on recent 

experience; (2) to determine if a change to the estimated hours for completion is 

needed; (3) to define or change what constitutes minimum completion of each 

inspectable area, if needed; and (4) to critically evaluate all of the inspectable 

areas together along with the PI program to ensure that the inspectable areas 

are adequately monitored for safety performance. In addition, a more detailed 

review and realignment of inspection resources will be performed at least 

biennially in accordance with IMC 0307, Appendix B, “ROP Realignment 

Process.” The focus of this effort is to adjust existing inspection resources to 

improve the effectiveness of the inspection program in identifying significant 

licensee performance deficiencies. 

Criterion: None; trend only. Summarize and evaluate the individual inspection procedure 

reviews and propose program adjustments as necessary to address noted 

inefficiencies. Provide basis for any meaningful increase or decrease in 

procedure scope, frequency, or level of effort as a result of the review. 

Goals Supported: Effective, Risk-Informed 

Analysis: An NRC working group completed its third biennial ROP realignment review 

during CY 2009. This review assesses the effectiveness of each ROP baseline 

inspection procedure by determining whether appropriate inspection resources 

were applied in each of the inspectable areas. The working group consisted of 

staff from the Office of Nuclear Reactor Regulation (NRR), NSIR, and each of the 

four regions. The staff made modifications and adjustments to the inspection 

effort across the baseline inspection program, but overall inspection resources 

for CY 2010 remain at CY 2009 levels. The CY 2009 ROP realignment also 

added new inspection activities and resources to verify compliance with the new 

requirements of Title 10 of the Code of Federal Regulations, Part 26 

(10 CFR 26), “Fitness-for-Duty Programs,” 10 CFR 73.55, “Requirements for 

physical protection of licensed activities in nuclear power reactors against 

radiological sabotage,” 10 CFR 50.54(hh), “Conditions of Licenses,” and lessons 

learned from Peach Bottom regarding inattentive security officers. Additionally, 

the staff adjusted some procedures in the reactor safety area to better align 

budgeted and expended inspection resources. The staff revised all radiation 

safety inspection procedures to provide a more performance-based inspection for 

each of the functional areas of a radiation safety program. It also made 

inspection resource adjustments to all security-related procedures, based on 

regional feedback and past inspection resources expended for each procedure. 

Additional details on the results of the CY 2009 ROP realignment process can be 

found at ADAMS Accession No. ML092090312. 

Metric Criterion Met: Yes 
- 16 -

III. SIGNIFICANCE DETERMINATION PROCESS METRICS 

SDP-1 Significance Determination Process (SDP) Results are Predictable, 

Repeatable, and Focus Stakeholder Attention on Significant Safety Issues

Definition: Annually, audit a representative sample (up to four per region) of inspection 

findings against the standard criteria set forth in IMC 0609, ASignificance 

Determination Process,@ and its appendices. To the extent available, samples 

should include potentially greater-than-green findings that were presented to an 

SDP/enforcement review panel. Findings should contain adequate detail to 

enable an independent auditor to trace through the available documentation and 

reach the same significance color characterization. 

Criteria: At least 90% of greater-than-green significance determinations are determined to 

be predictable and repeatable. Any SDP outcomes determined to be nonconservative will be evaluated, and appropriate programmatic changes will be 

implemented. 

Goals Supported: Risk-Informed, Predictable 

Analysis: Eleven findings had greater-than-green significance in CY 2009. The staff 

audited two findings from each region for a representative sample of eight 

findings having greater-than-green significance. The final risk significance of 

each finding was evaluated using the risk-informed process detailed in IMC 0609, 

Appendix A, “Determining the Significance of Reactor Inspection Findings for AtPower Situations.” The documentation of the final risk characterization of each 

finding included adequate detail to support the final risk significance 

determination; therefore, the final risk significance of each finding was 

predictable and repeatable. The staff determined that 100% of samples chosen 

for review were predictable and repeatable since CY 2005. Therefore, the metric 

criteria are met. 

Metric Criteria Met: Yes 
- 17 -

SDP-2 SDP Outcomes are Risk-Informed and Accepted by Stakeholders

Definition: Track the total number of appeals of final SDP results. 

Criteria: Expect zero appeals of SDP significance that result in a final determination being 

overturned across all regions. All successful appeals will be assessed to 

determine causal factors and to recommend process improvements. 

Goals Supported: Risk-Informed, Objective, Predictable 

Analysis: The staff received only one appeal letter, which was rejected because it failed to 

meet the criteria for invoking the appeal process. The metric criteria are met 

because there were no successful appeals of significance determinations. 

Metric Criteria Met: Yes 
- 18 -

SDP-3 Inspection Staff is Proficient and Finds Value in Using the SDP

Definition: Survey internal stakeholders using specific quantitative survey questions that 

focus on training, effectiveness, and efficiency. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Understandable, Risk-Informed 

Analysis: The staff did not conduct an internal survey in CY 2009, consistent with its 

biennial frequency defined by IMC 0307. However, this metric was met in CY 2008 based on 

the results of the CY 2008 internal ROP survey, which are documented in the CY 2008 ROP 

metric report (ADAMS Accession No. ML090550522). 

Metric Criterion Met: Yes 
- 19 -

SDP-4 The SDP Results in an Appropriate Regulatory Response to Performance 

Issues

Definition: Survey external and internal stakeholders asking if the SDP results in an 

appropriate regulatory response to performance issues. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Understandable, Objective, Predictable, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The CY 2009 external survey asked stakeholders if they thought SDP results 

fostered an appropriate regulatory response to performance issues. A majority of 

respondents indicated that the SDP for the most part resulted in the appropriate 

regulatory response, which is an improved response compared to the previous 

external ROP survey responses. Therefore, the metric criterion is met. 

 The survey respondents encouraged continued dialogue between NRC senior 

reactor analyst and licensee probabilistic risk assessment (PRA) staff. 

Respondents also provided observations and suggestions related to improving 

SDP consistency, clarifying NRC assumptions, using licensees’ PRA models, 

and improving SDP tools in the areas of fire protection, common cause failures, 

and human error probability. The staff will address these comments in the 

consolidated response to the ROP external survey comments. 

Metric Criterion Met: Yes 
- 20 -

SDP-5 Resources Expended (Direct Charges and Support Activities) are 

Appropriate

Definition: Track the percentage of total resource expenditures attributed to SDP activities 

to determine the effort expended by the regions in completing SDP evaluations 

as a percentage of the total regional direct inspection effort (DIE). 

Criterion: Total SDP expenditures should not exceed 10% of the total regional DIE with a 

stable or declining (i.e., improving) trend. 

Goals Supported: Effective, Predictable 

Analysis: The chart below presents the percentage of SDP resource expenditures to total 

DIE per region. Regional expenditures associated with SDP evaluations remain 

below the target goal of 10% of the total DIE. The national average also has 

remained stable over the past five years. Therefore, the metric criterion is met. 

SDP Resource Expenditures

0.0

2.0

4.0

6.0

8.0

10.0

12.0

1Q/052Q/053Q/054Q/051Q/062Q/063Q/064Q/061Q/072Q/073Q/074Q/071Q/082Q/083Q/084Q/081Q/092Q/093Q/094Q/09

Calendar Quarter

Percentage of DIE for SDP Efforts

R1 R2 R3 R4 National

Metric Criterion Met: Yes 
- 21 -

SDP-6 Final Significance Determinations are Timely

Definition: Conduct a quarterly audit of Reactor Program System data to identify the total 

number of inspection items finalized as having greater-than-green significance 

that were reviewed for more than 90 days since: 

(1) the date of initial licensee notification of the preliminary significance in an 

inspection report, or 

(2) the item was documented in an inspection report as an apparent violation 

pending completion of a significance determination and not counted in the 

above category. 

Criteria: At least 90% of all SDP results that are counted in accordance with the definition 

above should be finalized within 90 days. All issues greater than 90 days will be 

assessed to determine causal factors and to recommend process improvements. 

Goals Supported: Effective, Open, Predictable 

Analysis: The chart below presents the percentage of SDP results that were completed 

within 90 days. The completion of final significance determinations has been 

consistently timely for the past four years. Because more than 90% of all SDP 

results were finalized within 90 days in CY 2009, the metric criteria are met. 

SDP Timeliness

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Q1/2005Q2/2005Q3/2005Q4/2005Q1/2006Q2/2006Q3/2006Q4/2006Q1/2007Q2/2007Q3/2007Q4/2007Q1/2008Q2/2008Q3/2008Q4/2008Q1/2009Q2/2009Q3/2009Q4/2009

Calendar Quarter

Percentage Completed in 90 Days

Metric Criteria Met: Yes 
- 22 -

IV. ASSESSMENT PROGRAM METRICS 

AS-1 Actions are Determined by Quantifiable Assessment Inputs (i.e., PIs and 

SDP Results) and are Commensurate with the Risk of the Issue and Overall 

Plant Risk

Definition: Audit all assessment-related letters and count the number of deviations from the 

Action Matrix. Evaluate the causes for these deviations, and identify changes to 

the ROP, if any, to improve the guidance documents. 

Criterion: Expect few deviations, with a stable or declining (i.e., improving) trend. 

Goals Supported: Objective, Risk-Informed, Open 

Analysis: There have been a total of 16 deviations from the Action Matrix since the 

beginning of the ROP in CY 2000. No new deviations occurred in CY 2009. The 

only active deviation in CY 2009 involved the Indian Point Energy Center, for 

which the Executive Director for Operations approved heightened NRC oversight 

via a memorandum dated December 16, 2008. The staff subsequently 

determined that the objectives listed in the deviation memorandum were satisfied 

and therefore closed the deviation, as documented in Inspection Report 

05000247(286)/2009008. Because no new deviations were issued in CY 2009 

and the one existing deviation was closed, the metric criterion is met. 

Metric Criterion Met: Yes 
- 23 -

AS-2 Number and Scope of Additional Actions Recommended as a Result of the 

Agency Action Review Meeting Beyond Those Actions Already Taken are 

Limited

Definition: Review the results of the Agency Action Review Meeting (AARM). 

Criterion: Expect few additional actions, with a stable or declining (i.e., improving) trend. 

Goals Supported: Understandable, Predictable, Objective 

Analysis: The AARM was held on April 22, 2009, in Bethesda, Maryland. The Palo Verde 

Nuclear Generating Station was the only reactor facility that met the criteria for 

discussion at this AARM. NRC senior managers reviewed the agency’s 

completed and planned actions, confirmed the appropriateness and effectiveness 

of the actions taken, and determined that current regulatory tools are sufficient to 

address the issues. The staff determined that no actions beyond those already 

planned for this facility were necessary as a result of the AARM discussions. 

Therefore, the metric criterion is met. 

The next AARM is scheduled for April 27, 2010. 

Metric Criterion Met: Yes
- 24 -

AS-3 Assessment Program Results (Assessment Reviews, Assessment Letters 

and Public Meetings) are Completed in a Timely Manner

Definition: Track the number of instances in which the timeliness goals stipulated in 

IMC 0305, AOperating Reactor Assessment Program,@ were not met for: (1) the 

conduct of quarterly, mid-cycle, and end-of-cycle reviews; (2) the issuance of 

assessment letters; and (3) the conduct of public meetings. 

Criterion: Expect few instances in which timeliness goals were not met, with a stable or 

declining (i.e., improving) trend. 

Goals Supported: Effective, Open, Predictable 

Analysis: Timeliness goals for the following activities are: (1) quarterly reviews - within five 

weeks of the end of quarter; (2) mid-cycle reviews - within seven weeks of the 

end of the 2nd quarter; (3) end-of-cycle reviews - within seven weeks of the last 

quarter; (4) issuance of assessment letters - within two weeks of the quarterly 

review, within nine weeks of the mid-cycle review, and within nine weeks of the 

end-of-cycle review; and (5) public meetings - within 16 weeks of the end of the 

assessment period. 

 The chart below presents the number of untimely actions per calendar quarter. 

All timeliness goals were met in CY 2009 with an improving trend compared to 

CY 2008; therefore, the metric criterion is met. 

Timeliness of Assessment Results

0

1

2

3

Q1/2005Q2/2005Q3/2005Q4/2005Q1/2006Q2/2006Q3/2006Q4/2006Q1/2007Q2/2007Q3/2007Q4/2007Q1/2008Q2/2008Q3/2008Q4/2008Q1/2009Q2/2009Q3/2009Q4/2009

Calendar Quarter

Number of Untimely Action

s

Metric Criterion Met: Yes 
- 25 -

AS-4 NRC's Response to Performance Issues is Timely

Definition: Count the number of days between issuance of an assessment letter discussing 

an issue having more than very low safety significance and completion of the 

supplemental inspection (by exit meeting date, not issuance of the inspection 

report). 

Criterion: Expect a stable or declining (i.e., improving) trend. 

Goals Supported: Effective, Predictable 

Analysis: The chart below presents the average number of days between the issuance of 

the assessment letter and the completion date of the supplemental inspection for 

greater-than-green findings. Data collected in CY 2009 indicate a slightly 

improving trend since CY 2007 regarding the elapsed time between the issuance 

of an assessment letter and the completion of the corresponding supplemental 

inspection. The yearly average improved by 18 days from CY 2008; therefore, 

the metric criterion is met. 

Timeliness of NRC Response to Performance Issues

0

30

60

90

120

150

180

210

1Q/052Q/053Q/054Q/051Q/062Q/063Q/064Q/061Q/072Q/073Q/074Q/071Q/082Q/083Q/084Q/081Q/092Q/093Q/094Q/09

Calendar Quarter

Average Number of Elapsed Days

Metric Criterion Met: Yes 
- 26 -

AS-5 NRC Takes Appropriate Actions to Address Performance Issues 

Definition: Survey external and internal stakeholders asking whether the NRC takes 

appropriate actions to address performance issues for those plants outside the 

Licensee Response Column of the Action Matrix. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Understandable, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the actions taken by the NRC to address 

performance issues at plants outside the Licensee Response Column of the 

Action Matrix have been appropriate and more predictable. Two respondents 

expressed concern with staff comments occasionally made at public meetings 

that deviations from the Action Matrix are permitted on a case-by-case basis and 

should be considered an option. However, there were no new deviations in 

CY 2009, and the staff continues to recognize that the use of deviations is 

intended for rare instances in which the regulatory actions dictated by the Action 

Matrix may not be appropriate, consistent with IMC 0305. The staff will address 

these comments in the consolidated response to the ROP external survey 

comments. Because of the positive response from survey respondents and a 

stable positive perception over time, the metric criterion is met. 

Metric Criterion Met: Yes 
- 27 -

AS-6 Assessment Reports are Relevant, Useful, and Written in Plain Language

Definition: Survey external and internal stakeholders asking whether the information 

contained in assessment reports is relevant, useful, and written in plain English. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Understandable, Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the information contained in assessment 

letters is, for the most part, relevant, useful, and well written. They also indicated 

that receiving inspection schedules as a part of the assessment letters is a useful 

practice. 

Respondents expressed concern with the discussion of substantive cross-cutting 

issues (SCCIs) in assessment letters. Two respondents requested greater 

consistency in the language and the detailed discussion used across regions. 

The respondents also stated that the criteria for opening and closing SCCIs are 

not clearly defined. The staff will address these comments in the consolidated 

response to the ROP external survey comments. 

Prior to receiving these comments, the staff committed, in the CY 2008 ROP selfassessment, to explore ways to use cross-regional experience to further improve 

the implementation of guidance on SCCIs. In response to this commitment, the 

staff leveraged ongoing efforts initiated by the Deputy Regional Administrators to 

improve the reliability of ROP implementation, including the SCCI process. The 

regions are continuing to implement the reliability initiatives, with NRR support. 

Because of the positive response from survey respondents and efforts previously 

underway to improve regional reliability and consistency, the metric criterion is 

met with a stable positive perception over time. 

Metric Criterion Met: Yes 
- 28 -

AS-7 Degradations in Plant Performance are Gradual and Allow Adequate 

Agency Engagement of the Licensees

Definition: Track the number of instances each quarter in which plants move more than one 

column to the right in the Action Matrix (as indicated on the Action Matrix 

Summary). 

Criteria: Expect few instances in which plant performance causes a plant to move more 

than one column to the right in the Action Matrix. Provide a qualitative 

explanation of each instance in which this occurs. Expect a stable or declining 

(i.e., improving) trend. 

Goals Supported: Risk-Informed, Predictable 

Analysis: There were no instances in CY 2009 in which plants moved more than one 

column to the right in the Action Matrix. Because there were also no instances in 

CY 2008, the trend is stable. Therefore, the metric criteria are met. 

Metric Criteria Met: Yes 
- 29 -

AS-8 Perceived Effectiveness of Safety Culture Enhancements to ROP 

Definition: Survey external and internal stakeholders asking whether the ROP safety culture 

enhancements help in identifying licensee safety culture weaknesses and 

focusing licensee and NRC attention appropriately. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the ROP safety culture enhancements are 

not effective at identifying weakness or appropriately focusing licensee and NRC 

attention. The respondents generally commented that the ROP safety culture 

enhancements are broad, vague, and have thresholds that lack appropriate 

bases. Several respondents encouraged the use of a new industry-owned safety 

culture oversight process developed by the NEI and proposed as an alternative 

to the process for identifying substantive cross-cutting issues (SCCIs). The staff 

is aware of the industry’s concerns with the SCCI process and is observing 

aspects of the NEI proposal at the request of the NEI. The staff will continue to 

consider industry initiatives in this area and opportunities to leverage 

demonstrated results from those initiatives to gain efficiencies in the ROP. The 

staff also recognizes that there was a significant decrease in the number and 

cross-section of external survey respondents and notes that it would be prudent 

to obtain a broader perspective from other stakeholders before drawing specific 

conclusions on the ROP safety culture enhancements. 

 Because of the nature of these responses, the metric criterion is not met. The 

staff will address these comments in the consolidated response to the ROP 

external survey comments. 

Metric Criterion Met: No 
- 30 -

V. OVERALL ROP METRICS 

O-1 Stakeholders Perceive the ROP to be Predictable and Objective

Definition: Survey external and internal stakeholders asking if ROP oversight activities are 

predictable (i.e., controlled by the process) and reasonably objective (i.e., based 

on supported facts, rather than relying on subjective judgment). 

Criterion: Expect a stable or increasing positive perception over time. 

Goals Supported: Objective, Predictable, Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: Survey respondents indicated that the ROP is predictable and reasonably 

objective, but some opportunities exist for further improvement. Some 

respondents indicated that the SDP (particularly for fire protection issues), the 

NRC’s approach to safety culture, and the definition of availability in the PI 

program are unpredictable, subjective, and need improvement. The staff will 

address these comments in the consolidated response to the ROP external 

survey comments. Stakeholder feedback was similar to previous years. As a 

result of the positive perception, the metric criterion is met. 

Metric Criterion Met: Yes 
- 31 -

O-2 Stakeholders Perceive the ROP to be Risk-informed 

Definition: Survey external and internal stakeholders asking if the ROP is risk-informed, in 

that actions and outcomes are appropriately graduated on the basis of increased 

significance. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Risk-Informed, Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the ROP provides appropriate graduation 

of actions and outcomes on the basis of increased risk. Some respondents 

expressed concerns with the integration of the NRC’s safety culture assessments 

and traditional enforcement with the ROP as potentially eroding the risk-informed 

nature of the ROP. The staff will address these comments in the consolidated 

response to the ROP external survey comments. The metric criterion is met with 

a stable positive perception over time. 

Metric Criterion Met: Yes 
- 32 -

O-3 Stakeholders Perceive the ROP to be Understandable 

Definition: Survey external and internal stakeholders asking if the ROP is understandable 

and if the processes, procedures, and products are clear and written in plain 

English. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Understandable, Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that ROP procedures and products are 

generally clear and understandable. Respondents noted prior improvements 

made to the assessment guidance regarding the definition of Multiple/Repetitive 

Cornerstone Column and prevention of double-counting an inspection finding and 

PI with the same underlying cause. Some respondents indicated that the 

process for characterizing performance deficiencies could be clarified to ensure 

consistent implementation across the regions. The staff will address these 

comments in the consolidated response to the ROP external survey comments. 

The metric criterion is met with a stable positive perception over time. 

Metric Criterion Met: Yes 
- 33 -

O-4 Stakeholders Perceive that the ROP Provides Adequate Regulatory 

Assurance that Plants are Operated and Maintained Safely and Securely

Definition: Survey external and internal stakeholders asking if the ROP provides adequate 

regulatory assurance, when combined with other NRC regulatory processes, that 

plants are being operated and maintained safely and securely. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: Survey respondents indicated that the ROP, when combined with other 

regulatory processes, ensures plants are operated and maintained safely and 

securely. Some respondents noted that ROP implementation has contributed to 

plant performance improvements across the industry. The staff will address 

these comments in the consolidated response to the ROP external survey 

comments. The metric criterion is met with a stable positive perception over 

time. 

Metric Criterion Met: Yes
- 34 -

O-5 Stakeholders Perceive the ROP to be Effective (e.g., High Quality, Efficient, 

Realistic, and Timely) 

Definition: Survey external and internal stakeholders asking whether NRC actions related to 

the ROP are high quality, efficient, realistic, and timely. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents generally indicated that the ROP is effective, efficient, 

realistic, and timely. Some respondents noted that the NRC should further 

improve the timeliness and objectivity of the SDP and that the extensive 

resources being expended to further improve the mitigating systems performance 

index are not warranted. One respondent noted that realism is lacking in the 

new, inexperienced residents being sent to the field. The staff will address these 

comments in the consolidated response to the ROP external survey comments. 

The metric criterion is met with a stable and generally positive perception over 

time. 

Metric Criterion Met: Yes 
- 35 -

O-6 Stakeholders Perceive that the ROP Ensures Openness

Definition: Survey external and internal stakeholders asking if the ROP ensures openness in 

the regulatory process. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Open, Effective 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents noted that the ROP is generally a very open process. 

Some respondents noted that the security oversight process and the review of 

draft documents should be more open and allow for greater stakeholder input. 

The staff will address these comments in the consolidated response to the ROP 

external survey comments. The metric criterion is met with a stable positive 

perception over time. 

Metric Criterion Met: Yes 
- 36 -

O-7 Opportunities for Public Participation in the Process

Definition: Survey external and internal stakeholders asking if there are sufficient 

opportunities for the public to participate in the process. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Open, Effective 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents acknowledged the ample opportunities for public 

participation, such as the ROP monthly public meetings at NRC headquarters, 

annual public meetings conducted in the reactor communities, annual 

assessment meetings at each site, and ROP surveys. Some respondents noted 

that the oversight of security is not very open to the public, which is appropriate 

in most cases. The staff will address these comments in the consolidated 

response to the ROP external survey comments. The metric criterion is met with 

a stable positive perception over time. 

Metric Criterion Met: Yes 
- 37 -

O-8 Stakeholders Perceive the NRC to be Responsive to its Inputs and 

Comments

Definition: Survey external and internal stakeholders asking if the NRC is responsive to the 

public's inputs and comments on the ROP. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Open, Effective 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the NRC is responsive to inputs and 

comments and noted the published response to the CY 2007 ROP survey as a 

positive example. Respondents encouraged continued responses for future 

surveys. However, some respondents noted a particular example where the 

NRC response to the CY 2007 survey was disappointing. The response involved 

the interpretation of performance indicator guidance for a plant that had been 

shut down for years and was returning to service. The staff will address these 

comments in the consolidated response to the ROP external survey comments. 

Overall, the respondents’ feedback was generally favorable. The NRC will 

continue to publish its consolidated response to the external surveys and 

encourage public input and feedback. The metric criterion is met with a stable 

positive perception over time. 

Metric Criterion Met: Yes 
- 38 -

O-9 Stakeholders Perceive that the ROP is Implemented as Defined 

Definition: Survey external and internal stakeholders asking if the ROP has been 

implemented as defined by program documents. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Predictable, Understandable, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents generally indicated that the ROP is being implemented 

as defined by program documents. However, some respondents noted concerns 

with inconsistencies in the number of findings, violations, and cross-cutting 

aspects issued across the four regions. Prior to receiving these comments, the 

regions had commenced initiatives to improve the reliability of ROP 

implementation as discussed in Enclosure 1 to the ROP self-assessment SECY 

paper. The staff will address these comments in the consolidated response to 

the ROP external survey comments. The metric criterion is met with mostly 

positive comments and a stable positive perception over time. 

Metric Criterion Met: Yes 
- 39 -

O-10 Stakeholders Perceive that the ROP Does Not Result in Unintended 

Consequences 

Definition: Survey external and internal stakeholders asking if the ROP results in unintended 

consequences. 

Criterion: Expect stable or increasingly positive perception over time. 

Goals Supported: Effective, Open 

Note: The five responses to the CY 2009 external ROP survey were all from nuclear 

power industry stakeholders. Therefore, the metric analyses do not necessarily 

reflect the views of other stakeholders, including state and local agencies, public 

interest groups, and members of the public. The staff did not conduct an internal 

survey in CY 2009, consistent with its biennial frequency defined by IMC 0307.

Analysis: The survey respondents indicated that the ROP generally does not result in 

unintended consequences. One respondent provided several areas for 

improvement that should be discussed in detail at upcoming ROP monthly public 

meetings. Examples included the way corrections are made to the safety system 

functional failure PI, the amount of attention that licensees apply to incomplete 

data regarding safety culture, and diverted resources from important findings to 

findings with minimal risk significance. The staff will address these comments in 

the consolidated response to the ROP external survey comments. The metric 

criterion is met with a stable positive perception over time. 

Metric Criterion Met: Yes 
- 40 -

O-11 Analysis of NRC=s Responses to Significant Events 

Definition: Review reports from incident investigation teams (IITs) and augmented 

inspection teams (AITs) to collect lessons learned regarding ROP programmatic 

deficiencies (i.e., did the baseline inspection program inspect this area? Did the 

SDP accurately characterize resultant findings?). IITs already have the provision 

to determine NRC program deficiencies. AITs will be reviewed by NRR/DIRS to 

identify any weaknesses. 

Criterion: Expect no major programmatic voids. 

Goals Supported: Effective, Predictable 

Analysis: No AIT or IIT inspections were performed in CY 2009; therefore, no 

programmatic voids occurred, and the metric criterion was met. 

Metric Criterion Met: Yes 
- 41 -

O-12 Analysis of Inspection Hours and Resource Expenditures

Definition: Annually, collect and analyze resource data (e.g., direct inspection effort, 

preparation, documentation, and plant status hours) for baseline, supplemental, 

plant-specific, and safety issues inspections, and other ROP activities. 

Criteria: (1) Significant deviations are not expected on an annual basis. Explore reasons 

for any deviations that may be evident. (2) Track and trend resource usage for 

baseline, supplemental, and plant-specific inspections. Analyze causes for any 

significant departures from established trends. (3) Track and trend resource 

usage for preparation, documentation, and other ROP activities, and assess the 

effects on budgeted resources. 

 Note: This metric is intended primarily for tracking and trending resource usage 

for the ROP. The results are used to improve the efficiency and effectiveness of 

the ROP and to make management and budget decisions. A detailed ROP 

resource analysis is included in the annual ROP self-assessment Commission 

paper. 

Goals Supported: Effective, Predictable 

Analysis: The staff implements the ROP on a CY basis; however, it obtains and reports 

resource data on a fiscal year (FY) basis. Overall staff effort in FY 2009 

increased by 1.4 percent, compared with FY 2008, but decreased by 3% 

compared with FY 2007. Total ROP effort during the past three years has 

remained relatively stable at approximately 6,300 hours per site and is consistent 

with budgeted resources. 

Baseline inspection hours increased in 2009 primarily as a result of increased 

effort in performing Inspection Procedure 71152, “Identification and Resolution of 

Problems,” and Inspection Procedure 71130.03, “Contingency Response - Forceon-Force Testing.” Although more of these inspections were performed in 

FY 2009 than in FY 2008, the staff will consider this apparent increase in 

inspection hours during the next ROP realignment of inspection resources. The 

hours charged to other baseline procedures remained relatively unchanged. 

Plant-specific (i.e., supplemental, reactive, and infrequently performed) 

inspection effort decreased in FY 2009, compared with FY 2008, because of a 

decrease in event response and supplemental inspections with a corresponding 

decrease in preparation and documentation effort for these inspections. The 

decrease in supplemental inspection hours reflects the decrease in the number 

of plants in the Degraded Cornerstone and Multiple/Repetitive Degraded 

Cornerstone Columns of the ROP Action Matrix in FY 2009, compared with 

FY 2008. 

Generic safety issue (GSI) inspections are typically one-time inspections of 

specific safety and security issues, and GSI inspection effort can vary 

significantly from year to year. The decreased effort in GSI inspections in 

FY 2009 reflects reduced activity in this area in FY 2009 and the completion, in 

December 2008, of inspections related to the verification of site-specific 

implementation of B.5.b Phase 2 and 3 mitigating strategies. 
- 42 -

The effort reported for “other activities,” including inspection-related travel, 

significance determination process, and routine communications (which now 

encompasses regional support, enforcement support, and review of generic 

technical documents), increased slightly in 2009. The effort for these activities 

typically corresponds to the baseline inspection effort but can also be impacted 

by inspection resources used for public outreach for issues that garner high 

public interest. The regional effort for licensee performance assessments has 

remained relatively steady during the past three FYs and suggests that the 

performance assessment effort has reached a steady state. 

Metric Criterion Met: Yes 
- 43 -

O-13 Analysis of Resident Inspector Demographics and Experience 

Definition: Annually, collect and analyze data in order to determine the relevant inspection 

experience of the resident inspector (RI) and senior resident inspector (SRI) 

population. The following four parameters will be measured and analyzed for 

both RIs and SRIs to ensure that the NRC maintains a highly qualified resident 

inspection staff: 

 (1) NRC time - the total time the individual has accumulated as an NRC 

employee. 

 (2) Total resident time - the total time the individual has accumulated as an 

RI or SRI. 

 (3) Current site time - the total time the individual has spent as an RI or SRI 

at the current site. 

 (4) Relevant non-NRC experience - the total time the individual has gained 

relevant nuclear power experience outside of the NRC. Examples of 

relevant non-NRC experience are operation, engineering, maintenance, 

or construction experience with commercial nuclear power plants, naval 

shipyards, U.S Department of Energy facilities, or the U.S. Navy nuclear 

power program. 

Criteria: None; trend only. Provide reasons for any meaningful increase or decrease in 

these resident demographic metrics. 

 Note: This metric is intended primarily for tracking and trending resident 

inspection experience. The results are used to make any necessary 

modifications to the RI and/or SRI programs in order to attract and retain highly 

qualified inspectors to the respective programs. A detailed resident demographic 

and staffing analysis, including graphs, data, and analyses for these resident 

demographic metrics, is included in the annual ROP self-assessment 

Commission paper. 

Goals Supported: Effective, Predictable 

Analysis: RIs’ NRC time (nationally) has steadily increased and relevant non-NRC 

experience has steadily decreased. Both of these trends may have resulted from 

the reduction in turnover rate since CY 2007. Region II has significantly greater 

relevant non-NRC experience than the other regions. SRI experience varies little 

among the regions. However, there is wide variance among regions for all types 

of experience except current site time. Although the RI and SRI turnover rates 

have declined for three consecutive years, the staff continues to closely monitor 

the attraction and retention of RIs and SRIs to ensure an experienced and stable 

RI and SRI program. An NRC senior-level management working group 

developed strategies and initiatives to address these retention issues and 

reported them to the Commission in SECY 09-0050, “Actions to Enhance 

Relocation and Retention for Employees,” dated March 30, 2009. 
- 44 -

The sites continue to be staffed with knowledgeable and experienced RIs and 

SRIs. There is an improving trend in the RI turnover rates, and regional training 

efforts are having a positive impact on the NRC experience level for RIs. In 

addition, feedback from licensees noted that the inspectors performed high 

quality and effective inspections that correctly characterized the licensee’s 

performance. 

Many of the RI program incentives described in SECY 09-0050 have only 

recently been implemented or are in the process of being implemented. 

Therefore, improvements in the RI demographics are expected to continue. 

Notwithstanding, the NRC will continue to monitor RI staffing and retention to 

identify any adverse trends early. The effectiveness of the enhancements to the 

relocation and retention initiatives described in SECY 09-0050 will be discussed 

in a separate paper to the Commission in CY 2011 in accordance with its 

associated SRM dated June 26, 2009. 

Metric Criteria Met: Yes
- 45 -

O-14 Analysis of Site Staffing

Definition: Annually, collect and analyze data in order to measure the permanent inspector 

staffing levels at each of the reactor sites for both RIs and SRIs in order to 

evaluate the agency’s ability to provide continuity of regulatory oversight. 

Criteria: The criterion is set at a staffing level of 90% program-wide. Any single site that 

falls below a staffing level of 90% will be individually evaluated. Provide reasons 

for any meaningful increase or decrease in the inspector staffing level at reactors 

sites. 

Note: Inspectors assigned to the site permanently or through a rotation with a 

minimum duration of six weeks shall be counted. Inspectors on 6-week or longer 

rotational assignments will be identified as such. Inspectors assigned to the site 

for less than six weeks will not be counted but should be indicated as such. 

Additionally, the regions shall indicate sites where permanently assigned RIs or 

SRIs are away from the site for an extended period of time (one continuous time 

period which is greater than six weeks). Only inspectors, who have attained at 

least a basic inspector certification status, as defined by Appendix A to 

IMC 1245, “Qualification Program for Operating Reactor Programs,” shall be 

counted. 

Data will indicate number of days a qualified RI and SRI are permanently 

assigned to the site during the year divided by the number of days in the year. 

Number of days spent on training, meetings away from the site, participation in 

team inspections, leave, or other temporary duties (e.g. acting for branch chiefs 

in his/her absence) will not be counted against the metric unless the absences 

exceed six continuous weeks. 

Goals Supported: Effective, Predictable 

Analysis: The regions succeeded in meeting the site staffing metric of 90% program-wide. 

The average site staffing for all regions was 97.55% in CY 2009. The table 

below presents the number of sites since 2005 that did not meet the 90% site 

staffing goal. As shown in this table, five sites fell below the 90% site staffing 

requirement in CY 2009 because temporary inspectors were at the sites for less 

than six continuous weeks. However, all five sites were staffed above 76%, were 

not recurrences from the previous year, and were supplemented by region-based 

inspectors to assist in completing the baseline inspection program. Meeting this 

metric was challenging and had a significant impact on inspectors and 

management, but the recent relocation and retention enhancements may 

improve future site staffing metric results. 

Number of Sites Below 90% Staffing Level 

2005 2006 2007 2008 2009 

Number of Sites 3 1 9 5 5 

Metric Criteria Met: Yes 
- 46 -

O-15 Analysis of ROP Training and Qualifications

Definition: Annually, evaluate the implementation of IMC 1245, “Qualification Program for 

the Office of Nuclear Reactor Regulation Programs,” particularly as it pertains to 

ROP implementation. 

 

Criteria: None; trend only. Summarize and evaluate the training accomplished over the 

previous year and propose program improvements as necessary to address 

noted concerns. 

 Note: This metric is intended primarily for tracking and trending the effectiveness 

of the ROP training and qualifications programs. A discussion of training 

effectiveness is included in the annual ROP self-assessment Commission paper. 

Goals Supported: Effective, Predictable, Understandable 

Analysis: The staff continued to improve the initial and continuing inspector training 

programs to develop and maintain well-qualified, competent inspectors. The staff 

made recommendations, reviewed them in accordance with the ROP feedback 

process, and incorporated the improvements into inspection standards, as 

appropriate. The staff also developed three new inspector qualification 

standards, one for fire protection inspectors and two advanced-level standards 

for inservice inspection and fire protection inspectors. The staff conducted 

regional training on the integration of traditional enforcement into the assessment 

process, documenting issues in inspection reports, and licensee reporting 

requirements associated with the SSFF PI. In addition, the staff initiated periodic 

knowledge management seminars to improve the NRC’s understanding of the 

concept of safety culture and its aspects. The staff also developed and 

implemented industrial safety training as well as a comprehensive training 

curriculum to support security inspections, including Force-on-Force inspections. 

In the CY 2008 ROP self-assessment, the staff agreed to develop and implement 

additional SDP training to ensure the inspectors remain efficient and effective in 

determining the safety and security significance of identified performance issues. 

Although the staff began to develop additional SDP training, it deferred 

implementation to incorporate input from the partnering initiative, which provided 

valuable insights regarding areas where training was lacking or can be improved. 

These areas include fundamental and overview training for certifying inspectors, 

as well as risk-informed decision-making fundamentals and techniques for 

managers. The staff will resume its efforts to implement SDP training in 

CY 2010. 

Metric Criteria Met: Yes 
- 47 -

O-16 Analysis of Regulatory Impact

Definition: Annually, collect and analyze licensee feedback and develop a summary of 

regulatory impact forms that are critical of the ROP. 

Criteria: None; trend only. Summarize and evaluate the feedback received and propose 

program improvements as necessary to address common concerns. 

 Note: This metric is intended primarily for tracking and trending regulatory 

impact. A detailed regulatory impact summary is included in the annual ROP 

self-assessment Commission paper. 

Goals Supported: Effective, Open, Understandable 

Analysis: During the previous fiscal year, the staff received and compiled feedback from 

95 site visits to 43 reactor sites (68 units) across all four regions. These visits 

resulted in 178 distinct comments that fell into two main categories: formal 

communication with licensees and inspector performance. Of the comments 

compiled, 92% (163/178) were favorable, and 8% (15/178) were unfavorable. 

The number, distribution, and the favorable percentage of comments were similar 

to previous years. 

The staff concludes that communication between the NRC and its licensees is 

effective and that the reported communication problems were isolated instances. 

The staff also concludes that inspectors were professional, maintained effective 

working relationships, and appropriately characterized licensee performance. 

The staff reviewed the negative feedback on inspector performance for trends 

and found that each concern related to an isolated incident or a difference of 

professional opinion. 

Metric Criteria Met: Yes