March 30, 2006

MEMORANDUM TO: Marc L. Dapas, Deputy Regional Administrator, RI

Loren R. Plisco, Deputy Regional Administrator, RII

Geoffrey E. Grant, Deputy Regional Administrator, RIII

Thomas P. Gwynn, Deputy Regional Administrator, RIV

FROM: Stuart A. Richards, Deputy Director /RA/

Division of Inspection and Regional Support

Office of Nuclear Reactor Regulation

SUBJECT: REACTOR OVERSIGHT PROCESS ANNUAL SELF-ASSESSMENT 

PERFORMANCE METRICS

The Reactor Oversight Process (ROP) performance metrics use objective measures and

predetermined criteria to monitor the performance of the ROP as described in Inspection

Manual Chapter (IMC) 0307, “Reactor Oversight Process Self-Assessment Program.” IMC

0307 was recently revised to support the new safety performance measures of the NRC’s

Strategic Plan, to better define the ROP goals and intended outcomes, and to consolidate and

clarify several of the performance metrics. The calendar year (CY) 2005 ROP self-assessment

was performed in accordance with the revised IMC 0307 and associated performance goals. 

These metrics rely on information from various sources, including the Reactor Program System

(RPS), the inspection program, periodic independent audits, stakeholder surveys, and public

comments. The staff collects data quarterly and uses preestablished success criteria to 

analyze the data. In most cases, success is defined as a steady or improving trend. The NRC

solicited comments on the sixth year of ROP implementation from external stakeholders through

a Federal Register notice (FRN) in October 2005. Of the 21 responses, nine were from the

utilities, while four were from State agencies, seven were from public interest groups or public

citizens, and one was anonymous. No internal survey was conducted this year consistent with

the biennial periodicity established in IMC 0307.

Based on our review, most of the metrics met their established criteria. However, three metrics

in the performance indicator (PI) program area, one in the inspection program, two in the

significance determination process (SDP) area, and one in the assessment program failed to

meet expectations. Each of the “overall ROP” metrics met their criteria. The staff’s corrective

actions to address the unsuccessful metrics are discussed in the following paragraphs, in the

enclosed metric analyses, and in the CY 2005 ROP self-assessment Commission paper.

CONTACT: Ron Frahm, NRR/DIRS

301-415-2986
-2-

PI Results

In reviewing the data for this reporting period, the staff found that three of the seven PI metrics

did not meet the established criteria. The recent trend in PI verification issues and discrepant

PIs resulted in the failure of the consistent results given the same guidance metric (PI-1). Two

metrics based on public perception as to whether the PI program provides insights that help

ensure plant safety (PI-4), and whether there is an appropriate overlap between the PI and

inspection programs (PI-6), also failed to meet expectations. NRR continues to evaluate

several PIs, with inputs from internal and external stakeholders, in an effort to improve their

effectiveness at identifying poor performance.

Inspection Results

One of the eleven inspection program metrics did not meet program expectations. Specifically,

the temporary instruction (TI) timeliness metric (IP-5) failed to meet the established criteria of

completing all TIs within the TI requirements. NRR plans to consult with the regions and review

the reasons for the untimely completion of several TIs and recommend possible program

improvements in CY 2006.

SDP Results

Of the eight metrics for the SDP, two did not meet the established criteria. The metric

measuring SDP timeliness (SDP-7) failed to meet staff expectations, and the negative

perception that the SDP does not yield an appropriate and consistent regulatory response

across all ROP cornerstones continued (SDP-5). One of the eight SDP metrics (SDP-3) was

not included in the CY 2005 metric report because there was no internal survey in CY 2005

consistent with its biennial frequency as defined by IMC 0307. NRR continues to address these

and other issues through the SDP Improvement Plan, which is expected to produce continued

improvements in SDP timeliness and consistency.

Assessment Results

The staff found that one of the eleven assessment metrics did not meet the established criteria

in CY 2005. The metric which measured whether subjective judgment is minimized and is not a

central feature of the process and whether actions are determined by quantifiable assessment

inputs (AS-1) did not meet program expectations because there was an increase in Action

Matrix deviations over CY 2004. Based on a review of these deviations, IMC 0305 was revised

to allow the regional offices to continue some actions that are consistent with the

multiple/repetitive degraded cornerstone or degraded cornerstone columns of the Action Matrix,

during the transition out these columns. Further program changes are expected in CY 2006 as

a result of the NRC’s safety culture initiative.

Overall Metric Results

In addition to the specific program area metrics, there are 17 overall ROP metrics of a more

general nature. Each of these overall ROP metrics met the established criteria. Two of these

metrics had failed to meet their criteria during the CY 2004 evaluation, but staff corrective

actions and increased positive perception resulted in them meeting their criteria this year. 
-3-

Specifically, these metrics gauge whether the public perceives the NRC to be responsive to its

inputs and comments (O-8), and whether the public perceives that the ROP results in

unintended consequences (O-10).

Conclusions and Next Steps

The performance metrics provide the staff with valuable insights and lessons learned that lead

to continued improvements in ROP effectiveness. This report provides a significant input into

the annual ROP self-assessment and the resulting Commission paper. Aspects of this report,

particularly missed metrics, will be discussed in the self-assessment paper under the respective

program areas. The ROP self-assessment Commission paper will be used to support the

Agency Action Review Meeting in April 2006 and the subsequent Commission briefing in May

2006. Similar to last year, we also plan to prepare and distribute a consolidated response to

stakeholder comments from the CY 2005 external survey to ensure continued positive

perception that the NRC is responsive to the public's inputs and comments on the ROP.

Enclosure: As stated
-3-

Specifically, these metrics gauge whether the public perceives the NRC to be responsive to its

inputs and comments (O-8), and whether the public perceives that the ROP results in

unintended consequences (O-10).

Conclusions and Next Steps

The performance metrics provide the staff with valuable insights and lessons learned that lead

to continued improvements in ROP effectiveness. This report provides a significant input into

the annual ROP self-assessment and the resulting Commission paper. Aspects of this report,

particularly missed metrics, will be discussed in the self-assessment paper under the respective

program areas. The ROP self-assessment Commission paper will be used to support the

Agency Action Review Meeting in April 2006 and the subsequent Commission briefing in May

2006. Similar to last year, we also plan to prepare and distribute a consolidated response to

stakeholder comments from the CY 2005 external survey to ensure continued positive

perception that the NRC is responsive to the public's inputs and comments on the ROP.

Enclosure: As stated

DISTRIBUTION:

DRP/DRS

Adams accession number: ML060590135

* See Previous Concurrence

OFFICE NRR/DIRS/IPAB NRR/DIRS/IPAB NRR/DIRS/IRIB NRR/DIRS/DD

NAME RFrahm* JAndersen* RGibbs* SRichards

DATE 03/15/2006 03/21/2006 03/27/2006 03/30/2006

OFFICIAL RECORD COPY
-1-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Occurrences

PI-1 Consistent Results Given Same Guidance

Definition: Independently verify PIs using Inspection Procedure (IP) 71151, “PI Verification.” 

Count all PIs that either (a) result in a crossed threshold based on a data

correction by the licensee (as noted in the resultant inspection report), or (b)

have been determined to be discrepant by the staff in accordance with IP 71150,

“Discrepant or Unreported Performance Indicator Data.”

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Safety

Comments: The graph represents the number of significant deficiencies or discrepant PIs

reported for each quarter. Significant discrepancies are issues identified by the

NRC during a PI verification inspection that caused the PI to cross a threshold.

Analysis: During this assessment period, three PIs crossed threshold based on a data

correction by the licensee and one PI was identified as discrepant.

A Peach Bottom 2 Scram with Loss of Normal Heat Removal PI, and the

emergency preparedness Drill/Exercise Performance PI for Diablo Canyon 1 and

2 crossed the white threshold based on data corrections from the licensees. 

The Davis-Besse Alert and Notification System (ANS) Performance Indicator (PI)

data was determined to be discrepant, as discussed in Special Inspection Report

05000346/2004018(DRS), dated January 13, 2005.

The NRC identified a discrepant PI at Waterford 3 in the fourth quarter of 2005

(see Waterford 3 assessment followup letter dated November 4, 2005), however,
-2-

the discrepant PI inspection has not been conducted. Waterford 3 initially

reported the High Pressure Safety Injection System PI as Green. The NRC

subsequently determined that the system may not have been available between

November 2003 and September 2004. Consequently, the PI may be greater

than Green. If the PI is determined to be greater than green, the NRC will

include this issue in the calendar quarter in which the inspection report

documenting the deficiency is issued (using the report cover letter date). 

Note: Verification of the IE, BI, and MS performance indicators per IP 71151 was

suspended in CY 2005 in order to allocate additional resources to the Mitigating

Systems Performance Index (MSPI) evaluation effort.

Metric Criterion Met: No. The number of and recent trend in discrepancies identified during

this assessment period did not meet program expectations.
-3-

0

10

20

30

40

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of FAQs

New FAQs Approved/Withdrawn FAQs Open/Active FAQs

PI-2 Questions Regarding Interpretation of PI Guidance

Definition: Quarterly, count the number of frequently asked questions (FAQs). 

Criteria: Expect low numbers, with a stable or declining trend.

Goals Supported: Understandable, Risk-Informed, Predictable

Comments: Each quarter represents the total number of new FAQs introduced and approved

during the ROP NRC/Industry Working Group meetings held during the

respective quarter.

Analysis: For this assessment period, the number of unresolved interpretation questions

has decreased, and all of the FAQs that had previously remained open for a

significant amount of time have been closed. Modifications were made to NEI

99-02 to improve the FAQ process. Specifically, an appeal process was put in

place to resolve issues when the NRC and industry could not reach consensus. 

This process was used to reduce the backlog of FAQs.

Metric Criterion Met: Yes. This metric met its criteria based on the decreasing trend of open

interpretation issues and a relatively stable trend of new FAQs.
-4-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Occurrences

PI-3 Timely Indication of Declining Safety Performance

Definition: Quarterly, track PIs that cross multiple thresholds (e.g., green to yellow or white

to red). Evaluate and characterize these results to allow timely indication of

declining performance.

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Ensure Safety, Risk-Informed, Ensure Effectiveness

Analysis: There were no occurrences of PIs crossing multiple thresholds during this

assessment period. As noted in metric PI-1, the Waterford 3 High Pressure

Safety Injection System PI is under review by the NRC. The NRC is evaluating if 

the system was available between November 2003 and September 2004. 

Consequently, the PI may be greater than Green. If the Waterford 3 PI crosses

multiple thresholds, it will be included in this metric (based on the issue date of

the inspection report that determines the color of the PI).

Metric Criterion Met: Yes. This metric met its criteria based on zero occurrences of PIs

crossing multiple thresholds.
-5-

PI-4 PI Program Provides Insights to Help Ensure Plant Safety

Definition: Survey external and internal stakeholders asking whether the PI Program

provides useful insights to help ensure plant safety.

Criteria: Expect a low number of negative comments, with a stable or declining trend.

Goals Supported: Ensure Safety, Ensure Effectiveness, Risk-Informed

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005.

Analysis: Responses to this survey question show a sharp difference of opinion between

public interest groups/members of the public and industry groups/industry

representatives/state representatives.

Public stakeholder responses indicate that they feel the PI program promotes

plant safety between “somewhat less than needed” and “far less than needed.” 

Comments generally reflected the opinion that the PIs have become less useful

and tell the NRC and the public very little since most PIs always remain green.

In contrast to the rankings by the public stakeholders, industry and state

stakeholder responses averaged between “very much” and “somewhat.” Industry

comments indicate that the PI program provides useful insights to help ensure

plant safety. The Safety System Unavailability and Scrams with Loss of Normal

Heat Removal PIs were again called out as PIs that have the potential to cause

licensees to take actions that may impact plant safety. However, these

comments also pointed out the ongoing NRC/industry efforts on replacement

indicators for these PIs.

Metric Criterion Met: No. This metric did not meet its criteria based partly on the slight

negative trend in responses, and partly due to the sharp difference

of opinion between public stakeholders and utility representatives.
-6-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Untimely Actions

PI-5 Timely PI Data Reporting and Dissemination

Definition: Within 5 weeks of the end of each calendar quarter, track (count) late PI postings

on the NRC’s external Web site. Also note the number of late submittals from

licensees that did not meet the 21-day timeliness goal.

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: There were no late postings during this assessment period. There have been no

late PI data postings on the NRC's external web site since the inception of the

ROP.

Metric Criterion Met: Yes. The criteria for this metric has been met.
-7-

PI-6 Stakeholders Perceive Appropriate Overlap Between the PI Program and

Inspection Program

Definition: Survey external and internal stakeholders asking if appropriate overlap exists

between the PI program and the inspection program.

Criteria: Expect a low number of negative comments, with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Ensure Safety, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005.

Analysis: Similar to the comments made for metric PI-4, public interest groups/members of

the public commented that it is too easy to achieve all green PIs. The comments

indicated a need to increase inspections.

Similar to last year, industry groups/industry responses stated that, in general,

appropriate overlap exists between the PI program and the inspection program

and further commented that if anything, there was excessive overlap (e.g., in the

radiation protection area and using SDPs for single equipment failures already

included in the safety system unavailability PIs).

Overall, stakeholder satisfaction, as reported in the survey responses for initial

ROP implementation, last year’s assessment, and this year’s assessment,

showed a slight decrease in the satisfaction of overlap between the PI and

inspection programs. In addition, most of the negative comments were

associated with the PIs essentially remaining green all of the time, as also

commented on in metric PI-4. 

Metric Criterion Met: No. This metric did not meet its criteria based on the slight negative

trend in responses.
-8-

PI-7 Clarity of Performance Indicator Guidance

Definition: Survey external and internal stakeholders asking if NEI 99-02, “Regulatory

Assessment Performance Indicator Guideline,” provides clear guidance

regarding performance indicators.

Criteria: Expect a low number of negative comments or examples of interpretation issues,

with a stable or declining trend in the number of negative comments received.

Goals Supported: Understandable, Ensure Openness, Objective

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005.

Analysis: Utility/utility group respondents commented that the PI guidance is clear and that

the recent FAQ process change has been useful and efficient in resolving

questions on the PI guidance. Comments also indicated that work being

performed on the Scrams with Loss of Normal Heat Removal PI will make it more

objective and clearly defined.

Non-utility stakeholders, in general, had no comments or responded that it would

be more appropriate for the licensees to provide comments on the effectiveness

of the PI guidance.

Overall, stakeholder satisfaction, as reported in the survey responses for initial

ROP implementation, last year’s assessment, and the current year’s

assessment, were generally favorable and consistent.

Metric Criterion Met: Yes. This metric met its criteria with a stable and relatively positive

perception since ROP implementation.
-9-

70

75

80

85

90

95

100

2003 2004 2005

Program Calendar Averages

% Findings that Meet All Requirements

IP-1 Percentage of Inspection Findings Documented in Accordance With

Requirements

Definition: Audit inspection reports in relation to program requirements (IMC 0612, “Power

Reactor Inspection Reports”) for documenting green findings, greater-than-green

findings, and violations. Report the percentage of findings that meet the program

requirements.

Criteria: Expect a stable or improving trend in the percentage of findings documented in

accordance with program requirements.

Goals Supported: Objective, Risk-Informed, Predictable

 Comments: The graph represents the cumulative average for 46 inspection reports reviewed

by NRR during CY 2005. The staff issued a comprehensive revision to IMC 0612

in September 2005 to clarify definitions for NRC-identified, self-revealing and

licensee identified findings; to provide additional guidance on how to document

cross-cutting issues; and to reflect changes made to the document based on

inspector feedback to improve the program guidance on documentation of

inspection findings. 

Analysis: The staff audited integrated inspection reports from each branch and a number of

team inspection reports from each region. The percentage of findings

documented in accordance with IMC 0612 requirements was 94 percent, a slight

decline in compliance with IMC 0612 requirements from CY 2004, and the

inspection reports were well written. 

Metric Criterion Met: Yes. This metric met its criteria with a stable trend in the percentage of

findings documented in accordance with program requirements.
-10-

0

1

2

3

4

5

Q1/03Q2/03Q3/03Q4/03Q1/04Q2/04Q3/04Q4/04Q1/05Q2/05Q3/05Q4/05

Calendar Quarter

Number of Procedures Changed

IP-2 Number of Baseline Inspection Procedures Significantly Changed

Definition: Review all significant changes to baseline inspection procedures and count those

procedures whose scope or frequency of inspection changed, and count new

inspectable areas that relate to risk-informing the inspection.

Criteria: Expect relatively few significant changes, with a stable or declining trend.

Goals Supported: Risk-Informed, Ensure Safety, Predictable

Analysis: The scope of three baseline inspection procedures (IPs) were changed in CY

2005. First, IMC-2515, Appendix D, “Plant Status,” was revised to require

inspectors to monitor and trend RCS leakage indications. Additionally, IP

71111.20, “Refueling and Other Outage Activities,” was revised to add the

objective of evaluating licensee activities during reduced inventory and mid-loop

conditions. Lastly, IP 71111.21, “Component Design Bases Inspection,” was

revised to incorporate the inspection methodology discussed in Temporary

Instruction 2515/158, “Functional Review of Low Margin/Risk Significant

Components and Human Actions,” and the commitments specified in SECY-05-

0118, “Results of the Pilot Program to Improve the Effectiveness of NRC

Inspections of Engineering and Design Issues.”

Metric Criterion Met: Yes. This metric met its criteria based on relatively few significant

changes made to the baseline inspection program. 
-11-

0

5

10

15

20

25

30

35

40

45

Q1/03 Q2/03 Q3/03 Q4/03 Q1/04 Q2/04 Q3/04 Q4/04 Q1/05 Q2/05 Q3/05 Q4/05

Calendar Quarter

Number of Program Documents

1-3 4-6 7-9 10-12 13-15 >15 Forms Received

IP-3 Number of Feedback Forms per Document

Definition: Count the number of feedback forms received for each program document each

quarter. Use a histogram to chart the number of documents for which feedback

forms were received. Highlight those documents against which the most forms

are written.

Criteria: Expect a decreasing trend in the number of feedback forms received for program

documents.

Goals Supported: Understandable, Predictable, Objective

Analysis: The staff received 102 feedback forms in CY 2005. Approximately 53 percent of

all feedback forms received during this assessment period related to issues in

the areas of (1) Power Reactor Inspection Reports (IMC 0612), (2) Significance

Determination Process (IMC 0609), (3) Qualification Program for the Office of

Nuclear Reactor Regulation Programs (IMC 1245), (4) Inspection Procedure

71111.05, Fire Protection, (5) IP 71151, Performance Indicator Verification, and

(6) IP 71152, Problem Identification and Resolution. Of these areas, IMC 1245

received approximately 12 percent of all feedback forms, IMC 0612 received

10 percent, IMC 0609 received 10 percent, IP 71151 received 9 percent, IP

71111.05 received 6 percent, and IP71152 received 5 percent. The remaining

47 percent of feedback forms were spread across the other inspection manual

chapters and inspection procedures, with no individual document receiving more

than 5 percent of all feedback forms.
-12-

The concentration of older feedback forms in certain areas was reduced by

completing several improvement efforts. In particular, the staff issued a revision

to IMC 0612 which addressed 23 feedback issues. The SDP improvements

completed 14 feedback forms and inspector training guidance documents

completed 19 forms.

The staff resolved 125 feedback forms in CY 2005. This includes 63 feedback

forms received during CY 2005. Seventy-six feedback forms had been brought

forward from CY 2004. The staff resolved 62 forms (82 percent) leaving 14

forms remaining to be addressed in CY 2006. The three areas of focus for these

remaining forms are IMC 1245, MC 0612 and IMC 0609. A total of 58 feedback

forms will be brought forward into CY 2006. Fifty-two percent of the 125

feedback forms resolved resulted in changes to the inspection program

document. Eighty-five forms were issued in procedure revisions this year and 32

forms are pending a change notice. 

The number of feedback forms received in CY 2005 (102 forms) indicates a slight

downward trend in the number of feedback forms compared to the previous three

years (114 for CY 2004, 123 for CY 2003 and 112 for CY 2002). Quarterly data

indicated that the largest number of feedback forms were received in the first

quarter and gradually declined through the remaining three quarters. 

Metric Criterion Met: Yes. The metric data indicated that total number of feedback forms

received in CY 2005 decreased from the number of forms

received in the previous three years.
-13-

IP-4 Completion of Baseline Inspection Program

Definition: Annual completion of baseline inspection program.

Criteria: Defined as per IMC 2515, “Light-Water Reactor Inspection Program - Operations

Phase.”

Goals Supported: Ensure Safety, Predictable, Ensure Effectiveness

Analysis: All four regions completed their baseline inspections in CY 2005 in accordance

with IMC 2515, "Light-Water Reactor Inspection Program - Operations Phase." 

Each region documented completion of the program in a memorandum to the

Division of Inspection and Regional Support in NRR. These memoranda can be

found in ADAMS under ML060650523 (Region I), ML 060400590 & ML

060400584 (Region II), ML060390745 (Region III), and ML060300575 (Region

IV). As in the 2004 inspection cycle, all regions completed their baseline

inspections in 2005 with the allocated regional resources, without the need for

the coping measures experienced in CY 2002 and 2003. 

The NRC approved a delay in the completion of inspection activities associated

with the biennial emergency preparedness exercise at Waterford 3, and the

inspection is scheduled for completion during CY 2006. The biennial emergency

preparedness exercise and the inspection activities associated with the exercise

were initially planned for December 7, 2005. However, because of the impact

from Hurricane Katrina, the licensee requested that the exercise be rescheduled

to June 2006. 

Metric Criterion Met: Yes. The metric met its criteria because all four NRC regions

completed the baseline inspection program during ROP cycle 6

(CY 2005) in accordance with IMC 2515. 
-14-

75%

80%

85%

90%

95%

100%

Q1/2003Q2/2003Q3/2003Q4/2003Q1/2004Q2/2004Q3/2004Q4/2004Q1/2005Q2/2005Q3/2005Q4/2005

Calendar Quarter

Percentage of Timely Reports

Region I Region II Region III Region IV

IP-5 Inspection Reports are Timely

Definition: Obtain RPS data on the total number of reports issued and the number issued

within timeliness goals as stipulated in IMC 0612, “Power Reactor Inspection

Reports.”

Criteria: Expect 90 percent of inspection reports to be issued within program's timeliness

goals.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: A total of 995 inspection reports were issued during the CY 2005. The regions

met or exceeded the inspection report timeliness goal of 90 percent in each

quarter throughout the year. Overall, about 97 percent of all issued inspection

reports were timely. 

Metric Criterion Met: Yes. This metric met its criteria based on more than 90 percent of

inspection reports issued within program's timeliness goals for the

year.
-15-

0

0.5

1

1.5

2

2.5

Q1/03Q2/03Q3/03Q4/03Q1/04Q2/04Q3/04Q4/04Q1/05Q2/05Q3/05Q4/05

Calendar Quarter

Number of TIs Closed Within Goals

Region I Region II Region III Region IV

IP-6 Temporary Instructions (TIs) are Completed Timely

Definition: Audit the time to complete TIs by region. Compare the completion status in RPS

to TI requirements. Report by region the number of TIs closed within goals.

Criteria: Expect all TIs to be completed within TI requirements.

Goals Supported: Ensure Effectiveness, Ensure Safety, Predictable

Analysis: All regions were challenged in completing the six TIs during CY 2005 on time. 

None of the four regions were able to meet the timeliness goal during any single

quarter; however, some regions were able to complete their timeliness goals in

some quarters. The staff will review the reasons for untimely completion of TIs

and recommend possible solutions in CY 2006.

Metric Criterion Met: No. This metric did not meet its criteria because not all TIs were

completed within their timeliness goals.
-16-

95%

96%

97%

98%

99%

100%

Q1/03Q2/03Q3/03Q4/03Q1/04Q2/04Q3/04Q4/04Q1/05Q2/05Q3/05Q4/05

Calendar Quarter

Percentage of Timely Postings

Region I Region II Region III Region IV

IP-7 Public Communication Is Timely

Definition: NRR/DIRS posts inspection reports to the NRC's external (public) Web site within

ROP timeliness goals using the electronic version of inspection reports entered

into the Agency Document Access and Management System (ADAMS) by the

regions. NRR/DIRS also posts entries from the Plant Issues Matrix (PIM) to the

NRC's public Web site using data entered into RPS by the regions. In addition,

NRR/DIRS records the number of inspection reports not available in ADAMS and

the number of PIM entries not updated in RPS, as well as the number of

inspection reports and PIMs that are not posted to the NRC's public Web site

within the goals stipulated in IMC 0306, “Information Technology Support for the

Reactor Oversight Process.”

Criteria: Expect few untimely postings of PIMs or inspection reports, with a stable or

declining trend.

Goals Supported: Ensure Openness, Ensure Effectiveness. Predictable

Analysis: PIMs and inspection reports were posted within timeliness goals over 98% of the

time for each region in each quarter of CY 2005. There were two untimely

postings of inspection reports to the ROP web page during the 2nd and 3rd

quarters of CY 2005. These resulted in broken links on the web for a few days,

but these rare occurrences were isolated and the reports were quickly posted

once they were issued.

Metric Criterion Met: Yes. This metric met program expectations.
-17-

0

2

4

6

8

10

12

14

Q1/03Q2/03Q3/03Q4/03Q1/04Q2/04Q3/04Q4/04Q1/05Q2/05Q3/05Q4/05

Calendar Quarter

Number of Inaccuracies

IP-8 Public Communication Is Accurate

Definition: Each calendar quarter, sample information on the NRC's external (public) Web

site and count the number of times and reasons for regions changing PIMs or

inspection reports (i.e., inaccuracy, new information).

Criteria: Expect few inaccuracies, with a stable or declining trend.

Goals Supported: Ensure Openness, Ensure Effectiveness, Understandable

Analysis: There were few inaccurate postings of PIM entries or inspection reports on the

web identified during CY 2005. No region had more than four inaccurate

postings for any of the quarters, and although there were several inaccuracies

during the first quarter of 2005 across three of the four regions, these were

quickly identified and corrected within days of the posting. Greater attention to

detail was exhibited during the remaining quarters of CY 2005. 

Metric Criterion Met: Yes. This metric met its criteria based on few inaccurate postings 

during CY 2005.
-18-

IP-9 Inspection Reports Are Relevant, Useful, and Written in Plain Language

Definition: Survey external and internal stakeholders asking whether the information

contained in inspection reports is relevant, useful, and written in plain English.

Criteria: Trend average level of agreement.

Goals Supported: Ensure Effectiveness, Understandable, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: The majority of those who provided feedback to the question on whether the

information in the inspection reports were useful responded that the inspection

reports were clearly written and provided a better understanding of plant

operations. Other comments included:

• Should be more explicit in what they expect the utility to do about the

problems and issues identified.

• Don't like having information on a single problem spread out over 2 or 3

sections in the report. Also, it is impossible to determine what section you

are reading from. The formatting and outlining approach used though is

very difficult to follow and non-sensical.

• Listing in the reports of inspection scope is duplicative of the Inspection

Procedures and could be eliminated. 

• Recent definition changes to IMC 0612 to allow more credit for licensee

identified findings (NCVs) and improvements to Appendix ‘E’ (adding

additional examples of cross-cutting aspects) are applauded. 

• One area of concern to licensees is the recent change to IMC 0612 and

its expansion of what constitutes a performance deficiency. The current

guidance provided imposes standards beyond those specified or

committed to by a licensee in their licensing basis. 

• Additionally, the use of findings has greatly increased since its inception. 

Observations with insights now being developed in the findings are very

useful to the licensees; however, when documented as a “finding” in an

inspection report and on the PIM they take on the same weight as a

violation (NCV). This practice seems inappropriate and may result in

inappropriate resources being applied to the issue at the expense of

potentially more significant issues.

• One area that has improved but not as consistently as desired is the

incorporation of simplified drawings/schematics.

Overall, stakeholder satisfaction was generally favorable and consistent.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-19-

IP-10 Inspection Program Effectiveness and Adequacy in Covering Areas

Important to Safety

Definition: Survey external and internal stakeholders asking whether the inspection program

adequately covers areas that are important to safety and is effective in identifying

and ensuring the prompt correction of performance deficiencies. 

Criteria: Trend average level of agreement.

Goals Supported: Ensure Safety, Ensure Effectiveness, Risk-Informed

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: The responses to whether the inspection program adequately covers areas that

are important to safety and is effective in identifying and ensuring the prompt

correction of performance deficiencies were mixed. Although the majority of the

responses indicated that the inspection program covered the appropriate areas,

there were some concerns raised with the adequacy of Problem Identification

and Resolution inspections and the effectiveness of corrective actions taken by

licensees to correct deficiencies. Comments included:

• The Inspection Program has an adequate focus. However, the NRC is

weak in driving the prompt corrective actions necessary to correct

performance deficiencies by weakening its stance on enforcement.

• ROP has become too burdensome for inspectors because of the

complexity of forms and analysis. Need a better balance between

deterministic evaluations and use of risk assessment perspective.

• ASME requirements for accepable pressure vessel leakage was too

lenient (as exemplified by the prolonged leak at Davis-Besse). 

• While it may cover the appropriate areas, it is thoroughly ineffective at

ensuring prompt corrective actions. Even after the NRC places a reactor

in the regulatory response column, performance often continues to

decline into degraded cornerstones and multiple degraded cornerstones.

• The current inspection program can be improved by performing only one

ALARA inspection per cycle and combining the ALARA and Access

Control to Radiologically Significant Areas Inspections.

• The new effort to replace the “Safety System Design and Performance

Capability Inspection” by focusing on low margin systems/components

appears to be an improvement with regard to safety focus. Given the size

of the team and duration of the activity it may prove adequate to extend

the frequency beyond two years for this inspection procedure. 

• The NRC’s baseline inspection process and “Problem Identification and

Resolution Inspection” have been providing adequate focus on actions to

correct performance deficiencies.

• Inspection Program is better today than when the ROP hit the streets in

April 2000. Problem Identification and Resolution (PI&R) module is quite

simply inadequate and needs extensive overhaul. Its two major flaws are:
-20-

(1) poor criteria for selection of sample size and (2) poor criteria for

placing findings in context. The selection criteria are almost exclusively

linked to risk significance of systems/components, with some insights

from recent inspection findings. What is lacking from the selection criteria

is an explicit, formal attempt to probe the breadth of the corrective action

process. 

• The inspection program is effective in accomplishing its goals in covering

areas important to safety and ensuring that performance deficiencies are

identified and promptly corrected. Some of the inspected areas with

demonstrated good and improving performance, such as Radiation

Protection, may be over inspected. Some thought should be given to

adjusting the inspection schedule to more effectively distribute the

inspection resources.

Metric Criterion Met: Yes. Generally, the responses indicated that the inspection program

was effective in covering areas important to safety.
-21-

IP-11 Analysis of Baseline Inspection Procedures

Definition: Annually, review each baseline inspection procedure to determine its

effectiveness and contribution to the overall effectiveness of the baseline

inspection program. The objectives of the review are: (1) to determine if changes

in scope, frequency, or level of effort are needed based on recent experience,

(2) to determine if a change to the estimated hours for completion is needed,

(3) to define or change what constitutes minimum completion of each inspectable

area, if needed, and (4) to critically evaluate all of the inspectable areas together

along with the PI program to ensure that the inspectable areas are adequately

monitored for safety performance.

Criteria: None; trend only. Summarize and evaluate the individual inspection procedure

reviews and propose program adjustments as necessary to address noted

inefficiencies. Provide basis for any meaningful increase or decrease in

procedure scope, frequency, or level of effort as a result of the review.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff performed its annual review of each baseline inspection procedure

during CY 2005. The period assessed was from October 2004 through

September 2005. The focus of the review was to identify potential areas for

improvement in the baseline inspection program and to identify any notable

changes in inspection results. Based on this review, the staff did not identify any

significant issues which warranted changes to the inspection program. 

Additionally, the staff evaluated the number of findings associated with each

inspection procedure. The staff’s self-assessment of the inspection findings,

internal and external feedback forms, and other independent reviews of the ROP

indicated that the inspection program was generally successful in identifying

performance deficiencies in many of the areas inspected. The median number of

findings per 1000 hours of inspections (4.7) for CY 2005 remained about the

same as compared to the results from CY 2004. As in CY 2004, the success of

the inspection program in being able to identify performance deficiencies varied

with each inspection procedure. 

The staff also performed a more detailed analysis of the scope and level of effort

of the inspection procedures in CY 2005. The review consisted of an

examination of inspection results data from October 2001 through September

2004 with the purpose of identifying effectiveness of each inspection procedure

to identify performance deficiencies. The desired outcome from this review was

to improve the alignment of inspection resources to inspected areas which had

indication of risk-significant performance deficiencies in the past. The staff’s

recommended changes were incorporated into seven inspection procedures as

detailed in Change Notice 06-001 (reference ADAMS Accession No.

ML060060380) for implementation by the inspection staff in CY 2006.

Metric Criterion Met: Yes. This metric met program expectations.
-22-

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Percentage of Comforming Findings

SDP-1 The SDP Results Are Predictable and Repeatable and Focus Stakeholder

Attention on Significant Safety Issues

Definition: Annually, audit a representative sample (up to four per region) of inspection

findings against the standard criteria set forth in IMC 0609, “Significance

Determination Process,” and its appendices. To the extent available, samples

should include potentially greater-than-green findings that were presented to the

Significance Determination Process/Enforcement Review Panel (SERP).

Findings should contain adequate detail to enable an independent auditor to

trace through the available documentation and reach the same significance color

characterization. 

Criteria: The target goal is at least 90% are determined to be predictable and repeatable. 

Any SDP outcomes determined to be non-conservative will be evaluated and

appropriate programmatic changes will be implemented. 

Goals Supported: Ensure Safety, Risk-Informed, Predictable

Analysis: The Division of Risk Assessment (DRA) from NRR conducted a review of green

inspection findings. The findings reviewed by DRA came from inspection reports

issued in the last two quarters of CY 2004 and first two quarters of CY 2005. A

total of 114 green inspection findings were reviewed. The DRA conclusion

indicates that the screening process improved overall since the last report. No

inspection findings were identified as having a non-conservative risk assessment

or not meeting evaluation standards. The auditors determined that all findings
-23-

reviewed contained sufficient documentation to allow for any knowledgeable

person to conduct an evaluation and reach the same significance color

determination.

In addition, the Office of Nuclear Regulatory Research (RES) compared accident

sequence precursor (ASP) results and SDP evaluations for ASP analyses

completed during this assessment period. A total of 17 greater than green

inspection findings were reviewed. No significant differences were found

between the SDP findings and the ASP results.

Metric Criterion Met: Yes. Performance during this assessment period met program

expectations.
-24-

0

1

2

3

4

5

6

7

1Q/012Q/013Q/014Q/011Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Appeals

SDP Appeals Successful Appeals

SDP-2 SDP Outcomes Are Risk-Informed and Accepted by Stakeholders

Definition: Track the total number of appeals of final SDP results.

Criteria: Expect zero appeals of SDP significance that result in a final determination being

overturned across all regions. All successful appeals will be assessed to

determine causal factors and to recommend process improvements.

Goals Supported: Risk-Informed, Objective, Predictable

Analysis: There were ten appeals by licensees of SDP significance outcomes. Nine of the

appeals were aimed at findings of very low safety significance, GREEN. There

was one appeal of a WHITE finding for a security issue. In each case, the final

outcome was unchanged.

Metric Criterion Met: Yes. Performance during this assessment period met program

expectations based on no successful appeals.
-25-

SDP-3 Inspection Staff Is Proficient and Find Value in Using the SDP

Definition: Survey internal stakeholders using specific quantitative survey questions that

focus on training, effectiveness, and efficiency.

Criteria: Expect either a stable or an increasingly positive perception of the SDP process

over time.

Goals Supported: Ensure Effectiveness, Understandable, Risk-Informed

NOTE: There was no internal survey in CY 2005 consistent with its biennial frequency as

defined by IMC 0307. The next survey to internal stakeholders will be conducted

during the fourth quarter of CY 2006.
-26-

0

2

4

6

8

10

1Q/05 2Q/05 3Q/05 4Q/05

Calendar Quarter

Number of Revisions

IMC 0609 & CORRESPONDENT ATTACHMENTS

SDP-4 SDP Guidance is Stable Enough To Be Perceived as Predictable and SDP

Tools Reflect Current Plant Design and Licensee Operating Practices

Definition: Count the number of revisions to IMC 0609 and its appendices and substantive

revisions to the Phase 2 risk-informed inspection notebooks due to nonconservative technical flaws.

Criteria: Expect few revisions to IMC 0609 and its appendices, with a stable or declining

trend. Expect zero notebook retractions due to non-conservative technical flaws.

Goals Supported: Predictable, Risk-Informed, Ensure Safety

Analysis: There were seven significant changes in calendar year 2005 that affected IMC

0609 and its appendices. The Fire Protection SDP (IMC 0609, App F and its

appendices) was revised in 1Q2005 to change all references for expected and

high confidence fire intensity values and add additional applicable correlations

from NUREG-1805. The Shutdown Operations SDP (IMC 0609, App G and its

appendices) was revised in 1Q2005 to clarify the definition of “available.” The

Operator Re-qualification Human Performance SDP (IMC 0609, App I) was

revised in 3Q2005 to match current revisions to IP71111.11 (Operator

Requalification) and to fix several flaws that have been identified. In 4Q2005,

IMC 0609, 0609 App A, 0609.01 and 0609.02 were revised to reflect a concerted
-27-

effort to provide guidance which will help meet the Commission’s guidance on

the timeliness for finalizing the significant determination of inspection findings.

Metric Criterion Met: Yes. This metric met its criteria based on relatively few significant

changes made to the SDP program documents and no notebook

retractions due to non-conservative technical flaws.
-28-

SDP-5 Results of the Same Color Are Perceived by the Public to Warrant the Same

Level of Regulatory Attention for All Cornerstones

Definition: Survey external and internal stakeholders asking if the SDP yields an appropriate

and consistent regulatory response across all ROP cornerstones.

Criteria: Expect stable or increasingly positive perception of the SDP over time.

Goals Supported: Understandable, Objective, Predictable

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: Participants in the external ROP survey included 9 industry representatives, 4

State or local government agencies, and 7 private citizens or public interest

groups. Most responses skewed toward the negative but for varying reasons. 

Two believe that the NRC handling of the Davis-Besse risk evaluation is a clear

example of the lack of equivalency between the cornerstones. Three responses

pointed to the lack of inconsistent outcome between the risk-informed and the

more deterministic SDP formats. One respondent stated that there is a lack

consistency, however, was not sure how to assess the question of equivalency

between the cornerstones when risk informing findings. The general tone of the

responses is similar to that of the previous assessment period, that the SDP did

not yield equivalent results for issues of similar significance in all ROP

cornerstones. The survey question and resultant metric were revised in CY 2005

to clarify the staff’s intent that it is the level of regulatory attention (versus the

level of significance) that is expected to be consistent between the cornerstones.

Stakeholders have expressed concern about this issue since the early inception

of the ROP. However, the staff continues to believe that relative parity has been

achieved among the cornerstones, based on the potential impact on public health

and safety and the designated NRC response to specific findings.

Metric Criterion Met: No. This metric does not meet its criteria based on a stable negative

perception over the past six years of ROP implementation.
-29-

0

2

4

6

8

10

12

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Percentage of DIE for SDP Efforts

R1 R2 R3 R4 National

SDP-6 The Resources (Direct Charges and Support Activities) Expended Are

Appropriate

Definition: Track the percentage of total resource expenditures attributed to SDP activities

to determine the effort expended by the regions in completing SDP evaluations

as a percentage of the total regional direct inspection effort.

Criteria: Total SDP expenditures should not exceed 10 percent of the total regional direct

inspection effort (DIE) with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Predictable

Analysis: Regional expenditures associated with SDP evaluations remain stable and below

the target goal.

Metric Criterion Met: Yes. Performance during this assessment period met program

expectations.
-30-

0%

20%

40%

60%

80%

100%

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Actions Completed within 90 days

Region I Region II Region III Region IV Total

SDP-7a Final Significance Determinations Are Timely

Definition: Conduct a quarterly audit of RPS data to identify the total number of inspection

items finalized as greater than green that were under review for more than 90

days since:

(1) the date of initial licensee notification of the preliminary

significance in an inspection report, or 

(2) the date the item was formally transmitted to an NRR technical

branch for SDP assistance, or

(3) the item was otherwise documented in an inspection report as an

apparent violation pending completion of a significance

determination and not counted in either of the above categories.

Criteria: At least 85% of all SDP results that are counted per the criteria above (increasing

to 90% in CY 2006) should be finalized within 90 days. All issues greater than 90

days will be assessed to determine causal factors and to recommend process

improvements.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: Timeliness of final significance determinations increased from 48% in FY 2004 to

68% for FY 2005. However, the percentage of completed findings remained

below the associated timeliness metric - 85 percent within 90 days in FY 2005. 

This was due in part to the impact of closing several longstanding issues as

discussed further in the performance metric discussion. The staff noted that 5 of

the 7 late issues missed the limit by 6 or fewer days, with the oldest issue taking

191 days to finalize. However, had the five issues that missed the limit by 6 or
-31-

fewer days been completed on time, then 91 percent of the final significance

determinations would have been timely and the metric would have been met.

Although timeliness in reaching final significance remains a challenge, the

revised and new SDPs with the associated training, the standardized riskinformed inspection notebooks, the Phase 2 pre-solved tables, the enhanced

SPAR models, and other SDP process changes are all intended to achieve

efficiencies and improve the process.

Metric Criterion Met: No. Performance during this assessment period does not meet

program expectations.
-32-

SDP-7b Final Significance Determinations Are Timely

Definition: Conduct a quarterly audit of issues that were assessed by the Significance

Determination Process/Enforcement Review Panel (SERP) to identify the total

number of inspection items finalized as green or greater-than-green that were

under review for more than 90 days since:

(1) the date of initial licensee notification of the preliminary significance in an

inspection report, otherwise documented in an inspection report as an

“AV” pending completion of a significance determination, or 

(2) the date the item was presented to the SERP for review. 

Criteria: At least 90% of all SDP results that are counted per the criteria above should be

finalized within 90 average days and 100% in 180 days. All issues greater than

180 days will be assessed to determine causal factors and to recommend

process improvements.

This metric is being piloted as a potential replacement for the existing SDP

timeliness metric.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: This metric is being piloted as a potential replacement for the existing SDP

timeliness metric and will be reported for the first time in the CY 2006 ROP

Performance Metric Report.

Metric Criterion Met: Not Applicable.
-33-

0

1

2

3

4

5

6

7

8

9

10

2000 2001 2002 2003 2004 2005

Calendar Quarter

Number of Inaccuracies

SDP-8 SDP Results Are Communicated Accurately to the Public

Definition: Each calendar quarter, track the number of inspection findings that are

inaccurately communicated to the public (color of findings is inaccurately

reported) by auditing the inspection findings summary information available on

the NRC Web. The detailed review will include item type, significance

characterization, enforcement action status, and text descriptions of greater-thangreen inspection findings prior to release to external stakeholders.

Criteria: The target goal is zero inaccuracies, with a stable or declining trend. All

inaccuracies must be addressed.

Goals Supported: Ensure Openness, Understandable, Ensure Effectiveness

Analysis: During the current assessment period no inaccuracies were identified. 

Metric Criterion Met: Yes. Performance during this assessment period met program

expectations.
-34-

0

1

2

3

4

5

6

7

8

9

10

2001 2002 2003 2004 2005

Year

Number of Deviations

AS-1 Subjective Judgment Is Minimized and Is Not a Central Feature of the

Process. Actions Are Determined by Quantifiable Assessment Inputs

(Examine PIs and SDP Results)

Definition: Audit all assessment-related letters and count the number of deviations from the

Action Matrix. Evaluate the causes for these deviations and identify changes to

the ROP, if any, to improve the guidance documents.

Criteria: Expect few deviations, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Openness

Analysis: There have been a total of nine (9) deviations from the Action Matrix since the

beginning of the Reactor Oversight Program in Calender Year 2000. Four of

these deviations occurred in Calendar Year 2005.

• Point Beach deviation dated December 23, 2005, to not require a separate

supplemental inspection of a white finding in the emergency preparedness

cornerstone. This finding was an old inspection item that was not reflective of

current licensee performance and was delayed in resolution due to investigations

by the Office of Investigations and the Department of Justice.

• Indian Point 2 deviation dated October 28, 2005, to closely monitor the utility’s

performance in addressing issues associated with the unit 2 spent fuel pool,

including on-site tritium contamination, and improving the reliability and

availability of the Alert and Notification System, including implementation of the

backup power requirements in the Energy Policy Act of 2005.

• Salem/Hope Creek deviation dated July 29, 2005, to renew the deviation that
-35-

was dated August 20, 2004.

• Davis-Besse deviation dated May 16, 2005, to allow for increased level of

oversight as Davis-Besse transitioned out of the IMC 0350 Process to the normal

ROP assessment on July 1, 2005.

• Salem/Hope Creek deviation dated August 20, 2004, to provide heightened NRC

oversight to closely monitor the licensee’s actions to address significant issues

associated with safety conscious work environment (SCWE).

• Cooper deviation dated April 12, 2004, to provide heightened NRC oversight to

monitor the actions confirmed by the Confirmatory Action Letter (CAL), dated

January 30, 2003.

• Indian Point 2 deviation dated April 2, 2004, to closely monitor the utility’s

performance following the station’s recovery from longstanding problems.

• Indian Point 2 deviation dated March 18, 2003 to provide for heightened

oversight of the facility. 

• Oconee 1 deviation dated August 23, 2002, to permit for agency actions

consistent with the degraded cornerstone column, including the performance of

an IP 95002 vice IP 95003 supplemental inspection.

The staff’s evaluation of the 2005 Action Matrix Deviations concluded the following:

(1) Based on a review of the Davis-Besse deviation, IMC 0305 was revised to allow for

the regional offices to utilize additional follow-up actions for plants that are exiting the

IMC 0350 Process. These actions were previously available for plants exiting the

multiple/repetitive degraded cornerstone of the Action Matrix. The programmatic

changes made as a result of this deviation will prevent the need for similar deviations in

the future.

(2) The Salem/Hope Creek deviation was an extension of the previous deviation in CY

2004. The results of the NRC’s safety culture initiative, as directed in the Commission

SRM’s dated August 20, 2004, and December 21, 2005, will dictate the appropriate

programmatic changes.

(3) The Indian Point 2 deviation was issued to address a variety of performance issues

unique to the site and represented a customized approach as envisioned in IMC 0305.

No programmatic changes are anticipated as a result of this deviation. 

(4) The Point Beach deviation represented a unique situation where the NRC decided

not to re-inspect an old inspection item that was not reflective of current licensee

performance. The performance deficiency associated with this finding had been

previously inspected in accordance with IP 95003, “Supplemental Inspection for

Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow

Inputs, or One Red Input.” As part of the IP 95003 inspection, the NRC also performed

attachment 95003.01, “Additional Emergency Preparedness Cornerstone Inspection.” 
-36-

This finding was delayed in resolution due to investigations by the Office of

Investigations and the Department of Justice. No programmatic changes are anticipated

as a result of this deviation. 

Metric Criterion Met: No. This metric does not meet its criteria based on the number of

deviations increasing during CY 2005.
-37-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Inappropriate Actions

AS-2 The Program Is Well-defined Enough to Be Consistently Implemented

Definition: Audit all assessment letters and count the number of significant departures from

requirements in IMCs 0305, “Operating Reactor Assessment Program,” and

0350, “Oversight of Operating Reactor Facilities in an Extended Shutdown as a

Result of Significant Performance Problems.” Timeliness goals are counted in

metric AS-5.

Criteria: Expect few departures, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Effectiveness

Analysis: There were no significant departures from the requirements of IMC 0305 or 0350

as a result of an audit of assessment letters during the period between January

and December 2005.

Metric Criterion Met: Yes. This metric met its criteria based on no departures from the

requirement of IMC 0305 or 0350 and the trend is considered

stable. 
-38-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Inappropriate Actions

AS-3 Actions Taken Are Commensurate with the Risk of the Issue and Overall

Plant Risk

Definition: Review actions taken for greater-than-green inspection findings and PIs. Track

the number of actions (or lack of actions) taken by the regions that are not

appropriate for the significance of the issues and are non consistent with the

Action Matrix.

Criteria: Expect few departures, with a stable or declining trend.

Goals Supported: Risk-Informed, Ensure Effectiveness, Ensure Safety

Analysis: All actions taken by the regional offices were consistent with the Action Matrix

during the period between January and December 2005. However, additional

actions were taken at Salem and Hope Creek that were initiated through the

Allegations Program in CY 2004. A deviation was issued on August 20, 2004

and July 29, 2005, to provide heightened NRC oversight to closely monitor the

licensee’s actions to address significant issues associated with safety conscious

work environment (SCWE). The results of the NRC’s safety culture initiative, as

directed in the Commission SRM’s dated August 20, 2004, and December 21,

2005, will dictate the appropriate programmatic changes.

Metric Criterion Met: Yes. This metric met its criteria based on no departures from the ROP

regarding actions taken in response to greater-than-green findings

or PIs.
-39-

AS-4 The Number And Scope of Additional Actions Recommended as a Result of

the Agency Action Review Meeting (AARM) Beyond Those Actions Already

Taken Are Limited

Definition: Review the results of the Agency Action Review Meeting (AARM).

Criteria: Few additional actions, with a stable or declining trend.

Goals Supported: Understandable, Predictable, Objective

Analysis: The AARM was held on May 4, 2005, in Flintstone, Maryland. The participants

confirmed the appropriateness of agency actions for Point Beach 1 and 2, Perry,

Davis-Besse and Cooper. The participants did not recommend any additional

actions, beyond those already taken or planned. The next Agency Action Review

Meeting is scheduled for April 2006.

Metric Criterion Met: Yes. This metric met its criteria based on no recommended additional

actions beyond those already taken or planned. 
-40-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Untimely Actions

AS-5 Assessment Program Results (Assessment Reviews, Assessment Letters

and Public Meetings) Are Completed in a Timely Manner

Definition: Track the number of instances in which timeliness goals established in IMC 0305

were not met. The regions will collect timeliness data for the conduct of quarterly

reviews (within 5 weeks of the end of quarter); mid-cycle, and end-of-cycle

reviews (within 6 weeks of the end of quarter); issuance of assessment letters

(within 2 weeks of the quarterly review and 3 weeks of the mid-cycle and end-ofcycle reviews); assessment follow-up letters (on or before the next quarterly

review); and public meetings (within 16 weeks of the end of the assessment

period).

Criteria: Expect few instances in which timeliness goals were not met, with a stable or

declining trend.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: 4Q/2005: All quarterly reviews and three assessment follow-up letters were

completed within timeliness goals.

3Q/2005: All mid-cycle review meetings and mid-cycle letters were conducted

within timeliness goals. Additionally, one public meeting was completed within

timeliness goals.

2Q/2005: All quarterly assessment reviews and two assessment follow-up letters

were completed within timeliness goals. Additionally, forty-eight out of forty-nine

public meetings were completed within timeliness goals.
-41-

1Q/2005: All end-of-cycle meetings and annual assessment letters and one 

assessment follow-up letter were completed within timeliness goals. Additionally,

thirteen public meetings were completed within timeliness goals. 

Metric Criterion Met: Yes. This metric met its criteria based on the timeliness goals were met

and have a stable trend.
-42-

0

1

2

3

4

5

6

7

8

9

10

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Untimely Actions

AS-6 Public Availability of Assessment Letters Is Timely

Definition: Record the number of letters not available in ADAMS and number of letters not

posted to the Web site within goals as stipulated in IMC 0305, “Operating

Reactor Assessment Program.” 

Criteria: IIPB posts assessment letters to the NRC's external Web site using the

electronic version in ADAMS within 10 weeks after the end of mid-cycle and endof-cycle assessment periods and within 8 weeks of the end of intervening

quarters.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: 4Q/2005: All three assessment follow-up letters were posted to the web within

timeliness goals.

3Q/2005: All of the mid-cycle letters were posted to the web within timeliness

goals.

2Q/2005: One of the two assessment follow-up letters was not posted to the web

within timeliness goals.

1Q/2005: All annual assessment letters and one assessment follow-up letter

were posted to the web within timeliness goals.
-43-

Metric Criterion Met: Yes. This metric met its criteria based on the overall web posting

timeliness goal being met. 
-44-

0

1

2

3

4

5

2001 2002 2003 2004 2005

Calendar Year

Number of Program Revisions

AS-7 Assessment Program Procedures Are Stable Enough to Be Perceived as

Predictable

Definition: Count the number of revisions to IMCs 0305 and 0350.

Criteria: Expect few revisions, with a stable or declining trend.

Goals Supported: Predictable, Understandable

Analysis: During CY 2005, there was one revision to IMC 0305, “Operating Reactor

Assessment Program” and IMC 0350, “ Oversight of Reactor Facilities in a

Shutdown Condition Due to Significant Performance and/or Operational

Concerns”, which were issued on November 15, 2005 and December 21, 2005,

respectively. 

Metric Criterion Met: Yes. This metric met its criteria based on the fact that the number of

IMC 0305 and IMC 0350 revisions have remained stable.
-45-

0

50

100

150

200

250

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Average Number of Elapsed Days

AS-8 The NRC's Response to Performance Issues Is Timely

Definition: Count the number of days between issuance of an assessment letter discussing

an issue of more than very low safety significance and completion of the

supplemental inspection (by exit meeting date, not issuance of the inspection

report).

Criteria: Expect a stable or declining trend.

Goals Supported: Ensure Safety, Ensure Effectiveness

Comments: The data represents an average timeliness for the supplemental inspections

completed in each region in any given quarter.

Analysis: Data collected to date indicates a relatively stable long term trend regarding the

elapsed time between the issuance of an assessment letter and the completion

of the corresponding supplemental inspection. The staff will continue to monitor

this data set to determine if an adverse trend exists. 

Metric Criterion Met: Yes. This metric met its criteria based on the relatively stable long term

trend regarding the NRC’s timeliness in responding to

performance issues.
-46-

AS-9 NRC Takes Appropriate Actions to Address Performance Issues

Definition: Solicit feedback on the appropriateness of regulatory attention given to licensees

with performance problems via a survey question to both internal and external

stakeholders.

Criteria: Expect stable or improved perception.

Goals Supported: Ensure Safety, Ensure Effectiveness, Understandable

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: The industry and States generally agreed that actions taken by the NRC for

plants outside of the licensee response column have been appropriate. Public

interest groups were generally critical of NRC actions. Specific concerns were

expressed with NRC actions at plants with significant performance problems

such as Cooper, Davis-Besse, and Perry. The level of external stakeholder

satisfaction in this area was generally favorable and similar to previous years. 

Metric Criterion Met: Yes. This metric met its criteria with a stable positive perception over

the past six years of ROP implementation.
-47-

AS-10 Assessment Reports Are Relevant, Useful, and Written in Plain Language

Definition: Survey internal and external stakeholders asking whether the information

contained in assessment reports is relevant, useful, and written in plain English.

Criteria: Expect stable or improved perception of the relevance, usefulness, and

understandability of assessment reports.

Goals Supported: Understandable, Ensure Effectiveness, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: The industry and States generally agreed that the information contained in

assessment reports is relevant, useful, and written in plain English. One public

interest group stated that the assessment letters contained too much boilerplate

information which precluded substantive insights about performance at individual

sites. The level of external stakeholder satisfaction in this area was generally

favorable and similar to previous years.

Metric Criterion Met: Yes. This metric met its criteria with a stable positive perception over

the past six years of ROP implementation.
-48-

0

1

2

3

4

5

1Q/022Q/023Q/024Q/021Q/032Q/033Q/034Q/031Q/042Q/043Q/044Q/041Q/052Q/053Q/054Q/05

Calendar Quarter

Number of Occurrences

AS-11 Degradations in Plant Performance, as Measured in the Action Matrix, are

Gradual and Allow Adequate Agency Engagement of the Licensees

Definition: Track the number of instances each quarter in which plants move more than one

column to the right in the Action Matrix (as indicated on the Action Matrix

Summary).

Criteria: Expect few instances in which plant performance causes a plant to move more

than one column to the right in the Action Matrix. Provide a qualitative

explanation of each instance in which this occurs. Expect a stable or declining

trend from the first-year benchmark.

Goals Supported: Risk-informed, Ensure Safety, Predictable 

Analysis: Only one reactor site moved more than one column to the right in the Action

Matrix over the past few years. Palo Verde Units 1, 2, and 3 moved from the

licensee response column to the degraded cornerstone column in 4Q/2004 due

to a yellow finding associated with a failure to maintain design control of

containment sump recirculation piping. This finding was documented in the final

significance determination letter dated April 8, 2005 and began counting in the

assessment process at the end of inspection period which ended in 4Q/2004.

Metric Criterion Met: Yes. This metric met its criteria based on the fact that only one site

(with all three units impacted by the finding) moved two or more

columns to the right during the period in CY 2005.
-49-

O-1 Stakeholders Perceive the ROP To Be Predictable and Objective

Definition: Survey external and internal stakeholders asking if ROP oversight activities are

predictable (i.e., controlled by the process) and reasonably objective (i.e., based

on supported facts, rather than relying on subjective judgment).

Criteria: Expect a stable or increasing positive perception over time.

Goals Supported: Objective, Predictable, Ensure Effectiveness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally perceive the ROP to be predictable and

reasonably objective, particularly when compared to the previous process, but

both industry and public stakeholders stated some concerns and recommended

that additional improvements be considered. 

The responses from State and local agencies were favorable and did not identify

any specific concerns. 

Public perception was fairly neutral, though some public stakeholders stated that

the ROP is too predictable in that most issues are assessed as Green and that

the bar needs to be raised, and that the ROP is too influenced by industry input

that allows downgrading of the significance of problems. 

Industry stakeholders generally agreed that the ROP was predictable and

objective, and acknowledged the improved guidance on documenting crosscutting issues. Some industry responses expressed concerns with “reverse

SDP” techniques (i.e., predetermining significance of issues and then developing

the supporting arguments), the definition and use of “performance deficiency” in

identifying and documenting inspection findings, the secrecy of the security area,

and documenting fire protection issues.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-50-

O-2 Stakeholders Perceive the ROP To Be Risk-Informed

Definition: Survey external and internal stakeholders asking if the ROP is risk-informed, in

that actions and outcomes are appropriately graduated on the basis of increased

significance.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Risk-Informed, Ensure Effectiveness, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally acknowledged that the ROP is risk-informed and 

that actions and outcomes are appropriately graduated on the basis of increased

significance, but both public and utility stakeholders expressed some concerns

and noted that further improvements could be made.

. 

Public perception was neutral to slightly negative indicating their concern that the

ROP is somewhat less risk-informed than needed. Some members of the public

indicated that the ROP should be more rigorous and less open for manipulation

by licensees. 

State and local agencies responded positively and did not identify any specific

concerns. 

Similar to last year, the industry perceived that some SDPs are deterministic and

not appropriately characterized by risk insights. These SDPs include the

emergency preparedness, radiation protection, fire, and security areas. Industry

respondents stated that implementation of the Mitigating Systems Performance

Index will be an improvement in risk-informing the ROP.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-51-

O-3 Stakeholders Perceive the ROP To Be Understandable

Definition: Survey external and internal stakeholders asking if the ROP is understandable

and if the processes, procedures, and products are clear and written in plain

English.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Understandable, Ensure Effectiveness, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally acknowledged that the ROP is understandable

and that the processes, procedures, and products are clear and written in plain

English, but some public stakeholders expressed concerns and noted that further

improvements could be made.

Public responses to this question were fairly neutral to slightly negative,

indicating their concern that the ROP is somewhat less understandable than

needed. Some public stakeholders stated that the ROP is too complex and

cumbersome and that the organization of ROP information on the Website was

not easily accessible. However, there was acknowledgment that inspection

reports are generally well written in response to another survey question (see

metric IP-9).

State agencies and industry stakeholders agreed that the ROP is understandable

and written in plain English. The industry recognized improvements in the

inspection manual regarding cross-cutting issues and the definitions of “licenseeidentified,” “NRC identified,” and “self-revealing” findings, but noted that the fire

protection and steam generator SDPs were highly technical and difficult to follow.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-52-

O-4 Stakeholders Perceive That the ROP Provides Adequate Regulatory

Assurance That Plants Are Operated and Maintained Safely

Definition: Survey external and internal stakeholders asking if the ROP provides adequate

regulatory assurance, when combined with other NRC regulatory processes, that

plants are being operated and maintained safely.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Safety, Ensure Effectiveness, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. The

wording for this metric and the corresponding survey question has changed to

reflect the revision to the NRC’s performance goals as stipulated in the Strategic

Plan. The previous goal to maintain safety was replaced with the current goal to

ensure protection of public health and safety and the environment.

Analysis: Overall, external stakeholders generally acknowledged that the ROP provides

adequate regulatory assurance that plants are being operated and maintained

safely. However, public stakeholders were on the opposite end of the spectrum

from utility and State stakeholders.

Public responses to this question were fairly negative, indicating their concern

that the ROP is somewhat less to far less adequate than needed to provide

assurance that plants are being operated and maintained safely. Examples of

their concerns include that all of the Green PIs give a false sense of safety, that

the NRC does not identify safety issues in a timely manner, and that there is no

evidence to support a positive response.

However, both state and industry stakeholder responses were very positive

indicating that they believe the ROP adequately assures that the plants are

operated and maintained safely. Responses from State and local agencies were

far more positive than they had been in previous years.

Metric Criterion Met: Yes. This metric met its criteria with an increasingly positive perception

over the past six years of ROP implementation.
-53-

O-5 Stakeholders Perceive the ROP To Be Effective, Efficient, Realistic, and

Timely

Definition: Survey external and internal stakeholders asking whether the ROP is effective,

efficient, realistic, and timely.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Effectiveness, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. The

wording for this metric and the corresponding survey question has changed to

reflect the revision to the NRC’s performance goals as stipulated in the Strategic

Plan. The current goal added “timely” to the previous goal of ensuring that NRC

actions are effective, efficient, and realistic.

Analysis: Overall, external stakeholders generally acknowledged that the ROP is effective,

efficient, realistic, and timely, particularly when compared to previous processes. 

Most of the negative comments focused on the timeliness of the SDP which is

captured by metric SDP-7.

Public perception was fairly negative, noting their concern with the timely

identification and evaluation of performance issues. One member of the public

stated that the ROP is cumbersome for inspectors and promotes cleverness on

the part of the utility in answering versus fixing plant problems. 

State and local agencies responded positively and did not identify any specific

concerns. 

 

Industry stakeholders generally agreed that the ROP was effective, efficient,

realistic, and timely, noting that inspections are more focused, findings are

evaluated using a more structured tool, and performance assessment is more

objective. They further acknowledged the improvement in SDP timeliness. 

Some industry responses noted that the NRC should consider reevaluating the

frequency of certain inspections, that certain SDPs have become too complex,

and the need to improve the transparency and communication in the security

area.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-54-

O-6 Stakeholders Perceive That the ROP Ensures Openness

Definition: Survey external and internal stakeholders asking if the ROP ensures openness in

the regulatory process.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Openness, Ensure Effectiveness

NOTES: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. The

wording for this metric and the corresponding survey question has changed to

reflect the revision to the NRC’s performance goals as stipulated in the Strategic

Plan. The previous goal to enhance public confidence was basically replaced

with the current goal to ensure openness in the regulatory process.

Analysis: External stakeholders generally acknowledged that the ROP ensures openness

in the regulatory process, but both public and utility stakeholders expressed

some concerns and noted that further improvements could be made.

Public perception was neutral to slightly negative indicating their concern that the

ROP is somewhat less open than needed. One stakeholder noted that public

perception is that the NRC change from the SALP process to the ROP is handholding with the industry to provide elevated assessments of plant safety

performance.

The responses from State and local agencies were very favorable and one

stakeholder noted that their participation in the process is an example of the

openness.

Utility stakeholders reiterated that the ROP is generally a very open process, but

pointed out the security process and the Significance Determination Process and

Enforcement Review Panel (SERP) activities as specific areas that should be

more open and allow for greater stakeholder involvement. 

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation.
-55-

O-7 Opportunities for Public Participation in the Process

Definition: Survey external and internal stakeholders asking if there are sufficient

opportunities for the public to participate in the process.

Criteria: Expect positive responses or an improving trend.

Goals Supported: Ensure Openness, Ensure Effectiveness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally agreed that there are sufficient opportunities for

the public to participate in the process, though some stakeholders noted that

further improvements could be made.

Public perception was neutral to slightly positive, indicating their general

agreement that the public has ample opportunity to participate in the process. 

One respondent suggested that a more significant effort be made to seek

comments from nuclear professionals, while another felt that their comments

have often been ignored in the past. Another respondent acknowledged the

numerous opportunities for public participation (monthly public meetings at NRC

headquarters, annual public meetings conducted in the reactor communities,

annual solicitation of public comments, annual ROP Commission briefing, and

the staff’s consolidated response to last year’s comments), but suggested that

the NRC improve its feedback to public inputs and comments received during the

annual assessment meetings. 

The responses from State and local agencies were very favorable and one

stakeholder noted their participation in the process as an example.

Industry stakeholder responses were generally very positive, but pointed out that

the public does not have adequate opportunity to participate in the security

process.

Metric Criterion Met: Yes. This metric met its criteria with mostly positive comments and a

stable perception over the past six years of ROP implementation.
-56-

O-8 Stakeholders Perceive the NRC To Be Responsive to its Inputs and

Comments

Definition: Survey external and internal stakeholders asking if the NRC is responsive to the

public's inputs and comments on the ROP.

Criteria: Expect positive responses or an improving trend.

Goals Supported: Ensure Openness, Ensure Effectiveness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally acknowledged that the NRC has been

responsive to the public's inputs and comments on the ROP, and noted that the

staff’s efforts in CY 2005 to improve staff responsiveness were effective.

This metric did not meet its criteria during the previous self-assessment due to an

increasing negative perception over time. To address the continued concerns

that the NRC has been unresponsive to stakeholder feedback, the staff

consolidated the comments from the CY 2004 external survey responses by

question and provided a comprehensive response to each question. This

consolidated response, along with this Commission paper and the annual ROP

performance metric report, was posted to the ROP Web page and sent to each

respondent to the survey in August 2005.

Public perception was fairly neutral, with one stakeholder acknowledging that the

consolidated response to the CY 2004 survey was very good, and that if that

effort was matched by comparable feedback provided to public inputs and

comments received during the annual assessment meetings, their assessment

would be that the NRC was very responsive to public input and comments on the

ROP.

The responses from State and local agencies were very favorable and one

stakeholder noted their first hand experience as a supporting comment.

Industry stakeholder responses were generally very positive, but pointed out that

the NRC remains relatively unresponsive to public input on the security process. 

Examples of NRC responsiveness included the consolidated response to the CY

2004 survey, the action item list used to track and status issues discussed at the

monthly ROP meetings, and the NRC’s efforts to encourage participation at the

monthly public ROP meetings and the annual performance review meetings, in

which public comments are received, evaluated, and dispositioned in a

professional manner.

Metric Criterion Met: Yes. This metric met its criteria with a stable perception over the past

six years of ROP implementation and a recently increasing

positive perception.
-57-

O-9 Stakeholders Perceive That the ROP Is Implemented as Defined

Definition: Survey external and internal stakeholders asking if the ROP has been

implemented as defined by program documents.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Predictable, Understandable, Ensure Openness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally agreed that the ROP has been implemented as

defined by program documents, though some stakeholders noted that further

improvements could be made.

Public perception was essentially neutral to slightly positive, indicating their

general agreement that the ROP has been implemented as defined by program

documents. One respondent indicated that the ROP program documents are

difficult to find on the NRC Website, making it difficult to determine if the NRC is

abiding by them while implementing the ROP.

The responses from State and local agencies were very favorable and did not

identify any specific concerns. 

Industry stakeholder responses were generally very positive, recognizing the

improved guidance relative to cross-cutting issue identification, characterization,

and evaluation, but also pointed out that the security area was neither scrutable

nor predictable.

Metric Criterion Met: Yes. This metric met its criteria with mostly positive comments and a

stable positive perception over the past six years of ROP

implementation.
-58-

O-10 Stakeholders Perceive That the ROP Does Not Result in Unintended

Consequences

Definition: Survey external and internal stakeholders asking if the ROP results in unintended

consequences.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Safety, Ensure Effectiveness

NOTE: An internal survey was not conducted in CY 2005; therefore, the input to this

metric came solely from the external survey conducted in October 2005. 

Analysis: External stakeholders generally agreed that the ROP minimizes unintended

consequences, though both public and utility stakeholders had some concerns

and provided specific examples of unintended consequences. As one of the

respondents noted, this question was poorly worded (“does the ROP result in

unintended consequences”) in the survey since positive responses are on the

other side of the scale when compared to all other questions. This made it

difficult to determine the external stakeholders relative perception of whether the

ROP results in unintended consequences. The staff will reword the question in

future surveys to “does the ROP minimize unintended consequences?” which will

provide a more consistent response.

Public perception was essentially neutral to slightly negative, indicating their

concern that the ROP does result in some unintended consequences. Some

public respondents stated that licensees use poor judgment and waste a lot of

time and effort trying to avoid greater-than-green issues.

Industry stakeholder responses were essentially neutral to slightly positive,

noting that the recent revision to IMC 0612 which bettered defined “licenseeidentified” avoids an unintended consequence and will encourage licensees to

openly identify problems. Noted examples of potential unintended consequences

included the recent practice of identifying cross-cutting aspects for essentially all

NRC findings, two performance indicators that have the potential to cause

licensees to take actions that can adversely impact plant safety, and the security

“no comment policy” could have the unintended consequence of undermining

public trust and confidence. The responses from State and local agencies did

not identify any specific concerns.

This metric did not meet its criteria during the previous self-assessment due to an

increasing negative perception over time. However, this year’s responses

included some positive comments and had fewer negative comments.

Metric Criterion Met: Yes. This metric met its criteria based on an increasing positive

perception when compared to the past few years of ROP

implementation. 
-59-

O-11 Analysis of NRC’s Responses to Significant Events

Definition: Review reports from incident investigation teams (IITs) and augmented

inspection teams (AITs) to collect lessons learned regarding ROP programmatic

deficiencies (i.e., did the baseline inspection program inspect this area? did the

SDP accurately characterize resultant findings?). IITs already have the provision

to determine NRC program deficiencies. AITs will be reviewed by NRR/DIRS to

identify any weaknesses.

Criteria: Expect no major programmatic voids.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: No IITs nor AITs were conducted during the 2005 ROP cycle. Additionally, no

feedback forms were received for IP 93800.

Metric Criterion Met: Yes. This metric met its criteria based on no current programmatic 

voids.
-60-

O-12 Analysis of Significant Events

Definition: Annually review all accident sequence precursor (ASP) events that have a risk

significance of more than 10-6 to identify any ROP programmatic voids (i.e., did

the baseline inspection program inspect this area? did the SDP accurately

characterize resultant findings?).

Criteria: Expect no major programmatic voids.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: The Office of Nuclear Regulatory Research (RES) compared ASP results and

SDP evaluations for ASP analyses completed during this assessment period. A

total of 17 greater than green inspection findings were reviewed. No significant

differences between the SDP findings and the ASP results were noted and no

programmatic voids were identified.

Metric Criterion Met: Yes. This metric met its criteria based on no identified programmatic

voids.
-61-

O-13 Analysis of Inspection Hours and Resource Expenditures

Definition: Annually, collect and analyze resource data (e.g., direct inspection effort,

preparation/documentation, plant status hours) for Baseline, Supplemental/PlantSpecific, and Safety Issues Inspections, and other ROP activities.

Criteria: (1) Significant deviations are not expected on an annual basis. Explore

reasons for any deviations that may be evident.

(2) Track and trend resource usage for the baseline inspection program and

supplemental/plant-specific inspections. Analyze causes for any

significant departure from established trend.

(3) Track and trend resource usage for preparation, documentation, and

other ROP activities, and assess the effects on budgeted resources.

NOTE: This metric is intended primarily for tracking and trending resource usage for the

ROP. The results are used to improve the efficiency and effectiveness of the

ROP and to make management and budget decisions.

Goals Supported: Ensure Effectiveness, Predictable

Analysis: Overall staff effort in 2005 was 5.4 percent higher compared with 2004. An

increase was evident in all areas of the ROP except for performance assessment

which has remained relatively constant during the past few years. 

Baseline inspection effort in 2005 increased 6.1 percent compared with 2004. 

This increase was generally evenly distributed among all baseline procedures. 

The largest increase was in plant status activities due to increased requirements

in this area for daily corrective action review and Reactor Coolant System

leakage trend reviews resulting from Davis-Besse lessons learned. Overall effort

for plant-specific inspections remained unchanged at approximately 22,700

hours. The only area where a significant increase was seen was in the 2005

inspection effort for generic and safety issues. This increase was the result of

the high level of inspection activity associated with temporary instructions

continuing from 2004 into 2005, primarily in the area of safeguards, grid

reliability, and material control and accountability.

The increased inspection effort in 2005 was most likely the result of increased

regional inspection activity due to additional requirements that have been

imposed on the inspection staff in recent years. These additional requirements

include corrective action reviews, activities resulting from Davis-Besse lessons

learned, increased generic safety issues inspections, and increased efforts in the

areas of safety culture, security, performance indicators, and inspection

procedure development. The staff intends to further investigate the reasons for

the inspection resource increase over the past few years. A detailed ROP

resource analysis is included in Enclosure 8 of the CY 2005 ROP

Self-Assessment Commission paper.

Metric Criterion Met: Yes. This metric met its program expectations.
-62-

O-14 Analysis of Resident Inspector Demographics and Experience

Definition: Annually, collect and analyze data in order to determine the relevant inspection

experience of the resident inspector (RI) and senior resident inspector (SRI)

population. The following five parameters will be measured and analyzed for

both RIs and SRIs to ensure that the NRC maintains a highly qualified resident

inspection staff:

(1) NRC time - the total time the individual has accumulated as an NRC

employee.

(2) Total resident time - the total time the individual has accumulated as an

RI or SRI.

(3) Qualified total resident time - the total time the individual has been

assigned to an RI or SRI position after completing the reactor operations

inspector qualification requirements of IMC 1245.

(4) Current site time - the total time the individual has spent as an RI or SRI

at the current site.

(5) Relevant non-NRC experience - the total time the individual has gained

relevant nuclear power experience outside of the NRC.

Criteria: None; trend only. Provide reasons for any meaningful increase or decrease in

these resident demographic metrics.

NOTE: This metric is intended primarily for tracking and trending resident inspection

experience. The results are used to make any necessary modifications to the RI

and/or SRI programs in order to attract and retain highly qualified inspectors to

the respective programs.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: See charts on following pages. A detailed resident demographic and staffing

analysis is included in Enclosure 9 of the CY 2005 ROP Self-Assessment

Commission paper.

Metric Criterion Met: Yes. This metric met its program expectations.
-63-

(1) NRC Time: NRC time for RIs increased in Regions I and II and decreased in Regions

III and IV. NRC time for SRIs increased in Regions I and III and decreased in Regions II

and IV.

NRC Time (RIs) 

0

1

2

3

4

5

6

7

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values

NRC Time (SRIs) 

0

2

4

6

8

10

12

14

16

18

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values
-64-

(2) Total Resident Time: Total resident time for RIs increased for Regions I and II and

decreased in Regions III and IV. Total resident time for SRIs increased in Regions II

and IV and decreased in Regions I and III.

Total Resident Time (RIs) 

0

1

2

3

4

5

6

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values

Total Resident Time (SRIs) 

0

2

4

6

8

10

12

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values
-65-

(3) Qualified Total Resident Time: Qualified total resident time for RIs increased in

Regions I, II, and IV and decreased in Region III. Qualified total resident time for SRIs

increased in Regions II and IV and decreased in Regions I and III.

Qualified Total Resident Time (RIs)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values

Qualified Total Resident Time (SRIs)

0

2

4

6

8

10

12

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values
-66-

(4) Current Site Time: Current site time for RIs increased in Regions I and II and

decreased in Regions III and IV. Current site time for SRIs increased in all the Regions.

Current Site Time (RIs)

0

1

2

3

4

5

6

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values

Current Site Time (SRIs) 

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values
-67-

Relevant Non-NRC Time (RIs)

0

2

4

6

8

10

12

14

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values

(5) Relevant Non-NRC Experience: Relevant non-NRC experience increased or remained

the same for RIs in all Regions with the exception of Region III. Relevant non-NRC

experience for SRI’s increased in all of the regions.

Relevant Non-NRC Time (SRIs)

0

1

2

3

4

5

6

7

8

9

10

2001 2002 2003 2004 2005

Year

Years (median values)

Region I Region II Region III Region IV National Medain Values
-68-

O-15 Analysis of Site Staffing

Definition: Semiannually, collect and analyze data in order to measure the permanent

inspector staffing levels at each of the reactor sites for both RIs and SRIs in order

to evaluate the agency’s ability to provide continuity of regulatory oversight. 

Criteria: None; trend only. Provide reasons for any meaningful increase or decrease in

the inspector staffing level at reactor sites.

NOTE: This metric is intended primarily for tracking and trending the staffing levels of

RIs and SRIs.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: The metric counts (1) RIs and SRIs who are permanently assigned to the site

and (2) inspectors who are on rotational assignments to the site for 6 weeks or

longer. Only inspectors who have attained at least a basic inspector certification

status, as defined by IMC 1245, are counted. The metric does not count

permanently assigned RIs and SRIs who are away from their sites for longer than

6 weeks. Inspectors who are assigned to sites for less than 6 weeks are not

counted towards satisfying the metric.

The success criteria for the metric is 90 percent coverage for each site. This

year, the average site coverage for the regions was 98.2 percent, with all regions

exceeding 97 percent. However, three sites did not meet the success criteria of

90 percent (their scores were 85 percent and higher), primarily because of the

assignment of the SRI to special work. The staff’s evaluation determined that

oversight continuity was maintained and that these three sites were adequately

covered for one or more of the following reasons: (1) the permanent RI was

present, (2) an experienced SRI was temporarily assigned, and/or (3) the site

was covered with qualified inspectors on assignments for less then 6 weeks. At

no time did these sites remain without qualified inspectors. It is important to note

that for all three of these sites the permanent SRIs were temporarily assigned for

periods of up to 5 months to participate in projects related to nuclear security.

Metric Criterion Met: Yes. This metric met its program expectations.
-69-

O-16 Analysis of ROP Training and Qualifications

Definition: Annually, evaluate the implementation of IMC 1245, “Qualification Program for

the Office of Nuclear Reactor Regulation Programs,” particularly as it pertains to

ROP implementation.

Criteria: None; trend only. Summarize and evaluate the training accomplished over the

previous year and propose program improvements as necessary to address

noted concerns.

NOTE: This metric is intended primarily for tracking and trending the effectiveness of the

ROP training and qualifications programs.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff continued its efforts to improve the inspector training programs and

techniques in accordance with IMC 1245. During CY 2005, the staff developed

and distributed the first survey on inspector training effectiveness. The

responses to the survey were favorable and indicated that regional managers

have maintained an environment that encourages inspectors to identify

performance deficiencies and improve the inspection program. The staff

developed and implemented Web-based read-and-sign training on substantive

cross-cutting issues, documenting findings in inspection reports, and revisions to

the SDP guidance. An evaluation of training effectiveness is included in

Enclosure 6 of the CY 2005 ROP Self-Assessment Commission paper.

Metric Criterion Met: Yes. This metric met its program expectations.
-70-

O-17 Analysis of Regulatory Impact

Definition: Annually, collect and analyze licensee feedback and develop a summary of

regulatory impact forms that are critical of the ROP.

Criteria: None; trend only. Summarize and evaluate the feedback received and propose

program improvements as necessary to address common concerns.

NOTE: This metric is intended primarily for tracking and trending regulatory impact. 

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff receives and evaluates feedback from licensees on an annual basis as

part of the regulatory impact process, established in 1991 based on Commission

direction to develop a process for obtaining feedback from licensees and

reporting the feedback to the Commission. Over the past year, the staff received

feedback from 91 reactor licensees on 253 issues. The staff also received

feedback from the Regulatory Information Conference in March 2005. Of the

comments received, 83 percent were favorable and 17 percent were unfavorable. 

The comments fell into three main categories: formal communication with

licensees, inspector performance, and security and safeguards activities. A

summary of the feedback received, the staff's evaluation, and the proposed

improvement actions are provided in Enclosure 7 of the CY 2005 ROP

Self-Assessment Commission paper.

Metric Criterion Met: Yes. This metric met its program expectations.