March 26, 2007

MEMORANDUM TO: Marc L. Dapas, Deputy Regional Administrator, RI

Victor McCree, Deputy Regional Administrator, RII

Geoffrey E. Grant, Deputy Regional Administrator, RIII

Thomas P. Gwynn, Deputy Regional Administrator, RIV

FROM: Elmo E. Collins, Director/RA/

Division of Inspection and Regional Support

Office of Nuclear Reactor Regulation

SUBJECT: REACTOR OVERSIGHT PROCESS ANNUAL SELF-ASSESSMENT 

PERFORMANCE METRICS

The Reactor Oversight Process (ROP) self-assessment process uses objective measures and

predetermined criteria to monitor the performance of the ROP as described in Inspection

Manual Chapter (IMC) 0307, “Reactor Oversight Process Self-Assessment Program.” These

metrics rely on information from various sources, including the Reactor Program System (RPS),

the inspection program, periodic independent audits, stakeholder surveys, and public

comments. The staff collects data quarterly and uses preestablished success criteria to

analyze the data. 

The Nuclear Regulatory Commission solicited comments on the seventh year of ROP

implementation from external stakeholders in a Federal Register notice (FRN) in October 2006. 

Of the 16 responses, eight were from the utilities and their representatives, while three were

from State and other government entities, and five were from public interest groups or public

citizens. Along with the external survey, the staff conducted an internal survey in October 2006

which solicited and analyzed stakeholder feedback regarding the effectiveness of the ROP. A

total of 266 responses were received from internal stakeholders, including resident/senior

resident inspectors, regional-based inspectors and staff, senior reactor analysts, regional and

headquarters line management, and headquarters technical and program staff.

Based on our review, most of the metrics met their established criteria. Specifically, all the

metrics in the inspection (designated as the IP metrics) and assessment (AS) areas met the

criteria, but a few metrics in the performance indicator (PI) and significance determination

process (SDP) areas did not. In addition, all “Overall ROP” (O) metrics met their criteria. The

staff’s corrective actions to address these issues are discussed in the following paragraphs, in

the attached metric analyses, and in the CY 2006 ROP self-assessment Commission paper.

CONTACT: Bart Fu, NRR

301-415-2467
-2-

PI Program Results

In reviewing the data for this reporting period, the staff found that two of the eight PI metrics did

not meet the established criteria. The metric regarding whether the PI program provides

insights to help ensure plant safety (PI-4) did not meet its criteria because internal, public, and

state respondents gave significant feedback that the PIs do not provide an adequate indication

of declining safety performance and do not enhance public confidence. In addition, a second

metric (PI-8) did not meet its criteria because a significant number of the internal, public, and

state respondents did not concur that the PI program can effectively identify performance

outliers. The staff recognizes the need to improve the PI Program and is reviewing and revising

several PIs to provide more meaningful indications of declining plant performance.

SDP Results

Of the seven official metrics for the SDP, one did not meet the established criteria. The metric

regarding whether results of the same color are perceived by the public to warrant the same

level of regulatory attention for all cornerstones (SDP-4), did not meet its criteria based on a

stable negative perception by external stakeholders over the past six years of ROP

implementation. The staff continues to believe that relative parity has been achieved among

the cornerstones, based on the potential impact on public health and safety and the designated

NRC response to specific findings. Findings are continuously under review by the staff to

determine the need for adjustments to the SDPs in this area.

There has been significant improvement in the SDP metrics because six out of seven metrics

were met during this assessment period, compared to five of nine metrics being met during the

previous period in 2005. Of note, Metric SDP-6a (Final Significance Determinations Are

Timely) was met based on the fact that performance exceeded program expectations for the

first time since ROP implementation.

Inspection Program Results

All ten metrics in the inspection area met the established criteria. This is an improvement from

the last self-assessment (CY 2005), when the temporary instruction (TI) timeliness metric (IP-5)

failed to meet the established criteria of completing all TIs within the timeliness requirements.

Assessment Program Results

All of the assessment program metrics met program expectations (ten out of ten, one was not

applicable). This is an improvement from CY 2005, when the number of Action Matrix

deviations metric (AS-1) did not meet its criteria.
-3-

Overall ROP Results

All 17 metrics in the Overall ROP area met established program expectations.

Conclusions and Next Steps

The performance metrics provide the staff with valuable insights and lessons learned that lead

to continued improvements in ROP effectiveness. This report provides a significant input into

the annual ROP self-assessment and the resulting Commission paper. Aspects of this report,

particularly missed metrics, will be discussed in the self-assessment paper under the respective

program areas.

Enclosure: As stated
-3-

Overall ROP Results

All 17 metrics in the Overall ROP area met established program expectations.

Conclusions and Next Steps

The performance metrics provide the staff with valuable insights and lessons learned that lead

to continued improvements in ROP effectiveness. This report provides a significant input into

the annual ROP self-assessment and the resulting Commission paper. Aspects of this report,

particularly missed metrics, will be discussed in the self-assessment paper under the respective

program areas.

Enclosure: As stated

DISTRIBUTION:

DRP/DRS

ADAMS ACCESSION NUMBER: ML070720085

OFFICE NRR/DIRS/IPAB NRR/DIRS/IPAB NRR/DIRS/IRIB NRR/DIRS/IPAB NRR/DIRS

NAME Rfrahm for

ZBFu

RFrahm RGibbs JAndersen SRichards

DATE 3/19/2007 3/19/2007 3/20/2007 3/20/2007 3/26/2007

OFFICIAL RECORD COPY
-1- Enclosure

PI-1 Consistent Results Given Same Guidance

Definition: Independently verify PIs using Inspection Procedure (IP) 71151, “PI Verification.”

Count all performance indicators (PI) that either (a) result in a crossed threshold

based on a data correction by the licensee (as noted in the resultant inspection

report), or (b) have been determined to be discrepant by the staff in accordance

with IP 71150, “Discrepant or Unreported Performance Indicator Data.”

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Safety

Comments: The graph represents the number of significant deficiencies or discrepant PIs

reported for each quarter. Significant discrepancies are issues identified by the

NRC during a PI verification inspection that caused the PI to cross a threshold.

Analysis: During this assessment period there was one PI that crossed the green/white

threshold based on a data correction by the licensee and two PIs were identified

as discrepant.

The NRC identified an issue with reporting of a PI during the Temporary

Instruction inspection for the mitigating systems performance index (MSPI). As a

result, the Perry Nuclear Power Plant made a data correction report during the

4 quarter of 2006, to change the 2 quarter 2006 PI for Emergency AC Power th nd

from Green to White. However, a frequently asked question has been submitted

by the licensee to the PI program that questions the basis for the threshold

change. This FAQ is under review.

The NRC also identified two discrepant safety system PIs at Waterford 3 in the

fourth quarter of 2005 and performed a discrepant PI inspection (letter dated

August 30, 2006). The NRC determined that one train of each of the High

Pressure Safety Injection and the Residual Heat Removal systems were not

available between November 2003 and September 2004. These systems are
-2-

two-train systems. These PIs should not have been reported as Green, but Red

and Yellow respectively. 

There was only one threshold change based on data correction by the licensee

for the year, and that is the subject of a FAQ that is under review. The two

discrepant PIs at Waterford 3 resulted from the mispositioning of one valve in

2003, and the late discovery resulted in high fault exposure. This situation,

although important, had mitigating circumstances. The safety system

unavailability PIs were replaced by the MSPI PIs in the 2 quarter of 2006. As nd

noted in the July 25, 2006 action matrix deviation letter, and the August 30, 2006

letter to the licensee, using the same set of circumstances applied to the MSPI

PIs would have resulted in a Green outcome. This is because of the differences

in the way fault exposure is treated under the former safety system unavailability

(SSU) performance indicators and the current MSPI indicators. One of the noted

problems with the SSU PIs and the use of fault exposure time was that it

overestimated the risk significance of unavailability. Therefore, since the

Waterford 3 issue was not considered risk-significant, and if not counted there

was only one occurrence applicable to this metric in CY 2006, the staff

concluded that this metric was met. However, the two year elevated trend in the

PI data reporting area, and the fact that both 2006 issues were discovered by the

NRC, bears close monitoring going forward.

(Waterford 3 exited the discrepant PI process in the 4 quarter of 2006.) th

Metric Criterion Met: Yes.
-3-

PI-2 Questions Regarding Interpretation of PI Guidance

Definition: Quarterly, count the number of frequently asked questions (FAQs).

Criteria: Expect low numbers, with a stable or declining trend.

Goals Supported: Understandable, Risk-Informed, Predictable

Comments: Each quarter represents the total number of new FAQs introduced and approved

during the ROP NRC/Industry Working Group meetings held during the

respective quarter.

Analysis: For this assessment period, the number of unresolved interpretation questions

has remained low compared to historical trends. However, the last three

quarters have shown an increasing trend in open FAQs. The staff closed a

number of FAQs in January 2007. At the present time, this metric meets its

criteria based on the fairly low number of open interpretation issues compared to

historical patterns.

Metric Criterion Met: Yes.
-4-

PI-3 Timely Indication of Declining Safety Performance

Definition: Quarterly, track PIs that cross multiple thresholds (e.g., green to yellow or white

to red). Evaluate and characterize these results to allow timely indication of

declining performance.

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Ensure Safety, Risk-Informed, Ensure Effectiveness

Analysis: There were two occurrences of PIs crossing multiple thresholds during this

assessment period. 

As noted in metric PI-1, the NRC identified two discrepant safety system PIs at

Waterford 3 in the fourth quarter of 2005 and performed a discrepant PI

inspection (letter dated August 30, 2006). The NRC determined that one train of

each of the High Pressure Safety Injection and the Residual Heat Removal

systems were not available between November 2003 and September 2004. 

These PIs should not have been reported as Green, but Red and Yellow

respectively. The late discovery of the valve the mispositioning resulted in

elevated fault exposure which resulted in the crossing of multiple thresholds. As

described earlier (see PI-1), one of the noted problems with the SSU PIs and the

use of fault exposure time was that it overestimated the risk significance of the

unavailability. This issue was corrected with the implementation of MSPI.

Therefore, this issue is not considered significant enough to result in not meeting

this metric.

Metric Criterion Met: Yes.
-5-

PI-4 PI Program Provides Insights to Help Ensure Plant Safety

Definition: Survey external and internal stakeholders asking whether the PI Program

provides useful insights to help ensure plant safety.

Criteria: Expect a low number of negative comments, with a stable or declining trend.

Goals Supported: Ensure Safety, Ensure Effectiveness, Risk-Informed

Analysis: Internal Survey

Several internal survey questions addressed this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

PIs provide useful information on risksignificant areas.71% 67% 70% 79%

PIs help to maintain safety. 71% 68% 68% 72%

PIs provide an adequate indication of

declining safety performance.

58% 45% 43% 53%

PIs enhance public confidence. 56% 57% 60% 65%

Internal stakeholders mostly agree that the PI program helps provide useful risk

insights and maintain safety. However, just over half of the respondents feel the

PI program is able to provide an adequate indication of declining safety

performance. There is also a notable adverse trend regarding the staff’s view of

the public’s confidence in the PI program. Many comments noted that threshold

for the PIs is too high to identify declining performance and, because the PIs are

almost always green, they do not enhance public confidence. 

External Survey

Responses to the external survey question show a wide divergence of opinion. 

Responses from the public ranged from strongly agree to strongly disagree

whereas the feedback from the states was neutral. However, both the public

and states singled out the inability of the emergency planning PIs to garner

public confidence. As noted in the internal survey, public and state respondents

also questioned whether the thresholds in the PI program are set correctly

because the PIs are always “green” and do not provide insights into plant safety. 

State feedback also noted that when thresholds were crossed the staff is too

lenient and allows the color to go back to “green.” Industry responses were

generally in agreement that the PI program does provide useful insights to

ensure plant safety.

This metric did not meet its criteria because internal, public, and state

respondents gave significant feedback that the PIs do not provide an adequate

indication of declining safety performance and do not enhance public confidence.
-6-

As noted in last year’s self assessment, the staff recognizes the need to improve

the PI Program to provide more meaningful indications of declining plant

performance.

Metric Criterion Met: No.
-7-

PI-5 Timely PI Data Reporting and Dissemination

Definition: Within 5 weeks of the end of each calendar quarter, track (count) late PI

postings on the NRC’s external Web site. Also note the number of late

submittals from licensees that did not meet the 21-day timeliness goal.

Criteria: Expect few occurrences, with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: There were no late postings on the NRC’s external Web site. 

Six power plants provided PI submittals that were late; however, the submittals

were made within a few days after the date required by NEI 99-02, “Regulatory

Assessment Performance Indicator Guideline.” The late PI data submittals from

the licensees did not impact the NRC’s ability to post the results on the web page

in a timely manner; however, they did impact the schedule for regional staff to

prepare for the mid-cycle and end-of-cycle assessment meetings. This adverse

trend was raised with the industry representatives to reinforce the guidance

contained in their document.

The criteria for this metric has been met because there have been no late PI

data postings on the NRC's external web site since the inception of the ROP. 

The increased frequency of late submittals are a concern and will continue to be

tracked to determine if the trend requires further attention.

Metric Criterion Met: Yes.
-8-

PI-6 Stakeholders Perceive Appropriate Overlap Between the PI Program and

Inspection Program

Definition: Survey external and internal stakeholders asking if appropriate overlap exists

between the PI program and the inspection program.

Criteria: Expect a low number of negative comments, with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Ensure Safety, Ensure Openness

Analysis: Internal Survey

One internal survey question addressed this metric. The question and it’s

resultant percentage of agreement is presented below.

Measure 2006 2004 2002 2001

PIs provide an appropriate level of

overlap with inspection program.

78% 78% 74% 74%

Internal stakeholders continued to generally agree that the PI provides an

appropriate level of overlap with the inspection program. The data indicates a

stable or slightly improving historical trend.

External Survey

Public response varied from agreement to disagreement regarding proper

overlap between the PI and inspection programs. However public response

again noted that the PIs are not effective. State response indicated that there is

appropriate overlap but the NRC should periodically re-evaluate the PI and

inspection programs to ensure proper focus and effectiveness, and that where

PIs are not effective, inspection should be performed.

Industry comments noted appropriate overlap overall, but added that

improvement can be made in the security area and that the NRC should evaluate

inspection findings that overlap PIs.

Metric Criterion Met: Yes.
-9-

PI-7 Clarity of Performance Indicator Guidance

Definition: Survey external and internal stakeholders asking if NEI 99-02, “Regulatory

Assessment Performance Indicator Guideline,” provides clear guidance

regarding performance indicators.

Criteria: Expect a low number of negative comments or examples of interpretation issues,

with a stable or declining trend in the number of negative comments received.

Goals Supported: Understandable, Ensure Openness, Objective

Analysis: Internal Survey

Two internal survey questions addressed this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

PIs are clearly defined. 82% 79% 71% 72%

PIs are understandable. 82% 87% 76% 78%

Most respondents felt the PIs are clearly defined and understandable. The

exception appears to be MSPI, based on a number of comments noting that the

MSPI is confusing, hard to understand, and difficult to inspect.

External Survey

Public response varied widely and comments again noted the ineffectiveness of

the PI program which resulted in the PI-4 metric not being met. State

stakeholders generally felt that the guidance was clear but that it would be more

appropriate for the licensees to provide comments on effectiveness of the PI

guidance. 

Utility group respondents commented that the PI guidance is clear and that the

FAQ process is responsive in addressing questions.

This metric meets its criteria based predominantly on internal and industry

feedback. Based on comments, MSPI guidance needs improvement and this is

on-going.

Metric Criterion Met: Yes.
-10-

PI-8 PI Program Identifies Performance Outliers In an Objective and Predictable

Manner

Definition: Survey external and internal stakeholders asking if the PI program can

effectively identify performance outliers based on risk-informed, objective, and

predictable indicators.

Criteria: Expect a low number of negative comments, with a stable or declining trend.

Goals Supported: Risk-Informed, Objective, Predictable

Analysis: Internal Survey

This is a new metric that was added to get feedback on the effectiveness of the

PI program in general. One internal survey question addressed this metric; its

percentage of agreement is presented below.

Measure 2006 2004 2002 2001

PIs, including MSPI, can effectively identify

performance outliers based on riskinformed, objective, and predictable

indicators

61% N/A N/A N/A

Many comments noted that the thresholds are too high and that a number of

indicators have never triggered. Responses on MSPI were mixed, some felt it

was an improvement or noted that it is too early to judge MSPI, others remained

concerned because it is so complex that it lends itself to possible manipulation

that inspectors cannot readily identify.

External Survey

Comments from the public clearly state that the PI program cannot identify

outliers, especially the MSPI indicator because it can be manipulated and is

difficult to understand. State stakeholders noted that the PI program is always

green and, as noted earlier, the thresholds may need re-examination. They

noted, however, that the MSPI was a positive step being risk based and

incorporating unavailability and unreliability, but is dependent on the quality of

the licensee’s risk assessment. Industry comments were favorable about the PI

program and in particular regarding MSPI because it is risk based.

This metric does not meet its criteria because a significant number of the internal

survey respondents did not concur that the PI program can effectively identify

performance outliers. In addition, the public and state stakeholders both noted

that the PI program does not adequately identify outliers. All three groups

expressed concern that MSPI is too complex and subject to licensee risk

assessment quality. The staff is in the process of reviewing and revising several

PIs to provide more meaningful indications of plant performance and to better

identify performance outliers, including unplanned scrams with loss of normal
-11-

heat removal, reactor coolant system leakage, safety system functional failures,

and others in the emergency preparedness cornerstone. 

Metric Criterion Met: No.
-12-

IP-1 Inspection Findings Documented In Accordance With Requirements

Definition: Audit inspection reports in relation to program requirements (IMC 0612, “Power

Reactor Inspection Reports”) for documenting green findings, greater-than-green

findings, and violations. Report the percentage of findings that meet the

program requirements.

Criteria: Expect a stable or improving trend in the percentage of findings documented in

accordance with program requirements.

Goals Supported: Objective, Risk-Informed, Predictable

Comments: The graph represents the cumulative average for the sample of inspection

reports reviewed by NRR during CY 2006. 

Analysis: The staff audited integrated inspection reports from each branch and a number

of team inspection reports from each region. Of the 733 inspection reports

issued in CY 2006, 46 were included in this audit. The percentage of findings

documented in accordance with IMC 0612 requirements for the sample audited

was 97 percent. This represents a slight rise in compliance with IMC 0612

requirements from CY 2005. Overall, the inspection reports were well written. 

Metric Criterion Met: Yes.
-13-

IP-2 Number of Feedback Forms per Document

Definition: Count the number of feedback forms received for each program document each

quarter. Use a histogram to chart the number of documents for which feedback

forms were received. Highlight those documents against which the most forms

are written.

Criteria: Expect a declining trend in the number of feedback forms received for program

documents.

Goals Supported: Understandable, Predictable, Objective

Analysis: The staff received 99 feedback forms in CY 2006. Approximately 60 percent of

all feedback forms received during this period relate to issues in the areas of:

(1) IMC 0612, Power Reactor Inspection Reports - (13 Forms), 14%;

(2) IMC 0305, ROP Assessment - (10 Forms), 11%;

(3) IMC 0609F and 71111.05, Fire Protection SDP and Inspection -

(9 Forms), 9%;

(4) IMC 0609, Significance Determination Process (SDP) including all

Appendices except Fire - (10 Forms), 9%;

(5) IP 71152, Problem Identification and Resolution - (8 Forms), 8%;

(6) IMC 2515, LAR Inspection Program - Operational Phase (7 Forms),

7%; and

(7) IP 71151, Performance Indicator Verification - (5 Forms), 5%. 
-14-

Of the 99 feedback forms received, the staff resolved 62. Overall, the staff

resolved 106 feedback forms from a total of 157 feedback forms (~ 68%). The

total number of feedback forms includes 58 feedback forms brought forward

from CY 2005. Of the 58 older forms, the staff closed 44 forms to date (~76%).

There are 51 feedback forms currently open. This number includes the

remaining 37 forms received this year and 14 forms carried over from CY 2005. 

The breakdown in age of these forms are as follows:

CY 2006 (37)

- 10 forms are less than 90 days old, 

- 8 forms are between 90 -180 days old, 

- 19 forms are between 180 - 360 days old,

CY 2005 (14)

- 14 forms are greater than 360 days old.

Sixty-two percent of the 106 feedback forms were resolved in CY 2006 resulting

in changes to inspection program documents. Forty forms were addressed by

procedure revisions this year and 32 forms are pending a change notice. 

The number of feedback forms received in CY 2006 (99 forms) indicates a slight

downward trend in the number of feedback forms compared to the previous

three years (102 for CY 2005, 114 for CY 2004, and 123 for CY 2003).

The staff enhanced the ROP Feedback Program to improve the timeliness,

efficiency, and effectiveness of feedback resolution. The revised process, which

was implemented at the beginning of FY 2007, assigns action to the staff via a

Green Sheet issued through the NRR Work Planning and Control Center to track

the response for resource usage and timeliness. Improvements in staff

performance and responsiveness to feedback are expected in 2007. 

Metric Criterion Met: Yes.
-15-

IP-3 Completion of Baseline Inspection Program 

Definition: Annual completion of baseline inspection program.

Criteria: Defined as per IMC 2515, “Light-Water Reactor Inspection Program - Operations

Phase.”

Goals Supported: Ensure Safety, Predictable, Ensure Effectiveness

Analysis: All four regions completed their baseline inspections in CY 2006. Each region

documented completion of the program in a memorandum to the Division of

Inspection and Regional Support in NRR. These memoranda can be found in

ADAMS under ML070430041 (Region I), ML070330047 (Region II),

ML070470661 (Region III), and ML070470659 (Region IV). As in the 2005

inspection cycle, all regions completed their baseline inspections in 2006 with the

allocated regional resources. 

Metric Criterion Met: Yes.
-16-

IP-4 Inspection Reports Are Timely

Definition: Obtain RPS data on the total number of reports issued and the number issued

within timeliness goals as stipulated in IMC 0612, “Power Reactor Inspection

Reports.”

Criteria: Expect 90 percent of inspection reports to be issued within the program's

timeliness goals.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: A total of 733 inspection reports were issued during the CY 2006. Regions met

or exceeded the inspection report timeliness goal of 90 percent in each quarter

throughout the year. Overall as an inspection program, about 99 percent of all

issued inspection reports were timely.

Metric Criterion Met: Yes.
-17-

IP-5 Temporary Instructions (TIs) Are Completed Timely

Definition: Audit the time to complete TIs by region. Compare the completion status in RPS

to TI requirements. Report by region the number of TIs closed within goals.

Criteria: Expect all TIs to be completed within TI requirements.

Goals Supported: Ensure Effectiveness, Ensure Safety, Predictable

Analysis: All Regions completed all 5 TIs on time. This metric was not met in CY 2005;

however, timeliness of TIs in CY 2006 improved and this metric was met. The

staff is planning to change this metric to a percentage goal in CY 2007, vice

expecting all TIs to be completed on time. 

Metric criterion Met: Yes.
-18-

IP-6 Public Availability of Inspection Information Is Timely

Definition: NRR/DIRS posts inspection reports to the NRC's external (public) Web site

within ROP timeliness goals using the electronic version of inspection reports

entered into the Agency Document Access and Management System (ADAMS)

by the regions. NRR/DIRS also posts entries from the Plant Issues Matrix (PIM)

to the NRC's public Web site using data entered into RPS by the regions. In

addition, NRR/DIRS records the number of inspection reports not available in

ADAMS and the number of PIM entries not updated in RPS, as well as the

number of inspection reports and PIMs that are not posted to the NRC's public

Web site within the goals stipulated in IMC 0306, “Information Technology

Support for the Reactor Oversight Process.”

Criteria: Expect few untimely postings of PIMs or inspection reports, with a stable or

declining trend.

Goals Supported: Ensure Openness, Ensure Effectiveness. Predictable

Analysis: PIMs and inspection reports were posted within timeliness goals nearly 100% of

the time for each region in each quarter of CY 2006. The untimely postings

resulted in broken links on the web for a few days, but these rare occurrences

were isolated and the reports were quickly posted once they were located. 

Beginning with the third quarter of 2006, the staff implemented an automated

process to post inspection reports that works directly with ADAMS, replacing the

cumbersome manual process used previously. As a result, there were no late

postings in the final two quarters of 2006 and the staff expects few, if any, late
-19-

postings of inspection reports in the future, except perhaps for those reports that

are not issued within timeliness goals. Since metric IP-4 already tracks the

timeliness of issuing inspection reports, the staff plans to discontinue this metric

in CY 2007 and delete it from the ROP self-assessment program as described in

IMC 0307. 

Metric Criterion Met: Yes.
-20-

IP-7 Public Communication Is Accurate

Definition: Each calendar quarter, sample information on the NRC's external (public) Web

site and count the number of times and reasons for regions changing PIMs or

inspection reports (i.e., inaccuracy, new information).

Criteria: Expect few inaccuracies, with a stable or declining trend.

Goals Supported: Ensure Openness, Ensure Effectiveness, Understandable

Analysis: There were few inaccurate postings of PIM entries or inspection reports on the

web identified during CY 2006. No region had more than two inaccurate

postings for any of the quarters. This is a declining trend from CY 2005. 

Metric criterion Met: Yes.
-21-

IP-8 Inspection Reports Are Relevant, Useful, and Written in Plain Language

Definition: Survey external and internal stakeholders asking whether the information

contained in inspection reports is relevant, useful, and written in plain English.

Criteria: Trend average level of agreement.

Goals Supported: Ensure Effectiveness, Understandable, Ensure Openness

Analysis: Internal Survey

Three internal survey questions addressed this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP generally provides appropriate

communication effectiveness through use

of plain English in official correspondence

(e.g., inspection reports, ...)

82% 79% 74% 74%

The information on plant performance

(e.g., inspection reports, ...) provided on

the ROP Web page is understandable and

written in plain English

93% 89% 87% 89%

The baseline inspection program reports

are communicated accurately.

96% 87% 93% 89%

The majority of those who provided feedback on inspection report relevance,

usefulness, and if they are written in plain language responded that the reports

are useful and were clearly written. The three questions all showed an

improvement from 2004 and all are above 82%. This shows strong agreement

among internal stakeholders about the relevance and usefulness of inspection

reports.

External Survey 

The majority of those who provided feedback to the question on whether the

information in the inspection reports was relevant, useful, and written in plain

language responded that the inspection reports were clearly written and useful. 

There were, however, comments for improvement in this area, including:

• The referenced document section in the back of the inspection report is

irrelevant. These are documents that are not public so they provide no

real value in the report.

• Inspection report information is generally useful and the organization

helps to provide focus in problem areas. However, there is some

disagreement about whether the reports are written in “plain English.” It is

also noted that the link between an inspection finding and a cross-cutting
-22-

aspect is not always clearly articulated in the inspection reports. Often,

the inspection report language only states that the finding is related to the

cross-cutting aspect with no explanation of how the cross-cutting aspect

significantly contributed to the cause of the finding. The link between the

cross-cutting aspect and the cause of the finding should be clear to the

reader of the inspection report.

• Generally, the reports are relevant, useful and written in plain English.

We note, however, that the reports are growing in size and detail with

minimal added value. Preliminary experience with the NRC’s Safety

Culture initiative, indicates cross-cutting aspects associated with

inspection findings are appropriately documented. 

Overall, however, stakeholder satisfaction was generally favorable and

consistent.

Metric Criterion Met: Yes.
-23-

IP-9 Inspection Program Effectiveness and Adequacy in Covering Areas

Important to Safety

Definition: Survey external and internal stakeholders asking whether the inspection program

adequately covers areas that are important to safety and is effective in

identifying and ensuring the prompt correction of performance deficiencies.

Criteria: Trend average level of agreement.

Goals Supported: Ensure Safety, Ensure Effectiveness, Risk-Informed

Analysis: Internal Survey

Seven internal survey questions address this metric. These questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

Baseline Inspection Program appropriately

inspects for and identifies risk-significant

issues

89% 79% 73% 78%

Level of effort for conducting each

inspection is consistent with that estimated

in the inspection procedure

65% 57% 58% 47%

Baseline Inspection Program provides

appropriate coverage of plant activities and

operations important to safety

83% 77% 67% 63%

Procedures are adequate to address

intended cornerstone attributes

94% 86% 80% 81%

Procedures are clearly written 85% 73% 78% 75%

Procedures adequately sample risk

significant aspects of each inspectable

area

87% 80% 72% 76%

Procedures are conducted at an

appropriate frequency 86% 84% 79% 73%

Internal stakeholders continue to generally agree that the inspection program

and its procedures provide appropriate coverage of areas important to safety

and the program is effective. The data supporting this metric indicates a stable

and slightly increasing positive perception for these seven measures when

compared to the previous surveys.

Some of the more prevalent comments on the inspection program were that

inspection procedure scope and level of effort need to be reviewed and adjusted

as appropriate and that issues that screen out as minor are often important and

should be monitored.
-24-

External Survey 

The responses to whether the inspection program adequately covers areas that

are important to safety and is effective in identifying and ensuring the prompt

correction of performance deficiencies were generally positive with comments for

making improvements. Comments included:

• The inspection program does adequately cover areas important to safety

and does identify the issues that require prompt correction. However, the

process also identifies performance deficiencies that do not require

prompt correction since they are not risk significant, and the inspectors

are identifying and documenting issues that are not risk significant. The

NRC should be watchful about regulating excellence instead of

compliance.

• While the inspection program adequately covers areas important to

safety and identifies the issues that require prompt correction, the

process also identifies issues that are not risk significant. Periodic

reviews of the overall effectiveness of certain inspection modules should

be performed. The reviews should consider the resources spent on the

inspection against the numbers, and level of significance of, findings from

the performance of the inspections covered under certain inspection

modules. These reviews could be used to reallocate inspection resources

to areas of greater risk significance. 

• The NRC Inspection Program is intended to cover areas that are

important to safety, but there are opportunities for further improvements.

1. There is extensive use of resources on the part of the NRC staff and

the licensees to assess the significance of inspection findings, specifically

for greater than “Green” findings. This also applies to those findings that

are not of safety significance.

 2. The number of findings in the cross-cutting areas (human

performance, safety culture and problem identification and resolution) is

relatively high. Additionally, there are some plants that are experiencing

relatively high number of “substantive” cross-cutting issues. The

effectiveness of the ROP Inspection Program as it relates to the

identification and resolution of cross-cutting issues is subject to further

review.

 3. The role of the NRC in situations that do not involve regulations, but

might involve a performance deficiency should be examined and better

defined. 

4. The NRC should consider more frequent inspection of the licensees’

Corrective Action Program (CAP). This is important considering that the

ROP relies heavily on the CAP for timely resolution of issues or

problems. Additionally, it is clear that the effectiveness of the CAP varies

significantly within the industry and some utilities are not very effective in

this area. 

 

Metric Criterion Met: Yes.
-25-

IP-10 Analysis of Baseline Inspection Procedures

Definition: Annually, review each baseline inspection procedure to determine its

effectiveness and contribution to the overall effectiveness of the baseline

inspection program. The objectives of the review are: (1) to determine if

changes in scope, frequency, or level of effort are needed based on recent

experience, (2) to determine if a change to the estimated hours for completion is

needed, (3) to define or change what constitutes minimum completion of each

inspectable area, if needed, and (4) to critically evaluate all of the inspectable

areas together along with the PI program to ensure that the inspectable areas

are adequately monitored for safety performance. In addition, a more detailed

review and realignment of inspection resources will be performed at least

biennially. The focus of this effort is to adjust existing inspection resources to

improve the effectiveness of the inspection program in identifying significant

licensee performance deficiencies.

Criteria: None; trend only. Summarize and evaluate the individual inspection procedure

reviews and propose program adjustments as necessary to address noted

inefficiencies. Provide basis for any meaningful increase or decrease in

procedure scope, frequency, or level of effort as a result of the review.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff performed its annual review of each baseline inspection procedure for

FY 2006. The period assessed was from October 2005 through September

2006. The focus of the review was to identify potential areas for improvement in

the baseline inspection program and to identify any notable changes in

inspection results. The staff’s annual evaluation of the inspection procedures did

not reveal significant weaknesses in the inspection program’s ability to identify

risk-significant issues. The staff completed the development of a more in-depth

ROP realignment process in late 2006 and plans to perform a more detailed

analysis of the scope and level of effort of each baseline inspection procedure in

CY 2007. Any changes resulting from the ROP realignment review will be

reflected in the baseline inspection program for CY 2008.

Metric Criterion Met: Yes.
-26-

SDP-1 The SDP Results Are Predictable and Repeatable and Focus Stakeholder

Attention on Significant Safety Issues

Definition: Annually, audit a representative sample (up to four per region) of inspection

findings against the standard criteria set forth in IMC 0609, “Significance

Determination Process,” and its appendices. To the extent available, samples

should include potentially greater-than-green findings that were presented to the

Significance Determination Process/Enforcement Review Panel (SERP).

Findings should contain adequate detail to enable an independent auditor to

trace through the available documentation and reach the same significance color

characterization. 

Criteria: The target goal is at least 90% are determined to be predictable and repeatable. 

Any SDP outcomes determined to be non-conservative will be evaluated and

appropriate programmatic changes will be implemented. 

Goals Supported: Ensure Safety, Risk-Informed, Predictable

Note: Inspection findings identified as Green using the SDP phase 1 or initial

screening process were not included in the subject sample size and such

findings were not audited for this period. IMC 0307 will be reviewed to

determine the appropriate audit frequency and sample size for green

findings.

Analysis: The Office of Research (RES) identified a total of 5 greater-than-green

SDP findings for the reporting period. Each finding was assessed using

the risk-informed process detailed in IMC 0609 Appendix A, “Determining

the Significance of Reactor Inspection Findings for At-Power Situations,”

and was assigned a final white color. The review was completed using a

systematic approach (a detailed checklist for each SDP finding). For all 5

of these findings, RES concluded that the documentation was sufficient

to support the overall facility risk (increase in core damage frequency)

conclusions and that the SDP documentation was adequate to support

the final SDP conclusions. In addition, none of the five findings were

found to be non-conservative.

 .

Metric Criterion Met: Yes.
-27-

SDP-2 SDP Outcomes Are Risk-Informed and Accepted by Stakeholders

Definition: Track the total number of appeals of final SDP results.

Criteria: Expect zero appeals of SDP significance that result in a final determination being

overturned across all regions. All successful appeals will be assessed to

determine causal factors and to recommend process improvements.

Goals Supported: Risk-Informed, Objective, Predictable

Analysis: There were five appeals of findings of low to moderate safety significance,

WHITE. The appeals did not result in changes to the final outcome of the

findings.

Metric Criterion Met: Yes.
-28-

SDP-3 Inspection Staff Is Proficient and Find Value in Using the SDP

Definition: Survey internal stakeholders using specific quantitative survey questions that

focus on training, effectiveness, and efficiency.

Criteria: Expect either a stable or an increasingly positive perception of the SDP process

over time.

Goals Supported: Ensure Effectiveness, Understandable, Risk-Informed

Analysis: Nine internal survey questions addressed this metric. These questions and their

resultant percentage of agreement from internal stakeholders are presented

below.

Measure 2006 2004 2002 2001

SDP focuses NRC attention on

safety-significant issues

83% 75% 71% 79%

SDP provides basis for effective

communication of inspection findings to the

Licensee

84% 78% 73% 77%

SDP provides basis for effective

communication of inspection findings to the

public 

73% 60% 60% 59%

SDP provides for consistent results 74% 63% 61% 72%

SDP training is effective 57% 38% 33% N/A

Reactor safety SDPs are easy to use 54% 36% 20% 60%

Non-reactor safety SDPs are easy to use 58% 41% 26% 64%

Program guidance documents are clear 63% 41% 32% N/A

Resource expenditures are appropriate 60% 41% 32% N/A

The survey results indicate that the staff believes the SDP is effective in meeting

important program objectives such as focusing on identifying safety significant

issues and communicating results to the licensees and the public. Inspector

confidence utilizing the SDP tools increased since the last survey. The

increased inspector confidence in these tools may be attributed to significant

improvements made in SDP documents and training. In particular, regional

management has become more involved in training needs as SDP guidance is

revised or developed. The staff anticipates increased inspector proficiency as

inspectors gain more experience with these tools. While the trend is positive,

some areas of concern were noted by internal stakeholders which are addressed

in Enclosure 1 to the annual ROP self-assessment Commission paper.

Metric Criterion Met: Yes.
-29-

SDP-4 Results of the Same Color Are Perceived by the Public to Warrant the Same

Level of Regulatory Attention for All Cornerstones

Definition: Survey external and internal stakeholders asking if the SDP yields an

appropriate and consistent regulatory response across all ROP cornerstones.

Criteria: Expect stable or increasingly positive perception of the SDP over time.

Goals Supported: Understandable, Objective, Predictable

Analysis: Internal Survey

Five internal survey questions addressed this metric. These questions and their

resultant percentage of agreement from internal stakeholders are presented

below.

Measure 2006 2004 2002 2001

SDP results correctly characterize the risksignificance of inspection findings76% 66% 61% 71%

SDP results are verifiable 85% 76% 76% 84%

SDP results are realistic 78% 69% 62% 70%

SDP results are based upon clear

standards

69% 56% 46% 53%

SDP results are accurate 75% 66% 59% 65%

The internal survey results indicate a steady long term improvement in this area.

External Survey

Generally, the external survey respondents remain steadily negative on the SDP

yielding consistent regulatory response across all ROP cornerstones. The staff

continues to believe that relative parity has been achieved among the

cornerstones, based on the potential impact on public health and safety and the

designated NRC response to specific findings. As a result, the staff plans to

delete this metric from the ROP self-assessment program as described in IMC

0307 and remove the related question from future external surveys. Findings

are continuously under review by the staff to determine the need for adjustments

to the SDPs in this area. For example, based on a finding identified during this

assessment period, the outcome of the Public Radiation Safety SDP is being

evaluated by the staff to assure equivalence between the various SDP

outcomes.

Metric Criterion Met: No.
-30-

SDP-5 The Resources (Direct Charges and Support Activities) Expended Are

Appropriate

Definition: Track the percentage of total resource expenditures attributed to SDP activities

to determine the effort expended by the regions in completing SDP evaluations

as a percentage of the total regional direct inspection effort.

Criteria: Total SDP expenditures should not exceed 10 percent of the total regional direct

inspection effort (DIE) with a stable or declining trend.

Goals Supported: Ensure Effectiveness, Predictable

Analysis: Regional expenditures associated with SDP evaluations remain stable and below

the target goal. 

Metric Criterion Met: Yes.
-31-

SDP-6a Final Significance Determinations Are Timely

Definition: Conduct a quarterly audit of RPS data to identify the total number of inspection

items finalized as greater than green that were under review for more than 90

days since:

(1) the date of initial licensee notification of the preliminary

significance in an inspection report, or 

(2) the date the item was formally transmitted to an NRR technical

branch for SDP assistance, or

(3) the item was otherwise documented in an inspection report as an

apparent violation pending completion of a significance

determination and not counted in either of the above categories.

Criteria: At least 90% of all SDP results that are counted per the criteria above should be

finalized within 90 days. All issues greater than 90 days will be assessed to

determine causal factors and to recommend process improvements.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: Timeliness of final significance determinations increased from 68% in FY 2005 to

96% for FY 2006. Since implementation in April 2000, the SDP has gone

through several significant changes based on feedback from internal and

external stakeholders and the recommendations of two independent audits. As

a result, SDP timeliness has improved significantly, meeting its goal for the first

time since the implementation of the ROP. 

Metric Criterion Met: Yes.
-32-

SDP-6b Final Significance Determinations Are Timely

Definition: Conduct a quarterly audit of issues that were assessed by the Significance

Determination Process/Enforcement Review Panel (SERP) to identify the total

number of inspection items finalized as green or greater-than-green that were

under review for more than 90 days since:

(1) the date of initial licensee notification of the preliminary significance in an

inspection report, otherwise documented in an inspection report as an

“AV” pending completion of a significance determination, or 

(2) the date the item was presented to the SERP for review. 

Criteria: Average age of all SDP results that are counted per the criteria above should be

equal or less than 90 days and none of the results can be older than 180 days.

All issues greater than 180 days will be assessed to determine causal factors

and to recommend process improvements.

This metric is being piloted as a potential replacement for the existing SDP

timeliness metric.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: There was a total of 35 findings presented to the SERP and 4 were greater than

180 days. Of the 35 issues identified, there was 1 Yellow, 24 Whites and 10

Greens. The average age of all the SDP results that were presented to the

SERP during FY 2006 was 119 days. Three of the 4 issues greater than 180

days were dispositioned as Green findings. The Green findings involved the

same performance deficiency affecting 3 units, thus the 3 findings. It took 244

days to close these Green findings due to the complexity of this issue. The issue

required a phase 3 evaluation of the external event risk contribution which

included the time consuming process of developing an appropriate assessment

tool which is programmatically discouraged in the SDP. Additionally, regional

and headquarters staff collaboration on this issue was inefficient causing further

delays. The fourth finding was dispositioned as White and was associated with a

plant shutdown issue where it took 343 days to close due to programmatic

difficulties encountered. Examples of difficulties experienced include: (1)

availability of the appropriate risk analyst, (2) roles and responsibilities between

the region and headquarters were not clearly defined, and (3) disagreement with

some of the assumptions and methodology of the SDP analysis and the resulting

conclusion. Much of the time was spent resolving the differences. 

As a result of SDP timeliness issues, the regions started the implementation of

the “SDP Best Practices Initiatives” to address causes of SDP delays and

enhance SDP completion timeliness. Numerous recommendations were

identified which should continue to improve timeliness. One of the

recommendations was to develop an additional metric to further improve SDP

timeliness. The proposed metric (i.e., 90% within 90 days and 100% within 180

days) will encourage the staff to complete evaluations in a timely manner, even

when the evaluation has exceeded the 90 day goal. The staff will continue to

evaluate this metric and a decision may be made in CY 2007 whether the 180

day metric should be incorporated into the SDP timeliness metric.
-33-

SDP-7 SDP Results Are Communicated Accurately to the Public

Definition: Each calendar quarter, track the number of inspection findings that are

inaccurately communicated to the public (color of findings is inaccurately

reported) by auditing the inspection findings summary information available on

the NRC Web. The detailed review will include item type, significance

characterization, enforcement action status, and text descriptions of greaterthan-green inspection findings prior to release to external stakeholders.

Criteria: The target goal is zero inaccuracies, with a stable or declining trend. All

inaccuracies must be addressed.

Goals Supported: Ensure Openness, Understandable, Ensure Effectiveness

Analysis: During the current assessment period no inaccuracies were identified. 

Metric Criterion Met: Yes.
-34-

AS-1 Subjective Judgment Is Minimized and Is Not a Central Feature of the

Process. Actions Are Determined by Quantifiable Assessment Inputs

(Examine PIs and SDP Results)

Definition: Audit all assessment-related letters and count the number of deviations from the

Action Matrix. Evaluate the causes for these deviations and identify changes to

the ROP, if any, to improve the guidance documents.

Criteria: Expect few deviations, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Openness

Analysis: There have been a total of twelve (12) deviations from the Action Matrix since

the beginning of the ROP in CY 2000. Three of these deviations occurred in

CY 2006. Since CY 2004, the trend seems to be stabilized at an increased

number due to the number extensions of deviations in CY 2005 and CY 2006. 

This metric meets its criteria based on the number of deviations decreasing

during CY 2006, the unique circumstances addressed by the deviations, and the

approval of only one new deviation. The staff’s evaluation of the 2006 Action

Matrix Deviations concluded the following:

(1) The NRC issued a deviation for the Indian Point Energy Center in December

2006 to allow for an increased level of oversight for two issues: Groundwater

contamination from cracks in the Unit 2 spent fuel pool and problems with the

Alert and Notification System (ANS). The Indian Point 2 deviation was an

extension of the previous deviation in CY 2005. On September 1, 2005, the

NRC was informed by Entergy that cracks in a Unit 2 spent fuel pool wall were

discovered during excavation work inside the spent fuel pool building. Low levels

of radioactive contamination were found in the vicinity of the crack. Region I

continues to monitor Entergy's activities on this issue to ensure NRC regulations

are satisfied. Regarding the ANS issue, Entergy is required to comply with the

stipulations of the Energy Policy Act of 2005, and the associated NRC

Confirmatory Order dated January 31, 2006. Entergy is required to supply

backup power to the ANS; and, as a result, they have begun replacement of the

entire ANS, including the actuation system for the sirens in an effort to make the

entire system more reliable. Entergy has developed the Indian Point Energy

Center Prompt Alert and Notification System Design Report which contains the

specifics of the new system design. This report has been submitted to the

Department of Homeland Security for approval.

(2) The Davis-Besse deviation was an extension of the previous deviation in CY

2005. This action is necessary to continue to monitor the licensee’s efforts to

sustain improved plant performance following resolution of the long-standing

underlying problems that culminated in a Red finding associated with the severe

wastage that was discovered on the reactor vessel head. Davis-Besse was

placed under the IMC 0350 process for about 3 years. While the plant

transitioned from the IMC 0350 process, a deviation was authorized on

May 16, 2005, for the period of July 2005 through June 2006. 

(3) The Waterford Steam Electric Station, Unit 3 one-time deviation was issued

to address PIs that were determined by the staff to be Red, and Yellow,
-35-

respectively, following the conduct of a discrepant PI inspection. The deviation

was requested because the actions outlined in the Licensee Response column of

the Action Matrix are more appropriate for the situation at Waterford 3 than those

of the Multiple/Repetitive Degraded Cornerstone Column. This situation is not

likely to recur due to the replacement of the SSU PIs with the Mitigating Systems

Performance Index (MSPI) in April of 2006. 

Metric Criterion Met: Yes.
-36-

AS-2 The Program Is Well-defined Enough to Be Consistently Implemented

Definition: Audit all assessment letters and count the number of significant departures from

requirements in IMCs 0305, “Operating Reactor Assessment Program,” and

0350, “Oversight of Operating Reactor Facilities in an Extended Shutdown as a

Result of Significant Performance Problems.” Timeliness goals are counted in

metric AS-5.

Criteria: Expect few departures, with a stable or declining trend.

Goals Supported: Objective, Predictable, Ensure Effectiveness

Analysis: There were no significant departures from the requirements of IMC 0305 or 0350

as a result of an audit of assessment letters during the period between January

and December 2006. The regions have consistently complied with the

requirements of IMCs 0305 and 0350 while preparing and issuing their

assessment letters over the past several years. The earlier inconsistencies that

this metric was designed to track and trend appear to have been corrected. As a

result, the staff no longer plans to monitor this metric and plans to delete it from

IMC 0307 in CY 2007.

Metric Criterion Met: Yes.
-37-

AS-3 Actions Taken Are Commensurate with the Risk of the Issue and Overall

Plant Risk

Definition: Review actions taken for greater-than-green inspection findings and PIs. Track

the number of actions (or lack of actions) taken by the regions that are not

appropriate for the significance of the issues and are non consistent with the

Action Matrix.

Criteria: Expect few departures, with a stable or declining trend.

Goals Supported: Risk-Informed, Ensure Effectiveness, Ensure Safety

Analysis: All actions taken by the regional offices were consistent with the Action Matrix

during the period between January and December 2006. This metric met its

criteria based on no departures from the ROP regarding actions taken in

response to greater-than-green findings or PIs. All noted departures from the

Action Matrix over the past several years have been properly documented as

deviations and are discussed under metric AS-1. As a result, the staff no longer

plans to monitor this metric and plans to delete it from IMC 0307 in CY 2007.

Metric Criterion Met: Yes.
-38-

AS-4 The Number And Scope of Additional Actions Recommended as a Result of

the Agency Action Review Meeting (AARM) Beyond Those Actions Already

Taken Are Limited

Definition: Review the results of the Agency Action Review Meeting (AARM).

Criteria: Few additional actions, with a stable or declining trend.

Goals Supported: Understandable, Predictable, Objective

Analysis: The AARM was held on April 19, 2006, in Baltimore, Maryland. The participants

confirmed the appropriateness of agency actions for Point Beach 1 and 2, Perry,

and Davis-Besse. The participants did not recommend any additional actions

beyond those already taken or planned. The next Agency Action Review

Meeting is scheduled for April 2007.

Metric Criterion Met: Yes.
-39-

AS-5 Assessment Program Results (Assessment Reviews, Assessment Letters

and Public Meetings) Are Completed in a Timely Manner

Definition: Track the number of instances in which timeliness goals established in IMC 0305

were not met. The regions will collect timeliness data for the conduct of quarterly

reviews (within 5 weeks of the end of quarter); mid-cycle, and end-of-cycle

reviews (within 6 weeks of the end of quarter); issuance of assessment letters

(within 2 weeks of the quarterly review and 3 weeks of the mid-cycle and end-ofcycle reviews); assessment follow-up letters (on or before the next quarterly

review); and public meetings (within 16 weeks of the end of the assessment

period).

Criteria: Expect few instances in which timeliness goals were not met, with a stable or

declining trend.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: 4Q/2006: All quarterly reviews and five assessment follow-up letters were

completed within timeliness goals.

3Q/2006: All mid-cycle review meetings, mid-cycle letters and seven quarterly

assessment reviews were conducted within timeliness goals. Additionally, four

public meetings were completed within timeliness goals.

2Q/2006: All quarterly assessment reviews and two assessment follow-up letters

were completed within timeliness goals. Additionally, forty-seven public

meetings were completed within timeliness goals.

1Q/2006: Eighty-one of eighty-two end-of-cycle reviews, all assessment letters,

four quarterly assessment reviews and two assessment follow-up letters were

completed within timeliness goals. Additionally, ten public meetings were

completed within timeliness goals.
-40-

This metric met its criteria. Note the timing of some of the assessments was

recently changed in IMC 0305 and will be updated in IMC 0307 in CY 2007.

Metric Criterion Met: Yes.
-41-

AS-6 Public Availability of Assessment Letters Is Timely

Definition: Record the number of letters not available in ADAMS and number of letters not

posted to the Web site within goals as stipulated in IMC 0305, “Operating

Reactor Assessment Program.”

Criteria: IPAB posts assessment letters to the NRC's external Web site using the

electronic version in ADAMS within 10 weeks after the end of mid-cycle and endof-cycle assessment periods and within 8 weeks of the end of intervening

quarters.

Goals Supported: Ensure Effectiveness, Ensure Openness, Predictable

Analysis: All assessment letters were posted to the web within timeliness goals.

Metric Criterion Met: Yes.
-42-

AS-7 The NRC's Response to Performance Issues Is Timely

Definition: Count the number of days between issuance of an assessment letter discussing

an issue of more than very low safety significance and completion of the

supplemental inspection (by exit meeting date, not issuance of the inspection

report).

Criteria: Expect a stable or declining trend.

Goals Supported: Ensure Safety, Ensure Effectiveness

Comments: The data represents an average timeliness for the supplemental inspections

completed in each region in any given quarter.

Analysis: Data collected to date indicates a relatively stable long term trend regarding the

elapsed time between the issuance of an assessment letter and the completion

of the corresponding supplemental inspection. The staff will continue to monitor

this data set to determine if an adverse trend exists.

Metric Criterion Met: Yes.
-43-

AS-8 NRC Takes Appropriate Actions to Address Performance Issues

Definition: Survey external and internal stakeholders asking whether the NRC takes

appropriate actions to address performance issues for those plants outside the

Licensee Response Column of the Action Matrix.

Criteria: Expect stable or improved perception.

Goals Supported: Ensure Safety, Ensure Effectiveness, Understandable

Analysis: Internal Survey

Three internal survey questions address this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The assessment process utilizes

appropriate actions to address

performance issues for those licensees

outside of the Licensee Response Column

of the action matrix

87% 85% 80% N/A

The assessment process provides for an

appropriate range of actions for safety

issues

89% 80% 78% 82%

The ROP provides sufficient attention to

licensees whose performance is in the

Licensee Response Column 

87% 81% 76% 74%

Internal stakeholders continued to agree that the NRC takes appropriate actions

to address performance issues. The data supporting this metric indicates a

slightly increasing positive perception for this measure when compared to the

previous survey in 2004. 

External Survey

The industry and States generally agreed that actions taken by the NRC for

plants outside of the licensee response column have been appropriate. Public

interest groups were generally critical of NRC actions. Specific concerns were

expressed with NRC actions at plants with challenging issues such as Indian

Point. Another concern is that the NRC action in accordance with the Action

Matrix is consistent for single White findings, but appears less consistent for

more complex issues. The overall level of external stakeholder satisfaction in this

area was generally favorable and similar to previous years. 

Metric Criterion Met: Yes.
-44-

AS-9 Assessment Reports Are Relevant, Useful, and Written in Plain Language

Definition: Survey external and internal stakeholders asking whether the information

contained in assessment reports is relevant, useful, and written in plain English.

Criteria: Expect stable or improved perception of the relevance, usefulness, and

understandability of assessment reports.

Goals Supported: Understandable, Ensure Effectiveness, Ensure Openness

Analysis: Internal Survey

Two internal survey questions addressed this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP generally provides appropriate

communication effectiveness through use

of plain English in official correspondence

(e.g., assessment reports, ...)

82% 79% 74% 74%

The information on plant performance

(e.g., assessment reports, ...) provided on

the ROP Web page is understandable and

written in plain English

93% 89% 87% 89%

Internal stakeholders continued to agree that the information contained in

assessment reports is relevant, useful, and written in plain language. The data

supporting this metric indicates a slightly increasing positive perception for this

measure when compared to the previous survey in 2004.

External Survey

The industry and States generally agreed that the information contained in

assessment reports is relevant, useful, and written in plain English. One public

interest group stated that the assessment reports are limited and that only one or

two lines in the assessment report actually provide a written assessment.

Another external comment stated the assessment letters contain too much

boilerplate information which precluded substantive insights about performance

at individual sites. This has been a similar comment made in previous years. 

The overall level of external stakeholder satisfaction in this area was generally

favorable and similar to previous years.

Metric Criterion Met: Yes.
-45-

AS-10 Degradations in Plant Performance Are Gradual and Allow Adequate

Agency Engagement of the Licensees

Definition: Track the number of instances each quarter in which plants move more than one

column to the right in the Action Matrix (as indicated on the Action Matrix

Summary).

Criteria: Expect few instances in which plant performance causes a plant to move more

than one column to the right in the Action Matrix. Provide a qualitative

explanation of each instance in which this occurs. Expect a stable or declining

trend from the first-year benchmark.

Goals Supported: Risk-informed, Ensure Safety, Predictable

Analysis: Oconee Units 1, 2, and 3 moved from the licensee response column to the

degraded cornerstone column in 4Q/2006 due to a white MSPI in emergency AC

power (4Q/2006) and a white finding in the mitigating systems cornerstone (the

standby shutdown facility flood wall breach) effective 3Q/2006. This metric met

its criteria based on the fact that only one site (with all three units impacted by

the finding) moved two or more columns to the right in CY 2006.

Metric Criterion Met: Yes.
-46-

AS-11 Perceived Effectiveness of Safety Culture Enhancements to ROP

Definition: Survey external and internal stakeholders asking whether the ROP safety culture

enhancements help in identifying licensee safety culture weaknesses and

focusing licensee and NRC attention appropriately.

Criteria: Expect stable or improved perception over time. Trend average level of

agreement.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: Internal Survey

Five internal survey questions are related to this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP safety culture enhancements

help in identifying licensee safety culture

weaknesses and focusing licensee and

NRC attention appropriately

62% N/A N/A N/A

The assessment process provides effective

consideration of safety culture aspects

67% N/A N/A N/A

The inspection program provides adequate

guidance on safety culture aspects

64% N/A N/A N/A

The inspection procedures provide

adequate guidance on safety culture

aspects

65% N/A N/A N/A

Adequate training is available for the safety

culture aspects of the ROP inspection

procedures and manual chapters

59% N/A N/A N/A

This was the first time that the internal survey has included safety culture

questions (the related inspection documents were issued July 1, 2006). A

number of supplemental written comments were provided to elaborate on the

responses. The written responses from a number of individuals reflected that it

was too soon to answer the safety culture questions given the short time that the

revised program has been in place. There were a diversity of opinions

expressed as to whether the program changes were beneficial or not. The most

common concerns expressed were that the ROP safety culture changes are

overly burdensome with respect to the additional time required to assign crosscutting aspects and to assess the results; that there are too many cross-cutting

aspects so that the threshold to reach a substantive cross-cutting issue is too

high; that the process is too subjective; that the revised process is too

complicated; and that additional safety culture training is necessary.
-47-

For all of the safety culture questions, even at this early point in the

implementation of the enhanced program, more than one-half of the internal

respondents indicated that the changes to the ROP will help to identify licensee

safety culture weaknesses and to focus both licensee and NRC resources

accordingly. Responses to related questions about the adequacy of the

supporting infrastructure (process, procedures and training) again indicate that

more than one-half of the internal respondents consider that an adequate

infrastructure is currently in place. 

In parallel with the implementation of the enhanced ROP, the staff has been

working to update existing and new inspector training courses to integrate the

ROP safety culture changes. Mechanisms exist for inspection staff to provide

feedback and suggested changes on the ROP guidance documents. The staff

considers the ROP guidance documents to be a work-in-progress with respect to

safety culture content as lessons learned during the initial 18 month

implementation phase will be evaluated and the program will be further modified

as necessary.

External Survey

One external survey question asked whether the "ROP safety culture

enhancements help identify licensee safety culture weaknesses and focus

licensee and NRC attention appropriately." This was the first time that the

external survey has included a safety culture question. 

Five external respondents were in agreement that the safety culture

enhancements would help identify weaknesses while 6 disagreed. Written

responses were received from six entities. Four of the six stakeholders (three 

licensees and one state or local government) commented that it was too early in

the implementation phase of the related program documents to answer whether

the changes are helping to identify safety culture weaknesses and focusing

licensee and NRC resources appropriately. State and local government

feedback from one respondent opined that the NRC will never identify safety

culture weaknesses, since the primary source of this input will come directly to

the NRC from licensee staff. One member of the public was very complimentary

of the NRC actions in this area. 

As there is only 6 months of data, and given the relatively short period of time

that the enhanced ROP has been in effect, a trend has not been established. A

more meaningful assessment of the metric criterion will be achieved in follow-on

years as the safety culture changes have been implemented for a longer period. 

The safety culture performance criteria is considered not applicable at this time.

Metric Criterion Met: Not applicable at this time.
-48-

O-1 Stakeholders Perceive the ROP To Be Predictable and Objective

Definition: Survey external and internal stakeholders asking if ROP oversight activities are

predictable (i.e., controlled by the process) and reasonably objective (i.e., based

on supported facts, rather than relying on subjective judgment).

Criteria: Expect a stable or increasing positive perception over time.

Goals Supported: Objective, Predictable, Ensure Effectiveness

Analysis: Internal Survey

Five internal survey questions addressed this metric. These questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

ROP generally provides appropriate

objectivity to the oversight process

88% 81% 82% 85%

The assessment process provides

objective levels of assessment

88% 84% 78% 84%

Baseline Inspection Program leads to

objective findings whose significance can

be clearly documented

81% 73% 69% 71%

ROP generally provides a predictable

approach to oversight*

87% 73%* 69%* 75%*

ROP generally provides a consistent

approach to oversight*

84% 84%* 85%* 84%*

 * In prior years’ surveys, these last two questions were worded in the

context of comparing the attributes to the previous oversight process

Internal stakeholders continue to generally agree that the ROP is predictable and

objective. The data supporting this metric indicates a slightly increasing positive

perception for these measures when compared to the previous survey two years

ago. It is notable that some commented that the process has become too

predictable and that it becomes a tool to make the licensees look good. 

External Survey

Overall, a majority of the respondents (including utilities, state agencies and

public interest groups) agreed that the ROP is predictable and objective in

comparison to the previous process. The responses from licensees were mostly

in favor of the predictability and objectivity aspects in the ROP, similar to

previous years. On the other hand, the public, while most agree that the process

is predictable and objective, are more skeptical of the ROP. In comments, some

utilities and their representatives indicated that the substantive cross-cutting
-49-

issues are the most unpredictable areas that need improvement. There is some

agreement that the overall process is moving in the right direction.

Metric Criterion Met: Yes.
-50-

O-2 Stakeholders Perceive the ROP To Be Risk-informed

Definition: Survey external and internal stakeholders asking if the ROP is risk-informed, in

that actions and outcomes are appropriately graduated on the basis of increased

significance.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Risk-Informed, Ensure Effectiveness, Ensure Openness

Analysis: Internal Survey

Six internal survey questions addressed this metric. The questions and the

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

ROP provides an effective risk-informed

approach to oversight

79% 74% 73% 82%

The assessment process focuses

resources on areas of greatest safety

significance

78% 81% 80% 80%

The baseline inspection program

appropriately inspects for and identifies

risk-significant issues

89% 79% 73% 78%

Inspection procedures adequately sample

risk significant aspects of each inspectible

area

87% 80% 72% 76%

PIs provide useful information on risksignificant areas71% 67% 70% 79%

SDP focuses NRC attention on

safety-significant issues

83% 75% 71% 79%

Internal stakeholders generally agree that the ROP provides an effective

risk-informed approach to oversight. The data supporting this metric indicates a

stable positive perception for the measure when compared to the previous

survey in 2004 and is consistent with the positive perception in all previous

surveys. However, some of the comments collected regarding this aspect

revealed concerns from internal stakeholders who questioned the risk-informed

process, and stated that licensees mainly use the process to cut cost. 

External Survey

Overall, a majority of the respondents believe the ROP is more risk informed

than the previous process. Some comments from the public expressed
-51-

reservations over the direction of the risk informed process and requested that

more items that are risk significant be included in the risk basis.

In summary, stakeholder satisfaction as reported in the survey responses was

generally favorable and consistent.

Metric Criterion Met: Yes.
-52-

O-3 Stakeholders Perceive the ROP To Be Understandable

Definition: Survey external and internal stakeholders asking if the ROP is understandable

and if the processes, procedures, and products are clear and written in plain

English.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Understandable, Ensure Effectiveness, Ensure Openness

Analysis: Internal Survey

Three internal survey questions addressed this metric. The questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP generally provides

understandable procedures and guidance

documents that are clear and written in

plain English

80% 72% 74% 74%

The assessment process provides an

understandable level of NRC oversight

91% 77% 76% 74%

The information provided on the ROP Web

page is understandable and written in plain

English

93% 89% 87% 89%

Internal stakeholders continue to generally agree that the ROP is

understandable and written in plain English (with the exception of the complexity

of the SDP). The data supporting this metric indicates a stable positive

perception for these measures and is consistent with the positive perception

when compared to the previous surveys.

External Survey

In general, most of the stakeholders stated that the ROP is understandable and

that products are written in clear and plain English. Both Licensees and State

agencies expressed reservations about the public’s ability to understand the

SDP. Utilities also stated that there is no clear guidance, established process or

consistency for the closing of a substantive crosscutting issue. Similar to the

previous surveys, the SDP is recognized to be the most complex portion of the

ROP requiring some technical background for understanding.

Overall, the objective measure of stakeholder satisfaction as reported in the 

survey responses for the current ROP was favorable and consistent.

Metric Criterion Met: Yes.
-53-

O-4 Stakeholders Perceive That the ROP Provides Adequate Regulatory

Assurance That Plants Are Operated and Maintained Safely

Definition: Survey external and internal stakeholders asking if the ROP provides adequate

regulatory assurance, when combined with other NRC regulatory processes, that

plants are being operated and maintained safely.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Safety, Ensure Effectiveness, Ensure Openness

Analysis: Internal Survey

Three internal survey questions addressed this metric. These questions and

their resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP generally provides appropriate

assurance that plants are being operated

safely

89% 84% 80% 88%

The ROP generally provides appropriate

regulatory attention to licensees with

performance problems

87% 81% 76% 74%

The ROP generally provides appropriate

identification of declining safety

performance before there’s a significant

reduction in safety margins

68% 57% 51% 53%

Internal stakeholders continue to generally agree that the ROP maintains safety. 

The data supporting this metric indicates a stable positive perception for this

measure when compared to the previous surveys. However, stakeholder’s

comments also identified some areas that need improvement, and some

continued to note concerns with the ROP’s ability to detect declining

performance in a timely manner.

External Survey

The majority of utility stakeholders believe the ROP maintains safety while some

of the non-utility stakeholders (state agencies and public interest groups) feel it

does not. These comments are consistent with past surveys that have had some

negative comments from the public interest groups on the ROP maintaining

safety.

Overall, stakeholder satisfaction as reported in the survey responses for the

current ROP was generally favorable and consistent.

Metric Criterion Met: Yes.
-54-

O-5 Stakeholders Perceive the ROP To Be Effective, Efficient, Realistic, and

Timely

Definition: Survey external and internal stakeholders asking whether the ROP is effective,

efficient, realistic, and timely.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Effectiveness, Ensure Openness

Analysis: Internal Survey

Three internal survey questions address this metric. These questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The ROP generally provides a realistic

approach to oversight

84% 75% 74% 79%

The ROP generally provides a timely

approach to oversight

79% 67%* 64%* 78%*

The ROP generally provides appropriate

efficiency and effectiveness to the

oversight process

77% 71%* 70%* 75%*

 * In prior years’ surveys, these last two questions were worded in the

context of comparing the attributes to the previous oversight process

Most of internal stakeholders agree that the ROP provides a realistic, timely,

efficient and effective approach to oversight. The data supporting this metric

indicates a stable perception for these four measures when compared to the

previous surveys as indicated in the table above, and is consistent with the

positive perception. 

External Survey

In general, the respondents believe that the ROP is effective, efficient, realistic

and timely in comparison to previous programs. The utility stakeholder

responses asked for improvement in many areas. A common concern among

most respondents is the SDP and that the process may not be clear.

Overall, stakeholder satisfaction as reported in the survey responses was

generally consistent.

Metric Criterion Met: Yes.
-55-

O-6 Stakeholders Perceive That the ROP Ensures Openness

Definition: Survey external and internal stakeholders asking if the ROP ensures openness

in the regulatory process.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Openness, Ensure Effectiveness

Analysis: Internal Survey

Four internal survey questions addressed this metric. These questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

The information provided by the NRC

appropriately keeps the public informed of

the agency oversight activities related to

the plants

89% 77% 78% 74%

The ROP generally provides appropriate

inspector and licensee communication

95% 86% 82% 83%

The ROP generally provides appropriate

communication effectiveness through use

of plain English in official correspondence

82% 79% 74% 74%

The information provided on the ROP Web

page is adequate to keep NRC internal

stakeholders informed

95% 87% 74% 77%

Internal stakeholders continue to generally agree that the ROP ensures

openness. The data supporting this metric indicates a stable positive perception

for the measures compared to the previous survey in 2004, and is consistent

with the positive perception in all previous surveys. 

External Survey

External stakeholders generally acknowledged that the ROP ensures openness

in the regulatory process, but both public and utility stakeholders expressed

some concerns and noted that further improvements could be made. Public and

State and local agencies perception was neutral to slightly negative indicating

their concern that the ROP is somewhat less open than needed. One

stakeholder noted specifically that plant PRAs are not available for public

scrutiny. Other public stakeholder comments noted that the SDP is too complex

for public involvement.

Utility stakeholders reiterated that the ROP is generally a very open process, but

pointed out the security process and the SDP activities as specific areas that

should be more open and allow for greater stakeholder involvement.

Metric Criterion Met: Yes.
-56-

O-7 Opportunities for Public Participation in the Process

Definition: Survey external and internal stakeholders asking if there are sufficient

opportunities for the public to participate in the process.

Criteria: Expect positive responses or an improving trend.

Goals Supported: Ensure Openness, Ensure Effectiveness

Analysis: External stakeholders generally agreed that there are sufficient opportunities for

the public to participate in the process. Public perception was positive to neutral,

indicating their general agreement that the public has ample opportunity to

participate in the process. One respondent noted that the level of participation

by the public was low, although the NRC has provided opportunities. Most

respondents acknowledged the ample opportunities for public participation

(monthly public meetings at NRC headquarters, annual public meetings

conducted in the reactor communities, annual solicitation of public comments,

annual ROP Commission briefing, and the staff’s consolidated response to last

year’s comments).

The responses from State and other agencies were very favorable and noted

their participation in the process as an example.

Industry stakeholder responses were generally very positive.

Metric Criterion Met: Yes.
-57-

O-8 Stakeholders Perceive the NRC To Be Responsive to its Inputs and

Comments

Definition: Survey external and internal stakeholders asking if the NRC is responsive to the

public's inputs and comments on the ROP.

Criteria: Expect positive responses or an improving trend.

Goals Supported: Ensure Openness, Ensure Effectiveness

Analysis: Internal Survey

Four internal survey questions addressed this metric. These questions and their

resultant percentage of agreement are presented below.

Measure 2006 2004 2002 2001

Responses from feedback forms sent to

headquarters are responsive and address

the issues raised

68% 60% 54% 55%

Responses from feedback forms sent to

headquarters are accurate

79% 76% 64% 65%

Responses from feedback forms sent to

headquarters are understandable and

written in plain English

78% 77% 69% 67%

Responses from feedback forms sent to

headquarters are timely

50% 47% 30% 66%

In general, most internal stakeholders agree that the agency is responsive to

their feedback and input. About two-thirds of the respondents believe that the

feedback forms are responsive and address the issues raised. Many

respondents believe that the feedback forms were understandable, written in

plain English, and were accurate. However, only half of the respondents agree

that the responses to feedback forms sent to headquarters are timely. The

internal feedback process is also directly gauged by metric IP-2, which describes

an enhanced process that should further improve the timeliness of the feedback

resolution.

External Survey

The majority of stakeholders believe that the NRC is responsive to inputs and

comments; however, some public stakeholders feel that the NRC is responsive

to input and feedback only because it is required. Non-utility stakeholders

commented that there is a perception on the part of the public that the NRC does

not value their input.

Overall, stakeholder satisfaction as reported in the survey responses for the

ROP was generally favorable and consistent.

Metric Criterion Met: Yes.
-58-

O-9 Stakeholders Perceive That the ROP Is Implemented as Defined

Definition: Survey external and internal stakeholders asking if the ROP has been

implemented as defined by program documents.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Predictable, Understandable, Ensure Openness

Analysis: In general, most external stakeholders including utilities, state and other

agencies, and the public believe the ROP is being implemented as defined. 

Overall stakeholder comments are satisfied with this aspect of the process. 

Stakeholder satisfaction as reported in the external survey responses for the

ROP was generally favorable and consistent.

This aspect of the ROP was not specifically addressed in the internal survey. 

Relevant questions will be added to future internal surveys.

Metric Criterion Met: Yes.
-59-

O-10 Stakeholders Perceive That the ROP Does Not Result in Unintended

Consequences

Definition: Survey external and internal stakeholders asking if the ROP results in

unintended consequences.

Criteria: Expect stable or increasingly positive perception over time.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: In a change from previous external surveys, the majority of stakeholders

responding indicate that they believe the ROP does not result in unintended

consequences, although one state agency stakeholder commented that it may

still be premature to conclude that the ROP minimizes unintended

consequences. Further indication of improvement from the previous external

surveys, the overall stakeholder satisfaction as reported in the survey responses

for the ROP was generally favorable or neutral.

This aspect of the ROP was not specifically addressed in the internal survey. 

Relevant questions will be added to future internal surveys.

Metric Criterion Met: Yes.
-60-

O-11 Analysis of NRC’s Responses to Significant Events

Definition: Review reports from incident investigation teams (IITs) and augmented

inspection teams (AITs) to collect lessons learned regarding ROP programmatic

deficiencies (i.e., did the baseline inspection program inspect this area? did the

SDP accurately characterize resultant findings?). IITs already have the provision

to determine NRC program deficiencies. AITs will be reviewed by NRR/DIRS to

identify any weaknesses.

Criteria: Expect no major programmatic voids.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: No IITs were conducted during the 2006 ROP cycle. One AIT was conducted in

CY 2006. Staff review of the AIT did not identify any program weaknesses or

voids. No feedback forms were received for IP 93800.

Metric Criterion Met: Yes.
-61-

O-12 Analysis of Significant Events

Definition: Annually review all accident sequence precursor (ASP) events that have a risk

significance of more than 10 to identify any ROP programmatic voids (i.e., did -6

the baseline inspection program inspect this area? did the SDP accurately

characterize resultant findings?).

Criteria: Expect no major programmatic voids.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: The Office of Nuclear Regulatory Research (RES) had in the past performed a

comparison of SDP analysis approaches and results with Accident Sequence

Precursor (ASP) analysis. This comparison is no longer performed because

under the recently revised ASP process, SDP analyses are used in lieu of

separate ASP analyses. As a result, the staff no longer plans to monitor this

metric and plans to delete it from IMC 0307 in CY 2007.

Metric Criterion Met: Not applicable.
-62-

O-13 Analysis of Inspection Hours and Resource Expenditures

Definition: Annually, collect and analyze resource data (e.g., direct inspection effort,

preparation/documentation, plant status hours) for Baseline, Supplemental/PlantSpecific, and Safety Issues Inspections, and other ROP activities.

Criteria: (1) Significant deviations are not expected on an annual basis. Explore reasons

for any deviations that may be evident.

(2) Track and trend resource usage for the baseline inspection program and

supplemental/plant-specific inspections. Analyze causes for any significant

departure from established trend.

(3) Track and trend resource usage for preparation, documentation, and other

ROP activities, and assess the effects on budgeted resources.

NOTE: This metric is intended primarily for tracking and trending resource

usage for the ROP. The results are used to improve the efficiency and

effectiveness of the ROP and to make management and budget decisions. A

detailed ROP resource analysis is included in the annual ROP self-assessment

Commission paper.

Goals Supported: Ensure Effectiveness, Predictable

Analysis: Overall, staff effort in 2006 was essentially unchanged compared with 2005 with

a decrease of 0.4%, although there was significant variation in the distribution of

effort among the various elements of the ROP. 

Baseline inspection effort in 2006 was comparable with 2005. The reduction in

“plant status” effort is the result of accounting changes in mid-2006 for effort

related to daily reviews of licensee corrective action activities. This effort is now

charged to Inspection Procedure (IP) 71152 instead of “plant status.”

Plant-specific inspections include: supplemental inspections conducted in

response to inspection findings and degraded performance indicators; reactive

inspections such as Augmented Inspection Teams (AITs) and Special

Inspections (SIs) performed in response to events; and infrequently performed

inspections that are not part of the baseline or supplemental inspection program. 

The effort for supplemental inspections (IP 95001, IP 95002, and IP 95003)

decreased in 2006 compared with 2005 due to a reduced number of

“greater-than-green” inspection findings in 2006. However, this decrease was

offset by increases in the other plant-specific inspections resulting in an overall

increase in the 2006 inspection effort for plant-specific inspections. A noticeable

increase was reported in SIs in response to events and in infrequently performed

inspections. This increase reflects the effort in this area at several Region I sites

with approved deviations for additional inspection, and in increased inspection

activity at Browns Ferry. 

Also noteworthy are: (1) a decrease of 17.1% from 2005 to 2006 in inspection

effort related to Generic Safety Inspections (these are typically one time

inspections of specific safety issues with significant variability in effort possible

from year to year) and, (2) an increase in the hours reported for the “other

activities.” The increase in the “other activities” was in the aggregate of Routine
-63-

Communication/Regional Support/Enforcement Support/Review of technical

Documents. Hours charged for inspection-related travel and Significance

Determination Process remained similar to 2005 hours.

Metric Criterion Met: Yes.
-64-

O-14 Analysis of Resident Inspector Demographics and Experience

Definition: Annually, collect and analyze data in order to determine the relevant inspection

experience of the resident inspector (RI) and senior resident inspector (SRI)

population. The following four parameters will be measured and analyzed for

both RIs and SRIs to ensure that the NRC maintains a highly qualified resident

inspection staff:

(1) NRC time - the total time the individual has accumulated as an NRC

employee.

(2) Total resident time - the total time the individual has accumulated as an

RI or SRI.

(3) Current site time - the total time the individual has spent as an RI or SRI

at the current site.

(4) Relevant non-NRC experience - the total time the individual has gained

relevant nuclear power experience outside of the NRC. Examples of

relevant non-NRC experience are operations, engineering, maintenance,

or construction experience with commercial nuclear power plants, naval

shipyards, Department of Energy facilities, and/or the U.S. Navy nuclear

power program.

Criteria: None; trend only. Provide reasons for any meaningful increase or decrease in

these resident demographic metrics.

NOTE: This metric is intended primarily for tracking and trending resident

inspection experience. The results are used to make any necessary

modifications to the RI and/or SRI programs in order to attract and retain highly

qualified inspectors to the respective programs.

Goals Supported: Ensure Safety, Ensure Effectiveness
-65-

TABLE 1

SUMMARY OF RI GROUP EXPERIENCE LEVELS (IN YEARS)

Nov.

2002

Nov.

2003

Nov.

2004

Nov.

2005

Nov.

2006

NRC time average 6.39 5.34 5.60 5.80 5.89

median 5.61 4.13 3.42 3.36 4.04

Total resident time average 3.90 3.28 3.20 3.52 3.41

median 3.77 1.99 2.00 2.31 2.39

Current site time average 2.86 1.64 2.18 2.38 2.36

median 2.30 1.00 1.85 2.25 2.23

Relevant non-NRC experience average 9.68 10.26 11.01 12.55 12.14

median 9.29 10.00 10.00 10.63 10.75
-66-

TABLE 2

SUMMARY OF SRI GROUP EXPERIENCE LEVELS (IN YEARS)

Nov.

2002

Nov.

2003

Nov.

2004

Nov.

2005

Nov.

2006

NRC time average 11.85 11.30 11.57 11.30 10.22

median 12.11 11.00 8.80 8.84 9.28

Total resident time average 8.17 8.22 8.22 8.16 8.20

median 7.36 6.82 7.32 7.54 7.77

Current site time average 2.90 2.44 2.68 2.79 2.92

median 3.06 1.76 2.31 2.63 3.21

Relevant non-NRC 

experience

average 7.26 8.37 8.51 8.98 10.44

median 5.17 6.42 6.55 7.96 9.08
-67-

Analysis: RI Demographic Data—The NRC staff review of the demographics included

analysis of the overall program data for the RI and SRI groups (see Tables 1 and

2, as well as Figures 1 and 2).

Analysis of 2006 RI Group—RI demographic data for 2006 (see Table 1 and

Figure 1) reflect a stable population with no change or small increases in all

areas.

During 2006, 14 RIs left the RI program. Of the 14 RIs, 11 were promoted to

SRIs, 2 were either promoted or laterally reassigned to a region or to

headquarters, and 1 retired or resigned from the NRC. 

Data indicate that experienced engineers entered the program as RIs. On

average, the new RIs had about 13 years of relevant non-NRC experience,

compared to an average of 14 years in 2005. As noted in Table 3, all but one of

the 16 new RIs in 2006 had at least 3 years of relevant non-NRC experience. 

The percentages in this table represent the ratio of those RIs that entered the RI

program in that particular year who had fewer than 3 years of relevant non-NRC

experience to the total number of RIs that entered the program.

TABLE 3

PERCENTAGE OF NEW RIs WITH LESS THAN 3 YEARS OF RELEVANT NON-NRC

EXPERIENCE

1998 1999 2000 2001 2002 2003 2004 2005 2006

12%

(1/17)

0%

 (0/5)

31%

(4/13)

6%

(1/16)

20%

(3/15)

30%

(8/27)

21%

(3/14)

0%

(0/16)

6%

 (1/16)

Analysis of 2006 SRI Group—SRI demographic data for 2006 (see Table 2 and

Figure 2) reflect a stable population with small increases in all areas.

In 2006, 11 SRI positions were filled, which is the same as 2005. In 2006, 16

SRIs left the program; of those 16, 7 were promoted within the NRC, 7 were

laterally reassigned to headquarters or a region, and 2 retired or resigned from

the NRC.

Conclusions—In summary, the staff concluded the following:

• The experience levels of both RIs and SRIs are relatively high.

• The RI and SRI staffing levels are generally good.

• The staffing turnover rate for calendar year 2006 was not excessive.

In conclusion, the program continues to attract and retain quality staff. 

Therefore, no changes to the RI program are warranted at this time. However,

the staff plans to closely monitor resident demographics and site staffing in 2007

due to anticipated influences on the program as a result of the projected

expansion of the nuclear industry.

 

Metric Criterion Met: Yes.
-68-

O-15 Analysis of Site Staffing

Definition: Semiannually, collect and analyze data in order to measure the permanent

inspector staffing levels at each of the reactor sites for both RIs and SRIs in

order to evaluate the agency’s ability to provide continuity of regulatory oversight. 

Criteria: The criteria is set at 90% program-wide. Any single site that falls below 90% will

be individually evaluated. Provide reasons for any meaningful increase or

decrease in the inspector staffing level at reactors sites.

Goals Supported: Ensure Safety, Ensure Effectiveness

Analysis: The metric counts (1) RIs and SRIs who are permanently assigned to the site

and (2) inspectors who are on rotational assignments to the site for 6 weeks or

longer. Only inspectors who have attained at least a basic inspector certification

status, as defined by IMC 1245, are counted. The metric does not count

permanently assigned RIs and SRIs who are away from their sites for longer

than 6 weeks. Inspectors who are assigned to sites for less than 6 weeks are

not counted towards satisfying the metric.

The success criteria for the metric is 90 percent program-wide. This year, the

average site coverage for the regions was 98.8 percent, with all regions

exceeding 97.1 percent. However, one site did not meet the success criteria of

90 percent (the score was 88.2 percent), primarily because of the assignment of

the RI to special work. The staff’s evaluation determined that oversight

continuity was maintained and that this site was adequately covered for one or

more of the following reasons: (1) the permanent SRI was present, (2) the site

was covered with qualified inspectors on assignments for less then 6 weeks. At

no time did these sites remain without qualified inspectors. 

Metric Criterion Met: Yes.
-69-

O-16 Analysis of ROP Training and Qualifications

Definition: Annually, evaluate the implementation of IMC 1245, “Qualification Program for

the Office of Nuclear Reactor Regulation Programs,” particularly as it pertains to

ROP implementation.

Criteria: None; trend only. Summarize and evaluate the training accomplished over the

previous year and propose program improvements as necessary to address

noted concerns.

NOTE: This metric is intended primarily for tracking and trending the

effectiveness of the ROP training and qualifications programs. A discussion of

training effectiveness is included in the annual ROP self-assessment

Commission paper.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff continued its efforts to improve the inspector training programs and

techniques in accordance with IMC 1245. Improvement actions identified by the

staff were reviewed in accordance with the ROP feedback process and the

improvements incorporated into inspection standards, as appropriate. The staff

evaluated the results of inspector training questions contained in the biennial

internal survey and concluded that respondents generally agreed that training

was effective. The staff conducted safety culture training and the annual ROP

refresher training on the SDP during the regional counterpart meetings. The

staff developed and implemented Web-based read-and-sign training on safety

culture, outage training, and post-accident training. Additional discussion of

training effectiveness is included in the CY 2006 ROP Self-Assessment

Commission paper.

Metric Criterion Met: Yes.
-70-

O-17 Analysis of Regulatory Impact

Definition: Annually, collect and analyze licensee feedback and develop a summary of

regulatory impact forms that are critical of the ROP.

Criteria: None; trend only. Summarize and evaluate the feedback received and propose

program improvements as necessary to address common concerns.

NOTE: This metric is intended primarily for tracking and trending regulatory

impact. A detailed regulatory impact summary is included in the annual ROP

self-assessment Commission paper.

Goals Supported: Ensure Effectiveness, Ensure Safety

Analysis: The staff receives and evaluates feedback from licensees on an annual basis as

part of the regulatory impact process. The regulatory impact process was

established in 1991 based on Commission direction to develop a process for

obtaining feedback from licensees and reporting the feedback to the

Commission. Over the past year, the staff received feedback from 68 reactor

licensees on 191 issues. The comments fell into three main categories: formal

communication with licensees, inspector performance, and security and

safeguards activities. Of the comments received, 84 percent were favorable and

16 percent were unfavorable. A summary of the feedback received, the staff's

evaluation, and the proposed improvement actions are provided in Enclosure 4

of the CY 2006 ROP Self-Assessment Commission paper.

Metric Criterion Met: Yes.