POLICY ISSUE

INFORMATION

June 25, 2001 SECY-01-0114

FOR: The Commissioners

FROM: William D. Travers

Executive Director for Operations

SUBJECT: RESULTS OF THE INITIAL IMPLEMENTATION OF THE NEW REACTOR

OVERSIGHT PROCESS

PURPOSE:

This Commission paper presents the results and lessons learned from the first year of

implementation of the new reactor oversight process (ROP). In addition to a discussion and

evaluation of the lessons learned, this paper provides the Commission with the status of

program changes made to date and those that are planned in the future.

This paper also serves as the program evaluation of the new ROP as specified in the NRC

Fiscal Year 2000 Strategic Plan. Further, the paper updates the status of several long term

items identified in Commission paper SECY-00-0049, ìResults of the Revised Reactor Oversight

Process Pilot Program,î dated February 24, 2000. The staffís response to the Commissionís

direction in the SECY-00-0049 Staff Requirements Memorandum (SRM) dated May 17, 2000 is

also included. Finally, this paper responds to several recommendations for initial implementation

made by the Pilot Program Evaluation Panel as noted in the ìFinal Report of the Pilot Program

Evaluation Panel,î dated December 21, 1999.

SUMMARY:

The staff employed many activities during ROP initial implementation to collect internal and

external stakeholder feedback and comments and to evaluate the new oversight process for 

Contacts: Timothy Frye, NRR

301-415-1287

Michael Johnson, NRR

301-415-1257
- 2 -

lessons learned. As part of this effort, the staff performed a program self-assessment that 

utilized objective measures and pre-determined criteria to assess the performance of the ROP. 

Internal feedback and comments from NRC staff were obtained through periodic meetings

between Headquarters and regional staff, regional and site visits by Headquarters staff, the use

of a formal feedback process, and a staff survey. Feedback and comments on the ROP from

external stakeholders were solicited through monthly public meetings, a Lessons Learned

Public Workshop held at the end of initial implementation, and a Federal Register notice. 

Finally, an Initial Implementation Evaluation Panel (IIEP) was established by the Agency in

accordance with Federal Advisory Committees Act (FACA) requirements to serve as an

advisory committee to the Agency. 

Based on the results of the staff survey and other feedback received during initial

implementation, internal stakeholders generally had a more positive view of the ROP following

the first year of implementation than they had following the 6-month pilot program, and showed

a marked increase in their understanding and acceptance of the ROP. They felt that the

process provides appropriate regulatory attention to licensees with performance problems and

is an effective risk-informed approach to oversight. However, internal stakeholders did express

several concerns with the ROP. For example, a majority of internal stakeholders indicated that

the significance determination processes (SDPs) are not easy to use. Inspectors were

concerned that the threshold was too high for documenting findings that could be precursors to

more significant issues and with how cross-cutting issues are addressed in the ROP framework. 

A significant percentage of internal stakeholders continue to express concern regarding the

ROPís ability to provide appropriate identification of declining safety performance in a timely

manner.

In general, external stakeholders indicated that the ROP improved consistency, reduced

unnecessary regulatory burden, and increased the predictability of Agency actions. The

industry, and many public stakeholders, perceived the ROP as being more objective and

understandable, with an increase in regulatory focus on risk significance. Industry and public

stakeholders also had numerous comments and concerns targeted at improving various parts

of the ROP. For example, many felt that the characterization of safety significance by the SDPs

was slow and complex to the point of being burdensome. The industry also strongly felt that

inconsistencies and overlap in the safety system unavailability performance indicator definitions 

(e.g., between the ROP and Maintenance Rule) need to be addressed. Feedback from public

stakeholders was mixed. While some members of the public believe that the ROP is overall a

more effective oversight process, others felt that the ROP is poorly focused, will not identify

declining performers in a timely manner, will not result in adequate assurance of safe reactor

operation, and view the new ROP as a step backwards. 

The IIEP reviewed the results and lessons learned for ROP initial implementation and 

concluded that the ROP is a notable improvement over the previous licensee performance

assessment program. The IIEP determined that the ROP has made progress toward achieving

the Agencyís four performance goals. The panel recommended, however, that the staff take

certain actions to ensure that it achieves the Agency performance goals in the long-term, and

consider other actions to improve the process. Most of the IIEP conclusions and recommended

actions were consistent with the staffís evaluation of the results of initial implementation. 

In addition to utilizing feedback to provide insights regarding the efficacy of the ROP, the staff

took advantage of situations encountered during initial implementation to identify lessons 
- 3 -

learned and improvement opportunities. For example, following the February 2000 steam

generator tube failure at Indian Point Unit 2 (IP2), NRC inspectors identified what was

characterized as a Red inspection finding associated with the licenseeís previous steam

generator tube examination. This issue, in combination with other existing performance

concerns, placed IP2 in the Multiple/Repetitive Degraded Cornerstone column of the Action

Matrix which resulted in a supplemental inspection effort under the guidance of inspection

procedure (IP) 95003. This was the first experience under the new program with a plant in this

column and the first full exercise of this supplemental inspection procedure. This experience

highlighted that the resources associated with providing oversight for a plant in the

Multiple/Repetitive Degraded Cornerstone column of the Action Matrix are significant and will

challenge the regionís ability to perform routine inspections at other sites. Additionally, an

extensive effort was necessary to appropriately interact with external stakeholders. As a result

of specific lessons learned, adjustments to resource estimates and planning models were made

and changes were made to strengthen inspection procedures and guidance. 

In response to a Yellow Alert and Notification System Reliability performance indicator (PI) at

Kewuanee, supplemental inspection efforts identified significant weaknesses in the licensee's

efforts to promptly address the underlying performance issue. The specific actions taken by the

licensee eventually returned the PI to the licensee response band (i.e., Green) after four

quarters (the PI was Yellow for two quarters, then White for one quarter, before becoming

Green the following quarter). This was the first time that supplemental inspection had identified

inadequate corrective actions under the new program. The staff is strengthening the program

guidance in this area to ensure that the performance issue remains open until the licensee has

responded sufficiently to address the issue.

Finally, the operational safeguards response evaluation (OSRE) findings at Quad Cities Station

identified significant weakness in the physical protection SDP. In response, the staff

developed, and the Commission approved, an interim SDP that is currently in use. 

Development of a final revised physical protection SDP is currently underway.

Based on its assessment of stakeholder feedback and the results and lessons learned from

initial implementation, the staff has developed a much greater level of confidence that the ROP

has met the Commissionís direction to develop an oversight process that is more objective, riskinformed, understandable, and predictable. The ROP has been substantially exercised such

that the staff has gained valuable insights on many aspects of the ROP and identified issues

that were not revealed during the pilot program. The staff recognizes that the ROP will

continue to require close scrutiny and oversight and established a self-assessment program

that will identify additional areas for improvement. The staff, as part of the Agency Action

Review process, will continue to report to the Commission on an annual basis the results of its

self-assessment and any significant changes to the ROP. 

As it is currently designed and implemented, the resource requirements for the ROP are

comparable to the requirements for the previous oversight program, and the ROP can be

completed with the resources currently allocated. Potential areas for resource efficiencies have

been identified based on the results of the staffís evaluation of ROP initial implementation, and

savings resulting from these potential efficiency gains will be incorporated into the resource

projections for future years. However, these will need to be weighed against emerging

programs and policies that may impact future resources.
- 4 -

BACKGROUND:

On January 8, 1999, the staff issued SECY-99-007, ìRecommendations for Reactor Oversight

Process Improvements,î forwarding the staffís recommendations for a revised reactor oversight

process for commercial nuclear power plants. On March 22, 1999, the staff issued SECY-99-

007A, ìRecommendations for Reactor Oversight Process Improvements (Follow-up to SECY99-007),î that forwarded to the Commission additional information on the concepts for the ROP,

and presented the staffís plans for a 6-month pilot of the revised oversight processes at two

sites per region. The June 18, 1999, SRM on SECY-99-007 and SECY-99-007A approved the

scope and concepts for the ROP, approved the staff plan for the pilot program, and provided

additional issues and Commission comments for the staff to consider when evaluating the pilot

program results.

The results and lessons learned from the pilot program were documented in Commission paper

SECY-00-0049, ìResults of the Revised Reactor Oversight Process Pilot Program,î dated

February 24, 2000. Based on the results of the pilot program, the staff recommended that the

Commission approve initial implementation of the ROP at all operating commercial nuclear

power plants (except for the D.C. Cook plant). The staff recommended that the D.C. Cook

plant not be included in the initial implementation of the ROP at that time due to the licenseeís

continued efforts to implement corrective actions to address significant performance concerns

and the increased Agency focus to monitor and evaluate these licensee efforts. Following the

restart of both D.C. Cook units, the inspection manual chapter (IMC) 0350 Oversight Panel

continued to provide oversight due to the lack of valid historical PI data, and the continuing

need to assess operational performance and the effectiveness of the licenseeís long term

corrective actions. The status of the transition of D.C. Cook into the ROP will be discussed at

the Agency Action Review Meeting in June 2001.

The SRM for SECY-00-0049 was issued on March 28, 2000, and approved initial

implementation of the ROP as recommended by the staff. Initial implementation commenced

on April 2, 2000. A follow-up to the SECY-00-0049 SRM was issued on May 17, 2000, and

provided several additional issues for the staff to consider during initial implementation. This

SRM also directed the staff to report on the implementation of the ROP results after the first

year of use. This paper constitutes the staff response to that requirement.

DISCUSSION:

Initial Implementation Assessment Activities

During the first year of ROP implementation, the staff conducted many activities to evaluate the

new oversight process for lessons learned and to collect internal and external stakeholder

feedback and comments. A general discussion of the different means of assessing the efficacy

of the ROP and a summary of the overall results follows. A detailed discussion of the

evaluation and disposition of the stakeholder feedback and self-assessment results is provided

in the attachments to this paper for each component of the ROP. 
- 5 -

Internal Stakeholder Feedback

Internal feedback and comments were obtained using various methods, such as: (1) periodic

meetings between headquarters and regional staff, (2) visits by headquarters staff to the

regional and resident inspector offices, (3) the use of feedback forms to allow regional and

headquarters staff to document questions and concerns on the various components of the

ROP, and (4) an internal stakeholder survey conducted near the end of initial implementation. 

Based on the staff survey results and other feedback received, internal stakeholders generally

had a more positive view of the ROP following the first year of implementation than they had

following the 6-month pilot program, and showed a marked increase in their understanding and

acceptance of various components of the ROP. They felt that the process provides appropriate

regulatory attention to licensees with performance problems, is objective, and is an effective

risk-informed approach to oversight. Internal stakeholders did express several concerns with

the ROP. A majority of internal stakeholders felt that the level of effort for conducting each

inspection was not consistent with that estimated in the inspection procedures. Internal

stakeholders were also concerned that the thresholds for documenting inspection findings were

too restrictive and that the ROP does not appropriately integrate and provide insights into crosscutting areas. The majority of internal stakeholders indicated that the SDPs are not easy to

use, and did not believe they are based on clear standards. A significant percentage of internal

stakeholders expressed concern regarding the ROPís ability to provide appropriate

identification of declining safety performance before a significant reduction in safety margins

occurs. The results of these site and regional visits, meetings, workshops, and staff survey are

discussed in more detail in Attachment 1.

External Stakeholder Feedback

Comments and feedback on the ROP from external stakeholders such as public interest

groups, industry representatives, and State and local government agencies were also solicited

in many different ways. These included public working group meetings held at least monthly

during initial implementation, evening public meetings at each of the power plants with local

citizens, visits by headquarters staff to six sites per region, public forum meetings held in the

vicinity of each regional office at the mid-point of initial implementation, regional management

site visits, and a Lessons Learned Public Workshop held at the end of initial implementation.

Finally, public comment on the initial implementation of the ROP was requested in a Federal

Register notice (FRN). 

In general, external stakeholders indicated that the ROP improved consistency, reduced

unnecessary regulatory burden, and increased the predictability of Agency actions. The

industry, and many public stakeholders, perceive the ROP as being more objective and

scrutable, with an increase in regulatory focus on risk significance. Public and industry

stakeholders were generally very positive about the use of the NRC external internet web page

to provide pertinent reactor oversight information and assessment results in a timely and well

organized manner. 

Industry and public stakeholders did have numerous comments and concerns targeted at

improving various parts of the ROP. For example, industry and some non-industry

stakeholders generally are of the opinion that, with the development of additional PIs and the

refinement of some existing PIs, there are additional opportunities for increased efficiencies,
- 6 -

such as the reduction in frequency of some inspections. On the other hand, other external

stakeholders, particularly States, cautioned the NRC not to prematurely reduce inspection

effort. The industry also strongly felt that inconsistencies and overlap between the ROP,

Institute of Nuclear Power Operations (INPO), and Maintenance Rule indicator definitions for

safety system unavailability need to be addressed. A number of industry and public

stakeholders felt that the characterization of safety significance by the SDPs was slow and

complex to the point of being burdensome.

Feedback from public stakeholders was mixed. Entities such as the Commonwealth of

Pennsylvania and the Union of Concerned Scientists believe that the ROP is overall a more

effective oversight process. Others, such as the State of New Jersey, believe that the ROP is

poorly focused, will not identify declining performers in a timely manner, and will not result in

adequate assurance of safe commercial nuclear reactor operation. Some public stakeholders

even viewed the new ROP as a step backwards, and felt that the NRC has not communicated

successfully with the public regarding NRC oversight activities and the safety performance of

licensees. A summary of these various meetings, workshops, and FRN comments can be

found in Attachment 2. 

Self-Assessment Program

During initial implementation, the staff developed a self-assessment process to collect

additional lessons learned and gain insights from the implementation of the ROP. This ongoing

and continuous self-assessment process utilizes objective metrics and pre-determined criteria

to monitor the performance of various aspects of the ROP. 

In general, the self-assessment metrics indicate that the ROP has been successful in being

more objective, risk-informed, understandable, and predictable than the previous process. 

However, the metrics were less conclusive regarding the performance of the ROP against the

Agencyís performance goals (maintain safety; increase public confidence; improve

effectiveness, efficiency and realism of NRC activities and decisions; and reduce unnecessary

regulatory burden) due to the limited and inconclusive data obtained during initial

implementation. More time is needed to gauge the overall impact of the ROP on safety. The

ROP self-assessment program is a continuous process that includes responding to noted

deficiencies and incorporating improvements in an ongoing manner. Additional detail on the

ROP self-assessment program and the results and lessons learned from initial implementation

are discussed in Attachment 3.

Initial Implementation Evaluation Panel

The IIEP was established in accordance with FACA requirements to serve as an advisory

committee to the Agency. This panel was comprised of a cross-disciplinary group of NRC,

public, and industry stakeholders representing many different nuclear power interests. The

purpose of the IIEP was to monitor and evaluate the results of the first year of initial

implementation and to provide advice and recommendations to the Agency on reforming and

revising the ROP. The results of the IIEP review are included as Attachment 4 to this paper. 

Overall the IIEP concluded that the ROP is a notable improvement over the previous licensee

performance assessment program, and that it should be continued. The IIEP determined that

the ROP has made progress toward achieving the Agencyís four performance goals. The panel
- 7 -

recommended, however, that the staff take certain actions to ensure that the Agency

performance goals are met in the long-term, and to consider other actions to improve the

process. Most of the IIEP conclusions and recommended actions were consistent with the

staffís evaluation of the results of initial implementation. These panel recommendations were

factored into the staff evaluation, and are addressed by the process changes described

throughout this paper. These IIEP recommendations and conclusions, and a summary of the

staffís actions in response, are discussed in more detail in Attachment 5.

Summary of Results and Actions

The following discussion provides the significant lessons learned from ROP initial

implementation, process changes made resulting from the lessons learned, and remaining work

needed to address longer term issues. 

Performance Indicators

Feedback received on the PIs during initial implementation reemphasized industry concerns

with potential unintended consequences with the scram PIs and NRC staff concerns with

potential unintended consequences associated with the unplanned power change PI. The staff

also received significant feedback on the definitions and thresholds for the safety system

unavailability (SSU) indicators. The most significant SSU issues raised by stakeholders

included: (1) the use of fault exposure hours (FEH), both known and estimated (t/2), (2) the

impact of the thresholds on effective preventive maintenance, and (3) inconsistencies between

the various programs that monitor the unavailability of safety systems (e.g., Maintenance Rule,

Probabilistic Risk Assessments (PRA), INPO/World Association of Nuclear Operators (WANO)

indicators, and ROP indicators). Additionally, several outstanding issues remained regarding

the definitions and thresholds of some of the indicators, including the Alert and Notification

System (ANS) reliability, the Protected Area Security Equipment Performance Index, and the

use of reportable events in the security program performance indicators.

Several actions were taken prior to and during initial implementation to address these concerns

and the lessons learned from both the 6-month pilot program and initial implementation. Many

of the issues and lessons learned from the pilot program were addressed in Revision 0 to NEI

99-02, ìRegulatory Assessment Performance Indicator Guideline,î issued March 28, 2000. In

particular, the guidance for the SSU indicators was changed to resolve several of the FEH

issues. During initial implementation, through the frequently asked question (FAQ) process, the

guidance was clarified to describe the conditions under which on-line overhaul maintenance

may be performed without counting unavailable hours. Also during initial implementation, most

of the FAQs that had been resolved during both the 6-month pilot program and initial

implementation were incorporated into Revision 1 to NEI 99-02, issued April 23, 2001. 

Additionally, a 6-month pilot program for two new indicators to replace the two scram PIs was

completed in March 2001. The results are currently being evaluated against the criteria

established in Regulatory Issue Summary (RIS) 2000-21, ìChanges to the Unplanned Scram

and Unplanned Scram with Loss of Normal Heat Removal Performance Indicator.î 

Several longer term actions are also planned to address many of these issues. The staff has

worked with the industry and other external stakeholders to develop a replacement indicator for

the ìUnplanned Power Changes per 7,000 Critical Hoursî PI and plans to pilot two potential
- 8 -

alternative indicators in the near future. In addition, the staff is working with the industry and

external stakeholders to improve the SSU indicators by simplifying them, making them easier to

understand, and increasing their compatibility with all uses (e.g., Maintenance Rule, PRA,

INPO/WANO indicators, and ROP indicators). 

Additional detail and discussion of the initial implementation results, lessons learned, and

program changes made for PIs can be found in Attachment 6. 

Inspection Program

Although much of the data collected for the self-assessment metrics are preliminary and only a

limited set of definitive conclusions can be drawn from just one year of full implementation, the

inspection program is generally meeting the established goals and measures. However, some

lessons were learned and require additional work to resolve.

The majority of NRC internal stakeholders believe that inspection reports provide sufficient and

proper information. However, a criticism from several external stakeholders was that many

inspection reports didnít adequately document the bases for determining the significance of

findings. An ongoing independent audit of inspection reports by the NRC staff confirmed this

criticism. Inspectors noted that several inspection areas yielded issues whose significance

could not be easily determined, or that could not be documented in inspection reports. The two

most notable areas were security force-on-force exercises and Maintenance Rule inspections. 

The staff acknowledges that both areas require changes to the inspection procedures. As

discussed later, the physical protection SDP and documentation thresholds have already been

changed to address these issues.

Stakeholders expressed several other concerns and opinions regarding the inspection program. 

Many stakeholders felt that the use of collective dose as a screening criterion for inspection

findings involving licensee as low as reasonably achievable (ALARA) programs could cause

identical findings at different plants to be treated differently. Several industry stakeholders

expressed interest in using licensee self-assessments to reduce the scope and effort of the

baseline inspection program. Many internal stakeholders expressed dissatisfaction with what is

perceived to be the reactive nature of the baseline inspection program and its reduced 

emphasis on finding the causal factors for problems.

During initial implementation, a few baseline inspection program procedures had to be

significantly revised to modify their inspection scope, frequency, or level of effort. For example,

midway through initial implementation, the staff revised IMC 0610*, ìPower Reactor Inspection

Reports,î to clarify: (1) the thresholds for documenting inspection findings, (2) the definition of a

ìdocumentableî cross-cutting issue, and (3) the expectations for documenting the basis for the

significance of findings. 

Several additional baseline inspection program procedures currently are being changed to

adjust the inspection scope, frequency, or level of effort (including some potential increases),

based on lessons learned. For example: (1) the procedure for inservice inspections will be

changed based on staff recommendations as described in the steam generator action plan and

the lessons learned from the failed steam generator tube at IP2, (2) the maintenance

effectiveness inspection procedure is being rewritten to make it more risk-informed and

performance based, and (3) the approach to the periodic inspection of problem identification
- 9 -

and resolution (PI&R) is being changed to make the inspections more effective and efficient by

allowing a more in-depth (albeit less frequent) assessment of a licenseeís program, while

emphasizing the routine follow-up of individual issues through a more focused review of PI&R

samples throughout the year. 

The staff also plans several longer term actions to address lessons learned. The Physical

Protection inspection procedure will be revised over the next year to account for policy changes

associated with this cornerstone. The staff will evaluate how licensee self-assessments in this

area might be used to satisfy some of the requirements of the baseline inspection program. 

Likewise, the ALARA baseline inspection procedure and other program documents will be

revised after the bases for determining what constitutes an ALARA inspection finding is reanalyzed. Additional effort is needed to evaluate the quality of staff efforts under the ROP,

particularly the quality of inspection and the depth and scope of inspection preparation. These

areas are difficult to measure but are important.

Additional detail and discussion of the initial implementation results, lessons learned, and

program changes made for the inspection program are discussed in Attachment 7. 

Significance Determination Process

The SDP was judged to be generally effective during initial implementation. The SDP focused

NRC and licensee resources commensurate with the significance of findings, although further

efficiency improvements need to occur. Inspection audit findings and stakeholder feedback

indicated that some findings processed through the SDP may not have met the guidance for

level of detail in documenting assumptions made during the analysis. Some public feedback

continued to reflect their difficulty in understanding the bases for SDP results, particularly those

that were based on licenseeís risk analyses. Many stakeholders expressed concern regarding

the difficulties experienced in using the reactor safety and non-reactor safety SDPs and with the

effort and time needed to finalize the safety significance of issues.

The NRC staff will continue to work to improve the timeliness of the SDP process while

ensuring that the final significance determination adequately portrays the safety significance of

the issue. One of the strengths of probabilistic risk assessments is that uncertainties can be

treated explicitly. In fact, the SDP process helps to reveal influential assumptions that are used

to arrive at any particular significance determination. This openness is intended to provide

greater ability for stakeholders to challenge the validity of the assumptions in order to increase

the likelihood that invalid assumptions will not be used. Achieving convergence on influential

assumptions and thereby narrowing the uncertainties is very important, since risk assessments

having a range of uncertainty are ultimately evaluated against finite SDP thresholds under the

ROP. The openness of the SDP process improves the defensibility and credibility of the SDP

outcome. However, the resultant increase in the internal dialogue and interaction with external

stakeholders extends the time spent in arriving at a final significance determination. A

significant ongoing challenge with the SDP will be to balance the need for sufficient

communication to allow the staff to make an appropriately informed significance decision and

the need for timeliness of SDP results to support the performance assessment process. In

spite of these difficulties, the SDP was generally found to enhance public confidence by

accurately communicating information in a way that demonstrates that the NRC understands

the plantís performance. 
- 10 -

Some stakeholders also expressed concern regarding the lack of external events risk data, the

lack of detailed guidance for assessing concurrent performance deficiencies, and the method to

appropriately address long standing design basis deficiencies. There was also significant

stakeholder feedback on the Occupational Radiation Safety and ALARA SDPs. The

Occupational Radiation Safety SDP excluded all hot particle exposures, including whole body

(deep dose equivalent), as well as skin dose from hot particles not subject to the enforcement

discretion. The process screened out all ALARA issues identified at plants that have a rolling 

3-year average collective dose below the screening criteria, which resulted in a number of

unintended outcomes. Finally, the staff determined that the force-on-force inspection findings

posed a significant challenge in applying the Physical Protection SDP (PPSDP). Specifically,

weaknesses with the PPSDP methodology made the application of risk, as was originally

intended, unsuitable in that the risk-informed approach to the PPSDP could routinely result in

inappropriate NRC responses to certain OSRE findings. 

The staff made several changes to the SDP as a result of lessons learned and the stakeholder

feedback received during initial implementation. Revisions to the inspection reporting guidance

completed during initial implementation are expected to help address the lack of detail seen in

some inspection reports for documenting the assumptions used in reaching a significance

determination. In addition, the staff has established an SDP instructional guide on the internal

NRC web site that, when fully developed, will provide a wide spectrum of example performance

issues and how they should be processed using the various SDP tools. The fire protection SDP

was revised to provide additional information to enhance the inspectorsí ability to assess

functionality of fire protection defense-in-depth components and to provide guidance on the

development of credible fire scenarios. The staff issued a Commission approved interim

PPSDP to address significance determinations for findings identified during the force-on-force

exercises. This interim SDP appropriately focuses the significance determination on security

response and not on reactor recovery. Lastly, the development and issuance of the reactor

safety SDP plant/site specific Risk-Informed Inspection Notebooks is expected to be completed

during the Summer of 2001.

Several longer term efforts to continue improving the SDP are also in progress. Stakeholder

concerns regarding ease of use are being addressed for fire protection, reactor safety and

physical protection SDPs. Enhancements to the guidance are being evaluated with regard to

applying combined risk results of multiple concurrent deficiencies in a manner that is

commensurate with the ROP Action Matrix. The NRC Office of Research (RES) has been

engaged to evaluate the scope of effort necessary to improve the capability of the Reactor

Safety SDP by assessing the impact of external events on operating reactor safety related

issues. Additionally, changes to the ALARA SDP screening criteria are being considered,

including the possible elimination of the 3-year average collective dose threshold, and instead

using that data as a means to adjust the baseline inspection effort. The staff continues to

develop methodologies that will allow inspectors to develop realistic fire scenarios and improve

the accuracy of site specific data used in the assessment of risk associated with findings. The

staff will replace the interim PPSDP with a revised version that will be developed with internal

and external stakeholder input.

Additional discussion and detail on the initial implementation results, lessons learned, and

program changes for the SDP are described in Attachment 8. 
- 11 -

Assessment

The assessment process proved to be objective and predictable with the actions taken by each

regional office appropriate and consistent with the Action Matrix. No deviations from the Action

Matrix were required during the first year of implementation. Most stakeholders agreed that the

assessment process focused NRC resources on the areas of greatest safety significance and

that the regulatory attention given to licensees with performance problems was appropriate. 

Stakeholders also generally agreed that assessment results were issued and posted on the

external Agency Web page in a timely manner. 

The staff received numerous issues for consideration from both internal and external

stakeholders. For example, guidance had not been established on how the staff would treat

instances in which significant weaknesses were found in licensee actions to respond to

performance issues that resulted from a PI that crossed thresholds in a manner consistent with

how similar weaknesses were treated for an inspection finding. Also, the use of ìNo Colorî

findings in the oversight process was not clear and had contributed to program inconsistencies. 

Several program changes have been made to reflect lessons learned from implementing the

new assessment process. The staff incorporated many of these lessons learned into revisions

of IMC 0305 ìOperating Reactor Assessment Program.î For example, the staff revised IMC

0305 to ensure that the EDO pre-approves all deviations from the Action Matrix, and that the

Commission is informed of any such deviations. 

Several longer term actions are also planned to address lessons learned. In particular the staff

will continue to monitor the ROP to determine whether it would be appropriate to have a graded

approach for removing inspection findings from consideration in the assessment process after

adequate corrective actions are taken by the licensee. The staff will continue to evaluate

appropriate actions to address the concerns with ìNo Colorî findings and the appropriate

Agency response to ineffective actions taken by a licensee in response to a performance

deficiency identified by a PI. Additionally, the staff will implement any lessons learned from the

End-of-Cycle Review Meetings, the Agency Action Review Meeting, and the annual public

meetings with the licensees.

The results of initial implementation of the new assessment process, including lessons learned

and program changes, are discussed in more detail in Attachment 9.

Enforcement 

The NRC Enforcement Policy was revised on May 1, 2000, to incorporate the Commission's

decision to authorize blanket discretion for potential 10 CFR 50.9 issues related to inaccurate

licensee PI data submittals. This policy remained in effect through January 31, 2001. Although

this blanket enforcement discretion expired, this does not impede the staff's ability to exercise

enforcement discretion under Section VII.B.6 of the Enforcement Policy on a case-by-case

basis for inaccurate or incomplete PI data. Additionally, the staff may consider exercising

discretion under Section VII.B.6 for any new PIs developed. The Office of Enforcement (OE)

will continue to work with NRR and the regional offices in determining whether enforcement

discretion should be exercised for future PIs related to 10 CFR 50.9 violations. 
- 12 -

During initial implementation, OE monitored the impact of the ROP on the timeliness with which

the NRC staff processed significant inspection findings (i.e., White, Yellow, and Red) and

associated violations, if applicable. Analysis of the timeliness data for the SDP and

enforcement process shows that there has been an increase in the amount of time that it takes

to process findings under the revised ROP. As of the end of initial implementation phase, the

average process time for all cases was 98 days, compared to an average of 75 days achieved

in FY 1999 for processing escalated enforcement actions (under the previous process). A

major contributor to the increase was the greater emphasis on risk analysis to support the

significance characterization of inspection findings.

Experience with the Maintenance Rule Enforcement Review Panel during ROP initial

implementation demonstrated that many of the Maintenance Rule issues identified were able to

be dispositioned consistently without a formal panel meeting. In addition, under the ROP,

Maintenance Rule findings have either been minor or, if more than minor, of very low safety

significance (no greater than Green).

Many actions are planned to address these lessons learned. The staff will continue to pursue

ways to make the SDP and enforcement processes more timely for evaluating and

communicating safety significant issues. In the near term, the staff will issue revised

Maintenance Rule enforcement guidance to establish a less formal review process.

Lessons learned from the first year of implementing the new enforcement policy for the ROP

are discussed in further detail in Attachment 10. 

Key Policy Issues

Commission paper SECY-00-0049 identified three ìIssues of Noteî which were issues that

emerged from the 6-month pilot program that had a significant impact on the implementation of

the ROP, and that the staff had not been able to resolve in a manner that adequately

addressed the majority of stakeholder concerns. These issues involved: (1) the treatment of

cross-cutting issues in the ROP, (2) role of Barrier Integrity cornerstone PIs in the ROP, and (3)

how the ROP should treat performance issues that are outside the licensing and design basis

of the plant. The staff committed to present these issues, the differing points of view, and the

staffís proposal for their resolution to the Commission for consideration. Additionally, SECY-00-

0049 provided an update to several other key policy issues that needed to be considered and

addressed as part of the ongoing implementation of the ROP. These issues included: (1) the

impact of the ìN+1" resident inspector staffing policy change on the implementation of the ROP,

(2) the impact of the ROP on regional organization, including staff training needs, (3) the impact

of implementing the ROP on the Agencyís allegation program, and (4) staff efforts to develop

risk-based PIs and an industry trending program.

With regard to the impact on the ROP of the January 2000 resident inspector staffing policy

change (i.e., ìN+1" to ìNî), the staff concluded that baseline inspections can be accomplished

with ìNî resident staffing at dual unit sites with some assistance from region-based inspectors. 

Sites that continued to be staffed at ìN+1" at dual unit site tended to expend a level of effort

commensurate with site staffing, not program requirements. However, the staff believes that it

is still too soon to determine the full impact of the change in the resident staffing policy. The

early indications are that the ìNî staffing policy will challenge regional management to provide
- 13 -

extended training, rotational assignments and other professional development opportunities for

the resident inspector staff at multi-unit sites.

The status of all of these key policy issues is discussed in more detail in Attachment 11.

D.C. Cook Transition Plan

Prior to the initial implementation of the ROP in April 2000, the licensee for D.C. Cook

requested a delay in the implementation of the ROP at that site. The basis for the licenseeís

request was to minimize the impact of the transition to the new oversight process on plant staff,

which at that time was focused on Unit 2 restart activities. Both D.C. Cook units had been

shutdown since September 1997, and IMC 0350, ìOversight of Operating Reactor Facilities in

an Extended Shutdown Condition,î was being implemented by the NRC. As discussed in

SECY-00-0049, the NRC staff approved the delay in ROP implementation at D.C. Cook until

after the Unit 2 restart.

With the startup of the Unit 2 reactor in June 2000, the NRC began implementation of the ROP

baseline inspection program and the revised assessment process. Specifically, the baseline

inspection program was implemented with additional inspections conducted in areas where PI

data was either not available or was under development. Due to the extended outage, PIs that

rely on plant operating history could not be fully developed. The SDP was utilized to evaluate

inspection findings and the Action Matrix was considered in determining necessary follow up

actions. In addition, under the cognizance of the IMC 0350 Oversight Panel, inspections were

conducted in specific areas that needed to be evaluated to assess the restart readiness of D.C.

Cook Unit 1. Following the restart of D.C. Cook Unit 1 in December 2000, the IMC 0350

Oversight Panel continued to provide oversight of D.C. Cook because of the lack of validity in

the historical PI data, and the continuing need to assess operational performance and the

effectiveness of the licenseeís long term corrective actions. This augmented baseline

inspection program will continue at D.C. Cook until sufficient PI data has been accumulated to

permit ìnormalî implementation of the ROP. The status of the transition of D.C. Cook into the

ROP will be discussed at the Agency Action Review Meeting (AARM) to be held in June 2001. 

In addition, to support the increased baseline inspection effort, the D.C. Cook resident office

continues to be staffed at the ìN+1" level.

Staff Response to SECY-00-0049 Staff Requirements Memorandum (SRM)

The SECY-00-0049 SRM, dated May 17, 2000, listed several issues that the Commission

requested the staff to consider during the first year of ROP implementation, including: (1) the

EDO should pre-approve all deviations from the ROP Action Matrix, (2) the staff should

emphasize the importance of licensee corrective action programs in ongoing communication

efforts, and (3) the staff should ensure that the threshold for documenting inspection

observations is clearly understood and consistently applied. The staff has considered all of

these issues, along with the lessons learned from the ROP initial implementation, in

determining oversight process changes and improvements. Many of these issues were

described previous sections of this paper. All of these issues, along with recommendations of

the Pilot Plant Evaluation Panel, are discussed in detail in Attachment 12. 
- 14 -

RESOURCES:

As a result of the first year of ROP implementation, the staff has concluded that the resources

budgeted for initial implementation of the new oversight process were appropriate. Specifically,

the results of initial implementation indicated that: (1) current regional resources were adequate

to carry out the ROP effectively and to achieve its stated objectives, (2) the current regional

resource model, modified based on the results of the first year of ROP implementation,

provided reasonably accurate estimates of regional resource requirements to implement the

ROP, and (3) overall ROP resource requirements during initial implementation were

comparable to the overall requirements in the previous program, although the resources were

allocated differently. 

Based on the lessons learned to date, the staff believes that it would be premature to forecast

further reductions in the resources to implement the ROP. As discussed below, the staff does

believe that there are opportunities for continued efficiency gains in several areas. These areas

will be assessed and reported to the Commission with the ROP self-assessment results,

following the second year of implementation of the new program. It is important to note that

actual resource expenditures are directly related to industry and individual plant safety

performance.

Future resource reductions may be possible in the ROP through continued efficiency gains as a

result of: (1) the elimination of start-up costs, (2) inspection preparation time decreasing as staff

becomes more familiar with process and procedures, (3) improved documentation methods

(such as implementation of quarterly integrated inspection reports in all regions for routine

inspections), (4) the use of SDP workbooks at all sites, and (5) reduced frequency and level of

activities related to licensee performance assessment. Resource savings may also be

achieved if ongoing staff review and evaluation recommends a reduction in the scope and/or

frequency of major team inspections or other baseline inspections. 

The NRC has already implemented reductions in resource expenditures for reactor oversight

(inspection and performance assessment) over the past five years to reflect improved industry

performance and anticipated efficiencies in NRC oversight. Prior to the initial implementation

phase of the ROP, the Commission approved the staffís recommendation not to make 

additional resource adjustments to the inspection and assessment programs until adequate

experience was gained with the revised ROP. A full year of implementation beyond the 6-

month pilot program was planned to obtain reliable data on which to estimate the resources

needed to accomplish individual inspections as well as to execute the overall oversight process. 

Two fundamental issues were addressed in assessing the ROP resource requirements,

specifically: (1) whether the ROP could be completed within the estimated resources, and (2)

whether the ROP is more efficient in resource utilization compared with the previous inspection

and assessment programs.

In order to assess the resource requirements of the revised ROP, new Regulatory Information

Tracking System (RITS) codes were developed, and detailed guidance was provided to the

regional staff so that staff time spent on specific direct and indirect activities in the ROP could

be more accurately tracked. A comparison of the actual hours expended for baseline

inspection activities under the ROP to the initial estimates for single-, dual-, and triple-unit sites

shows that, with the exception of the two triple-unit sites, the actual resource requirements to

accomplish direct inspection, inspection preparation and documentation, and plant status
- 15 -

activities were less than initial estimates. However, indirect activities (e.g., inspection-related

travel, communication activities including stakeholder interface, significance determinations,

and regional technical support) were greater than expected and increased the overall workload

substantially. Although these indirect activities were not tracked separately between the

baseline and supplemental inspection program areas, much of the indirect effort involved

necessary support to baseline inspection activities.

The total direct inspection effort expended on plant-specific inspection during this first year

was reasonably close to the initial estimate; however, the resource allocations in this area

need to be refined. The inspection effort to address plants in the Regulatory Response

column of the Action Matrix were significantly lower than estimated, while the inspection effort

to address the plants in the Degraded Cornerstone column were accurately estimated. At IP2,

the only case involving a plant in the Multiple/Repetitive Degraded Cornerstone column of the

Action Matrix, the magnitude of effort required for supplemental inspection and to provide

oversight of the licenseeís performance improvement plan was unanticipated and significant. 

The Agencyís response to IP2 not only consumed considerable effort for direct inspection and

inspection preparation and documentation, but also significantly increased the expenditures of

other indirect activities. 

A comparison was also made of the overall resource expenditures of the ROP during the first

year of implementation with the resources expended under the previous oversight program for

the same time period prior to initial implementation. It should be noted that such a comparison

is difficult and problematic because neither the 52 weeks prior to or following the start of initial

implementation can be considered typical, and the two oversight programs differed

significantly (e.g., no Systematic Assessment of Licensee Performance (SALP) and substantial

ROP development effort). Notwithstanding this concern, the first year of implementing the

ROP resulted in the staff using slightly more resources overall than what was applied to the

oversight process in the prior year. As expected, the staff applied more resources in executing

the baseline inspection program than the previous core inspection program, and less

resources in conducting plant specific inspections (reactive and supplemental) than in the past

(which includes regional initiative).

Several other factors may impact future ROP resource requirements that are difficult to

estimate. These include: (1) institutional inefficiencies that may result from full implementation

of ìNî resident inspector staffing policy (e.g., potentially greater inspection preparation and

travel time), (2) reduced inspector efficiency due to increasing the number of vacancies filled

by entry level employees, (3) large demands on inspectors caused by unplanned issues and

events that result in significant plant specific inspections, and (4) the potential need for

additional inspection resources due to increases in the number of independent spent fuel

storage installation (ISFSI) facilities.

The staff will continue to collect additional data in order to improve the resource model. Areas

of consideration for potential model refinement include: (1) anomalies in regional resource

usage (e.g., increased travel requirements for Region IV inspectors; additional resources for

dual-unit sites that are significantly different in design, vintage or management), (2) impacts of

plants with multiple degraded cornerstones, (3) explicit inclusion of indirect inspection

activities, and (4) more accurate data for performance assessment expenditures.

Additional detail on the ROP resource analysis can be found in Attachment 13.
- 16 -

CONCLUSIONS:

The first year of implementation has improved the NRC staffís confidence that the ROP is

capable of ensuring adequate protection of public health and safety through a more objective

and risk-informed approach to the oversight of commercial nuclear power plant operation. The

initial implementation phase of the ROP did not identify any significant aspects of licensee

performance that were not adequately covered through the combination of PIs and inspection. 

The use of risk-informed safety thresholds for both PIs and inspection findings allowed the

NRC and licensees to focus their respective resources on issues with the most safety

significance. NRC actions specified by the assessment process Action Matrix were

appropriate for the safety concerns identified. The submittal of PIs by the licensees provided

performance data that were more timely and relevant than what was available under the

previous oversight process. Information was more readily accessible to the public through the

posting of the PIs and inspection findings on the NRCís internet web page. The bases for

NRC actions were more easily understood through the use of the assessment process Action

Matrix.

Initial implementation results indicated a notable shift in inspection resource expenditure, with

more time required for inspection preparation, and less time required for inspection

documentation. However, overall program resources, including inspection, assessment,

enforcement, inspection preparation and documentation time, were comparable to the

requirements in the previous program. Currently budgeted regional resources are adequate to

carry out the ROP effectively and achieve its stated objectives.

Based on industry and internal NRC feedback, the regulatory burden associated with the ROP

appears appropriate. Though more licensee resources were required to support the data

collection and reporting associated with PIs, this burden increase was offset by changes in the

inspection, assessment, and enforcement processes. These changes allowed licensees to

focus their resources more efficiently on issues with the greatest safety significance. Industry

representatives felt that the SDP in particular provided the means for both the NRC and its

licensees to better utilize their resources.

As described in Attachment 4, the IIEP concluded that the ROP should be continued because

it is a notable improvement over the previous licensee performance assessment program, and

that the ROP has made progress toward achieving the NRCís four performance goals. 

However, the panel also recommended that the staff take certain actions to ensure that they

continue to achieve the Agencyís performance goals in the long-term, and to consider other

actions to improve the process. Most of the IIEP conclusions and recommended actions were

consistent with the staffís evaluation of the results of initial implementation.

Even with the success achieved during ROP initial implementation, it is still too early to judge

the impact of the ROP in several areas. For example, it will take more than one year of

experience and data to understand the full impact of the ROP on overall industry performance

and determine whether cross-cutting issues can cause a rapid decline in performance without

prior indication through PIs and inspection. As described in Attachment 11, the staff is working

to develop a trending program to provide a means to monitor overall industry performance. 

Additionally, several issues have been identified which warrant improvements to the ROP to

address lessons learned from the first year of implementation. While many changes have
- 17 -

already been made, others involve longer term actions to develop and enhance the process. 

The following are the most significant long term actions planned by the staff:

Performance Indicators

ï Begin a pilot program in July 2001 for a replacement unplanned power changes

indicator entitled ìUnit Power Reductions per 7,000 Critical Hours.î

ï Continue to work with the industry to improve the existing SSU indicators to

simplify them, make them more consistent between the various programs that

monitor the unavailability of safety systems (e.g., Maintenance Rule, PRA,

INPO/WANO indicators, and ROP indicators), and make them more riskinformed.

Inspection Program

ï Continue to evaluate and, as necessary, revise the guidance for documenting

inspection findings to ensure that significance thresholds are consistently

applied.

ï Revise the Physical Protection cornerstone inspection procedure and its

attachments to account for significant changes and new policies in physical

security.

ï Continue to clarify the basis for evaluating ALARA inspection findings and will

revise the associated inspection procedure as needed.

ï Refine the estimates for the inspection effort and budget models based on

experience, continued data collection and analysis of future changes in the

inspection program scope. Additional effort is needed to evaluate the quality of

staff efforts under the ROP, particularly the quality of inspection and the depth

and scope of inspection preparation.

ï Evaluate how licensee self-assessments might be used to satisfy some

requirements of the baseline inspection program without compromising overall

outcome goals, including public confidence.

SDP

ï Issue the plant specific Reactor Safety SDP notebooks, including the Phase 2

worksheets.

ï Continue work to revise the ALARA SDP.

ï Replace the interim Physical Protection SDP with a revised SDP that will be

developed with internal and external stakeholder input. 
- 18 -

ï Continue to devise methodologies that will allow inspectors to develop realistic

fire scenarios and improve the accuracy of site specific data used in the

assessment of risk associated with fire protection findings, such as fire ignition

frequency. 

ï Develop a process to evaluate the risk significance of plant shutdown issues.

ï Improve the capability to assess the impact of external events on operating

reactor safety related issues. 

ï Enhance the guidance provided for the assessment of concurrent deficiencies.

Assessment

ï Develop additional guidance on how to address the situation where

supplemental inspection for PIs indicate that there are substantive inadequacies

in a licenseeís root cause evaluation or corrective actions.

ï Continue to evaluate how historical licensee performance issues should be

treated by the Action Matrix.

ï Determine whether a graded approach for removing inspection findings from

consideration in the Action Matrix is appropriate.

ï Continue to monitor ìNo Colorî findings during the ongoing inspection report

review process and evaluate changes in program guidance, as appropriate, to

minimize their use.

ï Implement lessons learned from the Spring 2001 end-of-cycle activities, such as

the End-of-Cycle assessments, the Agency Action Review Meeting, and the

annual meetings with licensees.

Enforcement

ï Continue to pursue ways to make the SDP and enforcement processes more

timely in evaluating and communicating safety significant issues.

ï Discontinue the special treatment of Maintenance Rule issues when the staff

has determined that the revised Maintenance Rule enforcement guidance is

sufficient and is being implemented in a consistent manner among the regions. 

Inspector Staffing

ï Develop criteria for the Regional Administrators, in consultation with NRR, to

allocate additional inspection resources to accommodate certain sites for other

than performance related issues (e.g., unique design or organization features). 

ï Develop appropriate ROP self-assessment metrics to monitor and trend

inspector demographics and program quality (e.g., inspector training time).
- 19 -

ï Continue to refine the ROP regional resource model to look for additional

efficiencies.

Based on its assessment of stakeholder feedback and the results and lessons learned from

initial implementation, the staff has developed a much greater level of confidence that the ROP

has met the Commissionís direction to develop an oversight process that is more objective, riskinformed, understandable, and predictable. The ROP has been substantially exercised such

that the staff has gained valuable insights on aspects of the ROP and identified issues that

were not revealed during the pilot program. The staff recognizes that the ROP will continue to

require close scrutiny and oversight and has established a self-assessment program that will

identify those areas for improvement. The staff, as part of the Agency Action Review process,

will continue to report to the Commission on an annual basis the results of its self-assessment

and any significant changes to the ROP. 

COORDINATION:

The Office of the General Counsel has reviewed this Commission paper and has no legal

objections to its content.

The Office of the Chief Financial Officer has reviewed this Commission paper for resource

implications and has no objections.

/RA/

William D. Travers

Executive Director

 for Operations

Attachments: 1. Internal Stakeholder Communication and Feedback

2. External Stakeholder Comment and Feedback

3. ROP Self-Assessment Program

4. IIEP Final Report

5. Staff Response to the IIEP Final Report

6. Performance Indicators 

7. Inspection Program 

8. Significance Determination Process 

9. Assessment Process

10. Enforcement Process

11. Key Policy Issues

12. ROP Initial Implementation Commitments

13. ROP Resource Analysis

Appendices: A. Cross-cutting Issues

B. Resident Inspector ìN+1" Policy

C. Staff Training Initiatives
-20-

COORDINATION:

The Office of the General Counsel has reviewed this Commission paper and has no legal

objections to its content.

The Office of the Chief Financial Officer has reviewed this Commission paper for resource

implications and has no objections.

/RA/

William D. Travers

Executive Director

 for Operations

Attachments: 1. Internal Stakeholder Communication and Feedback

2. External Stakeholder Comment and Feedback

3. ROP Self-Assessment Program

4. IIEP Final Report

5. Staff Response to the IIEP Final Report

6. Performance Indicators 

7. Inspection Program 

8. Significance Determination Process 

9. Assessment Process

10. Enforcement Process

11. Key Policy Issues

12. ROP Initial Implementation Commitments

13. ROP Resource Analysis

Appendices: A. Cross-cutting Issues

B. Resident Inspector ìN+1" Policy

C. Staff Training Initiatives

ACCESSION NO.: ML011410551

* See previous concurrence ** Concurred via e-mail

To receive a copy of this document, indicate in the box: "C" = Copy without enclosures "E" = Copy with enclosures "N" = No copy

OFFICE IIPB:DIPM E TECH ED N IIPB:DIPM E IIPB:DIPM E SPSB:DSSA N

NAME TJFrye * PKleene* MSatorius * MRJohnson* RBarrett* 

DATE 5/16/01 5/24/01 5/16/01 5/16/01 6/7/01 

OFFICE PPRB:PMAS N IOLB:DIPM N DIPM:NRR N ADIP:NRR D:OE N

NAME JSilber* GTracy* BBoger * RBorchardt* FCongel*

DATE 6/8/01 6/11/01 6/12/01 6/19/01 6/8/01

OFFICE RA:RI N RA:RII N RA:RIII N RA:RIV N D:NRR

NAME HMiller** LReyes** JDyer ** EMerschoff** RWB for SJCollins* 

DATE 6/8/01 6/6/01 6/7/01 6/5/01 6/19/01

OFFICE CIO N CFO OGC N EDO

NAME SReiter** N/A JLFunches* via E-mail JGoldberg* WDTravers

DATE 6/12/01 62201 (see

comments)

 6/19/01 06/25/01

OFFICIAL RECORD COPY
INTERNAL STAKEHOLDER COMMUNICATION AND FEEDBACK

Task Lead: August Spector, NRR/IIPB

Attachment 1
1

Internal Stakeholder Communication and Feedback

Introduction

During the first year of ROP implementation, the staff expended a great deal of effort in providing

information to inspectors and their managers, and obtaining feedback on the ROP and its

implementation. Inspection Program Branch (IIPB) staff developed a Communication Plan that

outlined activities designed to enhance understanding of the ROP among internal stakeholders. 

Activities conducted included stakeholder workshops and lessons-learned meetings, training

courses that dealt with specific aspects of the new program, weekly regional conference calls,

and management counterpart meetings. These activities provided stakeholders the means to

submit questions and feedback about the process. In addition, an internal ROP Web site was

made available to all NRC staff. These activities have served to improve the staffís

understanding and acceptance of the revised ROP. The communication plan will be revised to

reflect ongoing activities related to the ROP. Below is a more detailed discussion of key

communication activities conducted during the initial implementation period.

Training Courses

The NRC Technical Training Center (TTC) developed and presented a special training course

entitled ìReactor Inspection and Oversight Program (G-200).î This course was attended by 342

regional attendees, and 96 Headquarters attendees from November 1999 through August 2000. 

The courses were presented in each region and Headquarters, and at the TTC. The course

included topic areas representing each component of the ROP. Participants used case studies,

program procedures, and job-aids in interactive exercises.

The training program was designed to provide inspectors knowledge and skill in the new process

in a timely manner before initial implementation, and follow-up training during the early stages of

initial implementation. 

Training support will be an ongoing effort with the development of additional courses and

workshops. The staff formed a working group to review, assess and modify inspector training

and qualification requirements to ensure current and future inspectors are adequately trained to

effectively implement the ROP. Training initiatives are discussed in more detail in Appendix C of

this paper. 

Use of Internal ROP Web Site

An internal Web site (ROP Digital City) was developed and maintained to disseminate ROP

information in a timely and efficient manner, to provide internal stakeholders awareness of current

changes in the ROP, and to facilitate feedback. This site provided regional and Headquarters

staff draft inspection procedures, inspection manual chapters, and internal memoranda. Web

links to additional Web sites were provided to simplify retrieval of information.

Weekly Conference Calls

NRR and regional managers and staff conducted a weekly conference call to discuss issues

related to the ROP and to plan for upcoming activities. NRR participants included managers and
2

staff from IIPB and concerned staff from other NRR divisions and NRC offices. Regional

participants included Division of Reactor Projects (DRP) and Division of Reactor Safety (DRS)

Division Directors, Branch Chiefs, and concerned staff members. These sessions were

invaluable in reviewing the weekís activities, preparing for future activities, discussing open

issues, and providing feedback on program implementation. 

Counterpart Meetings and Briefings

Headquarters and regional managers and staff participated in counterpart meetings to review and

address emerging implementation issues and plan future programmatic direction. These

meetings were attended by the regional DRP/DRS division directors and the IIPB branch chief

and selected staff. In addition, IIPB and the DRP/DRS division directors participated in a video

conference with the Deputy Regional Administrators to ensure their awareness of program status

and to obtain feedback regarding ROP implementation.

The regions conducted a variety of special meetings, briefings and internal lessons learned

sessions designed to facilitate regional implementation of the ROP. Among these included

courses on the management of change and inspector counterpart meetings.

Internal Lessons Learned Workshop

In March 2001, the staff conducted an internal lessons learned workshop. The purpose of the

workshop was to discuss aspects of the ROP that challenged the staff during the first year of

implementation. Regional and Headquarters managers identified proposed improvement actions,

and those issues that called for external stakeholder input. These issues were subsequently

discussed at the public External Lessons Learned Workshop held at the end of March 2001. In

preparation for the internal workshop, ìfocus groupsî consisting of regional technical staff,

inspectors, managers and Headquarters technical staff were formed. Each focus group identified

the issues of concern in their assigned area and proposed improvement actions. 

Inspector Feedback

A formal feedback process was developed to enable inspectors and regional staff to provide NRR

with information important to improving the ROP and its implementation. The feedback process

is documented in Inspection Manual Chapter (IMC) 0801, ìProgram Feedback.î IMC 0801

describes the feedback process and provides feedback forms to be used by the staff to document

issues encountered in implementing the ROP. During the first year of implementation, IIPB

received feedback related to all aspects of the ROP and from all regions. Approximately 340

feedback forms were received. Most feedback forms concerned issues related to interpretation

of performance indicators (PIs) (80), significance determination process (SDP) interpretation (50),

and inspection report documentation (35). Many of the comments and suggestions were found to

be useful and were incorporated into the program during initial implementation. Other feedback

raised issues that were complex in nature or for which it was necessary to hold action in

abeyance until the end of initial implementation in order to enable the program office to evaluate

a broader range of data from all four regions before making changes to the program (e.g.,

suggested changes to procedure scope and resource estimates.) The information gained from

the use of ROP program feedback forms was invaluable in making modifications to procedures

and process related to initial implementation of the ROP. 
3

Site Visits by Headquarters Staff 

During the months of October and November 2000, IIPB conducted visits to six reactor sites in

each region. The purpose of these site visits was to provide an opportunity for the resident and

regional inspection staff and licensee managers to provide first-hand feedback on the ROP. 

During these visits, the inspection staff and licensee managers discussed their experiences and

observations regarding implementation of the ROP. There was a consensus on many aspects of

the ROP. Overall the ROP was considered to be meeting the criteria described by the

Commission when they directed the staff to revise its approach to nuclear power plant oversight. 

The ROP was considered more risk-informed, objective, understandable, and predictable. The

increased emphasis on risk-informing processes, such as the areas of emphasis in the inspection

program and the application of the SDP, focused both Agency and licensee resources and

attention more appropriately on areas of risk and safety importance.

The feedback received from these site visits provided useful information that helped staff shape

the issues that were eventually forwarded for discussion at the internal and public External

Lessons Learned Workshops held in March 2001. Feedback indicated that the staff had mixed

views regarding the efficacy of the ROP. Some NRC staff and managers were skeptical of the

ability of the ROP to maintain safety and enhance public confidence. Inspectors expressed

concern related to the thresholds for documenting issues that could be precursors to more

significant issues in inspection reports and the manner in which cross-cutting issues are

addressed in the ROP framework. Several inspectors expressed the view that the process left

them unable to firmly engage licensees on lower level issues that may be predictive of future

problems. Resident inspectors noted that the risk-informed baseline inspection program had

eliminated much of the peripheral, less important inspection activities found in the previous

inspection program. Many licensees also expressed the view that inspector efforts were more

focused on areas important to safety, thus licensee resources would be more appropriately

applied. However, several also noted that they did not see any notable reduction in overall

regulatory burden associated with the ROP. In addition, the IIPB staff sought specific information

on how well its feedback process was working and how well the training associated with the ROP

prepared inspectors for carrying out the process. While there had been improved initial response

to issues raised through the feedback process by IIPB, some inspectors were concerned that it

was not always clear if issues received appropriate consideration. It was noted that feedback

forms were not always submitted in a timely fashion by the regions when issues or questions on

ROP guidance or processes arose in the field. Many times direct verbal communication with the

Headquarters program office was necessary, therefore resulting in inefficient feedback.

Initial training on the ROP was viewed as positive. However, opportunities for additional training

on ROP related topics such as the SDP, inspection report preparation, and specific inspection

procedures were noted as desirable.

The insights obtained as a result of the site visits have been included in the appropriate ROP

area in order to improve ROP efficiency and identify needed improvements.
4

Internal Stakeholder Feedback Surveys

In March 2001, IIPB staff conducted a survey of those individuals within NRC who were involved

with the ROP initial implementation. The majority of respondents agreed that the ROP provides a

realistic approach to oversight, and assures that plants are being operated safely. They consider

that the process provides appropriate regulatory attention to licensees with performance

problems, is objective, and is an effective risk-informed approach to oversight. Compared to the

previous process, most respondents considered the new ROP to have increased predictability,

consistency, clarity, objectivity, timeliness, and efficiency. It is more risk-informed and reduces

unnecessary administrative burden on the NRC. Respondents agree that the ROP has resulted

in a reduction of unnecessary regulatory burden on stakeholders.

In January 2001 IIPB conducted a training program needs analysis survey among regional staff. 

This survey requested information about staff perception of the ROP training activities provided to

the regions. The majority of respondents indicated that training was offered in a timely manner

and provided the necessary skill and knowledge required for implementation of the ROP. 

In November 1999, at the end of the 6-month pilot program, IIPB conducted a survey to obtain

feedback from staff who were familiar with the ROP at that time. When comparing this survey

with the March 2001 survey, respondents generally indicated more positive ratings after the initial

implementation year compared to the pilot program period. The majority of respondents showed

a marked increase in their understanding and acceptance of various components of the ROP,

including the SDP, the baseline inspection program, the assessment program, PIs, and internal

and external communication activities.

Although some NRC inspectors may have initially indicated skepticism of the significant changes

being brought about by the new program, the end-of-program 2001 survey indicates a much

higher level of acceptance, and a better understanding and familiarity with the ROP. The 2001

survey data indicates that generally NRC internal stakeholders who have been involved with the

implementation of the new program and are familiar with its processes have more positive

acceptance than those who were surveyed after the pilot program initiative in 1999.

Survey Results 

March 2001 Survey

In the March 2001 survey, a total of 234 responses were received. This represented a 43%

return rate. Respondent replies were anonymous. Demographic data was kept on location,

position, grade, and length of service. The survey solicited responses in five major topic areas:

overall program, inspection program, assessment process, performance indicators, significance

determination process. All questions were rated on a four point scale (strongly agree, agree,

disagree, strongly disagree). An ìunable to answerî choice was also provided. In addition, space

was provided at the end of the questionnaire for additional comments.

The results of the five survey sections are provided below.

The numbers in parentheses in the summary below represent the combined percentage of

respondents who endorsed the stated view.
5

Section: Overall

The majority of respondents agreed that the ROP is a realistic approach to oversight

(79%), and assures that plants are being operated safely (88%). They consider that the

process provides appropriate regulatory attention to licensees with performance problems

(74%), is objective (85%), and is an effective risk-informed approach to oversight (82%). 

Compared to the previous process, the ROP increases predictability (75%), consistency

(84%), clarity (70%), objectivity (79%), timeliness (78%), efficiency (75%) and to a lesser

extent, effectiveness (57%). It is more risk-informed (96%) and reduces unnecessary

administrative burden on the NRC (69%). Respondents agree that the ROP has resulted

in a reduction of unnecessary regulatory burden on stakeholders (79%).

Respondents consider the information on plant performance provided on the ROP Web

site to be timely (91%), understandable (written in plain English) (89%), accurate (90%),

easily retrievable (86%), and adequate to keep NRC internal stakeholders informed

(77%). Respondents further agree that the information provided by the NRC appropriately

keeps the public informed of the Agency oversight activities related to the plants (74%).

Respondents believe that the ROP provides appropriate communication effectiveness

through the use of plain language in official correspondence such as inspection reports

and letters to licensees (74%) and there is appropriate inspector and licensee

communication (83%).

Although respondents agree that responses from feedback forms sent to Headquarters

are generally understandable (written in plain English) (67%) and accurate (65%), the

majority of respondents did not believe the feedback process is timely (66%) or that

feedback addresses the issues raised (55%). 

Section: Inspection Program

Respondents agree that the baseline inspection program appropriately inspects for and

identifies risk-significant issues (78%) which lead to objective findings whose significance

can be clearly documented (71%). They consider that inspection reports are

communicated accurately (89%) and in a timely fashion (95%). To a lesser extent,

respondents consider the baseline inspection program report format adequately

communicates relevant information to the licensees (63%), the public (60%), and to

internal NRC stakeholders (56%). 

Respondents strongly agree that the baseline inspection program procedures are

adequate to address intended cornerstone attributes (81%), are clearly written (75%),

place sufficient emphasis on planning (82%), adequately sample risk important aspects of

each inspectable area (76%), and are conducted at an appropriate frequency (73%). 

They believe that the baseline inspection program provides appropriate coverage of plant

activities and operations important to safety (63%). Less than half of the respondents

consider that the level of effort for conducting each inspection is consistent with that
6

estimated in the inspection procedures (47%). The majority of respondents agreed that

the resources needed to oversee licensees using the ROP are appropriate (61%).

Respondents agreed that the supplemental inspection procedures provide sufficient

information to confirm the adequacy of a licenseeís root cause and corrective action effort

(69%). It should be noted that 88 respondents did not (were ìunable to answerî) provide

a rating to this question. This may indicate that they did not have sufficient knowledge of

this area to provide a ìratingî.

The majority of respondents disagreed or strongly disagreed that the ROP appropriately

integrates and provides insights into cross-cutting areas (70%). Some inspectors

suggested that cross-cutting issues should be documented in inspection reports and that

additional inspection should be considered to address events that may have cross-cutting

implications or are not adequately covered by baseline inspection, PI, or SDP activities.

Section: Assessment Process

Respondents agreed that the assessment process provides an appropriate range of

actions for safety issues (82%). The majority of respondents felt that the assessment

process provides for timely resolution of issues commensurate with safety significance

(76%), focuses resources on areas of greatest safety significance (80%), provides

objective levels of assessment (84%), and understandable thresholds (74%). 

Respondents felt that the new assessment process minimizes duplication and rework in

preparation of assessment meetings such as mid-cycle, end-of-cycle, agency action

review, and public meetings (88%),

Respondents indicated that the ROP provides sufficient attention to licensees whose

performance is in the licensee response band (74%). However, they indicated less

agreement regarding the ROPís ability to provide appropriate identification of declining

safety performance before thereís a significant reduction in safety margins (53%). 

Respondents indicated that the issuance of non-cited violations (NCVs) and relying on the

licenseeís corrective action program provides for an adequate approach to resolve issues

of very low safety significance (i.e., Green findings) (80%). Respondents indicated less

agreement that the assessment process applies appropriate enforcement actions (67%). 

Respondents believe that the timeliness goals specified in IMC 0305 for documentation,

data collection, etc. can reasonably be met (91%). It should be noted that 57 respondents

did not (were ìunable to answerî) provide a rating to this question. This may indicate that

they did not have sufficient knowledge of this area to provide a ìrating,î because fewer

staff were involved with assessment and all assessments have not been completed (i.e.,

end of year, agency action reviews, etc.) 

Section: Performance Indicators

Respondents agreed that the PIs provide useful information on risk-significant areas

(79%), are clearly defined (72%), understandable (78%), and help to maintain safety

(72%). They consider that PIs do provide an appropriate level of overlap with inspection

findings (74%). 
7

Possibly due to the newness of the ROP, respondents showed less agreement as to

performance indicators providing an adequate indication of declining safety performance

(53%) and that the ROP may foster long-term self-improvement by licensees (56%).

Respondents considered that performance indicators enhanced public confidence (65%),

however, sixty-nine (69) no responses (ìunable to answerî) were given to this question. 

This may indicate skepticism on the part of internal stakeholders or lack of knowledge of

how the public understands the program. 

Section: Significance Determination Process

Respondents agree that SDP results correctly characterize the risk-significance of

inspection findings (71%). They agree that the SDP focuses NRC attention on safetysignificant issues (79%) and provides a basis for effective communication of inspection

findings to licensees (77%), and to a lesser extent to the public (59%). 

The majority of respondents consider the SDP to provide for consistent results (72%), are

verifiable (84%), realistic (70%), and provide accurate results (65%). To a lesser extent.

respondents agreed that the SDP is based upon clear standards (53%) and on the

completeness of SDP results (54%). 

The majority of respondents indicated that the significance determination process reactor

safety SDPs and non-reactor safety SDPs are not easy to use (60% and 64%

respectively). It should be noted that 25 and 127 respondents respectively did not (were

ìunable to answerî) provide a rating to these two questions. This may indicate that they

did not have sufficient knowledge of this area, especially non-reactor SDPs, to provide a

ìratingî. 

Demographic Summary

Respondents were requested to indicate four demographic issues: length of service with

NRC, location, position title, and grade. A statistical analysis was made to compare

respondent demographics for each question in order to show differences within each

demographic category.

The data indicated some disparity between regions in how they viewed particular areas,

such as the perception that the ROP increases effectiveness and objectivity, the response

to feedback forms, procedures used in the baseline inspection program, resources utilized

in the assessment process, and the SDPs ability to yield consistent results. Many of

these disparities are a result of varying experience with the new process, uniformity of

training activities, and specific plant issues. Generally, regional respondents (inspection

staff and technical staff) are favorable toward the ROP, although they do remain skeptical

of particular issues. Regional line managers, Headquarters technical or program staff,

and Headquarters line managers were shown to be more positive in their views regarding

the new process. The data indicated that staff who have been employed by NRC for less

than five years indicated some dissatisfaction with the understandability and
8

responsiveness of internal feedback from feedback forms submitted to the region and

Headquarters. 

Training Activities Survey of January 2001

In January 2001, IIPB conducted a training program needs analysis survey among regional staff. 

This survey, completed by 147 respondents, requested information about staff perception of the

ROP training activities provided to the regions. This survey found inspectors considered the

training in the new ROP was offered in a timely manner (76%), to have adequately explained the

basis for the oversight process change (85%), and communicated the roles of resident inspectors

(72%) and regional inspectors (75%) in the new process. Inspectors considered the courses to

have provided the necessary skill and knowledge required for implementation of baseline

inspection procedures (68%) and understanding of performance indicators (58%), but only 49%

agreed/strongly agreed the necessary training was provided related to the assessment program. 

Most inspectors indicated that training in the use of the SDP and inspection report documentation

was inadequate (62% and 70% respectively). These two areas have been under continual

development during the initial implementation period. Once these activities are finalized,

additional refresher training will be provided.

Comparison of March 2001 and November 1999 Surveys

At the end of the 6-month pilot program (November 1999) the staff conducted an internal

Program Survey designed to obtain feedback on the perceptions of those who were familiar with

the ROP at that time. This survey garnered responses from 94 respondents, primarily from the

regions. The data from the two surveys were compared. Although the questions asked in both

surveys were not completely identical (the pilot program survey was not as extensive, utilized a

five point scale as opposed to the four point scale, thus eliminating the neither agree/disagree

option found to be uninterpretable in the November 1999 survey, and the wording of some

questions was modified in the March 2001 survey) the surveys were similar enough in a number

of areas to permit a comparison. The survey data presented below provides the combined

agree/disagree response for those question similar in both surveys.

When comparing those questions similar in both surveys, respondents generally indicated more

positive ratings after the initial implementation year compared to the pilot program period. The

majority of respondents showed an increase in their understanding and acceptance of various

components of the ROP, including the SDP, the baseline inspection program, the assessment

program, PIs, and internal and external communication activities.

The majority of respondents to the 2001 survey agreed that the ROP provides appropriate

assurance that plants are being operated safely (88% in 2001 vs. 49% in 1999) and that the ROP

provides appropriate regulatory attention to licensees with performance problems (74% in 2001

vs. 41% in 1999). The majority of respondents consider that the ROP provides objectivity to the

oversight process (85% in 2001 vs. 69% in 1999), that the baseline inspection program has the

ability to identify risk-significant issues (78% in 2001 vs. 54% in 1999), and to produce adequate

inspection to address cornerstone attributes (81% in 2001 vs. 55% in 1999). Respondentsí rating

of the new ROP compared to the previous process with regard to predictability, consistency,

clarity, and objectivity of our inspection program improved as compared to those responding to

the end-of-pilot survey (78% in 2001 vs. 56% in 1999). 
9

As compared to those in 1999, respondents to the 2001 survey indicated much higher

acceptance related to performance indicators providing useful information on risk significant

areas (79% in 2001 vs 10% in 1999). The ability of PIs to provide an indication of declining

safety performance showed an increase (53% in 2001 vs. 24% in 1999). Compared to the pilot

survey, the majority of respondents to the initial implementation survey considered the SDP

results to correctly characterize risk significance of inspection findings (71% in 2001 vs. 32% in

1999). A majority of respondents to the 2001 survey consider the new program to provide

enhanced internal and external stakeholder information for keeping the public informed of the

NRCís regulatory program (74% in 2001 vs. 41% in 1999). They considered the new ROP Web

site to contribute to improved communications by providing internal and external stakeholders

adequate information (77% in 2001 vs. 56% in 1999). Respondents indicated increased

satisfaction toward the assessment program and the Action Matrix to provide an appropriate

range of actions for safety issues (82% in 2001 vs. 51% in 1999). They further agreed that the

assessment program is timely and efficient (76% in 2001 vs. 41% in 1999). Respondents were

more positive related to the issuance of NCVs and the ability of the licensees corrective action

program to provide an adequate approach to resolving issues of low safety significance (80% in

2001 vs. 73% in 1999).

Although NRC inspectors may have initially indicated skepticism of the significant changes being

brought about by the new program, the end-of-program 2001 survey strongly indicates more

acceptance, and a better understanding and familiarity with the ROP. The 2001 survey data

indicates that, in general, NRC internal stakeholders who have been involved with the

implementation of the new program and are familiar with its processes have more positive

acceptance than those who were surveyed after the 6-month pilot program initiative in 1999. The

data indicates that the staff considers the ROP to be heading in the direction specified in the

Agency Strategic Plan. Specific feedback gained from these surveys has been considered in

modifications to each area of the ROP.
EXTERNAL STAKEHOLDER COMMENT AND FEEDBACK

Task Lead: Leon Whitney, NRR/SPLB

Attachment 2
1

External Stakeholder Comment And Feedback

Introduction

This section contains a description of the initiatives implemented to solicit and obtain external

stakeholder comment and feedback and a summary of the results of these efforts.

Summary

NRC staff made a concerted effort to continue and improve on the initiatives implemented during

the pilot program to keep external stakeholders informed and to provide frequent and ongoing

opportunities to submit comment and feedback. These initiatives included the external Web

page, a Federal Register notice (FRN), and numerous forums during which public, industry, and

governmental stakeholders were participants. 

The staff utilized a FRN to inform the public of Agency activities and to solicit input on the initial

implementation of the ROP. The elicited feedback was utilized in the ROP self-assessment

program discussed in a separate section. To ensure widespread participation, copies of the

notice were mailed to all individuals and organizations that had participated in previous forums or

had previously submitted comments. A summary of the response to the public comment FRN is

discussed later in this attachment. 

Generally, overall FRN feedback indicated that initial implementation of the ROP was successful. 

The ROP is perceived to be an improvement over the previous processes in many respects. It

has improved regulatory consistency, reduced overall regulatory burden, and improved the

predictability of Agency actions. The industry is confident that the new ROP is providing

adequate assurance that reactor plants are being operated safely, that sufficient regulatory

attention is being given to licensees with performance problems, and that the new ROP has the

potential to develop into an improved and more effective program. The industry perceives the

process as being more objective and understandable, with an increase in regulatory focus on risk

significance, and a reduction in regulatory burden. Industry feedback indicates that the NRC staff

has maintained adherence to the new ROP as designed throughout the pilot and initial

implementation periods of the new program. 

In the view of industry, as expressed in their FRN responses, licensees and industry

organizations have been afforded adequate opportunity to provide input and comments on the

new ROP, and to become involved in the ROP development process. Industry and some other

responding organizations, such as the Union of Concerned Scientists and the State of

Pennsylvania, indicated that the NRC has conscientiously sought public participation in the

development of the new ROP. Despite providing an overall positive view of the ROP, industry

had many comments targeted at improving the ROP, most addressing the performance indicators

(PIs) and the significance determination process (SDP). Industry considers certain portions of

the SDP to be too complex (especially for the public to understand), and overall the SDP is

viewed as a work in progress. 

The FRN responses of the Union of Concerned Scientists and the Commonwealth of

Pennsylvania were more mixed regarding the new ROP. Their respective responses provided

selective observations and suggestions for ROP improvement.
2

The State of New Jersey, and the public interest groups Three Mile Island Alert and EFMR

Monitoring Group, were much less favorable regarding the new ROP. They expressed the belief

that the ROP is poorly focused, will not identify declining performers (at least in a timely manner),

and will not result in adequate assurance of safe commercial nuclear reactor operation. TMI

Alert, EFMR Monitoring Group, and Mr. Marvin Lewis of Philadelphia, Pennsylvania indicated that

the new ROP is a step backwards. EFMR Monitoring Group was critical of the NRCís

performance in communicating with the public regarding NRC oversight activities and the safety

performance of licensees. EFMR Monitoring Group stated that the NRC staffís public outreach

has missed the mark at the local community level (near the TMI and Peach Bottom sites). The

State of New Jersey expressed its frustration at the NRCís apparent lack of attention to its

numerous comments. 

External stakeholder feedback has been factored in to the spectrum of IIPB efforts to improve the

new ROP.

Communication and Feedback Initiatives

IIPB staff developed a Communication Plan that outlined activities designed to enhance the

understanding of the ROP among external stakeholders. Activities conducted included public

meetings at each power plant site, lessons learned workshops, monthly public meetings with the

industry and concerned stakeholders, establishment of an external Web site, and publication of a

plain language NUREG that explained the new process. 

In general, public and industry stakeholders expressed appreciation of the Agencyís efforts to

provide information on the ROP and to consider stakeholder input. The staff will continue to

conduct outreach activities, including routine public working meetings to discuss on-going

process development and revision, public meetings with citizens near power plants to provide

current plant status under the ROP and results, FRNs to solicit public comment, and continued

improvement and updating of the external Web site. The staff will also consider additional

Agency outreach activities in order to keep members of the public local to plant sites cognizant of

NRCís mission, role, and activities. 

The Office of Public Affairs, in conjunction with NRR, published an updated revision of NUREG1649, ìReactor Oversight Process,î in April and July 2000. This plain language pamphlet will be

periodically reviewed to determine if it should be revised to reflect process changes and updates.

Subsequent to initial implementation, the communication plan will be revised to reflect ongoing

activities related to the ROP. 

Below is a summary of key communication activities conducted during initial implementation.

ROP Web page

The staff has continued to maintain the ROP Web page. The primary goals of the ROP Web

page are to provide ROP information to internal and external stakeholders that is timely, useful,

accurate, and user-friendly. The ROP Home page (http://www.nrc.gov/NRR/ OVERSIGHT/-

index.html) serves as the primary gateway into the vast array of information available regarding

reactor oversight. It provides introductory remarks, a direct feedback mechanism, and hyperlinks
3

to several subpages, including the plant assessment results, ROP program documents, ROP

meeting notices and summaries, and a ìplain Englishî description of the ROP. The Plant

Assessment Results page (http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html) provides

the most recent information on the performance of the nation's nuclear power plants, using PIs

and NRC inspection findings to determine the applicable Action Matrix column for each plant. In

addition, the staff continues to make changes to improve the effectiveness of the Web

information. The most notable Web improvements include the development of summary matrices

for the PIs, inspection findings, and Action Matrix designations, and providing information on the

Initial Implementation Evaluation Panel process and results. 

The staff has solicited and received feedback on the effectiveness of the ROP Web page in

meeting its goals primarily through workshops, surveys, and direct feedback from the Web page

itself. The majority of the feedback, from both internal and external stakeholders, has been very

positive, although there have been some suggestions for improvement.

Specific comments of note on the ROP Web page and staff actions planned or taken include: 

(1) Plant performance should be organized by a top-down approach starting with

an overview of each plantís performance based on the Action Matrix outcomes,

with all of the site-related documents linked to the individual site pages.

The Action Matrix Summary Page (http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/-

actionmatrix_summary.html) was developed and has been revised to afford stakeholders

a top-level view of relative performance for all plants, with the ability to drill down into more

detailed individual plant performance information. 

(2) It is confusing to portray ìNo Colorî findings as blue on the Web pages. 

The Web pages have been revised to replace the blue color with a gray color. The

broader issue regarding the use of ìNo Colorî findings is addressed in the assessment

attachment (Attachment 9).

(3) The publicís focus has been on performance indicators due to their

predominant exposure on the Web pages.

The structure and format of the Web pages were improved to emphasize that the staffís

assessment of plant performance and resultant actions are based on both inspection

findings and PIs. 

(4) Enhance the use of plain English to make the ROP information more

understandable. 

The ROP Web pages have been improved to make them more user-friendly and

understandable to our stakeholders, though the staff recognizes that continued

improvements in this area are warranted and plans to add more ìplain Englishî

descriptions in the future.
4

(5) For only approximately one third of the inspection findings on the NRC Web site

was it possible for knowledgeable representatives of a public interest organization

to reconstruct the logic path which culminated in the findingís color.

The staff plans to improve the Web page design and inspection report writeups to more

clearly describe the finding and its significance. The staff is continuing to monitor

inspection report documentation to ensure improvements are made with respect to clear

descriptions and rationales utilized in arriving at the final significance of issues (see

Attachment 7, Inspection Program). In addition, the staff plans to improve Web page

linkages to make it easier to navigate to associated or supporting information. 

The staff will continue to solicit and make improvements to its ROP Web page to ensure that it is

a useful communication tool in providing accurate and timely ROP information to all stakeholders.

Public Meeting Outreach Efforts

The staff conducted public meetings at least monthly during initial implementation to discuss

ongoing development and address implementation issues, including responding to frequently

asked questions (FAQs) associated with the PI guidance contained in NEI 99-02, ìRegulatory

Performance Indicator Guidelineî. These meetings were working sessions between the staff and

the Nuclear Energy Institute, industry representatives, public interest groups, and interested

members of the public. These meetings were an important vehicle for continuing to provide

openness to the development and implementation of the process and being responsive to

external stakeholder concern. Much information was shared with all participants in an effort to

keep the public informed of staff activities.

During the first few months of initial implementation, the staff conducted evening public meetings

at each of the power plant sites. The purpose of these meetings was to orient local citizens with

the ROP. During each meeting, the staff explained the Agencyís mission and role, explained why

the Agency revised its process, and described how the new process worked. A presentation

template was developed for use by regional staff presenters to ensure that consistent key

messages were provided.

The staff also conducted interim lessons learned workshops in each region in the Fall of 2000. 

These 1-day public meetings were organized in cooperation with industry representatives to

assist the staff in gaining input from industry and regional staff and to address specific questions

that were of importance to stakeholders.

During the months of October and November 2000, the NRR staff conducted a series of visits to

sites in each region. The purpose of these site visits was to solicit feedback on the ROP. These

one day visits involved six sites in each region where both resident inspector staff and licensee

management were asked to discuss their experiences and observations regarding

implementation of the ROP. Following the site visits, regional management was asked to provide

similar information. There was general consensus on many aspects of the ROP that were

considered successes relative to its implementation to-date. There were also several areas that

were generally considered to require more information to make an informed judgement or

required additional effort to revise or modify the process. Lessons learned from these visits were

used to develop focus areas for subsequent working meetings.
5

The staff conducted a public External Lessons Learned Workshop, March 26-28, 2001, to bring

together the NRC and external stakeholders to discuss, review, and develop recommendations

associated with key issues that have emerged during the first year of initial implementation of the

new ROP. Specific topic areas discussed during that workshop included: reactor safety

performance indicators, fire protection, radiation safety, cross-cutting and problem identification

and resolution, maintenance effectiveness, physical protection, assessment and enforcement,

and communication issues. Other issues raised at this workshop by external stakeholders have

been documented and are being reviewed by the staff. A copy of the meeting minutes for this

workshop are available on the NRC ROP Web site.

The staff gave presentations at a variety of workshops and courses sponsored by various nuclear

oriented associations and universities. The staff also made presentations at several international

conferences and met with foreign visitors to the NRC.

Comments from the Federal Register Notice

The outreach FRN was issued on January 10, 2001, soliciting public comments regarding

suggestions for topics to be considered in the March 2001, public External Lessons Learned

Workshop and on the first year of initial implementation of the ROP. The deadline for ROP

comments was April 13, 2001. 

With respect to the first year of initial implementation of the ROP, the FRN requested responses

to 17 specific questions related to two general ROP areas: (1) the efficacy of the overall process,

and (2) specific ROP program areas. The NRC received 14 sets of external comments in

response. The responses were utilized in the ROP self-assessment program, described in a

separate section of this paper. The most salient comments received, and the staff response, are

discussed in more detail in the individual attachments of this paper for each of the oversight

processes. While each comment has not been addressed, all comments were appropriately

considered in the staffís efforts to make process refinements based on the entire spectrum of

stakeholder comments received. 

FRN ROP comments were received from the following:

- Union of Concerned Scientists (UCS)

- Nuclear Energy Institute

- Entergy Operations, Inc.

- Exelon Generation Company, LLC

- Southern California Edison

- Dominion (Virginia Electric Power Company)

- Rochester Gas and Electric

- Tennessee Valley Authority

- Strategic Teaming and Resource Sharing (STARS) (representing TXU Electric,

AmerenUE, Wolf Creek Nuclear Operating Corporation, Pacific Gas and Electric

Company, and STP Nuclear Operating Company)

- Pennsylvania Department of Environmental Protection

- New Jersey Department of Environmental Protection

- Three Mile Island Alert

- EFMR Monitoring Group
6

- Mayor of Auburn, Nebraska

- Mr. Marvin Lewis

Eleven overarching themes were synthesized from the FRN comments. These were:

ï Industry, and some non-industry, respondents were generally satisfied with the new ROP,

but had specific suggestions for improvement. Industry, and some non-industry,

responders appreciated the consistency and predictability of the new ROPís inspection,

SDP and assessment components. Further, industry respondents were optimistic

regarding the new ROPís potential to develop into an improved and more effective

program. However, given this overall positive view, these respondents still had specific

comments on how to improve the new ROP. 

ï Some non-industry responders did not have confidence in the new ROP. Some nonindustry responders stated that the new ROP is poorly focused, does not identify declining

performers, and does not result in adequate assurance of safe commercial nuclear

reactor operation. TMI Alert, EFMR Monitoring Group and Mr. Lewis view the new ROP

as a step backwards. However, no respondents from industry concurred in this viewpoint.

ï Some non-industry respondents had the opinion that the NRC has not successfully

communicated with the public regarding NRC oversight activities and the safety

performance of licensees (possibly, in part, due to the complexity of certain aspects of the

new ROP, such as the Significance Determination Process). One public interest group

(EFMR Monitoring Group) stated that the NRC staffís public outreach has missed the

mark at the local community level in the Three Mile Island and Peach Bottom site areas.

ï Respondents were generally satisfied with the content and utility of the new ROP Web

pages. Industry and peer regulatory body respondents were concerned that the Web

pages may tend to create confusion and erroneous conclusions when accessed by nonexperts in the nuclear field.

ï Industry, and some non-industry, respondents stated that the NRC has conscientiously

sought public participation in the development of the new ROP. However, the State of

New Jersey expressed frustration at the NRCís perceived lack of attention to its numerous

comments.

ï The PI component of the new ROP received a large variety of critical comments focused

on its improvement.

ï Industry, and some non-industry, responders expressed the opinion that resolving overlap

and/or inconsistencies between NRC, WANO, Maintenance Rule, and probabilistic safety

assessment (PSA) reporting requirements is of urgent importance. 

ï Although it is a ìwork in progress,î the SDP was considered by industry, peer regulatory

body and public interest groups to be an improvement over past inspection finding

significance determination efforts. However, some SDP methodologies in certain subject

areas were considered to be slow, and can be complex to the point of being burdensome. 

Further, these respondents stated that sometimes SDP methodologies are technically
7

beyond inspectorís implementation skills, and some methodologies are not fully

developed.

ï UCS stated that the ROP does not provide sufficient regulatory attention to utilities with

performance problems. TMI Alert was concerned that the NRC has stated that a given

reactor plant can have multiple Red PIs and continue to operate (ìthere are no failing

gradesî in performance assessment). The State of New Jersey broadly criticized the

assessment process (while conceding its risk focus) as complicated, cumbersome and

difficult to use, forcing interested persons to ìpiece together the performance puzzle.î 

Industry indicated that it was pleased with the assessment process, but offered specific

criticisms and suggestions for improvement. 

ï No public interest group, peer regulatory body or industry responding organization stated

that the overall regulatory burden associated with the new ROP has increased

inappropriately, and most respondents believe that the overall regulatory burden has

decreased. Industry had specific suggestions for further reducing regulatory burden. 

UCS stated that the burden associated with information reporting associated with PIs is

necessary. It was the Commonwealth of Pennsylvaniaís view that the licenseeís reporting

effort was considerable, but this effort was offset by other changes brought by the new

ROP. Industry stakeholders stated that overall regulatory burden has been reduced, but

saw opportunities for increased efficiencies (e.g., combined programmatic inspections

across multiple sites, inspection frequency reductions, NRC acceptance of licensee selfassessments in lieu of NRC inspections, and integrated PI, WANO, and Maintenance

Rule information reporting requirements). TVA pointed out that, for previously SALP

Category 1 licensees, regulatory burden may have increased even though overall industry

regulatory burden appears to have decreased under the new ROP. TVA saw radiation

protection inspection and parallel PI data collection as combining to result in an

unnecessary increase in regulatory burden. TVA endorsed a change from ìN+1" to ìNî

(ìNî = number of reactor units per site) resident inspectors using public risk per reactor

unit as justification. 

ï Industry stated that periods of enforcement discretion are appropriate when new PIs and

major ROP process changes are piloted or initially implemented nation-wide. In a related

vein, the industry endorsed the pilot process as an appropriate means of testing changes

to the new ROP.

The specific FRN questions, and summaries of the replies provided by the respondents, are

provided below:
8

I. Questions related to the efficacy of the overall process:

1. Does the ROP provide adequate assurance that plants are being operated

safely?

NEI indicated that the new ROP is ensuring safe reactor operation. However, the

responding public and a peer regulatory organization did not fully agree. The

State of New Jersey stated that the credibility of the PI system is suspect due to

itís sparse non-Green performance indications, there are too few inspection hours

allocated within the new ROP, relatively few non-Green inspection findings are

developed, and that a significant portion of inspection findings are licensee selfidentified (a concern echoed by Mr. Lewis). EFMR Monitoring Group alone stated

that onsite inspection and oversight has been reduced under the new ROP. UCS

recommended that the staff study the correlation between inspection hours and

the identification of inspection findings (to better focus the inspection component of

the new ROP). 

2. Does the ROP provide sufficient regulatory attention to utilities with

performance problems?

Industry stated that the new ROP provides sufficient regulatory attention to utilities

with performance problems. UCS did not agree, citing a specific instance involving

a steam generator tube integrity issue Red finding at Indian Point 2 (although the

Red finding indicated a ìsignificant reduction in safety margin,î the NRC waived

the 30-day deadline so that Consolidated Edison could focus on unit restart and

power ascension). The State of New Jerseyís position is that the attention paid to

licensees with performance problems is too narrow.

3. Does the ROP reduce unnecessary regulatory burden on licensees?

All respondents were of the opinion that regulatory burden has decreased overall,

and, in many cases, has increased appropriately in some narrow areas (e.g., PI

reporting). However, some respondents considered certain SDP methodologies in

certain subject areas to be slow, and complex to the point of being burdensome

(e.g. fire protection SDP). Industry had numerous suggestions for decreasing

regulatory burden further. [These suggestions are provided in answer sets

elsewhere within this section.]

4. Does the ROP improve the efficiency, effectiveness, and realism of the

regulatory process, focusing NRC resources on those issues with the most

safety significance?

NEI stated that the greatest improvement in inspection focus is in the reactor

safety area, where the PIs and reactor SDP have permitted the NRC and licensees

to allocate resources based on safety significance. NEI also stated that the gains

in inspection focus efficiency, effectiveness and realism have been less

pronounced in the radiation protection, physical security and safety system design

inspection areas. TVA commented that there appears to be an unwarranted
9

increase in inspection hours in the area of radiation protection, and TVA and

Entergy commented that the ìN+1" resident inspector policy leads to uneven

routine inspection burden for single-unit facilities. Peer regulatory bodies, public

interest groups and general members of the public did not respond directly to this

question.

5. Has the public information associated with the ROP been appropriate to

keep the public informed, in a timely manner and understandable fashion, of

NRC activities related to plant safety?

The State of New Jersey was satisfied with inspection report issuing goal of about

a month (for resident and regional DRS reports), as well as the staffís performance

in meeting that goal. However, the State of New Jersey had the opinion that nonGreen inspection findings take too long to assess, and the quarterly availability of

PI and inspection finding information makes the information less than current

(making the new ROP a lagging assessment program). Industry, public interest

groups, and general members of the public did not respond to this question. 

6. Does the ROP increase the predictability, consistency, clarity and objectivity

of the NRCís oversight activities?

Industry stated that the ROP increases the predictability, consistency, clarity and

objectivity of the NRCís oversight activities. Industry also commented that the

information is timely and the program is transparent. However, TVA stated that it

has not observed ìvery appropriate regulatory responsesî for issues identified in

the non-reactor safety cornerstones where less risk-based PIs and SDPs exist.

7. Has the public been afforded adequate opportunity to provide

input/comments and involvement in the ROP development process?

Although no respondent except EFMR Monitoring Group was critical of the NRCís

overall outreach effort (EFMR believing that, on the local level, the NRC has not

been successful), a variety of respondents had suggestions on how better to

conduct public meetings, obtain public and industry feedback on the ROP, and

display information, choose content, and display inspection report results on the 

Web pages. The Commonwealth of Pennsylvania observed that the SDP is a

complex and complicated process for the public to understand [and therefore the

SDP is considered by the Commonwealth of Pennsylvania to be a de facto barrier

to public understanding of the regulatory process]. The Commonwealth of

Pennsylvania pointed out the potentially confusing dichotomy between licensee

ìExcellenceî and the NRCís ìMaintaining Safetyî goals for members of the public. 

The State of New Jersey was critical of the difficulty it has had in putting the overall

reactor licensee performance picture together under the new ROP.
10

8. Has the NRC been responsive to input/comments provided by the public

regarding the ROP development process?

Industry stated that the NRC has been responsive to input by the public during the

ROP development process. However, the State of New Jersey complained that its

numerous comments have had little impact on the ROP development process. 

Public interest groups and general members of the public did not respond directly

to this question.

9. Please provide any additional (brief) information or issues related to the

reactor oversight process.

NEI and TVA were satisfied with the usefulness of the NRC Web pages, and

expressed appreciation for instances in which inspectors had provided verbal,

informal feedback on plant operations. TVA was satisfied with the objectivity of the

annual licensee/NRC ROP meetings. NEI and responding reactor licensees had

numerous comments on how to improve the new ROP and the process

surrounding itís further development. Public interest groups and general members

of the public did not respond directly to this question.

II. Questions related to specific ROP program areas

1. Do the performance indicators or other aspects of the ROP create

unintended consequences? (Please comment on the potential of unintended

consequences associated with the counting of manual scrams in the

Initiating Event Cornerstone Performance Indicators.)

NEI commented that increased regulatory oversight of planned unavailabilities of

equipment can have a number of unintended consequences, elaborating that it is

important that licensees not be unwisely penalized for taking appropriate actions to

operate their plants in a safe and economical fashion (e.g. conducting unplanned

mitigating system unavailabilities). Entergy commented that the NRC PI for safety

system unavailability may encourage more ìstackingî of system maintenance

during online maintenance (in order to manage the indicator) than might be

appropriate from a risk perspective. 

STARS commented that the Mitigating Systems PIs measure only unavailability

and are not a balance between unavailability and reliability. STARS commented

that maintenance on mitigating systems during (licensing basis approved) allowed

outage times (AOTs) may result in White PI values and additional inspections,

even though the AOT would have been obtained by demonstrating adequate

protection to the health and safety of the general public. 

STARS took exception to the practice within the current ALARA SDP to equate the

accuracy of ALARA job planning dose estimates directly to safety and safety

significance, since the dose estimates are set low to encourage proper worker

behaviors. STARS points out that this can be perceived as creating an unintended

disincentive to ALARA planning goals. 
11

TVA commented that experiences with the treatment of estimated fault exposure

time (t/2 time) have shown that this metric can arbitrarily raise the regulatory

significance of certain issues. 

The State of New Jersey responded that, unfortunately, the ROP is becoming a

two tiered system: plants that are all Green, and plants that are not all Green. 

Licensees focus great effort on getting non-Green findings reduced in color. The

State of New Jersey continued that the unintended consequence is that plant

owners will do everything possible to eliminate any PIs or change inspection

findings that are not Green. This minimizes the role of the inspectors in the

process.

2. Do any aspects of the ROP inappropriately increase regulatory burden? 

(Please comment on any unnecessary overlap between ROP reporting

requirements with those associated with INPO, WANO, or the Maintenance

Rule.)

Non-industry respondents stated that the regulatory burden associated with PI

reporting requirements is appropriate. NEI stated that overall regulatory burden

decreased under the new ROP. All industry respondents recommend early

integration of WANO, INPO, Maintenance Rule and NRC PI information reporting

requirements. NEI and Rochester Gas and Electric suggested decreasing

regulatory burden by reducing PI inspection effort, and offered a number of other

specific suggestions targeted at reducing current regulatory burden under the new

ROP. TVA points out that previously ìSALP 1" (top performing) reactor plants may

have experienced an increase in regulatory burden under the new ROP. UCS

stated (without elaboration) that necessary regulatory burden may have been

reduced as an unintended consequence of reducing unnecessary regulatory

burden. The Commonwealth of Pennsylvania stated that some members of the

public continue to be skeptical of the idea of ìreducing regulatory burdenî on

licensees [presumably these members of the public do not see such reduction as

an appropriate feature or concern of the regulatory process].

3. Is the Significance Determination Process (SDP) usable and does it produce

consistent and accurate results?

Although industry stated that the SDP process generally produces consistent and

accurate results, industry had numerous suggestions for improving the various

SDP methodologies and the SDP implementation process. There was general

agreement among all respondents that portions of the SDP (e.g., fire protection)

are time consuming and complex, and completion and distribution of the Phase 2

Worksheets for the Reactor Safety SDP is a much needed improvement. UCS

stated that information provided by licensees during the SDP process should be

subject to 10 CFR 50.9 sanctions (just like licensee provided PI information). UCS

stated that the SDP process is too slow. UCS stated that the SDP does not

provide consistent and accurate results, and provided two inspection finding case

histories from Beaver Valley Unit 1 to make the case that the SDP is
12

fundamentally flawed (in that it permits inadequate justifications for designating

findings as Green).

4. Are there areas of unnecessary overlap between the inspection program and

the performance indicators?

NEI commented that there is unnecessary overlap in the area of radiation safety

inspection and the Occupational Exposure Control Effectiveness PI.

5. Does the ROP assessment program provide timely, consistent, and relevant

assessment information?

Compared to the previous SALP process, the industry was satisfied with the

quality and timeliness of assessment information, although industry would

appreciate it if inspection report information on the Web were issued more timely

than quarterly. The State of New Jersey viewed the ROP as a ìlaggingî

assessment program, that suffers somewhat from its information being posted on

the Web generally one quarter after the information was first developed. STARS

had specific comments on restructuring the Action Matrix to better weight the

Reactor Safety strategic performance area.

6. Has the NRC implemented the ROP as defined by program documents?

NEI stated that the NRC is following the Action Matrix without exception, and in

general appears to be following its new ROP process procedures. However, NEI

commented that it has seen inconsistencies across the NRC regions. It is NEIís

interpretation that this is in part due to the process being only a year old. STARS

commented that throughout the pilot and initial implementation periods the staff

endeavored to maintain strict adherence to the program as designed. Peer

regulatory bodies, individual licensees, public interest groups, and general

members of the public did not respond directly to this question.

7. Please provide any additional (brief) information or comments on other

program areas related to the reactor oversight process. Other areas of

interest may be: the treatment of cross-cutting issues in the ROP, the riskbased evaluation process associated with determining event response, and

the reduced subjectivity and elevated threshold for documenting issues in

inspection reports.

NEI agreed with the ROP key premise that weaknesses in cross-cutting issues will

manifest themselves in PIs and inspection findings. NEI endorsed the proposed PI

replacement for counting automatic and manual scrams, and concurs that these

new PIs will place the emphasis on the proper area. TVA and NEI did not believe

that manipulation of the unplanned power change PI is occurring. Industry and the

Commonwealth of Pennsylvania had numerous comments targeted at future

improvements to the new ROP. UCS commented that the exclusion of design

deficiencies from the safety system unavailability PI non-conservatively inflates

safety system availability numbers. Three Mile Island Alert was concerned that the
13

NRC has stated that a given reactor plant can have multiple Red PIs and continue

to operate.
ROP SELF-ASSESSMENT PROGRAM

Task Lead: Alan Madison, NRR/IIPB

Attachment 3
1

Reactor Oversight Process Self-assessment Program

Background

The reactor oversight process (ROP) development model presented in SECY-99-007,

ìRecommendations for Reactor Oversight Process Improvements,î dated January 8, 1999,

included an ongoing self-assessment process that would utilize objective measures and predetermined criteria to monitor the performance of the ROP. During the 6-month ROP pilot

program, criteria were established to evaluate the results of implementing each of the

components of the ROP at the pilot plants. In addition, the staff employed a number of methods

to obtain internal and external stakeholder feedback. The results of these efforts were used by

the staff and the Pilot Program Evaluation Panel (PPEP) to aid in determining the efficacy of the

new process and the advisability of proceeding to initial implementation. The ROP transition plan

presented in SECY-00-0049, ìResults of the Revised Reactor Oversight Process Pilot Program,î

dated February 24, 2000, stated that the staff would continue to perform program selfassessments to collect additional lessons learned and gain insights from the new oversight

process. 

Development

Employing the lessons learned from the pilot program, the staff began development of a program

for monitoring the major components of the ROP. The major components of the ROP are: (1) the

performance indicators, (2) the inspection program, (3) the significance determination process

(SDP), and (4) the assessment process. The ROP self-assessment program utilizes eight

criteria. The first four criteria monitor the ROPís ability to be: (1) objective, (2) risk-informed, (3)

understandable, and (4) predictable. These criteria derive from the original design objectives of

the ROP. The other four criteria monitor the ROPís ability to: (1) maintain safety, (2) increase

public confidence, (3) make NRC activities and decisions more effective, efficient, and realistic,

and (4) reduce unnecessary regulatory burden on stakeholders. These criteria derive from the

Agencyís performance goals as outlined in the Strategic Plan. 

Via a series of facilitated working meetings involving selected program experts from the Office of

NRR, the Office of Enforcement, and the Office of Research (RES), the staff first developed a set

of questions with the objective of defining the ROP performance information necessary to assure

achievement of each criteria. Subsequent meetings focused on developing metrics to measure

or indicate ROP performance with respect to each question. The proposed criteria, questions

and metrics were presented to regional and Headquarters management for review and comment. 

Following incorporation of their comments, the results were presented to the Initial

Implementation Evaluation Panel (IIEP) for review and comment. The resultant criteria and

metrics were issued for implementation on October 18, 2000. The first set of partial data was

collected during November and December 2000, and the IIEP was briefed on initial results on

January 10, 2001.

Self-Assessment Metrics

As part of implementing a Planning, Budgeting, and Performance Management (PBPM) process,

the Agency has developed program-level operating plans, that include performance measures
2

and targets. The ROP self-assessment program is not meant to replicate or replace this activity,

however, many of the same or similar measures and criteria are used.

The proposed metrics rely on information from various sources, including the Reactor Program

System (RPS), the inspection program, periodic independent audits, stakeholder surveys, and

public comment. In addition to the metrics developed for each major component of the ROP,

metrics of a more general nature have been developed utilizing stakeholder feedback to gauge

overall performance.

With the exception of stakeholder surveys, Federal Register notice responses, and some audit

results, data is collected quarterly. Stakeholder surveys and Federal Register notices will be

issued periodically, as appropriate. Also, periodic equipment trending reports issued by RES are

reviewed to identify additional insights into ROP performance.

The data are compared to specific, pre-established criteria and an analysis of its meaning or

programmatic impact is completed. Thus a favorable comparison of data to criteria would

indicate the ROP met the process goals and objectives and likely no programmatic changes

would be recommended. However, in the event of an unfavorable comparison, more analysis

would be required to determine causal factors and develop planned process improvements.

Success criteria for each of the performance metrics have been established. In most cases,

success is defined as an improving trend. Quantitative success criteria for many of the

performance metrics could not be developed, because of the newness of the ROP and the

resultant lack of data needed to establish thresholds. For these metrics, baseline data was

collected and will be used to monitor trends and establish thresholds for action, as appropriate. 

Consequently, success of the first year of implementation can not be assessed by the metrics

alone.

Reports

Two types of reports will be issued. Each quarter, graphical presentations of performance

metrics, including current data compared with established criteria, and a summary of selfassessment performance metric outcomes that highlight any areas of concern and proposed

corrective actions will be issued. Each year the staff will also issue an annual report, consisting

of an overall summary and several attachments. The overall summary report will address the

following areas: (1) efficacy of the ROP, (2) industry performance trends, (3) resource

expenditures, (4) public interaction, and (5) any other considerations. The first annual report will

be issued by June 30, 2001. All reports will be made available on the ROP internal and external

Web sites.

Conclusions

While it is too early to draw detailed conclusions from the self-assessment data collected to date,

some general statements can be made regarding the efficacy of the ROP. In general, the ROP

has been successful in being more objective, risk-informed, understandable, and predictable than

the previous process. In addition to other objective measures, comments received from internal

and external stakeholders support this conclusion and represent an improving trend in perception

of performance since initial implementation. In contrast, internal stakeholders are critical of the

training provided prior to initial implementation and of the responsiveness by Headquarters staff
3

to regional feedback. The remaining criteria: maintain safety, increase public confidence,

improve effectiveness, efficiency and realism of NRC activities and decisions, and reduce

unnecessary regulatory burden, are harder to determine and will require additional time to

assess. For example, the impact on safety of any regulatory approach cannot be gauged quickly,

since: (1) many factors affect safety, (2) there is some normal variation in indicators, (3) there is

usually a significant lag time between a regulatory change and the full manifestation of its impact,

and (4) conditions that potentially impact safety may exist for some time before detection (e.g.,

design and latent errors). However, it should be noted that the ROP self-assessment program is

a continuous process that includes responding to noted deficiencies and incorporating

improvements in an on-going manner. 

It should also be noted that the self-assessment program itself is still undergoing development

and review. New metrics may be added to provide a more complete picture and some metrics

may be eliminated if found to not provide an accurate indication of ROP performance. ROP selfassessment program changes will be identified in the annual report.

Insights based on the results to date regarding individual aspects of the ROP are discussed in the

attachments of this paper focused on performance indicators, inspection program, significance

determination process, and assessment.
IIEP FINAL REPORT

Attachment 4
REACTOR OVERSIGHT PROCESS

INITIAL IMPLEMENTATION EVALUATION PANEL

FINAL REPORT

MAY 10, 2001

ADAMS ML011290025
TABLE OF CONTENTS

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Introduction ................................................................. 1

Approach and Objectives ...................................................... 1

Panel Conclusions and Recommendations ......................................... 2

Overall Reactor Oversight Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Significance Determination Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Assessment and Enforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

ROP Self-Assessment Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

IIEP Charter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 1

IIEP Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 2

Bibliography of Panel Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 3

Sources of Panel Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 4
i

EXECUTIVE SUMMARY

On May 17, 2000, the Commission directed the NRC staff to convene a panel under the Federal

Advisory Committee Act to evaluate the first yearís implementation of the Reactor Oversight

Process (ROP). The Initial Implementation Evaluation Panel (IIEP) was established on October

17, 2000. The panel met six times between November 2000 and April 2001.

The IIEP has concluded that the ROP is a notable improvement over the previous licensee

performance assessment program and should be continued. The reactor oversight process has

made progress toward achieving the Agencyís four performance goals: 1) maintain safety, 2)

increase public confidence, 3) increase regulatory effectiveness and efficiency, and 4) reduce

unnecessary regulatory burden. In addition, the process provides a more objective, riskinformed, predictable, and understandable approach to the oversight of commercial nuclear

reactor facilities. 

The NRC and the nuclear industry expended substantial time and effort communicating with their

staff members and public stakeholders about the process changes and paradigm shifts embodied

in the ROP. Although the change-management tools used to communicate and carry out the

changes were generally successful, the panel identified continuing tension as a result of three

changes in regulatory philosophy: maintaining safety rather than improving safety, applying riskinformed regulation rather than deterministic regulation, and using indicative measures of

performance rather than predictive measures of performance. To a large extent, stakeholder

concern with these regulatory changes are common denominators for the various issues

identified by the panel. The tension created by these underlying changes has affected the

degree to which the performance goals of the ROP can ever be fully achieved according to

various stakeholdersí perspectives about what constitutes a successful oversight process. On

the other hand, it is this tension that helps to ensure that the NRC, the industry, and public

stakeholders will continue to engage in creative dialogue focused on the safe operations of

nuclear power reactors. The panel considers the interaction among stakeholders throughout the

development process, and going forward, to be an important feature of the ROP. Continued

management attention is needed to sustain genuine and substantive interaction among

stakeholders, mindful of the challenges and creative potential inherent in the tensions of the

complex nuclear regulatory environment.

An ancillary common concern raised by the panel members in evaluating the specific issues

discussed in the report is the continuing need for sufficient resources to maintain the formal

processes and infrastructure for the ROP. NRC resources are needed to evaluate, pilot,

communicate, and implement future enhancements and ensure regional consistency. Although

the staff has learned many lessons from the first yearís implementation and has already made

numerous changes, there are many issues to be resolved and still other issues to be discovered

as the ROP continues to evolve. The many issues regarding the significance determination

process reflect the degree of change in the use of risk insights in the ROP and the substantial

work that remains to complete development of the staffís suite of tools. 

The panel considered the following recommendations as high priority:

! Establish a formal program and assign sufficient resources to enhance communications

necessary for improving the ROP. The program should accumulate lessons learned,

provide multiple and diverse opportunities for comment to all internal and external
ii

stakeholders, respond to stakeholdersí comments, and have a process for making timely

process changes.

! Revise the ROP communication plan to include outreach activities designed to inform the

public about the process and its relationship to the Agencyís mission of protecting the

public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the information

on the ROP Web page to improve and simplify public access to the information. Identify

methods, using stakeholder input, to improve public outreach efforts.

! Establish a structured ongoing program to evaluate long-term ROP effectiveness and to

test ROP assumptions. As a minimum, this includes integrating the insights of the ROP

self-assessment program and the overall assessment of industry performance. The staff

should also consider periodically engaging internal and external stakeholders to

independently assess the ROP.

! Continue the efforts of the cross-cutting issues task force and clarify the ROP guidance

on the identification and disposition of cross-cutting issues.

! Evaluate lessons learned from initial implementation to achieve parity in the treatment of

risk-significant inspection findings and crossed performance indicator thresholds. The

evaluation should verify that the outcomes from the performance indicators and inspection

findings accurately reflect the significance of the issues. The staff should consider

addressing this issue by adjusting the Green/White thresholds or modifying the Action

Matrix.

! Ensure that the staffís ROP self-assessment program identifies and evaluates any

unintended consequences or unnecessary regulatory burden caused by the performance

indicators and that changes are made where appropriate.

! Expedite the efforts to resolve the concerns about the safety system unavailability

performance indicators and implement any needed revisions to NEI 99-02, ìRegulatory

Assessment Performance Indicator Guideline,î to achieve consistency with other

applications.

! Evaluate the inspection approach for the physical protection cornerstone and revise the

inspection program as necessary after the pending safeguards performance assessment

pilot program and the physical security rulemaking are completed.

! Validate and issue the revised reactor safety significance determination process Phase 2

worksheets.

! Continue efforts to obtain improved and standardized risk analysis tools for the reactor

analysts.

! Review lessons learned from use of the fire protection significance determination process,

and improve the risk characterization tool to make it more meaningful, effective, and

efficient.
iii

! Continue development of an improved physical protection significance determination

process.

! Evaluate the need for other significance determination tools. The staff should carefully

evaluate any potential changes against the Agencyís goals. For example, a new

significance determination process that increases regulatory burden should have a

corresponding benefit in maintaining safety or increasing the Agencyís effectiveness.

! Evaluate lessons learned from initial implementation and revise the as-low-as-reasonablyachievable (ALARA) significance determination process as necessary.

! Evaluate a graded approach for resetting non-Green inspection findings as entry

conditions into the Action Matrix.

! Evaluate and clarify the guidance on the designation, definition, and use of what are

presently called ìNo Colorî findings, and find a more appropriate term for these findings.

As part of its evaluation, the panel reviewed the staffís self-assessment program and the

performance measures for the ROP. The panel concluded that the self-assessment program

has the necessary elements to evaluate the ROP against the Agencyís performance goals;

however, the panel could not evaluate the effectiveness of the program given that the first yearís

assessment data are not yet available. Nevertheless, the panel found that, for the most part, the

staff had identified the same issues the panel identified.

The IIEP had the benefit of information obtained during the first full year of nationwide

implementation of the ROP. During the year, the oversight process elements were extensively

exercised and many experiences that exemplified concerns and issues were available for the

panelís consideration. The panel observed that the staff actively solicited stakeholder input

throughout initial implementation, as it did during development and piloting of the ROP. The

panel believes that the level of stakeholder involvement has been unprecedented for an NRC

process change and is reflected in the quality of the process. Public workshops, public meetings

near all facilities, surveys, and formal internal feedback processes were critical to the staffís

efforts to further refine the ROP.
1

FINAL REPORT OF THE REACTOR OVERSIGHT PROCESS

INITIAL IMPLEMENTATION EVALUATION PANEL

Introduction

The NRC implemented, nationwide, a revised Reactor Oversight Process (ROP) for commercial

nuclear power plant licensees on April 2, 2000. Background information on the development of

the ROP and the results of the pilot program is given in Commission papers SECY-99-007,

ìRecommendations for Reactor Oversight Process Improvements,î dated January 8, 1999,

SECY-99-007A, ìRecommendations for Reactor Oversight Process Improvements (Follow-up to

SECY-99-007),î dated March 22, 1999, and SECY-00-049, ìResults of the Revised Reactor

Oversight Process Pilot Program,î dated February 24, 2000. These Commission papers

described the scope and content of performance indicator reporting, a new risk-informed baseline

inspection program, a new assessment process, and revisions to the enforcement policy. 

Commission paper SECY-00-049 also described the results from the Pilot Program Evaluation

Panel (PPEP), including a recommendation to proceed with initial implementation of the ROP at

all power reactor facilities. On March 28, 2000, the Commission approved initial implementation

of the ROP. 

On May 17, 2000, in a staff requirements memorandum in response to SECY-00-049, the

Commission directed the NRC staff to convene another evaluation panel under the Federal

Advisory Committee Act to evaluate the first yearís implementation of the ROP. The Initial

Implementation Evaluation Panel (IIEP) was established on October 17, 2000. The purpose of

the cross-disciplinary oversight panel was to independently monitor and evaluate the results of

the first yearís implementation of the ROP and provide advice and recommendations to the

Director of the Office of Nuclear Reactor Regulation on reforming and revising the ROP (see

Attachment 1). The panel, like the PPEP, was made up of a cross-section of stakeholders. An

NRC senior resident inspector and a senior reactor analyst were added to the panel at the

Commissionís request. The NRC selected the panel members to represent the views of diverse

groups that had an expressed interest in the changes to the ROP (see Attachment 2).

Approach and Objectives

The IIEP conducted six meetings during the first yearís implementation of the ROP. All meetings

were open to the public and all meeting material was placed in the NRCís public document room. 

Additionally, the meeting notices, summaries, and transcripts were placed on the NRCís ROP

Web page. Attachment 3 is a bibliography of the significant documents on the panelís activities. 

The IIEP worked as a management-level cross-disciplinary oversight group of experts to evaluate

whether the new regulatory oversight process can be effectively carried out and whether it has

achieved its overall objectives. The panel solicited and obtained additional views, to supplement

the membersí personal insights, from representatives of four States (New Jersey, Pennsylvania,

Illinois, and Vermont), the Union of Concerned Scientists, the Nuclear Energy Institute (NEI),

NRC resident inspectors, NRC senior reactor analysts, the NRC Office of Public Affairs, and

McGraw-Hill (see Attachment 4). The NRC staff members directly involved in the process

development reported on the status of the initial implementation and responded to questions and

comments.

During the IIEP meetings, the panel discussed and generally agreed to the following objectives: 
2

(1) Determine whether the ROP is achieving the following goals: 

ï Maintaining safety 

ï Increasing public confidence 

ï Increasing effectiveness and efficiency 

ï Reducing unnecessary regulatory burden 

ï Objective

ï Risk-informed

ï Predictable

ï Understandable

(2) Determine whether the more significant problem areas of the ROP have been identified. 

(3) Determine whether the NRC has developed a sound self-assessment program for the ROP 

 and, if so, whether it includes mechanisms for self-correction. 

To accomplish these objectives, each panel member was requested to provide a list of issues

regarding implementation of the ROP. The panel memberís issues were compiled, categorized,

and integrated with other issues presented to the panel during its meetings. The issues were

sorted according to the ROP elements they affected: performance indicators (P), inspection (I),

significance determination process (S), and assessment and enforcement (A). Those issues that

were identified as being pertinent to more than one element of the ROP were placed into an

overall (O) category.

With a facilitatorís assistance, the panel collectively evaluated the list of issues and developed

group consensus on the description of each issue, its priority, the primary ROP performance

goals affected, and the panelís recommendation for addressing the issue. The panel defined

consensus in its bylaws as no one panel member dissenting from the position taken. In practice,

this meant all panel members ìcould live withî the group position taken on the issue.

 

Each issue was assigned one of two priorities. The panel defined Priority 1 issues as those

issues that should receive high priority. Priority 2 issues were defined as issues for the staffís

consideration. Although the panel prioritized the various issues, it did not identify a timetable

within which the staff should resolve the issues. The panel recognized the complexity of some

issues and the need for additional time and information to evaluate solutions and the impacts of

any proposed changes. Furthermore, the panel did not lower the priority of an issue because the

staff was already resolving the issue. In developing its recommendations, the panel did not

specify how to resolve the issues identified in this report. The staff resolved some of the issues

the panel identified before the panel completed its activities. The panel did not include these

issues in the report.

Panel Conclusions and Recommendations

The IIEP concluded that the ROP is a notable improvement over the previous licensee

performance assessment program and should be continued. The reactor oversight process has

made progress toward achieving the Agencyís four performance goals: 1) maintain safety, 2)

increase public confidence, 3) increase regulatory effectiveness and efficiency, and 4) reduce

unnecessary regulatory burden. In addition, the process provides a more objective, riskinformed, predictable, and understandable approach to the oversight of commercial nuclear

reactor facilities.
3

The NRC and the nuclear industry expended substantial time and effort communicating with their

staff members and public stakeholders about the process changes and paradigm shifts embodied

in the ROP. Although the change-management tools used to communicate and carry out the

changes were generally successful, the panel identified continuing tension as a result of three

changes in regulatory philosophy: maintaining safety rather than improving safety, applying riskinformed regulation rather than deterministic regulation, and using indicative measures of

performance rather than predictive measures of performance. To a large extent, stakeholder

concern with these regulatory changes are common denominators for the various issues

identified by the panel. The tension created by these underlying changes has affected the

degree to which the performance goals of the ROP can ever be fully achieved according to

various stakeholdersí perspectives about what constitutes a successful oversight process. On

the other hand, it is this tension that helps to ensure that the NRC, the industry, and public

stakeholders will continue to engage in creative dialogue focused on the safe operations of

nuclear power reactors. The panel considers the interaction among stakeholders throughout the

development process, and going forward, to be an important feature of the ROP. Continued

management attention is needed to sustain genuine and substantive interaction among

stakeholders, mindful of the challenges and creative potential inherent in the tensions of the

complex nuclear regulatory environment.

! Maintaining safety rather than improving safety: One premise of the NRCís strategic plan

is that the nuclear power industryís performance has improved substantially over the past

10 years and nuclear reactors, collectively, are operating above acceptable safety levels

consistent with the Agencyís Safety Goal Policy. The staff designed the ROP to maintain

the current level of safety. The process is designed to improve safety performance before

it falls below acceptable levels, not to continually improve the safety margins that currently

exist. Some public stakeholders do not believe that current nuclear industry performance

is sufficient to assure public health and safety without continual improvement. This could

limit the publicís confidence in the process.

! Risk-informed regulation rather than deterministic regulation: In many ways the ROP is

ahead of the other regulatory processes in using risk insights. It is difficult to implement a

risk-informed oversight process while in a deterministic regulatory framework. For

example, the integration of the significance determination process with the NRCís

assessment program puts both the inspector and licensee in a conflicting situation. The

ROP focuses on risk-significant issues, but licensees must still comply with regulatory

requirements that are not risk-informed. This conflict has contributed to many of the

issues discussed in this report. An additional concern of public stakeholders is the

perceived overreliance on existing risk analysis tools for regulatory decision making. Over

the long term, the staff must continue risk-informing the regulations to close the gap

between the regulatory framework and the oversight process.

! Indicative measures of performance rather than predictive measures of performance: The

ROP, using performance indicator thresholds and the significance determination process,

is an indicative process, whereas the previous performance assessment process

attempted to be predictive by using performance issues of low safety significance to

identify declining performance trends. A premise of the ROP is that the licenseeís

corrective action program best handles low-level performance trends and that a regulatory

response is not required until a threshold is crossed. A related assumption is that a

licensee will not normally pass directly from the licensee response column to the

unacceptable performance column of the Action Matrix, giving the NRC time to respond
4

before plant performance becomes unacceptable. Many of the concerns about crosscutting issues and inspection report thresholds come from skepticism about this

assumption.

Though the panel focused on areas needing improvement, it noted many positive attributes and

outcomes. We note some of these in the introductory comments on each ROP element.

The panelís recommendations are as follows:

! Establish a formal program and assign sufficient resources to enhance communications

necessary for improving the ROP. The program should accumulate lessons learned,

provide multiple and diverse opportunities for comment to all internal and external

stakeholders, respond to stakeholdersí comments, and have a process for making timely

process changes. (O-1)

! Revise the ROP communication plan to include outreach activities designed to inform the

public about the process and its relationship to the Agencyís mission of protecting the

public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the information

on the ROP Web page to improve and simplify public access to the information. Identify

methods, using stakeholder input, to improve public outreach efforts. (O-2)

! Establish a structured ongoing program to evaluate long-term ROP effectiveness and to

test ROP assumptions. As a minimum, this includes integrating the insights of the ROP

self-assessment program and the overall assessment of industry performance. The staff

should also consider periodically engaging internal and external stakeholders to

independently assess the ROP. (O-3)

! Continue the efforts of the cross-cutting issues task force and clarify the ROP guidance

on the identification and disposition of cross-cutting issues. (O-4)

! Evaluate lessons learned from initial implementation to achieve parity in the treatment of

risk-significant inspection findings and crossed performance indicator thresholds. The

evaluation should verify that the outcomes from the performance indicators and inspection

findings accurately reflect the significance of the issues. The staff should consider

addressing this issue by adjusting the Green/White thresholds or modifying the Action

Matrix. (O-5)

! Ensure that the staffís ROP self-assessment program identifies and evaluates any

unintended consequences or unnecessary regulatory burden caused by the performance

indicators and that changes are made where appropriate. (P-1)

! Expedite the efforts to resolve the concerns about the safety system unavailability

performance indicators and implement any needed revisions to NEI 99-02, ìRegulatory

Assessment Performance Indicator Guideline,î to achieve consistency with other

applications. (P-2)

! Continue the efforts to identify and evaluate improvements to performance indicators. 

The staff should thoroughly evaluate any significant changes to the performance

indicators, following the structured process in Inspection Manual Chapter 0608,
5

ìPerformance Indicator Program.î Further, the staff should evaluate the effect of any

significant changes in the scope of information provided by performance indicators on the

Action Matrix and the baseline inspection program, and should identify any additional

costs and benefits. (P-3)

! Continue efforts to incorporate the answers to frequently asked questions into the

performance indicator guidance document, NEI 99-02, and to make the answers more

generic where possible. (P-4)

! Evaluate the inspection approach for the physical protection cornerstone and revise the

inspection program as necessary after the pending safeguards performance assessment

pilot program and the physical security rulemaking are completed. (I-1)

! Evaluate and revise guidance to inspectors as necessary to clarify and promote

consistency in documenting inspections. Continue conducting periodic audits of

inspection reports to identify and correct inconsistencies. The new documentation

thresholds for issues that have a defined level of safety or regulatory significance are

appropriate. Inspection observations and insights that do not reach the threshold should

continue to be communicated verbally to licensees for their consideration. (I-2)

! Evaluate inspection findings and performance indicator results for the first yearís

implementation and determine the appropriate level of effort to adequately assess risksignificant areas in the baseline inspections. Modify the process as appropriate. (I-3)

! After further experience with the ROP, review the results and consider whether to waive

certain parts of the baseline team inspections and let licensees assess themselves under

defined circumstances. (I-4)

! Validate and issue the revised reactor safety significance determination process Phase 2

worksheets. (S-1)

! Continue efforts to obtain improved and standardized risk analysis tools for the reactor

analysts. (S-2)

! Review lessons learned from use of the fire protection significance determination process,

and improve the risk characterization tool to make it more meaningful, effective, and

efficient. (S-3)

! Continue development of an improved physical protection significance determination

process. (S-4)

! Evaluate the need for other significance determination tools. The staff should carefully

evaluate any potential changes against the Agencyís goals. For example, a new

significance determination process that increases regulatory burden should have a

corresponding benefit in maintaining safety or increasing the Agencyís effectiveness. (S5)

! Evaluate lessons learned from initial implementation and revise the ALARA significance

determination process as necessary. (S-6)
6

! Use lessons learned from initial implementation to make the risk characterization process

expedient, scrutable, and understandable to all stakeholders. (S-7)

! Use lessons learned from initial implementation to clarify the definition of a performance

deficiency. (S-8)

! Evaluate a graded approach for resetting non-Green inspection findings as entry

conditions into the Action Matrix. (A-1)

! Evaluate and clarify the guidance on the designation, definition, and use of what are

presently called ìNo Colorî findings, and find a more appropriate term for these findings.

(A-2)

! Clarify the guidance on the objectives and structure of regulatory conferences and

communicate this guidance to the external and internal stakeholders. (A-3)

! Develop clear guidance on how to handle multiple related inspection findings, and

communicate the guidance to all stakeholders. (A-4)

The following five sections provide the consensus of the panel with respect to the description of

the issues, the priority, the primary performance goals affected, and the panelís

recommendations. Where appropriate, the panel has referred to the recommendations of the

Pilot Program Evaluation Panel when the two panels identified similar issues or made similar

recommendations.

Panel members expressed two minority views:

(1) The initial implementation does not appear to demonstrate that the ROP has improved

identification of design basis issues or validated recent initiatives to scope design basis

issues. NUREG-1275, Volume 14, ìCauses and Significance of Design-Basis Issues at

U.S. Nuclear Power Plants,î draws a clear connection between the number of

engineering (and design) inspection hours expended and the number of design basis

issues discovered. However, the number of engineering inspection hours in the first yearís

implementation has not yet been compiled, indicating to some observers that the staff is

not focused on design basis issues under the new process. Public confidence will not be

enhanced unless there is assurance that plants are properly designed, built as designed,

modified only with proper analysis, and properly maintained including sufficient attention to

"aging" phenomena. Uncertainties regarding design basis issues serve to undermine

confidence in Probabilistic Risk Assessments and the concept of "maintaining" safety. In

an October, 17, 1996 All Employees NRC meeting, Chairman Shirley Jackson opined that

one reason for the events at Millstone was that ". . . we stopped doing design basis

inspections too early, and relied on industry . . . without maintaining an appropriate

regulatory focus to assess whether in fact they were dealing with the issue in a timely

manner." 

(2) The IIEP make-up was weighted with regulators and industry to the extent that common

interest in moving the ROP forward, and other commonalties, may have limited the panel's

perspective. Given the common working culture, it is not surprising that the IIEP critique

was quite similar to the staff review group critique. Future panels might benefit from

inclusion of some additional individuals from outside of the NRC-licensee set, for example,
7

attorneys with nuclear specialties or academics. Panel builders might also add to the

quality of deliberation by a more even gender mix. While it enriches dialogue and

broadens perspective to add members from public interest groups, adding only one or two

with views likely to be quite alien to the remainder of the group on some issues is

problematic in terms of free and meaningful consensus building.
8

Overall Reactor Oversight Process

The ROP is a notable improvement over the previous licensee performance assessment program

and should be continued. The reactor oversight process has made progress toward achieving

the Agencyís four performance goals: 1) maintain safety, 2) increase public confidence, 3)

increase regulatory effectiveness and efficiency, and 4) reduce unnecessary regulatory burden. 

In addition, the process provides a more objective, risk-informed, predictable, and

understandable approach to the oversight of commercial nuclear reactor facilities. However, the

panel identified the following issues that the staff should address:

Issue O-1: Process improvements and stakeholder feedback 

Priority 1

Primary performance goals affected: Public confidence/effectiveness and efficiency

Issue description: As with any regulatory process, it is important that a formal infrastructure be

established to allow for stakeholder comments and questions. The infrastructure should promptly

review feedback and implement process improvements. The frequently asked questions process

used in the performance indicator program is a good mechanism for raising and resolving

licensee and inspector issues. This process promotes the open exchange of information and

establishment of uniform and consistent guidance. Other elements of the ROP, such as the

significance determination process, would benefit from a similar approach. This process should

also include a mechanism for the public to retrieve information on past questions and answers

and ensure that lessons learned and feedback information are communicated to the other

regions. 

The PPEP recommended continued feedback from inspectors and the ongoing modification of

procedures during the industrywide implementation to assure that the procedures are clear and

appropriately address the cornerstones.

Panel recommendation: Establish a formal program and assign sufficient resources to enhance

communications necessary for improving the ROP. The program should accumulate lessons

learned, provide multiple and diverse opportunities for comment to all internal and external

stakeholders, respond to stakeholdersí comments, and have a process for making timely process

changes.

Issue O-2: Public access to ROP information

Priority 1

Primary performance goals affected: Public confidence/understandable

Issue description: It is important that the public have confidence that the ROP provides the

regulator a means for accurately assessing the safety of plants and taking action where

necessary and that the process and actions be effectively communicated to the public. Likewise,

it is essential that the public have clear, unfettered access to accurate and meaningful information

to be able to reach its own conclusions.

The staff made significant improvements to the publicís access to plant performance information

during the initial implementation period. The staff established a Web site that displayed

information about the ROP. The publicís response to the Web site was generally very positive. 

The panel observed that the staff actively solicited stakeholder input throughout initial
9

implementation, as it did during development and piloting of the ROP. Public workshops, public

meetings near all facilities, and surveys provided opportunities for valuable input to the staff as

they sought to further refine the ROP.

However, much remains to be done to make the ROP understandable and accessible to public

stakeholders. At first, the public and media thought the new process relied solely on performance

indicators because the indicators were highlighted on the Web page. The highlighting caused a

few public stakeholders to believe that the NRC had abandoned the resident inspector program. 

Many stakeholders did not understand that insights from both performance indicators and

inspection findings were used to assess overall licensee performance. 

The staff has improved the structure and format of the Web pages to correct this perception. 

However, the Web pages need additional improvements. The ROP Web pages should start with

overview information and provide layered access to more detailed information (i.e., links to the

site-related documents). Site-specific pages could include a bulletin board with the status of

enforcement items and inspections. This would enable the public to understand, without

conducting exhaustive research, the status of important issues at the plant in their locale. A highlevel summary would also provide a means to differentiate the performance of one plant from

another. The timely posting of information is important to enhancing public confidence.

Panel recommendation: Revise the ROP communication plan to include outreach activities

designed to inform the public about the process and its relationship to the Agencyís mission of

protecting the public health and safety. Appropriate resources should be provided to revise and

implement the communication plan. Evaluate additional improvements to the information on the

ROP Web page to improve and simplify public access to the information. Identify methods, using

stakeholder input, to improve public outreach efforts. 

Issue O-3: Long-term process effectiveness

Priority 1

Primary performance goals affected: Maintain safety/effectiveness and efficiency/public

confidence

Issue description: The panel recognizes and agrees with the concern of members and

stakeholders that there are limits to what may be learned from a 1-year test of the ROP. The

staff should evaluate the long-term effectiveness of the process to determine whether the

performance indicators and inspection findings identify poorly performing plants. 

The significance determination process tools are a key element in ensuring the effectiveness of

the ROP. However, the significance determination process Phase 2 worksheets were not

available for much of initial implementation (see S-1). 

The ROP is based on certain assumptions (e.g., that licensee corrective action programs are

mature and support the basis for the licensee response band concept, that degraded

performance will reveal itself by ever increasing significant issues and crossed performance

indicator thresholds, and that all violations of NRC regulations do not require followup by the

NRC). As sufficient information and experience is obtained, assumptions either will be confirmed

or refuted. Whatever the result, there must be a validation process. In addition, there may be

unintended consequences of the ROP elements such as with some performance indicators (see

P-1). The final oversight process must focus on identifying issues of safety significance,
10

eliminating any underestimation of risk characterization determinations (false negatives), and

minimizing overestimation of issues (false positives).

The PPEP recommended that the staff continue to monitor industrywide implementation to

ensure that when a risk-significant event occurs, the event-specific response requires

reevaluation of the performance indicators and inspection results to address whether they missed

a cross-cutting or common-mode failure issue. The PPEP also concluded those process

assumptions had not been tested sufficiently.

Panel recommendation: Establish a structured ongoing program to evaluate long-term ROP

effectiveness and to test ROP assumptions. As a minimum, this includes integrating the insights

of the ROP self-assessment program and the overall assessment of industry performance. The

staff should also consider periodically engaging internal and external stakeholders to

independently assess the ROP.

Issue O-4: Cross-cutting issues

Priority 1

Primary performance goals affected: Maintain safety/public confidence

Issue description: During the development of the ROP, and initial implementation, some

inspectors were concerned about the identification and disposition of cross-cutting issues. The

concern was that licensee performance in the cross-cutting areas of human performance, safetyconscious work environment, and problem identification and resolution could become degraded

without being detected by the baseline inspection program and performance indicators. The

ROP addresses cross-cutting issues by highlighting them in inspection reports when they are

notable contributors to inspection findings or if an appreciable trend or pattern has emerged. The

staff further amplifies these concerns in assessment letters to the licensee when the concerns

constitute a substantive issue. 

The current process does not have sufficient criteria, thresholds, and definitions of cross-cutting

issues to ensure consistency in handling these issues. In addition, there is no predefined NRC

action if the inspection program identifies a substantive cross-cutting issue such as a deficient

corrective action program. The ROP does not provide for additional NRC engagement on crosscutting issues unless they are contributing causes to performance indicators or inspection

findings that have been characterized as White or greater. Some inspectors are also concerned

about the lack of a process to handle low-level human performance trends when it appears that

NRC actions could prevent the occurrence of a significant performance issue. The industry

believes the ROP should focus on performance outcomes, of which cross-cutting issues are but

one possible cause. 

One premise of the ROP is that either performance indicators or inspection findings will detect

degradation in the cross-cutting areas in time to allow for Agency action to protect the public

health and safety. Early data obtained from initial implementation suggests that there is a

correlation between cross-cutting issues and crossed thresholds consistent with the premise of

the process (i.e., the number of cross-cutting findings per plant appears to increase as you move

to the right in the Action Matrix).

The PPEP final report also highlighted the divergent views on the identification and disposition of

cross-cutting issues.
11

Panel recommendation: Continue the efforts of the cross-cutting issues task force and clarify

the ROP guidance on the identification and disposition of cross-cutting issues.

Issue O-5: Basis of Green/White thresholds

Priority 1

Primary performance goals affected: Public confidence/understandable

Issue description: The bases for the performance indicator Green/White thresholds are not riskinformed. The thresholds were selected to identify the 95-percent performance level (i.e.,

industry outliers). Since NRC action is the same for both White performance indicators and

White inspection findings, which are risk-informed, several problems have resulted. First, the

NRC and the licensee have a different perception of the impact and importance of White issues. 

Second, it is difficult to communicate to public stakeholders that a White performance indicator

may not be risk-significant when the NRC increases its regulatory response according to the

Action Matrix. This could impact public confidence in the NRC.

Panel recommendation: Evaluate lessons learned from initial implementation to achieve parity

in the treatment of risk-significant inspection findings and crossed performance indicator

thresholds. The evaluation should verify that the outcomes from the performance indicators and

inspection findings accurately reflect the significance of the issues. The staff should consider

addressing this issue by adjusting the Green/White thresholds or modifying the Action Matrix.
12

Performance Indicators

The integration of performance indicators into the ROP has provided objective measures for

assessing licensee performance. Additionally, licensees can accurately report performance

indicators without an excessive burden, and the public can easily understand the performance

data. The initial implementation period has verified that the performance indicators can focus

both licensee and NRC attention on issues that are either risk significant or relevant to promoting

desired performance. However, the panel identified the following issues that the staff should

address:

Issue P-1: Unintended negative consequences of performance indicators

Priority 1

Primary performance goals affected: Maintain safety/unnecessary regulatory burden

Issue description: The use of performance indicators may have unintended negative

consequences when they measure both desirable actions and performance issues. This could

lead to nonconservative decisions by licensees. In addition, NRC may impose unnecessary

regulatory burden when it takes actions based, in part, on licensee actions that are desirable and

appropriate.

The staff has noted that some licensees have altered normal operating and maintenance

practices solely to avoid conditions that may contribute to crossing a performance indicator

threshold. For example, the Unplanned Power Change performance indicator may cause a

licensee to delay needed equipment repairs for 72 hours to avoid counting a power reduction. In

contrast, a plant that appropriately conducts equipment repairs in a well-planned manner within

72 hours may be considered a poor performer. Another example is the Safety System

Unavailability performance indicator, which includes unavailability time for planned preventive

maintenance as well as unplanned corrective maintenance and equipment failures. A licensee

may consider delaying discretionary maintenance if it is near the performance indicator threshold. 

Despite these concerns, there have not been any known instances to date of unsafe actions by a

licensee because of the performance indicators.

Panel recommendation: Ensure that the staffís ROP self-assessment program identifies and

evaluates any unintended consequences or unnecessary regulatory burden caused by the

performance indicators and that changes are made where appropriate.

Issue P-2: Safety System Unavailability performance indicators

Priority 1

Primary performance goals affected: Maintain safety/risk-informed/understandable

Issue description: Many of the performance indicator frequently asked questions (FAQs) during

initial implementation involved the safety system unavailability performance indicators definitions

and guidance. The performance indicator definition of equipment unavailability is different from

that used by other NRC and industry programs that monitor or consider unavailability of safety

equipment (e.g., Maintenance Rule program and licenseeís probabilistic risk assessments). The

major hurdle in resolving this problem is determining whether the unavailability is to be measured

against the design basis or the risk analyses (i.e., operable versus functional). For example,

consideration of operator recovery actions are limited by this indicator but are allowed in other

programs. 
13

Other issues are how to treat fault exposure hours and what allowances should be made for

planned overhaul maintenance when a quantitative risk assessment has been previously

performed and approved by the NRC. The large number of generic and site-specific exceptions

to what equipment unavailability is counted in these performance indicators has made them

difficult to understand and may erode public confidence. Finally, these indicators also measure

appropriate actions by the licensee, such as planned preventive maintenance, so there is a

potential for unintended consequences (see P-1). 

Panel recommendation: Expedite the efforts to resolve the concerns about the safety system

unavailability performance indicators and implement any needed revisions to NEI 99-02,

ìRegulatory Assessment Performance Indicator Guideline,î to achieve consistency with other

applications.

Issue P-3: New performance indicators

Priority 2

Primary performance goals affected: Maintain safety/risk-informed

Issue description: Some current performance indicators and associated thresholds do not

directly correlate with risk. In addition, the panel found at least one unintended consequence of

performance indicators, specifically the misunderstanding that can occur because the

Green/White threshold for performance indicators is not related to risk, as with inspection findings

(see O-5). This misunderstanding causes the application of the Action Matrix to sometimes

appear inconsistent and calls into question the value of some performance indicators as an input

to performance assessment. For example, some emergency preparedness and physical

protection performance indicators do not directly correlate to risk to the public health and safety,

but are rather intended to identify weaknesses in licensee programs. The staff specifically avoids

the identification of such weaknesses in the more risk-focused cornerstone areas.

Panel recommendation: Continue the efforts to identify and evaluate improvements to

performance indicators. The staff should thoroughly evaluate any significant changes to the

performance indicators following the structured process in Inspection Manual Chapter 0608,

ìPerformance Indicator Program.î Further, the staff should evaluate the effect of any significant

changes in the scope of information provided by performance indicators on the Action Matrix and

the baseline inspection program, and should identify any additional costs and benefits.

Note: The panel cautions the staff not to eliminate a performance indicator solely because it does

not provide risk-informed information. The performance indicator may provide information useful

for other purposes (e.g., increasing public confidence, identifying programmatic issues). In this

case, the staff may need to adjust the threshold.
14

Issue P-4: Frequently asked questions

Priority 2

Primary performance goals affected: Understandable/effectiveness and efficiency

Issue description: During initial implementation of the ROP, licensee and NRC staff members

asked many questions regarding the performance indicator guidance. These questions were

documented, answered, and posted on the NRCís ROP Web site. Although this was a useful

approach for clarifying and interpreting the guidance in NEI 99-02, the large number of questions

made it difficult for all stakeholders to understand the guidance on the performance indicators. In

addition, the inspectors noted that some licensees used site-specific answers to the questions out

of context when applying it to their specific situation.

Panel recommendation: Continue efforts to incorporate the answers to frequently asked

questions into the performance indicator guidance document, NEI 99-02, and to make the

answers more generic where possible.
15

Inspection

The new inspection process has been effective in focusing the NRCís inspection efforts on areas

more important to risk and safety. The combination of baseline and supplemental inspections

provides sufficient coverage of the attributes of the seven safety cornerstones. The planning for

both the overall inspection effort at a given site and for the selection of individual procedural

samples has been more risk-informed. The improvements in procedural guidance have focused

NRC resources on reviewing and assessing performance as opposed to providing subjective

views of various licensee processes and programs. The new procedural guidance, being more

objective and risk-informed, has also improved the consistency among the regions and the

individual inspectors. Since the outcomes of the new inspection program are more risk-informed,

the dialogue between the licensee and NRC is now primarily focused on safety. However, the

panel identified the following issues that the staff should address:

Issue I-1: Physical protection cornerstone inspections

Priority 1

Primary performance goals affected: Objective/unnecessary regulatory burden

Issue description: Licensees have expressed concern regarding the NRCís approach to

inspecting the licenseeís response to contingency events (i.e., force-on-force drills) and applying

the physical protection significance determination process (see S-4). Before initial

implementation, the staff removed the evaluation of force-on-force exercises from the baseline

inspection procedure (71130.03 ìResponse to Contingency Events (Protective Strategy and

Implementation of Protective Strategyî) and continued the Operational Safeguards Response

Evaluation (OSRE) inspections under inspection procedure 81110. The industry has proposed a

pilot program for industry-conducted drills and exercises (SECY-01-0060). The industry has

requested that self-assessment initiatives be considered as part of the inspection process similar

to how the staff handles the evaluation of emergency drills in the emergency preparedness

cornerstone.

Panel recommendation: Evaluate the inspection approach for the physical protection

cornerstone and revise the inspection program as necessary after the pending safeguards

performance assessment pilot program and the physical security rulemaking are completed. 

Issue I-2: Inspection report documentation threshold

Priority 2

Primary performance goals affected: Public confidence/effectiveness and efficiency

Issue description: The staff significantly revised the guidance for documenting inspections

under the ROP to provide a more structured approach for deciding the issues to be documented. 

NRC Inspection Manual Chapter 0610*, ìPower Reactor Inspection Reports,î changed the

documentation threshold by eliminating discussions of positive performance attributes, minor

violations, licensee-identified findings, and nonregulatory concerns (such as general weaknesses

in programs or inspector observations) from the reports. The staff implemented these

documentation changes, in part, to address industry concerns that subjective performance

observations and inspector opinions in past reports were not based on regulatory requirements

and did not correspond to any observable performance problems. In parallel with development of

the ROP, the Office of Enforcement also developed improved guidance to clarify the threshold for

minor violations. This guidance was incorporated into MC 0610*. With the higher threshold for
16

issues discussed in the inspection reports, some public stakeholders are concerned that they

may receive less performance information than under the previous program. On the other hand,

some stakeholders have noted that the more focused, albeit reduced, information in inspection

reports allows for a more effective identification of significant regulatory and performance issues. 

Some inspectors are also concerned that they may miss low-level performance trends if they do

not include them in the inspection reports. Stakeholders have also expressed a concern that the

staff is not consistently implementing the new report documentation thresholds. 

NRC Inspection Manual Chapter 2515, ìLight Water Reactor Inspection Program - Operations

Phase,î endorses action by inspectors to provide licensees well-considered insights beyond those

to be documented in the inspection report. Although the documentation threshold has changed,

most licensees request that inspectors continue to share all of their observations at exit meetings. 

Since these observations do not meet the report documentation threshold, inspectors do not

provide them to the public. This practice creates the appearance that relevant performance

information is being intentionally withheld from the public. Some public stakeholders have

suggested that all exit meetings should be open to the public to address this concern.

Panel recommendation: Evaluate and revise guidance to inspectors as necessary to clarify and

promote consistency in documenting inspections. Continue conducting periodic audits of

inspection reports to identify and correct inconsistencies. The new documentation thresholds for

issues that have a defined level of safety or regulatory significance are appropriate. Inspection

observations and insights that do not reach the threshold should continue to be communicated

verbally to licensees for their consideration. 

Issue I-3: Appropriate level of baseline inspection

Priority 2

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: The inspection resource expenditures are about the same under the ROP as

under the previous inspection program. The staff has noted wide ranges in actual resource

expenditures, however, for certain procedures during the first year of implementation. Licensees

have commented that specific cornerstones, such as occupational radiation exposure, appear to

have too many resources applied when licensee performance trends and the previous inspection

program are considered objectively (i.e., inspection resources have increased although overall

occupational exposure has decreased).

The PPEP recommended that the resource levels required to plan and implement the baseline

inspection program be evaluated during industry wide implementation, but cautioned that process

effectiveness not be measured solely by increases or decreases in resource utilization. They

also recommended that the appropriateness of the inspection frequency and scope continue to

be assessed during industry wide implementation.

Panel recommendation: Evaluate inspection findings and performance indicator results for the

first yearís implementation and determine the appropriate level of effort to adequately assess risksignificant areas in the baseline inspections. Modify the process as appropriate.

Issue I-4: Use of licensee self-assessment information
17

Priority 2

Primary performance goals affected: Unnecessary regulatory burden/effectiveness and

efficiency/public confidence

Issue description: In the previous inspection program, there were cases where the NRC did not

conduct portions of specific team inspections if the licensee had conducted a rigorous selfassessment of the same area and placed the self-assessment in the public domain. The staff

outlined this process in Inspection Procedure 40501, ìLicensee Self-Assessments Related to

Team Inspections.î When NRC management approved the use of this procedure as an

alternative to independent NRC inspection, the staff reviewed the scope and results of the

licenseeís self-assessment and the qualification of team members, and monitored ongoing

portions of the licenseeís review. The ROP has not provided this flexibility for reducing

unnecessary regulatory burden. While this approach did not save a significant amount of NRC

inspection resources, it reduced regulatory burden on the licensees, and the licensee staff gained

a better understanding of their systems. However, it could reduce public confidence in the NRC

as an independent regulator. 

Panel recommendation: After further experience with the ROP, review the results and consider

whether to waive certain parts of the baseline team inspections and let licensees assess

themselves under defined circumstances.
18

Significance Determination Process

The significance determination process (SDP) has shown that risk information can be used in a

systematic, practical, and repeatable manner. The SDP has given NRC inspectors an objective

process for consistently characterizing inspection findings, and it has provided the NRC with

improved tools for prioritizing emergent issues and selecting individual inspection samples. The

SDP has focused both NRC and licensee attention on the risk associated with identified issues,

as opposed to focusing attention and resources on the compliance implications. However, the

panel identified the following issues that the staff should address:

Issue S-1: SDP Phase 2 Worksheets

Priority 1

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: The primary tools to be used by field inspectors in determining the risk

significance of reactor safety cornerstone inspection findings, the Phase 2 worksheets, were not

available to the inspectors during initial implementation. The draft Phase 2 worksheets did not

accurately reflect the current site probabilistic risk assessments and equipment configurations

and needed to be revised. It was necessary for the regional risk analysts to perform resourceintensive Phase 3 analyses of all potential non-Green reactor safety issues to determine the risk

significance of findings. The lack of adequate Phase 2 worksheets negatively affected the

effectiveness and efficiency of the ROP during the first year.

The PPEP highlighted the importance of having plant-specific SDP worksheets before

industrywide implementation.

Panel recommendation: Validate and issue the revised reactor safety SDP Phase 2 worksheets.

Issue S-2: Quality of NRC PRA tools

Priority 1

Primary performance goals affected: Maintain safety/public confidence

Issue description: The ROP relies on the quality and consistency of the probabilistic tools used

by the NRC risk analysts and inspectors for their risk characterizations and decisions. Currently,

the NRC relies heavily on the individual plant probabilistic risk assessments developed by the

licensees, but the quality of these tools varies. The lack of validated Phase 2 worksheets

magnified this concern during initial implementation (see S-1). For some findings that reached

the Phase 3 analysis stage, the licensees with state-of-the-art tools felt penalized when the staff

used their less sophisticated results to determine the risk significance of an issue. Their concern

was predictability, since the calculated risk significance may have been greater with a less

sophisticated tool. The lack of defined standards for methods and models also hampers the

staffís ability to obtain timely and consistent results when evaluating findings. 

Panel recommendation: Continue efforts to obtain improved and standardized risk analysis

tools for the reactor analysts. 
19

Issue S-3: Fire Protection SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/understandable

Issue description: The application of the fire protection SDP during the first year showed that it

was excessively complex and subjective. This has limited its usefulness as a tool in evaluating

some fire protection findings. Besides the complexity of the fire protection SDP, the resulting risk

characterization of the findings did not seem consistent with findings in other cornerstone

significance determinations.

Panel recommendation: Review lessons learned from use of the fire protection SDP, and

improve the risk characterization tool to make it more meaningful, effective, and efficient.

Issue S-4: Physical Protection SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/risk-informed

Issue description: The physical protection SDP was initially aligned to the reactor safety SDP. 

The staff found that the process was problematic in several cases during initial implementation. 

The SDP results seemed inconsistent with the actual risk significance. The staff made interim

revisions to the physical protection SDP to incorporate direction contained in Staff Requirements

Memorandum COMSECY-00-0036 dated January 25, 2001. 

Panel recommendation: Continue development of an improved physical protection SDP.

Issue S-5: Development of SDPs for other areas

Priority 1

Primary performance goals affected: Effectiveness and efficiency/predictable

Issue description: Substantial work is needed to complete the suite of SDP tools. During the

first yearís implementation, the established SDP did not provide an effective tool for evaluating all

inspector findings in certain areas (e.g., the staff identified the need for effective significance

determination tools concerning shutdown, containment, and external events). The staff cannot

easily assess other process-oriented inspection findings, such as those involving inadequate

application of the Maintenance Rule, with the existing SDP unless there is a measurable impact

on plant equipment.

Panel recommendation: Evaluate the need for other significance determination tools. The staff

should carefully evaluate any potential changes against the Agencyís goals. For example, a new

SDP that increases regulatory burden should have a corresponding benefit in maintaining safety

or increasing the Agencyís effectiveness.
20

Issue S-6: ALARA SDP

Priority 1

Primary performance goals affected: Effectiveness and efficiency/unnecessary

regulatory burden

Issue description: During initial implementation, the staff experienced problems when using the

ALARA SDP. The SDP screened out all issues identified at plants that had a 3-year average

collective dose equal to or below the screening criteria based on the median industry

performance. The unintended consequence of this SDP structure is that the inspectors cannot

document the occurrence of an ALARA failure at these better performers in the inspection report,

but they document the identical finding at a plant above the screening criteria. In addition, the

staff designed the SDP structure to evaluate the licenseeís performance in ALARA on a per job

basis, but did not define a job. Another potential unintended consequence is that some licensees

may estimate exposure in a very conservative manner during ALARA planning. Some

stakeholders have commented that the SDP screening criteria are an inappropriate de facto

definition of ALARA for occupational doses at nuclear power plants.

Panel recommendation: Evaluate lessons learned from initial implementation and revise the

ALARA SDP as necessary.

Issue S-7: Process for evaluating and communicating SDP conclusions

Priority 2

Primary performance goals affected: Effectiveness and efficiency/understandable/public

confidence

Issue description: Using an SDP to place all inspection insights into a risk-informed context has

been beneficial, but it has resulted in challenges. The breadth of potential issues and the

uniqueness of each plantís design and associated risk profile are leading to a highly complex and

time-consuming process that is challenging public confidence. The experience with

implementation of the SDP during the first year has been that the final risk characterizations are

often untimely and the process is not always transparent to the licensees and external

stakeholders. 

Inherent to the SDP is the fact that regional risk analysts and inspectors discuss technical

information and risk analysis assumptions with the licenseeís technical staff. These discussions

are used to ensure the NRCís risk analysis is technically sound, but they have given the

perception to many stakeholders that the finding is being debated out of the public view. Several

public stakeholders have expressed concern that ìnegotiationsî occur between the NRC staff and

licensees during the risk characterization process. Public stakeholders have also observed that

the communication of the basis for the final risk significance determination is not clear in all

inspection reports, and does not always provide sufficient information for any interested party to

independently reconstruct the analysis. Some stakeholders have suggested that all of the

information used in the SDP, including licensee probabilistic risk assessments, should be

docketed. 

The time and resources committed to process individual potential non-Green issues have been

higher than expected, and many final determinations have not met the Agencyís timeliness goals. 

The guidance concerning Agency decisions emphasizes risk-based criteria as opposed to riskinformed. This could encourage protracted ìPRA battlesî that will hold up NRC actions and could
21

reduce the publicís confidence in the new processís effectiveness and efficiency. Stakeholders

have observed that excessive time is also spent resolving disagreements regarding Green

inspection findings, which appears contrary to being risk-informed and efficient. 

The PPEP noted that the staff needs to shorten the turnaround time for Phase 3 evaluations. 

The PPEP also recommended that the process for interactions between the NRC and licensee

during SDP evaluations be better defined, and that attention be focused on explaining the basis

for color assignments.

Panel recommendation: Use lessons learned from initial implementation to make the risk

characterization process expedient, scrutable, and understandable to all stakeholders. 

Issue S-8: Definition of a performance deficiency

Priority 2

Primary performance goals affected: Public confidence/maintain safety

Issue description: Early in the implementation of the ROP, the staff developed guidance in NRC

Inspection Manual Chapter 0609, ìSignificance Determination Process,î that required inspectors

to demonstrate a licensee performance deficiency before entering the SDP. This policy caused

concern among some inspectors because the NRC might appear to be overlooking risksignificant issues simply because it had not established a clear performance deficiency. In

addition, the staff had difficulty developing a licensee performance deficiency if the licensee could

not establish a root cause for an equipment failure. There is a potential to erode public

confidence by giving the appearance that the NRC is not taking consistent actions on risksignificant issues.

Panel recommendation: Use lessons learned from initial implementation to clarify the definition

of a performance deficiency.
22

Assessment and Enforcement

The assessment process associated with the ROP has effectively reduced the subjectivity of the

previous assessment process. The use of a predetermined Action Matrix, that places objective

performance indicators and inspection findings in a risk-informed context, has made the

determination and communication of NRC actions more streamlined and predictable. This has

improved public confidence by providing a clear road map for understanding Agency-licensee

interactions and regulatory decisions. Enforcement actions are now performance-based, with an

emphasis on placing the significance of the noncompliance into a safety context and ensuring

that the licensee is taking corrective actions. The response of the Agency is more timely under

the new assessment scheme than under the previous process. However, the panel identified the

following issues that the staff should address:

Issue A-1: Length of time inspection finding is included in Action Matrix

Priority 1

Primary performance goals affected: Unnecessary regulatory burden/understandable

Issue description: According to NRC Inspection Manual Chapter 0305, ìOperating Reactor

Assessment Program,î a non-Green inspection finding is normally carried forward in the

assessment program (i.e., Action Matrix) for a total of four calendar quarters. Performance

indicators are recalculated quarterly. Licensees have proposed that there be a graded approach

for how long findings remain active rather than the fixed 1 year. Considering the risk significance

of the various findings, it may be beneficial to establish a graded approach for resetting

inspection findings in the Action Matrix.

Panel recommendation: Evaluate a graded approach for resetting non-Green inspection

findings as entry conditions into the Action Matrix. 

Issue A-2: ìNo Colorî inspection findings

Priority 1

Primary performance goals affected: Understandable/public confidence

Issue description: The ROP has proceduralized the use of ìNo Colorî findings. The role of ìNo

Colorî findings, however, is not clear and has contributed to process inconsistencies and

confusion among many stakeholders. ìNo Colorî findings are associated with specific

extenuating circumstances listed in NRC Inspection Manual Chapter 0610*. These findings

typically address regulatory issues that are more than minor violations, but do not meet the

threshold for entry into the existing cornerstone significance determination process. These

issues do not receive a severity level or color and, therefore, the NRC does not characterize their

importance. Early in the process, the staff guidance was nonspecific and the result was the

appearance of a new finding classification. 

Licensees are concerned that these findings may be inappropriately used to artificially inflate the

significance of individual issues; likewise, they believe that many of the issues do not warrant

inclusion in inspection reports. External stakeholders have noted that the staff has established a

new undefined category of findings. Furthermore, ìNo Colorî findings are colored blue on the

NRCís Web page, and their role in the process is not understandable.
23

Panel recommendation: Evaluate and clarify the guidance on the designation, definition, and

use of what are presently called ìNo Colorî findings, and find a more appropriate term for these

findings.

Issue A-3: Purpose of the Regulatory Conference

Priority 2

Primary performance goals affected: Public confidence/effectiveness and

efficiency/understandable

Issue description: The purpose of a regulatory conference is to gain a complete understanding

of the risk significance of an inspection finding and to obtain information pertinent to

understanding any apparent violations. During initial implementation of the ROP, stakeholders

noted that the objectives of the regulatory conferences were not clear. The structure of the

regulatory conferences was very similar to that of the enforcement conferences in the previous

program. The regulatory conference discussions sometimes focused more on enforcement and

corrective actions rather than on the determination of the risk significance of the issue. 

Additionally, licensee and NRC managers have expressed discomfort with holding public

meetings concerning a potentially risk-significant issue and not including all potential decisionmakers. 

Panel recommendation: Clarify the guidance on the objectives and structure of regulatory

conferences and communicate this guidance to the external and internal stakeholders.

Issue A-4: Multiple related inspection findings

Priority 2

Primary performance goals affected: Unnecessary regulatory burden/effectiveness and

efficiency

Issue description: It is not clear how the NRC should disposition multiple inspection findings that

are related to the same technical problem or root cause. For example, would it be appropriate to

characterize an inspection finding involving five related issues (3 Green, 1 White, 1 Yellow) as

one finding or five separate findings? If they are handled as separate findings, the NRC

response, as determined by the Action Matrix, will be different from if they are considered one

finding. Experience has shown that significant events and conditions are often caused by

multiple performance failures.

Panel recommendation: Develop clear guidance on how to handle multiple related inspection

findings, and communicate the guidance to all stakeholders.
24

ROP Self-Assessment Program

Through briefings to the IIEP by the NRC staff and review of the preliminary self-assessment

metrics, the panel evaluated the ROP self-assessment program to determine whether the

developed program was sound and whether it included mechanisms for self-correction. The staff

developed the self-assessment program to determine whether the ROP is meeting its objectives

(including the Agencyís performance goals), to gather information about overall industry

performance, and to develop information to support possible improvements. The selfassessment program includes more than 75 metrics for measuring the success of the overall

ROP and how effectively it supports the four ROP elements (performance indicators, inspection

program, significance determination process, assessment program). The metrics are aligned to

the ROP objectives (risk-informed, predictable, understandable, objective) and the Agencyís

performance goals (maintain safety, increase public confidence, increase effectiveness and

efficiency, reduce unnecessary regulatory burden). The staff evaluates the metrics on a periodic

basis using information from various sources, including the inspection program, performance

indicators, periodic audits, stakeholder surveys, and public comments.

Based on the timing of the panelís activities and the parallel collection and assessment of selfassessment data, limited data and results were available for the panel to review. Although the

panel acknowledges the significant efforts of the staff in developing the self-assessment

program, it is best described as a noteworthy work in progress, that may require further

enhancements and refinements based upon the evaluation of the data collected. Although

individual panel members provided comments to the staff for consideration in further defining and

revising the individual metrics, the panel, as a whole, took no consensus position on the overall

adequacy and acceptability of the metrics. They are, even now, continuing to be refined by the

staff and the data for assessing their overall efficacy were not available. Nevertheless, the panel

concluded that the self-assessment program has the necessary elements to evaluate the ROP

against the Agencyís performance goals. The panel, however, could not evaluate the

effectiveness of the program given that the first yearís assessment data was not yet available. 

The panel was encouraged by the fact that, for the most part, the staff had identified the same

issues the panel identified.
IIEP CHARTER

UNITED STATES NUCLEAR REGULATORY COMMISSION CHARTER

 REACTOR OVERSIGHT PROCESS INITIAL IMPLEMENTATION EVALUATION PANEL

1. The Committeeís official designation:

 Reactor Oversight Process Initial Implementation Evaluation Panel (IIEP)

2. The Committees objectives and the scope of its activity:

The NRC has implemented a revised reactor oversight process (ROP) for commercial

nuclear power plant licensees. The ROP is described in NRC Inspection Manual Chapter

2515. Background information on the development of the ROP is contained in

Commission papers SECY-99-007, ìRecommendations For Reactor Oversight Process

Improvements,î dated January 8, 1999, SECY-99-007A, ìRecommendations For Reactor

Oversight Process Improvements (Follow-up to SECY-99-007),î dated March 22, 1999,

and SECY-00-049, ìResults of the Revised Reactor Oversight Process Pilot Program,î

dated February 24, 2000. These Commission papers describe the scope and content of

performance indicator reporting, a new risk-informed baseline inspection program, a new

assessment process, and revisions to the enforcement policy. Commission paper SECY00-049 also describes the results from the Pilot Program Evaluation Panel (a previous

Federal Advisory Committees Act (FACA) panel), including a recommendation from the

panel to proceed with initial implementation of the ROP at all power reactor facilities. On

March 28, 2000, the Commission approved initial implementation of the ROP, and on May

17, 2000, the Commission directed the NRC staff to convene another evaluation panel

under FACA to evaluate the first year of implementation of the ROP. The staff has

established this IIEP in response to the Commissionís directions.

The IIEP will function as a cross-disciplinary oversight group to independently monitor and

evaluate the results of the first year of initial implementation of the ROP and provide

advice and recommendations to the Director of the Office of Nuclear Reactor Regulation

on reforming and revising the ROP. IIEP meetings will be announced to the public in

advance and (unless closed according to the provisions of FACA) open to the public. 

Subject to the Freedom of Information Act, all material made available to or prepared for

or by the IIEP will be made available to the public. Meetings will be transcribed and

meeting summaries will be prepared following each meeting to document the results of

the meetings. The transcripts and meeting summaries will be publicly available.

The IIEP will evaluate the ROP results against performance measures. The IIEP will

provide a written report containing an overall evaluation of the ROP to the Director of the

Office of Nuclear Reactor Regulation. This report will include the consensus views of the

panel, or the majority and minority views when panel consensus cannot be achieved. The

NRC staff will use the IIEP evaluation as a major input to its deliberative process to

determine what modifications, if any, are needed to the ROP following initial

implementation.

3. The period of time necessary for the Committee to carry out its purpose:

Nine months

4. The NRC official to whom this Committee will report:
Committee Chairman

Loren Plisco

Director, Division of Reactor Projects

Region II

5. The NRC office and individual responsible for providing support for the Committee:

Office of Nuclear Reactor Regulation

Designated Federal Official

John Monninger

Technical Assistant, Associate Director for Inspection & Programs

Office of Nuclear Reactor Regulation

6. A description of the duties for which the Committee is responsible, and if such duties are

not solely advisory, a specification of the authority for such functions:

The panel will provide advice and recommendations only.

7. The estimated annual operating costs, in dollars* and staff years, for the Committee:

a. $120,000 (*Includes travel and per diem)

b. 0.50 FTE

8. The estimated number and frequency of the Committee meetings:

Approximately 3 meetings will be held, on an as needed basis, during the period the panel

is in existence.

9. Organization - The IIEP will create any subcommittees which may be necessary to fulfill

the IIEPís mission. In addition, NRC and IIEP will establish such operating procedures as

are required to support the group, consistent with the Federal Advisory Committee Act, as

amended.

10. The Committeeís termination date, if less than two years from the date of establishment:

 July 31, 2001

11. Charter Filing Date:

October 17, 2000

/RA/

___________________________

Andrew L. Bates

Advisory Committee Management Officer

Office of the Secretary of the Commission
Attachment 2

IIEP MEMBERS

In selecting members for the Initial Implementation Evaluation Panel, the NRC considered

interested persons and groups with professional, technical, or personal qualifications or

experience that could contribute to the functions and tasks of the panel. The NRC considered

several factors in appointing IIEP members, including (1) the requirements in 10 CFR Part 7,

ìAdvisory Committees,î directing balance in advisory committee membership in terms of the

points of views represented and the functions to be performed, (2) Commission direction in the

staff requirements memorandum for SECY-00-0049 for a panel with a cross-section of

stakeholders similar to those who participated in the original evaluation panel, and with the

addition of at least one resident inspector and one senior reactor analyst, (3) a desire for

independence from the NRC office responsible for development and oversight of the ROP (Office

of Nuclear Reactor Regulation) and a focus on those stakeholders most affected by the ROP,

and (4) a desire to provide both continuity and new perspectives in terms of the individual panel

members selected. 

Panel Members

Loren Plisco, NRC, Region II (Chairman)

Randolph Blough, NRC, Region I

Kenneth Brockman, NRC, Region IV

Richard Borchardt, NRC, Office of Enforcement*

Mary Ferdig, Benedictine University; Ferdig, Inc.**

Steve Floyd, Nuclear Energy Institute 

David Garchow, PSEG Nuclear LLC

Richard Hill, Southern Nuclear Operating Company

Rod Krich, Exelon Corporation

Robert Laurie, California Energy Commission

David Lochbaum, Union of Concerned Scientists**

James Moorman, NRC, Region IV (Senior Resident Inspector)

Steven Reynolds, NRC, Region III 

Edward Scherer, Southern California Edison

James Setser, Georgia Department of Natural Resources

Raymond Shadis, New England Coalition on Nuclear Pollution**

James Trapp, NRC, Region I (Senior Reactor Analyst)

Support to the Panel

Facilitator: Francis X. Cameron - NRC, Office of the General Counsel

Designated Federal Official: John D. Monninger - NRC, Office of Nuclear Reactor Regulation

* Richard Borchardt was originally appointed to the panel and participated in all panel activities

through the fourth meeting of February 26-27, 2001. Subsequently, he was appointed to the

position of NRR Associate Director for Inspection and Programs and assumed those duties as of

March 19, 2001. In recognition of the desire for independence in panel membership from NRR,

Mr. Borchardt decided to recuse himself from panel activities effective April 2, 2001.

** David Lochbaum was originally appointed to the panel and participated in the first panel

meeting on November 1-2, 2000. He resigned from the panel on November 6, 2000. On

December 4, 2000, Mary Ferdig and Ray Shadis were appointed to the panel.
Attachment 3

BIBLIOGRAPHY OF PANEL ACTIVITIES

1. Results of the Revised Reactor Oversight Process Pilot Program, SECY-00-049, February

24, 2000, ADAMS ML003683227

2. Staff Requirements Memorandum on SECY-00-0049, ìResults of the Revised Reactor

Oversight Process Pilot Program (Part 2),î May 17, 2000, ADAMS ML003715823

3. IIEP Charter, October 17, 2000, ADAMS ML003760300

4. Letter to General Services Administration, October 17, 2000, ADAMS ML003760307

5. Letter to Library of Congress, October 17, 2000, ADAMS ML003760327

6. Letters to Congressional Oversight Committees, October 17, 2000, ADAMS

ML003763800

7. First IIEP Meeting Summary and Transcript, November 1-2, 2000, ADAMS ML003774521

8. Second IIEP Meeting Summary and Transcript, December 11-12, 2000, ADAMS

ML010090359 & ML010530300

9. Third IIEP Meeting Summary and Transcript, January 22-23, 2001, ADAMS

ML010530104

10. Fourth IIEP Meeting Summary and Transcript, February 26-27, 2001, ADAMS

ML010880350

11. Fifth IIEP Meeting Summary and Transcript, April 2-3, 2001, ADAMS ML011140513

12. Sixth IIEP Meeting Summary and Transcript, April 25, 2001, ADAMS ML011280333

13. Final IIEP Report, May 10, 2001, ADAMS ML011290025
Attachment 4

SOURCES OF PANEL INFORMATION

December 11-12, 2000

Bill Sherman, Vermont Department of Public Service

Gary Wright, Illinois Department of Nuclear Safety

January 22-23, 2001

Joseph Brady, NRC Senior Resident Inspector

Sonia Burgess, NRC Senior Reactor Analyst

Stephen Campbell, NRC Senior Resident Inspector

Jeffrey Clark, NRC Senior Resident Inspector

Steven Jones, NRC Senior Resident Inspector

William Jones, NRC Senior Reactor Analyst

Jill Lipoti, New Jersey Department of Environmental Protection

James Trapp, NRC Senior Reactor Analyst

Dennis Zannoni, New Jersey Department of Environmental Protection

February 26-27, 2001

Victor Dricks, NRC Office of Public Affairs

Steve Floyd, Nuclear Energy Institute

Rich Janati, Pennsylvania Department of Environmental Protection

Judith Johnsrud, ECNP, Sierra Club

Steve Kerekes, Nuclear Energy Institute

David Lochbaum, Union of Concerned Scientists

Jenny Weil, McGraw Hillís Inside NRC
STAFF RESPONSE TO THE IIEP FINAL REPORT

Attachment 5
1

Staff Response to the IIEP Final Report

Introduction

In its final report, included as Attachment 4 to this paper, the Initial Implementation Evaluation

Panel (IIEP) presented its evaluation of the results of the first year of implementation of the

reactor oversight process (ROP). The overall panel conclusion was that the ROP is a notable

improvement over the previous oversight process, and it should be continued. The ROP has

made progress toward achieving the Agencyís four performance goals. The panel

recommended, however, that the staff take certain actions to ensure they achieve the Agency

performance goals in the long-term, and consider other actions to improve the process.

Most of the IIEP conclusions and recommended actions were consistent with the staffís

evaluation of the results of ROP initial implementation. These panel recommendations were

factored into the staff evaluation of the ROP, and are addressed throughout this paper. The IIEP

recommendations, and a summary of staff actions, are discussed below. The reference number

following each recommendation refers to the issue number in the IIEP report.

ï Establish a formal program and assign sufficient resources to enhance

communications necessary for improving the ROP. The program should

accumulate lessons learned, provide multiple and diverse opportunities for

comment to all internal and external stakeholders, respond to stakeholdersí

comments, and have a process for making timely process changes. (O-1)

Inspection Manual Chapter (IMC) 0307, ìReactor Oversight Process Self-Assessment

Program,î describes the formal program developed by the staff to continue open

communication with all stakeholders to enhance the ROP. The ROP self-assessment

program, and additional background on its development during the first year of ROP

implementation, is also described in Attachment 3 to this paper.

As described in IMC 0307, on a periodic basis the self-assessment program collects

information from various sources, including the Reactor Program System (RPS), the

inspection program, the performance indicator (PI) program, additional industry level PIs,

periodic independent audits, internal and external stakeholder surveys, and public

comment. Based on this information, an assessment of ROP success in the

programmatic areas of PIs, inspection program, significance determination process, and

assessment will be performed. In addition, an assessment of overall ROP efficacy will be

made and recommendations for program improvement will be developed.

The staff will continue to evaluate ways to improve the ROP self-assessment program,

including how best to respond to stakeholder comment and feedback.

ï Revise the ROP communication plan to include outreach activities designed to

inform the public about the process and its relationship to the Agencyís mission of

protecting the public health and safety. Appropriate resources should be provided

to revise and implement the communication plan. Evaluate additional

improvements to the information on the ROP Web page to improve and simplify
2

public access to the information. Identify methods, using stakeholder input, to

improve public outreach efforts. (O-2)

Throughout initial implementation, the staff made a concerted effort to continue and

improve on the initiatives implemented during the ROP pilot program to keep the public 

informed and to provide frequent and ongoing opportunities to submit comment and

feedback. The staff developed a Communication Plan that outlined the activities designed

to enhance the understanding of the ROP among all external stakeholders. The

communication tools used by the staff included the external Web page, a Federal Register

notice, and numerous forums during which public, industry, and governmental

stakeholders were participants. 

As described in Attachment 2, the staff is working to update the Communication Plan to

reflect ongoing activities related to the ROP and will continue to look for ways to improve

public outreach based on lessons learned from the first year of ROP implementation. 

ï Establish a structured ongoing program to evaluate long-term ROP effectiveness

and to test ROP assumptions. As a minimum, this includes integrating the insights

of the ROP self-assessment program and the overall assessment of industry

performance. The staff should also consider periodically engaging internal and

external stakeholders to independently assess the ROP. (O-3)

As described in Attachment 3, the staff has developed a periodic self-assessment process

to evaluate the effectiveness of the ROP on an ongoing basis and to test program

assumptions. In addition to the metrics developed for each major component of the ROP,

metrics of a more general nature have been developed utilizing stakeholder feedback to

gauge overall performance. Industry level performance metrics are also being developed

and will be part of the self-assessment program. Both the self-assessment metrics and

the industry level performance metrics evaluate the efficacy of the fundamental premises

of the ROP that form the basis of the program. Although not planned for the upcomng

year, the staff will consider engaging internal and external stakeholders in the future to

independently assess the ROP.

ï Continue the efforts of the cross-cutting issues task force and clarify the ROP

guidance on the identification and disposition of cross-cutting issues. (O-4)

The cross-cutting issues task force is continuing its effort to address many of the issues

discussed in Appendix A of this paper pertaining to the treatment and role of cross-cutting

issues in the ROP. Several additional planned actions regarding the evaluation of the role

of cross-cutting issues in the ROP are also discussed in Appendix A. In particular, during

periodic reviews of issued inspection reports, the staff will evaluate whether cross-cutting

issues are being adequately identified and documented during inspection activities. As

part of the self-assessment metrics developed for assessing the ROP, the staff will review

the circumstances surrounding plants that jump two or more columns in the Action Matrix

to see if these performance weaknesses were due to cross-cutting issues, and if so,

whether inspections or PIs have identified similar concerns. And during annual ROP selfassessment, the staff will evaluate whether the ROP allowed for sufficient NRC
3

engagement at facilities that reached the degraded cornerstone column of the Action

Matrix.

ï Evaluate lessons learned from initial implementation to achieve parity in the

treatment of risk-significant inspection findings and crossed performance indicator

thresholds. The evaluation should verify that the outcomes from the performance

indicators and inspection findings accurately reflect the significance of the issues. 

The staff should consider addressing this issue by adjusting the Green/White

thresholds or modifying the Action Matrix. (O-5)

Lessons learned from implementing the ROP indicated that risk-significant inspection

findings and crossed PI thresholds had parity in the oversight process and received an

appropriate Agency response as determined by the assessment process Action Matrix. 

The staff will continue to evaluate this issue as part of the ongoing ROP self-assessment,

and will adjust Green/White thresholds or the columns of the Action Matrix as appropriate.

ï Ensure that the staffís ROP self-assessment program identifies and evaluates any

unintended consequences or unnecessary regulatory burden caused by the

performance indicators and that changes are made where appropriate. (P-1)

The staff has established specific metrics and success criteria in the ROP selfassessment program to periodically assess the PI program to identify and evaluate any

instances of unintended consequences or unnecessary regulatory burden. The selfassessment program is discussed in Attachment 3 of this paper, and described more fully

in IMC 0307.

ï Expedite the efforts to resolve the concerns about the safety system unavailability

performance indicators and implement any needed revisions to NEI 99-02,

ìRegulatory Assessment Performance Indicator Guideline,î to achieve consistency

with other applications. (P-2)

As described in Attachment 6, an NRC/Industry working group has been created to take a

clean-slate approach to the SSU indicators to produce a set that is simpler, easier to

understand, compatible with all intended uses, and more risk-informed than the current

indicators. This effort will start by defining the objective of the indicators and will construct

a new indicator from the ground up, including reevaluation of the changes already made

to accommodate several of the issues associated with the unavailability PIs. 

ï Continue the efforts to identify and evaluate improvements to performance

indicators. The staff should thoroughly evaluate any significant changes to the

performance indicators, following the structured process in Inspection Manual

Chapter 0608, ìPerformance Indicator Program.î Further, the staff should evaluate

the effect of any significant changes in the scope of information provided by

performance indicators on the Action Matrix and the baseline inspection program,

and should identify any additional costs and benefits. (P-3)

As described in Attachment 6 to this paper, the staff is continuing to pursue several efforts

to evaluate improvements to the PIs, such as ongoing efforts by the NRC/Industry working
4

group to develop improvements to the unavailability PIs. Any proposed changes to the PI

program will be handled in accordance with the requirements of IMC 0608, which the staff

issued on April 12, 2001. As described in IMC 0608 any new or revised PIs would be

evaluated for their impact on the other parts of the ROP, including the affect on the

baseline inspection program and evaluating the additional costs and benefits associated

with the change.

ï Continue efforts to incorporate the answers to frequently asked questions into the

performance indicator guidance document, NEI 99-02, and to make the answers

more generic where possible. (P-4)

Revision 1 to NEI 99-02 was issued on April 23, 2001, and incorporated numerous

frequently asked questions (FAQs) that were generated during the first year of ROP

implementation. As described in IMC 0608, NEI 99-02 will continue to be updated

periodically, as appropriate, to clarify the PI reporting guidance based on insights from the

ongoing resolution of FAQs.

ï Evaluate the inspection approach for the physical protection cornerstone and

revise the inspection program as necessary after the pending safeguards

performance assessment pilot program and the physical security rulemaking are

completed. (I-1)

As described in Attachment 7, the Physical Protection cornerstone inspection procedure

(IP 71130) and its attachments will be revised over the next year to account for the

significant changes and new policies in physical security. The feedback from external

stakeholders on the 10 CFR Parts 26 and 73 rulemaking and the OSRE program, along

with internal inspector feedback on the baseline inspection program, will form the basis for

drafting the revisions to the inspection procedure. The staff also will write a new

attachment to the inspection procedure to inspect the industryís self-assessment program.

ï Evaluate and revise guidance to inspectors as necessary to clarify and promote

consistency in documenting inspections. Continue conducting periodic audits of

inspection reports to identify and correct inconsistencies. The new documentation

thresholds for issues that have a defined level of safety or regulatory significance

are appropriate. Inspection observations and insights that do not reach the

threshold should continue to be communicated verbally to licensees for their

consideration. (I-2)

Since January 2001, the staff has been auditing inspection reports issued by the regions

to determine whether the regions are documenting inspection results in accordance with

the requirements of IMC 0610*, ìPower Reactor Inspection Reports.î The focus of these

audits has been on determining whether the appropriate threshold was applied to

document a finding, evaluating whether the safety significance of findings is clearly

documented, and whether ìNo Colorî findings are being properly used and placed in the

proper context. These inspection report audits are performed as part of the periodic ROP

self-assessment discussed earlier, and will continue beyond initial implementation. Based

on these audit results, the staff will evaluate and revise the guidance for documenting

inspection findings as necessary to ensure consistency.
5

General inspection policies are described in IMC 2515, ìLight-Water Reactor Inspection

Program - Operations Phase.î As discussed in IMC 2515, communicating inspection

observations is an integral and important part of every inspection, whether done daily

during the course of an inspection, or periodically with status meetings. Observations or

insights that do not reach the threshold for describing in an inspection report as defined by

IMC 0610* should be conveyed to the licensee for its consideration, but are not to be used

in the NRCís reactor assessment program.

ï Evaluate inspection findings and performance indicator results for the first yearís

implementation and determine the appropriate level of effort to adequately assess

risk-significant areas in the baseline inspections. Modify the process as

appropriate. (I-3)

Based on lessons learned from the first year of implementing the ROP, a number of

baseline inspection program procedures are being changed to adjust the inspection

scope, frequency, or level of effort as appropriate. The inspection procedures being

revised include inservice inspection, 10 CFR 50.59 review, maintenance effectiveness,

ALARA, and problem identification & resolution. Additional detail on the procedures being

revised, and the basis for the revision, is provided in Attachment 7 to this paper. 

ï After further experience with the ROP, review the results and consider whether to

waive certain parts of the baseline team inspections and let licensees assess

themselves under defined circumstances. (I-4)

As described in Attachment 7, the staff will continue to evaluate how licensee selfassessments could be used in satisfying some requirements of the baseline inspection

program.

ï Validate and issue the revised reactor safety significance determination process

Phase 2 worksheets. (S-1)

The development and issuance of the reactor safety SDP plant/site specific Risk-Informed

Inspection Notebooks, including the Phase 2 worksheets, is currently ongoing. All of the

70 notebooks are scheduled to be completed and issued for use by the end of July 2001. 

Following issuance, the last important step remaining is a benchmarking process that

compares the results of the SDP Phase 2 notebooks with the licensee risk model results

to ensure that the SDP is generally conservative. The benchmarking process continues at

a rate of two site visits per month. Additional detail regarding the issuance of the Phase 2

worksheets can be found in Attachment 8.

ï Continue efforts to obtain improved and standardized risk analysis tools for the

reactor analysts. (S-2)

The improvement and standardization of the ROP risk analysis tools is an ongoing major

effort associated with all of the SDPs. For several of the SDPs (e.g., fire protection,

safeguards, ALARA) the improvement effort is mostly being performed by the NRC staff,

while other SDP improvements such as reactor safety and shutdown risk involve both the
6

staff and the experts of a national laboratory. Details of the ongoing and planned

improvements can be found in Attachment 8.

ï Review lessons learned from use of the fire protection significance determination

process, and improve the risk characterization tool to make it more meaningful,

effective, and efficient. (S-3)

Based on initial implementation lessons learned, the staff revised the fire protection SDP

to incorporate additional information to enhance the inspectorsí ability to assess

functionality of fire protection defense in depth components such as fire brigade,

automatic detection, and protection performance. The staff continues to develop the

methodologies that will allow inspectors to determine fire scenario development and

improve the accuracy of site specific data used in the assessment of risk associated with

findings, such as ignition frequency. The results of these efforts will undergo functional

evaluation and are expected to be made available to the inspectors by the end of 2001. 

Additional information regarding lessons learned from implementing the fire protection

SDP can be found in Attachment 8 of this paper.

ï Continue development of an improved physical protection significance

determination process. (S-4)

The staff determined that the Physical Protection SDP (PPSDP), with its link to the reactor

safety SDP, can over-estimate the risk significance of an issue, and therefore lead to a

higher level of NRC response and engagement than is warranted. The problem is most

noticeable when processing a deficiency through the PPSDP that is identified by a forceon-force exercise. To remedy the problem the staff determined that the PPSDP has to be

revised. However, while a longer term effort to revise the PPSDP is underway, the staff,

with the approval of the Commission, issued an interim SDP. Details of the interim

guidance were provided to the Commission in COMSECY-00-0036, which addresses

significance determinations for findings developed during multiple scenarios in force-onforce exercises. 

ï Evaluate the need for other significance determination tools. The staff should

carefully evaluate any potential changes against the Agencyís goals. For example,

a new significance determination process that increases regulatory burden should

have a corresponding benefit in maintaining safety or increasing the Agencyís

effectiveness. (S-5)

Currently, all SDP related improvement efforts are designed to streamline and simplify the

process, thereby increasing inspector efficiency and reducing regulatory burden. For

example, a proposal to create a Maintenance Rule SDP is under review and consideration

will be given to assure that the appropriate balance between maintaining safety and any

increase in regulatory burden is achieved.

ï Evaluate lessons learned from initial implementation and revise the ALARA

significance determination process as necessary. (S-6)
7

As described in Attachment 8 of this paper, the staff is taking several actions to revise the

ALARA SDP based on ROP initial implementation lessons learned. The staff will revise

the Radiation Exposure Control section of the Occupational Radiation Safety SDP to

clarify how the SDP reflects the Commissionís policy on enforcement discretion for skin

over-exposures from hot particles (or discrete radioactive particles). Additionally changes

are being considered to the screening question for the ALARA SDP and to use the 3-year

average collective dose comparison to adjust the baseline inspection effort. 

ï Use lessons learned from initial implementation to make the risk characterization

process expedient, scrutable, and understandable to all stakeholders. (S-7)

The staff continues to evaluate the causes for not meeting the SDP guidelines for

expediency (timeliness) during initial implementation. Enhanced SDP tools and

continued experience with the process should provide some improvements in this area. 

Ongoing and longer term improvements, including improving the scrutability of

assumptions, should make the process more understandable in general to all

stakeholders. Additional detail regarding these process improvements can be found in

Attachment 8.

ï Use lessons learned during initial implementation to clarify the definition of a

performance deficiency. (S-8)

ROP initial implementation demonstrated that inspectors were generally able to identify

licensee performance deficiencies associated with appropriately characterized issues. 

However, a specific definition of licensee performance deficiency has not been fully

developed by the staff. Such a definition is under discussion, with proposals expected to

be presented to regional management and inspectors at future inspector counterpart

meetings. 

ï Evaluate a graded approach for resetting non-Green inspection findings as entry

conditions into the Action Matrix. (A-1)

As described in Attachment 9, the staff discussed this issue with both internal and external

stakeholders at the External Lessons Learned Workshop held in March 2001. The staff

will take the feedback received at the workshop and continue to evaluate the ROP to

determine whether a graded reset approach for inspection findings is appropriate.

ï Evaluate and clarify the guidance on the designation, definition, and use of what

are presently called ìNo Colorî findings, and find a more appropriate term for these

findings. (A-2)

The staff also discussed this issue with both internal and external stakeholders at the

External Lessons Learned Workshop. As described in Attachment 9, the workshop

participants could not reach consensus on a appropriate means to deal with ìNo Colorî

findings. As an interim measure, the group recommended that the Agency: (1) address

the perception problems with ìNo Colorî findings, (2) continue to monitor ìNo Colorî

findings and adjust program office guidance to minimize the number of these findings, and
8

(3) evaluate these findings to determine if they represent a weakness in the ROP. Over

the longer term, the staff will also evaluate the role of ìNo Colorî findings in the ROP. 

ï Clarify the guidance on the objectives and structure of regulatory conferences and

communicate this guidance to the external and internal stakeholders. (A-3)

As described in Attachment 9, the staff has developed several actions to clarify the

objective and structure of regulatory conferences. These actions include conducting the

meetings in such a manner as to ensure that the safety significance is discussed first and

having someone besides the regional enforcement coordinator open the meetings.

ï Develop clear guidance on how to handle multiple related inspection findings, and

communicate the guidance to all stakeholders. (A-4)

As described in Attachment 8, the staff is working to improve the guidance provided for

the assessment of concurrent deficiencies. The existing guidance in IMC 0609,

ìSignificance Determination Process,î provides for concurrent performance deficiencies to

be assessed collectively to determine total contribution to change in the core damage

frequency (CDF). However, each concurrent performance deficiency should be assigned

a color individually. When multiple issues stem from the same common cause, risk

analysis techniques account for the potentially greater risk significance of the combined

issues (i.e., delta CDF or delta large early release frequency (LERF)). However, in all

cases the assessment process Action Matrix was designed to combine multiple issues for

determining the appropriate NRC response. Although it is expected that in most cases

the regulatory response will not differ if multiple issues are treated either in combination or

independently, ROP guidance is being evaluated for enhancement with regard to applying

combined risk results of multiple issues in a manner that is appropriate for use in the ROP

Action Matrix.
PERFORMANCE INDICATORS

Task Lead: Don Hickman, NRR/IIPB

Attachment 6
1

Performance Indicators

Open Issues Addressed

There were several open issues and commitments that the staff continued to evaluate and work

on during ROP initial implementation. Several of the issues involved longer term items that the

staff identified in Commission paper SECY-00-0049 as needing additional work following initial

implementation. The Commission directed the staff to consider several additional issues in the

SECY-00-0049 Staff Requirements Memorandum (SRM) dated May 17, 2000. And several

recommendations for staff consideration during initial implementation were made by the Pilot

Program Evaluation Panel (PPEP) in their final report. The issues and recommendations for the

inspection program are as follows: 

SECY-00-0049

Consistency of Performance Indicator Definitions: There are a number of programs that

collect similar performance data, including the Reactor Oversight Process (ROP)

Performance Indicator (PI) program, Probabilistic Risk Assessment (PRA) programs, the

Maintenance Rule, and the Institute of Nuclear Power Operations (INPO)/World

Association of Nuclear Operators (WANO) performance indicator program. The NRC staff

and the industry agree that, to the extent practical, there should be common definitions of

performance indicators for all uses. More importantly, there should be one data set

reported by licensees that can meet the needs of all of the above programs. The NRC

has developed common definitions wherever possible. The staff has also communicated

with INPO/WANO and the industry in an effort to move toward common definitions. 

Because the ROP PI program is an evolving one (toward plant-specific, more riskinformed indicators), there may be some inconsistencies that can only be worked out

when all elements are in place. In the meantime, the staff continues to work with industry

to reduce the differences. There is currently a significant effort underway with industry

(including INPO and WANO) to develop a common definition and a common data set for

the Safety System Unavailability indicators, which are the ones that have produced the

most questions and concerns during initial implementation.

Process for Changing, Adding, or Deleting Indicators: During initial implementation, some

changes to PIs became necessary to provide consistency among plants and with other

regulatory requirements. Inspection Manual Chapter (IMC) 0608, ìPerformance Indicator

Program,î was developed to provide a deliberate, formal process for PI program changes

to ensure that they are carefully thought through and achieve the desired objective. The

IMC 0608 process was used during initial implementation to address questions from

stakeholders and to develop and evaluate potential replacement indicators (see Manual

Scram Issue below).

Guidance on How to Handle a Plant in an Extended Non-Regulatory Shutdown: When a

plant has been shut down for an extended period, some PIs no longer provide useful

information about that plantís performance (e.g., Unplanned Scrams per 7,000 Critical

Hours, Unplanned Power Changes per 7,000 Critical Hours). IMC 0608 defines an

extended shutdown and references Inspection Procedure 71150, ìDiscrepant or

Unreported Performance Indicator Data,î for guidance on how to monitor plant
2

performance through inspection during extended shutdowns when some performance

indicator data is not meaningful. 

Guidance and Thresholds for RCS Activity and Leakage: These two indicators are

intended to monitor the integrity of the fuel cladding and the primary coolant system, the

first two barriers to the release of radioactive fission products. But because of differences

in the method and frequency of licensee measurements of these parameters, as well as

licensee actions to control them, they do not necessarily indicate the true condition of the

barriers. In accordance with the PI change process, the staff plans to work with external

stakeholders to improve the RCS Activity indicator, including improving calculational

methods and thresholds, to more accurately reflect fuel cladding barrier performance. 

Guidance on Unplanned Power Changes per 7,000 Critical Hours: For this PI, an

unplanned power change is one that was initiated less than 72 hours following the

discovery of an off-normal condition. The staff is concerned about unintended

consequences resulting from this 72 hour requirement. There have been instances during

initial implementation when a licensee has deferred corrective maintenance for more than

72 hours to avoid counts in this indicator. Accordingly, the NRC and industry have agreed

to pilot an NRC proposed replacement PI entitled ìUnit Power Reductions per 7,000

Critical Hours,î that counts most power reductions in average daily power level of greater

than 20 percent. This eliminates the distinction between ìplannedî and ìunplanned,î

which the staff has not considered to be an important aspect of the indicator. The pilot will

also include an alternate PI proposed by industry that would count power reductions due

to equipment problems or operator errors. The NRC and industry plan to begin the pilot of

the possible replacement indicators in the near future.

Guidance on the Relationship of Fault Exposure Hours to Operability and Reportability:

Fault exposure hours (FEH) are used in the Safety System Unavailability (SSU)

performance indicators in two ways: to indicate the hours a train was unavailable due to

an undetected, failed condition (an appropriate use in an unavailability indicator), and to

provide an indication of the reliability of the monitored trains to start and run (as a

surrogate for a reliability because there is none in the SSU). The SSU indicators have

generated more Frequently Asked Questions (FAQs) and concerns than any other, many

of them related to fault exposure hours. Responding to these questions on specific

events or conditions has taken considerable time on the part of both the staff and the

industry. Many issues, including site-specific concerns, have been resolved using the

FAQ process. While incorporating reliability indicators could provide better indication of

equipment performance and would reduce the need for FEH, it would not totally eliminate

them from the unavailability indicators. The staff has, in cooperation with the industry,

begun a significant effort to revise the SSU indicators to improve their ability to identify

risk-important events or conditions.

Protected Area Security Equipment Performance Index Definition: Some stakeholders

have questioned the use of security guard compensatory hours as a measure of security

equipment performance rather than the unavailable hours of the equipment. An

NRC/Industry working group reexamined this issue and concluded that compensatory

hours are easier to collect, adequate for the intended purpose of the PI, and have proven

successful during initial implementation. Therefore there is no need to change the PI.
3

Guidance on Security-Related Reportable Events: The Physical Protection Cornerstone

indicators for Personnel Screening Program Performance and Fitness-for-Duty Program

Performance have been considered to be ineffective at identifying declining performance

prior to major programmatic problems. Due to resource limitations, the NRC/Industry

working group was not able to undertake an effort to develop alternate indicators for

security-related reportable events during the past year.

Guidance for Loss of Key Control/Surveys: The NRC/Industry working group addressed

the issue regarding the need for guidance related to loss of key control/surveys. This

guidance was incorporated into the Nuclear Energy Institute (NEI) guidance document,

NEI 99-02, Revision 1, ìRegulatory Assessment Performance Indicator Guideline.î

Guidance on Alert and Notification System (ANS) Reliability: Public stakeholders,

licensees and inspectors have expressed a concern that the ANS Reliability PI does not

include known periods of unavailability. The national standard for evaluating ANS

performance is FEMA-REP-10, 1985, ìGuide for the Evaluation of Alert and Notification

Systems for Nuclear Power Plants,î which has been used by many licensees for 15 years. 

Since FEMA-REP-10 uses reliability as the measure for evaluating ANS performance, the

staff considered it to be appropriate to use the same measure in the ANS PI, especially

since there has been no indication that the guidance of FEMA-REP-10 was inadequate. 

Nevertheless, the NRC did consider using an alternate performance indicator that would

count unavailable hours. This effort was abandoned as unfeasible due to the variety of

systems, test methods, and procedures throughout the industry that would greatly

complicate the PI and produce results that are not comparable from plant to plant. 

However, the lack of an unavailability PI does not mean that there is important information

not captured by the ROP. Baseline Inspection Procedure 71114.02, ìAlert and

Notification System Testing,î would be expected to identify repetitive or significant ANS

outages and to assess the associated corrective actions for adequacy.

Impact of Multi-Unit Sites or Multi-Input Indicators on Site-Wide Indicators: The staff

committed to monitor site-wide indicators in light of the concern expressed by some that

good performance at one unit of a multi-unit site, or in one part of a multi-input PI, could

mask poor performance at the other unit or in the other part of the PI. The NRC has

reviewed the use of site-wide indicators for multi-unit sites in the Emergency

Preparedness, Occupational Radiation Safety, and Physical Protection cornerstones, and

multi-input indicators in the latter two cornerstones, for possible masking of performance

issues, with the following results:

Emergency Preparedness Cornerstone - The concern is that good performance in

the Drill/Exercise Performance PI at one unit (or crew), or in one part of the PI

(e.g., protective action recommendation (PAR) development) could mask poor

performance at the other unit (or another crew), or in another part of the PI (e.g.,

classification or notification). The staff has reviewed the EP data from initial

implementation and not found any evidence that masking occurred, although the

potential exists. However, the ROP is more likely to reveal a masking problem

than previously, since, in the past, only one crewís performance was observed

every two years, and the same team could perform in sequential exercises. Also,

Baseline Inspection Procedure 71114.05 ìCorrection of Emergency Preparedness

Weaknesses and Deficiencies,î addresses the potential for masking. The
4

procedure directs inspectors to review the licenseeís identification and correction

of drill performance problems with emphasis on classification, notification and PAR

development. Masking of significant problems should be revealed through review

of critique results and subsequent corrective actions. The staff believes that the

ROP adequately addresses the potential for masking poor performance.

Occupational Radiation Safety Cornerstone - The radiation protection program at a

site is a site-wide program. Every unit at a multi-unit site is assessed a PI count

for each occurrence captured by the PI, regardless of the unit in which it occurred,

and the site value is same as the unit value. Therefore "good" performance in one

Unit cannot mask "poor" performance in another. Also, because the Green/White

threshold for this PI is low (>2 occurrences), "good" performance in one area (i.e.,

exposure control) is unlikely to mask "poor" performance in another area (i.e.,

Technical Specification high radiation area compliance).

Physical Protection Cornerstone - The physical protection program at a site is a

site-wide program. Every unit at a multi-unit site is assessed the same PI values,

and the site value is same as the unit value. Therefore "good" performance in one

unit cannot mask "poor" performance in another. The Protected Area Security

Equipment Performance Index is a multi-input PI, and good performance of the

closed circuit televison cameras can, and has, masked poor performance of the

intrusion detection system. The staff has proposed a resolution to this problem

and is working with industry to incorporate a change to the PI.

SECY-00-0049 SRM

Manual Scrams: In a letter to Chairman Meserve dated May 19, 2000, Mr. Joe F. Colvin of

NEI, Mr. James T. Rhodes of INPO, and Mr. Zack T. Pate of WANO expressed their

concern about the possibility of unintended consequences that may result from the use of

an indicator that counts manual reactor scrams. In the SECY-00-0049 SRM, the

Commission noted that the staff has commenced a dialogue with industry on the issue

and directed the staff to expeditiously report to the Commission potential resolutions to the

concern. Accordingly, the staff has worked with industry to develop replacements for both

the ìUnplanned Scrams per 7,000 Critical Hoursî and the ìScrams With Loss of Normal

Heat Removalî performance indicators. Those replacements are intended to collect the

same information as the current indicators in a way that minimizes the potential that an

operator could be influenced to not scram the reactor to avoid a count against a PI. The

indicators are entitled ìUnplanned Reactor Shutdowns per 7,000 Critical Hoursî and

ìUnplanned Reactor Shutdowns With Loss of Normal Heat Removal.î An unplanned

reactor shutdown is defined as follows:

... the shutdown of the reactor in response to off-normal conditions

or events by the unplanned addition of negative reactivity by any

means (e.g., insertion of control rods, boron, or opening reactor trip

breakers). Unplanned reactor shutdowns are those that bring the

reactor from criticality to a shutdown mode within 15 minutes of

commencing to insert negative reactivity.
5

In accordance with the formal PI change process, the staff and industry conducted a pilot

for these two indicators that commenced in October 2000 and was completed with the

data submission for March 2001 that was received by the NRC on April 23, 2001. The

results are currently being evaluated against pre-determined criteria, which comprise the

following: (1) differences between the data collected for the Unplanned Reactor

Shutdowns per 7,000 Critical Hours PI and those collected for the Unplanned Scrams per

7,000 Critical Hours PI; (2) comparability of the data reported for the Unplanned Reactor

Shutdowns With Loss of Normal Heat Removal PI and both the Unplanned Scrams With

Loss of Normal Heat Removal PI and the appropriate sections of NUREG/CR-5750,

ìRates of Initiating Events at U.S. Nuclear Power Plants, 1987 - 1995;î (3) the ability of

licensees to report the requested data accurately and with minimal need for clarification;

(4) the ability of each alternate PI to reduce the potential for unintended consequences

without introducing other unintended consequences; and (5) whether there are minimal

changes in reporting burden for licensees. In addition, the NRCís databases at the Idaho

National Engineering and Environmental Laboratory and the Oak Ridge National

Laboratory are being utilized to identify previously reported events that would not be

captured by the proposed replacement indicators. Upon completion of the evaluation, the

staff will make a determination regarding implementation of the replacement indicators.

PPEP Final Report

Early Performance Indicator Verification Inspections: The Pilot Program Evaluation Panel

recommended that verification inspections of the performance indicators be conducted

early in initial implementation for the non-pilot plants. Accordingly, Temporary Instruction

2515/144, Revision 1, was issued early in initial implementation for resident inspectors at

each non-pilot plant to review the licenseeís PI data collection and reporting process to

determine whether the licensees are appropriately implementing the NRC/Industry

guidance.

Resolve the Issues Regarding the Appropriateness and Thresholds of Several Indicators: 

At the end of the pilot program, there were outstanding issues concerning the following

indicators and their thresholds that have been addressed as described:

ERO Drill/Exercise Performance (DEP) and Drill Participation Indicators: NEI 99-

02, Revision 0, included language explaining that these two indicators are linked. 

That is, for key ERO members who are responsible for classification, notification or

PAR development to get credit in the ERO Drill Participation PI, their performance

must be assessed and must contribute to the DEP PI value. The issue of linkage

between the DEP PI and ERO PI was largely resolved early during initial

implementation. However, the linkage was not apparently universally understood

and, in a few instances, was misinterpreted or overlooked. The clarity of the

linkage was improved in NEI 99-02, Revision 1, and NEI committed to conduct

further industry training to ensure understanding. The indicators allow the licensee

to run as many drills as deemed necessary, above a minimum set by regulation. If

performance is declining toward the White band, licensees have generally used

this option rather than to allow the PI to cross the threshold.

ANS Reliability Indicator: The concern expressed by public stakeholders,

licensees and inspectors about the appropriateness of the ANS Reliability PI is
6

discussed above. There was also a concern about the thresholds for this

indicator. The thresholds were established by expert opinion, based on 60 plantyears of data. The Green/White threshold was set at 94 percent in recognition of

the typically high ANS reliability. When this threshold was developed, only one

plant was below the threshold and one near it. Therefore the threshold was

deemed to accurately reflect the performance level at which greater NRC

involvement in licensee maintenance of the ANS was appropriate. This premise

was borne out during initial implementation. The number of White indicators has

been about the expected rate of 5 percent, and they were primarily due to

problems with ANS maintenance. The Yellow band threshold was set at 90

percent, the reliability value that requires a formal corrective action plan to be filed

with FEMA. Although no member of the NRC/Industry EP working group was

aware of an ANS with annual reliability so low, one did occur during initial

implementation. This poor reliability could have been masked under the normal

FEMA reporting scheme because ANS statistics for that plant would have been

combined with a neighboring plant. Inspection of the associated corrective actions

revealed a poor root cause analysis and poor corrective action implementation by

the licensee, which indicated that the threshold was appropriate.

Protected Area Security Equipment Performance Index: As described above, the

NRC/Industry working group examined the use of unplanned unavailable hours in

lieu of compensatory hours in this PI and concluded that there is no need to

change. Also, as described above, this PI is a multi-input PI and the staff has

proposed a resolution to the masking problem. In addition, both the White band

and Yellow band thresholds, that had been a concern during the pilot program,

were changed when the historical data provided by all plants prior to the start of

initial implementation was evaluated. The staff determined that the Green band

should be widened by adjusting the Green/White threshold from 0.05 to 0.08. The

Yellow band was eliminated because the NRC/Industry working group agreed that

posting a guard when security equipment is inoperable does expend additional

resources but generally provides adequate compensation so that a degraded

condition does not exist. The working group also agreed that human

compensation measures, while effective in the short-term, may become degraded

after an extended period of time due to the inability of a human to remain at a full

attentiveness level. The human tendency to continue to expect no activity after a

significant period of no activity is a factor that may lead to a degraded condition as

well. Therefore, the performance indicator has been determined to provide a valid

indication for a siteís protected area perimeter security systems.

Containment Leakage Indicator: As a consequence of several significant

problems with this indicator, it was removed from the program prior to initial

implementation. The problems included infrequent measurement of leakage

(typically, only during refueling outages), lack of a common method among

licensees for measuring leakage, and untimely (after-the-fact) indication of

containment integrity.

Treatment of Significant Events: The steam generator tube failure at Indian Point Unit 2

raised questions about the need for a PI to monitor steam generator tube performance
7

during plant operation. While such a PI could be developed, the staff determined that it

would not provide useful information and the effort was therefore terminated.

Self-Assessment Metric Results

The ROP self-assessment program evaluates the overall success of the ROP being objective,

risk-informed, understandable, and predictable as well as its success in meeting the Agencyís

performance goals of maintaining safety, increasing public confidence, making NRC activities and

decisions more effective, efficient, and realistic, and reducing unnecessary regulatory burden on

stakeholders. On a periodic basis, the self-assessment program collects information from

various sources, including the Reactor Program System (RPS), the inspection program, the PI

program, additional industry level PIs, periodic independent audits, stakeholder surveys, and

public comment. Based on this information, an assessment of ROP success is performed in the

programmatic areas of performance indicators, inspection, significance determination process,

and assessment. Listed below are discussions of the self-assessment metrics associated with

the PI program. Due to the lack of historical data, an in-depth analysis is not possible in most

cases. However, where appropriate, some conclusions were reached.

Measures for the PI program were developed as part of the overall assessment of the ROP. The

measures demonstrate attributes of the PI program that were designed to give feedback in key

areas that monitor performance. Those attributes, a description of the measures, and a summary

of the preliminary self-assessment results follow.

Objective, Understandable, Predictable

These attributes are all closely linked. The staff determined that the PI program would be

considered objective, understandable, and predictable based on the number of: (1) significant

deviations that cross thresholds and (2) discrepancies in licensees reporting PI data, plus the

volume of questions regarding interpretations of PI guidance. These attributes were met as

determined by successful achievement of the following measures:

a. The low number of significant deviations that crossed thresholds. 

Significant deviations that crossed a threshold remained low during the entire first

year of initial implementation. Two instances were cited while conducting PI

verification inspections. The first occurred in the first quarter of initial

implementation and the second was in the fourth quarter. (Significant deviations

are defined as reporting inaccuracies that would cause a PI to cross a threshold

when corrected.)

b. The low number of discrepancies in reporting PI data plus the volume of questions

regarding interpretations (internal and external frequently asked questions). 

Reporting discrepancies trended downward from the initial implementation period,

with a ìspikeî upward in the third quarter of 2000. This was primarily due to

problems with the SSU PI guidance. The staff is currently addressing this issue,

hence the reporting discrepancies are approaching stability, as seen from quarter

to quarter.
8

Maintains Safety

The PI program maintains safety by: (1) tracking, trending, evaluating, and characterizing PIs that

cross multiple thresholds (i.e. Green to Yellow or Red) to allow timely interaction and (2)

surveying licensees for feedback regarding undesirable actions taken in order to remain within

the Green band. While it is too early to draw conclusions on the success of this attribute, positive

feedback obtained on these measures are as follows:

a. Quarterly reporting on the number of multiple crossed thresholds.

During the first quarter of initial implementation, one PI (High Pressure Safety

Injection) crossed multiple thresholds. The remainder of the full year of the

implementation of the ROP saw a decreasing trend resulting in zero multiple

thresholds crossed. 

b. Minimizing potential for licensee actions taken in response to the PI program that

adversely impact plant safety.

Responses to internal surveys on the PI program revealed that respondents

agreed that the PIs provide useful information on risk-significant areas (79%), are

clearly defined (72%), understandable (78%), and help to maintain safety (72%).

Efficiency, Effectiveness, Realistic

The attribute: efficiency, effectiveness, and realistic used five measures to obtain feedback to

monitor performance: (1) information is provided in a timely manner, (2) the process remains

stable over time, (3) the information is reported accurately, (4) timely indication received of

declining safety performance, and (5) minimizing potential for licensee actions taken in response

to the performance indicator that adversely impacts plant safety. The last three of these five

measures share the same success criteria as the objective, understandable, predictable

(Measure 3), and maintains safety attributes (Measures 4 and 5). The attribute was considered

met based on successful achievement of the aforementioned and the following measures:

a. The ability to provide information in a timely manner, which was measured by:

Tracking late PI postings on the NRCís external Web site

PI postings are due on the external Web site within five weeks of the end of each

quarter. To date, there have been no late PI submissions.

b. Process stability over time, which was measured by:

The number of changes made to the existing PIs. The expectation was to have

low numbers of PI changes, with a stable or decreasing trend. 

To date, two proposed changes to PIs have been pilot tested. The staff is

currently conducting an evaluation to determine if the proposed changes to the

unplanned scram and unplanned scram with loss of normal heat removal PIs

warrant replacement.
9

Enhances Public Confidence

The PI program enhances public confidence by assuring that accurate and understandable

information is provided in a timely manner. This attribute uses four measures. Three of the four

measures share the same measures previously achieved in the objective, understandable, and

predictable; and the efficient, effective, and realistic attributes. While it is too early to draw

conclusions on the success of this attribute, positive feedback obtained on this measure is as

follows:

a. Results of an external survey of stakeholders perception on the clarity of the

guidance contained in NEI 99-02.

External stakeholders offered a wide variety of comments, most of them related to

the effectiveness of the PIs in ensuring that declining performance is identified

before safety is compromised. Industry representatives agreed with a key premise

of the ROP, that weaknesses in cross-cutting issues, such as the corrective action

program, will manifest themselves in the PIs as well as in inspection findings. An

industry monitoring group tacitly endorsed the PIs by submitting a petition to the

NRC to make reporting of the indicators mandatory. The State of New Jersey,

however, pointed out that the ability of PIs to find performance problems is suspect

because they are consistently and overwhelmingly Green.

Reduces Unnecessary Regulatory Burden

While all respondents to internal and external surveys associated with the ROP indicated that

they believed that unnecessary regulatory burden has decreased overall, external stakeholders

held the view that regulatory burden in the area of the PI program had increased, but increased 

in an appropriate manner. The staff concluded that this attribute had been met because the

appropriate increase in burden associated with PI data collection was, on balance, a contributor

to the overall reduction of unnecessary regulatory burden associated with the ROP.

External survey results, as well as interaction with license and industry stakeholders at public

meetings and during IIPB staff visits to license facilities, indicate that licensees are concerned

with the duplication of reporting requirements between the PI program, the Maintenance Rule,

and INPO/WANO. The staff is working closely with its stakeholders through the ROP Industry

Working Group to resolve these data reporting issues.

Stakeholder Feedback (Internal/Public/Industry)

Most of the feedback on PIs was provided by resident and region-based inspectors, and by the

industry through FAQs. Much of this input concerned the SSU indicators. In addition, some

industry managers perceived the potential for negative consequences from counting manual

scrams, and the staff had concerns about the unintended consequences of the Unplanned Power

Changes per 7,000 Critical Hours PI. Also, inspectors and their managers provided feedback on

the extent of licenseesí efforts to avoid a White PI.
10

The most significant SSU issues include (1) credit for operator recovery actions, (2) treatment of

support system unavailable hours, (3) counting unavailable hours for on-line maintenance, (4)

removal of large blocks of fault exposure hours (FEH) due to a single event, (5) the use of nonconservative default values for the hours a train is required, (6) the use of FEH, both known and

estimated (t/2), (7) the impact of the thresholds on effective preventive maintenance, and (8)

inconsistencies between the various programs that monitor the unavailability of safety systems

(e.g., Maintenance Rule, PRA, INPO/WANO indicators, and ROP indicators). All of these issues

have been discussed in the staffís monthly public meetings with NEI and industry representatives. 

Those discussions have resulted in changes to the program to address the first four issues, as

described in the following section. Resolution of the four remaining issues has been more

difficult. Therefore, a new NRC/Industry working group has been created to take a clean-slate

approach to the SSU indicators to produce a set that is simpler, easier to understand, compatible

with all intended uses, and more risk-informed than the current indicators. This effort will start by

defining the objective of the indicators and will construct a new indicator from the ground up,

including reevaluation of the changes made to accommodate the first four issues. (The last

unresolved issue on the list above - the inconsistencies between the various programs that

monitor the unavailability of safety systems - is discussed in detail in the section on Consistency

of PI Definitions under SECY-00-0049 above).

Some industry managers are concerned about the potential for negative consequences from

counting manual scrams. This issue is discussed in detail in the section on Manual Scrams under

Staff Requirements Memorandum on SECY-00-0049 above.

Some internal stakeholders are concerned about the potential for unintended consequences in

the use of the Unplanned Power Changes indicator. The issue is discussed in detail in the

section on Guidance on Unplanned Power Changes per 7,000 Critical Hours under SECY-00-

0049 above.

A concern stated by some in the industry, in response to a Federal Register notice requesting

comments on the first year of initial implementation, was that the PI verification inspection could

place an unnecessary regulatory burden on licensees. The stated reason for this is that some

inspectors are trying to find undercounts in the SSU indicator in amounts that are so small as to

not be worth the effort of the inspector or the licensee to resolve. The staff believes that, given

the importance placed on the PIs, verification is necessary. Inspections have discovered

situations where increased licensee attention is warranted. As a part of resource tracking, the

staff is monitoring effort devoted to PI verification and will make warranted adjustments. Thus

far, it appears that a slight increase may be warranted. 

Perhaps a more significant source of burden of the PI program is generated by the perception of

many licensee that performance outside the licensee response band will have unacceptable

consequences that must be avoided if at all possible. The concern that licensees express to the

NRC is that a White PI puts them one more White PI or inspection finding away from a degraded

cornerstone, which would result in a significantly increased NRC supplemental inspection. In

addition, licensees are apprehensive about a public reaction that could result in unfavorable

publicity, increased insurance rates, and lower bond ratings. Because of these concerns, some

licensees, to avoid crossing the Green/White threshold, are striving for zero counts in those

indicators where any count is bad. This has caused several licensees to go well beyond what the

staff considers reasonable, in performing engineering analyses, simulator exercises, or tests of

mockups of plant conditions, to prove that a train would have functioned. This puts a heavy
11

burden on resident inspectors to investigate licenseesí responses to determine if they are

acceptable. This issue has had a significant impact on the PI program through increased

numbers of FAQs and regional feedback forms that must be resolved, through increased

inspector hours to review the licenseeís actions to avoid PI counts, and through increased

Headquarters staff time to respond to telephone and e-mail correspondence concerning the

appropriateness (or lack thereof) of the licensees actions. The staff will continue to evaluate

these concerns and take appropriate action.

In late 2000, the Union of Concerned Scientists (UCS), pursuant to 10 CFR 2.802, submitted a

petition for rulemaking to the NRC . The requested rule would require nuclear plant owners to

submit the PI information needed for the ROP. This petition was noticed in the Federal Register. 

The petition and responses to the Federal Register notice are under review by the staff.

The UCS also is concerned about not including design errors in the SSU. (This issue is being

addressed in the NRC/Industry SSU working group described above.) Other stakeholders

believe that the indicators are not definitive and the thresholds are not appropriate or that both

indicators and thresholds should be plant-specific. It is important to remember that the ROP

includes baseline inspections along with performance indicators to determine when the NRC

should engage the licensee. It is also important to remember that the ROP is a living program,

and it is the NRCís intent to make both the indicators and the thresholds more plant-specific and

more risk-informed as the program matures.

The State of New Jersey questioned the credibility of indicators that show 99% of the plants in

the Green band, a number that may be obtained when looking at the PI results for a single

quarter. Thresholds were based upon identifying approximately 5% of plants as outliers in one

year. This was done for the IE and MS cornerstones only and for the three year period from 1995

through 1997. If 1% of the indicators are non-Green in one quarter, then up to 4% may be nonGreen for one year (this can only be determined by looking at the performance of individual

plants). For the other cornerstones, the thresholds were set based on expert opinion, that, with a

few adjustments, turned out to be quite accurate. In some cases, such as the Protected Area

Security Equipment Performance Index and Emergency Preparedness Drill Participation,

licensees responded and fixed programs that had not been effective. These are PI program

successes that caused licensee performance to improve over what it was in 1995 through 1997.

Hence the number of non-Green indicators in those cornerstones has gone down. It is expected

that licensees will respond by devoting more resources to performance measures that are being

tracked and made publicly available. Therefore we would expect performance to improve over

time in each of the indicators. 

Program Changes Made as a Result of Lessons Learned

The SSU indicators were changed as follows:

ï Credit for operator recovery actions was expanded to allow credit for recovery from

operator errors and equipment malfunctions. The latter has generated a number of

questions and resulted in some licensee actions that are well beyond what was intended. 

ï A provision was added that allowed licensees who perform on-line maintenance to not

count those hours as unavailable if the licensee has in place a quantitative risk
12

assessment that demonstrates that the increase in risk is small. This was done to allow

licensees to perform on-line maintenance and not count unavailable hours if the risk is

comparable to doing that maintenance while shut down, when the hours would also not be

counted.

ï A provision was added to allow the removal of FEH for blocks of FEH of 336 hours or

greater due to a single event after 4 quarters have elapsed, if the licensee has competed

all necessary corrective action and the NRC has inspected and closed out the issue. This

will reset the indicator so that future problems could cross the threshold and result in

supplemental inspection.

Long-term Program Changes Planned

Plans for long-term changes to the PI program include the following:

ï Evaluate the results of the two Initiating Events Cornerstone pilots for possible

replacement of the three indicators in this cornerstone (Scrams, Scrams with Loss of

Normal Heat Removal, and Unplanned Power Changes). 

ï Work with industry to improve the existing SSU indicators to simplify them, make them

easier to understand, more compatible with all uses, and more risk-informed; and evaluate

the RBPIs as possible additions to or replacements for the SSU indicators.

ï Continue to work on the barrier indicators (RCS activity, RCS leakage, and a containment

PI) to improve the usefulness of those indicators.

ï Work with industry to remove the potential to mask poor performance in the protected

area security equipment index PI.

ï Improve the usefulness of the Personnel Screening Program Performance and the

Fitness-for-Duty/Personnel Reliability Program Performance indicators. These PIs count

reportable events, which have been considered ineffective in identifying declining

performance.
INSPECTION PROGRAM

Task Lead: Steve Stein, NRR/IIPB

Attachment 7
1

Inspection Program

Open Issues Addressed

There were several open issues and commitments that the staff continued to evaluate and work

on during ROP initial implementation. Several of the issues involved longer term items that the

staff identified in Commission paper SECY-00-0049 as needing additional work following initial

implementation. The Commission directed the staff to consider several additional issues in the

SECY-00-0049 Staff Requirements Memorandum (SRM) dated May 17, 2000. And several

recommendations for staff consideration during initial implementation were made by the Pilot

Program Evaluation Panel (PPEP) in their final report. The issues and recommendations for the

inspection program are as follows:

SECY-00-049

ï Evaluate during initial implementation and beyond the supplemental and event response

portions of the inspection program; collect feedback on how cross-cutting issues are

handled by the inspection program; evaluate and incorporate feedback into the program

and continue to improve the program to ensure it provides sufficient information to assess

cornerstone attributes.

ï Issue a supplemental inspection procedure for human performance and continue to

assess the adequacy of the supplemental portion of the inspection program.

ï Evaluate the need for additional inspections of fuel movement during operations.

SECY-00-049 SRM

ï Ensure the threshold for documenting observations is clearly understood and that

observations do not undermine consistent and objective inspection and enforcement.

PPEP Final Report

ï Ensure program effectiveness is not measured solely based on resource utilization.

ï Evaluate significant events for insights into effectiveness of inspection program results.

Self-Assessment Results

The ROP self-assessment program evaluates the overall success of the ROP being objective,

risk-informed, understandable, and predictable as well as its success in meeting the Agencyís

performance goals of maintaining safety, increasing public confidence, making NRC activities and

decisions more effective, efficient, and realistic, and reducing unnecessary regulatory burden on

stakeholders. On a periodic basis, the self-assessment program collects information from

various sources, including the Reactor Program System (RPS), the inspection program, the ROP

performance indicator (PI) program, additional industry level PIs, periodic independent audits,

stakeholder surveys, and public comment. Based on this information, an assessment of ROP

success is performed in the programmatic areas of PIs, inspection, significance determination
 1

ìNo Colorî findings are inspection issues whose significance can not be evaluated by the

SDP and, therefore, can not be assigned a color.

2

process (SDP), and assessment. Listed below are discussions of the self-assessment metrics

associated with the inspection program. The metrics do not directly measure the quality of

inspections and one year of implementation may not be sufficient to draw conclusive results. 

However, where appropriate, some conclusions were reached.

Objective

The program was developed to be objective and included guidance for establishing a minimum

threshold for inspection findings and for explaining the objective basis for the significance of the

findings. Therefore, the programís focus on objective findings and minimization of reliance on

subjective matter is measured by how well findings that appear in inspection reports meet the

programís guidance. Although the data is preliminary, it indicates that most of the issues being

documented comply with the programís requirements for adequately explaining the significance

and reasons for the findings, and therefore are objectively arrived at. The data also indicates that

there is room for improvement. The guidance for documenting issues in inspection reports was

changed significantly during initial implementation. The revised guidance clarified the thresholds

for documenting findings and the expectations for documenting the significance of findings. 

Additional changes will be made based on lessons learned and feedback from inspectors and

public stakeholders to continue to improve in this area.

The survey of internal stakeholders asked if the program fosters objective findings whose

significance can be clearly explained. Almost three quarters of the respondents agree that the

program does.

Risk-Informed

The risk-informed nature of the inspection program is indicated by several measures: (1) how well

all findings that appear in inspection reports meet the programís guidance, (2) how well ìNo

Colorî1

 findings that appear in inspection reports meet the programís guidance, (3) the number of

successful appeals of SDP determinations, (4) the number of changes to program documents

that improve the risk-informed aspects of the program, and (5) the number of changes to baseline

inspection program documents that affect scope of what is inspected or frequency of inspection. 

The measures are augmented by a survey of inspectors and other NRC staff associated with the

ROP.

The preliminary data indicates that the baseline inspection program is risk-informed. Findings are

generally documented as required by the program and few final SDP determinations are

successfully challenged. However, the number of program documents that have been changed

to improve the risk-informed nature of the program indicate that further improvements can be

expected. The majority of NRC personnel surveyed believe the program is appropriately riskinformed.
3

Understandable

The ability of program requirements to be understood by the NRC staff responsible for

implementing the program, and the ability of inspection reports to communicate relevant

information are measured by responses to an internal survey. While the majority of those

implementing the baseline inspection program find its procedures to be understandable, a large

percentage do not. The staffís audit of inspection reports also finds that a large percentage of

inspection findings are not clearly described in the reports.

Predictable

The inspection program is considered to be predictable if it is implemented as defined and if it is

planned and conducted consistently across the regions. The measures used to determine

predictability are: (1) the rate of completion of the baseline inspection program over the year,

(2) the percentage of inspections whose schedules change, and (3) a comparison and analysis of

inspection effort. The measures are augmented by a survey of inspectors and other NRC staff

associated with the ROP.

Preliminary results indicate that the program was generally consistently applied across the

regions. Although the rate of changes to inspection schedules varied during the year, it averaged

under 9 percent overall. The largest number of changes (42 percent) were due to inspectors

being unavailable or perturbations in schedules to accommodate a plant the region needed to

focus on, such as Indian Point 2 (IP2) in Region I. Schedule changes to make the inspection

efforts more effective or efficient, such as by combining inspections or because the licenseeís

schedule or program changed making site conditions inappropriate for the inspection, accounted

for 22 percent of the changes. Changes made to accommodate a licensee (regulatory impact

changes) accounted for 19 percent of the schedule changes. Another 6 percent of the changes

were because of a conflict with another NRC inspection or meeting. The reasons for 10 percent

of the changes were not identified. This level of schedule change was acceptable and the staff

will continue to monitor it.

The resources used in implementing the program were close to the estimated resources. The

analysis of resources for the ROP are discussed in ROP resources attachment to this paper.

Maintains Safety

The inspection program helps to maintain the current level of plant safety if the program covers

all the areas deemed necessary for plant safety and the program is fully implemented. If the

scope and depth of inspections are adequate and the program is fully implemented, then the

program maintains safety. Therefore, this aspect of the program is measured by: (1) the

number of program documents changed that effect the scope or frequency of inspection, and

(2) the number of changes to inspection schedules for reasons other than regulatory impact. 

The measures are augmented by a survey of inspectors and other NRC staff associated with the

ROP.

Only a small percentage of program documents were revised effecting the scope or frequency of

inspection. In general the scopes were somewhat reduced or frequencies decreased (a biennial

inspection in physical protection was changed to triennially). Although the rate of inspection

schedules that changed varied during the year, it averaged under 9 percent of the total activities
4

scheduled. The survey of NRC personnel implementing the new program shows the majority of

them find that the program covers activities important to safety and addresses the attributes of

the cornerstones of safety.

Therefore, the staff concludes that the inspection program does contribute to maintaining plant

safety.

Efficiency, Effectiveness, Realistic

How efficient the inspection program is measured by a comparison of resources, both direct

effort by NRC inspectors and contractor support dollars. The programís effectiveness and

whether it is realistic are measured by: (1) the number of changes to inspection schedules and

the reasons for the changes, (2) the programís stability as evidenced by the number of

significant changes to the program, and (3) the timeliness of issuing inspection reports and

completing temporary instructions. The measures are augmented by a survey of inspectors and

other NRC staff associated with the ROP.

The analysis of resources for this first year of implementation found that the regional resources

were adequate to effectively carry out the ROP, including the inspection program. The data will

be used to better estimate the resources necessary for future implementation of the program

and to develop a more realistic budget model for inspection. See the ROP resources analysis

attachment to this paper.

Changes were made to about 9 percent of the inspections scheduled during the year of initial

implementation. The reasons for many of the changes (42 percent) were unavailable inspectors

and making inspectors available for plants that two of the regions needed to focus on. About 13

percent of the changes were made to be more efficient and effective by combining inspections

or assigning inspections to an inspector already on site. Another 8 percent of the changes were

because a licensee changed its schedule or program, which made site conditions inappropriate

for the inspection.

The number of change notices issued that significantly effected the program increased during

initial implementation. Further significant changes are expected.

The majority of contracted support has been in the areas expected: design and fire protection for

baseline team inspections. The only other contract support needed was mechanical and

electrical engineering for the IP 95003 supplemental inspection at IP2.

Almost all inspection reports were issued within the programís timeliness requirements and

temporary instructions were completed within deadlines. The temporary instruction for

evaluating licenseesí processes for reporting performance indicators had to be extended several

months. The instruction was not performed at two plants because they had been shutdown and

were not collecting data for the indicators.

The survey of NRC personnel implementing the ROP indicates that the baseline inspection

program appropriately emphasizes inspection planning.
5

Although generally meeting the goals of efficiency, effectiveness, and realistic, the inspection

program can be improved in these areas. The program should stabilize after the changes

following the year of initial implementation have been incorporated into the program documents.

Enhances Public Confidence

Because public confidence is not measured directly, the self-assessment metrics are based on

the premise that the inspection program enhances public confidence when information is timely

and accurate. How well the program meets this goal is measured by: (1) the timely posting of

inspection results on NRCís Internet Web site, (2) the number of inaccuracies in the posted and

issued inspection material, and (3) a survey of inspectors and other NRC staff associated with

the ROP. The preliminary data indicates that the posting of inspection results is timely and

accurate. Only a few problems with posted information were identified and usually related to not

properly updating the original Plant Information Matrix entries for findings after their significance

had been finally determined.

Reduces Unnecessary Regulatory Burden

How well the inspection program, as part of the overall ROP, helps to reduce unnecessary

regulatory burden on licensees is not directly measured. It is indicated by an analysis of

comments from the industry and other public stakeholders received in response to a Federal

Register notice request for comments. The general response from public stakeholders is that

the overall burden of the ROP compared to the previous process has been reduced, although

some of the public stakeholders do not see this as a good outcome. The industry attributes the

reduction to better focus on risk significant issues and integration of enforcement into the

assessment process.

Several industry stakeholders commented on areas for further reductions in unnecessary burden

within the inspection program. These areas included decreasing the frequency of team

inspections, combining inspections of programs that are common across a utilityís different

reactor sites, and using a licenseeís self-assessments for satisfying baseline inspection program

requirements. NEI stated that improvements in the focus, efficiency, and effectiveness of the

inspection program were most notable in the reactor safety cornerstones and less pronounced in

the radiation safety and physical protection cornerstones. One utility commented on the

increase of inspection over the previous program in the radiation safety areas despite the advent

of performance indicators for monitoring a licenseeís performance, and NEI stated that the

inspection and the occupational exposure effectiveness indicator unnecessarily overlap.

Although the overall burden has declined, the industry stakeholders commented on the

increased inspection at what were previously considered superior performers (SALP category 1

plants). One utility commented on the seemingly inconsistent allocation of resources between

single unit and dual unit plants, with single unit sites receiving a disproportionate amount of

inspection. The change in the ìN+1" resident policy was mentioned as a contributing factor; i.e.,

both single and dual unit plants have two resident inspectors. Conversely the State of New

Jerseyís comments stated that the level of inspection is too low and the ROP doesnít clearly

define the appropriate levels based on performance. However, both the industry and State of

New Jersey agree that the ROP focuses resources on risk significant issues.
 2

IMC 0610* contains specific guidance for documenting inspections using IPs 95001 and

95002. The reporting guidance is less constrictive for IP 95003 and the report form is left to the

inspection team leaderís customization because of the diagnostic nature of the inspection.

6

Although the ROP has reduced overall burden on licensees and the inspection program is

focusing the NRC and licensees on the more important issues, there are opportunities to

improve the burden imposed on licensees by the inspection program. The areas recommended

by industry for possibly reducing inspection burden have not yet been evaluated.

Evaluation of Supplemental Inspection Program

In SECY-00-049, the staff committed to evaluate the supplemental and event follow-up portions

of the inspection program, as these portions of the program had not been sufficiently exercised

during the pilot process. During the first year of initial implementation of the ROP, experience

was gained with all three supplemental inspection procedures (IPs). IP 95001, ìInspection For

One Or Two White Inputs In A Strategic Performance Area,î which is performed at facilities that

reach the regulatory response column of the Action Matrix, was used 30 times. IP 95002,

ìInspection for One Degraded Cornerstone or Any Three White Inputs in a Strategic

Performance Area,î which is performed at facilities that reach the degraded cornerstone column

of the Action Matrix, was used four times. And IP 95003, ìSupplemental Inspection For

Repetitive Degraded Cornerstone, Multiple Degraded Cornerstone, Multiple Yellow Inputs, Or

One Red Input,î which is performed at facilities that reach the Multiple/Repetitive Degraded

Cornerstone column of the Action Matrix, was used once.

Throughout this first year of the ROP, NRR staff reviewed all supplemental inspection reports. 

The results of that review indicate a high degree of uniformity in the implementation of the

inspections. The degree of uniformity can be largely attributed to the inspection report guidance

contained in Inspection Manual Chapter (IMC) 0610*, ìPower Reactor Inspection Reports,î

which requires the documentation of all supplemental inspection requirements, along with the

inspectorís findings for each requirement.2

 This guidance helps to enhance public confidence by

providing a clearly documented NRC assessment of the licenseeís evaluation and corrective

actions associated with significant issues. The requirements for documenting supplemental

inspections also ensure that the supplemental inspections are highly predictable.

Based on reviewing these inspection reports and feedback received from the staff conducting

the inspections, the staff believes that the supplemental inspection program will be effective at

maintaining safety. The supplemental inspections conducted during initial implementation have

helped licensees improve the effectiveness of their root cause evaluations of risk significant

issues. During several inspections, the NRC identified significant deficiencies in the licenseeís

evaluation of the issues. In some cases, these deficiencies were found to be programmatic and

resulted in broad improvements to the licenseeís corrective action program. By allowing

licenseeís the opportunity to first evaluate significant issues before the NRC intervenes with

inspection, the supplemental inspection program also highlights the importance of our role as

regulators and the licenseeís role as operators of the facility.

The supplemental inspection program also seems to be very efficient. The 30 inspections

conducted using IP 95001 totaled 852 hours of direct inspection effort. The four inspections

conducted using IP 95002 totaled 739 hours and the one inspection conducted using IP 95003
7

took 1724 hours. The total direct inspection hours charged to the supplemental inspections

during the first year of the ROP were 3315, or an average of 32 hours for each of the 103

operating units. While a one-to-one comparison can not be made to the previous inspection

program, the supplemental inspection effort under the ROP was significantly less than the plantspecific inspection effort under the previous program.

Evaluation of Event Follow-Up

Management Directive (MD) 8.3, ìNRC Incident Investigation Program,î was revised early this

year to risk inform the Agencyís response to significant operational events at power reactors. It

provides three levels of event follow-up as a function of deterministic and risk criteria: Incident

Investigation Teams, Augmented Inspection Teams (AITs), and Special Inspections. IP 93812,

ìSpecial Inspection,î was issued to provide guidance on regional follow-up to events that needed

more attention than a quick review under the baseline inspection program, but were not

significant enough to warrant an incident investigation or augmented inspection. Baseline

inspection procedure IP 71153, ìEvent Follow-Up,î requires the inspector to evaluate licensee

events in order to provide input regarding the deterministic and risk criteria, which assists

management in deciding the level of NRC response.

During the first year of the ROP, a draft version of MD 8.3 was used until the revision was issued

in March 2001 and the regions provided feedback in its use. The feedback resulted in including

degraded conditions within the scope of significant operational events at power reactors, and

revising IP 71153 to clarify that licensee notifications in accordance with 10 CFR 50.72 are one

means of activating event follow up.

During the first year of the ROP, eight events at power reactors (one AIT follow-up and seven

Special Inspections) required a follow-up inspection beyond the baseline inspection program, for

a total direct inspection effort of 1926 hours. For most of these events, the region consulted with

NRRís Inspection Program Branch (IIPB) regarding implementation of MD 8.3 and IP 71153 in

determining the appropriate level of event follow-up. The staffís review of these events and

follow-up validated the effectiveness of determining the level of event follow-up based on a

combination of deterministic and risk criteria.

Stakeholder Feedback (Internal/Public/Industry)

The staff collected the majority of the feedback on the inspection program from several sources,

including: (1) feedback forms from inspectors and other NRC staff, (2) a survey of NRC staff

who were involved in the initial implementation of the ROP, (3) a Federal Register survey of the

public and industry, and (4) public meetings held near Washington, DC and all four regions. In

addition, about half way through initial implementation IIPB visited 24 sites and all four regions to

interview resident and regional inspectors, licenseesí staff, and regional managers.

The staff addressed questions and recommendations as feedback was received on individual

procedures or other program documents. Many of the comments and suggestions were

incorporated during initial implementation. Other feedback, such as estimated time necessary to

complete a procedure, were held in abeyance until the end of initial implementation so the staff

could evaluate a broader range of data from all four regions before making changes to the

program.
 3

 The document with the most feedback forms was IMC 0608, ìPerformance Indicator

Program,î because of the large number of questions raised regarding interpretation of the

indicators. The individual SDP procedures also received many feedback forms, and when

combined exceed the number received for IMC 0610*.

8

A summary of the major feedback themes is as follows.

Clarity of Findings In Inspection Reports

Although the majority of stakeholders believe that inspection reports provide sufficient and

proper information, a common criticism is that inspection findings in inspection reports donít

always conform to program requirements for documenting the basis for determining the findingís

significance. That guidance is in IMC 0610*, ìPower Reactor Inspection Reports.î The staffís

audit of inspection reports confirms that a high percentage of findings are not adequately

documented in reports. The staff is using the audit results to disseminate examples contrasting

properly and improperly written findings to the inspectors and regions and to clarify the program

guidance for writing inspection reports.

IMC 0610* also was the subject for many feedback forms from NRC inspectors. Of all of the

program documents, the third highest number of feedback forms were written against the report

writing guidance.3

 Many of the forms questioned implementation of the new aspects of

documenting inspection findings in the ROP, such as cross-cutting issues, threshold for

documenting issues, and licensee-identified violations, and pointed out seeming contradictions.

Two issues with documenting the results of inspections in the ROP were the subject of task

groups and discussed at the External Lessons Learned Workshop in March 2001: (1) the role of

cross-cutting areas and (2) ìNo Colorî findings. The resolution of the issues are discussed in

other attachments to this paper. The role of cross-cutting areas is discussed in the cross-cutting

attachment and documenting ìNo Colorî findings is discussed in the assessment process

attachment.

Physical Protection Inspections

Much of the feedback from the first year of implementation related to challenges in determining

the significance of and documenting findings from the inspection of force-on-force exercises. 

The staff got only limited experience using the ROP in overseeing security force-on-force

exercises during initial implementation because the exercises was not initially inspected under

the baseline inspection program. The SDP aspect of this concern is discussed in the

significance determination process attachment to this Commission paper. The other aspect of

this concern dealt with the scope and threshold criteria for the screening questions in IMC 0610*,

that determine if an issue should be documented in an inspection report. The staff addressed

these issues by implementing an interim physical protection SDP and a revision to the screening

questions in IMC 0610*. A final SDP for physical protection will be issued upon additional

feedback and experience gained over the upcoming year.

Another topic of feedback received concerned the nuclear power industryís proposal for a selfassessment program that would demonstrate the capability of a plantís physical protection

contingency response to defeat a design basis threat, as prescribed by 10 CFR Part 73. 
9

Feedback from the stakeholders regarding the Safeguards Performance Assessment (SPA)

program focused on the revisions to the baseline inspection program necessary to provide

sufficient regulatory oversight for the SPA program, and ensuring that the SPA program parallels

and complements the current 10 CFR 73.55 rulemaking effort.

Maintenance Rule Inspections

Feedback from inspectors indicated that the frequency and level of effort for IP 71111.12,

ìMaintenance Rule Implementation,î were too high and that the programmatic aspects of this

inspection were neither risk-informed nor met the minimum threshold for documenting findings in

an inspection report. Based on the results of IP 71111.12 inspections, the most significant

finding can only be classified as very low significance (Green). As a result, inspectors

questioned the value of the procedure and suggested that it be enhanced. As originally written,

IP 71111.12 verifies the licenseeís implementation of its Maintenance Rule program (i.e., it is

compliance based) and doesnít verify the effectiveness of the licenseeís maintenance efforts

(i.e., performance based). Only minor comments were received from inspectors on other

baseline inspection procedures related to maintenance effectiveness.

A maintenance effectiveness focus group, consisting of regional senior resident inspectors and

staff from NRR and OE, reviewed this issue, developed several recommendations for

improvements, and discussed the issue and recommendations at a public workshop in March of

this year. One of the groupís recommendations was to revise IP 71111.12 to have the

inspectors evaluate equipment performance issues, such as availability and reliability preferably

relating to trains of high safety-significant systems not covered by PIs and systems with negative

performance trends or known equipment performance problems. Other recommendations

included adding requirements and guidance into the annual problem identification and resolution

inspection procedure, IP 71152, to determine the adequacy of licenseeís corrective actions in

addressing performance issues associated with maintenance errors and Maintenance Rule

implementation issues. The staff is incorporating the focus groupís recommendations into the

procedures.

Inspection of ALARA Programs

Many stakeholders were concerned with the baseline inspection programís use of collective

dose as a screening criteria for inspection findings for as low as reasonably achievable (ALARA)

programs. Some stakeholders believed it was inappropriate that the screening criteria (a 3-year

rolling average) could cause identical findings at different plants to be treated differently. Such

findings would not be documented in inspection reports for plants below the criteria and

documented for plants above the criteria. Plants above the criteria also could be susceptible to

entering the Degraded Cornerstone column of the Action Matrix from findings during one outage.

The issues with the ALARA inspection and findings were discussed at the public workshop in

March 2001. The discussions resulted in several approaches and suggestions for changes to

the ALARA portion of the baseline inspection program. These approaches included removing

the significance criteria from the screening criteria (Group 2 questions in IMC 0610*); developing

a new Group 2 screening question for ALARA to clarify that the basis for a finding in this area is

a breakdown in or failure to implement the ALARA Program resulting in ìunintended doseî; and

using the 3-year rolling average for collective dose to adjust the level of effort of the baseline
10

inspection for individual plants. The staff is evaluating the approaches and will make any

necessary changes to the program to implement the approaches taken.

Industry stakeholders have consistently commented on the increased level of inspection effort

over the previous program in the radiation safety area, even with the advent of a performance

indicator. The estimated inspection hours will be revised after the ALARA baseline inspections

have been restructured.

Refueling and Other Outage Activities

In SECY-00-0049, the staff stated that lessons learned from using the refueling and outage

activities inspection procedure and its adequacy in addressing spent fuel cooling systems would

be incorporated into the procedure. The staff also committed to evaluate the need for additional

inspections of fuel handling during power operations during the first year of implementing the

ROP. 

The spent fuel pool cooling system is integral to decay heat removal for shutdown safety, but is

not covered by the Shutdown Operations SDP. The staff is evaluating how to incorporate this

area in the SDP. In addition, the staff is evaluating how the SDP should address inspection

procedures for independent spent fuel storage installations.

The staff evaluated feedback from inspectors, and revised the baseline inspection program

procedure to address areas such as the scope of outage activities to inspect, estimated

resources to conduct the inspection, and inconsistencies between inspection requirements and

licensee technical specifications. Some feedback indicated that the procedure needs to focus

on areas that are addressed by the Shutdown Operation SDP and are more risk significant. The

next procedure revision will address this, primarily by eliminating areas that are not addressed

by the SDP.

Other Feedback

Several industry stakeholders have expressed interest in using self-assessments in anticipation

of major baseline inspections to either find significant issues before the NRC finds them, or to

reduce the scope and effort of the baseline inspection. Currently, the staffís practice is to not

take credit for licensee efforts in satisfying baseline program requirements, at least until the NRC

has had sufficient experience in implementing the new program.

The baseline inspection program was designed to be indicative and focused on results, not

predictive and focused on causes. However, a common theme in the comments received with

the internal stakeholders survey is a dissatisfaction with what is perceived to be the reactive

nature of the baseline inspection program and itís reduced emphasis for finding the causal

factors for problems. Many respondents stated that allowing inspectors to document minor

issues (those below the level of very low significance) and their causes would be useful in

predicting problems in areas such as human performance before they manifest themselves in

other ways, such as failed equipment. The Commonwealth of Pennsylvania, in its response to

the Federal Register notice, also expressed its view that the baseline inspection program should

focus more on cross-cutting areas.
11

The latest survey of NRC personnel can not be statistically analyzed against the 1999 survey

conducted of those who implemented the pilot program, but some general comparisons can be

made. For those questions that were comparable, the responses were generally the same or

showed some improvement. The largest improvement was in the responses to the questions

about whether inspection reports communicate relevant information to the public and licensees. 

Another area of marked improvement was the responses to the question about whether the

baseline inspection procedures adequately address cornerstone objectives.

One of the comments from the Nuclear Energy Institute (NEI), in its response to the Federal

Register notice, was that the baseline inspection programís verification of PIs could become

overly burdensome if not properly managed. NEI cites as an example inspectors who focus on

uncovering time that should have been counted as part of a PI of such short durations as to be

insignificant to the reported indicator. The objective of verifying PIs in the baseline inspection

program is to sample the raw data being used to calculate the indicators to provide assurance of

the indicatorsí accuracy without reconstructing the reported numbers. The inspection procedure

for verifying PIs, in conjunction with IMC 0610*, emphasizes finding and reporting information

that is significant enough to cause an indicator to cross thresholds, and doesnít allow

documenting conditions that could not appreciably affect the indicators. Therefore, the program

provides a disincentive for inspectors to focus on insignificant factors. In addition, the staffís

review of this aspect of the baseline inspection program indicates that the focus of PI verification

inspections is usually appropriate although the resources necessary to complete the verifications

are higher than originally estimated.

Program Changes Made as a Result of Lessons Learned

During the year of initial implementation, the following program documents were changed in

response to feedback and lessons learned:

ï IMC 0610*, was revised to clarify: (1) the thresholds for documenting inspection findings,

(2) what constitutes a documentable cross-cutting issue, and (3) documenting the basis

for significance of findings. Agency decisions about ìNo Colorî findings will affect the

guidance in IMC 0610*, which will be revised to implement the decisions. Examples of

properly and improperly documented findings from past inspection reports will be added

to the manual chapter.

ï A supplemental inspection procedure (IP 71841, ìHuman Performanceî) for evaluating

corrective actions for human performance deficiencies was issued on December 12,

2000. (SECY-00-049 commitment)

ï The baseline inspection procedure for following up events (IP 71153) was revised to

compliment the risk-informed aspects of MD 8.3.

ï The baseline inspection procedure for refueling and other outages (IP 71111.20) was

revised to clarify its basis for fuel movement and to add guidance on the increased risk

during such times.

ï The baseline inspection procedure IP 71111.13, ìMaintenance Risk Assessment and

Emergent Work Control,î was revised to incorporate 10 CFR 50.65 (a)(4) rule
12

requirements and inspection guidance. A supplemental inspection procedure (IP 62709,

ìConfiguration Risk Assessment and Risk Management Processî) for performing

independent assessment of the conditions associated with 10 CFR 50.65 (a)(4)

implementation problems was issued on December 28, 2001. 

A number of baseline inspection program procedures are being changed based on the staffís

review of initial implementation. The procedures whose scope, frequency, or level of effort are

being changed significantly include the following:

ï The procedure for inservice inspections will be changed based on staff recommendations

in the steam generator action plan and the lessons learned from the steam generator

tube failure at Indian Point 2. At multi-unit sites, the inspection will be conducted at every

refueling outage for each reactor unit instead of at one unit during an outage about every

two years. Additional inspection requirements will be added for pressurized water

reactors with older steam generators.

ï The inspection frequency for evaluating licenseesí programs that implement 10 CFR

50.59 will be changed from annual to biennial because the inspection is more effective

when combined with other design related inspections that are conducted every two years. 

An initial evaluation of licenseeís meeting the recently revised rule will be accomplished

with a one-time temporary instruction inspection.

ï The Maintenance Rule inspection procedure (IP 71111.12) is being rewritten to make it

more risk-informed and performance based.

ï The ALARA procedure will be revised after the staff restructures the screening and

significance criteria for ALARA findings.

ï The annual problem identification and resolution inspection will be changed to biennial. 

Other changes to the baseline inspection program will ensure continued oversight of this

cross-cutting area every year. The changes will make the inspections more effective and

efficient by allowing a more in-depth (albeit less frequent) assessment of a licenseeís

program while emphasizing routinely following up individual issues. (See the crosscutting attachment to this paper.)

Long-term Program Changes Planned 

The Physical Protection cornerstone baseline inspection program procedure (IP 71130) and its

attachments will be revised over the next year to account for any changes in policy or

regulations. The feedback from external stakeholders on the 10 CFR Parts 26 and 73

rulemaking and the OSRE program, along with internal inspector feedback on the baseline

inspection program, will form the basis for drafting the revisions to the inspection procedure. 

The staff also is developing a new inspection procedure on the NRCís oversight of the SPA pilot

program. The staff is considering any scheduling efficiencies between the SPA pilot program

inspection procedure and the baseline inspection program.

Refinements to the estimates for the inspection effort and budget models will continue as a

result of these changes in inspection program scope.
13

The staff needs to evaluate how licensee self-assessments could be used in satisfying some

requirements of the baseline inspection program.

The staff is publically meeting with stakeholders to restructure the assessment of ALARA

findings (screening criteria and significance criteria) and will revise the associated inspection

program documents to accommodate the consensus outcomes.
SIGNIFICANCE DETERMINATION PROCESS

Task Lead: Peter Koltay, NRR/IIPB

Attachment 8
1

Significance Determination Process

Open Issues Addressed

There were several open issues and commitments that the staff continued to evaluate and work

on during ROP initial implementation. Several of the issues involved longer term items that the

staff identified in Commission paper SECY-00-0049 as needing additional work following initial

implementation. The Commission directed the staff to consider several additional issues in the

SECY-00-0049 Staff Requirements Memorandum (SRM) dated May 17, 2000. And several

recommendations for staff consideration during initial implementation were made by the Pilot

Program Evaluation Panel (PPEP) in their final report. The issues and recommendations for the

Significance Determination Process (SDP) are as follows: 

SECY-00-0049

ï The staff will continue to evaluate NRC risk analyst staffing needs throughout the first

year of implementation. It is anticipated that additional risk analyst expertise will be

needed to augment the regional senior reactor analysts (SRAs) on a long-term basis. 

Three senior residents were selected for the training program for regional SRAs. 

Additionally, twelve region based staff were selected to receive advanced risk training

that will enable them to perform and interpret basic quantitative risk analyses.

ï The staff will continue to evaluate the efficacy of the various SDPs. 

This is an ongoing process. The staff has continued to evaluate the efficacy through 

continuous monitoring of the process. Additional details on completed and planned

changes to the SDP process are provided under Program Changes Made as a Result of

Lessons Learned section of this document. 

ï The staff has planned to visit each plant site to discuss the plant-specific SDP Phase 2

worksheets with the licensee. The objective of these visits is to ensure that the individual

plant examination (IPE)-based SDP worksheets are modified as needed to include any

plant changes since the IPE was written or any new risk insights generated by the

licenseeís most current risk analysis. 

The site visits have been completed. The resulting changes will be incorporated into the 

worksheets and the documents will be issued for use by the inspectors by the end of July

2001.

ï The staff initiated a longer-term effort to develop a Phase 2 SDP process for issues

affecting reactor safety during shutdown conditions will continue beyond initial

implementation.

This is an ongoing activity utilizing the expertise of a national laboratory.

ï An integrated review of inspector training requirements is being conducted to better align

the NRC training courses in reactor technology, regulatory practices, and probabilistic
2

risk assessment (PRA) to support the revised reactor oversight program. An evaluation

of the type of training needed on new or significantly revised SDPs will be made after

development of these are complete.

This is an ongoing activity.

ï The current SDP does not explicitly address certain categories of violations of 10 CFR

50.65 (Maintenance Rule). 

A proposal to develop an SDP addressing Maintenance Rule (a)(4) violations is under

review.

ï The staff will develop an SDP for licensed operator re-qualification inspection issues.

The ìOperator Requalification Human Performance Significance Determination Processî

was issued as Appendix I to MC 0609 in December of 2000.

SECY-00-0049 SRM

ï The staff should continue to work with NRC stakeholders to resolve the "Performance

Issues Outside Licensing and Design Basis" issue while maintaining its appropriate

sensitivity to backfitting implications. The staff should specifically discuss this issue when

they report to the Commission on the results of the first year of ROP implementation.

Four examples of licensee performance outside the licensing or design basis of the plant

were addressed during this period. Although no compliance issues existed, the Agency

was able to take appropriate action through the assessment process and supplemental

inspection to address the safe significance of the issues. No backfitting evaluations were

performed during the period.

PPEP Final Report

ï The independent SDP review panel should be maintained during and following

industry-wide implementation to continue to reinforce and ensure desired consistency as

more inspectors are exposed to the process.

The panel is identified as the SDP/Enforcement Review Panel (SERP). The SERP 

reviews of each inspection finding potentially greater than Green contributes to the

consistent implementation of the ROP. Participants include the Headquarters and region

based SRAs, enforcement specialists, and appropriate NRR technical and project

management staff. The panel is chaired by the Chief, Inspection Program Branch (IIPB)

or by an IIPB section chief who is responsible for obtaining consensus on the

determination of significance of the findings. Detailed guidance on the SERP functions is

incorporated into a draft document that will become Attachment 0609.01 to Inspection

Manual Chapter (IMC) 0609, ìSignificance Determination Process,î when issued. 
3

Self-assessment Metric Results

The ROP self-assessment program evaluates the overall success of the ROP being objective,

risk-informed, understandable, and predictable as well as its success in meeting the Agencyís

performance goals of maintaining safety, increasing public confidence, making NRC activities

and decisions more effective, efficient, and realistic, and reducing unnecessary regulatory

burden on stakeholders. On a periodic basis, the self-assessment program collects information

from various sources, including the Reactor Program System (RPS), the inspection program, the

ROP performance indicator (PI) program, additional industry level PIs, periodic independent

audits, stakeholder surveys, and public comment. Based on this information, an assessment of

ROP success is performed in the programmatic areas of performance indicators, inspection,

significance determination process, and assessment. Listed below are discussions of the selfassessment metrics associated with the SDP. Due to the lack of historical data, an in-depth

analysis is not possible in most cases. However, where appropriate, some conclusions were

reached.

Objective

The programís design to focus on findings that minimize reliance on subjective inspection input

is measured by the number of SDP packages rejected by the SERP because the

characterization of the finding and the proposed preliminary significance determination does not

meet the guidelines outlined in IMC 0609. During the first year of implementation, no SDP

packages have been returned for not meeting established standards for technical accuracy

and/or format. However, audit findings indicate that some inspection findings processed through

the SDP lacked documentation detailing the actual assumptions used in coming to a significance

determination. The required information was obtained by the SERP without returning the subject

packages to the region. Revisions to inspection report documentation guidance completed

during the initial implementation period should improve the SDP implementation process.

Risk-informed

The SDP is considered to be risk-informed by design. No metrics were specifically established

to evaluate this attribute.

Understandable 

The ability of the SDP to be understood by the NRC staff responsible for implementing the

program, and the ability of the SDP packages to communicate relevant information are

measured by internal audits and stakeholder feedback. The available metrics data indicates that

improvements are needed to ensure that SDP decisions are auditable and can be easily

reconstructed using publicly available documentation. Although all of the necessary information

is captured, there is not a single source of information that will allow for simple auditing of the

process.

Feedback from survey provides original input in this area. Additional detail is provided in the

Stakeholder Feedback section below.
4

Predictable

The SDP is considered to be predictable if it is consistently implemented as defined. The

measures used to determine predictability are: (1) SDP results are reproducible, (2) SDP

standards and processes do not require substantial changes, and (3) SDP tools reflect current

plant design and licensee operating practices, and the SDP results are perceived to translate to

the same level of concern for all cornerstones. 

Initial results indicate that the process resulted in reproducible outcomes. However, based on

the stakeholder feedback and initial implementation experience, it was recognized that some

revisions to the current SDP guidance are necessary in some instances to reduce the complexity

of the process and in others to produce appropriate regulatory response. Some enhancements

to SDP guidance for Fire Protection, Security and Safeguards, as low as reasonably achievable

(ALARA) programs, and Transportation have been completed, others are under consideration. 

Stakeholder surveys of findings from various inspection areas and cornerstones assessed

through the applicable SDP indicate that the classification of the findings by safety significance

and corresponding colors led to the expected response band of the regulatory process.

Maintains Safety

The SDP helps to maintain the current level of safety if the SDP is effective in focusing NRC and

licensee attention on safety significant issues. This attribute of the SDP is measured by the

tracking the total number of over-conservative and non-conservative initial SERP results. During

the initial implementation period, no non-conservative findings were identified. This is expected

because the SDP incorporates generally conservative assumptions and therefore the results

tend to be conservative. The potential implication is that the process is over-conservative, thus

repeatedly causing reduction in preliminary characterization of significance of findings following 

use of more accurate assumptions. Additional program experience must be gained before the

optimum balance between conservative and more accurate preliminary SDP assessment can be

determined. The metric results also indicate that the SERP process has been effective in

identifying over-conservative initial significance determinations prior to final licensee notification.

Efficiency, Effectiveness, and Realistic

The efficiency of the SDP is measured by evaluating whether NRC resource expenditures are

commensurate with the significance of the issues identified. This attribute is measured by

tracking: (1) NRC interactions with the licensee, (2) whether SDP timeliness goals are met, and

(3) the accuracy of SDP results.

The metrics indicates that SDP results have been accurate. The timeliness goals established in

MC 0609 have not been met. Specifically for findings greater than Green, the time required to

process the finding through the SDP was significantly longer than originally was expected by the

staff. Some of the delay can be attributed to a learning phase by both the staff and the

licensees. However, additional evaluations to ascertain the effort required to arrive at

preliminary significance assessments of the findings is necessary to ensure that the established

timeliness goals are realistic, and appropriate adjustments made as necessary. It is expected

that through continued use of the SDP, staff proficiency levels will improve. The results indicate

that the licensees have also become familiar with the process and have indicated that SDP
5

outcomes have been predictable and consistent. Refer to substantial stakeholder feedback

below. 

Enhances Public Confidence

Although public confidence is not readily measurable directly, the SDP is believed to enhance

public confidence when information is timely and communicated in a way that demonstrates that

the NRC understands the plantís performance. This attribute of the SDP is measured by

verifying the accuracy of facts and the timeliness of communicating the facts to the licensee. As

previously stated, the inability to meet established timeliness criteria for developing preliminary

significance determinations and the SDPís tendency to provide over-conservative preliminary

results will be evaluated to determine if changes to the process are necessary.

Reduces Unnecessary Regulatory Burden

The SDP program reduces unnecessary burden by focusing NRC and licensee resources

commensurate with the significance of findings. This attribute is measured by licensee feedback

on the appropriateness of resource expenditures. Metrics results indicate that the licenseeís

have accepted the SDP process and believe that the SDP provides for realistic and effective use

of NRC and licensee resources.

Stakeholder Feedback (Internal/Public/Industry)

The staff captured internal feedback, regional and resident inspector comments as well as

comments by NRC managers, through a standardized feedback process. Issues relating to the

SDP were evaluated, incorporated, or scheduled to be incorporated as deemed applicable and

appropriate. External feedback was solicited and received through workshops, routine meetings

with NEI, at industry forums such as fire protection, that were attended by IIPB. Public

comments were received at several workshops. Licensees offered significant criticism during

the development, presentation and assessment of all findings, of all levels of safety significance. 

The staff also received comments by the Initial Implementation and Evaluation Panel (IIEP). 

The IIEP represented a cross section of stake holders tasked with the assessment of the first

year of implementation of the ROP. 

Applicable to All Cornerstones.

In general, the time spent by NRC staff on the development of the preliminary significance

determination was greater than originally anticipated, and often protracted, resource intensive,

interaction with licensees ensued the identification of the finding.

The IIEP recommended that the staff should carefully evaluate any potential changes against

Agency goals. For example the need for the development of additional SDPs should be

questioned in terms of increased regulatory burden versus benefits in maintaining safety. The

IIEP recommended that the staff Improve the process for determining risk characterization of

issues so that they are expedient, scrutable, and understandable to all stakeholders. 

Additionally there were several requests to clarify the definition of performance deficiency.
6

Reactor Safety SDP

The majority of internal and external stakeholders expressed concern regarding the complexity

of the risk-informed assessment of inspection findings. Additionally, a significant number of

stakeholders questioned the reliability of the initial Risk-Informed Inspection Notebooks, the

Phase 2 process of the SDP, that were based on IPEs and were known to under-estimate the

potential significance of inspection findings. This resulted in the time consuming detailed risk

analyses of each inspection finding that passed through the Phase 1 screening process as

potentially greater than Green, while site visits were conducted to collect the information

required to improve the notebooks. Recommendations were made to continue efforts that

provide improved and standardized risk analysis tools beyond the Phase 2 Notebooks. 

Stakeholders also expressed concerns regarding the lack of external events risk data,

application of operator response credit, lack of detailed guidance for assessing concurrent

performance deficiencies and the method to appropriately address long standing design basis

deficiencies. Stakeholders concerns with the guidance provided for assessing shutdown and

containment related findings were also noted. 

Fire Protection SDP

Concerns with the complexity of the Fire Protection SDP were expressed by some stakeholders. 

Lack of detailed written guidance for fire scenario development and fire brigade performance

evaluation, both key inputs to the SDP, resulted in extensive expenditure of SRA, risk analyst,

and fire protection engineer resources. There were also several examples of disagreement

between the NRC staff and licensees on the fire protection design bases such as in the

evaluation of fire induced circuits failures of associated circuits that would result in hot shorts

and spurious equipment actuations.

Maintenance Rule SDP

The feedback received from stakeholders indicated that the significance of (a)(4) Maintenance

Rule performance issues cannot be assessed with the existing Reactor Safety SDP. 

The existing Reactor Safety SDP does not clearly address issues related to risk assessment and

risk management associated with performance of maintenance activities. The existing SDP

Phase 1 worksheet may inappropriately screen risk-significant plant maintenance configurations

to Green. In addition, the Phase 2 site-specific inspection notebooks lack the necessary level of

detail and completeness to assess maintenance configurations with multiple equipment out-ofservice. The licensees are already using Phase 3 type analyses (and tools) to assess the atpower risks of maintenance configurations.

The staff has developed a draft SDP for evaluating the significance of Maintenance Rule (a)(4)

performance issues (failure to perform an adequate risk assessment and failure to manage risk)

for at-power and shutdown conditions. In part, the metrics used for the proposed SDP are

based on configuration-specific incremental core damage probability (ICDP), incremental large

early release probability (ILERP), and degradation of key safety functions. Findings regarding

inadequate risk management are proposed to be assessed qualitatively. The proposed SDP

was presented to the internal and external stakeholders during a lessons learned public

workshop in order to obtain their feedback and recommendations. During the workshop the

stakeholders raised a number of concerns regarding the proposed SDP and provided several
7

recommendations for improvement. The staff will continue to develop a Maintenance Rule (a)(4)

SDP and work closely with stakeholders in resolving the issues. 

Transportation SDP

Based on inspection findings, the Transportation SDP was found to result in inappropriately high

significance in certain situations. 

Physical Protection SDP

The staff determined that force-on-force inspection findings posed a significant challenge in

applying the Physical Protection SDP (PPSDP). These challenged were associated with the

linkage of the PPSDP to the Reactor Safety SDP. Given the inability to appropriately assess the

probability of a successful adversary attack on the facility, the staff found that the PPSDP would

generate inappropriately high outcomes for certain force-on-force exercise findings. The linkage

to the Reactor Safety SDP also caused the process to focus on operator recovery of a damaged

facility rather than on the security program.

Shutdown SDP

Internal stakeholders noted that the Shutdown SDP does not include reactivity guidelines, and

the associated check lists do not correlate well with the base line inspection guidance. 

Shutdown related inspection findings continue to require a Phase 3 assessment by a risk

analyst, which involves the extensive use of regional SRAs and Headquarters based risk

analysts. A Phase 2 inspection tool that will allow inspectors to make a more detailed

assessment of safety significance, is under development. 

Occupational Radiation Safety SDP

Stakeholders noted that the Occupation Radiation Safety SDP appears to exclude all hot particle

exposures, including whole body (deep dose equivalent) as well as skin dose from hot particles

not subject to the enforcement discretion. This is not consistent with the Commission

enforcement policy. 

The current Occupational Radiation Safety Group two screening criteria in MC 0610*, Power

Reactor Inspection Reports,î screens out all ALARA issues identified at plants that have a rolling

3-year average collective dose that does not exceed the screening criteria. The unintended

consequence of this screening is that, for these licensees, a failure to implement appropriate

ALARA procedures or engineering controls, cannot be documented in the inspection report. The

process also resulted in a number of unintended outcomes such as: (1) inspectors note that they

cannot document a finding for these plants no matter how egregious the performance issue they

identify, (2) the Institute of Nuclear Power Operations (INPO) pointed out that the screening

criteria can be viewed as a de facto definition of ALARA for power reactors, and (3) some

licensees, with a 3-year average collective dose above the screening criteria, feel they are at risk

for multiple White findings from one poor outage, which would result in NRC action that may be

too harsh considering the Agencyís previous enforcement history with ALARA. 

The question was raised whether a violation associated with multiple White findings (escalated

enforcement) is too harsh a response considering the Agencies previous enforcement history
8

with ALARA. This issue was discussed at the March 2001 External Lessons Learned Workshop. 

The consensus opinion of the participants indicated that although no scenario could be

constructed where a single ALARA failure should result in a degraded cornerstone (i.e., a Yellow

finding), it is appropriate for a degraded cornerstone to be assessed for licensees that have

multiple White findings in the assessment period. 

Significance and Enforcement Review

Attachment 0609.01 to IMC 0609, designed to provide guidance on the conduct on SERPs,

Regulatory Conferences, and for the issuance of preliminary and final decisions on SDP and

enforcement, was not formally issued at the time of initial implementation. However, the staff

implemented a draft of the document that continued to undergo enhancements based on

lessons learned from the implementation of the oversight process throughout the period. Final

comments are being solicited from all internal stakeholders. The document will be issued during

the Summer 2001.

Program Changes Made as a Result of Lessons Learned

Reactor Safety SDP

Phase 2 Notebooks:

The staff determined that the most important change to the reactor safety SDP, currently

ongoing, is the development and issuance of the plant/site specific Risk-informed Inspection

Notebooks. The notebooks incorporate site specific information collected and verified during

site visits by SRAs and NRC Headquarters risk analysts. The notebooks include the Phase 2

worksheets of the SDP giving the inspectors the tool they need to assess inspection findings in

the field. All of the 70 notebooks will be completed and issued for use by the inspectors by the

end of July 2001. Following issuance, the last important step remaining is a benchmarking

process that compares the results of the SDP Phase 2 notebooks with the licensee risk model

results to ensure that the SDP is generally conservative. The latter effort requires site visits that

started in April 2001. To date, several benchmarking efforts indicate that SDP results using the

notebooks are conservative as expected. However, there is also indication that some licensees

did not provide adequate comments during the initial site visits resulting in some notebooks that

do not accurately reflect certain accident sequences. While the process is ready for full

implementation by the inspectors, continued SRA involvement and verification by the SERP

remain important aspects of the process. The benchmarking of the notebooks continues at a

rate of two site visits per month.

Credit for Operator Actions:

The staff revised the credit given for operator actions by changing from two categories, recovery

under normal conditions and recovery under stressful conditions, to four categories of Human

Error Probabilities (HEP) ranging from 0.1 to 1E-4. Phase 2 worksheets are in the process of

being updated to include plant and site specific information. 
9

Transportation SDP:

For the Transportation / Part 61 area (i.e., classification of radioactive materials for shipment and

disposal); the staff accepted a proposed revision to the Part 61 portion of the SDP that was

submitted by NEI on June 12, 2000. This portion of the SDP assesses the risk from the failure

of a licensee to correctively classify a radioactive waste shipment (i.e., under classify waste

shipments by assigning a less restrictive classification, such as classifying Class B waste as

Class A waste). Prior to the change, all findings that involved radioactive materials being under

classified received a White finding. NEI proposed that the SDP flowchart be expanded to offer

additional decision questions to refine the SDP process into separate steps that correspond to

graded levels of risk to the public. NEI maintained that there are cases where there is low risk to

workers, members of the public, the waste disposal facility, and the environment. For such

cases, the SDP should reflect this low risk and the SDP should be include a risk assessment of

Green. The NRC agreed with the proposed revision. Adjustments to the SDP and associated

flowcharts were made and subsequently tested to assure that the SDP will screen inspection

findings to the appropriate licensee and NRC response bands. It was also agreed that a White

finding was still appropriate for cases that involved Class C waste and for Class B waste that did

not meet the requirements of 10 CFR 61.56.

Physical Protection SDP:

The staff determined that linking the Reactor Safety SDP to the PPSDP would result in reactor

safety findings that were at a higher level of NRC response and engagement than is warranted. 

The problem was most noticeable when processing simulated equipment damage that resulted

from mock adversary actions (force-on-force exercises) not prevented by security measures. To

remedy the problem, the staff recommended an interim PPSDP to be used to evaluate force-onforce exercise findings. The Commission accepted the staff recommendations and provided

additional guidance in COMSECY-00-0036, which directed the staff to continue the effort to

develop more permanent guidance. This guidance will be developed and finalized based upon

stakeholder feedback and experience with the interim PPSDP.

Fire Protection SDP:

The staff decided to temporarily suspend the review of fire induced circuit failures of associated

circuits, as part of the triennial inspection program. This is in response to an industry initiative to

improve the understanding of the behavior of electrical circuits while exposed to a fire. Details of

the ongoing study, which involves fire tests of various circuit arrangements, are being closely

followed by the staff. This effort is expected to last approximately one year. Upon completion,

new inspection guidance based on the evaluation of the test results will be incorporated into the

inspection procedure.

Near the end of the initial implementation period additional information to enhance the

inspectorsí ability to assess functionality of fire protection defense in depth components such as

fire brigade, automatic detection and protection performance as they relate to the SDP, was

incorporated into an attachment to the Fire Protection SDP.
10

Long-term Program Changes Planned 

All Cornerstones

The staff will continue to enhance the SDP through improving existing, and developing new

significance determination tools. However, changes to the process will be closely scrutinized to

ensure the balance between increased regulatory burden and benefit in maintaining safety and

Agency effectiveness. For example, changes to the process to address the ìNo Colorî category

of findings are under extensive review to assure that the change improves the process. The

development of a maintenance activity related SDP is under similar evaluation by the staff. The

staff will continue efforts to reduce time spent on preliminary safety significance determinations.

For example, the site specific notebooks for the reactor safety SDPs, and enhancements to the

Fire Protection SDP, should result in improvements of timeliness and reduction of resources. 

Incorporating comments into the SDPs associated with other cornerstones should also result in

improvement of efficiencies.

Reactor Safety SDP

The staffís capability to assess the impact of external events on operating reactor safety related

issues needs to be improved. To this affect the staff requested the Office Nuclear Regulatory

Research to conduct scoping evaluation of the specific plants that may warrant development of

improved external events tool. 

The staff also needs to improve guidance provided for the assessment of concurrent

deficiencies. Existing guidance in IMC 0609 provides for concurrent performance deficiencies to

be assessed collectively to determine total contribution to change in the core damage frequency

(CDF). However, each concurrent performance deficiency should be assigned a color

individually. When multiple issues stem from the same common cause, risk analysis techniques

account for the potentially greater risk significance of the combined issues (i.e., delta CDF or

delta large early release frequency (LERF)). However, in all cases the Action Matrix was

designed to combine multiple issues for determining the appropriate NRC response. Although it

is expected that in most cases the regulatory response will not differ if multiple issues are treated

either in combination or independently, ROP guidance is being evaluated for enhancement with

regard to applying combined risk results of multiple issues in a manner that is appropriate for

use in the ROP Action Matrix.

Fire Protection SDP

The staff continues to develop the methodologies that will allow inspectors to determine fire

scenario development and improve the accuracy of site specific data used in the assessment of

risk associated with findings, such as ignition frequency. The results of these efforts will

undergo functional evaluation and be made available to the inspectors by the end of 2001.

Shutdown SDP

The development of a Shutdown SDP Phase 2 process, a tool that will allow for the conservative

assessment of the risk associated with a finding by the inspectors in the field, is under way with

the assistance of risk analysts at Brookhaven National Laboratories. The effort is expected to
11

be complete by the end of 2001. The results of the Phase 2 process should provide colors that

correspond to our existing definitions of significance based on incremental increase in core

damage frequency and expected Action Matrix Response. 

Occupational Radiation Safety SDP

The staff is evaluating options to revise the Radiation Exposure Control section of the

Occupational Radiation Safety SDP to clarify how the SDP reflects the Commissionís policy on

enforcement discretion for skin over exposures from hot particles (or discrete radioactive

particles). 

The current Occupational Radiation Safety Group 2, Question 1 (concerning ALARA) will be

removed. The rolling 3-year average collective dose comparison will be used to adjust the

baseline inspection level of effort (hours of inspection) and possibly as a SDP significance

criteria. The other criteria (e.g., ìgreater than 5 person-rem,î and ìactual dose exceeds

estimated [planned] by 50%î ) will be incorporated into the baseline inspection procedure as

guidance on what constitutes a ìmore than minorî issue. A new Group 2 Question will be

developed to clarify that the basis for a finding in this area is an ALARA program failure that

results in ìunintended collective doseî for a job, similar to the current Occupational Radiation

Safety Group 2, Question 2.

Containment SDP

Interim guidance for assessing significance of containment related inspection findings is

provided in Appendix H to IMC 0609. Appendix H is based on a draft (August 2000) technical

basis document by Brookhaven National Laboratory. The technical basis document is being

revised to address residual technical issues and incorporate additional guidance based on

feedback and questions from regions. Appendix H will be updated and expanded following

revision of the basis document. The target completion date is December 2001. Issues being

addressed in the revision include: additional screening guidance for BWR main steam isolation

valve leakage and suppression pool bypass findings, refinement of failure criteria for ice

condenser hydrogen igniter and ice bed doors, and extension of the guidance to cover

containment integrity during shutdown operations.

Spent Fuel SDP.

Spent fuel issues are currently evaluated by a risk analyst using Phase 3 analysis. The

feasibility for the development of an assessment tool that can be implemented by the inspector

is under evaluation. 

Additional Work 

Occupational Radiation Safety SDP. The proposed revised basis for ALARA performance

assessment will require additional public meetings with NEI, and other stakeholders, to develop

a Group 2 Question and clearly define applicable terms such as ìunintended collective doseî

and ìa jobî. 
ASSESSMENT PROCESS

Task Lead: Bob Pascarelli, NRR/IIPB

Attachment 9
1

Assessment Program

Open Issues Addressed

There were several open issues and commitments that the staff continued to evaluate and work

on during ROP initial implementation. Several of the issues involved longer term items that the

staff identified in Commission paper SECY-00-0049 as needing additional work following initial

implementation. The Commission directed the staff to consider several additional issues in the

SECY-00-0049 Staff Requirements Memorandum (SRM) dated May 17, 2000. And several

recommendations for staff consideration during initial implementation were made by the Pilot

Program Evaluation Panel (PPEP) in their final report. The issues and recommendations for the

assessment process are as follows: 

SECY-00-0049

Assessment program not fully exercised during the pilot program

There were certain aspects of the assessment process, such as the End-of-Cycle (EOC)

Review Meeting and Agency Action Review Meeting (AARM), that were not exercised

during the pilot program. However, the EOC Review meetings were exercised at the pilot

plants during May 2000. Lessons learned from the pilot plants have been incorporated

into the latest revision of IMC 0305 ìOperating Reactor Assessment Programî in order to

support these meetings at all plants in May 2001. Additionally, the first AARM will be

conducted in June 2001 and appropriate lessons learned will be incorporated into draft

Management Directive (MD) 8.14 ìAgency Action Review Meetingî. 

SECY-00-0049 SRM

Deviations from the Action Matrix

The Commission expressed concern in their response to the staff on the results of the

ROP pilot program (SRMís dated 03/28/00 and 05/17/00) regarding the use of deviations

from the Action Matrix. Specifically, the Commission directed the staff to:

ï Obtain pre-approval by the EDO, during the initial implementation phase, prior to

issuing Action Matrix deviations.

ï Provide in the report to the Commission on the results of the first year of the

ROP, a summary of the deviations from the Action Matrix, recommendations on a

method to assure Agency-wide consistency when deviating from the Action

Matrix, and recommendations on how to keep the Commission currently informed

of such deviations.

ï Inform the Commission, during initial implementation, whenever the staff

determines a deviation is warranted.

ï Develop a process for deviating from the Action Matrix during the initial

implementation phase of the ROP.
2

The staff revised IMC 0305 to clarify expectations for deviating from the Action Matrix. 

IMC 0305 now requires that the Executive Director for Operations (EDO) pre-approve all

deviations and that the Commission be informed of any such deviations. This ensures

that deviations that are granted are appropriately considered and are rare. The Agency

is considering the development of further guidance that would describe the types of

issues that may be considered for deviations from the Action Matrix. Although deviations

from the Action Matrix were considered on a few occasions during initial implementation

of the ROP, none were issued. 

Impact of cross-cutting issues on the ROP

The Commission directed the staff to discuss every case in which enhanced NRC action

was taken based on cross-cutting issues in this report. During initial implementation of

the ROP, enhanced NRC action was not taken by the Agency based solely upon findings

associated with cross-cutting issues. 

PPEP Final Report

Significance of inspection findings

A recommendation from the PPEP was that the staff clarify guidance relative to the use

of the Action Matrix when the significance of the inspection findings are under review at

the time the assessment is published (mid-cycle or annual). Guidance has been added

to IMC 0305 to discuss regional office actions when there are inspection findings whose

significance have not been determined at the end of the assessment cycle.

For example, IMC 0305 states that the date used for consideration in the assessment

program is the end-date of the pertinent inspection period for the inspection finding. 

After final determination of the significance of the finding, the regional office shall refer

back to the appropriate date to determine if any additional action would have been taken

had the significance of the inspection finding been known at that time. Additionally, the

mid-cycle or annual assessment letter would state the existence of potentially greater

than Green findings and that additional inspection may be warranted if the finding is

determined to be greater than Green. 

Self-Assessment Results

The ROP self-assessment program evaluates the overall success of the ROP being objective,

risk-informed, understandable, and predictable as well as its success in meeting the Agencyís

performance goals of maintaining safety, increasing public confidence, making NRC activities

and decisions more effective, efficient, and realistic, and reducing unnecessary regulatory

burden on stakeholders. On a periodic basis, the self-assessment program collects information

from various sources, including the Reactor Program System (RPS), the inspection program, the

ROP performance indicator (PI) program, additional industry level PIs, periodic independent

audits, stakeholder surveys, and public comment. Based on this information, an assessment of

ROP success is performed in the programmatic areas of performance indicators, inspection,

significance determination process (SDP), and assessment. Listed below are discussions of the
3

self-assessment metric results for the assessment program. Due to the lack of historical data,

an in-depth analysis is not possible in most cases. However, where appropriate, some

conclusions were reached.

Objective

The programís focus on objective assessment of licensee performance with minimal

subjective inputs is measured by: (1) the number of deviations from the Action Matrix, (2)

the number of appeals of risk-significant SDP outcomes, and (3) the number of

significant departures (not including timeliness goals) from IMC 0305 and IMC 0350,

ìStaff Guidelines for Oversight of Operating Reactor Facilities in an Extended Shutdown

as a Result of Significant Performance Problemsî. The assessment program was

considered to be objective based on:

(1) There have been no deviations from the Action Matrix during the ROP.

(2) There has been a challenge to 3 White findings in the Occupational Radiation Safety

Cornerstone at Callaway Nuclear Station. This appeal was processed in accordance with

IMC 0609 ìSignificance Determination Processî and the original determination was

subsequently upheld by the Agency. 

(3) The review of assessment letters during the ROP have revealed very few

discrepancies and those discrepancies were limited to the first quarter of ROP initial

implementation.

Risk-Informed 

The assessment program is considered to be risk-informed if the actions taken by the

regions are commensurate with the overall plant risk as reflected in the Action Matrix. 

The assessment program was found to be risk-informed based on the fact that the

actions that have been taken by the regions have been consistent in all cases with the

Action Matrix.

Predictable

The assessment program is considered to be predictable if: (1) the regions arrive at the

correct Action Matrix column designation for each plant and subsequently take the

appropriate actions for the same set of circumstances, (2) the resources that are

expended on the assessment program are appropriate and consistent across regions, (3)

there are limited additional actions recommended by senior Agency management at the

AARM, (4) the timeliness goals, established in IMC 0305, and ADAMS availability goals

were met, and (5) there were few unplanned, substantive revisions to IMC 0305 and IMC

0350. 

Based upon the staffís review, the regions arrived at the correct Action Matrix designation

for each plant and subsequently took the appropriate actions. Regional resource

expenditures on the assessment program were incomplete due to the unavailability of

hours expended on the preparation and conduct of the End-of-Cycle Review meetings as

well as those hours used to prepare the annual assessment letters. Additionally, analysis
4

is complicated by the different methods in which individual regions have captured actual

expenditures. This analysis is discussed in the ìROP Resource Analysisî attachment of

this paper. The first AARM will be conducted in June 2001 and there is no data with

which to evaluate the associated metric. The review of assessment letters during the

ROP and the posting of these letters in ADAMS have revealed very few discrepancies. 

Timeliness goals have been met with very few exceptions and there has been only one

unplanned, substantive revision to IMC 0350. The staff believes that, based on the

results of data collected to date, the assessment program has been predictable.

Maintains Safety

The assessment program maintains safety if: (1) the regulatory attention given to

licensees with performance problems is appropriate and (2) NRC actions are timely by

measuring the lag time between the issuance of an assessment follow-up letter

(discussing a safety significant PI or inspection finding) and the completion of the

associated supplemental inspection.

The staff has received a majority of positive responses on the survey with regards to the

appropriateness of regulatory attention given to licensees with performance problems.

Additionally, the staff has received some limited data on the timeliness of addressing

risk-significant issues. Based upon these results the program office is unable to draw

any conclusions with the amount of data that has been reported.

Efficient, Effective, and Realistic

The efficiency and effectiveness of the assessment program is measured by evaluating:

(1) the appropriateness of regulatory attention given to plants with performance

problems, (2) the efficiency with which potentially risk significant issues are processed

once a final risk determination is completed, (3) the number of timeliness goals that are

not met, and (4) the appropriateness of the level of management involvement with

respect to overall plant performance. Based upon our evaluation of these results, the

assessment program was found to be efficient, effective, and realistic.

Enhances Public Confidence

The assessment program is considered to enhance public confidence if the process is

timely and predictable, the Agencyís inspection and assessment reports are

understandable, and if few deviations are taken from the Action Matrix. Based upon the

feedback from surveys and evaluation of metric data, the assessment program was

found to enhance public confidence. 

Reduces Unnecessary Regulatory Burden 

 

The assessment program is considered to reduce unnecessary regulatory burden if: (1)

the ROP focuses licensee resources on areas of greatest significance and minimizes

rework or duplication, (2) the ROP minimizes inconsistencies between the regional

offices, and (3) each region takes similar actions for plants in the same column of the

Action Matrix. Based upon the results of the external survey, the assessment program

was found to reduce unnecessary regulatory burden. 
5

 

Stakeholder Feedback (Internal/Public/Industry)

A. Treatment of Inadequate Corrective Actions

It was identified during initial implementation that inadequate corrective actions for a PI

were not treated consistently with inadequate corrective actions for inspection findings. 

IMC 0305 states that an inspection finding is normally carried forward in the assessment

process for a total of four calender quarters. IMC 0305 requires that an inspection

finding is not to be removed from consideration of future Agency actions (via the Action

Matrix) until the identified weaknesses in the root cause evaluation have been corrected

by the licensee. There is no such provision for safety-significant PIs. However, it was

always intended that the original performance deficiency (whether it is an inspection

finding or a PI) will not be removed from consideration of future Agency actions (i.e. the

Action Matrix) until the licensee has corrected the issue.

This situation occurred at Kewaunee Nuclear Station last year. The licenseeís

evaluation of a Yellow Alert and Notification System PI was determined to be inadequate

after two supplemental inspections were conducted on this issue. In this case, the staff

issued a parallel Yellow inspection finding that corresponded to the original performance

deficiency from the Yellow PI.

This issue was discussed at the External Lessons Learned Workshop that was held

between March 26-28, 2001. The goal of this session was to solicit licensee input and

arrive at a consensus approach. The Agency asked the workshop attendees to provide

their input on several implementation issues such as:

ï How much of an opportunity (if any at all) should be provided to the licensee to

correct the deficiencies in the evaluation prior to the issuance of the parallel

inspection finding.

ï What criteria should the Agency consider in evaluating whether a parallel

inspection finding should be issued.

ï How to provide licensees the opportunity to provide their perspective on the

identified weaknesses prior to issuance of the proposed parallel finding.

ï Whether the finding is a placeholder for the original performance deficiency or a

separate finding directed against the CAP that would require a separate

evaluation and corresponding supplemental inspection. This question would

apply whether the original performance deficiency was an inspection finding or a

PI.

ï The appropriate time and method to remove the inspection finding from

consideration in the assessment program. 

The participants in the workshop reached alignment on the following implementation

issues:
6

ï The performance issue should remain open as a finding of the same color as the

original performance issue and be considered as a finding against the corrective

action program.

ï There should be a strong causal link between the original performance deficiency

and the ineffective corrective actions in order to issue the parallel inspection

finding.

ï That the licensee should be provided an opportunity to provide input at the

supplemental inspection exit meeting. 

B. Role of ìNo Colorî findings in the inspection and assessment programs

The staff has received feedback from external stakeholders throughout initial

implementation of the ROP that the role of ìNo Colorî findings in the oversight process is

not clear and has contributed to program inconsistencies. There was also some concern

that the existence of ìNo Colorî findings represented a weakness in the ROP. ìNo Colorî

findings are associated with specific extenuating circumstances as discussed in IMC

0610*, ìPower Reactor Inspection Reportsî. Early guidance to the regions was nonspecific, which resulted in the significance of these findings being confusing to

stakeholders and their role in the assessment program unclear. Clarification to existing

program office guidance appears to have reduced the number of ìNo Colorî findings. 

The assessment program was not designed to include ìNo Colorî findings and their

existence may undermine public confidence in the ROP. 

The staffís intention was that ìNo Colorî findings are greater than minor findings of a

regulatory nature (e.g., do not impact a cornerstone) that can not be evaluated by the

current SDP. These findings should only include violations of regulatory requirements or

notable adverse performance trends or patterns associated with cross-cutting issues. 

These findings were expected to be small in number.

This issue was discussed at the External Lessons Learned Workshop. The goal of this

session was to provide a proposed solution followed by a group discussion of the

proposal. The workshop participants could not reach consensus on the staffís proposal

to color these findings as Green. Some of the participants were concerned that the

proposal would give the appearance that these findings had been evaluated through the

SDP and had the same risk significance as other Green findings. As an interim

measure, the group recommended that the Agency: (1) address the perception problems

with ìNo Colorî findings, (2) continue to monitor ìNo Colorî findings and adjust program

office guidance to minimize the number of these findings, and (3) evaluate these findings

to determine if they represent a weakness in the ROP. Over the longer term, the staff

will continue to evaluate the role of ìNo Colorî findings in the ROP. 

C. Role of historical issues in the Action Matrix

Historical issues, such as those that are captured during design inspections, are not

necessarily reflective of current licensee performance. In particular, those issues already

identified by and appropriately addressed by the licensee may be reflective of good

licensee performance. The assessment program determines appropriate Agency actions
7

based upon the most current licensee performance. The most current licensee

performance is determined on a quarterly basis by reviewing the current PI and

inspection results. 

This issue was discussed at the External Lessons Learned Workshop. The goal of this

session was to identify possible approaches with convergence towards a consensus

viewpoint. The staff asked the participants to consider the following three fundamental

questions when considering historical issues in the reactor oversight process:

ï Whether the ROP should be reflective of current plant conditions or current

licensee performance.

ï What types of issues would not be considered reflective of current plant

conditions or licensee performance.

ï Can the approach for treating historical issues be structured such that it does not

create disincentives for licensees aggressively seeking to identify and resolve

issues. 

The consensus approach from the breakout session was that an issue with current risk

significance (e.g., greater than Green) is a performance issue, regardless of whether it

reflects current licensee organizational performance. Licensees and NRC staff had

differing views on the desirability of creating disincentives for licensees and how these

findings should be treated in the assessment process. Consensus was not reached on

whether to: (1) treat these findings like all other findings in the assessment process or

whether to define a class of findings that may warrant discretion or (2) keep the finding

open for four quarters or remove it once the appropriate corrective actions have been

completed. 

D. Purpose of a Regulatory Conference and a Regulatory Performance Meeting

The staff has received feedback throughout initial implementation of the ROP that there

was some confusion between the purpose of a Regulatory Conference and a Regulatory

Performance Meeting. The purpose of a Regulatory Conference is to gain a complete

understanding of the significance of an inspection finding as well as information pertinent

to understanding any apparent violations. In some cases, this requires a technical

discussion of the probabilistic inputs and assumptions used to characterize the risk

significance of the issue. The role of NRC and licensee management has changed from

their role during enforcement conferences held in the past. The Regulatory Conference

was not intended to be a forum for a discussion of the adequacy and effectiveness of

licensee corrective actions. However, a significant amount of attention has been devoted

to the discussion of licensee corrective actions during Regulatory Conferences. This

may be due in part to the efficiency in conducting a detailed discussion of corrective

actions at this meeting in conjunction with a discussion of the significance of an

inspection finding and any apparent violations.

This issue was discussed at the External Lessons Learned Workshop. The staff

proposed that the guidance in IMC 0609, attachment 1 be revised to more accurately
8

characterize the purpose of the meeting and to clearly differentiate it from the Regulatory

Performance Meeting. 

Licensees are normally offered an opportunity for a Regulatory Conference to discuss

potentially safety significant inspection findings, whether or not violations are involved. A

secondary purpose of the meeting is to provide an opportunity to address any apparent

violations that may be associated with the finding. This meeting enables the Agency to

obtain the licenseeís perspective in order to come to a common understanding of the

facts and the significance of the findings. The Regulatory Conference is not a meeting to

negotiate sanctions or discuss the adequacy of any current or proposed licensee

corrective actions. If a licensee is in agreement with the issues then they may opt not to

have a Regulatory Conference. 

Regulatory Performance Meetings are held between licensees and the Agency to discuss

the effectiveness of a licenseeís root cause evaluation and corrective actions associated

with safety significant inspection findings after the completion of the associated

supplemental inspection. Each safety significant assessment input shall be discussed in

one of the following forums listed below in order to arrive at a shared understanding of

the performance issues, underlying causes, and planned licensee actions. These

discussions may take place at supplemental inspection exit meetings between the

Agency and the licensee, conference calls, or public meetings. This meeting should be

documented in an inspection report or a public meeting summary as appropriate. NRC

management, as specified in the Action Matrix, conducts the Regulatory Performance

Meeting.

The participants generally agreed to the staffís proposal and provided the following

recommendations: (1) due to possible public perception problems, the licensees should

be allowed to address corrective actions after the discussion of safety significance and

any apparent violations at the Regulatory Conference, (2) someone other than the

regional enforcement coordinator should open the meeting, and (3) the Regulatory

Conference should be chaired by the appropriate level of NRC management (e.g., in

accordance with the Action Matrix). Licensees also recommended that the NRC

standardize the process for sharing the basis and assumptions of safety significant

findings prior to the Regulatory Conference.

E. Timeframe for counting an inspection finding in the assessment process

The Agency has received both internal and external feedback that there should be a

graded approach for removing inspection findings from consideration in the assessment

process. IMC 0305 states that an inspection finding is normally carried forward in the

assessment process for a total of four calender quarters. The program currently states

that an inspection finding will not be removed from consideration of future Agency actions

(via the Action Matrix) until the identified weaknesses in the root cause evaluation have

been corrected by the licensee. During development of the ROP, the period of four

quarters was selected as the appropriate time frame for counting inspection findings in

the assessment process. The date used for consideration in the assessment process is

the date of the end of the pertinent inspection period for the finding. Stakeholders

recommended the following approaches for consideration: (1) apply a graded approach

such that White inspection findings can be removed from consideration in the
9

assessment process when the regional offices are satisfied with the results of the

licenseeís corrective actions. Under this proposal, Yellow and Red findings would

normally remain for four quarters or (2) Apply a graded approach to all risk significant

(White, Yellow, or Red) findings.

An internal lessons learned workshop was held March 6-8, 2001 to discuss proposed

topics for the upcoming External Lessons Learned Workshop. This topic was discussed

with regional and Headquarters management in order to determine if a consensus

approach existed and to recommend whether this issue should be a topic at the External

Lessons Learned Workshop. This was not forwarded to the external workshop due to

the limited experience in the disposition of risk significant findings and a concern that the

regions may feel unwarranted pressure to close out a finding prior to a proper evaluation

of the cumulative circumstances surrounding the finding. However, the Agency

committed to re-evaluating the proposal after more experience is gained over the next

year of ROP full implementation. 

Program Changes Made as a Result of Lessons Learned

As a result of ongoing feedback between the program office and internal and external

stakeholders, a revision to IMC 0305 was issued on March 23, 2001 to reflect the following

major program changes:

ï Added guidance that the EDO is responsible for authorizing all deviations from the Action

Matrix (see SRM on SECY 00-0049).

ï Added a description of the roles of the Agency Allegations Advisor, Office of

Enforcement, Office of Investigations, Office of Research, and DLPM during the EOC

Review meetings. 

ï Added additional guidance for the regional offices regarding preparation for and conduct

of the Mid-Cycle and EOC Review meetings. 

ï Added an EOC summary meeting, between the regional offices and the Director of NRR

(or another member of the Executive Team), at the conclusion of the EOC Review

meeting.

ï Changed the requirement for issuance of the annual assessment letters to three weeks

after the EOC Review meeting from 1 week after the AARM meeting. This change was

made to recognize the fact that there should be no direct tie between the AARM and

issuance of the annual assessment letters.

ï Added a note to clarify that; ìregulatory actions listed in the Multiple/Repetitive Degraded

Cornerstone column of the Action Matrix are not mandatory. However, the regional office

should consider each of these regulatory actions when new performance information

regarding licensee performance becomes available.î This note has also been added to

the Action Matrix (exhibit 5 of IMC 0305).

ï Updated sample assessment letters to reflect lessons learned from the previous MidCycle and EOC Review meetings.
10

ï Added additional guidance regarding the process for approval of a deviation from the

Action Matrix. 

Long Term Program Changes Planned

ï Implement any lessons learned from the EOC Review meetings, the AARM, and the

annual public meetings with the licensees.

ï Add additional guidance on how to address supplemental inspection for performance

indicators when there are substantive inadequacies in the evaluation of the root causes

of the original performance deficiency, the extent of the performance problems, or the

associated corrective actions.

ï Continue to evaluate how historical licensee performance issues should be treated by the

Action Matrix.

ï Consider the development of further guidance that would describe the types of issues

that may be considered for deviations from the Action Matrix. 

ï Monitor the ROP to determine whether a graded reset approach for inspection findings is

appropriate.

ï Continue to monitor ìNo Colorî findings in the Inspection Program Branch inspection

report review process.

ï Clarify guidance in IMC 0609 attachment 1, IMC 0305, and the Enforcement Policy

regarding the purposes of the Regulatory Conferences and the Regulatory Performance

Meetings. 
ENFORCEMENT PROCESS

Task Lead: Chris Nolan, Office of Enforcement

Attachment 10
1

Enforcement

The Office of Enforcement (OE) is encouraged by the results of the first year of implementation

of the reactor oversight process (ROP) and in particular, the manner in which the significance

determination process (SDP) assesses the safety significance of inspection findings. OE finds

that the assessment and enforcement programs have reached a degree of integration never

before achieved under previous oversight programs. While implementation issues still remain,

OE has concluded that the ROP provides a sound framework for assessing the performance of

power reactor licensees and utilizes the NRCís enforcement authority appropriately through a

process that is both objective and transparent.

Open Issues Addressed

A. Enforcement Discretion Regarding the Submittal of Performance Indicator Data

The Enforcement Policy was revised on May 1, 2000, to describe the Commission's

decision to provide blanket discretion to power reactor licensees regarding the treatment

of performance indicator (PI) data submittals. This policy was developed in recognition

that some errors in the data submitted to the NRC were expected given the time

constraints to gather and submit historical data and the learning process for the

submission and review of quarterly PI data. This policy remained in effect through

January 31, 2001. The NRC staff is sensitive to stakeholder concerns regarding this

issue and the evolving nature of PIs within the ROP. Expiration of the interim policy has

not impeded the staff's ability to exercise enforcement discretion under Section VII.B.6 of

the Enforcement Policy on a case-by-case basis for inaccurate or incomplete PI data. 

Section VII.B.6 provides the NRC with the flexibility to refrain from issuing a civil penalty

or a Notice of Violation based on the merits of the case, including, but not limited to, such

factors as the significance of the violation and the clarity of the requirement. Exercising

enforcement discretion on a case-by-case basis under Section VII.B.6 takes into account

the completion of the initial learning curve and the recognition of ongoing PI development

activities. For example, the staff may consider exercising discretion if new PIs are

developed. The staff does not anticipate a significant number of cases involving

inaccurate or incomplete PI data. The staff notes that it exercised enforcement

discretion four times for inaccurate or incomplete PI data (three cases under the blanket

discretion policy and one on a case-specific basis). Further, OE will continue to have

oversight of cases involving inaccurate or incomplete PI data submitted to the NRC that

would cause a PI to change color. OE will continue to work with the regional offices and

NRR in determining whether enforcement discretion should be exercised for future PIrelated 10 CFR 50.9 violations. This position was communicated to external

stakeholders at the public External Lessons Learned Workshop held March 26-28, 2001,

with no negative feedback received.

Open Issues Unresolved

A. Timeliness of Escalated Enforcement Actions

OE has been monitoring the impact of the ROP on the timeliness with which the NRC

staff processes findings that are characterized as greater than Green and associated
2

violations, if applicable. Timeliness has been traditionally used by the OE as a measure

of organizational efficiency. Timeliness was selected for discussion in this Commission

paper because it directly impacts two of the NRC performance goals: (1) making NRC

decisions more effective, efficient, and realistic; and (2) increasing public confidence. 

Maximizing process efficiency is a critical component in effective regulation when faced

with the constraint of limited resources. Timely and predictable decision making is also

important in the Agency's effort to increase public confidence.

Analysis of the timeliness data for the assessment and enforcement process shows that

there has been a substantive increase in the amount of time that it takes to process

findings under the revised ROP. When taken in total, the ROP identifies fewer issues

that are considered for escalated enforcement. Currently, the average process time for

all open and closed cases identified since the initial implementation of the ROP is 98

days. This compares to an average timeliness of 75 days that was achieved in FY 1999

under the old enforcement program. This review included significant inspection findings

(i.e., White, Yellow, Red) regardless of whether they are associated with an enforcement

action. It has been recognized that the ROP places new requirements on the NRC staff

regarding the characterization of the significance of inspection findings. These

requirements demand the involvement of centralized NRC staff risk expertise in

combination with an increased reliance on licensee risk assessment tools and

information. These activities provide the basis for the initial significance characterization

and occur prior to the SDP and Enforcement Review Panel (SERP). OE has reviewed all

of the cases identified since ROP initial implementation and determined that if the NRC

staff began tracking issues from the point that they were identified, rather than the point

at which they are brought to a SERP, the average cycle time for these significant

inspection findings would increase to approximately 150 days.

This timeliness review demonstrated that there are three critical phases of the

assessment and enforcement process. First, the time from when an issue is first

identified by an inspection or revealed in an event to the time that a SERP is held. This

represents the time that it takes the NRC staff to provide an initial risk characterization

for an issue. Currently, this phase of the process is not monitored or tracked by any

management metric although this is being considered by the staff. Second, the

conference preparation time is the time required to develop a case from the SERP to a

regulatory conference. This phase has been the most time intensive for the cases

handled to date. Licensee responsiveness plays an important role in the time delays

associated with this area. Finally, the Agency decision time measures the time period

from the regulatory conference to the issuance of the final determination letter. This

portion of the process is similar to the traditional enforcement program and only becomes

a major impact on timeliness for the few cases that require policy decisions.

The decrease in timeliness for these potentially significant issues is attributable to many

factors that include familiarity with new processes, greater reliance on limited NRC staff

risk expertise, continuing development of NRC risk tools, greater reliance on licensee risk

tools and information, and a greater involvement of the licensee in the process. The

effects of some of these factors may diminish as the NRC staff gains more experience

under the ROP. However, many of these factors are inherent in the new process and will

need to be managed closely to attain optimal organizational performance.
3

B. Integration of Traditional Enforcement Action with the Agency Action Matrix

During the development of the ROP, the NRC staff recognized that it was not appropriate

to assess the significance of violations associated with actual consequences, willfulness,

and impeding the regulatory process under the ROP/SDP. As such, these issues have

always been reserved for traditional enforcement. OE supports this arrangement as

violations associated with actual consequences are captured within the ROP by either the

SDP or performance indicators. Violations in the areas of willfulness and impeding the

regulatory process are of concern to the Agency as they impact the ability of the NRC to

effectively regulate. However, the assignment of risk significance to these types of

issues is problematic and application of the ROP/SDP is difficult and impractical. Under

the old oversight program, violations and civil penalties in the areas of willfulness and

impeding the regulatory process were integrated into the overall assessment of

licensees, though admittedly in a fairly subjective manner. Under the revised ROP, the

ability to consider traditional enforcement issues is limited. IMC 0305, "Operating

Reactor Assessment Program," Section 06.01, "Overall Assessment Process," indicates

that traditional enforcement issues may be considered in determining the range of

Agency action within the appropriate column of the Action Matrix. However, there is no

explicit link to allow integration of traditional enforcement findings to the Action Matrix due

to the lack of direct linkage to risk.

C. Maintenance Rule Enforcement Review Panels

Enforcement Guidance Memorandum 96-001 established the Maintenance Rule

Enforcement Review Panel (the Panel) to help ensure consistency in enforcement of

10 CFR 50.65. The Panel process has served well to achieve this objective. However,

recent experience has shown that many of the issues were able to be dispositioned with

consistency without a formal panel meeting. Instead, they have been resolved through

consultation and agreement among the key representatives of the organizations

comprising the Panel. In addition, under the ROP, findings have either been minor or, if

more than minor, of very low safety significance (i.e., no greater than Green).

Accordingly, the revised Maintenance Rule enforcement guidance to be issued shortly

will establish a less formal enforcement review process to be used routinely to ensure

consistency. Resident inspectors and cognizant regional staff will be expected to

continue to consult with the Office of Enforcement and Maintenance Rule-cognizant NRR

staff to arrive at a consensus on the proper disposition of the Maintenance Rule findings. 

Nevertheless, if in the judgement of any of the cognizant parties, the circumstances of a

particular Maintenance Rule issue should warrant a formal enforcement panel, such a

panel would be convened.

The staff will continue the enforcement review process until such time that the

enforcement guidance is verified to be effective and reasonable consistency amongst the

regions is achieved. When the staff has determined that these conditions have been

satisfied, the special treatment of Maintenance Rule issues will be discontinued without

prior notification of the Commission.
4

D. Enforcement Actions Associated With Findings That Are Particularly Significant

The staff communicated the proposed revisions to the Enforcement Policy required to

implement the ROP in SECY-00-0061, ìProposed Revision to the Enforcement Policy to

Address the Revised Reactor Oversight Process,î dated March 9, 2000. The

Commission approved the staff's recommendation in a Staff Requirements Memorandum

(SRM) dated April 11, 2000, with one additional requirement. In the SRM, the

Commission requested consultation prior to exercising discretion and issuing a civil

penalty for "violations associated with findings that the reactor oversight program's

significant determination process (SDP) evaluated as having low to moderate, or greater

significance (i.e., White, Yellow, or Red) that are particularly significant" (Enforcement

Policy, Section IV.A.5.b). In addition, the Commission requested that the staff provide a

recommendation on continuing such consultation on this type of enforcement action as

part of its report on the first year of implementation of the revised ROP.

The staff has not identified any violations under the ROP that would be classified as

particularly significant and require the assessment of a civil penalty. This is consistent

with the expectation that particularly significant inspection findings would be rare. The

existing Enforcement Policy ensures that the Commission will receive prior written

notification (i.e., Enforcement Notification) of all enforcement actions that involve a civil

penalty. However, in order to highlight these rare cases, the Office of Enforcement

intends to revise the Enforcement Notification (EN) guidance in the Enforcement Manual

to specifically require a discussion in the EN of the basis for assessing a civil penalty for

any violation that is characterized through the SDP in accordance with the ROP. 

 
Key Policy Issues

Attachment 11
1

Key Policy Issues

Commission paper SECY-00-0049 identified three ìIssues of Note,î issues that emerged from

the pilot program that had a significant impact on the implementation of the ROP, and that the

staff had not been able to resolve in a manner that adequately addressed the majority of

stakeholder concerns. The staff committed to present these issues, the differing points of view,

and the staff proposal for resolution to the Commission for consideration. Additionally, SECY00-0049 provided an update to several other key policy issues that need to be considered and

addressed as part of the implementation of the ROP. 

The three ìIssues of Note,î and their status are as follows.

1. Significantly different viewpoints existed among stakeholders on how crosscutting issues and programmatic breakdowns should be addressed in the ROP. 

One viewpoint supported the original tenet that programmatic breakdowns would reveal

themselves by performance indicators (PIs) crossing thresholds and the identification of

significant inspection findings. The other view supported the opinion that significant

programmatic breakdowns could occur without prior indication through PIs or inspection

findings, and that the specific evaluation of licensee performance in the cross-cutting

areas would provide a timely opportunity for the NRC to respond to these concerns. 

As committed to in SECY-00-0049, a working group was formed consisting of

Headquarters and regional management and staff to focus on the concerns expressed

regarding cross-cutting issues. The first meeting was held on April 5, 2000 where the

group developed five key issues to focus on in the cross-cutting area (see Appendix A for

additional detail on the five key issues). 

A total of three working group meetings were held, with members of the public, industry,

and the Institute of Nuclear Power Operations (INPO) invited to attend the third meeting

and present their perspective on the cross-cutting issue concerns. In addition to

discussing the five key issues, the group also discussed the feasibility of developing

performance indicators in the problem identification and resolution (PI&R) area. The

general consensus expressed was that this would be very difficult as the PI&R programs

are very site specific. Additionally, the group discussed the current approach the NRC

uses to assess safety-conscious work environment. Currently, the only inspection

activity where the NRC assesses safety-conscious work environment is the annual PI&R

inspection. Some NRC representatives expressed a concern that this was a cursory

review that might lead to false positives, that is, falsely assuming that the licensee has

established a safety-conscious work environment. Some industry representatives

expressed a concern that NRC inspectors may not be trained to accurately assess this

area and that this area is more subjective in nature, making it difficult to assess. There

was also some discussion about whether this topic is really a separate cross-cutting

issue or is it just imbedded in a licenseeís overall PI&R. 

Cross-cutting issues was a topic that was discussed in both the internal and external

NRC workshops held in March of 2001. In the external workshop, four issues were

discussed that were closely aligned with the five key issues identified by the above
2

working group. In addition, a recommendation to change the approach for inspecting the

area of PI&R was also discussed at the workshop. 

As discussed at the workshop, experience from the initial implementation of the ROP

thus far tends to support one of the fundamental premises of the ROP; that degradation

in the cross-cutting areas will be detected by either PIs or inspections in a sufficient time

frame to allow for Agency action to protect public health and safety. However, these

results are not conclusive since one year represents a relatively short experience base,

particularly when trying to determine whether the premise holds under all reasonably

credible situations. Further, declines in licensee performance typically occur over an

extended period of time, so it is too soon to reach definite conclusions in this case. 

During initial implementation, there have been no significant precursors to a reactor

accident. Also, during initial implementation of the ROP, the staff has not identified any

cross-cutting issues that would warrant special treatment. Currently, the ROP addresses

cross-cutting issues by highlighting them in inspection reports when they are notable

contributors to inspection findings or if there is an appreciable trend or pattern that has

emerged; and in assessment letters to the licensee when they constitute a substantive

issue. 

A revision to IMC 0610*, issued October 6, 2000, better explains when and how crosscutting issues should be documented in inspection reports. The ROP does not allow for

additional NRC engagement on cross-cutting issues unless they are contributing causes

to PIs or inspection findings that are characterized as White or greater. The staff has not

engaged licensees on cross-cutting issues outside of the guidance contained in the

Action Matrix. NRC engagement has occurred per the Action Matrix when significant

cross-cutting issues have moved licensee performance beyond the licensee response

column. 

Following the external workshop, an internal focus group was formed to further study the

recommendation discussed at the workshop to improve the effectiveness of the PI&R

inspection approach. The focus group recommended changing the baseline frequency

of the annual PI&R inspection to once every two years for all facilities. Also, the focus

group recommended revising the recommended hours budgeted per inspection to 250

hours from the current 210 hours. In addition, the group recommended adding 60 hours

of effort annually to inspection procedure 71152 to allow for follow-up/assessment of

PI&R samples throughout the inspection period. The above recommendations will result

in an estimated overall reduction of direct inspection effort of 25 hours per year in this

area. Lastly, for facilities that reach the degraded cornerstone column of the Action

Matrix, the focus group recommended consideration of an additional PI&R team

inspection as part of the supplemental inspection response to assess the extent of

condition of the identified performance issues. Overall these changes should result in a

more effective allocation of NRC resources in this area. Implementation of these

recommended changes is currently planned to begin in January 2002.

Based on the lessons learned during initial implementation of the ROP, the staff has not

identified a need for significant modifications to address cross-cutting issues; however,

the staff will continue to assess events and inspection findings to look for safety

significant areas not adequately covered by the current baseline inspections, PIs, or the
3

significance determination processes (SDP). Further details regarding the staffís

assessment of cross-cutting issues during initial implementation of the ROP and on

future planned actions to assess this issue are discussed in Appendix A of this paper.

2. The Barrier Integrity cornerstone PIs are fundamentally different from the other

indicators used in the ROP. The Barrier Integrity PIs are intended to provide

indications of the integrity of the three barriers to the release of radioactive material from

the reactor core. They use readily available information that licensees are required to

collect by technical specifications (TS). The thresholds are set as percentages of the TS

limit. However, in practice plants typically operate very far below the TS limits, so that

the Green/White threshold would rarely be exceeded. The indicators therefore serve

primarily a public confidence role to indicate how much margin there is to any safety

concern in the performance of these barriers, as opposed to an indication, at least at the

Green/White threshold, that there is a deviation from nominal industry performance. In

addition, because TS requirements vary from plant to plant (e.g., for Reactor Coolant

System leakage, in addition to unidentified leakage some plants measure identified

leakage while others measure total leakage), and because licensees use a variety of

methods to measure compliance with these TS (e.g., some measure as-found

containment leakage while other record only as-left leakage), the data reported can vary

considerably from plant to plant. 

The staff committed to work to improve the meaningfulness of the Barrier Integrity

cornerstone PIs during ROP initial implementation. However, no changes were made to

these PIs during initial implementation due to the need to address many of the higher

priority PI issues previously discussed in this paper. The staff intends to continue work

on the barrier integrity indicators following initial implementation to improve their

usefulness. 

3. There was disagreement on how the ROP should treat inspection findings

involving licensee performance issues that are outside the licensing and design

basis of the plant. The ROP requires the staff to assess the significance of inspection

findings involving deficient licensee performance and to input any results that are of

greater than Green (very low risk significance) into the Action Matrix to determine the

most appropriate Agency responses to licensee performance declines. The key feature

of this policy is that it does not distinguish between findings involving regulatory noncompliance and those findings that do not violate the licensing or design basis but

nevertheless represent an unintended increase in plant risk resulting from deficient

licensee performance. If such an issue were to achieve sufficient risk significance to

meet the regulatory analysis guidelines for backfit, then the staff would consider

implementing such a backfit. However, based on the pilot program experience, it is very

possible for the risk significance of such issues not to achieve this threshold, but still be

characterized as greater than Green. 

Lessons learned from initial implementation of the ROP indicate that the oversight

process can appropriately treat licensee performance issues that are outside the

licensing and design basis of the plant. Of the 22 safety significant inspection findings

evaluated by the SDP during initial implementation, four did not involve compliance with

regulatory requirements. Although no compliance issues existed, licensees took
4

appropriate action to address the underlying safety concerns. At this time, changes to

the ROP are not necessary to address this issue.

The status of the other key policy issues previously discussed in SECY-00-0049 are as follows:

The ROP may have a long-term impact on the ìN+1" policy for resident inspector

staffing due to the potential for improved efficiency and changes in the division of

responsibility for implementing the new processes. The staffís proposal for

addressing the ìN+1" resident inspector staffing policy under the ROP was forwarded to

the Commission by SECY-99-227, ìN+1 Resident Inspector Staffing Policy,î dated

September 13, 1999. By SRM dated January 11, 2000, the Commission approved the

staffís recommendation to modify the policy to require ìNî resident inspectors at dual and

triple-unit plants, but maintain ìN+1" resident inspector staffing at single unit plants. 

As part of the first year implementation of the ROP, the staff evaluated the impact of the

resident inspector staffing policy change on the ability of the resident inspectors to

complete the inspection program. The details of this evaluation are included as Appendix

B of this paper. Based on the first year of ROP implementation, the staff concluded that:

ï Baseline inspections can be accomplished by ìNî resident staffing at dual unit

sites with some assistance from region-based inspectors.

ï Most regions gained substantive experience completing the baseline inspections

at dual unit sites with ìNî residents during the first year of ROP implementation.

ï Sites that were staffed at ìN+1" tend to expend a level of effort commensurate

with site staffing, not program requirements. 

ï There is an increased inspection-related travel requirement for the region-based

inspectors because the regions had to rely on region-based inspectors and

project engineers to augment the resident inspector staff during extended

absences and vacancies at the dual unit sites. 

ï It is too soon to determine the full impact on the resident inspector program from

the change to the ìN+1" resident staffing policy. However, early indications are

that the ìNî staffing policy will challenge the regions in providing extended

training, rotations, and other professional development assignment opportunities

for the resident inspector staff at multi-unit sites.

ï The regions did not gain the immediate increase in inspection flexibility as

originally envisioned by the change to the ìN+1î resident inspector policy. As

regions moved to ìNî staffing, it was generally in response to retirements,

promotions, and voluntary reassignments. Therefore, many times a hew hire was

brought onboard as a resident inspector position was closed, resulting in an

overall reduction in inspector expertise available in the regions. 
5

ï Regional effort to support the ROP, evaluation and development activities was

substantial and impacted the resources available to support site-based activities

as regions moved to ìNî staffing.

ï Regional management must manage the resident inspector program more closely

to assure that upcoming RI vacancies are posted and filled in a timely manner

and to assure that appropriate site coverage is maintained during RI transitions.

Based on these conclusions, the staff plans to:

ï Develop criteria for the Regional Administrators, in consultation with NRR, to

allocate additional inspection resources for other than performance related issues

(e.g., unique design or organization features). 

ï Develop appropriate ROP self-assessment metrics to monitor and trend inspector

demographics and resident inspector program quality attributes (e.g., inspector

training time; site coverage; rotational opportunities).

ï Determine and adjust, as appropriate, the budget model to reflect efficiency

differences in converting a resident inspector to a region-based inspector.

The regional and Headquarters organizational structures may need to be changed

to support the cornerstone framework and the ROP. The development of the ROP

has resulted in changes in the necessary inspection skills and expertise for regional and

resident inspectors. Two areas of the baseline inspection program where this is

particularly true are for the design and fire protection inspections. Fire protection triennial

inspections in particular require a breadth of fire protection knowledge that does not

currently exist in all regions. Contractor resources are currently being utilized to augment

region-based inspectors in the design and fire protection areas. Additionally, riskinforming the ROP and the development of the SDP has identified the need to enhance

the risk expertise of regional staff.

Several evaluations were performed during initial implementation to address the resident,

regional, and Headquarters resources (including contractor resources) required to

implement the ROP. The results indicate that adequate resources are currently

budgeted to implement the ROP. The regional resource model that was used to

establish resource estimates needs to be refined to reflect the ROP. While the staff has

developed a revised resource model based on the initial implementation results, it will be

refined as more data is gathered and evaluated. The staff has identified areas for

potential efficiencies in the ROP as well as emerging budgetary issues to be considered

in refining the resource model. While no significant regional or Headquarters

organizational changes were necessary to implement the ROP, adjustments were made

to the roles of certain regional staff to participate more in inspection. Additional detail on

resident inspector staffing levels can be found in Appendix B and details on the ROP

resource analysis can be found earlier in this paper and in Attachment 13.

A working group was formed in August 2000, and charged with reviewing, assessing and

modifying the inspector training and qualification requirements contained in inspection
6

manual chapter (IMC) 1245, ìInspector Qualifications for the Office of Nuclear Reactor

Regulation Inspection Program,î to support the ROP. Since its inception, the working

group has completed the identification of tasks performed by inspectors under the ROP

and has surveyed existing inspectors to determine the level of difficulty they have

experienced while trying to implement those tasks. The knowledge, skills and attitudes

needed by inspectors to successfully perform the various reactor oversight tasks have

been identified and grouped into 11 competencies. The IMC 1245 working group is

defining a new set of competency-based training and qualification requirements for

inspectors. The content and methods used in the existing training program are being

reviewed to identify which program areas already support the newly defined

competencies, where improvements are needed, and where new training must be

developed. Definition of the requirements is expected to be completed in the summer

2001. However, the new requirements will be implemented over time to allow for the

development or revision of the necessary training and qualification materials. Changes

to IMC 1245 will not affect the qualification status of currently qualified inspectors.

A separate working group on risk expertise was formed in July 2000, with the purpose of

improving the risk expertise among regional staff. The goals of this working group

included: (1) ensuring that every reactor inspector in the region is capable of using and

understanding the SDP and (2) ensuring that a subset of these inspectors receive

advanced risk training. One of the recommendations of the risk expertise working group

was to develop near-term training to improve the use and understanding of the SDP

through the use of examples. This instructional guide was completed in June 2001 and

includes twelve reactor SDP examples, including a fire protection SDP example. 

In November 2000, a solicitation of interest notice was issued to select region-based staff

for advanced risk training that would enable them to utilize NRC Probabilistic Risk

Assessment (PRA) software tools to perform and interpret basic quantitative risk

analysis. A total of twelve regional selections for the advanced risk training were made,

with these individuals attending a series of seven PRA classes commencing in April

2001. Additionally, the course on îPRA Technology and Regulator Perspectiveî (P-111)

was redesigned to include more SDP instruction and examples, and to include a scenario

in which students are required to defend an SDP analysis during a mock SERP. Finally,

the Senior Reactor Risk Analyst training and qualification program was formalized in

April, 2001, via a revision to IMC 1245. 

Additional detail on the inspector training and regional risk expertise working groups can

be found in Appendix C to this paper.

The implementation of the ROP will have an impact on the conduct of the Agencyís

allegation program. Four options for modifying the allegation program were forwarded

by the staff in Commission paper SECY-99-273, ìImpact of Changes to the Inspection

Program for Reactors on Implementing the Allegation Program,î dated November 23,

1999. As documented in the SECY-99-273 SRM dated January 27, 2000, the

Commission approved the staffís proposal to solicit stakeholder input prior to a final

Commission decision on one of the options. The staff presented the results of

stakeholder feedback on the four options in Commission paper SECY-00-0177, 
7

îImplementing the Allegation Program under the Revised Reactor Oversight Process,î

dated August 22, 2000.

By the SECY-00-0177 SRM dated October 11, 2000, the Commission disapproved

further pursuit of risk-informing the allegation program, and directed the staff to continue

to implement the existing allegation program. The staff was directed to ensure that

allegations are handled and documented consistently from region to region through a

systematic process that interfaces with the ROP. Further, upon completion of the staff's

evaluation of the SDP, it should be employed to determine those issues developed from

allegations that should be documented in an inspection report in the same way the SDP

is utilized to assess findings not related to allegations. The staff is currently

implementing this direction from the Commission.

While considering the impact of the ROP on the allegation program, the staff was

concerned that the timeliness of allegation closeout might decrease as the regions had

less discretion for conducting regional initiative inspection, during which much of the

allegation follow-up was conducted. The staff was also concerned that it might be harder

to protect the identify of allegers during allegation follow-up inspections and while

documenting the allegation results using the revised inspection report writing guidance.

During the first year of initial implementation, the staff noted a slight decrease in the

timeliness of allegation closeout as the regions scheduled more allegation follow-up

inspections with previously scheduled baseline inspections. However, average allegation

closeout timeliness was still well within the Agency goal of 180 days. Additionally,

experience during initial implementation did not demonstrate a problem with protecting

the identity of allegers, either during the conduct of the allegation follow-up or while

documenting the results.

The staff has worked closely with the Office of Nuclear Regulatory Research (RES)

to investigate the feasibility of selected risk-based performance indicators (RBPIs)

as a potential enhancement to the PIs for the ROP. In response to a User Need

Memorandum from NRR, RES is examining the feasibility of developing indicators to

enhance the current set of ROP indicators in areas of plant performance where there are

currently none, such as equipment reliability, shutdown operations, fire protection

systems, and containment performance. RES is examining these selected indicators as

part of its broader initiative to examine the feasibility of developing a comprehensive set

of RBPIs. Some the RBPIs have the potential to provide insights on component, train,

and system performance, as well as allowing for the use of plant-specific thresholds. 

RES has documented the results of the feasibility study in a draft RBPI Phase 1 report. 

NRR and RES will assess continued RBPI development based on feedback received

from stakeholders on the Phase 1 report. The Phase 1 Report was published for public

comment in a Federal Register Notice (FRN) in February 2001, and feedback has been

received in response to the FRN and as part of two public meetings on the report. NRR

will review the stakeholder feedback and provide direction to RES on the continued

development of RBPIs in August 2001.

In addition, the staff is developing an industry trends program as a means of assessing

whether the safety of operating power plants is being maintained by the nuclear industry
8

and to enhance public confidence in the efficacy of NRC processes. The NRC will use

indicators to identify adverse trends, evaluate them, and take appropriate Agency

actions. One important output of this program is to report to Congress each year on the

measure ìno statistically significant adverse industry trends in safety performanceî as

part of the NRCís Performance and Accountability Report. No statistically significant

adverse trends have been identified to date based on the information currently available

from the industry-wide indicators developed by the former Office for Analysis and

Evaluation of Operational Data (AEOD) and the Accident Sequence Precursor (ASP)

program implemented by RES. The NRC staff will continue to use these indicators to

monitor industry trends and use them as the basis for the report to Congress. In

addition, the staff will develop additional indicators that are more risk-informed and

aligned with the cornerstones of safety for use in the industry trends program. The

additional indicators will be derived from the plant-specific PIs of the ROP and from

operating experience information in RES. They will be developed and qualified for use in

the industry trends program and the report to Congress in phases. In determining its

response to adverse trends, the staff will use a graded approach based on the safety

significance of the issues. The staff will be examining a more objective and risk-informed

approach to assessing the safety significance of changes to indicators, including

potentially establishing thresholds for Agency response. 

Changes to the reactor oversight process need to be closely coordinated with the

Agency initiative to risk-inform 10 CFR Part 50. The ROP increases focus on certain

risk-significant requirements and decreases focus on certain other less risk significant

requirements. This could result in situations where very low significance findings, even if

numerous, would be evaluated and treated with little to no follow-up by the Agency. 

Similarly, the ROP may identify risk significant conditions, not covered by current

regulations, that warrant some level of Agency follow-up beyond the baseline inspection

program. Risk-informing 10 CFR Part 50 may provide further insights into aspects of our

current regulations that may impact the inspection program. The staff will continue to

work to ensure that the ROP is consistent with the results of this effort.

The staff has also worked to apply many of the ROP concepts developed and

lessons learned identified to the oversight of nuclear fuel cycle facilities. The

Office of Nuclear Material Safety and Safeguards (NMSS) forwarded its proposal for

evaluating and proposing revisions to the oversight process in Commission paper SECY99-188, ìEvaluation and Proposed Revision of the Nuclear Fuel Cycle Facility Safety

Inspection Program,î dated July 21, 1999. NMSS forwarded an update to the status of

their development efforts in Commission paper SECY-00-0222, ìStatus of Nuclear Fuel

Cycle Facility Oversight Program Revision,î dated November 27, 2000, and briefed the

Commission on the status of these efforts on December 20, 2000. NRR staff will

continue to work closely with NMSS to share lessons learned from ROP initial

implementation and support its work to develop a more risk-informed process for the

regulatory oversight of nuclear fuel cycle facilities. 
ROP INITIAL IMPLEMENTATION COMMITMENTS

Attachment 12
1

ROP Initial Implementation Commitments

There were several open issues and commitments that the staff continued to evaluate and work

on during the first year of ROP implementation. Several of the issues involved longer term items

that the staff identified as needing additional work following initial implementation in Commission

paper SECY-00-0049, dated February 24, 2000. The Commission directed the staff to consider

several additional issues in the SECY-00-0049 SRM dated May 17, 2000. Also several

recommendations for staff consideration during initial implementation were made by the Pilot

Program Evaluation Panel (PPEP) as noted in the ìFinal Report of the Pilot Program Evaluation

Panel,î dated December 21, 1999. All of these issues were considered by the staff during initial

implementation as described throughout this Commission paper and its attachments. The status

of these issues is summarized below.

SECY-00-0049 SRM

ï The staff should convene another evaluation panel under the Federal Advisory

Committee Act during the first year of initial implementation. The staff should

include at least one resident inspector and one senior reactor analyst on this

ìInitial Implementation Evaluation Panel.î

An Initial Implementation Evaluation Panel (IIEP) was established by the Agency in

accordance with Federal Advisory Committees Act (FACA) requirements to serve as an

advisory committee to the Agency. This panel was a cross-disciplinary group of NRC,

public, and industry stakeholders representing many different nuclear power interests,

including the Nuclear Energy Institute, power plant licensees, the New England Coalition

on Nuclear Pollution, the Georgia Department of Natural Resources, a PhD candidate

from Benedictine University, and the California Energy Commission. NRC staff on the

IIEP included a regional Senior Resident Inspector and a Senior Risk Analyst, in addition

to Headquarters and regional managers. The purpose of the IIEP was to monitor and

evaluate the results of the first year of initial implementation and to provide advice and

recommendations to the Agency on reforming and revising ROP. The IIEP first met on

November 1, 2000 and conducted a total of six meetings throughout the initial

implementation of the ROP. The IIEP final report was issued on May 10, 2001, and the

recommendations were addressed throughout this Commission paper.

ï During the initial implementation phase, ending in June 2001, Action Matrix

deviations should be pre-approved by the EDO. 

Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program,î was

revised on April 24, 2000, to ensure that the EDO approves all proposed deviations from

the Action Matrix.

ï In its report to the Commission on the initial implementation phase of the ROP, the

staff should provide a summary of the deviations from the Action Matrix,

recommendations on a method to assure Agency-wide consistency when deviating

from the Action Matrix, and recommendations on how to keep the Commission

currently informed of such deviations. Furthermore, during the initial
2

implementation phase, the staff should inform the Commission whenever it

determines that such a deviation is warranted. 

Although deviations from the Action Matrix were considered on a few occasions during

initial implementation of the ROP, none were issued. The staff revised IMC 0305 to

ensure that the Commission is informed of any such deviations. IMC 0305 provides for

Agency-wide consistency by ensuring that deviations from the Action Matrix are rare and

are made with the appropriate level of NRC management approval. 

ï NRC management should undertake continuing dialogue with the staff, including

in particular the inspection staff, to stay abreast of concerns with initial

implementation of the ROP and to assure that these concerns are thoroughly aired

and addressed. 

Internal feedback and comments from NRC staff were obtained during the first year of

ROP implementation using various methods. Weekly teleconferences were held with

regional management to solicit feedback and address implementation issues. 

Counterpart meetings were held with the regional Division Directors approximately every

two months and a briefing of the four Deputy Regional Administrators was conducted to

discuss implementation issues and solicit feedback and comments on the ROP. 

Feedback forms were developed and used for the inspection procedures and oversight

process to allow regional staff to document questions and concerns on the various

components of the ROP. Regional and site visits were made to each of the four regional

offices and six sites per region to solicit program feedback at the mid-point of initial

implementation. Based on feedback received, internal staff focus groups were

established to address the major issues, which were then discussed at an Internal

Lessons Learned Workshop conducted in March 2001. Finally, an internal stakeholder

survey of the regional inspection staff was conducted near the end of the first year of

ROP implementation to gather additional insights to be considered while evaluating the

lessons learned. 

ï In its ongoing communication efforts the staff should emphasize the importance of

the licensee corrective action programs (CAPs) and the integral role that each

licensee must play, i.e., that the new process places greater responsibility on the

utilities that operate nuclear power plants. The staff should more clearly articulate

the role of licensee CAPs, with self-assessment, as the beginning and the end of

this process. 

In several ongoing efforts, the staff has communicated the importance of licensee CAPs

and their role in the ROP. For example, NRC staff and inspectors received training on

the problem identification and resolution (PI&R) baseline inspection procedure during the

G-200 course, ìReactor Inspection and Oversight Programî, provided by the Technical

Training Center. In addition, IMC 0610* was revised to allow documentation of crosscutting issues, including PI&R issues, associated with greater than Green inspection

issues. Similarly, substantive trends in cross-cutting issues have been documented in

mid-cycle and end-of-cycle assessment letters. Finally, the staff has continually

reinforced the importance of the licensee CAP and PI&R baseline inspections with both

internal and external stakeholders at the various presentations, public meetings,
3

workshops, and other venues conducted as part of the Communication Plan during ROP

initial implementation. 

ï NRR and regional management should ensure that the threshold for documenting

observations is clearly understood and consistently applied. They should also

take steps to assure that inspector observations are placed in an appropriate

context and do not undermine the overall effort to put inspection and enforcement

efforts on a more objective and consistent foundation.

IMC 0610* generally does not allow the documentation of observations, except in rare

instances where extenuating circumstances exist (e.g., in response to an allegation). 

When appropriate, these observations can then be documented as ìNo Colorî findings in

accordance with the requirements of IMC 0610*. 

A revision to IMC 0610* was issued on October 6, 2000, which provided additional

guidance and clarification to the thresholds for documenting inspection findings, including

the criteria and thresholds for ìNo Colorî findings. Since January 2001, the staff has

been auditing inspection reports issued by the regions to determine whether the regions

are documenting inspection results in accordance with the requirements of IMC 0610*. 

The focus of these audits has been on evaluating whether the safety significance of

findings is clearly documented and whether ìNo Colorî findings are being properly used

and placed in the proper context. These inspection report audits are performed as part

of the periodic ROP self-assessment discussed earlier, and will continue beyond initial

implementation.

ï The staff should show that cross-cutting issues they identify have a clear and

strong link to significant inspection findings or degraded PIs before the staff

attempts to take action on programmatic concerns.

The ROP addresses cross-cutting issues by highlighting them in inspection reports when

they are notable contributors to inspection findings or if there is an appreciable trend or

pattern that has emerged; and in assessment letters to the licensee when they constitute

a substantive issue. Recent changes made to Inspection Manual Chapter 0610* better

explain when and how cross-cutting issues should be documented in inspection reports. 

The ROP does not allow for additional NRC engagement on cross-cutting issues unless

they are contributing causes to PIs or inspection findings that are characterized as White

or greater. 

ï In its report to the Commission on the initial implementation phase of the RROP,

the staff should discuss every case in which enhanced NRC action was taken

based on cross-cutting issues. 

During initial implementation of the ROP, enhanced NRC action was not taken by the

Agency based solely upon findings associated with cross-cutting issues. 

ï The staff should continue to work with NRC stakeholders to resolve the

ìPerformance Issues Outside Licensing and Design Basisî issue while

maintaining its appropriate sensitivity to backfit implications. The staff should
4

specifically discuss this issue when they report to the Commission on the results

of the first year of RROP implementation. 

Lessons learned from initial implementation of the ROP indicate that the oversight

process can appropriately treat licensee performance issues that are outside the

licensing and design basis of the plant. Of the 21 safety significant inspection findings

evaluated by the SDP during initial implementation, four did not involve compliance with

regulatory requirements. Although no compliance issues existed, licensees took

appropriate action to address the underlying safety concerns.

PPEP Commitments

ï Conduct required verification inspections early for the performance indicator data

provided by non-pilot plants.

Early during ROP initial implementation, resident inspectors at each non-pilot plant

conducted an inspection in accordance with Temporary Instruction 2515/144, Revision 1. 

The purpose of this inspection was to review the licenseeís performance indicator (PI)

data collecting and reporting process to determine whether the licensees are

appropriately implementing the NRC/Industry guidance. 

ï Resolve the outstanding issues regarding the appropriateness and threshold

values of the emergency preparedness response training and drill participation,

security equipment performance, containment integrity, and siren notification

systems PIs. These issues should be resolved during the first year of industrywide implementation and included as part of the report to the Commission. 

ERO Drill/Exercise Performance (DEP) and Drill Participation Indicators: NEI 99-02,

Revision 0, included language explaining that these two indicators are linked. That is,

for key emergency response organization (ERO) members who are responsible for

classification, notification or protective action recommendation (PAR) development to get

credit in the ERO Drill Participation PI, their performance must be assessed and must

contribute to the DEP PI value. The issue of linkage between the DEP PI and ERO PI

was largely resolved early during initial implementation. However, the linkage was not

apparently universally understood and, in a few instances, was misinterpreted or

overlooked. The clarity of the linkage was improved in NEI 99-02, Revision 1, and NEI

committed to conduct further industry training to ensure understanding. 

Security Equipment Performance Index: The NRC/Industry working group examined the

use of unplanned unavailable hours in lieu of compensatory hours in this PI and

concluded that there is no need to change. In addition, changes were made to both the

White band and Yellow band thresholds that had been a concern during the pilot

program. When the historical data provided by all plants prior to the start of initial

implementation was evaluated, the staff determined that the Green band should be

widened by adjusting the White band threshold from 0.05 to 0.08, and the Yellow band

was eliminated.
5

Containment Leakage Indicator: As a consequence of several significant problems with

this indicator, it was removed from the program prior to initial implementation.

ANS Reliability Indicator: The Green/White threshold for this indicator was set at 94

percent in recognition of the typically high ANS reliability. When this threshold was

developed, only one plant-year was below the threshold and one near it. Therefore the

threshold was deemed to accurately reflect the performance level at which greater NRC

involvement in licensee maintenance of the ANS was appropriate. This premise was

borne out during initial implementation. The number of White indicators has been about

the expected rate of 5 percent, and they were primarily due to problems with ANS

maintenance. The Yellow band threshold was set at 90 percent, the reliability value that

requires a formal corrective action plan to be filed with FEMA. Although no member of

the NRC/Industry EP working group was aware of an ANS with annual reliability so low,

one did occur during Initial Implementation. Inspection of the associated corrective

actions revealed a poor root cause analysis and poor corrective action implementation by

the licensee, which indicated that the threshold was appropriate in indicating that

mandatory regulatory involvement was appropriate.

ï Ensure that the program effectiveness is not measured solely based on the

increase or decrease of resource utilization. 

The staff developed a comprehensive self-assessment program to evaluate the efficacy

of the ROP. A total of 20 measures for the inspection program were developed as part

the overall self-assessment process. Of those 20, only 3 measures collect and compare

inspection resources: 2 measures use NRC inspector time and 1 measure uses

contractor support dollars. Therefore, the self-assessment program measures ROP

effectiveness by more than resource utilization.

ï Significant events should be evaluated from a program perspective as related to

the effectiveness of performance indicators and risk-informed baseline inspection

results. 

An Indian Point 2 Lessons Learned Task Group was formed following the February 2000

Steam Generator (SG) Tube Failure to review the event for lessons learned that could be

applied to the ROP, including the new baseline inspection program and PIs. The task

group made several recommendations regarding the Inservice Inspection baseline

inspection procedure and improvements to the PIs that could better reflect the integrity of

the SG tubes. As described earlier in this Commission paper, the staff evaluated these

recommendations and is considering appropriate changes to the baseline inspection

program. The staff determined that no changes were necessary to the PIs.

ï A monitoring program that seeks to confirm the continued validity of the

underlying assumptions regarding cross-cutting issues such as human

performance is needed for the first year of implementation.

A working group was formed consisting of Headquarters and regional management and

staff to focus on the concerns expressed regarding cross-cutting issues. Three working

group meetings were held during initial implementation to review lessons learned and
6

experience with cross-cutting issues. Data obtained from the initial implementation of the

ROP thus far tend to support one of the fundamental premises of the ROP; that

degradation in the cross-cutting areas will be detected by either PIs or inspections in a

sufficiently pro-active time frame to allow for Agency action to protect public health and

safety. Also, during initial implementation of the ROP, the staff did not identify any crosscutting issues that would warrant special treatment. 

ï The staff should clearly articulate in writing both the approach and the rationale

for using core damage probability (CDP) not core damage frequency (CDF) for

event response and enforcement, and CDF and not CDP for SDP and enforcement. 

The approach and rationale for using CDF and not CDP for the SDP safety evaluations

and enforcement was documented in a revision to IMC 0609, ìSignificance Determination

Process,î Appendix A, Attachment 2, issued on December 28, 2000.

ï The independent SDP review panel should be maintained during and following

industry-wide implementation to continue to reinforce and ensure desired

consistency as more inspectors are exposed to the process. 

The panel is identified as the SDP/Enforcement Review Panel or SERP. The SERP 

contributes to the consistent implementation of the ROP by reviewing each inspection

finding potentially greater than Green. Participants include the Headquarters and region

based senior reactor analysts (SRAs), enforcement specialists, and appropriate NRR

technical and project management staff. The panel is chaired by the Chief, Inspection

Program Branch or by an IIPB section chief who is responsible for obtaining consensus

on the determination of significance of the findings. Detailed guidance on the function of

the SERP is provided in Attachment 0609.01 to Manual Chapter MC 0609. 

ï The staffís guidance should be clarified relative to the use of the Action Matrix

when the significance of the inspection findings are under review at the time the

assessment is published (mid-cycle or annual).

Guidance was added to IMC 0305, ìOperating Reactor Assessment Program,î to discuss

how the regions should document in assessment letters those potentially safety

significant inspection findings whose final safety evaluation is pending.

ï Update the program basis document and make it publically available as soon as

possible. 

The staff is in the process of finalizing an update to the ROP Basis Document that will

provide the background behind the development of the PIs, the new inspection program,

the revised assessment process, the revised enforcement policy, and the SDP. The

basis document will not only describe the rationale and decision-making process for the

key attributes of each of the oversight processes, but will also describe many of the

program aspects that were considered during ROP development, but not included in the

oversight process. The staff expects to issue the ROP basis document as a NUREG

during the summer of 2001, and will periodically update the document to keep it current

with program changes. 
7

ï Set up a process for ongoing confirmation of assumptions underlying the process. 

The staff has developed an ROP self-assessment program to continue to perform

periodic program self-assessments following initial implementation to collect additional

lessons learned and gain insights from the new oversight process. This ongoing selfassessment process utilizes objective measures and pre-determined criteria to monitor

the performance of the ROP. As described in IMC 0307, ìROP Self-Assessment

Program,î these periodic self-assessments will monitor the major components of the

ROP utilizing eight criteria. The first four criteria would monitor the ROPís ability to be: 1)

objective, 2) risk-informed, 3) understandable, and 4) predictable. These criteria derive

from the original design objectives of the ROP. The other four criteria would monitor the

ROPís ability to: 1) maintain safety, 2) increase public confidence, 3) make NRC activities

and decisions more effective, efficient, and realistic, and 4) reduce unnecessary

regulatory burden on stakeholders. These criteria derive from the Agencyís performance

goals as outlined in the Strategic Plan. 
ROP RESOURCE ANALYSIS

Task Lead: Armando Masciantonio, NRR/IIPB

Attachment 13
1

ROP Resource Analysis

Introduction

Prior to initial implementation of the revised Reactor Oversight Process (ROP), the staff stated in

SECY-00-0049 that there would be no resource adjustments to the inspection and assessment

programs until adequate experience was gained with the revised ROP. A full year of

implementation beyond the pilot program was planned to obtain reliable data on which to

estimate the resources needed to accomplish individual inspections as well as to execute the

overall ROP. 

In order to assess the ROP from a resource perspective, the staff focused on two areas; (1) can

the ROP be completed within the estimated resources, and (2) is the ROP more efficient in

resource utilization compared with the previous inspection and assessment programs.

These issues have been addressed by separately examining the following aspects of the ROP:

ï The actual resources used during the first year of ROP implementation compared with

the assumptions and estimated resource requirements of the ROP for the specific

reactor oversight activities including baseline, plant specific, and generic issues

inspection areas and reactor performance assessment. 

ï Total resources expended in each inspection and assessment program area compared

with estimated resources in those program areas.

ï Regional variation in resource utilization.

ï Comparison of total annual resource expenditures for the revised ROP with the previous

program.

Initial Estimates of ROP Resource Requirements

Resource requirements for the ROP were estimated prior to initial implementation based on

assumptions relative to the time needed to complete each baseline procedure at the appropriate

depth and scope, and the frequency for performing the inspection. The expected needs for

supplemental and event response inspections, generic safety issues, and performance

assessment were also considered.

An expert panel of inspectors and senior staff reviewed the individual inspection procedures and

arrived at a consensus as to the content, scope and the hours required to complete the baseline

procedure for single-, dual-, and triple-unit sites. The resource requirements for other elements

of the ROP were similarly estimated as described below.

Baseline Inspections

An annual estimate of resource requirements for the baseline inspections was calculated based

on the estimated procedure hours and frequency. In addition, the annual effort for plant status

activities was estimated at 630 hours, 700 hours, and 840 hours, respectively, for a one-unit,
2

two-unit, and three-unit site. Plant status activities are not considered direct inspection effort but

reflect the time needed for inspectors to stay cognizant of plant conditions and issues. These

activities include time spent by resident inspectors gathering and analyzing information related to

current plant status and ongoing activities that are directly applicable to inspection planning,

including control room and plant walkdown, attendance at licensee status meetings, and

participation in inspection entrance and exit meetings. 

Based on these assumptions, the resulting annual baseline resource requirements for direct

inspection, preparation and documentation, and plant status effort were estimated at 4714 hours

for a one-unit site, 5030 hours for a two-unit site, and 5506 hours for a three-unit site.

Plant Specific Inspections

Annual supplemental inspection effort was estimated as 45 hours per site plus 200 hours per

region, based on a planning assumption there would be an average of 1.5 White findings per

site and one Yellow finding per region. An additional equal number of hours were added to the

estimate for inspection preparation and documentation. 

Special Inspections (SI) and Augmented Inspection Team (AIT) inspections in response to

events were estimated at 1275 hours per region (three event response inspections at 250 hours

each for 750 hours of direct inspection effort, and 525 hours for inspection preparation and

documentation).

Generic Safety Issues Inspections 

Generic Safety Issues inspection effort was estimated at 60 hours per site per year (30 hours

direct inspection effort and 30 hours for inspection preparation and documentation).

Licensee Performance Assessment

Performance assessment was estimated at 192 hours per site per year. This estimate was

based on best available recent historical data.

A summary of the above assumptions and resulting regional allocation of resources for the

revised ROP based on the above assumptions are provided in Tables 1, 2, and 3.

Table 1 summarizes the baseline inspection procedure resource estimates and annual

equivalents for single-, dual-, and triple-unit sites.

Table 2 provides a summary of the assumptions and bases for the regional resource model.

Table 3 summarizes the regional allocation of resources based on this model and the specific

distribution of units and sites per region.

Actual Resource Expenditures During Initial Implementation of ROP

Tracking resource expenditures during the first year provided the means to benchmark the

resource estimates and will permit improvements to the regional resource model and help
3

identify areas where efficiencies may be possible. Detailed Regulatory Information Tracking

System (RITS) codes were developed in order to track time spent on specific activities in the

ROP. Guidance was provided to regional staff, and inspectors were instructed to complete the

procedures and implement the program as required. This guidance was formalized in the ìRITS

Users Guideî and is included in draft NRC Inspection Manual Chapter 0306, ìInformation

Technology Support for the Reactor Oversight Process.î

A necessary step in evaluating resource implications of the revised ROP was to gather the

necessary data, tabulate it, and analyze the results. The hours expended to carry out the ROP

at operating power reactors during the first year of implementation were obtained through

selective searches of data in the RITS. In addition, contractor effort was included, as

appropriate, for all inspections that were supplemented with contractor assistance. The results

are tabulated several different ways in Tables 4 through 8 and are discussed in detail in the

pages that follow.

(It should be noted that there may be small variations in the actual hours reported within the

different tables in this section. These variations are the result of the minor corrections and

revisions made to the hours reported in the Regulatory Information Tracking System (RITS) from

one week to the next, such that the same search performed on different days may yield slightly

different results especially for the more recent RITS entries. The actual hours expended were

obtained during the second and third weeks of April 2001) 

Baseline Inspections

The expenditures for baseline inspection activities are provided in Tables 4 and 5.

Table 4 shows the actual hours expended for baseline inspection activities compared with initial

estimates for single-, dual-, and triple-unit sites. The results show that, with the exception of the

two triple-unit sites, the actual resource requirements to accomplish direct inspection, inspection

preparation and documentation, and plant status activities were less than initial estimates. 

However, as discussed below, these specific activities do not represent the total effort needed to

conduct routine inspection activities.

Table 5 provides the same information by office. Again the results show that actual

expenditures for these activities are less than initial estimates across all regions. The discussion

that follows provides some perspective to this comparison. 

Except for inspection preparation and documentation, the resource requirements for other

indirect inspection activities were not explicitly identified in the regional resource model. It was

assumed that the time expended on these activities would be small and would be

accommodated within allocated program resources; however, anecdotal regional feedback

suggested that indirect activities that were not accounted for; i.e., regional support of inspections

(REG), routine communications (COM), and significance determination process (SDP),

consumed a measurable amount of inspector effort. In order to determine the actual effort

required for these activities, expenditures for the indirect inspection activities; i.e., inspection

related travel (AT), routine communications (COM), regional support of inspections (REG),

inspection preparation (BIP) and documentation (BID), significance determination process

(SDP), enforcement support (ENF) and review of technical documents (RLD), were tracked

during the first year of initial implementation. 
4

The actual hours charged to the indirect inspection activities are provided, by region, in Table 6. 

The results show that, in some cases, expenditures in these activities are significant, such as the

Region I expenditures for inspection related travel, routine communications and regional

assistance. These indirect activities were charged against the three inspection program areas

(baseline inspections, plant specific inspections, and GSI inspections) in proportion to the

expected effort in the three areas. Therefore, if the effort for the support activities shown in

Table 6 is added to the totals shown in Table 5, the difference between the actual baseline hours

and estimated hours in Table 5 becomes less.

Tracking the actual effort in these activities provides a basis to improve the accuracy of the

regional resource allocation model by either explicitly accounting for these indirect activities or

adjusting the allocation for inspection preparation/documentation. The staff will continue

tracking expenditures in these activities and will make appropriate adjustments in the regional

model for baseline inspection effort. 

Plant Specific Inspections

A comparison of actual total hours expended for plant-specific inspections vs. estimated hours,

by region, is provided in Table 7.

As already discussed, the regional allocation for plant specific direct inspection effort was based

on an assumption that 1.5 White findings per site would result from performance indicators and

inspection findings (67 sites x 1.5 = 100.5 total White findings) at 30 hours per finding = 3,015

hours, four Yellow findings would be identified at 200 hours each (800 hours), and 3 event

response inspections per region would be conducted at 250 hours per inspection (3,000 hours)

for a total direct inspection effort of 6,815 hours. It was assumed that there would be no Red

findings and thus, no resources were allocated for supplemental inspections to resolve Red

findings.

In actuality during this first year, Inspection Procedure (IP) 95001, inspection for one or two

White inputs in a strategic performance area, was implemented 30 times for a total direct

inspection effort of 852 hours; IP 95002, inspection for a degraded cornerstone, was performed

four times for a total direct inspection effort of 739 hours; and IP 95003, supplemental inspection

for a Multiple/Repetitive Degraded Cornerstone or one Red input, was completed at one plant for

a total direct inspection effort of 1,724 hours.

One follow-up inspection to an AIT and seven Special Inspections were performed as

determined from the hours charged to Inspection Procedures IP 93800 and IP 93812,

respectively. The total direct inspection effort charged to these two procedures was 1,926 hours

compared with an initial estimate of 3,000 hours (750 hours per region). 

The total NRC staff inspection effort charged to plant-specific inspection at operating power

reactors, including direct inspection effort and inspection preparation and documentation at all

sites, was 17,980 hours. This includes supplemental inspections, event response inspections

and all other plant specific inspections. 

Based on an analysis of the resources expended in this area compared to the regional resource

model, we can conclude the following as a result of the first year:
5

ï The number of plants where IP 95001 was applied (White inspection findings or White

performance indicators) and the effort to resolve these were significantly lower than

estimated (852 actual hours vs. 3,015 estimated hours).

ï The number of times IP 95002 for a degraded cornerstone was exercised (Yellow or

multiple White findings) and the effort to resolve these were both accurately estimated (4

actual vs. 4 estimated; 739 actual hours vs. 800 estimated hours for direct effort).

ï The inspection effort and oversight associated with a plant in a multiple/degraded

cornerstone was significant. The required regulatory oversight not only consumed

considerable effort for direct inspection and inspection preparation and documentation,

but also significantly increased the expenditures of other indirect activities such as

inspection related travel, routine communication, significance determination process, and

regional technical support. 

Expenditures for plant-specific inspections depend directly on performance indicators, the results

of baseline inspections, and events that may occur during the year. The total direct inspection

effort expended on plant specific inspection during this first year is reasonably close to the initial

estimate; however, the resource allocations in this area need to be refined. In the coming year,

the staff will continue tracking events and expenditures in these activities and will make needed

adjustments in the resource model and assumptions to more accurately reflect the number and

resource requirements for White, Yellow and Red findings.

Generic Safety Issues Inspections

The GSI/SI expenditures vs. estimates are shown in Table 7. Due to the limited effort in this

area during this first year of implementation, it is not appropriate to draw definitive conclusions

regarding the initial assumptions. The total expenditures in this area depend on the number of

issues that surface during the year. During this first year total expenditures for direct inspection

were reported as 1,389 hours, while 1,030 hours were charged to inspection preparation and

documentation, for a total of 2,419 hours. The actual hours are less than the 4,026 hours

estimated. This is consistent with a recent trend of reduced numbers of GSI inspections and

may lead to inspection resource savings. The estimated annual 60 hours per site will be

maintained for the near term until the assumptions can be validated with greater confidence to

see if this estimate can be reduced.

Summary of First Year Expenditures

Table 8 provides a comparison of actual total hours charged to the ROP inspection and

assessment program areas during the first year of implementation compared with the initial

estimated resource needs. All inspection and assessment related activities are included in the

actual total charges, including the indirect activities such as inspection related travel, regional

support, enforcement support, etc. 

The following can be inferred from the results shown in Table 8:

ï Baseline Inspection: The effort required for the baseline program element was

underestimated for Regions I and III, and overestimated for Region II. These variances

need to be investigated in more detail to understand their basis.
6

ï Plant Specific Inspections: The initial estimate for plant specific inspections only

considered direct inspection effort and inspection preparation and documentation. If only

these activities are considered, the initial estimate is fairly accurate as can be seen from

the data in Table 7. However, if all activities are considered, as shown in Table 8, the 

resource expenditures for the plant specific inspection program element were

underestimated to varying degrees for all four regions. Although the Region I variance

can be attributed, in part, to the increased activities related to Indian Point 2, as already

discussed, the greater than anticipated effort in this program area indicates that

adjustments to the regional resource model are required.

ï GSI/SI Inspections: As already discussed, no adjustments to the GSI/SI allocation will be

made in the near term until the quality of the data can be validated and potential resource

savings are identified.

ï Performance Assessment: Table 8 also provides actual expenditures charged to licensee

performance assessment activities. It must be noted that the time period on which these

assessment numbers are based does not capture the major activities for the first end-ofcycle plant assessments in the revised ROP. Those major assessment activities are

scheduled six weeks after completion of the inspection cycle and are not reflected in the

hours in Table 8. Also, a significant portion of the hours charged to performance

assessment were incorrectly reported to SALP activity without a corresponding docket

number. Because of these inadequacies in the database and in recording and reporting

the time actually spent performing assessment activities, the quality of these data can not

be confirmed with absolute certainty. The staff will continue tracking assessment

expenditures with more accurate reporting to determine if adjustments to the regional

model are required.

The initial resource estimates for the ROP were based on the best available information and

expert opinion. As discussed above, a number of issues were identified that indicate the

estimates need to be refined as more experience is gained implementing the process and more

accurate, longer term data are available.

Comparison with Previous Reactor Oversight Program

Table 9 provides a side-by-side display of the resource expenditures of the ROP during the first

year of implementation with the resource expenditures for the same time period prior to initial

implementation. The numbers shown in Table 9 include all direct and indirect inspection effort at

operating power reactors expended by NRR and the regional offices during the two time periods. 

Although there are similarities between the previous inspection/assessment programs and the

ROP, a number of significant, elemental differences make a direct comparison difficult and

potentially problematic. It should also be noted that, because of the transition in inspection

processes, neither the 52 weeks prior to implementation of the ROP, nor the 52 weeks of initial

implementation can be considered a typical inspection year. 

The previous program consisted of core inspections with considerable latitude for regional

initiative based on licensee performance as determined by the results of the systematic

assessment of licensee performance (SALP). In contrast, the revised ROP applies the same
7

baseline inspection effort to all sites and the extent of supplemental inspection is determined by

the significance of baseline inspection findings and performance indicators. Also, in the ROP,

the effort required for specific indirect activities such as; significance determination process,

plant status, regional support, and routine communication, is tracked separately. In the previous

program these activities were included as part of other elements of the program. However, even

with these differences it is possible to make some general observations.

As in the previous program, the majority of inspection effort is in baseline/core activities. Also,

as expected, even with the effort at Indian Point 2, plant specific inspection activities in the ROP

are a much smaller fraction of total inspection effort than in the previous program. 

The effort for direct inspection activities and plant status for Baseline/Core and Plant Specific

inspections is less for the ROP than for the previous program (183,493 hours for the ROP and

192,867 hours for the previous program).

The effort for direct inspection, plant status, and inspection preparation and documentation for

Baseline/Core and Plant Specific inspections is comparable for the ROP and the previous

program (306,111 hours for the ROP and 307,925 for the previous program). We expect,

however, that greater familiarity with process and procedures will decrease preparation and

documentation time. 

A direct comparison of the GSI/SI hours and Performance Assessment hours is not appropriate

for the reasons already discussed.

Additional Considerations

Contractor assistance was used in conducting a number of major team inspections during the

past year. These hours have been applied and are reflected in the various tables presented.

Contractor assistance is equivalent to approximately 8915 hours of additional inspection

resources. A breakdown by inspection area and region during the first year of implementation is

shown in Table 10. Future availability of contractor assistance will have significant resource

implication for the ROP absent any other changes. 

Also, as noted in Attachment 7, Inspection Program, the Physical Protection Cornerstone

baseline inspection process is continuing to develop and will likely change based on rule

changes and licensee self-assessment efforts. Appropriate resources to address these changes

will be evaluated as the changes are finalized.

Conclusions

This first year of ROP implementation provided a benchmark for future decisions on resource

expenditures. Based on the results of the first year of ROP implementation, the staff concludes

that:

ï Current regional resources were adequate to effectively carry out the ROP and achieve

its stated objectives during initial implementation. Regional variation in resource usage is

not excessive and may be explained by unique circumstances (IP2 activities), or specific

process efficiencies.
8

ï Overall resource requirements for the ROP during initial implementation were

comparable to the overall requirements in the previous program although the resources

are allocated differently.

ï Future resource reductions may be possible through efficiency gains as a result of:

elimination of start-up costs; inspection preparation time decreasing as staff becomes

more familiar with process and procedures; improved documentation methods (such as

implementation of quarterly integrated inspection reports in all regions for routine

inspections); use of SDP workbooks at all sites; and reduced frequency and level of

activities related to licensee performance assessment. Resource savings may result if

ongoing staff review and evaluation recommends a reduction in the scope and/or

frequency of major team inspections or other baseline inspections. 

ï A number of other factors may impact resource requirements for the ROP in the future

that are difficult to estimate. These include: institutional inefficiencies that may result

from full implementation of ìNî resident inspector staffing policy (greater inspection

preparation and travel time); reduced inspector efficiency due to increases in vacancies

filled at an entry level; unplanned issues/events that result in significant plant specific

inspections or unplanned demand on inspectors; and additional inspection resources due

to the increase in independent spent fuel storage installation (ISFSI) facilities.

ï In total, the current regional resource model, modified based on the results of the first

year of ROP implementation, provides reasonably accurate estimates of regional

resource requirements to implement the ROP. The model will be refined as more data

are gathered and evaluated. Areas considered for potential review include anomalies in

regional resource usage relative to the resource model (e.g.; more onerous travel

requirements for Region IV, additional resources for dual-unit sites that are significantly

different in design, vintage or management), refinement of the model to include resource

impacts of multiple degraded cornerstones, explicit inclusion of indirect inspection

activities in the resource model, and more accurate data for performance assessment

expenditures.

These factors will be addressed and evaluated for resource impacts as part of the ongoing

improvement process; and efficiency gains will be implemented over the next several years. As

part of the effort to develop the FY 2004 budget, the staff will reflect savings resulting from

efficiency gains in FY 2003 and beyond.

The following should also be given consideration in any decision related to resource

requirements for the ROP:

- There has already been a substantial decrease in inspection hours expenditures

over the past five years to reflect improved industry performance and anticipated

efficiencies from the ROP.

- Although current resources appear to be appropriate for the near term, additional

data will be collected and analyzed as we move out of the start-up phase and

complete the transition to ìNî resident staffing.
9

- Efficiency gains can be implemented over the next several years. The selfassessment program will provide for ongoing review of the ROP.

- Factors such as entry level hires, resident inspector quality of life.

- The impact of growth areas such as construction inspection and ISFSI. 
10

Table 1

Baseline Inspection Resource Estimates

Consensus Notes One Unit Two Units 

Procedure Attachment Title Hours Delta Hours Delta Hours Delta 

71111 1 Adverse Weather 18 3 0 18 3 18 3 71111 2 Evaluation of Changes, Tests... 32 6 1 30 4 32 6 71111 4 Equipment Alignment 80 10 1 72 9 80 10 71111 5 Fire Protection monthly 33 3 0 33 3 33 3 71111 5 Fire Protection Triennial 200 40 4 67 13 67 13 71111 6 Flood Protection 20 4 0 20 4 20 4 71111 7 Heat Sink 6 1 0 6 1 6 1 71111 7 Heat Sink Biennial 32 2 3 16 1 16 1 71111 8 ISI Biennial per site 32 4 0 16 2 16 2 71111 11 Lic Operator Requal 12 2 0 12 2 12 2 71111 11 Lic Op Requal Biennial 96 10 3 48 5 48 5 

71111 12 Maintenance Rule 216 20 0 216 20 216 20 

71111 12 Maint Rule Bienniel per site 40 10 3 20 5 20 5 

71111 13/03 Emergent Work & Maint risk 120 20 1 108 18 120 20 

71111 14 Non-routine evolutions 102 10 1 92 9 102 10 

71111 15 Operability Evaluations 3.5hr per

18/22/26

72 8 1 63 7 77 8 

71111 16 Operator Workarounds 35 3 1 32 3 35 3 71111 17 Plant Mods Biennial 80 8 0 40 4 40 4 

71111 17 Permanent Plant Mods 16 2 0 16 2 16 2 71111 19 Post Maint Testing 84 8 0 84 8 84 8 71111 20 RFO per refuel per unit 80 8 2 54 5 107 11 

71111 21 SSDI Biennial 420 60 3 210 30 210 30 71111 22/09/10 Surv Testing & IST 132 12 0 132 12 132 12 

71111 23 Temp Mods 40 6 1 36 5 40 6 71114 1 EP drills & exercise Biennial 12 2 3 6 2 6 2 71114 1 EP drills & exer Biennial 52 6 3 26 3 26 3 71114 2 Alert & Notif system Biennial 8 2 3 4 1 4 1 71114 3 Response Org Augmen Biennial 8 2 3 4 1 4 1 

71114 4 EAL evaluation and EP Changes 16 2 0 16 2 16 2 

71114 5 EP Weaknesses & Def Biennial 12 2 3 6 1 6 1 71114 6 Drill Evaluation 10 2 0 10 2 10 2 

71121 1 Rad Access Controls 32 3 0 32 3 32 3 
11

71121 2 ALARA Biennial 120 12 3 60 6 60 6 

Consensus Notes One Unit Two Units 

Procedure Attachment Title Hours Delta Hours Delta Hours Delta 

71121 3 Rad Instruments 32 3 0 32 3 32 3 71122 1 Effluents Biennial 32 3 3 16 2 16 2 

71122 2 Rad Mat. Shipping Biennial 32 3 3 16 2 16 2 

71122 3 Environmental Biennial 32 3 3 16 2 16 2 71130 1 Security Access Authorization 12 2 0 12 2 12 2 71130 2 Security Access Control 24 2 0 24 2 24 2 71130 3 Contingency Events Biennial 104 10 3 52 5 52 5 

71130 4 Security Plan Changes Biennial 16 2 3 8 1 8 1 

71151 NA PI verif Dependent on # of units ** * 5 35 3 50 5 

71152 NA PI&R 210 30 0 210 30 210 30 

71153 NA Event Follow-up 18 2 1 16 2 18 2 

TOTAL DIE 2042 250 2165 266 Plant Status 630 68 700 755 Total DIE 2042 250 2165 266 Est.Prep/Doc

2042 250 2165 266 

BaselineProgram

4714 568 5030 607 
12

Notes

0 Nominal procedural estimates are applied to all sites regardless of number of units.1 Procedural estimates are for a 2-unit site. 1-unit sites are estimated at 90% of nominal and 3-unit sites are esti2 Procedural estimates are per occurrence for an estimated 18-month refueling cycle. All units receive the inspe3 These procedures are performed biennially. Per occurrence estimates are applied at 50% per year at all sites.4 This procedure is performed triennially. Per occurrence estimate is applied at 33% per year at all sites.5 PI verification is dependent on the number of units. For a one unit facility, the verification of the PIs is estimate

overlap of PIs, for each additional unit, this procedure will require an additional 15 hours for a total of 50 hours ahours for a three unit facility. This is exclusive of the hours to accomplish the PI TI which is estimated at 25 hou

Assumptions

ï The total of Preparation and Documentation is charged at a rate of 1 hour to every hour of inspection.ï Only baseline inspection program hours were included in the assessment.ï The security and safeguards, radiological safety and emergency preparedness inspection procedures were rev

reallocated based on the results of the review. Hour estimates are believed to be reliable.

ï The ìConsensus Hoursî for IP 71111.02 were revised from 40 hours to 32 hours annually. This reflects the tran

changesî from this procedure into EP and security procedures.
13

Table 2

REACTOR OVERSIGHT PROCESS

REGIONAL RESOURCE MODEL

Baseline Inspections:

1-unit site 2-unit site 3-unit site

Direct Inspection Effort 2042 hours 2165 hours 2333 hours

Inspection Prep/Doc 2042 2165 2333

Plant Status 630 700 840

Total 4714 hours 5030 hours 5506 hours

Annual Regional Inspection Hours Required = (4714 x number of 1-unit sites in region) 

+ (5030 x number of 2-unit sites in region) 

+ (5506 x number of 3-unit sites in region) 

Plant Specific Inspections:

Direct Inspection Effort = (30 hours/White finding x 1.5 White findings/site x number of sites in

region) + (200 hours/Yellow finding x 1 Yellow finding/region)

Inspection Prep/Doc = Direct Inspection Effort

Special Inspections/AIT Effort = 250 hours/inspection x 3 inspections = 750 hours/year/region

Inspection Prep/Doc = 70% of direct effort = 525 hours/year/region

GSI inspections:

Direct Inspection Effort = 30 hours/site x number of sites in region

Inspection Prep/Doc = Direct Inspection Effort

Performance Assessment:

Effort is estimated at 192 hours/site x number of sites per region
14

Table 3

Estimated Regional Resource Requirements For Reactor Oversight Process

By Program Element

(Hours)

Program Element Region I

(12 single-unit sites)*

(7 dual-unit sites)

Region II

(5 single-unit sites)

(12 dual-unit sites)

(1 triple-unit site)

Region III

(8 single-unit sites)

(8 dual-unit sites)

Baseline Direct Inspection EffortBaseline Inspection Prep/DocPlant Status Activities Sub-Total

396593965912460

91778

385233852312390

89436

336563365610640

77952

Supplemental Direct Inspection EffortSupplemental Inspection Prep/DocSpecial Inspection/AIT Direct EffortSpecial Inspection/AIT Prep/Doc Sub-Total 

10551055 750 525

3385

10101010 750 525

3295

 920 920 750 525

3115

GSI/SI Direct Inspection EffortGSI/SI Inspection Prep/Doc Sub-Total

 570 570

1140

 540 540

1080

 480 480

 960

Performance Assessment 3648 3456 3072 Total Estimated RegionalRequirements 99951 hours 97267 hours 85099 hours 

* IP2, IP3, Millstone 2, Millstone 3 are considered single-unit sites. Hope Creek and Salem 1 & 2 are considered a single-unit and a dual-unit site, respectively.
15

Table 4

BASELINE INSPECTION ACTIVITIES

Actual Total Hours vs. Estimated Hours

By Number of Units

(Includes Hours Charged by Regional Offices, NRR and Contractor Hours)

4/2/00 - 4/1/01

Activity Single-Unit Sites Dual-Unit Sites Tri

Actual Hours Est. Hours* Actual Hours Est. Hours* Actual Ho

Baseline Direct InspectionEffort 60449 67386 67046 69280 4818 Baseline InspectionPreparation/Documentation 54040 67386 59763 69280 5991 Plant Status Activities 20184 20790 21798 22400 1717  Total Hours 134673 155562 148607 160960 12526 Average Baseline DirectInspection (hours/site) 1832 2042 2095 2165 2409 Average Plant Status(hours/site) 612 630 681 700 859 Prep+Doc/Direct Insp Effort 0.89 1.0 0.89 1.0 1.24 

* Based on 33 single-unit sites, 32 dual-unit sites, and 2 triple-unit sites
16

Table 5

BASELINE INSPECTION ACTIVITIES

Actual Total Hours vs. Estimated Hours

By Region

4/2/00 - 4/1/01

Activity Code

Region I

(12 single-unit sites)

(7 dual-unit sites)

Region II

(5 single-unit sites)

(12 dual-unit sites)

(1 triple-unit site)

Region III

(8 single-unit sites)

(8 dual-unit sites)

Regi

(8 single-

(5 dual-u

(1 triple-

Actual Hrs Est Hours Actual Hrs Est Hours Actual Hrs Est Hours Actual Hrs 

Direct InspectionEffort 39798 39659 30803 38523 33506 33656 23581 InspectionPreparation/Documentation 33925 39659 29006 38523 27516 33656 23418 Plant Status 12449 12460 11559 12390 10880 10640 8863 Total Staff Hours 86172 91778 71368 89436 71902 77952 55862 Contractor Hours 1245 2110 2260 2580 Total BaselineHours 87417* 91778 73478* 89436 74162* 77952 58442* 

* These totals only include direct inspection effort, inspection preparation/documentation, plant status hours, anThe portion of the hours shown in Table 6 expended for indirect activities to support baseline inspections are no
17

Table 6

Regional and NRR Hours Charged To Indirect Inspection Activities 

At Operating Power Reactors

4/2/00 - 4/1/01

Activity

Region I

19 sites

Region II

18 sites 

Region III

16 sites

Region IV

14 sites

TotalHours*

Hours/site

TotalHours*

Hours/site

TotalHours*

Hours/site

TotalHours*

Hours/site

Inspection RelatedTravel 5782 304 4693 261 4347 272 5125 366 Routine Communication

6628 349 3412 190 2244 140 2901 207 

Technical DocumentsReview 106 6 226 13 2 - 182 13 Regional Assistance

1334 70 465 26 843 53 418 30 

SignificanceDetermination 1268 67 954 53 1208 76 909 65 Enforcement Support

794 42 1139 63 330 21 927 66 

Total 15912 10889 8974 10462 

*Actual total hours (regular + non-regular) charged 4/2/00 to 4/1/01. These hours were not explicitly accounted resource model.
18

Table 7

PLANT SPECIFIC/GSI INSPECTIONS

Actual Total Hours vs. Estimated Hours

4/2/00 - 4/1/01

Activity

Region I

(12 single-unit sites)

(7 dual-unit sites)

Region II

(5 single-unit sites)

(12 dual-unit sites)

(1 triple-unit site)

Region III

(8 single-unit sites)

(8 dual-unit sites)

(8 si

(5 d

(1 t

Actual Hrs Est Hours Actual Hrs Est Hours Actual Hrs Est Hours Actual 

Plant SpecificInspections Direct Effort 4676 1805 1979 1760 2350 1670 1088 Plant SpecificInspectionPrep/Doc 3316 1580 1096 1535 559 1445 1332  Total hours

7992 Staff 720 Contr

8712 3385 3075 3295 2909 3115 2420 

Generic/SafetyInspections Direct Effort 308 570 292 540 483 480 306 GSI/SI InspectionPrep/Doc 431 570 315 540 221 480 63  Total hours 739 1140 607 1080 704 960 369 
19

Table 8

Actual Total Hours vs. Estimated

By Program Element

(All Inspection and Oversight Activities and Contractor Effort at Operating Power Reactors

4/2/00 - 4/1/01

Program Element Region I Region II Region III Region 

Actual Hrs Est Hours Actual Hrs Est Hours Actual Hrs Est Hours Actual Hrs 

BaselineInspections 100346 91778 80510 89436 79632 77952 65057 Plant SpecificInspections 9663 3385 4715 3295 4078 3115 3655 GSI/SIInspections 908 1140 711 1080 778 960 445 PerformanceAssessment 3632 3648 3101 3456 6332 3072 2979 ContractorAssistance 1965 2110 2260 2580Total 116514 99951 91147 97267 93080 86099 74716 
20

Table 9

Resources Expended

ROP vs. Previous Program

(Total Staff Effort Expended at Operating Power Reactors)

(NRR, RI, RII, RIII, RIV)

52 weeks prior to 52 weeks post

implementation implementation

4/4/99-4/1/00 4/2/00-4/1/01

Baseline/Core

Direct Inspection Effort 142,259 128,447

Inspection Prep/Doc 108,730 115,935

Plant Status 74 43,751

Subtotal 251,063 288,133

Plant Specific Inspections

Direct Inspection Effort 50,534 11,295

Inspection Prep/Doc 6,328 6,683

Subtotal 56,862 17,978

GSI/SI 2,709 2,416

Performance Assessment 23,746 21,017

Inspection Related Travel 16,524 20,193

Routine Communication 15,185

Regional Support 3,272

Enforcement Support 4,408 3,397

Significance Determination Process 4,628

Review of Technical Documents 1,270 515

Total Staff Effort (reg + nonreg hrs) 356,582 hours 376,734 hours
21

Table 10

Contractor Inspection Hours Expended

4/2/00 - 4/1/01

Engineering

Inspections

Fire Protection

Inspections

Region Total

(hours)

Region I 1650* 315 1965

Region II 1160 950 2110

Region III 1630 630 2260

Region IV 1630 950 2580

Total Effort (hours) 6070 2845 8915

* Includes approximately 720 hours, one-time, contractor support for IP2 restart
CROSS-CUTTING ISSUES

Task Lead: Jeff Jacobson, NRR/IIPB

Appendix A
1

Cross-cutting Issues

Background

In the framework for the revised reactor oversight process (ROP) described in SECY-99-007,

the staff explained how cross-cutting issues potentially impacting more than one cornerstone

were addressed during the cornerstone development process. Specifically, the staff explained

how the cross-cutting issues of human performance, safety-conscious work environment, and

problem identification and resolution (PI&R) were linked to either performance indicators (PIs) or

baseline inspection areas. In the past, these cross-cutting issues have been shown to be the

root causes of performance issues. 

With regard to human performance, a premise of the ROP is that if risk-informed inspections

and PIs together indicate that plant performance is meeting the cornerstone objectives, then

those findings also provide an indication of the acceptability of the associated human objectives. 

As such, the baseline inspection program does not include a specific human performance

inspection. In a similar fashion, the ROP does not include a specific baseline inspection to

assess safety-conscious work environment, since it was thought that a lack of a safetyconscious work environment would result in an increase in problems and events; however,

insights into this cross-cutting issue are obtained during periodic reviews of licenseeís PI&R

programs and via the Agencyís review of allegations. This periodic PI&R baseline inspection

also provides the NRC with the opportunity for early indication of potentially declining plant

performance. 

In SECY-00-0049, the staff described feedback obtained from stakeholders with regard to how

cross-cutting issues were being treated in the ROP. The feedback was essentially divided into

two views. One view supported the above assumptions that significant weaknesses in the crosscutting areas would reveal themselves by PIs crossing thresholds or via the identification of

inspection findings. The other view supported the opinion that significant programmatic

breakdowns could occur without prior indication through PIs or inspection findings, and that the

specific evaluation of licensee performance in the cross-cutting areas would provide a timely

opportunity for the NRC to respond to these concerns. Due to the inability to reach consensus

on this issue, the staff recommended the formulation of a working group specifically to address

cross-cutting issues.

In its recommendations to the Commission, the Pilot Plant Evaluation Panel (PPEP) also

recommended that the staff seek to confirm the validity of the underlying assumptions made

regarding cross-cutting issues during the first year of implementation.

In the staff requirements memorandum to SECY-00-049, the Commission stated that crosscutting issues should have a clear and strong link to significant inspection findings or degraded

PIs before action is taken on programmatic concerns.

Work Group Activities

As a result of the above commitments and recommendations, a working group was formed

consisting of Headquarters and regional management and staff to focus on the concerns
2

expressed regarding cross-cutting issues. The first meeting was held on April 5, 2000, and the

group developed five key issues to focus on in the cross-cutting area. The five key issues were:

1. Do the PIs and baseline inspection program provide sufficient information regarding

performance in the cross-cutting areas of human performance, safety-conscious work

environment, and PI&R? For the purpose of this issue, ìsufficient informationî can be

thought of as information of sufficient depth and scope and within a sufficient time frame

to allow for appropriate levels of Agency interaction.

2. Are there other cross-cutting issues that warrant additional consideration in the ROP?

3. Does the ROP (e.g. inspection program, significance determination process, Action

Matrix) provide for proper treatment of cross-cutting issues when they are identified? 

Should the approach be the same for all cross-cutting issues or should the approach

vary?

4. What would be the definition of a ìsubstantialî cross-cutting issue that would require

additional Agency actions beyond what the current process would provide?

5. Currently, what is the guidance for capturing cross-cutting issues in inspection reports?

It was recognized by the group that resolution of these issues would likely be a long term effort

and would be aided by the additional experience that would be acquired during the initial

implementation of the ROP.

In its second meeting conducted on September 14, 2000 the group focused on initial

experiences gained during implementation of the annual PI&R inspection. The group also

discussed newly issued guidance that described the approach for documenting cross-cutting

issues in inspection reports. This guidance addressed issue #5 above regarding documentation

of cross-cutting issues. 

In its third meeting conducted on December 11, 2000 the group invited members of industry and

the Institute of Nuclear Power Operations (INPO) to attend and present their perspective on the

cross-cutting issue concerns. INPO gave a presentation on recent activities it has taken to

assess licensee corrective action and self-assessment programs. The group also discussed the

feasibility of developing indicators in the PI&R area. The general consensus expressed was that

this would be very difficult as the PI&R programs are very site specific. There was some

agreement that some generic indicators could be developed, but that the thresholds for the

indicators would have to be site specific. In general, there was little support for such an effort

and some concern was expressed that generic PIs in this area might lead to unintended

consequences. Some members from the industry detailed some of the site specific PIs that they

use in the PI&R area such as one for repeat events.

The group discussed the current approach the NRC uses to assess safety-conscious work

environment. Currently, the only inspection activity where the NRC assesses safety-conscious

work environment is the annual PI&R inspection. Some NRC representatives expressed a

concern that this was a cursory review that might lead to false positives, that is, falsely assuming

that the licensee has established a safety-conscious work environment. There was a general

consensus that the resident inspectors might be in a better position to perform this assessment
 1

There have no events classified as significant precursors to a reactor accident during 2000

and 2001. A significant precursor is defined as an event that has a 1/1000 or greater

probability of leading to a reactor accident.

3

as part of the baseline inspections that they are assigned. Some industry representatives

expressed a concern that NRC inspectors may not be trained to accurately assess this area and

that this area is more subjective in nature, making it difficult to assess. There was also some

discussion about whether this topic is really a separate cross-cutting issue or is it just imbedded

in a licenseeís overall PI&R. 

Preliminary Conclusions Regarding Cross-cutting Issues and Other Stakeholder Feedback 

Cross-cutting issues was a topic that was discussed in both the internal and external NRC

workshops held in March of 2001. In the external workshop, three issues were discussed that

were closely aligned with the first three key issues identified by the above working group. An

additional recommendation to change the approach for inspecting PI&R was also discussed at

the workshop. For each of these issues, the NRC staff presented its preliminary findings based

upon the first year of initial implementation, allowed for questions or comments, and then

discussed its planned future actions.

With regard to key issue number one, the staff explained that experience obtained from the

initial implementation of the ROP thus far tends to support one of the fundamental premises of

the ROP; that degradation in the cross-cutting areas will be detected by either PIs or inspections

in a sufficiently pro-active time frame to allow for Agency action to protect public health and

safety. Examples of where this has proven true include Indian Point 2 (IP2), Kewaunee,

Millstone, and Cooper. At these facilities, problems have been identified during the initial

implementation of the ROP that have been attributed to one of the three cross-cutting areas

(mainly PI&R) and the NRC has performed supplemental inspections due to PIs and/or baseline

inspection findings crossing thresholds. Also, during initial implementation, there have been no

significant precursors to a reactor accident.1

 A significant precursor that was caused by a crosscutting issue might be indicative of a weakness in the ROP approach in this area. Since the

first year of initial implementation of the ROP represents a relatively short experience base,

particularly when trying to determine whether the premise holds under all reasonably credible

situations, these results are preliminary. Further, declines in licensee performance typically

occur over an extended period of time, so it is too soon to reach definite conclusions in this case. 

Feedback received from the Nuclear Energy Institute (NEI) in response to the NRCís Federal

Register notice is supportive of the staffís above preliminary conclusions.

During initial implementation of the ROP, the staff has not identified any cross-cutting issues that

would warrant special treatment. There has been a tacit recognition that there are programs,

such as the maintenance effectiveness and erosion/corrosion programs, that are essentially

elements of a licenseeís PI&R process and thus have cross-cutting aspects to them, but not

necessarily to the degree they should be called out as an individual cross-cutting area. 

Numerous changes have been made to the PIs, baseline inspection program, and significance

determination processes (SDPs) to better direct our resources to those areas of most safety

significance, including areas were not adequately addressed in the past. This assessment

addresses key issue number two identified above. 
4

With regard to key issue number three, the ROP addresses cross-cutting issues by highlighting

them in inspection reports when they are notable contributors to inspection findings or if there is

an appreciable trend or pattern that has emerged; and in assessment letters to the licensee

when they constitute a substantive issue. Recent changes made to Inspection Manual Chapter

0610*, ìPower Reactor Inspection Reports,î better explain when and how cross-cutting issues

should be documented in inspection reports. The ROP does not allow for additional NRC

engagement on cross-cutting issues unless they are contributing causes to PIs or inspection

findings that are characterized as White or greater. The Commission previously directed the

staff to specifically inform them if the NRC decides to engage licensees outside of the Action

Matrix because of cross-cutting issues. To date, the staff has not engaged licensees on crosscutting issues outside of the guidance contained in the Action Matrix. Engagement has occurred

per the Action Matrix when significant cross-cutting issues have contributed to issues that have

moved licensee performance outside of the Licensee Response column. 

Based on the staffís preliminary assessment, initial ROP data, and feedback received from

internal and external stakeholders, the staff has not identified a need for significant modifications

to the ROP to address cross-cutting issues; however, as detailed below, the staff will continue to

assess events and inspection findings to look for safety significant areas not adequately covered

by the current baseline inspections, PIs, or SDPs. As such, the staffís preliminary assessment

of key issue number four is that there is not a need to take additional actions for ìsubstantialî

cross-cutting issues, beyond what the current process already provides.

In addition to the key issues identified above, the staff also discussed at the external workshop

some proposed changes that would increase the effectiveness and efficiency of the NRCís

approach to inspecting a licenseeís PI&R program. These proposed changes were the result of

experience gained during the initial year of implementation of the ROP. Following the external

workshop, an internal focus group was formed to study the proposed changes and to develop

additional specific recommendations to improve the effectiveness of the PI&R inspection

approach. The focus group recommended changing the baseline frequency of the annual PI&R

inspection to once every two years for all facilities. Also, the focus group recommended revising

the estimated hours per inspection to 250 hours from the current 210 hours. In addition, in order

to maintain the ability for timely detection of degradation of licensee performance in the PI&R

area, the group recommended adding 60 hours of effort annually to inspection procedure 71152

to allow for focused follow-up/assessment of PI&R samples throughout the inspection period. 

The above recommendations will result in an estimated overall reduction of direct inspection

effort of 25 hours per year in this area. Lastly, for facilities that reach the Degraded Cornerstone

column of the Action Matrix, the focus group recommended the consideration of an additional

PI&R team inspection as part of the supplemental inspection response to assess the extent of

condition of the identified performance issues. Overall these changes should result in a more

effective allocation of NRC resources in this area. Implementation of these recommended

changes is currently planned to begin in January of 2002. Plans are for the focus group to

continue to assess what changes can be made to the inspection approach in this area in order to

make the inspections more effective, efficient, and objective.
 2

 ASP events are events with a conditional core damage probability of equal to or greater

than 1.0 X 10E-6.

5

Additional Planned Actions

During the next year of implementation of the ROP the following activities are being pursued or

considered to further evaluate the adequacy of the ROP with regard to cross-cutting issues:

ï ASP events2

 and inspection findings classified as Yellow or Red will be reviewed to

determine if weaknesses in one of the three cross-cutting areas contributed to the event

or finding. If cross-cutting issues were a major contributor to the event, The staff will

assess whether the PIs or inspection program provided for sufficiently pro-active NRC

engagement on the issues to protect public health and safety.

ï During periodic reviews of issued inspection reports, the staff will evaluate whether crosscutting issues are being sufficiently captured when identified during inspection activities.

ï As part of the self-assessment metrics developed for assessing the ROP, the staff will

review the circumstances surrounding plants that jump two or more columns in the Action

Matrix to see if these performance weaknesses were due to cross-cutting issues, and if

so, whether inspections or PIs have identified similar concerns. 

ï During our annual assessment of the ROP we will evaluate whether the ROP allowed for

sufficient NRC engagement at facilities that reached the Degraded Cornerstone column

of the Action Matrix.

ï A workgroup will be formed to assess whether changes should be made to how the NRC

assesses safety-conscious work environment.

ï Office of Nuclear Regulatory Research staff will complete research to: (1) characterize

the extent to which the effects of human performance on safety are accounted for

through the current oversight process, (2) characterize the types of human performance

problems that are not accounted for and assess the significance of their effects, and (3)

identify how these effects could be addressed though enhancements to the ROP.

ï The Inspection Program Branch of NRR will evaluate the need for additional refinements

to inspection and assessment guidance regarding the definitions of what constitutes an

ìappreciable trend or patternî or a ìsubstantialî cross-cutting issue. 
RESIDENT INSPECTOR ìN+1" POLICY

Task Lead: Jim Isom, NRR/IIPB

Appendix B
1

Resident Inspector ìN+1" Policy

Background

On January 11, 2000, the Commission approved a revision to the resident inspector staffing

policy to require two resident inspectors at single and dual unit sites and three resident

inspectors at triple unit sites. The Commissionís revision to the resident inspector staffing policy

required that the staff evaluate and recommend inspection staffing issues as part of the first year

implementation of the revised reactor oversight process (ROP). Specifically, the Commission

requested that the staff make recommendation on how the inspection resources should best be

allocated among Headquarters, the regions, and the sites. This attachment provides the staffís

analysis of the resident inspector resources (sites) required to complete the baseline

inspections. 

Assumptions

Inspection hours associated with activities that were either attributable to the resident or the

senior resident inspectors were totaled for each plant during the period from April 2, 2000

through April 1, 2001. The activity codes were selected in an attempt to fully capture the time

spent by the RIs during the first year (April 2, 2000 to April 1, 2001) of ROP implementation to

complete the baseline inspection activities. Some activity codes from the previous core program

(e.g., CO: core inspections) were used because it appeared that time spent on performing

baseline activities were charged to the old core activity codes. These hours were then converted

to full time equivalent staff (FTE) values to determine the resources expended to complete the

portion of the ROP assigned to the RIs (Table 2). These activities included:

APP: Inspection-Related Prep/Doc ASM: Reactor Performance Assessment

AT: Inspection Related Travel BA2: Normal Enforcement Activities

BH2: Escalated Enforcement BI: Baseline Inspections

BID: Baseline Inspection Documentation BIP: Baseline Inspection Preparation

CO: Core Inspections COM: Routine Communications

PS: Plant Status SDP: Significant Determination Process

SI: Safety Issues Program ENF: Enforcement 

Additionally, hours that were charged by region-based inspectors for completing the following

baseline inspections were also totaled over the initial year of ROP implementation and 

converted to a FTE value (Table 3). These expended regional resources were also adjusted for

inspection-related prep/doc and for inspection related travel. Finally, RI and regional resources

were summed to obtain the total resources expended to perform the part of the ROP assigned to

be completed by the RIs (Table 1). 

Procedure: 71111

Attachment Title: Attachment Title:

1 Adverse Weather 4 Equipment Alignment

5 Fire Protection Monthly 6 Flood Protection

7 Heat Sink 11 License Operator Requalification
2

12 Maintenance Rule 13/03 Emergent Work & Maint Risk

14 Non-routine evolutions 15 Operability Evaluations

16 Operator Workarounds 17 Permanent Plant Mods

19 Post Maint Testing 20 RFO 

22/09/10 Surveillance Testing & IST 23 Temp Mods

Procedure: 71114

Attachment Title: Attachment Title:

1 EP drills & exercise 6 Drill Evaluation

Procedure 71152: PI Verification

Procedure 71153: Event Follow-up

Table 1 is a tabulation of resources (expressed in Full-time equivalent staff (FTE)) required to

complete the baseline inspections assigned to the resident inspectors. The table also includes

the FTE expended by region-based inspectors (from either Division of Reactor Projects or

Division of Reactor Safety) who assisted the RIs to complete the baseline inspections. The

resources in Table 1 do not include the FTE required to complete other baseline inspections

assigned to be completed by the Division of Reactor Safety (DRS). 

Inspection resources associated with Indian Point 2 (IP2) in Region I and D.C. Cook in Region III

were not included in any of the tables. IP2 was a performance outlier (Multiple/Repetitive

Degraded Cornerstone column of the Action Matrix) and D.C. Cook is under the IMC 0350

process and is transitioning to oversight under the ROP. Therefore they received many more

inspection resources than the typical facility. 

Two methods were used to convert the hours of inspection performed to the number of

inspectors required to perform the baseline inspections. For the region-based inspectors, 1167

hours of inspections per FTE was used to convert the total inspection hours to an FTE value. 

This value is used by NRR in its budget model. The conversion factor of 1167 hours to an FTE

was empirically developed by taking data averaged over several years, including all regions and

all regional inspectors. 

For the resident inspectors, a conversion factor was derived by dividing the average hours of

inspection performed at the site by the number of inspectors assigned to the site. A different

conversion factor was derived for the RIs because the staff recognized that the RIs were more

efficient at performing baseline inspections than the region-based inspectors. The inspector

number was adjusted for overtime performed by the resident inspectors (RIs) since use of

overtime was equivalent to availability of additional resources at the site to perform inspections. 

A unique conversion factor, which was averaged, was used for the following categories: single

unit site; dual unit sites staffed at ìNî RIs; and dual unit sites staffed at ìN+1" RIs for each of the

four regions. This conversion factor was based on data obtained during the initial

implementation of the ROP and generally resulted in a conversion factor which was greater than

the NRR official budget model of 1167 hours of inspections for an FTE. 
3

Resident Inspector Inspection Hours Data

Table 1 displays the FTE required to complete baseline inspections by both resident and regionbased (either DRP or DRS) inspectors for different types of sites in each of the Regions. 

Table 1

FTE Expended By RIs and Region-Based Inspectors to Complete

Baseline Inspections

Single unit sites Dual unit sites 

at "N"

Dual unit sites

 at "N+1"

3 unit sites 

Region I 2.23 2.31 3.16 0 (note 1) 

Region II 2.07 2.14 3.07 4.03 (note 2) 

Region III 2.24 2.36 2.71 0 (note 1) 

Region IV 2.22 2.25 0 (note 3) 3.14 (note 4)

National

Avg

2.19 2.26 2.98 3.58

National

Median

2.22 2.28 3.07 3.58 

Resources

Estimated 

2.36 2.63 2.63 3.02

Table 2 displays the FTE expended to complete baseline inspections by only the resident

inspectors for different types of sites in each of the Regions. 

Table 2

FTE Expended By Only RIs to Complete Baseline Inspections

Single unit sites Dual unit sites 

at "N"

Dual unit sites

 at "N+1"

3 unit sites 

Region I 2.09 2.05 3.09 0 (note 1) 

Region II 2.04 2.05 3.05 4.03 (note 2) 

Region III 2.11 2.15 2.55 0 (note 1) 

Region IV 2.07 2.1 0 (note 3) 3.1 (note 4) 

National

Avg

2.07 2.08 2.89 3.56 

National

Median

2.08 2.07 3.05 3.56 
4

Table 3 displays the additional FTE required to augment the resident inspector resources in

order to complete the baseline inspections. 

Table 3

FTE Expended By Region Based Inspectors 

to Complete Baseline Inspections

Single unit sites Dual unit sites 

at "N"

Dual unit sites

 at "N+1"

3 unit sites 

Region I 0.14 0.26 0.07 0 (note 1) 

Region II 0.03 0.09 0.02 0 (note 2) 

Region III 0.13 0.21 0.16 0 (note 1) 

Region IV 0.15 0.15 0 (note 3) .04 (note 4) 

National

Avg

0.11 0.17 0.08 0.02 

National

Median

0.13 0.18 0.07 0.02 

note 1: no 3-unit sites in the Region

note 2: staffed to ìN+1" resident inspectors

note 3: no dual-unit sites staffed to ìN+1" resident inspectors in R-IV

note 4: staffed at ìNî resident inspectors

Transition to ìNî Resident Staffing Status

Table 4 shows that the Agency is rapidly completing its transition to the ìNî resident inspector

staffing. Greater than 50 percent of the multi unit sites were at ìNî resident staffing status

during the first year of implementation under the ROP.

 Table 4

Multi-unit August 2000 June 2001

Sites @ N 20 27

Sites @ N+1 13 6

Sites with 

exemptions:

2 2

Multi-unit

Sites Total:

35 35
5

Plants @ N+1 (17.14%)

Plants w/ exemptions (5.71%)

Plants @ N (77.14%)

Status of Implementing the Transition 

From ìN+1" to ìNî (06/2001)

Table 5 displays how the ìN+1" resident inspector vacancies are being filled. Nearly all of the

vacancies which resulted from the Agencyís transition to ìNî resident inspector staffing at dual

unit sites were filled by new hires. The positions filled by the departing ìN+1" RIs included:

ï promotion to SRI

ï assigned as RI to another site in one of the four regions

ï assigned or promoted to other organizations in the Agency other than the Division

of Reactor Projects (DRP)

 Table 5

Regions No. of sites 

Transition to ìNî

(01/2000 - 05/2001)

No. of RIs 

Rotated to

Region

*No. of Vacancies 

Filled By New Hires

I4 0 4

II 4 0 4

III 3 3 0

IV 3 0 3

Total 14 3 11

 * Generally, all positions filled by new hires were transferred to 

 Division of Reactor Safety

Feedback from the Regions

The staff requested the regions to assess the degree to which the regions have been able to

allocate resources on the basis of licensee performance through the ìNî resident staffing policy

during the first year of its implementation and its impact on implementing the ROP. 
6

Additionally, the staff requested that the regions provide an assessment of the impact of

transitioning to the ìN" resident staffing policy as it relates to:

ï increased flexibility to facilitate site coverage (e.g., to support scheduled training and

vacations)

ï improved ability to provide support to reactive inspections and to support other baseline

inspections

ï overall ability to support implementation of the ROP

ï any challenges that have resulted that need to be accommodate either through ROP

program guidance or considered in the budgeting process

The following responses from the regions were received by the staff:

Increased Flexibility to Facilitate Site Coverage

ï The current EDO policy of site coverage is still achievable, however, there must be closer

management oversight and coordination. This policy has not yet posed a significant

problem for the sites staffed at ìNî and we recommend retaining the policy as a public

confidence consideration. It should be recognized, however, that the frequency of one

and two day periods with no resident inspector coverage will increase with the revised

policy. There is the potential that over the long term this policy could negatively affect

resident inspector morale when an inspector gets "stuck" as the sole coverage person

because the other resident inspector is off on a long term assignment. Even when

regional resources are used to supplement site coverage needs, they rarely are used for

weekend event response coverage. Although the Agency policy is to be able to respond

to an event at the site within eight hours, the current regional management expectation is

for the residents to be available to respond to significant site events and emergencies. 

Increased planning and coordination will be necessary to implement this expectation,

although this challenge has always been the case at single unit sites.

Improved Ability to Provide Support to Reactive/Baseline Inspections

ï The revised resident inspector staffing policy will impact the regionís ability to support

developmental assignments, developmental training, vacations, and inspections at other

sites of greater than one week duration by the resident inspectors. Since the margin

between available resources and program requirements is small, the staff will have to

pay greater attention to planning ahead and trying to anticipate future needs for any

extended time away from the site. Additionally, because of the small margin between

available resources and program requirements, going to ìNî resident inspectors will

require more frequent support of the resident inspector office by region-based inspectors.

ï Project engineer support is a necessity at multi-unit sites to complete the resident

baseline portion of the ROP, and this results in scheduling challenges. At single unit

sites, project engineer support is needed if there is any loss of inspector availability (e.g.,

turnover, rotation, extended training class, protracted medical leave [like

maternity/paternity]). 
7

ï The current resident inspector staffing levels are now such that small perturbations in

workload have a much larger consequence than they did under the core inspection

program. As a result, the flexibility in responding to changes in plant performance and to

providing site coverage has been greatly reduced. 

ï DRS inspectors are usually rigidly scheduled and, therefore, are less available than

resident inspectors to support reactive efforts. Alternatively, since the resources have

been given to DRS, there are specialty skills that have been retained that would have,

otherwise, been lost. Bottom line - we have more specialized personnel on staff to

support reactive efforts; however, our overall staffing levels are so tight that we are

challenged in utilizing them. 

Overall Ability to Support Implementation of the ROP

ï Significantly more management oversight and planning are necessary to ensure resident

inspector program requirements such as refresher training, back-up site visits, objectivity

visits, and counterpart meetings do not impact completion of the reactor oversight

process. In addition, extended annual or sick leave (more than one week) and military

leave will create new challenges for management to ensure the baseline program

requirements are met.

ï The flexibility in both resident and region-based inspection resources that existed in the

past is no longer present and we must make prioritization decisions on every activity that

we now assign to inspectors. Also, we now find it more difficult to support program

development initiatives since having gone to ìNî resident inspectors. 

ROP Program Guidance Issues

ï The policy should allow staffing of some sites at ìN+1" when warranted by the siteís

unique design and/or organizational features that may challenge the ability of the

program to provide an adequate indication of licensee performance.

ï The policy should be revised to allow the creation of resident positions at sites to provide

opportunities for career development and succession planning. In addition, we

recommend allowing regions the option to assign the replacement resident before the

incumbent residentís rotation date. This would help us maintain an ROP inspection

tempo and also provide some ìpipeline flexibility.î To support this revision, we would shift

some region-based inspection activities for that site to the resident staff along with

making those additional resident inspectors available for inspections at other sites.

Other Comments

ï It may be too soon to tell the full impacts of the policy change on the resident inspector

program.

ï The most recognizable impacts of the policy change have been the unintended effects on

career development for the staff and on succession planning. Career development and

succession planning efforts have been made more difficult as a result of the reduced
8

number of resident inspector positions and the longer assignments at a particular site. 

Similarly, our ability to retain highly qualified staff in the region has been adversely

affected.

ï Several inspectors have commented that another outcome of the resident inspector

staffing changes will be fewer inspection findings overall from the resident inspectors

simply because fewer resources are applied to the inspection program. 

Conclusions:

The staff made the following conclusions with respect to the impact of transitioning to ìNî

resident inspectors at multi-unit sites.

ï Baseline inspections can be accomplished by ìNî resident staffing at dual unit sites with

some assistance from region-based inspectors.

ï Most regions gained substantive experience completing the baseline inspections at dual

unit sites with ìNî residents during the first year of ROP implementation.

ï Sites that were staffed at ìN+1" tend to expend a level of effort commensurate with site

staffing, not program requirements. 

ï There is an increased inspection-related travel requirement for the region-based

inspectors because the regions had to rely on region-based inspectors and project

engineers to augment the resident inspector staff during extended absences and

vacancies at the dual unit sites. 

ï It is too soon to determine the full impact on the resident inspector program from the

change to the ìN+1" resident staffing policy. However, early indications are that the ìNî

staffing policy will challenge the regions in providing extended training, rotations, and

other professional development assignment opportunities for the resident inspector staff

at multi-unit sites.

ï The regions did not gain the immediate increase in inspection flexibility as originally

envisioned by the change to the ìN+1î resident inspector policy. As regions moved to

ìNî staffing, it was generally in response to retirements, promotions, and voluntary

reassignments. Therefore, many times a hew hire was brought onboard as a resident

inspector position was closed, resulting in an overall reduction in inspector expertise

available in the regions. 

ï Regional effort to support the ROP, evaluation and development activities was

substantial and impacted the resources available to support site-based activities as

regions moved to ìNî staffing.

ï Regional management must manage the resident inspector program more closely to

assure that upcoming RI vacancies are posted and filled in a timely manner and to

assure that appropriate site coverage is maintained during RI transitions.

Planned Actions:
9

Based on these conclusions, the staff plans to:

ï Develop criteria for the Regional Administrators, in consultation with NRR, to allocate

additional inspector resources to certain sites for other than performance related issues

(e.g., unique design or organization features). 

ï Develop appropriate ROP self-assessment metrics to monitor and trend inspector

demographics and resident inspector program quality attributes (e.g., inspector training

time; site coverage; rotational opportunities, etc. ).

ï Determine and adjust as appropriate the budget model to reflect efficiency differences in

converting a resident inspector to a region-based inspector.
STAFF TRAINING INITIATIVES

Task Leads: Mary Ann Ashley, NRR/IIPB

Fiona Tobler, NRR/IIPB

Appendix C
1

Staff Training Initiatives

The staff undertook several significant initiatives during initial implementation to evaluate staff

training needs as a result of implementing the ROP. A working group was established to review

and modify, where appropriate, the inspector training and qualification requirements. And a

second working group was established to determine how best to enhance the risk analysis

expertise of regional staff.

Inspector Training and Qualification

Purpose, Objectives, and Scope

A working group was formed in August 2000 and charged with reviewing, assessing and

modifying the inspector training and qualification requirements contained in Inspection Manual

Chapter (IMC)1245, ìInspector Qualifications for the Office of Nuclear Reactor Regulation

Inspection Program,î to support the reactor oversight process. The working group comprises

representatives from the regional offices, the technical training staff and the Office of Nuclear

Reactor Regulation (NRR). The working group is guided in its efforts by a steering committee

comprised of Headquarters and regional managers.

Results Achieved to Date

Since its inception, the working group has completed the identification of tasks performed by

inspectors under the oversight program and has surveyed existing inspectors to determine the

level of difficulty they have experienced while trying to implement those tasks. Current

inspectors were also asked to rate the effectiveness of training they received during the

inspector qualification process. The difficulty information is serving as the basis for determining

the appropriate level of training needed to prepare future inspectors to implement the oversight

program, while the effectiveness ratings are being used to identify the best methods for

presenting that training. 

The knowledge, skills and attitudes needed by inspectors to perform the various reactor

oversight tasks successfully have been identified and grouped into 11 competencies. The

competency areas are as follows: Regulatory Framework, Fundamental Plant Design and

Operation, Inspection-area Technical Expertise, Inspection, Emergency Response, Problem

Analysis, Assessment and Enforcement, Communication, Teamwork, Self-Management, and

Information Technology. The organization of knowledge, skills and attitudes into competencies

will focus the training and qualification process on inspector capabilities in the field and ensure

that each inspector (1) will understand the legal basis for and the regulatory processes used to

achieve the NRCís regulatory objectives; (2) will understand the technology and have a

demonstrated ability to apply concepts in various technical areas to allow the NRC to carry out

its overall responsibilities; (3) will have mastered the techniques and skills needed to collect,

analyze, and integrate information using a safety focus to develop supportable regulatory

conclusions; and (4) will have personal and interpersonal skills to carry out the regulatory

activities either individually or as part of a team.

The IMC 1245 working group is defining a new set of competency-based training and

qualification requirements for inspectors. The content and methods used in the existing training
2

program have been reviewed to identify which program areas already support the newly defined

competencies, where improvements are needed, and where new training must be developed. 

Definition of the requirements is expected to be completed this summer. However, the new

requirements may be implemented over time to allow for the development or revision of the

necessary training and qualification materials. The requirements of the newly designed training

and qualification program will be applied to staff hired after its implementation and will not affect

the qualification status of currently qualified inspectors. 

Longer Term Program Changes Planned

IMC 1245 will be revised to incorporate the changes to the content of the inspector training and

qualification requirements and also to incorporate improvements to the administrative aspects of

program implementation. The administrative changes are based on feedback from inspectors

and regional managers and will include improvements to the structure of on-the-job training and

self-study to better define the desired outcomes and thereby improve consistency. The revised

manual chapter will also include a requirement for the Inspection Program Branch to conduct

regular assessments of the effectiveness of the training and qualification process in preparing

inspectors to implement the reactor inspection program. The results of those assessments will

serve two purposes: defining the continuing and refresher training needs of inspectors and

providing an ongoing means for maintaining the initial inspector training and qualification

program content.

Enhanced Regional Risk Analysis Training

Purpose, Objectives, and Scope

A Working Group on Risk Expertise was formed in July 2000, with the purpose of improving risk

expertise among staff. The goals of this working group included: 1) ensuring that every reactor

inspector in the Divisions of Reactor Projects and Reactor Safety is capable of using and

understanding the risk-informed Significance Determination Processes (SDP) and, 2) ensuring

that a subset of these inspectors receive advanced risk training. 

Results Achieved to Date

One of the recommendations of the Working Group on Risk Expertise was to improve the

understanding and use of the reactor safety SDP through the use of examples. A Web-based

instructional guide was completed in June 2001. The SDP instructional guide contains case

studies of actual ROP issues processed through the various SDPs. 

.

On November 11, 2000, guidance on a solicitation of interest notice for advanced risk training for

competitively selecting region-based staff for training was sent to the Regional Administrators. 

The purpose of the solicitation of interest notice was to offer GG-14 

region-based inspectors a series of advanced risk training that would enable them to utilize NRC

Probabilistic Risk Assessment (PRA) software tools to perform and interpret basic quantitative

risk analysis. This solicitation notice culminated in a total of twelve regional selections for the

advanced risk training. The advanced risk training consists of a series of 7 classes and is a

subset of the training offered to Senior Risk Analysts (SRA). 
3

The course on îPRA Technology and Regulator Perspectiveî (P-111) has been redesigned to 

include more SDP instruction and examples, and to include a scenario in which students must

defend an SDP analysis during a mock SERP.

Program Changes Made:

The staff formalized the SRA training program via a revision to IMC 1245, issued in April 2001. 

Long Term Program Changes Planned

Staff is exploring the use of a comprehensive computer-based training course for all of the

SDPís. This effort will be coordinated with the IMC 1245 Task Group. 

Additional Work:

The Inspection Program Branch will continue to review risk analysis training requirements.