OFFICE OF

THE INSPECTOR GENERAL

U.S. NUCLEAR

REGULATORY COMMISSION

Review of NRC’s Significance Determination 

Process

OIG-02-A-15 August 21, 2002 

AUDIT REPORT

All publicly available OIG reports (including this report) are accessible through 

NRC’s website at: 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/
August 21, 2002

MEMORANDUM TO: William D. Travers

Executive Director for Operations 

FROM: Stephen D. Dingbaum/RA/

Assistant Inspector General for Audits

SUBJECT: REVIEW OF NRC’S SIGNIFICANCE DETERMINATION

PROCESS (OIG-02-A-15)

Attached is the Office of the Inspector General’s audit report titled, Review of NRC’s

Significance Determination Process (SDP). This report reflects the results of our review to

determine whether SDP is achieving its objectives, staff clearly understand the process, and

staff are using the process in accordance with agency guidance.

NRC staff, licensees, and stakeholders view SDP as an improvement over NRC’s previous

method for establishing the significance of inspection findings. While SDP is meeting its

objectives and agency staff are using SDP in accordance with guidance, additional refinements

are needed. Specifically, NRC should (1) develop an action plan to correct Phase 2 analysis

weaknesses or eliminate this portion of the SDP, because Phase 2 provides conservative

results that have been subsequently changed, is used infrequently, and adds cost and time to

the process; (2) discontinue the expenditure of about $1,050,000 remaining to develop Phase 2

until the action plan is completed, (3) provide guidance for using information from licensee risk

assessments in SDP evaluations; (4) take action to improve SDP timeliness; (5) improve its

web site to more fully inform the public; and (6) improve SDP training and guidance. These

refinements will help to ensure that SDP is implemented successfully.

This report makes 11 recommendations to help enhance SDP effectiveness.

The comments your office provided at a March 19, 2002, exit meeting and in your 

August 13, 2002, written response to the draft report have been incorporated into the report,

where appropriate. Appendix D contains the written response in its entirety.

If you have any questions, please call Russ Irish at 415-5972 or me at 415-5915.

Attachment: As stated

cc: J. Craig, OEDO
cc: R. McOsker, OCM/RAM

B. Torres, ACMUI

G. Hornberger, ACNW

G. Apostolakis, ACRS

J. Larkins, ACRS/ACNW

P. Bollwerk III, ASLBP

K. Cyr, OGC

J. Cordes, Acting OCAA

S. Reiter, CIO

J. Funches, CFO

P. Rabideau, Deputy CFO

J. Dunn Lee, OIP

D. Rathbun, OCA

W. Beecher, OPA

A. Vietti-Cook, SECY

W. Kane, DEDR/OEDO

C. Paperiello, DEDMRS/OEDO

P. Norry, DEDM/OEDO

M. Springer, ADM

R. Borchardt, OE

G. Caputo, OI

P. Bird, HR

I. Little, SBCR

M. Virgilio, NMSS

S. Collins, NRR

A. Thadani, RES

P. Lohaus, OSTP

F. Congel, IRO

M. Federline, NMSS

R. Zimmerman, RES

J. Johnson, NRR

H. Miller, RI

L. Reyes, RII

J. Dyer, RIII

E. Merschoff, RIV

OPA-RI

OPA-RII

OPA-RIII

OPA-RIV
Review of NRC’s Significance Determination Process

iii

EXECUTIVE SUMMARY

BACKGROUND

The United States Nuclear Regulatory Commission (NRC) regulates the Nation’s

103 operating commercial nuclear power plants through its reactor oversight

program. NRC staff use inspection findings and performance indicators to

assess plant operations within a framework of seven “cornerstones” of safety. A

key feature of the inspection program is the significance determination process

(SDP).

SDP is a series of analytical steps that NRC staff use to evaluate inspection

findings. The process uses four colors — Green, White, Yellow, and Red — to

indicate the significance of inspection findings. NRC provides the public with the

results of its inspections and performance assessments on its web site and

through public meetings. SDP evaluations are generally specific to each

cornerstone of safety. 

PURPOSE

The objectives of the audit were to determine whether (1) SDP is achieving

desired results, (2) NRC staff clearly understand it, and (3) NRC staff are using

SDP in accordance with agency guidance. 

RESULTS IN BRIEF 

NRC staff, licensees, and stakeholders view the significance determination

process as an improvement over NRC’s previous method for establishing the

significance of inspection findings. While SDP is meeting its objectives and

agency staff are using SDP in accordance with guidance, additional refinements

are needed. Specifically, NRC should (1) develop an action plan to correct

Phase 2 analysis weaknesses or eliminate this portion of the SDP, because

Phase 2 provides conservative results that have been subsequently changed, is

used infrequently, and adds cost and time to the process; (2) discontinue the

expenditure of about $1,050,000 remaining to develop Phase 2 until the action

plan is completed, (3) provide guidance for using information from licensee risk

assessments in SDP evaluations; (4) take action to improve SDP timeliness;

(5) improve its web site to more fully inform the public; and (6) improve SDP

training and guidance. These refinements will help to ensure that SDP is

implemented successfully.

RECOMMENDATIONS

This report makes 11 recommendations to help enhance SDP effectiveness. A

consolidated list of recommendations is on page 17.
Review of NRC’s Significance Determination Process

iv

AGENCY COMMENTS

On August 13, 2002, the Executive Director for Operations (EDO) provided a

response to our April 10, 2002, draft report. We modified the report as we

determined appropriate in response to comments provided at a March exit

meeting and in the EDO’s August 13 response. The EDO’s response can be

found at Appendix D.
Review of NRC’s Significance Determination Process

v

ABBREVIATIONS AND ACRONYMS

NRC United States Nuclear Regulatory Commission

OIG Office of the Inspector General

PRA probabilistic risk assessment

SDP significance determination process
Review of NRC’s Significance Determination Process

vi

[Page intentionally left blank.]
Review of NRC’s Significance Determination Process

TABLE OF CONTENTS

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

ABBREVIATIONS AND ACRONYMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

I. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

II. PURPOSE ........................................................3

III. FINDINGS ........................................................3

A. DEVELOP AN ACTION PLAN TO CORRECT PHASE 2 ANALYSIS WEAKNESSES

OR ELIMINATE IT ...........................................3

B. PROVIDE GUIDANCE FOR USING LICENSEE RISK ASSESSMENT INFORMATION

IN SDP EVALUATIONS ........................................7

C. TAKE ACTION TO IMPROVE SDP TIMELINESS .......................9

D. IMPROVE THE WEB SITE TO MORE FULLY INFORM THE PUBLIC . . . . . . . . . 11

E. IMPROVE SDP TRAINING AND GUIDANCE . . . . . . . . . . . . . . . . . . . . . . . . . 14

IV. CURRENT MANAGEMENT ACTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

V. CONSOLIDATED LIST OF RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . 17

VI. OIG DISCUSSION OF AGENCY COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

 APPENDIXES

A. SCOPE AND METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

B. REACTOR OVERSIGHT PROCESS PERFORMANCE AREAS AND

CORNERSTONES - CHART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

C. REACTOR OVERSIGHT PROCESS PERFORMANCE AREAS AND

CORNERSTONES - DESCRIPTION . . . . . . . . . . . . . . . . . . . . . . . . . . 23

D. AGENCY COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Review of NRC’s Significance Determination Process

viii

[Page intentionally left blank.]
Review of NRC’s Significance Determination Process

1

I. BACKGROUND

The United States Nuclear Regulatory Commission’s (NRC) mission is to ensure

adequate protection of the public health and safety, the common defense and

security, and the environment in the civilian use of nuclear materials in the

United States. As part of this responsibility, the agency regulates the Nation’s

103 operating commercial nuclear power plants.

NRC uses its reactor oversight process to monitor the safety performance of

nuclear power plants and verify that they are being operated by licensees in

accordance with agency regulations. In April 2000, after incorporating lessons

learned from a 6-month pilot program, NRC implemented a revised reactor

oversight process that is less subjective than the previous process and focused

on areas of greatest safety significance. 

The revised oversight process assesses three key performance areas: reactor

safety, radiation safety, and safeguards. This report focuses primarily on the

reactor safety performance area. Each area contains “cornerstones” of safety

that reflect the essential aspects of safe plant operation. NRC staff use

inspection findings and performance indicators to assess plant operations within

the cornerstones. Satisfactory licensee performance in each cornerstone

provides reasonable assurance of safe facility operation. Appendix B shows the

performance areas and cornerstones within NRC’s regulatory framework and

Appendix C describes the cornerstones.

A key feature of the new oversight process is the significance determination

process (SDP) used for evaluating the safety significance of inspection findings. 

SDP objectives are to (1) characterize the significance of an inspection finding

for the NRC licensee performance assessment process, using risk insights as

appropriate; (2) provide all stakeholders an objective and common framework for

communicating the potential safety significance of inspection findings; and

(3) provide a basis for assessment and/or enforcement actions associated with

an inspection finding. SDP evaluations are generally specific to each

cornerstone of safety. 

Nuclear power plant safety is based, in part, on having multiple systems or

components available to control equipment malfunctions or other adverse events

that may occur during plant operations. Licensee staff and NRC inspectors

identify degraded or deficient conditions, and NRC staff evaluate this information

using SDP to determine what actions NRC should take in response. Evaluation

of inspection findings using SDP provides one of the following results.
Review of NRC’s Significance Determination Process

2

 Result Safety significance of the finding

 Green finding Very low safety significance

 Low to moderate safety significance

 Yellow finding Substantial safety significance

 Red finding High safety significance

In general, SDP colors indicate the seriousness of the degraded or deficient

conditions. For example, in the reactor safety performance area, a Green finding

indicates that three or more systems were available to mitigate a condition, no

matter how frequently it occurs. In contrast, a Red finding indicates that one or

no systems were available to mitigate the condition, even if the condition occurs

infrequently. The following are examples of findings in the reactor safety area at

each level of risk significance:

Green: A licensee incorrectly scheduled preventive maintenance for an

emergency diesel generator. As a result, the generator was taken out of service

unnecessarily. This increased the unavailability of a safety system component. 

This condition was of very low safety significance because additional mitigating

equipment was available during the period the generator was out-of-service for

maintenance.

White: A piece of hose became lodged in the intake of an important

pump, rendering the pump inoperable for more than the allowable number of

hours while the plant was operating. This condition resulted in a mitigating

safety system also being inoperable during that time and was determined to

have low to moderate safety significance.

Yellow: A licensee did not complete a required maintenance activity in

accordance with procedures. This error likely led to the failure of an auxiliary

pump which provides cooling water to the steam generators when the main

water supply is unavailable. This issue had substantial safety significance based

on the pump’s function and the length of time the condition persisted.

Red: A licensee’s overall direction and execution of steam generator

examinations was deficient. As a result, flawed steam generator tubing was left

in service. These deficiencies resulted in a significant reduction in safety margin

while the plant was operating because of the increased probability of a tube

failure. Subsequent to the examination, one tube failed during operations.

For greater-than-Green findings (White, Yellow, or Red), NRC gives licensees

the choice of accepting NRC’s decision or presenting additional information that

might change the color of a finding. If the licensee provides further information,

NRC regional office staff, with headquarters’ participation, make a final decision

on the significance/color of the finding. If the licensee disagrees with the

decision, it may appeal to the appropriate NRC regional administrator. 
Review of NRC’s Significance Determination Process

3

The agency uses its web site and public meetings to inform the public of

inspection findings and performance assessments.

II. PURPOSE

The objectives of this audit were to determine whether (1) SDP is achieving

desired results, (2) NRC staff clearly understand the process, and (3) NRC staff

are using SDP in accordance with agency guidance. Appendix A provides

additional information about the scope and methodology.

III. FINDINGS

NRC staff, licensees, and stakeholders view the significance determination

process as an improvement over NRC’s previous method for establishing the

significance of inspection findings. While SDP is meeting its objectives and

agency staff are using SDP in accordance with guidance, additional refinements

are needed. Specifically, NRC should (1) develop an action plan to correct

Phase 2 analysis weaknesses or eliminate this portion of the SDP, because

Phase 2 provides conservative results that have been subsequently changed, is

used infrequently, and adds cost and time to the process; (2) discontinue the

expenditure of about $1,050,000 remaining to develop Phase 2 until the action

plan is completed, (3) provide guidance for using information from licensee risk

assessments in SDP evaluations; (4) take action to improve SDP timeliness;

(5) improve its web site to more fully inform the public; and (6) improve SDP

training and guidance. These refinements will help to ensure that SDP is

implemented successfully.

A. DEVELOP AN ACTION PLAN TO CORRECT PHASE 2 ANALYSIS

WEAKNESSES OR ELIMINATE IT

Phase 2 analysis is intended to enable inspection staff to characterize the risk

associated with inspection findings in the reactor safety performance area. 

However, Phase 2 analyses have not been effective because (1) the risk

information used is incomplete, (2) Phase 2 was designed to give generally

conservative results, and (3) inspectors use it infrequently. These factors have

led to Phase 2 results that have been subsequently changed and decreased

public confidence in the process. In addition, continuing to develop guidance for

Phase 2 analysis is producing unnecessary program duplication and costs. 

Although NRC will expend more than $2.2 million to develop and implement

Phase 2, the agency is also developing more complete computer-based models

that can be used to perform these analyses. Because there are significant

questions about the usefulness of Phase 2 analysis, the agency needs to

prepare an action plan to correct Phase 2 weaknesses or eliminate it. In

addition, until the action plan is complete, NRC should discontinue expenditure

of about $1,050,000 remaining to develop Phase 2.
Review of NRC’s Significance Determination Process

1 NRC required all power plants to develop Individual Plant Examinations in a Generic Letter,

Individual Plant Examination For Severe Accident Vulnerabilities - 10 CFR 50.54(f) (Generic Letter

No. 88-20), November 23, 1988. The general purpose of the examination was for each utility to

perform a systematic examination to identify any plant-specific vulnerabilities to severe accidents

[risk assessments] and report the results to the Commission.

4

Phase 2 Analysis

NRC’s inspection manual provides guidance for evaluating the risk significance

of a finding in each cornerstone of safety. First, inspectors perform pre-SDP

steps to screen out minor issues. After the initial screening, inspectors use SDP

to evaluate the significance of findings. In the reactor safety strategic

performance area, the reactor safety SDP provides for a three-phase analysis of

inspection findings. The three phases are:

Phase 1: Phase 1 analysis is a characterization of the finding and initial

screening of very low safety significance findings (Green) for

disposition by the licensee's corrective action program. Most

inspection issues do not proceed beyond Phase 1. 

Phase 2: For more significant inspection issues, Phase 2 analysis is used to

determine the potential risk associated with the finding. The

agency intended that, when NRC officials agreed on the results of

the Phase 2 analysis, the final results would be documented in an

inspection report and no further review would be needed.

Phase 3: Phase 3 analysis relies on more advanced risk assessment

techniques and is intended to confirm or modify Phase 2 results.

Phase 2 Risk Information Is Incomplete

Guidance for performing Phase 2 risk analysis is based on information from risk

assessments licensees submitted to NRC in the early 1990s1

 and on recent site

visits to update the information. Those assessments did not address the impact

of certain events on the plant (i.e., internal fires, high winds/tornadoes,

transportation accidents, external floods, and earthquakes). However, these

events, especially internal fires, can have considerable influence on risk in plant

operations. This limitation extends to Phase 2 evaluations, which do not

consider the impact of external events. As a result, to consider the potential

impact of external events, NRC must perform Phase 3 analysis.

Phase 2 Analysis Was Designed To Give Generally Conservative

Results

NRC designed Phase 2 analysis to produce generally conservative results. 

Conservative results tend to err on the side of caution; to be more likely to

identify findings as preliminarily more safety significant than they are. For
Review of NRC’s Significance Determination Process

2 “Field inspectors” is used here to mean inspectors resident at power plants and also region-based

inspectors. 

3 The remaining 15 findings initially characterized as greater-than-Green were in other SDP areas. 

5

example, Phase 2 analysis might characterize a finding as Yellow, while a more

sophisticated analysis would result in a White finding. 

One NRC official estimated that 30 to 40 percent of inspection findings initially

characterized as greater-than-Green are subsequently reduced to a lower risk

significance level. This can occur, for example, when more accurate risk

information is provided by licensees. Between April 2000 and February 2001, 3

of 10 findings using the reactor safety SDP, initially evaluated as greater-thanGreen, were reduced to Green findings. 

Phase 2 Analysis Is Used Infrequently

Phase 1 and 2 analyses are intended to be accomplished primarily by field

inspectors2

 and their first-line managers. However, these inspectors use Phase

2 infrequently. There are more than 130 resident inspectors at nuclear power

plants. However, only 11 issues were screened in Phase 2 during the first year

of the new oversight process (April 2000 - April 2001) so the great majority of

inspectors did not use Phase 2 analysis. Resident inspectors stated that their

infrequent use of Phase 2 analysis led to an inability to effectively use that part of

SDP. As a result, inspectors are unable to evaluate various issues that surface

and NRC risk experts are providing assistance for Phase 2 analyses.

Phase 2 Analysis Results Have Not Been Final

NRC expected that when staff and management agreed on Phase 2 analysis

results, no further review would be needed. However, during the first year of the

new oversight process, Phase 2 analysis did not provide a final characterization

for any inspection finding. In that period, 632 findings were initially characterized

as Green and 26 were initially characterized as greater-than-Green. Phase 2

analysis was applied to each of the 11 findings that used the reactor safety SDP

and were initially characterized as greater-than-Green.3

 Although Phase 2

analysis was intended to provide a final significance determination, all 11 were

also evaluated using Phase 3 analysis. Based on the additional analysis, 2 of

the 11 findings were eventually characterized as Green and 9 as greater-thanGreen. In addition, NRC guidance states that the significance of greater-thanGreen inspection findings will be confirmed by a Phase 3 analysis.

Decreased Public Confidence

Building and maintaining public trust is an important performance goal for NRC. 

In addition, the agency strives toward regulation that is based on the best

available knowledge from research and operational experience. Phase 2 results

that are subsequently changed have a negative impact on the public’s
Review of NRC’s Significance Determination Process

6

confidence in NRC’s position on findings. In fact, members of the public have

expressed concern about the reductions in finding colors. One stakeholder

described the reductions as NRC “negotiating” with licensees and “losing” the

discussion. Another stakeholder cited the failure of an important feedwater

pump which NRC initially determined to be a Yellow finding yet months later

reduced to a White finding. The stakeholder noted that these “changed-onappeal” determinations give the appearance of “collusion, political intimidation, or

incompetence.”

Program Duplication and Costs

NRC has been developing guidance for performing Phase 2 analysis for each

power plant since fiscal year 2000. An NRC official estimated the total cost of

that effort to be $2.2 million. About $1,050,000 remains to be expended in fiscal

years 2002-2004 to complete the project. Additionally, maintenance costs of

$250,000 per year have been budgeted for fiscal years 2004 and 2005.

In addition, since 1999, NRC has been developing computer-based models to

increase its ability to perform more comprehensive risk analyses. Results from

these models are more detailed and more complete than results from Phase 2

analysis. This effort reflects NRC’s policy to use state-of-the-art risk assessment

methods whenever practicable. An NRC official estimated these more detailed

models will cost about $3.5 million and should be completed in 2004.

Phase 2 has been in development and use for more than 2 years, and is being

used to make important regulatory decisions. However, a senior agency official

acknowledged that there are still important Phase 2 development issues that

may take 2 to 3 years to resolve. Because there are significant questions about

its usefulness, the agency needs to prepare an action plan to correct these

weaknesses or eliminate Phase 2. In addition, until the action plan is complete,

NRC should discontinue expenditure of about $1,050,000 remaining to develop

Phase 2.

RECOMMENDATIONS

The Office of the Inspector General (OIG) recommends that the Executive

Director for Operations:

1. Develop an action plan by September 6, 2002, to correct Phase 2

weaknesses or eliminate it. 

2. Discontinue expenditure of about $1,050,000 remaining to develop Phase

2 until the action plan is complete.
Review of NRC’s Significance Determination Process

7

B. PROVIDE GUIDANCE FOR USING LICENSEE RISK ASSESSMENT

INFORMATION IN SDP EVALUATIONS

Effective risk-informed regulation of nuclear power plants should be anchored in,

among other things, risk assessments that define the safety significant

structures, systems, or components of a power plant. SDP relies on risk

information provided by NRC licensees and the agency has stated that it will

require risk assessment quality commensurate with the particular use. NRC

provides guidance for using risk information in areas other than SDP. However,

NRC experts evaluating the quality of licensee risk information for SDP use must

rely, for example, on their personal knowledge of plant operations or on NRC’s

computer-based risk models, which are primarily based on risk information

provided by licensees. Although NRC’s Strategic Plan emphasizes the

importance of its regulatory independence, the agency has not developed

guidance for an independent verification process to provide assurance that

licensee risk assessment results are acceptable for SDP purposes and provide a

sound basis for regulatory decisions.

Risk Assessment

Risk assessments systematically examine complex systems to identify and

estimate the public health, environmental, and economic risks of nuclear plants. 

They attempt to quantify the probabilities and consequences of an accident’s

occurrence. By their nature, risk assessments are statements of uncertainty that

identify and assign probabilities to events.

Probabilistic risk assessment (PRA) is a risk assessment methodology used in

the nuclear power industry that systematically answers (1) what can go wrong

(accident scenarios)?, (2) how likely is it to occur (probability or frequency)?, and

(3) what will be the outcome (consequences)? SDP utilizes information from

licensee PRAs to analyze the significance of inspection findings, primarily in the

reactor safety performance area.

Within NRC, senior reactor analysts have significant responsibilities related to

the use of these licensee risk assessments. These experts are generally

responsible for evaluating the potential risk significance of plant events and

inspection findings. 

NRC’s Strategic Plan emphasizes the importance of the agency’s regulatory

independence and states that NRC must be viewed as an independent and

reliable regulator. The agency has committed to requiring PRA quality

commensurate with the particular use of the information. Maintaining the quality

of the technical basis for NRC decisions, which includes licensee PRA

information, helps NRC to ensure adequate protection of public health and

safety, and the environment. In addition, public confidence is enhanced when

the agency is consistent in carrying out its mission in a thorough, disciplined, and

timely manner. To accomplish these goals, staff require clear guidance for

assessing the quality of licensees’ PRA information.
Review of NRC’s Significance Determination Process

4 “Licensing actions” are licensee requests, such as license amendments or exemptions, that

require review and approval by NRC staff before they may be implemented by the licensee. Some

licensing actions use PRA information.

8

NRC Provides Guidance for Using PRA Information in Other Areas

NRC uses licensee PRA information in many of its regulatory activities, and

agency staff ensure this information is of sufficient scope and technical quality

for those activities. NRC determines, on an application-specific basis, whether

licensee PRA information is of sufficient quality to support its use when making

regulatory decisions. For example, NRC established a framework for using

licensees’ PRA information in licensing actions4

 and provided guidance for

determining whether that risk information is acceptable to use for those actions. 

Such guidance helps ensure consistent, thorough, and disciplined regulation.

Guidance Is Needed for Ensuring PRA Information Is Acceptable

Information from licensee risk assessments generally forms the basis of SDP

Phase 2 and Phase 3 analysis used in the reactor safety performance area. 

NRC staff use this information in evaluating potentially greater-than-Green

findings. In addition, to augment NRC’s SDP evaluation, a licensee may provide

risk information to support its position on the significance of an inspection

finding. This additional information must also be evaluated for adequacy. 

Currently, NRC senior reactor analysts evaluating licensee risk assessments

must rely, for example, on their personal knowledge of plant operations or on

NRC’s computer-based risk models, which are also based on risk information

provided by licensees. 

Senior NRC officials confirmed that the agency is highly reliant on information

from licensee risk assessments. Agency officials also noted that there are no

PRA standards, no requirements for licensees’ PRAs to be updated or accurate,

and that the quality of the assessments varies considerably among licensees. 

NRC officials stated that they depend on NRC’s senior reactor analysts to

determine the acceptability of licensee risk assessments. 

The importance of assessing the acceptability of licensee PRA data was

highlighted by senior reactor analysts who stated they have identified errors in

licensee PRAs. Those errors could impact the validity of a final determination of

the risk significance of a finding. 

However, despite its importance, NRC has not provided guidance for using

licensee PRA information in SDP evaluations. Such guidance would, for

example, specify an acceptable level of documentation that would enable staff to

conclude that the licensee has performed a sufficiently comprehensive and

acceptable PRA analysis.
Review of NRC’s Significance Determination Process

5 The SDP and Enforcement Review Panel provides a management review of potential findings and

related apparent violations.

9

Lack of Sound Basis for Regulatory Decisions

NRC senior reactor analysts confirmed they do not have guidance for reviewing

and validating licensee risk assessments. As one NRC risk expert stated,

independent assessment tools to determine the acceptability of licensees’ PRAs

“do not exist.” Without such assurance, questions exist as to whether SDP risk

evaluations are providing a sound basis for regulatory decisions.

RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

3. Develop and implement guidance for using PRA in SDP. 

4. Develop and implement guidance for providing independent assurance of

the quality of licensee risk information used to support SDP decisions.

C. TAKE ACTION TO IMPROVE SDP TIMELINESS

NRC’s performance goals include making agency activities and decisions more

effective, efficient, and realistic. The goals also call for the agency to establish

program metrics and a method for addressing identified inefficiencies. NRC

officials and stakeholders have expressed concerns that SDP evaluations are

not timely. In addition, an internal review found a “substantive” increase in the

amount of time it takes to process greater-than-Green findings under SDP when

compared to the previous program. However, current metrics do not capture the

entire process. As a result, agency managers are not able to effectively monitor

the entire process and ensure that delays are resolved and inefficiencies are

addressed. 

SDP Evaluations Are Not Timely

NRC’s inspection manual states that staff should make the final determination of

the significance of a finding within 90 days following the exit meeting at which the

licensee was officially notified of the finding. However, the 90-day goal does not

reflect the time it takes to complete the entire process. Although time expended

prior to the exit meeting is not tracked, a number of SDP activities may take

place in that period. For example, initial assessment of a finding and the SDP

and Enforcement Review Panel5

 are typically completed prior to the exit meeting

with the licensee. 
Review of NRC’s Significance Determination Process

6 October 16, 2001, First Annual Reactor Oversight Process Self-Assessment Report.

7 NRC Office of Enforcement; Audit of the Timeliness of Escalated Cases Handled Under the

Revised Reactor Oversight Program; August 2, 2001.

10

In a recent NRC assessment of the reactor oversight process, stakeholders were

critical of the time taken to finalize the safety significance of findings.6

 One

stakeholder pointed out that lower risk-significant issues are resolved more

rapidly than higher risk-significant issues but that higher risk-significant issues

should be resolved more quickly due to their greater safety significance. NRC

officials also stated that there were many instances where evaluating findings

using SDP took too long. For example, NRC regional officials expressed

concerns about delays in obtaining information needed from NRC headquarters

related to finding evaluations. One item of requested information was more than

300 days overdue and another issue being jointly evaluated was more than 440

days behind schedule.

An NRC internal review of SDP timeliness found that the time it took to process

escalated findings (greater-than-Green under SDP) had increased substantially

as a result of implementing SDP. The review also found that pre-exit

assessment activities (from the date of the “event” or from NRC identification of

the issue) increased the average process time by approximately 53 days; from

98 to 151. 

Metrics Do Not Capture the Entire Process

Time elapses between the date of the finding “event” itself and the date NRC

inspectors identify the issue and begin to evaluate it. OIG found that NRC

managers do not monitor this span of time, and, therefore, may not identify and

address related delays that may occur. NRC needs to monitor this interval to

ensure that the entire inspection process is effective in identifying performance

problems at power plants.

Management Actions

During this audit, NRC’s Office of Enforcement recommended expanding SDP

metrics to include processing time prior to exit meetings.7

 NRC assigned a

senior manager to monitor SDP timeliness and to provide weekly reports for

management review, flagging potentially untimely SDP results for increased

management attention. NRC also revised the reactor oversight process to

systematically monitor key SDP timeliness metrics. NRC regional offices submit

data quarterly to support the self-assessment process. However, these actions

still do not capture the entire process of identifying and assessing findings and

additional actions are needed to improve timeliness.
Review of NRC’s Significance Determination Process

8 http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html

11

RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

5. Establish metrics to capture the entire process of identifying and

assessing findings.

6. Establish a mechanism for agency managers to resolve identified delays.

D. IMPROVE THE WEB SITE TO MORE FULLY INFORM THE PUBLIC 

The revised reactor oversight process is consistent with NRC’s goal of providing

information to enhance the public’s confidence that the agency is carrying out its

mission. To that end, information about inspection findings is posted at NRC’s

web site.8

 However, NRC’s web site does not provide information about

inspection findings sufficient for the public to make informed decisions about

plant performance and NRC oversight. This is because important information is

missing. Specifically, additional information is needed in the inspection findings

summary and NRC should more fully document licensee corrective actions. 

Providing this information will help to increase the public’s understanding of, and

confidence in, NRC’s oversight process.

Additional Information Is Needed in the Inspection Findings Summary

NRC uses its web page to provide information to the public about the agency’s

oversight of power plants, including inspection findings and SDP results. Table 1

shows an example of the Inspection Findings Summary table for a plant. The

summary shows the seven cornerstone areas and the most significant final

finding color, over the previous four quarters, as of a given date in each

cornerstone.

Table 1: Inspection Findings Summary for a Sample Plant

Plant

Initiating

Events

Mitigating

Systems

Barrier

Integrity

Emergency

Preparedness

Occupational

Radiation

Safety

Public

Radiation

Safety

Physical

Protection

Plant 1 Green Green White Green No findings Yellow No findings

OIG reviewed a judgmental sample of inspection findings and determined that:

 Inspection reports do not show final results for greater-than-Green

findings because licensees can provide additional information related to

the finding after the inspection report is issued. Therefore, inspection
Review of NRC’s Significance Determination Process

9 After the SDP and Enforcement Review Panel agrees on the final determination of significance,

the licensee is informed of the final color of the finding in a letter.

12

reports show some finding results as “to-be-determined”. However, the

NRC letter to licensees specifying the final color for findings9

 is not linked

to the web page. For example, one White finding was shown as a

“preliminary” Yellow finding in the inspection report but there is no

information available at the web page to tell the public why the finding

appears as White in the findings summary table.

 The web site does not provide complete inspection report results. The

text summaries supporting the inspection findings summary table discuss

Green and greater-than-Green findings and provide a link to inspection

reports containing those findings. However, there is no discussion of or

links to inspection reports where no operational deficiencies were

observed. Therefore, the public does not see the complete picture of

plant inspection results.

 Only one color is displayed in the findings summary for each cornerstone. 

For example, if a licensee has both Yellow and White findings in a

cornerstone, only the most risk significant—the Yellow finding—will be

displayed. To ensure that the public can make reasoned judgments

about a plant’s performance, such additional information should be

available.

RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

7. Revise the web page to provide a link from the findings summary web

pages to documents that support any changes from preliminary

inspection report significance determinations. 

8. Expand the web page to provide complete access to inspection report

results, not just those that identify operational deficiencies.

9. Expand the web page to display all significant finding colors in a

cornerstone.

NRC Should More Fully Document Licensee Corrective Actions

NRC enforcement actions focus on ensuring that the licensee is taking corrective

actions. In accordance with that, the Commission requested that staff

emphasize the importance of licensee corrective action programs in ongoing

communication efforts. NRC guidance directs staff to discuss licensee corrective

actions in inspection reports, but information about licensee corrective actions is
Review of NRC’s Significance Determination Process

13

not generally available at the web site. As a result, the public cannot readily

obtain this important information.

Based on the importance of licensee corrective actions and requirements to

discuss corrective actions in inspection reports, OIG reviewed information at the

oversight process web site to determine whether NRC is providing the public with

information about licensee corrective actions. OIG found that the Plant

Assessment Results web page describing NRC’s policies for the assessment of

plant performance does not discuss licensee corrective actions.

Accessing and reading inspection reports, which can be highly technical and

lengthy, is a time-consuming process. Therefore, NRC summarizes information

about inspection findings in a short text description that is accessed via links in

the inspection findings summary table. Important information about findings

should be located in the text summary, including licensee corrective actions. 

Table 2 shows the results of OIG’s review of the short text descriptions of

inspection findings for three plants.

Table 2: Inspection finding descriptions that include corrective action 

Finding Color/Plant Number of

findings

Number - Text description

mentions corrective action

 Point Beach 1 12 2

 Harris 1 6 1

 Seabrook 1 9 1

 Harris 1 1 1

 Seabrook 1 1 0

These results indicate that the oversight process web site does not provide

adequate information to the public in this important area. Although the

Commission has emphasized the importance of licensee corrective actions and

the new oversight process places additional focus on corrective action, there is

no general information available at the Plant Assessment Results web page

about licensee corrective actions. In addition, there is only infrequent mention of

corrective action in the inspection findings summaries. Because of the

importance of licensee corrective actions to the effective regulation of power

plants, information about those actions should be readily available to the public. 

Lacking this information, the public cannot be fully informed of how licensees are

addressing performance deficiencies at their plants.
Review of NRC’s Significance Determination Process

14

RECOMMENDATION

OIG recommends that the Executive Director for Operations:

10. Revise the web site to fully describe licensee corrective action related to

each finding. 

E. IMPROVE SDP TRAINING AND GUIDANCE

Resident inspectors provide NRC’s major onsite presence for direct observation

and verification of licensee activities. To effectively use SDP, these inspectors

must have specific training and guidance. However, results from an agency

internal survey and interviews during this audit indicate that opportunities exist to

improve SDP training and guidance. Suggestions from NRC staff indicate the

agency has not been sufficiently proactive in identifying potential improvements

in these areas. The agency needs to revise its periodic survey of inspectors and

others working with SDP to specifically identify evolving training and guidance

needs. With these improvements, inspectors will be more effective in monitoring

and assessing conditions at power plants.

Staff Suggested Improvements

Resident inspectors are a critical component of NRC’s inspection program. 

These inspectors must have the necessary knowledge, skills, and guidance to

successfully implement the agency’s oversight process. In November 1999,

NRC conducted a survey to obtain feedback on the revised reactor oversight

program. SDP was viewed negatively primarily because the process was

perceived as not being easy to use. Limitations noted also included complexity

and extensive time to use. OIG found that these same difficulties remain, in

addition to others noted below. 

OIG evaluated SDP-related training and guidance by meeting with a judgmental

sample of resident inspectors at 10 power plants and others who work with

resident inspectors, including supervisors and licensee officials. The inspection

staff were generally working at plants that had been issued a greater-than-Green

finding and, thus, had more experience using SDP. The following are areas

where they stated NRC should improve SDP training and guidance.

Training

 Provide additional training in the application of risk to plant operations. 

For example, inspectors cited a need for more training in (1) relating

inspection findings to core damage frequency, a key SDP element;

(2) evaluating licensed operator actions; (3) using PRA information

specific to their plant; and (4) integrating SDP and enforcement. 
Review of NRC’s Significance Determination Process

10 These are the Group 1, 2, and 3 questions in NRC’s inspection manual, chapter 0610*.

15

 Provide periodic training for SDPs that residents use only occasionally

such as plant shutdown, fire protection, and containment integrity. 

Inspectors cannot use them effectively without refresher training to

compensate for their infrequent use.

Guidance

 Clarify the initial screening questions that lead into SDP.10 For example,

several inspectors stated that the term “credible impact on safety” was

too subjective and difficult to apply. The degree of subjectivity results in

inconsistent application among inspectors and places additional burden

on senior reactor analysts to provide clarification. One resident inspector

noted that staff have extensive debates on the meaning of that term

because there is wide variation of interpretation.

 Clarify guidance on how to address issues that involve more than one

SDP. In one instance, an issue involved both the fire protection and

shutdown SDPs and guidance was not clear about which SDP should be

entered first.

 Provide more guidance for reviewing licensee corrective actions and

examples of licensee corrective action program deficiencies. NRC has

reduced inspection time in this area although the oversight process

places more emphasis on licensees taking adequate corrective actions. 

In addition, changes in inspection focus mean that resident inspectors

may not be able to review all of a licensee’s condition reports. Inspectors

stated that it is prohibitive at some plants to go through each issue using

SDP guidance because of the number of corrective action items. A

mechanism is needed to help run through those more quickly. 

 Provide guidance on document retention. For example, what should be

documented and retained for responses to the initial screening

questions? Without clear guidance, important material may be discarded

after the inspection report is issued.

 Provide guidance on the disposition of issues that are not included in

inspection reports. Both NRC and licensee officials are encouraging

inspectors to continue to note minor issues. Residents routinely meet

with licensee management and discuss issues that are not included in

inspection reports. Also, these minor issues may be discussed during an

exit conference. Guidance is needed to ensure that inspectors manage

these interactions as the agency expects.
Review of NRC’s Significance Determination Process

16

 Annotate revisions to the inspection manual. Inspectors cannot currently

determine what has been changed without reading through the entire

manual and comparing it to the previous version. 

Inspectors are key to SDP implementation. OIG interviews with inspectors and

other officials indicate that improvements can be made to training and guidance. 

The agency needs to continue examining those needs by periodically surveying

inspectors and others involved with SDP. NRC most recently surveyed staff

about the revised oversight process in March 2001. However, survey questions

were not targeted to training and guidance needs. Improvements based on

more specific survey questions and results will help the agency provide better

guidance and training to enable staff to more effectively implement SDP.

RECOMMENDATION

OIG recommends that the Executive Director for Operations:

11. Revise the NRC Reactor Oversight Process Survey to capture more

specific information about SDP training and guidance.

IV. CURRENT MANAGEMENT ACTIONS

Although the new oversight process has been in place since April 2000, the

agency had not issued guidance for conducting SDP and Enforcement Review

Panels and regulatory conferences as of OIG’s draft report in March 2002. All

findings with a potential assessment greater-than-Green are reviewed by an

SDP and Enforcement Review Panel. In addition, NRC allows licensees to

request a public regulatory conference with NRC management to discuss

potential findings. NRC officials said these meetings can lack direction, fail to

reach conclusions, and can be too lengthy, thereby adding to SDP timeliness

issues. 

In October 2001, after completing field work in this area, OIG provided

comments to the agency on a draft attachment to the NRC inspection manual. 

The attachment provides guidance for conducting SDP and Enforcement Review

Panels, regulatory conferences, and caucuses. OIG suggested modifications to

the guidance to address issues related to timeliness and the effective functioning

of the meetings, among other things. In April 2002, NRC issued final guidance

which should address OIG’s concerns in this area.
Review of NRC’s Significance Determination Process

17

V. CONSOLIDATED LIST OF RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

1. Develop an action plan by September 6, 2002, to correct Phase 2

weaknesses or eliminate it. 

2. Discontinue expenditure of about $1,050,000 remaining to develop Phase

2 until the action plan is complete.

3. Develop and implement guidance for using PRA in SDP. 

4. Develop and implement guidance for providing independent assurance of

the quality of licensee risk information used to support SDP decisions.

5. Establish metrics to capture the entire process of identifying and

assessing findings.

6. Establish a mechanism for agency managers to resolve identified delays.

7. Revise the web page to provide a link from the findings summary web

pages to documents that support any changes from preliminary

inspection report significance determinations. 

8. Expand the web page to provide complete access to inspection report

results, not just those that identify operational deficiencies.

9. Expand the web page to display all significant finding colors in a

cornerstone.

10. Revise the web site to fully describe licensee corrective action related to

each finding. 

11. Revise the NRC Reactor Oversight Process Survey to capture more

specific information about SDP training and guidance.
Review of NRC’s Significance Determination Process

18

VI. OIG DISCUSSION OF AGENCY COMMENTS

On March 19, 2002, OIG discussed its initial draft report with agency officials

who generally agreed with OIG’s recommendations. Also in March, NRC began

an SDP Improvement Initiative. That initiative plans numerous, important

improvements to the process, some of which are also noted in this report. 

On August 13, 2002, the Executive Director for Operations (EDO) responded to

our April 10, 2002, final draft report. The EDO stated that he has directed staff

to develop a plan to address OIG's recommendations and also those of an

internal review panel which examined some of the same issues. The EDO

requested that staff provide a final approved plan by September 6, 2002, and

that the work be completed by November 8, 2002. Where appropriate, this

report incorporates the agency’s suggestions provided at the March meeting and

in the EDO’s August 13 response. The EDO's response can be found at

Appendix D.
Review of NRC’s Significance Determination Process

19

Appendix A

SCOPE AND METHODOLOGY

The objectives of this audit were to determine whether (1) SDP is achieving

desired results, (2) NRC staff clearly understand the process, and (3) NRC staff

are using SDP in accordance with agency guidance. To address the audit

objectives, OIG reviewed relevant program documentation and conducted

interviews with more than 80 individuals, including:

 NRC headquarters program officials, 

 senior officials in the reactor safety and materials safety areas at all four

NRC regional offices;

 senior reactor analysts at all four NRC regional offices;

 Office of Enforcement officials at NRC regional offices;

 NRC resident inspectors at 11 power plants; 

 a public interest group active in this area; 

 the Nuclear Energy Institute; and 

 licensee officials at 10 power plants, including experts in the use of

probabilistic risk assessment and senior operating officials. 

This audit was conducted from May through October 2001 in accordance with

generally accepted Government auditing standards and included review of

management controls related to the objectives of the audit. The major

contributors to this report were William McDowell, Team Leader; Robert Moody,

Audit Manager; and David Horn, Senior Auditor.
Review of NRC’s Significance Determination Process

20

[Page intentionally left blank.]
Review of NRC’s Significance Determination Process

21

Protect Public Health and Safety 

in the Operation of Commercial 

Nuclear Power Plants

Reactor Safety Radiation Safety

Initiating 

Events

Mitigating 

Systems

Barrier 

Integrity

Emergency 

Preparedness

Occupational 

Radiation 

Safety

Public 

Radiation 

Safety

Safeguards

Mission:

Performance Areas:

Cornerstones:

Physical 

Protection

Appendix B

REACTOR OVERSIGHT PROCESS PERFORMANCE AREAS AND

CORNERSTONES - CHART
Review of NRC’s Significance Determination Process

22

[Page intentionally left blank.]
Review of NRC’s Significance Determination Process

23

Appendix C

REACTOR OVERSIGHT PROCESS PERFORMANCE AREAS AND

CORNERSTONES - DESCRIPTION

Reactor Safety

Initiating Events - This cornerstone focuses on operations and events at a

nuclear plant that could lead to a possible accident if plant safety systems did not

intervene. These events could include equipment failures leading to a plant

shutdown, shutdowns with unexpected complications, or large changes in the

plant’s power output. 

Mitigating Systems - This cornerstone measures the function of safety systems

designed to prevent an accident or reduce the consequences of a possible

accident. The equipment is checked by periodic testing and through actual

performance. 

Barrier Integrity - There are three important barriers between the highly

radioactive materials in fuel within the reactor and the public and the

environment outside the plant. These barriers are the sealed rods containing the

fuel pellets, the heavy steel reactor vessel and associated piping, and the

reinforced concrete containment building surrounding the reactor. The integrity

of the fuel rods, the vessel, and the piping is continuously checked for leakage,

while the ability of the containment to prevent leakage is measured on a regular

basis. 

Emergency Preparedness - Each nuclear plant is required to have

comprehensive emergency plans to respond to a possible accident. This

cornerstone measures the effectiveness of the plant staff in carrying out its

emergency plans. Such emergency plans are tested every 2 years during

emergency exercises involving the plant staff and local, State, and, in some

cases, Federal agencies. 

Radiation Safety

Occupational Radiation Safety - NRC regulations set a limit on radiation doses

received by plant workers, and this cornerstone monitors the effectiveness of the

plant’s program to control and minimize those doses. 

Public Radiation Safety - This cornerstone measures the procedures and

systems designed to minimize radioactive releases from a nuclear plant during

normal operations and to keep those releases within Federal limits. 
Review of NRC’s Significance Determination Process

24

Appendix C

Safeguards

Physical Protection - Nuclear plants are required to have well-trained security

personnel and a variety of protective systems to guard vital plant equipment, as

well as programs to assure that employees are constantly fit for duty through

drug and alcohol testing. This cornerstone measures the effectiveness of the

security and fitness-for-duty programs. 

In addition to the seven cornerstones, the reactor oversight program features

three “cross-cutting” elements, so named because they affect, and are therefore

part of, each of the cornerstones: 

 human performance;

 management attention to safety and workers' ability to raise safety issues

(the "safety-conscious work environment"); and

 finding and fixing problems (the utility's corrective action program).
Review of NRC’s Significance Determination Process

25

Appendix D

AGENCY COMMENTS 
Review of NRC’s Significance Determination Process

26

Appendix D
Review of NRC’s Significance Determination Process

27

Appendix D