SIGNIFICANCE

DETERMINATION

PROCESS

TASK GROUP REPORT

December 13, 2002
ii

SIGNIFICANCE DETERMINATION PROCESS TASK GROUP REPORT

TABLE OF CONTENTS

EXECUTIVE SUMMARY

Objective and Scope ...................................................v

Background ..........................................................v

Observations and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

1.0 INTRODUCTION ......................................................1

1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Scope and Method ...............................................1

2.0 ISSUE SUMMARY AND BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Issue Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 Background ....................................................4

3.0 EVALUATION RESULTS AND RECOMMENDATIONS ........................6

3.1 Evaluation of Reactor Oversight Process and Significance Determination

Process Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1.1 Phase 2 SDP Contribution to ROP and SDP Objectives . . . . . 10

3.1.1.2 Insights Regarding the Viability of the Phase 2 Process . . . . . 12

3.1.1.3 Alternatives to the Current Phase 2 Process . . . . . . . . . . . . . 13

3.1.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.1.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2 Performance Expectation for the Significance Determination Process . . . . . . 24

3.2.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.2.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.2.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.3 Expectations for Use of Significance Determination Process Phase 2 . . . . . . 27

3.3.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.4 Applicability of the Current Significance Determination Process to the 

Range of Inspection Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.4.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.4.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.4.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.5 Significance Determination Process Training and Guidance . . . . . . . . . . . . . . 32

3.5.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.5.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.5.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
iii

3.6 Significance Determination Process Benchmarking and the Need 

for Peer Review Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.6.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.6.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.6.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.7 Use of Uncertainty in the Significance Determination Process . . . . . . . . . . . . . 37

3.7.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.7.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.7.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.8 Evaluation of How Other Inputs are Factored Into the Significance 

Determination Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.8.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.8.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.8.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.9 Significance Determination Process Timeliness . . . . . . . . . . . . . . . . . . . . . . . . 43

3.9.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.9.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.9.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.10 Reactor Oversight Process Web Site Improvements . . . . . . . . . . . . . . . . . . . . 47

3.10.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.10.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.10.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.11 Evaluation of Other Reactor Oversight Process and Significance 

Determination Process Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.11.1 Accident Sequence Precursor Program and Significance 

Determination Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.11.1.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.11.1.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.11.1.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.11.2 Objectivity of Probabilistic Risk Assessments . . . . . . . . . . . . . . . . . . . . 51

3.11.2.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.11.2.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.11.2.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.11.3 Insights on Performance of ROP Action Matrix . . . . . . . . . . . . . . . . . . 52

3.11.3.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.11.3.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.11.3.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

FIGURES

Figure 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
iv

APPENDICES: 

APPENDIX A - Consolidated List of Recommendations . . . . . . . . . . . . . . . . . . . . . . . 54

APPENDIX B - List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

APPENDIX C - List of Documents Reviewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

APPENDIX D - List of Persons Contacted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
v

EXECUTIVE SUMMARY

Objective and Scope

The Executive Director for Operations (EDO) directed the formation of an NRC task group to

perform an independent and objective review of the Significance Determination Process (SDP). 

This review was prompted, in part, by issues described in a Differing Professional Opinion

(DPO) Panel Response dated June 28, 2002, and an Office of Inspector General (OIG) Audit

Report dated August 21, 2002. The Charter for the Significance Determination Process Task

Group was established in a memorandum dated September 18, 2002, from Samuel J. Collins,

Director, Office of Nuclear Reactor Regulation (NRR) to Victor M. McCree, the Task Group

Chairperson. The overall objective of the Task Group was to review the issues raised in both

the DPO Panel Response and the OIG Audit Report and provide observations, conclusions,

and recommendations to address the underlying concerns, including whether the current

reactor safety Phase 2 SDP approach should be continued, modified, or replaced. 

The scope of the review, as indicated in the Charter, consisted of the key SDP issues outlined

in the EDO’s tasking memorandum dated August 6, 2002, including: (1) achievement of SDP

and Reactor Oversight Process (ROP) objectives; (2) performance expectation for the SDP;

(3) applicability of the SDP to inspection findings; (4) consideration of uncertainty in the SDP;

(5) consideration of “other inputs” in the ROP/SDP; (6) expectations for inspector use of the

reactor safety SDP; (7) the need to continue, modify, or replace the current Phase 2 SDP tool;

(8) implementation of the SDP by appropriate agency personnel; (9) improvements in SDP

training and guidance; and (10) other ROP process changes. The Task Group’s review also

included: as described in the DPO Panel Report, (11) development and peer review of criteria

for SDP benchmarking; and, as indicated in the OIG Audit Report, (12) SDP timeliness and

(13) ROP web site improvements.

Consistent with the Charter, the Task Group’s review focused on the SDP for the Reactor

Safety Strategic Performance Area and, in particular, issues pertaining to the SDP for the

Initiating Events (IE), Mitigating Systems (MS) and Barrier Integrity (BI) Cornerstones. As a

result, the Task Group did not perform a detailed review of the SDP for the Radiation Safety

Performance Area or Safeguards Performance Area. In addition, because the Emergency

Preparedness (EP) Cornerstone SDP was not the focus of the DPO Panel Response or OIG

Audit Report, and because the relevant EP SDP issues are the focus of other NRC review

activities, the Task Group did not emphasize this area in its review.

The Task Group met with a broad spectrum of persons involved with the SDP to obtain their

views and recommendations regarding the SDP. Task Group members met with NRC staff and

managers in Headquarters and in each Regional office, and interviewed resident inspectors,

licensee managers, and licensee risk analysts at two sites in each Region. Altogether, the

Task Group obtained input from 160 stakeholders, including 118 NRC staff, 36 licensee

representatives and 6 external stakeholders.

Background

In SECY-99-007, “Recommendations for Reactor Oversight Process Improvements,” dated

January 8, 1999, the staff provided its recommendations to the Commission for improving the
vi

reactor regulatory oversight processes, including proposed changes to the NRC’s inspection,

assessment, and enforcement processes. The staff’s efforts to develop the proposed changes

was guided by three objectives: (1) improve the objectivity of the [reactor] oversight process so

that subjective decisions were not central process features; (2) improve the scrutability of these

processes so that NRC actions have a clear tie to licensee performance; and (3) risk-inform the

process so that NRC and licensee resources are focused on those aspects of performance

having the greatest impact on safe plant operations. 

With respect to the assessment process, the staff sought to develop a process that would allow

the integration of various information sources relevant to licensee safety performance. In

SECY-99-007, the staff concluded that adequate assurance of licensee performance would be

achieved through the use of risk-informed performance indicators (PIs) and inspection findings.

The staff also highlighted the need to develop a method for characterizing the risk of inspection

findings and indicated that a “level of risk significance, based on a risk scale, will be determined

and documented for the findings.”

In SECY-99-007A, “Recommendations For Reactor Oversight Process Improvements”

(follow-up to SECY-99-007), Attachment 2, dated March 22, 1999, the staff introduced the

Significance Determination Process (SDP) as the method for characterizing the risk of

inspection findings. The SDP was designed to assess only those inspection findings

associated with at-power operations in the Reactor Safety Strategic Performance Area

cornerstones of IE, MS and BI; however, concepts for characterizing the risk significance of

inspection findings in the emergency preparedness, radiation safety, and safeguards areas

were under development. The SDP provided a means to screen out inspection findings that

have minimal or no risk significance and trigger a more detailed analysis of potentially

risk-significant findings.

To support the start of the initial implementation the revised Reactor Oversight Process (ROP)

in April 2000, the staff issued Inspection Manual Chapter (IMC) 0609, “Significance

Determination Process.” Appendix A to IMC 0609 provided guidance for the staff to estimate

the unintended increase in risk during at-power plant conditions caused by deficient licensee

performance. The guidance was intended to provide a simplified probabilistic framework for

use by the staff in identifying potentially risk significant findings in the reactor safety area--either

the IE, MS, or BI cornerstones. 

When the ROP was initially implemented in April 2000, the staff‘s efforts to develop the Phase 2

notebooks for each nuclear plant were still in progress. As a result, the draft notebooks that

were made available for staff use at initial ROP implementation were considered to be

incomplete. By late 2000, the staff had made sufficient progress in the site visits associated

with the development of Phase 2 SDP notebooks, that it began to issue the “Revision 0”

notebooks to the sites. After issuance of the first Rev 0 notebooks, the staff identified problems

with the accuracy of the notebooks and concluded that benchmarking was needed to confirm

the adequacy of the notebooks. Using NRC risk analysts and contractor resources, the staff

began its efforts to benchmark the notebooks in April 2001. As of November 12, 2002, the staff

had issued 24 Revision 1, Phase 2 notebooks.

In a memorandum dated November 8, 2001, Troy Pruett, Senior Reactor Analyst, Region IV,

submitted a differing professional view (DPV) to the Director of the Division of Reactor Safety in
vii

Region IV. The DPV expressed concerns about the performance of the SDP Phase 2

analyses. An Ad Hoc Panel, appointed by the Regional Administrator by memorandum dated

November 16, 2001, was formed to review the DPV and make appropriate recommendations. 

The DPV Panel documented its findings in a report to the Region IV Administrator dated

January 10, 2002. This report was forwarded to the Director, NRR, for program office

consideration and appropriate action. In a memorandum dated February 18, 2002, the Director,

NRR informed Mr. Pruett of the results of the review of his DPV. Mr. Pruett expressed several

concerns with the results of the DPV review and, in a memorandum to the EDO dated March

15, 2002, recommended an independent review of the concerns in his DPV. Through a

memorandum dated April 9, 2002, the EDO convened an Ad Hoc panel to review Mr. Pruett’s

DPO.

The DPO Panel completed its review and issued conclusions and recommendations in a report

dated June 28, 2002. The DPO Panel generally agreed with the overall analysis performed by

the DPV panel and its response to Mr. Pruett’s recommendations. The DPO Panel found that

“NRC management and staff are in the process of addressing many of the Ad Hoc DPV Panel’s

observations and recommendations in the SDP Improvement Initiative.” However, the DPO

Panel also recommended that the NRC conduct an independent review of the SDP assessment

tools.

Between May and October 2001, the OIG conducted an audit of the SDP. The objectives of the

audit, as indicated in the OIG’s report (OIG-02-A-15) dated August 21, 2002, were to determine

whether (1) the SDP is achieving desired results, (2) NRC staff clearly understand the process,

and (3) NRC staff are using [the] SDP in accordance with agency guidance. In its report, OIG

concluded that “while the SDP is meeting is objectives and agency staff are using SDP in

accordance with guidance, additional refinements are needed.” The report provided a number

of recommendations, including that the NRC develop an action plan to correct Phase 2 analysis

weaknesses or eliminate this portion of the SDP. 

In a memorandum to the Director, NRR dated August 6, 2002, the EDO directed that a plan be

developed to address both the DPO Ad Hoc Panel and OIG recommendations. The EDO’s

memorandum indicated that this “plan shall address the DPO Panel recommendation for an

overall objective review of the SDP.” The plan developed by the Director, NRR included the

formation of the SDP Task Group to conduct an independent review of the SDP. 

Observations and Conclusions

The SDP, including the Phase 2 process, has generally succeeded in meeting the ROP

objectives of providing a more objective, scrutable, and risk-informed process. In addition, the

four SDP objectives have generally been met by the current SDP process. Although the

contribution of the Phase 2 process to the SDP objectives was limited because of its relatively

infrequent use, feedback from stakeholders indicated that the Phase 2 process has contributed

to the staff’s efforts to characterize the significance of inspection findings, facilitate stakeholder

communication, provide a basis for assessment and enforcement actions, and risk-inform the

inspection program. 

The Task Group concluded that the SDP should retain the Phase 2 process to facilitate

(resident and region-based) inspector involvement in the screening of Green findings and the
viii

characterization of potentially greater-than-green inspection findings. However, the SDP should

be enhanced to address the current problems with the Phase 2 notebooks, including the

technical inadequacies and the cumbersome, complex and time-consuming characteristics of

the worksheets. In this regard, the Task Group evaluated six options to enhance the SDP and

recommended the development of enhanced pre-solved Phase 2 SDP tables (Option 3) to

replace the Phase 2 SDP notebooks as the primary tool for inspectors to perform Phase 2

evaluations. However, the notebooks should be retained for optional use by the NRC staff to

gain additional risk insights.

The performance expectations for Phase 1 of the SDP were clearly documented in IMC 0609,

well understood by the staff. The Phase 1 process has been used to appropriately screen out

inspection findings of very low safety significance. However, the original performance

expectations regarding the use and accuracy of the Phase 2 process have not been fully

realized due, in large part, to the untimely development and issuance of the Phase 2 notebooks

and the errors associated with the Revision 0 notebooks. The current SDP performance

expectations, as well as the expectations for inspectors to use the SDP, are described in an

August 9, 2002, memorandum from the Director, NRR. The Task Group concluded that the

expectations were understandable and capable of being implemented by the Regions; however,

based on the interviews in the Regions, some staff were unfamiliar with the guidance and

others were unsure of whether and how it had been implemented.

The Task Group determined that resident and region-based inspectors rarely used the Phase 2

notebooks to characterize an inspection finding because the majority of the findings screened

(out) as minor or Green prior to meeting a condition that required use of the Phase 2. Also,

most of the inspector interviewees expressed difficulty in using the Phase 2 notebooks because

the guidance was complex and the worksheets were time consuming and not user-friendly. 

Notwithstanding these concerns, most of the NRC interviewees supported the continued use of

the Phase 2 process, provided that the current problems are addressed, and felt that it was

essential for inspectors to participate in determining the significance of inspection findings

beyond the initial screening, Phase 1.

IMC 0609 and IMC 0612, “Power Reactor Inspection Reports,” provide some guidance on the

types of inspection findings that can be evaluated using the SDP. The reactor safety Phase 2

SDP notebooks do not currently allow for treatment of inspection findings that are related to

external events, shutdown modes and Large Early Release Frequency (LERF). However, the

staff has initiated action to refine the SDP guidance documents for evaluation of findings related

to shutdown modes, containment, and fire protection. The degraded condition of a Structure,

System or Component (SSC) can also be treated using the SDP if agreement can be reached

on characterizing the impact of the finding on equipment unavailability or initiating event

frequency. The Task Group also concluded that external events can have a significant effect

on risk analyses and, therefore, should be included in the SDP. In this regard, the Task Group

concurred in the ongoing NRR initiatives to develop better tools and guidance for external

events so that the risk insights can be applied in a consistent manner.

A variety of NRC training courses and seminars were administered over a relatively short time

frame and a number of training courses were established to enhance the knowledge of NRC

staff and managers in the area of risk. Although the general consensus from the Task Group’s

interviews of internal and external stakeholders was that the staff's knowledge and ability to
ix

communicate risk insights has improved due, in part, to staff training, many commented on

areas for improvement. In particular, many inspectors felt that the training could have been

better and that it did not make them proficient in use of the Phase 2 tool. Although training was

a factor in inspector proficiency, the Task Group concluded that the complexity of the Phase 2

notebooks, the unavailability of benchmarked notebooks, and the infrequent use of the Phase 2

process were the main hindrances to inspectors’ proficiency in use of the Phase 2 SDP. The

Task Group concluded that simplifying the Phase 2 SDP by using enhanced pre-solved tables

will alleviate the need for inspectors’ current reliance on the SRAs to conduct the Phase 2

evaluation. The Task Group also recommended that a systematic assessment of agency

training in the area of risk be conducted. 

Although the current Phase 2 SDP guidance is complex, it was deemed adequate to implement

the current Phase 2 worksheets. In addition, the Task Group concluded that the enhanced

pre-solved Phase 2 SDP tables will prompt the development of more user-friendly

implementation guidance. The actions and schedule in the SDP Task Action Plan to enhance

the current Phase 3 guidance are appropriate. Although no examples were cited where the

limited Phase 3 SDP guidance has resulted in an unsatisfactory outcome (i.e., inadequate or

poor quality Phase 3 analysis), the Task Group concurred in the ongoing NRR initiative to

enhance the Phase 3 guidance.

The Task Group concluded that the ongoing benchmarking contributes significantly to

improving the quality of the Phase 2 SDP notebooks and Standardized Plant Analysis Risk

(SPAR) models. Many of the recommendations contained in the DPO Ad Hoc Panel report

regarding the benchmarking process, including staff qualification and peer review

recommendations, have already been included in the current process. However, the Task

Group noted several areas for further improvement, including documentation of the

“construction rules” for developing the Phase 2 SDP notebooks and comparison between the

results of the benchmarked SPAR models and the benchmarked notebooks. In addition,

because future licensee probabilistic risk assessment (PRA) changes and/or plant modifications

can affect the adequacy of the notebooks, the Task Group concluded that a plan to update the

Phase 2 tool was warranted. 

The Task Group reviewed the use of uncertainty in SDP evaluations in light of the three types

of uncertainty highlighted in Regulatory Guide (RG) 1.174, “An Approach for Using Probabilistic

Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing

Basis:” parameter uncertainty; model uncertainty; and completeness uncertainty. The Task

Group concluded that parameter uncertainty was a relatively minor contributor for the purpose

of the SDP, and that model uncertainties, particularly those associated with the characterization

of the impact of the inspection finding on the function of an SSC, represented the greatest

source of uncertainty. The Task Group also noted that the staff should consider the

contribution from external events, and low power and shutdown events when making a final

assessment of risk significance of a finding (for completeness). The Task Group concurred in

the ongoing initiatives to develop better tools and guidance in the area of low power and

shutdown. The Task Group also noted that the SDP does not provide for explicit consideration

of traditional engineering analysis (deterministic) inputs in the risk significance estimation

process. This is in contrast to RG 1.174 which uses considerations of defense-in-depth and

safety margins to assess whether license amendments should be granted. In that context,

defense-in-depth and safety margins are seen as mechanisms to account for incompleteness.
x

The Task Group concluded that the SDP should not be modified to include consideration of

“other inputs.” There were no examples where the current provisions for treating other inputs in

the ROP had limited the staff’s ability to focus resources in appropriate areas. In addition, the

existing guidance in IMC 0305, “Operating Reactor Assessment Program,” regarding deviations

from the Action Matrix can be employed, where appropriate, to consider “other inputs” in

determining agency response. However, the Task Group also concluded that the current

guidance in IMC 0305 does not promote consistency in the identification of substantive

cross-cutting issues by the Regions and the guidance is vague on the expected NRC and

licensee response to a substantive cross-cutting issue.

The Task Group concluded that the ROP timeliness goal of completing the SDP within 120

days of the first inspection exit meeting and within 90 days of the issuance of the inspection

report is reasonable. However, the goals were not well understood by all members of the staff.

The Task Group recognized that some findings involve complex engineering issues and may

require additional time to adequately assess the safety significance. The Task Group

determined that significant progress has been made in tracking the timeliness of findings that

are greater than Green and establishing accountability within the responsible offices. However,

the Task Group concluded that improved management decisiveness in determining the

engineering assumptions used in risk evaluations would also contribute to the timeliness of the

SDP. The Task Group concluded that a recent revision to the Office of Public Affairs (OPA)

policy regarding issuance of a press release for a White findings was an improvement and

noted that further revision to link press releases to a finding(s) that results in a Degraded

Cornerstone was warranted.

The OIG Audit Report, (OIG-02-A-15) provided several recommendations to enhance the ROP

web site. The Task Group concluded that the recommendation to improve the web site links for

all relevant documents for greater-than-green findings was appropriate and should be adopted

by NRR. The recommendation to link the summary of findings to the inspection report had

already been implemented. The recommendation to revise IMC 0612 to include a brief

summary of corrective actions in the inspection report summary of findings was reasonable. 

The Task Group concluded that the recommendation to display all significant findings colors in

a cornerstone should not be adopted. Web site users are provided ready access to all findings

in a cornerstone by hyper-linking to the next web page, and the difficulty and costs of

implementing this change would likely exceed the benefit. 

The Task Group determined that the issuance of different NRC risk characterizations from the

Accident Sequence Precursor (ASP) Program and SDP can have a potentially negative impact

on public confidence. Consequently, program requirements and/or an Office of Nuclear

Regulatory Research (RES)/NRR office protocol should ensure reasonable consistency in the

output of the programs to promote public confidence. The Task Group concluded that

efficiencies could be gained through a better coordination and/or integration of these two

programs. 

The Task Group evaluated concerns about the perceived overemphasis on the objectivity of 

PRA results and the quality of the PRA models used to determine the final risk significance of

inspection findings. The Task Group concluded that the staff routinely uses engineering

judgement in the SDP and that this practice is consistent with the NRC's PRA Policy Statement. 

With respect to the quality and scope of licensee PRAs, the Task Group concluded that
xi

guidance should be developed to allow the staff to determine whether the results of a licensee’s

risk analysis of a finding is of sufficient quality to use as input to the staff’s final significance

determination.

The Task Group concluded that overall, the Action Matrix has fulfilled its purpose in providing

an objective, scrutable and predictable framework for NRC actions in response to licensee

performance problems. The oversight process also provides sufficient flexibility in use of the

Action Matrix for NRC managers to use discretion in decisions concerning the scope and timing

of the agency’s response to licensee performance problems. In addition, the Task Group

concluded that the staff used reasoned judgement in its decision to use two White inputs in the

same cornerstone as part of the criteria for defining a “Degraded Cornerstone.” Although a

detailed analysis or evaluation was not developed to support this decision, the Task Group did

not identify data or obtain information from its interviews to suggest that the criteria were

inappropriate. 

Recommendations

The Task Group determined that the NRC should take specific actions to address the issues

described in Section 3 of this report. A consolidated list of recommendations is provided in

Appendix A.
SIGNIFICANCE DETERMINATION PROCESS

TASK GROUP

DECEMBER 13, 2002

Prepared by SDP Task Group:

V. McCree, SDP Task Group Chairperson /RA/ 

L. Dudes, NRR /RA/ 

H. Hamzehee, RES /RA/ 

I. Jung, NRR /RA/ 

G. Parry, NRR /RA/ 

D. Roberts, NRR /RA/ 

J. Trapp, RI /RA/ 

Task Group Administrative Support provided by:

C. Raynor
1.0 INTRODUCTION

1.1 Objective

The Executive Director for Operations (EDO) directed the formation of an NRC task group to

perform an independent and objective review of the Significance Determination Process (SDP). 

This review was prompted, in part, by issues described in a Differing Professional Opinion

(DPO) Panel Response dated June 28, 2002, and an Office of Inspector General (OIG) Audit

Report dated August 21, 2002. The Charter for the Significance Determination Process Task

Group was established in a memorandum dated September 18, 2002, from Samuel J. Collins,

Director, Office of Nuclear Reactor Regulation (NRR) to Victor M. McCree, the Task Group

Chairperson. The objective, scope and management interface of the Task Group are defined in

the memorandum and its attachment. The overall objective of the Task Group was to review

the issues raised in both the DPO Panel Response and the OIG Audit Report and provide

observations, conclusions, and recommendations to address the underlying concerns, including

whether the current reactor safety Phase 2 SDP approach should be continued, modified, or

replaced. 

1.2 Scope and Method

The scope of the review, as indicated in the Charter, consisted of the key SDP issues outlined

in the EDO’s tasking memorandum dated August 6, 2002, including: (1) achievement of SDP

and Reactor Oversight Process (ROP) objectives; (2) performance expectation for the SDP;

(3) applicability of the SDP to inspection findings; (4) consideration of uncertainty in the SDP;

(5) consideration of “other inputs” in the ROP/SDP; (6) expectations for inspector use of the

reactor safety SDP; (7) the need to continue, modify, or replace the current Phase 2 SDP tool;

(8) implementation of the SDP by appropriate agency personnel; (9) improvements in SDP

training and guidance; and (10) other ROP process changes. The Task Group’s review also

included: as described in the DPO Panel Report, (11) development and peer review of criteria

for SDP benchmarking; and, as indicated in the OIG Audit Report, (12) SDP timeliness and

(13) ROP web site improvements.

Consistent with the Charter, the Task Group’s review focused on the SDP for the Reactor

Safety Strategic Performance Area and, in particular, issues pertaining to the SDP for the

Initiating Events (IE), Mitigating Systems (MS) and Barrier Integrity (BI) Cornerstones. As a

result, the Task Group did not perform a detailed review of the SDP for the Radiation Safety

Performance Area or Safeguards Performance Area. In addition, because the Emergency

Preparedness (EP) Cornerstone SDP was not the focus of the DPO Panel Response or OIG

Audit Report, and because the relevant EP SDP issues are the focus of other NRC review

activities, the Task Group did not emphasize this area in its review.

A Steering Committee, consisting of four senior managers, was formed to guide the Task

Group and to provide counsel in areas of concern. The Task Group kept the Steering

Committee informed of its activities and periodically briefed the members during its review. In

addition, on October, 16, 2002, the Task Group met with the OIG to discuss the Task Group

Charter. On October 25, 2002, the Task Group held a public meeting at NRC Headquarters to

solicit input from public stakeholders on the SDP.

The Task Group met in September 2002, and developed a method for data gathering and

information management that included techniques used in NRC Incident Investigation Team
-2-

reviews and other methods used in the past to successfully evaluate agency programs. The

Task Group’s review was accomplished in three phases: a Preparatory Phase, an Evaluation

Phase and a Report Phase. The six members of the Task Group were split into three,

two-member groups to promote synergy and to distribute the workload.

During the Preparatory Phase, the Task Group conducted a range of activities to prepare for

the Evaluation and Report Phases, including an extensive review of background documents 

associated with the SDP (see Appendix C). Several orientation briefings were conducted to

familiarize the Task Group members with various aspects of the review. The Task Group

developed an extensive list of interview questions during this phase to facilitate the interviews

and provide a basis for comparing the opinions and recommendations of the various

stakeholders. This phase of the review was also used to identify and arrange interviews with

NRC staff, licensee representatives and other external stakeholders.

During the Evaluation Phase, the Task Group met with a broad spectrum of persons involved

with the SDP to obtain their views and recommendations regarding the SDP. Task Group

members met with NRC staff and managers in Headquarters and in each Regional office, and

interviewed resident inspectors, licensee managers, and licensee risk analysts at two sites in

each Region. Altogether, the Task Group obtained input from 160 stakeholders, including 118

NRC staff, 36 licensee representatives and 6 external stakeholders.

The 118 NRC interviewees consisted of: the Regional Administrators and most of the

reactor-program senior managers in each Region (18); at least four reactor-program Branch

Chiefs in each Region (19); Regional inspectors, project engineers and resident inspectors

(34); most of the Regional Senior Reactor Analysts (SRAs) (6); most of the NRR senior

managers responsible for the development and/or implementation of the ROP/SDP (13); NRR

Section Chiefs responsible for the development and/or implementation of the ROP/SDP

(3); some NRR Senior Risk Analysts and other NRR staff, including seven recently reassigned

resident inspectors (14) ; Senior Managers, Section Chiefs and/or Risk Analysts from the Office

of Nuclear Regulatory Research (RES) (7); Office of Enforcement (OE) (3); Office of General

Counsel (OGC) (1) and Office of Nuclear Security Incident and Response (NSIR) (1). 

The 35 licensee interviewees included Plant Managers and Regulatory Affairs Managers (22)

and Licensee PRA Managers and Risk Analysts (13). External Stakeholders who participated

in the October 25, 2002, SDP Task Group public meeting included a representative from the

State of Illinois, the State of Pennsylvania, the Union Concerned Scientists, the New England

Coalition on Nuclear Pollution, and the Nuclear Energy Institute. A representative from the

State of New Jersey also submitted a letter to the Task Group providing views on the SDP. 

The Task Group considered all the comments received during its review. 

2.0 ISSUE SUMMARY AND BACKGROUND

2.1 Issue Summary

In a memorandum dated November 8, 2001, Troy Pruett, SRA, Region IV, submitted a differing

professional view (DPV) to the Director of the Division of Reactor Safety in Region IV. The

DPV expressed concerns about the performance of the SDP Phase 2 analyses. An Ad Hoc
-3-

panel, appointed by the Regional Administrator by memorandum dated November 16, 2001,

was formed to review the DPV and make appropriate recommendations. The DPV Panel

documented its findings in a report to the Region IV Administrator dated January 10, 2002. 

This report was forwarded to the Director, NRR, for program office consideration and

appropriate action. In a memorandum dated February 18, 2002, the Director of NRR informed

Mr. Pruett of the results of the review of his DPV. Mr. Pruett expressed several concerns with

the results of the DPV review and, in a memorandum to the EDO dated March 15, 2002,

recommended an independent review of the concerns in his DPV. Through a memorandum

dated April 9, 2002, the EDO convened an Ad Hoc panel to review Mr. Pruett’s DPO.

The DPO Panel completed its review and issued its conclusions and recommendations in a

report dated June 28, 2002. The DPO panel generally agreed with the overall analysis

performed by the DPV panel and its response to Mr. Pruett’s recommendations. Included in the

DPO Panel’s report were the following conclusions: 

(1) The SDP, in its entirety, appropriately addresses the safety significance of inspection

findings.

(2) The extensive dependency on the SRAs has contributed to inefficiencies and

ineffectiveness of the Phase 2 reviews. 

(3) The current implementation of SDP does not result in an unnecessary burden to the

licensee. 

(4) The use of the Phase 2 notebooks currently provides marginal benefit within the ROP

since not all of the notebooks have been benchmarked. 

(5) There is no basis to limit the number of risk assessment tools currently utilized by the

NRC staff. The development and expenditure of resources for both the Phase 2

notebooks and the Standardized Plant Analysis Risk (SPAR) model should not be

suspended. 

(6) It is conceivable that different tools could be applicable to varying categories of

inspection findings. SPAR models have the potential to provide safety significance

insights beyond the plant specific notebooks, while the notebooks, with planned

improvements, could be an effective screening tool.

The DPO Panel found that “NRC management and staff are in the process of addressing many

of the Ad Hoc DPV Panel’s observations and recommendations in the SDP Improvement

Initiative.” However, the DPO Panel also made the following specific recommendations:

(1) The NRC should conduct an independent review of the SDP assessment tools.

(2) The SDP should incorporate uncertainty analysis in the inspection finding assessments.

(3) Criteria for benchmarking the SDP should be subjected to peer review.

(4) Guidance for performing SDP (Phase 3) reviews should be developed and applied

consistently across the regions. 
-4-

Between May and October 2001, the OIG conducted an audit of the SDP. The objectives of the

audit, as indicated in the OIG’s report (OIG-02-A-15) dated August 21, 2002, were to determine

whether: (1) the SDP is achieving desired results; (2) NRC staff clearly understand the

process; and (3) NRC staff are using [the] SDP in accordance with agency guidance. In its

report, the OIG concluded that “while the SDP is meeting its objectives, and agency staff are

using SDP in accordance with guidance, additional refinements are needed.” The report

provided the following recommendations for the NRC:

(1) Develop an action plan to correct Phase 2 analysis weaknesses or eliminate this portion

of the SDP;

(2) Discontinue the expenditure of about $650,000 remaining to develop Phase 2 until the

action plan is completed;

(3) Provide guidance for using information from licensee risk assessments in SDP

evaluations;

(4) Improve SDP timeliness; 

(5) Improve the NRC’s [ROP] web site to more fully inform the public; and

(6) Improve SDP training and guidance.

In a memorandum dated May 14, 2002, the Deputy Executive Director for Reactor Programs

informed the OIG that, in general, the staff agreed with many of the observations and

recommendations in the OIG’s draft report. The memorandum indicated that the staff had

already initiated various SDP improvement initiatives and forwarded specific comments on the

OIG draft report. In a memorandum to the Director of NRR, dated August 6, 2002, the EDO

directed that a plan be developed to address both the DPO Ad Hoc panel and the OIG

recommendations. The EDO’s memorandum indicated that this plan shall address the DPO

Panel recommendation for an “overall objective review of the SDP.” The plan developed by the

Director of NRR included the formation of the SDP Task Group to conduct an independent

review of the SDP. 

2.2 Background

In SECY-99-007, “Recommendations for Reactor Oversight Process Improvements,” dated

January 8, 1999, the staff provided its recommendations to the Commission for improving the

reactor regulatory oversight processes, including proposed changes to the NRC’s inspection,

assessment, and enforcement processes. The staff’s efforts to develop the proposed changes

were guided by three objectives: (1) improve the objectivity of the [reactor] oversight process

so that subjective decisions were not central process features; (2) improve the scrutability of

these processes so that NRC actions have a clear tie to licensee performance; and (3) riskinform the process so that NRC and licensee resources are focused on those aspects of

performance having the greatest impact on safe plant operations. 

With respect to the assessment process, the staff sought to develop a process that would allow

the integration of various information sources relevant to licensee safety performance. In
-5-

SECY-99-007, the staff concluded that adequate assurance of licensee performance would be

achieved through the use of risk-informed performance indicators (PIs) and inspection findings.

The staff also highlighted the need to develop a method for characterizing the risk of inspection

findings, indicating that a “level of risk significance, based on a risk scale, will be determined

and documented for the findings” and that both the PIs and inspection findings would be

evaluated against risk-informed thresholds. 

In SECY-99-007A, “Recommendations For Reactor Oversight Process Improvements”

(Follow-up to SECY-99-007), Attachment 2, dated March 22, 1999, the staff introduced the SDP

as the method for characterizing the risk of inspection findings. The SDP was designed to

assess only those inspection findings associated with at-power operations in the Reactor Safety

Strategic Performance Area cornerstones of initiating events, mitigation systems, and barrier

integrity; however, concepts for characterizing the risk significance of inspection findings in the

emergency preparedness, radiation safety, and safeguards areas were under development. 

The SDP provided a means to screen out inspection findings that have minimal or no risk

significance and trigger a more detailed analysis of potentially risk-significant findings. As

indicated in Appendix 1 to SECY 99-007A, the following objectives were use to guide the staff’s

development of the SDP: (1) characterize the risk significance of an inspection finding

consistent with the regulatory response thresholds used for PIs and (2) provide a risk-informed

framework for discussing and communicating the potential significance of inspection findings.

To support the start of the initial implementation the revised ROP in April 2000, the staff issued

Inspection Manual Chapter (IMC) 0609, “Significance Determination Process.” Appendix A to

IMC 0609 provided guidance for the staff to estimate the unintended increase in risk during atpower plant conditions caused by deficient licensee performance. The guidance was intended

to provide a simplified probabilistic framework for use by the staff in identifying potentially risk

significant findings in the reactor safety area--either the initiating events, mitigation systems, or

barrier integrity cornerstones. 

The reactor safety SDP uses a graduated, three-phase process to differentiate inspection

findings on the basis of their risk significance. Phase 1 of the SDP provides a characterization

of the finding and an initial screening of very low safety-significance findings for disposition by

the licensee’s corrective action program. Phase 2 of the SDP provides an initial approximation

of the risk significance of the finding and develops the basis for the significance determination. 

The Phase 2 SDP is performed using risk-informed inspection notebooks, which the staff

develops for each nuclear plant. The Phase 2 notebooks contain plant-specific worksheets

used by the inspectors to determine the safety-significance (color) of the inspection finding. 

Phases 1 and 2 of the SDP are intended to be accomplished primarily by inspectors and their

supervisors or managers. Phase 3 of the SDP, which is performed by an NRC risk analyst,

involves a review and, as needed, refinement of the risk significance estimate from Phase 2. A

Phase 3 evaluation is also performed for inspection findings which cannot be evaluated using

the plant-specific risk-informed Phase 2 notebooks.

When the ROP was initially implemented in April 2000, the staff‘s efforts to develop the Phase 2

notebooks for each nuclear plant were still in progress. As a result, the draft notebooks that

were made available for staff use at initial ROP implementation were considered to be

incomplete. By late 2000, the staff had made sufficient progress in the site visits associated

with the development of Phase 2 SDP notebooks, that it began to issue the “Revision 0”

notebooks to the sites. After issuance of the first Rev 0 notebooks, the staff identified problems
-6-

with the accuracy of the notebooks and concluded that benchmarking was needed to confirm

the adequacy of the notebooks. Using NRC risk analysts and contractor resources, the staff

began its efforts to benchmark the Phase 2 SDP the notebooks in April 2001. As of November

12, 2002, the staff had issued 24 Revision 1, Phase 2 notebooks.

In March 2002, the staff revised IMC 0609, Appendix A, to provide guidance for evaluating

concurrent inspection findings and clarified the process of accounting for external event core

damage initiators in the risk significance characterization of inspection findings. Attachment 1

to the revision incorporated rules for using the Phase 2 SDP notebooks, including a "counting

rule" convention for estimating the risk significance of inspection findings based on the internal

initiating events that lead to core damage. Also, in a memorandum dated March 18, 2002, the

Director NRR issued an integrated plan to coordinate and complete initiatives aimed at

improving process, tools and knowledge issues associated with the SDP. 

In a memorandum to the Regional Administrators dated August 9, 2002, the Director, NRR,

issued guidance to clarify and emphasize the program office’s expectations for inspector use of

the SDP. The memo provided guidance in six areas; however, particular emphasis was placed

on two areas: (1) the importance of inspectors using their initial SDP evaluation of a finding as

a basis for discussing the finding with the licensee at the earliest opportunity following the initial

characterization of the finding; and (2) guidance to inspectors and SRAs on the use of the

Phase 2 risk-informed inspection notebooks, including when it is necessary for the SRAs to

perform additional analysis of an inspection finding beyond the Phase 2 process. 

3.0 EVALUATION RESULTS AND RECOMMENDATIONS

The SDP, including the Phase 2 process, has generally succeeded in meeting the ROP

objectives of providing a more objective, scrutable, and risk-informed process. In addition, the

four SDP objectives have generally been met by the current SDP process. Although the

contribution of the Phase 2 process to the SDP objectives was limited because of its relatively

infrequent use, feedback from stakeholders indicated that the Phase 2 process has contributed

to the staff’s efforts to characterize the significance of inspection findings, facilitate stakeholder

communication, provide a basis for assessment and enforcement actions, and risk inform the

inspection program. 

The Task Group concluded that the SDP should retain the Phase 2 process to facilitate

(resident and region-based) inspector involvement in the characterization of potentially greaterthan-green inspection findings. However, the unbenchmarked SDP notebook should be

enhanced to address the current problems with the Phase 2 Worksheets, including the

technical inadequacies and the cumbersome, complex and time-consuming characteristics of

the worksheets. In this regard, the Task Group evaluated six options to enhance the SDP and

recommended the development of enhanced pre-solved Phase 2 SDP tables (Option 3) to

replace the Phase 2 SDP notebooks as the primary tool for inspectors to perform Phase 2

evaluations. However, the notebooks should be retained for optional use by the NRC staff to

gain additional risk insights.

The performance expectations for Phase 1 of the SDP were clearly documented in IMC 0609,

well understood by the staff, and have been used to appropriately screen out inspection
-7-

findings of very low safety significance. However, the original performance expectations

regarding the use and accuracy of the Phase 2 process have not been fully realized due, in

large part, to the untimely development and issuance of the Phase 2 notebooks and the errors

associated with the Revision 0 notebooks. The current SDP performance expectations, as well

as the expectations for inspectors to use the SDP, are described in an August 9, 2002,

memorandum from the Director, NRR. The Task Group concluded that the expectations were

understandable and capable of being implemented by the Regions, however, based on the

interviews in the Regions, some staff were unfamiliar with the guidance and others were unsure

of whether and how it had been implemented.

The Task Group determined that resident and region-based inspectors rarely used the Phase 2

notebooks to characterize an inspection finding because the majority of the findings screened

(out) as minor or Green prior to meeting a condition that required use of the Phase 2. Also,

most of the inspector interviewees expressed difficulty in using the Phase 2 notebooks because

the guidance was poorly organized and the worksheets were time consuming and not userfriendly. Notwithstanding these concerns, most of the NRC interviewees supported the

continued use of the Phase 2 process, provided that the current problems are addressed, and

felt that it was essential for inspectors to participate in determining the significance of inspection

findings beyond the initial screening, Phase 1.

IMC 0609 and IMC 0612, “Power Reactor Inspection Reports,” provide some guidance on the

types of inspection findings that can be evaluated using the SDP. The reactor safety Phase 2

SDP notebooks do not currently allow for treatment of inspection findings that are related to

external events, shutdown modes and Large Early Release Frequency (LERF). However, the

staff has initiated action to refine the SDP guidance documents for evaluation of findings related

to shutdown modes, containment, and fire protection. The degraded condition of a Structure,

System or Component (SSC) can also be treated using the SDP if agreement can be reached

on characterizing the impact of the finding on equipment unavailability or initiating event

frequency. The Task Group also concluded that external events can have a significant effect

on risk analyses and, therefore, should be included in the SDP. In this regard, the Task Group

concurred in the ongoing NRR initiatives to develop better tools and guidance for external

events so that the risk insights can be applied in a consistent manner.

A variety of NRC training courses and seminars were administered over a relatively short time

frame and a number of training courses were established to enhance the knowledge of NRC

staff and managers in the area of risk. Although the general consensus from the Task Group’s

interviews of internal and external stakeholders was that the staff's knowledge and ability to

communicate risk insights has improved due, in part, to staff training, many commented on

areas for improvement. In particular, many inspectors felt that the training could have been

better and that it did not make them proficient in use of the Phase 2 tool. Although training was

a factor in inspector proficiency, the Task Group concluded that the complexity of the Phase 2

notebooks, the unavailability of benchmarked notebooks, and the infrequent use of the Phase 2

process were the main hindrances to inspectors' ability to use the Phase 2 SDP. The Task

Group concluded that simplifying the Phase 2 SDP by using enhanced pre-solved tables will

alleviate the need for inspectors’ current reliance on the SRAs to conduct the Phase 2

evaluation. The Task Group also recommended that a systematic assessment of agency

training in the area of risk be conducted. 
-8-

Although the current Phase 2 SDP guidance is complex, it was deemed adequate to implement

the current Phase 2 worksheets. In addition, the Task Group concluded that the enhanced presolved Phase 2 SDP tables will prompt the development of more user-friendly implementation

guidance. The actions and schedule in the SDP Task Action Plan to enhance the current

Phase 3 guidance are appropriate. Although no examples were cited where the limited Phase 3

SDP guidance has resulted in an unsatisfactory outcome (i.e., inadequate or poor quality Phase

3 analysis), the Task Group concurred in the ongoing NRR initiative to enhance the Phase 3

guidance.

The Task Group concluded that the ongoing benchmarking contributes significantly to

improving the quality of the Phase 2 SDP notebooks and SPAR models. Many of the

recommendations contained in the DPO Ad Hoc Panel report regarding the benchmarking

process, including staff qualification and peer review recommendations, have already been

included in the current process. However, the Task Group noted several areas for further

improvement, including documentation of the “construction rules” for developing the Phase 2

SDP and comparison between the results of the benchmarked SPAR models and the

benchmarked notebooks. In addition, because future licensee probabilistic risk assessment

(PRA) changes and/or plant modifications can affect the adequacy of the notebooks, the Task

Group concluded that a plan to update the Phase 2 tool was warranted. 

The Task Group reviewed the use of uncertainty in SDP evaluations in light of the three types

of uncertainty highlighted in Regulatory Guide (RG) 1.174, “An Approach for Using Probabilistic

Risk Assessment in Risk-Informed Decisions on Plant Specific Changes to the Licensing

Basis:” parameter uncertainty; model uncertainty; and completeness uncertainty. The Task

Group concluded that parameter uncertainty was a relatively minor contributor for the purpose

of the SDP, and that model uncertainties associated with the characterization of the impact of

the inspection finding on the function of an SSC represented the greatest source of uncertainty. 

The Task Group also noted that the staff should consider the contribution from external events,

and low power and shutdown events when making a final assessment of risk significance of a

finding (for completeness). The Task Group concurred in the ongoing initiatives to develop

better tools and guidance in the area of low power and shutdown.

The Task Group concluded that the SDP should not be modified to include consideration of

“other inputs.” There were no examples where the current provisions for treating other inputs in

the ROP had limited the staff’s ability to focus resources in appropriate areas. In addition, the

existing guidance in IMC 0305, “Operating Reactor Assessment Program,” regarding deviations

from the Action Matrix can be employed, where appropriate, to consider “other inputs” in

determining agency response. The Task Group also concluded that the current guidance in

IMC 0305 does not promote consistency in the identification of substantive cross-cutting issues

by the Regions and the guidance is vague on the expected NRC and licensee response to a

substantive cross-cutting issue.

The Task Group concluded that the ROP timeliness goal of completing the SDP within 120

days of the first inspection exit meeting and within 90 days of the issuance of the inspection

report is reasonable, however, the goals were not well understood by all members of the staff.

The Task Group recognized that some findings involve complex engineering issues and may

require additional time to adequately assess the safety significance. The Task Group

determined that significant progress has been made in tracking the timeliness of the issues that

are greater than Green and establishing accountability within the responsible offices. However,
-9-

the Task Group concluded that improved management decisiveness in determining the

engineering assumptions used in risk evaluations would contribute to the timeliness of the SDP. 

The Task Group concluded that a recent revision to the Office of Public Affairs (OPA) policy

regarding issuance of a press release for a White findings was an improvement and noted that

further revision to link press releases to a finding(s) that results in a Degraded Cornerstone was

warranted.

The OIG audit report, (OIG-02-A-15) provided several recommendations to enhance the ROP

web site. The Task Group concluded that the recommendation to improve the web site links for

all relevant documents for greater-than-green findings was appropriate and should be adopted

by NRR. The recommendation to link the summary of findings to the inspection report had

already been implemented. The recommendation to revise IMC 0612 to include a brief

summary of corrective actions in the inspection report summary of findings was reasonable. 

The Task Group concluded that the recommendation to display all significant findings colors in

a cornerstone should not be adopted. Web site users are provided ready access to all findings

in a cornerstone by hyper-linking to the next web page, and the difficulty and costs of

implementing this change would exceed the benefit. 

The Task Group determined that the issuance of different NRC risk characterizations from the

Accident Sequence Precursor (ASP) Program and SDP can have a potentially negative impact

on public confidence. Consequently, program requirements and/or an RES/NRR office protocol

should ensure reasonable consistency in the output of the programs to promote public

confidence. The Task Group concluded that efficiencies could be gained through a better

coordination and/or integration of these two programs. 

The Task Group evaluated concerns about the perceived overemphasis on the objectivity of

PRA results and the quality of the PRA models used to determine the final risk significance

inspection findings. The Task Group concluded that the staff routinely uses engineering

judgement in the SDP and that this practice is consistent with the NRC’s PRA Policy Statement

(“Use of Probabilistic Risk Assessment Methods in Nuclear Activities: Final Policy Statement,”

dated August 16, 1995). With respect to the quality and scope of licensee PRAs, the Task

Group concluded that guidance should be developed to allow the staff to determine whether the

results of a licensee’s risk analysis of a finding is of sufficient quality to use as input to the

staff’s final significance determination.

The Task Group concluded that overall, the Action Matrix has fulfilled its purpose in providing

an objective, scrutable and predictable framework for NRC actions in response to licensee

performance problems. The oversight process also provides sufficient flexibility in use of the

Action Matrix for NRC managers to use discretion in decisions concerning the scope and timing

of the agency’s response to licensee performance problems. In addition, the Task Group

concluded that the staff used reasoned judgement to support its decision to use two White

inputs in the same cornerstone as part of the criteria for defining a “Degraded Cornerstone.” 

Although a detailed analysis or evaluation was not developed to support this decision, the Task

Group did not identify data or obtain information from its interviews that suggested that the

criteria were inappropriate. 

Sections 3.1 through 3.11 describe the Task Group’s recommendations to address the

identified issues. 
-10-

3.1 Evaluation of Reactor Oversight Process and Significance Determination Process

Objectives

As directed by the Charter, the Task Group evaluated whether the SDP, particularly the

Phase 2 portion of the process, had succeeded in meeting the objectives of the ROP and SDP,

and whether the Phase 2 SDP should be continued, modified or replaced. In its evaluation, the

Task Group reviewed various background documents, including SECY-99-007, SECY-99-007A,

and IMC 0609. The Task Group also considered in its evaluation, insights obtained from

interviews with internal stakeholders, licensee representatives and other external stakeholders.

3.1.1 Observations

The three objectives of the ROP, which were established in SECY-99-007, are to improve the

objectivity, improve the scrutability, and to risk-inform the inspection and oversight process. 

The first two SDP objectives were established in SECY-99-007A: (1) to characterize the

significance of an inspection finding for the NRC licensee performance assessment process,

using risk insights as appropriate; and (2) to provide all stakeholders an objective and common

framework for communicating the potential safety significance of inspection findings.

Subsequently, two additional SDP objectives were identified in IMC 0609: (3) provide a basis

for assessment and/or enforcement actions associated with an inspection finding; and

(4) provide inspectors with plant-specific risk information for use in risk-informing the inspection

program.

3.1.1.1 Phase 2 SDP Contribution to ROP and SDP Objectives 

Many of the NRC staff interviewed by the Task Group indicated that the ROP objectives,

particularly Objectivity and Scrutability, have been met. A common point of comparison among

the interviewees was their reference to the previous inspection and enforcement process. 

Many stated that the ROP and SDP have reduced the subjectivity that characterized NRC

inspection, enforcement, and assessment decision-making prior to ROP, and provide for a

more understandable and predictable process.

Although many NRC staff interviewees agreed that the SDP had contributed to risk-informing

the ROP, almost all of the NRC staff interviewees stated that they believe the process had

become too risk-based. Many stated that significant staff resources are often expended for

potentially greater-than-green findings in an effort to refine the quantitative output of the SDP

(Phase 3). Some indicated that this refinement, and the associated staff effort, results from the

staff’s desire to develop a precise measurement of risk associated with findings, while others

attributed the perceived need for a precise determination to the fact that any greater-than-green

finding is strongly challenged by the licensee. Some noted that potential differences between

final SDP determinations and the Accident Sequence Precursor (ASP) program evaluations had

prompted a greater sensitivity, on the part of staff conducting SDP evaluations, towards

developing the “best” possible SDP determination. Still others indicated that the SDP is

predisposed to being viewed as a risk-based process because the process requires a

comparison of point estimates to discrete response thresholds (i.e., 10-6 per reactor year,

Green/White; 10-5 per reactor year, White/Yellow; and 10-4 per reactor year, Yellow/Red) to

determine the required regulatory actions. Another indication to some NRC staff that the SDP

is risk-based, is the fact that IMC 0609 does not explicitly include provisions for considering
-11-

traditional engineering considerations (i.e., defense-in-depth and safety margins) during the

SDP deliberation process.

With respect to the four SDP objectives, most NRC staff interviewees also indicated that the

current SDP process had successfully met the first, third, and fourth objectives. There was a

general belief that the SDP had risk-informed what was previously a more subjective, designbasis-oriented approach to regulation. By providing a common frame of reference from which

to evaluate performance issues associated with the initiating event and mitigating system

cornerstones, most interviewees credited the Phase 2 process with providing a consistent

repeatable basis for assessment and/or enforcement actions, particularly where the Phase 2

process screened the finding as one of having very low safety significance (Green). Yet, a

number of NRC staff interviewees shared the view that the contribution of the Phase 2

worksheets to the SDP objectives was unclear because the worksheets have been used

infrequently as the sole input to the final significance determination.

Although it was not identified as the primary tool for inspectors to gain risk insights for 

inspection planning (the fourth SDP objective), the Phase 2 SDP was credited as one of many

tools from which inspectors gained risk insights. Other tools used to acquire risk insights

included licensee online risk monitor programs, maintenance rule scoping summaries,

plant-specific PRA summaries, simplified lists of Risk Achievement Worth values for plant

equipment, etc.

The Task Group’s interviews revealed mixed views on whether the Phase 2 SDP notebooks

have provided an objective and common framework for communicating the potential safety

significance of inspection findings. Some of the NRC staff interviewees indicated that the

Phase 2 process had met this SDP objective because the dialogue between inspectors who

have used the Phase 2 SDP and licensees indicates greater awareness of risk significant plant

issues. Several of the licensee interviewees indicated that they do not use the current Phase 2

notebooks to evaluate the significance of issues and they consider the notebooks to be a tool

for use by NRC inspectors. They also indicated that because the worksheets were designed to

be conservative in their initial assessments of significance, licensees rely more on their own

plant-specific probabilistic risk assessments to analyze findings and plant conditions. However,

despite the fact that they do not routinely use the Phase 2 SDP notebooks, most of the licensee

interviewees indicated that communications with NRC on the risk significance of findings was

good. This was attributed, in part, to the framework of the Phase 2 SDP which allowed the

NRC and the licensee to focus on the influential assumptions.

During a public meeting that the Task Group held on October 25, 2002, external stakeholders

were asked their views on whether the Phase 2 SDP had met the ROP and SDP objectives. 

Their response indicated that timeliness and clarity in the NRC communication of significance

determinations were of greater importance (than the stated objectives) when evaluating the

efficacy of the SDP. One stakeholder indicated that because some findings have not been

processed in a timely manner, the resultant risk characterization was out of phase with the

performance indicators. Another indicated that the NRC’s release of final significance

determinations as much as a year after the fact serves no purpose. The external stakeholders

acknowledged that the use of the Phase 2 worksheets by inspectors does not significantly

influence the timeliness of SDP determinations and that other factors in the SDP process have

a greater impact, such as NRC decisions on basic assumptions (see Section 3.9 , SDP

Timeliness). 
-12-

Several external stakeholders also expressed concern about the amount of information

available to the public concerning the significance determinations of findings, including the fact

that the site-specific SDP notebooks are not currently publicly available documents. While

acknowledging that the SDP notebooks have not been released publicly for security purposes,

they indicated that this underscored the importance of the NRC clearly articulating in its reports

the basis for final significance determinations, particularly when the final determination differs

from the preliminary determination. In general, the external stakeholders indicated that the

SDP (as a whole) had not succeeded in providing timely and clear communications to the public

regarding the significance of findings; thus, the objective of providing a common framework for

NRC staff and stakeholders to communicate the safety significance of findings had not been

fully achieved. When asked about their specific views on the use of Phase 2 worksheets, most

of the external stakeholders were not inclined to support NRC’s use of any particular risk

assessment tool, as long as the results are communicated in a clear and timely manner. 

3.1.1.2 Insights Regarding the Viability of the Phase 2 Process

The interviews with stakeholders revealed mixed views on whether the Phase 2 process should

be continued, modified or replaced. When asked about their specific views on the use of Phase

2 worksheets, most of the external stakeholders did not recommend that NRC use any

particular risk assessment tool, but reiterated their concern that the results of the NRC’s risk

assessments be clear and timely. Although most licensee interviewees indicated that they do

not routinely use the current Phase 2 notebooks to evaluate the significance of issues, several

of the licensee risk analysts indicated that the Phase 2 SDP worksheets represent a useful tool

to screen Green findings and to identify potentially greater-than-green findings for follow-on

review by NRC SRAs. Most of the licensee risk analysts also commented that the Phase 2

worksheets were overly complex and not user-friendly, and suggested that they be made more

simple (as they were initially intended) for use by inspectors to screen Green findings. 

Some NRC staff interviewees indicated that the current Phase 2 SDP had provided

questionable results and that the Phase 2 worksheets are cumbersome and complicated. 

Experience with inadequacies in the Revision 0 notebooks and frustration with the perceived

complexity of the Phase 2 worksheets were the primary reasons for this view. Some

interviewees also indicated that the Phase 2 process was unnecessary, particularly in light of

the staff’s success in using the Phase 1 to initially screen and Phase 3 to estimate the risk

significance of inspection findings. It was noted that the Phase 2 worksheets have been used

infrequently as the sole input to the final significance determination for green or greater-thangreen findings affecting the mitigating systems and initiating events cornerstones. Since the

initial implementation of the ROP in April 2000, most findings were either screened as green in

Phase 1 or, if potentially greater-than-green, the final significance determinations involved a

Phase 3 analysis (due to the complexity of the finding or the lack of confidence in the nonbenchmarked Phase 2 notebooks). Based on data provided by the Regional offices, 54 Green

inspection findings were processed through Phase 2 of the SDP during the 12-month period,

October 1, 2001 to September 30, 2002: eight in Region I; 24 in Region II; nine in Region III;

and 13 in Region IV. An even smaller number was determined to be greater than green after

passing through Phase 2. As a result, some interviewees believed that the time and effort

expended to conduct Phase 2 evaluations with the current notebooks is unwarranted. 
-13-

Notwithstanding the problems encountered to date with use of the Phase 2 SDP, most of the

NRC interviewees support the continued use of the process to screen out Green findings and

identify potentially greater-than-green findings. Most NRC interviewees indicated that the

benchmarking of the Phase 2 SDP workbooks would address many of the problems and that it

was essential for the SDP to continue to facilitate inspector involvement in significance

determinations beyond the initial screening, Phase 1. Many indicated that the Phase 2 process

benefits from the site specific knowledge of resident inspectors in determining the significance

of inspection findings and that placing this responsibility on the Regional SRAs would be less

effective and inefficient. Most NRC interviewees also suggested that the Phase 2 SDP should

be made more simple and user friendly, which would allow the Phase 2 process to be more

easily implemented by inspectors and reduce the burden of screening Green findings by the

SRAs. 

3.1.1.3 Alternatives to the Current Phase 2 Process

In light of the insights obtained from interviews with stakeholders, the Task Group evaluated

seven different options for determining the significance of inspection findings, as summarized in

Figure 1. One of the options was to continue with the current process and complete the

benchmarking of the remaining notebooks (i.e., the “Base Case”). The other six options were

as follows:

Option 1: Retain the Phase 2 SDP, but terminate benchmarking of the remaining

Phase 2 SDP notebooks;

Option 2: Develop Corel Quattro-Pro Spreadsheet Presentation of the Phase 2

SDP Worksheets; complete benchmarking of the remaining Phase 2 SDP

notebooks to support development of spreadsheets. This option

represents a computerized version of the existing worksheets that

requires the user to (i) select affected equipment , (ii) enter exposure

time, and (iii) enter recovery credit, and then the spreadsheet

automatically presents the significance determination (color);

Option 3: Develop Enhanced Pre-Solved Phase 2 SDP Tables using the Phase 2

Worksheets; complete benchmarking of the remaining Phase 2 SDP

notebooks to support development of the tables. The tables are

pre-solved worksheets, with the accident sequences already solved for

each mitigating strategy and for each of the three exposure time ranges

(<3 days, 3 - 30 days, and > 30 days) that are used in the current Phase

2 worksheets. In addition, the table includes easy-to-understand

narrative discussion on the risk insights associated with each of the

mitigating functions or initiating events for which consideration was given

in the original worksheets. A sample table is provided in Figure 2;
-14-

FIGURE 1 - EVALUATION OF SIGNIFICANCE DETERMINATION PROCESS OPTIONS

EVALUATION

CRITERIA--->

 Produces Objective Results

 Produces Scrutable Results

 Produces Risk-Informed Results 

 Facilitates Stakeholder Communication 

 Provides for Assessment and Enforcement

*Promotes Maintaining Safety

*Promotes Efficiency, Effectiveness & Realism

*Reduces Unnecessary Regulatory Burden

*Enhances Public Confidence

 Provides Reasonably Accurate Results 

Provides Independence from Licensee PRA

*User Friendly

 *Facilitates Inspector Involvement in SDP

 *Developmental Costs/Time to DevelopE

Base Case(statusquo) 0000000000000 0 

Option 1 0 0 0 -1 0 0 -2 0 0 -1 0 -2 0 +2 Option 2 0 0 0 0 0 0 +2 0 0 0 0 +2 0 -2 Option 3 0 0 0 +1 0 0 +2 0 0 0 0 +2 0 -2 Option 4 0 0 0 0 0 0 +2 0 0 +1 -1 +2 0 -4† 

Option 5 0 0 0 0 0 0 -2 0 0 +1 n/a 0 -2 +2 Option 6 0 0 0 0 0 0 +2 0 0 +1 +1 +2 0 -4† 

Note: Base Case: Retain Phase 2 SDP; complete benchmarking of remaining notebooks

Option 1: Retain Phase 2 SDP; terminate benchmarking of remaining notebooks Option 2: Develop Quattro-Pro Spread Sheet Presentation; complete benchmarking of remaining notebooksOption 3: Develop Enhanced Pre-Solved Table; complete benchmarking of remaining notebooksOption 4: Develop Enhanced Pre-Solved Table using Licensee’s PRA; terminate benchmarking of remaining notebooksOption 5: Use SDP Phase 1 and SDP Phase 3; terminate benchmarking of remaining notebooks Option 6: Develop SPAR-driven User-friendly Input/Output Device; terminate benchmarking of remaining notebooks

Key: Meets the Base Case = 0 Does Not Meet the Base Case = -1 Exceeds the Base Case = +1

Double Weighted - * Development Cost/ Time to Develop significantly more than the Base Case- †
-15-

FIGURE 2 

ENHANCED PRE-SOLVED PHASE 2 SIGNIFICANCE DETERMINATION PROCESS 

BWR PLANT X

UNAVAILABLE

EQUIPMENT

DURATION

RISK INSIGHTS

<3

DAYS

3-30

DAYS

>30

DAYS

HPCI G W Y The HPCI System is important because it provides 1 of 3 high pressure in

vessel inventory. The following sequences are the primary contributors to In the event of a failure of the feedwater system, either HPCI or RCIC is rIf HPCI and RCIC were unavailable, the operators would need to manuallow pressure injection pumps to maintain core cooling. (2) An inadvertenrequire HPCI to maintain core cooling. In this case, high containment prefeedwater would not be available for vessel injection. Additionally, RCIC provide vessel makeup with a SORV open. Therefore, if HPCI fails, operdepressurize and use low pressure injection to maintain vessel inventory. 

RCIC G W Y The RCIC system is important because it provides 1 of 3 high pressure in

vessel inventory. The sequence of events that contributes to RCIC’s imp

followed by the loss of both HPCI and RCIC. In this case, operators woureactor vessel and use low pressure injection pumps to maintain core coo

RHR

TRAIN A

GW Y The RHR Train A is important because Train A can be used for containm

cooling). If the normal heat removal path to the condenser is lost, suppremaintain containment integrity. It is assumed if containment heat removaresulting in failure of all reactor vessel injection sources, located in the resequence is a loss of turbine building closed cooling water (TBCCW) folloremoval. The failure of TBCCW results in a loss of the condenser and noalso results in the loss of all potential injector sources outside of the reactsuppression pool cooling (SPC) is important because it is necessary to mTBCCW-17]

RHR

TRAIN B

GG G The loss of Train B is not risk significant because it can’t be used for supp

train of four LPCI trains or one of two trains of LPCS trains can be used foSignificance Sequence: TPCS-3, SORV-3, MLCCA-3]

EDG

TRAIN A

GG W The EDGs are important because they provide an alternate means to pow

power is lost. If offsite power is lost and not recovered for a long period odeplete and all reactor vessel injection sources will be lost. However, Placan be used to recharge the station batteries. Therefore, operators can amaintaining the station batteries charge with this portable diesel generato
-16-

Option 4: Develop Enhanced Pre-Solved Phase 2 SDP Tables using the licensee’s

PRA as a basis; terminate benchmarking of the remaining Phase 2 SDP

notebooks. This option provides similar information for the user as

Option 3, however, it would require “benchmarking” and/or quality review

of a licensee PRAs;

Option 5: Use SDP Phase 1 and SDP Phase 3 only; terminate the benchmarking of

the remaining Phase 2 SDP notebooks. This option would require

inspectors to complete the initial screening of inspection findings using

Phase 1 and the SRAs to evaluate all potentially greater-than-green

findings using Phase 3 tools (SPAR models and/or insights from licensee

PRAs);

Option 6: Develop SPAR-driven Input/Output Device for inspectors to conduct

Phase 2 SDP reviews; terminate benchmarking of the remaining Phase 2

SDP notebooks. This option would use the SPAR engine and an

interface device to convert the output of the SPAR to intellectually

manageable information for inspectors.

The Task Group identified 14 criteria to evaluate the merits of the different options. The

criteria, which included the ROP objectives, the SDP objectives, the four NRC Performance

Goals, and other considerations that reflect the predominant views of most stakeholders, were

as follows: Produces Objective Results; Produces Scrutable Results; Produces Risk-Informed

Results; Facilitates Stakeholder Communication; Provides for Assessment and Enforcement;

Promotes Maintaining Safety; Promotes Efficiency and Effectiveness; Reduces Unnecessary

Regulatory Burden; Enhances Public Confidence; Provides Reasonably Accurate Results; User

Friendly; Facilitates Inspector Involvement in SDP; Development Costs/Time to Develop.

Each of the criteria was used to evaluate the six options relative to the Base Case. A grading

scale was also employed to aid the evaluation: -1 = does not meet the Base Case; 0 = meets

the Base Case; and +1 = exceeds the Base Case. For ease of comparison, the Base Case

was assigned a zero in all 14 criteria; however, the Task Group recognized that the current

Phase 2 worksheets did not meet the Base Case for several criteria, as noted in the discussion

of each criterion. The six options were compared to the Base Case in determining their scores.

A (double) weighting scheme was developed and applied to seven of the criteria to recognize

their relative importance to enhancing the SDP. These criteria included the four Agency

Performance Goals, User Friendly, Facilitates Inspector Involvement in SDP and Development

Costs/Time to Develop. To recognize the importance of evaluating existing programs and new

initiatives against the achievement of the agency performance goals, as outlined in the

Strategic Plan, the Task Group determined that additional weight should be given to the agency

performance goal criteria. The Task Group also recognized that the basis for much of the

criticism of the Phase 2 SDP was the cumbersome, complex and time-consuming

characteristics of the worksheets. As a result, the Task Group determined that the User

Friendly criteria should be weighted to emphasize its importance. The vast majority of the NRC

interviewees also cited the importance of continued inspector involvement in the SDP beyond

the initial screening (SDP Phase 1) to exploit their knowledge of the design and condition of the

plant and to enhance the knowledge of inspectors through the practical application of risk

concepts. As a result, the Task Group gave added weight to its importance. Finally, the Task

Group thought it prudent to weight both the Developmental Costs/Time to Develop criteria to
-17-

distinguish the business and practical implications of the various options. For each option, the

range of possible scores was -23 to +21. 

The following paragraphs provide a description of the 14 evaluation criteria and insights into the

basis for the Task Group’s assessment (and scoring) for the Base Case and the six options. A

summary of the Task Group’s evaluation is shown in Figure 1:

(1) Produces Objective Results. The primary factor used to assess the options

against this criterion was whether or not the option would facilitate an objective

determinations of risk when compared to the Base Case. The Task Group

concluded that none of the six options would inherently provide more or less

objective result than the current notebooks. The Task Group considered the fact

that each of the options, including the Base Case, has risk as a key factor in

determining the safety significance of inspection findings. Consequently, each

option reduces the subjectivity involved in significance determinations and were

considered to “meet the Base Case.”

(2) Produces Scrutable Results. The primary factor used to assess the options

against this criterion was whether or not the option would promote

understandable, predictable, repeatable, and traceable results when compared

to the Base Case. Also, key to the Task Group’s assessment of this criterion

was whether knowledgeable stakeholders could review the product and

understand the method by which the agency arrived at its significance

determination. The Task Group concluded that each option would “meet the

Base Case” because the results of each option (a significance determination)

would provide similar input to the agency’s inspection report process–no option

provided a result that was more or less scrutable than the Base Case. 

(3) Produces Risk-Informed Results. The primary factor used to assess the options

against this criterion was the extent to which the option would use PRA concepts

(e.g., initiating event frequencies, equipment failure rates, minimum cut-set

accident sequences, etc.) when compared to the Base Case. The Task Group

found the benchmarked notebooks to be reasonably risk-informed due to their

use of such concepts and because of continued staff efforts to benchmark the

notebooks against the licensee’s PRA. The Task Group concluded that,

because all of the options were rooted in PRA concepts, none was more or less

risk-informed than the other, thus, they all “meet the Base Case.”

(4) Facilitates Stakeholder Communication. The primary factor used to assess the

options against this criterion was the extent to which the option would provide a

more common framework for communicating risk insights between NRC and its

stakeholders when compared to the Base Case. Although the Task Group

received mixed views from interviewees regarding the contribution of the current

Phase 2 notebooks to quality communications between NRC and all of its

stakeholders, the Base Case was assigned a zero to facilitate a comparison of

the options.

Of the six options, the Task Group felt that four (Options 2, 4, 5 and 6) would not

provide more or less of a common frame of reference from which to enter into
-18-

discussions with stakeholders on plant risk significance. Option 1 (retain

notebooks; terminate benchmarking) was considered to “not meet the Base

Case” because the inspectors’ use of non-benchmarked notebooks would not

contribute to a common frame of reference for communicating plant risk

significance. Option 3, however, was viewed as exceeding the Base Case

because the proposed tables would require completion of the benchmarking and

include risk insights using plain-English narrative text for each of the affected

systems. The Task Group believed that these risk insights will promote a more

common understanding of the risk insights among NRC staff (particularly

inspectors) and stakeholders.

(5) Provides for Assessment and Enforcement. The primary factor used to assess

the options against this criterion was the extent to which the option would provide

a more discernible advantage or disadvantage in how it contributed to the

assessment and enforcement processes when compared to the Base Case. In

evaluating each option against this criterion, the Task Group recognized that the

each option would provide comparable input to the enforcement and assessment

processes in the form of safety significance determination (color). Consequently,

all seven options were considered to “meet the Base Case.”

(6) Promotes Maintaining Safety. The primary factor used to assess the options

against this criterion was the extent to which the option contributed more or less

to the agency’s ability to focus on maintaining safety at power reactors when

compared to the Base Case. The Task Group determined that the significance

determinations provided by each of the options would provide an adequate

contribution to the assessment process and that the ROP affords sufficient

defense-in-depth to assure that risk-significant findings would be identified. 

Therefore, each option was considered to “meet the Base Case.”

(7) Promotes Efficiency, Effectiveness and Realism. The primary factors used to

assess this criterion by comparing the options against the Base Case were the

extent to which the option would promote balancing of the risk evaluation

workload, capitalize on information technology, and provide a basis for

consistent quality decision-making. The Task Group noted that the

aforementioned problems with the Phase 2 notebooks (e.g., cumbersome,

complex and time-consuming) detract from its ability to support consistent quality

decision-making; however, the Base Case was assigned a score of zero to

facilitate a comparison of the options. 

The Task Group acknowledged that either of the options would generally support

improvement in consistency and predictability because each option provides a

risk-informed approach to determining the significance of findings. However, the

Task Group determined that Option 1 did "not meet the Base Case" because

terminating the benchmarking of the Phase 2 notebooks would not further the

accuracy of the worksheets and, as a result, there would be a diminished basis

for consistent quality decision-making. 

For Options 2, 3, 4 and 6, the Task Group determined that the improved

accuracy of the tools would provide a basis for more consistent quality decision-
-19-

making. In addition, the Phase 2 SDP worksheets and licensee PRAs, which

have been used to evaluate the safety significance of inspection findings, would

form the basis for the Quattro Pro spreadsheet presentation (Option 2) and the

enhanced pre-solved tables (Options 3 and 4), respectively. For Option 6, a

SPAR-driven, user- friendly input/output device would be used by inspectors to

complete the Phase 2 evaluation. Because these options involve inspectors to

complete the Phase 2 process, there is a sharing of the risk evaluation burden

between inspectors and SRAs. Options 2, 3, 4 and 6 were considered to

"exceed the Base Case.”

Option 5 did "not meet the Base Case" because it would require the SRAs rather

than inspectors to assume the burden of evaluating all findings that screen out of

Phase 1 (as potentially greater-than-green). Although inspectors would still be

required to gather information to support the Phase 3 evaluations, the Task

Group determined that omitting inspectors from the (Phase 2) evaluation of

potentially greater-than-green findings would not exploit their site specific

knowledge and, thereby, result in a less efficient distribution of the risk

assessment workload within a Region. 

(8) Reduces Unnecessary Regulatory Burden. The primary factor used to assess

the options against this criterion was the extent to which the option would

contribute to action that is necessary and sufficient to assure safety when

compared to the Base Case. The Task Group assumed that for each option the

output (color) would continue to be used in the Action Matrix to guide the staff’s

response to inspection findings. As a result, the degree of regulatory burden

imposed on a licensee by either of the SDP options is directly related to the

degree to which the color of the finding, particularly a greater-than-green finding,

adequately represents its significance. For potentially greater-than-green

findings, either of the options would involve the use of an advanced risk

assessment (Phase 3) tool and require formal dialogue with the licensee before

reaching a final significance determination, as is currently done for the Base

Case . Therefore, the Task Group determined that use of either option would

produce adequate significance determinations and result in staff actions that are

necessary and sufficient to assure safety. Each option was found to “meet the

Base Case.”

(9) Enhances Public Confidence. The primary factor used to assess the options

against this criterion was the degree to which the options would provide

stakeholders with clear and timely information about the significance of

inspection findings, as compared to the Base Case. Insights gained from

discussions with external stakeholders indicated that, in general, the public is

more concerned about the NRC’s efforts to assure the clarity and timeliness of

significance determinations than with the agency’s use of a particular SDP tool. 

The Task Group determined that there was no discernible difference between

the options (in terms of their output) that would impact the clarity and timeliness

of the information that is conveyed to the public about the significance of

inspection findings. Other elements of the ROP, such as the inspection reporting

requirements and implementation of timeliness goals, have a greater impact on
-20-

the clarity and timeliness of significance determinations. The Task Group found

that each option “meets the Base Case.”

(10) Provides Reasonably Accurate Results. The primary factor used to assess the

options against this criterion was the degree to which the options provided a

reasonable estimate of the risk associated with an identified plant condition, as

compared to the Base Case. The Task Group decided that the measure of

accuracy was the degree to which the option would minimize the likelihood that

the Phase 2 output would be significantly modified after a Phase 3 evaluation

was conducted.

The Task Group determined that Option 1 “did not meet the Base Case”

because the use of non-benchmarked notebooks would not consistently provide

reasonable estimates of risk. Options 2 and 3 were determined to “meet the

Base Case” because they essentially represent the “answers” derived from the

worksheet methodology, but in a more user-friendly format.

Options 4, 5, and 6 would employ more advanced risk assessment tools, such as

the SPAR models, which are benchmarked to the licensees’ PRAs and reduce

the potential for calculational errors. In this regard, the Task Group assumed

that, in most cases, the licensees’ PRAs (if performed in accordance with an

NRC-approved PRA standard) would be the best tools available for determining

the risk significance of a finding. Options 4, 5 and 6 were found to “exceed” the

Base Case.

(11) Provides Independence from Licensee PRA. The primary factor used to assess

the options against this criterion was the degree to which the options would

provide for an objective assessment of inspection findings by inspectors when

compared to the Base Case. Although the licensees’ PRAs may, in many cases,

represent the best-available tool for judging the “accuracy” of SDP risk

significance determinations, the Task Group determined that the SDP must be

consistent with the NRC Principles of Good Regulation and, therefore, assure a

degree of independence and objectivity when assessing the significance of

inspection findings. For the Base Case, the site-specific Phase 2 SDP

worksheets were developed using generic data, consistent construction rules

and functional level accident sequences for similar reactor types. During the

benchmarking site visits, the results of selected unavailable equipment cases are

compared between the licensee’s PRA and the plant-specific notebook, areas of

difference are recognized, reasons for the differences are understood, and

appropriate changes to the notebooks are made.

Similar to the Base Case, Options 1, 2, and 3 use the Phase 2 SDP worksheets

to support their respective significance determinations. As a result, the Task

Group did not identify a discernible difference in the ability of these options to

provide for an unbiased assessment. Thus, Options 1, 2 and 3 “met the Base

Case.” The Task Group determined that Option 4 would “not meet the Base

Case” because it would be based solely on the licensees’ PRA. 
-21-

This criterion was determined to “not apply” to Option 5 because this option

would not include a Phase 2 process, as does the Base Case. Option 6 was

judged to be more objective, in that the development of the SPAR models had

entailed a more extensive independent review by the staff and the risk

assessment tool itself was not solely based on the licensee’s PRAs. The Task

Group determined that Option 6 “exceeded the Base Case.”

(12) User-Friendly. The primary factors used to assess the options against this

criterion was the amount of user training required, the amount of time and effort

required to determine the significance of a finding, the apparent complexity of

the tool, and the extent to which information technology (IT) was employed, as

compared to the Base Case. The Task Group observed that the benchmarked

notebooks were generally viewed as not user friendly, instead they were often

characterized as being cumbersome, time-consuming, and difficult to use. 

However, to allow for comparison with the other options, the Base Case was set

to zero. 

Option 1 was viewed to be less advantageous than the Base Case because the

current limitations of the Base Case would be preserved. Also, by terminating

the benchmarking of the remaining Phase 2 notebooks, the time and effort

required for inspectors to determine the significance of some findings would

continue to be extended. Option 1 did “not meet the Base Case.” 

Options 2, 3, and 4 would likely reduce the amount of time and effort required for

inspectors to make significance determinations. In addition, each option would

take advantage of IT tools to simplify and automate the process, and would

require minimal training for users. Similarly, Option 6 would employ IT tools,

allow for more timely significance determinations, and present a simplified

conceptual interface for users. Although more extensive training may be

warranted for Option 6, the Task Group determined that Options 2, 3, 4, and 6

“exceed the Base Case.” 

Option 5 would eliminate the current Phase 2 process from the SDP and require

SRAs rather than inspectors to evaluate the significance of potentially greaterthan-green findings. The Task Group determined that although this option would

increase the SRA workload, it would not significantly impact the training, time

and effort, complexity of the tools, or the use of IT by SRAs. The Task Group

determined that it “meets the Base Case.”

(13) Facilitates Inspector Involvement in the SDP. The primary factor used to assess

the options against this criterion was the degree to which the options allowed

inspectors to be involved in the assessment of potentially greater-than-green

findings, as compared to the Base Case. In the Base Case, inspectors are

required to conduct the Phase 2 review by completing the Phase 2 SDP

worksheets. 

Options 1, 2, 3, 4, and 6 were determined to be comparable to the Base Case in

terms of inspector involvement. Although the remaining notebooks would not be

benchmarked in Option 1 and 4, inspectors would continue to use the
-22-

worksheets and the enhanced presolved tables, respectively, to complete the

Phase 2 significance determination. Options 2, 3 and 4 would require

completion of the benchmarking of the remaining notebooks and would provide

user-friendly tools for inspectors to conduct the Phase 2 evaluations. In Option

6, inspectors would use a SPAR-driven user-friendly interface to complete the

Phase 2 determination. Options 1, 2, 3, 4, and 6 “meet the Base Case.”

Option 5 would eliminate the current Phase 2 process from the SDP and require

SRAs rather than inspectors to evaluate the significance of potentially greaterthan-green findings, therefore, it would “not meet the Base Case.”

(14) Developmental Costs / Time to Develop. The primary factors used to assess the

options against this criterion were whether or not the cost and time to develop

the options would be the same (0), less than (+2), more than (-2), or significantly

more than (-4) the Base Case. For the Base Case, the Task Group used the

cost estimates and schedule associated with completing the benchmarking of

the remaining Phase 2 notebooks. Based on data obtained from NRR, as of

November 12, 2002, the staff had completed benchmarking of 41 Revision 1

notebooks; 24 were issued for staff use and 17 were undergoing quality reviews. 

Notebooks for 30 plants still required benchmarking. The estimated cost to

complete the remaining notebooks, by the end of FY03, was $25K per notebook,

for a total estimated cost of $750K. 

Option 1 would retain the use of the Phase 2 worksheets by inspectors, but

would terminate the benchmarking of the remaining notebooks. Option 5 would

eliminate the current Phase 2 process altogether and terminate the

benchmarking of the remaining notebooks. Options 1 or 5 could be implemented

immediately and the estimated costs of completing the benchmarking of the

remaining notebooks would be avoided, thus, both options would “exceed the

Base Case.”

Options 2, 3, 4, and 6 were determined to involve costs above the Base Case, as

well as time to develop beyond the end of FY03; therefore, they did “not meet

the Base Case.” Option 2 would require the development of a Quattro-Pro

Spreadsheet Presentation of the Phase 2 SDP Worksheets for each plant at a

cost of approximately $850K. Option 3 would require the development of presolved Phase 2 SDP tables for each plant at an estimated cost of $386K ($216K

for the 41 benchmarked notebooks and $170K for the 30 non-benchmarked

notebooks). Both Options 2 and 3 would still require the completion of notebook

benchmarking for the remaining plants. 

Although Option 4, would not require the benchmarking of the remaining Phase

2 notebooks, it would require the staff to develop pre-solved tables for each plant

and a review of the quality of all licensee PRAs. This review would be needed to

promote independence and to ascertain the adequacy of licensee PRAs since

they would form the basis for the pre-solved tables. The Task Group found that

the scope and costs associated with an NRC review of licensee PRAs was

difficult to predict. However, the Task Group assumed that the costs and time to

develop would significantly exceed those for the Base Case. 
-23-

Option 6 would eliminate the need to complete the benchmarking of the

remaining notebooks, however, the Task Group concluded that the research,

development, implementation, and training costs, and developmental time

associated with a site specific SPAR-driven interface would significantly exceed

those for the Base Case.”

The scores for all six options ranged between -4 and +3. The scores for three of the options

(Options 2, 3, and 6) provided very little discrimination because they all fell in the +2 to +3

range. This indicated that each of these options were feasible enhancements to the current

Phase 2 process. The team considered these options and determined that Option 3, which also

had the highest score (+3), was the preferred Phase 2 SDP enhancement. The Task Group

determined that Option 3 eliminates for inspectors the time-consuming calculations associated

with the current worksheets. In fact, the tables are simply pre-solved worksheets, with the

minimum cut-set sequences already solved for each mitigating strategy and for each of the

three exposure time ranges (<3 days, 3 - 30 days, and > 30 days) that were used for the

original notebook concept. In addition, the table includes a column that provides an easy-tounderstand narrative discussion on the risk insights associated with each of the mitigating

functions or initiating events for which consideration was given in the original worksheets. By

having the information readily available on the computer and/or in hard-copy notebooks, the

inspectors would not have to resort to long mathematical exercises to gain risk insights for

inspection planning purposes.

3.1.2 Conclusions

(1) The SDP, including the Phase 2 process, had generally succeeded in meeting the ROP

objectives of providing a more objective, scrutable, and risk-informed process,

particularly when compared to the previous NRC inspection, enforcement and

assessment processes. 

(2) The four SDP objectives have generally been met by the current SDP process. 

Although the contribution of the Phase 2 process to the SDP objectives was limited

because of its relatively infrequent use, feedback from stakeholders indicates that the

Phase 2 process has contributed to the staff’s efforts to characterize the significance of

inspection findings, facilitate stakeholder communication, provide a basis for

assessment and enforcement actions, and risk inform the inspection program. 

(3) The SDP should continue to facilitate (resident and region-based) inspector involvement

in the significance determination of inspection findings beyond the initial screening,

Phase 1.

(4) The SDP should be enhanced to address the current problems with the Phase 2

Worksheets, including the technical inadequacies and the cumbersome, complex and

time-consuming characteristics of the worksheets.

(5) Of the six options that the Task Group evaluated, Option 3 was the preferred method to

replace the Phase 2 SDP notebooks as the primary tool for inspectors to perform Phase

2 evaluations. However, the notebooks should be retained for optional use by the NRC

staff to gain additional risk insights. 
-24-

(6) Most of the NRC interviewees viewed the SDP process as risk-based vs risk-informed. 

This opinion was based on the fact that the SDP requires a comparison of point

estimates to discrete response thresholds and the understanding that considerable staff

resources have been expended to develop a precise quantitative estimate of risk for

many potentially greater-than-green findings. The interviewees also indicated that this

aspect of the SDP appeared to conflict with its initial performance expectation, which

was to provide reasonably accurate estimates of risk for timely decisions regarding

follow-up inspection (see Section 3.2, Performance Expectation for the SDP). The Task

Group observed that the Phase 1 SDP process uses a risk-informed decision logic to

screen (Green) findings or determine that the finding needs a more detailed analysis. 

However, both Phase 2 and Phase 3 of the SDP use probabilistic tools to assess the

significance of an inspection finding. This SDP output is then used in the ROP Action

Matrix to determine the agency’s and expected licensee response to the finding. 

RG 1.174 indicates that the integration of risk information with traditional engineering

decisions signifies that a process is risk-informed. Using the RG 1.174 approach to riskinformed decision-making as a model, the Task Group concluded that the degree to

which the SDP represents a risk-informed process is reflected in the degree to which

traditional engineering analysis as well as the SDP output are used in the decisionmaking process. In addition, the current practice of basing the color solely on the SDP

output gives credence to the assertion that the SDP has been implemented in a more

risk-based than risk-informed manner. Section 3.7.3(2) provides a recommendation for

additional staff guidance regarding the use of traditional engineering analysis.

3.1.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) The NRC should retain the Phase 2 process as a screening method for inspectors to

screen findings of very low safety significance (Green) and identify potentially greaterthan-green findings for further review by SRAs. 

(2) The NRC should develop plant-specific Enhanced Pre-Solved Phase 2 SDP Tables

(Option 3) and complete the benchmarking of the remaining Phase 2 SDP notebooks. 

Once developed, these tables should replace the current Phase 2 notebooks.

3.2 Performance Expectation for the Significance Determination Process 

The DPO Ad Hoc Panel report dated June 28, 2002, indicated that the NRC should develop

and issue a performance expectation for the SDP. To evaluate this recommendation, the Task

Group reviewed SECY-99-007, SECY-99-007A, and IMC 0609, and interviewed internal

stakeholders, licensee representatives and other external stakeholders.

3.2.1 Observations

In SECY-99-007, the staff concluded that adequate assurance of licensee performance would

be achieved through the use of risk-informed PIs and inspection findings. The staff also

highlighted the need to develop a method for characterizing the risk of inspection findings,

indicating that a “level of risk significance, based on a risk scale, will be determined and
-25-

documented for the findings” and that both the PIs and inspection findings would be evaluated

against risk-informed thresholds. SECY-99-007A introduced the SDP as the agency’s method

for characterizing the risk of inspection findings. The SDP was designed to provide a means to

screen out inspection findings that have minimal or no risk significance and trigger a more

detailed analysis of potentially risk-significant findings. 

The staff developed IMC 0609, Appendix A to provide guidance for the staff to estimate the

unintended increase in risk during at-power plant conditions caused by deficient licensee

performance. The guidance was intended to provide a simplified probabilistic framework for

use by the staff to identify potentially risk significant findings in the reactor safety

area–specifically, the initiating events, mitigation systems, and barrier integrity cornerstones. 

The reactor safety SDP uses a graduated, three-phase process to differentiate inspection

findings based on their risk significance. Phase 1 of the SDP provides a characterization of the

finding and an initial screening of very low safety-significance findings for disposition by the

licensee’s corrective action program. Phase 2 of the SDP provides an initial approximation of

the risk significance of the finding and develops the basis for the significance determination.

Phases 1 and 2 of the SDP were intended to be accomplished primarily by inspectors and their

supervisors or managers. Phase 3 of the SDP requires an NRC Senior Reactor Analyst (SRA)

of NRR Risk Analyst to review and refine, as needed, the risk significance estimate from Phase

2. SRAs were also expected to perform a confirmatory analysis using the SPAR models if a

finding screened from Phase 2 as potentially greater-than-green. A Phase 3 review was also

required for findings that cannot be evaluated using the Phase 2 (e.g., external events,

shutdown and, LERF. 

Most NRC interviewees indicated that they understood and were satisfied with the SDP Phase 1

screening expectations. They indicated that the Phase 1 screening appropriately screened

inspection findings of very low safety significance, thereby allowing resources to be focused on

issues of greater safety significance. 

In addition, most NRC interviewees understood that the Phase 2 was intended to be a simple

tool that would allow for a relatively easy characterization of the risk significance of a finding. 

They also indicated that the initial performance expectation for the Phase 2 SDP notebooks

was that it would serve as the primary tool to characterize the majority of potentially greaterthan-green inspection findings, represent a reasonably accurate estimation of the risk of an

inspection finding, and facilitate timely agency decisions regarding the allocation of resources. 

However, some NRC interviewees observed that the Phase 2 was designed to provide

conservative results and that, as a result, a Phase 3 would likely be performed before final

significance determinations were made. 

The Task Group determined that challenges experienced during and prior to the initial

implementation of the Phase 2 notebooks impacted the ability of users to achieve the original

performance expectation of the Phase 2 process. For example, the staff planned to have

licensees review and comment on the Revision 0 notebooks, and to modify the notebooks to

correct any significant errors. However, the Revision 0 notebooks were not distributed to all

licensees, many of the licensees that received the notebooks did not provide comments to the

staff, and some of the licensees that provided comments to the staff did not receive feedback

on how their comments were treated. After initial issuance of the first Revision 0 notebooks,

the staff realized that the notebooks contained errors, such as missing initiating events,
-26-

modeling errors and inappropriate success criteria, that were deemed significant enough to

warrant an independent benchmarking initiative to add quality to the notebooks.

In addition, most Regional interviewees revealed that during the initial implementation of the

ROP, the Phase 2 notebooks had not been issued and were not available for the inspectors to

use. This fact led the Regions to develop an early reliance on SRAs to evaluate potentially

greater-than-green findings. There was no indication that the ongoing benchmarking efforts

had reduced this reliance on SRAs. In general, the Regional interviewees considered the

Phase 2 worksheets to be cumbersome and difficult to use, and noted that on average, the

worksheets were used only a few times a year to evaluate findings. As a result, despite the

expectation for inspectors to use the Phase 2 worksheets for potentially greater-than-green

findings, questions about the adequacy, infrequent use, and difficulties in use of the worksheets

continued to cause many inspectors to seek assistance from an SRA.

The Task Group’s interviews highlighted an unanticipated result from the NRC’s use of risk

assessment tools to evaluate inspection findings. Many of the interviewees indicated that

licensees routinely use their PRAs to develop their own estimate of risk (i.e., a quantitative

estimate of the change in core damage frequency ( CDF) and/or LERF) in response to an

inspector’s identification of a preliminary finding of greater-than-green safety significance. This

had caused the staff to employ Phase 3 tools routinely to confirm the risk estimate and support

the formal deliberations with licensees on the CDF and/or LERF, as well as the influential

assumptions and sensitivities that effect the inputs to the risk models. This approach differed

from the initial performance expectation, which envisioned use of the Phase 2 tool by inspectors

to develop the estimate of risk significance for the majority of potentially greater-than-green

inspection findings. During interviews with NRR staff, the Task Group determined that a

document was under development that would describe the precepts and principles that were

used in the development of the SDP (SDP Basis Document). This document will also include

the SDP performance expectations. 

In a memorandum to the Regional Administrators dated August 9, 2002, the Director of NRR

issued guidance to clarify and reemphasize the program office’s (current) expectations for

inspector use of the SDP. The memo provided guidance in six areas, including the expectation

that: (i) inspectors use their initial SDP evaluation of a finding as a basis for early discussion of

the finding with the licensee; (ii) inspectors use the most current version of the Phase 2 SDP

notebooks, including Revision 0, to better understand differences between the notebooks and

the licensee’s risk model, including differences in influential assumptions; and (iii) SRAs

perform a Phase 3 when the inspector doubts the results of the Phase 2 and if the results are

within one order or two orders of magnitude of the Green/White threshold, using the Revision 1

or the Revision 0 notebooks, respectively. However, based on the Task Group interviews with

Regional personnel, some were unfamiliar with the guidance and others were unsure of

whether or how it had been implemented. 

3.2.2 Conclusions

(1) The performance expectations for Phase 1 of the SDP were clearly documented in

IMC 0609, well understood by the staff, and have been used to appropriately screen out

inspection findings of very low safety significance.
-27-

(2) The original performance expectations regarding the use and accuracy of the Phase 2

process have not been fully realized due, in large part, to the untimely development and

issuance of the Phase 2 notebooks and the errors associated with the Revision 0

notebooks. 

(3) Because of the problems with development and implementation of the Phase 2

notebooks, the potential benefits (e.g. facilitating stakeholder communication, risk

informing the inspectors and reducing SRA workload) of the Phase 2 process have not

been fully realized. 

(4) The current SDP performance expectations, which are described in an August 9, 2002,

memorandum from the Director, NRR, were understandable and capable of being

implemented by the Regions. Based on interviews in the Regions, however, some staff

were unfamiliar with the guidance and others were unsure of whether and how it had

been implemented.

(5) The use of SDP Phase 3, including the SPAR models or a best-available tool, by SRAs

for confirmatory analysis and to facilitate formal discussions between the staff and a

licensee on a potentially greater-than-green inspection finding is consistent with the

current performance expectation outlined in the August 9, 2002, memorandum from the

Director, NRR to the Regional Administrators.

3.2.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should issue the proposed SDP basis document, including the current

performance expectations for the Phase 2 notebooks. 

(2) NRR should engage the Regions to confirm their understanding and implementation of

the expectations regarding use of the SDP provided in the August 9, 2002,

memorandum from the Director of NRR to the Regional Administrators.

(3) NRR should reevaluate the performance expectation of the SDP tools after the

remaining notebooks have been benchmarked and modify program guidance, as

appropriate, to reflect any revision to the expectation. 

3.3 Expectations for Use of Significance Determination Process Phase 2 .

The purpose of the Task Group’s review in this area was to respond to issues raised in the OIG

audit report (OIG-02-A-15) regarding inspector use of the SDP. The OIG’s report indicated that

inspectors use the Phase 2 SDP infrequently and are, as a result, hindered in their ability to

effectively use the Phase 2 as a risk characterization tool. The Task Group’s review also

addresses the question raised in the Charter regarding whether the SDP is implemented by the

most appropriate agency personnel. To respond to these issue, the Task Group reviewed

IMC 0609, IMC 1245, ”Inspector Qualification Program For the Office of Nuclear Reactor

Regulation Inspection Program,” and interviewed internal stakeholders.
-28-

3.3.1 Observations

Specific guidance for inspectors to use the Phase 1 SDP screening and Phase 2 SDP risk

characterization process is provided in Attachment 1 to IMC 0609. In general, the guidance

requires inspectors to use the Phase 1 to initially screen very low safety-significance (Green)

findings and the Phase 2 notebooks to screen Green findings and characterize potentially

greater-than-green findings. The guidance also indicates that an SRA should perform a

confirmatory analysis using the SPAR models for those findings that are characterized as

potentially greater-than-green finding from Phase 2, and to evaluate findings that cannot be

evaluated using the Phase 2 SDP. As discussed in Section 3.2.1, the current expectations for

inspector use of the SDP were also provided in a memorandum dated August 9, 2002, from the

Director, NRR to the Regional Administrators. The memo clarified and reemphasized guidance

in six areas, including the expectation that inspectors use the most current version of the Phase

2 SDP notebooks and use their initial SDP evaluation of a finding as a basis for early

discussions with the licensee.

With few exceptions, the inspectors interviewed by the Task Group indicated that, while they

understood the expectations for use of the Phase 2 SDP, they rarely used their site specific

Phase 2 notebooks to screen Green findings or to evaluate potentially greater-than-green

findings. They indicated that most inspection findings screen out as either Minor, using the

criteria for minor findings in IMC 0612, “Power Reactor Inspection Reports,” or as Green, using

the Phase 1 screening criteria. This feedback appears to be supported by data that the Task

Group obtained from the Reactor Programs System and the Regions, which indicated that

during FY02, over 500 findings in the IE, MS and BI cornerstones were screened to Green

using the Phase 1. 54 findings were evaluated as Green using the Phase 2 (see Section

3.1.1.2). Another 14 findings (12 White & 2 Red) were evaluated using Phase 3, some of which

involved an initial risk estimation using Phase 2.

Some inspectors also indicated that they are not confident in the results of the current Phase 2

worksheets because of the extensive usage rules, confusing guidance and organization of the

notebooks, and the “perceived” inaccuracies of the notebooks. Most inspectors indicated that

they usually engage the SRAs when they use the Phase 2 SDP.

Some of the inspectors interviewed indicated that they use their site specific notebooks for

inspection planning. Most inspectors indicated that they also use the licensee’s licensee online

risk monitor programs, maintenance rule scoping summaries, plant-specific PRA summaries,

simplified lists of importance measures (i.e., risk achievement worth values) for plant

equipment, etc., to plan inspections.

Although many Regional interviewees indicated that they were aware of the August 9, 2002,

memorandum from the Director of NRR and the general subject areas that it addressed,

evidence that the guidance had been implemented was unclear. However, the Task Group

noted some cases where Regional training and discussions had been conducted to review the

guidance. 

As discussed in Section 3.1.1.2, notwithstanding the problems encountered to date with use of

the Phase 2 SDP, most of the NRC interviewees support the continued use of the process,

particularly if the benchmarking of the Phase 2 SDP notebooks is completed and the tool is

enhanced to make it more user friendly and less time consuming. In addition, most of the
-29-

interviewees stated that it was essential for inspectors to participate in significance

determinations beyond the initial screening, Phase 1.

3.3.2 Conclusions

(1) Resident and region-based inspectors rarely used the Phase 2 notebooks to

characterize an inspection finding because the majority of the findings screen (out) as

minor or Green prior to meeting a condition that would require use of the Phase 2. Also,

most of the inspector interviewees expressed difficulty in using the Phase 2 notebooks

because the guidance was poorly organized and the worksheets were time consuming

and not user-friendly. 

(2) Some inspectors have used the Phase 2 notebooks to gain risk insights for inspection

planning, however, it is one of many tools that inspectors have used for this purpose. 

(3) Most of the NRC interviewees supported the continued use of the Phase 2 process,

provided that the current problems are addressed. They also considered it essential for

inspectors to participate in determining the significance of inspection findings beyond

the initial screening, Phase 1.

(4) The expectations for inspector use of the SDP are described in IMC 0609 and in an

August 9, 2002, memorandum from the Director of NRR. The expectations in the

Director’s August 9, 2002, memorandum are understandable and capable of being

implemented by the Regions, however, based on Regional interviews, some staff were

unfamiliar with the guidance and others were unsure of whether and how it had been

implemented. This conclusion is similar to Conclusion 3.2.2.4.

3.3.3 Recommendations

To address the issues discussed in this section of the report, the Task Group supports

Recommendations 3.1.3(1) and 3.1.3(2). 

3.4 Applicability of the Current Significance Determination Process to the Range of

Inspection Findings

The purpose of the Task Group’s review in this area was to determine whether the applicability

of the current SDP to the range of possible inspection findings is clearly articulated in the SDP

program guidance. In order to address this question, the Task Group reviewed IMC 0609 and

interviewed internal stakeholders. 

3.4.1 Observations

The majority of the interviewees stated that the inspection findings related to the following areas

cannot be evaluated using the Phase 2 SDP notebooks because the notebooks are limited to

core damage sequences for internal initiating events at power: 

(1) External initiating events (e.g., fire, seismic, and high winds). The Task Group noted

that there are two types of findings associated with external events:
-30-

(a) findings associated with plant features that are designed to protect the plant

against the physical impact of the external initiating event, e.g., tornado missile

barriers, seismic restraints, fire barriers; and

(b) findings associated with mitigating systems that are used in response to

accidents initiated by external events;

(2) Shutdown; and 

(3) LERF (or more generally containment issues)

However, the Task Group noted IMC 0609 provides guidance for evaluating shutdown and

LERF issues in Appendix G and Appendix H, respectively. 

The Task Group determined that, in general, there were no restrictions on Phase 3 evaluations

other than the fact that NRC staff does not have ready access to the tools to analyze the risk

from external events and for the low power and shutdown modes. However, the staff has had

difficulty dealing with other types of findings because the characterization of their impact on the

plant components is complex. These types of findings, which have a similar effect on both

Phase 2 and Phase 3, include:

(1) Degraded conditions;

(2) Concurrent failures of multiple systems/components (guidance was recently provided in

IMC 0609); and 

(3) Cross-cutting issues

Many interviewees stated that the inspection findings related to internal events and mitigating

systems can and have been evaluated using the SDP. They also acknowledged that IMC 0609

and IMC 0612 provide some guidance on what types of inspection findings can be evaluated

using the Phase 2 notebooks. The majority of the inspectors stated that the guidance is difficult

to follow and that this is exacerbated by their infrequent use of the Phase 2 notebooks. No

specific recommendations were provided to improve the guidance in this area.

Following the first year of ROP implementation, the staff recognized that the quality of the

Revision 0, Phase 2 SDP notebooks was not adequate to assess the significance of inspection

findings. NRR initiated a project to enhance the quality of the Phase 2 SDP notebooks by

comparing the results of the notebooks with other available PRA tools (benchmarking). 

Lessons learned from an early benchmarking effort highlighted the importance of including

external events risk contributions in SDP analyses. The benchmarking included an examination

of four plant-specific PRAs with integrated external initiating events risk modeling to determine

if the absence of external events in the phase 2 notebooks would result in an underestimation

of risk. In three of the four PRAs, systems and components were identified for which the

significance of inspection findings using the internal events PRA model would result in an order

of magnitude underestimation (e.g., by one color) compared to the integrated PRA model. 

Many NRC interviewees stated that the inclusion of external events risk insights is important to

properly characterize the risk of inspection findings. The SDP guidance (IMC 0609, Appendix

A, Attachment 1, Step 2.5) states that for Phase 2 SDP evaluations that represent an increase
-31-

in risk of greater than 1E-7 per reactor-year (Risk Significance Estimation of 7 or less), that an

SRA or other NRC risk analyst should perform a Phase 3 analysis to estimate the increase in

risk due to external initiators. However, the SRA interviewees stated that the quality of

licensee’s external events risk information varies significantly and, as a result, there has been

general inconsistencies in the inclusion of external event risk insights in SDP analyses. To

address this concern, the SRA’s requested additional guidance and tools, and recommended

that the treatment of external events in the SDP be temporarily suspended. The

recommendation to temporarily suspend the treatment of external events was under NRR

evaluation during the period of the Task Group review, consequently, the Task Group did not

review this area in detail. Recently, however, NRR decided to not suspend the use of external

events in the SDP. 

3.4.2 Conclusions

(1) The reactor safety Phase 2 SDP notebooks do not treat inspection findings that are

related to external events, shutdown modes and LERF. However, the NRC staff is

currently refining the guidance documents for SDP evaluation of shutdown modes,

containment issues that relate to LERF (NRR SDP Improvement Task Action Plan

(Task 3, Objectives 3.3), and an SDP for fire protection. The Task Group noted that the

fire SDP that is under development only addresses fire protection issues, and is not an

SDP for fire as an external initiating event in the traditional PRA sense.

(2) Degraded conditions can be treated using the SDP if agreement can be reached on

characterizing the impact of the finding on equipment unavailability or initiating event

frequency. However, the notebooks are not benchmarked for multiple findings and the

results would be uncertain. 

(3) Because external events can have a significant effect on risk analyses, they should be

included in the SDP. Therefore, the current guidance and tools should be improved so

that external event risk insights can be applied by the SDP in a consistent manner . 

NRR should continue its initiatives to develop better tools and guidance in the area of

external events as outlined in SDP Improvement Task Action Plan (Task 3,

Objectives 3.1a, 3.3g, 3.7). 

3.4.3 Recommendations

The Task Group did not provide any recommendations in this area.

3.5 Significance Determination Process Training and Guidance

The purpose of the Task Group’s review in this area was to determine if improvement was

warranted in SDP training and guidance. This review was prompted by recommendations

made by the OIG and the DPV Ad Hoc Panel. The OIG Audit Report (OIG-02-A-15) included a

recommendation for the NRC to improve SDP training and guidance for inspectors. The DPV

Ad Hoc Panel report, dated January 10, 2002, recommended that improved guidance be

developed for Phase 3 analysis. To evaluate these recommendations, the Task Group

obtained information on SDP and PRA training, reviewed IMC 0609, and interviewed NRC staff

and managers. 
-32-

3.5.1 Observations

The OIG audit report (OIG-02-A-15, Section E) documented two staff suggested improvements

regarding SDP training program. The improvements include developing training on the

application of risk to plant operations and SDP refresher training for infrequently used SDPs. 

The DPV also indicated that the Phase 2 SDP notebooks are difficult to use and there has been

inadequate training on their use. The SDP Task Action Plan includes objectives to partially

address these concerns. The SDP Task Action Plan (Objective 4.1) provides actions to

conduct SDP training on the Revision to IMC 0609A (dated March 18, 2002) at each Region

and to generate schedules for the development SDP refresher and initial training. NRC

IMC 1245 does not currently require any PRA/SDP refresher training for inspectors. 

The inspection staff have been provided two formal SDP training sessions and one computer

based training exercise. The first formal SDP training was conducted prior to the initial

implementation of the ROP (Spring 2000). The Phase 2 SDP training conducted during the

initial implementation of the ROP training was largely ineffective because the SDP notebooks

had not been developed at that time. The inspection staff were recently provided additional

Phase 2 SDP training (Fall 2002) following the revision to IMC 0609, Appendix A, on March 18,

2002, that significantly revised the methodology for conducting Phase 2 SDP. This training

provided the necessary skills to conduct a Phase 2 SDP evaluation using the SDP notebooks;

however, this training was attended on an as-available basis and therefore not all inspectors

have received this training. The computer-based SDP exercise was completed by all

inspectors.

In addition to SDP training, the NRC offers 13 courses on reactor PRA and risk analysis. All

qualified inspection staff are required to complete the PRA Technology and Regulatory

Perspectives Course (P-111). Many managers have completed the PRA for Technical

Managers Course (P-107). Also, several inspectors, project engineers and/or Branch Chiefs, in

each Region have completed the series of advanced risk training courses. 

Most of the inspectors interviewed stated that they have not been provided adequate Phase 2

SDP training to be proficient at using the notebooks. They stated that the infrequent uses of

the Phase 2 SDP made it difficult to retain the skills needed to efficiently implement Phase 2

SDP. The inspectors also indicated that the SRAs were proficient at implementing the SDP and

were routinely used to validate that the Phase 2 notebooks were properly completed. While

many of the inspectors interviewed indicated that they did not attend the recent Phase 2 training

on the notebooks, those who attended this training found it beneficial. Nearly all inspectors

interviewed expressed a desire to receive periodic refresher training on Phase 2 of the SDP. 

Those individuals who had completed the advanced risk training found the training informative

and believed that it improved their ability to implement the SDP.

The guidance for implementing all Phases of the SDP is documented in IMC 0609. On

March 18, 2002, the guidance for conducting the Phase 2 and 3 SDP (IMC 0609, Appendix A,

Attachment 1) was significantly modified. The general consensus among NRC interviewees

was that the Phase 2 analysis guidance was complex, but adequate. 

The DPV and DPO review panels recommended that the guidance for Phase 3 SDP analyses

be enhanced. Additionally, the OIG report recommended that the guidance for using the

licensee’s PRA in Phase 3 analyses be improved. Based on the Task Group’s interviews, many
-33-

of the users of the Phase 3 SDP indicated that the guidance could be improved; however, no

specific areas for supplementing the current guidance were provided. The SDP Task Action

Plan Objective 3.6 includes an action to enhance the guidance provided for Phase 3 analyses. 

3.5.2 Conclusions

(1) Several different SDP training courses and seminars were provided to inspectors,

however, many inspectors felt that the training could have been better and that it did not

make them proficient in use of the Phase 2 tool. Although training was a factor in

inspector proficiency, the Task Group concluded that the complexity of the Phase 2

notebooks, the unavailability of benchmarked notebooks, and the infrequent use of the

Phase 2 process were the main hindrances to inspectors’ ability to use the Phase 2

SDP. Inspectors routinely used the SRAs as a resource to validate Phase 2 analyses

and, with the assistance of the SRAs, the SDP was implemented successfully. As noted

in Section 3.1, the Task Group believes that simplifying the Phase 2 SDP will alleviate

the need for inspectors’ current reliance on the SRAs to conduct the Phase 2 review. 

The simplified process is also expected to involve minimal training for inspectors.

(2) A variety of NRC training courses and seminars were administered over a relatively

short time frame and a number of training courses have been established to enhance

the knowledge of NRC staff and managers in the area of risk. Also, several staff in

each Region have participated in the advanced risk training course series, but there has

been no significant change in their job task assignments to take advantage of their

additional expertise in performing SDP evaluations. Although the general consensus

from the Task Group’s interviews of internal and external stakeholders was that the

staff's knowledge and ability to communicate risk insights has improved due, in part, to

staff training, many commented on areas for improvement. Some of the comments

centered on the lack of an integrated concept of the knowledge, skills, and abilities

(KSAs), and the associated training, required for staff and managers to implement the

SDP. This prompted the Task Group to recognize that a systematic assessment of

agency training in the area of risk has not been conducted. Such an assessment would

consider the ROP requirements, identify the positions that require risk KSAs (e.g.,

inspectors, Regional Branch Chiefs, senior managers, etc.), identify those KSAs, and

develop and/or identify the type and frequency of training required to establish and

maintain the KSAs.

(3) Although the current Phase 2 SDP guidance is complex, it was deemed adequate to

implement the current Phase 2 worksheets. Adoption of the Task Group’s

recommended approach to enhance the Phase 2 process, as discussed in Section

3.1.3(2), will prompt the development of more user-friendly implementation guidance. 

(4) The actions and schedule in the SDP Task Action Plan to enhance the current Phase 3

guidance are appropriate. The Task Group noted that while there have been several

recommendations for enhancing the Phase 3 guidance, no examples were cited where

the limited guidance has resulted in an unsatisfactory outcome (i.e., inadequate or poor

quality Phase 3 analysis).
-34-

3.5.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should encourage the Regions to hold refresher training on the Phase 2 SDP at

least annually. This training should be led by a Regional SRA and can also be used to

convey best practices and give feedback on common questions and problem areas. 

(The need for periodic Phase 2 SDP refresher training should be significantly diminished

with the use of the enhanced pre-solved SDP tables discussed in Section 3.1.1.3.)

(2) NRR should implement SDP Task Action Plan Objective 3.6, as planned and scheduled

to enhance the guidance for conducting Phase 3 analyses. 

(3) NRR should conduct a systematic assessment of training in the area of risk, with a

particular focus on identifying and advancing the knowledge, skills, and abilities (KSAs)

for implementing the SDP. 

3.6 Significance Determination Process Benchmarking and the Need for Peer Review

Criteria

The purpose of the Task Group’s review in this area was to address the recommendations in

the DPO Ad Hoc Panel report regarding the effectiveness of staff’s SDP benchmarking effort. 

The DPO Panel recommended that: (1) criteria for benchmarking the SDP be subjected to peer

review; (2) individuals conducting benchmarking should have statistical and PRA expertise;

(3) sequences and cutsets within licensee and NRC plant-specific PRA models should be

validated during the benchmarking process; (4) benchmarking should include some

independent review of the licensee’s PRA; (5) benchmarking should include a comparative

analysis of competing tools using standard problems; and (6) benchmarking process should be

continuous to capture plant and PRA model changes.

3.6.1 Observations

Following the first year of ROP implementation, the staff recognized that the quality of the

Revision 0, Phase 2 SDP notebooks was not adequate to assess the significance of inspection

findings. NRR initiated a project to enhance the quality of the notebooks by comparing the

results of the notebooks with each licensee’s plant-specific probabilistic risk assessments

(PRAs) (benchmarking). The benchmarking effort, which began in the Spring 2001, is

conducted for each plant during a several-day interface meeting between NRC staff and a

licensee’s PRA experts or representatives. As of November 2002, the staff had completed

benchmarking of approximately 50 percent of the notebooks. The current plan is to complete

benchmarking of the remaining notebooks by the end of the fiscal year (FY) 2003. The

technical assistance budget allocated for the effort is approximately $850K. However, there is

currently no plan established to periodically update the notebooks beyond the current

benchmarking effort. 

The benchmarking site visits are conducted in accordance with the guidance provided in

“Benchmarking Standard for SDP Phase Two Worksheets.” The Task Group determined from
-35-

its interviews that this guidance was developed by NRR staff and peer-reviewed by the

Regional SRAs prior to implementation. 

A regional SRA, NRC Headquarters SRA/Risk Analyst, and a PRA expert from Brookhaven

National Laboratory (BNL) typically conduct the benchmarking site visits. The benchmarking

site visits have normally also included a PRA expert from Idaho National Engineering and

Environmental Laboratory (INEEL), who is responsible for enhancing the agency’s plant specific

PRA models (SPAR). All individuals participating in the benchmarking trips have experience

and/or advanced training in PRA and the statistical methods used in PRAs.

During the benchmarking site visits, the results of selected unavailable equipment cases are

compared between the licensee’s PRA and the plant-specific notebook, areas of difference are

recognized, reasons for the differences are understood, and appropriate changes to the

notebooks are made. When the results of the notebook and the licensee’s PRA differ,

sequences and cutsets are typically reviewed to determine the cause(s) of the differences. 

However, when the results of the licensee’s PRA and the notebooks match, no additional

evaluation is performed to determine if the dominant sequences are consistent. At this point,

BNL prepares a benchmarking trip report which includes a comparison of the results from the

notebook with that of the licensee’s PRA. The report includes the benchmarked Revision 1

notebook and the enhancements that result from the benchmarking site visit.

The Task Group interviewed some of the NRC staff who have participated in the benchmarking

visits. They indicated that it was not practical to thoroughly review the quality of the licensee’s

PRA during the benchmarking visits. They also indicated that, in general, the quality of the

benchmarking visits was not strictly dependent on the quality of the licensee’s PRA because the

notebooks are simple, use generally conservative assumptions, use generic initiating event

frequencies and data, and employ simplified event trees that are generally consistent for each

major reactor design. In addition, the interviewees stated that the notebooks are developed

based on “construction rules” that are consistently applied for all plants, however, the Task

Group noted that the staff has not documented this information. 

The benchmarking process includes a comparison analysis, for selected hypothesized

equipment failures, between a plant-specific notebook, licensee’s PRA, and, sometimes, the

corresponding SPAR model. The comparison between the notebook and licensee’s PRA is

documented in the benchmarking report. Following the benchmarking site visit, INEEL has

made any required changes to the SPAR model and subsequently issued the revised (Revision

3) SPAR model. The documentation of each Revision 3 SPAR model contains data that can be

used for a comparison with the results of the corresponding notebook. However, routine

comparison of the benchmarking data between the notebooks and SPAR models has not yet

been implemented. 

The Task Group determined that the ROP feedback forms can be used to initiate a change to

correct errors in the current Phase 2 SDP Notebooks. However, there is currently no

established agency plan or process to periodically update the Phase 2 SDP notebooks. Some

of the interviewees suggested that the NRC should conduct a review or update of the

notebooks at least every 5 years. This update would be led by a Regional SRA with additional

support, as needed from a Headquarters Risk Analyst and/or contract PRA specialist. The

objective of the review would be to identify and resolve any significant differences between the

Phase 2 tool and the licensee's updated PRA. The periodic review should target 14 sites per
-36-

year beginning in 2008. In the intervening years, the Regions should use inspection insights

from Inspection Procedures (IP) 71111.12, "Biennial Periodic Maintenance Effectiveness

Evaluation Inspections," and IP 71111.17, "Permanent Plant Modification" to determine if any

significant plant or procedure modifications have occurred that may warrant an earlier review

and update of the SDP assessment tools. The ROP feedback system should be used to initiate

any changes needed to correct errors or omissions in the Phase 2 tool. 

The Task Group determined that the staff plans to complete the benchmarking effort to develop

the Revision 3 SPAR models by the end of FY 2003. The staff also intends to upgrade and

maintain the Revision 3 SPAR models, as needed, to support its use in risk-informed licensing

reviews, event risk assessments for event response, the ASP Program, as well as for Phase 3

SDP evaluations. The Task Group determined that there are no near-term plans for a major

revision to the SPAR models.

Nearly all the internal and external stakeholders who were interviewed about the benefit of

benchmarking the notebooks indicated that the effort significantly improved the quality of the

Phase 2 notebooks. There was a general consensus that the current Revision 0 Notebooks are

generally of inadequate quality to assess the risk significance of inspection findings and that the

benchmarked notebooks exhibited a significant improvement in the adequacy of risk estimates. 

This view was supported by a BNL report entitled, “Lessons Learned from Early Benchmarking

of Inspection Notebooks,” dated May 23, 2002. The report cited significant differences between

hypothetical findings evaluated using the Revision 0 notebooks and licensee PRAs, highlighting

the need for benchmarking of the notebooks. The benchmarking trip reports also indicated that

benchmarking results in a substantial improvement in the quality of the Notebooks. 

3.6.2 Conclusions

(1) Benchmarking significantly contributes to improving the quality of the Phase 2 SDP

notebooks and is also necessary for the development of the enhanced pre-solved tables

described in Section 3.1.1.3. This conclusion was based on the common support for

benchmarking from the stakeholders, the insights from the benchmarking trip reports

and the BNL report that cited lessons learned from comparing the results of hypothetical

findings evaluated using the Revision 0 notebooks and licensee PRAs.

(2) The staff has not routinely compared the results of the benchmarked SPAR models to

those of the benchmarked notebooks. Such a comparison would contribute to validating

the adequacy of the notebooks and promote independence from the licensee’s PRA . 

(3) Many of the recommendations contained in the DPO Ad Hoc Panel report have already

been included in the current benchmarking process. Specifically, each of the staff who

conducted the SDP benchmarking was qualified and the benchmarking guidance was

peer reviewed by SRAs prior to implementation of the benchmarking effort. The

benchmarking process uses generic information and applies construction rules to

maintain consistency and to make the notebooks independent of licensee’s PRA. 

(4) The “construction rules” for developing the notebooks have not been documented. A

clear discussion of how the notebooks are created would enhance stakeholder

knowledge and appreciation for the independence of the notebooks.
-37-

(5) There is currently no established agency plan or process to periodically update the

Phase 2 SDP notebooks. Because future licensee PRA changes and/or plant

modifications can affect the adequacy of the notebooks, a plan to maintain the Phase 2

tool current is warranted.

3.6.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should complete the benchmarking of the remaining Phase 2 SDP notebooks to

support the current Phase 2 SDP as well as the development of the enhanced presolved Phase 2 table recommended in Section 3.1.3(2).

(2) NRR should develop a plan to periodically review and update the Phase 2 SDP

assessment tools to address any licensee PRA changes and/or plant modifications.

(3) NRR and RES should implement a process to compare the results of the benchmarked

SPAR model with the results of the benchmarked Phase 2 notebooks. 

(4) NRR should document the “construction rules” for the Phase 2 SDP notebooks. 

Consideration should be given to including (or referencing) the construction rules in the

proposed SDP Basis Document.

3.7 Use of Uncertainty in the Significance Determination Process

The purpose of the Task Group’s review in this area was to address the recommendations in

the DPO Ad Hoc Panel report regarding the use of uncertainty analysis in the staff’s

assessment of inspection findings. The Panel’s report included the observation that “it is

difficult to ascribe meaning to the point estimates that are derived using the plant-specific

notebooks. With order of magnitude approximations for the unavailability of systems and

components, and the large uncertainties associated with the unreliability of human actions,

questions regarding the efficacy of comparing delta core damage frequencies on the order of

1E-6 and 1E-5 naturally arise.” The Task Group’s recommendations in this area were

influenced by the insights obtained from interviews of internal and external stakeholders.

3.7.1 Observations

It has become customary to discuss uncertainty in PRA analyses as originating in one of three

ways according to RG 1.174:

(1) parameter uncertainty - which recognizes that the value of such parameters as initiating

event frequencies, component failure probabilities or failure rates, human error

probabilities cannot be known with precision

(2) model uncertainty - which recognizes that the relationship between the real plant and its

mathematical representation is not known with certainty
-38-

(3) completeness uncertainty - which recognizes that the model may not model every

aspect, either because it may relate to an unknown issue, or because models do not

exist for some aspects 

PRAs are capable of addressing parameter uncertainty explicitly. Model uncertainties that

underlie the development of the base PRA model are typically handled by making assumptions

which become part of the definition of the PRA model. When there are alternate assumptions

that are equally plausible, sensitivity analyses may be conducted, using the alternate

assumptions, to assess the robustness of the results with respect to those assumptions. 

Completeness uncertainties cannot be addressed analytically since, by definition, they result

from contributors that are missing from the model. The traditional engineering approaches

adopted to deal with uncertainties that cannot be readily analyzed are the establishment of

defense-in depth, and of adequate safety margins. For a more complete discussion of

uncertainty analysis see RG 1.174. 

There is no explicit treatment of uncertainty in the SDP. As described in SECY-99-007A, the

Phase 2 SDP was designed to provide a generally conservative estimate of the risk significance

of inspection findings. The output of the SDP would then be allocated to a response band,

each a decade in width, with boundaries based on the values used in the acceptance guidelines

of RG 1.174. The current SDP notebooks are constructed on the basis of accident sequences

resulting from initiating events at full power from internal initiating events. Within that context,

the use of the notebooks results in generally conservative risk estimates. However, this is an

incomplete assessment of the risk significance since many of the potentially significant

contributors to risk, including external events (i.e., initiating events resulting from internal fires,

internal floods, seismic events, high winds, etc.,) and events during low power and shutdown

modes of operation are not addressed. The significance of these contributors is plant-specific,

but internal fires and shutdown modes of operation have been found to be significant in many

PRA studies. Seismic and high wind contributors may also be significant for plants in certain

geographic regions. In reaching a conclusion about the risk significance of a finding, these

contributions should be taken into account.

Many of the external and internal interviewees expressed concerns that the use of the SDP may

convey a false impression of the precision of the PRA derived determination of the risk

significance of a finding. Their concern centered on the use of point estimates and recognition

that the estimates of CDF obtained from PRAs typically have a significant amount of

uncertainty. Some of the concern was associated with the ROP itself in that it requires a

comparison of point estimates to discrete response boundary. This is in contrast to the bands

used in Management Directive 8.3, “NRC Incident Investigation Program,” that overlap by an

order of magnitude, allowing other deterministic factors to be considered in deciding the

appropriate response. To some interviewees, this contributes to the belief that the use of the

SDP is risk-based vs risk-informed. (See Section 3.1.2.)

Based on insights from the Task Group’s interviews, one of the most significant sources of

uncertainty involves the characterization of the impact of the inspection finding for input to the

SDP. This typically involves making assumptions about the impact of the inspection finding on

the reliability or unavailability of an SSC or its impact on the initiating event frequency. The

view that this is a significant source of uncertainty is substantiated by the staff’s experience in

its assessment of many contentious issues, including the findings related to the Indian Point

Unit 2 steam generator tube leak, the D.C. Cook service water system fouling, and the
-39-

degraded reactor vessel head at Davis Besse. For these findings, the complicating factors

were independent of the risk assessment tool (i.e., Phase 2 or Phase 3 tools). It is the

differences between the staff’s engineering judgement and that of the licensees about the

assumed impact of the degraded condition that drove the differences in the assessment of

safety significance and, hence, generated significant uncertainty about the safety significance. 

3.7.2 Conclusions

(1) In light of the uncertainty arising from incompleteness and model uncertainty, parameter

uncertainty is a relatively minor issue. However, there are some conclusions that can be

drawn in relation to parameter uncertainty:

(a) Since the thresholds between response bands were based on the acceptance

guidelines of RG 1.174, it is appropriate to compare mean values of CDF with

those thresholds. The DPO Ad Hoc Panel’s concerns with the use of point

estimates can be addressed only if the point estimates are close enough to

mean values for the stated purpose of the SDP.

(b) There are several issues that determine whether the Phase 2 evaluation can be

regarded as close enough to a mean value to be adequate for the stated

purpose of the SDP. First, the values used in the notebooks for system and train

unavailabilites and human error probabilities are point estimates that are

intended to be somewhat conservative with respect to what would be found in a

typical PRA. Typical PRA values would generally represent mean values or, in

the case of system or train unavailabilities, point estimates based on using mean

values for all the input parameters. The common cause failure (CCF)

contributions are included conservatively - redundant systems are given a credit

of 1E-3, while a single train gets 1E-2, equating to a (conservative) Beta-factor

of 0.1. The principal concern with not performing a complete parametric

uncertainty analysis is that the “state of knowledge” correlation, discussed in RG

1.174, cannot be taken into account. Taking this correlation into account can

alter the mean overall CDF by up to a factor of 2 in some cases. In the context

of the SDP, it is more of a concern if the CDF resulting from the inspection

finding is dominated by cutsets that involve several components whose failure

probabilities are estimated based on the same parameter and that parameter

value is highly uncertain. There are relatively few situations where this is

expected to be significant. 

(c) When inspection findings require a Phase 3 evaluation, more complex models

(e.g., SPAR and/or insights from a licensee’s PRA model) may be used. In rare

cases, particularly if the contributors to CDF are such that the state-ofknowledge correlation can have an impact on the evaluation of CDF, parameter

uncertainty can and should be propagated analytically to derive the real mean

value.

(2) It is important to consider the contribution from external events, and low power and

shutdown events when making a final assessment of risk significance of a finding

(completeness). However, simplified methods similar to the notebooks have not yet

been developed. Guidance is needed on how the staff should engage in determining
-40-

the significance of a finding associated with the prevention or mitigation of accidents

resulting from external events (see Section 3.4.2(4)), or during low power and shutdown. 

NRR should continue its initiatives to develop better tools and guidance in the area of

low power and shutdown as outlined in SDP Improvement Task Action Plan (Task 3,

Objectives 3.3e,).

(3) Model uncertainties associated with the characterization of the impact of the inspection

finding on the function of an SSC represent the greatest source of uncertainty. This was

particularly clear in the evaluation of several findings that involved a degraded SSC. 

Relating this degradation to component reliability or the frequency of a failure of a

pressure or fission-product boundary can represent a considerable challenge to the staff

and the uncertainty is strongly tied to the analytical assumptions, which can vary

significantly. When there is substantial disagreement between analysts, the ‘mean’

value of the assessments has no real significance, and should not be used to determine

the color associated with the finding. 

(4) The SDP does not provide for explicit consideration of traditional engineering analysis

(deterministic) inputs. This is in contrast to RG 1.174 which uses considerations of

defense-in-depth and safety margins to assess whether license amendments should be

granted. In that context, defense-in-depth and safety margins are seen as mechanisms

to account for incompleteness.

3.7.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should develop guidance to allow the staff to recognize situations where the “state

of knowledge” correlation, which is described in RG 1.174, might warrant a Phase 3

analysis.

(2) NRR should modify IMC 0609 guidance to explicitly indicate that traditional engineering

analysis considerations (e.g., reduction of safety margin, or significant loss of defensein- depth) should be used to determine an appropriate color to associate with findings

where the uncertainty in the risk evaluation arising from the characterization of the

impact of the inspection finding is large enough that the color is indeterminate on the

basis of the risk analysis. This guidance should be such that it promotes consistency in

the staff’s use of such analyses and should only be applied to those findings where the

uncertainty is significant (e.g., when alternate assumptions yield results which vary over

more than two orders of magnitude). 

3.8 Evaluation of How Other Inputs are Factored Into the Significance Determination

Process

The purpose of the Task Group’s review in this area was to respond the Charter directive that

the Task Group evaluate whether the ROP (SDP) appropriately allows for consideration of other

inputs independent of a best-estimate risk assessment. The Task Group’s recommendations in

this area were influenced by its review of IMC 0609 and the insights obtained from interviews of

internal and external stakeholders.
-41-

3.8.1 Observations

The Revised Oversight Program focuses NRC resources in areas where there are existing

safety significant performance problems. As a result, the process is largely indicative, in that it

directs resources based on a licensee’s past performance. However, the ROP also includes a

predictive performance element to assess licensee performance in “other” (cross-cutting) areas,

which are recognized in the ROP as human performance, problem identification and resolution,

and safety conscious work environment.

The ROP was developed based on the presumption that deficiencies in cross-cutting areas

would be revealed by safety significant findings and PIs. Accordingly, there must be a

significant level of agency concern to believe there is a substantive cross-cutting issue at the

plant. Substantive cross-cutting issues are identified in the NRC’s Mid-Cycle or Annual

Assessment letters when there is evidence of a significant number of current inspection findings

with common causal factors, in the area of human performance, problem identification and

resolution (PI&R), or safety conscious work environment. There is no additional NRC

engagement (i.e. inspections, meetings, correspondence, etc.) with licensees beyond the

baseline program as a result of identifying a substantive cross-cutting issue. The licensee’s

progress in resolving the cross-cutting issue is documented in subsequent assessment letters.

Most of the NRC managers interviewed by the Task Group indicated that they believed that the

ROP/SDP would be enhanced by consideration of “other inputs” in the significance

determination process. However, nearly all managers cited challenges that would be posed by

any effort to include “other inputs” in the ROP in an objective and predictable manner. The

“other inputs” that could be considered for including varied considerably and included such

areas as the timeliness of issue identification, the adequacy of corrective actions, the

effectiveness of extent of condition reviews, programmatic breakdowns, organizational

effectiveness, and design control.

Based on insights obtained during the interviews, the Task Group developed several

approaches for incorporating “other inputs” into the ROP/SDP. A brief description of the

approaches are as follows:

(1) The Significance and Enforcement Review Panel (SERP) guidance could be modified to

allow Senior NRC Managers to modify the significance determination (color) of

inspection findings based on “other inputs.” This process would be similar to that used

in the enforcement process for escalating or mitigating civil penalties. The “other inputs”

mitigators and escalators could include items such as: (1) who identifies the issue; (2)

timeliness of problem identification; (3) completeness and timeliness of corrective

actions; and (4) extent of condition. Guidelines and/or standards to facilitate an

objective and predictable consideration of “other inputs” would also need to be

developed.

(2) The ROP could be modified such that “other inputs” could be considered as part of the

ROP annual assessment process. The guidance in IMC 0305 could be modified to

redirect additional NRC resources into areas where substantive “other inputs” have

resulted in degraded performance. 
-42-

(3) The existing Action Matrix deviation process could be used to include “other inputs”

when the ROP Action Matrix does not provide an appropriate agency response, as

determined by NRC management. The Task Group noted that guidance for action

matrix deviations is provided in IMC 0305, “Operating Reactor Assessment Program,”

Section 06.06f.

Most licensee interviewees were less receptive to including other inputs in the ROP. Some

representatives indicated that this would represent a return to the Systematic Assessment of

License Performance (SALP) system. Some licensee managers also observed that the

Institute of Nuclear Power Operations (INPO) has a program to assess “soft issues” such as

organizational effectiveness and human performance. They indicated that either the NRC could

allow INPO to monitor this area or the NRC could assess the information collected by INPO.

The Task Group asked interviewees to provide examples of where the current ROP provisions

regarding the use of "other inputs" may have had an adverse impact on the agency's oversight

responsibilities. There were no examples cited where the current provisions for the use of

"other inputs" in the ROP had resulted in the agency not allocating resources for a licensee

performance problem in a timely manner. Additionally, the Task Group noted that, as

discussed in SECY-02-0058, "Results of the Industry Trends Program for Operating Power

Reactors and Status of Ongoing Development," dated April 1, 2002, there have been no

statistically significant adverse industry trends in safety performance through FY 2001.

NRC Inspection Manual Chapter 0305, Section 06.06.h, “Operating Reactor Assessment

Program” provides guidance on how substantive cross-cutting issues are addressed in the

ROP. This guidance requires multiple findings documented in the assessment period with

causal factors that support the cross-cutting issue. The basis for the substantive cross-cutting

issue is then documented in the mid-cycle or annual assessment letter. The guidance does not

recommend redirecting NRC resources in response to the identification of a substantive

cross-cutting issue nor does it address the expected licensee response. However, it does allow

the staff to consider increasing the frequency of the PI&R inspection if the plant enters the

degraded cornerstone column of the action matrix. Many NRC Regional managers indicated

that the current guidance for defining a substantive cross-cutting issue appears to be applied

inconsistently across the Regions. 

3.8.2 Conclusions

(1) There were no examples where the current provisions for treating “other inputs” in the

ROP had limited the staff’s ability to focus resources in appropriate areas. In addition,

the ROP currently allows for deviations from the action matrix when deemed appropriate

by senior NRC management; the guidance does not preclude use of “other inputs” as

factors in the justification for the deviation.

(2) Consideration of “other inputs” in the SDP would require the development of guidelines

and/or standards to support the staff use of them in objective and predictable manner.

Although feasible, the cost and time to develop such guidelines and/or standards would

likely be substantial.
-43-

(3) The existing guidance in IMC 0305 regarding deviations from the Action Matrix can be

employed, where appropriate, to consider “other inputs” in determining agency

response.

(4) The guidance in IMC 0305 for the identification of significant cross-cutting issues has

not been consistently implemented by the Regions. The current guidance is vague on

the expected NRC and licensee response to a substantive cross-cutting issue.

3.8.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) The NRC should not modify the SDP to include consideration of “other inputs.” This

decision should be re-evaluated as part of the annual ROP self assessment to

determine if licensee performance indicates that different treatment of “other inputs” in

the ROP is warranted. 

(2) NRR should enhance its oversight of the implementation of the guidance in IMC 0305,

Section 06.06.h, for the identification of substantive cross cutting issues to promote

consistency application by the Regions. 

(3) NRR should revise the ROP guidance to include consideration of a response to the

identification of a substantive cross-cutting issue (only when there is at least one White

PI or finding). This response could include a redirection of inspection resources,

management meetings, and/or a docketed licensee response describing actions

planned or taken to address the cross-cutting issue. This guidance should also include

a description of how the NRC will close a substantive cross-cutting issue.

(4) NRR should supplement the guidance in IMC 0305, Section 06.06.f, with additional

guidance that lists the type of information that should be included in a Region’s request

to deviate from the Action Matrix (e.g., synopsis of the findings affecting the licensee’s

performance; the actions (column) stipulated by the Action Matrix; the Region’s rationale

or considerations for taking action different from that stipulated in the Action Matrix, etc.)

3.9 Significance Determination Process Timeliness

The purpose of the Task Group’s review in this area was to respond to recommendations in the 

OIG audit report (OIG-02-A-15) regarding the timeliness of SDP determinations. The OIG

report indicated that SDP evaluations that result in greater-than-green findings are not

processed in a timely manner and that the current ROP metrics do not capture the total time of

inspection finding evaluations. The Task Group’s recommendations in this area were

influenced by its review of IMC 0609 and the insights obtained from interviews of internal and

external stakeholders.

3.9.1 Observations

IMC 0609, Exhibit 5 contains “Suggested Timeliness Criteria for SDP Inspection Findings.” The

criteria requires the completion of SDP characterizations within 120 days of the first inspection
-44-

exit meeting or within 90 days of the issuance of the inspection report. A Staff Requirements

Memorandum (SRM) dated August 2, 2001, directs the staff to meet the timeliness goals stated

in IMC 0609 for 100% of the inspection findings.

The official data base for tracking the timeliness of the SDP is the Enforcement Action Tracking

System (EATS). This data base, which is maintained by staff in the Office of Enforcement OE,

provides a comprehensive list of the timeliness milestones for those inspection findings that are

potentially greater-than-green, (e.g., inspection exit date, SERP date, Choice Letter issuance

date, regulatory conference date, etc.). In addition, the Inspection Programs Branch (IIPB) in

NRR, tracks the timeliness of inspection findings from the issuance of the inspection report until

issuance of the final significance determination letter. The Task Group determined (based on a

review of the NRR tracking system, the data contained in the Reactor Program System, the

monthly OE input to the Congressional Report and the EATS data) that the NRC inspection

program yielded approximately 600 to 800 inspection findings per year. Of these 600 to 800

findings, approximately 20 to 25 across all cornerstones were preliminarily characterized as

greater-than-green, and of these, about 10 to14 did not meet the timeliness goals. Thus, the

data indicates that less than about three percent of the total number of inspection findings did

not meet the ROP timeliness goals. 

Almost half of the findings that have not met the ROP timeliness goals, have involved findings

of potentially higher risk significance. In addition, some of the findings that did not meet the

timeliness goals involved fire protection and shutdown conditions, where there are recognized

inadequacies in the staff’s risk evaluation guidance (see Section 3.7). Most NRC interviewees

indicated that the factors that have most significantly impacted SDP timeliness, aside from the

aforementioned areas where guidance is lacking, include the development of human reliability

assumptions and engineering assumptions regarding the impact of a degraded condition on the

ability of an SSC to satisfy its safety function. Differences among NRC staff and between the

staff and licensees regarding such assumptions has significantly influenced timeliness. 

Both internal and external stakeholders believe that SDP timeliness is a major factor in

promoting public confidence in the ROP. External stakeholders indicated that public confidence

is particularly affected when findings that appear to have a greater impact on risk are untimely. 

In addition, several external stakeholders indicated that they believe that NRC management

should devote more attention to assuring the timeliness of SDP evaluations of greater-thangreen findings. Both internal and external stakeholders stated that they favor the use of a

“greater-than-green” characterization for preliminary safety significance determination (Choice)

letters. Many of the interviewees believe that this characterization will alleviate the staff’s

perceived need for precision and allow for greater use of reasonable assumptions in the

preliminary safety significance determinations. They indicated that the timeliness of the SDP,

including the regulatory conferences and the staff’s decisions on the final significance

determinations, should improve with the issuance of more timely Choice letters. 

Based on feedback from NRC interviewees, many were not aware of the current ROP

timeliness goals and most assumed that the goals were the same as the previous enforcement

action goals. The OE operating plan measures enforcement timeliness for non-investigation

violations (i.e., cases that do not involve an OI investigation) from the date of the exit meeting. 

The OE operating plan metric requires 100% completion within 180 days, with an average

completion time of 120 days. Many NRC interviewee’s also indicated that the timeliness of the

SDP has often been hindered by protracted deliberations with licensees who routinely challenge
-45-

the staff on potentially greater-than-green findings. As discussed in Section 3.2.1, this was an

unanticipated result from the NRC’s use of risk assessment tools to evaluate inspection

findings. 

Interviews with several licensee stakeholders indicated that the primary factors that cause

licensees to frequently dispute potential greater-than-green findings are the negative reaction

within the financial community, the potential adverse effect on bond ratings, and public criticism

of licensee performance. While they acknowledged that these factors were beyond the

agency’s control, the licensee interviewees noted other contributing factors for why licensee’s

challenge potentially greater-than-green findings, including the potential for incomplete

characterization of the significance of the findings in NRC inspection reports, letters, and press

releases. 

With respect to NRC press releases, most NRC interviewees observed that the policy for

issuing a press release for each White finding was misaligned with the significance of many

such findings. Some recommended that the issuance of press release be linked to a licensee's

performance as indicated by the Action Matrix rather than the licensee's performance reflected

in a single White finding. Recently, the Office of Public Affairs (OPA) revised its policy

regarding issuance of press releases for White findings. The revised policy requires the

issuance of a press release for a single White finding only for those plants that the Director

OPA has determined to have a significant amount of public interest. In addition, a press

release will be issued for a plant that has two White findings, even if the findings are in different

cornerstones. OPA plans to continue its practice of issuing a press release for all public

regulatory conferences and regulatory performance meetings.

The Task Group determined that NRR recently adopted an Operating Plan metric which allows

for a targeted improvement in SDP timeliness. In FY03, the metric requires 75% of greaterthan-green findings to meet the ROP timeliness goals. The metric will be adjusted upwards by

5% until a 90% target is reached in FY06. 

3.9.2 Conclusions

(1) The SDP was designed to generate a reasonably accurate estimate of the risk

significance of an inspection finding to facilitate timely agency decisions regarding the

allocation of resources. As a result, the Task Group determined that the ROP

timeliness goal of completing the SDP within 120 days of the first inspection exit

meeting and within 90 days of the issuance of the inspection report is reasonable. 

However, the Task Group recognized that some findings involve complex engineering

issues and may require additional time to adequately assess the safety significance. 

The NRR Operating Plan metric for greater-than-green findings should provide an

appropriate management tool to address the timeliness challenge for the relatively few

findings expected to involve such safety significance.

(2) Significant progress has been made in tracking the timeliness of the issues that are

greater than Green and accountability within the responsible offices has been

established by identifying responsible managers in the IIPB SDP timeliness matrix. In

addition, although the number of untimely SDP evaluations is relatively small, additional

management attention is warranted to further improve SDP timeliness. Many of the

NRC interviewees indicated that improved management decisiveness in determining the
-46-

engineering assumptions used in risk evaluations would contribute to the timeliness of

the SDP. The oversight of findings through the SDP, beginning with the initial

identification of issues of potential greater-than-green safety significance, through the

exit meeting, the SERP, the initial documentation of the finding, and the follow-on public

deliberations with the licensees warrants closer management attention to promote

timeliness. 

(3) The timeliness goals for characterizing inspection findings in the SDP process were not

well understood by all members of the staff. The Task Group also noted that it was

likely that licensee representatives were similarly unclear on the agency’s SDP

timeliness goals.

(4) The issuance of a press release for a White finding may contribute to the licensees’

tendency to dispute preliminary White findings. A White finding, which is indicative of a

licensee performance deficiency, represents an unintended (low to moderate) increase

in risk and results in actions specified by the Regulatory Response Column of the Action

Matrix. However, because a plant in the Regulatory Response Column has still “fully

met the related cornerstone objectives,” the issuance of a press release for a single

White finding may, in many cases, result in unwarranted emphasis on a particular

finding. The Task Group concluded that the recent revision to the OPA policy, which

requires issuance of a press release for a White findings at plants with significant public

interest and to announce regulatory conferences and regulatory performance meetings,

was an improvement. 

3.9.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should reinforce the current SDP timeliness goals through management

interactions, identifying responsible mangers, and establishing accountability for the

resolution of overdue significance determinations. The NRR SDP timeliness metrics

should be incorporated into the Regional Operating Plans. 

(2) NRR should communicate the agency's SDP timeliness goals to licensees (e.g., include

text in Choice Letters to inform licensees of NRC's plan to reach a final significance

determination within 90 days; discuss the timeliness goal during regulatory conferences;

discuss SDP timeliness, including the timeliness goal during the annual Regulatory

Information Conference, etc.)

(3) NRR should rectify the difference between the metrics identified in the NRR Operating

Plan (i.e., 75% timeliness goal in FY03 increasing to 90% timeliness goal by FY06) with

the goals stated in IMC 0609 and reiterated by the August 2, 2001, SRM (i.e., 100%

timeliness goal). 

(4) NRR should change the guidance in IMC 0609 to allow the Regions the option to use a

"preliminary greater-than-very low safety significance" characterization for preliminary

safety significance determination (Choice) letters when there is sufficient uncertainty

regarding the preliminary risk estimate. 
-47-

(5) OPA should further modify its policy for issuing press releases for ROP findings to link

the issuance of press release to a finding(s) that results in a Degraded Cornerstone. 

This modification will better align the emphasis that results from the issuance of a press

release with the "moderate degradation in safety performance" associated with a

Degraded Cornerstone (i.e., two White inputs (findings and/or Performance Indicators),

one Yellow input, or three White inputs in the same Strategic Performance Area). 

(6) NRR should modify IMC 0305 to reference the OPA policy for issuing press releases for

ROP inspection findings.

3.10 Reactor Oversight Process Web Site Improvements

The purpose of the Task Group’s review in this area was to respond to OIG recommendations

regarding the ROP web site. To respond to this issue, the Task Group interviewed the NRR

staff and managers who are responsible for the content and maintenance of the web site.

3.10.1 Observations

Section D, “Improve the Web Site to More Fully Inform the Public,” of the OIG Audit Report,

(OIG-02-A-15) provided the following recommendations to enhance the ROP web site: 

(1) (Recommendation #7) Revise the web page to provide a link from the findings summary

web pages to the documents that support any changes from preliminary inspection

report significance determinations.

(2) (Recommendation #8) Expand the web page to provide complete access to inspection

report results, not just those that identify operational deficiencies.

(3) (Recommendation #9) Expand the web page to display all significant findings colors in a

cornerstone. 

(4) (Recommendation #10) Revise the web site to fully describe licensee corrective action

related to each finding.

The Task Group reviewed a draft response to the OIG report and discussed the responses with

cognizant NRR staff. The draft response to the OIG audit report findings was completed by

NRR. The responses to the four OIG recommendations provided in the draft NRR response

are as follows: 

(1) (Recommendation #7) The NRR draft response letter agrees with this finding and

indicates that the web page would be revised by March 2003. 

(2) (Recommendation #8) The web page had been revised to incorporate this comment. 

The current summary of findings provides a link to the inspection report that

documented the finding. The web site was accessed and it was validated by the Task

Group that this action is complete.

(3) (Recommendation #9) The plant findings summary web page does not indicate the

number of White findings (if any) that are present in a cornerstone with a Yellow finding. 
-48-

A cornerstone with a only Yellow findings or White findings is correctly displayed on the

summary page. In the case where a Yellow and White findings exist in the same

cornerstone, only the Yellow (most significant) finding is displayed. A web site user

could easily determine the number of White findings in the cornerstone with a Yellow

finding by clicking into a subsequent web page that provides a summary of all findings in

the cornerstone. The Action Matrix column designation for a single Yellow finding or a

Yellow finding with multiple White findings is identical. Therefore, the current display

would correctly indicate that the cornerstone was degraded. There will be very few

instances where there will be simultaneous White and Yellow findings in the same

cornerstone (currently there are none). The Task Group determined that although it

would be feasible to modify the web site to accommodate this recommendation, it would

also be complex and costly. 

(4) (Recommendation #10) The IMC 0612 does not provide guidance to describe corrective

actions in the summary of findings entry. The Task Group verified that IMC 0612,

Section 05.30, “Summary of Findings” does not mention including corrective actions

taken by the licensee for all items included in the summary of findings. The report

example, Appendix D of IMC 0612 provides an example of a summary of finding entry

where the corrective actions are not mentioned. The NRR staff responsible for IMC

0612 indicated that green findings are by definition in the licensees response band and

licensee’s are only required to issue a corrective action report for green findings. The

corrective actions taken for greater-than-green findings are thoroughly discussed in the

supplemental inspection report. Therefore, he indicated that it would not be beneficial to

include a detailed discussion of corrective actions in the summary of findings.

3.10.2 Conclusions

(1) The proposed change for OIG Report recommendation #7, to improve the links for all

relevant documents for greater-than-green findings, is appropriate. The proposed

completion date for implementing this change is reasonable.

(2) The OIG Report recommendation #8, to link the summary of findings to the inspection

report, had been implemented.

(3) The OIG Report recommendation #9 would not substantially improve the quality of the

information provided on the ROP web page and that there were currently no plans to

implement the recommendation. Users are provided ready access to all findings in a

cornerstone by hyper-linking to the next web page, and the difficulty and costs of

implementing this change appears to exceed the benefit. 

(4) The OIG Report recommendation #10, to revise IMC 0612 to include a brief summary of

corrective actions in the inspection report summary of findings, is reasonable. A brief

description of the corrective actions taken by the licensee to restore compliance with

NRC regulations should be included in the summary of findings.

3.10.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:
-49-

(1) NRR should change IMC 0612 to require a brief description in the inspection report

summary of findings of corrective actions taken by the licensee to restore compliance

with NRC regulations, where applicable. (OIG Report recommendation #10)

(2) The EDO should inform the OIG that the ROP web site will not be revised as described

in recommendation #9. Users currently have ready access to all findings in a

cornerstone by hyper-linking to the next web page and the OIGs’ recommendation

would not substantially improve the quality of the information available to users. The

difficulty and costs of implementing this change appears to exceed the benefit. (OIG

Report recommendation #9)

3.11 Evaluation of Other Reactor Oversight Process and Significance Determination

Process Issues

As allowed by the Charter, the Task Group reviewed several other topics related to the SDP to

determine if any process changes were warranted. Based on insights from interviews with

internal and external stakeholders, the Task Group identified the following three topics for

separate discussion: ASP Program and SDP; Limitations on the use of PRA methods in the

SDP; and Performance of ROP Action Matrix.

3.11.1 Accident Sequence Precursor Program and Significance Determination Process 

In a technical assistance user-need memorandum dated January 31, 2002, (NRR-2002-004),

NRR requested RES to validate the output of the SDP for all greater-than-green findings by

comparing the significance determinations with preliminary ASP results. Based on the

feedback from of the RES validation reviews, many NRC interviewees expressed concern about

the impact that potential differences between an ASP and SDP risk assessment might have on

public confidence and questioned the efficiency and effectiveness of maintaining both

processes. Although the Task Group did not perform an extensive review of this area, the

following insights were gained from a review of ASP and SDP basis documents and interviews

with NRC staff and managers.

3.11.1.1 Observations

The ASP program provides a measure of how the agency meets two of its Strategic Plan

performance goals in the Reactor Safety Arena. These are: (1) no more than one event per

year identified as a significant accident precursor (i.e., Conditional Core Damage Probability

(CCDP) or importance > 1E-3) and (2) no statistically significant adverse industry trends in

safety performance (the Industry Trends Program is discussed in SECY 01-0111,

“Development of an Industry Trends Program for Operating Power Reactors.” ASP results are

used as input to an annual report to Congress on whether those goals are met. In this regard,

the ASP program is not constrained by the SDP timeliness goals, although final analyses are

usually completed within two years of an identified plant condition. This allows the ASP risk

analysts additional time to confirm the assumptions and inputs used in their risk evaluations. 

The ASP evaluates all potentially risk significant conditions and events that existed at the time

of the identified plant condition, which differs from the SDP which only evaluates inspection

issues that result from the licensee’s failure to meet a requirement or standard that have an

associated licensee performance deficiency (i.e., findings). 
-50-

The SDP employs three phases: an initial screening - Phase 1; an estimation of risk using a

simplified PRA tool - Phase 2; and an estimation of risk using more sophisticated PRA tools -

Phase 3. The SDP was intended for use by NRC inspectors, risk analysts and managers to

assess the safety significance of inspection findings and facilitate a timely decision regarding

the allocation of agency resources. Although the primary risk assessment tool that is used by

SRAs to perform the Phase 3 SDP, the SPAR model, is the same tool employed by risk

analysts for ASP analyses, the SDP was not intended to provide a precise measurement of the

risk of an inspection finding.

Some of the NRC interviewees indicated that it was inappropriate for the ASP program and

SDP to reach different risk significance conclusions for the same issue. They believed that

public confidence would be eroded if NRC were to issue an ASP and an SDP significance

determination (i.e., CDF) with conflicting results. In addition, some of the interviewees stated

that they believe that the purposes of ASP can be assumed by the SDP and that the ASP

program should be suspended or subsumed by the SDP. They considered the use of both

programs to conduct similar independent risk assessment of the same issue was not an

efficient use of agency resources.

However, other NRC interviewees stated that the ASP program and the SDP serve different

purposes and should continue to be implemented as independent programs. They indicated

that the verification of SDP analyses using ASP provides a positive benefit. Given the fact that

SDP was not designed nor intended to provide a precise risk determination, the role of the ASP

review was to provide an ongoing confirmation of the general assumption that SDP would

provide for reasonably accurate assessments of the risk of inspection findings.

Some interviewees suggested that better integration of RES staff responsible for ASP into the

development of preliminary significance determinations could enhance both the SDP

evaluations and the subsequent ASP analyses. However, given the current construct of the

ASP program, which includes multiple peer reviews by the licensees and NRC technical staff, it

is not practical to complete ASP analysis within the timeliness goals of the SDP. 

The Task Group determined that NRR had identified the need to improve the consistency

between ASP and SDP approaches, Objective 6.1 of the SDP Improvement Plan Initiative. 

This task involves the development of guidance to clearly delineate the role of RES in the SDP

and minimize the potential for unexpected or unreasonable differences in the results of the SDP

and ASP processes. The action is scheduled for completion by April 2003.

3.11.1.2 Conclusions

(1) The concern for the potential negative impact on public confidence from the issuance of

different NRC risk characterizations for the same issue has merit. Program

requirements and/or an RES/NRR office protocol should ensure reasonable consistency

in the output of the programs to promote public confidence. The Task Group noted that

RES, NRR and Regional management recognize the potential impact of different

ASP/SDP risk assessments, as evidenced by the significant resources that were

expended to develop a shared understanding of several recent preliminary ASP and

SDP analyses.
-51-

(2) The schedule to complete ASP evaluations currently precludes using ASP insights in the

SDP; however, many NRC interviewees believe that there are efficiencies that can be

gained through a better coordination and/or integration of these two programs. 

(3) The Task Group noted significant differences among many staff in their understanding

of the purpose and uses of the ASP. Efforts to enhance the staff’s knowledge of the

ASP would contribute to better coordination and integration of the ASP and the SDP.

3.11.1.3 Recommendations

(1) To address the issues discussed in this section of the report, the Task Group made the

following recommendation: NRR should broaden the NRR SDP Task Action Plan

Objective 6.1 to initiate a cooperative effort between NRR and RES to explore efficiency

and quality enhancements that would result in better coordination and/or integration of

these two programs. 

(2) NRR and RES should identify avenues to enhance the staff’s knowledge of the ASP

program, including adding a module to the P-111 course regarding the ASP program.

3.11.2 Objectivity of Probabilistic Risk Assessments 

During the Task Group’s interviews, some stakeholders expressed a concern that the objectivity

introduced by the use of PRA results had been overemphasized. Others expressed a related

concern about the quality of the PRA models used to determine the final risk significance. A

brief discussion of the staff’s insights in these areas follows.

3.11.2.1 Observations

Objectivity of Assessment of Safety Significance - One of the three objectives of the ROP is to

improve the objectivity of the reactor oversight process so that subjective decisions and

judgement are not central to the process. Based on insights from internal and external

stakeholders, the staff has made progress in achieving this important objective. An apparent

effect of using the SDP to advance this objective, however, has been the lack of recognition,

and perhaps undervaluation, of the importance of informed engineering judgement by the staff. 

The staff routinely uses engineering judgement in the identification of licensee performance

deficiencies, which allows inspection issues that result from the licensee’s failure to meet a

requirement or standard to be placed in context (i.e., a finding). Engineering judgement is also

routinely used by the staff in formulating its assumptions about whether or not a degraded SSC

would have performed its safety function given a specific demand. As noted in Section 3.7.1,

such assumptions are often the primary determinants of the safety significance of a finding and,

hence, the key to the uncertainty of the underlying risk estimate. While there is often a

subjective element in the determination of risk significance, the use of the PRA provides a

context for that subjectivity. 

Quality of Licensee PRAs - Licensee PRAs vary considerably in both quality and scope. This is

evidenced, in part, by significant differences in whether and how licensee’s account for external

events, low power and shutdown modes of operation.
-52-

3.11.2.2 Conclusions

(1) The staff routinely uses engineering judgement in the SDP. The Task Group noted that

this practice is consistent with the NRC’s PRA Policy Statement which encourages the

use of PRA technology to complement the staff’s deterministic decisions. It is

noteworthy that many of the internal and external stakeholders interviewed by the Task

Group, indicated that the perceived problems with the SDP, including timeliness, can be

attributed more to the challenges associated with making informed engineering

decisions (or assumptions), than with limitations of PRA. 

(2) In light of the differences in quality and scope of licensee PRAs, it is inadvisable for the

staff to use the results from the licensees’ PRA as the sole basis for SDP

determinations. Additional guidance is needed for the staff to determine whether the

results of a licensee’s risk analysis of a finding is of sufficient quality to use as input to

the staff’s final significance determination. 

3.11.2.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendation: 

(1) NRR should develop guidance to allow the staff to determine whether the results of a

licensee’s risk analysis of a finding is of sufficient quality to use as input to the staff’s

final significance determination. This guidance should focus on the use of DG-1122,

which addresses the Nuclear Energy Institute’s (NEI) peer review process (NEI-00-02),

and the proposed update, that includes a licensee self-assessment process to confirm

conformance with ASME RA-S-2002, “Standard for Probabilistic Risk Assessment for

Nuclear Power Plant Applications,” dated April 5, 2002.

3.11.3 Insights on Performance of ROP Action Matrix

During the Task Group’s interviews, NRC staff and licensee representatives expressed

concerns that the use of two White findings in a cornerstone as a criterion for entry into a

Degraded Cornerstone was overly conservative.

3.11.3.1 Observations

Most internal and external interviewees agreed that the ROP Action Matrix has functioned as

designed. However, many NRC interviewees suggested that the criteria in IMC 0305 regarding

the number of White inspection findings that result in a “Degraded Cornerstone” should be

revisited. Currently, two White inspection findings in a given cornerstone are treated in the

Action Matrix the same as a Yellow inspection finding. Some interviewees observed that two

White inspection findings could each be near the lower end of the White performance band

(i.e., 1.0E-6 per reactor year), and in such cases, the cumulative impact of those two findings

could result in a CDF change that is less than 1.0E-5 per reactor year, which is in the White

performance band. Some licensee representatives indicated that use of this criterion had

resulted in unnecessary burden due to the financial implications and public response to the

Degraded Cornerstone. 
-53-

Some interviewees suggested that NRC should consider using three (or more) White inputs as

the criteria for determining when a Degraded Cornerstone exists. They also indicated that the

risk associated with a White input in the EP cornerstone is not commensurate with the risk in

the IE, MS or BI cornerstones, however, such inputs are treated similarly in the Action Matrix

and could result in a Degraded Cornerstone. Because the combination of such findings often

results in more NRC action than is warranted by the licensee’s performance, the interviewees

suggested that the NRC review the appropriateness of this criteria. 

Despite these concerns, internal and external interviewees indicated that the NRC’s response

at those plants that have entered the Degraded Cornerstone column of the Action Matrix was,

in most cases, merited by the licensee’s performance. 

3.11.3.2 Conclusions

(1) Overall, the Action Matrix has fulfilled its purpose in providing an objective, scrutable

and predictable framework for NRC actions in response to licensee performance

problems. The oversight process also provides sufficient flexibility in use of the Action

Matrix for NRC managers to use discretion in decisions concerning the scope and timing

of the agency’s response to licensee performance problems. 

(2) The staff used reasoned judgement to support its decision to use two White inputs in

the same cornerstone as part of the criteria for defining a “Degraded Cornerstone.” 

Although a detailed analysis or evaluation was not developed to support this decision,

the Task Group did not identify data or obtain information from its interviews that

suggested that the criterion was inappropriate. 

3.11.3.3 Recommendations

To address the issues discussed in this section of the report, the Task Group made the

following recommendations:

(1) NRR should review the Action Matrix annually to assess its impact on stakeholders and

the appropriateness of the criteria for determining the combination of inputs that dictate

a licensee’s placement in the Action Matrix. The results of this assessment should be

provided in a report to management with recommendations for adjustments, as

necessary.
-54-

APPENDIX A - CONSOLIDATED LIST OF RECOMMENDATIONS

Recommendation

Number

Recommendation

3.1.3(1) The NRC should retain the Phase 2 process as a screening method

for inspectors to screen findings of very low safety significance

(Green) and identify potentially greater-than-green findings for further

review by SRAs. 

3.1.3(2) The NRC should complete the benchmarking of the remaining Phase

2 SDP notebooks and develop plant-specific Enhanced Pre-Solved

Phase 2 SDP Tables (Option 3) as the primary tool for inspectors to

perform Phase 2 evaluations. However, the notebooks should be

retained for optional use by the NRC staff to gain additional risk

insights. 

3.2.3(1) NRR should issue the proposed SDP basis document, including the

current performance expectations for the Phase 2 notebooks. 

3.2.3(2) NRR should engage the Regions to confirm their understanding and

implementation of the expectations regarding use of the SDP

provided in the August 9, 2002, memorandum from the Director, NRR

to the Regional Administrators.

3.2.3(3) NRR should reevaluate the performance expectation of the SDP tools

after the remaining notebooks have been benchmarked and modify

program guidance, as appropriate, to reflect any revision to the

expectation. 

3.5.3(1) NRR should encourage the Regions to hold refresher training on the

Phase 2 SDP at least annually. This training should be led by a

Regional SRA and can also be used to convey best practices and

give feedback on common questions and problem areas. (The need

for periodic Phase 2 SDP refresher training should be significantly

diminished with the use of the enhanced pre-solved SDP tables

discussed in Section 3.1.1.3.)

3.5.3(2) NRR should implement SDP Task Action Plan Objective 3.6, as

planned and scheduled to enhance the guidance for conducting

Phase 3 analyses. 

3.5.3(3) NRR should conduct a systematic assessment of training in the area

of risk, with a particular focus on identifying and advancing the

knowledge, skills, and abilities (KSAs) for implementing the SDP.

3.6.3(1) NRR should complete the benchmarking of the remaining Phase 2

SDP notebooks to support the current Phase 2 SDP as well as the

development of the enhanced pre-solved Phase 2 table

recommended in Section 3.1.3(2).
-55-

3.6.3(2) NRR should develop a plan to periodically review and update the

Phase 2 SDP assessment tools to address any licensee PRA

changes and/or plant modifications.

3.6.3(3) NRR and RES should implement a process to compare the results of

the benchmarked SPAR model with the results of the benchmarked

Phase 2 notebooks. 

3.6.3(4) NRR should document the “construction rules” for the Phase 2 SDP

notebooks. Consideration should be given to including (or

referencing) the construction rules in the proposed SDP Basis

Document.

3.7.3(1) NRR should develop guidance to allow the staff to recognize

situations where the “state of knowledge” correlation, which is

described in RG 1.174, might warrant a Phase 3 analysis.

3.7.3(2) NRR should modify IMC 0609 guidance to explicitly indicate that

traditional engineering analysis considerations (e.g., reduction of

safety margin, or significant loss of defense-in-depth) should be used

to determine an appropriate color to associate with findings where the

uncertainty in the risk evaluation arising from the characterization of

the impact of the inspection finding is large enough that the color is

indeterminate on the basis of the risk analysis. This guidance should

be such that it promotes consistency in the staff’s use of such

analyses and should only be applied to those findings where the

uncertainty is significant (i.e., when alternate assumptions yield

results which vary over more than two orders of magnitude). 

3.8.3(1) The NRC should not modify the SDP to include consideration of

“other inputs.” This decision should be re-evaluated as part of the

annual ROP self assessment to determine if licensee performance

indicates that different treatment of “other inputs” in the ROP is

warranted.

3.8.3(2) NRR should enhance its oversight of the implementation of the

guidance in IMC 0305, Section 06.06.h to promote consistent

application by the Regions. The guidance in IMC 0305,

Section 06.06.h should be enhanced to provide a more predictable

standard/criteria for determining what constitutes a substantive cross

cutting issue. Additional oversight in this area is needed to ensure

consistency across the regional offices. 
-56-

3.8.3(3) NRR should revise the ROP guidance to include consideration of a

response to the identification of a substantive cross-cutting issue

(only when there is at least one White PI or finding). This response

could include a redirection of inspection resources, management

meetings, and/or a docketed licensee response describing actions

planned or taken to address the cross-cutting issue. This guidance

should also include a description of how the NRC will close a

substantive cross-cutting issue.

3.8.3(4) NRR should supplement the guidance in IMC 0305, Section 06.06.f,

with additional guidance that lists the type of information that should

be included in a Region’s request to deviate from the Action Matrix

(e.g., synopsis of the findings affecting the licensee’s performance;

the actions (column) stipulated by the Action Matrix; the Region’s

rationale or considerations for taking action different from that

stipulated in the Action Matrix, etc.).

3.9.3(1) NRR should reinforce the current SDP timeliness goals through

management interactions, identifying responsible mangers, and

establishing accountability for the resolution of overdue significance

determinations. The NRR SDP timeliness metrics should be

incorporated into the Regional Operating Plans. 

3.9.3(2) NRR should communicate the agency's SDP timeliness goals to

licensees (e.g., include text in Choice Letters to inform licensees of

NRC's plan to reach a final significance determination within 90 days;

discuss the timeliness goal during regulatory conferences; discuss

SDP timeliness, including the timeliness goal during the annual

Regulatory Information Conference, etc.)

3.9.3(3) NRR should rectify the difference between the metrics identified in the

NRR Operating Plan (i.e., 75% timeliness goal in FY03 increasing to

90% timeliness goal by FY06) with the goals stated in IMC 0609 and

reiterated by the August 2, 2001, SRM (i.e., 100% timeliness goal). 

3.9.3(4) NRR should change the guidance in IMC 0609 to allow the Regions

the option to use a "preliminary greater-than-very low safety

significance" characterization for preliminary safety significance

determination (Choice) letters when there is sufficient uncertainty

regarding the preliminary risk estimate. 

3.9.3(5) OPA should further modify its policy for issuing press releases for

ROP findings to link the issuance of press release to a finding(s) that

results in a Degraded Cornerstone. This modification will better align

the emphasis that results from the issuance of a press release with

the "moderate degradation in safety performance" associated with a

Degraded Cornerstone (i.e., two White inputs (findings and/or

Performance Indicators), one Yellow input, or three White inputs in

the same Strategic Performance Area). 
-57-

3.9.3(6) NRR should modify IMC 0305 to reference the OPA policy for issuing

press releases for ROP inspection findings.

3.10.3(1) NRR should change IMC 0612 to require a brief description in the

inspection report summary of findings of corrective actions taken by

the licensee to restore compliance with NRC regulations, where

applicable. (OIG Report recommendation #10)

3.10.3(2) The EDO should inform OIG that the ROP web site will not be revised

as described in recommendation #9. Users currently have ready

access to all findings in a cornerstone by hyper-linking to the next

web page and the OIGs’ recommendation would not substantially

improve the quality of the information available to users. The difficulty

and costs of implementing this change appears to exceed the benefit.

(OIG Report recommendation #9)

3.11.1.3(1) NRR should broaden the NRR SDP Task Action Plan Objective 6.1 to

initiate a cooperative effort between NRR and RES to explore

efficiency and quality enhancements that would result in better

coordination and/or integration of these two programs. 

3.11.1.3(2) NRR and RES should identify avenues to enhance the staff's

knowledge of the ASP program, including adding a module to the

P-111 course regarding the ASP program.

3.11.2.3(1) NRR should develop guidance to allow the staff to determine whether

the results of a licensee’s risk analysis of a finding is of sufficient

quality to use as input to the staff’s final significance determination. 

This guidance should focus on the use of DG-1122, which addresses

the Nuclear Energy Institute’s (NEI) peer review process (NEI-00-02),

and the proposed update, that includes a licensee self-assessment

process to confirm conformance with ASME RA-S-2002, “Standard

for Probabilistic Risk Assessment for Nuclear Power Plant

Applications,” dated April 5, 2002.

3.11.3.3(1) NRR should review the Action Matrix annually to assess its impact on

stakeholders and the appropriateness of the criteria for determining

the combination of inputs that dictate a licensee’s placement in the

Action Matrix. The results of this assessment should be provided in a

report to management with recommendations for adjustments, as

necessary.
-58-

APPENDIX B - LIST OF ACRONYMS

CDF Change in Core Damage Frequency

LERF Change in Large Early Release Frequency

ASP Accident Sequence Precursor

BI Barrier Integrity

BNL Brookhaven National Laboratory

CCDP Conditional Core Damage Probability 

CDF Core Damage Frequency

DPO Differing Professional Opinion

DPV Differing Professional View

EATS Enforcement Action Tracking System

EDO Executive Director for Operations

IE Initiating Events

IIPB Inspection Program Branch, Division of Inspection Program Management, Office

of Nuclear Reactor Regulation

IMC Inspection Manual Chapter

INEEL Idaho National Engineering and Environmental Laboratory

IT Information Technology

LERF Large Early Release Frequency

MS Mitigating Systems

NRC Nuclear Regulatory Commission

NRR Office of Nuclear Reactor Regulation

NSIR Office of Nuclear Security Incident and Response

OE Office of Enforcement

OIG Office of the Inspector General

OPA Office of Public Affairs

PRA Probabilistic Risk Assessment

RES Office of Nuclear Regulatory Research

RG Regulatory Guide

ROP Reactor Oversight Process

SDP Significance Determination Process

SECY Office of the Secretary

SERP Significance and Enforcement Review Panel

SPAR Standardized Plant Analysis Risk 

SRA Senior Reactor Analyst

SRM Staff Requirements Memorandum

SSC Structure, System, or Component
-59-

APPENDIX C - LIST OF DOCUMENTS REVIEWED

(1) SECY-99-007, “Recommendations for Reactor Oversight Process Improvements”

(2) SECY-99-007A, “Recommendations for Reactor Oversight Process Improvements

(Follow-up to SECY-99-007)”

(3) Inspection Manual Chapter 0609, Sections 01 and 02, “Significance Determination

Process”

(4) Inspection Manual Chapter 2515, “‘Operating Reactor Assessment Program”

(5) Inspection Manual Chapter 0612, “Power Reactor Inspection Reports”

(6) Correspondence: ML013550087 Staff response to ACRS comments on the SDP

(1/10/02)

(7) Correspondence: ML020110121 Staff response to RIV & OE request for review of

SDP(1/15/02)

(8) Correspondence: ML020420587, ML020370605, ML020420589 Staff response to DPV

(2/18/02)

(9) Correspondence: ML020920470 Staff response to Commission SRM (2/5/02)

(10) Correspondence: ML020440182 (pkg.) SDP Improvement Plan issuance (3/18/02)

(11) Correspondence: G20020209 Staff comments on OIG draft audit report on the SDP

(5/14/02)

(12) Correspondence: ML021760004 Response to SRM M020319 Differences between

SDP, ASP, & INES (7/12/02)

(13) Correspondence: ML021750054 Expectations for Inspector Use of the SDP (8/9/02)

(14) Correspondence: ML013530458 Update of Active NRR Requests for Assistance

(1/31/02)

(15) Correspondence: ML020810004 Reactor Oversight Process Self-Assessment Support

(3/21/02)

(16) Correspondence: ML021770453 Reactor Oversight Process Self-Assessment Support

(6/26/02)

(17) Correspondence: ML022410392 Reactor Oversight Process Self-Assessment Support

(8/29/02)

(18) “Understanding Risk - Informing Decisions in a Democratic Society,” (summary),

National Academy Press, 1996

(19) Documents related to SDP results and analyses for:

Cooper environmental qualification finding

Indian Point steam generator tube degradation finding

Davis-Besse reactor vessel head degradation significance characterization (no

finding yet)

(15) ROP and SDP Program Guidance
-60-

APPENDIX D - LIST OF PERSONS INTERVIEWED

HEADQUARTERS - Management & Supervisors

Baranowsky, Patrick, Branch Chief, RES

Barrett, Richard, Director, DE, NRR

Beckner, William, Program Director, Operating Reactor Improvements, NRR

Black, Suzanne, Deputy Director, DSSA, NRR

Boger, Bruce, Director, DIPM, NRR

Borchardt, R. William, ADIP, NRR

Carpenter, Cynthia, Branch Chief, IIPB, DIPM, NRR

Cheok, Michael, Assistant Branch Chief, OERAB, DRAA, RES

Coe, Douglas, Section Chief, RIS, IIPB, NRR

Collins, Samuel, Director, NRR

Congel, Frank, Director, OE

Dean, William, Deputy Director, DE, NRR 

Gillespie, Frank, Deputy Director, DRIP, NRR

Johnson, Jon, Deputy Director, NRR

Johnson, Michael, Branch Chief, SPSB, DSSA, NRR

Leuhman, James, Deputy Director, OE

Lieberman, James, Special Counsel for Rulemaking & Fuel Cycle ,OE

Madison, Alan, Branch Chief, RSOS, NSIR

Newberry, Scott, Director, DRAA, RES

Reinhart, F. Mark, Licensing Section, SPSB, DSSA, NRR

Rubin, Mark, Section Chief, SPSB, DSSA, NRR

Ruland, William, Project Directorate IV, DLPM, NRR

Sheron, Brian, ADPT, NRR

Strosnider,Jack, Deputy Director, RES

HEADQUARTERS - Staff

Arrighi, Russell, Project Manager, RLEP, DRIP, NRR

Franovich, Michael, Risk Analyst, SPSB, DSSA, NRR

Gibbs, Russell, Sr. Reactor Analyst, IIPB, DIPM, NRR

Houghton, James, RES

Jacobson, Jeffrey, Program Manager, IIPB, DIPM, NRR

Johnson, James, Special Assistant, RES

Koltay, Peter, IIPB, DIPM, NRR (former RI at DC Cook, Region III)

Long, Steven, Sr. Reliability and Risk Analyst, SPSB, DSSA, NRR

Nelson, David, Enforcement Specialist, OE

O’Neal, Daniel, Risk Analyst, SPSB, DSSA, NRR

O’Reilly, Patrick, Sr. Reliability and Risk Engineer, RES

Sykes, Marvin, Reactor Operations Engineer, IIPB, DIPM, NRR

Wilson, Peter, PRAB, DSSA, NRR

Wong, See-Meng, Sr. Reactor Analyst, SPSB, DSSA, NRR
-61-

APPENDIX D - LIST OF PERSONS INTERVIEWED (Cont’d.)

Regional Managers

Brockman, Kenneth, DRP, Division Director, Region IV

Caniano, Roy, Deputy Director, DRS, Region III

Christensen, Harold, Deputy Director, DRS, Region II

Collins, Elmo, Acting Division Director, DRS, Region IV

Dyer, James, Regional Administrator, Region III

Grant, Geoffrey, DRP Division Director, Region III

Grobe, Jack, DRS, Division Director, Region III

Gwynn, Thomas P., Deputy Regional Administrator, Region IV

Holian, Brian, Deputy Director, DRP, Region I

Lanning, Wayne, Division Director, DRS, Region I

Merschoff, Ellis, Regional Administrator, Region IV

Miller, Hubert, Regional Administrator, Region I

Pederson, Cindy, Director, DRS, Region III

Plisco, Loren, Director, DRP, Region II

Reyes, Luis, Regional Administrator, Region II

Wiggins, James, Deputy Regional Administrator, Region I

Regional Supervisors

Bonser, Brian, Branch Chief, DRP, Region II

Burgess, Bruce, Branch Chief, DRP, Region III

Cahill, Stephen, Branch Chief, DRP, Region II

Conte, Richard, Branch Chief, DRS, Region I

Doerflein, Lawrence, Branch Chief, Systems Branch, DRS, Region I

Gody, Tony, Operations Branch Chief, DRS, Region IV

Haag, Robert, Branch Chief, Region II

Jones, William, Branch Chief, DRP, Region IV

Kennedy, Kriss, Branch Chief, DRP, Region IV

Landis, Kerry, Branch Chief, DRP, Region II

Lanksbury, Roger, Branch Chief, DRP, Region III

Lew, David, Branch Chief, DRS, Region I

Marschall, Charlie, Branch Chief, DRS, Region IV

McDermott, Brian, Branch Chief, DRP, Region I

Meyer, Glenn, Branch Chief, DRP, Region I

Ogle, Charles, Branch Chief, Region II

Riemer, Kenneth, Branch Chief, DRS, Region III

Shanbaky, Mohamed, Branch Chief, DRP, Region I

Smith, Linda, Branch Chief, DRP, Region IV

Vegel, Anton, Branch Chief, DRP, Region III

Wert, Leonard, Branch Chief, DRP, Region II
-62-

APPENDIX D - LIST OF PERSONS INTERVIEWED (Cont’d.)

Regional Inspectors/Project Engineers

Brown, Eva, Resident Inspector, Brunswick, RII (currently DLPM, NRR)

Caldwell, Robert, Resident Inspector, Farley, Region II (currently RORP, DRIP, NRR)

Carroll, Robert, DRP, Region II

Clark, Jeff, Project Engineer, Region IV (former SRI, Cooper, Region IV)

Cook, William, Sr. Project Engineer, Branch 2, DRP, Region I

Coyne, Kevin, Resident Inspector, DC Cook, Region III (currently IEHB, DIPM, NRR)

Falevits, Zelig, Electrical Inspector, DRS, Region III

Gage, Paul, Reactor Inspector, Division of Reactor Safety, Region IV

Gray, Mel, Sr. Reactor Inspector, Systems Branch, DRS, Region I

Haire, Mark, Reactor Operations Inspector, Division of Reactor Safety, Region IV

Loughead, Patricia, Mechanical Inspector, DRS, Region III

MacDonald, George, DRP, Region II

McKenzie, Thomas, Inspector, DRS, Region II

Passehl, David, Region III

Paulk, Chuck, DRP, Region IV

Schin, Robert, Inspector, DRS, Region II

Schmidt, Wayne, Sr. Reactor Inspector, DRS, Region I

Walker, Wayne, DRP, Region IV

Residents

Allen, Don, Sr. Resident Inspector, Commanche Peak

Bower, Fred, Resident Inspector, Salem 1 & 2

Bywater, Russ, SRI, Arkansas Nuclear One

Cox, Mark, Resident Inspector, Indian Point 3

Dipalo, Eugene, Resident Inspector, McGuire 

Drysdale, Peter, Sr. Resident Inspector, Indian Point 3

Duncan, Eric, Sr. Resident Inspector, La Salle

Habighorst, Peter, Sr. Resident Inspector, Indian Point 2

James, Lois, Resident Inspector, Indian Point 2

Krohn, Paul, Sr. Resident Inspector, Point Beach

Lorson, Raymond, Sr. Resident Inspector, Salem 1 & 2

McCoy, Gerald, Resident Inspector Surry

Morris, R. Michael, Resident Inspector, Point Beach

Sanchez, Fred, Resident Inspector, Commanche Peak

Schoppy, Joseph, Sr. Resident Inspector, Hope Creek

Shaeffer, Scott, Sr. Resident Inspector, McGuire

Weaver, Kathy, RI, Arkansas Nuclear One

SRAs

Bernhard, Rudolph, Region II

Burgess, Sonia, Region III

Cobey, Eugene, Region I

Loveless, David, Region IV

Pruett, Troy, Region IV

Rasmussen, Richard, SPSB, DSSA, NRR (in training)

Rogers, Walt, Region II
-63-

APPENDIX D - LIST OF PERSONS INTERVIEWED (Cont’d.)

External Interviews

Anderson, Ross, Licensee Risk Analyst, North Anna

Ashley, Glenn, Licensing Manager, Arkansas Nuclear One, Region IV Licensee

Bement, Robert, General Manager, Arkansas Nuclear One, Region IV Licensee

Berchall, William, PRA Manager, Exelon

Brewer, Duncan, PRA Supervisor, Duke Energy

Bryant, Julius, Licensing Engineer, McGuire, Duke Energy

Bucheit, David, Licensee Risk Analyst, Supervisor, North Anna

Canyia, Fred, Site Vice President, Point Beach

Crossman, James, Licensing Manager, North Anna, Dominion

DeRoy, Joseph, Plant Manager, Indian Point 3

Fricker, Carl, Operations Manager, Salem

Ho, Wei, Salem

Kaegi, Glen, Regulatory Affairs Manager, La Salle

Kitlan, Michael, Jr., PRA Engineer, Duke Energy

Krause, Charles, Sr., Licensing Engineer, Point Beach

Kreieger, Kurt, Salem

Lanc, Terry, Site Risk Analyst, La Salle, Exelon

Masterlark, James, Licensee Risk Analyst, Point Beach, Nuclear Management Company

McCann, John, Licensing Manager, Indian Point 2

Michael, Lloyd, Region IV Licensee, Arkansas Nuclear One

Moore, David, Plant Manager, Commanche Peak

Nagle, John, Salem

Ritzman, Robin, Salem

Salamon, Gabe, Salem

Schiavoni, Mark, Plant Manager, La Salle

Schwarz, Christopher, Plant Manager, Indian Point 2

Small, Michal, Licensing Manager, Surry, Dominion

Sowers, Thomas, III, Director, Station Operations and Maintenance, Surry

Steinmetz, John, PRA Senior Engineer, Exelon

Thomas, Charles, Licensing Manager, McGuire, Duke Energy

Tirsun, Daniel, Risk Analyst, Region IV Licensee, Comanche Peak

Waldinger, Lon, Salem

Walker, Jessica, Region IV Licensee, Arkansas Nuclear One

Walker, Roger, Licensing Manager, Commanche Peak

Walker, Woody, Region IV Licensee, Arkansas Nuclear One

Webb, Thomas, Regulatory Affairs Manager, Point Beach

NEI

Floyd, Steven

Other External

Dyckman, Dennis, Pennsylvania Department of Environmental Protection 

Lipoti, Jill, New Jersey Department of Environmental Protection 

Lochbaum, David, Union of Concerned Scientists

Settles, Cecil, Illinois Department of Nuclear Safety

Shadis, Raymond, New England Coalition on Nuclear Pollution