Audit of NRC’s

Significance Determination 

Process for Reactor Safety 

OIG-16-A-21

September 26, 2016

All publicly available OIG reports (including this report) 

are accessible through NRC’s Web site at

http://www.nrc.gov/reading-rm/doc-collections/insp-gen
UNITED STATES

NUCLEAR REGULATORY COMMISSION

WASHINGTON, D.C. 20555-0001

OFFICE OF THE

INSPECTOR GENERAL September 26, 2016

MEMORANDUM TO: Victor M. McCree

Executive Director for Operations

FROM: Steven E. Zane \RA\

Acting Assistant Inspector General for Audits

SUBJECT: AUDIT OF NRC’S SIGNIFICANCE DETERMINATION 

PROCESS FOR REACTOR SAFETY (OIG-16-A-21)

Attached is the Office of the Inspector General’s (OIG) audit report titled, Audit of NRC’s 

Significance Determination Process for Reactor Safety.

The report presents the results of the subject audit. Following the August 29, 2016, exit 

conference, agency staff indicated that they had formal comments for inclusion in this 

report. These comments are included in Appendix B of this report.

Please provide information on actions taken or planned on each of the 

recommendations within 30 days of the date of this memorandum. Actions taken

or planned are subject to OIG followup as stated in Management Directive 6.1.

We appreciate the cooperation extended to us by members of your staff during the 

audit. If you have any questions or comments about our report, please contact me at 

(301) 415-5915 or Paul Rades, Team Leader, at (301) 415-6228.

Attachment: As stated
Office of the Inspector General

U.S. Nuclear Regulatory Commission

Defense Nuclear Facilities Safety Board

Results in Brief

NRC needs to establish, communicate, and document clear and consistent 

expectations for staff and managers to complete their roles in the SDP; clarify 

more-than-minor screening questions; and implement controls to ensure 

independent audits of greater than Green inspection findings are performed 

and documented. 

NRC does not know the amount of resources used by staff and managers to 

complete various steps in the SDP. Additionally, unclear questions used to 

screen issues identified during inspections have made it difficult for 

inspectors to determine whether issues should be categorized as either 

minor or more-than-minor. Also, NRC is unable to provide documentation of 

independent audits performed on greater than Green inspection findings 

despite recently reporting that these inspection findings were assessed by an 

independent auditor.

This report makes recommendations to improve overall management of SDP 

workflow, clarify issue screening questions for inspection staff, and 

implement controls to ensure independent audits are performed and 

documented.

Agency management stated their general agreement with the 

recommendations in this report and provided additional comments.

Audit of NRC’s Significance Determination Process for 

Reactor Safety

What We Found

What We Recommend

Why We Did This Review

The U.S. Nuclear Regulatory 

Commission (NRC) Significance 

Determination Process (SDP) is 

used to determine the safety 

significance of inspection 

findings identified within the 

Reactor Oversight Process (ROP) 

cornerstones of safety. 

NRC inspectors perform

inspections at nuclear reactor 

sites to identify licensee failures 

to meet a regulatory 

requirement or self-imposed 

standard that a licensee should 

have met.

The SDP consists of several steps 

and activities performed by 

agency staff and management to 

determine and categorize the 

significance of licensee 

performance deficiencies 

identified through inspections. 

The SDP also requires an 

independent audit of inspection 

findings to ensure significance 

determination results are 

predictable and repeatable.

The audit objective was to assess 

the consistency with which NRC 

evaluates power reactor safety 

inspection findings under the 

SDP. 

 

 

 

 

 

 

OIG-16-A-21

September 26, 2016
Audit of NRC’s Significance Determination Process for Reactor Safety

ABBREVIATIONS AND ACRONYMS .......................................................... i

I. BACKGROUND ................................................................................1

II. OBJECTIVE......................................................................................3

III. FINDINGS.........................................................................................3

A. SDP Resource Use and Expectations for Time Spent on the 

SDP Are Unknown ................................................................3

B. Unclear Issue Screening Questions ......................................6

C. Unable To Verify Independent Audits ..................................19

IV. CONSOLIDATED LIST OF RECOMMENDATIONS.......................12

V. AGENCY COMMENTS...................................................................13

APPENDIXES 

A. OBJECTIVE, SCOPE, AND METHODOLOGY...............................14

B. AGENCY FORMAL COMMENTS...................................................16

TO REPORT FRAUD, WASTE, OR ABUSE.............................................20

COMMENTS AND SUGGESTIONS..........................................................20

 TABLE OF CONTENTS
Audit of NRC’s Significance Determination Process for Reactor Safety

i

NRC Nuclear Regulatory Commission

OIG Office of the Inspector General

ROP Reactor Oversight Process

SDP Significance Determination Process

SERP Significance and Enforcement Review Panel

SRA Senior Reactor Analyst

 ABBREVIATIONS AND ACRONYMS
Audit of NRC’s Significance Determination Process for Reactor Safety

1

NRC’s Reactor Oversight Process

The U.S. Nuclear Regulatory Commission’s (NRC) Reactor Oversight 

Process (ROP) is an approach to assessing plant performance in key 

areas. Each plant performance goal established by the ROP has 

cornerstones that reflect the essential safety aspects of facility operation. 

Licensee performance that meets NRC requirements in these 

cornerstones provides reasonable assurance of safe facility operation.

Significance Determination Process

The Significance Determination Process (SDP) is a process used by NRC 

to determine the safety significance of inspection findings identified within 

the ROP cornerstones of safety.1 Before the SDP is conducted, 

inspectors located at reactor sites and NRC regional offices perform 

inspections and identify potential performance deficiencies as part of preSDP steps. Performance deficiencies are licensee failures to meet a 

regulatory requirement or self-imposed standard that a licensee should 

have met. NRC assesses performance deficiencies as minor or morethan-minor by using screening questions. Minor performance deficiencies 

are not documented; however, the SDP is conducted for more-than-minor 

performance deficiencies, or findings. Findings are categorized as either 

Green, White, Yellow, or Red.

Green findings2 have a very low safety significance while White findings 

have a low to moderate safety significance, and Yellow findings have a 

substantial safety significance. Red findings have a high safety 

significance and move the operating plant to NRC’s highest category of 

 1 The ROP regulatory framework is comprised of three strategic performance areas that each contain cornerstones 

reflecting essential safety aspects of facility operation. These three strategic performance areas and their 

associated cornerstones are: (1) Reactor Safety (initiating events, mitigating systems, barrier integrity, and 

emergency preparedness); (2) Radiation Safety (public radiation safety and occupational radiation safety); and (3) 

Safeguards (security). This audit focused on the Reactor Safety strategic performance area and its initiating events, 

mitigating systems, and barrier integrity cornerstones. 

2 Green findings are documented by NRC and do not undergo additional risk analyses.

 I. BACKGROUND
Audit of NRC’s Significance Determination Process for Reactor Safety

2

oversight, which entails thousands of additional hours of inspection. In 

2015, there were 17 greater than Green findings issued under all ROP 

cornerstones.

Figure 1. SDP Workflow

Source: OIG

Senior reactor analysts (SRA) perform risk analyses in step four to inform 

the significance determination for the finding. Division directors, branch 

chiefs, and other staff such as regional administrators provide general 

oversight, guidance, direction, and resource allocation to support the SDP 

during several steps. Division directors and branch chiefs also attend or 

provide representatives, as requested, for Significance and Enforcement 

Review Panels (SERP) and licensee Regulatory Conferences in steps five 

through seven. NRC staff and managers are also responsible for 

reviewing SDP results for timeliness and accuracy. 

Recently, NRC took steps to improve the SDP through a business process 

improvement initiative with the goals of reviewing SDP resource and 

timeliness data, evaluating SDP practices and guidance, and identifying 

opportunities for improvement. NRC has also proposed SDP changes 

associated with milestones and resource allocation.
Audit of NRC’s Significance Determination Process for Reactor Safety

3

The audit objective was to assess the consistency with which NRC evaluates power 

reactor safety inspection findings under the SDP.

Opportunities exist for NRC to more consistently manage the SDP by 

communicating clear and consistent expectations, clarifying issue 

screening questions, and ensuring independent audits are performed and 

documented.

A. SDP Resource Use and Expectations for Time Spent on the 

SDP Are Unknown 

NRC does not know the extent of resources used to complete each SDP 

step, and staff and managers do not know how much time they have to 

complete each SDP step. This is because NRC has not evaluated the 

resources used to complete each SDP step, nor has it established or 

communicated clear and consistent expectations for how long staff and 

managers should spend completing each step. As a result, NRC (1) does 

not have a complete view of how resources are used, (2) may undertake 

unnecessary changes to improve timeliness, and (3) cannot compare 

actual performance to planned performance. Additionally, without clear 

and consistent expectations, step completion time is driven by individual 

staff and manager priorities.

Resource Use and Expectations Should Be Known

Federal internal control standards require NRC to use quality information 

to achieve program objectives and have controls in place to compare 

actual program performance to planned or expected results. NRC should 

What Is Required

 II. OBJECTIVE

 III. FINDINGS
Audit of NRC’s Significance Determination Process for Reactor Safety

4

know the resources allocated and used for different SDP steps and 

establish clear and consistent expectations for how long each step should 

take. Further, staff and managers responsible for performing SDP steps 

should know how much time they should spend to complete their work.

Resource Use and Expectations Unknown

NRC does not know the amount of resources used to complete each SDP 

step. NRC relies on aggregate hours charged to an SDP activity code to 

identify how much time it takes to complete significance determinations.

For example, NRC’s most recent data shows approximately 13,000 hours 

were spent on the SDP in 2014.

3 However, auditors found staff and 

managers charge time spent performing SDP steps in different ways. For 

example, SRAs charge time to an SDP activity code, and inspectors might

sometimes charge time to this code. However, branch chiefs and other 

management such as division directors do not charge time to the SDP

activity code even when performing SDP related work.

NRC also relies on the total number of days it takes to issue a final 

significance determination as a measure of timeliness, but sometimes the 

clock does not start until most of the work for a significance determination 

has already been completed. Preliminary analysis of significance can 

entail several months of work by inspectors, SRAs, and branch chiefs. 

Further, staff can start charging time to the SDP at different steps.

Staff and managers do not know how much time they should spend to 

complete each SDP step. A manager contended inspectors and SRAs 

should spend no more than 90 days (the current metric for the entire 

process) performing the steps. Other managers maintained the overall 

amount of time spent dispositioning findings should vary based on the 

complexity of the finding.

 3 NRC recently changed the time codes and was unable to provide comparable data for 2015.

What We Found
Audit of NRC’s Significance Determination Process for Reactor Safety

5

Evaluation Not Performed and Expectations Not Communicated

NRC has not evaluated the resources used to perform each SDP step. 

NRC recently made efforts to identify how long it takes to complete 

significance determinations, but these efforts are incomplete. For 

example, NRC recently evaluated the number of hours spent completing 

significance determinations and compared those hours to the number of 

significance determinations completed over the last 13 years. However, 

the data used in the analysis excluded time spent by branch chiefs and 

other managers on the SDP. NRC also identified that the total number of 

SDP findings had decreased since 2001, but observed that overall time 

spent on the SDP had increased. However, NRC did not include all hours 

spent on the SDP by all participants nor did it analyze how much time staff 

and managers took to perform each SDP step. 

Additionally, NRC has not established and communicated clear and 

consistent expectations for how much time staff and managers should 

spend completing each SDP step. There is an aggregate metric for the 

SDP, but no metrics for specific steps.

Risk of Unreasonable Timeliness Standards, Inability to Measure 

Performance, and Inconsistent Priorities

Because NRC does not have a complete view of how SDP resources are 

used, it risks undertaking unnecessary changes to improve SDP 

timeliness. For example, current reliance on hourly data that always 

excludes manager participation and focuses on the total number of days it 

takes to issue a final significance determination, could lead to inaccurate 

timeliness assessments. Also, timeliness assessments lack consistency 

because these assessments can have different starting points within the 

SDP. 

Why This Occurred

Why This Is Important
Audit of NRC’s Significance Determination Process for Reactor Safety

6

Additionally, NRC risks developing new timeliness metrics based on 

incomplete data. For example, NRC has undertaken an effort to revise 

the SDP by modifying guidance and changing timeliness metrics. 

However, these efforts are underway without knowing how much time or 

resources are used for the SDP.

Further, NRC is unable to compare actual performance of SDP steps to 

planned performance. Without clear and consistent expectations for how 

much time staff and managers have to complete each SDP step, 

completion time is driven by individual staff and manager priorities.

Therefore, there are varying thoughts among staff and management on 

how much time is appropriate for each SDP step.

Recommendation

OIG recommends that the Executive Director for Operations

1. Assess SDP workflow, and establish, communicate, and document 

clear and consistent expectations for staff and managers to 

complete their roles in the SDP. 

B. Unclear Issue Screening Questions

Inspectors sometimes have difficulty determining whether issues identified 

during inspections should be categorized as either minor or more-thanminor because issue screening questions in Inspection Manual Chapter 

0612 Appendix B, “Issue Screening,” are unclear. As a result, inspectors 

might needlessly spend time or resources formally documenting minor 

issues and miss opportunities to document more significant issues. 

Further, NRC risks the potential for inconsistent screening of performance 

deficiencies.
Audit of NRC’s Significance Determination Process for Reactor Safety

7

More-than-Minor Screening Process Should Be Readily Understood 

and Easily Applied

NRC requires agency positions to be readily understood and easily 

applied. The more-than-minor screening questions should be clear to staff 

and managers applying the criteria to issues identified in inspections.

More-than-Minor Screening Process Is Not Readily Understood or 

Easily Applied

Inspectors sometimes have difficulty understanding and applying the 

questions used to screen inspection findings as either minor or more-thanminor. IMC 0612 Appendix B more-than-minor screening questions ask if 

a performance deficiency has “the potential to lead to a more significant 

safety concern” and “adversely affect the associated cornerstone 

objective.”4 Some inspectors could not readily define, based on the 

guidance, which aspects of a performance deficiency have potential to 

cause a more significant safety concern. Some inspectors said they are 

left to rely on past experience to get through the questions and expressed 

concern that the questions do not provide a clear and consistent approach 

to determining whether an issue is minor or more-than-minor. 

 4 Recently, NRC initiated an effort staffed by headquarters and regional representatives, to develop several options 

to improve consistency in the more-than-minor screening process.

What Is Required

What We Found
Audit of NRC’s Significance Determination Process for Reactor Safety

8

Figure 2. More-than-Minor Screening Questions

Source: NRC Inspection Manual

IMC 0612 Appendix B Screening Questions Are Unclear

Unclear issue screening questions have made it difficult for inspectors to 

determine whether issues should be categorized as either minor or morethan-minor. The questions promote a large degree of subjectivity in the 

more-than-minor determination process.5 Some inspectors noted it is 

unclear which conditions have potential to lead to a more significant safety 

concern. Additionally, inspectors and managers explained that nearly any 

performance deficiency can be presented to adversely affect the 

associated cornerstone objective because the cornerstone objectives are 

broad.

Unclear Screening Questions Could Lead to Use of Unnecessary 

Resources and Inconsistency 

Unclear issue screening questions could cause inspectors to needlessly 

spend time or resources formally documenting minor issues and miss 

 5 NRC management noted that results from the more-than-minor screening process often vary given specific 

circumstances associated with individual issues. 

Why This Occurred

Why This Is Important
Audit of NRC’s Significance Determination Process for Reactor Safety

9

opportunities to document more significant issues. Inspectors and 

managers explained that excessively long periods of time can be spent 

deliberating whether a performance deficiency is minor or more-thanminor according to the IMC Appendix B screening questions. Inspectors 

said that the time spent determining if a performance deficiency is minor 

or more-than-minor can take away time and resources that could be better 

used toward other inspection activities.

NRC also risks the potential for inconsistent screening of performance 

deficiencies and inspectors might miss opportunities to document issues 

in inspection reports and the ability to share potentially similar findings 

across regions.

Recommendation

OIG recommends that the Executive Director for Operations

2. Clarify IMC 0612 Appendix B issue screening questions so that 

they are readily understood and easily applied.

C. Unable To Verify Independent Audits

NRC is unable to provide documentation of independent audits performed 

on greater than Green inspection findings because NRC does not have 

controls in place to ensure the audits are performed and documented. As 

a result, NRC risks inaccurately presenting agency activities in agency 

reporting documents and could miss opportunities to implement needed 

programmatic changes identified by audits.

Independent Audits of Greater than Green Inspection Findings 

Should Be Performed

Inspection Manual Chapter 0307 Appendix A, “Reactor Oversight Process 

Self-Assessment Metrics,” requires independent audits of greater than 

Green inspection findings to be performed to ensure significance 

determination results are predictable and repeatable. One metric, 

“Predictability and Repeatability of Significance Determination Results,” is 

What Is Required
Audit of NRC’s Significance Determination Process for Reactor Safety

10

evaluated based on an independent auditor’s ability to conclude that the 

finding’s significance characterization is “reasonably justifiable from both 

programmatic and technical positions.” Appendix A adds that appropriate 

programmatic changes will be implemented if the independent auditor 

finds the significance characterization is not reasonably justifiable.

NRC Cannot Confirm Independent Audits Are Performed

NRC is unable to provide documentation of independent audits being 

performed on greater than Green inspection findings. NRC’s annual selfassessment of the ROP for calendar year 2015 stated all greater than 

Green inspection findings had reasonably justifiable significance 

characterizations “as verified by an independent auditor using available 

documentation.” However, NRC management responsible for overseeing 

the ROP self-assessment was not able to provide documentation of the 

audits.

No Controls for Independent Audit Performance and Documentation

NRC does not have controls in place to ensure audits of greater than 

Green inspection findings are performed and documented. Some NRC 

staff and managers involved in the SDP were not familiar with the 

predictability and repeatability metric or its requirement that an 

independent audit be performed. Some of those who were aware of the 

metric and independent audit requirement were unable to specify who had 

been responsible for conducting the audits in the past or verify whether 

these audits had been performed at all. Neither staff nor management 

could specify where documentation of independent audits could be found.

What We Found

Why This Occurred
Audit of NRC’s Significance Determination Process for Reactor Safety

11

Risk of Inaccurate Presentation and Missed Opportunities For 

Improvement

NRC risks inaccurately presenting agency activities in agency reporting 

documents, and could miss opportunities to implement needed 

programmatic changes identified by audits. The lack of audit 

documentation could lead to inaccurate presentation of agency activities 

because the information reported cannot be verified. Additionally, NRC 

might miss opportunities to improve the SDP because documentation of 

needed programmatic changes is not available to staff.

Recommendations

OIG recommends that the Executive Director for Operations

3. Implement controls to ensure independent audits of greater than 

Green inspection findings are performed.

4. Document independent audits of greater than Green inspection 

findings.

Why This Is Important
Audit of NRC’s Significance Determination Process for Reactor Safety

12

OIG recommends that the Executive Director for Operations

1. Assess SDP workflow, and establish, communicate, and document 

clear and consistent expectations for staff and managers to 

complete their roles in the SDP.

2. Clarify IMC 0612 Appendix B issue screening questions so that 

they are readily understood and easily applied.

3. Implement controls to ensure independent audits of greater than 

Green inspection findings are performed.

4. Document independent audits of greater than Green inspection 

findings.

 IV. CONSOLIDATED LIST OF RECOMMENDATIONS
Audit of NRC’s Significance Determination Process for Reactor Safety

13

An exit conference was held with the agency on August 29, 2016. Agency 

managers reviewed a discussion draft and provided comments that have 

been incorporated into this report, as appropriate. During the exit 

conference, and in post exit discussions, agency managers agreed with 

the report’s recommendations. Additionally, agency managers opted to 

provide additional formal comments on the findings. These comments can 

be found in Appendix B.

 V. AGENCY COMMENTS
Audit of NRC’s Significance Determination Process for Reactor Safety

14

Appendix A

Objective

The audit objective was to assess the consistency with which NRC 

evaluates power reactor safety inspection findings under the SDP.

Scope

The audit focused on evaluating procedures used to determine the safety 

significance of inspection findings. We conducted this performance audit 

from February 2016 through June 2016. Internal controls related to the 

audit objective were reviewed and analyzed. Throughout the audit, 

auditors were aware of the possibility of fraud, waste, and abuse in the 

program.

Methodology

OIG reviewed relevant criteria for this audit, including the NRC Inspection 

Manual and Government Accountability Office’s Standards for Internal 

Control in the Federal Government.

To understand how NRC staff and managers write, interpret, and 

implement procedures used to characterize the significance of findings, 

OIG reviewed additional sources such as SDP timeliness and resource 

data, guidance, SDP improvement initiatives, self-assessments, and SDP 

tracking tools. OIG also interviewed NRC staff and managers from the 

Office of Nuclear Reactor Regulation and all regions.

We conducted this performance audit in accordance with generally 

accepted government auditing standards. Those standards require that 

we plan and perform the audit to obtain sufficient, appropriate evidence to 

provide a reasonable basis for our findings and conclusions based on our 

audit objectives. We believe that the evidence obtained provides a 

reasonable basis for our findings and conclusions based on our audit 

objectives. 

 OBJECTIVE, SCOPE, AND METHODOLOGY
Audit of NRC’s Significance Determination Process for Reactor Safety

15

The audit was conducted by Paul Rades, Team Leader; Levar Cole, Audit 

Manager; John Thorp, Senior Technical Advisor; Larry Vaught, Auditor; 

Anna Boyle, Management Analyst; and Urvi Banerjee, Student 

Management Analyst.
Audit of NRC’s Significance Determination Process for Reactor Safety

16

Appendix B

Agency Formal Comments
Audit of NRC’s Significance Determination Process for Reactor Safety

17
Audit of NRC’s Significance Determination Process for Reactor Safety

18
Audit of NRC’s Significance Determination Process for Reactor Safety

19
Audit of NRC’s Significance Determination Process for Reactor Safety

20

Please Contact:

Email: Online Form

Telephone: 1-800-233-3497

TDD 1-800-270-2787

Address: U.S. Nuclear Regulatory Commission

Office of the Inspector General

Hotline Program

Mail Stop O5-E13

11555 Rockville Pike

Rockville, MD 20852

If you wish to provide comments on this report, please email OIG using this link.

In addition, if you have suggestions for future OIG audits, please provide them using 

this link.

 TO REPORT FRAUD, WASTE, OR ABUSE

 COMMENTS AND SUGGESTIONS