Page 1 of 83 

Web-based Course on NUREG-1855

Guidance on the Treatment of Uncertainties 

Associated with PRAs in Risk-Informed Decisionmaking

Text-Only Version

Last Updated: April 12, 2017
Page 2 of 83 

Title Slide

Welcome to the web-based course on NUREG-1855, titled “Guidance on the Treatment 

of Uncertainties Associated with PRAs in Risk-Informed Decisionmaking.”

This course is divided into 18 modules. Each module contains 1-2 main level slides 

which are designated with a letter A following the title number. Further details supporting 

the primary information on the main slides is provided in slides with letter B, C, D, etc. 

following the slide number.

U.S. NRC Logo:

United State Nuclear Regulatory Commission

Protecting People and the Environment

EPRI Logo:

Electric Power Research Institute
Page 3 of 83 

Slide 1A - Need for Guidance

1 Why is this guidance needed and what is the purpose of NUREG-1855 (and the 

associated EPRI Reports)? 

This course is needed because uncertainties associated with the risk analysis need to be 

recognized and understood so that the decision maker is able to make a sound, riskinformed decision.

A risk-informed decision relies on quantitative risk acceptance guidelines that are used to 

compare with the risk results from the licensee’s risk analysis, for example, a 

Probabilistic Risk Assessment.

There are, however, uncertainties associated with the PRA model, and therefore, with 

the results.

For example, the decision maker needs to: understand to what extent the results are 

impacted by the uncertainties understand whether there are uncertainties that are 

impacting the results that may challenge the acceptance guidelines and determine if the 

driver for the large uncertainties can be identified and remediated.

Guidance is needed for how the applicant will address uncertainties and how the staff 

will integrate the treatment into their risk-informed decisionmaking process.

The primary objective of NUREG-1855 and the associated EPRI reports is to provide 

guidance on how uncertainties associated with the PRA results are treated in the Riskinformed Decisionmaking process. 

NUREG-1855 and the EPRI reports can be found here:

NUREG-1855, “Guidance on the Treatment of Uncertainties Associated with PRAs in 

Risk-Informed Decision Making,” [https://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1855/]

EPRI Report 1013492, “Probabilistic Risk Assessment Compendium of Candidate 

Consensus Models,” 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

13492/] EXIT: This link is external to the NRC website

EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic 

Risk Assessments,” 
Page 4 of 83 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

16737/] EXIT: This link is external to the NRC website

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 5 of 83 

Slide 1B - Associated EPRI Reports

The associated EPRI reports provide guidance on how uncertainties associated with 

PRA results are treated in the Risk-informed Decisionmaking process.

Links to the EPRI reports:

EPRI Report 1013492, “Probabilistic Risk Assessment Compendium of Candidate 

Consensus Models,” 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

13492/] EXIT: This link is external to the NRC website

EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic 

Risk Assessments,” 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

16737/] EXIT: This link is external to the NRC website

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website

Link to NUREG-1855:

NUREG-1855, “Guidance on the Treatment of Uncertainties Associated with PRAs in 

Risk-Informed Decision Making,” [https://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1855/]
Page 6 of 83 

Slide 2A – Course Learning Objectives

2 What should you learn from this course?

The purpose of this course to provide a high level understanding of the content in 

NUREG-1855 and the associated EPRI reports. After completing this course, you will be 

able to identify:

• The importance of uncertainties in the decisionmaking process

• Key concepts associated with risk-informed decisionmaking

• Types of uncertainties and their impact on PRA results and

• The approach used to treat PRA uncertainties in the decisionmaking process 

This course is not intended to provide guidance on how to:

• Implement the detailed guidance contained in NUREG-1855 (and the associated 

EPRI reports)

• Quantify the uncertainties in developing the PRA model
Page 7 of 83 

Slide 2B - More Information on Uncertainties in PRA

You should also gain an appreciation for the following:

• Decision-making on complex issues is always subject to uncertainty. 

• Underlying uncertainties exist whether or not it is acknowledged.

• The purpose of treating uncertainties in PRA is to ensure that the decisionmaker 

clearly understands whether the PRA uncertainties have:

• been adequately addressed in the application, and 

• the potential to affect the decision.

• PRA complements deterministic decision criteria by providing structure to the issue of 

uncertainty.

• When appropriately addressed, the uncertainties analysis adds technical support and 

value to risk-informed decisionmaking.

• The analysis of uncertainties in risk models allows the decisionmaker to make a more 

robust decision.
Page 8 of 83 

Slide 3A - Background Information

To best benefit from this course, it is useful to have a foundational knowledge of riskinformed decisionmaking. 

There are some key concepts that you should understand before taking this course.

Please answer each of the following five questions in order to explore and review these 

concepts.

Question #1 - PRA Models

What does a PRA model do? Choose all that apply:

A) Examines a spectrum of possible outcomes with a discrete set of representative 

scenarios.

B) Ignores the impact of modeling assumptions to produce a best estimate risk outcome.

C) Creates a new source of uncertainty that needs to be accounted for in a 

decisionmaking process. 

The correct answer is: A. Examines a spectrum of possible outcomes with a 

discrete set of representative scenarios.

• Fundamentally, a PRA is a model that represents the accident sequences which 

could occur as a result of the various events (initiating events) that could challenge 

the normal operation of the plant. 

• A PRA model is based on engineering (deterministic) analyses, such as the 

identification of system success criteria (e.g., based on thermo-hydraulic analyses), 

containment capacity (e.g., based on structural analysis) and plant specific 

information so that the model represents the as-built and as-operated plant.

• The PRA has probabilistic models whose parameters (e.g., initiating event 

frequencies, component failure rates, etc.) are estimated based on available 

knowledge or data.
Page 9 of 83 

• The PRA results provide insights on both what is important in causing the risk and 

what can reduce the risk, and provide insights to the magnitude (potential impact) 

of parameter, model, and completeness uncertainties.

• A full-scope PRA addresses three levels of risk (i.e., core damage, radiological 

releases, and consequences) from both internal and external hazards, and all 

operating states (at-power, low power and shutdown) 

• Not all risk-informed activities require a full-scope PRA. The needed scope is 

dependent on the specific decision/application under consideration.

• The acceptability of the base PRA model is determined by the current state-ofpractice (as codified in the PRA Standard and as endorsed by Regulatory Guide 

1.200). 

Link to Regulatory Guide 1.200:

Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities 

[https://www.nrc.gov/docs/ML0904/ML090410014.pdf]

Question #2 - Difference between Risk-informed and Risk-based

Risk-based and Risk-informed decisionmaking methods use the same information to 

make a decision. 

A) True

B) False

The correct answer is: FALSE

• A "risk-based" approach to regulatory decisionmaking is an approach in which 

such decisionmaking is solely based on the numerical results of a risk assessment. 
Page 10 of 83 

• A "risk-informed" approach to regulatory decisionmaking is an approach in which 

decisionmaking is not solely based on the numerical results of a risk assessment. 

A risk-informed approach takes into account risk insights (including the numerical 

results) along with other factors such as defense-in-depth, monitoring, safety 

margins. 

Question #3 - Risk-informed Decisionmaking Principles

What are the principles, as outlined in Regulatory Guide 1.174, in the risk-informed 

decisionmaking process? Choose all that apply: 

A) Regulation is met or explicit rule exemption

B) Consistent with defense in depth

C) Maintain safety margins

D) Performance monitoring

E) Deterministic criteria

F) As low as is reasonably achievable

G) Risk increases are small and consistent with safety goal

H) Minimization of risk

The correct answer is:

There are five principles constituting the integrated decisionmaking process: 

• Regulation is met or explicit rule exemption

• Consistent with defense in depth

• Maintain safety margins

• Performance monitoring

• Risk increases are small and consistent with safety goal
Page 11 of 83 

The existence of uncertainties does not stop us from using the results of a risk 

assessment (or a deterministic assessment) in our decisionmaking. We employ the 

other elements of the risk-informed decisionmaking process to address the PRA 

uncertainties that can impact the decision. See Regulatory Guide 1.174 for more about 

the risk-informed decisionmaking principles. PRA results are only one portion of a riskinformed decisionmaking process. All the principles are used in the decisionmaking

process.

Link to Regulatory Guide 1.174:

Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities 

[https://www.nrc.gov/docs/ML0904/ML090410014.pdf]

Question #4 - PRA Uncertainties

Engineering analyses account for uncertainties. 

A) True

B) False

The correct answer is: TRUE

Knowledge is imperfect regardless of how an analysis is conducted. Uncertainties exist 

in both engineering and risk analyses. Consequently, uncertainties from both 

engineering and risk analyses, need to be addressed in the decisionmaking process. 

However, as opposed to deterministic analyses, PRAs make the impact of uncertainties 

on the results explicitly observable by accounting for uncertainties in the quantification of 

the risk metrics.

Definition of Uncertainty:

Uncertainty is the understanding that knowledge is imperfect, and this imperfect 

knowledge needs to be considered in decisionmaking.

Question #5 – Risk-informed Terminology
Page 12 of 83 

Which of the following statements are false? Choose all that apply:

A) A frequency and probability have the same meaning. 

B) Uncertainties are only associated with our ability to accurately predict the likelihood of 

equipment performance.

C) A risk analysis and a PRA are identical.

The correct answer is: All the answers are false.

• A frequency and probability DO NOT have the same meaning. Frequency is a rate 

of occurrence of an event, and probability is an estimate of the likelihood that an 

event (for example 2 heads in 5 coin tosses) will happen at any given time. 

• Uncertainties are NOT only associated with our ability to accurately predict the 

likelihood of equipment performance. There are uncertainties, for example, in our 

ability to predict human performance, phenomena, hazard frequency and 

magnitude. 

• A risk analysis and a PRA are NOT identical. A PRA is one type of risk analysis. 

A risk analysis can be a qualitative or quantitative analysis of the risk metric(s). A 

PRA, however, is a type of quantitative analysis of the risk metric(s).

Definitions for the risk terms used in this course (and used in NUREG-1855 and the 

associated EPRI reports), can be found in NUREG-2122 (“Glossary of Risk-Related 

terms in Support of Risk-Informed Decisionmaking”). This glossary is a great reference 

for you as a risk-informed decision maker! 

Link to NUREG-2122:

NUREG-2122, “Glossary of Risk-Related terms in Support of Risk-Informed 

Decisionmaking,” [http://pbadupws.nrc.gov/docs/ML1215/ML121570620.pdf]
Page 13 of 83 

Slide 4A – Decisionmaker Guidance

4 What type of guidance is provided to a decision maker?

This course explains the guidance that is provided for addressing PRA uncertainties as 

part of the risk-informed decision making process. 

It is important to have a basic understanding of this process.

This decisionmaking process has five main elements. 
Page 14 of 83 

Slide 4A (Continued) - Risk-informed Decisionmaking Process 

Let’s take a closer look at the risk-informed decisionmaking process. 

First, the decision under consideration is defined.

Secondly, the regulatory requirements that can be impacted by the decision are 

identified, including assessing how the ability to meet the requirement may be impacted. 

Third, a risk analysis is performed to evaluate how the decision may impact the risk to 

public health and safety from the nuclear power plant. 

This analysis includes assessing the PRA uncertainties in two different ways, both from a 

deterministic perspective and from a probabilistic perspective. These perspectives 

require assessing the decisions impact on defense in depth and safety margins. Both 

perspectives require an assessment of uncertainties in their own way. 

This course is focused on the probabilistic perspective. 

Fourth, identify how monitoring the performance of the impacted equipment or operation 

would be performed. The assessment of uncertainties be incorporated into the 

implementation and monitoring program and that program should be accounted for in the 

assessment of uncertainties.

Fifth, the decision maker considers the PRA uncertainties along with the other elements 

as part of the risk-informed decisionmaking process, which includes an assessment of 

the uncertainties.
Page 15 of 83 

Slide 5A – Uncertainties Treated Differently

5 Uncertainties exist in both deterministic and probabilistic analyses, are they 

treated differently?

Uncertainties exist in both deterministic and probabilistic analyses. 

However, each of these approaches addresses uncertainties differently.

Deterministic decisionmaking addresses the uncertainties using calculations that are 

deliberately conservative to establish absolute criteria.

In this approach, the possibility that structures, systems and components may operate 

past their regulatory limit is generally not considered by the decisionmaker.

Probabilistic decisionmaking addresses the uncertainties using calculations that are 

intended to be realistic to establish a range of possible outcomes that are factored into 

the decisionmaking process. 

In this approach, the possibility that structures, systems and components may operate 

past their regulatory limit is generally considered by the decision maker. 
Page 16 of 83 

Slide 5B – Deterministic Approach

Deterministic Approach

The deterministic approach addresses uncertainties by generally establishing 

conservative bounds (e.g., safety margins) to compensate for the uncertainties. For 

example, a battery is considered depleted once the established regulatory limit has been 

exceeded (although the limit is considered to be conservative with a certain amount of

margin). 
Page 17 of 83 

Slide 5C – Probabilistic Approach

Probabilistic Approach

The probabilistic approach addresses uncertainties by establishing a range of possible 

outcomes with associated probabilities for each of the various outcome. In this example, 

the battery is considered depleted at different limits where there is a level of confidence 

established for each limit. For an example where a power from a battery is needed to 

ensure system functionality, at limit “x,” there is 5% chance of depletion, at limit “y,” there 

is an average chance of depletion, and at limit “z,” there is a 95% chance of depletion. A 

decisionmaker will likely be more comfortable at the “x” limit than the “z” limit, but there is 

not hard transition point between acceptability and unacceptability. 
Page 18 of 83 

Slide 6A – Types of Uncertainties

6 There are different types of uncertainties associated with PRAs: What are they?

Fundamentally there are three types of uncertainties associated with PRAs that reflect 

our imperfect knowledge.

Completeness uncertainties: Addresses the confidence in our knowledge (or ability) that 

all the risk significant events and physical phenomenon have been identified and 

included in the risk analysis. 

Parameter uncertainties: Addresses the confidence in our state of knowledge (or ability) 

to accurately predict the frequency of an event, the probability of equipment successfully 

performing, or the probability of a successful human action. 

Model uncertainties: Addresses the confidence in our state of knowledge (or ability) to 

accurately model the performance of structures, systems or components under adverse 

conditions.

given our lack of understanding of the conditions (phenomenon) being modeled or the 

nature of the possible mechanisms causing the structures’, systems’ and components’ 

failure. 

EPRI Report 1016737 provides a list of uncertainties for internal events 

EPRI Report 1026511 provides a list of uncertainties for fire, seismic, low-power 

shutdown and level 2.

Link to EPRI Report 1016737:

EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic 

Risk Assessments,” 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

16737/] EXIT: This link is external to the NRC website

Link to EPRI Report 1026511:

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 19 of 83 

Slide 6B – Completeness Uncertainties Example

Completeness uncertainties are those associated with phenomena or failure modes that 

are not included in the PRA model. Some may not be included because there are no 

analytical methods for dealing with them. Others are simply because we don’t know 

about them; they are the “unknown unknowns”. For example, modeling of degradation 

mechanisms of a material is based on an understanding of what the underlying 

environmental conditions are for that material (e.g., temperature, chemical environment, 

etc.). If the possible range of those conditions are not completely identified, then 

important degradation mechanisms can be missed. The Davis-Besse plant found severe 

degradation of the reactor vessel head as a result of prolonged exposure to concentrated 

boric acid. Operating experience reveals instances in which corrosion rates were 

significantly underestimated for identified boric acid leaks because of erroneous 

assumptions regarding the nature of the leakage, environmental conditions, the 

relationship between the actual leakage and experimental data, or other factors. As a 

consequence, in some instances, carbon steel components have been corroded to a 

much greater extent than anticipated.
Page 20 of 83 

Slide 6C – Parameter Uncertainties Example

The various parameters used to quantify the PRA model, including the initiating event 

frequencies, component failure probabilities, and human error probabilities cannot be 

known with certainty. This uncertainty is characterized by a probability distribution on the 

parameter value. For an example from EPRI 1026511, the frequency of a large 

earthquake is a parameter that can have substantial uncertainty around it. Generally, 

very large earthquakes are rare events, so there is not sufficient data to produce a 

statistically robust frequency estimate. Because we do not know the true frequency, we 

assign an uncertainty distribution to capture the level of confidence in that estimate.
Page 21 of 83 

Slide 6D – Model Uncertainties Example

• Model uncertainties result from a lack of knowledge of how structures, systems and 

components behave under the conditions arising during the development of an 

accident. Model uncertainties can arise because the phenomenon being modeled 

itself is not completely understood. For some phenomena, some data or other 

information may exist, but it needs to be interpreted to infer behavior under 

conditions different from those in which the data were collected. Model 

uncertainties may also arise because the nature of the failure mode is not 

completely understood or is unknown.

• An example of model uncertainty, taken from EPRI 1026511, is the different 

models used to predict core melt progression. MELCOR and MAAP are two 

models used to determine the core melt progression for severe accidents. These 

models differ in some fundamental assumptions in how the underlying phenomena 

are treated (e.g., to what extent can solid debris block a flowpath, how particulate 

debris move, how crust formation effects movement of corium through the reactor 

vessel, etc.) due to limits in our basic understanding of the physics and 

computational limitations. While both codes are benchmarked against available 

data, and both perform well on those cases, there is limited data that might resolve 

these differences. These underlying differences in capturing the relevant 

phenomena can translate to differences in the output (e.g., temperatures, amount 

of hydrogen generated, etc.) that is then used in the PRA.
Page 22 of 83 

Slide 7A - Overall Guidance in NUREG-1855

7 What is the overall process in NUREG-1855 and the associated EPRI reports?

The process for treating uncertainties in risk-informed decisionmaking is comprised of 

seven stages divided into three parts.

The first part provides guidance to both the licensee and the NRC staff and is only 

comprised of Stage A which addresses whether the guidance in NUREG-1855 and the 

associated EPRI reports should be used for the risk-informed decision under 

consideration. 

The second part provides guidance to the licensee and is comprised of Stage B through 

Stage F and represent the licensee’s process. 

In Stage B, guidance is provided for the licensee to assess whether the scope and level 

of detail of the PRA is adequate to support the application. 

In Stage C, guidance is provided to the licensee to assess the significance of the missing 

scope and missing level of detail. 

In Stages D and E, guidance is provided to the licensee on how to treat parameter and 

model uncertainties. Stages C through E are iterative. 

The results of stages C D and E are feed into Stage F, where guidance is provided to the 

licensee on the possible strategies for addressing the key uncertainties. 

Stage G provides a discussion of the approach used by the staff in their risk review of 

the licensee’s application referred to her as the NRC Decisionmaking process. Stage G 

focuses on the determination of:

• PRA Acceptability

• Appropriate Use of Bounding Analyses of Inadequate Scope

• Treatment of Parameter and Key Model Uncertainties

The NRC Decisionmaking process iterates with the Licensee process until a decision is 

reached. 

The guidance in Stages A through G are based on the NRC risk-informed 

decisionmaking process, and the treatment of uncertainties is consistent with the 

ASME/ANS PRA standard.
Page 23 of 83 

Slide 8A - Relation of NUREG-1855 and Associated EPRI Reports

8 How are the NUREG-1855 and associated EPRI reports related?

NUREG-1855 and the associated EPRI reports are designed to be used together in 

addressing uncertainties in a risk-informed application. They facilitate:

• Understanding the Application

• The role of the PRA Standard

• The role of the PRA Model

NUREG-1855 provides the overall guidance on the process and focuses on Stage A 

through G, while the associated EPRI reports provide detailed guidance on only stages B 

through F of the process.

The key contributions from the NUREG and EPRI reports are:

NUREG Report:

• Uncertainty from Non-Modeled Risk Contributors

• Characterization and Propagation of Parameters

• Identification of Key Model Uncertainties

EPRI Reports:

• Detailed and Approximate Methods for Parameters

• Identification and Characterization of the Model Uncertainties

• Detailed Example
Page 24 of 83 

Slide 8B - More Information on Uncertainties in PRA

EPRI 1026511 Steps in Relation to Stages of NUREG-1855, Rev. 1

The information below provides the process in EPRI 1026511 that a licensee should 

follow in preparing a risk-informed application, and how uncertainty is considered in each 

step of the process. The diagram also provides links between the licensee process and 

the steps in NUREG-1855.

Step 1 (NUREG-1855 Step B1): Define applications:

-Acceptance guidelines

-Scope of risk contributions

-Cause-effect relationship

Step 2 (NUREG-1855 Step B2): Assess adequacy of existing PRA to model cause-effect 

relationship

If the cause and effect relationships in the PRA are not adequate, refine model or 

redefine application (NUREG-1855 Step B3) and go back to step 2. 

If the cause and effect relationships in the PRA model it are adequate, the licensee 

needs to asses if they need a full scope PRA or if the existing PRA scope is adequate?

If the existing PRA scope is not adequate, then proceed to step 3. 

Step 3: Perform initial comparison with acceptance guidelines. Identify significant 

contributors and role of affected function(s).

Once step 3 is finished, conduct step 4.

Step 4 (NUREG-1855 Step C): Assess adequacy of existing PRA to model cause-effect 

relationship. 

If the cause and effect relationships in the PRA are not adequate, refine model or 

redefine application (NUREG-1855 Step B3) and go back to step 2. 
Page 25 of 83 

If the cause and effect relationships in the PRA are adequate, proceed to step 5. If the 

answer to the previously asked question, is the existing PRA scope adequate, was yes, 

then insights from EPRI 1016737 are incorporated and the licensee would move directly 

to step 5 from that question. 

Step 5 (NUREG-1855 Steps D and E): Comparison with acceptance guidelines and 

identify significant contributors

-Identify conservatisms

-Parameter uncertainty

-Model uncertainty

If the Acceptance Guidelines are exceeded (Section 5), then the licensee needs to refine 

the model or redefine the application and return to Step 5.

If the acceptance guidelines are not exceeded, proceed to Step 6. 

Step 6 (NUREG-1855 Steps F): Integrated decision-making

Link to EPRI Report 1026511:

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 26 of 83 

Slide 9A - Activities that Use NUREG-1855

9 When is the guidance in NUREG-1855 and associated EPRI reports used?

Stage A provides guidance to both the licensees and the NRC staff for determining 

whether the detailed guidance in NUREG-1855 and the associated EPRI reports should 

be used for the risk-informed application under consideration. 

The goal of this stage is to determine whether the results from a PRA are used in the 

risk-informed application and to determine how the PRA results are to be used to support 

the decision. 
Page 27 of 83 

Slide 9B - Stage A Process

Stage A involves two steps: 

Step A-1 determines the type of risk metric results used in the application. If the risk 

metric is independent of the PRA, treatment of uncertainties is addressed through a 

different process than NUREG-1855 (See Section 3.4). If the risk metric is PRA 

dependent, proceed to step A-2.

Step A-2 determines how the PRA results are used to support the decision requested in 

the application. If the PRA results are continuous or immediately evaluated, treatment of 

uncertainties is addressed through a different process than NUREG-1855 (See Section 

3.4). If the application of the PRA results is a regulatory submittal, proceed to step B, 

assessing the PRA Scope and Level of Detail.

Although not all decisions explicitly use the guidance in NUREG-1855 and the 

associated EPRI reports, the guidance in these documents can be used in a generic 

fashion as described in the next screen.

For those decisions that will use the guidance in NUREG-1855 and the associated EPRI 

reports, the analyst then follows Stages B through F.
Page 28 of 83 

Slide 9B – Step A-1: Type of Results

In this step, the analyst determines if the risk results used in the application are 

dependent on development of a PRA, either by evaluating changes to the core damage 

frequency and/or large early release frequency values or by utilizing importance 

measures to gain insights into risk significant components or actions. Some example 

risk-informed activities include: 

• Component Risk Ranking (e.g., motor- operated valves) -- Utilizes the FussellVesely importance measure (IM) calculated from a PRA to rank components.

• Regulatory Guide 1.174 (licensing basis changes) -- Uses core damage frequency 

and large early release frequency calculated from the baseline PRA and the 

potential change as a result of the proposed modifications to the plant design or 

operation reflected in the PRA.

• 10 CFR 50.69 (special treatment of structures, systems and components) -- Uses 

Fussell-Vesely and risk achievement worth importance measures calculated from a 

PRA. Also uses the core damage frequency and large early release frequency 

calculated from the baseline PRA and the potential change as a result of the 

proposed modifications to the plant design or operation reflected in the PRA.

• Notice of Enforcement Discretion (NOED) – Uses an incremental conditional core 

damage probability.
Page 29 of 83 

Slide 9C – Step A-2: How Results Are Used

In this step, the analyst determines how the application, e.g., decision, uses the PRA 

results based on the following criteria:

1. The risk metrics from the PRA are continuously being evaluated such that at any 

time, the risk is known – not subject to the detailed approach discussed in NUREG1855. Example answers for this criteria for various activities would be: Component 

risk ranking, no; Reg Guide 1.174, no; 10 CFR 50.69, no; NOED, no. 

2. The decision under consideration is based on reviewing the PRA risk results 

against specified regulatory acceptance guidelines – subject to the detailed 

approach discussed in NUREG-1855. Example answers for this criteria for various 

activities would be: Component risk ranking, only importance measures; Reg 

Guide 1.174, yes; 10 CFR 50.69, yes; NOED, possibly.

3. The risk significance of a decision is being evaluated at the time of occurrence of 

an event and there is insufficient time to evaluate the impact of uncertainties – not 

subject to the detailed approach discussed in NUREG-1855. Example answers for 

this criteria for various activities would be: Component risk ranking, no; Reg Guide 

1.174, no; 10 CFR 50.69, no; NOED, yes.
Page 30 of 83 

Slide 9D – Generic Use

Stage A starts with a determination of the type of risk-informed activity that is being 

evaluated (non-risk-informed activities are outside the scope of this report) and the type 

of risk metric results used in the application. If the risk metric results do not depend on 

the development of a PRA, the detailed approach in NUREG-1855 does not apply. Next, 

with regard to how the PRA is being used in the decision, if the activity involves a 

continuous evaluation of risk or an evaluation of risk at the time of an event, the detailed 

approach in NUREG-1855 does not apply. However, the detailed approach in NUREG1855 can be generically applied for all risk-informed activities.
Page 31 of 83 

Slide 10A – Generic Application of NUREG-1855

10 Is the guidance in NUREG-1855 (and associated EPRI 

Risk-informed activities not covered under Stage A still require an assessment of the 

uncertainties, but, perhaps to a lesser degree. 

For these activities, the guidance in NUREG-1855 is not applied verbatim; the guidance 

is generic and can be broadly applied.

This generic process is comprised of the following three steps:

1. Understanding the risk-informed activity.

2. Understanding the sources of uncertainties. 

3. Addressing the uncertainties in the decisionmaking. 
Page 32 of 83 

Slide 10B – Understanding the Risk-Informed Activity

• Need to identify what results are needed to evaluate the specific aspect of plant 

design or operation being assessed and how they are to be used to inform the 

decision.

• There can be occasions where qualitative risk insights are sufficient and thus a 

qualitative risk method could be sufficient to make a risk-informed decision.

• The level of detail of the risk assessment can vary because of choices in the 

modeling assumptions and approximations made in order to limit the need for 

potentially resource-intensive detailed analysis.
Page 33 of 83 

Slide 10C – Understanding Sources of Uncertainties

• The effect of parameter, model and completeness uncertainties on the risk 

assessment results must be identified to make proper risk-informed decisions.

• Identification and evaluation of these uncertainties for risk assessments other than

PRA follows the same approach as provided in Stages C, D and E.

• It is necessary to assess whether the uncertainties have the possibility of changing 

the evaluation of risk significantly enough to alter a decision.
Page 34 of 83 

Slide 10D – Addressing Uncertainties

• It is important to understand how uncertainties can affect the decision – the 

proximity of the mean value to a quantitative acceptance guideline can influence 

the depth of analysis required to make a decision. 

• The missing scope or level-of-detail items must be addressed in the decision 

process. Screening assessments can be used to show that the missing items are 

not important to the decision. In some cases, the risk-informed activity can be 

altered such that the missing scope items do not affect the decision process.

• The results of sensitivity studies performed to address model uncertainties can 

confirm that the acceptance guidelines are still met even when alternative 

assumptions are used (i.e., the conclusion generally remains the same with 

respect to the acceptance guideline).
Page 35 of 83 

Slide 11A – PRA Scope and Level of Detail

11 Does the missing PRA scope and level of detail matter for the application? 

Stage B provides guidance to the licensee to determine whether the PRA scope and 

level of detail is sufficient to support the risk-informed decision under consideration.

The goal of this stage is to determine whether the hazard groups, initiating events, 

component failure modes, accident sequences, human reliability analysis, etc. are 

sufficient to support the risk-informed decision under consideration.
Page 36 of 83 

Slide 11A (Continued) Stage B Process

Stage B, applicability of approach, consists of three steps: 

Step B-1 involves understanding the risk-informed application and decision. 

Step B-2 involves identifying the PRA scope and level of detail needed for the riskinformed application. 

Step B-3 involves assessing the PRA to determine if there is scope or level of detail 

needed for the risk-informed application that is missing from the PRA.

If the PRA structure is satisfactory for the PRA application, continue to stage D, 

assessing Parameter Uncertainty.

If the PRA structure is unsatisfactory for the PRA application, the licensee can redefine 

the application or conduct a screening analysis as defined in Stage C. Once completed, 

both these approaches will return the licensee to Step B-1. The licensee can also refine

the PRA and proceed to Stage D, assessing Parameter Uncertainty.

Associated EPRI report 1026511 provides additional guidance to the analyst on how to 

conduct this step.

Step 1 (NUREG-1855 Step B1): Define applications:

-Acceptance guidelines

-Scope of risk contributions

-Cause-effect relationship

Step 2 (NUREG-1855 Step B2): Assess adequacy of existing PRA to model cause-effect 

relationship

If the cause and effect relationships in the PRA are not adequate, refine model or 

redefine application (NUREG-1855 Step B3) and go back to step 2. 

If the cause and effect relationships in the PRA model it are adequate, the licensee 

needs to asses if they need a full scope PRA or if the existing PRA scope is adequate?

Link to EPRI Report 1026511:
Page 37 of 83 

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 38 of 83 

Slide 11B - Step B-1: Understanding Application and Decision

In this step, the analyst need to identify what structures, systems, and components and 

activities would be affected by a proposed application. In addition, a key aspect in this 

process is identifying the risk metrics needed to support the application.

Stage B

Step B-1: Understanding the risk-informed application and decision

Step B-2: Identify the PRA scope and level of detail needed for the risk-informed 

application

Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk 

change associated with the risk-informed application?
Page 39 of 83 

Slide 11C – Step B-2: Identify Scope and Level of Detail

In this step, the analyst needs to determine the cause-and-effect relationship between 

the application and the structures, systems and components and plant activities affected 

by the proposed risk-informed activity. This determination will then dictate the required 

scope of the PRA. It is then necessary to identify if the existing PRA contains the 

elements that would be affected by an application and are important to evaluating the 

risk measure(s) applicable to the decision.

Stage B

Step B-1: Understanding the risk-informed application and decision

Step B-2: Identify the PRA scope and level of detail needed for the risk-informed 

application

Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk 

change associated with the risk-informed application?
Page 40 of 83 

Slide 11D – Step B-3: Addressing Missing Scope and Level of Detail

In this step, the analyst, depending on whether the PRA does not contain the scope and 

level of detail needed to evaluate the risk change associated with a risk-informed 

application, chooses one of the following options: (1) upgrade the PRA to include the 

required scope and level of detail, (2) redefine the application such that the missing PRA 

scope or level of detail is not needed in the evaluation process, or (3) perform a 

conservative/bounding assessment of the missing items to determine if they are 

significant to the decision.

Stage B

Step B-1: Understanding the risk-informed application and decision

Step B-2: Identify the PRA scope and level of detail needed for the risk-informed 

application

Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk 

change associated with the risk-informed application?
Page 41 of 83 

Slide 12A - Completeness Uncertainties

12 How are completeness uncertainties addressed? 

Stage C provides guidance to the licensee on how to address completeness 

uncertainties; that is, how to address the scope and level of detail items that are not 

modeled in the PRA.

The goal of this stage is to determine whether the missing scope and level of detail are 

significant to the decision under consideration.
Page 42 of 83 

Slide 12A (Continued) - Stage C Process

If Stage B determines that screening is required, the licensee will enter Stage C. 

Stage C involves two major steps: In Step C-1, the Licensee Performs screening 

analyses to determine the significance of the missing PRA scope or level of detail to the 

risk-informed decision. If the missing scope or level of detail are determined to be nonsignificant contributors to the decision, the licensee can move to Stage D, assessing 

parameter uncertainty. 

In Step C-2, the Licensee determines if the PRA model needs to be updated or if the 

application needs to be modified to address the missing PRA scope or level of detail 

significant to the decision.

If there is not an NRC-endorsed PRA standard for the missing scope and level-of-detail, 

the licensee can proceed to Stage D. If a missing scope or level-of-detail item cannot be 

screened, the licensee will need to redefine the application by returning to Stage B. The 

licensee can also conduct upgrades per the PRA standard as endorsed in Regulatory 

Guide. 1.200, or submit the results of a conservative analysis, bounding analysis, or both 

to address the missing items. This upgrade process would allow the application to 

proceed to Stage D. 

Associated EPRI report 1026511 provides additional guidance to the analyst on how to 

conduct this step.

Step 3: Perform initial comparison with acceptance guidelines. Identify significant 

contributors and role of affected function(s).

Once step 3 is finished, conduct step 4.

Step 4 (NUREG-1855 Step C): Assess adequacy of existing PRA to model cause-effect 

relationship. 

Link to Regulatory Guide 1.200

Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities 

[https://www.nrc.gov/docs/ML0904/ML090410014.pdf]

Link to EPRI Report 1026511
Page 43 of 83 

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 44 of 83 

Slide 12B – Step C-1: Perform Screening

In this step, the analyst determines the risk significance of a missing scope or level of 

detail PRA item A screening analysis is performed to demonstrate that the particular 

non-modeled PRA scope or level-of-detail item can be eliminated from further 

consideration in the risk-informed decision. Screening analyses are either qualitative or 

quantitative in nature or a combination of both types.

If a missing scope or level-of-detail PRA item is not screened out after it is analyzed 

using a bounding, conservative, or limited realistic analysis, AND it is not significant to 

the risk-informed decision, then the results of that analysis can be incorporated into the 

application as an acceptable expansion of the PRA scope or level-of-detail. 

Stage C

Step C-1: Perform screening analyses for missing scope 

and level of detail items to assess significance

Step C-2: Determine strategy for addressing risk from each significant un-modeled scope 

or level-of-detail item 
Page 45 of 83 

Slide 12C – Step C-2: Determine Strategy

In this step, the analyst addresses the risk from each significant un-modeled scope or 

level-of-detail item using a PRA model that is developed in accordance with a consensus 

standard for that item that has been endorsed by the NRC staff. If there is no PRA 

standard that addresses the missing scope or level-of-detail item in question, the 

licensee can submit the results of the quantitative screening analysis as part of the input 

into the decisionmaking process. When this situation occurs, the analyst can then 

proceed to evaluate the parameter and model uncertainties in the context of the 

application. 

Stage C

Step C-1: Perform screening analyses for missing scope 

and level of detail items to assess significance

Step C-2: Determine strategy for addressing risk from each significant un-modeled scope 

or level-of-detail item 
Page 46 of 83 

Slide 13A - Parameter Uncertainties

Stage D provides guidance to the licensee on how to address the parameter 

uncertainties associated with the PRA that is used in support of risk-informed 

applications.

The goal of this stage is to calculate the PRA risk metrics determine how the results 

compare with the quantitative acceptance criteria and determine if the uncertainty of the 

PRA risk results, due to the underlying parameter uncertainties, impacts this comparison.
Page 47 of 83 

Slide 13A (Continued) – Stage D Process

Once completeness has been assessed in Stage C, the licensee will enter Stage D. 

Stage D involves three major steps:

Step D-1 characterizes the uncertainty in the quantification of the parameters in the basic 

events and other inputs of the PRA model.

Step D-2 quantifies the risk metrics, accounting for parameter uncertainty and the State 

of Knowledge Correlation (SOKC). 

Associated EPRI report 1016737 provides detailed guidance on the state-of-knowledgecorrelation.

Step D-3 compares the application risk metric results with the application acceptance 

guidelines. 

The ultimate purpose here is to determine the extent to which the mean value of the risk 

metric results may challenge the quantitative acceptance guidelines.

This determination includes an assessment of the significance of the state-of-knowledge 

correlation.

However, parameter uncertainty is only one type of uncertainty in the PRA results that 

need to be considered. 

Therefore, the next step in the process is consideration of model uncertainties. 

Associated EPRI report 1026511 provides additional guidance to the analyst on how to 

conduct this step. 

Link to EPRI Report 1026511

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website

Link to EPRI Report 1016737

EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic 

Risk Assessments,” 

[http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010

16737/] EXIT: This link is external to the NRC website
Page 48 of 83 

Slide 13B – Step D-1: Characterize Parameter Uncertainties

In this step, the analyst characterizes the parameter uncertainties in the form of an 

intervals (i.e., a range of probability values within which the actual input value lies). 

However, such a characterization will not lend itself to propagation of the parameter 

uncertainties through the PRA. Therefore, it is more typical (and necessary if uncertainty 

propagation is to be achieved) to characterize the uncertainties in terms of a probability 

distribution on the value of the quantity of concern.

Stage D

Step D-1: Characterize parameter uncertainty in the quantification 

of basic events and other PRA inputs.

Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation.

Step D-3: Compare the risk results with the acceptance guidelines.
Page 49 of 83 

Slide 13C – Step D-2: Quantify the Risk Metrics

In this step, the analyst quantifies the risk metrics, i.e., the output of the PRA, as well as 

assessing the uncertainties associated with the quantification. The uncertainties of the 

risk metric results is a consequence of the parameter uncertainty in the PRA input 

values. For the simplest approach, the risk metric can be quantified as a point estimate, 

in which case the uncertainty associated with the risk metric can only be expressed as 

an uncertainty interval. For most applications ultimately requiring an NRC decision, the 

recommended approach is to propagate the parameter uncertainties and calculate the 

risk metrics in order to obtain a probability distribution from which a mean value can be 

derived. The effect of the state-of-knowledge-correlation on the results needs to be 

included in the calculation or, if neglected, justified as being negligible.

Stage D

Step D-1: Characterize parameter uncertainty in the quantification 

of basic events and other PRA inputs.

Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation.

Step D-3: Compare the risk results with the acceptance guidelines.
Page 50 of 83 

Slide 13D – Step D-2: Compare to Acceptance Guidelines

In this step, the analyst compares the relevant risk metric results with the acceptance 

guidelines to be used for the particular application being considered. This comparison 

reveals if the acceptance guidelines are satisfied using the mean value of the risk metric 

results, arising from the propagation of the uncertainties in parameter values of the PRA 

inputs.

Stage D

Step D-1: Characterize parameter uncertainty in the quantification 

of basic events and other PRA inputs.

Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation.

Step D-3: Compare the risk results with the acceptance guidelines.
Page 51 of 83 

Slide 14A – Model Uncertainties

14 How are model uncertainties treated?

Stage E provides guidance to the licensee for addressing sources of model uncertainties 

and associated assumptions related to the base PRA and the application PRA. 

The goal of this stage is to determine whether and the degree to which the risk metric 

results challenge or exceed the quantitative acceptance guidelines

Any such source of model uncertainty that could cause the risk metric results to 

challenge or exceed the acceptance guidelines are considered to be key.
Page 52 of 83 

Slide 14A (Continued) Stage E Process

Once parameter uncertainty has been assessed in Stage D, the licensee will enter Stage 

E. 

Stage E involves two major steps:

Step E 1 identifies any potential model uncertainties as a result in constructing the base 

PRA model.

Step E 2 identifies sources of model uncertainties key to the application.

Associated EPRI reports 1016737 and 1026511 provide guidance on identification and 

sources of model uncertainties and a generic list of sources of model uncertainties for 

internal events, internal floods, internal fires, seismic, low power and shutdown and Level 

2; additionally, associated EPRI report 1016737 provides guidance on performing 

sensitivity studies, including logical groupings of model uncertainties.

At this stage of the process, the applicant has identified the sources of uncertainties and 

has determined their impact on the results, thereby, the impact regarding the extent to 

which the acceptance guidelines are met. 

The applicant then develops a strategy to address any key uncertainties.

Once the existing PRA scope is determined to be adequate, then insights from EPRI 

1016737 are incorporated and the licensee would move directly to step 5 from that 

question. 

Referring back to the flow chart in EPRI 1026511, the following steps correspond to 

Stage E. 

Step 5 (NUREG-1855 Steps D and E): Comparison with acceptance guidelines and 

identify significant contributors

-Identify conservatisms

-Parameter uncertainty

-Model uncertainty

If the Acceptance Guidelines are exceeded (Section 5)? Then the licensee needs to 

refine the model or redefine the application and return to Step 5.
Page 53 of 83 

If the acceptance guidelines are not exceeded, proceed to Step 6. 

Step 6 (NUREG-1855 Steps F): Integrated decision-making

Link to EPRI Report 1026511

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 54 of 83 

Slide 14B - Step E-1: Identify Potential Model Uncertainties

In this step, the analyst identifies the sources of model uncertainties in the base PRA. 

The base PRA is reviewed to identify and characterize the sources of model 

uncertainties. Some sources may be generic, and some may be plant-specific. These 

sources of model uncertainties are those that result from developing the PRA. The 

sources of model uncertainties associated with the base PRA are reviewed to identify 

those that are relevant to the application under consideration; that is, some sources of 

model uncertainties may impact part of the PRA model that is not relevant to the 

application. New sources of model uncertainties that may be introduced by the 

application are also identified. This identification is based on an understanding of the 

type of application and the associated acceptance guidelines. 

Stage E

Step E-1: Identify potential model uncertainties and related assumptions and determine 

their significance.

Step E-2: Identify model uncertainties or related assumptions that are key to the 

decision.
Page 55 of 83 

Slide 14C – Step E-2: Identify Key Uncertainties

In this step, the analyst reviews the sources of model uncertainties that are relevant to 

the application to identify those that are key to the application. For the situation where 

the risk metric results from Stage D already challenges or exceeds the acceptance 

guidelines, it still is necessary to determine the potential significance of these sources of 

model uncertainties and related assumptions when they result in a further challenge to or 

exceedance of the acceptance guidelines. This review involves performing a 

quantitative analysis to identify the importance of each relevant source of model 

uncertainty. The relevant sources of model uncertainties that have the potential to 

challenge or exceed the acceptance guidelines are considered key for the application. 

Stage E

Step E-1: Identify potential model uncertainties and related assumptions and determine 

their significance.

Step E-2: Identify model uncertainties or related assumptions that are key to the 

decision.
Page 56 of 83 

Slide 15A Risk-informed Application Strategies

15 What strategy should licensees use in developing their risk-informed 

application?

Stage F provides guidance to the licensee on the process of developing a risk-informed 

application submittal. 

The goal of this stage is to help ensure that adequate justification is provided for the 

acceptability of the risk-informed application. 
Page 57 of 83 

Slide 15A (Continued) Stage F Process

The Application Development Process, Stage F, involves three options:

The first option consists of redefining the application.

The second option involves refining the PRA, while the third option uses compensatory 

measures or performance monitoring requirements. 

The three options may be used separately or together, and to varying degrees, when 

assessing the effect of uncertainties for a particular application. 

The licensee may decide sufficient justification has been provided, and therefore, not 

pursue any of the three options.

Consequently, this stage is performed in an iterative fashion. 

The licensee must ensure that: (1) adequate justification has been provided for the 

acceptability of the risk-informed application and (2) that the documentation provides a 

clear and concise presentation of this argument.

Associated EPRI report 1026511 provides additional guidance on how a licensee can 

provide integrated information to the decisionmaker and how feedback from Stage F can 

be incorporated into Stage B. Stage F and Stages C, D, and E will impact each other 

and must be coherent before moving onto Stage G, the NRC’s risk informed review 

process. 

Link to EPRI Report 1026511:

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 58 of 83 

Slide 15B – Option 1: Redefine the Application

Redefine the Application

In this option, the licensee elects to redefine the application, when a PRA model is 

incomplete in its coverage of significant risk contributors, the scope of a risk-informed 

application can be restricted to those areas supported by the PRA. The licensee 

must return to Stage B—as indicated by the dashed gray line—to determine scope of 

the redefined application and subsequently reassess the effect of all uncertainties via 

Stages C, D, and E. 

Stage F - Application Development Process

Option 1: Redefine Application

Option 2: Redefine PRA

Option 3: Compensatory Measures or Performance Monitoring
Page 59 of 83 

Slide 15C – Option 2: Redefine the PRA

In this option, the licensee elects to refine (i.e., upgrade) the PRA, and will either 

1. address the missing scope and level of detail impacted by the application, or 

2. upgrade the model of concern impacted by the application to reduce its 

uncertainty. 

The licensee must return to Stage D and then Stage E to recalculate the risk metrics, 

reassess the effect of model and parameter uncertainties, and determine the 

acceptability of the new risk measure results relative to the acceptance guidelines. 

Stage F - Application Development Process

Option 1: Redefine Application

Option 2: Redefine PRA

Option 3: Compensatory Measures or Performance Monitoring
Page 60 of 83 

Slide 15D – Option 3: Use Compensatory or Performance Monitoring

In this option, the licensee elects other means for justifying their application. 

Additional justification can involve, for example, compensatory measures and 

performance monitoring requirements for a given application. Compensatory 

measures can be used to neutralize the expected negative effect of some feature of 

plant design or operation on risk. Performance monitoring can be used to 

demonstrate that, following a change to the design of the plant or operational 

practices, there has been no degradation in specified aspects of plant performance 

that are expected to be affected by the change. This monitoring is an effective 

strategy when no predictive model has been developed for plant performance in 

response to a change.

Stage F - Application Development Process

Option 1: Redefine Application

Option 2: Redefine PRA

Option 3: Compensatory Measures or Performance Monitoring
Page 61 of 83 

Slide 15E - How a licensee can provide integrated information to the 

decisionmaker

Associated EPRI report (1026511) provides additional guidance on how a licensee 

can provide integrated information to the decisionmaker. 

Special topics addressed include:

• Development of compensatory measures and performance monitoring using 

insights from the uncertainties analysis

• Assessment of defense-in-depth

• Dealing with very large uncertainties

• Appendix A provides a complete example application 

Link EPRI Report 1026511:

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 62 of 83 

Slide 15F - Dealing with Large Uncertainties

-For some contributors, core damage frequency and large early release frequency) can 

be very large. The causes of large uncertainties are generally associated with: 

• There is little or no data to support its characterization although the potential 

for the challenge to the plant to occur is known (note rewrite of script)

• The need to extrapolate or interpolate from limited data for the particular use, 

or

• The need to depend on models of underlying phenomena for which there is 

no direct validation. 

-Large uncertainties can be manifested in PRA models and results in a variety of ways 

and can appear as uncertainties on parameters or in the model structure: 

• Characterization of the hazard

• Impact of the hazard

• Response to hazard

-The uncertainties may be so large that the analyst does not have confidence that the 

mean value resulting from the analysis is meaningful

-Defense-in-depth is recognized as a way of addressing uncertainties in both the 

traditional and risk-informed approaches

-The qualitative insights obtained from the understanding of the conditional core damage 

probability analyses provides an understanding of the margins built into the plant to 

enable a focus on those aspects for which defense in depth is weakest.
Page 63 of 83 

Slide 16A - NRC Staff Review Process

16 What process does the NRC staff use in their risk-informed review?

Stage G describes the process used by the staff for determining whether a licensee’s 

risk-informed application demonstrates an acceptable treatment of uncertainties and 

whether the proposed application represents an acceptable risk to the plant. 

The risk-informed review process described here only involves those aspects of the 

staff’s risk-informed review process that specifically relate to the treatment of 

uncertainties.

In determining whether the acceptance guidelines have been met, the staff seeks to 

answer the following general questions: 

1. Is the scope and level of detail of the PRA appropriate for the application? 

2. Is the PRA model acceptable? 

3. What is the proximity of the risk metric results to the acceptance guidelines?

4. How do parameter uncertainties affect the risk metric results? 

5. How do model uncertainties affect the risk metric results?

6. Is the acceptability of the application adequately justified? 

Note: The risk-informed review process described here only involves those aspects of 

the staff’s risk-informed review process that specifically relate to the treatment of 

uncertainties. Guidance on the other aspects of the staff’s risk-informed review process 

may be found in other application-specific guidance documents.
Page 64 of 83 

Slide 16B - Appropriateness of Scope and Level of Detail

The staff addresses the acceptability of the scope and level of detail by comparing the 

scope and level of detail required (as requested in the specific risk-informed activity staff 

guidance) to that in the licensee’s submitted application PRA. This comparison identifies 

those PRA scope and level-of-detail items that have either not been modeled in the PRA 

or have been intentionally excluded for that application. The staff reviews the screening 

analyses provided by the licensee to determine if there is adequate justification for the 

exclusion of PRA items that are missing from the application.
Page 65 of 83 

Slide 16C - PRA Acceptability

The acceptability of the base PRA is addressed in Regulatory Guide 1.200. If an 

unacceptability issue is identified, the staff then determines whether the issue is 

significant. If the staff determines that the issue is significant, the application may be 

rejected as a risk-informed application. If the issue is determined not to be significant, 

the staff continues with the risk-informed review process.
Page 66 of 83 

Slide 16D - The Proximity of the Risk Results to the Acceptance Guidelines

The staff first determines the numerical difference between the licensee’s PRA model 

risk metric results and the acceptance guideline, then subsequently determines if the 

proximity to the guidelines is significant (i.e., challenging or exceeding the guidelines). If 

the proximity is not significant, the staff continues on with the risk-informed review 

process. If the proximity is significant, the staff determines whether adequate 

justification has been provided for the acceptability of its proximity. The adequacy of the 

justification for the proximity may not be fully evident until the staff has reviewed the 

licensee’s treatment of both parameter and model uncertainties. 
Page 67 of 83 

Slide 16E - Parameter Uncertainties

The staff’s parameter uncertainty review addresses whether the licensee has: 

• Demonstrated an adequate understanding of the effect of parameter 

uncertainties (i.e., the significance of state-of-knowledge correlation) and 

• Provided adequate justification for the proximity of the PRA risk metric results 

to the acceptance guidelines. 

Since both parameter and model uncertainties may affect the application’s risk results, 

the staff may refrain from rejecting the application as a risk-informed application on the 

basis of the comparison of the mean value derived from the propagation of parameter 

uncertainties, until the review of the licensee’s treatment of model uncertainties has been 

performed.
Page 68 of 83 

Slide 16F - Model Uncertainties

The staff’s model uncertainty review addresses whether the licensee has: 

• adequately identified key sources of model uncertainties and related assumptions; 

• demonstrated an adequate understanding of the effect on the risk metric results 

from key model uncertainties or related assumptions; and 

• provided adequate justification for the acceptability of the comparison between the 

risk metric results and the acceptance guidelines such as qualitative arguments 

that the sensitivity studies that challenge the guidelines may be considered 

compensated for by plant practices not accounted for in the PRA model

The staff reviews the application’s technical bases and sensitivity analyses to 

determine whether the licensee has provided adequate justification for the 

acceptability of the effect of key sources of model uncertainties and related 

assumptions.
Page 69 of 83 

Slide 16G - Adequate Justification

The staff determines whether adequate justification has been provided by reviewing the 

licensee’s argument for the acceptability of the application, and may include, but is not 

limited to, a review of the application technical bases supporting analyses, compensatory 

measures or monitoring requirements, and other qualitative considerations.
Page 70 of 83 

Slide 17A - Aggregation Treatment

17 How are different levels of detail, conservatisms (e.g. aggregation) in the PRA 

treated?

The process for treating different levels of detail and conservatisms in the PRA involve 

the following:

Contributors are Combined -- meaning that for all applications, it is necessary to address 

all applicable hazards and plant operating states when quantifying the risk metrics.

Different Levels of Detail and Conservatisms -- addresses the fact that when combing 

the results of PRA models, the level of detail may differ from one hazard to the next or 

among the plant operating states, with some being more conservative than others.

Biasing of Results -- discusses the idea that significantly higher levels of conservative 

bias can exist in the level of detail and scope which may introduce varying levels of 

realism in the corresponding PRA results.

Proximity to Acceptance Guidelines -- details how the process of aggregation can 

influence the proximity of the risk result to the acceptance guidelines and therefore 

influence the risk-informed application.
Page 71 of 83 

Slide 17B - Contributors Are Combined

For all applications, it is necessary to consider the contributions from the applicable 

hazards and/or plant operational states when quantifying the risk metrics such as core 

damage frequency, large early release frequency, or an importance measure. Because 

the hazards and plant operating states are independent, addition of the contributions 

(i.e., aggregation of the results) is mathematically correct. However, issues such as 

varying level of detail in the different hazard or plant operational states analyses should 

be considered when combining the results from different hazards and plant operational 

states.
Page 72 of 83 

Slide 17C - Different Levels of Detail and Conservatisms

When combining the results of PRA models for several hazards (e.g., internal events, 

internal fires, seismic events) as required by many acceptance guidelines, the level of 

detail may differ from one hazard to the next with some being more conservative than 

others. The level of modeling can vary even within a hazard such as an internal event, 

at-power PRA. For example, the evaluation of room cooling and equipment failure 

thresholds can be conservatively evaluated leading to a conservative time estimate for 

providing a means for alternate room cooling while other evaluations are more realistic 

(e.g., defining the onset of core damage for a boiling water reactor occurring at 2’ above 

the bottom of the active fuel rather than at core uncovery which would be considered a 

more conservative evaluation). Moreover, at-power PRAs follow the same general 

process as used in the analysis of other hazards, with regard to screening: low-risk 

sequences can be modeled to a level of detail sufficient to prove they are not important 

to the results.
Page 73 of 83 

Slide 17D - Biasing of Results

-Significantly higher levels of conservative bias can exist in external hazards, low power 

and shutdown, and internal fire PRAs. These biases can result from several factors, 

including the unique methods or processes and the inputs used in these PRAs as well as 

the scope of the modeling. For example, the fire modeling performed in a fire PRA can 

use simple scoping models or more sophisticated computer models or a mixture of 

methods and may not mechanistically account for all factors such as the application of 

suppression agents. Moreover, in an effort to reduce the number of cables that have to 

be located, fire PRAs do not always credit all mitigating systems. To a certain level, 

conservative bias will be reduced by the development of detailed models and 

corresponding guidance for the analysis of external hazards, fires, and low power and 

shutdown that will provide a similar level of rigor to the one currently used in internal 

events at-power PRAs. However, as with internal events at-power PRAs, the evaluation 

of some aspects of these other contributors will likely include some level of conservatism 

that may influence a risk-informed decision.

-The level of detail, scope, and resulting conservative biases in the PRAs may introduce 

varying levels of realism in the corresponding PRA results. In addition, because the 

phenomena to be taken into account can be complex, the impact of model uncertainties 

on the PRA results can be larger for different risk contributors. However, these concerns 

do not preclude the aggregation of results from different risk contributors; but it does 

require that the decision-maker understand the differences in the confidence with which 

the significant contributors to the risk metrics are representative of the associated risk.
Page 74 of 83 

Slide 17E - Proximity to Acceptance Guidelines

The process of aggregation can be influenced by the type of risk-informed application. 

For example, it is always possible to add the core damage frequency or large early 

release frequency, or the changes in core damage frequency or changes in large early 

release frequency contributions from different hazards for comparison against 

corresponding acceptance guidelines. However, in doing so, one should always 

consider the influence of known conservatism when comparing the results against the 

acceptance guidelines, particularly if they mask the real risk contributors (i.e., distort the 

risk profile) or result in exceeding the guidelines. If the acceptance guidelines are 

exceeded due to a conservative analysis, then it may be possible to perform a more 

detailed, realistic analysis to reduce the conservatism and uncertainty. For applications 

that use risk importance measures to categorize or rank structures, systems and 

components according to their risk significance (e.g., revision of special treatment), a 

conservative treatment of a hazard can bias the importance measures calculated for that 

hazard. Moreover, the importance measures derived independently from the analyses 

for different hazards cannot be simply added together. An integration of different risk 

models would not provide a good set of integrated importance measures due to impact 

of different levels of conservatism in the various hazard analyses. Thus, risk-informed 

decisionmaking based on the use of importance measures will have to consider the 

uncertainty of the values from different hazard assessments.
Page 75 of 83 

Slide 18A – Treatment of Uncertainties

18 How are the PRA uncertainties treated in risk-informed decisionmaking?

The staff review of a risk-informed application begins with determining the proximity of 

the application risk metric results to the acceptance guidelines and the licensee’s 

justification. 

The justification needed to demonstrate the acceptability of a given risk-informed 

application, and therefore the extent of the staff review, is largely dictated by the 

proximity of the risk metric results to the acceptance guidelines.

In Regime 1, the risk metric results are well below the acceptance guidelines.

In Regime 2, the risk metric results are closer to, but do not challenge the acceptance 

guidelines. 

In Regime 3, the risk metric results challenge the acceptance guidelines.

In Regime 4, the risk metric results clearly exceed the acceptance guidelines. 

For each regime, NUREG-1855 provides guidance on considerations for decisionmakers 

when the risk metric results fall in that regime.

Associated EPRI report 1026511 provides guidance to licensees on what information to 

provide decisionmakers for the respective regimes in order to support that 

decisionmaking.

Link to EPRI Report 1026511:

EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment 

in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” 

[https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is 

external to the NRC website
Page 76 of 83 

Slide 18B - General Justification

The justification for a given application should be commensurate with the proximity of the 

risk metric results to the acceptance guidelines. In general, more justification will be 

needed for a given application when the risk metric results are closer to challenging or 

exceeding the acceptance guidelines than when the risk metric results are further below 

the acceptance guidelines.
Page 77 of 83 

Slide 18C - Regime 1

In Regime 1, the risk metric results are considered to be well below the acceptance 

guidelines when the mean value of the risk metric results are less than the acceptance 

guidelines by approximately one order of magnitude or more. 

The staff would look for an assessment which demonstrates that the state-of-knowledge 

correlation does not affect the risk metric results of the PRA, would also evaluate the 

application to determine whether the validity of the assumptions made in the application 

PRA will be appropriately monitored via the implementation of specific measures and 

criteria, would look to see whether degraded performance can be detected in a timely 

fashion and would likely place little importance on the licensee’s use of compensatory 

measures, depending on the justification that is provided. 
Page 78 of 83 

Slide 18D - Regime 2

In Regime 2, the application PRA risk metric results do not challenge the application 

acceptance guidelines and the mean value of the risk metric results are approximately 

within an order of magnitude of the acceptance guidelines.

The staff would look for an assessment which shows that the state-of-knowledge 

correlation does not affect the risk metric results of the PRA, would examine the 

application to ensure that the proposed performance monitoring is appropriate and 

adequate for the application and whether degraded performance can be detected in a 

timely fashion. The staff would examine the peer review (as discussed in Regulatory 

Guide 1.200) findings with a higher degree of scrutiny than for applications that fall into 

Regime 1 so as to better understand how particular findings were resolved as well as the 

general impact of the findings. In general, it is unlikely the staff would perform an audit 

on the application PRA for those applications that fall into Regime 2.

Link to Regulatory Guide 1.200:

Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities 

[https://www.nrc.gov/docs/ML0904/ML090410014.pdf]
Page 79 of 83 

Slide 18E - Regime 3

In Regime 3, the risk metric results are considered to challenge the acceptance 

guidelines when the mean value of the risk metric results are either slightly less than or 

slightly greater than the acceptance guidelines.

The staff expects that a quantitative assessment would be provided which shows that the 

state-of-knowledge correlation does not affect the rick metric results of the PRA. The 

staff would examine the application to ensure that the proposed performance monitoring 

is adequate and determine whether degraded performance can be detected in a timely 

fashion, would expect compensatory measures to be in place, would examine the peer 

review (as discussed in Regulatory Guide 1.200) findings with an even higher degree of 

scrutiny so as to better understand how particular findings were resolved as well as the 

general effect of the findings, would likely consider the parameter uncertainties and the 

identified key sources of model uncertainties to determine if greater depth of review of 

the uncertainty aspects of the application should be performed, would likely perform an 

audit of the application PRA to determine the cause of the change in risk and would 

generally be limited to an investigation of the significant issues in the PRA and the key 

sources of model uncertainties.

Link to Regulatory Guide 1.200:

Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of 

Probabilistic Risk Assessment Results for Risk-Informed Activities 

[https://www.nrc.gov/docs/ML0904/ML090410014.pdf]
Page 80 of 83 

Slide 18F - Regime 4

In Regime 4, the risk metric results are considered to significantly exceed the 

acceptance guidelines when the mean value of the risk metric results are more than just 

slightly greater than the acceptance guidelines.

The staff would typically not accept such applications unless the application represents 

an overall reduction in plant risk. The staff would expect the licensee to identify those 

aspects of the PRA that are conservative and bounding and then perform a more 

realistic assessment to support the application, would look for a quantitative assessment 

which shows that the state-of-knowledge correlation does not affect the rick metric 

results of the PRA, would examine the application to ensure that the proposed 

performance monitoring is adequate and determine whether degraded performance can 

be detected in a timely fashion, would determine the appropriateness of the 

compensatory measures, would thoroughly review the licensee’s PRA peer review 

findings, would review in greater detail the parameter uncertainties and the identified key 

sources of model uncertainties and their effects on the risk metric results, would perform 

a more in-depth audit of the application PRA.
Page 81 of 83 

Slide 18G – Application Acceptance/Rejection

If accepted by the staff, the risk-informed application is considered to have (1) an 

acceptable treatment of uncertainties, and, (2)to have met the fourth risk-informed 

decisionmaking principle of posing an acceptable risk effect to the plant. 

Conversely, if the staff rejects the application, the risk-informed application is considered 

to have 1) an unacceptable treatment of uncertainties, or 2) poses an unacceptable risk 

effect to the plant. 
Page 82 of 83 

Slide 18H – Application Acceptance

In many cases, a risk-informed application requires consideration of the risk effect from 

multiple hazards and plant operational states and, as such, the risk contribution from 

each analysis must be combined, or aggregated, into a single risk metric, such as core 

damage frequency or large early release frequency. When combining the results of PRA 

models for several hazards (e.g., internal events, internal fires, seismic events) as 

required by many acceptance guidelines, the level of detail and level of approximation is 

likely to differ from one hazard to the next with some being more conservative than 

others. Often conservative bias can be larger for external hazards, internal fire, and low 

power and shutdown risk contributors, and can be one reason why the associated 

uncertainties can be larger. Therefore, the staff determines whether the licensee 

demonstrates adequate understanding of both the level of detail associated with the 

modeling of each of the hazards as well as the hazard-specific parameter and model 

uncertainties. This involves determining whether the licensee demonstrates an 

adequate understanding of:

1. The individual risk contributions and the parameter uncertainties associated with 

the analysis of each hazard or plant operational state and 

2. The sources and effects of conservatisms and model uncertainties that significantly 

affect the application results.
Page 83 of 83 

Course Completion Slide

Congratulations! You have completed this course and should now have a good 

understanding of the guidance presented in NUREG-1855 and the Associated EPRI 

Reports. 

For training on how to implement the guidance in these documents, please sign up for 

the training workshop or course that goes into detail, using various examples of riskinformed activities in order to illustrate application of the guidance.