Guidance on the Treatment 

of Uncertainties 

Associated with PRAs in 

Risk-Informed Decision 

Making 

Draft Report for Comment 

Office of Nuclear Regulatory Research 

NUREG-1855
Guidance on the Treatment 

of Uncertainties 

Associated with PRAs in 

Risk-Informed Decision 

Making 

Draft Report for Comment 

Manuscript Completed: November 2007 

Date Published: November 2007 

Prepared by: 

M. Drouin1

, G. Parry2 

J. Lehner3

, G. Martinez-Guridi3 

J. LaChance4, T. Wheeler4 

Other Contributors: 

L. Killian1 

3

Brookhaven National Laboratory 

Upton, NY 11973 

4

Sandia National Laboratories 

Albuquerque, NM 87185 

M. Drouin, NRC Project Manager 

1

Office of Nuclear Regulatory Research 

2

Office of Nuclear Reactor Regulation 

NUREG-1855
Draft for Comment iii NUREG-1855

ABSTRACT

This document provides guidance on how to treat uncertainties associated with probabilistic risk

assessment (PRA) in risk-informed decision making. The objectives of this guidance include

fostering an understanding of the uncertainties associated with PRA, the impact of the uncertainties

on the results of the PRA, and the uncertainties in the context of the decision making. The

guidance in this document focuses on the use of PRA insights and results and ways to address the

associated uncertainties. Consequently, the scope of the guidance contained in this report is

limited to addressing the uncertainties associated with the use of the results of risk models.

In implementing risk-informed decision making, the US Nuclear Regulatory Commission expects

that appropriate consideration of uncertainty will be given in analyses and interpretation of findings.

Such consideration should include using a program of monitoring, feedback, and corrective action

to address significant uncertainties. To meet this objective, it is necessary to understand the role

that PRA results play in the context of the decision process. Defining the context includes providing

an overview of the risk-informed decision making process itself.

With the context defined, the characteristics of a risk model and, in particular, a PRA need to be

understood. This understanding includes a recognition of the different forms of uncertainty which

include aleatory and epistemic. A PRA, as a probabilistic model already characterizes aleatory

uncertainty. The focus of this document is epistemic uncertainty. Therefore, guidance is given on

 identifying and describing the different types of sources of epistemic uncertainty including the

different ways that they are treated. The different types of epistemic uncertainty include parameter,

model, and completeness uncertainties.

The final part of the guidance includes addressing the uncertainty in PRA results in the context of

risk-informed decision making and, in particular, the interpretation of the results of the uncertainty

analysis when comparing PRA results with the acceptance criteria established for a specified

application. In addition, guidance is provided for addressing the other elements contributing to

completeness uncertainty in risk-informed decision making (e.g., unknown phenomena that have

not been recognized or factors that have been identified but for which there is no agreed on method

for addressing them in PRAs).
Draft for Comment v NUREG-1855

FOREWORD

In its safety philosophy, the U.S. Nuclear Regulatory Commission (NRC) has always recognized

the importance of addressing uncertainties as an integral part of its decision making. Two

fundamental elements of the NRC’s safety philosophy for addressing uncertainties are defense in

depth and safety margins. These elements are more directly coupled with the uncertainties

associated with the NRC’s traditional deterministic approach, that is, to address the uncertainties

associated with the state of knowledge regarding design, materials, fuel, and other issues. With

the increase use of probabilistic risk assessment (PRA) in the NRC’s risk-informed decision making,

the uncertainties associated with the PRA need to be considered. These uncertainties, and their

potential impact on the comparison of PRA results with acceptance criteria, need to be understood

so that informed decisions are made. When dealing with completeness uncertainties, the use of

bounding analyses to address potential risk contributors not included in the PRA needs to be

understood. This document provides guidance on how to treat uncertainties associated with PRA

in risk-informed decision making. The Electric Power Research Institute (EPRI), in parallel with the

NRC, has been developing guidance documents on the treatment of uncertainties. The activities

of the NRC and EPRI are meant to be complementary.

This draft NUREG is being issued for public review and comment.

Paper work Reduction Act Statement

This NUREG does not contain information collection requirements and, therefore, is not subject to

the requirements of the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq.)

Public Protection Notification

The NRC may not conduct or sponsor, and a person is not required to respond to, a request for

information or an information collection requirement unless the requesting document displays a

currently valid OMB control number.
Draft for Comment vii NUREG-1855

TABLE OF CONTENTS

Section Page

ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

FOREWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

CONTENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

ABBREVIATIONS AND ACRONYMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

1.1 Background and History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

1.2 Objectives and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3

1.3 Approach Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4

1.4 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5

2. OVERALL APPROACH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

2.1 Risk-Informed Decision Making Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

2.2 Understanding the PRA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5

2.3 Identifying and Characterizing Epistemic Uncertainties . . . . . . . . . . . . . . . . . . . . 2-6

2.3.1 Parameter Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6

2.3.2 Model Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6

2.3.3 Completeness Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-7

2.4 Understanding the Impact of the Uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-7

3. UNDERSTANDING THE PRA MODEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

3.1 The Risk Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

3.2 Characteristics of a PRA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2

3.2.1 The Structure of a PRA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2

3.2.2 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2

3.2.3 Scope of a PRA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3

3.2.4 Precision and Level of Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4

3.2.5 Results of the PRA (Aggregation of the Results) . . . . . . . . . . . . . . . . . . . 3-5

3.3 PRA Models and Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6

3.3.1 Examples of Epistemic Uncertainty in PRA . . . . . . . . . . . . . . . . . . . . . . . 3-6

3.3.2 Types of Epistemic Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7

3.3.3 Assessing the Impact of Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9

4. PARAMETER UNCERTAINTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

4.1 Characterization of Parameter Uncertainty for Basic Events . . . . . . . . . . . . . . . . 4-1

4.1.1 General Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2

4.1.2 Bayesian Estimation of Parameter Uncertainty . . . . . . . . . . . . . . . . . . . . . 4-6

4.1.3 Additional Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
NUREG-1855 viii Draft for Comment

TABLE OF CONTENTS (continued)

Section Page

4.2 Propagation of Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9

4.2.1 Monte Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10

4.2.2 Latin Hypercube Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10

4.2.3 Interpretation of Uncertainty of Risk Metric . . . . . . . . . . . . . . . . . . . . . . 4-11

4.2.4 State of Knowledge Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12

4.3 Treatment of Parameter Uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-16

5. MODEL UNCERTAINTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2

5.2 Base PRA Sources of Model Uncertainties and Related Assumptions . . . . . . . . 5-3

5.2.1 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4

5.2.2 Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5

5.2.3 Qualitative Screening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-7

5.3 Relevant Sources of Model Uncertainties and Related Assumptions . . . . . . . . . . 5-8

5.3.1 Understanding the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9

5.3.2 Performing Qualitative Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-11

5.4 Potentially Key Sources of Model Uncertainty and Related Assumptions . . . . . 5-12

5.4.1 Applications Involving Cumulative Acceptance Criteria . . . . . . . . . . . . . 5-14

5.4.2 Applications Involving Incremental Acceptance Criteria . . . . . . . . . . . . . 5-18

5.5 Key Sources of Model Uncertainty and Related Assumptions . . . . . . . . . . . . . . 5-24

5.5.1 Define and Justify Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25

5.5.2 Perform Quantitative Realistic Screening . . . . . . . . . . . . . . . . . . . . . . . . 5-26

5.5.2.1 Applications Involving Cumulative Acceptance Criteria . . . . . . 5-27

5.5.2.2 Applications Involving Incremental Acceptance Criteria . . . . . . 5-28

6. COMPLETENESS UNCERTAINTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

6.1 Determining the Required Scope of an Application . . . . . . . . . . . . . . . . . . . . . . . 6-2

6.2 Defining the Types of Bounding Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-4

6.3 Selecting and Using Bounding Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-6

6.3.1 Examples of Screening of Risk Contributors . . . . . . . . . . . . . . . . . . . . . . . 6-6

6.3.1.1 Qualitative Screening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8

6.3.1.2 Quantitative Screening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10

6.3.2 Examples of Bounding Risk Contributors . . . . . . . . . . . . . . . . . . . . . . . . 6-13

7. RISK-INFORMED DECISION MAKING: DEALING WITH UNCERTAINTY . . . . . . . . . 7-1

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

7.2 Presenting the Results of PRA Uncertainty Analyses . . . . . . . . . . . . . . . . . . . . . . 7-2

7.3 Comparison of PRA Results with Acceptance Criteria . . . . . . . . . . . . . . . . . . . . . 7-4

7.3.1 An Example of Risk Acceptance Guidelines Using RG 1.174 . . . . . . . . . 7-4

7.3.2 Addressing Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-5

7.3.2.1 Parameter Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-6

7.3.2.2 Model Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-6

7.3.2.3 Completeness Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-6

7.3.3 Interpretation of the Results of the PRA . . . . . . . . . . . . . . . . . . . . . . . . . . 7-7

7.3.3.1 Uncertainty Arising from Level of Detail . . . . . . . . . . . . . . . . . . . 7-7

7.3.3.2 Parametric Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-8

7.3.3.3 Model Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-8

7.3.3.4 Completeness Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9

7.3.3.5 Combining PRA Results (Integrated Assessment) . . . . . . . . . . . 7-9

7.4 Addressing Uncertainty in SSC Categorization . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
Draft for Comment ix NUREG-1855

TABLE OF CONTENTS (continued)

Section Page

7.5 Using Qualitative Approaches in Integrated Decision Making . . . . . . . . . . . . . . 7-11

7.5.1 Performance Monitoring Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 7-11

7.5.2 Limiting Scope of Plant Modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11

7.5.3 Use of Compensatory Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12

8. REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1

APPENDIX A

STAFF POSITION ON ELECTRIC POWER RESEARCH INSTITUTE, "GUIDELINE FOR THE

TREATMENT OF UNCERTAINTY IN RISK-INFORMED APPLICATIONS” . . . . . . . . . A-1

LIST OF FIGURES

Figure Page

2-1 Principles of Risk-Informed Decision Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2

2-2 Elements of Integrated Risk-Informed Decision Making Process . . . . . . . . . . . . . . . . 2-3

2-3 Steps Needed for the Treatment of Uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4

5-1 Process to Develop List of Key Sources of Uncertainty and Related Assumptions . . 5-1

5-2 Process to Identify and Characterize the Sources of Model Uncertainty and Related

Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4

5-3 Process to Identify Relevant Sources of Model Uncertainty and Related Assumptions

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9

5-4 Process to Identify Potentially Key Sources of Model Uncertainty and Related Assumptions

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13

5-5 NRC RG 1.174 Acceptance Guidelines for CDF and LERF . . . . . . . . . . . . . . . . . . . 5-19

5-6 Process to Identify Key Sources of Model Uncertainty and Related Assumptions . . 5-25

7-1 Integrated Decision Making Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

7-2 Risk Acceptance Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4

LIST OF TABLES

Table Page

4-1 Characteristics of Methods for Assessing Parameter Uncertainty . . . . . . . . . . . . . . . 4-4

5-1 Definitions For Model Uncertainty and Related Assumption . . . . . . . . . . . . . . . . . . . 5-2

5-2 Examples of Regulatory Applications and Risk-Informed Acceptance Criteria . . . . 5-10

5-3 Ordered Pairs of CDF and ΔCDF and Comparison against Acceptance Criteria . . . 5-20
Draft for Comment xi NUREG-1855

ABBREVIATIONS AND ACRONYMS

ACRS Advisory Committee on Reactor Safeguards

ANS American Nuclear Society

AOT Allowed Outage Time

ASME American Society of Mechanical Engineers

BWR Boiling Water Reactor

CCF Common Cause Failure

CDF Core Damage Frequency

CFR Code of Federal Regulations

CRD Control Rod Drive

EPRI Electric Power Research Institute

HEP Human Error Probability

HLR High Level Requirement

HPCI High Pressure Coolant Injection

HRA Human Reliability Analysis

LERF Large Early Release Frequency

LHS Latin Hypercube Sampling

LOCA Loss of Coolant Accident

LOOP Loss of Offsite Power

LPSD Low Power and Shutdown

LWR Light Water Reactor

MCS Minimal Cut Set

MOV Motor Operated Valve

NEI Nuclear Energy Institute

NPP Nuclear Power Plant

NRC U.S. Nuclear Regulatory Commission

PDF Probability Density Function

POS Plant Operating State

PRA Probabilistic Risk Assessment

PWR Pressurized Water Reactor

RAW Risk Achievement Worth

RCIC Reactor Core Isolation Cooling

RCP Reactor Coolant Pump

RG Regulatory Guide

SBO Station Blackout

SER Safety Evaluation Report

SOKC State of Knowledge Correlation

SSC Structure System and Component

YR Year
Draft for Comment 1-1 NUREG-1855

1. INTRODUCTION

1.1 Background and History

In its safety philosophy, the U.S. Nuclear Regulatory Commission (NRC) has always recognized

the importance of addressing uncertainties as an integral part of its decision making. Two

fundamental elements of the NRC’s safety philosophy are defense-in-depth and safety margins.

In a 1995 policy statement [NRC,1995a], the NRC encouraged the use of probabilistic risk

assessment (PRA) in all regulatory matters. The policy statement declares that “the use of PRA

technology should be increased to the extent supported by the state of the art in PRA methods and

data and in a manner that complements the NRC’s deterministic approach....PRA and associated

analyses (e.g., sensitivity studies, uncertainty analyses and importance measures) should be used

in regulatory matters....” The Commission further notes in the policy statement that the “treatment

of uncertainty is an important issue for regulatory decisions. Uncertainties exist...from knowledge

limitations....A probabilistic approach has exposed some of these limitations and provided a

framework to assess their significance and assist in developing a strategy to accommodate them

in the regulatory process.”

In a white paper entitled “Risk-Informed and Performance-Based Regulation” [NRC,1999a], the

Commission defined the terms and described its expectations for risk-informed and

performance-based regulation. The Commission indicated that a “risk-informed” approach explicitly

identifies and quantifies sources of uncertainty in the analysis (although such analyses do not

necessarily reflect all important sources of uncertainty) and leads to better decision making by

providing a means to test the sensitivity of the results to key assumptions.

Since the issuance of the PRA policy statement, the NRC has implemented or undertaken many

uses of PRA, including modification of the NRC’s reactor safety inspection program and initiation

of work to modify reactor safety regulations. Consequently, confidence in the information derived

from a PRA is an important issue. The technical content must be of quality sufficient to justify the

specific results and insights to be used to support the decision under consideration. The treatment

of the uncertainties associated with the PRA is an important factor in establishing this quality.

Regulatory Guide (RG) 1.174 [NRC, 2002a], RG 1.200 [NRC, 2007a], and the national PRA

consensus standard [ASME, 2005a] all recognize the importance of the identification and

understanding of uncertainties that are part of PRA (and of any deterministic analysis as well), and

these references provide guidance on this subject to varying degrees. However, they do not

provide explicit guidance on the treatment of uncertainties in risk-informed decision making.

RG 1.174 states that a PRA should include a full understanding of the impacts of the uncertainties

through either formal quantitative analysis or more simple bounding or sensitivity analyses. The

guidance also maintains that the decisions “must be based on a full understanding of the

contributors to the PRA results and the impacts of the uncertainties, both those that are explicitly

accounted for in the results and those that are not.”
1. Introduction

NUREG-1855 1-2 Draft for Comment

RG 1.200 states that a full understanding of the uncertainties and their impact is needed (i.e.,

sources of uncertainty are to be identified and analyzed). Specifically, RG 1.200 notes the

following:

“An important aspect in understanding the base PRA results is knowing what are the

sources of uncertainty and assumptions and understanding their potential impact.

Uncertainties can be either parameter or model uncertainties, and assumptions can

be related either to PRA scope and level of detail or to model uncertainties. The

impact of parameter uncertainties is gained through the actual quantification

process. The assumptions related to PRA scope and level of detail are inherent in

the structure of the PRA model. The requirements of the applications will determine

whether they are acceptable. The impact of model uncertainties and related

assumptions can be evaluated qualitatively or quantitatively. The sources of model

uncertainty and related assumptions are characterized in terms of how they affect

the base PRA model (e.g., introduction of a new basic event, changes to basic event

probabilities, change in success criterion, introduction of a new initiating event).”

The national consensus standard on PRA requires that sources of model uncertainty in the base

PRA be identified and provides requirements for the identification and characterization of both

parameter and model uncertainties, both parameter and the model. However, the standard

provides requirements on “what to do” and not “how to.”

In a letter dated April 21, 2003 [ACRS, 2003a], the Advisory Committee on Reactor Safeguards

(ACRS) provided recommendations for staff consideration in Draft Guide 1122 (now RG 1.200).

One recommendation was to include guidance on how to perform sensitivity and uncertainty

analyses. In its letter, the ACRS noted the following:

“...a systematic treatment should include rigorous analyses for parametric

uncertainties, sensitivity studies to identify the important epistemic uncertainties, and

quantification of the latter. In a risk-informed environment, the proper role of

sensitivity studies is to identify what is important to the results, not to replace

uncertainty analyses.” 

In its response [NRC, 2003a], the staff noted that the standard provides requirements for the

performance of sensitivity and uncertainty analyses. However, the staff agreed to examine the

requirements in more detail to identify where additional guidance may be needed. 

In a subsequent letter dated May 16, 2003 [ACRS, 2003b], the ACRS provided recommendations

to improve the quality of risk information for regulatory decision making. One recommendation was

for the staff to develop guidance for the quantitative evaluation of model uncertainties. In its letter,

the ACRS noted the following:

• If methods other than PRAs are used to compensate for missing scope items, they can

result in nonconservative decisions.

• Models that are included in the PRAs can be important sources of uncertainty. For

example, using only one of the several models for human performance yields results with
1. Introduction

Draft for Comment 1-3 NUREG-1855

unknown uncertainties, since the use of another model could produce different results. Yet

this model uncertainty is rarely considered.

• Most licensees have not included a systematic treatment of uncertainties in their PRAs. A

systematic treatment would include analyses of parametric uncertainties, sensitivity studies

to identify the important model uncertainties, and quantification of the latter.

• Tools for performing analyses of parametric uncertainties are readily available and are

included in most of the widely used PRA software. The disciplined use of sensitivity studies

to address model uncertainties is not as well understood. Developing guidance for

quantifying model uncertainty is not infeasible. Such an effort would build on past practice

and the literature.

• More guidance regarding sensitivity and uncertainty analyses would contribute greatly to

confidence in risk-informed regulatory decision making. Such guidance should include a

clear discussion of the roles of sensitivity and uncertainty analyses, as well as practical

procedures for performing these analyses. It should address not only how uncertainties

should be treated in the PRA, but, also, how they impact decision making with examples to

show the pitfalls if uncertainties are inadequately addressed.

In response to the ACRS [NRC, 2003b], the NRC agreed that guidance is needed on the treatment

of uncertainties in risk-informed decision making (i.e., the role of sensitivities and uncertainty

analyses), specifically guidance regarding acceptable characterization of other methods, such as

bounding analyses, to ensure that reasonable approaches are used. This report provides the

needed guidance.

1.2 Objectives and Scope

This document provides guidance on how to treat uncertainties associated with PRA in riskinformed decision making. The objectives of this guidance include fostering an understanding of:

• the uncertainties associated with PRA

• the impact of the uncertainties on the results of the PRA

• the uncertainties in the context of the decision making

The Commission has defined risk-informed decision making as an integrated process whereby risk

insights are considered together with other factors to better focus both licensee and regulatory

attention on design and operational issues commensurate with their importance to public health and

safety. These factors are the key principles identified and described in detail in RG 1.174 and

include the following:

• meeting current regulations

• consistency with defense-in-depth philosophy

• maintenance of safety margins

• use of PRA insights and results

• use of performance measures to monitor impact of decision
1. Introduction

NUREG-1855 1-4 Draft for Comment

The guidance in this document focuses on the use of PRA insights and results and ways to address

the associated uncertainties. Consequently, the scope of the guidance contained in this report is

limited to addressing the uncertainties associated with the use of the results of risk models.

The guidance is intended to be consistent with the NRC’s PRA policy statement and subsequent,

more detailed guidance in RG 1.174 and RG 1.200. It is also intended to support these documents

and other NRC documents that address risk-informed activities including, at a minimum, the

following:

• RG 1.201, “Guidelines for Categorizing Structures, Systems, and Components in Nuclear

Power Plants According to Their Safety Significance” [NRC, 2006a]

• RG 1.205, “Risk-Informed, Performance-Based Fire Protection for Existing Light-Water

Nuclear Power Plants” [NRC, 2006b]

• RG 1.206, “Combined License Applications for Nuclear Power Plants (LWR Edition)” [NRC,

2007b]

• Standard Review Plan, Section 19.0, “Probabilistic Risk Assessment and Severe Accident

Evaluation for New Reactors,” Revision 2; Section 19.1, “Determining the Technical

Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities”;

Section 19.2, “Review of Risk Information Used to Support Permanent Plant-Specific

Changes to the Licensing Basis: General Guidance” [NRC, 2007c]

• regulatory guides for specific applications such as for inservice testing, inservice inspection,

and technical specifications [NRC, 1998a, NRC, 2003c; NRC, 1998c]

The guidance is also intended to support guidance provided by standards-setting and nuclear

industry organizations [NEI, 2005a; NEI, 2006a; NEI, 2006b]. In particular, the Electric Power

Research Institute (EPRI), in parallel with the NRC, has been developing guidance documents on

the treatment of uncertainties. This work is meant to complement the guidance in this document.

Where possible the NRC guidance will refer to the EPRI work for acceptable treatment of

uncertainties [EPRI, 2004a; EPRI, 2006a].

1.3 Approach Overview

In implementing risk-informed decision making, the NRC expects that appropriate consideration of

uncertainty will be given in analyses and interpretation of findings. Such consideration should

include using a program of monitoring, feedback, and corrective action to address significant

uncertainties. To meet this objective, it is necessary to understand the context that the role of

uncertainties play in the decision process. Defining the context includes providing an overview of

the risk-informed decision making process itself.

With the context understood, the characteristics of a risk model and, in particular, a PRA need to

be defined. This definition understanding the different forms of uncertainty which include aleatory

and epistemic. A PRA, as a probabilistic model already characterizes aleatory uncertainty. The

focus of this document is epistemic uncertainty. Therefore, the definition includes identifying and

describing the different types of sources of epistemic uncertainty including the different ways that
1. Introduction

Draft for Comment 1-5 NUREG-1855

they are treated. The different types of epistemic uncertainty include parameter, model, and

completeness uncertainties.

Parameter uncertainty — Guidance is provided on how to address parameter uncertainty in the

use of PRA results for decision making. This guidance involves characterization of parameter

uncertainty, propagation of uncertainty, assessment of the significance of the state-of-knowledge

correlation, and comparison of results with acceptance criteria.

Model uncertainty — Guidance is provided for identifying and characterizing model uncertainties

in PRAs. This guidance involves assessing the impact of model uncertainties on insights used for

risk-informed applications. It is provided in the context of two different types of applications:

(1) The base PRA is used in a regulatory application, for example, to evaluate various design

options or to determine the baseline risk profile as part of a license submittal for a new

plant.

(2) The base PRA is used as input to evaluating proposed changes to a plant’s licensing basis,

for example, by using RG 1.174, and the emphasis is on the change in risk due to the

proposed plant changes.

Completeness uncertainty — Guidance is provided on addressing one aspect of completeness

uncertainty (i.e., missing scope) in risk-informed applications. This guidance involves the

performance of a conservative or bounding analysis as one means to address items missing from

a plant’s PRA scope.

The final part of the guidance includes addressing the uncertainty in PRA results in the context of

risk-informed decision making and, in particular, the interpretation of the results of the uncertainty

analysis when comparing PRA results with the acceptance criteria established for a specified

application. In addition, guidance is provided for addressing the other elements contributing to

completeness uncertainty in risk-informed decision making (e.g., unknown phenomena that have

not been recognized or factors that have been identified but for which there is no agreed on method

for addressing them in PRAs).

1.4 Report Organization

The remainder of this report is divided into the following chapters:

• Chapter 2 — Provides an overview of the overall approach used to address uncertainties

in risk-informed decision making, which includes the context of the uncertainties and an

overview of the risk-informed decision making process itself. This chapter serves as a

roadmap to the rest of the report.

• Chapter 3 — Defines the characteristics of a risk model (in particular, a PRA) and the

sources and types of uncertainty.

• Chapter 4 — Provides the guidance for the treatment of parametric uncertainty.

• Chapter 5 — Provides the guidance for the treatment of model uncertainty.
1. Introduction

NUREG-1855 1-6 Draft for Comment

• Chapter 6 — Provides the guidance for the treatment of completeness uncertainty.

• Chapter 7 — Provides guidance on addressing the uncertainty in PRA results in the context

of risk-informed decision making.

• Chapter 8 — Provides the references.

• Appendix A — Provides the staff position on EPRI guidance documents on treatment of

uncertainties in risk-informed regulatory applications (to be written)
Draft for Comment 2-1 NUREG-1855

2. OVERALL APPROACH

The purpose of this chapter is to provide an overview of the guidance provided in the subsequent

chapters of this report to address the treatment of uncertainties in risk-informed decision making.

The proposed approach focuses on providing an understanding of the decision making process,

the role of the PRA model in the process, the uncertainties associated with the risk model and their

impact on the model, and ultimately, the treatment of the uncertainties in the decision making.

2.1 Risk-Informed Decision Making Process

In its white paper “Risk-Informed and Performance-Based Regulation,” [NRC, 1999a] the

Commission defined risk-informed regulation as an approach to regulatory decision making that

represents a philosophy whereby risk insights are considered together with other factors to

establish requirements. The requirements thus established focus licensee and regulatory attention

on design and operational issues commensurate with their importance to public health and safety.

This philosophy was elaborated in RG 1.174 [NRC, 2002a] to develop a risk-informed decision

making process for licensing changes.

In developing this process, the NRC defined in RG 1.174 a set of key principles that it expects to

be followed for decisions regarding licensing changes. For the most part, the principles are global

in nature and can be generalized to all applications that are the subject of risk-informed decision

making as shown in the following examples:

• Principle 1: RG 1.174 states, “Change meets current regulations unless it is explicitly

related to a requested exemption or rule change.” This principle can be generalized to

“Current Regulations Met.”

• Principle 2: RG 1.174 states, “Change is consistent with defense-in-depth philosophy. This

principle is already a general principle, “Maintenance of Defense-in-Depth.”

• Principle 3: RG 1.174 states, “Maintain sufficient safety margins.” This principle is already

a general principle, “Maintenance of Safety Margins.”

• Principle 4: RG 1.174 states, “Proposed increases in CDF or risk are small and are

consistent with the Commission’s Safety Goal Policy Statement.” This principle in essence

is requiring the performance of a risk assessment demonstrating acceptable risk be

performed. It can be generalized to “Acceptable Risk Analysis.”

• Principle 5: RG 1.174 states, “Use performance-measurement strategies to monitor the

change.” This principle can be generalized to “Monitor Performance.”

Figure 2-1 illustrates the five principles of risk-informed decision making. Of the five principles,

Principle 4, Risk Analysis, introduces uncertainties. Consequently, Principle 4 is of most interest

to this report with its focus on providing guidance for an acceptable treatment of uncertainty in a

risk analysis. To address the uncertainties associated with the risk analysis, an understanding is

needed of the use of all these principles in decision making. 
2. Overall Approach

NUREG-1855 2-2 Draft for Comment

Figure 2-1 Principles of Risk-Informed Decision Making

The principles themselves, although expected to be observed, are not the process or approach that

is used in risk-informed decision making. RG 1.174 presents an approach that ensures the

principles will be met for risk-informed decision making involving licensing changes. This approach

can be generalized and applied to all risk-informed decision making.

The approach, generalized from RG 1.174, is a structured process that integrates all the insights

and requirements that relate to the safety or regulatory issue of concern. These insights and

requirements include recognition of any mandatory requirements resulting from current regulations,

as well as the insights from deterministic and probabilistic analyses performed to help make the

decision. It also includes provisions for implementing the decision and for monitoring the results

of the decision. The approach includes the five following elements:

(1) define the decision

(2) identify and assess the applicable requirements

(3) perform a risk-informed analysis

(4) define an implementation and monitoring program

(5) integrate the above into a decision

The elements of this integrated process, as illustrated in Figure 2-2, are described below.

Principle 4

Risk-Informed 

Analysis

Principle 1

Current 

Regulations Met

Principle 3

Maintenance of 

Safety Margins

Principle 5

Monitor 

Performance

Integrated RiskInformed Decision 

Making

Principle 2

Defense-in-Depth 

Consistency

Principle 4

Risk-Informed 

Analysis

Principle 1

Current 

Regulations Met

Principle 3

Maintenance of 

Safety Margins

Principle 5

Monitor 

Performance

Integrated RiskInformed Decision 

Making

Principle 2

Defense-in-Depth 

Consistency
2. Overall Approach

Draft for Comment 2-3 NUREG-1855

Figure 2-2 Elements of Integrated Risk-Informed Decision Making Process

Element 1: Define the decision under consideration. The first step is to define the issue to be

addressed by the decision making process. Examples of the types of issues that the NRC would

need to address include the following:

• decisions related to the design or operation of the plant

• the plant technical specifications/limits and conditions for normal operation

• the periodicity of inservice inspection, inservice testing, maintenance, and planned outages

• the allowed combinations of safety system equipment that can be removed from service or

configuration control during power operation and shutdown modes

• the emergency operating procedures and accident management methods 

Element 2: Identify and assess the applicable existing requirements. A review will be needed

to determine the current requirements that apply to the decision under consideration. This review

must be followed by a determination of the effect of the decision on these requirements. This

element implements Principle 1 of risk-informed decision making shown in Figure 2-1.

Element 3: Perform a risk-informed analysis to support the decision. In this step, an

assessment is made, in terms of a risk-informed analysis, to obtain insights regarding the decision.

The risk-informed analysis includes both deterministic and probabilistic components, which will of

necessity involve uncertainties. The appropriate treatment of the uncertainties in the probabilistic

analysis is implicitly required to implement Principle 4 of risk-informed decision making shown in

Figure 2-1. Treatment of these uncertainties is the focus of this report, as discussed below. 

1. Define the 

Decision

2. Identify and 

Assess the 

Applicable 

Requirements

4. Define 

Implementation & 

Monitoring 

Program

“Assess”

Uncertainties

5. Integrated 

Decision

3. Perform a RiskInformed Analysis

Deterministic Probabilistic

Analysis Analysis

1. Define the 

Decision

2. Identify and 

Assess the 

Applicable 

Requirements

4. Define 

Implementation & 

Monitoring 

Program

“Assess”

Uncertainties

5. Integrated 

Decision

3. Perform a RiskInformed Analysis

Deterministic Probabilistic

Analysis Analysis
2. Overall Approach

NUREG-1855 2-4 Draft for Comment

Element 4: Make provisions for implementing the decision and monitoring the results. A

part of the decision making process is looking ahead to the implementation of a positive decision

in terms of needed changes and required effort. In addition, consideration should be given to a

performance-based means of monitoring the results of the decision. This monitoring will allow

future assessment of whether the decision has been implemented effectively and if there are any

adverse effects. This element implements Principle 5 of risk-informed decision making as shown

in Figure 2-1.

Element 5: Make the decision. Here the inputs from Elements 1 through 4 are integrated, and

the decision is made whether to accept or reject a proposed design, plant change, regulatory

change, or etc. This requires that the individual insights obtained from the other parts of the

decision process are weighted and combined to reach a conclusion. In integrating the inputs from

the elements, an essential aspect is the consideration of uncertainties, including those associated

with the PRA.

PRAs can address many uncertainties explicitly. These uncertainties are the epistemic

uncertainties (i.e., uncertainties arising from limitations in or lack of knowledge). However, there

is a specific type of uncertainty that risk analyses, whether deterministic or probabilistic, cannot

address. This type of uncertainty involves the incompleteness of the state of knowledge concerning

potential failure modes or mechanisms. Because these failure modes or mechanisms are

unknown, they cannot be addressed analytically (whether the analysis be deterministic or

probabilistic). Other principles of safety, such as defense in depth, safety margins, and

performance monitoring, address these uncertainties. However, the focus of this report is on the

treatment of the uncertainties associated with the PRA.

To address the treatment of uncertainties associated with a PRA in the risk-informed decision

making process, it is necessary to understand the role the PRA plays in the decision, the different

sources of uncertainties, and the impact of the uncertainties. This process is shown in Figure 2-3.

The figure also shows the arrangement of the remaining chapters of this report.

Figure 2-3 Steps Needed for the Treatment of Uncertainties

Subsequent sections of this chapter provide an overview of the process for treating uncertainties.

Later chapters provide the detailed guidance.

U n d e rsta n d ing 

th e P R A M o d e l 

( Cha p ter 3 )

U n d e rsta n d ing th e 

Im p a ct of th e 

U n ce rta in tie s 

( Cha p ter 7 )

P a ra m e tric 

U n ce rta in tie s 

(C hapter 4) 

C o m p le te n e ss 

U n ce rta in tie s 

(C hapter 6) 

M o d e l 

U n ce rta in tie s 

(C hapter 5) 

Id e n tific a tio n a n d C h a ra cte riza tio n o f E p is te m ic U n c e rta in tie s 
2. Overall Approach

1

Risk is defined as the potential health effects based on the design and operation of the plant.

Draft for Comment 2-5 NUREG-1855

2.2 Understanding the PRA Model

To address the treatment of uncertainties inherent in a PRA model in decision making, a thorough

understanding of the model is required. This understanding will identify the underlying assumptions

and limitations, thereby indicating the sources of uncertainty, the parts of the PRA model that could

be affected, and ultimately the results from the PRA model that may be impacted.

PRA models that are developed to predict the performance of nuclear power plants in terms of their

risk1

 are complex models consisting of many elements. Each element may involve the

development of a logic structure (e.g., fault trees), input to the structure (e.g., failure data), or both.

Both the structures and input can introduce uncertainties that could have a significant impact on

the predictions of the PRA model, and these uncertainties must be addressed. Examples of

sources of uncertainty include the following:

• Uncertainty about the capability of some systems to perform their function under the

conditions specified by the developed scenarios. This capability uncertainty leads to

uncertainty in characterizing the success criteria for those functions. This, in turn, has an

impact on the logic structure of the overall PRA model.

• For some basic events, there can be uncertainty in the characterization of failures, which

results in uncertainties in the probabilities of those failures.

• Because most of the events for which data are needed (e.g., initiating events, operator

errors, and equipment failures) are rare, the data are generally scarce. Thus, statistical

uncertainties may occur in the parameter estimations (e.g., uncertainty in the computational

approach chosen to estimate the event probability).

• Some events for which the computational approach is generally agreed on may have

uncertainties associated with interpreting the data to be used for estimation. 

Therefore, the uncertainty associated with the structure of, and input to, the PRA model could

reside in the choice of the logic structure, in the computational approach, or in the values of the

parameters. 

While uncertainties in the PRA model have different sources, there are two basic forms of

uncertainties, aleatory and epistemic. Aleatory uncertainty is associated with the random nature

of events such as initiating events, and is implicit in the structure of the PRA model. The guidance

provided in this report on the treatment of uncertainties pertains to epistemic uncertainties.

Epistemic uncertainties arise because of limitations in the knowledge of the analysts carrying out

the PRA. The epistemic uncertainties relate to the degree of belief that the analysts have in the

model itself and the predictions of the model; that is, how well the model reflects the design and

operation of the plants and, therefore, how well it predicts the performance of the plant. The three

types of epistemic uncertainties found in a PRA are parameter uncertainty, model uncertainty, and

completeness uncertainty. The understanding, identification, and treatment of these three types

of epistemic uncertainties found in a PRA are the principal subject of the remainder of this report.
2. Overall Approach

NUREG-1855 2-6 Draft for Comment

The focus of Chapter 3 is the role uncertainties play in the decision process. The purpose of

Chapter 3 is to provide an overview of the PRA model’s characteristics and the different sources

of uncertainty associated with the model. 

2.3 Identifying and Characterizing Epistemic Uncertainties

The following is an overview of the treatment of the three different types of epistemic uncertainties

(parameter, model, and completeness). Chapters 4, 5, and 6, respectively, provide detailed

guidance on these three uncertainty types. 

2.3.1 Parameter Uncertainty

Parameter uncertainty relates to the uncertainty in the computation of the parameter values for

initiating event frequencies, component failure probabilities, and human error probabilities that are

used in the quantification process of the PRA model. These uncertainties can be characterized by

probability distributions that relate to the analysts’ degree of belief in the values that these

parameters could take. Most of the PRA software in current use has the capability to propagate

these uncertainties through the analysis and calculate the probability distribution for the results of

the PRA. To make a risk-informed decision, the numerical results of the PRA, including their

associated uncertainty, must be compared with the appropriate decision criteria.

Chapter 4 discusses the treatment of parameter uncertainty at length. The purpose of Chapter 4

is to provide guidance on how to address parameter uncertainty in the use of PRA results for

decision making. The chapter includes characterization of parameter uncertainty, propagation of

uncertainty, and comparison of results with decision criteria.

2.3.2 Model Uncertainty

Model uncertainty relates to the uncertainty in the assumptions made in the analysis and the

models used. Examples of model uncertainty include the assumptions made as to how a reactor

coolant pump in a pressurized-water reactor would fail following loss of seal cooling and/or

injection, the approach used to address common cause failure in the PRA model, and the approach

used to identify and quantify operator errors. In general, model uncertainties are addressed by

studies to determine the sensitivity of the results of the analysis if different assumptions are made

or different models are used.

The treatment of model uncertainty in decision making depends on how the PRA will be used. The

expectation is that the focus will be on identifying and evaluating sources of uncertainty that are

relevant to the specific application at hand. The applications for which the treatment of sources of

uncertainties that could impact a risk-informed decision is needed generally are of two types:

(1) The base PRA is used in a regulatory application, for example, to evaluate various design

options or to determine the baseline risk profile as part of a license submittal for a new

plant.

(2) The base PRA is used as input to evaluating proposed changes to a plant’s licensing basis,

for example, by using RG 1.174, and the emphasis is on the change in risk due to the

proposed plant changes.
2. Overall Approach

Draft for Comment 2-7 NUREG-1855

Chapter 5 focuses on the treatment of model uncertainty. The purpose of Chapter 5 is to provide

guidance for identifying and characterizing model uncertainties in PRAs and then assessing the

possible impact of those uncertainties on insights about risk-informed applications.

2.3.3 Completeness Uncertainty

Completeness uncertainty, in the context of this report, relates to contributions to risk that have

been excluded from the PRA model. This class of uncertainties may have a significant impact on

the predictions of the PRA model and must be addressed. Examples of sources of incompleteness

include the following:

• The scope of the PRA does not include some classes of initiating events, hazards, or

modes of operation.

• There is no agreement on how the PRA should address certain elements, such as the

effects on risk resulting from aging or organizational factors.

• The analysis may have omitted phenomena, failure mechanisms, or other factors because

their relative contribution is believed to be negligible.

When a PRA is used to support an application, its scope and level of detail must be examined to

determine if they match what is required for the risk-informed application. If the scope or level of

detail of the existing base PRA is incomplete, then the PRA must be upgraded to include the

missing piece(s), or it must be demonstrated that the missing elements are not significant risk

contributors. Conservative or bounding type analyses are used to demonstrate the risk significance

and the associated uncertainty. 

Chapter 6 focuses on the treatment of completeness uncertainty. The purpose of Chapter 6 is to

provide the guidance for the performance of conservative or bounding analysis to address items

missing from a plant’s PRA scope or level of detail.

2.4 Understanding the Impact of the Uncertainties

In making an integrated decision, the decisionmaker must consider each of the elements in

Figure 2-2. When the risk analysis element is considered, the uncertainties in the results of that

analysis need to be addressed to understand the robustness of the conclusions of the analysis.

Depending on their significance, the impacts of the uncertainties could influence the decision under

consideration. Under the best circumstances, the impact of the uncertainties can be directly

quantified, and the analyst can determine whether the planned defense-in-depth measures, the

safety margins, and other factors adequately address the uncertainties and then make his or her

decision accordingly. 

However, in some cases, no agreed upon theoretical or empirical basis for representing the impact

of a change on the plant exists, so the effect on risk cannot be quantified without some degree of

arbitrariness. In such a case, one solution may be an increased reliance on the performance

monitoring element. The adoption of a performance monitoring strategy can ensure that the

subsequent observed changes in plant performance are no greater than those assumed in the risk

analysis supporting the change. 
2. Overall Approach

NUREG-1855 2-8 Draft for Comment

In other cases, an analyst may decide not to use a PRA model to address a potentially significant

contributor to risk. This decision may be made because of resource limitations, for example. One

way of quantitatively addressing the uncertainty in the PRA results that arises from using an

incomplete PRA model is to use bounding approaches demonstrating that a missing PRA scope

contributor to risk is not significant. However, when the missing contributors cannot be

demonstrated to be insignificant to the decision, an alternative approach is needed. One possibility

is to impose conditions on the implementation of the decision to compensate for the uncertainty by

essentially arguing that the implementation does not affect the unquantified portion of risk.

Chapter 7 treats all of these cases in some depth.

Chapter 7 discusses in detail the integration of the risk-informed analysis within the risk-informed

decision making process. The purpose of that chapter is to give guidance on addressing the

uncertainty in PRA results in the context of risk-informed decision making and in particular on

interpreting the results of the uncertainty analysis when comparing PRA results with the acceptance

criteria established for a specified application. Since the risk information required will depend on

the formulation of the acceptance criteria against which the information will be compared, Chapter 7

also discusses the coordination of the development of the risk information with the acceptance

criteria.
Draft for Comment 3-1 NUREG-1855

3. UNDERSTANDING THE PRA MODEL

The purpose of Chapter 3 is to provide an overview of the characteristics of the PRA model and the

different sources of uncertainty associated with a PRA model. The discussion focuses on the

nature of the model used to characterize risk (i.e., a PRA model), defines the types of uncertainty

that result from the modeling process, and briefly describes how these uncertainties are addressed.

3.1 The Risk Model

In general terms, a model can be described as an analyst’s attempt to represent a “system.” A

system model in the physical sciences or engineering disciplines is usually a mathematical model,

which is to say that it has a mathematical structure that can be used to produce numerical results

that represent certain aspects of the system’s behavior. Such a mathematical model will generally

have several parameters, which require numerical estimates. 

In general, since it is not possible to capture all the subtleties of the system behavior in a tractable

mathematical form, most models are approximations. Therefore, uncertainties are associated with

the formulation of the model and with its predictions. For some models, however, this uncertainty

is so small that it can essentially be ignored. For example, the mathematical formulation of many

of the models created by physicists to explain natural phenomena are well supported or verified

such that the models are very precise in their predictions, and their uncertainty is sufficiently small

to be ignored. An example of one such model is Newtonian mechanics and Newton's law of gravity.

This model is capable of very accurately predicting such things as planetary motion and can be

used to define the trajectories of planets or space vehicles with great accuracy.

PRA models are used to perform risk analysis of complex systems such as nuclear power plants.

A PRA of a nuclear power plant is not as precise as the well-established physical models as in the

above example for a number of reasons. For example, PRA models are probabilistic models that

predict the occurrence of rare events and their predictions cannot be verified directly. Further, the

phenomena arising from a severe accident and the equipment behavior under adverse conditions

are not always well understood. These factors result in a PRA model involving varying degrees of

approximation, and based on a number of assumptions. The uncertainties associated with a PRA

can range from being very small to having a significant effect on the calculated results.

To ensure that the decisionmaker is making an informed decision, an understanding of the

uncertainties associated with the PRA model is essential. The analyst must understand the

different kinds of uncertainties, identify the uncertainties, and then determine their impact on the

results. Essential to this understanding is knowledge of the characteristics of a PRA model,

including the structure of the model, the underlying assumptions made in developing the model, and

the scope and level of detail of the model.

The analyst constructing the PRA model determines its scope and level of detail. Therefore, these

two characteristics can vary to the extent that the PRA addresses significant contributors to a riskinformed decision. However, for some risk contributors, alternate approaches, other than inclusion

in the PRA model, may be used, particularly for items that are not significant contributors to the

decision. In the context of risk-informed decision making, these alternate approaches should be

capable of producing estimates of risk, and therefore, they are typically conservative or bounding

models. Such alternate models (as discussed in Chapter 6) will have some characteristics of fully

developed PRA models. Therefore, the discussion in Section 3.3 is applicable to the extent that

the alternate model reflects the characteristics of a PRA model.
3. Understanding the PRA Model

NUREG-1855 3-2 Draft for Comment

3.2 Characteristics of a PRA Model

The characteristics of a PRA model include the structure of the model, the underlying assumptions

made in developing the model, the scope of the model, and the level of detail of the model. The

following sections describe each of these characteristics.

3.2.1 The Structure of a PRA Model

A PRA model is a complex model consisting of many elements. The structural basis of the PRA

model is a logic model, which is constructed using logic structures such as event trees and fault

trees. The event trees identify the different plant responses that could occur given an initiating

event. The fault trees identify the different combinations of more elementary events, called basic

events, that could lead to undesired system states. These logic models represent a simplification

(discretization) of the potentially unlimited range of scenarios into a manageable set that is

supposed to be representative of, and encompass the range of consequences of, that larger set.

The types of basic events found in PRAs include events that represent the following:

• occurrence of initiating events

• the states of unavailability or failure of systems, structures, or components

• the human failures that contribute to the failure of the systems designed to protect against

the undesirable consequences should an initiating event occur 

A frequency is estimated for an initiating event occurrence, while probabilities are estimated for the

other basic events. These frequencies and probabilities are derived from the basic event models.

For example, the occurrence of an initiating event is modeled as a random process with an

associated frequency of occurrence. The other basic events discussed above are typically events

such as the failure of a pump to start, the failure of a pump to run for 24 hours, and the failure of

an operator to take the appropriate actions to prevent system damage. Usually, these basic events

are also regarded as resulting from random occurrences with respect to the demand created by the

initiating event. They are described by simple probabilistic models, such as the constant probability

of failure on demand (the binomial process) or a constant failure rate (the Poisson process). 

While the above examples are representative of most of the basic events in the models, other types

of basic events are also included. For example, when analyzing reactor coolant pump (RCP) seal

loss-of-coolant accidents (LOCAs), there may be events that delineate the different sizes of LOCAs.

Furthermore, PRA models used to analyze the risk from fires or internal floods may also include

events that are used to represent the occurrence of certain states of fire/flood damage. The

probabilities of these events are derived from more complex models. For example, the occurrence

of a fire damage state may be evaluated from the interplay between a model of fire growth and a

model of fire detection and suppression.

3.2.2 Assumptions

The development of any model is generally based on a number of assumptions, and a PRA model

is no exception. An assumption is a decision or judgment that is made in the development of the
3. Understanding the PRA Model

Draft for Comment 3-3 NUREG-1855

PRA model. An assumption is either related to a source of model uncertainty or related to scope

or level of detail.

An assumption related to a model uncertainty is made with the knowledge that a different

reasonable alternative assumption exists. A reasonable alternative assumption is one that has

broad acceptance in the technical community and for which the technical basis for consideration

is at least as sound as that of the assumption being made. This is discussed briefly in Section 3.4

and in more detail in Chapter 5. 

By contrast, an assumption that is related to scope or level of detail is one that is made for modeling

convenience. Such assumptions result in defining the boundary conditions for the PRA model.

Sections 3.2.3 and 3.2.4 address this topic.

3.2.3 Scope of a PRA Model

The scope of the PRA is defined in terms of (1) the metrics used to evaluate risk, (2) the plant

operating states for which the risk is to be evaluated, and (3) the types of initiating events that can

potentially challenge and disrupt the normal operation of the plant and, if not prevented or mitigated,

would eventually result in core damage and/or a large release.

Risk metrics are the end-states quantified in a PRA to evaluate risk. In a PRA, these end-states

or risk metrics are generally defined as a Level 1, limited Level 2, Level 2, or Level 3 analysis.

These metrics are defined as follows:

• Level 1 PRA — involves the evaluation and quantification of the frequency of the sequences

leading to core damage

• Limited Level 2 PRA — involves the evaluation and quantification of the mechanisms, and

probabilities of subsequent radioactive material releases leading to large early releases

from containment

• Level 2 PRA — involves the evaluation and quantification of the mechanisms, amounts, and

probabilities of all the subsequent radioactive material releases from the containment

• Level 3 PRA — involves the evaluation and quantification of the resulting consequences to

both the public and the environment from the radioactive material releases

Plant operating states (POSs) are used to subdivide the plant operating cycle into unique states,

such that the plant response can be assumed to be the same for all subsequent accident initiating

events. Operational characteristics (such as reactor power level; in-vessel temperature, pressure,

and coolant level; equipment operability; and changes in decay heat load or plant conditions that

allow new success criteria) are examined to identify those relevant to defining POSs. These

characteristics are used to define the states, and the fraction of time spent in each state is

estimated using plant-specific information. The risk perspective is based on the total risk

associated with the operation of the reactor, which includes not only full-power operation, but also

low-power and shutdown conditions.

Initiating events have the ability to challenge the condition of the plant. These events include

failure of equipment from either internal plant causes (such as hardware faults, operator actions,

floods, or fires) or external plant causes (such as earthquakes or high winds).
3. Understanding the PRA Model

NUREG-1855 3-4 Draft for Comment

The scope of the model may be limited on the assumption that a certain challenge or challenges

do not significantly affect the risk. On the other hand, the analyst may simply choose not to model

a certain contributor and to deal with it in another manner, as discussed in Chapter 7. 

3.2.4 Precision and Level of Detail

The level of detail of a PRA is defined in terms of the degree to which (1) the logic models are

discretized and (2) plant representation is modeled. Ultimately, the degree of detail required of the

PRA is determined by how it is intended to be used.

 

The logic models of a PRA (i.e., the event trees and fault trees) are a simplified representation of

the complete range of potential accident sequences. For example, modeling all the possible

initiating events or all the ways a component could fail would create an unmanageable and

unwieldy model. Consequently, simplifications are achieved by making approximations. For

example, in developing an accident sequence timeline, a representative sequence is generally

chosen that assumes that all the failures of the mitigating systems occur at specific times (typically

the time at which the system is demanded). However, in reality, the failures could occur over an

extended time period (e.g., the system could fail at the time demanded or could fail at some later

time). Developing a model that represents all the possible times the system could fail and the

associated scenarios is not practical. The time line is used to provide input to the human reliability

analysis. Typically, a time is chosen that provides the minimum time for the operator to receive the

cues and to complete the required action. This minimized time maximizes the probability of failure.

This simplification, therefore, leads to an uncertainty in the evaluation of risk which is essentially

unquantifiable without developing more detailed models that more explicitly model different

timelines. The assumption is that the simplification is adequate for the purpose for which the model

is being developed. It is also generally assumed that the simplification results in a somewhat

conservative assessment of risk.

The degree to which plant performance is represented in the PRA model also has an effect on the

precision of the evaluation of risk. For each technical element of a PRA, the level of detail may vary

by the extent to which the following occur:

(1) Plant systems and operator actions are credited in modeling the plant design and operation.

(2) Plant-specific experience and the operating history of the plant’s structures, systems, and

components are incorporated into the model.

(3) Realism is incorporated in the deterministic analyses to predict the expected plant

responses.

The level of detail in the way the logic models are discretized and the extent to which plant

representation is modeled is at the discretion of the analyst. The analyst may screen out initiating

events, component failure modes, and human failure events, so that the model does not become

encumbered with insignificant detail. For example, not all potential success paths may be modeled.

However, there is a certain level of detail implicit in the requirements of the PRA standard. While

an analyst chooses the level of detail, the PRA model must be developed enough to correctly model

the major dependencies (e.g., those between front line and support systems) and to capture the

significant contributors to risk. Nonetheless, this discretion in choosing the level of detail chosen

leads to a lack of precision, which results in uncertainty with respect to the predictions of the model.

The generally conservative bias that results could be removed by developing a more detailed
3. Understanding the PRA Model

Draft for Comment 3-5 NUREG-1855

model. The need to do so would be driven by a particular application. The purpose of Chapters

4, 5, and 6 of this document is not to address this issue but to address the uncertainty in the

formulation of the model constructed under the boundary conditions created by the discretization

and level of detail. 

3.2.5 Results of the PRA (Aggregation of the Results)

Typically, the results produced by the PRA model include the following:

• core damage frequency (CDF)

• large early release frequency (LERF)

• identification of the relative importance of various contributors including

— initiating events

— accident sequences

— component failures or unavailabilities

— human failure events

For many applications it is necessary to sum the contributions from all the contributors to a specific

risk metric such as CDF, LERF, or an importance measure. This summation has sometimes been

referred to as “aggregation.” Since the initiating events are independent, addition of the

contributions is mathematically correct. However, there are several issues that should be

considered when aggregating PRA results. First, it is important to note that, when combining the

results of PRA models for several risk contributors (e.g., internal initiating events, internal fires,

seismic events), as required by many acceptance criteria, the level of detail and level of

approximation may differ from one contributor to the next, with some being more conservative than

others. This modeling difference is true even for an internal events, at-power PRA. For example,

the evaluation of room cooling and equipment failure thresholds can be conservatively evaluated

leading to a conservative time estimate for providing means for alternate room cooling.

Furthermore, at-power PRAs follow the same general process as used in the analysis of other

contributors with regard to screening: low risk sequences can be modeled to a level of detail

sufficient to prove they are not important to the results. 

Significantly higher levels of conservative bias can exist in PRAs for external events, lower power

and shutdown (LPSD), and internal fire PRAs. These biases result from several factors including

the unique methods or processes and the inputs used in these PRAs, as well as the scope of the

modeling. For example, the fire modeling performed in a fire PRA can use simple scoping models

or more sophisticated computer models and may not mechanistically account for all factors such

as the application of suppression agents. Further, fire PRAs do not always credit all mitigating

systems in an effort to reduce the number of cables that have to be located. To a certain level,

conservative bias will be reduced by the development of PRA standards, detailed models, and

corresponding guidance for the analysis of external events, fires, and LPSD that will provide a

similar level of rigor currently used in at-power PRAs. However, as with internal at-power PRAs,

the evaluation of every aspect of these other contributors will likely include some level of

conservatism that may influence a risk-informed decision.

The level of detail, scope, and resulting conservative biases in a PRA introduces uncertainties in

the PRA results. Since conservative bias can be larger for external events, fire, and LPSD risk

contributors, the associated uncertainties can be larger. The propagation of parametric
3. Understanding the PRA Model

NUREG-1855 3-6 Draft for Comment

uncertainties through the model and the use of mean CDF and LERF values can account for

parameter uncertainty. Model uncertainty should be addressed using the guidance provided in

Chapter 5. The outcome of that effort is that more model uncertainties will be identified and have

to evaluated in order to support a risk-informed application. However, a higher level of uncertainty

does not preclude the aggregation of results from different risk contributors; but it does require

consideration of all uncertainties that have a significant impact on the risk-informed application.

Finally, it is important to note that the process of aggregation can be influenced by the type of riskinformed application. For example, it is always possible to add CDF and LERF, and changes in

CDF and LERF contributions from different risk contributors for comparison against corresponding

criteria. However, in doing so, one should always consider the influence of known conservatism

when comparing the results against the criteria, particularly if they mask the real risk contributors

or result in exceeding the criteria. If the criteria are exceeded due to a conservative analyses, then

it is always possible to perform a more detailed, realistic analysis to reduce the conservatism and

uncertainty, or address the issue in another way, such as taking compensatory measures to

address the cause of conservatism. For applications that use risk importance measures to prioritize

specific actions (e.g., special treatment), a conservative bias can significantly impact the identified

risk ranking. Furthermore, some importance measures can not be added together from different

analyses and thus would require a true integration of the different risk models in order to evaluate

them. 

Chapter 7 provides guidance on how to aggregate the result from different risk contributors for use

in risk-informed decision making. To facilitate this effort, it is best that results and insights from all

of the different risk analyses relevant to the application be provided to the decision maker in

addition to the aggregated results. This information will allow for consideration of all conservatism

associated with any of the risk contributors and will help focus the decision-maker on those aspects

of the analysis that have the potential to influence the outcome of the decision.

3.3 PRA Models and Uncertainty

PRA models are constructed as probabilistic models to reflect the random nature of the constituent

basic events such as initiating events and component failures. Randomness is one manifestation

of a form of uncertainty that has come to be called aleatory uncertainty. A PRA is, therefore, a

probabilistic model that characterizes the aleatory uncertainty associated with accidents at nuclear

power plants. The focus of this document is guidance in dealing with another area of uncertainty,

namely epistemic uncertainty. This uncertainty is associated with the incompleteness in the

analysts’ state of knowledge and has an impact on the results of the PRA model. The following

sections identify and describe examples of sources of epistemic uncertainty.

3.3.1 Examples of Epistemic Uncertainty in PRA

Uncertainties in the PRA models arise for many different reasons, including the following:

• For many of the basic events of the PRA model, there are generally accepted probability

models. These models are typically simple, with only one or two parameters. Examples

include the simple constant failure rate reliability model, which assumes that the failures of

a component while it is on standby occur at a constant rate, and the uniformly distributed

(in time) likelihood of an initiating event. The model for both these processes is the wellknown Poisson model. The parameter(s) of such models can be estimated using
3. Understanding the PRA Model

Draft for Comment 3-7 NUREG-1855

appropriate data, which in the examples above comprise the number of failures observed

in a population of like components in a given time, and the number of occurrences of a fire

scenario in a given time, respectively. Statistical uncertainties are associated with the

estimates of the parameters of the model. Since most of the events that constitute the

building blocks of the risk model (e.g., some initiating events, operator errors, and

equipment failures) are relatively rare, the data are scarce, and the uncertainties can be

relatively significant.

• For some events, while the basic probability model is generally agreed on, uncertainties

may be associated with interpreting the data to be used for estimation. For example, when

collecting data on component failures from maintenance records, it is not always clear

whether the failure would have prevented the component from performing the mission

required of it to meet the success criteria assumed in the risk model.

• For some basic events, there can be uncertainty as to how to model the failures, which

results in uncertainties in the probabilities of those failures. One example is the behavior

of RCP seals on loss of cooling. Another example is the modeling of human performance

and the estimation of the probabilities of human failure events.

• There can be uncertainty about the capability of some systems to perform their function

under the conditions specified by the developed scenarios. This leads to uncertainty in

characterizing the success criteria for those functions, which has an impact on the logic

structure of the model.

As seen in these examples, the uncertainty associated with the structure of and input to the PRA

model can be in the choice of the logic structure, the mathematical form of the models for the basic

events, or it can be in the values of the parameters of those models, or it can be in both. To the

extent that changes in parameter values are little more than subtle changes in the form of the

model, it can be argued that there is really no precise distinction between model uncertainty and

parameter uncertainty. However, as discussed below, parameter uncertainties and model

uncertainties are dealt with differently. It should also be noted that, while the Poisson and binomial

models are typically adopted for the occurrence of initiating events and for equipment failures, these

are assumptions and may not be appropriate for all situations.

3.3.2 Types of Epistemic Uncertainty

As noted above, it is helpful to categorize uncertainties into those that are associated with the

parameter values and those that involve aspects of models. This categorization is primarily

because the methods for the characterization and analysis of uncertainty are different for the two

types. However, a third type of uncertainty exists, namely uncertainty about the completeness of

the model. While this type of uncertainty cannot be handled analytically, it must be considered

when making decisions using the results of a risk analysis.

Parameter Uncertainty: Parameter uncertainty is the uncertainty in the values of the parameters

of a model, given that the mathematical form of that model has been agreed to be appropriate.

Current practice is to characterize parameter uncertainty using probability distributions on the

parameter values [SNL, 2003a]. When the parameters are combined algebraically to evaluate the

PRA numerical results, or some intermediate result such as a basic event probability, these

uncertainty distributions can be mathematically combined in a simple way to estimate the
3. Understanding the PRA Model

NUREG-1855 3-8 Draft for Comment

uncertainty of those numerical results. When there is uncertainty as to which model to use to

estimate the probability of the basic events, this is more appropriately addressed as a model

uncertainty.

Model Uncertainty: Model uncertainty is related to an issue for which there is no consensus

approach or model and where the choice of approach or model is known to have an effect on the

PRA model (e.g., introduction of a new basic event, changes to basic event probabilities, change

in success criterion, introduction of a new initiating event). A model uncertainty results from a lack

of knowledge of how systems, structures, and components behave under the conditions arising

during the development of an accident. A model uncertainty can arise for the following reasons:

• The phenomenon being modeled is itself not completely understood (e.g., behavior of

gravity-driven passive systems in new reactors, crack growth resulting from previously

unknown mechanisms).

• In a subset of the preceding item, some data or other information may exist, but it must be

interpreted to infer behavior under conditions different from those in which the data were

collected (e.g., RCP seal LOCA information).

• The nature of the failure modes is not completely understood or is unknown (e.g., digital

instrumentation and controls).

As indicated in Section 3.3.1, model uncertainty may be manifested in uncertainty about the logic

structure of the PRA model or in the choice of model to estimate the probabilities associated with

the basic events. 

Completeness Uncertainty: Completeness is not in itself an uncertainty, but a reflection of a

limitation in the scope of the model. The result is, however, an uncertainty about where the true

risk lies. This can also be thought of as a type of model uncertainty. However, completeness

uncertainty is discussed separately because it represents a type of uncertainty that cannot be

quantified and because it represents those aspects of the system that are, either knowingly or

unknowingly, not addressed in the model. 

The problem with completeness uncertainty in PRAs is that, because it reflects an unanalyzed

contribution, it is difficult (if not impossible) to estimate its magnitude. Incompleteness in the model

can be thought of as arising in two different ways:

• Some contributors/effects may be knowingly left out of the model for a number of reasons.

For example, methods of analysis have not been developed for some issues, and these

gaps have to be accepted as potential limitations of the technology. Thus, for example, the

impact on actual plant risk from unanalyzed issues such as the influences of organizational

performance cannot now be explicitly assessed. As another example, the resources to

develop a complete model may be limited, leading to a decision not to model certain

contributors to risk (e.g., seismically induced fires).

• Some phenomena or failure mechanisms may be omitted because their potential existence

has not been recognized.
3. Understanding the PRA Model

Draft for Comment 3-9 NUREG-1855

3.3.3 Assessing the Impact of Uncertainty

While there may be many sources of uncertainty in a PRA, they do not all have an impact on a

particular decision. Those that can affect the results used to support a decision are identified as

key sources of uncertainty, and these demand attention. The way that the uncertainties are

assessed is a function of the way the acceptance criteria (or acceptance guidelines) for the decision

are defined and the way that the uncertainties are characterized with regard to the acceptance

guidelines.

 

For example, in the context of RG 1.174 [NRC, 2002a], the acceptance guidelines are defined to

demonstrate, with reasonable assurance, that the risk change is small. This decision must be

based on a full understanding of the contributors to the PRA results and the impacts of both the

uncertainties that are explicitly accounted for in the results and those that are not. This process is

somewhat subjective, and the reasoning behind the decisions must be well documented. The

acceptance guidelines of RG 1.174, for example, are not intended to be interpreted as being overly

prescriptive. They are intended to indicate in numerical terms what is considered acceptable.

RG 1.174 uses the term “acceptance guidelines” rather than “acceptance criteria” primarily to

recognize that the numerical results do not capture all of the analysis uncertainty.

The following provides an overview of the assessment of each type of uncertainty (parameter,

model, and completeness). Detailed guidance appears in Chapters 4, 5, and 6, respectively.

Parameter Uncertainty: For many parameters (e.g., initiating event frequencies, component

failure probabilities or failure rates, human error probabilities) the uncertainty may be characterized

as subjective probability distributions. Chapter 4 discusses the methods for propagating these

uncertainties through the analysis to characterize the uncertainty in the numerical results of the

analysis. In this manner, the impact of the parameter uncertainties on the numerical results of the

PRA can be assessed integrally. However, many of the acceptance criteria or guidelines used in

regulatory decision making (e.g., the acceptance guidelines of RG 1.174) are defined such that the

appropriate measure for comparison is the mean value of the uncertainty distribution on the

corresponding metric. In this case, as discussed in Chapter 4, the primary issue with parameter

uncertainty is its effect on the calculation of the mean, and specifically, on the relevance and

significance of the state-of-knowledge correlation.

Model Uncertainty: While the analysis of parametric uncertainty is fairly mature and is addressed

adequately through the use of probability distributions on the values of the parameters, the analysis

of the model and completeness uncertainties cannot be handled in such a formal manner. The

typical response to a modeling uncertainty is to choose a specific modeling approach to develop

in the PRA model. While it is possible to embed a characterization of model uncertainty into the

PRA model by including several alternate models and providing weights (probabilities) to represent

the degree of credibility of the individual models, this is not usual. Therefore, when using the

results of the model, it is necessary to demonstrate that, for the key uncertainties, reasonable

alternative hypotheses, adjustment factors, or modeling approximations or methods would not

significantly change the assessment. Chapter 5 discusses methods for performing such

demonstrations. 

In dealing with model uncertainties, it is helpful to identify whether the model uncertainties can alter

the logic structure of the PRA model, or whether their primary impact is on the probabilities of the

basic events of the logic model. For example, an uncertainty associated with the establishment of
3. Understanding the PRA Model

2

Note that this definition is somewhat different from that in the Electric Power Research Institute's "Guideline

for the Treatment of Uncertainty in Risk-Informed Applications: Technical Basis Document" [EPRI, 2004a], although

the intent is similar.

NUREG-1855 3-10 Draft for Comment

the success criterion for a specific system can result in an uncertainty as to whether one or two

pumps is required for a particular scenario. This uncertainty would be reflected in the choice of the

top gate in the fault tree for that system. On the other hand, those model uncertainties that are

associated with choosing the model to estimate the probabilities of the basic events do not alter the

structure of the model. Because of this, tools such as importance analyses can be used to explore

the potential impact of these uncertainties in a way not possible for the former.

One approach to dealing with a specific model uncertainty is to adopt a consensus model, which

essentially removes that uncertainty from having to be addressed in the decision making. In the

context of regulatory decision making, a consensus model can be defined as follows:

Consensus model: in the most general sense, as a model that has a publicly

available published basis and has been peer reviewed and widely adopted by an

appropriate stakeholder group. In addition, widely accepted PRA practices may be

regarded as consensus models. Examples of the latter include the use of the

constant probability of failure on demand model for standby components and the

Poisson model for initiating events.2

 For risk-informed regulatory decisions, the

consensus model approach is one that the NRC has utilized or accepted for the

specific risk-informed application for which it is proposed.

The definition given here ties the consensus model to a specific application. This restriction is

because models have limitations that may be acceptable for some uses and not for others. In some

cases (e.g., the Westinghouse Owners’ Group 2000 RCP seal LOCA model), this consensus is

documented in a safety evaluation report (SER) [NRC, 2003d]. In such cases, the tendency is for

the model to be considered somewhat conservative. While no further analysis of the model

uncertainty is typically needed, it is important to recognize the potential for conservatism if it could

mask other contributors that may be important to a decision. Many models can already be

considered consensus models without the issuance of an SER. For example, the Poisson model

for initiating events has been used since the very early days of PRA.

Completeness Uncertainty: The issue of completeness of scope and level of detail of a PRA can

be addressed for those scope and level of detail items for which methods are available and,

therefore, some understanding of the contribution to risk exists. For example, the out-of-scope and

level of detail items can be addressed by supplementing the PRA with additional analysis to enlarge

the scope or increase the level of detail, or by bounding analyses to demonstrate that, for the

application of concern, the out-of-scope or level of detail contributors are not significant. Chapter 6

discusses these approaches. An alternative is to design the proposed change such that the major

sources of uncertainty will not have an impact on the decision making process, as discussed in

Chapter 7.

 

The true unknowns (i.e., those related to issues whose existence is not recognized) cannot be

addressed analytically. However, in the interests of making defendable decisions, these unknowns

must still be addressed during the decision making as discussed in Chapter 2. The principles of

safety margins and defense in depth play a critical role in addressing this source of uncertainty.
Draft for Comment 4-1 NUREG-1855

4. PARAMETER UNCERTAINTY

The purpose of this chapter is to provide guidance on how to address parameter uncertainty in the

use of PRA results for decision making. Guidance is provided for the characterization of parameter

uncertainty, propagation of parameter uncertainty, and the treatment of parameter uncertainties in

comparison of PRA results with acceptance criteria. 

RGs 1.174 [NRC, 2002a] and 1.200 [NRC, 2007a] and the ASME PRA Standard [ASME, 2005a]

all recognize the importance of the identification and understanding of the uncertainties in a PRA.

More specifically, the ASME PRA Standard (as endorsed by the staff in RG 1.200) requires that

uncertainties in the PRA are characterized, and that sources of model uncertainty and related

assumptions are identified and their impact on the results understood. In addition, there are

specific requirements for characterization of parameter uncertainties. The standard (and as

endorsed by RG 1.200) requires either (1) an estimate of the uncertainty interval of the overall CDF

results associated with the parameter uncertainties, or (2) the propagation of the parameter

uncertainties (and those model uncertainties explicitly characterized by a probability distribution).

The specific parameter uncertainties to be characterized are also identified in the standard and

include the basic events of initiating event frequencies, component failures and unavailabilities, and

human-error probabilities (HEPs). For each of the basic events, the standard requires the

characterization of the uncertainty interval on the parameter values used to estimate the basic

event probability. This approach allows the uncertainty in the final risk metric to be quantified.

The requirements described above are the extent of the level of detail provided in the standard.

This chapter provides more detailed guidance on the characterization, propagation, and treatment

of parameter uncertainties. Accordingly, this chapter has three main sections:

• Characterization of Parameter Uncertainty for Basic Events — This section discusses the

characterization of uncertainty on the parameters used to quantify the basic event

probabilities.

• Propagation of Uncertainty — This section discusses the propagation of the uncertainties

on the input parameters to provide an assessment of the uncertainty on the quantitative

results of the PRA (e.g., accident sequence frequencies, CDF).

• Treatment of Parameter Uncertainties — This section describes how the resultant

uncertainty in the PRA results is used in comparison with acceptance criteria or guidelines

associated with an application.

4.1 Characterization of Parameter Uncertainty for Basic Events

Guidance is provided in this section for the characterization of the parameter uncertainties. The

parameter uncertainties are characterized by the probability distributions used for the parameters

of the basic event model. There are two statistical approaches used for this characterization,

namely the frequentist and Bayesian methods, but usually the latter is used. In some cases,

however, detailed analyses and evidence (i.e., data) about the event are very limited or unavailable.

In these cases, it is not feasible to use a mathematical model, and it is necessary to rely on the

knowledge of experts in the specific technical field associated with the basic event. Accordingly,

the guidance involves the following:
4. Parameter Uncertainty

3

In the rare circumstance that there is no agreement on the choice of probability model for the basic event,

this disagreement or choice can be regarded as a model uncertainty and would be addressed as discussed in

Chapter 5.

NUREG-1855 4-2 Draft for Comment

• General Considerations — This section discusses general considerations for characterizing

parameter uncertainty of basic events and compares the two statistical approaches. 

• Bayesian Estimation of Parameter Uncertainty — This section presents the fundamental

principles of the use of Bayes’ Theorem for estimating parameter uncertainty distributions.

• Additional Issues — This section provides two other topics for consideration when

characterizing parameter uncertainty.

4.1.1 General Considerations

This section discusses general considerations for characterizing parameter uncertainty of basic

events and compares the two statistical approaches used for this characterization, namely the

frequentist and Bayesian methods. As described below, for practical reasons the Bayesian

approach is the one that is most commonly used. 

The process for estimating the probability of occurrence of a basic event comprises two major

steps:

(1) Determine the appropriate probability model to represent the basic event. These probability

models typically have one or more parameters.

(2) Estimate the values of these parameters. This estimation is based on the most applicable

and available data.

Both steps introduce some uncertainty into the PRA model.3

 For many of the basic events of the

PRA model, however, there are generally accepted basic event models (i.e., probability models),

which are typically simple. Examples include the simple constant failure rate reliability model, which

assumes that the failures of a component while it is in standby occur at a constant rate, and the

uniformly distributed (in time) likelihood of an initiating event. The model for both these processes

is the well-known Poisson model. The parameter(s) of such models can be estimated using

appropriate data, which in the examples above comprise the number of failures observed in a

population of like components in a given time and the number of occurrences of an initiating event

in a given time, respectively. 

There is a statistical uncertainty associated with the estimation of parameters based on data. This

chapter discusses this kind of uncertainty. Since most of the basic events that constitute the

building blocks of the risk model (e.g., some initiating events, and equipment failures) are relatively

rare events, the data for estimating the parameters for each basic event consist only of a relatively

small set of randomly generated data or are even unavailable. Therefore, this uncertainty can be

significant.

Usually an analyst associates a basic event model (i.e., a probability distribution) with each basic

event. For example, initiating events are typically modeled by a Poisson process, so the number
4. Parameter Uncertainty

4

This chapter denotes a random variable by an uppercase letter (e.g., X) and represents an observed or

observable value of the random variable (a number) by a lowercase letter (e.g., x). The probability that a random

variable X takes the value x is denoted by Pr(X=x). A probability density function (PDF) and a cumulative distribution

function (cdf) are denoted as f(x) and F(x), respectively.

Draft for Comment 4-3 NUREG-1855

of occurrences X during some fixed exposure time t is a Poisson distributed random variable, and

the probability of x initiating events in time t is

Pr(X=x) = e-λt

 (λt)x / x! Equation 1

where λ, the event frequency, has units of events per unit time. This Poisson distribution is a basic

event model and has one parameter, λ. In this model, the exposure time t is treated as fixed, and

the number of events is treated as random.4

As noted above, there are two basic statistical approaches for estimating parameters and their

associated uncertainty—the frequentist, or classical, method and the Bayesian method. These two

methods and a comparison of them are discussed below.

Frequentist Approach

In the frequentist approach, the probability of a random event is defined as the long-term fraction

of times that the event would occur in a large number of trials. Probabilities are used only for

random quantities, the possible data values. Probability distributions are never used to describe

parameters because parameters are not considered to be random.

The primary use of the frequentist approach is to conduct a preliminary examination of the data,

to check the correctness of model assumptions, and to decide which model to use. For example,

frequentist methods can help the analyst decide whether data sets may be pooled or whether a

trend is present. Goodness-of-fit tests and calculation of statistical significance are commonly used

frequentist tools in this context.

Bayesian Approach

In the Bayesian approach, probability is a measure of uncertainty, a quantification of degree of

belief. The Bayesian methodology is used to modify uncertainty in a logically coherent way, so that

“degree of belief” is rational, not merely personal opinion.

When estimating a parameter and associated uncertainty for PRA, data from reliable equipment

are typically sparse, with few or even zero observed failures. In such cases, it is reasonable to

draw on other sources of information. The Bayesian approach provides a mechanism for

incorporating such information as prior belief. It also allows straightforward propagation of basic

event uncertainties through a logical model to produce an uncertainty on the frequency of the

undesirable end state, such as core damage. To do this, an analyst estimates a probability

distribution for each of the unknown parameters and typically enters the distributions of the

parameters into a PRA computer code. The code then draws a random sample from each

distribution and constructs the corresponding sample for the frequency of the undesirable end state.
4. Parameter Uncertainty

NUREG-1855 4-4 Draft for Comment

PDFs used in PRAs to characterize uncertainty include, in alphabetical order, beta, gamma,

inverted chi-squared, logistic-normal, log-normal, maximum entropy, normal, Poisson, and uniform.

The properties and parameters of all these PDFs are presented in statistical textbooks and in

reports providing guidance for conducting a PRA. In particular, NUREG/CR-6823 [SNL, 2003a] is

a handbook on estimating parameters and their associated uncertainty. Appendix A of that

NUREG/CR provides the basics of probability theory and useful properties of several probability

distributions commonly used in PRAs.

Frequentist and Bayesian Comparison

The frequentist method considers that a parameter is a constant that is fixed. On the other hand,

the Bayesian approach considers that a parameter is a constant, but it is assigned a probability

distribution, measuring the current state of belief. Accordingly, the two approaches differ in the way

they treat uncertainty of unknown parameters. Therefore, the way the uncertainty of an unknown

parameter is characterized depends on the method used to assess the parameter and its

uncertainty. 

Since the frequentist approach cannot handle complicated propagation of uncertainties except by

rough approximations for estimating a parameter and associated distribution and uncertainty, the

Bayesian approach works better.

Frequentist estimates are often simpler to calculate than Bayesian estimates and therefore are

useful for rough approximate calculations.

The characteristics of these two methods for assessing the uncertainty of unknown parameters in

a PRA are summarized in Table 4-1, extracted from Appendix B to the Data Handbook.

Table 4-1 Characteristics of Methods for

Assessing Parameter Uncertainty

Bayesian Approach Frequentist Approach

Because Bayesian probability intervals can be

interpreted as probability statements about a

parameter, they are easily combined with

other sources of uncertainty in a PRA using

the laws of probability.

A confidence interval cannot be directly

interpreted as a probability that the parameter

lies in the interval.

Bayesian distributions can be propagated

through fault trees, event trees, and other

logic models. 

It is difficult or impossible to propagate

frequentist confidence intervals through fault

and event tree models common in PRA to

produce corresponding interval estimates on

output quantities of interest.

For the reasons mentioned above, the frequentist approach is not commonly used for assessing

the uncertainty of a parameter as part of a PRA; hence, it is not discussed further in this chapter.

An overview of the use of the Bayesian method for estimating an unknown parameter and its

associated uncertainty is provided.
4. Parameter Uncertainty

5

An example of other relevant failures is the occurrence of a LOCA due to the failure of the seals of the

RCPs of a pressurized-water reactor (PWR). An example in a level-2 PRA is the failure of the containment.

Draft for Comment 4-5 NUREG-1855

In addition, it is important to understand that the way that the uncertainty of each basic event is

characterized is dependent on the type of basic event. The types of basic events generally found

in PRAs include events that represent the following:

• initiating events

• failures to start or change state of components

• failures to run or maintain state of components

• unavailabilities of components from being out of service

• failures to recover structures, systems, and components (SSCs), such as failure to recover

offsite power within a certain time

• common-cause failures (CCFs) of components

• human errors that contribute to the occurrence of initiating events or failure of the mitigating

SSCs

• other relevant failures5

Although the guidance provided in concept applies to all the types of basic events listed above, for

the latter three, the reader is advised to use specific guidance and therefore consult other sources,

such as the following:

CCFs of components

• NUREG/CR-5497 [INL, 1998a]

• NUREG/CR-6268 [INL, 1998b]

• NUREG/CR-5485 [INL, 1998c]

• NUREG/CR-4780 [EPRI, 1988a]

• EPRI NP-3967 [EPRI, 1985a]

Human errors

• NUREG-1792 [NRC, 2005a] 

Other relevant failures

• uncertainty related to these failures may be estimated using expert judgment elicitation.
4. Parameter Uncertainty

NUREG-1855 4-6 Draft for Comment

4.1.2 Bayesian Estimation of Parameter Uncertainty

This section presents the fundamental principles of the use of Bayes’ Theorem for estimating

parameter uncertainty distributions. This overview is mainly based on Appendices A and B to the

Data Handbook; the reader is advised to see this handbook or other reputable document for a more

complete discussion on this method. In addition, Siu and Kelly [Siu, 1998a] give a thorough

introduction to Bayesian estimation for PRA. 

Bayesian estimation of parameter uncertainty incorporates degree of belief and information beyond

that contained in the data sample. The prior belief about a parameter’s value is contained in what

is referred to as the prior distribution, which describes the state of knowledge (or subjective

probability) about the parameter, prior to obtaining the data sample. Therefore, in the Bayesian

approach, the parameters of the sampling distribution have probability distributions. These

probabilities do not model random variability of the parameters but the analyst’s degree of belief

about the true values of the unknown parameters. The distributions are sometimes called

uncertainty distributions. A Bayesian uncertainty distribution satisfies all the rules of a probability

distribution.

The Bayesian method uses Bayes’ Theorem, which is given here for continuous PDFs. Suppose

X is a discrete or continuous random variable, with PDF depending on parameter θ, and with

conditional PDF of X, given θ, specified by f(x/θ). Also suppose that θ has a continuous probability

distribution with PDF g(θ). In the case of parameter estimation, θ is an uncertain parameter with

a subjective uncertainty distribution. Call g(θ) the prior PDF. Then for every x such that f(x) > 0

exists, the posterior PDF of θ, given X = x, is

g x Equation 2 fx g

f x (/) ( / )()( ) θ θ θ =

where

Equation 3

is the marginal PDF of X. The prior and posterior PDFs can be used to represent the probability

of various values θ prior to and posterior to observing a value of another random variable X. 

Bayesian estimation consists of two main areas, both of which use the notion of subjective

probability. The first area involves using available data to assign a subjective, prior distribution to

a parameter, such as a failure rate. The degree of belief about the uncertainty in a parameter value

is expressed in the prior distribution. This assignment of a prior distribution does not involve the

use of Bayes’ Theorem. The second area of Bayesian estimation involves using additional or new

data to update an existing prior distribution. This is called Bayesian updating and directly uses

Bayes’ Theorem. 

Bayes’ Theorem can be seen to transform the prior distribution by the effect of the sample data

distribution to produce a posterior distribution. The sample data distribution, f(x/θ), can be viewed

as a function of the unknown parameter, instead of the observed data, xi

, producing a likelihood

function.

fx fx g d () ( / )() = ∫ θ θθ
4. Parameter Uncertainty

Draft for Comment 4-7 NUREG-1855

Using the likelihood function, the fundamental relationship expressed by Bayes’ Theorem is

Posterior Distribution = (Prior Distribution) (Likelihood)

Marginal Distribution

The marginal distribution serves as a normalizing constant. 

In Bayesian updating, the sampling distribution of the data provides new information about the

parameter value. Bayes’ Theorem provides a mathematical framework for processing new sample

data as they become sequentially available over time. With the new information, the uncertainty

of the parameter value has been reduced, but not eliminated. Bayes’ Theorem is used to combine

the prior and sampling distributions to form the posterior distribution, which then describes the

updated state of knowledge (still in terms of subjective probability) about the parameter. Point and

interval estimates of the parameter can then be obtained directly from the posterior distribution,

which is viewed as containing the current knowledge about the parameter. This posterior

distribution can then be used as the prior distribution when the next set of data becomes available.

Thus, Bayesian updating is successively implemented using additional data in conjunction with

Bayes’ Theorem to obtain successively better posterior distributions that model a plant-specific

parameter. 

Bayesian point and interval estimates are obtained from both the prior and posterior distributions.

The interval estimates are subjective probability intervals, or credible intervals. The terminology

is not yet universally standard. A credible interval can be interpreted as a subjective probability

statement about the parameter value, unlike classical interval estimates. That is, the interpretation

of a 95 percent Bayesian posterior probability interval (a, b) is that, with 95 percent subjective

probability, the parameter is contained in the interval (a, b), given the prior and sampling

distributions. 

Bayesian parameter estimation involves four steps. The first step is identification of the

parameter(s) to be estimated, which involves consideration of the assumed distribution of the data

that will be collected. The second step is development of a prior distribution that appropriately

quantifies the state of knowledge concerning the unknown parameter(s). The third step is collection

of the data sample. The fourth and final step is combining the prior distribution with the data

sample using Bayes’ Theorem to produce the desired posterior distribution.

For PRA applications, determining the prior distribution is usually based on generic data and the

data sample usually involves site-specific or plant-specific operating data. The resulting posterior

distribution would then be the site-specific or plant-specific distribution of the parameter.

The plant-specific data collected are assumed to be a random sample from an assumed sampling

distribution. The data are used to update the prior distribution, producing the posterior distribution.

Point estimates, such as the most likely value (the mode), the median, or (most commonly) the

mean value, and probability interval estimates of the parameter can then be obtained. Other

bounds and other point values can also be obtained with the Bayesian approach because the

posterior parameter distribution is entirely known and represents the available knowledge about the

parameter.
4. Parameter Uncertainty

NUREG-1855 4-8 Draft for Comment

Using the posterior PDF, a symmetric 100(1 - γ) percent two-sided Bayes probability interval

estimate of the parameter is obtained by solving the two equations for the lower limit θL and the

upper limit θU.

Equation 4

Equation 5

The interval estimate (θL, θU) would then be such that

 Pr(θL < θ < θU / x) = 1 - γ 

The Bayes interval estimate is an explicit probability statement of the true parameter contained in

the interval. For example, if γ = 0.05, there is a 0.95 probability that this interval really includes the

unknown parameter. 

Thus, the uncertainty in θ is expressed by the posterior distribution, which is a probability

distribution. The distribution’s mean and width are the best estimate of θ and a measure of the

uncertainty in θ, respectively. A wide or narrow distribution represents large or small uncertainty

in the true value of θ, respectively.

4.1.3 Additional Issues

There are other issues that need to be considered when characterizing parameter uncertainty.

Parameter uncertainty gets larger with increasing scarcity of data. However, uncertainty associated

with these data may also be relevant — uncertainty from the nonrepresentativeness of the data

sources and uncertainty in the data counts themselves.

One issue to consider is that the data are sometimes from settings that do not perfectly match the

problem of interest. For example, suppose one situation is of interest, but the data are from

equipment with a different manufacturer or different design, or from equipment operated under

different conditions, or maintained with different practices. It is then difficult to quantify the

relationship between the data and the problem of interest. Engineering judgment is used, and, to

be conservative, the uncertainty distribution is often assigned a larger uncertainty than would be

called for by the data alone.

If a case can be made that the data are somewhat representative, then the situation becomes more

tractable. One example is uncertainty of the value of a parameter for one data source (such as one

nuclear power plant (NPP)) when data are available from many similar but not identical data

sources (other NPPs). This case can be formulated in terms of a hierarchical model and analyzed

by empirical Bayes or hierarchical Bayes methods, as discussed in Chapter 8 of the Data

Handbook.

g xd L(/) θ θ

θ γ = ∫ − ∞ 2

g xd

U

(/) θ θ γ

θ = ∞

∫ 2
4. Parameter Uncertainty

Draft for Comment 4-9 NUREG-1855

There can be uncertainty in the data counts themselves. For example, it may be unclear whether

a particular event should be counted as a failure, or the number of demands may not be known

exactly. A Bayesian method for dealing with uncertainty in PRA data was first proposed by Siu and

Apostolakis [Siu, 1984a; Siu, 1986a] and has been used by several authors, including Mosleh

[Mosleh, 1986a], EPRI [EPRI, 1988a], and Martz and Picard [Martz, 1995a]. As outlined by Atwood

and GentilIon [Atwood, 1996a], uncertainty in classifying the data yields a number of possible data

sets, each of which can be assigned a subjective probability. One approach is to use an “average”

data set, a “best estimate” of the data, and analyze it. The uncertainty in the data is ignored, lost,

at that point. A second approach is to analyze each data set and combine the results. Each

analysis produces a Bayesian distribution for the unknown parameter(s), and the final result is a

mixture of these distributions. This approach includes the data uncertainty in the analysis and

results in wider uncertainty intervals than the first approach. A third approach is to perform

sensitivity analyses using the sets of data separately to see if there are significant differences in

the results.

4.2 Propagation of Uncertainty

Guidance is provided in this section for the propagation of the uncertainties on the input parameters

through the analysis to the PRA results. 

It is necessary that the uncertainties of the parameters of the PRA basic events be expressed in

a way that allows them to be combined to estimate the uncertainty of the risk metrics of the PRA,

such as the CDF, the LERF, or some intermediate result such as the frequency of an accident

sequence. In other words, the uncertainty in the parameters of the basic event models should be

quantified in a way that allows the uncertainty in the final risk metric to be quantified. In this way,

the uncertainty in the parameters of these models can be propagated to the intermediate results

and to the plant-level measures of risk, such as the CDF. Therefore, the impact of the parameter

uncertainties on these measures can be assessed in an integral manner. Finally, the top-level

results, such as CDF and LERF, including their assessed uncertainties, can be compared against

acceptance guidelines to use these risk metrics for decision making.

As discussed in Section 4.1, each basic event probability is treated as a random variable with an

associated uncertainty (i.e., probability, distribution). Since a risk metric such as CDF is a function

of these random variables, it is itself a random variable. The term “propagation of uncertainty”

refers to the process of determining the distribution of a risk metric (or of another intermediate

result, such as the probability of a top event) from the individual distributions of the basic events.

This propagation is needed to provide an assessment of the uncertainty on the quantitative results

of the PRA (e.g., accident sequence frequencies, CDF). This guidance covers the following:

• Monte Carlo — This section discusses the most fundamental approach for propagating

uncertainty — repeated quantification of the probabilistic metric of interest.

• Latin Hypercube Sampling (LHS) — This section discusses a stratified sampling technique

that tries to improve on the accuracy and precision of the Monte Carlo approach.

• Interpretation of Uncertainty of Risk Metric — This section discusses the representation and

interpretation of the uncertainty measures obtained with the above two methods.
4. Parameter Uncertainty

NUREG-1855 4-10 Draft for Comment

• State-of-Knowledge Correlation (SOKC) — This section discusses an issue for risk

calculations that, under some circumstances, can result in incorrect estimates of the mean

and uncertainty of the distribution of a risk metric.

4.2.1 Monte Carlo

 Monte Carlo simulation consists of making repeated quantifications of the probabilistic metric of

interest. For each quantification or Monte Carlo run, the following steps are automatically carried

out by a computer code — (1) for each basic event model of the PRA, the probability distribution(s)

of its parameter(s) are sampled using a random number generator, (2) the basic event models of

the PRA are then evaluated using their corresponding parameters, and (3) the risk metric is

calculated using the resulting values from these models. This procedure is repeated a

predetermined number of times, and the various outcomes of the risk metric are used to obtain

empirical estimates of the desired attributes of this metric, such as mean and variance. 

As a risk metric is evaluated using more and more Monte Carlo samples, the precision of the

empirical estimates of the distribution of this metric improves. However, Martz et al. [LANL, 1983a]

point out that the rate of convergence to the true distribution tends to decrease as the number of

samples increases. 

A “random number seed” or simply “seed” is provided by the analyst and is the seed to start the

random number generator. It is important to verify convergence of the simulation by demonstrating

that the results are not sensitive to the choice of seeds and increases in the number of samples.

Many computer codes have been written to perform Monte Carlo simulations. For example, the

Reactor Safety Study [NRC, 1975a] used the code SAMPLE. More recently, the Monte Carlo

method has been included in several codes, such as the Systems Analysis Programs for Hands-on

Integrated Reliability Evaluations (SAPHIRE) code (see, for example, NUREG/CR-6116, Vol. 1

[EG&G, 1993a]). The online help for this code states that “...Any number of samples greater than

or equal to ten will be allowed [by the code], but a number of at least 1000 is probably a better value

for improving the reliability of the Monte Carlo results.”

The Monte Carlo method has the following advantages:

• The method offers complete flexibility in the selection of parameter distributions of the basic

event models. 

• Any specified precision of the risk metric attributes can be achieved, limited only by cost and

roundoff error. 

• The method is reasonably easy to implement. 

4.2.2 Latin Hypercube Sampling

LHS was developed to improve upon the accuracy and precision of Monte Carlo simulations in

estimating functions of multiple random variables [SNL, 1984a]. LHS is a stratified sampling

technique, where n different values are selected for each parameter of the basic event models. The

values are selected by dividing each parameter’s probability distribution into n intervals, each of
4. Parameter Uncertainty

Draft for Comment 4-11 NUREG-1855

equal probability. Within each interval, one value of each parameter’s probability is randomly

selected. The probability of a basic event is evaluated using the corresponding n values of its

parameter. The n values of this particular random variable (basic event) are then combined with

the n values of the other basic events. The result is an n x k matrix (k is the number of sampled

basic events), where the ith row of the matrix contains specific values of each random variable to

be used in the ith run or calculation of the risk metric value. The risk metric’s distribution and

quantiles are empirically estimated from the n runs. 

The LHS method also has been implemented in the SAPHIRE code. The online help for this code

states that “...The LHS technique gives its best results if the number of samples is at least twice the

total number of unique (basic) events [in the PRA model].”

As mentioned in the discussion of the Monte Carlo method, the analyst should verify convergence

of the simulation by demonstrating that the result is not sensitive to the choice of seeds and

increases in the number of samples. 

LHS has the same advantages as associated with Monte Carlo methods, as well as the following

additional advantages:

• The accuracy in modeling the risk metric uncertainty is improved with respect to direct

Monte Carlo methods for the same number of samples. In other words, LHS usually can

significantly reduce the time required for an evaluation, while obtaining similar accuracy,

because it requires less samples than Monte Carlo. 

• Use of LHS ensures that the tails of the parameter distributions are included in the

propagation of uncertainty. 

4.2.3 Interpretation of Uncertainty of Risk Metric 

This section discusses the representation and interpretation of the uncertainty measures obtained

with the above two methods. As mentioned above, a Monte Carlo or LHS run is repeated a

predetermined number of times, n. In this way, a sample of n random variables from the distribution

of the risk metric are obtained, that is, X1, X2, ..., Xn. If n is large enough, the sampled values will

represent this distribution well enough to permit inference about the true distribution. 

Mathematical expectation and moments provide characteristics of distributions of random variables.

These ideas can also be used with observations from a random sample from a distribution to

provide estimates of the parameters that characterize that distribution.

A statistic is a function of one or more random variables that does not depend on any unknown

parameters. A function of random variables that can be computed from the collected data sample

is thus a statistic. A function of random variables is also a random variable that has its own

probability distribution and associated characteristics.

The sample mean and sample variance are calculated as follows, where X1, X2, ..., Xn denote a

random sample of size n from the distribution of the risk metric. The sample mean is
4. Parameter Uncertainty

NUREG-1855 4-12 Draft for Comment

Equation 8

and the sample variance is 

Equation 9

Other frequently calculated statistics are the median and the 5th and 95th percentiles. 

One of the common uses of statistics is estimating the unknown parameters of the distribution from

which the sample was generated. For example, the sample mean is used to estimate the mean of

the distribution of the risk metric. Similarly, the sample variance, S2, is used to estimate the

variance of this distribution. 

The two most important characteristics of the distribution of the risk metric are central tendency and

dispersion. The sample mean is a measure of central tendency of this distribution. If this

probability distribution is interpreted as a mass distribution, the mean corresponds to the center of

gravity. 

The sample variance reflects the degree of dispersion or spread of the distribution of the risk metric,

since this variance will be zero if and only if each and every Xi

 has the same value. The more the

random variables (X1, X2, ..., Xn) differ from each other and the sample mean, the larger the

variance will be. 

The 5th and 95th percentiles (0.05 and 0.95 quantiles) also provide an indication of the dispersion

or spread of the distribution of the risk metric. These percentiles mean that there are 0.05 and 0.95

probabilities that the risk metric takes values less than or equal to x0.05 and x0.95, respectively.

Accordingly, these two percentiles may be interpreted as low and high values of the risk metric. 

Usually, a plot of the empirical uncertainty distribution of the risk metric also is obtained. 

4.2.4 State of Knowledge Correlation

In the evaluation of the PRA model to assess a risk metric or an intermediate value, such as the

frequency of an accident sequence, it is important to account for the correlation between the

estimates of the parameters of some of the basic events of the model. Otherwise, the assessment

of the uncertainty of this metric potentially would be underestimated; in one case, ignoring this

correlation also would potentially result in an underestimation of the mean value of this metric. This

potential underestimation is explained below. The correlation occurs because for the same type

of basic events the state of knowledge about their probability is the same. In other words, the

events are not independent but are related to each other.

As mentioned in Chapter 3, many of the acceptance criteria or guidelines used in regulatory

decision making, for example, the acceptance guidelines of RG 1.174, are defined such that the

X X

n

i

i

n

= =

∑

1

S X X

n

i

i

n

2

2

1 1 = −

= − ∑ ( )
4. Parameter Uncertainty

Draft for Comment 4-13 NUREG-1855

appropriate measure for comparison is the mean value of the uncertainty distribution of the

corresponding metric. Since ignoring the SOKC may result in an underestimation of the mean

value of a risk metric, care should be exercised to account for it so that the resulting metric(s) can

be compared in a meaningful way with such criteria or guidelines. 

The SOKC stems from the fact that for nominally identical components in a given NPP, the state

of knowledge about their failure probabilities is the same, as explained below. This correlation was

described by Apostolakis and Kaplan [Apost, 1981a], and this discussion is mainly based on their

paper. 

As described in Section 4.1, an analyst’s state of knowledge about the possible values of a

parameter θ is expressed in terms of a PDF, such as h(θ). For a particular basic event i, its

associated PDF is denoted here by hi

(θi). This distribution represents the probability of a certain

failure mode of a class of nominally identical components (e.g., failure of motor-operated valves

(MOVs) to open). Generic risk studies, like the Reactor Safety Study interpret hi

(θi) as expressing

mainly the plant-to-plant variability of θi

. In other words, at a specific plant, θi has a specific

numerical value, which will be revealed after sufficient statistical evidence has been collected. This

numerical value is, in general, different from one plant to another and hi

(θi) quantifies the analyst’s

state of knowledge about this variability. 

Consistent with this interpretation of the generic distributions, it is natural and convenient to adopt

a model in which all components of the same type, at the same NPP, have the same PDF.

Therefore, in assessing the risk at a specific NPP, it is appropriate to group together the experience

of all the nominally identical components in that NPP. Thus, all the failures of MOVs to open in that

NPP are aggregated into a single number. Likewise, the total number of demands in that NPP are

aggregated. As described in Section 4.1, this evidence (represented by x) can then be used in

Bayes’ Theorem to obtain the posterior distribution, g(θ/x), representing the updated state of

knowledge about a parameter. 

When evaluating the PRA model of an specific NPP, suppose that θ1 and θ2 represent the

parameters of two physically distinct but nominally identical MOVs. Since this discussion assumes

that all such MOVs have the same parameter, it is necessary to set h1 = h2. Furthermore, since the

analyst’s state of knowledge is the same for the two valves, it follows that 

h1(θ1) = h2(θ2) Equation 10

Thus, when the PRA model is evaluated to obtain attributes of a risk metric of a NPP, such as the

variance of the distribution of this metric, h1 and h2 must be regarded as being not only equal

distributions, but they also must be treated as completely dependent distributions. 

The SOKC not only applies to distributions of parameters, such as h(θ), but also to other kinds of

basic events, such as CCFs and human errors. An example of human errors of the same type (i.e.,

that are correlated) is forgetting to return valves to their normal position after testing. 
4. Parameter Uncertainty

6

An MCS is a minimal set of basic events that causes an undesired end state, such as core damage.

NUREG-1855 4-14 Draft for Comment

The mean of a minimal cut set (MCS)6 that contains basic events that are correlated is

underestimated because

E(Xn

) > En(X) Equation 11

where X is a random variable corresponding to a basic event that is correlated with other basic

events in the MCS, E(Xn

) is the expected value of the random variable X elevated to the nth power,

and En

(X) is the nth power of the expected value of X. 

To illustrate this underestimation for n = 2, consider the simple case where two MOVs are in

parallel, represented by variables X1 and X2 that are correlated, and system failure occurs when

both fail to open. The correct equation is

T = X2 Equation 12

where T represents system failure. This equation expresses the fact that the failure probabilities

of the two MOVs are identical (i.e., the distributions of the failure probabilities express the same

state of knowledge). 

If X1 and X2 are incorrectly considered to be independent, the equation used for system failure

would be

T = X1 X2 Equation 13

This equation leads to an underestimation of the mean of T, as can be seen by taking the expected

value in Equations 12 and 13. When using Equation 12,

E(T) = E(X2

) = E2(X) + σ2

X Equation 14

where E2

(X) is the second power of the expected value of X, and σ2

X is the variance of X. 

If Equation 13 is used,

E(T) = E(X1 X2) = E2

(X) Equation 15

It can be seen by comparing Equations 14 and 15 that the mean value of the system failure (i.e.,

the expected value of this failure E(T)) is underestimated when the SOKC is ignored. 

The underestimation of the mean of an MCS that contains basic events that are correlated is

particularly significant when

E(Xn

) >> En(X) Equation 16
4. Parameter Uncertainty

Draft for Comment 4-15 NUREG-1855

This condition occurs when an MCS contains more than two basic events that are correlated or

when the uncertainty (i.e., spread) of the distribution of the correlated basic events in an MCS is

large. 

This example of two MOVs in parallel also can be used to illustrate the potential underestimation

of the uncertainty as expressed by the variance of the distribution of system failure. Considering

that the distributions of the failure probabilities of the two MOVs express the same state of

knowledge, the correct equation is

σ2

T = E(X4

) - E2(X2) Equation 17

where σ2

T is the variance of T. 

If X1 and X2 are incorrectly considered to be independent, the equation used for the variance of the

distribution of system failure would be

σ2

T = E2

(X2) - E4(X) Equation 18

In typical evaluations, Equation 17 yields a greater variance than Equation 18. It is important to

note that the uncertainty of the distribution of system failure potentially will be underestimated even

if the correlated events are not in the same MCS.

Accordingly, failing to take into account the SOKC when evaluating a risk metric or an intermediate

value, such as the frequency of an accident sequence, has the following potential impacts:

• In evaluating the uncertainty of the risk metric, ignoring the SOKC will result in

underestimating the uncertainty. 

• For MCSs containing basic events that are correlated, ignoring the SOKC will underestimate

the mean of each MCS containing basic events that are correlated. This point has

implications in generating the MCSs from the logic model of the PRA and in using the mean

values of the MCSs to estimate the mean of the intermediate results and final risk metrics,

as follows:

– Since the number of MCSs in a PRA model can be extremely large, it is common

to use a truncation value in solving the logic model of the PRA. In this way, only

MCSs above this value are obtained, while the rest are neglected. If the mean value

or other point estimate of an MCS is obtained without accounting for the SOKC, and

this value is used for comparison with the truncation value, some MCSs containing

basic events that are correlated may be incorrectly eliminated.

In addition, after the MCSs have been generated from the logic model of the PRA,

sometimes they are screened to identify some of them for detailed evaluation, thus

eliminating some from further consideration. Consideration of the SOKC can be

important when screening of the MCSs is performed using their means because

some MCSs may be incorrectly eliminated. 
4. Parameter Uncertainty

NUREG-1855 4-16 Draft for Comment

– Since the means of the MCSs that are above the truncation value are combined to

estimate the mean of the risk metric, and several of these MCSs may contain basic

events that are correlated, the potential effect of underestimating the mean of these

MCSs may cause a potential cumulative underestimation of the mean of the risk

metric. 

The combined effect of incorrectly removing some MCSs (which are below the truncation value)

from the quantitative evaluation and underestimating the mean of MCSs (which are above the

truncation value) containing basic events that are correlated potentially will cause a cumulative

underestimation of the mean of the risk metric or other intermediate values. 

One approximate approach proposed in NUREG/CR-4836 [SNL, 1988a] can be used for

addressing the issues raised in the first point. It is advisable that the practicality of this approach

be demonstrated for large scale PRAs.

In summary, failing to take into account the SOKC when evaluating a risk metric or an intermediate

value has the potential impacts of underestimating the mean and the uncertainty of the distribution

of this metric. For the simple example above where two MOVs are in parallel, it is easy to see that

the underestimation of the mean can be significant, especially if the variance of the distribution is

large. 

It is important to determine in which cases such underestimations are significant for a large-scale

PRA. EPRI has prepared some guidance for industry on when the analyst must account for the

SOKC. The U.S. Nuclear Regulatory Commission (NRC) staff position on this guidance is

documented in Appendix A of this report.

4.3 Treatment of Parameter Uncertainties

Guidance is provided in this section for the treatment of parameter uncertainties in comparison of

PRA results with acceptance criteria. After the parameter uncertainties are characterized by the

probability distributions obtained for the parameter(s) of the basic event models, and the

uncertainties are propagated to obtain the uncertainties for the intermediate and final results of the

analysis, these results, along with their uncertainties, are compared to the acceptance criteria.

Since most acceptance criteria currently are explicitly or implicitly couched in terms of the mean

values of the risk metrics, robust mean values are usually the key results from the risk analysis for

most cases. 

Several different approaches to the comparison of PRA results with acceptance guidelines have

been advocated in the past. The comparison of these results with numerical acceptance guidelines

and alternative approaches for making the comparisons are discussed in SECY-97-221,

“Acceptance Guidelines and Consensus Standards for Use in Risk-Informed Regulation” [NRC,

1997a], and summarized in SECY-97-287, “Final Regulatory Guidance on Risk-Informed

Regulation: Policy Issues” [NRC, 1997b]. The comparative approaches are not restricted to

parameter uncertainties but apply to any uncertainties that are characterized in terms of a

probability distribution on the value of a risk metric, where the probability associated with a

particular value represents a measure of the analyst’s degree of belief that the value is a bound on

the true value. In SECY-97-221, three possible approaches were identified for comparison of such

distributions with acceptance guidelines.
4. Parameter Uncertainty

Draft for Comment 4-17 NUREG-1855

The first approach involves the use of the mean values of the metrics (and possibly their

increments) in the comparison with the acceptance guidelines. The use of mean values is

conceptually simple and is consistent with classical decision making. Furthermore, evaluation of

the mean value incorporates consideration of those uncertainties explicitly captured in the model.

However, as pointed out in SECY-97-287, the mean, as with any other single estimate derived from

a distribution, is a summary measure that does not fully take into account the information content

of the probability distribution.

The second approach involves the use of percentile measures. Assuming the uncertainty is

characterized in terms of a probability distribution of the numerical PRA result, then an approach

would be to overlay the distribution on the value associated with the acceptance criteria (which

could be expressed as a single value or, in theory, could also be in terms of a probability

distribution) and determine at what level of confidence the criteria are met. This would require a

policy decision concerning what level of confidence would be acceptable. Historically, it is typical

to see assurance levels of 0.95 as being characteristic of acceptability. 

Such an approach is intuitively appealing, but as pointed out in SECY-97-221, there are several

concerns with this approach. First, the forms of the distributions for characterizing the parameter

uncertainties are arbitrary. In particular, the tails of the distributions are usually not the focus when

deciding on an appropriate distribution; it is the central 90 percent of the distribution that generally

receives attention. Therefore, comparison to high percentile values involving the tails of the

distribution may be overly conservative or may give a false sense of assurance. In addition, with

regard to the acceptable level of confidence, it is not just a question of how much of the distribution

lies above the acceptable value, but also how it is distributed. Finally, since many of the existing

NRC guidelines, such as the Commission’s Safety Goals and their subsidiary objectives, were

meant to be compared with mean values, this option would require a reevaluation of the guidelines

themselves.

A third alternative is one where one criterion is defined for comparison to the mean value of the

distribution characterizing the uncertainty, and a companion criterion is defined for comparison with,

for example, the 95th percentile. This approach treats the acceptance criteria not as a simple

go/no-go limit, but rather as a tolerance band. This approach would require a change in policy to

determine the form of the acceptance criteria and in particular to establish the upper (percentile)

criteria. Again, the mean and especially the higher percentiles are sensitive to fluctuations in the

tails of the distributions. 

As SECY -97-287 points out, from a theoretical standpoint, there is no clear advantage to choosing

any one of these approaches over the others. They are all subject to the criticisms that the

complete distribution is not being fully used, that the form of the distributions for characterizing the

input uncertainties is to some extent arbitrary, and that both the mean and the higher percentiles

are sensitive to changes in the tails of the distributions.

The recommendation in SECY-97-287 is to address parametric uncertainty (and any explicit model

uncertainties) in the assessment using mean values to compare to acceptance criteria. The SECY

further recommends using sensitivity studies to evaluate the impact of using alternate models for

the principal implicit model uncertainties. To address incompleteness, the SECY advocates the use

of quantitative analyses or qualitative analyses, as necessary and as appropriate to the decision

and the acceptance guidelines. The mean value (or other appropriate point estimate if it can be
4. Parameter Uncertainty

NUREG-1855 4-18 Draft for Comment

argued to be close enough to the mean value) is appropriate for comparing with the acceptance

guidelines. This approach has the major advantage that it is consistent with the state of the art,

since current acceptance criteria such as the Commission's Safety Goals and subsidiary objectives

were meant to be compared with mean values. The SECY also points out that for the distributions

generated in typical PRAs, the mean values typically corresponded to the region of the 70th to 80th

percentiles, and coupled with a sensitivity analysis focused on the most important contributors to

uncertainty, can be used for effective decision making.

The form of the acceptance criteria will also play a role in determining the appropriate uncertainty

comparison. For example, the acceptance criteria for PRA results in RG 1.174 require comparison

against the risk metrics of CDF and LERF as well as their increments. In this example, the means

of the risk metrics and the means of their increments need to be established for comparison with

the figure of acceptable values.
Draft for Comment 5-1 NUREG-1855

5. MODEL UNCERTAINTY

The purpose of this chapter is to provide guidance for identifying and characterizing model

uncertainties and related assumptions in a PRA and then assessing what might be their impact on

insights used to support risk-informed applications. In particular, the chapter provides guidance on

the following:

• definition of the concepts of key model uncertainties and related key assumptions

• a process used to identify and characterize key model uncertainties and related

assumptions in PRA

This process comprises four major steps, as illustrated in Figure 5-1.

Figure 5-1 Process to Develop List of Key Sources of

Uncertainty and Related Assumptions

Step 1: Base PRA Sources of Model Uncertainties and Related Assumptions — Both

generic and plant-specific sources of model uncertainty and related assumptions for the

base PRA are identified and characterized. These sources of uncertainty and related

assumptions are those that result from developing the PRA model.

Step 2: Relevant Sources of Model Uncertainties and Related Assumptions — The sources

of model uncertainty and related assumptions that are relevant to the application are

identified by performing a qualitative analysis. This analysis is based on an

understanding of the type of application and the associated acceptance criteria. In

addition, new sources of model uncertainty and related assumptions that may be

introduced by the application are also identified.

Step 3: Potentially Key Sources of Model Uncertainties and Related Assumptions — A

quantitative analysis of the importance of the sources of uncertainty and related

assumptions is performed. This importance analysis identifies those sources of model

uncertainty and related assumptions that can be screened from further consideration.

Review Base 

PRA

Perform 

Qualitative 

Analyses

Perform 

Realistic 

Sensitivity 

Analyses

Perform 

Quantitative 

Screening 

Analyses

Step 1 Step 2 Step 3 Step 4

List of Key Sources of Uncertainties and Related Assumptions

Relevant 

Sources

Potentially 

Key Sources

All 

Sources

Identify in Context of Application

Review Base 

PRA

Perform 

Qualitative 

Analyses

Perform 

Realistic 

Sensitivity 

Analyses

Perform 

Quantitative 

Screening 

Analyses

Step 1 Step 2 Step 3 Step 4

List of Key Sources of Uncertainties and Related Assumptions

Relevant 

Sources

Potentially 

Key Sources

All 

Sources

Identify in Context of Application
5. Model Uncertainty

7

At the present time, the definitions in the ASME standard for these terms are under ballot for

approval; if approved, the above definitions will be consistent with the standard.

NUREG-1855 5-2 Draft for Comment

Step 4: Key Sources of Model Uncertainties and Related Assumptions — A sensitivity

analysis is performed on the retained sources of model uncertainty and related

assumptions to identify any realistic alternative modeling hypotheses. These

hypotheses are used to identify which of the retained sources of model uncertainty and

related assumptions are key with regard to the application.

5.1 Definitions

In order to provide guidance on how to address the impact of model uncertainty and related

assumptions, it is necessary to understand what is meant by these terms. The following definitions,

which are consistent with RG 1.200 [NRC, 2007a] and the American Society of Mechanical

Engineers (ASME) PRA standard [ASME, 2005a]7

, are provided below in Table 5-1.

Table 5-1 Definitions For Model Uncertainty and

Related Assumption

A source of model uncertainty is one that is related to an issue in which there

is no consensus approach or model and where the choice of approach or model

is known to have an effect on the PRA model (e.g., introduction of a new basic

event, changes to basic event probabilities, change in success criterion,

introduction of a new initiating event). 

A source of model uncertainty is labeled key when it could impact the PRA results

that are being used in a decision and, consequently, may influence the decision

being made. Therefore, a key source of model uncertainty is identified in the

context of an application. This impact would need to be significant enough that

it changes the degree to which the risk acceptance criteria are met and, therefore,

could potentially influence the decision. For example, for an application for a

licensing basis change using the acceptance criteria in RG 1.174 [NRC, 2002a],

a source of model uncertainty or related assumption could be considered “key”

if it results in uncertainty regarding whether the result lies in Region II or Region

I, or if it results in uncertainty regarding whether the result becomes close to the

region boundary or not.
5. Model Uncertainty

Table 5-1 Definitions For Model Uncertainty and

Related Assumption

Draft for Comment 5-3 NUREG-1855

An assumption is a decision or judgment that is made in the development of the

PRA model. An assumption is either related to a source of model uncertainty or

is related to scope or level of detail.

An assumption related to a model uncertainty is made with the knowledge that

a different reasonable alternative assumption exists. A reasonable alternative

assumption is one that has broad acceptance within the technical community

and for which the technical basis for consideration is at least as sound as that of

the assumption being made.

An assumption related to scope or level of detail is one that is made for

modeling convenience.

An assumption is labeled key when it may influence (i.e., have the potential to

change) the decision being made. Therefore, a key assumption is identified in

the context of an application.

5.2 Base PRA Sources of Model Uncertainties and Related Assumptions

(Step 1)

The goal of this step is to identify and characterize those sources of uncertainty and related

assumptions in the base PRA model. Identifying and characterizing a source of uncertainty and

related assumptions means determining how they have been addressed in developing the base

PRA model. Once sources of uncertainty and related assumptions have been identified, they may

be evaluated against qualitative screening criteria to identify those issues that can be eliminated

from further consideration as sources of uncertainty based on the existence of some accepted

model approach, such as a consensus model for a particular issue. 

The process used to identify and characterize the sources of model uncertainty and related

assumptions is illustrated below in Figure 5-2.
5. Model Uncertainty

NUREG-1855 5-4 Draft for Comment

Figure 5-2 Process to Identify and Characterize the Sources

of Model Uncertainty and Related Assumptions

This process, discussed below, involves the following:

• Identification of sources of model uncertainty and related assumptions

• Characterization of sources of model uncertainty and related assumptions

• Qualitative screening of model uncertainty and related assumptions

5.2.1 Identification

The sources of model uncertainty and related assumptions that are a result of developing the PRA

model are identified. This identification is performed by using the structure of the PRA, and more

explicitly by examining each step of the construction of the PRA to identify if it involved a model

uncertainty or related assumption as defined previously in Section 5.1. This process provides a

systematic method of identifying the sources of model uncertainty and related assumptions. This

process basically involves the following:

• Using the ASME PRA standard as the PRA structure, for each high-level requirement

(HLR), review each supporting requirement to determine if a model uncertainty or related

assumption is involved. This identification is required in the ASME PRA standard. For each

HLR, the standard requires that the sources of model uncertainty are documented.

• For completeness, a literature search review may be performed to identify potential sources

of model uncertainty or related assumptions not identified as a result of the above step.

EPRI Report

Guideline for the Treatment of 

Uncertainty in Risk-Informed Applications

List and Characterization of Model 

Uncertainties and Related

Identify Sources of Model 

Uncertainty and Related 

Assumptions

• Identify sources against 

PRA structure

• Perform literature search

Qualitative Screening of Model 

Uncertainty and Related 

Assumptions

• Consensus model

• Bounding, conservative 

model

Characterize Sources of Model 

Uncertainty and Related 

Assumptions

• Identify part of PRA model 

affected

• Identify model or 

assumptions selected

• Identify impact on the PRA 

model

EPRI Report

Guideline for the Treatment of 

Uncertainty in Risk-Informed Applications

List and Characterization of Model 

Uncertainties and Related

Identify Sources of Model 

Uncertainty and Related 

Assumptions

• Identify sources against 

PRA structure

• Perform literature search

Qualitative Screening of Model 

Uncertainty and Related 

Assumptions

• Consensus model

• Bounding, conservative 

model

Characterize Sources of Model 

Uncertainty and Related 

Assumptions

• Identify part of PRA model 

affected

• Identify model or 

assumptions selected

• Identify impact on the PRA 

model
5. Model Uncertainty

8

At this time, the staff and EPRI, under a Memorandum of Understanding, are working together to ensure

that the two efforts are complementary.

Draft for Comment 5-5 NUREG-1855

For this process, EPRI 1009652, “‘Guideline for the Treatment of Uncertainty in Risk-Informed

Applications’ Technical Basis Document” [EPRI, 2004a], may provide an example of an acceptable

approach as discussed in Appendix A. Furthermore, EPRI 1009652 provides a generic list of

sources of model uncertainty and related assumptions as a result of implementing the process.

This list can serve as a starting point to identify the set of plant-specific sources of model

uncertainty and related assumptions.8

5.2.2 Characterization

The identified sources of model uncertainty and related assumptions are characterized. This

characterization identifies the impact of the sources on the PRA model (e.g., introduction of a new

basic event, changes to basic event probabilities, change in success criteria, introduction of a new

initiating event). Characterization of the source of model uncertainty or a related assumption is a

three step (and can be iterative), which involves identification of the following:

• part of the PRA model affected

• model used or assumption made

• impact on the PRA model

Part of the PRA Model Affected

The part of the model affected by the source of model uncertainty or a related assumption is

identified. This identification is needed because not every application involves every aspect of the

PRA model. Therefore, as discussed further in Step 2, if the application deals with an aspect of the

PRA model not affected by the source of model uncertainty, then the source is not relevant to the

application.

There are two ways in which the PRA model can be affected:

(1) at the basic event level

(2) in the logic structure

There are three types of basic events found in PRAs. These events represent the following:

(1) occurrence of initiating events

(2) states of unavailability or failure of SSCs

(3) human failures that contribute to the failure of the systems designed to protect against the

undesirable consequences should an initiating event occur

Sources of model uncertainty and related assumptions may be identified with the definitions and

the methods of evaluation of each of these types of basic events. 
5. Model Uncertainty

NUREG-1855 5-6 Draft for Comment

There can be uncertainty associated with whether it is necessary to include specific initiating

events, whether it is necessary to include specific failure modes of SSCs, how to address the

consequences of SSC failure (e.g., digital instrumentation and control systems), the assessment

of success criteria, the need for support systems for specific missions of frontline systems, the

modeling of failures of complex pieces of equipment (e.g., RCP seals), and other issues. As is

discussed below, the way in which these uncertainties are addressed can have an impact on

accident sequence definitions or system fault tree structures and thus affect the logic model

structure.

In addition, it needs to be noted that a source of model uncertainty or a related assumption could

have multiple impacts on the PRA model. That is, it could impact both a basic event (or several

basic events) and the logic structure. Furthermore, the impact on the logic structure could also

occur in either a single place or in multiple places.

Model Used or Assumption Made

At this point in the process, the model or related assumption selected for each source is identified.

Different models or different assumptions are available to address the identified source of

uncertainty. As examples, different models have been proposed for modeling RCP seal failures

on loss of cooling, different thermohydraulic computer codes have been used to derive success

criteria, different human reliability analysis (HRA) models are used for deriving HEPs, and different

assumptions can be made with regard to equipment performance under adverse conditions. For

all these examples, the analyst’s choice of model or assumption will affect the PRA model. This

effect may introduce, for example, new initiating events or accident sequences, or it could affect

the computational results. Some examples are given below. Each model that is available to the

analyst as a choice for any particular uncertainty issue or related assumption will uniquely impact

the PRA model.

Impact on the PRA Model

The PRA model is constructed based on the particular models used or assumptions made to

address the sources of uncertainty. Therefore, in characterizing the impact of the source of model

uncertainty or a related assumption, it is necessary to identify how it is reflected in either the logic

structure or the basic events of the PRA model. The existence of model uncertainty results in

uncertainty about the predictions of the PRA with respect to alternate models and assumptions, as

shown in the following examples:

• An alternate computational model may produce different initiating event frequencies, SSC

failure probabilities, or unavailabilities.

• An alternate HRA model may produce different HEPs, or introduce new human failure

events.

• An alternate assumption regarding phenomenological effects on SSC performance can

impact the credit taken for SSCs for some accident sequences.

• An alternate success criterion may lead to a redefinition of a system fault tree logic.
5. Model Uncertainty

9

At this time, the staff and EPRI, under a Memorandum of Understanding, are working together to ensure

that the two efforts are complementary.

10Note that this definition is somewhat different from that in EPRI 1009652, “Guideline for the Treatment of

Uncertainty in Risk-Informed Applications: Technical Basis Document,” issued December 2004, although the intent is

similar.

Draft for Comment 5-7 NUREG-1855

• An alternate screening criteria may lead to adding or deleting initiating events, accident

sequences, or basic events.

• An alternate assumption that changes the credited frontline or support systems included in

the model may change the incremental significance of sequences.

• An alternate seal LOCA model can result in a different event tree structure, different LOCA

sizes, and different probabilities. 

For this process, EPRI 1009652 may provide an example of an acceptable approach as discussed

in Appendix A. Furthermore, EPRI 1009652 provides a generic list of sources of model uncertainty

and related assumptions as a result of implementing the process. This list can serve as a starting

point to identify the set of plant-specific sources of model uncertainty and related assumptions.9

5.2.3 Qualitative Screening

Once sources of model uncertainties and related assumptions have been identified and

characterized, they may be qualitatively screened to identify those that warrant and do not warrant

further consideration as potential key sources of uncertainty and related assumptions. That is,

there may be sources or uncertainty or related assumptions that, for qualitative reasons, do not

need to be considered in the decision making process.

Qualitative criteria are developed that identify those sources of model uncertainty and related

assumptions that do not need to be pursued. The following are two examples of qualitative

screening criteria:

(1) use of consensus models

(2) use of qualitative bounding assessments

Adopting a consensus model will remove an issue as a source of model uncertainty to be

addressed in the decision making. With the use of a consensus model, an alternative hypothesis

need no longer be explored. The definition of a consensus model (as defined in Section 3.3.3) is

as follows:

Consensus model: in the most general sense, as a model that has a publicly

available published basis and has been peer reviewed and widely adopted by an

appropriate stakeholder group. In addition, widely accepted PRA practices may be

regarded as consensus models. Examples of the latter include the use of the

constant probability of failure on demand model for standby components and the

Poisson model for initiating events.10 For risk-informed regulatory decisions, the
5. Model Uncertainty

NUREG-1855 5-8 Draft for Comment

consensus model approach is one that the NRC has utilized or accepted for the

specific risk-informed application for which it is proposed.

Note that the definition given here ties the consensus model definition to a specific application. This

is because models have limitations that may be acceptable for some uses and not for others. 

Examples of consensus models include the following:

• Poisson model for initiating events

• Bayesian analysis

• PWR RCP seal LOCA model for Westinghouse plants

There may be a de facto consensus of acceptance when certain conservative NRC licensing criteria

are used as the basis to model certain issues. An example of such a conservative criterion is the

2–4-hour coping time for battery depletion during a loss of alternating current (ac) power event,

since it is expected that station batteries would be available for several more hours if loads were

to be successfully shed. A model reflective of the licensee’s licensing basis is generally perceived

as incorporating the conservative attributes of the deterministic licensing criteria. However, such

de facto models must be assessed for potential impact based on the application at hand. For some

applications, the use of conservative criteria in one area can mask the significance of another part

of the risk picture, which might be the part that is needed for an application.

5.3 Relevant Sources of Model Uncertainties and Related Assumptions (Step 2)

The goal of this step is to identify those sources of model uncertainty and related assumptions that

are relevant to an application. To establish which of the sources of uncertainty identified in Step 1

are relevant, it is essential to assess them in the context of the application for which the PRA results

are to be used. That is, if the source of model uncertainty or a related assumption does not impact

the part of the PRA model relevant to the application, then it is also not relevant to the application.

Therefore, it is necessary to identify the purpose of the application and the acceptance criteria

against which the results of the risk-informed analysis will be evaluated. This process identifies

which parts of the base PRA are relevant to the application, and therefore which of the sources of

uncertainty identified in Step 1 need to be addressed. 

For some applications, the base PRA may be modified to perform an assessment of the change

in risk given a plant change. For these applications, the modifications themselves may introduce

new sources of model uncertainty or related assumptions. These “new” sources of model

uncertainty and related assumptions need to be identified.

The result of Step 2 is an understanding of the type of application and its associated acceptance

criteria, and the finalized list of characterized sources of model uncertainty, including both those

identified from the base PRA in Step 1 and those sources of model uncertainty and related

assumptions that have been introduced as a consequence of modifications to or expansions of the

PRA model to account for the application. All of these are inputs to the process for identifying those

sources of model uncertainty and related assumptions that are potentially key in terms of the

application (described in Section 5.4). The process used to identify those sources of uncertainty

that are relevant to the application is illustrated below in Figure 5-3.
5. Model Uncertainty

Draft for Comment 5-9 NUREG-1855

Figure 5-3 Process to Identify Relevant Sources of Model

Uncertainty and Related Assumptions

This process, discussed below, involves the following:

• understanding the application

• performing qualitative analyses

5.3.1 Understanding the Application

To identify those sources of model uncertainty and related assumptions that are relevant to an

application, it is necessary to identify the application and its associated acceptance criteria. Each

application has specific criteria or guidelines against which to assess PRA results to demonstrate

that the risk impact of the application is acceptable. A source of model uncertainty or a related

assumption is relevant to the application if the acceptance criteria are challenged, and therefore

the decision may be influenced. A source of model uncertainty or a related assumption that does

not challenge the acceptance criteria would then be considered not relevant and would not have

an influence on the decision under consideration by the application.

For example, for an application for a licensing basis change using the acceptance criteria in

RG 1.174, a source of model uncertainty or a related assumption would be considered relevant if

it results in uncertainty regarding whether the result lies in region II or region I, or if it results in

uncertainty regarding whether the result becomes close to the region boundary or not. It is the

numerical acceptance criteria that provide the basis for identifying which sources of model

Base PRA 

Applications

Identify Acceptance Criteria 

Associated with Application

Modification to the 

PRA Applications

Identify Parts of PRA 

Needed for 

Acceptance Criteria

Identify and Characterize 

Sources of Uncertainty 

Associated with Changes to 

PRA Model

Sources of Model 

Uncertainty or Related 

Assumptions Involves 

Acceptance Criteria?

Sources of Model 

Uncertainty and 

Related Assumptions

Sources of Model 

Uncertainty and Related 

Assumptions NOT

Relevant to Application

Sources of Model Uncertainty and Related 

Assumptions Relevant to Application

No

Yes

Base PRA 

Applications

Identify Acceptance Criteria 

Associated with Application

Modification to the 

PRA Applications

Identify Parts of PRA 

Needed for 

Acceptance Criteria

Identify and Characterize 

Sources of Uncertainty 

Associated with Changes to 

PRA Model

Sources of Model 

Uncertainty or Related 

Assumptions Involves 

Acceptance Criteria?

Sources of Model 

Uncertainty and 

Related Assumptions

Sources of Model 

Uncertainty and Related 

Assumptions NOT

Relevant to Application

Sources of Model Uncertainty and Related 

Assumptions Relevant to Application

No

Yes
5. Model Uncertainty

NUREG-1855 5-10 Draft for Comment

uncertainty and related assumptions can potentially challenge the decision and, therefore, which

ones would then be identified as relevant.

The acceptance criteria for the application will include cumulative objectives, incremental

objectives, or both. For example, an cumulative objective may be a maximum acceptable CDF, and

an incremental objective may be a maximum change in CDF. The specific objectives that are used

for the acceptance criteria for the application under consideration are identified.

Examples of application and acceptance criteria are listed in Table 5-2

Table 5-2 Examples of Regulatory Applications and Risk-Informed

Acceptance Criteria

Application

Category

Application [NRC, 2007d] Acceptance Criteria

Operational Uses Maintenance rule (10 CFR 50.65) TO BE IDENTIFIED (see NOTE)

Oversight

(industrywide)

Accident Sequence Precursor program 

Events assessment

Generic issue resolution

Oversight

(licensee)

Significance Determination Process (Phase 2)

Significance Determination Process (Phase 3)

Event assessment (Management Directive 8.3)

Notice of Enforcement Discretion

Mitigating System Performance Index

Licensing Changes to the licensing basis (RG 1.174) • CDF and LERF

• ΔCDF and ΔLERF

Risk-informed inservice testing (RG 1.175) TO BE IDENTIFIED (see Note)

Risk-informed technical specifications

(RG 1.177)/Exigent technical specifications

Risk-informed inservice inspection (RG 1.178)

10 CFR 50.69 SSC categorization

Technical specification initiatives (especially 4b)

10 CFR 50.46 (future)

Guidance on response to emergent conditions

License renewal/severe accident mitigation

alternatives

Advanced reactor design (10 CFR Part 52)
5. Model Uncertainty

Table 5-2 Examples of Regulatory Applications and Risk-Informed

Acceptance Criteria

Application

Category

Application [NRC, 2007d] Acceptance Criteria

Draft for Comment 5-11 NUREG-1855

10 CFR 50.36(c)(2)(ii)(D) criterion 4 (SSCs found by

a PRA to be significant to public health and safety) 

Standard Review Plan Section 18.0/NUREG-1764,

regarding the review of changes to human actions

Rulemaking 10 CFR 50.69 TO BE IDENTIFIED (see Note)

10 CFR 50.44

10 CFR 50.46

Manual actions (Appendix R related)

NOTE:

This information to be completed in conjunction with staff position on EPRI document on Treatment of

Uncertainty.

5.3.2 Performing Qualitative Analyses

In addition to the above, only those sources of model uncertainty or related assumptions that impact

the part of the PRA model associated with the application are relevant. That is, if the application

is only concerned with, for example, the LOCA accident sequences, then those sources of model

uncertainty and related assumptions impacting the other sequences would not be considered

relevant to the application.

To perform the identification, it is necessary to identify what results from the PRA are necessary

to support the application. Therefore, the basis or process used for determining which part(s) of

the PRA model is impacted needs to be documented. Furthermore, it is essential that the process

examine and take into consideration the dependencies of the PRA.

Having understood what PRA results are necessary to support the application, it is straightforward

to identify those parts of the base PRA needed to assess the risk implications. For some

applications (e.g., a simple allowed outage time (AOT) extension), the complete PRA model may

not need to be exercised. Only those sources of uncertainty that affect the parts of the base PRA

needed to support the application need to be retained for further evaluation.

For example, when the application addresses an AOT extension for a diesel generator, only those

parts of the PRA that involve the diesel generator need be exercised, namely those sequences that

involve a loss of offsite power (LOOP). On the other hand, an application such as the

implementation of Title 10, Section 50.69, “Risk-Informed Categorization and Treatment of

Structures, Systems, and Components for Nuclear Power Reactors,” of the Code of Federal

Regulations (10 CFR 50.69) requires the categorization of SSCs into low and high safety

significance, and since these are relative measures, such an assessment would involve the

complete PRA.
5. Model Uncertainty

NUREG-1855 5-12 Draft for Comment

The application either involves cumulative or incremental acceptance criteria as described above.

If the application involves cumulative criteria, then the base PRA is being used without making any

changes to the PRA. However, if the application involves incremental criteria, then the PRA model

is being revised or modified. This modification may introduce additional sources of model

uncertainty or related assumptions. For example, the effect of changing certain plant conditions

or operating practices on SSC availability or reliability may need to be modeled. An example is the

modeling of the impact of reduced special treatment of SSC reliability. This can introduce additional

uncertainty.

For those newly identified sources of model uncertainty and related assumptions associated with

the modification of the base PRA, a qualitative screening, as discussed in Step 1, should be

performed.

5.4 Potentially Key Sources of Model Uncertainty and Related Assumptions

(Step 3)

The goal of this step it to identify sources of model uncertainty and related assumptions potentially

key to the application. The input to this step is a list of sources of model uncertainties that are

relevant to the application (i.e., can impact the PRA model either in toto or in parts). These sources

of model uncertainties or related assumptions may now be quantitatively screened to identify those

that have the potential to impact the results of the PRA sufficient to impact a decision regarding the

application. This determination is made by evaluating the importance of the source of model

uncertainty or a related assumption to the acceptance criteria. Those determined to be important

are potentially key sources of model uncertainty or related assumptions. 

As discussed in Step 2, the application involves either (1) cumulative maximum acceptance criteria

or (2) maximum incremental acceptance criteria—that is, the application involves the entire base

PRA or only portions of the base PRA model without necessitating a change to the model, or the

application involves making changes to the base PRA model. 

The process to do this is illustrated in Figure 5.4. The details for performing the analysis are

different depending on whether the acceptance criteria are in terms of cumulative or incremental

measures of risk, as discussed in Sections 5.4.1 and 5.4.2, respectively.
5. Model Uncertainty

Draft for Comment 5-13 NUREG-1855

Figure 5-4 Process to Identify Potentially Key Sources of

Model Uncertainty and Related Assumptions

In the following, the approach to screening is discussed with respect to these two types of

acceptance criteria for each of the following classes:

a. sources of model uncertainty or related assumptions linked to a single basic event

b. sources of model uncertainty or related assumptions linked to multiple basic events

c. sources of model uncertainty or related assumptions linked to the logic structure of PRA

models

d. sources of model uncertainty or related assumptions linked to logical combinations

In light of the associated acceptance criteria, or each type of application and for each class, the

sources of model uncertainty or related assumptions are examined to perform the initial screening.

This screening process is discussed below.

Identify Acceptance Criteria 

Associated with Application

Identify Application 

Case

Sources of Model Uncertainty 

and Related Assumptions 

Relevant to Application

Sources of Model 

Uncertainty and Related 

Assumptions NOT Key to 

Application

Sources of Model Uncertainty and Related 

Assumptions Potentially Key to Application

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Logical Combinations of Sources of Model 

Uncertainty that Can Affect the Application

Screen

Screen

Sources of Model 

Uncertainty and Related 

Assumptions Challenge 

Acceptance Criteria?

No

Yes

Perform Quantitative Screening

Identify Acceptance Criteria 

Associated with Application

Identify Application 

Case

Sources of Model Uncertainty 

and Related Assumptions 

Relevant to Application

Sources of Model 

Uncertainty and Related 

Assumptions NOT Key to 

Application

Sources of Model Uncertainty and Related 

Assumptions Potentially Key to Application

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Logical Combinations of Sources of Model 

Uncertainty that Can Affect the Application

Screen

Screen

Sources of Model 

Uncertainty and Related 

Assumptions Challenge 

Acceptance Criteria?

No

Yes

Perform Quantitative Screening
5. Model Uncertainty

NUREG-1855 5-14 Draft for Comment

5.4.1 Applications Involving Cumulative Acceptance Criteria

Guidance is provided for performing screening for each of the above four cases for applications

involving a cumulative maximum acceptance criterion.

Case 1a Sources of Model Uncertainty or Related Assumptions Linked to a Single

Basic Event

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant only to a single basic event. For each identified source of

uncertainty, an importance analysis can be used for screening.

An approach to determining the importance of the source of model uncertainty or related

assumptions is to calculate a maximum acceptable risk achievement worth (RAW) (denoted as

RAWmax) associated with the metric of interest, such as maximum allowable CDF or LERF. The

RAW for each relevant basic event can be compared to the RAWmax associated with the maximum

acceptable CDF. For all basic events with a RAW less than RAWmax, the associated model

uncertainty or related assumption issue cannot be key since it is not mathematically possible for

the impact of that issue to cause the result of the PRA model to be greater than the maximum

acceptable metric. “Acceptable” must be defined within the context of the application for which the

licensee intends to use the PRA, which is the purpose of Step 2, but it would most likely be defined

in terms of a maximum acceptable risk metric, such as CDF, incremental CDF deficit, or

incremental core damage probability. 

For the j

th basic event, the definition of RAW is

RAW Equation 1

CDF

CDF j base

j base

base

,

,

=

+

Where:

RAWj is the value of RAW for basic event j as calculated in the base PRA

CDFbase is the value of the CDF mean estimate in the base PRA

CDF+

j,base is the base PRA CDF mean estimate with the basic event j set to logical

TRUE.

Thus, given that the acceptance criterion defines the maximum acceptable CDF(denoted as CDF+

),

Equation 1 can be solved directly for the “maximum acceptable” RAW, RAWmax, that any basic

event can have without constituting a potential key uncertainty. CDF+

, the metric of interest in this

case, is substituted into Equation 1 in place of CDF+

j,base and the equation is solved for RAWmax.

To illustrate the concept of a maximum acceptable RAW, suppose that for a particular base PRA

the CDF is 3.0x10-5/year (yr). Suppose further that the maximum acceptable CDF for a particular

application of the base PRA is 5.0x10-5/yr. Hence, from Equation 1
5. Model Uncertainty

Draft for Comment 5-15 NUREG-1855

RAW , therefore, Equation 2

CDF

CDFbase

max =

+

RAWmax

.

. = ×

×

−

−

50 10

30 10

5

5

RAWmax = 17.

For this example, any source of model uncertainty or a related assumption linked to a basic event

from the base PRA that has a RAW greater than 1.7 would have the mathematical potential to be

a key source of model uncertainty or a related assumption. All such sources of model uncertainty

or related assumptions must be assessed with a realistic sensitivity analysis to determine whether

or not they constitute a key source of model uncertainty or a related assumption. Expressed more

generally, if

RAW RAW Equation 3 j base , max ≤

is a true expression for the j

th basic event, then the source of model uncertainty or a related

assumption linked to the j

th basic event is screened from further consideration, otherwise it is a

potential key uncertainty and must be assessed with a realistic sensitivity analysis (see

Section 5.5). It has to be remembered that the criteria or guidelines are typically not “hard” criteria,

and there may be some flexibility in their application. However, given that the RAW measure is an

extremely sensitive measure in that it takes the failure probability to one, this is an acceptable

screening approach and may indeed be somewhat conservative for many cases.

It must be emphasized that the result in Equation 3 is relevant only for those sources of model

uncertainty and related assumptions identified in Step 1 that are linked to a particular basic event.

Case 1b Sources of Model Uncertainty or Related Assumptions Linked to Multiple

Basic Events

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant only to multiple basic events. For each identified source of

uncertainty, an importance analysis is performed.

The RAW importance measures for several basic events cannot be used collectively to assess the

combined impact of the sources of model uncertainties or related assumptions associated with the

group of basic events. However, the concept of setting all relevant basic events to logical TRUE

simultaneously and then reevaluating the PRA model yields the same perspective for a group of

basic events as does the RAW importance measure analysis for an individual basic event. Hence,

by setting all basic events relevant to a source of uncertainty to logical TRUE to calculate a CDF+

j

where j represents the set of basic events relevant to the j

th source of model uncertainty or related

assumption, the following equation can be evaluated:

CDF CDF j base , Equation 4 + + ≤
5. Model Uncertainty

NUREG-1855 5-16 Draft for Comment

If the relationship in Equation 4 is true, then the source of uncertainty is not a key uncertainty as

there is no mathematical possibility that any quantification of the values of the basic events linked

to that source of uncertainty could achieve an unacceptably high CDF. Otherwise, the source of

uncertainty is a potential key uncertainty and must be evaluated with a realistic sensitivity analysis

(see Section 5.6).

Case 1c Sources of Model Uncertainty or Related Assumptions Linked to the Logic

Structure of the PRA Model

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant only to the logic structure of the PRA model.

In general, there is no straightforward method to quantitatively screen sources of model uncertainty

or related assumptions that impact the logic structure of the PRA model. Alternative methods or

assumptions that could possibly introduce new cut sets in existing sequences by changing fault tree

models, new sequences by changing the structure of event trees, or even entirely new classes of

accident sequences by introducing new initiating events must be assessed by manipulating or

altering the PRA model to reflect these alternatives. A new CDF can be compared to the maximum

acceptable CDF in a manner similar to that shown in Equation 4. If the new CDF is greater than

CDF+

, then the issue is potentially key and must be evaluated with a realistic sensitivity analysis

(see Section 5.6).

The above approach can involve significant resources. However, in some cases, it may be possible

to perform an approximate bounding evaluation (see Chapter 6) that can demonstrate that the

potential impact of the alternate assumption or model will not produce a result that challenges the

decision criteria. As an example, this demonstration can be achieved if the effect of the model

uncertainty or related assumption is limited to the later branches of the lower frequency accident

sequences, and the frequency of the portion of the sequences up until the branch points is low

enough.

Case 1d Sources of Model Uncertainty or Related Assumptions Linked to Logical

Combinations

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant to combinations of basic events and logic structure. For these

cases, the combination may impose a synergistic impact upon the uncertainty of the PRA results.

The resulting uncertainty from their total impact may be greater than the sum of their individual

impacts. For example, several issues could relate to the same dominant cut sets, or certain

sequences, or a particular event and the success criteria for systems on that event tree, or to the

same plant damage state. In other words, such issues overlap each other by impacting jointly on

the same parts of the risk profile modeled in the PRA. Thus, to accurately assess the full potential

for the impact of uncertainty, such issues also must be grouped together. 

A simple example can be found in the relationship of two models — recovery of offsite power and

recovery of failed diesel generators — to the overall uncertainty of the model. Both models

represent the failure to restore ac power to critical plant systems through different but redundant

power sources. Hence, the potential total impact of uncertainty associated with the function of

supplying ac power to emergency electrical buses would involve a joint assessment of the
5. Model Uncertainty

Draft for Comment 5-17 NUREG-1855

uncertainty associated with both models. Another example is with regard to the interaction between

uncertainties associated with the direct current (dc) battery depletion model and those associated

with the operator actions to restore power; specifically the interrelationship between operator

performance and the performance of key electrical equipment under harsh conditions (e.g., smoke,

loss of room cooling). How long dc batteries can remain sufficiently charged and successfully

deliver dc power to critical components relies on the shedding of nonessential electrical loads,

which is achieved through the actions of reactor operators and equipment operators who operate

eletromechanical equipment such as motor control centers and circuit breakers, as well as through

procedures and the availability of required tools (e.g., lighting, procedures, communication devices).

Uncertainty associated with these operator actions and the potential harsh environmental impacts

on both operators and equipment should be jointly assessed for a perspective on the potential total

impact of uncertainty upon the dc battery depletion model.

Furthermore, the choice of HRA method can impact the uncertainty of PRA results in several areas.

An interface exists between the human actions necessary to restore diesel generator operation

after either failing to start or run and the time to dc battery depletion. Many diesel generators

depend on dc power for field flashing in order for successful startup. If equipment operators fail to

successfully restore diesel generator operation before the dc batteries become depleted, then the

diesel generators cannot be restored to operation. Hence, the potential impact of uncertainties

associated with the HEPs in the model to recover failed diesel generators and uncertainties

associated with the dc battery depletion model should be assessed together. 

In the above examples, the uncertainty issues were linked by their relationship to a given function,

namely establishing power. However, uncertainties related to different issues can also have a

synergistic effect. As an example, consider the case of an uncertainty associated with the modeling

of high-pressure coolant injection (HPCI) in a PRA for a boiling-water reactor. In core damage

sequences of transient event trees, failure of the HPCI is either coupled with failure of other highpressure injection systems (reactor core isolation cooling (RCIC), recovery of feedwater, control

rod drive (CRD)) and failure of depressurization, or failure of other high-pressure injection systems

and failure of low-pressure injection. The importance of HPCI is affected by the credit taken for

additional injection systems (over and above RCIC). For example, taking credit for fire water (as

an additional low-pressure system), CRD, or recovery of feedwater (as a high-pressure system) can

lessen the importance of HPCI. 

In the LOOP/station blackout (SBO) tree, a significant function of HPCI is to provide a delay to give

time to recover the offsite power. Therefore, the modeling of recovery of offsite power in the short

term (given that HPCI has failed), the frequency of LOOP, and the CCF probability of the diesels

and the station batteries all have an impact on the importance of HPCI.

 

HPCI importance is therefore affected by the following:

• transient frequencies

• HEP for depressurization

• credit for motor-driven feedwater pumps

• credit for alternate injection systems (e.g., fire water, service water cross-tie, CRD)
5. Model Uncertainty

11The discussion that follows is in terms of CDF but can also be applied to LERF.

NUREG-1855 5-18 Draft for Comment

• LOOP frequency, CCF of diesels and batteries, and the factors associated with the shortterm recovery of ac power given a LOOP

In general, uncertainties associated with any of these issues could interact synergistically to impact

the overall model uncertainty associated with the modeling of the HPCI.

Detailed guidance on grouping issues into logical groupings can be found in EPRI 1009652. The

analyst’s judgement and insight regarding the PRA should yield numerous logical groupings specific

to the PRA in question. Certain issues may readily fall into more than one logical grouping

depending on the nature of the other issues.

Once the various combinations have been identified, the importance analyses can be performed.

The situation wherein a logical group of sources of uncertainty is relevant to and impacts a single

basic event is similar to Case 1a. This case lends itself to an analysis in which a maximum

mathematically possible CDF is estimated using the RAW of the relevant basic event. The method

for Case 1a can be employed directly.

The situation wherein a logical group of sources of uncertainty is relevant to and impacts more than

one basic event is similar to Case 1b with regard to quantitative screening. The concept of setting

all relevant basic events to logical TRUE simultaneously and then reevaluating the PRA model

yields the same perspective for a logical grouping of sources of uncertainties as for a single source

of uncertainty that impacts several basic events. Hence, by setting all basic events relevant to a

logical grouping of sources of uncertainty to logical TRUE to calculate a CDF+

j

 where j represents

the set of basic events relevant to the j

th logical grouping of sources of uncertainty, Equation 4 can

be evaluated to determine if the logical grouping is potentially a key uncertainty. If Equation 4 is

true, then the logical grouping of sources of uncertainly does not present a potential key uncertainty

as there is no mathematical possibility that the values of the basic events linked to the particular

logical grouping could achieve an unacceptably high CDF. Otherwise, the logical grouping of

sources of uncertainty is a potential key uncertainty and must be evaluated with a realistic

sensitivity analysis (see Section 5.5).

5.4.2 Applications Involving Incremental Acceptance Criteria

In general, these types of applications (i.e., ones that involve a change to the PRA and acceptance

criteria involving maximum incremental metrics) are license amendment applications. For example,

quantitative assessment of the risk impact in terms of changes to the CDF (or LERF)11 metric (i.e.,

ΔCDF) is used in comparison against the RG 1.174 acceptance guidelines (Figure 5-5) or

guidelines derived from those of RG 1.174.
5. Model Uncertainty

Draft for Comment 5-19 NUREG-1855

Figure 5-5 NRC RG 1.174 Acceptance Guidelines for CDF

and LERF

Because the criteria involve two metrics—an unacceptable range for CDF horizontally in the figures,

and an unacceptable range for ΔCDF vertically—it is necessary to assess the potential impact of

a model uncertainty issue not only with respect to CDF but to ΔCDF as well, since an acceptable

result along one axis could be offset by an unacceptable result along the other. Hence, just as for

applications involving only the base PRA (see Section 5.5.1), one is interested in assessing the

potential impact of model uncertainty upon CDFbase, but it is also an issue that must be assessed

for its impact upon ΔCDF. Therefore, the following metrics are of interest for applications involving

a change to the licensing basis:

CDFbase the value of the CDF mean estimate in the base PRA 

CDFafter the value of the CDF mean estimate in the modified base PRA to account

for changes proposed to the licensing basis 

CDF+

j,base the CDF mean estimate in the base PRA with the basic event j set to logical

TRUE

CDF+

j,after the CDF mean estimate in the modified PRA with the basic event j set to

logical TRUE

Using these four quantities, the terms ΔCDF and ΔCDF+

j

 are defined as follows:

ΔCDF CDF CDF Equation 5 after base = −

ΔCDF CDF CDF Equation 6 j j after j base

++ + = − , ,

Equations 5 and 6 allow for the assessment of the potential vertical movement into unacceptable

regions of RG 1.174 acceptance criteria.
5. Model Uncertainty

NUREG-1855 5-20 Draft for Comment

Each of the following four cases, repeated from the beginning of Section 5.4, is discussed

separately below:

a. sources of model uncertainty or related assumptions linked to a single basic event

b. sources of model uncertainty or related assumptions linked to multiple basic events

c. sources of model uncertainty or related assumptions that impact the logic structure of PRA

models

d. sources of model uncertainty or related assumptions linked to logical combinations

Case 2a Sources of Model Uncertainty or Related Assumptions Linked to a Single

Basic Event

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine that which are relevant only to a single basic event. For each identified source of

uncertainty, an importance analysis is performed.

The terms on the right-hand side of Equations 5 and 6 are readily calculable—in Equation 5 by

exercising the base PRA and the modified PRA, and in Equation 6 by recalculating the base and

modified PRAs with the value of the relevant basic event (or the j

th basic event) set to logical TRUE

in both the base and modified PRAs. Exercising the base and modified PRAs to calculate the righthand terms of Equations 5 and 6 and then solving Equations 5 and 6 for ΔCDF and ΔCDF+

j

,

respectively, allows for the plotting of the four ordered pairs, shown in Table 5-3 against the

acceptance criteria from RG 1.174 in Figure 5-5.

Table 5-3 Ordered Pairs of CDF and ΔCDF and Comparison against

Acceptance Criteria

Ordered Pair Perspective of Comparison

(CDFbase, ΔCDF) Comparison of the mean CDF and mean ΔCDF against the acceptance

criteria. Provides the analyst’s best judgement of the impact of the change in

risk.

(CDFbase, ΔCDF+

j

) Comparison of the mean CDF and the largest shift possible in ΔCDF, as

defined with the jth basic event quantified as logical TRUE, against the

acceptance criteria. Provides a perspective on the potential shift in the ΔCDF

value resulting from an alternate model or assumption.

(CDF+

j,base, ΔCDF) Comparison of the greatest possible shift in the base CDF, as defined with the

j

jth basic event quantified as logical TRUE, and the mean ΔCDF against the

acceptance criteria. Provides a perspective on the potential shift in the CDF

value resulting from an alternate model or assumption.
5. Model Uncertainty

Table 5-3 Ordered Pairs of CDF and ΔCDF and Comparison against

Acceptance Criteria

Draft for Comment 5-21 NUREG-1855

(CDF+

j,base, ΔCDF+j

) Comparison of the greatest possible shift in the base CDF and the greatest

possible shift in the ΔCDF, as defined with the jth basic event quantified as

logical TRUE, against the acceptance criteria. Provides a perspective on the

potential shift in both the ΔCDF and CDF value resulting from an alternate

model or assumption.

A source of uncertainty can influence a decision by moving the ordered pair into a different region

of Figure 5-5. For those sources of uncertainty associated with the modeling of the impact of the

change associated with the application, this can only occur by changing ΔCDF. However, sources

of uncertainty associated with the base PRA can impact both the CDF and the ΔCDF. 

If any of the four ordered pairs were to fall within (or very close to) region I, the maximum possible

impact of the model uncertainty associated with the j

th basic event would be to either render the

application unacceptable or (if near the acceptance criteria) impose the need for greater regulatory

vigilance. Similarly, if the impact of the alternate model or assumption is to move any of the

ordered pairs into region II from region III, this could also affect the decision, by, for example,

adopting a different performance monitoring strategy or adopting specific compensatory measures.

Hence, in either case, the issue is potentially key and must be assessed with a realistic uncertainty

analysis, discussed in Section 5.5. 

The significance of the ordered pairs (CDF+

j,base, ΔCDF) and (CDF+j,base, ΔCDF+j

) is that they give

the decision maker a perspective of the combined impact of an issue on both the base PRA and

the modified PRA. An issue that might be perceived as not influencing the decision on the basis

of where the ordered pairs (CDFbase, ΔCDF) and (CDFbase, ΔCDF+

j

) lie on Figure 5-5 might be

perceived otherwise if the combined impact of the issue is factored into the decision.

Another method to assess the potential impact of a model uncertainty relevant to a single basic

event is the method of Reinert and Apostolakis [Reinert, 2006]. Reinert and Apostolakis employ

a method wherein they define the concept of a “threshold RAW” value (analogous to the use of

RAWmax in Case 1a) for basic events with regard to both CDF and ΔCDF. Their definition of RAW

with regard to CDF is directly from standard PRA practice:

RAW and Equation 7

CDF

CDF j CDF base

j base

base

,

,

−

+

= RAW

CDF

CDF j CDF after

j after

j after

,

,

,

−

+

=

RAW with regard to ΔCDF is defined as

RAW Equation 8

CDF

CDF j CDF

j

,Δ

Δ

Δ =

+

where ΔCDF and ΔCDF+

j

 are defined as in Equations 5 and 6. Substituting Equations 5 and 6 into

Equation 8 yields:
5. Model Uncertainty

NUREG-1855 5-22 Draft for Comment

RAW Equation 9

CDF CDF

CDF CDF j CDF

j after j base

after base

,

, ,

Δ = −

−

+ +

Solving the relationships in Equation 7 for the respective ΔCDF+

j

 and inserting the results into

Equation 9 yields

Equation 10 ( ) ( ) ( ) ( ) RAWj CDF

RAWj CDF after CDFafter RAWj CDF base CDFbase

CDFafter CDFbase ,

, ,

Δ = − × − − ×

−

All of the right-hand terms of Equation 10 are readily calculable, which allows the analyst to

calculate a RAW with regard to ΔCDF. Reinert and Apotolakis use the relationships in Equations

8 and 10 to calculate threshold RAWs with regard to both CDF and ΔCDF by selecting maximum

acceptable values for CDF and ΔCDF (which they refer to as CDFthreshold and ΔCDFthreshold,

respectively) and then substitute these threshold values for CDF+

j,base and ΔCDF+j

 in Equations 7

and 8 to yield

RAW Equation 11

CDF

CDF CDF threshold

threshold

base

, = Δ

RAW Equation 12

CDF

CDF CDF threshold

threshold

base

, = Δ

Equations 11 and 12 yield threshold values for the RAW with regard to the base PRA CDF defined

in Equation 7 and the RAW with regard to ΔCDF defined in Equation 10. The base PRA model and

the modified PRA model are exercised, which yields RAWj,CDF-base values for all basic events in the

base PRA and which allows for the solving of Equation 10 to calculate RAWj,Δ CDF values. The

resulting values for RAWj,cdf-base and RAWj,Δ CDF are compared to the threshold values calculated by

Equations 11 and 12 to determine if any model uncertainty associated with a single basic event

poses a potential key model uncertainty.

In employing the method of Reinart and Apostolakis, care should be given with regard to assessing

the potential combined impact of a model uncertainty on both CDF and ΔCDF. This method does

not automatically investigate the potential that an acceptable impact upon CDF and an acceptable

impact upon ΔCDF could nonetheless result in an overall unacceptable result, which is the function

of the order pairs (CDF+

j,base, ΔCDF) and (CDF+j,base, ΔCDF+j

) in the ordered pair approach discussed

above. Reinart and Apostolakis do address this issue by selecting more than one threshold value

for CDF and ΔCDF based on the horizontal and vertical transitions between regions I and II and

between regions I and III. 

Reinert and Apostolakis provide a case study to illustrate this method. 
5. Model Uncertainty

Draft for Comment 5-23 NUREG-1855

Case 2b Sources of Model Uncertainty or Related Assumptions Linked to Multiple

Basic Events

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant to multiple basic events. For each identified, an importance

analysis is performed. An example would be the choice of model to quantify human errors and

recovery actions, or an assumption that affects the quantification of a particular failure mode of

several redundant components (e.g., stuck-open safety relief valve). 

The RAW importance measures for several basic events cannot be used collectively to assess the

combined impact of the uncertainties associated with the group of basic events. However, the

concept of setting all relevant basic events to logical TRUE simultaneously and then reevaluating

the PRA model yields the same perspective for a group of basic events as does the RAW

importance measure for an individual basic event. Hence, all basic events relevant to a particular

source of uncertainty are set to logical TRUE to calculate the ordered pairs in Table 5-2, where j

would connote the set of basic events relevant to the j

th source of uncertainty. If any of the four

ordered pairs were to fall within or close to region I, the maximum possible impact of the source of

uncertainty would be to render the application unacceptable or require the implementation of

increased regulatory vigilance. Similarly, if the impact of the alternate model or assumption is to

move any of the ordered pairs into region II from region III, this could also affect the decision, by,

for example, adopting a different performance monitoring strategy or adopting specific

compensatory measures. Hence, in either case, the source of uncertainty must be assessed with

a realistic uncertainty analysis. 

Case 2c Sources of Model Uncertainty or Related Assumptions Linked to the Logic

Structure of the PRA Model

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant to the logic structure of the PRA model. For each identified, an

importance analysis is performed.

In general, there is no straightforward method to quantitatively screen sources of model uncertainty

or related assumptions that impact the logic structure of the PRA model. Alternative methods or

assumptions that could possibly introduce new cut sets in existing sequences by changing fault tree

models, new sequences by changing the structure of event trees, or even entirely new classes of

accident sequences by introducing new initiating events must be assessed by manipulating or

altering the PRA model to reflect these alternatives. New estimates for CDF+

j,base and CDF+j,after can

be developed, where these terms are now defined as follows:

CDF+

j,base The base PRA CDF mean estimate where the base PRA has been modified to

address the j

th source of model uncertainty or related assumption that is linked to the

logic structure of the PRA model.

CDF+

j,after The base PRA CDF mean estimate where the PRA, as modified for the application,

has been further modified to address the j

th source of model uncertainty or related

assumption that is linked to the logic structure of the PRA model.
5. Model Uncertainty

NUREG-1855 5-24 Draft for Comment

Using Equations 5 and 6, the analyst can calculate values for the terms of the ordered pairs in

Table 5-2 and compare the plots of those ordered pairs to the acceptance criteria shown in

Figure 5.5. If any of the four ordered pairs were to fall within or close to region I, the maximum

possible impact of the source of uncertainty would be to render the application unacceptable or

require the implementation of increased regulatory vigilance. Similarly, if the impact of the alternate

model or assumption is to move any of the ordered pairs into region II from region III, this could also

affect the decision, by, for example, adopting a different performance monitoring strategy or

adopting specific compensatory measures. Hence, in either case, the source of uncertainty must

be assessed with a realistic uncertainty analysis. 

Case 2d Sources of Model Uncertainty or Related Assumptions Linked to Logical

Combinations

The sources of model uncertainty and related assumptions identified in Step 2 are reviewed to

determine those that are relevant to combinations of basic events and logic structure. One should

not, however, restrict oneself to a short list of generic logical groupings. The analyst’s judgement

and insight regarding the PRA should yield numerous logical groupings specific to the PRA in

question. Certain issues may readily fall into more than one logical grouping depending on the

nature of the other issues. For these cases, the combination may impose a synergistic impact upon

the uncertainty of the PRA results. Detailed guidance on grouping issues into logical groupings can

be found in EPRI 1009652. (Refer to the discussion for Case 1d in Section 5.4.1.)

The situation wherein a logical group of sources of uncertainty is relevant to and impacts more than

one basic event is similar to Case 2b with regard to quantitative screening. The concept of setting

all relevant basic events to logical TRUE simultaneously and then reevaluating the PRA model

yields the same perspective for a logical grouping of sources of uncertainties as for a single source

of uncertainty that impacts several basic events. Hence, all basic events relevant to a particular

model issue or to a logical group of issues are set to logical TRUE to calculate the ordered pairs

in Table 5-2, where j represents the set of basic events relevant to the j

th logical group of model

issues. If any of the four ordered pairs were to fall within region I, the maximum possible impact

of the uncertainty would be to render the application unacceptable. Similarly, if the impact of the

alternate model or assumption is to move any of the ordered pairs into region II from region III, this

could also affect the decision, by, for example, adopting a different performance monitoring strategy

or adopting specific compensatory measures. Hence, in either case, the source of model

uncertainty or a related assumption must be assessed with a realistic uncertainty analysis (see

Section 5.5). 

5.5 Key Sources of Model Uncertainty and Related Assumptions (Step 4)

The goal of this step it to identify sources of model uncertainty and related assumptions that are

key to the application. The input to this step is a list of sources of model uncertainties that are

potentially key to the application. These sources of model uncertainties or related assumptions

may now be further screened by performing realistic sensitivity analyses. Those determined to

impact the results of the PRA sufficient to actually influence a decision regarding the application

are identified as key sources of model uncertainty or related assumptions. The process to do this

screening is illustrated in Figure 5-6.
5. Model Uncertainty

Draft for Comment 5-25 NUREG-1855

Figure 5-6 Process to Identify Key Sources of Model

Uncertainty and Related Assumptions

This process, discussed below, involves the following:

• defining acceptable sensitivity analyses to perform realistic screening

• performing realistic screening to identify the key sources of model uncertainty and related

assumptions

5.5.1 Define and Justify Sensitivity Analysis

Defining acceptable sensitivity analyses involves developing reasonable alternatives or hypotheses

to those adopted for the base model. However, the set of sensitivity analyses that are needed to

obtain a realistic understanding of the impact of the source of model uncertainty or related

assumption is dependent on the particular source of model uncertainty or related assumption. To

develop the alternatives, an indepth understanding of the issues associated with the source of

model uncertainty or related assumption is needed. What is known about the issue itself will

Define and Justify Sensitivity 

Analyses

Sources of Model Uncertainty 

and Related Assumptions 

Potentially Key to Application

Sources of Model 

Uncertainty and Related 

Assumptions NOT Key to 

Application

Sources of Model Uncertainty and Related 

Assumptions Key to Application

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Logical Combinations of Sources of Model 

Uncertainty that Can Affect the Application

Screen

Screen

Sources of Model 

Uncertainty and Related 

Assumptions Challenge 

Acceptance Criteria?

No

Yes

Perform Realistic Screening

Define and Justify Sensitivity 

Analyses

Sources of Model Uncertainty 

and Related Assumptions 

Potentially Key to Application

Sources of Model 

Uncertainty and Related 

Assumptions NOT Key to 

Application

Sources of Model Uncertainty and Related 

Assumptions Key to Application

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Sources of 

Model Uncertainty that 

Impact Basic Events

Identify Sources of Model 

Uncertainty that Impact 

Multiple Basic Events

Identify Sources of 

Model Uncertainty that 

Impact Logic Model

Identify Logical Combinations of Sources of Model 

Uncertainty that Can Affect the Application

Screen

Screen

Sources of Model 

Uncertainty and Related 

Assumptions Challenge 

Acceptance Criteria?

No

Yes

Perform Realistic Screening
5. Model Uncertainty

12Examples relevant to the following text are discussed in the EPRI “Guidelines for the Treatment

of Uncertainty in Risk-Informed Regulatory Applications,” Draft Report, [EPRI, 2005a]

NUREG-1855 5-26 Draft for Comment

essentially dictate possible alternatives to be explored. However, previous experience from the

PRA community suggest several ideas for developing reasonable alternatives.12

One example of previous experience that would provide reasonable alternatives would be to identify

variations in the way a particular source of model uncertainty or related assumption has been

addressed in other PRA analyses, both for base PRA evaluations and for related sensitivity

analyses, and that has generally been accepted as reasonable in the literature. One example of

a common issue that generally been accepted as reasonable in the literature is the reliability of

equipment operating after loss of room cooling for which no specific calculations exist. An accepted

conservative model assumption is to assign a probability of failure of 1.0. On the other hand, it may

be worthwhile to explore whether the issue of room cooling is relevant to the decision by performing

a sensitivity analysis under the assumption that room cooling is not needed. 

Another example is the use of alternate models regarding the validity of varying a parameter value

to address uncertainty that can be exercised to derive a range of reasonable values. For example,

consider the issue of dc battery life. If a conservative licencing-basis model has been used,

consider increasing battery life by 50 percent to represent the potential to extent battery life through

operator actions. If a nonconservative life-extension model that credits load shedding has been

used, decrease battery life by 50 percent to reflect the possibility that equipment operators fail to

successfully perform all tasks under stressful conditions. 

It is common to use factors of, for example, 2, 5, or 10 on specific parameter values or groups of

parameter values as sensitivity analyses. For these sensitivity analyses to be meaningful, it is

necessary to have a justification for the factors based on an understanding of the issue that results

in the uncertainty. An alternative approach to justifying the factor is to implement a monitoring

program that would verify that the assumption of the factor is not invalidated. 

For this step of the process, EPRI 1009652 may provide examples of an acceptable approach, as

discussed in Appendix A.

5.5.2 Perform Quantitative Realistic Screening

As discussed in Step 2, the application involves either cumulative maximum acceptance criteria or

maximum incremental acceptance criteria; that is, the application involves the entire or parts of the

base PRA model without changing the model, or the application involves changes to the base PRA

model. These acceptance criteria are applied to one of the following, repeated from the beginning

of Section 5.4:

a. sources of model uncertainty or related assumptions linked to a single basic event

b. sources of model uncertainty or related assumptions linked to multiple basic events

c. sources of model uncertainty or related assumptions that impact the logic structure of PRA

models
5. Model Uncertainty

Draft for Comment 5-27 NUREG-1855

d. sources of model uncertainty or related assumptions linked to logical combinations

The process by which sources of model uncertainty or related assumptions for each of the above

are quantitatively screened is discussed below.

5.5.2.1 Applications Involving Cumulative Acceptance Criteria

Guidance is provided for performing an importance analysis for each of the above four cases for

applications involving an cumulative maximum acceptance criterion.

Case 1a Sources of Model Uncertainty or Related Assumptions Linked to a Single

Basic Event

Any basic event associated with a source of uncertainty identified in Step 3, for which the

relationship in Equation 3 (Case 1a in Step 3) is false, is a potential key uncertainty. The issue

must be evaluated with a realistic sensitivity analysis on the basic event value. The basic event

value that would achieve a false relationship in Equation 3 must be estimated. If the value of the

basic event that results in a maximum acceptable CDF (CDF+

) is realistic or reasonable, then the

issue is a key uncertainty because there exists at least one reasonable hypothesis regarding the

source of model uncertainty or a related assumption that could yield an unacceptably high CDF.

If the basic event value necessary to achieve a false relationship in Equation 3 is unrealistic, then

it is not reasonable to expect that the source of model uncertainty or a related assumption could

result in an undesirable risk profile, and hence it is not a key uncertainty.

Case 1b Sources of Model Uncertainty or Related Assumptions Linked to Multiple

Basic Events

If the expression in Equation 4 (Case 1b in Step 3) is false, then the potential exists that the source

of uncertainty is a key uncertainty. The analyst must select reasonable options for alternate models

for the particular issue. Then for each reasonable alternate model, the base PRA must be

requantified and the relationship between the maximum acceptable CDF (CDF+

) and the new CDF

estimate as expressed in Equation 4 (Case 1b in Step 3) must be reevaluated. The multiple results

create a range of potential base PRA results that must be compared to the acceptance criteria. If

any of the results are unacceptable, then the issue must be considered as a key uncertainty

because at least one reasonable hypothesis exists that would result in an unacceptable result.

Case 1c Sources of Model Uncertainty or Related Assumptions Linked to the Logic

Structure of the PRA Model

Some issues, depending on their nature as to how they impact the PRA model, may require indepth manipulation of the PRA models, such as restructuring of fault trees or event trees, or the

definition of new failure modes and events. Once again, realistic uncertainty analyses for issues

should ultimately be conducted for each logic grouping in order to evaluate whether or not the

issues are key sources of uncertainty. 

The analyst must select reasonable options for alternate models for the particular issue. Then for

each reasonable alternate model, the base PRA must be requantified and the relationship between

the maximum acceptable CDF (CDF+

) and the new CDF estimate as expressed in Equation 4
5. Model Uncertainty

NUREG-1855 5-28 Draft for Comment

(Case 1b in Step 3) must be reevaluated. The multiple results create a range of potential base PRA

results. If any of the results are unacceptable, then the issue is a key uncertainty because at least

one reasonable model results in an unacceptable result.

Case 1d Sources of Model Uncertainty or Related Assumptions Linked to Logical

Groupings

If the expression in Equation 4 (Case 1b of Step 3) is false, then the potential exists that the logical

grouping of sources of uncertainty constitutes a key uncertainty. The analyst must select

reasonable options for alternate models for the particular issues in the logical grouping. Then for

each reasonable alternate model, the PRA must be requantified. The multiple results create a

range of potential base PRA results. If any of the results are unacceptable, then the logical

grouping must be considered as a key uncertainty because at least one reasonable model results

in an unacceptable result.

Recall the example logical grouping of issues relevant to SBO sequences (Section 5.2.1, Case 2d)

— recovery of LOOP, dc battery depletion time, HRA relevant to recovery of diesel generators, and

others. These multiple issues impact multiple basic events. This represents a multivariable

problem, considerably more complex than the case where a single issue impacts a single basic

event or group of basic events.

5.5.2.2 Applications Involving Incremental Acceptance Criteria

Guidance is provided for performing a sensitivity analysis for each of the four cases given in Section

5.5.2.1 for applications involving incremental maximum acceptance criteria.

Case 2a Sources of Model Uncertainty or Related Assumptions Linked to a Single

Basic Event

Sources of uncertainty that could be related directly to a specific basic event were conservatively

evaluated in both the base PRA and the modified PRA in Case 2a of Step 3 by setting the value

of the relevant basic event to logical TRUE. The issue must be evaluated with a realistic sensitivity

analysis on the basic event value. The terms for the ordered pairs in Table 5-2 must be

reevaluated for any reasonable hypothesis developed for any source of model uncertainty or related

assumption linked to the j

th basic event. For any such reasonable hypothesis, if any of the ordered

pairs in Table 5-2 yields a result in or close to region I, then the source of model uncertainty or

related assumption is a key uncertainty. 

As discussed in Case 2a of Step 3, Reinert and Apostolakis provide an alternate method to test for

whether or not a source of model uncertainty or a related assumption linked to a single basic event

constitutes a potential key uncertainty. The application of that method is continued here. In order

to perform a realistic uncertainty analysis on model issues that can be related to specific basic

events, Reinert and Apostolakis continue to employ the concept of a threshold RAW. The term

“threshold” RAW importance measure coined by Reinert and Apostolakis can also be thought of

as a “maximum acceptable” RAW importance measure in that it represents the largest possible

value for the RAW importance measure for which it would be mathematically impossible for the
5. Model Uncertainty

13These are Equations 12 and 13 in [Reinart, 2006a].

Draft for Comment 5-29 NUREG-1855

uncertainty associated with a particular basic event to cause an unacceptable PRA result.

Threshold values for the RAWcdf and the RAWacdf are calculated as13

RAW Equation 13

CDF

CDF CDF threshold

threshold

base

, =

RAW Equation 14

CDF

CDF CDF threshold

threshold

Δ

Δ

Δ , =

where CDFthreshold is the value of CDF that corresponds to the vertical line between the applicable

regions in Figure 5-5, and ΔCDF threshold is the value of ΔCDF that corresponds to the horizontal line

between the applicable regions in Figure 5-5. 

Once the RAWcdf, threshold and the RAWacdf, threshold have been calculated, each model issue can be

evaluated by investigating the model for the basic events relevant to each issue. For any particular

issue, the value for a particular relevant basic event j is adjusted and both the base PRA model and

the modified PRA are reevaluated until a result for the base PRA is found that yields one of the

following:

RAW RAW j CDF CDF threshold , , ≈ Equation 15

RAW RAW j CDF CDF trheshold , , Δ Δ ≈ Equation 16

If the value of the j

th basic event that corresponds to the approximation in either Equation 15 or 16

is based on a reasonable hypothesis for the basic event’s probability, then the source of uncertainty

linked to the jth basic event is a key uncertainty. If this is the case, the sensitivity analysis should

continue so that a “reasonable” maximum value for the jth basic event and its corresponding “high”

estimates for CDFj

 and ΔCDF j are calculated. These values for CDFj and ΔCDF j will most likely

be less than the values of CDF+

j, base and ΔCDF+j

, respectively, that were calculated by setting the

value of the j

th basic event to logical TRUE. These “reasonable” high values for CDF j

 and ΔCDFj

will be necessary for the comparison of the risk-informed application to the RG 1.174 acceptance

criteria. If the value of the jth basic event that yields the approximation in either Equation 15 or 16

is based on an unreasonable hypothesis, then the issue is not a key uncertainty since the

uncertainty associated with the issue could not reasonably cause a result in the PRA models that

would be within an unacceptable region in Figure 5-5.
5. Model Uncertainty

NUREG-1855 5-30 Draft for Comment

Case 2b Sources of Model Uncertainty or Related Assumptions Linked to Multiple

Basic Events

If, in Case 2b of Step 3, the plot of any of the ordered pairs in Table 5-2 is not acceptable, then the

potential exists that the issue is a key uncertainty. The analyst must select reasonable options for

alternate models or assumptions for the particular issue. Then, for each reasonable alternate

model, the base and modified PRAs must be requantified. The multiple results create a range of

potential base PRA and modified PRA results. If any of the results yield values for the terms in the

ordered pairs of Table 5-2 such that the plotting of any of the ordered pairs against the acceptance

criteria of Figure 5-5 falls in or close to region I, then the issue is a key uncertainty because at lest

one reasonable model results in an unacceptable result.

Case 2c Sources of Model Uncertainty or Related Assumptions Linked to the Logic

Structure of the PRA Model

Some issues, depending on their nature as to how they impact the PRA model, may require indepth manipulation of the PRA models, such as restructuring of fault trees or event trees, or the

definition of new failure modes and events. Once again, realistic uncertainty analyses for issues

should ultimately be conducted for each logic grouping in order to evaluate whether or not the

issues are key sources of uncertainty. 

The analyst must select reasonable options for alternate models for the particular issue. Then, for

each reasonable alternate model, the base and modified PRAs must be requantified. The multiple

results create a range of potential base PRA and modified PRA results. If any of the results yield

values for the terms in the ordered pairs of Table 5-2 such that the plotting of any of the ordered

pairs against the acceptance criteria in Figure 5-5 falls in or close to region I, then the issue is a key

uncertainty because at least one reasonable model results in an unacceptable result.

Case 2d Sources of Model Uncertainty or Related Assumptions Linked to Logical

Groupings

If, in Case 2d of Step 3, the plot of any of the ordered pairs in Table 5-2 is not acceptable, then the

potential exists that the issue is a key uncertainty. The analyst must select reasonable options for

alternate models for the particular issue. Then, for each reasonable alternate model, the base and

modified PRAs must be requantified. The multiple results create a range of potential base PRA and

modified PRA results. If any of the results yield values for the terms in the ordered pairs of Table

5-2 such that the plotting of any of the ordered pairs against the acceptance criteria of Figure 5-5

falls in or close to region I, then the issue is a key uncertainty because at least one reasonable

model results in an unacceptable result.
Draft for Comment 6-1 NUREG-1855

6. COMPLETENESS UNCERTAINTY

The purpose of this chapter is to provide guidance on addressing one aspect of completeness

uncertainty — incomplete PRA scope or incomplete PRA level of detail. This chapter provides

guidance on the use of conservative or bounding analyses to address missing items. The types

of bounding approaches that can be utilized in these efforts are identified and guidance is provided

on their use.

As indicated in Chapter 2, a risk-informed decision making process integrates insights from

deterministic and risk analyses along with considerations of defense in depth and safety margins.

Generally, what is used as the risk input to decision making are the quantitative risk results from

PRAs (e.g., CDF). Depending on the decision, the needed PRA scope and level of detail can vary.

Furthermore, an existing PRA may not always match the scope or contain the level of detail

required for a specific risk-informed decision. For example, the PRA may not include analysis of

accidents during low-power and shutdown (LPSD) modes of operation, specific initiating events

such as seismic events, specific accident sequences such as those resulting from an anticipated

transient without scram, or some component failure modes (e.g., spurious component operation or

pipe ruptures). For such a situation, a licensee will have the following four options:

(1) upgrade the PRA to address the required scope or level of detail

(2) use bounding analyses to address the scope and level of detail not included in the PRA

model

(3) use a bounding analysis to demonstrate that the missing scope items are not significant to

the decision 

(4) modify the application such that the missing scope or level of detail is not affecting the

decision

The approach in the second and third options, namely the use of bounding analyses, is the subject

of this chapter. The guidance for use of bounding analyses, as described below, includes the

following:

• Determining the Required Scope of an Application — This section discusses the process

for assessing the application in terms of identifying the PRA scope and level of detail

required to support the risk-informed application.

• Defining the Types of Bounding Analyses — This section discusses the types of bounding

analyses that can be used to supplement the PRA when it does not contain the required

scope and level of detail needed to support the risk-informed application.

• Selecting and Using Bounding Approaches — This section provides guidance with

examples on how to use different types of bounding analyses in an application to address

missing PRA scope items.
6. Completeness Uncertainty

14POSs are used to subdivide the plant operating cycle into unique states, such that the plant

response can be assumed to be the same within the given POS for a given initiating event. Operational

characteristics (such as reactor power level; in-vessel temperature, pressure, and coolant level; equipment

operability; and changes in decay heat load or plant conditions that allow new success criteria) are

examined to identify those relevant to defining POSs. These characteristics are used to define the states,

and the fraction of time spent in each state is estimated using plant-specific information.

NUREG-1855 6-2 Draft for Comment

6.1 Determining the Required Scope of an Application

Guidance is provided in this section for assessing the application in terms of identifying the PRA

scope and level of detail required to support the risk-informed application. As discussed in

Chapter 3, the scope of the PRA is defined in terms of the following:

• the metrics used to evaluate risk

• the POSs for which the risk is to be evaluated

• the types of initiating events that can potentially challenge and disrupt the normal operation

of the plant and, if not prevented or mitigated, would eventually result in core damage, a

release, and/or health effects

The level of detail of a PRA is defined in terms of the degree to which the logic models are

discretized and plant representation is modeled.

Sections 3.2.3 and 3.2.4 provide detailed discussions on PRA scope and level of detail,

respectively. The required PRA scope and required PRA level of detail is determined by the

application. The application is reviewed to determine the needed scope and level of detail. Once

this determination is established, the PRA model is then reviewed to determine if it has the needed

scope and level of detailed defined by the application.

PRA Scope

Three decisions are made by the analyst — what risk characterization, what POS, and what

initiating events need to be addressed by the PRA.

The risk metrics relevant to the application are defined by the acceptance criteria. For example,

for a licensing-basis change, RG 1.174 [NRC, 2002a] defines the risk metrics as CDF, LERF,

ΔCDF, and ΔLERF. Therefore, if the acceptance criteria use CDF, LERF, ΔCDF, and ΔLERF, then

the PRA scope must also address these.

The POSs to be considered are determined by the impact of the application. The various operating

states include full power, low power, and shutdown.14 Not every application will necessarily impact

every operating state. In deciding this aspect of the required scope, the SSCs affected by the

application are identified. This step includes a determination of the cause-and-effect relationships

between the proposed plant application and its impact on the SSCs. It is then determined if the

affected SSCs are required to prevent or mitigate accidents in the different POSs and if the impact
6. Completeness Uncertainty

15Initiating events are the events that have the ability to challenge the condition of the plant. 

These events include failure of equipment from either internal plant causes (such as hardware faults,

operator actions, floods, or fires) or external plant causes (such as earthquakes or high winds).

Draft for Comment 6-3 NUREG-1855

of the proposed plant change would impact the prevention and mitigation capability of the SSC in

those POSs. A plant change could affect the potential for an accident initiator in one or more POSs

or reduce the reliability of a component or system that is unique to a POS or required in multiple

POSs. Once the cause-and effect-relationship on POSs is identified, the PRA model is reviewed

to determine if it has the scope needed to reflect the effect of the application on plant risk.

The types of initiating events to be considered are also determined by the impact of the application.

The types of initiating events include internal events, internal fire, and external events.15 In deciding

this aspect of the required scope, the process is similar to that described above for POSs. The

SSCs affected by the application are determined, and the resulting cause-and-effect relationships

are identified. It is then determined if the affected SSCs are required to prevent or mitigate both

internal and external events and if the proposed plant change would affect that capability. The

impact of the proposed plant change on the SSCs could include the introduction of new accident

initiating events, affect the frequency of initiators, or affect the reliability of mitigating systems

required to respond to multiple initiators. Once the cause-and-effect relationship on the accidentinitiating events is identified, the PRA model is reviewed to determine if it has the scope needed

to reflect the effect of the application on plant risk.

For example, consider an application that involves a licensing-basis change where a seismically

qualified component is being replaced with a nonqualified component. If the new component’s

reliability for responding to nonseismic events is not changed, the nonseismic part of the PRA is

not impacted and only the seismic risk need be considered; that is, the other contributors to the risk

(e.g., fire) are not needed for this application. If, on the other hand, the reliability of the new

component is different for responding to nonseismic events, the nonseismic part of the PRA may

be impacted and, therefore, must be included in the scope.

PRA Level of Detail

The minimum level of detail for a base PRA, independent of an application, is determined by the

requirements of the ASME standard [ASME, 2005a]. However, it is recognized that the detail

specified by the technical requirements may not be needed for a given application, while, in other

cases, the minimum level of detail may not be sufficient. The level of detail needed is that detail

required to capture the effect of the application. That is, the PRA model needs to be of sufficient

detail so that the impact of the application can be assessed.

This determination is generally accomplished by considering the cause-and-effect relationship

between the application and its impact on the plant risk. A proposed application can impact one

or more PRA technical elements. Examples of potential impacts include the following:

• introduces a new initiating event or requires modification of an initiating event group

• changes a system success criterion
6. Completeness Uncertainty

NUREG-1855 6-4 Draft for Comment

• requires the addition of new accident sequences

• requires additional failure modes of SSCs

• alters system reliability or changes system dependencies

• requires modification of parameter probabilities 

• introduces a new CCF mechanism

• eliminates, adds, or modifies a human action

• changes important results used in other applications, such as importance measures

• changes the potential for containment bypass or failure modes leading to a large early

release

• changes the SSCs required to mitigate external hazards such as seismic events

• changes the reliability of systems used during LPSD modes of operation

Once the potential impacts are identified, the PRA model is reviewed to determine if its level of

detail is sufficient to assess the impact.

Insufficient Scope or Level of Detail

The Commission has directed that, if a PRA standard for a specific scope item is available and has

been endorsed by the staff, then the PRA used to support a risk-informed application should be

performed against that standard. [NRC, 2003e] That is, if the missing scope or missing level of

detail is addressed by a standard, the PRA should be upgraded according to the standard.

However, for a given application, if it can be demonstrated that the missing scope or missing of

level of detail does not impact the risk, the PRA does not need to be upgraded. In this situation,

it is acceptable to use a bounding analysis to demonstrate that the missing scope or the missing

level of detail is not a contributor to risk. If a standard does not exist to address the missing scope

or the standard does not address the missing level of detail, then a bounding analysis can be used

to bound the risk contribution instead of upgrading the PRA.

6.2 Defining the Types of Bounding Analyses

Guidance is provided in this section for defining the characteristics of the different types of bounding

analyses used for the missing scope and level of detail in the PRA model. There are different

circumstances when use of a bounding analysis is acceptable:

• to screen the risk contribution from further consideration in the risk estimate

• to bound the risk contribution for consideration in the risk estimate
6. Completeness Uncertainty

16“Standard for Probabilistic Risk Assessment for Nuclear Power Plant Applications,” ASME/ANS

RA-S-2007, anticipated to be published in January 2008.

Draft for Comment 6-5 NUREG-1855

In the first situation, a bounding analysis is used to demonstrate that the risk from the missing

scope (or level of detail) is not a contributor. In the second situation, a bounding analysis is used

in lieu of a PRA to estimate the risk contribution from the missing scope (or the missing level of

detail).

In developing the guidance for a bounding analysis, the first question that needs to be addressed

is “what is a bounding analysis” and then “what makes the bounding analysis acceptable?”

A bounding analysis, in the context of a missing scope item, in general, is one that captures the

worst credible outcome of a set of outcomes that result from the event (or hazard) being assessed.

The event (or hazard) being assessed is the risk contributor missing from the PRA (e.g., the risk

from LPSD accidents, the risk from internal fire events). The worst outcome is the one that is the

most challenging to the defined risk metric. Furthermore, a bounding analysis should be bounding

both in terms of the potential outcome and the likelihood of that outcome. Consequently, a

bounding analysis in general considers both frequency of the event and outcome of the event.

This definition is consistent with, but more inclusive than, that provided in the ASME/American

Nuclear Society (ANS) standard16, which defines a bounding analysis as “Analysis that uses

assumptions such that the assessed outcome will meet or exceed the maximum severity of all

credible outcomes.” The same definition can, in principle, be applied to a missing level of detail

item.

How the above definition is applied is dependent on whether a bounding analysis is to bound the

risk or to screen the item as a potential contributor to risk. If a bounding analysis is being used to

bound the risk (i.e., determine the magnitude of the risk of an event), then both its frequency and

outcome are considered. However, if a bounding analysis is being used to screen the event

(i.e., demonstrate that the risk from the event does not contribute), then it can be screened on

frequency, outcome, or both, depending on the specific event.

Given the above discussion on what constitutes a bounding analysis, it needs to be determined

what makes the bounding analysis acceptable. The criteria for acceptability of a bounding analysis

are related to the type of event being bound or screened. The bounding assumptions made by the

analyst will be specific to the event under consideration. Nonetheless, each bounding analysis

needs to address the following to be acceptable: 

• Completeness of Potential Impacts and Their Effects

• Frequency
6. Completeness Uncertainty

NUREG-1855 6-6 Draft for Comment

Completeness of Potential Impacts and Their Effects —

The spectrum of potential impacts of the missing item and the effects on the evaluation of risk has

been addressed such that impacts or effects that could lead to a more severe credible outcome

have not been overlooked.

For potential events, the full spectrum should be addressed such that impacts are not overlooked

that could have a worse credible outcome. That is, the potential impacts that could contribute have

been identified and considered. Given this identification and consideration, there would be little

likelihood that an event with a more severe outcome was overlooked.

For example, suppose that the PRA did not initially address LOCAs. If a bounding analysis were

to be used to address LOCAs, the full spectrum of break sizes would need to be considered. If the

spectrum only included break sizes from 8 inches to 24 inches, the bounding analysis would not

be bounding since the break sizes not considered (e.g., greater than 24 inches) could have a more

severe outcome.

For potential effects, the different effects (e.g., accident progression) resulting from the events have

been addressed such that a more severe credible outcome has not been overlooked. That is, the

different accident progressions have been identified and understood to the extent that a different,

more severe credible outcome is unlikely.

Frequency —

The frequency used in the bounding analysis should be greater than, or equal to, the maximum

credible collective frequency of the spectrum of impacts analyzed.

6.3 Selecting and Using Bounding Approaches

Although there are general characteristics of an acceptable bounding analysis, as provided above,

acceptability of a bounding analysis is also dependent on the type of event being bound or

screened. That is, the bounding assumptions made by the analyst will be specific to the event

under consideration. This dependency is illustrated in the examples of various applications of

bounding analyses discussed below. First, examples of bounding analyses used to screen risk

contributions from further consideration are presented (Section 6.3.1), then some examples of

analyses used to bound the risk contribution to be included in the risk estimate are illustrated

(Section 6.3.2).

6.3.1 Examples of Screening of Risk Contributors

The general process for the screening of missing PRA scope items is a progressive process that

can involve different levels of qualitative and/or quantitative screening. In general, qualitative

screening is performed prior to any quantitative screening analysis. Qualitative screening generally

involves an argument that the missing scope cannot impact plant risk or is not important to the

change in risk (i.e., CDF or LERF) associated with the proposed plant modification. An example

of the former is that the potential for specific external events can be eliminated for many plant sites.
6. Completeness Uncertainty

Draft for Comment 6-7 NUREG-1855

An example of the latter is that changing an at-power technical specification does not impact LPSD

risk. Quantitative risk screening involves a conservative estimate of the risk or change in risk from

the proposed plant modification related to a missing scope item. Examples of this include analyses

that show that a missing initiating event has a low frequency or analyses that indicate that a plant

change does not significantly change the unavailability of a system.

Both qualitative and quantitative approaches can utilize conservative deterministic analyses to

support the screening process. An example of a deterministic analysis in qualitative screening is

the use of a conservative thermal-hydraulic evaluation that shows that the conditions for a

phenomenon such as pressurized thermal shock cannot occur for a given scenario, eliminating the

need to include vessel rupture in the risk evaluation. The performance of a structural analysis that

indicates that a building cannot be damaged by an external hazard such as an explosion is another

example.

An example of a deterministic analysis supporting a quantitative screening process is provided in

the quantitative fire compartment screening process documented in NUREG/CR-6850,

“EPRI/NRC-RES Fire PRA Methodology for Nuclear Power Facilities,” [EPRI, 2005b] where

conservative fire modeling analyses are used to identify fire sources that can potentially cause

damage to important equipment. This analysis allows for eliminating fire sources that cannot cause

damage, thus reducing the fire frequency for compartments that is used in the quantitative

screening process.

Qualitative and quantitative arguments for screening missing scope items are based on the

application of appropriate screening criteria. Recommended screening criteria for eliminating items

from the scope of a base PRA are provided in the available ASME PRA standard [ASME, 2005a]

and ANS external events PRA standard [ANS, 2007a] and in the LPSD and fire PRA standards that

are currently in development. These screening criteria are applicable for PRAs, subject to their

endorsement or clarification as provided in RG 1.200 [NRC, 2007a]. In addition, screening criteria

in NRC guidance documents can also be utilized. An example of this is the screening criteria

specified in the joint EPRI/NRC fire PRA methodology. 

The screening process performed for a base PRA must be reviewed for a risk-informed application

to verify that the screening is still appropriate. This can be accomplished by confirming that the

proposed plant modification or operational change does not change the reason for the screening

in the base PRA. The cause-and-effect relationship that is used to establish the impact of the

proposed change on SSCs and the required scope of the required PRA will provide the information

necessary to accomplish this review. The cause-and-effect determination can also be utilized for

qualitative screening of missing scope items. In this case, if there is no identified effect on missing

scope items, then there will be no change in the risk parameters used to evaluate the plant change.

However, if there is an identified effect, then either a screening or a detailed quantitative analysis

must be performed to determine the impact on the required risk parameter. 
6. Completeness Uncertainty

NUREG-1855 6-8 Draft for Comment

6.3.1.1 Qualitative Screening

Qualitative screening is utilized in many facets of establishing a base PRA. It involves the

application of approved screening criteria such as those specified in PRA standards and guidance

documents to eliminate potential risk contributors from the PRA. Since the performance of a base

PRA is a potential application, these screening criteria are applicable to those applications.

Furthermore, as previously indicated, the screening process used in the base PRA must be

reviewed for applicability following a proposed plant change. Some examples of these criteria are

provided below to illustrate the general nature of the qualitative screening criteria that can be

utilized.

The primary application of qualitative screening criteria is in the analysis of spatially related initiators

such as internal fire and flooding and external events. For example, in internal fire and flood

analyses, compartments can be eliminated from analysis if they contain no equipment that can

cause a plant trip (manual or automatic) or require the need for an immediate plant shutdown and

no SSCs required to mitigate the event are located in the area. Variations of these criteria exist and

in some cases require the performance of a deterministic analysis. For example, the requirements

for internal flood areas provided in the ASME PRA standard also allow screening of flood areas if

no plant trip or shutdown will occur AND if the flood area has flood mitigation equipment such as

drains capable of preventing flood levels from causing damage to equipment. A deterministic

analysis would be required in this case to show that the drains are sufficient in size to prevent the

water level from all flood sources from rising to a level that would result in equipment or structure

damage.

Qualitative screening can also be accomplished by considering whether there is a hazard source

in an area and if there is a potential for propagation. For example, a flood area could also be

screened from consideration if there are no identified flood sources in the area and if floods from

other areas cannot propagate to the area.

External event risk analysis typically begins with a screening process to eliminate events that are

not possible at a plant or that can be combined with other events. Although the list of possible

external events is large (see appendix ANS External Events PRA standard), most are typically

screened from analysis in PRAs. External events included in most PRAs include both natural and

manmade events. Natural external events typically include earthquakes, high winds and tornados,

and external flooding. Manmade external events are evaluated less frequently, but the most typical

events analyzed include airplane crashes, explosions at nearby facilities, and impacts from nearby

transportation activities.

The ANS External Events PRA standard has endorsed the following set of five external event

screening criteria:

(1) The event would result in equal or lesser damage than the events for which the plant has

been designed. This requires an evaluation of plant design bases in order to estimate the

resistance of plant structures and systems to a particular external event.
6. Completeness Uncertainty

Draft for Comment 6-9 NUREG-1855

(2) The event has a significantly lower mean frequency of occurrence than another event,

taking into account the uncertainties in the estimates of both frequencies, and the event

could not result in worse consequences than the other event.

(3) The event cannot occur close enough to the plant to affect it. Application of this criterion

must take into account the range of magnitudes of the event for the recurrence frequencies

of interest.

(4) The event is included in the definition of another event.

(5) The event is slow in developing, and it can be demonstrated that there is sufficient time to

eliminate the source of the threat or to provide an adequate response.

Most external events are screened from further consideration based on Criteria 1 and 4. In general,

the following external events must be considered further in a risk assessment:

• aircraft impacts

• external flooding

• extreme winds and tornados (including generated missiles)

• external fires

• accidents from nearby facilities

• internal fires

• internal flooding

• pipeline accidents (e.g., natural gas)

• release of chemicals stored at the site

• seismic events

• transportation accidents

• turbine-generated missiles

The ASME/ANS PRA standard also indicates that a secondary external event screening step can

be performed, which allows that if the current as-built and as-operated plant conforms to the designbasis requirements for an external event in the 1975 Standard Review Plan (SRP) [NRC, 1975b],

then that event can be screened from further analysis. This criterion was accepted by the NRC for

use in the individual plant examination of external events based on the judgment that the

contribution to core damage from an external event that meets the criteria in the SRP would be less

than10-6/yr. It is noted that the SRP requires analysis of certain design-basis events that have

frequencies between 10-7/yr and 10-6/yr. However, this criterion should not be used for screening

seismic events, and seismic events were explicitly excluded in NUREG-1407 [NRC, 1991a] and

from the ANS External Events PRA standard from being screened in such a manner. 

Even for nonseismic events, the analyst should use caution in applying the above criterion. One

of the pitfalls in using the SRP criterion is that emphasis is placed on comparisons of event lists with

the design bases of the safety-related systems and structures. However, PRAs have shown that

there are important risk contributions from nonsafety-related systems, and their capacities are

generally not evaluated. More importantly, there is also the possibility that the magnitude of an

external event may exceed the plant design basis. In fact, if the exceedence frequency for the
6. Completeness Uncertainty

NUREG-1855 6-10 Draft for Comment

external event is relatively flat around the design-basis magnitude, then the contribution from larger

magnitude events could be important. In addition, a significant risk contribution from lower

magnitude events is also possible if the susceptibility of the plant to damage (fragility) is also

relatively insensitive to the magnitude of the event. In order to provide a convincing case that an

external event can be excluded based on the SRP screening criterion, it may be necessary to

perform some bounding estimates of the risk for both lower and higher magnitude events.

Qualitative screening criteria are also used in the evaluation of internal events but to a lesser

extent. For example, a random initiator such as a loss of heating, ventilation, and air conditioning

can be screened if the event does not require the plant to go to shutdown conditions until sufficient

time has expired, during which the initiating event conditions can be detected and corrected before

normal plant operation is terminated (either automatically or administratively in response to a

limiting condition of operation). Deterministic analyses would be required to utilize this criterion.

Qualitative screening is also performed in the evaluation of preaccident human errors to restore

equipment following test and maintenance. In this case, factors such as automatic realignment of

equipment on demand, the performance of postmaintenance tests that would reveal misalignment,

position indicators in the control room, and frequent equipment status checks all can be used as

justification for screening out preaccident human errors. However, there is an implied probabilistic

(quantitative) argument associated with such screening.

For a risk-informed application, the screening process performed for a base PRA must be reviewed

to verify that the screening is still appropriate. As mentioned previously, this can be accomplished

by confirming that the proposed plant modification or operational change does not change the

reason for the screening in the base PRA. In addition, missing scope items can also be

quantitatively screened. The cause-and-effect relationship that is used to establish the impact of

the proposed change on SSCs and the required scope of the required PRA will provide the

information necessary to accomplish both reviews. If there is no identified effect on missing scope

items, then there will be no change in the risk parameters used to evaluate the plant change.

However, if there is an identified effect, then either a screening or a detailed quantitative analysis

must be performed to determine the impact on the required risk parameter.

6.3.1.2 Quantitative Screening

Quantitative screening is utilized in most of the technical elements that comprise a base PRA. It

involves the application of approved screening criteria such as those specified in PRA standards

and guidance documents to eliminate potential risk contributors from the base PRA. The screening

criteria are developed and must be correctly utilized to ensure that the screening process does not

eliminate elements of a PRA model that can provide a significant contribution to the risk estimate.

These screening criteria are also applicable for use in risk-informed applications and can be used

for screening missing scope items.

Quantitative screening criteria can be either purely quantitative or incorporate both quantitative and

qualitative components. They can include consideration of the individual contribution of a screened

item and the cumulative contribution of all screened items. Special consideration is generally given

to those items that can result in containment bypass. Some of the criteria also use comparative
6. Completeness Uncertainty

Draft for Comment 6-11 NUREG-1855

information as a basis for screening (e.g., an item whose contribution is significantly less than the

contribution from another item can be screened). Some examples of these criteria are provided

below to illustrate the general nature of the quantitative screening criteria that can be utilized:

• Initiating events can be screened if their frequency is less than 10-7/yr as long as the event

does not include a high consequence event such as an interfacing system LOCA,

containment bypass, or reactor vessel rupture. Alternatively, initiating events can be

screened if their frequency is less than 10-6/yr and core damage could not occur unless two

trains of mitigating systems are failed independent of the initiator. [ASME, 2005a]

• A component may be excluded from a system model if the total failure probability of all the

component failure modes resulting in the same effect on system operation is at least two

orders of magnitude lower than the highest failure probability of other components in the

same system resulting in the same effect on system operation. A component failure mode

can be excluded from the system model if its contribution to the total failure probability is

less than 1 percent of the total failure probability for the component, when the effect on

system operation of the excluded failure mode does not differ from the effects of the

included failure modes. However, if a component is shared among different systems (e.g.,

a common suction pipe feeding two separate systems), then these screening criteria do not

apply. [ASME, 2005a]

• An internal flood initiating event can be screened if it affects only components in a single

system and if it can be shown that the product of the flood frequency and the probability of

SSC failures given the flood is two orders of magnitude lower than the product of the

nonflooding frequency for the corresponding initiating events in the PRA and the random

(nonflood induced) failure probability of the same SSCs that are assumed failed by the

flood. [ASME, 2005a]

• A flood area can be screened if the product of the sum of the frequencies of the flood

scenarios for the area and the bounding conditional core damage probability is less than

10-9/yr. [ASME, 2005a]

• A fire compartment can be screened if the CDF is less than 10-7/yr and LERF is less than

10-8/yr. The cumulative risk from the screened fire compartments should be less than

10 percent of the total internal events risk. [EPRI, 2005b]

The ANS external events PRA standard provides guidance for probabilistic analyses of seismic,

high-wind, and external flooding events. A set of screening criteria is provided for eliminating

external event initiators, component failures, and accident sequences. For example, an initiating

event can be screened out if any of the following applies:

• The event meets the criteria in the NRC’s 1975 SRP (or a later revision).

• It can be shown using a demonstrably conservative analysis (i.e., bounding analysis) that

the mean value of the frequency of the design-basis hazard used in the plant design is less
6. Completeness Uncertainty

NUREG-1855 6-12 Draft for Comment

than approximately 10-5/yr and that the conditional core damage probability is less than 0.1,

given the occurrence of the design-basis-hazard event.

• It can be shown using a demonstrably conservative (i.e., bounding) analysis that the CDF

is less than 10-6/yr.

As noted previously, since a proposed plant change can change the basis for screening, the

screening process used in the base PRA must be reviewed for applicability following a proposed

plant change. In addition, the type of screening criteria identified above can be used in conjunction

with bounding risk evaluations to eliminate items missing from the scope of a PRA required to

support a risk-informed application.

One quantitative method consists of a progressive screening approach that is performed using the

PRA technical elements as a guide. The screening process begins with screening out initiating

events with sufficiently low frequencies and proceeds to screening specific sequences if necessary.

Screening of LPSD conditions is done on a POS level. A quantitative screening approach that can

be used to screen certain specified shutdown POSs from requiring further quantitative risk

assessment (in the context of a specified application) is based on demonstrating that the risk for

these POSs is lower than some predetermined limiting value. The POSs that may be screened by

this approach involve only the POSs in which a plant is shut down; this approach does not address

POSs involving low-power operation or power transition (i.e., only POSs that span cold shutdown

or refueling as defined by a plant’s technical specifications). This approach performs a qualitative

comparison of the plant systems and configurations being examined with reference plant systems

and configurations. If sufficient similarity exists, then bounding numerical risk metric results (e.g.,

CDF) can be determined. If sufficient similarity does not exist, the method cannot be used. The

basis for the numerical results is expected to ensure that the actual numerical value for the plant

being examined is equal to (or less than) the bounding results. If the bounding result is above a

specified screening value, the POS cannot be screened.

Quantitative screening can be used to either obtain an approximation of the appropriate risk metric

(e.g., CDF or LERF) for a POS or to eliminate initiating events within a POS. There are two ways

by which an initiating event may be eliminated:

(1) elimination based on frequency times the fraction of time in a POS

(2) elimination based on the amount of time before core damage begins

An initiating event may be eliminated from further consideration if the product of its frequency times

the fraction of time a POS occurs (per year) is less than a specified screening value (e.g., 10-8/yr).

To be considered for elimination, the initiating event must not lead directly to a bypass of the

containment (e.g., steam generator tube rupture).

An initiating event may also be eliminated if the time to core damage is greater than a specified time

period (e.g., 24 hours) and the initiating event does not necessarily preclude or hamper potential

recovery actions (e.g., a large seismic event might be expected to hamper potential offsite recovery

actions).
6. Completeness Uncertainty

Draft for Comment 6-13 NUREG-1855

6.3.2 Examples of Bounding Risk Contributors

A bounding risk method can involve a simplified risk assessment where the failure probabilities of

available SSCs are conservatively estimated and combined with the (conservative estimate of the)

initiating event frequency. The bounding estimation of the SSC failure probability can be based on

bounding the known information (e.g., based on the failure probability of other similar SSCs or

generic information), based on a bounding detailed analysis (e.g., by generating a fault tree for a

system), or by a bounding deterministic analysis (structural evaluation of a structure). It is critical

in this process that the simplified bounding analysis reflect the as-built and as-operated plant and

include all dependencies. However, care must be taken to ensure that the bounding results do not

mask important risk contributors.

The evaluation of an external event can be bounded using a hazard analysis and some bounding

assessment of the plant damage and consequences. The effects of the external event can be

evaluated conservatively using upper bound assumptions. For example, the maximum magnitude

of a gas explosion could be used to determine the overpressure that would be applied to structures,

the potential for missile generation, and the magnitude of a fire. The effects of the event on

accident initiators and mitigating systems can also be bounded. For example, all systems in a

building could be assumed damaged by an aircraft impact that is conservatively assessed to result

in building damage.

Specific guidance for performing bounding assessments of external events is provided in

NUREG/CR-4832 [SNL, 1992a] and NUREG/CR-4839 [SNL, 1992b]. This guidance was utilized

in the risk methodology integration and evaluation program study and is valid for use in

risk-informed applications. It addresses aircraft crashes, high winds and tornados, transportation

accidents, turbine-generated missiles, accidents at nearby facilities, and external flooding. The

guidance can be used to establish the baseline risk from these external events and to determine

the risk from a proposed risk-informed change to the design or operation of the plant

PRAs typically combine similar initiating events into groups for evaluation. For example, a small

LOCA could be conservatively modeled as a large LOCA if the system success criteria for the large

LOCA are more limiting than the small LOCA. However, the following other factors must be

considered besides the system success criteria to ensure that the resulting risk estimate is

conservative:

• The impact of the initiating event on the SSCs must be included in the analysis. 

• The success criteria for systems must be more restrictive during all phases of an accident.

• Mitigating system requirements (i.e., support systems) and dependencies for the

unanalyzed event should be bounded by the event analyzed.

• The timing of the accident sequences and the impact on the operator actions must be

bounded.
6. Completeness Uncertainty

NUREG-1855 6-14 Draft for Comment

• Phenomenological conditions created by the missing accident must be included in the

analysis.

• The potential for a large, early release is bounded by the analysis.

It may be possible to estimate the risk associated with certain LPSD POSs with a bounding

analysis. These POSs include those for which the technical specifications are unchanged from

power operation. If the technical specifications for a POS are the same as those at power, the CDF

or LERF may be approximated by simply multiplying the fraction of time (per year) spent in the POS

by the appropriate risk metric determined by the at-power PRA. This approach can be used as long

as there are no new initiating events for the POS in question nor any change to the frequency of

initiating events as defined in the at-power PRA. If changes to initiating event frequencies are

possible, then the at-power PRA model must be either requantified (if the initiating frequency

decreases) or resolved (if the initiating frequency increases). Once the new at-power risk metric

results have been obtained, the approach described in the previous paragraph can be used to

approximate the risk associated with the POS. Note that if an initiating event’s frequency

decreases, one may approximate the risk by simply performing the calculation described in the

previous paragraph; the resulting risk metric will simply be more conservative and less “realistic.”
Draft for Comment 7-1 NUREG-1855

7. RISK-INFORMED DECISION MAKING: DEALING WITH

UNCERTAINTY

7.1 Introduction

The purpose of this chapter is to provide guidance on addressing the uncertainty in PRA results in

the context of risk-informed decision making. Specifically, guidance is provided on the

interpretation of the results of the uncertainty analysis when comparing results derived from a PRA

with the acceptance criteria or guidelines established for a specific application. 

As discussed in Section 2.1, in a risk-informed environment, an evaluation of the risk implications

is one of several inputs to making a decision. Figure 7-1, based on RG 1.174 [NRC, 2002a],

illustrates the integrated nature of decision making. 

Figure 7-1 Integrated Decision Making Process

When providing input to the decision makers, it is the responsibility of the risk analyst to ensure that

the conclusions of the risk analysis are documented and communicated clearly. An important part

of this documentation is a discussion of the robustness of, or the confidence in, the conclusions

drawn from that analysis. To address robustness, the key sources of uncertainty and the key

assumptions need to be addressed. This chapter focuses on the way in which the key uncertainties

and key assumptions are addressed in drawing conclusions related to the risk implications relevant

to the decision. 

As discussed in previous chapters of this report, uncertainties in the risk evaluation can be dealt

with in different ways. Where possible, they are addressed explicitly in a quantitative manner.

However, in some cases, the characterization of the uncertainty in the PRA results in a quantitative

manner may be challenging. This challenge can occur, for example, when there is no agreed-upon

theoretical or empirical basis for representing the impact on the plant of the issue being evaluated,

so that the effect on risk cannot be quantified without some degree of arbitrariness.

Principle 4

Risk-Informed 

Analysis

Principle 1

Current 

Regulations Met

Principle 3

Maintenance of 

Safety Margins

Principle 5

Monitor 

Performance

Integrated RiskInformed Decision 

Making

Principle 2

Defense-in-Depth 

Consistency

Principle 4

Risk-Informed 

Analysis

Principle 1

Current 

Regulations Met

Principle 3

Maintenance of 

Safety Margins

Principle 5

Monitor 

Performance

Integrated RiskInformed Decision 

Making

Principle 2

Defense-in-Depth 

Consistency
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-2 Draft for Comment

An example is the modeling of the effect of a reduction in special treatment on the unreliability of

SSCs. In this case, one solution is the adoption of a performance-monitoring strategy to ensure

that the degradation in SSC performance is no larger than that assumed in the demonstration that

the change in risk is small. 

In other cases, an analyst may decide not to address a potentially significant contributor to risk

using a PRA model for that contributor. This decision may be made for one of a number of

reasons, including lack of expertise in the methods or resource limitations. Chapter 6 discussed

the use of bounding approaches to demonstrate that a missing PRA scope contributor to risk does

not contribute significantly. This approach is one way of quantitatively addressing the uncertainty

in the PRA results that arises from using an incomplete PRA model. However, when the missing

contributors cannot be demonstrated to be insignificant contributors to the decision, an alternative

approach is needed. One possibility, rather than attempting to quantify the risk impact, is to adopt

conditions on the implementation of the decision to compensate for the uncertainty, by essentially

arguing that the unquantified portion of risk is not affected by the implementation. An

understanding of the uncertainties as they affect the risk insights in a more qualitative sense can

also provide input to defining performance measurement strategies or compensatory measures.

This chapter, in addressing the above issues, provides guidance on the following:

• presenting the results of PRA uncertainty analysis

• taking uncertainty into account in addressing the comparison of PRA results with

quantitative acceptance criteria and as a special case

• addressing uncertainty in SSC categorization

• using qualitative approaches to address uncertainty in integrated decision making

7.2 Presenting the Results of PRA Uncertainty Analyses

Guidance is provided regarding the presentation of the results of a PRA uncertainty analysis. In

order to provide input to a decision, the results of a risk analysis have to be compared against some

acceptance criteria or guidelines that provide a means of measuring the significance of the risk

implications. Consequently, it is important that the PRA results are presented in a manner that is

compatible with how the results are going to be used. Typically, the ways in which the uncertainty

in PRA results is presented include the following:

• a continuous probability distribution on numerical results

• a discrete probability distribution representing the impact of different models or assumptions

• sensitivity studies that provide a discrete set of results that represent the results of making

different assumptions or using different models, or that represent the impact of varying key

parameters in the model that have significant uncertainty, without providing weights or

probabilities to the members of the set

• bounds or ranges of results that represent the results of the extreme assumptions
7. Risk-informed Decision Making: Dealing with Uncertainty

Draft for Comment 7-3 NUREG-1855

The first methods, a continuous probability distribution on numerical results, is only feasible when

input uncertainties are represented as probability distributions and there is an analytical means of

propagating uncertainty. As discussed in Chapter 4, this is the approach used to address

parameter uncertainties. The second approach, a discrete probability distribution representing the

impact of different models or assumptions, is only practical when the analysts are willing to weigh

their beliefs in the different models or assumptions that may be used. In practice, this is rarely

employed, but it has been used for the representation of seismic hazard (e.g., [Parry, 1986a]).

Such a distribution could be propagated through to provide a probability distribution on the final

results

In the most general case, a number of these means of characterizing uncertainty may be used in

any one analysis to address the different types of uncertainty. Furthermore, different approaches

may be used for the different risk contributors, particularly when some contributors are addressed

by detailed PRA models whereas others are addressed by approximate methods based on

alternate approaches, such as margins studies, or bounding analyses. 

Regardless on how the results of the risk analysis are presented, the analyst has to understand and

be able to communicate what is driving the uncertainty. Interpreting the significance of the results

of a PRA in light of the uncertainties is important if the PRA results are to be applied to making

meaningful decisions about changes in design or operating practices, or if they are to be used for

economic decisions. While it is important to characterize the overall uncertainty, it is equally

important to understand which factors drive the uncertainty and also the implications of making

specific assumptions. For example, it is important for a decision maker to know whether a

particular contributor is being evaluated in a very conservative manner, particularly if it is influencing

the decision significantly. Even if the results of alternate models are probabilistically combined in

the final risk results, it is essential to be able to distinguish between the results of the alternate

models. 

Several methods are available for identifying the drivers of uncertainty. The two principal tools are

importance analysis and sensitivity analysis:

• Importance Analysis — Several different importance measures are in common usage.

These include the Fussell-Vesely and Birnbaum importance measures, risk achievement

worth, and risk reduction worth [Cheok, 1998a]. When applied to the results of a risk

analysis, they measure the significance of a basic event to the overall result. Once a basic

event is identified as significant, the uncertainty associated with the evaluation of that basic

event is also significant to the results.

• Sensitivity Analysis — Sensitivity analysis is a powerful tool for both investigating and

characterizing the impact of uncertainty on results. However, to be meaningful, the

sensitivity analyses must be chosen carefully and must represent reasonable alternatives

to the model of choice. Because uncertainties can have synergistic effects on the results,

as discussed in Chapter 5, it may be necessary to perform sensitivity analyses of more than

one of the uncertain issues simultaneously.
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-4 Draft for Comment

7.3 Comparison of PRA Results with Acceptance Criteria 

Guidance is provided on approaches to assessing the impact of the uncertainties when comparing

PRA results with the acceptance criteria used in decision making to delineate between acceptance

and rejection with regard to the application or decision under consideration. The acceptance

criteria are defined in terms of a value, or values, of metrics (e.g., CDF). This section addresses

the following:

• a brief review of the risk acceptance guidelines of RG 1.174 as an example of a widely used

set of quantitative acceptance guidelines.

• approaches for dealing with the three categories of uncertainty:

– parameter uncertainty

– model uncertainty

– completeness uncertainty

• interpretation of the results of the PRA

7.3.1 An Example of Risk Acceptance Guidelines Using RG 1.174

The acceptance guidelines of RG 1.174 are used to illustrate the issues associated with

comparison of PRA results to acceptance criteria or guidelines in a risk-informed environment. The

use of the RG 1.174 guidelines to illustrate the issues is driven by the fact that they are among the

most widely used set of acceptance guidelines in the regulatory arena for nuclear reactors.

Furthermore, their formulation was informed by other acceptance criteria or guidelines in common

use, such as those in the Regulatory Analysis Guidelines [NRC, 2004a], and the subsidiary

objective to the Safety Goal Policy Statement [NRC, 1986a].

The acceptance guidelines used in RG 1.174 for one of the metrics, CDF, is shown in Figure 7.2.

Figure 7-2 Risk Acceptance Guidelines
7. Risk-informed Decision Making: Dealing with Uncertainty

Draft for Comment 7-5 NUREG-1855

In the context of RG 1.174, the intent of comparing the PRA results with the acceptance guidelines

is to demonstrate with reasonable assurance that the principle that requires the risk change to be

small is indeed being met. This decision must be based on a full understanding of the contributors

to the PRA results and the impacts of the uncertainties, both those that are explicitly accounted for

in the results and those that are not. This is a somewhat subjective process, and the reasoning

behind the decisions must be well documented.

As discussed earlier in this report, when establishing a decision criterion, it is crucial to define how

uncertainty is to be addressed when comparing calculated results with the criterion. This is

discussed in RG 1.174 in Section 2.2.5.5, where parameter uncertainty, model uncertainty, and

completeness are addressed in somewhat general terms. This section provides more guidance

on this comparison. 

With respect to parametric uncertainty, when establishing the acceptance criteria for RG 1.174, the

following alternatives were discussed. The measure to be used for comparison should be (1) the

mean value of the probability distribution representing the epistemic uncertainty or (2) a specified

percentile of the uncertainty distribution, corresponding to a specified confidence level. The

guidelines were finally chosen such that the appropriate measure for comparison is the mean value

of the corresponding uncertainty distribution. The reason for this was to some extent historical,

since the guidelines were based on prior guidelines for which mean values were to be used.

However, in addition, it was considered that there is a philosophical problem associated with

determining what would be the appropriate confidence level to choose if Option (2) above were

adopted. More details of the reasoning behind the approach can be found in SECY-97-221 [NRC,

1997a].

The metrics chosen are such that, for a risk-informed decision, all contributors to risk should be

addressed, and each significant contributor to risk is taken into account. In this context, a

significant contributor is one whose risk contribution can have an effect on the decision. 

The acceptance guidelines are not intended to be interpreted as overly prescriptive. They are

intended to provide an indication, in numerical terms, of what is considered acceptable. This is

largely in recognition that the state-of-knowledge, or epistemic, uncertainties associated with PRA

calculations, and, in particular, those that are not explicitly included in the uncertainty analysis used

to calculate the mean values, preclude a definitive decision with respect to in what region the

results used in the application belong, based purely on the numerical results. Furthermore, it is

recognized that there may be unquantified contributions that could increase or decrease risk. In

the case of RG 1.174, the term “acceptance guidelines” was used rather “acceptance criteria,”

primarily to recognize that not all of the analysis uncertainty was captured in the uncertainty

distribution. The implications of this are discussed in Section 7.3.3.5.

7.3.2 Addressing Uncertainty

As discussed in Chapter 3, epistemic uncertainty is generally categorized into three types —

parameter, model, and completeness uncertainty. Because they are characterized in different

ways, the approaches to addressing them are different as discussed below.
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-6 Draft for Comment

7.3.2.1 Parameter Uncertainty

Methods for addressing parameter uncertainty in the calculation of PRA results are described in

Chapter 4. Since the impact of parameter uncertainty can be addressed in terms of a probability

distribution on the numerical results of the PRA, it is straightforward to compare a point value, be

it the mean, the 95th percentile, or some other representative point value, with an acceptance

guideline or criterion. As stated previously, the point value to be used should be that specified

when the acceptance criterion is established. For example, the appropriate numerical measures

to use in the comparison of the PRA results with the acceptance guidelines of RG 1.174 are mean

values. The mean values referred to are the arithmetic means of the probability distributions that

result from the propagation of the uncertainties on the input parameters.

While a formal propagation of the uncertainty is the best way to correctly account for state-ofknowledge uncertainties that arise from the use of the same parameter values for several basic

event probability models, under certain circumstances, a formal propagation of uncertainty may not

be required if it can be demonstrated that the state-of-knowledge correlation is unimportant. This

will involve, for example, a demonstration that the bulk of the contributing scenarios (cut sets or

accident sequences) do not involve multiple events that rely on the same parameter for their

quantification. This is discussed in detail in Section 4.3.2.

It should be noted that, if model uncertainties have been explicitly represented in the PRA model

using a discrete probability distribution and propagated through the analysis, this would also be

represented in the mean value. 

7.3.2.2 Model Uncertainty

Methods for identifying and assessing the impact of key uncertainties are discussed in Chapter 5.

In the context of decision making, it is necessary to assess whether these uncertainties have the

possibility of changing the evaluation of risk significantly enough to alter a decision. As discussed

in Chapter 5, this assessment can take the form of reasonable and well-formulated sensitivity

studies or qualitative arguments.

In this context, “reasonable” is interpreted as implying some precedent for the alternative

assumption or model represented by the sensitivity study(ies), such as previous use by other

analysts, and that there is a physically and theoretically reasonable and documented basis for the

alternative. It is not necessary for the search for alternatives to be exhaustive and arbitrary. The

sensitivity studies provide a discrete set of results that demonstrate the impact of the alternative

models or assumptions on the mean values calculated as discussed above. 

7.3.2.3 Completeness Uncertainty

In this section, what is classified as a completeness issue is a concern about the limitation of the

scope of the model. The issue of completeness of the scope of a PRA can be addressed

quantitatively for those scope items for which limited methods are in principle available, and

therefore some understanding of the contribution to risk exists.
7. Risk-informed Decision Making: Dealing with Uncertainty

Draft for Comment 7-7 NUREG-1855

For example, the out-of-scope items can be addressed by supplementing the analysis with

additional analysis by alternative methods to a fully developed PRA. Approaches to supplementing

the PRA model by using alternate models are discussed in Chapter 6. These methods can either

be approximations or bounding analyses. 

However, neither an approximate method nor a bounding analysis can be used in the same way

as a full PRA model to fully understand the contributions to risk and thereby gain robust risk

insights. Thus their use is somewhat limited. The principal use of such methods is to demonstrate

that the risk contribution from that contributor, and any change to that risk contribution resulting

from a change to the plant, are small. For this to be the case, the result of the analyses should be

demonstrably conservative. When this is the case, there is no need to explicitly capture the

uncertainty in the risk analysis, since it has been demonstrated that the contribution(s) so

dispositioned is not significant to the decision.

The degree to which supplementary arguments can be used to support the claim that the

uncertainties do not impact the decision depends on the proximity to the guidelines. When the

contributions from the modeled contributors are close to the guideline boundaries, the argument

that the contribution from the missing items is not significant must be more convincing than when

the results are further away from the boundaries and in some cases may require additional PRA

analyses. When the margin is significant, a qualitative argument may be sufficient. 

7.3.3 Interpretation of the Results of the PRA

This section provides guidance on how to interpret the results derived from a PRA, taking into

account the results of the analysis of uncertainty. While the focus in this report is the treatment of

parameter, model, and completeness uncertainty, it is also important to recognize the unquantified

uncertainty that is introduced by making modeling choices regarding the level of detail with which

the PRA model is constructed. 

7.3.3.1 Uncertainty Arising from Level of Detail

Even in the absence of model uncertainty or completeness uncertainty, differences can arise in

PRA models performed by two groups of analysts for the same plant. Differences can arise

because one model may be developed to a lower level of detail than the other. For example, when

developing a PRA model for a BWR, one analyst may choose to include the fire water system as

an additional low-pressure injection system, whereas another might not. The latter may have

decided that the contribution to the frequency of core damage of the accident sequences resulting

from failure of the low-pressure injection function is low enough without taking into account systems

in addition to the low-pressure coolant injection and low-pressure core spray.

Regardless of the level of detail chose, iIf developed in accordance with the ASME standard, both

models would, however, be expected to have captured the risk-significant sequences, and the

differences between the two models should be in the less significant contributors. Therefore, the

differences between the two model results that are a consequence of the level of detail would be

expected to be minor.
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-8 Draft for Comment

Consequently, in reality, there is essentially no difference in quantitative results that differ by a very

small amount. What the scale for this small amount should be is not defined. However, using the

acceptance guidelines for RG 1.174 as the basis for generally defining “small,” a very small change

is defined as being less than 10-6 (ΔCDF). This value was chosen recognizing that this was an

assessment of a change that could be negated by a (legitimate) refinement of the model, and thus

it represented a measure of the level of discrimination afforded by a PRA model.

Generally, the less detailed model would be expected to be slightly conservative with respect to the

more detailed model. The level of conservatism that is incorporated in a PRA model must be

considered when using the results in a risk-informed application, as discussed later in this section.

This is a particular concern when results from PRAs for different contributors are combined as

discussed below.

7.3.3.2 Parametric Uncertainty

The comparison of the appropriate point value derived from the probability distribution on the

corresponding PRA result, be it the mean value or a percentile of the distribution, represents the

first level of uncertainty analysis. In RG 1.174, the interpretation of the acceptance guidelines is

that they should not be regarded as overly prescriptive. This interpretation is a reflection of the

realization that there are residual uncertainties not captured in the mean value. For example, there

may be conservatism in the model or things that have been left out. Therefore, acceptance or

rejection is not based on a strict interpretation whereby a result just inside one region would be

acceptable, whereas one just outside would not.

The analyst should qualify the result recognizing any conservatisms or nonconservatisms

associated with the level of detail of the PRA model. When the point value is well away from the

criteria boundaries, any concerns about the impact of a potential bias caused by the choice of level

of detail tends to be a lesser concern. However, when the result is close to the boundary, then the

level of detail could make the difference between being above or below the boundary.

This situation is a recurring issue with Phase 3 Significance Determination Process evaluations,

where often the discussion centers about whether the assessment of the risk significance of a

finding is above or below the boundary between, for example, a green and a white finding. When

both the staff and the licensee values are relatively close, and the assumptions made to model the

significance of the issue are agreed upon, the difference between the values should not be an

issue. This problem arises from treating the calculated values as absolute when comparing with

the criteria for determining the “color” of the inspection finding. 

7.3.3.3 Model Uncertainty

The second level of analysis addresses model uncertainty. As discussed above, model uncertainty

is typically represented in terms of the results of sensitivity studies that determine the effect of

alternate hypotheses or models. 
7. Risk-informed Decision Making: Dealing with Uncertainty

Draft for Comment 7-9 NUREG-1855

When the results of the sensitivity studies confirm that the guidelines are still met even under the

alternative assumptions (i.e., change generally remains in the appropriate region), the risk principle

can be determined to be met with confidence. 

However, when some alternative hypotheses lead to a significant change in the relationship of the

PRA result to the acceptance criteria, the analyst should provide the decision maker with the basis

for considering the credibility of the alternatives. For example, if reasons can be given as to why

they are not appropriate for the particular application or for the particular plant, they need not be

taken into account. If, however, the decision maker does not have sufficient confidence that the

alternative hypothesis that challenges the decision criteria can be discounted, the tendency will be

to err on the side of caution. Alternatively, the analysis can be used to identify candidates for

compensatory actions or increased monitoring (see Section 7.5). 

When the risk estimate is derived from a number of PRA models providing estimates for the risk

from the different contributors, if the model uncertainty affects several of the constituent models,

the analysis of model uncertainty should be performed in an integral manner. In other words, the

sensitivity analysis of a model uncertainty that is pertinent to the plant response model should be

demonstrated for the sum of all the affected risk contributors.

7.3.3.4 Completeness Uncertainty

When a particular risk contributor is not evaluated by a PRA model, then either the effect on the

application has to be bounded and shown not to be significant, or other measures have to be taken

to ensure that the assumption of no risk increase is supported (see Section 7.5). 

7.3.3.5 Combining PRA Results (Integrated Assessment)

PRA methods exist, in principle, for analyzing each of the potentially significant risk contributors

(i.e., internal events, internal fires, external events, and events during low power and shutdown

(LPSD) operation). The methods employed to address internal fires, external events, and LPSD

events typically make use of the models developed for the analysis of internal events at full power

to address the plant response to the various challenges to the plant; therefore, they are subject to

the sources of uncertainty associated with those models. However, in addition, each of the PRA

analyses for these other scope items has its own unique sources of uncertainty. For example, in

a fire PRA, there are uncertainties associated with the modeling of fire growth and suppression as

well as other areas of the analysis, such as the potential for fire damage; in a seismic PRA, there

are uncertainties associated with the assessment of the seismic hazard and with the structural

response of the SSCs. 

Furthermore, the models for the different contributors are often developed to different levels of

detail. In particular, the set of mitigating equipment that is credited with fulfilling the critical safety

functions may, for one reason or another, differ from one scope item to the next. For example,

when analyzing internal fires, only those systems that are known to be unaffected by fires may be

credited. The demonstration that a particular system that is not part of the safe-shutdown train is

not affected by fire may involve cable tracing. If the fire-initiated CDF is low enough without taking
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-10 Draft for Comment

credit for that system, it may be decided not to perform cable tracing, and the system would not be

included in the fire PRA even if credit were taken for that system in the internal events PRA. Thus,

the PRA models for the different scope items may reflect a different level of detail in modeling plant

response. 

In addition, when constructing a PRA model for the risk from a specific risk contributor, screening

approaches are typically employed to limit the number of scenarios that need to be modeled. The

specific approach to screening varies between the PRA models used for the different risk

contributors. This can result in some of the models yielding results of differing degrees of realism

or conservatism that can be particularly significant when assessing the impact of changes to the

plant. In addition, it is claimed that, for some risk contributors, particularly fires and seismic events,

the methods used are intrinsically conservative and the uncertainties in the results are therefore

of a different nature and magnitude. Thus, while for the purposes of comparison with acceptance

guidelines or criteria it is necessary to add the results from the various contributors to risk, the result

must be interpreted carefully to take into account the differing degrees of uncertainty associated

with each of the constituent PRA models.

Therefore, when results from different risk contributors are combined, because of the concern that

the representation of the risk from some contributors may be relatively more conservative than

others, it is important to determine the relative contributions from each of the contributors and to

identify whether there is significant conservatism associated with any of these contributors. This

helps focus the decision maker on those aspects of the analysis that have the potential to influence

the outcome of the decision. This becomes important for applications in which there is a need to

assess the total risk impact, particularly when the total risk metric value approaches the boundary

between regions of acceptability. The differing level of detail in constituent analyses is particularly

important when considering risk ranking of SSCs, since the absence of SSCs from some of the

models will distort the overall results.

When analyzing the results, it is essential not only to focus on the numerical integrated result but

also to identify the main contributors. If there are contributors that are considered to be particularly

conservative in such a way that they could lead to challenging meeting the risk acceptance criteria,

it is incumbent upon the analyst to provide a convincing demonstration of the extent of the

conservatism, if necessary, by refining the PRA model. Finally, consideration has to be given to

the nonquantified uncertainty (e.g., the conservatism resulting from the modeling approach or the

improvement to safety from programs or design that are not captured in the PRA model). In some

cases, this can be dealt with by proposing compensatory measures designed to negate any

potential negative impact on risk.

7.4 Addressing Uncertainty in SSC Categorization

Categorization of SSCs according to their risk significance is an integral part of many applications,

such as the implementation of 10 CFR§50.69. Nuclear Energy Institute (NEI) 00-04[NEI, 2005b]

addresses uncertainty primarily by using the point estimates of the calculations to perform the initial

categorization, and then performing sensitivity studies that vary some of the key groups of

parameters (e.g., HEPs, CCF probabilities) and address some of the model uncertainties identified
7. Risk-informed Decision Making: Dealing with Uncertainty

Draft for Comment 7-11 NUREG-1855

by the peer review as potentially being significant. These sensitivity studies are used to identify

changes in the categorization, and the most conservative categorization is used.

The reasonableness of this approach is justified in two ways. First, EPRI TR 1008905, “Parametric

Uncertainty Impacts on Option 2 Safety Significance Categorization” [EPRI, 2003a], demonstrates

that the impact of performing a full uncertainty analysis to categorize SSCs does not significantly

change the categorization resulting from using mean value point estimates in the importance

calculations. Second, in regulatory applications, there is a requirement to demonstrate that the

impact of the change resulting from using the categorization to change plant practices is small

changes to risk, using the RG 1.174 acceptance guidelines discussed above. 

The differing level of detail in the models for the different risk scope items requires care in

interpreting the results of using those models to categorize the SSCs.

7.5 Using Qualitative Approaches in Integrated Decision Making

When the analysis of risk does not address all significant risk contributors, or when the impact of

a change on risk has not been estimated, it may be acceptable to adopt one of a number of

strategies to deal with the unknown impact. These strategies, discussed below, include the

following:

• adopting performance monitoring requirements

• limiting the scope of application of plant changes

• establishing compensatory measures

7.5.1 Performance Monitoring Requirements

Monitoring can be used to demonstrate that, following a change to plant design or operational

practices, there is no degradation in specified aspects of plant performance. This is an effective

strategy when there is no predictive model for plant performance in response to a change. One

example of such an instance is the impact of the relaxation of special treatment requirements on

equipment reliability. For monitoring to be effective, the plant performance must be measurable in

a quantitative way, and the criteria used to assess acceptability of performance must be realistically

achievable given the expected quantity of data.

7.5.2 Limiting Scope of Plant Modification

When a PRA model is incomplete in its coverage of significant risk contributors (e.g., it does not

address fire risk), the implementation of a risk-informed change can be restricted so that any SSCs

that would be expected to be used to mitigate the risk from fires would be unaffected, and therefore

that contribution to risk would remain unchanged. This is the strategy adopted in NEI 00-04 for

categorizing SSCs according to their risk significance to determine for which SSCs the special

treatment requirements can be relaxed.
7. Risk-informed Decision Making: Dealing with Uncertainty

NUREG-1855 7-12 Draft for Comment

7.5.3 Use of Compensatory Measures

The purpose of implementing compensatory measures is to neutralize the expected impact of a

change. Examples include the establishment of a fire watch to compensate for a faulty fire barrier,

or the implementation of a manual action (suitably proceduralized and accounted for in training) to

replace an automatic actuation of a system (e.g., the initiation of depressurization of a BWR

following loss of high-pressure injection that is necessitated by the inhibition of the automatic

(automatic depressurization system) function). In many cases, it is not possible to make a

completely convincing case that the compensation is numerically equivalent.
Draft for Comment 8-1 NUREG-1855

8. REFERENCES

[Apost., 1981a] Apostolakis, G., and Kaplan, S., “Pitfalls in Risk Calculations,” Reliability

Engineering, Vol. 2, pp. 135-145, 1981.

[Atwood, 1996a] Atwood, C. L., and Gentillon, C. D., “Bayesian Treatment of Uncertainty in

Classifying Data: Two Case Studies,” Probabilistic Safety Assessment and

Management 96 (ESREL '96 -- PSAM-Ill), P. C. Cacciabue and I. A.

Papazoglou, ed., Springer, London, pp. 1283-1288, 1996. 

[ACRS, 2003a] Advisory Committee on Reactor Safeguards, Letter from M. Bonaca, ACRS

Chairman, to Chairman Diaz, NRC, "Proposed Resolution of Public Comments

on Draft Regulatory Guide (DG)-1122, ‘An Approach for Determining the

Technical Adequacy of Probabilistic Risk Results for Risk-Informed Decision

Making,'" Washington, DC, April 21, 2003.

[ACRS, 2003b] Advisory Committee on Reactor Safeguards, Letter from M. Bonaca, ACRS

Chairman, to Chairman Diaz, NRC, "Improvement of the Quality of Risk

Information for Regulatory Decision Making," Washington, DC, May 16, 2003.

[ANS, 2007a] American Nuclear Society, “External events PRA methodology” standards,

ANS-58.21-2007, La Grange Park, IL, March 2007. 

[ASME, 2005a] American Society of Mechanical Engineers, "Standard for Probabilistic Risk

Assessment for Nuclear Power Plant Applications," ASME RA-Sb-2005, New

York, New York, December 30, 2005.

[Cheok, 1998a] M. C. Cheok, G.W. Parry, and R. R. Sherry, "Use of Importance Measures in

Risk-Informed Regulatory Applications," Reliability Engineering and System

Safety, Vol. 60, pp. 213-226, 1998.

[EG&G, 1993a] EG&G Idaho, Inc., "Systems Analysis Programs for Hands-on Integrated

Reliability Evaluations (SAPHIRE) Version 5.0 - Technical Reference Manual,"

NUREG/CR-6116, Vol. 1, ID, December 1993. 

[EPRI, 1985a] Electric Power Research Institute, “Classification and Analysis of Reactor

Operating Experience Involving Dependent Events,” EPRI NP-3967, Palo Alto,

CA, June 1985.

[EPRI, 1988a] Electric Power Research Institute, “Procedures for Treating Common Cause

Failures in Safety and Reliability Studies,” NUREG/CR-4780, EPRI NP-5613,

PLG-0547, Vol. 1, Palo Alto, CA, January 1988.

[EPRI, 2003a] Electric Power Research Institute, “Parametric Uncertainty Impacts on Option

2 Safety Significance Categorization,” EPRI TR 1008905, Palo Alto, CA, June

2003.
8. References

17The laboratory’s name has since been changed to Idaho National Laboratory (INL). 

NUREG-1855 8-2 Draft for Comment

[EPRI, 2004a] Electric Power Research Institute, "Guideline for the Treatment of Uncertainty

in Risk-Informed Applications: Technical Basis Document," EPRI TR 1009652,

Palo Alto, CA, December 2004.

[EPRI, 2005a] Electric Power Research Institute, “Guidelines For The Treatment of Uncertainty

in Risk-Informed Regulatory Applications,” Draft Report, Palo Alto, CA, October

2005.

[EPRI, 2005b] Electric Power Research Institute,"EPRI/NRC-RES Fire PRA Methodology for

Nuclear Power Facilities" NUREG/CR-6850.

[EPRI, 2006a] Electric Power Research Institute, "Guideline for the Treatment of Uncertainty

in Risk-Informed Applications: Applications Guide," EPRI 1013491, Palo Alto,

CA, 2006.

[INL17, 1998a] Idaho National Engineering and Environmental Laboratory, “Common-Cause

Parameter Estimations,” NUREG/CR-5497, INEEL/EXT-97-01328, Idaho Falls,

ID, October 1998.

[INL, 1998b] Idaho National Engineering and Environmental Laboratory, “Common-Cause

Failure Database and Analysis System: Overview,” NUREG/CR-6268,

INEEL/EXT-97-00696, Idaho Falls, ID, June 1998.

[INL, 1998c] Idaho National Engineering and Environmental Laboratory, “Guidelines on

Modeling Common-Cause Failures in Probabilistic Risk Assessments,”

NUREG/CR-5485, INEEL/EXT-97-01327, Idaho Falls, ID, November 1998.

[LANL, 1983a] Los Alamos National Laboratory, “A Comparison of Methods for Uncertainty

Analysis of Nuclear Power Plant Safety System Fault Tree Models,”

NUREG/CR-3263, Los Alamos, NM, April 1983. 

[Martz, 1995a] Martz, H. F., and Picard, R. R., “Uncertainty in Poisson Event Counts and

Exposure Time in Rate Estimation,” in Reliability Engineering and System

Safety, Vol. 48, pp. 181-190, 1995.

[Mosleh, 1986a] Mosleh, A., “Hidden Sources of Uncertainty: Judgment in the Collection and

Analysis of Data,” in Nuclear Engineering and Design, Vol. 93, pp. 187-198,

1986.

[NARA, 2007a] National Archives and Records Administration, The Code of Federal

Regulations, Title 10–Energy, Washington, DC, January 2007.

[NEI, 2005a] “Process for Performing Follow-on PRA Peer Reviews Using the ASME

PRA Standard,” NEI-05-04, Washington, DC, January 2005.
8. References

Draft for Comment 8-3 NUREG-1855

[NEI, 2005b] Nuclear Energy Institute, "10 CFR 50.69 SSC Categorization Guideline",

NEI-00-04, ADAMS Accession #ML052910035, Washington, DC, July 2005.

[NEI, 2006a] Nuclear Energy Institute,"Probabilistic Risk Assessment (PRA) Peer Review

Process Guidance," Update to Revision 1, NEI-00-02, Washington, DC, October

2006.

[NRC, 1975a] U.S. Nuclear Regulatory Commission, "Reactor Safety Study: An Assessment

of Accident Risks in U.S. Commercial Nuclear Power Plants," WASH-1400,

NUREG-75/014, Washington, DC, October 1975.

[NRC, 1975b] U.S. Nuclear Regulatory Commission, “Standard Review Plan for the Review of

Safety Analysis Reports for Nuclear Power Plants,” NUREG-0800, Washington,

DC, November 1975. 

[NRC, 1986] U.S. Nuclear Regulatory Commission, "Safety Goals for the Operations of

Nuclear Power Plants: Policy Statement," Federal Register, Vol 51, p. 30028 (51

FR 30028), Washington, DC, August 4, 1986.

[NRC, 1995a] U.S. Nuclear Regulatory Commission, “Final Policy Statement ‘Use of

Probabilistic Risk Assessment (PRA) Methods in Nuclear Regulatory Activities’,"

Washington, DC, 1995.

[NRC, 1997a] U.S. Nuclear Regulatory Commission, "Acceptance Guidelines and Consensus

Standards for Use in Risk-Informed Regulation," SECY-97-221, Washington,

DC, September 30, 1997.

[NRC, 1997b] U.S. Nuclear Regulatory Commission, "Final Regulatory Guidance on

Risk-Informed Regulation: Policy Issues," SECY-97-287, Washington, DC,

December 12, 1997.

[NRC, 1991a] U.S. Nuclear Regulatory Commission, NUREG-1407, Procedural and Submittal

Guidance for the Individual Plant Examination of External Events (IPEEE) for

Severe Accident Vulnerabilities,” Washington, DC, June 1991.

[NRC, 1998a] U.S. Nuclear Regulatory Commission, “An Approach for Plant-Specific,

Risk-Informed Decision Making: Inservice Testing,” RG 1.175, Washington, DC,

August 1998.

[NRC, 1998c] U.S. Nuclear Regulatory Commission, “An Approach for Plant-Specific,

Risk-Informed Decision Making: Technical Specifications,” RG 1.177,

Washington, DC, August 1998.

[NRC, 1999a] U.S. Nuclear Regulatory Commission, “Commission Issuance of White Paper

on Risk-informed and Performance-based Regulation,” Yellow Announcement

# 019, Washington, DC, dated March 11, 1999.
8. References

NUREG-1855 8-4 Draft for Comment

[NRC, 2001a] U.S. Nuclear Regulatory Commission,“NRC Incident Investigation Program,”

Management Directive 8.3, Washington, DC, March 2001.

[NRC, 2002a] U.S. Nuclear Regulatory Commission, "An Approach for Using Probabilistic Risk

Assessment in Risk-Informed Decisions on Plant-Specific Changes to the

Licensing Basis," Regulatory Guide 1.174, Washington, DC, November 2002.

[NRC, 2003a] U.S. Nuclear Regulatory Commission, Letter from William Travers, Executive

Director for Operations, to M. Bonaca, ACRS Chairman, "Proposed Resolution

of Public Comments on Draft Regulatory Guide (DG)-1122, ‘An Approach for

Determining the Technical Adequacy of Probabilistic Risk Results for

Risk-Informed Decision Making'," Washington, DC, June 4, 2003.

[NRC, 2003b] U.S. Nuclear Regulatory Commission, Letter from William Travers, Executive

Director for Operations, to M. Bonaca, ACRS Chairman, "Improvement of the

Quality of Risk Information for Regulatory Decision Making," Washington, DC,

August 4, 2003.

[NRC, 2003c] U.S. Nuclear Regulatory Commission, “An Approach for Plant-Specific

Risk-Informed Decision Making for Inservice Inspection of Piping,” RG 1.178,

Washington, DC, September 2003. 

[NRC, 2003d] U.S. Nuclear Regulatory Commission, "Safety Evaluation of Topical Report

WCAP-15603, Revision 1, ‘WOG 2000 Reactor Coolant Pump Seal Leakage

Model for Westinghouse PWRs," ADAMS Accession No. ML031400376,

Washington, DC, May 30, 2003.

[NRC, 2003e] U.S. Nuclear Regulatory Commission, ”PRA Quality Expectations and

Requirements,” Staff Requirements Memorandum COMNJD-03-0002,

Washington DC, December 18, 2003.

[NRC, 2004a] U.S. Nuclear Regulatory Commission, "Regulatory Analysis Guidelines of the

U.S. Nuclear Regulatory Commission (rev. 4)", NUREG/BR-0058, Washington,

DC, September 2004.

[NRC, 2005a] U.S. Nuclear Regulatory Commission, “Good Practices for Implementing Human

Reliability Analysis,” NUREG-1792, Washington, DC, April 2005. 

[NRC, 2006a] U.S. Nuclear Regulatory Commission,“Guidelines for Categorizing Structures,

Systems, and Components in Nuclear Power Plants According to Their Safety

Significance,” RG 1.201, Washington, DC, May 2006.

[NRC, 2006b] U.S. Nuclear Regulatory Commission, “Risk-Informed, Performance-Based Fire

Protection for Existing Light-Water Nuclear Power Plants,” RG 1.205,

Washington, DC, May 2006.
8. References

Draft for Comment 8-5 NUREG-1855

[NRC, 2007a] U.S. Nuclear Regulatory Commission, "An Approach for Determining the

Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed

Activities," Regulatory Guide 1.200, Revision 1, Washington, DC, January 2007.

[NRC, 2007b] U.S. Nuclear Regulatory Commission, “Combined License Applications for

Nuclear Power Plants (LWR Edition)” RG 1.206, Washington, DC, June 2007.

[NRC, 2007c] U.S. Nuclear Regulatory Commission, “Standard Review Plan, Section 19.0,

‘Probabilistic Risk Assessment and Severe Accident Evaluation for New

Reactors,’” Revision 2, Washington, DC, June 2007.

[NRC, 2007d] U.S. Nuclear Regulatory Commission, “Update on the Improvements to the

Risk-informed Regulation Implementation Plan,” SECY-07-0074, Washington,

DC, April 26, 2007.

[Parry, 1986a] Parry, G. W., "A Discussion on the Use of Judgement in Representing

Uncertainty in PRAs", Nuclear Engineering and Design, Vol. 93, pp. 135-144,

1986.

[Reinert, 2006a] Reinert, J. M., and Apostolakis, G. E., “Including Model Uncertainty in

Risk-informed Decision Making,” Annals of Nuclear Energy, Vol. 33, pp.

354-369, January 19, 2006.

[Siu, 1984a] Siu, N., and Apostolakis, G., “Models for the Detection and Suppression of Fires

in Nuclear Power Plants and Some Related Statistical Problems,” UCLA-ENG8440, 1984. 

[Siu, 1986a] Siu, N., and Apostolakis, G., “A Methodology for Analyzing the Detection and

Suppression of Fires in Nuclear Power Plants,” Nuclear Science and

Engineering, Vol. 94, pp. 213-226, 1986.

[Siu, 1998a] Siu, N., and Kelly, D., “Bayesian Parameter Estimation in Probabilistic Risk

Assessment,” Reliability Engineering and System Safety, Vol. 62, pp. 89-116,

October 1998.

[SNL, 1984a] Sandia National Laboratories, "A FORTRAN 77 Program and User's Guide for

the Generation of Latin Hypercube and Random Samples for Use with Computer

Models," SAND83-2365, NUREG/CR-3624, Albuquerque, NM, March 1984. 

[SNL, 1988a] Sandia National Laboratories, "Approaches to Uncertainty Analysis in

Probabilistic Risk Assessment," SAND87-0871, NUREG/CR-4836, Albuquerque,

NM, January 1988.

[SNL, 1992a] Sandia National Laboratories, “Analysis of the LaSalle Unit 2 Nuclear Power

Plant: Risk Methods Integration and Evaluation Program (RMIEP),”

SAND92-0537, NUREG/CR-4832, Albuquerque, NM, October 1992.
8. References

NUREG-1855 8-6 Draft for Comment

[SNL, 1992b] Sandia National Laboratories, “Methods for External Event Screening

Quantification: Risk Methods Integration Evaluation Program (RMIEP)

Methods Development,” SAND87-7156, NUREG/CR-4839, Albuquerque, NM,

July 1992.

[SNL, 2003a] Sandia National Laboratories, "Handbook of Parameter Estimation for

Probabilistic Risk Assessment," SAND2003-3348P, NUREG/CR-6823,

Albuquerque, NM, September 2003.
Draft for Comment A-1 NUREG-1855

APPENDIX A

STAFF POSITION ON ELECTRIC POWER RESEARCH INSTITUTE,

"GUIDELINE FOR THE TREATMENT OF UNCERTAINTY IN

RISK-INFORMED APPLICATIONS”

The Electric Power Research Institute (EPRI) has published draft versions of a "Guideline for the

Treatment of Uncertainty in Risk-Informed Applications” which includes a Technical Basis

Document and an Applications Guide. The staff position on the final publication of these

documents will be provided in this appendix.