NUREG-1563

Branch Technical Position on the

Use of Expert Elicitation in the

High-Level Radioactive

Waste Program

Manuscript Completed: August 1996

Date Published: November 1996

J. P. Kotra, M. P. Lee, N. A. Eisenberg, NRC

A. R. De~ispelare, CNWRA

Center for Nuclear Waste Regulatory Analyses

San Antonio, IX 78228-0510

Division of Waste Management

Office of Nuclear Material Safety and Safeguards

U.S. Nuclear Regulatory Commission

Washington, DC 20555-0001
AlESTRACT

The U.S. Department of Energy (DOE) is

conducting a program of site characterization to

gather enough information, about the Yucca

Mountain (Nevada) site, to be able to evaluate the

waste isolation capabilities of a potential geologic

repository. Should the site be found suitable,

DOE will apply to the U.S. Nuclear Regulatory

Commission for permission to construct and then

operate a proposed geologic repository for the

disposal of spent nuclear fuel and other high-level

radioactive waste at Yucca Mountain. In deciding

whether to grant or deny DOE's license

application for a geologic repository, NRC will

closely examine the facts and expert judgment set

forth in any potential DOE license application.

NRC expects that subjective judgments of

individual experts and, in some cases, groups of

experts, will be used by DOE to interpret data

obtained during site characterization and to

address the many technical issues and inherent

uncertainties associated with predicting the

performance of a repository system for thousands

of years. NRC has traditionally accepted, for

review, expert judgment to evaluate and interpret

the factual bases of license applications and is

expected to give appropriate consideration to the

judgments of DOE's experts regarding the

geologic repository. Such consideration, however,

envisions DOE using expert judgments to

complement and supplement other sources of

scientific and technical information, such as data

collection, analyses, and experimentation. In this

document, the NRC staff has set forth technical

positions that: (1) provide general guidelines on

those circumstances that may warrant the use of a

formal process for obtaining the judgments of

more than one expert (i.e., expert elicitation); and

(2) describe acceptable procedures for conducting

expert elicitation when formally elicited judgments

are used to support a demonstration of

compliance with NRC's geologic disposal

regulation, currently set forth in 10 CFR Part 60.

In this NUREG, the staff also provides an

expanded definition of "peer review" over that

provided earlier in NUJREG-1297.

iii NUREG-1563
CONTENTS

Page

ABSTRACT ...................................................................... iii

FOREWORD............................................................................. vii

ACKNOWLEDGMENTS .................................................................. ix

1 INTRODUCTION ...................................................................... 1

1.1 Background ........................................................................ 2

1.2 What are Expert Judgment, Expert Elicitation, and Peer Review? ........................ 3

1.2.1 Expert Judgment . ............................................................. 3

1.2.2 Expert Elicitation . ............................................................. 3

1.2.3 Peer Review .................................................................. 5

1.3 Selected Examples of NRC Use of Expert Elicitation ............................... .5 

13.1 Severe Accident Risk Analysis ................................................. 5

1.3.2 Seismic Hazard Analysis ................. ..................................... 6

1.3.3 HLW 'Tchnical Analyses ................. ..................................... 7

1.4 Purpose of the BTP . ................................................................ 7

1.5 BTPs as Technical Guidance ..................... .................................... 9

2 REGULATORY FRAMEWORK ......................................................... 11

2.1 10 CFR Part 60 ..................................................................... 11

2.2 Commission Policy Statement Concerning the Use of PRA Methods in Nuclear

Regulatory Activities . ................................................................ 13

3 BRANCH TECHNICAL POSITIONS .................................................... 15

4 DISCUSSION ...................................................................... 19

5 REFERENCES ........................................................................ 31

FIGURE

1 Components in An Acceptable Expert Elicitation Process .................................. 16

v NUREG-1563
FOREWORD

This Branch Technical Position (BTP) was

developed only for application to the high-level

waste program, as part of the staff's Iterative

Performance Assessment efforts. The technical

positions contained here reflect staff experience

gained from both monitoring the U.S. Department

of Energy's site characterization program at

Yucca Mountain, Nevada, and developing the U.S.

Nuclear Regulatory Commission's independent

regulatory capability.

Although there are several examples of the use of

expert elicitation in a nuclear regulatory context,

no formal Agency guidance on this subject exists.

Thus, in developing this BT, the Division of

Waste Management (DWM) staff has also drawn

from previous staff experience of other NRC

program offices, in the use of expert elicitation. In

this regard, DWM staff has relied on certain

Agency resource documents, such as: "Risk

Assessment: A Survey of Characteristics,

Applications, and Methods Used by Federal

Agencies for Engineered Systems"; 'A Review of

NRC Staff Uses of Probabilistic Risk Assessment"; and "Recommendations for Probabilistic

Seismic Hazard Analysis: Guidance on

Uncertainty and Use of Experts," to help

formulate its position statements. Consequently,

the reader will find that this BTP is largely consistent with these other resource documents, in

substance.

Subsequent to the finalization of this BTP, the

staff may elect to develop guidance on the use of

expert judgment in other areas of nuclear

regulatory regulation.

vii NUREG-1563
ACKNOWLEDGMENTS

The authors gratefully acknowledge substantial

contributions from the staffs of the Center for

Nuclear Waste Regulatory Analyses (CNWRA),

the Division of Waste Management (DWM), the

Office of the General Counsel, the Office of

Nuclear Reactor Regulation (NRR), the Office of

Nuclear Regulatory Research (RES), and the

Office for Analysis and Evaluation of Operational

Data (AEOD), which have been incorporated into

the text of this Branch Technical Position. In this'

regard, the following staff deserve special

recognition for their comments and/or reviews:

Robert G. Baca/CNWRA

Joseph A. Murphy/RES

EJ. Bonano/Beta Corporation International

John D. Randall/RES

Michael C. Cheok/NRR

Charles E. Rossi/AEOD

Anthony H. Hsia/NRR

Phyllis A. Sobel/DWM

AK. IbrahimlDWM

John S. Trapp/DWM

Christiana H. Lui/DWM

Sandra L. Wastler/DWM

Steven E. Mayi/AEOD

-Michael F. Veber/DWM

Timothy J. McCartin/DWM

Ernst G. Zurflueh/RES

Keith I. McConnell/DWM

Thanks also to George A. Apostolakis, Department of Nuclear Enginebring/Massachusetts

Institute of Technology, under a grant from RES,

for his review comments. Ellen Kraus provided

editorial guidance.

L5; NUREG-1563
APPENDICES

Page

A GLOSSARY ........................................ A-1

B AN ELICITATION ON FUrtURE CLIMATE: LESSONS LEARNED .B-1

C FINAL COMMISSION POLICY STATEMENT ON THE USE OF PRA

METHODS IN NUCLEAR REGULATORY ACTIVITIES .C-1

D DISPOSITION OF PUBLIC COMMENTS ON FEBRUARY 28, 1996,

DRAFT BRANCH TECHNICAL POSITION ......................................... D-1

E STAFF VIEWS ON DOE'S 1995 PRINCIPLES AND GUIDELINES AND THE

POSSIBLE DISPOSITION OF SCA COMMENT 3 ................. E-1

F DISPOSITION OF ACNW` COMMENTS ................. F-1

NUREG-1563 x
1 INTRODUCTION -:

The U.S.'Nucleai Regulatory Commission's

regulations for the licensing of a geologic

repository for the disposal of spent nuclear fuel

and other high-level radioactive waste (HLWV) are

intentionally non-prescriptive in that 10 CFR Part

60 leaves to the U.S. Department of Energy.

(DOE) the opportunity and responsibility to

determine how best to site and design a potential

geologic repository that can meet the performance

objectives contained in Subpart E of those

regulations. DOE is conducting a program of site

characterization, at Yucca Mountain, Nevada.'

Through this program, DOE is gathering

information on the characteristics of the Yucca

Mountain site, to evaluate the waste isolation - capabilities of the proposed site, as it would

perform in concert with DOE's repository design, in order to meet the performance objectives.

DOE will need to interpret the geologic record

and demonstrate that the repository site and

design will comply with explicit numerical

performance standards established by the U.S.

Environmental Protection Agency (EPA) and

adopted by NRC.1

For the Commission's part, if it is to reach an

affirmative licensing decision, the Commission

will need to confirm that the numerical

performance standards have been met and must

.satisfy itself that DOE's analyses of the'site and

design are sufficiently convincing, that'their.

limitations are well understood, and that DOE

has demonstrated that its analyses have made

appropriate allowance for the time period,

hazards;and uncertainties involved.

Nearly every aspect of site characterization and

performance assessment will involve'significant

uncertainties. The primary method to evaluate,'

and perhaps reduce, these uncertainties' should be

collection of sufficient data and information

during site characterization. However, factors

Ibe staff recognizes that revised EPA standards, specific to the

Yucca Mountain site, required by the Energy Policy Act of 1992

(Public Law 102486) must be based on and consistent with recent

findings and recommendations of the National Academy of Sciences

(NAS-see National Research Council, 1995a) Once EPA promulgates regulations establishing its final standards, NRC will modify its

requirements at 10 CFR Part 60 to conform to the new standards.

Notwithstanding these forthcoming revisions to both EPA's;

standards and NRC's conforming regulations, the staff presumes that there will continue to be a requirement for some kind of a quantitative performance assessment to estimate the long-term,

pet5-closure performance of the overall repository system.,

such as temporal and spatial variations in the

data, the possibility for multiple interpretations of

the same data; and the absence of validated

theories for predicting the performance of a.

repository for thousands 'of years, will make it

necessary to complement and supplement the

data obtained during site characterization with

the interpretations and subjective judgments of

technical experts (i.e., expert judgments-see

definition in Section 1.2.1).

Expert judgment is ubiquitous in almost every

scientific or technical endeavor. As important as

its role may be, however, for the purposes'of the

HLW program, the subjective judgment of experts

should be distinguished from both measured data

or technical 'calculations based on'accepted scientific laws and principles. It should be viewed

as an alternative, and employed when other means

of obtaining requisite data or information have

beei thoroughly considered and it has been

concluded that such means are not practical to

implement. Thus, expert judgment, informal as

well as formally elicited, may be used by DOE in

its 'demonstrations of compliance with NRC's

geologic disposal regulation. Moreover, the NRC

staff will continue to accept, for review, those

compliance demonstrations and other analyses

employing inf6rmal expert judgment and expert elicitation.

With this notion in mind, current NRC policy is to

encourage the use of probabilistic risk assessment

(PRA) state-of-the-art'technology and methods as

a compleiment to the deterministic approach in

nuclear regulatory activities (NRC, 1995; 60 FR

42622)2 and in keeping with the Commission's

paramount responsibility to protect public health

and safety. Expert judgment may provide an

essential part of the information used'in PRAs (or

performance assessments,- in the case of waste.

management' systems),'and may also be used in

deterministic analyses. Consistent with the

Commission's' policy, the NRC staff has developed

this' Branch Technical Position (BTP) to identify

acceptable procedures for the formal elicitation of

2EPA's 1985 HLW standards (50 FR 38066) adopted a probabilistic perspective when making compliance determinations. Because of

the uncertainties inherent in the geologic disposal of nuclear waste,

it is anticipated that a probabilistic treatment of the performance of

the waste disposal system will continue to be the regulatory

approach.

1 1 ~~~~~NUREG-15
such judgments in the area of HLW The staff

believes that PRA methods, such as described in

this guidance, can be applied to the HLW

program in a manner that would contribute

significantly to the necessary confidence that a

geologic repository could be licensed without

undue risk to the health and safety of the public.

1.1 Background

A geologic repository is a complex system, the

future performance of which must be predicted

over many thousands of years. Because of the

nature of the task and the limitations of scientific

understanding in many pertinerit technical fields,

DOE (and its contractors) will use expert judgment

throughout site characterization ad later, in the

preparation of a license application, to construct a

potential geologic repository for HLW. DOE and

its contractors will also use expert judgment in the

identification and screening of events and

scenarios; development and selection of models

that describe the geology and hydrology of the

repository system; assessment of model

parameters; collection of data; assessment of

volcanic and seismic hazard potential; and for

strategic decision-making, about the repository's

design, that could affect its long-term

performance (e.g., DOE, 1986; Dennis, 1991;

Seismic Methods Peer Review Panel,- 1991; Test

Prioritization Task Force, 1991; DOE, 1991b;

Barnard et al., 1992; Andrews et al.; 1994, Wilson

et al., 1994; and Schenker et al., 1995).

Although acquisition and analysis of physical data

should be the primary manner in which licensing information is collected, many considerations may

preclude the collection of such information

necessary for licensing. As with all complex technical analyses, "professional" judgment,

usually informal and implicit, is used routinely by

the scientists, engineers, and technical program

managers who contribute to the repository

program, to supplement and interpret this

information, indeed, even to determine how to

obtain the data or perform the analyses. The staff

believes that its ability to evaluate a potential

SLW license application will, in large measure,

depend on the transparency with which data are

collected, analyzed, and interpreted, and

safety-related decisions are made. Therefore, the

staff believes that it is important for all program

participants to have a common understanding of

the general circumstances under which it may be

worthwhile to obtain and apply expert judgments in a more formal manner, and of the appropriate methods for doing so.

In reviewing DOE's Site Characterization Plan

(SCP-see DOE, 1988), the staff identified

concerns related to DOE's proposed use of expert judgment as part of a potential license

application. In its Site Characterization Analysis (SCA), the staff noted that DOE's SCP relied too

heavily on the elicitation of expert judgment as a

substitute for quantitative data and analyses (NRC, 1989; pp. 4-8-4-10). Subsequently, the

NRC staff has criticized specific DOE uses of

expert elicitation in the Calico Hills Risk/Benefit

Analysis (DOE, 1991a) and the Early Site

Suitability Study (Science Applications International Corporation, 1992). The NRC staff

found fault with both the manner in which these

elicitations were conducted and the way in which

the elicitation results were used to make site

characterization decisions (see Linehan, 1990 and

1991; and Holonich, 1992). As with other types of

information, expert judgment can be misinterpreted, misrepresented, and misused. The

NRC staff is particularly concerned with the

potential for over-reliance on expert judgment as

a basis for decision-making, as well as its

potential misuse as an inappropriate justification

to avoid gathering additional objective data.

Since 1990, the Nuclear Waste Technical Review

Board (NWTRB) and the NAS have also

addressed DOE's plans to use expert judgment. Both the NWIRB and the NAS, independently, have expressed concerns with these plans and, in

particular, with how DOE addresses the potential for "bias" and "conflicts of interest" when

conducting expert eicitations (see NWTRB

(1990a, p. 21; 1990b, p. 26; 1991, pp. 29-30; 1994,

pp. 31-35) and National Research Council (1990,

p. 24)).3 For example, DOE has been criticized for

relying almost entirely on its own scientific experts

and contractors, with little or no external peer

review.

More recently, DOE has announced a series of

initiatives that would lead to the restructuring of

its geologic repository program, including site

characterization activities (see DOE, 1994 (pp.

1-3-1-4) and 1995 (pp. 1-3-1-8)). The exact

3DOE's response to some of the Board's recommendations are reprintedinNTRB (1992 (pp. E-11-E-12) and 1995 (pp. H-20-H1-2 1)).

NUREG-1563 2
details of the re-structuring plan have not yet been

disclosed, nor have the impacts of recent budget

constraints imposed on DOE's repository.

program been fully evaluated in the 'context of this

plan. That being said, the staff generally ' '

understands that DOE will, both by intent and

necessity,'c6llect fewer data and, instead, rely

extensively on bounding analyses to both form its

HLW programmatic decisions and to support a

potential license application it would submit

pursuant to NRC's geologic disposal regulation

(see DOE, 1994; pp. 1-3-14).

1.2 What are Expert Judgment, Expert

Elicitation, and Peer Review?

1.2.1 Expert Judgment

Expert judgment is infornation, 4 provided by a

technical expert, in his or her subject matter area

of expertise, based on opinion, or on a belief

based on reasoning. Questions are usually posed

to experts because they cannot be'answered by

other means. Expert judgments can be evaluations

of theories, models, or experiments, or theyjcan be

recommendations for further research. Expert'

judgments may also be opinions that can be"

analyzed and interpreted, and used in subsequent

technical assessments. Expert judgments can be

either qualitative or quantitative. Expert

judgments can also be judgments about uncertain

quantities or judgments about value preferences.

Expert judgment has also been called expert

opinion, subjective judgment, expert forecast, best

estinate, educated guess, and, most recently expert . knowledge (see Meyer and Booker, 1990; p. 3).' Regardless of how one defines it, expert judgment

ultimately reflects the technical expert's evaluation

and interpretation of some scientific knowledge

' base, to the extent that the knowledge' base exists.

Moreover, expert judgment does-not create

knowledge, rather it "synthesizes disparate and

often conflicting sources of information to

produce an integrated picture" (see Hora,' 1993).

The distinction between judgmental information

(e.g., is there life on other planets ?) and more

straightforward, factual information (e.g., what is

£ dr/x ?) is not sharp. The use of expert judgment

in technical and scientific work, including that

4 Expert judgment is sometimes referred to as 'data" (e.g., for purposes of aggregating the judgments of multiple experts).

routinely reviewed by the NRC staff, is

ubiquitous.' This'infonnal use of expert judgment

is implicit in the choice of mathematical

equations to describe a system, the methods used

for testing, the interpretation of data, the methods

'used to obtain numerical results, and other

aspects of analysis and testing.

Judicial and administrative proceedings frequently

involve "expert witnesses" who, by virtue of their

training and/or experience, are judged capable of

providing useful opinions or conclusions about

certain matters in issue. These experts perform

their roles under the rules governing the particular

proceeding, whether it be a judicial or an

administrative setting. However, in this BTP, the

word "expert" will be used more broadly and with

reference to the "pre-licensing" and "licensing

phases" (Johnson, 1994) of the HLW program. In

the context of this BT, "experts" refer to those

knowledgeable individuals, in engineering and

science, who, by the nature'6f their experience

and academic achievement,'can' speak to the

understanding of certain scientific laws and.

principles. This BTP makes no attempt to identify

who would be an "expert"' for.the purposes of a

judicial or an administrative proceeding.

However, an "expert" used in an' elicitation

process may also be used as an "expert" in a

judicial proceeding.

1.2.2 Expert Elicitation

1.. .2 .x

Expert elicitation is a formal, highly structured,

and well-documented process whereby expert

judgments,'usuallyof multiple experts, are

obtained. Although informal expert judgment

involves 'only subject-matter experts, formal expert

elicitations usually involve normative experts,

generalists, and subject-matter experts (see -

Appendix A). The normative expert has training

and experience in statistics, decision analysis, and

probability encoding; this expert's main function

is to structure the formal elicitation and train the

subject-matter experts in probability encoding.

The generalist understands the context in which

the results of the expert elicitation will be used,-

guides the structure of the elicitation to produce

the needed results, provides relevant information

and 'documentation to the subject-matter experts,

' and helps to train them. Often the generalist's

expertise overlaps that of the subject-matter

experts or is in a closely-related, allied field. For

example, a performance assessment expert could

3 NUREG-1563
be the generalist in an expert elicitation for

climate change. The subject-matter experts, of

course, provide the subjective judgments that are

usually encoded as probabilities (e.g., a

probability distribution or a point estimate for an

uncertain parameter).

In the context of the HLW program, expert

elicitation may play several roles in DOE's

decision-making process for a potential geologic

repository. Typically an elicitation is conducted to

evaluate uncertainty. The uncertainty could be

associated with: the value of a parameter to be

used in a model; the likelihood and frequency of

various future events; or the relative merits of

alternative conceptual models. In each of these

cases, the information regarding uncertainty would be represented by encoding the subjective probabilities from each subject-matter expert. For

example, a few boreholes may yield data necessary

to calculate the permeability of a rock type in a

particular hydrogeologic unit. Given that these

data represent a small sample from a highly

variable population and given further that

undiscovered structures and features could greatly

influence the effective permeability of the unit,

each expert might be asked to generate a

probability distribution for the effective

permeability. of the hydrogeologic unit. Similarly,

probabilities may be attached to the likelihood of

various futures, frequencies of various events, and

validity of various models. These probability

distributions could be used as direct input to

probabilistic performance assessments.

Alternatively,. discrete distributions describing alternative conceptual models could be used to

characterize the outcomes of analyses using each

of the alternative concepts. In either event, the

outputs of these performance assessments are

expected to provide important inputs to DOE's

ongoing site characterization and design

processes, as well as to indicate the relative

importance of various programmatic activities for

performance assessment. In addition, these

"iterative" assessments are expected to provide

important feedback on the nature and importance

of improvements to be made in the analytical

basis for the performance assessment. Ultimately the performance assessment, supported in part by

expert elicitation, is expected to be the central

focus of any potential DOE license application,

wherein the various lines of evidence (field data,

laboratory experiments, natural analogues,

theoretical and semi-empirical analyses) are

drawn together in an effort to demonstrate

compliance. Because the NRC staff has an

interest and oversight role5 in these various

aspects of performance assessment and the

associated expert elicitation(s), the staff has

developed this guidance document.

Finally, it should be noted that NRC evaluation of

any potential DOE license application would

involve an opportunity for a hearing. If a hearing is requested and if the Commission appoints an

Atomic Safety and Licensing Board to preside at

the hearing, then that Licensing Board will be

responsible for conducting a hearing at which

evidence is taken on contested issues. The

Licensing Board has considerable liberty in the

selection of those values or judgments it would

use to reach any potential licensing decision.

Although the Board's decision can be based only

on evidence in the record, it can exercise wide

discretion in the weight it attributes to any particular piece of evidence. Thus, for any particular issue or case, the staff cannot predict what weight, if any, the Licensing Board would

attribute to expert opinion derived from an expert elicitation, as opposed to expert opinion obtained

by other means.

5

1n addition to the review of the site characterization activities

specified under 10 CFR 60.18, the Commission noted in its final

rule that it contemplated an ongoing review of information on site

investigation and site characterization, such as those with long lead-time procurement actions, so as to allow for the early identification and resolution of potential licensing issues. Moreover, NRC's

strategic planning assumptions call for the early identification and

resolution, to the extent practicable, at the staff level, before the

receipt of a potential license application for a geologic repository (see Johnson, 1994). The principal means for achieving this goal is

through informal, pre-licensing consultation with DOE the State of

Nevada, Indian Thbes, and affected units of local government. This

approach attempts to reduce the number of, and to better define, issues that will be litigated during a potential licensing hearing, by obtaining input and striving for consensus from the technical

community, interested parties, or other targeted groups on such

issues.

In this regard, the staff has undertaken the development of this

BTP as a means for closure on acceptable procedures for con- ducting expert elicitation when formally elicited judgments are used

to support a demonstration of compliance with NRC's geologic disposal regulations The staff believes that rigid adherence to the

specific steps proposed in the IP is not sought so much as the use of a consistent process that produces an accurate and properly documented assessment of the state of scientific uncertainty. Moreover, the staff believes that effective implementation of a good

elicitation process cannot guarantee acceptance of the technical

conclusions; however, use of a flawed process or improper imple- mentation of a good process cannot help but cast serious doubt on the quality of the conclusions.

NUREG-1563 4
1.23 Peer Review6

Much of scientific and engineering development is

subjected to the normal review process of critical

evaluation by colleagues in various venues. These

so-called peer reviews are typically documented,- 

critical reviews that evaluate the acceptability and

adequacy of some particular form of original

research, performed by peers who are :

independent of the work being reviewed. A peer

review can be conducted by obtaining input

separately from a number of peers or by

convening a panel to conduct the review. Also,

discussions among the panel members can

generate useful information not available from a

set of independent reviews. The most common...

peer review process (i.e., pre-publication technical

review of a scientific article) typically uses

informal expert judgment to evaluate scientific

methods and results. However, in principle, the - nature of peer review is sufficiently flexible that its

rigor and formality are commensurate with the

study being reviewed. For example, the NAS is

frequently called on to review reports or

conclusions as a group of technical experts (see

National Research Council, 1995b and 1995c).

Peer reviews can also be conducted using a formal

process' to 'review the solution'of problems of high importance. Formal'peer review ha'some of the

same basic attributes of the formal expert

elicitation process (e.g., disclosure of potential"'

conflicts of panelists, documentation behind

decision-making).'

The peers are recognized experts in the'domain of

interest as evidenced by their scientific

qualifications. The' peers may comment on the

validity of the assumptions, the appropriateness

and limitations 'of the methodology and

procedures; the accuracy of the calculations, the'

validity of the conclusions, and the uncertainty of

-the results and consequences of the work; They

may also offer -alternative explanations of the!

results and comment on the adequacy of the

information and data used to obtain them.

The peer review process requires the expert

judgments of peers. However, it is important to

note that peer review as an expert judgment

process is different from' the formal elicitation of

expert judgments in the context of this staff

6Tbis discussion is an cxpansion of the earlier definition of peer review provided by Altnan et al (1988, p. 2).

position. The reference to expert judgment herein

denotes judgments, opinions, or information

provided by subject-matter experts that give rise

to or contribute to the generation of a scientific

stance or solution to a given problem. Peer review,

by contrast,'seeks judgments from subject-imatter

experts regarding the soundness and quality of an

*existing or proposed scientific stance or solution

to a'problem. In this context, expert judgments'

can be the subject of peer review. The admittedly

subtle differences between the elicitation of expert judgment and independent peer review

notwithstanding, both processes contribute in a

positive way to enhancing quality.

1.3 Selected Examples of NRC Use of

Expert Elicitation

In addition to reviewing DOE's site character-

'ization activities, the NRC staff has drawn on

experience obtained in' other NRC regulatory

programs (see PRA Working Group, 1994) and

has-be'en exploring, independently, the ways in

which expert elicitation' may be applied in the

geologic repository program. Some of the staff's

program activities are summarized below.

13.1 Severe'Accident Risk Analysis

The formal use of expert elicitation in NRC PRAs

was introduced, during the mid-1980s, with the

development of NUREG-1150, an assessment of

severe accident risk at five U.S. nuclear power

plants (NRC, 1990). Earlier, in 1975, NRC had

completed its first study of the probabilities and

consequences of severe reactor accidents at two

commercial nuclear power reactors. This work, for

the first time, used the analytical technique of

PRA for the study of core meltdown accidents

(see NRC, 1975). After completion of these first

PRAs, NRC, industry groups, and the utilities

-initiated programs to improve PRA technology to

measure and enhance nuclear power reactor

safety, and NRC gradually introduced rRAs into

its regulatory process.

In the late 1980s, the staff updated the 1975 PRA

and in doing so, reassessed the 1975 estimates,

using improved PRA techniques; the results of

this reassessment were documented in

NUREG- 1150. One of the major accomplishments of the NUREG-1150 study, which

continues to be one of the most sophisticated'

. 5 ~~~~~NUREG-15
I

applications of PRA performed by the staff to

date, was the adoption of a formal protocol to

elicit expert judgment in areas of the risk studies

where little or no operational data existed. Expert

judgment was needed to supplement and interpret the available data, and to explicitly address the

uncertainties in the analysis itself. The elicitation

process relied on a formal set of procedures that

are described in detail by Gorham-Bergeron et al.

(1986).

This approach was subsequently reviewed and

modified,7 based, in large part, on

recommendations made by Kouts et al. (1987) and

Kastenberg et al. (1988) after peer reviews of the

first draft of NUREG-1150. Based on these

recommendations, the elicitation process for the

final NUREG-1150 report was made more formal

and rigorous by the identification of nine discrete

process steps; these process steps are described in

Appendix A of the final report.

More recently, expert elicitation was applied to

uncertainty assessment for two new probabilistic

accident consequence codes. In a joint effort with

the Commission of European Communities, the

NRC Office of Nuclear Regulatory Research

conducted a formal expert elicitation of 16

international experts to develop a library of

uncertainty distributions for selected consequence

parameters. Distributions of measurable

atmospheric dispersion and deposition

parameters were successfully elicited from 16

international experts involved in the many

phenomenological areas of consequence analysis

(see Harper et al, 1995). For the most part, the

elicitation protocol used by the NRC/Commission

of European Communities followed the same

principles that guided the NUREG-1150 process.

One noteworthy exception, however, was the

expectation in the later study that, should it

become necessary (e.g., to support an independent

peer review), the subject-matter experts should be

willing to be identified with their elicited

probabilities and the rationales for those

probabilities. In NUREG-1150, experts were

permitted complete anonymity if they so desired.

1.3.2 Seismic Hazard Analysis

Expert elicitation is also widely recognized as

integral to probabilistic seismic hazard analyses

(see National Research Council, 1988). In the

mid-1980s, NRC sponsored a major study of

probabilistic seismic hazard in the eastern United

States conducted by Lawrence Livermore

National Laboratory (LLNL-see Bernreuter et

al., 1980-83, 1985, and 1989). This study of seismic

hazard at 69 reactor sites relied heavily on the

judgments of experts whose interpretations of

geophysical, seismological, and geologic data were

individually obtained using a formal elicitation

process. In conjunction with funding the LLNL

study, NRC recommended that the nuclear power

industry perform an independent study to provide

a coordinated utility position on seismic hazard

estimates. The Electric Power Research Institute

(EPRI), with funding from a consortium of

nuclear power utilities, developed its own

elicitation methodology and applied it to hazard

estimates for 56 of the 69 reactor sites examined

by LLNL (see Seismicity Owners Group/EPRI,

1986).

In both the LLNL and EPRI methodologies,

seismic hazard curves were developed for U.S.

commercial nuclear power reactor sites east of the

Rocky Mountains, using expert judgment to

interpret the available data. Both the LLNL and

the EPRI programs were based on the premise that available geologic data were not sufficient to

fully predict seismic phenomena, and they

adopted the use of expert opinion to characterize

the uncertainties in the data. Although both

methodologies used essentially the same sets of

data, the two methodologies produced

significantly different results. The differences in

the approaches and results were subsequently

reviewed and critically evaluated (see Bernreuter

et al., 1987). It was concluded that, in general, the

differing results could be attributed to how

elicited information was aggregated in the

respective elicitation processes (op cit., pp.

254-258). LLNL has subsequently performed a

re-elicitation of the seismicity and ground motion

experts to improve its earlier estimates of

uncertainty in seismicity parameters and ground

motion models (see Sobel, 1994).

Most recently, in a separate effort jointly

sponsored by NRC, DOE, and EPRI, a "Senior

Seismic Hazard Analysis Committee" has

developed methodological guidance on how best

7Gorham-Bergeron et aL (1986) was available in draft form (as

NUREG/ClR-4551 in the Public Document Room) as a complementary report to NUREG-1150, when NUREG-1150 was first

issued in draft form, for public comment, in 1987. The final version

of NUREG/CR-4551 was published as Gorham ct aL (1993).

NUREG-1563 6
to perform a probabilistic seismic hazard analysis,

with special emphasis on the formal process for

eliciting expert opinion. Results of this project

were published in late 1995 (see Senior Seismic

Hazard Analysis Committee (SSHAC), 1995) and

are currently being reviewed by an NAS

committee.

Appendix A ("Geologic Siting and Design Criteria

for Nuclear Power Plants") to 10 CFR Part 100

contains existing NRC siting and design policy;

related to geological and seismological hazards for

commercial nuclear power reactors. In

conjunction with the Standard Review Plan for

'nuclear power reactors and'other applicable

regulatory guides, Appendix A'sets forth a

regulatory fraimework that guides the NRC staff in

its evaluation of the adequacy of an applicant's investigations of geologic phenomena and

proposed desigri parameters for nuclear power

reactors. Also, independent spent fuel storage

installations, monitored retrieval storage systems,

and mine-tailings'dams for uranium. processing'

mills refer to Appendix A for guidance on faulting

and seismic criteria. Recently, the Commission

proposed revisions to the requirements and

application of Appendix A'to 10 CFR Part 100'

(see NRC, 1994; 59 FR 52255). Under review as

part of this reassessment are recommendations

that NRC's geological and seismological investigations and design criteria be modified to

reflect better the current state of the art in these

areas. Although probabilistic seismic hazard

assessments have been discussed previously in

Safety Evaluation Reports for nuclear power

reactors, these revised requirements, as proposed, would explicitly recognize the use of probabilistic

techniques, as'a valid means to assess uncertainty

associated with the analysis of and'engineering design for seismic phenomena. Specifically, the

Commission has proposed the use of either of the

probabilistic methodologies independently

developed by LLNL'or EPRI as aceptable

approaches to the evaluation of seismic hazard,

uncertainty.' Because both methodologies rely, extensively on the formal elicitation of expert

judgment to assess uncertainty in seismic hazard

estimates, this proposed rule; if promulgated in

final form,' would represent the Commission's first

explicit regulatory recognition of the value of

formal expert elicitation as a technique for

assessing uncertainty.

1.33 HLW Technical Analyses

The process of formally eliciting the judgments'of

multiple experts has also been studied and

applied in support of the staff's independent

performance assessment activities. For example,

Bonano et al. (1990) discussed the state of the art

of formal expert elicitation and its possible'

application to HLW performance assessments.

DeWispelare et al. (1993) subsequently 'aplied the

formal process of elicitation to the prediction of

future climate, with associated parameter distributions, at Yucca Mountain, and to the

estimation of corresponding probabilities'of

occurrence. As part of the NRC staff's

independent Iterative Performance Assessment

efforts to develop a performance assessment

review capability, the staff has relied on informal

elicitations to identify and screen scenarios (see

Codell et al. (1992) and Wescott et al. (1995)). A

structured elicitation process was used by'the' staff to evaluate potential quantitative criteria to

clarify the ".. substantially complete containment requirement" (10 CFR 60.113(a)(1)(i)(A)-

see Tchoepe and Abramson, 1992). Broader

'.applications of expert judgment were exanined by

DeWispelare et al. (1994) who identified situations

where the use of expert judgment might be

appropriate in the HLW program.'Fro nthis

review, the staff was able to verify' that lessons

learned from'these direct experiences had

relevance for broader. applications. In pursuing these reviews of prior Agency experience with

expert elicitation, as well as its own elicitation

activities, the NRC staff has acquired a better

understanding of the critical issues associated

with the development and use of expert elicitation

(and, more generally,-expert judgment).'

1.4 Purpose of the BTP

The NRC staff recognizes that expert judgment is

implicit in all scieritific inquiry and engineering endeavors, and is generally applied in an informal

manner. It is the purpose of this BTP to:

(1) provide general guidelines on those

circumstances that may. warrant the use of a

formal process for obtaining the judgments of

more than one expert (i.e., expert elicitation); and

(2) describe'acceptable procedures for conducting

expert elicitation when formally elicited judgments

are used to support a demonstration of

compliance with NRC's geologic repository

disposal regulations. Included in this BTP is a

7 NUREG-1563
I

recommended procedure for selecting experts,

structuring a formal elicitation, and documenting

the elicitation process.

The NRC staff believes that formal elicitation

procedures, used prudently and appropriately, can

help ensure that expert judgments are wvelldocumented and that the technical reasoning used

to reach those judgments is openly displayed for

review. If conducted optimally, formal elicitation

can reveal a wide range of scientific and technical

interpretations, thereby exposing (and possibly

quantifying) the uncertainties in estimates

concerning repository siting, design, and

performance attributable to limitations in the

state of technical knowledge. Formal procedures

may also help groups of experts resolve

differences in their estimates by providing a

common scale of measurement and a common

vocabulary for expressing their judgments.

In preparing this BTP, the NRC staff has drawn

on the specific recommendations of DeWispelare

and Bonano (1995), as well as earlier, more

general recommendations in Winkler et al. (1992)

and Bonano et al. (1990). It should also be noted

that this BTP has attempted to incorporate, as

appropriate, a number of NWIRB and NAS

recommendations with regard to the use of expert

elicitation as a formal decision-aiding

methodology, including the treatment of "bias."

(See Section 4 for a discussion of staff

consideration of specific NWTRB and NAS

recommendations.) Moreover, this BTP has not

attempted to prescribe the specific technical

issues for which expert judgment should (or

should not) be applied. The staff has viewed such

determinations to be the prerogative of DOE.

However, with respect to performance assessment,

the staff is considering the potential need for

future guidance to identify those specific aspects

of a performance assessment for which the

application of expert judgment may or may not be

appropriate.

The positions and discussions in this BTP are

based on the premise that, under appropriate

circumstances, it is acceptable to supplement data

and analyses with the opinions of experts as part

of the support for demonstrating compliance with

NRC's geologic disposal regulation, and that, in

some cases, these opinions are best obtained

using a formal and well-documented process. This

BTP gives specific guidance by which DOE may

determine if formal expert elicitation would be

useful and provides guidelines for an acceptable

process for obtaining it. Section 2 summarizes the

principal regulatory requirements and

considerations that relate to this topic. The staff's

technical position statements are listed in Section

3, and Section 4 provides a discussion of the

supporting rationale behind each statement of

position.

Definitions of key terms used in the BTP are

provided as Appendix A. Summarized in

Appendix B are lessons learned from a recent

expert elicitation exercise performed by the NRC

staff and its contractor, the Center for Nuclear

Waste Regulatory Analyses (CNWRA). Appendix

C contains the final Commission Policy Statement

with regard to the use of PRA methods in its

regulatory activities, including HLW. The staff's

response to the public comments received on an

earlier draft of the BTP, noticed in the Federal

Register on February 28, 1996 (61 FR 7568), are

contained in Appendix D. Appendix E contains

the staff's views with regard to a possible course

of resolution for NRC SCA Comment 3. Lastly,

Appendix F contains the staff's response to

comments received from the Advisory Committee

on Nuclear Waste on the final draft of the BTP

The NRC staff recognizes that DOE has the

flexibility to determine whether the costs and

benefits of performing an expert elicitation are

advantageous compared with the costs and

benefits of performing theoretical analyses andlor

gathering additional field and experimental data.

That being said, however, the use of expert

elicitation should not be considered as an

acceptable substitute for traditional analyses

based on adequate field or experimental data,

when such data are reasonably available or

obtainable, or the analyses are practicable to

perform. Nor can the use of a formal elicitation

process, even when conducted in a manner

consistent with guidance provided in this BT?,

guarantee that specific technical conclusions will

be accepted and adopted by the staff, a Licensing

Board, the Commission itself, or any other party

to a potential HLW licensing proceeding. Rigid

adherence to a sound elicitation process, in and of

itself, in no way guarantees that the resulting

judgments will be sufficient to meet the

applicant's burden of proof regarding the

substantive issues addressed by the elicitation.

Nonetheless, expert judgments obtained through

NUREG-1563 8
an evidently flawed or poorly documented process

will weaken their ability to support demonstrations of compliance.

It should also be noted that nothing in this BTP

precludes the use of expert judgment obtained

through informal means by DOE, in the

preparation of a potential license application. As

has been the case in previous regulatory activities,

the staff will accept for review the results of

formal or informal judgment so long as the'

rationale associated with the judgment is

adequate, transparent, and sufficiently documented.

1.5 BTPs as Technical Guidance

BTPs are issued to describe, and make available

to the public, methods acceptable to the NRC

staff, for implementing specific parts of the

Commission's regulations, and to provide

regulatory guidance to regulated entities such as

DOE. BTPs are not substitutes for regulations,

and compliance with them is not required. Methods and solutions differing from those set

out in the BTP will be acceptable if they provide a

sufficient basis for the findings requisite to the

issuance of a permit or license by the

Commission.

This BTP constitutes informal pre-licensing

activity between the NRC staff and a prospective

applicant under 10 CFR 2.101(a)(1) and is not

part of a proceeding under the Atomic Energy

Act (Public Law 83-703), as amended. Nothing in

this BTP constitutes a commitment to issue any

authorization or license, nor in any way affects the

authority of the Commission, the Atomic Safety and Licensing Boards, other presiding officers, or

the Director, in any such proceeding.

Published BTPs will be revised, as appropriate, to

accommodate comments and to reflect new

information and experience.

9 NUREG-1563
2 REGULATORY FRAMEWORK

'The following discussion describes the current

regulatory framework and Commission policy that

underpin the staff's technical positions presented

in this BTP.

2.1 10 CFR Part 608

As noted earlier, the Commission's regulations

found at 10 CFR Part 60 address the licensing of

a mined geologic repository for the disposal of

HLW Subpart E of those regulations sets forth

specific performance objectives, along with a

number of general siting and design criteria.

Section 60.112 of Subpart E currently establishes

compliance with EPA standards for the disposal

of HLW (The previously applicable EPA

standards would have limited the release of

radioactive material to the accessible environment

(weighted by a factor approximately proportional

to radiotoxicity, and integrated over a period of

time) as the overall system performance objective

for the geologic repository after permanent

closure.)

With regard to 10 CFR 60.112, Section

60.101(a)(2) states that:

While these performance objectives and

criteria are generally stated in unqualified

terms, it is not expected that complete

assurance that they will be met can be

presented. A reasonable assurance, on the

basis'of the record before the Commission,

that the objectives and criteria will be met is

the general standard that is required. For

§60.112, and other portions of this subpart

that impose objectives and criteria for

repository performance over long times into

the future, there will inevitably be greater

uncertainties. Proof of the future

performance of engineered barrier systems

and the geologic setting over time periods of

many hundreds or many thousands of years

is not to be had in the ordinary sense of the

word. For such long-term objectives and -

BAs noted earlier, the need for future revision to 10 CFR Part 60 is

under consideration. Should 10 CFR Part 60 be revised, the

language of the citations quoted in this section of the BTP would

need to be re-examined, and if necessary, be revised accordingly. It

is not expected, however, that these revisions would require any

change to the staffs technical positions set out here.

criteria, what is required is reasonable

assurance, making allowances for the time

period, hazards, and'uncertainties involved,

that the outcome will be in conformance with

those objectives and criteria. Demonstration

of compliance with such objectives and

criteria will involve the use of data from

accelerated tests and predictive models that

are supported by such measures as field and

laboratory data and natural analog studies.

In a subsequent proposal to conform the 10 CFR

Part 60 regulations to then extant EPA standards

for management and disposal of HLW, the

Commission further elaborated on what was

necessary for a satisfactory demonstration of

compliance (NRC, 1986; 51FR 22288):

Demonstration of compliance with the - performance objectives of §60.112 will also

involve predicting the likelihood and

consequences of events and processes that

may disturb the repository. Such predictions

may involve complex computational models,

analytical theories and prevalent expert

judgment. Substantial uncertainties are likely

to be encountered and sole reliance on

numerical predictions to determine

compliance may not be appropriate. In

reaching a'determination of reasonable

assurance, the Commission may supplement

numerical analyses with qualitative judgments

including, for example, consideration of the

degree of diversity or redundance among multiple barriers of a specific repository.

A primary consideration of a decision to

authorize construction of a repository will be

whether the site and design comply with the

performance-objectives and criteria contained in

Subpart E. DOE must interpret the geologic'

record and provide a demonstration that the

repository site and'design will comply with explicit

numerical performance standards. As noted in the

regulation, there will be substantial and

unavoidable uhcertainties 9 in predicting the

long-term performance of a geologic repository.

9Uncertainties may include, but not be limited to the: (a) identification of basic phenomena and their potential effects on repository performance; (b) development and validation of models to describe

these phenomena; (c) accuracy of available data; and (d) calculational uncertainties.

11 NUREG-1563
Conclusions as to the performance of the geologic

repository and of particular barriers over long

periods of time, by necessity, will be based largely

on inference, as it will not be possible to carry out

test programs of sufficient duration or that

simulate the full range of potential conditions

expected over the period of regulatory concern

(see NRC, 1983; 48 FR 28204). Given these

uncertainties, it will be necessary for DOE to

adopt a variety of design features, develop

sophisticated models, perform tests, acquire data,

and undertake other measures to be able to

demonstrate that the performance objectives will

be met.

For its part, in reaching a potential construction

authorization decision, the Commission has stated

that ". . . a reasonable assurance, on the basis of

the record before the Commission, that the

objectives and criteria will be met is the general

standard that is required." (10 CFR 60.101(a)(2))

To reach a "reasonable assurance" finding, the

Commission has said it will need to be able to do

at least two things (48 FR 28201). It must first

determine that DOE has demonstrated

compliance with the numerical performance

standards, and, second, it must satisfy itself that

DOE's analysis of the site and design is

sufficiently convincing, that its limitations are

well-understood, and that DOE has demonstrated

that its analyses have made appropriate allowance

for the time period, hazards, and uncertainties

involved.10

Confidence in the adequacy of DOE's data,

analyses, and other items and activities associated

with the repository program will be enhanced to

the extent that they are obtained through a quality

assurance (QA) program consistent with Subpart

G of 10 CFR Part 60. In this regard, the staff has

acknowledged that external peer reviews may be

used as part of the QA actions necessary to

provide confidence in the work submitted. The

NRC staff has provided guidance on how these

peer reviews would be conducted in

NUREG-1297 (see Altman et al., 1988).

One of the greatest challenges facing the

Commission in making the determinations

necessary for the licensing of a potential HLW

1 For a more detailed discussion of the Commission's views on the

'reasonable assurance" concept, in the context of the geologic repository regulation, see NRC (1983 and 1986).

repository will be to assess;the validity of DOE's

treatment of uncertainty. Various methods may be

used (e.g., probability distributions, conservative

"bounding" analyses), and the Commission will

evaluate quantitative and nonquantitative

arguments to assess their application. For

purposes of demonstrating compliance with the

10 CFR Part 60 performance objectives, the

Commission has acknowledged that the treatment

of uncertainty in DOE's compliance

demonstrations is expected to ". . . rely heavily on

[the use of some form of] expert judgment. . . in

the selection of an appropriate [uncertainty

reduction] method and for the application of that

technique .... " (NRC, 1986; 51 FR 22292)11 This

is consistent with previous licensing experience

where the NRC Licensing Board has admitted

expert opinion evidence by one expert or by a

panel of experts in the analysis of risks at nuclear

pover plants where little or no data exist or the

available data are unreliable.

As with other NRC licensing actions, a Licensing

Board's decision to grant or deny a license for a

proposed repository will be based on submitted

evidence that is a combination of fact and

opinion. The subjective judgments of individual

experts and, in some cases, groups of experts,

would likely be presented to interpret DOE's data

and analyses, and also to address the technical

issues presented in the hearing. Such assessments

may be not only quantitative but qualitative as

well. In certain instances, it may not be possible

to develop a "correct" estimate of some event or

process. Rather, it may only be possible to

develop a "range" of estimates. Thus, both DOE

and NRC may have no choice but to present the

informed judgment of qualified experts.

That being said, however, in its review of DOE's

statutory SCP (10 CFR 60.16), the NRC staff

noted its concerns with DOE's site characterization programs, specifically calling into question

DOE's potential over-reliance on the use of expert

judgment to supply the necessary information in

any potential license application.12

tin the context of the containment requirements (10 CFR 60.112), EPA also noted that"... sole reliance on [the] numerical

prediction of [geologic repository performance) may not be

appropriate. . [an ethe implementing agencies may choose to

supplement such predictions vith non-quantitative judgments"

(EPA, 1985; 50 FR 38088).

12

See the staff's SCA Comment 3 (in NRC, 1989). This open item

has not been resolved at this time.

NUREG-1563 12
2.2 Commission Policy Statement

Concerning the Use of PRA

Methods in Nuclear Regulatory

Activities

NRC has generally regulated the use of nuclear

material based on deterministic approaches and

used probabilistic approaches only in certain

specialized areas. However, after the Three Mile

Island incident, NRC has increasingly used PRA

techniques to augment the traditional

non-probabilistic methods of analyzing nuclear

safety. PRA techniques have been applied

successfully in several regulatory areas (see DOE

et al., 1992) and have proven to be a valuable

complement to engineering approaches.

Recently, Commission policies and regulations

have been based, in part, on PRA methods and

insights (NRC, 1995; 60 FR 42623). In light of

these developments, the Commission issued an

overall policy statement on the expanded use of

PRA in nuclear regulatory activities, including the

area of nuclear waste disposal (60 FR 42622).

Among other things, the Commission Policy

Statement (see Appendix C) called for the use of

"... state-of-the-art... [PRA] methods ... ." in a

manner that complements the deterministic

approach. (60 FR 42628) (According to a recent

staff survey, such PRA methods include, among

other things, methods to obtain and process

expert judgment (see PRA Working Group, 1994).)

For the purposes of the regulatory actions in the

waste management program, the Commission

noted, in the "Statement of Considerations for the

final Policy Statement," that it .... agrees with

[public] comments regarding [the treatment] of

uncertainties in projecting repository performance ... [through] the use of technical expert judgment ... ." (60 FR 42627)

To ensure evenness and consistency in the staff's

future uses of PRA methods, NRC recently

established a PRA Working Group to: (a) evaluate

the staff's current uses of PRA; (b) identify areas

for improvement in that use; and (c) recommend

the tenets of some basic principles and guidance

for application in the future. In the area of expert

elicitation, the PRA Working Group made a

number of general recommendations on formal

techniques for obtaining, evaluating, and

processing expert judgment (PRA Working

Group, 1994; pp. C-129-C-148). To ensure the

quality and reproducibility of the elicited

information, the Working Group recommended

that any formal elicitation procedure contain the

following process steps (op cit., pp. C-130-

C-135):

* Selecting and defining technical issues.

* Selecting experts.

* Organizing assessments.

* Preparing for the elicitation.

* Processing expert judgment.

* Documenting.

The technical positions cited in Section 3 are

consistent with these recommended process steps.

13 NUREG-1563
3 BRANCH TECHNICAL POSITIONS

In view of the aforementioned policy

considerations and statements of regulatory consideration underpinning 10 CFR Part 60, the

staff has adopted the following technical positions concerning the use of expert elicitation in

demonstrating compliance with the geologic

repository disposal regulations. (As a supplement

to the technical positions here, Appendix A

provides definitions for certain key terms.)

(1) In matters important to the demonstration of

compliance, the use of formal expert

elicitation should be considered whenever one

or more of the followingconditions exist:

(a) Empirical data are not reasonably

obtainable, or the analyses are not

practical to perform;

(b) Uncertainties are large and significant to

a demonstration of compliance;

: . More than one conceptual model can

explain, and be consistent with, the

available data; or

provided to them, and the form of the

judgments that will be 'required.

Step No. 2-Selection of Experts

Before selection of the subject-matter experts,

whose judgments will be elicited, two other

types of experts should be recruited- the

normative expert and the generalist. Because

these types of experts may influence the

outcome of the elicitation by the manner in

which judgments are elicited, analyzed, or

used, care should be taken in their selection

to ensure that they can perform in an

objective and impartial manner. Working together, the normative experts and

generalists generate and apply specific

criteria for the selection' of the subject-matter

experts.

The subject-matter experts selected for

elicitation should be individuals who: (a)

possess the necessary knowledge and

expertise;13 (b) have demonstrated their

ability to apply their knowledge and expertise;

(c) represent a broad diversity of independent

opinion and approaches for addressing the

topic(s) in question; (d) are willing to be

identified publicly with their judgments; and

(e) are willing to publicly disclose all potential

conflicts of interest.

The criteria used to select the various experts

of the elicitation team should be documented.

Step No. 3-Refinement of Issues and Problem

Decomposition

The generalists and normative experts should

work with'the subject-matter experts to

decompose the broad objectives of the

elicitation by clearly and precisely specifying

more focused and simpler sub-issues.

Step No. 4-Assembly and Dissemination of

Basic Information

Assembly of background information should

be initially conducted by the generalists and

13 Vsth regard to Item (a), it would be useful for members of the

expert panel to possess at least some rudimentary knowledge of

both decision-making theory and statistics. However, the possession or the lack of this knowledge should not be used as a selection

criterion.

(d) Technical judgments are required to'

assess whether bounding assumptions or

calculations are appropriately

conservative.

- (2) (a) When formally eliciting expert judgment,

the applicant should use a consistent and

systematic procedure that will ensure that the

results obtained accurately reflect what is

known and not known about the topic in

question. The components in an acceptable -- elicitation process are described below and -

are illustrated in Figure 1. Although written

largely for the elicitation of individual

subject-matter experts, the same approach

can be applied to a panel (or a team) of

subject-matter experts.

Step No. .1-Definition of Objectives

The objectives of the elicitation should be

defined explicitly and in a manner that

reflects a clear understanding of how the

judgments obtained will be used. The

explication of these objectives should then

guide the choice of experts, the information

15 15 .NUREG-1563 ~ ~ ~ ~ 
Tvni 0

Step No. I I

DEFINITION

OF #

OBJECTIVES

I

a

_ ,~~~~~~~

:

:

,......................

Stop No. 2

SELECTION

OF -

EXPERTS

Step No. 3 v .

REFINEMENT

OF

ISSUES

! Stop No. 4

ASSEMBLY AND

DISSEMINATION

OF BASIC

INFORMATION

.... ...........

a

a

Step No. 5 Step No. 6 Step No. 7 Step No. 8

PRE-| ELICITATION POST-O DSPARATE

eEUICITATION X OF ELICITATION bVIEWS AND

TRAINING JUDGMENTS FEEDBACK AGGREGATION _ _ A~~~~~~~~~~~~----. OFJUDG MENTS

.6

Step No. 9

P

| = ~~D 0 C U M E N T A T I NX

* E.g., te data, evidence, or knowledge base used by the subjct-watexperts

to arrive at their wspective judgments.

Figure 1 Components in an acceptable expert elicitation process. (Numbers next to the process

blocks correspond to the technical position statements described in the text.

Dashed lines indicate a feedback loop.)
normative experts. As the elicitation process

proceeds, the subject-matter experts may be

able to recommend additional sources of

information. Bias in the selection of this

background material should be avoided such

that a full range of views is represented and

the necessary data and information are

provided in a uniform, balanced, and timely

fashion to all subject-matter experts.

Step No. 5-Pre-Elicitation Training

Individual (or teams of) subject-matter

experts should be provided training before

the elicitations to: (a) familiarize them with

the subject matter (including the necessary

background information on why the

elicitation is being performed and how the

results will be used); (b) familiarize them with

the elicitation process; (c) educate them in

both' uncertainty and probability encoding

and the expression of their judgments, using

subjective probability; (d) provide them

practice in formally articulating their

judgments as well as explicitly identifying

their associated assumptions and rationale;

and (e) educate them with regard to possible

biases that could be present and influence

their judgments.

Step No. 6-Elicitation of Judgments

The individual elicitation session with each

subject-matter expert (or teams of

subject-matter experts) should be held in a

private setting conducive to uninterrupted

discussion. The generalists and normative

experts should be in attendance for the

complete session with each subject-matter

expert. At the start of the session for each

subject-matter expert, the normative expert

should summarize the issues to be covered

and outline the logistics of the elicitation. All

definitions and assumptions agreed to by the

group during pre-elicitation meetings should

be reviewed. All subject-matter experts

should be queried in a uniform manner and

asked to provide specific answers to

questions about the issues considered and the

reasoning behind their responses. Care

should be taken to ensure that the required

information is obtained and that it is

internally consistent. Responses of all

subject-matter experts should be documented

thoroughly with one or more of the' following:

written notes, transcription, and audio or

video tape.

Step No. 7-Post-Elicitation Feedback

Each subject-matter expert (or teams of

subject-matter experts) should be provided

feedback from the elicitation team on the

results of his or her elicitation as soon as

practical after the elicitation sessions are

completed. Each expert should be queried as

to the need for revision or clarification of his

or her respective judgments based on that

feedback. As is the case for all the elicited

judgments, the rationale for any revisions

should be documented scrupulously.

Step No.' 8-Aggregation of Judgments

(Including Treatment of Disparate ievs)

Whatever aggregation method is employed,

the individual expert's opinions must be

preserved, documented, and provided to the

NRC staff. Transparency in the aggregation

process will render these judgments,

including disparate views or outliers, 4 useful

for subsequent analyses. If disparate

'judgments are aggregated or combined, the

applicant should: (a) provide some rationale

for the specific aggregation techniques

employed and provide documentation

sufficient to trace the impact of the individual

expert's judgment on the consolidated

judgment; and (b) show what effect, if any,

the disparate views would have on design

and/or performance.

When widely disparate opinions arise, extra

effort should be taken to document

thoroughly the bases for the differing views.

Subject-matter experts with differing views

should be asked to comment on opposing

views during and/or after their individual

elicitations. Should the disparity in views

persist, then each of the significantly varying

views should be provided as output of the

elicitation so that it may be incorporated

directly into technical analyses and

performance assessments, or used to

represent the extremes in a sensitivity

analysis.

1As used in this guidance, outliers" refers to those opinions which

lie apart from the views or expected (average) views af other

experts.

17 NUREG-1563
-

Step No. 9-Documentation

Proper documentation of a formal expert

elicitation should indicate what was done,

why, and by whom. The resulting judgments

should be clearly described along with the

reasoning supporting these judgments. The

specific issues addressed by the elicitation

should be precisely defined. Unambiguous

definitions of all specific terms should be

provided and any assumptions used in the

elicitation should be explicitly stated. The

judgments, as they are stated by each

subject-matter expert, should be provided,

accompanied by the logic and information on

which they are based. Any calculations that

the experts considered important in

determining judgments or models used

should be recorded and all literature used,

whether public or restricted, should be

properly referenced. Proper documentation

should clearly distinguish between that

information provided directly by each

subject-matter expert and any subsequent

processing of that information, such as

smoothing, interpolation, extrapolation, or

aggregation of the judgments of different

experts.

(b) The approach described above envisions

that all of these process steps would be part

of a procedure for an expert elicitation. If

preferred, some of these steps can be

combined as long as all of the elements of the

process are addressed.

If one or more of the process steps are

omitted from the recommended procedure,

the staff may need additional information for

its consideration before accepting the results

of an elicitation for its review and evaluation.

(3) If information from an expert elicitation is to

be submitted in support of a license application, and if additional data or information

becomes available, subsequent to the completion of the elicitation, which could change

opinions or judgments obtained in the formal

elicitation, the results of the elicitation should

be re-examined and updated, as appropriate.

In addition to the information requested

above, documentation should include a

detailed description of the updating process.

NUREG-1563 18
4 DISCUSSION

The technical positions outlined in Section 3 are

motivated by several primary purposes.

First, as noted in Section'1.1, the NRC staff has

traditionally accepted, for review, the appropriate

judgments of technical experts regarding the basis

of a license application. In this regard, the

Commission has already acknowledged that it

expects that expert judgment will play an,

important role in any potential geologic repository

licensing proceeding (see NRC, 1986; 51'FR

22292). Thus,' the staff believes that these technical

positions are consistent with both previous

Commission practice and Commission

expectations on how'the geologic repository

disposal regulations will be implemented.

In defining its technical positions, the staff has

attempted to draw from previous Commission

experience with the application of formally elicited

expert judgment in a regulatory context. The most

prominent examples of the Commission's use of

this process can be found in the area of

commercial nuclear reactor regulation. As

discussed in Section 1.1, the techniques of expert

elicitation have been applied heretofore, in NRC's

regulatory program, to the assessment of- ' uncertainty associated with seismic hazards and

faulting parameters, and to the assessment of risk

from severe accidents.

Second, the NRC staff recognizes that, in certain

instances, the collection'of data is not feasible and

that expert judgment must be used to complement

existing information, in order to support

demonstrations of compliance with the.;

requirements of 10 CFR Part 60. Topics'where

staff expects that expert judgment will be brought

to bear are discussed in more detail in Bonano et

al. (1990) and DeWispelare et al. (1994). Among

these are: scenario formulation, development, and

probability estimation; development of and

confidence building for computational models;

parameter estimation; and identification of where

additional data and information gathering are

necessary and where they may be impractical. As

important as its role may be,' however, expert

judgment should be distinguished from both

measured data'or technical calculations based on

accepted scientific laws and principles. It should

be viewed as an alternative, and employed when

other means'of obtaining requisite data or

information have been thoroughly considered and

it has been concluded that such means are not

practical to implement.

,in evaluating the practicality of obtaining the

needed information (or data), DOE is expected to

consider cost, schedule, resource availability, and

other programmatic factors. DOE will need to

evaluate the programmatic costs and risks ofusing expert elicitation as an alternative to more

objective data-gathering methods, and the

likelihood that the elicitation itself may introduce

greater uncertainty into the demonstration of

compliance than will the use of objective

information.

Lastly, when expert judgments are used to support

a demonstration of compliance, whether they are

formally elicited or not, sufficient documentation

should exist to allow external examination of what

the judgments were, how the judgments were

arrived at (their basis), how the judgments were

used, and why'the judgments were used instead of

obtaining objective information (e.g., obtaining

the needed data). Such documentation supports a

broader understanding and acceptance of the

expert judgment. For expert judgment to be

defensible under external peer review, the basis

for the judgment should be well-documented.

Peers must be able to trace origins of specific

judgments from initial assumptions through.

integration of results and conclusions. There

should be no gaps in the documentation, so that

an evaluation can be based on a thorough

understanding of the work presented. In addition,

the availability of such documentation makes

possible technical discussions in terms of

underlying principles rather than just the

individual outcomes. Documentation should also

include the justification for using expert judgment

to complement other data and analyses in

reducing the residual uncertainty.

All assumptions 'about the technical issues that

were used to develop the expert judgments should

be identified. The judgments 'as they are stated by

each individual subject-matter expert should be

provided in the documentation. Any calculations

that a subject-matter expert considered important

in forming judgments, or models used, should be

documented.' All literature used, whether public or

19 NTREG-1563
restricted, should be properly referenced. Any

subsequent manipulation of the individual

expert's judgments, such as aggregation or

combining with other opinions or data, should be

documented thoroughly.

In closing, it should be noted that this NUREG

discusses the rationale for and the potential benefits derived from the use of expert judgment in a number of places. Without intending to limit

those discussions, the following summary is

provided. Section 1 describes the ubiquitous

nature of expert judgment in scientific and

technical activities. Section L2.1 provides a

description of typical outputs of an expert elicitation and the potential uses of the expert

judgments in the evaluation of repository

performance assessment. Section 13 provides actual examples of the NRC use of expert

judgments, including those used in PRA. Section 3

provides the technical position on conditions

warranting the consideration of a formal expert elicitation. Later in this section, the staff

elaborates on these conditions and discuss the

benefits derived from obtaining expert judgments.

Finally, Appendix A provides a discussion of

expert judgment from a definitional perspective

and Appendix B provides results and lessons

learned from the NRC-sponsored formal climate

expert elicitation.

The following discussion parallels the list of

technical positions given in Section 3.

(1) Conditions That May Warrant Consideration

of a Formal. Process of Expert Elicitation

So-called "professional" or "engineering"

judgment is exercised by scientists, engineers, and

technical program managers routinely. More often

than not, this is done informally and in a

non-explicit manner. However, as noted in Section

1, the staff believes that confidence in DOE's

HLW program will be enhanced by the

transparency of the decision-making process.

(This may also enhance public confidence, as

well.) Thus, it may be appropriate for DOE to

"formalize" the way in which certain decisions on

issues or problems are made, as noted below.

One way in which the necessary transparency can

be achieved in an expert elicitation process is to

use a structured procedure, based on proven

decision analysis methods, to gather the necessary

judgments from recognized subject-matter

experts. The use of such formal methods, as

discussed in Technical Position No. 2 and

depicted in the figure shown previously, promotes the accurate, consistent, and efficient collection

and processing of the expert judgments.

Moreover, the use of a formalized process increases the scrutability of the resulting judgments and enhances the communication of

the results.

The question thus arises when or under what

circumstances might it be appropriate to

undertake formal methods of expert elicitation in

the HLW program. As discussed in Section 3, the

staff believes that formal methods of expert elicitation may be of the greatest value to the

program and should be considered in the

following situations: (a) empirical data are not

reasonably obtainable or the analyses are not

practical to perform; (b) uncertainties are large and significant to a demonstration of compliance; (c) more than one conceptual model can explain, and be consistent with, the available data; or

(d) technical judgments are required to assess

whether bounding assumptions or calculations are

appropriately conservative, as discussed below.

Precise criteria for determining when an expert

elicitation is to be undertaken are not presented here. Programmatic concerns such as timing, cost,

and compliance demonstration will have a major impact on determining whether an expert elicitation or some other form of information

gathering should be used. For example, expert elicitations may be more costly than a particular

type of data-gathering investigation, but they may

provide results in a more timely manner.

Programmatic concerns dominate the choices of:

(a) gathering additional field or laboratory data;

(b) undertaking additional theoretical analyses;

(c) using expert elicitation; or (d) altering the

compliance demonstration strategy, to lessen or

eliminate the need to resolve a particular issue.

Therefore, the intent of this technical position is

to allow DOE the maximum flexibility in choosing

an approach, as long as an effective

demonstration of compliance with the regulations

can be made.

* Empirical data are not reasonably obtainable or

the analyses are not practical to perform-In

some cases, data directly relevant to a

problem, question, or issue may be lacking or

incomplete (e.g., do not cover the entire

NUREG-1563 20
period of regulatory interest), or are not

reasonably obtainable or practical to

perform. Under such circumstances, existing

data may be supplemented with expert judgments. These judgments may be obtained

using a formal elicitation process. Examples

of situations where field data are not

reasonably or practicably obtainable include:

(i) the site characteristics important to waste

isolation would be irreversibly compromised

by extensive data collection in such a way. that could potentially disqualify the site;

(ii) it is infeasible or impossible to collect

data over the temporal or spatial scales.

appropriate to adequately address a

particular issue;'and (iii) the cost of collecting

the comprehensive suite of data may be

prohibitive.

Uncertainties are large and significant to a

demonstration of compliance-Because of the

extremely long period of regulatory concern, a

combination of experimental methods,

studies of natural analogues, and

mathematical models'will be used in the

technical and performance assessments

necessary to support any potential license

application. Mathematical models are

expected to be the primary tools for

estimating the long-term future performance of the repository. Identification of external

conditions to which the'repository will be

subjected during the period of regulatory

concern is an essential requirement for

applying the mathematical models. These

external conditions include evolving

tectonism, volcanism, seismicity, climate,

hydrology, geochemistry, and other physical

processes that may affect repository

performance. Because of the complex interactions of these processes, as well as the

temporal and spatial scales involved, more

than one credible interpretation of existing data is frequently possible.

As a result of these practical limitations, after

every reasonable effort has been made to -

reduce the uncertainties affecting repository siting, design, and performance, large

uncertainties will probably still persist. How

the applicant intends to examine and resolve

specific residual uncertainties has not yet

been identified. However, regardless of the

specific approach adopted to accommodate

these uncertainties, there is no doubt that

expert judgments will be pervasive (Fehringer

and Coplan, 1992). Where such judgments have a direct and significant bearing on

compliance, formal elicitation methods may

be appropriate.

* More than one conceptual model can explain,

and he consistent with the available

data-Conceptual modeling of a HLW

disposal site is based on a combination of the

application of fundamental physical and

chemical principles and data interpretation.

Data interpretation and conceptual model

development rely extensively on expert judgments and it is not uncommon for

multiple or alternative conceptual models to

-emerge that are consistent with available - data. This is particularly true when the data

available are limited and amenable to

multiple, equally valid interpretations.

Because conceptual models provide the

underpinning for the development of the

mathematical models and computer codes'

that will be used in the quantitative estimates

of performance measures, the selection or

rejection of a conceptual model could have a

considerable impact on the results of

computational analyses: These judgments,

when made by DOE to support its

calculations in a potential license application, will be subject to considerable'scrutiny by the

NRC staff (Park et al., 1994), and therefore

are appropriately derived from a formal

elicitation process.

* Technicaljudgments are required to assess

whether bounding assumptions or calculations

are appropriately conservative-Bounding'

assumptions or calculations are used as a

technical approach to providing scientifically

based estimates when'the level of uncertainty

is very high,when the subject matter is

complex, and when approximations are

sufficient to resolve the issue at hand. If such

assumptions and calculations are used as a

basis to terminate'or curtail further data

collection or analyses, judgments must be

made, and justified, that such estimates are

sufficiently conservative for their intended

application. For issues critical to compliance

- 21 21 ~~~~NUREG-156
-

demonstration, formal elicitation of these

judgments and justifications may be of value.

(2) A Consistent and Systematic Process for

Elicitation Should Be Applied

If expert elicitations are to render accurate

representations of the legitimate range of

scientifically supportable interpretations among

the informed technical community, they should be

conducted using a structured procedure, based on

proven decision analysis methods. A structured,

thoroughly documented procedure allows reviewers

to reconstruct the logic and events involved in the

elicitation and use of expert judgment. Use of

such a protocol establishes and maintains the

quality of the process in much the same way as

use of a QA program for field data collection.

Adherence to such a protocol is also critical in

supporting internal and external reviews of the

resulting judgments and to foster confidence in

the integrity of the process. That being said,

however, scrupulous adherence to a sound

elicitation process, in and of itself, in no way

guarantees that the resulting judgments will be

sufficient to meet the applicant's burden of proof

regarding the substantive issues addressed by the

elicitation. Nonetheless, expert judgments

obtained through an evidently flawed or poorly

documented elicitation process might ultimately

undermine the credibility of any demonstrations

of compliance supported by those judgments.

The procedure set forth in Section 3 in summary

form, and repeated below, has elements of several

other protocols-see, for example, NRCNtREG/CR-5411 (Bonano et al., 1990) and

NUREG-1150 (NRC, 1990); EPRI (Coppersmith

et al., 1993); the Sandia National Laboratories

(Trauth et al., 1994); and the CNWRA

(DeWispelare et al., 1993 and 1994). It is provided

here, but not with the intent that it be rigidly

applied in every instance where judgments are

elicited. Instead, it should be viewed as a general

framework for a formal elicitation acceptable to

the NRC staff. The applicability of any of the

proposed process steps and thus the degree to

which the overall process is implemented should

be evaluated in each separate elicitation, and the

process may be customized or revised, as needed,

for the elicitation of interest.

In addition, although the process steps are listed

in numerical order, it is not necessary that the

individual steps be performed in the exact

sequence presented. In fact, it is expected that

several of these process steps will proceed or can

be initiated concurrently, subject to repeated

iterations and opportunities for feedback from the

subject-matter experts. This may be especially

true for Step Nos. 2, 3, and 4, which are depicted

as parallel process steps with feed-backs, as

shown in the preceding figure. What this figure

shows, for example, is that once the subjectmatter experts are identified (in Step No. 2), they

can help to better define the objective of the

elicitation (Step No. 1) and thus aid in the

decomposition of the elicitation issue into its

constituent parts (Step No. 3). Moreover, the

subject-matter experts can also aid in the

identification of additional information that could

facilitate the elicitation (Step No. 4).15

Step No. 1-Definition of Objectives

Perhaps the most important step in any

elicitation process is the precise definition of

its ultimate objective(s). Proper definition of

the objective(s) calls for an understanding of

how the judgments will be used in subsequent

analyses. This understanding should direct

the overall content of the elicitation by

defining the nature of the expertise that

should be brought to bear (Step No. 2); the

assumptions and information that will be

provided to the appropriate subject-matter

experts (Step Nos. 3 and 4); and the form of

the judgments that should result (Step No. 6).

This is a critical step that should include

input from generalists who are familiar with

the overall project and the specific

information needs and intended uses.

Step No. 2-Selection of Experts

7jpes of Experts: Three types of experts are

ordinarily recruited to participate in a formal

elicitation: generalists, normative experts, and

subject-matter experts (the last of whom are

usually referred to as "the experts").

Generalists and normative experts are

recruited first and together comprise the

elicitation team, which is responsible for

1

5

Alternatively, before deciding on the final panel of subject-matter

experts (Step No. 1), in contacting candidate experts, it might be

useful to solicit their input on refining the general problem (Step

No. 3) that was previously formulated bv the generalists and the

normative experts. In this way, a much arger knowledge base is

available to fine-tune the issues that will ultimately be addressed by the final group of subject-matter experts. Further, final refinement

of the sub-issues (and questions-Step No. 3) should be performed

ultimately by the selected panel of subject-matter experts.

NUREG-1563 22
organizing, conducting, and documenting the

elicitation process.

Generalists are individuals knowledgeable

about various overall, and one or more

specific aspects related to site

characterization, repository design, and

performance assessment. Typically,

generalists have substantive knowledge in one

discipline (e.g., hydrology, geology, material

science, transport phenomena, etc.) and a

solid general understanding of the technical

aspects 'of the problem.

Normative experts have training in probability

theory, psychology, and decision analysis.

They assist the generalists and subject-matter

experts in articulating their professional

judgments and thought processes in a form

suitable for input into a particular technical

assessment.

Subject-matter experts are the experts from

whom judgments are elicited. These are

individuals who are at the forefront of a

specialty relevant to geologic waste disposal,

and are recognized by their peers'as

authorities because of their sustained and

significant research on the topic.

In selecting each of the three types of experts,

especially the subject matter experts, it may

be useful to seek qualified nominations from

outside sources, or recognized peers in the

field. This would include, for example: the

National Academies of Sciences and

Efgifiering; academia; recognized -

professiofial'societies (e.g., Sigma Xi, The

Geological Society of America, The American

Society of Civil Engineers), National

laboratories; knowledgeable Federal Agencies

and International Organizations; private

industry; State development and regulating

bodies; representative public interest groups;

;and interested stakeholders. (Examination of

the frequency of citations in the scientific

literature may also be of help in this regard.)

The elicitation team members, and their

respective assignments, would then be

selected from the list of nominees (for

example, see Appendix A in DeWispelare et

al., 1993).

Selection Criteria: Among its many tasks, the

elicitation team generates and applies criteria

for the selection of the subject-matter experts.

For this reason, and because the elicitation

team members may influence the outcome of

the elicitation by the manner in which

judgments are elicited, analyzed, or used,

special care should be taken in the selection

of generalists and normative experts to ensure

that they can perform in an objective and

impartial manner. Although the selection

criteria discussed below focus primarily on

those attributes necessary to establish the

suitability and substantive'knowledge'of

prospective subject-matter experts, the same

critiria-should be applied to the selection of

generalists and normative experts insofar as

they relate to their expertise, experience, and

ability to cany out their respective roles. Tis is

particularly true as it relates to the criterion

concerning the appearance of bias or conflict of

interest owing to the influence the generalist

and normative expert can have on the outcome

of any potential elicitation, as discussed below.

As stated in the technical position, the panel

of experts selected for elicitation should

comprise individuals who: (a) possess the

necessary knowledge and expertise; (b) have

demonstrated their ability to apply their

knowledge and expertise; (c) represent a

broad diversity of independent opinion.and

approaches for addressing the topic(s) in

question; (d) are willing to be identified

publicly with their judgments; and (e) are

: willing to identify, for the record, any

potential conflicts of interest.

-Technical knowledge, expertise, and the

ability to address the topic of the elicitation

by a subject-matter candidate can be

established through examination of his or her

educational background, professional

experience (including research and consulting

activities in related problems or studies),

publication record, previous experience as a

peer reviewer for the work of others,

membership and leadership positions in

professional societies, and awards and other

indications of peer recognition.

A subject-matter expert's ability to apply his

or her substantive knowledge to the task at

hand can be determined by examining the

expert's record of published research and

23 NUREG-1563
participation in consulting activities on

related problems or studies, prior participation in other expert elicitations, as

well as experience as peer reviewer of the

work of others. Although difficult to quantify,

an expert's flexibility of thought and ability to

objectively consider evidence that challenges

his or her own conventional wisdom, as well

as an ability to explain complex topics in

clear and straightforward terms, is also of

value for a successful elicitation.

In selecting subject-matter experts, it is

important to identify candidates who have

the requisite credentials, including a variety

of backgrounds and experiences, to ensure

that the full range of legitimate opinions on a

particular scientific topic is represented and

is incorporated into the study. Particular care

should be taken to avoid selecting experts with similar educational backgrounds and

experiences, because this increases the

likelihood that they will invoke similar

assumptions and approaches to arrive at

their respective judgments. Dependence

among subject-matter experts (so-called

"expert dependence"-see Clemen and

Winkler (1985); and Chhibber and

Apostolakis (1993)) can significantly diminish

the diversity of opinion. In this regard, care

should be taken to select experts who have

had training at different academic

institutions and, to the extent possible,

represent a diversity of scholarly approaches.

Conflict of Interest: The credibility of the

judgments from any formal elicitation will be

increased when the subject-matter experts have fewer conflicts of interest in the areas of

institutional influences, financial or

professional gain, or promotion of a social or

political agenda, to the extent practicable. If

some subject-matter experts are encumbered

by such conflicts of interest, an attempt

should be made to balance the influences by

other panelists. The NWTRB has commented

frequently on this point and has

recommended using balanced panels of

experts, incorporating experts from outside

DOE and its contractors, who have

independent and varying perspectives on an

issue. This is important to building credibility

into the judgments of the experts (NWIRB,

1990a, 1990b, 199L 1993, and 1995).

Perhaps the most frequent conflict-of-interest

concern is that of "set bias" by virtue of who

employs the experts. The issue here, for

example, is whether the subject-matter

experts derive employment or income from

organizations charged with conducting the

overall performance assessment or with the

construction and licensing of the repository.

Other potential conflicts of interest may

involve subject-matter experts' close-working

relationships with individuals involved in

repository characterization or development. Subject-matter experts should be asked to

provide written statements of any potential or

potentially perceived conflicts of interest,

each of which should be made a part of the

record of the elicitation.

The staff position does not assume, however,

that any individual with a perceived or real

conflict of interest will permit this conflict to

influence his or her professional judgments. Furthermore, the staff does not wish to

exclude crucial information from an expert elicitation simply because a knowledgeable

individual has a potential conflict of interest.

The HLW program is a small technical

program in many respects and in certain key subject disciplines, there are relatively few

experts. The population of known experts has

probably been involved with one or more of

the interested/affected parties, and therefore

some conflicts of interest may be

unavoidable.

Consequently, an elicitation should be

designed such that the knowledge and

reasoning of experts with potential conflicts,

along with the fact and nature of those

potential conflicts, should be made available

to all subject-matter experts. Convening

balanced elicitation panels and disclosures of

possible conflicts are appropriate measures

to counter the appearance of a potential

conflict of interest.

Also, credibility of the elicitation can be

enhanced when nominations for candidate

subject-matter experts can come from

organizations such as professional and

academic societies, peers in the field, or

reviews of the scientific literature. The

subject-matter experts should be selected

from the list of nominees, using explicit

criteria, and the entire selection process

NUREG-1563 24
should be systematic and thoroughly.

documented.

Finally, this staff position is not intended to

preclude the ability of DOE to use its own

staff or its contractor's staff in any elicitation

panel. NRC has traditionally considered, for

review, the technical analyses prepared by the

licensee (and its contractors) and will

continue to do so.

Step No. 3-Refinement of Issues and Problem

Decomposition

After definition of the objective of the

elicitation (Step No. 1), perhaps the next most

important process step in an elicitation would

be decomposition of the problem into concise

and distinct questions. (As noted earlier, this

step may be initiated concurrently with the

selection of the subject-matter experts.)

Generalists and normative experts comprising

the elicitation team may make a preliminary

attempt to identify key sub-issues and to

decompose the problem. Problem

decomposition in elicitation refers to

breaking down issues to provide for easier

and less complex assessments that can be

recombined into a probability distribution or

utility function for the quantity of interest.

The recombination is usually accomplished'

using one or more mathematical models that

express the value of interest as a

mathematical function of component

quantities. -

Definition of the issues should be precise;

clarity of the issues is important for the

elicitation design. The issues can range from

general to specific and from simple to

complex. Conventionally, complex problems

are broken up into smaller and simpler

components to facilitate the solution of the

problem. The basic tenet of problem

decomposition is that the solution of the

smaller components is more tractable than

that of the'entire problem. Problem'

decomposition related to the elicitation of

expert judgments is advocated by many (see

Chhibber et al. (1992); and Hora 'and Iman

(1989), among others) as the vehicle to

increase the likelihood that the judgments are

focused on issues with which'the ` '

subject-matter experts are thoroughly

familiar. If possible, issue definition and

problem decomposition should first place

emphasis on making explicit the subjectmatter expert's direct knowledge, based on

experience and evidence, and second on his

or her ability to process or encode this

knowledge into probability estimates (as in

the approach advocated by Kaplan (1992).

The normative expert, drawing on his or her

expertise, may assist the subject-matter

experts in the framing'of key questions or

sub-issues in such a way as to minimize the

introduction of certain types of bias. For

example, it is known that soliciting input on

the extremes of a probability distribution can

help reduce the tendency for experts to

"anchor" on a central value and, thereby,

reduce bias from over-confidence (a form of

cognitive bias).

A preliminary statement of the issues and

assumptions may be prepared by the

elicitation team, presented to the subjectmatter experts, and then, later, refined, based -on their feedback. Alternatively, the

elicitation team can work with the subjectmatter experts in'structured interactions to

develop the ehtire list of issues and

assumptions. In either case,' it is important

that the subject-matter experts have ample

opportunity to provide input to the

formulation of technical questions and the

decomposition of the problem'into more

tractable sub-issues. Structured interactions

among the subject-matter experts to discuss

and decompose the problem enhance the

likelihood that all share a common

understanding of the problem, relevant

sub-issues, and appropriate boundary

conditions that are used to define the

problem.

By having the subject-matter experts agree on

a common decomposition of a particular

problem, the views of the various experts on

the sub-issues can be compared in a

consistent fashion. Ad hoc or arbitrary

decomposition of a problem, by each expert,

may not permit such comparisons.

Step No. 4-Assembly and Dissemination of

Basic Information

When subject-matter experts are asked to

provide judgments, they are being asked to

25 25 ~~~~~NUTREG
analyze information provided to them about a

given question, issue, or problem. The fact

that expert judgments are being sought is an

indication that there is either a lack of

directly relevant information or major gaps in

the available information about the question

or problem being addressed, and more

importantly, that obtaining such information

or closing critical gaps, using experimental or

other traditional approaches, is not feasible.

The goal of the elicitation, therefore, is to

identify the true state of uncertainty within

the scientific community-be it aleatory

(stochastic) or epistemic (state-of-knowledge)

uncertainties (see SSHAC, 1995; pp. 13-14).

Accordingly, it is reasonable that, as a starting

point, the sponsors of the elicitation should

assist the elicitation team by assembling a

preliminary body of basic information,"'

which, in the view of the sponsors, is

necessary to allow the subject-matter experts

to arrive at their judgments. It is important to

recognize, however, that the judgments of the

subject-matter experts may be influenced by

the type of information they receive, and the

manner in which that information is

presented. Information that covers the full

range of views (i.e., information that is

all-inclusive) on the subject of the elicitation

should be provided. Biasing may be

introduced at this very influential point, and

credibility of the elicitation could be reduced,.

if a suitably broad range of available

information on a particular issue or sub-issue

is not made available.

As the subject-matter experts interact to

identify issues and decompose the problem

(Step No. 3), they may identify additional

data and information they will need to assess

the relevant sub-issues. Therefore, feedback

from the subject-matter experts and

suggestions of additional sources of

information should be solicited before the

elicitations. It is important that data and

information identified as necessary by the

subject-matter experts are provided in a

uniform, balanced, and timely fashion to all

16e.g., knowledge base-can be in the form of data, evidence, models,

analyses, parameters, or statistics

members of the expert panel as they prepare

for the elicitations.

Step No. 5-Pre-Elicitation Training

Because many subject-matter experts are

unfamiliar with the purpose and mechanics

of formal expert elicitation, it is important

that sufficient training be provided by the

elicitation team to adequately prepare them

for the elicitation of their technical

judgments. This training should:

(a) familiarize them with the elicitation

process; (b) educate them in uncertainty

encoding and the expression of their

judgments using subjective probability;

(c) provide them practice in formally

articulating their judgments as well as

explicitly identifying their associated

assumptions and rationale; and (d) educate

them with regard to possible biases that

could be present and influence their

judgments.

Before conducting an elicitation, the

subject-matter experts should have a clear

understanding of the objective of the

elicitation, the reason for obtaining their

judgments through a formal process, and the

manner in which their judgments will be

used. In most expert elicitations, the

subject-matter experts are specialists with

extensive knowledge and experience in a

highly refined, but perhaps very narrow field

of study. If, for example, the purpose of the

elicitation is to obtain probability judgments

that are to serve as input to a performance

assessment, it is important that the

subject-matter experts receive some basic

overview training in performance assessment

to provide a context for how their specific

expert judgments and any associated

mathematical functions or distributions will

be used.

As a designated member (or members) of the

elicitation team explains the purpose and

process of the elicitation, it should be

emphasized that the goal is not consensus,

but rather a realistic description of the true

state of scientific knowledge and uncertainty

about the subject area in question. 'Paining

should consist of familiarizing the

subject-matter experts with the elicitation

process; motivating them to provide

NUREG-1563 26
judgments; giving them practice in formally

expressing their judgments'as well as the

assumptions and rationale'for the judgments (especially uncertainty encoding); and

educating them about possible biases that

could be present and influence the

judgments.

One important aspectof training is the

conduct of rehearsal sessions to assist the

subject-'matter experts in becoming familiar

and comfortable with offering judgments and

the corresponding underlying reasoning and

assumptions. Training helps ensure that the

judgments-represent the subject-matter

experts actual state of knowledge about the

problem of interest.

The most common method of encoding

uncertainty associated with an expert's

judgment is through elicitation of a'subjective

probability distribution. When subject-matter

experts are asked to express degrees of belief

in terms of subjective probabilities, they must

become familiar with the techniques that will

allow them to perform this task. Rehearsal

sessions provide'the means for the

subject-matter experts to practice the

application of these techniques.

Familiarizing the subject-matter experts with

possible biases that maybe present and

influence their opinions is another reason for

conducting elicitation training. There are two

general classes of bias: motivational and

cognitive. Motivational biases occur because a

subject-matter expert has a vested interest in

'an issue and consciously or'unconsciously

distorts his or her judgment. Exarples of

circumstances that might contribute to

motivational bias were noted'under'Step No.

'2 (above). Cognitive biases occur because of a

failure to process, aggregate, or integrate the

available data and information: Motivational

biases can generally be reduced, or at least

mitigated, by careful selection of the

subject-matter experts and by sensitizing the

'subject-matter experts to the motivational

bias' potential for influence. Cognitive biases

are more difficult to address, and to date, the

'best approach 'to 'deal with them is to become

familiar with and practice the application of

debiasing techniques during elicitation

training sessions.

-In addition to their expertise, it should also

be noted that subject-matter experts, themselves, are likely to bring preconceived

notions (e.g., cognitive biases) to the

questions or problems they are addressing. These are' most often the result of many years

of involvement in research in the specific area

in which their judgments are sought-one of

the main reasons why'specific individuals are

recruited in the first place. It is inevitable

that these notions or biases will significantly

influence the expert's judgments. Because of

their extensive experience, subject-matter

experts may tend to focus quickly on

solutions for problems of a similar nature

without closely examining the manner in

which the current problem may differ from

past experiences and the impact those

differences should have on their judgments of

the problem at hand. Slovic (1991) concluded

that most individuals have great difficulty in

accepting information that does not confirm

prior experience, and generally tend to

discard information that does not fit within

their reasoning paradigms. Morgan and

Henrion (1990) suggest that subject-matter

experts need to be presented with, and

required to consider, information that

challenges 'their conventional wisdom. These

investigators state that subject-matter experts

are most likely to exhibit over-confidence and

bias when they fail to examine information

that supports a point of view different from

their own. If, instead,'subject-matter experts

explicitly consider and address information

that contradicts their prior judgments,

Morgan and Henrion argue that

over-confidence biases are reduced and

overall judgments tend to improve. For these

reasons, normative experts should challenge

the subject-matter experts to consider afresh

conflicting information, especially when new

information becomes available that could

result in a re-evaluation of earlier judgments.

Step N. 6-Elicitation of Judgments"

The actual elicitation of the judgments is the

climax of the process for the individual

subject-matter experts'(6r teams' of

subject-matter experts-i.e., all activities

preceding the elicitation were aimed at

preparing for it). The elicitation must be

tailored to the specific question or issue at

hand, the type of judgments required (e.g.,

27 NUREG-1563
identification of events and phenomena,

subjective probabilities, probability

distributions, etc.); the resources available for

the elicitation; the availability of the

subject-matter experts; and the individual

professional preferences (e.g., the scheduling

and length of elicitation sessions, format for

conducting the elicitations, etc.); These

factors may influence the methods and

techniques employed in the elicitation.

Regardless of the manner in which the

elicitation is carried out, reviewers should be

able to discern not only the judgments

themselves, but also the reasons,

assumptions, approaches, and information

that each of the subject-matter experts used.

A dry run or rehearsal of the elicitation is

very beneficial to familiarize the elicitation

team with the procedure. This rehearsal

should occur in advance of the individual

elicitation sessions.

Step No. 7-Post-Elicitation Feedback

The subject-matter expert (or teams of

subject-matter experts) should be given

individual feedback from the elicitation team

on the results of the elicitation as soon as

practical after the elicitation sessions are

completed. In particular, they should be

provided with numerical, graphical, and/or

other useful representations of their

judgments. The subject-matter experts should

be allowed to revise their judgments based on

the feedback. However, the rationale for any

revisions should be carefully documented.

The elicitation team should seek confirmation

of the conclusions from each subject-matter

expert, but guard against attempting to force

consensus or influence their outcome, during

the individual feedback session, if disparate

opinions exist. Finally, this step also allows

the elicitation team members the opportunity

to verify data codification and check for

encoding errors.

Step No. 8-Aggregation of Judgments

(Including Treatment of Disparate Views)

The preceding steps make no assumptions

with respect to whether individual expert

judgments (or teams of judgments) rather

than combined or aggregated results are

ultimately needed. The advantages of

focusing on the individual judgments, up to

this point, include traceability of specific

judgments to an individual and the voicing of

diverse views and perspectives that increase

the likelihood that the actual state of

knowledge on a given problem or issue has

been captured. The use of multiple

subject-matter experts as a means to capture

the existing diversity of opinion about the

answer to a given question or the solution of

a problem of interest gives rise to multiple

judgments-typically at least one different

judgment for each subject-matter expert. In

those cases where a similar or the same

conclusion is arrived at from different paths,

the confidence in this conclusion is obviously

enhanced.

To render differing judgments from multiple

subject-matter experts useful for certain

practical analyses and assessments, or for

discrete input to a performance assessment

calculation, it may be necessary to aggregate,

or combine, in some fashion, the individual

judgments. Two general approaches to

combining expert judgments are commonly

identified as behavioral and mechanical

aggregation. Behavioral aggregation usually

entails the bringing together of the

subject-matter experts to discuss and

combine their judgments. Such interactions

allow the thinking, logic, and experience base

of the different experts to be exchanged. This

may bring about some reconciliation of

differences and result in a single consensus

representation of the state of knowledge, or it

may minimize the differences among experts.

At a minimum such interactions should

reduce the potential for unintentional

disagreement. The behavioral approach is

most beneficial when subject-matter experts

have basic differences in fundamental

assumptions, on which their judgments are

based, which have not been made explicit.

Interactions among the differing experts can

thereby illuminate these assumptions and

may lead to a more thorough understanding

and documentation of conflicting approaches.

Mechanical aggregation techniques (also

known as analytical aggregation) consist of

logic, and formulas consistent with that logic,

that have been developed by normative

experts for combining individual judgments.

NUREG-1563 28
Individual judgments are combined

mathematically such that the sum of the

weighted individual judgments is normalized

(e.g.,' equal to one). Among the obvious

advantages of mechanical combination are

ease of use,' amenability to extensive

sensitivity analyses, and the fact that

individual experts need have no influence on

the judgments of other experts, after

elicitation. The most common and

straightforward mechanical aggregation is a

simple average that assigns equal weights

(parity) to the judgment of each expert.

However, differential weighting techniques

have also been used'to account for relative

expertise or experience of individual experts

(see PRA Working Group, 1994; pp.

C-139-C-142).

A third approach to aggregation is one'

'recently discussed by SSHAC (1995), whereby

b6th the behavioral (judgrniental)'and

'mechanistic schemes are "blended" or

combined. In this approach, the

mathematical models used to produce

preliminary elicitation results are shown to

the experts, along with an explanation of the

'assumptions used to construct the models.

The experts are subsequently 'asked if they

wish' to revise their judgments, given the

knowledge about the consequences of the

results, and they thus develop a "consensus"

representation or aggregate distribution

behaviorally based on their revised

judgments.

In the stepwise approach to elicitation, the

value of structured, iterative, interactions

among the subject-matter experts is

recognized. The staff believes that

interactions, among the experts, properly

structured to permit exchange of reasoning,

data, and assumptions, may outweigh the

potential disadvantage previously ascribed to

a behavioral approach, namely that some

experts may be dominated or "forced" to

suppress their ideas and contribute to an

artificial consensus. Should interaction

among the experts, after the individual

elicitations, result in any changes of

judgments by the individual experts (as in the

manner cited by SSHAC (1995), above (e.g.,

repeating Step Nos. 6-8 after some initial

integration has been achieved)), the

'descriptions and implications of the changes

should be included in updated

representations of the individual experts'

states of knowledge. If such interactions lead

to a commonly held representation of the

state of knowledge, then the representation of

each individual should also reflect the

representation for the group. More

commonly, however, residual differences

between the individual experts will persist,

and these can then be'combined using

mechanical techniques.

It is not the intent of the NRC staff to

prescribe any particular algorithm or

aggregation technique, or require aggregation

itself. Choice of an appropriate method or

methods of aggregation may be highly

issue-specific. That being said, however, the

staff believes that adherence to a step-wise

elicitation process similar to that identified in

this technical position will tend to foster

conditions where equal weighting of

individual judgments'may be most

appropriate. Regardless of which aggregation

techniques are ultimately selected, however,

sufficient documentation must be provided to

trace the impact of each individual

subject-matter opert's judgment on the

consolidated position. It cannot be

emphasized enough that, because of the

reviewer's potential need to examine an

individual expert's judgments and reasoning

bases, the professional judgment of each

subject-matter expert must be explicitly

documented as opposed to that of a person

who is a panel spokesman or facilitator. A

unanimous, consensus, or summary opinion

without such documentation, will generally

not be suitable to support a compliance

demonstration, for purposes of licensing.

Step No. 9-Documentation

An essential element of a formal elicitation

process is thorough documentation of all

aspects of the process, the judgments

acquired, and the rationale and basis for the

judgments. The reasons for documenting the

use of expert judgment for technical problems

are derived from the following objectives:

(a) to improve decision-making associated

with public policy; (b) to enhance

communication; (c) to facilitate peer review,

appraisal, and acceptance; (d) to recognize

29 29 ~~~~NUREG-
and minimize biases in expert judgment;

(e) to indicate the current state of knowledge

about important technical and scientific

matters; and (f) to provide a basis for

updating that knowledge.

Since credibility and acceptability are

objectives of any expert elicitation associated

with controversial issues in the licensing and

public acceptance of a repository, as

suggested by the NWTRB (1990a), an

organized, thoroughly documented procedure

allows reviewers to reconstruct the logic and

events involved in the elicitation and use of

expert judgment The availability of such

documentation supports a broader

understanding and acceptance of what was

undertaken. In addition, it makes possible

technical discussions in terms of underlying

principles rather than just the individual

outcomes. Comprehension of the elicitation

results and the utility of their. use will be

greatly enhanced by the use of a uniform and

consistent reporting format for documenting

the elicitation of each subject-matter expert.

Documentation is a continuous task that

begins as soon as an issue is identified as a

candidate for expert judgment elicitation.

Precise and complete documentation is

pivotal to the success of the elicitation (and

ultimately, acceptance of the results). For

example, the documentation should include a.

discussion of all steps in the elicitation

procedure. Each step should be described

and the results presented. Moreover, the

documentation should also reflect what

specific information was used or relied on by

the experts (e.g., rationale) to reach a

particular judgment (reached in Step No. 4).

As emphasized above, the results should be

provided for each subject-matter expert as

well as for any aggregated judgments.

(3) Elicited Judgments Should Be Updated as

Warranted

When new data or information becomes available

before license application submittal, it could

potentially change a DOE position with regard to

the design and perhaps the performance of the

geologic repository. To the extent practicable, any

potential license application should address the

significance and impact that any new information

might have on the validity of all previously

existing data and elicited judgments used. If the

impacts are determined to be significant, then the

data and expert judgments should be updated to

incorporate the new data or information, as the

information becomes available. Of course, the new

information may resolve the issue by providing

the objective data needed and thus obviate the

need for a new elicitation.

The methods of updating the expert judgments

range from the use of Bayes' Theorem, for

statistical updating, to conducting another set of

individual elicitations for the same or a different

set of experts. Whichever method is used for

incorporating the new data or information into

the existing expert opinions, it should be

thoroughly documented to provide a transparent

view of the updating process and resulting

judgments.

NUREG-1563 30
-

5 REFERENCES

Altman, WD., et al., "Peer Review for High-Level

Nuclear Waste Repositories," U.S. Nuclear

Regulatory Commission, Generic Technical

Position, NUREG-1297, February 1988.

Andrews, R.W, T.F. Dale, and J.A. McNeish,

"Total-System Performance Assessment-1993:

An Evaluation of the Potential Yucca Mountain

Repository," Las Vegas, Nevada, INTERA, Inc.,

March 1994. [Prepared for the U.S. Department

of Energy/Yucca Mountain Site Characterization

Project.]

Barnard, R.W, et al., "TSPA: 1991 An Initial

Total-System Performance Assessment for Yucca

Mountain," Albuquerque, New Mexico, Sandia

National Laboratories, SAND91-2795, September

1992. [Prepared for the U.S. Department of

Energy/Yucca Mountain Site Characterization

Project.]

Bernreuter, D.L., C. Minichino, and the TERA

Corporation, "Seismic Hazard Analysis," U.S.

Nuclear Regulatory Commission, NUREG/

CR-1582, 5 vols., 1980-83. [Prepared by the

Lawrence Livermore National Laboratory and the

TERA Corporation.]

Bernreuter, D.L., et al., "Seismic Hazard

Characterization of the Eastern United States,"

Livermore, California, Lawrence Livermore

National Laboratory, UCID-20421, 2 vols., April

1985. [Prepared for the U.S. Nuclear Regulatory

Commission.]

Bernreuter, D.L., J.B. Savy, and R.W Mensing,

"Seismic Hazard Characterization of the Eastern

United States: Comparative Evaluation of the

LLNL and EPRI Studies," U.S. Nuclear

Regulatory Commission, NUREG/CR-4885, May

1987. [Prepared by the Lawrence Livermore

National Laboratory.]

Bernreuter, D.L., et al., "Seismic Hazard

Characterization of 69 Nuclear Power Plant Sites

East of the Rocky Mountains," U.S. Nuclear

Regulatory Commission, NUREGICR-5250,

January 1989. [Prepared by the Lawrence

Livermore National Laboratory.]

Bonano, E.J., et al., "Elicitation and Use of Expert

Judgment in Performance Assessment of HighLevel Radioactive Waste Repositories," U.S.

Nuclear Regulatory Commission, NUREG/

CR-5411, May 1990. [Prepared by the Sandia

National Laboratories.]

Clemen, R.T, and R.L. Winkler, "Limits for the

Precision and Value of Information from

Dependent Sources," Journal of Operations

Research, 33:427-442 [1985].

Chhibber, S., G. Apostolakis, and D. Okrent, 'A

Taxonomy of Issues Related to the Use of Expert

Judgments in Probabilistic Safety Studies,"

Reliability Engineering and System Safety, 38:2745

[1992].

Chhibber, S. and G. Apostolakis, "Some

Approximations Useful to the Use of Dependent

Information Sources," Reliability Engineering and

System Safety, 42:67-86 [1993].

Code of Federal Regulations, "Disposal of

High-Level Radioactive Wastes in Geologic

Repositories," Part 60, Chapter I, Title 10,

"Energy."

Code of Federal Regulations, 'Appendix A to Part

100-General Design Criteria for Nuclear Power

Plants," in "Reactor Site Criteria," Part 100,

Chapter I, Title 10, "Energy."

Codell, R.B., et al., "Initial Demonstration of the

NRC's Capability to Conduct a Performance

Assessment for a High-Level Waste Repository,"

U.S. Nuclear Regulatory Commission,

NUREG-1327, May 1992.

Coppersmith, KJ., R.C. Perman, and R.R.

Youngs, "Earthquakes and Tectonics Expert

Judgment Elicitation Project," Palo Alto,

California, Electric Power Research Institute,

EPRI TR-102000, February 1993. [Prepared by

Geomatrix Consultants, Inc.]

Dennis, A.W (ed.), "Exploratory Studies Facility

[ESF] Alternatives Study: Final Report,"

Albuquerque, New Mexico, Sandia National

Laboratories, SAND91-0025, 2 vols, September

1991. [Prepared for the U.S. Department of

Energy. Also see A.L. Stevens and L.S. Costin,

31 NUREG-1563
Findings of the ESF Alternatives Study,. -.

Albuquerque, New Mexico, Sandia National

Laboratories, SAND90-3232, March 1991.]

DeWispelare, A.R., et al., "Expert Elicitation of

Future Climate in the Yucca Mountain VicinityIterative Performance Assessment Phase 2.5," San

Antonio, Texas, Center forNuclear Waste

Regulatory Analyses, CNWRA 93-016, August1993. [Prepared for the U.S. Nuclear Regulatory

Commission.]

DeWispelare, A.R., et al, "Background Report on

the Use and Elicitation of Expert Judgment," San

Antonio, Texas, Center for Nuclear Waste

Regulatory Analyses, CNWRA 94-019, September

1994. [Prepared for the U.S. Nuclear Regulatory

Commission.]

DeWispelare, A.R., and EJ. Bonano, "Input to

the Draft Staff Technical Position on Elicitation of

Expert Judgment," San Antonio, Texas, Center for

Nuclear Waste Regulatory Analyses, CNWRA

95-006, March 1995. [Prepared for the U.S.

Nuclear Regulatory Commission.]

Fehringer, D. J., and S.M. Coplan, "Uncertainty in

Regulatory Decision-Making," in American - Nuclear Society/American Society of Civil

Engineers, High-Level Radioactive Waste Management: Proceedings of the Third International

Conference, April 12-16, 1992, Las Vegas, Nevada,

1:106-109 [1992].

Gorham-Bergeron, E.D., et al., "Evaluation of'

Severe Accident Risks: Methodology for the

Accident Progression, Source Term, Consequence,

Risk Integration, and Uncertainty Analyses," U.S.

Nuclear Regulatory Commission'NUREGf

CR-4551 (Draft Revision 1), Vol. 1 [1986]..

[Prepared by the Sandia National Laboratories.]

Gorham, E.D., et al., "Evaluation of Severe

Accident Risks: Methodology for the ,

Containment; Source Term; Consequence, and'

Risk Integration Analyses,".U.S Nuclear

Regulatory Commission, NUREG/CR-4551

(SAND8S6-1309), Vol. -1, Revision 1, December

1993. [Prepared by the Sandia National

Laboratories.]

Harper, .T, et at, "Probabilistic Accident

Consequence Uncertainty Analysis; A Joint

Report Prepared by U.S. Nuclear Regulatory

- Commission and Commission of European

Communities, U.S. Nuclear Regulatory

Commission, NUREG/CR-6244, 3 vols., January

1995. [Prepared by the Sandia National

Laboratories].

Holonich, JJ., Office of Nuclear Material Safety

and Safeguards/Division of High-Level Waste

Management, letter to J.P. Roberts, U.S.

Department of Energy/Office of Civilian

Radioactive Waste Management [Subject: "Review

of Report of Early Site Suitability Evaluation of

the Potential Repository Site at Yucca Mountain,

Nevada"], U.S. Nuclear Regulatory Commission,

July 22, 1992.

Hora, S.C., and R.L. Iman, "Expert Opinion in

Risk Analysis: The NUREG-1L50 Methodology,"

Nuclear Science and Engineering, 102 323-331

[1989].

Hora, S.C., 'Acquisition of Expert Judgment:

Examples from Risk Assessment," Journal of

Energy Engineering, 118:136-148 [1993]. :

Johnson, R.L, "Overall Review Strategy for the

Nuclear Regulatory Commission's High-Level

Waste Repository Program," U.S. Nuclear

Regulatory Commission, NUREG-1495,

November 1994.

Kaplan, S., "Expert Information Versus Expert

Opinions: Another Approach to the Problem of

Eliciting/Combining!Using Expert Knowledge in

PRA," Journal of Reliability Engineering and

System Safety, 35:61-72 [1992].

Kastenberg, WE., et a., Findings of the Peer

Review Panel on the Draft Reactor Risk

Document-NUREG-1150," Livermore, CA,

Lawrence Livermore National Laboratory,

UCID-21346,- May 1988. [Prepared for the U.S.

* Nuclear Regulatory Commission as

NUREG/CR-5113.]

Kouts, H., et al., "Methodology for Uncertainty

Estimation in NUREG-1150 [Draft]: Conclusions

of a Review Panel," Upton, New York, Brookhaven National Laboratory, BNL-NUREG52119, December 1987. [Prepared for the U.S.

Nuclear Regulatory Comrnission as

NUREG/CR-5000.]

Linehan, JJ., Office of Nuclear Material Safety

and Safeguards/Division of High-Level Waste

NUREG-1563 32
Management, letter to R. Stein, U.S. Department

of Energy/Office of Civilian Radioactive Waste

Management [Subject: "U.S. Department of

Energy Plans for Conducting the Calico Hills

Risk/Benefit Analysis"], U.S. Nuclear Regulatory

Commission, January 5, 1990.

Linehan, JJ., Office of Nuclear Material Safety

and Safeguards/Division of High-Level Waste

Management, letter to D.E. Shelor, U.S.

Department of Energy/Office of Civilian

Radioactive Waste Management [Subject

"Transmittal of Meeting Summary for the U.S.

Nuclear Regulatory Commission-U.S.

Department of Energy Meeting on the Calico

Hills Risk/Benefit Analysis and the Exploratory

Shaft Facility Alternatives Study"], U.S. Nuclear

Regulatory Commission, August 27, 99L

Meyer, M.A. and J.M. Booker, "Eliciting and

Analyzing Expert Judgment: A Practical Guide,"

U.S. Nuclear Regulatory Commission,

NUREG/CR-5424, January 1990. [Prepared by

the Los Alamos National Laboratory.]

Morgan, M.G., and M. Henrion, Uncertainty: A

Guide to Dealing with Uncertainty in Quantitative

Risk and PolicyAnalysis, Cambridge,

Massachusetts, Cambridge University Press, 1990.

National Research Council, "Probabilistic Seismic

Hazard Analysis," Washington, D.C., National

Academy Press, Commission on Physical

Sciences, Mathematics, and Resources, 1988.

National Research Council, "Rethinking HighLevel Radioactive Waste Disposal: A Position

Statement of the Board on Radioactive Waste

Management," Washington, D.C., National

Academy Press, Commission on Geosciences,

Environment, and Resources, July 1990.

National Research Council, "Technical Bases for

Yucca Mountain Standards," Washington, D.C.,

National Academy Press, Commission on

Geosciences, Environment, and Resources, July

1995a.

National Research Council, "Ward Valley: An

Examination of Seven Issues in Earth Sciences

and Ecology," Washington, D.C., National

Academy Press, Commission on Geosciences,

Environment, and Resources, [May] 1995b.

National Research Council, "Review of U.S.

Department of Energy Technical Basis Report for

Surface Characteristics, Preclosure Hydrology,

and Erosion," Washington, D.C., National

Academy Press, Commission on Geosciences,

Environment, and Resources, [December] 1995c.

Nuclear Waste Technical Review Board, First

Report to the US. Congress and the U.S. Secretary

of Energy, U.S. Government Printing Office,

March 1990a.

Nuclear Waste Technical Review Board, Second

Report to the US. Congress and the US. Secretary

of Energy, U.S. Government Printing Office,

November 1990b.

Nuclear Waste Technical Review Board, Fourth

Report to the US. Congress and the US. Secretary

of Energy, U.S. Government Printing Office,

December 1991.

Nuclear Waste Technical Review Board, Fifth

Report to the US. Congress and the US. Secretary

of Energy, U.S. Government Printing Office, June

1992.

Nuclear Waste Technical Review Board, Report to

the US. Congress and the US. Secretary of Energy

January to December 1993, U.S. Government

Printing Office, May 1994. [Tenth in a series of

reports]

Nuclear Waste Technical Review Board, Report to

the US. Congress and the US. Secretary of Energy:

1994 Findings and Recommendations, U.S.

Government Printing Office, March 1995.

[Eleventh in a series of reports]

Park, J.R., et al., "The Nuclear Regulatory

Commission Strategic Plan for Postclosure

Performance Assessment Activities for the

High-Level Waste Geologic Repository: Draft

Report," San Antonio, Texas, Center for Nuclear

Waste Regulatory Analyses, Letter Report, March

1994. [Prepared for the U.S. Nuclear Regulatory

Commission.]

PRA Working Group, 'A Review of NRC Staff

Uses of Probabilistic Risk Assessment," U.S.

Nuclear Regulatory Commission, NUREG-1489,

March 1994.

Science Applications International Corporation,

"Report of Early Site Suitability Evaluation of the

33 NUREG-1563
Potential Site at Yucca Mountain, Nevada," Las

Vegas, Nevada, 2 vols., SAIC-91/8000O1, January

1992. [Prepared for the U.S. Department of

Energy/Yucca Mountain Site Characterization

Project.]

Seismic Methods Peer Review Panel, "Final

Recommendations of the Peer Review Panel for

the Use of Seismic Methods for Characterizing

Yucca Mountain and Vicinity," Report submitted

to the U.S. Department of Energy/Office of

Civilian Radioactive Waste/Yucca Mountain Site

Characterization Project, January 22, 1991.

Seismicity Owners Group/Electric Power

Research Institute, "Seismic Hazard Methodology

for the Central and Eastern United States [Final

Report]," Palo Alto, California, EPRI NP-4726, 10

vols., July 1986. [Prepared by Risk Engineering, Inc., Woodward-Clyde Consultants, Inc.,

Geomatrix Consultants, Inc., and the CYGNA

Corporation.]

Senior Seismic Hazard Analysis Committee,

"Recommendations for Probabilistic Seismic

Hazard Analysis: Guidance on Uncertainty and

Use of Expert," Livermore, California, Lawrence

Livermore National Laboratory,

UCRLID-122160, 2 vols., August 1995.

[Sponsored by the U.S. Department of Energy,

the U.S. Nuclear Regulatory Commission, and the

Electric Power Research Institute. Also expected

to be published as NUREG/CR-6372.]

Schenker, A.R., et al., "Stochastic Hydrogeologic

Units and Hydrogeologic Properties Development

for Total-System Performance Assessments,"

Albuquerque, New Mexico, Sandia National

Laboratory, SAND94-0244, September 1995.

[Prepared for the U.S. Department of Energy/

Yucca Mountain Site Characterization Project.]

Slovic, P., "Beyond Numbers: A Broader

Perspective on Risk Perception and Risk

Communication," in D.G. Mayo and R.D.

Hollander (eds.), Acceptable Evidence. Science

and Values in Risk Management, New York,

Oxford University Press, 1991, pp. 48-65.

Sobel, PA.,- "Revised Livermore Seismic Hazard

Estimates for Sixty-Nine Nuclear Power Plant:

Sites East of the Rocky Mountains Final

Report," U.S. Nuclear Regulatory Commission,

NUREG-1488, April 1994.

lbst Prioritization Thsk Force, "Testing Priorities

at Yucca Mountain: Recommended Early Tests to

Detect Potentially Unsuitable Conditions for a

Nuclear Waste Repository," U.S. Department of

Energy/Office of Civilian Radioactive Waste

Management, 2 vols., March 1, 1991.

Trauth, KM., S.C. Hora, and R.V. Guzowski,

'Expert Judgment on Markers to Deter

Inadvertent Human Intrusion into the Waste

Isolation Pilot Plant," Albuquerque, New Mexico,

Sandia National Laboratories, SAND92-1382,

November 1993.

Tschoepe,' E., Iml, and L.R. Abramson,

"'Substantially Complete Containment'.

Elicitation Report," San Antonio, Texas, Center

for Nuclear Waste Regulatory Analyses, CNWRA

92-016, August 1992. [Prepared for the U.S.

Nuclear Regulatory Commission.]

U.S. Environmental Protection Agency,

"Environmental Standards for the Management of

Spent Nuclear Fuel, High-Level and THansuranic

Wastes [Final Rule]," Federal Register, Vol. 50,

No. 182, September 19, 1985, pp. 38066 - 38089.

U.S. Department of Energy, 'A Multiattribute

Utility Analysis of Sites Nominated for

Characterization for the First Radioactive Waste

Repository-A Decision-Aiding Methodology,"

Office of Civilian Radioactive Waste

Management, DOE/RW 0074, May 1986.

U.S. Department of Energy, Chapter 8,

Section 8.1, Rationale," in "Site Characterization

Plan, Yucca Mountain Site, Nevada Research and

Development Area, Nevada," Office of Civilian

Radioactive Waste Management, Nevada

Operations Office/Yucca Mountain Project Office,

Nevada, DOE/RW-0199, Vol. IV, Part B,

December 1988.

U.S. Department of Energy, "Record Memorandum: Risk/Benefit Analysis of Alternative

Strategies for Characterizing the Calico Hills Unit

at Yucca-Mountain (Revision 0), Office of Civilian

Radioactive Waste Management, 2 vols.,

YMP/91-6, January 1991a.

U.S. Department of Energy, "Unsaturated Zone

Hydrology Peer' Review Record Memorandum,"

Office of Civilian Radioactive Waste :

Management/Yucca Mountain Project Office,

June 24, 1991b.

NUREG-1563 34
U.S. Department of Energy, et al., "Risk Assessment: A Survey of Characteristics, Applications,

and Methods Used by Federal Agencies for

Engineered Systems (Submitted to the Federal

Coordinating Council for Science, Engineering,

and Technology, Ad Hoc Working Group on Risk

Assessment)," Washington, D.C., U.S. Nuclear

Regulatory Commission, November 1992.

[Contributions from eight Federal agencies,

including NRC.]

U.S. Department of Energy, "Site Characterization Progress Report: Yucca Mountain,

Nevada, October 1, 1993-March 31, 1994

(No. 10)," Office of Civilian Radioactive Waste

Management, DOE/RW-0450, October 1994.

U.S. Department of Energy, "Site Characterization Progress Report: Yucca Mountain,

Nevada, April 1, 1994-September 30, 1995

(No. 11)," Office of Civilian Radioactive Waste

Management, DOE/RW-0463, March 1995.

U.S. Nuclear Regulatory Commission, "Reactor

Safety Study: An Assessment of Accident Risks in

U.S. Commercial Nuclear Power Plants,"

WASH-1500 (NUREG-75/014), October 1975.

U.S. Nuclear Regulatory Commission, 'Disposal

of High-Level Radioactive Wastes in Geologic

Repositories [Final Rule]," Federal Register,

Vol. 48, No. 120, June 21, 1983, pp. 28194-28229.

U.S. Nuclear Regulatory Commission, "Disposal

of High-Level Radioactive Wastes in Geologic

Repositories [Conforming Amendments]," Federal

Register, Vol. 51, No. 118, June 19, 1986, pp. 22286

-22301.

U.S. Nuclear Regulatory Commission, "NRC Staff

Site Characterization Analysis of the Department

of Energy's Site Characterization Plan, Yucca

Mountain, Nevada," Office of Nuclear Material

Safety and Safeguards, NUREG-1347, August

1989.

U.S. Nuclear Regulatory Commission, "Severe

Accident Risks: An Assessment for Five U.S.

Nuclear Power Plants-Final Summary Report,"

Office of Nuclear Regulatory Research,

NUREG-1150, 3 vols., December 1990.

U.S Nuclear Regulatory Commission, "Reactor

Site Criteria Including Seismic and Earthquake

Engineering Criteria for Nuclear Power Plants

and Proposed Denial of Petition from Free

Environment, Inc., et al. [Proposed Rule and

Proposed Denial of Petition]," Federal Register,

Vol. 59, No. 199, October 17, 1994, pp. 52255 -

52273.

U.S. Nuclear Regulatory Commission, "Use of

Probabilistic Risk Assessment Methods in

Nuclear Regulatory Activities [Final Policy

Statement]," Federal Register, Vol. 60, No. 158,

August 16, 1995, pp. 42622 - 42630.

U.S. Nuclear Regulatory Commission,

'Availability of Draft Branch Technical Position

on the Use of Expert Elicitation in the High-Level

Waste Program [Notice of Availability]," Federal

Register, Vol. 61, No. 40, February 28, 1996, pp.

7568 - 7569.

Wescott, R.G., et al. (eds.), "NRC Iterative

Performance Assessment Phase 2: Development of

Capabilities for Review of a Performance Assessment for a High-Level Waste Repository," U.S.

Nuclear Regulatory Commission, NUREG-1464,

October 1995.

Wilson, M.L., et al., "Total-System Performance

Assessment for Yucca Mountain-SNL Second

Iteration (TSPA-1993)," Albuquerque, New

Mexico, Sandia National Laboratory, SAND93-

2675, 2 vols., April 1994. [Prepared for the U.S.

Department of Energy/Yucca Mountain Site

Characterization Project.]

Winkler, R.L., S.C. Hora, and R.G. Baca, "The

Quality of Expert Probabilities Obtained through

Formal Elicitation Techniques," San Antonio,

Texas, Center for Nuclear Waste Regulatory

Analyses, Letter Report, September 1, 1992.

[Prepared for the U.S. Nuclear Regulatory

Commission.]

35 NUREG-1563
----

APPENDIX A

GLOSSARY

As used in this guidance:

'Aggregation of judgments" refers to the

combining of the individual elicited judgments of

more than one subject-matter expert to produce a

single judgment, point estimate, range, or

uncertainty distribution.

"Behavioral aggregation methods" use personal

interactions among the subject-matter experts

employing consensus techniques to combine

multiple individual judgments.

"Cognitive bias" occurs when a subject-matter

expert fails to process, include, or integrate the

available data or information available.

"Debiasing techniques" refers to training the

subject-matter experts in dealing with and

reducing the effects of cognitive biases on the

elicitation results. Examples of these techniques

are: (i) familiarity and practice with the elicitation

task; and (ii) awareness of the biases through personal experiences, making use of feedback.

"Elicitation team" refers to the group of

generalists and normative experts conducting and

facilitating the elicitation. This group may also be

referred to as the project team or panel.

"Expert judgment" refers to the data or information provided by a subject-matter expert. It is the

subject-matter experts opinion or belief based on

reasoning. Expert judgments can be evaluations of

theories, models, experiments, or recommendations for further research. Expert judgments may

also be opinion that can be analyzed and

interpreted and can be used in performance

assessment and other technical models. Expert

judgments can be either qualitative or quantitative. Expert judgments can also be judgments about uncertain quantities or judgments about

value preferences. A subject- matter expert may

provide a probability distribution or a point

estimate for an uncertain parameter.

"Expert elicitation" is a formal, highly structured,

and well-documented process for obtaining the

judgments of multiple experts.

"Generalist" is an individual with substantial

technical background in one or more of the

disciplines needed to solve the problem of

interest-but whose understanding of the problem

typically spans beyond the particular discipline- and is well-versed on how the judgments will be

used in the solution of the problem. Generalists

may be selected from the project staff and they work with the normative experts in the conduct of

the elicitation. They serve several roles: (i) propose the problem decomposition; (ii) prepare the

issue definition; (iii) provide assistance to the

subject-matter experts by explaining how their

judgments will be used; and (iv) together with the

normative experts and with input of the subjectmatter experts, orchestrate the final presentation and, where appropriate, aggregation of the elicited

judgments.

"Mechanical aggregation methods" rely on

analytic formulae (such as weighted averaging) to

combine multiple individual judgments.

"Motivational bias" occurs when a subject-matter

expert has a vested interest in an issue, institution,

political agenda, or personal relationship, and

when that vested interest consciously or

unconsciously acts to distort his or her judgment.

"Normative expert" refers to individuals with a

sound theoretical and conceptual knowledge of

probability and practical experience in the

elicitation of judgments from individuals.

Normative experts are well-versed in the

psychological and cognitive processes that

subject-matter experts follow in the analysis of

information to produce the desired judgments.

'"eer review" is frequently described as a form of

expert judgment; however, in the context of this

BTP, peer review does not fall within the aforementioned definition of expert judgment. The staff

recognizes a subtle, yet fundamental, difference

between peer review and expert judgment as used

here. The former refers to judgments provided xegarding the soundness and quality of an existing

solution to a given problem, whereas the latter

refers to judgments which, themselves, give rise to

or contribute to a scientific stance or solution. In

this context, the staff expects expert judgments to

be elicited and used in a manner that would allow

A-1 NUREG-1563
-

the elicitation and the judgments to be the subject

of peer review.

"Subject-matter expert" is the individual from

whom the expert judgment will be elicited. The

subject-matter expert is an individual recognized

by his or her peers as an authority in a specific

subject matter or topic. Subject-matter experts

typically gain recognition as such because of

significant and sustained research in the subject

matter or topic, and their knowledge is believed

by others to represent the current state of the art

in that subject or topic.

NUREG-1563 A-2
APPENDIX B

AN ELICITATION ON FUTURE CLIMATE: LESSONS LEARNED

B-1 Introduction

In 1992-93, the Center for Nuclear Waste

Regulatory Analyses (CNWRA) conducted an

elicitation, to familiarize the U.S. Nuclear

Regulatory Commission staff with a state-of-theart formal expert judgment elicitation. This work

was performed as part of the staff's Iterative

Performance Assessment efforts to develop an

independent license application review capability

(see Appendix D in Johnson, 1994). The subject of

the elicitation was the future climate in the Yucca

Mountain vicinity. The details of this formal

elicitation itself, as well as the results, are

described in DeWispelare et al. (1993).

B-2 Rationale

A formal elicitation was selected for this subject,

for the following reasons:

(1) The state of climate science and modeling

does not support accurate sub-regional

long-term projections based on historic or

current meteorologic data. Expert judgment

is a way to integrate and supplement the

output of general circulation models.

(2) There are great uncertainties associated with

predictions of future climate at Yucca

Mountain. This is especially evident when one

considers that climate varied during the

Quaternary Period (ranging from pluvial to

arid conditions), and that the influences of

such variations on infiltration can potentially

dominate the predicted performance of a

geologic repository for high-level waste.

(3) Because of the state of science in long-term

climate prediction, there exist a number of

conceptual approaches. These range from

general circulation models conditioned with

combinations of past and present meteorologic data, to energy-balance models based on

current physical data, to empirical historic

data used to establish past conditions.

(4) The published record contains a variety of

data and opinion that establishes various

bounding limits, some of which have been

interpreted to be conservative.

Several lessons were garnered, from the climate

elicitation, that have relevance for the process of

expert elicitation, generally:

* The ability to understand and compare the

judgments of each participant in an expert

elicitation is far easier when uniform

procedures are used to elicit judgments and

document results. The generalists all

commented that the clarity and logic that the

subject-matter experts provided in the

judgments and supporting rationale were

attributable to their overall expertise.

* It is possible to have a defensible process for

selection of subject-matter experts. To

address concerns regarding the lack of bias

or independence in the selection of the

subject-matter experts, a documented process

of peer nomination and selection can be

conducted even within a relatively short time

schedule.

* Debiasing training of the subject-matter

experts is essential to a smooth elicitation.

Most subject-matter experts have had only

limited or no experience at producing

consistent subjective probability distributions. All the subject-matter experts agreed

with the normative experts and generalists

that this training was essential to the process.

* The elicitation team had considerable

difficulty in constructing a behaviorally-based

aggregation because of the variation in the

individual judgments and the subject-matter

experts conviction regarding their judgments.

After an attempt at behavioral aggregation, a

mechanical aggregation of the subject-matter

experts judgments was attempted. This

approach was faster and easier to implement

than a behavioral aggregation. When it is

necessary to aggregate the individual

judgments of the subject-matter experts after

they have been elicited, it is efficient to use a

B-1 NUREG-1563
mechanical aggregation scheme. This scheme

can be easily documented to provide

traceability.

* It is valuable to have the subject-matter

experts visit the site. The subject-matter

experts indicated that experience gained

through the site visit was important to their

preparing for the elicitation, and facilitating the post-elicitation validation process. The

subject-matter experts noted that by visiting

the site, they were able to place the data and

research assembled for the elicitation into the

context of the physical setting at Yucca

Mountain, thus leading to what the

subject-matter experts regarded as a more

realistic interpretation of this information.

* Individual documentation is critical to a

successful elicitation. The documentation

consisted of two parts. First, a short paper, which formed the scientific basis for a

particular judgment, served as the reference

for understanding the technical reasoning expressed in the elicitation. The second part

consisted of the elicitation team's documentation of the elicited judgments (e.g.,

probability distributions associated with the

climate variables) and accompanying specific

reasoning resulting from each expert's

elicitation session, to ensure that the rationale

used by each subject-matter expert was

well-understood and expressed consistently.

Video taping of each elicitation session

helped the team to check session notes and

served as a permanent record of each session.

B-3 References

DeWispelare, A.R., et al., "Expert Elicitation of

Future Climate in the Yucca Mountain VicinityIterative Performance Assessment Phase 25," San

Antonio, Texas, Center for Nuclear Waste

Regulatory Analyses, CNWRA 93-016, August

1993.

Johnson, R.L., 'Appendix D: NRC Post-Closure

Performance Assessment Strategy for a

High-Level Nuclear Waste Repository" in "Overall

Review Strategy for the Nuclear Regulatory

Commission's High-Level Waste Repository

Program," U.S. Nuclear Regulatory Commission,

NUREG-1495, November 1994.

NUREG-1563 B-2
APPENDIX C

FINAL COMMISSION POLICYSTATEMENT ON THE USE OF PRA METHODS

IN NUCLEAR REGULATORY ACTIVITIES

C-1 Introduction

The following statement presents the policy that

the U.S. Nuclear Regulatory Commission will

adopt in the use of probabilistic risk assessment

(PRA) methods in nuclear regulatory matters.

This policy was developed because the Commission believed that the potential applications of

PRA methodology could improve public health

and safety decision-making while promoting

stability and efficiency in the regulatory process

and reducing unnecessary burdens on licensees.

After a public workshop, the Policy Statement was

published in draft form in the Federal Register

(NRC, 1994; 59 FR 63389). On receipt and

consideration of public comments, it was

published in final form (see NRC, 1995; 60 FR

42622).

C-2 The Commission Policy (at 60 FR

42628)

1. The use of PRA technology should be

increased in all regulatory matters to the

extent supported by the state of the art in

PRA methods and data and in a manner that

complements NRC's deterministic approach

and supports NRC's traditional defense-indepth philosophy.

2. PRA and associated analyses (e.g., sensitivity

studies, uncertainty analyses, and importance

measures) should be used in regulatory

matters, where practical within the bounds of

the state of the art, to reduce the unnecessary

conservatism associated with current

regulatory requirements, regulatory guides,

license commitments, and staff practices.

Where appropriate, PRA should be used to

support the proposal for additional regulatory requirements in accordance with 10 CFR

50.109 (Backfit Rule"). Appropriate procedures for including PRA in the process for

changing regulatory requirements should be

developed and followed. It is, of course,

understood that the intent of this policy is

that existing rules and regulations shall be

complied with unless these rules and

regulations are revised.

3. PRA evaluations in support of regulatory

decisions should be as realistic as practicable, and appropriate supporting data

should be publicly available for review.

4. The Commission's safety goals for nuclear

power plants and subsidiary numerical

objectives are to be used with appropriate

consideration of uncertainties in making

regulatory judgments on the need for

proposing and backfitting new generic

requirements on nuclear power plant

licensees.

C-3 References

U.S. Nuclear Regulatory Commission, "Use of

Probabilistic Risk Assessment Methods in

Nuclear Regulatory Activities [Proposed Policy

Statement]," Federal Register, Vol. 59, No. 235,

November 8, 1994, pp. 63389-63391.

U.S. Nuclear Regulatory Commission, "Use of

Probabilistic Risk Assessment Methods in

Nuclear Regulatory Activities [Final Policy

Statement]," Federal Register, Vol. 60, No. 158,

August 16, 1995, pp. 42622-42630.

C-1 NUREG-1563
APPENDIX D - DISPOSITION OF PUBLIC COMMENTS1 ON FEBRUARY 28, 1996, DRAFT

BRANCH TECHNICAL POSITION (61 FR 7568)

State of Nevada

The following comments were submitted by the

State of Nevada (see Loux, 1996):

The Agency'for Nuclear Projects/Nuclear

Waste Project Office has reviewed the

subject Draft Branch 'Tchnical Position

(BTP) and finds that'it is, in general, favorably responsive to concerns transmitted

in our letter of July 25, 1995 (see Loux, 1995),

regarding the U.S. Department of Energy's

(DOE's) June 5, 1995, Principles and

Guidelines for Form"al Use of Exjqert Judgment

by the Yucca Mountain Site Characterization

-Project Office (see Brocoum, 1995).

The BTP properly emphasizes' the need to'

formalize and thoroughly document the

process of preparing for and eliciting expert

opiniofi, aggregating judgments, and i

performing follow-up, when appropriate.

However, missingfrom the discussion is

explicit guidance that it is equally important

that DOE's decision, and its basis, to

employ the expert elicitation process, be

thoroughly documented and trans'parent to

-future reviewers. Also;'the documentation of

'the decision should be maintained'as part of

the record of the'expert elicitation process.

It will be particularly important to know at

the time of license application review. . whether cost was a primary consideration in

the decision to employ expert judgment rather than performing theoretical analyses

and/or gathering additional field and

experimental data, as the BTP suggests it

might be (pages 11 and 24). This is

important because the Yucca Mountain

Project has' a continuing history of changing priorities on field and lab data collection

and 'analysis that often is driven from year to

year by available funds. What may seem to

'DOE to be practical and 'necessary one year

can be deemed imiipractical the next year,:

with the issue then set aside as if it were

IThe indented portions of this appendix quote the public comments.

closed and unnecessary to pursue. An

example of this which may persist into a

license application is the question of

additional field study to understand the high hydrologic gradient north of Yucca Mountain. At present, it appears that some

informal expert judgment has determined

that it is not important to understand why

this condition exists, whereas in the past it

has been considered important by DOE

when more money was available to the

project for surface-based testing.

The BTP states the four conditions under

which the use' of expert elicitation should be

considered (page 17 and elaborated on pages

24-25):

(a); Empirical data are not reasonably obtainable, or the analyses are not

'practical to perform;

(b) -Uncertainties are large and significant to

a demonstration of compliance;

(c) More than one conceptual model can

explain, and be consistent with, the

available data; or

(d) Technical judgments are required to

assess whether bounding assumptions or

calculations are appropriately

conservative.

*Throughout the text of the BTP, the use of

expert judgment is described in various

manners that are only in some cases

obviously consistent with the above list, e.g., 'an altemative'when other means of obtaining

data and information'are not practical to

implement; a means of reducing uncertainty; 'an assessment of the state'of scientific

uncertainty; a meais of exploring the state of

knowledge on a particular topic conple- menting and supplementing other sources of

-sientific and technical information; etc. It

would be helpful for the BTP to collect and

discuss all of the various descriptions of the

beneficial use of expert judgment in one ' place in 'order for the reader to better

D-1 D-1 ~~~~NUREG-156
interpret the Staff's apparently broader view

on when the use of expert elicitation might

be appropriate, and to what purpose. This

further discussion possibly could be added

to [Section] 4(1) as a supplement. It also

appears that the final three items in the

above list of conditions are often simply a

subset of the first condition. Each of the

final three conditions can be the result of a

DOE decision that further data collection

and analysis is not practical, but, in some

cases such as the above example, the condition likely could be mitigated by additional

data collection and analysis. The BTP

repeatedly admonishes against substituting

expert judgment for traditional analyses, but

in each case makes cost and practicality a

prominent consideration. As the regulator,

the NRC should be most concerned with

safety considerations, which can only be

derived from a high-quality license application firmly grounded on data and analyses.

As in every regulatory arena, it is the

applicant's responsibility to weigh cost

against quality in the preparation of its

license application, and it is the regulator's

responsibility to judge whether the product

is adequate. The BTP gives helpful guidance

regarding how the regulator might view the

quality of DOE's application, but, [we]

believe, errs in emphasizing cost and

practicality on the part of the applicant as a

potential measure of compromise in

determining the "reasonable assurance" that

the applicant's safety case is adequately

proven. The text of this BTP should be

revised to reflect that the NRC's primary

regulatory role, and highest priority, is to

promote safety. As it stands, the unfortunate

implication in this BTP is that cost and

practicality for the applicant are acceptable

measures against which to weigh safety in

the regulatory proceeding.

And finally, despite DOE's view2 that much

of [Sections] 1 and 2 of the BTP is an

unnecessary review of past experience with

the use of formal expert judgment exercises,

r[he State] believes that it is a useful

description that adds basis to the staff's

positions as set out in [Section] 3 and further

2 Expre:ed during a joint NRC/DOEState of Nevada telephone conference call on April 23, 1996.

discussed in [Section] 4. The use of expert

judgment in a licensing proceeding in this

unique case will remain a prominent issue

throughout, and it will be important to

understand as thoroughly as possible the

current staff's basis for the guidance which

it provided to the applicant and other

parties prior to preparation of the license

application. It must be remembered that, in

the licensing proceeding the product of this

guidance may or may not be found

acceptable, and the original basis for the

guidance could be integral to that decision.

Response

With respect to the State of Nevada's first general

comment, that missing from the guidance is some

discussion that it is important for DOE to

document its decisions, and their bases, when it

employs an expert elicitation process, the staff is

fully aware of this issue and generally shares a

somewhat related concern. In fact, DOE's use of

expert judgment, during site characterization, was

first identified as an issue by the NRC staff in the

course of its review of DOE's Site Characterization Plan (SCP-see DOE, 1988). In its Site

Characterization Analysis (SCA) Comment 3, the

staff expressed the concern that DOE might

"... rely [too] heavily on the use of expert

judgments (e.g., expert elicitations) to supply the

licensing information or to substitute for quantitative analyses" (NRC, 1989; pp. 4-8-4-40).

Because of NRC's regulatory interest and

oversight role in the repository program, the staff

has consistently expressed the view that DOE

needs to document its decision-making record so

as to allow for the identification and resolution, at

the staff level, of any potential licensing issues.

The staff repeated its concerns later, during the

conduct of DOE's site characterization program.

That being said, it is generally recognized that

DOE's repository program has evolved

significantly beyond that described in the 1988

SCP As DOE prepares its future program plans

for site characterization and the repository

Viability Assessment, the staff believes that DOE

understands NRC's overall intent to ensure that

there is transparency in its (DOE's) decisionmaking process and fully expects DOE to provide

sufficient documentation to support its decisions,

including those that relate to the use of formal

expert judgment. Consistent with DOFs own

NUREG-1563 D-2
Principles and Guidelines for Formal, Use of Expert

Judgment by the Yucca Mountain Site Characterization Project Office (hereafter referred to as'the

Principles and Guidelines), the staff also expects

that DOE would prepare the necessary documentation, as recommended by'the State in

its comment. The staff believes that this point is

apparent throughout Section 1 of the BTP and'

acknowledged specifically elsewhere in the BTP

(e.g., in Section 4, refer to the following locations

in the discussion: paragraph 2 (in the introduction); Technical Position 1, paragraph 1; and

Technical Position 2, Step No. 9, paragraph 1).

Therefore, the staff does not agree with the State

that it is necessary to provide further guidance or

to include additional information in the BTP.

NWith respect to the State of Nevada's second

general comment, ". . . to collect and discuss all of

the various descriptions of the beneficial use of

expert judgment in one place. . .'," the staff has no

objection to including'additional information. The

staff has added the following'paragraph to the

introduction of Section 4 of the BTP to address

the State's request:

In closing, it should be noted that this

NUREG discusses the rationale for' and the

potential benefits derived from the use of

expert judgment'in a number of places.

Without intending to limit those discussions,

the following summary is provided. Section 1

describes the ubiquitous nature of expert

judgment in scientific and technical

activities. Section 1.2.1 provides a

description of typical outputs of an expert

elicitation and the potential uses of the

expert judgments in the evaluation of

repository performance assessment. Section

1.3'provides actual examples of the NRC use

of expert judgments, including those used in

PRA. Section 3 provides the' technical

position on conditions warranting the

consideration of a formal expert elicitation.

Later in this section, the staff elaborates on

these conditions and discuss the benefits

derived from obtaining expert judgments.

Finally, Appendix A provides a discussion of

expert judgment from a definitional

perspective and Appendix B provides results

and lessons learned from the

NRC-sponsored formal climate expert

elicitation..

In its third general comment, the State of Nevada

observed that the ". . . implication in this BTP is

that:cost and practicality for the applicant (DOE)

are acceptable measures against which to weigh

safety'in the regulatory proceeding... ." The staff

does not share this view of the implications of the

BTP. For example, paragraph 2 of Section 1.4 of

the BTP states that:

.. . under appropriate circumstances, it is

acceptable to supplement data and analyses

with the opinions of experts as part of the

support for demonstrating compliance with

NRC's geologic disposal regulation, and that,

under certain circumstances, these opinions

can be obtained using a formal and

well-documented process...."

In making this statement, in recognition that

compliance with NRC regulations is the measure

for judging safety in an NRC licensing proceeding,

the staff also provided additional caveats (highlighted) to the BTP (see Section 1.4, paragraph 5):

The NRC staff recognizes that DOE has the

flexibility to determine whether the costs and

benefits of performing an expert elicitation are

advantageous when compared with'the costs

and benefits ofperforming theoretical analyses

andlor gathering additionalfield and

experimental data. That being said, however,

the use of expert elicitation should not be

considered as an acceptable substitute for

traditional analyses based on adequate field

or experimental data, when such data are

reasonably available or obtainable, or the

analyses are practicable to perform. Nor can

the use of a formal elicitation process, even

when conducted in a manner consistent with

guidance provided in this BTPE guarantee that

specific technical conclusions will be accepted

and adopted by the staff a Licensing Board,

the Commission itself or any other party to a

potential HLWlicensing proceeding. Rigid

adherence to a sound elicitation process, in

and of itself, in'no way guarantees that the

resulting judgments will be sufficient to meet

the applicant's burden of proof regarding the

substantive issues addressed by the

elicitation. Nonetheless, expert judgments

obtained through an evidently flawed or

poorly documented process will weaken their

D-3 NUREG-1563
poorly documented process will weaken their

ability to support demonstrations of

compliance.

It should also be noted that nothing in this

BTP precludes the use of expert judgment

obtained through informal means by DOE,

in the preparation of a potential license

application. As has been the case in previous

regulatory activities, the staff will accept for

review the results of formal or informal

judgment so long as the rationale associated

with the judgment is adequate, transparent,

and sufficiently documented.

However, in order to avoid any ambiguity on this

subject, the staff has revised the first sentence of

the fifth paragraph of the "Introduction" to

Section 1 of the BTP to emphasize that NRC's

primary regulatory role and highest priority is to

ensure public health and safety; the revision will

read as follows:

"With this notion in mind, current NRC

policy is to encourage the use of

probabilistic risk assessment (PRA)

state-of-the-art technology and methods as a

complement to the deterministic approach in

nuclear regulatory activities (NRC, 1995; 60

FR 42622)4 and in keeping with the

Commission's paramount responsibility to

protect public health and safety...."

2EPA's 1985 HLW standards (50 FR 38066) adopted a probabilistic perspective when making compliance determinations Because of the uncertainties inherent in the

geologic disposal of nuclear waste, it is antiipated that a probabilistic treatment of the performance of the waste

disposal system will continue to be the regulatory approach.

In providing this clarification, the staff wishes to

finally note that it is DOE's ultimate prerogative

and responsibility to adopt a strategy for

demonstrating compliance with NRC's

regulations. The intent of this NUREG, therefore,

is to allow DOE the maximum flexibility in

choosing an approach, as long as an effective

demonstration of compliance with the regulations

can be made.

In its final comment, the State of Nevada disputes

DOE's views regarding the value of or the need

for the background information contained in

Sections 1 ("Introduction") and 2 ("Regulatory

Framework") of the BTR The staff believes that

this information provides a helpful context for the

technical positions themselves insofar as it

summarizes the history and considerations that

relate to guidance on this particular subject. The

information may be of value, for example, to a

member of the public and thereby furthers NRC's

interest in conducting its regulatory activities in

such a way that those activities are understandable to all interested parties. (See NRC

(1996) for a discussion of the Commission's

"openness policy" with respect to how the

Commission conducts its regulatory activities.)

U.S. Department of Energy

The following comments were submitted by DOE

(see Brocoum, 1996):

The NRC's "Branch Technical Position on

the Use of Expert Elicitation in the

High-Level Radioactive Waste Program"

provides guidance to develop a structured

process to conduct formal expert elicitations.

DOE and NRC seem to have consistent

positions on the general steps that are

appropriate for these structured exercises.

DOE has identified no substantive

disagreements with respect to the process

the NRC has outlined for elicitation and its

associated documentation. DOE followed

each of the nine steps specified in the NRC's

process while conducting its recently

completed "Probabilistic Volcanic Hazard

Analysis for Yucca Mountain, Nevada"

(Geomatrix Consultants/TRW Environmental Safety Systems, Inc., 1996).

DOE's only major comment on the BTP

centers on the possibility that the NRC may

offer additional guidance on the use of

expert elicitation in the area of performance

assessment. The Department believes that

additional guidance is unnecessary.

As stated in DOE's June 1, 1995, letter to

NRC (Brocoum, 1995), DOE's elicitation

process will identify and document the basis

for any judgment, and this basis could

include both site-specific information

developed during site characterization

(including qualitative, descriptive, and

quantitative analytical information) as well

NUREG-1563 Do
as information from other relevant or similar

settings.

Response

With regard to DOE's first overall comment, that

".. . DOE and NRC seem' to have consistent

positions on the general steps [for the formal

elicitation of expert judgments]. . ." and that DOE

has ". . . identified no substantive disagreements

with respect to the process the NRC has outlined

for elicitation and its associated documentation... ," the staff welcomes DOE's view. The

staff also believes that it may be possible to

resolve, at the staff level, the staff's earlier

concerns 'expressed in SCA' Comment 3 (e.g.,

criteria for the use of expert elicitations are

needed in light of the requirement for a license

application to be as complete as possible given

reasonably available information). (See the

dis'cussion in Appendix E for more on the staff's

reasoning in this area.)

The staff is mindful, however, that despite this

apparent convergence in' thinking, 'the instruction

contained in DOE's Principles and Guidelines is

not identical to the guidance contained in this

BT? DOE should view this BIT as the primary guidance, in this area, although as with other

NRC staff guidance, it may choose to pursue alternative approaches, at its own risk. In

addition, consistent with the BTP, the staff has

identified a number of specific concerns with

DOE's Principles and Guidelines documentation

itself, which appear, along with a possible path to

their resolution, in Section E-2 of this document.

With regard to DOE's second overall comment,

that additional guidance is not necessary in the

area of performance assessment, the'staff will take

this comment into account in due course but has

made no decision as yet on this option. In view of

the potential that new performance' measures may

be established for a potential repository at Yucca

Mountain, pursuant to 'the Energy Policy Act of

1992, and consistent with the'findings and

recommendations of the recent National Academy of Sciences (NAS) report (see National Research

Council, 1995),3 the staff wishes to provide for the

3

1n addition to the recent NAS recommendations the staff notes that

there are pending legislative proposals that could affect the

rrgulation of HIP at Yucca 'Mountain. See summary in the 1994

Fndins and Recommendations of the Nuclear Vaste ¶Ichnical Review Board (NWTRB-see NWTF.B, 1995; pp. 46-48).

possibility that site-specific standards may necessitate the development of additional

implementing guidance. The staff, therefore,

continues to leave open this option, as noted in

Section 1.4, paragraph 3 of the BIT (e.g., that it

J"...is considering the potential need for future

guidance to identify those specific aspects of a

performance assessment for which the application of expert judgment may or may not be

appropriate.-. ."). Currently, the staff has no plans

for, or resources devoted to, preparing any additional guidance in this area. If the staff

decides that additional guidance should be

considered, it will explain its reasons and offer the

opportunity for comment by DOE (and others).

General Comments (Prescribed Use of

Elicitations in the Area of Performance

Assessment)

In Section 1.4 of the BTP, NRC is appropriately' silent about the specific technical issues for which

expert elicitation should or should not be applied,

except in the area of performance assessment.

DOE would like clarification of why the staff

believes they should consider additional guidance

(page 11)". . to identify those specific' aspects of

a performance assessment for which the

application of expert judgment may or may not be

appropriate." DOE believes that once the NRC

staff have set out the process, as described in the

BTP, it is the applicant's prerogative to decide if

and how its use is advantageous to support

arguments for licensing.

DOE believes that the BTT may suggest generic circumstances when use of expert elicitation is

appropriate, but that it is not appropriate to

prescribe categories or topical requirements for

these exercises.' DOE will have to balance many

factors in selecting the topic and scope for expert

elicitations. DOE is currently evaluating the

advisability of conducting expert elicitations in

several areas supporting development of its Total

System Performance Assessment (ISPA) for the

Viability Assessment,' including: scenario analysis "and associated estimates of probability,

parameter'uncertainty and bounding case

identification, and certain aspects of process

model abstraction'and conceptual model evaluation. DOE's plans currently do not include an

elicitation on the totality of the TSPA submittal

forming the basis of a license application. DOE

D-5 NUREG-1563
expects to discuss our future plans in this area

with the NRC when our planning has matured.

DOE notes that development of prescriptive

guidance for performance assessment is in

conflict with the BTP description (page 11) which

states ". . . under appropriate circumstances, it is

acceptable to supplement data and analyses with

opinions of experts as part of the support for

demonstrating compliance... ." Decisions as to

when or whether to conduct an elicitation

exercise, the identification of specific issues or

topics, or other "appropriate circumstances" are

management prerogatives of the DOE. DOE is

concerned with the potential for prescriptive

definitions or circumstances that might limit

management's alternatives or options to use

expert elicitations as part of the technical basis

for our compliance argument(s). DOE intends

that any use of expert elicitations in the area of

performance assessment be consistent with its

Principles and Guidelines.

Response

In this general comment, DOE has requested

clarification regarding why the staff believes it

might consider additional guidance"... to

identify those specific aspects of a performance

assessment for which the application of expert

judgment may or may not be appropriate... ." As

noted above, in response to DOE's second overall

comment, it is premature to decide whether and,

if so, what additional guidance is appropriate or

to attempt to predict those specific aspects of a

performance assessment for which the application

of expert judgment may or may not be appropriate. If the staff decides that additional

guidance should be considered, it will explain its

reasons and offer the opportunity for comment by

DOE (and others).

tangential or marginal issues. Section 2.2 of the

BTP is a good example.

1. Section 1 ("Introduction"): The

"Introduction" section is somewhat confusing

because it is not clear whether the NRC's

intent is to provide guidance on the generic

use of expert elicitation (i.e., in any repository

program) or restrict the guidance to the

Yucca Mountain Project. It is also not clear

to which part of the NRC this BTP applies.

DOE suggestions: Include a statement of

"scope" to describe the NRC's intentions in

developing the BTP generic or projectspecific guidance. A suitable description of

scope appears to be available (with minor

modifications) in Section 4, Discussion, on

pages 22 and 23.

Also, include a reference to the Division of

Waste Management, or the Branch(es) within

it that are responsible for preparation of the

BTP. This information is not obvious and

only now occurs in the "Foreword."

Response

The staff believes that the clarification requested

by this comment already exists within the body of

the BTP, as discussed below.

With regard to DOE's first comment, regarding

the applicability of the BTP to NRC's regulatory

programs, it should be noted that this BTP

applies to the HLW program. There are a number

of references to that effect, both explicitly as well

as implicitly throughout the BT. For example, see

the "Foreword," 'Abstract," and "Introduction"

(5th paragraph). Because this guidance was

developed for application to the HLW program, it

is the staff's view that it would apply, generically,

to any potential repository licensed by NRC.

As regards DOE's.second comment, as indicated

on both the cover page and in the "Foreword" of

the guidance, the BTP was prepared by Division

of Waste Management (DNVM) staff (Kotra, Lee,

and Eisenberg) with assistance from its technical

assistance contractor, the Center for Nuclear

Waste Regulatory Analyses-CNWRA

(DeWispelare). As indicated, these authors

incorporated the many useful comments and

suggestions of other DWM and CNWRA staff

members, representatives of the Office of the

Specific Comments

The specific comments below (Nos. 1-5) refer to

format and content issues or subject headings that

may be prescribed for BTPs. DOE's theme in

these comments is that, early on and for some

length, they tend to focus the document on a look

back instead of forward. DOE's general concern

is that together Sections 1 and 2 tend to cloud the

points of agreement between the two agencies in

Sections 3 and 4 by including and discussing

NUREG-1563 D-6
General Counsel, and members of the staff from

the Office of Nuclear Reactor Regulation, the

Office of Nuclear Regulatory Research, and the

Office for Analysis and Evaluation of Operational

Data. (These individuals are identified in the

'Acknowledgments" section of the BTP.) Lastly,

the final technical positions reflect discussions

with and recommendations from NRC's Advisory

Committee for Nuclear Waste (ACNW), which are

documented in Appendix F. As noted in the

"Foreword," separately and at a later time, the

staff may expand and refine this guidance for

application in other areas of NRC regulation.

2. Section 1.1: The Background" ection is

distracting. To revisit and critique DOE's

past uses of expert judgment or prior

elicitations recapitulates a comment record

already made by the NRC on the DOE's past

efforts and documentation. It distracts from

the BIT's purpose in establishing;

expectations on how these exercises are

performed in the future.

DOE suggestion: Delete this section or

condense significantly.

establish process expectations for future

elicitation applications.

DOE suggestion:'Condense significantly or

move the material in Section 15 either to

front material such as a "Foreword," or an

Appendix.

Response

In conformance with a standard format for all

DWM staff guidance documents, the information

in question, in Section 1.5, is typically contained

in staff technical positions, and is intended to

communicate the staff's overall goals in issuing

regulatory guidance.

4. Section 2.1 ("Regulatory Framework10 CFR Part 60"): The last paragraph in the

section is' a good example of raising issues

that are not relevant to the purpose of the

BTP, i.e., establishing process expectations for

expert elicitations. It refers to a past

comment and response dialog on the DOE's

SCP.

DOE suggestion: Delete the paragraph.

Response

The staff does not share DOE's view that the

"Background" section (Section 1) of the BTP

distracts from the technical position statements.

'Although the information is historical in nature,

the staff believes this information provides a

useful context for the technical positions

themselves insofar as it summarizes the history,

reasoning, and staff considerations that go into a

decision to issue guidance on a particular subject.

As noted in its response to the State of Nevada

(above), the staff believes that this information

could provide a helpful context for the 'iechnical

positions themselves insofar as it summarizes the

history and considerations that relate to guidance

on this particular subject.'The info'ination"may

be of value, for example, to a-member of the 

public and to this extent, furthers NRC's interest

in conducting its regulatory activities in such a"

way that those activities are understandable to all

interested parties.

3. Section 15: The "Branch Technical Positions

as Technical Guidance" section provides little

value toward the purpose of.theguidance, to

Response

The staff does not share DOE's view concerning

the relevance of the paragraph in question to the

BTP. As a result of the staff's review of DOE's

statutory SCP, several subsequent reviews of DOE

site characterization activities, and recommendations of certain advisory bodies-the ACNW

and the NWIRB (all of which are summarized in

Section 1.1), the staff decided to develop guidance

in this area.

5. Section 2.2 ("Commission Policy Statement

Concerning the Use of PRA Methods in

Nuclear Regulatory Activities"): This section

has little apparent relevance to the development of guidance for the use of expert

elicitation, and only the last sentence in the

entire section states the essence of the section

that could bear the purpose of the BT. The

"Regulatory Framework" section, in general,

is distracting and does not appear to be

relevant to the purpose of the BTP as process

guidance.

DOE suggestion: Delete Section 2.24 move the

last sentence [with edits], "The technical

D-7 NUREG-1563
positions cited in Section 3 are consistent

with the recommended process steps [from

the NRC's staff's Probabilistic Risk

Assessment Working Group."] to an

appropriate place in Section 3. Consider

condensing Section 2 and especially Section

1 significantly.

Response

The staff does not agree with DOE's comment to

delete the material contained in Section 2.2 of the

BTP. In the staff's view, a brief discussion of the

Commission's Policy Statement, concerning the use

of probabilistic risk assessment (PRA) methods in

nuclear regulatory activities (see NRC, 1995; 60

FR 42623), contributes useful information to the

BTP that places it in a broader regulatory context.

The staff believes that the Commission's decision

regarding "emerging" PRA technologies in the

Policy Statement is supportive of the BTP.

6. Section 3 ("Branch Technical Positions")-

page 17, Technical Position 2(a), line 2: The

inclusion of the word "defensible" in the

description of the procedure includes a

criterion whose definition is arbitrary until it

can be determined whether the procedure

was in fact successfully or unsuccessfully

defended. Furthermore, "defensibility" or the

"need to defend" are management

considerations which are inappropriate topics

for the guidance.

7. Technical Position No. 3 in Sections 3 and

4-pages 21 and 36, respectively: The time

frame implied in the description on page 21

is not clear. One interpretation would

indicate that expert judgment, expert

elicitation, and peer review are intended to be

snap-shots in time of the experts' views of the

issue in question based on the information

available at the time the exercise was

conducted. An alternative interpretation

would imply that the guidance envisions an

elicitation process that is iterative and might

never be completed if additional data were

continuously available.

On page 36, the expectation for re-examining

and updating the results of past elicitations

to new, relevant data needs clarification. It is

the DOE's prerogative and responsibility to

ensure that the materials submitted to the

NRC for a licensing action are current.

When new information becomes available,

DOE will evaluate and document its

relationship to the assumptions and range

variation established in prior elicitations. It

is not necessary, however, for the BTP to

establish expectations for how an elicitation

is re-evaluated in light of new information,

beyond the means of being well documented.

DOE suggestion. Use wording that does not

imply a process-specific means of

re-examining the results of a prior elicitation

in light of new data.

There are similar references to "defensible"

in Section 4 ("Discussion"), p. 25, Item (2)

and p. 29, 3rd paragraph.

DOE suggestion: Revise these references to

"defensible" to describe thoroughly

documented processes.

Response

The staff's intent behind the use of the word

"defensible" in Technical Position 2(a) (and

elsewhere) was to reference the obligation of the

potential applicant (i.e., DOE) to demonstrate the

acceptability of both the elicitation process and

the outcome of that process. With that

understanding in mind, the staff has no objection

to making the requested change.

Response

NRC expects that a potential license application

for a geologic repository to be complete as

possible in light of the information that is

reasonably available at the time of docketing (10

CFR 60.24(a)). As a potential applicant, the intent

of this technical position was to remind DOE of

the staff's expectation on this matter. However, to

avoid any additional confusion in this area, the

staff has revised Technical Position No. 3 to read

as follows:

If information from an expert elicitation is to

be submitted in support of a license

application, and if additional data or

information becomes available, subsequent

to the completion of the elicitation, which

could change opinions or judgments

NUREG-1563 D-8
obtained in the formal elicitation, the results

of the elicitation should be reexamined and

updated, as appropriate. In addition'to the

information requested above, documentation

should include a detailed 'description of the

updating process.

Consistent with these changes, the staff has also

revised'the second paragraph of the rationale for

this technical position (in Section 4) to read as

follows:

The methods'of updating the expert judgments'range from the use of Bayes' Theorem, for statistical updating, to

conducting another set of individual

elicitations for the same or a'different set of

experts. Whichever method is used for

incorporating the new data or information

into the existing expert opinions, it should be

thoroughly documented to provide a'

transparent view of the updating process and

resulting judgments.

8. :Section 4 ("Discussion" behind Technical

Position No. 2)-page 26, 2nd full paragraph, lines' 6-9: The staff's expectations with regard

to the use of subject-matter experts, to better

define the objective of the elicitation, are not

clear.Using subject-matter experts to define

the'objectives of the elicitation on 'which they - have been asked to participate,'represents a

potential to create coiflicts of interestespecially financial and pfofessional conflicts

* ' of interest.'Discussion of the'same

consideration on page 30 is clear and does - not contain the apparent ambiguities found

in the discussion on page 26.

DOE suggestion: Delete the sentence "What

this figure shows,' into' its constituent parts

(Step No.'3)" to remove this ambiguity and

potential inconsistency that could create a

conflict of interest. ' ':-' i

Response

The staff does not agree with DOE's characterization'of this issue. It is the staff's view that there

is no ambiguity regarding the sentence'in question

(introductory discussion in Section 4 behind'

Technical Position No. 2-page 26). The staff is

expressing the 'view in this section of the BTP that

subject-matter experts, working with the

generalist, can help to better define the overall

objectives of any particular elicitation. Although

not expressed in the BTP itself, it is the staff's

view that the normative expert and generalist,

working with the elicitation sponsor (i.e., DOE),

reserve the right to limit the scope and extent of

any proposed elicitation given, of course, contrary

'(documented) advice from the subject-matter

experts themselves.

9. Appendix C: This Appendix is relevant only

to Section 2.2 of the draft Branch Technical

Position.

DOE suggestion: Delete Appendix C if

Section 2.2 is deleted, as suggested.

Response

The staff's response to this comment is addressed

under DOE specific comment no. 5.

U.S. Nuclear Waste Technical Review Board

The following comments Were submitted by the

NWIRB (see Cantlon, 1996):

The Board has had a long-standing interest

in the use of formal expert judgment by

DOE as it characterizes the potential repository site at Yucca Mountain and

moves toward a possible application to

construct a permanent underground

repository there. The Board has addressed

'the need for DOE to develop sound

elicitation methodologies, to involve outside

experts in any formal elicitation conducted,

and to resolve possible conflicts with the

NRC well before the submission of any

license application.

The BTP lays out in a thoughtful and wellargued manner the key issues involved in

carrying out a successful and reliable formal

elicitation. For the most part,' the BTP has

incorporated the best'curint thinking of

decision analysts who have examined this

'area'as well as appropriate lessons from

previous NRC experience. The BTP correctly

recognizes that DOE ultimately bears the

burden of convicing a Licensing Board, and

probably'others as well, that its use of expert

' judgment' on a" 'parti6ular issue has properly

NUREG-1563
characterized the relevant uncertainties and

that their magnitude is, in fact, acceptable.

There are, however, some areas and issues

where the Board feels greater specificity in

the BTP might be helpful.

* Based on its own analyses, does the NRC

[staff] believe that there are technical

issues that are so critical to demonstrating the safety of a repository system that

their resolution should be based almost

exclusively on primary data, minimizing

the reliance on expert judgment? If so,

what are they?

* Has the NRC staff developed views or

guidance as to when "it is infeasible or

impossible to collect data" and what

types of management challenges would

support a determination that "data

collection [has] become "prohibitively

expensive" (p. 24)?

* Are there any circumstances where the

NRC staff might not accept the results of

a DOE elicitation that was conducted in

accordance with the process outlined in

the BTP? If so, what are they?

* Can some guidance be offered to DOE

on the conditions and circumstances that

justify departure from equal weighting of

experts' judgments?

NRC might wish at some point to explore,

with DOE, the related question of how

biases of experts might be minimized when

their judgment is rendered informally,

although such an effort could be outside of

the scope of the BTP.

Response

With respect to the NNVTRB's first comment, it

should be noted that it is DOE's responsibility, in

the first instance, to formulate a strategy for

demonstrating compliance; the staff is, therefore,

reluctant to constrain DOE by identifying

technical issues that must be resolved almost

exclusively by the use of primary data, rather than

expert elicitation. For issues DOE elevates to

critical importance, DOE should make an

adequate, but decisive, case. The responsibility

will then fall to the staff to evaluate the

acceptability of DOE's analysis, in the context of

a complete license application. DOE will have to

decide how much reliance to place on various

features and components of the repository system

in order to demonstrate compliance with the totalsystem performance objective. The staff should

not attempt to predict, in the BTP, whether DOE

may choose to take credit at all for certain

potentially helpful components or features (e.g.,

cladding of spent fuel as an additional barrier

isolating the waste). The repository system is

highly complex and nonlinear; modeling the

repository system is based on limited data.

Determination of critical issues is difficult and

might be subject to change under these

conditions, even presupposing that DOE has a

firm strategy for demonstrating compliance. Some

important issues of repository performance (e.g.,

the validity of long-term predictions), must

necessarily be based on scant primary data. For

all these reasons, the staff is not inclined to

specify issues, in this BTP, requiring resolution

almost exclusively using primary data. These

points are either made already in the responses to

public comments or may be found in Section 1 of

the BTP.

With respect to the NWTRB's second comment,

concerning possible guidance on when it might be

". . . infeasible or impossible to collect data . . or when the cost of data collection might be

"prohibitive," the staff tends to think that this

matter, like the first NWTRB comment, is better

left for later consideration, in the context of a

topical report or a completed license application,

for example. However, the staff believes that it is

impossible to collect some data, such as direct

confirmation of long-term predictions of

radionuclide migration at the site. Nevertheless,

the staff has not developed guidance on these

issues. Furthermore, DOE, the party responsible

for collecting the necessary site characterization

data, as the potential applicant, has not requested

further guidance at this time. Moreover, as the

party responsible for preparing a potential license

application, it might be more appropriate to defer

this question to DOE. In this regard, the staff

notes in Section 1.4 ("Purpose of the BTP"-see

paragraph 6) of the BTP that:

"DOE has the flexibility to determine

whether the costs and benefits of performing

an expert elicitation are advantageous

NUREG-1563 D-10
compared to the costs and benefits of

performing theoretical analyses and/or

gathering additional field and experimental

data."4

In response to the NWTRB's third comment,

concerning .. . circumstances where the NRC

staff might not accept the results of a DOE*

elicitation that was conducted in accordance with

the process outlined in the BTP. . . ," the staff's

preference is to be cautious on what adherence to

the BTP can provide and not to predict where it

can fail. Hypothetically, though, circumstances

could arise in which the staff might not accept the

results of an elicitation. For example, one

circumstance might be the existence of

subsequent analyses or opinions that contradict

the conclusions reached by an elicitation. In

anticipation that such situations cannot be

completely discounted in the future, the staff

acknowledged this possibility (see Section 1.4 of

the BTP). However, the staff believes that -

following the guidance set forth in the BTP

enhances the chances of the acceptance of the

elicitation and its results.

In response t the NWTRB's final comment

regarding conditions or circumstances that might

justify departure from equal weighting of experts'

judgments, although tere may be some

circumstances in which unequal weighting might

be appropriate, the staff prefers not to speculate

on what those conditions or. circumstances might

be. The staff believes that the response to the

NWTRB's question is more appropriately

addressed by DOE, again, as the practitioner.

Whatever weighting factors are used, DOE should

document its rationale for selecting them.

However, the reader is referred to-DeWispelare et

al. (1994) and the Senior Seismic Hazard Analysis

Committee (1995) for discussion and examples of

how this issue has been dealt with in specific

applications.

* Lastly, with respect to the NWTRB's recommendation, that NRC and DOE explore the related

question of how biases of experts might be

minimized when their judgment is rendered

4As noted in the staff's response to the State of Nevada's third

general comment, this BTP is conistent with current NRC policy

vwhich encourages the use of PRA state-ofthe-art technology and

methods as a con Wiement to the deterministic approach in nuclear

regulatory activities and is also in keeping with the Commission's

paramount responsibilit to protect public health and safety.

informally, the staff agrees with the NWTRB that

this issue is beyond the scope of the BTP and

should be addressed separately.

References

Brocoum, S.J., U.S. Department of Energy/Office

of Civilian Radioactive Waste Management, letter

to JJ. Holonich, Office of Nuclear Material Safety

and Safeguards/Division of Waste Management

[Subject: " 'Principles and Guidelines for Formal

Use of Expert Judgment by the Yucca Mountain

Site Characterization Project Office' and

Resolution of Site Characterization Analysis

Comment 3"], Yucca Mountain Site

Characterization Office, June 1, 1995.

Brocoum, S.J., U.S. Department of Energy/Office

of Civilian Radioactive Waste Management, letter

to Secretary, Docketing and Services Branch/U.S.

Nuclear Regulatory Commission [Subject: "U.S.

Department of Energy Comments on [the] U.S.

Nuclear Regulatory Commission Draft Branch

'Technical Position on the Formal Use of Expert

Elicitation in the High-Level Waste Program"],

Yucca Mountain Site Characterization Office,

May 13, 1996.

Cantlon, J.E., U.S. Nuclear Waste Technical

Review Board, letter to Secretary, Docketing and

Services Branch/U.S. Nuclear Regulatory

Commission [Subject: "NWTRB Comments on

the NRC's Draft BTP on the Formal Use of

Expert Elicitation in the HLW Program"],

Arlington, Virginia, May 10, 1996.

DeWispelare, A.R., et al., "Background Report on

the Use and Elicitation of Expert Judgment," San

Antonio,`Texas, Center for Nuclear Waste

Regulatory Analyses, CNWRA 94-019, September

1994. [Prepared for the U.S. Nuclear Regulatory

Commission.]

Geomatrix Consultants/TRW Environmental

Safety Systems, Inc., "Probabilistic Volcanic

Hazard Analysis for Yucca Mountain, Nevada,"

San Francisco, California, Document No.

BAOOOOOOO-1717-2200-00082 (Rev. 0), June 1996.

[Prepared for the U.S. Department of

Energy/Office of Civilian Radioactive Waste

Management.]

Holonich, JJ., Office of Nuclear Material Safety

and Safeguards/Division of Waste Management,

letter to R.R. Loux, State of Nevada/Agency for

D-11 D1NUREG-1563
.

Nuclear Projects [Subject: "State of Nevada

Comments on U.S. Department of Energy

Activities Related to the Use of Expert Elicitation

in the High-Level Waste Program"], U.S. Nuclear

Regulatory Commission, August 22, 1995.

Loux, R.R., State of Nevada/Agency for Nuclear

Projects, letter to JJ. Holonich, Office of Nuclear

Material Safety and Safeguards/Division of Waste

Management [Subject: "'Principles and

Guidelines for Formal Use of Expert Judgment by

the Yucca Mountain Site Characterization Project

Office' and Resolution of Site Characterization

Analysis Comment 3"], Carson City, Nuclear

Waste Project Office, July 25, 1995.

Loux, R.R., State of Nevada/Agency for Nuclear

Projects, letter to J.H. Austin, Office of Nuclear

Material Safety and Safeguards/Division of Waste

Management [Subject: "State of Nevada

Comments on 'Draft Branch Technical Position on

the Formal Use of Expert Elicitation in the

High-Level Waste Program' (February 1996)"],

Carson City, Nuclear Waste Project Office,

May 15, 1996.

National Research Council, "Technical Bases for

Yucca Mountain Standards," Washington, D.C.,

National Academy Press, Commission on

Geosciences, Environment, and Resources, July

1995.

Senior Seismic Hazard Analysis Committee,

"Recommendations for Probabilistic Seismic

Hazard Analysis: Guidance on Uncertainty and

Use of Experts," Livermore, California, Lawrence

Livermore National Laboratory, UCRLID122160, 2 vols., August 1995. [Sponsored by the

U.S. Department of Energv, the U.S. Nuclear

Regulatory Commission, and the Electric Power

Research Institute. Also expected to be published

as NUREG/CR-6372.]

U.S. Department of Energy, "Site Characterization Plan, Yucca Mountain Site, Nevada

Research and Development Area, Nevada," Office

of Civilian Radioactive Waste Management,

Nevada Operations Office/Yucca Mountain

Project Office, Nevada, DOE/RW-0199, 9 vols.,

December 1988.

U.S. Environmental Protection Agency,

"Environmental Standards for the Management of

Spent Nuclear Fuel, High-Level and Transuranic

Wastes [Final Rule]," Federal Register, Vol. 50, No.

182, September 19, 1985, pp. 38066 - 38089.

U.S. Nuclear Regulatory Commission, "NRC Staff

Site Characterization Analysis of the Department

of Energy's Site Characterization Plan, Yucca

Mountain, Nevada," Office of Nuclear Material

Safety and Safeguards, NUREG-1347, August

1989.

U.S. Nuclear Regulatory Commission, "Use of

Probabilistic Risk Assessment Methods in

Nuclear Regulatory Activities [Final Policy

Statement]," Federal Register, Vol. 60, No. 158,

August 16, 1995, pp. 42622 - 42630.

U.S. Nuclear Regulatory Commission, "Public

Involvement in the Nuclear Regulatory Process,"

NUREG/BR-0215, 1996.

U.S. Nuclear Regulatory Commission,

'Availability of Draft Branch Technical Position

on the Use of Expert Elicitation in the High-Level

Waste Program [Notice of Availability]," Federal

Register, Vol. 61, No. 40, February 28, 1996, pp.

7568 - 7569.

Wescott, R.G., et al. (eds.), "NRC Iterative

Performance Assessment Phase 2: Development of

Capabilities for Review of a Performance

Assessment for a High-Level Waste Repository,"

U.S. Nuclear Regulatory Commission,

NUREG-1464, October 1995.

NUREG-1563 D1-12
APPENDIX E

STAFF VIEWS ON DOE'S 1995 PRINCIPLES AND GUIDELINES AND THE

POSSIBLE DISPOSITION OF SCA COMMENT 3

E-1 Introduction

The staff's original intent, as indicated in

correspondence to both the U.S. Department of

Energy (DOE) and the State of Nevada (see

respectively, Austin (1995) and Holonich (1995)),

was to comment on both DOE's Principles and

Guidelines for Formal Use of Expert Judgment by

the Yucca Mountain Site Characterization Project

Office (Broc6um, 1995) (hereafter referred to as

the rinciples and Guidelines) and, in that 

comment, to reflect its consideration of the State's

review (Loux, 1995), thereof, after a Fall 1995 U.S.

Nuclear Regulatory Commission/DOE technical

exchange. However, the proposed technical

exchange was cancelled, at DOE's request. In light

of the canciellation, the staff decided to defer -

comment until after public comments had been

received on the draft Branch Technical Position

(BTP).

In its comments on the draft BTh, DOE indicated

that it is now in "substantial agreement'? with the

NRC staff's technical positions on the formal use

of expert elicitation in the high-level waste

program (see Brocoum, 1996). For its part, the

'State commented that the draft BTP was

"favorably responsive" to its earlier concerns (see

Loux, 1995). Therefore, the staff is inclined to

believe that with publication of the BTP, the

staff's original intention, to comment on the DOE

Principles and Guidelines, has been overtaken by

events (i.e., issuance and acceptanceof the NRC

guidance). e

With these thoughts in mind, the NRC staff offers

* the following comments and describes a possible

path to resolution, at the staff level, of the

particular Site Characterization Analysis (SCA)

open item, SCA Comment 3, to which this issue

applies.

E2- -DOE's Principles and Guidelines

To address the concerns raised by the staff in its

SCA Coinment 3, the staff made two

recommendations to DOE (NRC, 1989; p. 4-10):

" State the criteria for the formal use of expert

judgment to ensure that objective, quantitative analyses based on empirical data are

used in preference to expert elicitation,

wherever possible.

* Modify the Site Characterization Plan, in an

early update, to assure that the requisite data

will be available."

Consistent with the staff's first iecommendation, .. . to state criteria for the formal use of expert

judgment. . ," DOE issued its 1995 Principles and

Guidelines and in doing so, has argued that the

information contained in it provides the necessary

criteria. Along with the information and direction

contained in the QualityAssurance Requirements

Document (QARD), DOE suggests that its 1995

Principles and Guidelines would be adequate to

resolve, at the staff level, this particular open

item.

The staff has reviewed DOE's Principles and

Guidelines and has a number of concerns. The

first is that DOE's QARD addresses only thesubject of "peer review" (see DOE, 1995; Section

2.2.9) and does not treat the issue of elicited

expert judgments (either f6rmally or informally).

Second, language in the Principles and Guidelines,

in many places, appears to confuse the concepts

of "expert judgment," "expert elicitation," and

"peer review," concepts- that are, in the staff's

opinion, distinct. Because the staff believes there

is frequent confusion in the use and application of

these terms, it decided to provide the necessary

clarification in its own BTP, including expanding

the definition of "peer review" over that provided

earlier in Altman et al. (1988). These three

subjects are distinct and should be addressed

separately. A-third deficiency identified by the

staff in the Principles and Guidelines document is

that it contains no substantive discussion

regarding potential procedures per se that would

be used to conduct a formal elicitation. If DOE

intends to rely on its Principles and Guidelines as

instruction to Department management and staff,

the NRC staff believes that a "how-to" statement,

such as that contained in the recommended

nine-step process in Section 3 of the BT? is

E1 E-1 ~~~~NUREG-156
needed. Lastly, the staff believes that DOE's

Principles and Guidelines needs to acknowledge or

address, as the staff did in the BTP (see Section

4), how DOE management and staff would deal

with the potential for conflict of interest when

conducting a formal elicitation.

With respect to the'second staff SCA recommendation .. . to modify the Site Characterization

Plan (SCP-DOE, 1988), in an early update, to

assure that the requisite data will be available ... ," DOE has not modified the SCP, as first

recommended. However, it is generally recognized

now that DOE's repository program has evolved

beyond that which was earlier described in the

1988 SCP, for a variety of reasons. As DOE

prepares its future program plans for site

characterization and the repository Viability

Assessment, the staff fully expects DOE to

provide sufficient documentation to support its

decisions, including those that relate to where it

might rely on formal use of expert judgment.

Therefore, until the Department has had an

opportunity to revise and update its site

characterization plan, it may be appropriate for

the staff to consider retraction of this earlier

recommendation, at this time.

E-3 State of Nevada July 1995

Comments on DOE'S Principles

and Guidelines

As noted above, and subject to specific comments

and recommendations, the State has indicated

that the draft BTP is "favorably responsive" to its

earlier concerns expressed to the staff concerning

DOE's Principles and Guidelines (Loux, 1995).

Further, as described below, DOE is being asked

to revise its Principles and Guidelines, consistent

with the BTP. Thus, in light of this, the staff

believes that further comment on the State's July

1995 letter would serve no useful purpose.

However, the staff does wish to point out, as it

did in its initial response to the State on these

comments, that the views of DeWispelare et al.

(1994) as well as the views of all other contractor

reports (e.g., Bonano et a!. (1990) and Senior

Seismic Hazard Analysis Committee (1995)) are

the views of those authors and do not, necessarily,

reflect the views on regulatory positions of the

NRC staff. To the extent that these or any other

contractor reports are cited in this BTP, the only

staff endorsement these documents receive is

limited to the particular points for which they are

referenced. Moreover, as regards the BTP's

reference to or consistency with any other

documents, the reader is reminded that for the

purposes of any potential license application, the

benchmark that will be used by the staff to judge

the acceptability of the process of any formal

elicitation is the staff's technical positions

described in this NUREG.

E-4 Staff Recommendation on the

Disposition of SCA Comment 3

In light of the aforementioned comments and

observations, and because DOE is in "substantial

agreement" with the NRC staff's technical

positions in the BTP, the staff believes that there

is a sufficient basis to recommend that SCA

Comment 3 be closed, at the staff level, although

on somewhat different grounds from those

suggested by DOE in its 1996 letter to the staff

(see Brocoum, 1996). However, in making this

recommendation, the staff believes that DOE will

need to agree to the following course of action,

with the attendant commitments or, an equivalent

course of action.

1. The 1995 Principles and Guidelines should be

revised to reflect DOE's acceptance of the

staff's BTP. Moreover, DOE's Principles and

Guidelines should be revised to address the

recommended changes to format and content,

as noted by the staff in Section E-2, above.

To summarize, these would include:

* Correction and clarification regarding

DOE's use of the terms "expert

judgment," "expert elicitation," and

"peer review."

* Substantive discussion regarding the

specific procedures per se that the

Department and its contractors would

follow when conducting a formal

elicitation.

* Direction to DOE management and staff

regarding how to address the potential

for conflict of interest when conducting a

formal elicitation.

2. The current version of the QARD (DOE,

1995) should be revised to include a

discussion of the treatment of "formal" expert

NUREG-1563 E-2
elicitation comparable to the discussion

which already exists for "peer reviews."

3. DOE decisions on the need to use formal

expert elicitations should be transparent.

DOE's Principles and Guidelines should be

revised to ensure that its management and

staff prepare the necessary documentation to

permit tracking of such decision-making.

E5 References

Altman, WD., et aL, "Peer Review for High-Level

Nuclear Waste Repositories," U.S. Nuclear

Regulatory Commission, Generic Technical

Position, NUREG-1297, February 1988.

Austin, J.H., Office of Nuclear Material Safety

and Safeguards/Division of Waste Management,

letter to SJ. Brocoum, U.S. Department of

Energy/Office of Civilian Radioactive Waste

Management [Subject: "Use of Expert Elicitation

in the High-Level Waste Program and Status of

Site Characterization Analysis Comment No. 3"],

U.S. Nuclear Regulatory Commission, July 21,

1995.

Bonano, EJ., et al., "Elicitation and Use of Expert

Judgment in Performance Assessment of HighLevel Radioactive Waste Repositories," U.S.

Nuclear Regulatory Commission,

NUREG/CR-5411, May 1990. [Prepared by the

Sandia National Laboratories.]

Brocoum, SJ., U.S. Department of Energy/Office

of Civilian Radioactive Waste Management, letter

to J.J. Holonich, Office of Nuclear Material Safety

and Safeguards/Division of Waste Management

[Subject: "'Principles and Guidelines for Formal

Use of Expert Judgment by the Yucca Mountain

Site Characterization Project Office' and

Resolution of Site Characterization Analysis

Comment 3"], Yucca Mountain Site

Characterization Office, June 1, 1995.

Brocoum, SJ., U.S. Department of Energy/Office

of Civilian Radioactive Waste Management, letter

to Secretary, Docketing and Services Branch/U.S.

Nuclear Regulatory Commission [Subject: "U.S.

Department of Energy Comments on [the] U.S.

Nuclear Regulatory Commission Draft Branch

Technical Position on the Formal Use of Expert

Elicitation in the High-Level Waste Program"],

Yucca Mountain Site Characterization Office,

May 13, 1996.

DeWispelare, A.R., et al., "Background Report on

the Use and Elicitation of Expert Judgment," San

Antonio, Texas, Center for Nuclear Waste

Regulatory Analyses, CNWRA 94-019, September

1994. [Prepared for the U.S. Nuclear Regulatory

Commission.]

Holonich, J.J., Office of Nuclear Material Safety

and Safeguards/Division of Waste Management,

letter to R.R. Loux, State of Nevada/Agency for

Nuclear Projects [Subject: "State of Nevada

Comments on U.S. Department of Energy

Activities Related to the Use of Expert Elicitation

in the High-Level Waste Program"], U.S. Nuclear

Regulatory Commission, August 22, 1995.

Loux, R.R., State of Nevada/Agency for Nuclear

Projects, letter to JJ. Holonich, Office of Nuclear

Material Safety and Safeguards/Division of Waste

Management [Subject: "'Principles and

Guidelines for Formal Use of Expert Judgment by

the Yucca Mountain Site Characterization Project

Office' and Resolution of Site Characterization

Analysis Comment 3"], Carson City, Nuclear

Waste Project Office, July 25, 1995.

Senior Seismic Hazard Analysis Committee,

"Recommendations for Probabilistic Seismic

Hazard Analysis: Guidance on Uncertainty and

Use of Experts," Livermore, California, Lawrence

Livermore National Laboratory, UCRL-ID122160, 2 vols., August 1995. [Sponsored by the

U.S. Department of Energy, the U.S. Nuclear

Regulatory Commission, and the Electric Power

Research Institute. Also expected to be published

as NUREG/CR-6372.]

U.S. Department of Energy, "Site Characterization Plan, Yucca Mountain Site, Nevada

Research and Development Area, Nevada," Office

of Civilian Radioactive Waste Management,

Nevada Operations Office/Yucca Mountain

Project Office, Nevada, DOEJRNV-0199, 9 vols.,

December 1988.

U.S. Department of Energy, "Quality Assurance

Requirements and Description (for the Civilian

Radioactive Waste Management Program),"

Washington, D.C., Office of Civilian Radioactive

Waste Management, DOE/RW/0333P (Rev. 5),

October 1995.

E-3 NUREG-1563
U.S. Nuclear Regulatory Commission, "NRC Staff

Site Characterization Analysis of the Department

of Energy's Site Characterization Plan, Yucca

Mountain, Nevada," Office of Nuclear Material

Safety and Safeguards, NUREG-1347, August

1989.

NUREG-1563 E-4
APPENDIX F

DISPOSITION OF ACNW COMMENTS

After receipt and analysis' of public comments on

the February 1996 draft Branch 'Rchnical Position

(BlP), the staff briefed the Advisory Committee

on Nuclear 'Waste (ACNW) on the staff's

disposition of public comments and any proposed

revisions to the BTP based on those public

comrnents. As a result of that briefings dated

August 22, 1996, the ACNW submitted the

following' commentsl to the Commission.

Recommendation

This BTP provides important guidance to the

(potential) applicant (the U.S. Department of

Energy-DOE), affected units of government, and

interested parties, on the use of formally elicited

expert judgment. The ACNW anticipates that the

BTP will be immediately useful to the NRC staff,

for example, in its evaluations of (or comments

on) DOE's Probabilistic Volcanic Hazard

Analysis for Yucca Mountain, Nevada"

(Geomatrix Consultants/TRW Environmental

Safety Systems, Inc., 1996) and later comments on

DOE's (ongoing) probabilistic seismic hazard

assessments In the long term, the BTP will

provide valuable guidance to the DOE in the

preparation of its license application and to other

parties carrying out expert elicitations in

connection with the facility licensing process.

We wish to commend the NRC staff for

completing the final draft BTT, which is desirably brief and nonprescriptive. The applicant is left to

its own creativeness on how to handle such

important issues as probabilities, methods of

aggregating uncertainties, data updating, and the

t

1he indented ortions of this appndix quote the ACNW

comnments. Moreover, the staff hs responded o minor comments

* ~on the BTP made by the ACNW, as noted in the Transcript for the

85th Meet of the ACNW-August 21-23. 1996. 

2It should be noted that during the August 22. 1996, briefing of the

ACNW, DOE indicated that it contemplated a series of additional

expert elicitations as part of its total-systemn performance assess- ment efforts. In an October 9, 1996, public meeting with the

Nuclear Waste Thchnical Review Board, DOE tentatively identified

the general subject areas for these additional focused elicitalions as: unsaturated zone hydrology, waste package canister

degradation, Thermal hydrology, waste form dissolution, and

saturated zone hydrology.

final form of the results. The Committee strongly

recommends the prompt completion and

publication of the final draft BTP.

Residual Concerns

Although the ACNW welcomes and supports the

subject draft BTP, we have several residual

concems'regarding the use of formally elicited

expert judgment in the decision-making process.

The Committee does not intend that these

concerns delay publication of the draft BT? We

realize that these concdrns could be addressed by

a variety of means outside the BTP, including

workshops,' letters, NUREGs, technical exchanges, and so on. These concerns are discussed below.

The Committee believes that these residual

concerns should not delay the prompt publication

of the BTP. The ACNW looks forward to working with the staff to address these concerns through

other avenues.

1. Subject-Matter Experts

The Committee believes that the nomination

process for selecting subject-matter experts

should include organizations such as the

National Academies of Sciences (NAS) and

Engineering, private industry, State development and regulating bodies, and representative public interest groups.

We also believe that the process of formulat-

-ing the problem to be solved, the issues to be

addressed, and the detailed questions to be

answered should take place, primarily, before

and during the process of selecting experts.

The Committee's suggested approach is that

before deciding on the final panel of experts,

a much larger number of experts be contacted and their input be elicited on refining the general problem that has been formulated

by the generalists and the normative experts.

In this way, a much larger knowledge base is

available to fine tune the issues, and the

opportunity exists for a very effective group

of experts to evolve that will eventually make

up the panel. Further refinement of the issues

F-1 F-i .. ~ NUREG-1563 ~ ~ ~ ~ 
and questions should be performed by the

selected panel of subject-matter experts.

Response

Regarding the first portion of the ACNW's

comment, the staff's intent behind Step No. 2

("Selection of Experts") of Technical Position No.

2 ('A Consistent and Systematic Process for

Elicitation Should be Applied") of the BTP was to

encourage the enlistment of qualified individuals

representing or affiliated with the organizations/

entities identified by the ACNW in its comment.

To make the staff's intent clearer, the first two

sentences of the fifth paragraph of the "Discussion" section (Section 4) behind Technical Position

No. 2/Step No. 2 have been modified as follows:

In selecting each of the three types of experts,

especially the subject-matter experts, it may be useful to seek qualified nominations from

outside sources, or recognized peers in the

field. This would include, for example: the

National Academies of Sciences and

Engineering; academia; recognized

professional societies (e.g., Sigma Xi, The

Geological Society of America, the American

Society of Civil Engineers); National

laboratories; knowledgeable Federal Agencies and International Organizations; private

industry; State development and regulating bodies; representative public interest groups;

and interested stakeholders.

Apropos the second portion of the ACNW's

comment, regarding the timing of final subjectmatter expert selection and problem formulation

(Step No. 3-"Identification of Issues and

Problem Decomposition" of Technical Position

No. 2), the staff believes that the BTP is

compatible with the ACNW's views on this issue.

Although the BTP may not state so in the same

way the ACNW has, the staff believes that the

BTP recognizes that it may be appropriate to

"iterate" on problem formulation (Step No. 3)

before subject-matter expert selection (Step No. 1)

is finalized, as noted in the third introductory

paragraph (in Section 4, "Discussion") to

Technical Position No. 2:

In addition, although the process steps are

listed in numerical order, it is not necessary that the individual steps be performed in the

exact sequence presented. In fact, it is

expected that several of these process steps

will proceed or can be initiated concurrently,

subject to repeated iterations and oppor- tunities for feedback from the subject-matter

experts. This may be especially true for Step

Nos. 2, 3, and 4, which are depicted as

parallel process steps with feed-backs, as

shown in the preceding figure. What this

figure shows, for example, is that once the

subject-matter experts are identified (in Step

No. 2), they can help to better define the

objective of the elicitation (Step No. 1) and

thus aid in the decomposition of the

elicitation issue into its constituent parts

(Step No. 3). Moreover, the subject-matter

experts can also aid in the identification of

additional information that could facilitate

the elicitation (Step No. 4).

However, the staff's intent here may not be clear

to all readers. Accordingly, the staff has added

the following footnote to the paragraph in

question:

Alternatively, before deciding on the final

panel of subject-matter experts (Step No. 1),

in contacting candidate experts, it might be

useful to solicit their input on refining the

general problem (Step No. 3) that was

previously formulated by the generalists and

the normative experts. In this way, a much

larger knowledge base is available to

fine-tune the issues that will ultimately be

addressed by the final group of subjectmatter experts. Further, final refinement of

the sub-issues (and questions-Step No. 3)

should be performed ultimately by the

selected panel of subject-matter experts.

Moreover, the staff has revised the title of Step

No. 3-"Identification of Issues and Problem

Decomposition" of Technical Position No. 2 to

read as "Refinement of Issues and Problem

Decomposition" to better reflect the staffs intent

in this particular process step.

2. Aggregation of Results

The Committee believes that the results from

expert elicitation should clearly display the

uncertainties in the chosen performance

measures for a particular issue. Therefore,

the aggregation of the results of the expert

panel should also be clear in terms of the

NUREG-1563- F-L
uncertainties in the individual judgments of

the panel members and the method of

aggregation and integration of bottom-line

results that include the quantification of

uncertainties. This'property of the elicitation

process becomes' especially important to the

regulators in the consideration of multiple

elicitations covering similar or identical

issues. The scientific process considers a full

range of alternatives on the basis of the

technical knowledge base of each and the

associated reasoning processes, all of which

should be exposed in the decision-making

process. This documentation will facilitate the

regulator's ability to discriminate between

different alternatives on the basis of the

evidence presented.

In this regard, major guidance Would come

from an illustration of the aggregation

process that embraces the notion of

combining and integrating probability

distributions. The idea would not be to

prescribe a process but rather to illustrate in

graphical and analytical terms'an example of

what is meant by the aggregation process.'It

is believed that such an aid Would greatly facilitate and add meaning to-the use of

probability methods in the licensing process in general, and in expert elicitation in

particular.

Responsee

The staff agrees with this comment and notes that

the BTP is not inconsistent with the ACNW's

views. The BTP emphasizes the importance of an

individual expert's opinion. In this regard, the

BT?, as part of the "Documentation" process step

(Step No. 9 of Technical Position No. 2), notesthat the documentation associated with any.

'formal elicitation'should be adequate insofar as it

accurately reflects the subject-matter expert's

opinion but also captures the rationale for that

opinion. This would include the prelirinary'and

constituent opinions thatare used'to form a final

opinion. If, during the feedback process (Step No.

7), a subject-matter expert modifies his' or her

opinion, then the staff believes that both the

original (Step No. 6) and modified opinions (Step No. 6), along with the rationale for any change,

should be included as part of the elicitation

record (Step No. 9).

Accordingly, given the underlying "transparency"

theme of the BT, the staff expects that the

documentation of the aggregation method used

(Step No. 8), whatever that aggregation method

might be, must adequately record, among other

things, the individual subject-matter expert's

opinions, and their attendant effect on results of

the elicitation itself, so that they are also

traceable, within 'the aggregate. As noted in'the

BTP, the staff expressed no opinion regarding

what type of aggregation should be used in a

'formal elicitation because this issue is considered

beyond the scope of this guidance. Furthermore,

providing an example on this aspect of an expert elicitation is inconsistent with the level of detail

sought in this'particular guidance document.

However, the Center for Nucleair Waste

Regulatory Analyses' climate elicitation

(DeWispelare el al., 1993) does provide a

graphical example illustrating one type of

mechanical aggregation. NUREG-1489 (PRA

Working Group, 1994; pp. C-139-C-142) also

provides some detailed discussion regarding the

mathematical aggregation of expert's judgments.

3. Interpretation of the Results

-The Committee wishes to emphasize that as a

result of the flexibility of the process, the

applicant should not conclude that following

the guidance implies automatic acceptance of

the results. The results, and the detailed

bases thereof, are the desired outcome of the

elicitation process. The credibility of the.

results has to be principally based on the

individual's reasoning process, the method of

aggregation, and the supporting knowledge base, including the use of specific data

wherever possible.

Response

'The staff agrees with this comment and notes that

the BTP, in particular, the sixth and seventh

paragraphs in Purpose of the BTP" (Section 1.4),

* address this issue.

4. a Application of Expert Elicitation

Although the Committee was pleased that the

BTP was not overly prescriptive on the

matter of how to conduct expert elicitations,

there is a need for additional guidance on

candidate issues for application. A discussion

F-3 NUREG-1563
- -

of appropriate applications would illustrate

the limitations and the general intent of the

process.

Response

As noted in the staff's response to DOE's second

overall comment (in Appendix D), on the need for

additional guidance in the area of performance

assessment, the staff will take this comment into

account in due course but has made no decision

as yet on this option. In view of the possibility

that new performance measures may be

established for a potential repository at Yucca

Mountain, pursuant to the Energy Policy Act of

1992, and consistent with the findings and

recommendations of the recent NAS report (see

National Research Council, 1995), the staff wishes

to provide for the possibility that site-specific

standards may necessitate the development of

additional implementing guidance. The staff,

therefore, continues to leave open this option, as

noted in Section 14, paragraph 3 of the BTP (e.g.,

that it ". . . is considering the potential need for

future guidance to identify those specific aspects

of a performance assessment for which the

application of expert judgment may or may not be

appropriate... ."). Currently, the staff has no

plans for, or resources devoted to, preparing any

additional guidance in this area. If the staff

decides that additional guidance should be

considered, it will explain its reasons and offer the

opportunity for comment by the ACNW (and

others).

Observation

Additionally, the Committee believes that the

Commission, consistent with its Policy

Statement on probabilistic risk assessment

(see NRC, 1995; 60 FR 42623), may wish to

examine the decision-making process to take

greater advantage of results developed

through state-of-the-art expert elicitations.

For example, there may be an impact on the

admissibility for testimony of a valid

elicitation resulting from the unavailability of

one or more subject matter experts. Although

there are legal arguments for the need for a

''sponsoring witness," such an individual may

not be able to represent, as his or her own,

the full range of the technical arguments

contained in the original elicitation.

Response

This comment concerns Commission policy and

adjudicatory issues that are beyond the scope of

the BTP.

References

DeWispelare, A.R., et al., "Expert Elicitation of

Future Climate in the Yucca Mountain VicinityIterative Performance Assessment Phase 2.5," San

Antonio, Texas, Center for Nuclear Waste

Regulatory Analyses, CNWRA 93-016, August

1993. [Prepared for the U.S. Nuclear Regulatory

Commission.]

Geomatrix Consultants/TRW Environmental

Safety Systems, Inc., "Probabilistic Volcanic

Hazard Analysis for Yucca Mountain, Nevada,"

San Francisco, California, Document No.

BAOOOOOOO-1717-2200-00082 (Rev. 0), June 1996.

[Prepared for the U.S. Department of

Energy/Office of Civilian Radioactive Waste

Management.]

National Research Council, "Technical Bases for

Yucca Mountain Standards," Washington, D.C.,

National Academy Press, Commission on

Geosciences, Environment, and Resources, July

1995.

PRA Working Group, "A Review of NRC Staff

Uses of Probabilistic Risk Assessment," U.S.

Nuclear Regulatory Commission, NUREG-1489,

March 1994.

U.S. Nuclear Regulatory Commission, "Use of

Probabilistic Risk Assessment Methods in

Nuclear Regulatory Activities [Final Policy

Statement]," Federal Register, Vol. 60, No. 158,

August 16, 1995, pp. 42622 - 42630.

NUREG-1563 F-4
NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER

(2-89) (Assigned by NRC, Add Vol.,

NRCM 1102, Supp., Rev., and Addendum Num3201, 3202 BIBLIOGRAPHIC DATA SHEET bars If any.)

(See nstructions on the reverse)

2. TITLE AND SUBTITLENUREG-1563

Branch Tchnical Position on the Use of Expert Elicitation in the 3. DATE REPORT PUBLISHED

High-Level Radioactive Waste Program MONTH I YEAR

November 1996

4. FIN OR GRANT NUMBER

5. AUTHOR(S) 6. TYPE OF REPORT

J. P. Kotra*, M. P. Lee*, N. A. Eisenberg*, and A. R. DeWispelare** Technical

7. PERIOD COVERED (Inclusive Dates)

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (it NRC, provide Division, Office or Region, U.S. Nuclear Regulatory Commission, and

maling address; I contractor, provide name and maling address.)

Division of Waste Management **Center for Nuclear Waste Regulatory Analyses

Office of Nuclear Material Safety and Safeguards 6220 Culebra Road

Washington, DC 20555-0001 San Antonio, TX 78228-0510

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (it NRC, type Sarne as above'; If contractor, provide NRC Division, Office or Region, U.S. Nuclear Regulatory Commission, and mailIng address.)

Same as Item 8, above

10. SUPPLEVENTARY NOTES

11. ABSTRACT (200 words or less)

The U.S. Nuclear Regulatory Commission staff expects that subjective judgments of individual experts and, in some

cases, groups of experts, will be used by the U.S. Department of Energy (DOE) to interpret data obtained during site

characterization and to address the many technical issues and inherent uncertainties associated with predicting the performance of a repository system for thousands of years. NRC has traditionally accepted, for review, expert judgment to

evaluate and interpret the factual bases of license applications and is expected to give appropriate consideration to the

judgments of DOE's experts regarding the geologic repository. Such consideration, however, envisions DOE using expert judgments to complement and supplement other sources of scientific and technical information, such as data collection, analyses, and experimentation. In this document, the NRC staff has set forth technical positions that: (1) provide general guidelines on those circumstances that may warrant the use of a formal process for obtaining the judgments of more than one expert; and (2) describe acceptable procedures for conducting expert elicitation when formally

elicited judgments are used to support a demonstration of compliance with NRC's geologic disposal regulation, currently set forth in 10 CFR Part 60.

12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers In locating the report.) 13. AVAILABILITY STATEMENT

Unlimited

14. SECURITY CLASSIFICATION

(This Page)

Expert elicitation Peer review Unclassified

Expert judgment Performance assessment (This Report)

Geologic repository Probabilistic risk assessment Unclassified

High-level waste Yucca Mountain 1s. NUMBER OF PAGES

16. PRICE

NRC FORM 335 (2-89)
Federal Recycling Program