Nuclear Science
NEA/NSC/R(2019)7
March 2023
www.oecd-nea.org

Existing Practices forMulti-physics Validation
Report on Sub-Task 3,
Task Force 2

NEA/NSC/R(2019)7  1
Nuclear Energy Agency

NEA/NSC/R(2019)7
Unclassified

English text only
2 March 2023

NUCLEAR ENERGY AGENCY
NUCLEAR SCIENCE COMMITTEE

Cancels & replaces the same document of 4 October 2022

Existing Practices for Multi-physics Validation
Report on Sub-Task 3, Task Force 2This document is available in PDF format only.

JT03513412
OFDE

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION
Unclassified
This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory,
to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.

2  NEA/NSC/R(2019)7
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENTThe OECD is a unique forum where the governments of 38 democracies work together to address the economic,
social and environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand andto help governments respond to new developments and concerns, such as corporate governance, the information
economy and the challenges of an ageing population. The Organisation provides a setting where governments cancompare policy experiences, seek answers to common problems, identify good practice and work to co-ordinate
domestic and international policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, Chile, Colombia, Costa Rica, the Czech
Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan,
Korea, Latvia, Lithuania, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, theSlovak Republic, Slovenia, Spain, Sweden, Switzerland, Türkiye, the United Kingdom and the United States. The
European Commission takes part in the work of the OECD.
OECD Publishing disseminates widely the results of the Organisation’s statistics gathering and research on
economic, social and environmental issues, as well as the conventions, guidelines and standards agreed by its
members.

NUCLEAR ENERGY AGENCYThe OECD Nuclear Energy Agency (NEA) was established on 1 February 1958. Current NEA membership consistsof 34 countries: Argentina, Australia, Austria, Belgium, Bulgaria, Canada, the Czech Republic, Denmark, Finland,
France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands,
Norway, Poland, Portugal, Romania, Russia (suspended), the Slovak Republic, Slovenia, Spain, Sweden,
Switzerland, Türkiye, the United Kingdom and the United States. The European Commission and the International
Atomic Energy Agency also take part in the work of the Agency.
The mission of the NEA is:– to assist its member countries in maintaining and further developing, through international co-operation, thescientific, technological and legal bases required for a safe, environmentally sound and economical use of
nuclear energy for peaceful purposes;– to provide authoritative assessments and to forge common understandings on key issues as input to government
decisions on nuclear energy policy and to broader OECD analyses in areas such as energy and the sustainable
development of low-carbon economies.
Specific areas of competence of the NEA include the safety and regulation of nuclear activities, radioactive waste
management and decommissioning, radiological protection, nuclear science, economic and technical analyses of thenuclear fuel cycle, nuclear law and liability, and public information. The NEA Data Bank provides nuclear data andcomputer program services for participating countries.

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to thedelimitation of international frontiers and boundaries and to the name of any territory, city or area.
Corrigenda to OECD publications may be found online at: www.oecd.org/about/publishing/corrigenda.htm.
© OECD 2023
You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own
documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of the OECD as source and copyright owner is given. All requests
for public or commercial use and translation rights should be submitted to neapub@oecd-nea.org. Requests for permission to photocopy portions of this material for public
or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at info@copyright.com or the Centre français d'exploitation du droit de copie (CFC)
contact@cfcopies.com.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  3

Foreword
The modelling and simulation of nuclear systems allow scientists and engineers to probe
aspects of the operation of these systems beyond what has been precisely measured. This
has numerous applications, including interpolation between known operational scenariosand extrapolation into the unknown, providing information to underpin safety, optimise
performance and design new facilities and experiments. As computational resources haveincreased in the past decades and with higher-fidelity modelling approaches, leveraging
novel simulation methods has become critical to the nuclear industry. However, thepotential value of all simulation is tightly bound to the body of evidence, which supports
the accuracy of the methods in the application domains.
The Organisation for Economic Co-operation and Development (OECD) Nuclear Energy
Agency (NEA) Expert Group on Multi-physics Experimental Data, Benchmarks andValidation (EGMPEBV) was created in 2014, under the aegis of the Nuclear Science
Committee (NSC), to address the needs of member countries in certifying and utilisingbenchmarks for the validation of novel multi-physics codes. These benchmarks and theguidance on their use will help experts validate their codes, enabling their use in the wider
nuclear community. Under the second task force created in the EGMPEBV, experts
compared their approaches and their relative strengths and limitations, including a criticalreview of the current state-of-the-art in verifying, validating and performing uncertainty
quantification with coupled multi-physics simulation.
This report provides a self-contained review of the current approaches for single and multiphysics validation, including both coupled and uncoupled uncertainty quantification andpropagation. Through a detailed inter-comparison study, this report compiles the common
elements that are required for best practices.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

4  NEA/NSC/R(2019)7

Acknowledgements
The Nuclear Energy Agency (NEA) expresses its sincere gratitude to Dr Maria Avramova(North Carolina State University, United States), who co-ordinated the efforts of the Expert
Group on Multi-Physics Experiments, Benchmarks and Validation on existing practices for
multi-physics validation, and for preparing this report.
The NEA acknowledges the work of A. Boldyrev (Russian Federation), D. Cuervo (Spain),
M. DeHart (United States), J. Hou (United States), E. Ivanov (France), K. Ivanov (UnitedStates), E. Moiseenko (Russian Federation), N. Mosunova (Russian Federation),
A. Petruzzi (Italy), U. Rohatgi (United States), S. Shevchenko (Russian Federation) and
K. Velkov (Germany), who contributed to the report, and to J. Dyrda (NEA), J.P. Hudelot(France), and T. Valentine (United States), who reviewed the content.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  5

Table of contents
List of abbreviations .............................................................................................................................. 8
Executive summary ............................................................................................................................. 14
1. Introduction and definition of terms ............................................................................................. 15
1.1 Background .................................................................................................................................. 15
1.2 Objective of the present report ..................................................................................................... 17
1.3 Structure of the report .................................................................................................................. 17
1.4 Definitions ................................................................................................................................... 17
2. Validation of single physics............................................................................................................. 25
2.1 Core reactor physics..................................................................................................................... 25
2.2 Fuel performance ......................................................................................................................... 28
2.3 Core thermal-hydraulics............................................................................................................... 33
2.4 System thermal-hydraulics........................................................................................................... 35
Reactor system and containment coupled codes ............................................................................ 35
The scaling issue ............................................................................................................................ 37
3. Validation of multi-physics codes − Examples and current approaches .................................... 40
3.1 Multi-physics validation hierarchy and multi-level validation methodology .............................. 40
3.2 Coupled thermal-hydraulics system/neutron kinetics codes ........................................................ 41
3.3 Coupled thermal-hydraulics core/neutronics simulators.............................................................. 45
3.4 Coupled neutronics/fuel performance simulators ........................................................................ 46
3.5 Multi-physics multi-scale code systems ...................................................................................... 47
4. Uncertainty quantification approaches ......................................................................................... 49
4.1 Overview of uncertainty methodologies ...................................................................................... 49
4.2 Propagation of input errors methodology .................................................................................... 50
Sources of uncertainties ................................................................................................................. 51
4.3 Summary of general frameworks for uncertainty quantification ................................................. 52
Code scaling applicability and uncertainty (CSAU) ...................................................................... 52
GRS Uncertainty quantification methodology ............................................................................... 57
GRS Sampling-based uncertainty analysis codes .......................................................................... 58
Automated Statistical Treatment of Uncertainty Method (ASTRUM) .......................................... 60
Uncertainty Methodology based on Accuracy Extrapolation − UMAE ........................................ 62
Code with capability of IAU (CIAU) and code with capability of IAU for coupled thermalhydraulic/neutronics calculation (CIAU-TN) ................................................................................ 64
4.4 New methods ............................................................................................................................... 70
CASUALIDAD method ................................................................................................................. 70
IMTHUA − Integrated Methodology for Thermal–Hydraulics Uncertainty Analysis.................. 78
SEC NRS Methodology ................................................................................................................. 80
IBRAE RAN Methodology ............................................................................................................ 83

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

6  NEA/NSC/R(2019)7
4.5 Traditional single physics uncertainty quantification (UQ) ......................................................... 84
Concepts ......................................................................................................................................... 84
Propagation of uncertainties in single physics calculations ........................................................... 85
4.6 Traditional multi-physics uncertainty quantification (UQ).......................................................... 87
Propagation of uncertainties in multi-physics calculations ............................................................ 87
5. Summary of validation and uncertainty quantification state of practice ................................... 91
5.1 Summary of practices for validation and uncertainty quantification for single physics codes .... 91
System analysis codes .................................................................................................................... 91
Neutronics (reactor physics) codes................................................................................................. 94
Fuel performance codes ................................................................................................................. 95
Core thermal-hydraulics codes ....................................................................................................... 97
5.2 Summary of practices for validation and uncertainty quantification for multi-physics codes... 101
5.3 Use of Best Estimate Plus Uncertainty (BEPU) by regulation in licensing calculations ........... 105
Phenomena identification and ranking table (PIRT) Process ....................................................... 107
Code validation and uncertainty analyses in licensing ................................................................. 113
6. Conclusions .................................................................................................................................... 125
References .......................................................................................................................................... 126

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  7

TablesTable 2.1. HZP validation results .......................................................................................................... 28
Table 4.1. Minimum number N of calculations..................................................................................... 50

FiguresFigure 1.1. Phases of validation process................................................................................................ 19
Figure 1.2. Design and utilisation of validation experiments ................................................................ 21
Figure 1.3. Concept for multi-physics model validation ....................................................................... 21
Figure 1.4. Multi-physics validation experiment hierarchy .................................................................. 22
Figure 1.5. Multi-physics coupling mechanisms and feedback effects in a reactor core ...................... 22
Figure 1.6. Multi-physics coupling mechanisms in LWR core ............................................................. 23
Figure 1.7. Multi-physics coupling mechanisms for short-time transient phenomena in
LWR core ............................................................................................................................ 23
Figure 1.8. Multi-physics validation hierarchy ..................................................................................... 24
Figure 2.1. Histogram of HZP BOC measured – predicted differences for critical Boron ................... 27
Figure 2.2. Radial discretisation for the finite differencing method used in
FRAPCON/FRAPTRAN models ........................................................................................ 29
Figure 2.3. Measured and predicted by FRAPCON centreline temperature for the
UO2 assessment cases throughout life ................................................................................. 32
Figure 2.4. FRAPTRAN-1.5 predictions of RIA FGR for CABRI and NSRR UO2 rods ..................... 32
Figure 2.5. Comparisons of the overall bundle pressure drop CTF predictions vs.
measurements ...................................................................................................................... 34
Figure 2.6. Concept of coupled SPACE and CAP coder system .......................................................... 36
Figure 3.1. Steady-state core average axial void fraction (mean and std. deviation) ............................ 42
Figure 3.2. Steady-state core average normalised axial power distribution (measured vs.
mean and standard deviations) ............................................................................................ 43
Figure 3.3. Best-estimate case ̶ Transient fission power comparison (average vs. measured –zoom) ................................................................................................................................... 43
Figure 3.4. Best-estimate case – Transient dome pressure (average vs. measured) .............................. 44
Figure 4.1. Illustration of the PIE method and results interpretation .................................................... 52
Figure 4.2. The CSAU methodology flow chart ................................................................................... 54
Figure 4.3. UMAE flow diagram (also adopted within the process of development
and application of CIAU) .................................................................................................... 64
Figure 4.4. Accuracy behaviour as a function of Kv ............................................................................. 66
Figure 4.5. Definition of quantity and time errors to be included into the quantity and time
uncertainty database ........................................................................................................... 67
Figure 4.6. The error filling process and the error extraction process .................................................. 69
Figure 4.7. Uncertainty evaluation based on CASUALIDAD methodology ....................................... 71
Figure 4.8. The basic idea and framework of the CASUALIDAD method .......................................... 73
Figure 4.9: Flow chart of CASUALIDAD method: Development process .......................................... 75
Figure 4.10. Flow chart of CASUALIDAD method: Application process ........................................... 78
Figure 4.11. Process for evaluation of code output error ...................................................................... 82
Figure 4.12. Key stages in assessment of model parameters uncertainty .............................................. 82
Figure 4.13. Validation procedure according to IBRAE RAN methodology........................................ 84

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

8  NEA/NSC/R(2019)7

List of abbreviations
1D

One-dimensional

2D

Two-dimensional

3D

Three-dimensional

APEX

Advanced plant EXperiment

ASAP

Adjoint sensitivity analysis procedure

ASTRUM

Automated statistical treatment of uncertainty method

ATWS

Anticipated transient without scram

BCL

Battelle Columbus Laboratory

BE

Best estimate

BEMUSE

Best-estimate methods – Uncertainty and sensitivity evaluation

BEPU

Best Estimate Plus Uncertainty

BFBT

Full-size Fine-Mesh Bundle Tests

BIC

Boundary and initial conditions

BOC

Beginning of cycle

BOL

Beginning of life

BT

Bayesian Theorem

BWR

Boiling water reactor

CANDU

CANada deuterium uranium

CASL

Consortium for advanced simulations of Light Water Reactors

CASUALIDAD

Code with the capability of Adjoint Sensitivity and Uncertainty AnaLysis by
Internal Data ADjustment and assimilation

CAP

Nuclear Containment Analyses Package

CEA

Alternative Energies and Atomic Energy Commission (Commissariat à l'énergie
atomique et aux énergies alternatives, France)

CCTF

Cylindrical core test facility

CCVM

CSNI code validation matrix

CDF

Cumulative distribution function

CFD

Computation fluid dynamics

CFR

Code of federal regulations

CHF

Critical heat flux
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  9

CIAU

Code with capability of IAU

CIAU-TN

Code with capability of IAU for coupled thermal-hydraulic/neutronics calculation

CSAU

Code scaling, applicability and uncertainty

CSNI

Committee on the Safety of Nuclear Installations (NEA)

CT

Computer tomography

CTF

Core test facility

CWO

Core-wide oxidation

DAA

Data adjustment/assimilation

DAKOTA

Design Analysis Kit for Optimisation and Tera-scale Applications

DNB

Departure from nucleate boiling

DNBR

Departure from nucleate boiling ratio

ECC

Emergency core cooling

ECCS

Emergency core cooling system

EG

Expert group

EGMPEBV

Expert Group on Multi-physics Experimental Data, Benchmarks and Validation
(NEA)

EGUAM

Expert Group on Uncertainty Analysis in Modelling (NEA)

EM

Evaluation model

EMDAP

Evaluation model development and assessment

ENDF

Evaluated Nuclear Data File library (United States)

EOC

End of cycle

EOL

End of life

ESBWR

Economic simplified boiling water reactor

FGR

Fission gas release

FOM

Figure of merit

FP

Fuel performance

FSAP

Forward sensitivity analysis procedure

FSA

Fractional scaling analyses

FSAR

Final safety analysis report

FFTBM

Fast Fourier transform-based method

GASAP

Global adjoint sensitivity analysis procedure

GRS

Gesellschaft für Anlagen- und Reaktorsicherheit gGmbH (Germany)GUI

Graphical user interface

H2TS

Hierarchical two-tiered scaling

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

10  NEA/NSC/R(2019)7
HFP

Hot full power

HZP

Hot zero power

HPI

High-pressure injection

IBRAE RAN

Nuclear Safety Institute of the Russian Academy of Sciences

ICSBEPInternational Criticality Safety Benchmark Evaluation Project (NEA)IFPEInternational Fuel Performance Experiments (NEA)

IRPhE

International Reactor Physics Experiment Evaluation (NEA)

IAU

Internal assessment of uncertainty

IE

Improved estimate

IET

Integral effect test

IMTHUA

Integrated methodology for thermal-hydraulics uncertainty analysis

INL

Idaho National Laboratory (United States)

IRWST

In-containment refuelling water storage tank

ISP

International standard problem

ITC

Isothermal temperature coefficient

ITF

Integral test facility

JEFF

Joint Evaluated Fission and Fusion library (Europe)KS

Kolmogorov/Smirnov

LBLOCA

Large break loss-of-coolant accident

LMO

Local maximum oxidation

LOCA

Loss-of-coolant accident

LOFT

Loss-of-fluid test

LSTF

(Japanese) Large Scale Test Facility

LWR

Light water reactor

MOX

Mixed-oxide

MP

Multi-physics

MSLB

Main steam line break

M&S

Modelling and simulation

NA-SA

Nucleoeléctrica Argentina

NDC

Nuclear Development Committee (NEA)

NEA

Nuclear Energy Agency

NF

Nuclear facility

NJOY

Nuclear Data Processing Code

NRS

Nuclear and radiation safety

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  11

NSC

Nuclear Science Committee (NEA)

NUPEC

NUclear Power Engineering Corporation

OECD

Organisation for Economic Co-operation and Development

PN

Spherical harmonics method

PCI

Pellet-clad interaction

pcm

Per cent mille

PCMI

Pellet-clad mechanical interaction

PCT

Peak cladding temperature

PDF

Probability density function

PHWR

Pressurised heavy water reactor

PIE

Propagation of input errors

PIRT

Phenomena identification and ranking table

PKL

Primarkreislauf − large-scale test facility

PML

Principle of the maximum likelihood

ppm

Parts per million

PP

Properties and phenomena

PREMIUM

Post-BEMUSE reflood Model Input Uncertainty Methods

PSBT

PWR sub-channel and bundle tests

PTS

Pressurised thermal shock

PVM

Parallel virtual machine

PWR

Pressurised water reactor

QoI

Quantities of interest

QPIRT

Quantified Phenomena Identification and Ranking Table

R&D

Research and development

REA

Rod ejection accident

RIA

Reactivity initiated accident

rms

Root mean square

ROM

Reduced order modelling

RP

Reactor physics

SAPIUM

Systematic APproach for Input Uncertainty quantification Methodology

SBLOCA

Small break loss-of-coolant accident

SCC

Stress corrosion cracking

SCCRED

Standardised and Consolidated Calculated and Reference Experimental Database

SCTF

Slab core test facility

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

12  NEA/NSC/R(2019)7
SCWR

Super critical water reactor

SEC

Scientific and Engineering Centre

SET

Separate effect test

SETF

Separate effect test facility

SETS

Stability enhancing two-step

SFR

Sodium fast reactor

SN

Discrete ordinates method

SMR

Small modular reactor

SRV

Safety relief valvesSS

Steady-stateST

Sub-task

SPACE

Safety and Performance Analysis Code

SUSA

Software for Uncertainty and Sensitivity Analysis

S&U

Sensitivity and uncertainty

TF

Task force

TH

Thermal-hydraulics

TIETHYSThe International Experimental Thermal HYdraulics Systems database (NEA)

TMI-1

Three Mile Island Unit 1 (United States)

TREND

Treatment and Representation of Evaluated Nuclear Data

SEC NRS

Scientific and Engineering Centre for Nuclear and Radiation Safety of the Russian
regulatory authority (Rostechnadzor)

TREATTransient Reactor Test Facility (Idaho National Laboratory, United States)

TT

Turbine trip

UMS

Uncertainty method study

UAM

Uncertainty analysis in modelling

UMAE

Uncertainty methodology based on accuracy extrapolation

UPI

Upper plenum injection

UPTF

Upper Plenum Test Facility (Germany)

US DOE

United States Department of Energy

US NRC

United States Nuclear Regulatory Commission

UQ

Uncertainty quantification

VVER

Water-water energetic reactor (Russia)

V&V

Verification and validation

VUQ

Validation and uncertainty quantification

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  13

WGAMA

Working Group on Accident Management and Analysis (NEA)WPRSWorking Party on Scientific Issues of Reactor Systems (NEA)

XSUSA

Cross-section uncertainty and sensitivity analysis

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

14  NEA/NSC/R(2019)7

Executive summary
The Nuclear Energy Agency (NEA) Expert Group on Multi-physics Experimental Data,
Benchmarks and Validation (EGMPEBV) was created in 2014 under the auspices of the
NEA Nuclear Science Committee (NSC) to establish the processes for the certification ofexperimental data and development of benchmark models for validation of Multi-Physics(MP) Modelling and Simulation (M&S) computational systems. The aim of the group is toprovide member countries with guidelines and recommendations for validating andimproving their novel multi-physics simulations and access to key experimental data.
The organisation of the EGMPEBV relies on three task forces (TFs), focused on:1) experimental data qualification and benchmark evaluation; 2) methods and standards;
and 3) specific applications.
Task Force 2 was designed to develop validation methods and guidelines and uncertainty
qualification for the multi-physics, multi-scale codes in the context of emerging demands
such as longer fuel cycles and power uprate.
Task 3 of Task Force 2 was designed to develop guidelines for current (traditional) multiphysics validation and uncertainty quantification. The present report summarises thecurrent practice in the validation process, discusses uncertainty analyses and current
methodologies used to extrapolate beyond the validation domain.
Code developers, research organisations and professional societies have developed various
practices for validation of M&S tools and data. The approaches may vary depending on thelimitations of the M&S tools and data as well as the intended application of such M&S
tools. This report summarises some of the primary practices, including recommendations
and guidelines that have been developed within NEA member countries.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  15

1. Introduction and definition of terms

1.1 Background
The Expert Group on Multi-physics Experimental Data, Benchmarks and Validation(EGMPEBV) was formed under the auspices of the Nuclear Energy Agency (NEA)
Nuclear Science Committee (NSC) in 2014 [1]. The expert group (EG) has the task ofestablishing the processes for the certification of experimental data and development ofbenchmark models for validation of Multi-Physics (MP) Modelling and Simulation (M&S)
computational systems. In addition, this EG will seek to establish appropriate processes
and procedures for the use of data and benchmark models for validation of M&S tools anddata. In detail, the objectives of the EG are to provide member countries with:

the guidance and processes for certifying experimental data for its use as abenchmark or for its use in general testing of modelling and simulation tools;

access to certified experimental data from the contributions of individual member
countries;

guidance and recommendations for developing benchmark models from certified
experimental datasets;

access to standardised benchmark models with detailed uncertainty evaluations
and uncertainty methodology guidelines;

recommendations and guidelines for the range of applicability of the certified
experimental datasets;



guidelines and consensus recommendations for validating MP simulations;a limited number of demonstrations of the validation recommendations.

The EGMPEBV is currently organised into three task forces:

Task Force 1 on Experimental Data Qualification and Benchmark Evaluation: The
TF-1 team has the aim first to identify the main challenges for validating MP M&S
tools and then to develop and implement processes for evaluating and producing
experimental datasets that could be used for validating novel MP M&S tools.

Task Force 2 on Validation Guidelines and Needs: The TF-2 team has been formed
to identify needs and to develop best practices for development of methodologies
for validation.



Task Force 3 on Example Application of Validation Experiments: The TF-3 teamhas the objective to organise MP M&S benchmarks involving validation
experiments.

In order to achieve the above objectives and given the multidisciplinary nature of theactivities, the three TF teams will liaise closely with other relevant NEA working parties
and expert groups, including those operating under the guidance of the Nuclear

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

16  NEA/NSC/R(2019)7
Development Committee (NDC) and the Committee on the Safety of Nuclear Installations(CSNI), to ensure that the respective programmes are complementary, to provide advice
and support where required and undertake common work where appropriate. Several
different expert groups of the NEA have established rigorous processes for theaccumulation and preservation of existing and new experimental data for single physics
phenomena. Under the auspices of:

Nuclear Science Committee (NSC):–

the International Criticality Safety Benchmark Evaluation Project (ICSBEP);–

the International Reactor Physics Experiment Evaluation (IRPhE) Project;–

the International Fuel Performance Experiments (IFPE) database;Committee on the Safety of Nuclear Installations (CSNI):–

the CSNI Code Validation Matrix (CCVM) of Separate Effect Test Facilities(SETFs);–the CSNI Code Validation Matrix (CCVM) of Integral Test Facility (ITF).

Several processes have been developed by which experimental data is “qualified” for use
in benchmarks and several benchmark models are developed for the validation of modellingof single physics or limited multi-physics phenomena.
Additionally, the Expert Group on Uncertainty Analysis in Modelling (EGUAM) within
the Working Party on Scientific Issues of Reactor Systems (WPRS) has been leading aninternational effort to establish the processes for performing coupled neutronics andthermal-hydraulics simulations for a variety of reactor types but with a focus on light water
reactors (LWRs). The efforts of these working groups will form the basis for the processes
and priorities of the EGMPEBV to reduce duplication and take advantage of the knowledge
built by these groups over the years.
The EGMPEBV effort will strive to leverage the efforts of the NSC and the CSNI into ademonstrated methodology for qualifying experimental data, creating “benchmarks” andestablishing validation guidelines for MP M&S tools.
The principal objectives of Task Force 2 initiative are:

development of consensus guidelines for validation of multi-physics M&S tools
and data;development of guidelines for performing uncertainty qualification and evaluating
ranges of applicability for predicting M&S performance outside of the validation
domain;

identifying the needs for specific experiments with the intended purpose ofvalidating multi-physics M&S tools and data.

The present report summarises the status of the existing practices for multi-physics
validation. Code developers, research organisations and professional societies havedeveloped various practices for validation of M&S tools and data. The approaches mayvary depending on the limitations of the M&S tools and data as well as the intended
application of such M&S tools. The report is developed that summarises some of the
primary practices, including recommendations and guidelines that have been developed
within the member countries of the NEA.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  17

1.2 Objective of the present report
This report identifies several commonly used assessment frameworks and summarises thegeneral approach, the strengths, the weaknesses and ease of use of these frameworks. The
report includes an overview of the phenomena identification and ranking table (PIRT)
process along with the key issues that must be considered when identifying relevant
physical phenomena and the coupling among the phenomena such as thermal-hydraulics
and neutronics. The report includes a summary of the principal issues to consider whencoupling physical phenomena that have varying degrees of fidelity in the temporal, spatial
and/or energy domains. Likewise, the report summarises current approaches for estimating
total uncertainty at various levels of the validation hierarchy and the cautions andchallenges in such estimates. The report also provides recommendations as to the treatmentof experimental uncertainty and the inference of measured data with respect to simulated
responses of interest. A critical topic in estimating the total uncertainty at the application
conditions of interest is how to estimate the model form uncertainty using validation
metrics. Finally, the report includes a summary of the primary recommendations for theuse of multi-physics modelling and simulation tools extrapolated beyond the validation
domain.

1.3 Structure of the report
The report is organised in six chapters including the introduction and definition of terms,
validation of single physics, validation of multi-physics, uncertainty quantification
approaches, summary of validation and uncertainty quantification (VUQ) state of practiceand conclusions.
Examples and current approaches for validation of single physics calculations are provided
in Chapter 2 while examples and current approaches for validation of multi-physics
simulations are given in Chapter 3. Uncertainty quantification frameworks for single andmulti-physics simulations are described in Chapter 4. A summary of VUQ practices isprovided in Chapter 5 and conclusions are given in Chapter 6.

1.4 DefinitionsFirst, some definitions will be introduced such as single physics, pre-traditional andtraditional multi-physics models, integral effects, separate effects and plant measurable
parameters. In addition, the definition of validation and uncertainty quantification as well
as discussion of traditional versus validation experiments will be provided.
Single physics modelling and simulation focus on single physics phenomena and consider
the interactions with other physics phenomena via boundary conditions.
Multi-physics modelling and simulation describes non-linear multi-physics phenomena by
on-line treatment of feedback effects of different physics.
While integral effects describe the behaviour of a reactor system at nominal, off-nominalor accident conditions, the separate effects describe the behaviour of single component orcharacteristics of one phenomenon.
Plant measurable parameters are integral parameters or local distributions, which can be directly
or indirectly measured during the plant operation and/or transient conditions. The obtained
measured values must be supplemented with corresponding experimental/measurement
uncertainties.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

18  NEA/NSC/R(2019)7
There are many interactions between different physics phenomena at different scale indifferent components of nuclear power plants. Especially important are the multi-physics
interactions in reactor cores. In the past, these different interactions were treated either asboundary conditions (i.e. each physics calculation was performed independently, and theimpact of other physics phenomena was taken into account through boundary conditions)or using very simplistic models for some of the physics phenomena. Examples for the latterare the point kinetics model implemented in the system thermal-hydraulic models or onedimensional thermal-hydraulic models implemented in neutronics core design codes. Thesesimulation models are defined as pre-traditional multi-physics tools.
The traditional multi-physics tools include mostly neutronics/thermal-hydraulics coupling
into reactor core on assembly/channel basis. The coupling of reactor core to the system andcoupling of the system to containment also belongs to traditional multi-physics simulations.
The novel multi-physics tools, which include high-fidelity coupling on pin/sub-pin (pinresolved)/sub-channel level of several physics’ phenomena in the reactor core such as
neutronics (reactor physics), thermal-hydraulics, fuel performance, structural mechanics,
chemistry, etc.
Verification and validation (V&V) and uncertainty quantification (UQ) are used as ameasure of performance of the reactor physics system. In the literature and among different
reactor vendors and regulatory agencies there are different terminologies of V&V and UQ.
The most common are the following definitions:verification is a process of determining that a model implementation accurately
represents the developer’s conceptual description of the model and the solution to
the model;

validation is a process of determining that a model is an adequate representation of
the real world from the perspective of the intended uses of the model;qualification is a process of determining the degree to which a model represents adomain of real-world solutions;

uncertainty quantification is a determination of the degree of accuracy of the model
representation for intended applications.

As discussed in [2], the process of determining the degree to which a computational model
provides an accurate representation of the real world from the perspective of the intended
uses of the code is generally referred to as validation. This is depicted in the schematicshown in Figure 1.1, which shows that the validation is performed in many phases.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  19

Figure 1.1. Phases of validation process

Source: Oberkampf and Trucano, 2004.

Although one of the goals of the various modelling and simulation projects is to reducereliance on expensive experimental validation, the complexity of these new computational
tools presents an even greater validation challenge. The amount of data needed to validatea single figure of merit (FOM) resolved over a fine mesh is orders of magnitude greaterthan that used for an integral code experiment. When dealing with multiple physics andscales, not only does the volume of data increase geometrically but one must account forthe coupling terms as well. Additional challenges arise from the fact that some of the key
parameters are not directly measurable but must be derived from other measurable
quantities, injecting even more uncertainty into the result. Furthermore, results of suchcomplex simulations are more sensitive to user effects due to many options, which are
difficult to isolate and quantify.
Thus, the difficulty of validating high-fidelity, multi-physics codes has rendered traditional
methods used for single physics and integral codes entirely inadequate. New experimental
and analytical techniques must be devised, tested and compared. In addition, new types ofexperiments are needed as validation experiments [2]. The objective of a validation
experiment is to determine the predictive accuracy of a model (or group of models), which
is different from objectives of the so-called traditional experiments. The traditional
experiments are usually categorised as:scientific discovery experiments (with objective to improve fundamental
understanding of physics (a process or phenomena) – for example departure from
nucleate boiling);calibration experiments (with objective of constructing and improving a model fora well-understood process i.e. determining parameters in existing mathematicalmodels – for example experiment for bubbly flows);

system performance experiments (with objective to determine reliability,
performance and/or safety of components or systems i.e. assessing subsystem orcomplete system performance – for example loss-of-coolant experiment).

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

20  NEA/NSC/R(2019)7
According to [3], a model validation experiment is an experiment that is designed andexecuted to quantitatively estimate a mathematical model’s ability to simulate awell-characterised experiment. The customer of a model validation experiment is usually amodel developer or computational analyst. The six characteristics of a validation
experiment are defined as follows [3]:a validation experiment should be jointly designed and executed byexperimentalists and computation specialists (close working relationship during thewhole process);a validation experiment should be designed to capture the relevant physics, allinitial and boundary conditions, and all auxiliary data needed for a simulation(computational simulation input data should be measured in the experiment and key
modelling assumptions understood as well as characteristics and imperfections of
the experimental facility should be measured and included in the simulation);

a validation experiment should use any possible synergisms between experimentand computational approaches (offset strengths and weaknesses of computation andexperiment);

independence between computational and experimental results should bemaintained where possible (the flavour of a blind comparison should be maintained
if possible and all input data needed for the simulation should be measured andprovided);a hierarchy of experimental measurements should be made, which presents anincreasing range of computational difficulty (the data should include functionals,
local variables, derivatives of local variables and computational solution data
should be processed in a manner similar to the experimental measurement data);

a validation experiment must carefully employ experimental uncertainty analysis
procedures to delineate and quantify random and correlated bias errors(experimentalist should provide uncertainty estimates on system response data andinput quantities needed by the code, use traditional or statistical design ofexperiments methods to estimate random and correlated bias errors inmeasurements, and if possible, conduct experiments using different diagnostic
techniques or different experimental facilities).

The design and utilisation of validation experiments, as proposed by Professor Dr Paul
Turinsky from North Carolina State University (NCSU), are illustrated in Figure 1.2.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  21

Figure 1.2. Design and utilisation of validation experiments

Source: NCSU, 2019.

During the past several years, methodological guidelines and procedures for designing andconducting a validation experiment have been developed [2]. These guidelines have beenutilised by Professor Dr Nam Dinh from NCSU to develop a concept for multi-physics
model validation as shown in Figure 1.3. The validation experiments provide a rigorous
foundation for validation of multi-physics simulations. Schematics of the validation
experiment hierarchy for a multi-physics model benchmark is depicted in Figure 1.4.
Figure 1.3. Concept for multi-physics model validation

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

22  NEA/NSC/R(2019)7
Figure 1.4. Multi-physics validation experiment hierarchy

Source: NCSU, 2019.

The multi-physics coupling mechanisms and feedback effects in a reactor core are shownin Figure 1.5. These interactions occur at different time scales, which we can separate asshort (seconds and minutes) and long (months and years) time phenomena. The utilised
approach is to design separate series of progressive validation experiments (to assess theimpact of individual physics on multi-physics simulations) for both short and long multiphysics time phenomena in a reactor core.
Figure 1.5. Multi-physics coupling mechanisms and feedback effects in a reactor core

Source: CEA, 2019.

The currently developed reactor modelling and simulation capabilities that couple multiple
physics phenomena have the following validation challenges:consistent multi-physics validation path from single- to two-, and to n-physics
validation;

complexity and high fidelity of these new computational tools;using new types of experiments ̶ validation experiments.

Usually in LWR core simulations the modelling of structural mechanics could be neglectedthus reducing the multi-physics coupling mechanisms to four physics as shown in Figure
1.6.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  23

Figure 1.6. Multi-physics coupling mechanisms in LWR core

Source: NCSU, 2019.

The short-time transient phenomena in a LWR core is of high importance and is related toindustry safety concerns such as design basis and anticipated transients without scram. The
multi-physics coupling in this case includes coupling of neutronics reactor physics (RP),thermal-hydraulics (TH), and fuel physics fuel performance (FP) as shown in Figure 1.7.
Figure 1.7. Multi-physics coupling mechanisms for short-time transient phenomena in LWR core

Source: CASL, 2018.

In this report, the validation of traditional multi-physics tools is discussed with focus on
the above-mentioned three-physics coupling between RP, TH and FP. The first step in
multi-physics validation is the validation of different single physics models involved in the
coupling for the specific application of interest. The second step is to validate different twophysics coupling mechanisms (RP-TH, RP-FP, TH-FP) using experimental data. The thirdand final step is to validate the full multi-physics coupling (RP-TH-FP). This approach
includes progression of exercises (based on experiments and measurements) as shown in
Figure 1.8. In real life, usually not all needed experimental data is available, especially

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

24  NEA/NSC/R(2019)7
when based on traditional experiments that have been used to validate the pre-traditional
and traditional multi-physics tools. For example, assuming that the experimental data,
which addresses “light green” physics areas in Figure 1.8, is available while the data forwhite areas is not. The light green areas will cover just part of the multi-physics validation
pyramid but will follow a logical path (blue line) to decompose the impact of each single
physics effect and consistently include coupling with other physics as described below.
Figure 1.8. Multi-physics validation hierarchy

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  25

2. Validation of single physics

Examples and current approaches for validation of single physics calculations are provided
in this chapter.

2.1 Core reactor physics
The current core reactor physics calculations process is a two-step process – first single
assembly lattice physics calculations are performed for preparation of a parametrised
few-group nodal parameter library, which is used by the core nodal simulators for corewide calculations.
Examples and validation matrices for both steps are presented below. In general, theverification and validation (V&V) approach for light water reactor (LWR) physics software
includes the following components [4]:verification:– self-consistency tests (convergence in space, time, energy);– comparisons to numerical benchmark solutions;– comparisons to higher-order methods;– comparisons to other code results.

validation:– comparisons to critical measurements;– comparisons to isotopic measurements;– comparisons to power reactor physics tests;– comparisons to power reactor core-follow measurements;– comparisons to power reactor transient measurements.

Different data libraries and software systems within the framework of typical reactor
physics calculation schemes are verified and validated. The fundamental data libraries
Evaluated Nuclear Data Files (ENDF) are among the most difficult things to verify. Intercomparisons are performed between different data libraries like ENDF and Joint EvaluatedFission and Fusion (JEFF) libraries. Nuclear Data Processing Code (NJOY) is usually used
to process the data in multi-group format. The lattice physics codes are usually verified byusing single assembly and mini-core (colour-set) numerical benchmarks and Monte Carloreference solutions. The isotopic predictions of lattice codes are validated using
measurements. Some “interpretation” is required to make these measurements lattice“benchmarks”. Isotopic validation is useful but limited in terms of sensitivity to reactivity.
The lattice physics codes are usually validated using critical experiments. Hundreds ofthose are available [5] but some of the best (for LWRs) are proprietary. About 100 per centmille (pcm) (one sigma) spread is routinely obtained in the comparisons with measured
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

26  NEA/NSC/R(2019)7
data. Some critical experiments can also be run directly with nodal codes. For the nodal
core simulators, the best verification model is a comparison with full-core lattice model
since the cross-sections for the nodal core model are generated by the lattice model. Usuallyacceptable results in such comparisons are if keff is within 50 pcm, nodal powers root meansquare (rms) are less than 1%, and pin-wise rms are less than 2%. For nodal simulators
validation using measured data at beginning of cycle (BOC) hot zero power (HZP)
conditions is very important.
Note that the core thermal-hydraulic models are usually validated using measured loop test
data like pressure drop across the core and for different axial layers between the spacer
grids. The coupled nodal/thermal-hydraulic core steady-state models are validated by using
data from core measurements. The transient coupled core simulators are verified using
numerical benchmarks (kinetics – without feedback and dynamics with feedback) in which
the cross-section libraries are provided. The validation sometimes also requires coupling to
the vessel and system thermal-hydraulic models for simulation of different transients forwhich measured data exist.
Development and qualification of new reactor physics code systems as well as merging theexisting tools in unified systems capable of analysing different reactor types is a lengthy
process. The term of qualification stands for V&V, which includes [6]:
comparisons to high-order methods, such as Monte Carlo methodologies;

comparisons to critical experiments;

comparisons to plant measurement (this the heaviest part and the most important).

The number of critical experiments and spent fuel analyses is limited. Mostly, theseexperiments are at exposure zero and the existent fuel samples available for spent fuelvalidation do not cover the particular interests of modern assembly designs (burn-up steps
up to 70 GWD/MTU or more, various void levels, 0 up to 80% void, different enrichments
and gadolinium content). Gamma scans pin by pin are even more uncommon than thecritical experiments, and the errors in the history of the samples can be significant. Someof the experiments are proprietary. The reactivity changes with exposure cannot be assessedby measurements unless a three-dimensional (3D) core simulator is used, which is anindirect way to prove the quality of the spent fuel analysis. Therefore, “numericalexperimentsˮ as performed by Monte Carlo criticality, Monte Carlo based depletion andcoupled Monte Carlo/thermal-hydraulic calculations are provided to bridge gaps where
measurements are lacking, to make comparisons for specific fuel assemblies.
The following integral experiments are required for reactor physics software qualification:

mock-up experiment;chemical assays on irradiated fuels;reactivity versus burn-up;

fundamental experiments.

Interest in high-quality integral benchmark data is increasing as efforts to quantify andreduce calculation uncertainties accelerate to meet the demands of next generation reactor
and advanced fuel cycle concepts. The International Reactor Physics Experiment
Evaluation Project (IRPhEP) and the International Criticality Safety Benchmark Evaluation
Project (ICSBEP) continue to expand their efforts and broaden their scope to identify,
evaluate, and provide integral benchmark data for method and data validation [7,8].

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  27

Benchmark model specifications provided by these two projects are used heavily by theinternational reactor physics, nuclear data and criticality safety communities.
The plant operating data used for qualification of reactor physics software consist of startup and cycle follow-up measurements:


critical eigenvalue:– hot, operating conditions;– cold shut-down/start-up critical configurations.



in-core instrumentation comparison;



gamma scans (power, exposure, and fission gas);

post-irradiation examination radiochemical analysis.

The United States Nuclear Regulatory Commission (US NRC) requires in reactor physics
software licensing submittals to have comparisons with plant measurements. The US NRC
will not issue a licence without plant data. The qualification of the nuclear core design
system serves two purposes:

to demonstrate the capability of the system to perform accurate nuclear design
calculations;to provide a basis for licensing the methodology with the US NRC.

Qualification calculations are always performed for parameters that are measured at plant
start-up. For a pressurised water reactor (PWR), these parameters usually include: BOC
HZP critical boron, BOC HZP isothermal temperature coefficient (ITC) and BOC HZP rod
worth. Reactor physics calculations are usually performed for all these parameters for all
cycles in the qualification plant database where existing measurement data were available.
Figure 2.1 shows a sample histogram of the differences between measured data andcalculation critical boron results. These results show very good performance, with anaverage difference very close to zero and a standard deviation on the order of 20 parts per
million (ppm).
Figure 2.1. Histogram of HZP BOC measured – predicted differences for critical Boron

Source: Mayhue et al., 2008.

One example of reactor physics validation are the results of recent validation studies for
the WIMS/PANTHER code package for PWR reactors [9]. A wide range of reactor cores

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

28  NEA/NSC/R(2019)7
have been analysed by this route, ranging in size in both axial and radial directions. The
fuel types vary in both fuel geometry and fuel composition. A range of comparisons ofpredictions with measurements has been carried out and the resulting agreement is reported
in the paper. These measurements included the following:

measurement at HZP:
– critical boron concentrations at various rod bank configurations;– bank worth of various rod bank configurations;– isothermal temperature coefficients;– Boron worth.

measurements at power:– Doppler coefficients;– flux maps at various power levels.

The summary of the statistical analysis of the validation results over 90 cycles is given in
Table 2.1:
Table 2.1. HZP validation results
HZP BOC Parameters

Predicted ̶ measured

Units

Mean

1s

Sample size

Critical boron concentrations
ARO

[ppm]

1

26

96

All rodded configurations

[ppm]

-13

19

231

Rod bank worth

Δr [%]

0.4

5.4

295

MTC

[pcm/C]

-2.7

1.3

90

Boron worth

Δr [%]

-2.7

5.5

92

Where Δr denotes a (PANTHER/Measured – 1) difference.
Source: Hutton et al., 2000.

2.2 Fuel performance
Fuel performance codes are usually developed to solve equations embodying thephenomena as thermal performance, structural mechanics, actinide behaviour and fission
product behaviour, etc. It is common that traditional fuel performance codes employ theso-called 1.5D representation of the fuel rod geometry, that is, the fuel rod simulations are
performed on an axisymmetric, axially-stacked, one-dimensional radial model. Examplesof fuel performance codes using such a simplified geometrical representation include
FRAPTRAN [10,17,24], SCANAIR [11], TRANSURANUS [12] and SFPR [13].
Take the FRAPTRAN and its steady-state companion code FRAPCON [15,22,23] forexample, the finite differencing approach is utilised to calculate the fuel performance
related parameters at each of the user specified slices of the fuel rod, as seen in
Figure 2.2 [15], each representing one axial node at a specific elevation defined by the user.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  29

Each axial fuel node is radially subdivided into a set of concentric rings or annuli, providing
nodes with constant volume, thickness, or other properties, such as power or fuel burn-up[15]. When variables at any given axial node are calculated, they are linked to those at otheraxial nodes via shared properties, such as 1D temperature linked via coolant heat transfer.
Figure 2.2. Radial discretisation for the finite differencing method used in
FRAPCON/FRAPTRAN models

Source: Geelhood et al., 2011.

The FRAPTRAN code is developed to model the phenomena that influence theperformance of fuel rods in general and the temperature, embrittlement and stress of thecladding. The code has a heat conduction model to calculate the transfer of heat from thefuel to the cladding and a cooling model to calculate the transfer of heat from the cladding
to the coolant. The code has an oxidation model to calculate the extent of cladding
embrittlement and the amount of heat generated by cladding oxidation. A mechanicalresponse model is included to calculate the stress applied to the cladding by the mechanicalinteraction of the fuel and cladding, by the pressure of the gases inside the rod, and by thepressure of the external coolant.
The SCANAIR code [11] consists of three main modules dealing with thermal dynamics,
structural mechanics and gas behaviour. These modules are closely coupled in thefollowing way: during each time step, a first convergence is reached between thermal andthermal-hydraulics solutions. Then, using the calculated temperatures, a second iteration
occurs between the gas behaviour module and the mechanical solutions. Upon achieving
the convergence of this second resolution, temperatures are re-calculated and so on until
full convergence of the different modules.
The SFPR code [13] was developed for mechanistic multi-scale modelling of single fuelrod behaviour under various regimes of LWR operation (normal and off-normal, including
severe accidents). The code simulates evolution of fuel defect structure (microscale,

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

30  NEA/NSC/R(2019)7~10-5 m), pellet-cladding stress strain state (mesoscale, ~10-2 m), heat conduction andchanges in rod geometry (macro-scale ~1 m). It provides self-consistent description of thefollowing processes: release of gaseous fission products from UO2 or mixed-oxide (MOX)
fuels; fuel pellet swelling; mechanical and physico-chemical “pellet-cladding” interactions;mechanical properties degradation of cladding material due to oxidation/hydrogenation.
The SFPR has a modular structure and includes three main modules: structural mechanics,
thermal conduction and fuel behaviour modules. The last module was developed on thebase of stand-alone fuel performance code MFPR [14]. The MFPR code self-consistently
describes: fission products generation and intragranular transport accompanying byformation of secondary phases – precipitates at the grain boundaries; evolution of fuelmicro-structure (point defects, such as vacancies and interstitials, and extended defects,
such as gas bubbles, sintering pores and dislocations), which strongly influences the intraand intergranular diffusion transport of fission product atoms in irradiated fuel; evolutionof the intergranular porosity resulting in the fuel swelling; radial diffusion of oxygen intemperature gradient in a fuel pellet resulting in significant changes in local oxygen
potential and, consequently, in thermos-physical properties of fuel; fission producttransport through intergranular to open porosity and fuel-cladding gap; partial fission
product condensation in the gap and formation of solid fission product layer (“Joint Oxide
Gainˮ or JOG structure).
The microscopic parameters characterising the crystal defect structure naturally arise,
however, being physically grounded; these can be fixed from the analysis of available
experimental data (and/or from atomic scale simulation: DFT, classical MD, KMC) andthen used without any artificial tuning in further calculations. This is the main goal of themechanistic approach realised in the MFPR code. For improvement and validation of the
1D thermo-mechanical models the capabilities of the 3D finite element method (FEM) wereused.
During last five years, on the basis of approaches and methods realised in the SFPR code,
the mechanistic multi-scale BERKUT code has been developed that describes the fuel rod
behaviour in the fast reactors.
The validation of fuel performance codes is primarily performed by comparing the code
predicted parameters with that obtained in the experiments. For example, an integral
assessment has been performed to quantify the predictive capabilities of FRAPTRAN using
a set of experimental data from reactivity initiated accident (RIA) and loss-of-coolant
accident (LOCA) operating conditions [16,23].
The validation metrics for the RIA accident assessment are the following:

residual cladding hoop strain;

cladding failure (and enthalpy at failure);

fission gas release.

The list below gives the validation metrics for LOCA accident assessment:

cladding ballooning, failure, and pressure;

residual hoop strain;high-temperature oxidation predictions.

The integral validation assessment of steady-state and cycle depletion fuel performance
calculations is usually done by comparing the code predictions for fuel temperatures,
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  31

fission gas release (FGR), rod internal void volume, fuel swelling, cladding creep/growth,
cladding corrosion, and hoop strain to data from integral irradiation experiments and postirradiation examination programmes [25]. The cases used for code assessment are usually
selected based on the following criteria: well-characterised design and operational data
were provided, and the reported results spanned ranges of interest for both design andoperating parameters. For LWRs, the fuel rod cases are selected to represent both boilingwater reactor (BWR) and PWR fuel types, with pellet-to-cladding gap sizes within, above,and below the normal range for power reactor rods. The fill gas is pure helium in mostcases, but cases are included for which helium-xenon fill gas mixtures were used to assess
the gap conductance model. The linear heat generation rates at beginning of life (BOL)
range up to 60 kW/m (18 kW/ft), and during end of life (EOL) power ramps, they range up
to 47 kW/m (14 kW/ft). The rod-average fuel burn-ups range up to 62 GWd/MTU. The
EOL FGR ranges from less than 1 percent to greater than 50% of the produced quantity.
The integral validation assessment of transient fuel performance calculations is usually
done to evaluate the predictions of the thermal and mechanical behaviour of LWR fuel rods
during reactor power and coolant transients such as reactivity accidents, BWR power
oscillations without scram, and LOCAs at burn-up levels up to 62 gigawatt-days per metricton of uranium (GWd/MTU). The assessment is performed by comparing the fuelperformance code calculations to data from selected integral irradiation experiments andpost-irradiation examination programmes [25]. The cases used for code assessment are
selected on the criteria of having well-characterised design and operational data andspanning the ranges of interest for both design and operating conditions. Two principal setsof data were used: 1) data from recent RIA test programmes and 2) data from LOCA test
programmes.
Most reactor system and core thermal-hydraulics codes, as well as some reactor physics
core simulators have their have their own simplified fuel rod and heat transfer models with
conduction solutions [26]. These models are validated using the same approaches describedabove for the fuel performance codes.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

32  NEA/NSC/R(2019)7
Figure 2.3. Measured and predicted by FRAPCON centreline temperature
for the UO2 assessment cases throughout life

Source: Geelhood and Luscher, 2014.

Figure 2.4. FRAPTRAN-1.5 predictions of RIA FGR for CABRI and NSRR UO2 rods

Source: Geelhood and Luscher, 2014.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  33

2.3 Core thermal-hydraulics
Modelling of nuclear reactor core thermal-hydraulics is performed using different fidelity
methods including system, sub-channel, porous media and Computational Fluid Dynamics(CFD) codes. The V&V of the thermal-hydraulic feedback core model is an extensive
programme [27], which includes numerical and experimental benchmarks for a variety of
LWR steady-state and transient simulations. The V&V work is meant to demonstrate thatthe code behaves as expected and produces physically correct and accurate results for these
intended applications. Therefore, the intended modelling applications drive the
V&V work [28]. Each application has a set of physical phenomena upon which it will bedependent. The core thermal-hydraulics code must be capable of accurately modelling eachof these physical phenomena. Usually, a series of tables are prepared that summarise thelist of intended applications, the important physical phenomena in these applications, themodels that were implemented into the code to capture these important physicalphenomena, and the selected experimental tests that are used to show how accurately thesemodels capture the phenomena. There are two columns for tests; one provides a link tovalidation tests and the other provides links to verification tests. A separate table isprovided for each intended application of the code. A LWR core thermal-hydraulics codeis usually developed for the following modelling applications: normal PWR and BWR
operating behaviours, Departure from Nucleate Boiling (DNB) analysis of PWRs andCritical Heat Flux (CHF) analysis of BWRs, RIA analysis in PWRs and BWRs, etc. In
summary, the V&V work seeks to provide a certain level of certainty and confidence in thepredictive capabilities of the code for the scenarios it was designed to model: rod bundle
geometries with operating conditions that are representative of prototypical PWRs and
BWRs in both normal and accident conditions. This is done by modelling a variety ofexperiments that simulate these scenarios and then presenting a qualitative and quantitative
analysis of the results that demonstrates the accuracy to which the core thermal-hydrauliccode can capture specific quantities of interest.
For example, the PWR verification matrix for the modernised version of COBRA-TF –CTF (which is considered to replace current core thermal-hydraulic models in vendor’score analysis code systems because of its advanced two-phase three-field modelling)includes [27]:

PWR sample case;

PWR core calculations:– steady state – fuel assembly (FA)-by-FA level;– steady state ̶ cell-by-cell level;– flow reduction transient ̶ FA-by-FA level;– flow reduction transient ̶ cell-by-cell level;– power rise transient ̶ FA-by-FA level;– power rise transient ̶ cell-by-cell level;– pressure reduction transient ̶ FA-by-FA level;– pressure reduction transient ̶ cell-by-cell level;– main steam line break transient.

The core test facility (CTF) calculations are compared with the results from other codes.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

34  NEA/NSC/R(2019)7
The validation matrix of CTF consists of tests selected to validate the CTF modelling of
the following physics phenomena [28]: heat transfer, pressure loss, single phase turbulent
mixing, void content, two-phase turbulent mixing and void drift, dry-out, solid structure
models and natural circulation.
The CTF BWR validation matrix includes different tests such as Bennett tube test, GE 3x3experiment, PELCO-S 4x4 test and NEA/NRC Full-size Fine-Mesh Bundle Tests (BFBT),
based on the NUclear Power Engineering Corporation (NUPEC) BWR bundle data
benchmark [29]. The comparisons of CTF predictions of BWR 8x8 bundle pressure dropwith measured values from the NEA/NRC BFBT benchmark are shown in Figure 2.5.
Figure 2.5. Comparisons of the overall bundle pressure drop CTF predictions vs. measurements
Single-Phase Pressure Drop

Two-Phase Pressure Drop
150

Calculated Pressure Drop, [kPa]

Calculated Pressure Drop, [kPa]

40

30

20

10
-------- 10 %

100

50
10%

σ = 1.9
0

0
0

10

20

30

Measured Pressure Drop, [kPa]

40

0

50

100

150

Measured Pressure Drop, [kPa]

Source: PSU, 2018.

When core thermal-hydraulics codes are developed, experimental test must be used todevelop and calibrate the models in the closure laws for the fluid equations. The facilities
try to reproduce the phenomena that can be observed in the fuel assembly. These facilities
reproduce totally or partially an assembly in the reactor of interest and the problem isrepresented at full scale. To analyse specific phenomenon encountered in the reactor core,
the test conditions are set in such a way that this phenomenon is predominant at thoseconditions.
Validation of core thermal-hydraulics codes has been performed traditionally through inhouse experiments carried out by fuel vendors interested in demonstrating cooling
capacities of the fuel and safety margins to the nuclear regulatory body. An example of thisis the GE 3x3 test developed by General Electric to assess model for inter-sub-channel
mixing at single and two-phase conditions [30].
Some of the most known experiments for validation, not only for core thermal-hydraulics
codes but also for CFD codes applied to fuel assembly analysis, were those performed withdata made available from the NUPEC database obtained in Japan. This corporation
performed between years 1987 and 1995 a series of void measurement tests using full-size
mock-up tests for both BWRs and PWRs. The void distribution was visualised at the meshsize smaller than the sub-channel under actual plant conditions using state-of-the-art
computer tomography (CT) technology. NUPEC also performed steady-state and transient
critical power test series based on the equivalent full-size mock-ups. The first set of

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  35

experiments released included BWR experiments and became the NEA/NRC BWR BFBT
benchmark. This corporation also decided to release the data obtained for PWR analysis tocarry out the NEA/NRC PWR Sub-channel and Bundle Tests (PSBT) benchmark [31].

2.4 System thermal-hydraulics
Reactor system and containment coupled codes
In traditional nuclear reactors the sole role of containment was to be the last barrier torelease of radioactive emissions. There are many safety systems in the containment thatprevent increase in the containment pressure and challenge integrity of the containment incase of LOCAs. The reactor system is also isolated from the containment. Because of
special components and phenomena, the containment was modelled with special codes suchas CONTAIN, GOTHIC, SPACE, MAP, LOTIC, etc. [32-35]. Most of these are based ona lumped parameter formulation with very large volumes. In addition, containment plays aspecial role for safety under severe accident conditions when molten fuel flows to
containment, and very different phenomena occurs.
The reactor system codes such as TRAC, RELAP5 were coupled at the time of LOCA
where the containment pressure provided boundary conditions to the system codes and thesystem codes provided mass and energy flow to containment as boundary conditions forcontainment codes. The system codes were essentially decoupled to containment as thebreak flow is initially choked and as the primary side depressurises the flow becomesunchoked. The break flow at a later stage will be controlled by the containment pressure.
However, the containment volume is very large and the pressure increase is small, so thebreak essentially sees a fixed containment pressure as a boundary condition. The coupling
between system codes and containment codes is accomplished through an executive
programme, parallel virtual machine (PVM).
GOTHIC is a containment modelling code. It receives coolant inflow conditions from aseparate reactor system code. The coupling approach is explicit. The LOCA code is run for
full transient and coolant flow and enthalpy is then provided as a boundary condition to
GOTHIC. However, GOTHIC has a simple reactor version model and the larger of the twofor coolant flow is used for conservative analyses.
New advanced codes are being developed where the reactor system and containment are
tightly coupled with safety for reactor system coupled to containment. This situation isvalid for AP1000 and economic simplified boiling water reactor (ESBWR) and for small
modular reactors (SMRs) such as the NuScale design. As containment participates inemergency core cooling (ECC) injection and condenses the flow from the reactor system,
the system codes have been expanded to include models for containment. In this case thecoupling is implicit. There are cases where two codes are still being coupled, the couplingis tight with synchronous coupling and information may be transferred at every time step.
The Korean nuclear industry has developed a system of two codes to simulate reactor andcontainment performance under accident conditions [36]. The reactor system is modelledwith Safety and Performance Analysis Code (SPACE), and the containment is modelledwith Nuclear Containment Analyses Package (CAP). CAP can model both design basis
accidents and severe accidents. It requires boundary conditions for mass and energy from
coolant, molten corium, fission products and high-pressure melt injection. The concept is
illustrated in Figure 2.6.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

36  NEA/NSC/R(2019)7
Figure 2.6. Concept of coupled SPACE and CAP coder system

Source: Hong et al., 2015.

In Russia within the “Development of virtual-digital nuclear power plant with VVER
reactor” Project, the system thermos-hydraulic code HYDRA-IBRAE/H2O has beendeveloped by the Nuclear Safety Institute of the Russian Academy of Sciences (IBRAE
RAN) to model both reactor systems and containment. The code allows modelling of
nuclear power plant behaviour under normal operation, during the transients and accidents.
The code includes sub-channel module SCORE to calculate the processes in the core.
Validation against experimental data is essential in the process of system code development
and improvement. The models implemented and used in a code are generally developed
based on experimental tests performed in specific facilities. It is possible to distinguishamong:


Basic facilities: in these facilities the fundamental phenomena are reproduced; theresults are used to improve the equations of the single model or to derive
empirically the relation between the relevant parameters; such facilities aredesigned with the goal to reproduce the specific phenomenon to be investigated.

Separate effect facilities: in these facilities some relevant zones of the nuclear
power plant are reproduced by a suitable scaling law to investigate the local
occurrence of a phenomenon; the results of the experiments performed in thesefacilities are used to create and to validate the (several) models to be included in acode.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  37

Integral tests facilities: these facilities are simulators of reference nuclear power
plants. All the relevant parts and systems of a nuclear power plant are reproduced
by a suitable scaling law. The whole plant is reproduced and the global plant
responses are obtained as results. The results are used to realise and improve themodels and to check the code capabilities.

It should also be noted that the data from nuclear power plants can be used, if available.
However, in a nuclear power plant the data obtained are those recorded by the system ofcontrol of the plant while, typically, the facilities are equipped with many sensors and many
detailed data are generated making the instrumentation of the facilities more suitable forcode validation. Huge effort was done by the NEA/CSNI from 1991 to 1997 in theconstruction of the Separate Effects Test Facility Code Validation Matrix (SETF-CCVM,
published 1994) for thermal-hydraulic system codes [37]. Integral Test Facility (ITF)
matrices for validation of realistic thermal-hydraulic system computer codes were alsoestablished by CSNI focused mainly in PWRs, and BWRs. The ITF-CCVM [38] validation
matrix was issued in 1987 and updated in 1996. By the validation matrices, the best sets of
openly available experimental data for code validation, assessment and improvement werecollected in a systematic way. Quantitative code assessment with respect to thequantification of uncertainties in the modelling of individual phenomena by the codes isalso an outcome of the matrix development. In addition, the construction of such matrices
was an attempt to record information of the experimental work which has been generated
around the world over the last years in the LWR-safety thermal-hydraulics field. 187
facilities covering 67 relevant phenomena for LOCA and non-LOCA transient applicationsof PWRs and BWRs within a large range of useful parameters were identified and about
2 094 tests were included in the SETF-CCVM matrix. Most of these phenomena are also
relevant to advanced water-cooled reactors. The major elements of the SETF-CCVM havebeen already integrated into the validation matrices of the major best-estimate thermalhydraulic system codes, e.g. RELAP5, CATHARE, TRACE and ATHLET.
A total number of 177 PWR and BWR integral tests have been selected as potential sources
for thermal-hydraulic code validation in the ITF-CCVM report. Counterpart tests, similar
tests and NEA International Standard Problem (ISP) tests were also introduced in thereport. Counterpart tests and similar tests in differently scaled facilities are consideredhighly important for code validation and therefore they were included in the tables of ITF
selected experiments. Moreover, over the last thirty years, the CSNI has promoted morethan 50 ISPs [39]. The main objectives of the ISPs are: to contribute to a better
understanding of postulated events, to compare and evaluate the capability of codes (mainly
best-estimate codes), to suggest improvements to the code developers, to improve theability of code users and to address the so-called scaling effect. ISPs were performed indifferent fields as in-vessel thermal-hydraulic behaviour, fuel behaviour under accident
conditions, fission product release and transport, core/concrete interactions, hydrogen
distribution and mixing and containment thermal-hydraulic behaviour. ISP experiments
were carefully controlled, documented and evaluated.

The scaling issue
A nuclear power plant is characterised by high power (up to thousands of MW), highpressure (tens of MPa) and large geometry (hundreds of m3), thus it is well understandable
the impossibility to perform experiments preserving all these three quantities. The term
scaling is in general understood in a broad sense covering all differences existing betweena real full-size plant and a corresponding experimental facility. An experimental facility
may be characterised by geometrical dimension and shape, arrangements and availability
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

38  NEA/NSC/R(2019)7
of components, or by the mode of operation (e.g. nuclear vs. electrical heating). All thesedifferences have the potential to distort an experimental observation precluding its direct
application for the design or operation of the reference plant. Distortion can be defined asa partial or total suppression of physical phenomena caused by only changing the size
(geometric dimension) or the shape (arrangement of components) of the facility [41].
The scaling analysis is therefore the necessary link between the experiments performed in
ITF and SETF and their utilisation in the code validation process. Three main objectives
can be associated to the scaling analysis:the design of a test facility;the code validation, i.e. the demonstration that the code accuracy is scale
independent;the extrapolation of experimental data (obtained in an ITF) to predict the nuclear
power plant behaviour.

For test facility design, three types of scaling principles can be adopted:
Time reducing scaling: rigorous reduction of any linear dimension of the test rig
would result in a direct proportional reduction in time scaling. This is of advantage
only for cases where body forces due to gravitational acceleration are negligible
compared to the local pressure differentials.

Time preserving scale: based on a scale reduction of the volume of the loop system
combined with a direct proportional (power to volume scaling) scaling of energy
sources and sinks (keeping constant the core power to system volume ratio).



Idealised time preserving modelling procedures: based on the equivalency of themathematical representation of the full-size plant and of the test rig. It is deduced
from a separated treatment of the conservation equations for all involved volumemodes and flow paths assuming homogeneous fluid.

ITFs are normally designed to preserve geometrical similarity with the reference reactor
system. Generally, all main components (e.g. rector pressure vessel, down-comer, rod
bundle, loop piping, etc.) and the engineered safety system (HPIS, LPIS, accumulators,
auxiliary feed water, etc.) are represented. ITF are used to investigate, by direct simulation,
the behaviour of a nuclear power plant in case of off-normal or accident conditions. The
geometrical similarity of the hardware of the loop systems has been abandoned in favourof a preservation of geometric elevations, which are decisive parameters for gravity
dominated scenarios (e.g. in case of natural circulation processes). Thus, the reduction of
the primary system volume is largely achieved by an equivalent reduction in vertical flowcross-sections. Due to the impossibility to perform relevant experiment at full scale (i.e. ina nuclear power plant), the use of ITF or SETF is unavoidable.
To address the scaling issue, different approaches have been proposed and are available
from literature. However, a comprehensive solution has not yet been achieved andmoreover, it is evident that the attempt to scale up all thermal-hydraulic phenomena that
occur during an assigned transient, results in a myriad of factors which have counteracting
effect [42]. For instance, consider a two-phase flow condition in a vessel of a facility whena small break loss-of-coolant accident (SBLOCA) scenario is postulated. The two-phasecritical flow is affected by phenomena like the vapour pull through and the subcooled
vapour formation by the sharp edge cavitations, the heat losses, the fluid temperature
stratification, etc. All these phenomena cannot be scaled up and are characterised by

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  39

parameters that appear neither in any balance equations nor in any scalable mechanistic
models. This is a typical situation in which a scaling criterion is not applicable.
Nevertheless, the influence of those phenomena is time-restricted in relation to the entire
transient and thus they can be considered as local phenomena. Therefore, a way to solve
the scaling problem is to consider only those phenomena and parameters that have a real
impact on the whole problem under investigation. Focusing on a single phenomenon which
occurs during a limited time (compared with the entire duration of the problem) should be
avoided because it is governed by factors that are not scalable. Therefore, a hierarchy inthe definition of the scaling factors is necessary and a global strategy is needed [42] todemonstrate that those phenomena are effectively local and cannot affect the overallbehaviour of the main thermal-hydraulic parameters selected to describe the transient.
One possible strategy to address the scaling issue is embedded into the Uncertainty
Methodology based on Accuracy Extrapolation (UMAE) methodology where the scaling
analysis plays a relevant role, firstly during the code assessment process (as the codedevelopment and improvement is based on experimental data obtained in scaled test
facilities) and secondly during the demonstration of the qualification of a nuclear power
plant nodalisation (that is a necessary step to perform a reliable nuclear power plantcalculation). Following the UMAE methodology, the strategy to adopt for solving thescaling problem consists of:

qualifying the code against experimental data measured in facilities (ITF andSETF);

demonstrating that the code accuracy (i.e. discrepancy between measured andcalculated trends) only depends upon boundary and initial conditions (BIC) values(within the assigned variation ranges) and is not affected by the scale of theconcerned ITF;

applying such code to predict the same relevant phenomena that are expected to befound in a similar experiment (or transient) performed at a different scale;performing nuclear power plant Kv-scaled calculation and explaining thediscrepancies (if any) between nuclear power plant Kv-scaled calculation andmeasured trends in ITF considering only BIC values and hardware differences (i.e.
distortions).

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

40  NEA/NSC/R(2019)7

3. Validation of multi-physics codes − Examples and current approaches
The description of the state of current practices for verification and validation (V&V) of
multi-physics calculations for specific applications focuses on V&V of current traditional
multi-physics methods and tools. It includes a discussion on the current V&V approacheswith examples. Multi-physics validation is a complex problem and cannot be even feasible
for some applications. Experimental installations for this type of validation are very limited
or even unavailable. Consequently, data is obtained from nuclear power plantmeasurements. In this case is difficult to define the data requirements for adequate V&Vof the codes for the phenomena of interest.
In multi-physics systems, each physics has its own length and time scales. The closure
laws, in addition, can create new length and time scales. Transients require that all thephysical length and time scales (including closure laws) are resolved by the mesh and time
step.
Multi-physics simulator validation has been performed traditionally using international
multi-physics benchmarks whose measured data have been supplied by nuclear power
plants. These are complex benchmarks, which use different phases, exercises and scenarios
to validate different models and the coupling between them. The available measured data
is utilised in combination with detailed code-to-code comparisons. The cross-section
libraries are generated usually by the benchmark teams, which removes the uncertainties
introduced by using different cross-section generation and modelling procedures. Thisapproach allowed development of a more in-depth knowledge of the capabilities of thecurrent generation of neutronics/thermal-hydraulics simulators. Traditional multi-physics
benchmark problems scope helps to define requirements for the definitions of futureinternational exercises.

3.1 Multi-physics validation hierarchy and multi-level validation methodology
The qualification procedure of coupled multi-physics code systems is based on thequalification framework (verification & validation) of separate physics models/codes andincludes in addition verification and validation of the coupling methodologies of thedifferent physics models. The extended verification procedure involves testing thefunctionality, the data exchange between different physics models, and their coupling,which is designed to model combined effects determined by the interaction of models. The
extended validation procedure compares the predictions from coupled multi-physics code
systems to available measured data and reference results. It is important to emphasise thatsuch validation should be based on a multi-level approach like the one utilised in validating
coupled neutronics/thermal-hydraulics codes in international standard problems.


use of the validation hierarchy (see for example the “pyramidˮ representation in
Figure 1.8);

coupling of physical phenomena (ad-hoc vs. first principles);

coupling phenomena with varying degrees of fidelity: temporal, spatial, and energy;integration of the levels of the validation hierarchy with uncertainty propagation.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  41

3.2 Coupled thermal-hydraulics system/neutron kinetics codes
The development of multi-physics multi-scale coupled methodologies for light water
reactor (LWR) analysis requires comprehensive validation and verification procedures,
which include well-established benchmarks developed in international co-operation. The
Nuclear Energy Agency (NEA) has provided such framework, and over the years many
LWR benchmarks have been developed and successfully conducted. The first set of NEA
benchmarks that permits testing of the neutronics/thermal-hydraulics coupling andverifying the capability of the coupled codes to analyse complex transients with coupledcore/plant interactions have been summarised in [43]. Such benchmarks are the
NEA/United States Nuclear Regulatory Commission (US NRC) pressurised water reactor(PWR) Main Steam Line Break (MSLB) benchmark, the NEA/NRC boiling water reactor(BWR) Turbine Trip (TT) Benchmark and the NEA/US Department of Energy(DOE)/Commissariat à l'énergie atomique et aux énergies alternatives (CEA) VVER-1000
Coolant Transient (V1000CT) benchmark. These benchmarks provide a validation basis
for the current generation of coupled neutronics/thermal-hydraulics best-estimate codes.
To meet the objectives of the validation of “best-estimate” coupled codes, a multi-level
methodology has been introduced to evaluate systematically the transients. Since thesebenchmarks are based on both code-to-code and code-to-data comparisons, further
guidance for presenting and evaluating results has been developed. The presented fourbenchmarks are developed in an international co-operation to provide a validation basis forthe new generation of best-estimate codes. Based on the previous experience, several
benchmark exercises were defined for each benchmark consisting of different steady states
and transient cases, which allow for a consistent and comprehensive validation process.
The introduction of extreme transient scenarios contributes to the study of different
numerical and computational aspects of coupled simulations. The participants use a crosssection library, generated by the benchmark team, which removes the uncertainties
introduced with using different cross-section generation and modelling procedures. The
defined benchmark cross-section modelling approach is a direct interpolation in
multidimensional tables with complete representation of the instantaneous cross-section
cross-term dependencies. In this way the benchmarks provide the opportunity to study the
impact of different thermal-hydraulics and neutronics models as well as the coupling
between them on code predictions and to identify the key parameters for modelling
important transients for different types of power reactors. This allowed the evaluation ofthese key parameters, through the performance of sensitivity studies, which led theparticipants to develop a more in-depth knowledge of the capabilities of the current
generation best-estimate thermal-hydraulic system codes.
The results of the NEA light water reactor (LWR) benchmark activities are discussed on
the example of Exercise 3 (coupled neutronics/thermal-hydraulic system modelling) for the
NEA/NRC BWR TT benchmark. Turbine trip transients in a BWR are pressurisation eventsin which the coupling between core spatially-dependent neutronics phenomena and system
dynamics plays an important role. The data made available from actual experiments carriedout at the Peach Bottom 2 plant make the present benchmark particularly valuable. Exercise
3 consists of the base test case (the so-called best-estimate case) and hypothetical cases (theso-called extreme scenarios). The purpose of Exercise 3 Best Estimate Case is to provide
comprehensive assessment of all the participating coupled codes in analysing complex
transients. To validate such assessments, available measured plant data are utilised for thiscase during the comparative analyses presented in this report. In addition to the base case,the analysis of the extreme scenarios provides a further understanding of modelling

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

42  NEA/NSC/R(2019)7
limitations of coupled codes and the interplays between different feedback mechanisms aswell as testing the coupled code capabilities in extreme situations. The following four
extreme scenarios are analysed by the participants during this exercise: Extreme Scenario 1: Turbine trip (TT) with steam bypass relief system failure;Extreme Scenario 2: TT without reactor scram;

Extreme Scenario 3: TT with steam bypass relief system failure without reactor
scram;

Extreme Scenario 4: TT with steam bypass system failure, without scram andwithout safety relief valves (SRV) opening.
Extreme Scenario 1 (TT without bypass system relief opening) and Extreme Scenario 2(TT without scram) can be considered as single failures and therefore provide informationfrom the perspective of the safety of the plant. Extreme Scenario 3 (combination of 1 and2), which considers the coincidence of two independent failures, and Extreme Scenario 4(in addition to 3 no opening of safety relief valves), which considers the coincidence of
three independent failures are extremely unlikely from a safety perspective, while they help
with the understanding of the short-time dynamics of the reactor system. In the base case,
SRVs are not opened during the transient, although this happens in the extreme scenarios
1, 2 and 3. In hypothetical cases, the dynamical response of the system due to the interactionof the flow in the steam line with the dynamics of SRVs happens to be more challenging
for the coupled codes. Hence, Extreme Scenario 4 serves as clear comparison of physicalmodels of the participants’ codes. It should be noted that no comparison with measured
data is possible for the extreme cases since they are hypothetical scenarios. Therefore,
submitted extreme scenario results are compared with an “averageˮ of the results of thebenchmark participants.
The averaged spatial and time-history distributions over results from 15 participants (using
different coupled code systems) are presented in Figure 3.1 through Figure 3.4 anddiscussed to outline common modelling tendencies. The most challenging part of the BWR
steady-state analysis is the prediction of the void fraction distribution. It can be seen from
Figure 3.1 that the standard deviation increases in the lower (bottom) part of the core whichindicates differences in the void modelling in terms of subcooled boiling and vapour slip.
Subsequently these deviations are also reflected in the comparison of the axial power
profile predictions due to the void feedback mechanism (see Figure 3.2).
Figure 3.1. Steady-state core average axial void fraction (mean and std. deviation)
0.70

0.60

Void Fraction

0.50

0.40

0.30

0.20

0.10

0.00
0

3

6

9

12

15

18

21

24

Axial Nodes

Source: PSU, 2018.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  43

Figure 3.2. Steady-state core average normalised axial power distribution(measured vs. mean and standard deviations)
1.6

1.4

Normalized Power

1.2

1.0

0.8

0.6

0.4

Measured and Std. Dev.
Average and Std. Dev.

0.2

0.0
0

5

10

15

20

25

Axial Nodes

Source: PSU, 2018.

When analysing the power history results in Figure 3.3, it has to be taken into account thatin the TT2 test, the thermal-hydraulic feedback alone limited the power peak and initiatedthe power reduction. The void feedback plays the major role while the Doppler feedback
plays a subordinate role. The reactor scram then inserted additional negative reactivity andcompleted the power reduction and eventual core shut-down. The accurate prediction ofvoid feedback during the transient depends on the modelling of the pressure wave
propagation into the vessel. The indication the modelling degree of success is the predictionof dome pressure time history (see Figure 3.4). The participants’ results were in good
agreement with measured data for the parameters dome pressure, time history and power
history. These results increased the trustworthiness of the coupled code capabilities.
Figure 3.3. Best-estimate case ̶ Transient fission power comparison (average vs. measured – zoom)

Fission Power Delta Changes (W)1.00E+10

8.00E+09

Average of Fission Power

6.00E+09

Measured (fission)

4.00E+09

2.00E+09

0.00E+00-2.00E+09
0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2

Time (s)Source: PSU, 2018.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

44  NEA/NSC/R(2019)7
Figure 3.4. Best-estimate case – Transient dome pressure (average vs. measured)
0.50

Pressure Delta Changes (MPa)

0.45
0.40
0.35
0.30
0.25

Average

0.20
Measured

0.15
0.10
0.05
0.00
0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Time (s)Source: PSU, 2018.

It is expected that the application of coupled multi-physics codes for safety analyses willgrow continuously. In fact, they are the only means to perform best-estimate calculations
for accident conditions with a tight coupling of neutronics and thermal-hydraulics effects.
Developing multi-level benchmarks, which utilise the available experimental data (such as
the described in this paper NEA/NRC BWR TT Benchmark) for consistent validation andverification of coupled multi-physics codes in international co-operation, is important.
The NEA LWR benchmark activities have also stimulated follow-up developments andbenchmarks to test new developments [44]. The models utilised have been improved whenmoving from one benchmark to the next and this created a need to validate these refinedmodels using high-quality experimental data. This new set of NEA benchmarks have been
initiated by the Expert Group on Uncertainty Analysis in Modelling (EGUAM), under the
Nuclear Science Committee (NSC), to address the current trends in the development of
LWR multi-physics and multi-scale modelling and simulation as well as some of the issues
identified in the first set of NEA benchmarks. These new benchmarks include the following
common features:utilisation of high-quality experimental data;

refined local scale modelling in addition to global predictions;

more detailed comparisons and analysis;



including uncertainty and sensitivity analysis of modelling.

The major task of the EGUAM is to address verification, validation and uncertainty
analysis of multi-scale and multi-physics modelling and simulation. EGUAM has createda roadmap along with a schedule and organisation for the development and validation of
methods and codes required for uncertainty analysis, including the benchmarks adequate
to meet those goals.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  45

Reference [45] discusses each of these new benchmarks by providing background andobjectives, description and discussion of comparative analysis of obtained results. The
following benchmark activities are summarised: NEA/NRC Oskarshamn-2 BWR stability
benchmark, and NEA Kalinin-3 water-water energetic reactor (VVER-1000) coupled code
benchmark. Since the concept of benchmarks are used throughout this report, its meaning
for the discussed NEA benchmarks is defined. Achieving the next level of credibility incomputational simulations of nuclear reactor design and safety requires concerted anddetermined efforts. The goal of these efforts is to improve the reliability of the computer
software, the estimation of numerical accuracy, the quality of the physics models used, andthe quantification of uncertainty. This report focuses on one aspect of the needed
improvements to software reliability and physics modelling, namely, the construction anduse of highly demanding V&V benchmarks. The NEA benchmarks of interest are thoserelated to the accuracy and reliability of physics models and codes. These benchmarks are
combinations of verification and validation benchmark exercises which involve both useof experimental (measured) data and reference solutions i.e. code-to-data and code-to-code
comparisons. When comparing code predictions to experimental data in these benchmarks
usually the experimental uncertainties are provided.

3.3 Coupled thermal-hydraulics core/neutronics simulators
The previous name of NSC EGUAM was the Expert Group on Reactor Stability and LWR
Transient Benchmarks. This expert group dealt throughout the 1990s with modelling andbenchmarking issues in the field of 3D coupled neutronics/thermal-hydraulics transientsfor nuclear reactor cores. This was the first set of NEA multi-physics benchmarks for LWR
cores mostly focused on multi-physics verification since no comparisons with experimental
data were involved. Later coupling of core phenomena and system dynamics for nuclear
reactor core design and safety analysis was introduced along with comparisons toexperimental/measured data developing the second set of NEA multi-physics benchmarks.
This set was discussed above along with the latest set of multi-physics benchmarks which
added to validation, the aspects of sensitivity and uncertainty analysis.
As mentioned above the first set of benchmarks aimed at performing code-to-code
comparisons although the benchmark reactor core models are based on real reactor data.
The first benchmark is a PWR 3D core transient benchmark [46] based on a rod ejection
accident. This is the ejection of a control assembly from the core, which may occur becauseof the rupture of the drive mechanism casing located on the reactor pressure vessel top.
This event can produce significant, well localised perturbations of the neutronic andthermal-hydraulic core parameters, without exceeding the safety margins. Hence, a rather
realistic standard reactor situation is defined, that efficiently utilises the neutronics andthermal-hydraulics sub-models of the reactor dynamics code. The second one is a BWR
core benchmark. The BWR problems consist of sub-prompt critical reactivity excursions
generated by rapid cold-water injection or core pressurisation events. This set of problems
was chosen because it is felt that the analysis of two such cases ̶ in which the interplaysof the neutronic and thermal-hydraulic effects are markedly different ̶ represents a direct
and effective way to fulfil the objectives of the benchmark.
The third one applies the same core model as the PWR rod ejection benchmark and is basedon modelling an uncontrolled withdrawal of control rods at zero power accident on a
PWR [47]. The accident consists basically of a continuous reactivity insertion, limited bya reactor trip, which occurs a certain delay after high flux or high flux increase detection.
Like the rod ejection accident, the peak power usually takes place before the scram and the

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

46  NEA/NSC/R(2019)7
main factor limiting the consequences of the accident is the Doppler effect. The temperaturerise, however, is stopped by the trip, that must therefore be considered. It should be notedthat the peak power occurs while important power distribution changes take place in thecore, because the rod extraction continues until reactor trip. Two prompt neutron groups,
i.e. thermal and fast neutrons, and six delayed neutron groups are used for neutronmodelling. The boundary condition for the solution of the neutron diffusion equation is flux
vanishing at the outer reflector surface. A complete set of macroscopic cross-sections fortransport, scattering, absorption and fission and their derivatives with respect to the boron
density, moderator temperature, moderator density and fuel temperature is defined for eachcomposition. The feedback or coupling between neutronics and thermal-hydraulics ischaracterised by definition of channel regions. In the benchmark, each FA must be defined
as a channel region (“closed channels” model), ignoring inter-assembly cross-flow andmixing and using one TH feedback axial mesh per assembly. A flat profile of the radial
distribution of the power density inside the fuel is assumed. The inlet mass flow throughthe core is distributed uniformly among the channels. The transient to be analysed as afunction of time in 3D or 1D is generated by the withdrawal of control rod banks from an
initially critical core at HZP. The withdrawal speed is a typical bounding value.
The fourth verification core benchmark addresses a rod ejection transient in a PWR coreloaded partially with weapons-grade, MOX fuel [48]. The neutronics characteristics ofplutonium are sufficiently different from uranium to significantly change the kinetics
response of the reactor core. The reactor core chosen for the simulation is based on a fourloop Westinghouse PWR power plant like the reactor proposed for plutonium disposal in
the United States.
In addition to the above core verification benchmarks the second exercises of the LWR
core/plant benchmarks (PWR MSLB, BWR TT, V1000 CT, Kalinin-3 VVER-1000
transient and Oskarshamn-2 BWR stability benchmarks) described previously can be usedfor validation of multi-physics core models. These are benchmark exercises with provided
core thermal-hydraulic boundary conditions from system calculations performed by the
benchmark teams. The available measured data for integral parameters and local
distributions inside LWR cores can be used for code-to-data comparative analyses.

3.4 Coupled neutronics/fuel performance simulators
The heat transfer phenomena inside the fuel rod are comprised of heat conduction in thefuel pellets, heat transfer across the gap between fuel pellets and cladding inside surface,and heat conduction across the cladding thickness. The combination of these effects, alongwith the geometrical fuel pin design, will influence the fuel pin overall time constant at
power variations, though the heat transfer across the gap (gap conductance) will have thegreatest influence. All the effects are dependent on the fuel pin burn-up and willconsequently also have a variation in the axial direction.
Modelling of fuel rod behaviour in multi-physics calculations is mostly important forpredictions of fuel temperatures for feedback purposes i.e. Doppler feedback modelling.
Usually this is done by incorporating simplified fuel rod models in system thermalhydraulics, or sub-channel (core) thermal-hydraulics, or neutronics core simulators. Thesemodels are efficient but need to be improved for accurate predictions of fuel temperatures.
The list of heat conduction and fuel-related phenomena or related computational aspects
that may require further investigation to validate the results of multi-physics calculations
includes the following: minimum number of concentric rings for the fuel rod that bring to
the convergence of results, pellet gap performance during the transient with main reference
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  47

to variations in physical properties and geometry, and fuel deformation and change inphysical properties during the transient (namely steep power rise).
Currently there are two tendencies to improve the accuracy of predictions of Doppler(effective fuel temperature) feedback temperatures in multi-physics calculations: using
higher-fidelity fuel performance codes to inform simplified fuel rod models [49] and tocouple directly fuel performance codes with neutronics and thermal-hydraulic coremodels [50]. The issue with the second tendency is that usually the fuel performance codesmodel a single fuel rod, which requires many parallel calculations and affects
computational efficiency.
Fuel performance codes analyses the thermal, mechanical and internal gas responses of fuelrod components with the goal of predicting rod condition and integrity. Modelling ofthermal behaviour during normal and transient conditions includes surface heat transfer,the heat transfer across the fuel-to-cladding gap, thermal conductivity of fuel and cladding,power generation distribution in the fuel and solution of the conduction equation. These
codes can model steady-state and cycle depletion as well as transient fuel behaviour [51].
One example of coupling simplified fuel rod models with neutronics codes are the couplingof INTERPIN with the Studsvik CMS codes ̶ CASMO and SIMULATE [52] for steadystate and cycle depletion calculations as well as the development of SIMULATE-3K
explicit fuel pin modelling [53] for transient calculations.
In traditional multi-physics calculations, the coupling is performed on a node/assemblybasis and for the purposes of modelling thermal-hydraulic feedback effects (including
Doppler feedback) the physical fuel pins within a node (assembly) are modelled by anaverage/lumped fuel rod. The capability to explicitly represent every fuel pin in the reactor
core has been recently added to novel high-fidelity multi-physics calculations by coupling
neutronics, thermal-hydraulics and fuel performance codes on fuel pin/sub-channel
basis [54].

3.5 Multi-physics multi-scale code systems
The traditional multi-physics methodologies have reached maturity and have several
common features [55]:

the 3D core neutronics model is based on two-group (or few-group) coarse-meshdiffusion approach using nodal nuclear data (library of parameterised nodal
equivalent parameters);

the spatial neutronics/thermal-hydraulicassembly/channel level;simplified fuel rod models are utilised as part of the core thermal-hydraulic codes;the temporal neutronics/thermal-hydraulic coupling is performed in the operatorsplitting approach using nested loop iteration (fixed point iterations) for steadystate simulations and sequential parameter exchange (explicit coupling) at eachtime step for transient simulations;the evaluation of safety related parameters on pin/sub-channel level for the hot
assembly/channel is performed via pin-power reconstruction combined with subchannel calculations using boundary conditions provided by the coupled coremodel.

coupling

isperformed

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2on

48  NEA/NSC/R(2019)7
These multi-physics methodologies can predict the non-symmetrical core power
perturbations in a more realistic manner and they can predict safety margins moreaccurately by means of 3D core model with a spatial resolution at fuel assembly level for awide range of normal operation including cycle depletion, operational transients andpostulated accidents. The approximation level is acceptable, however, the safety-relevant
parameters, that determine the accident consequences, such as fuel rod enthalpy, departurefrom nucleate boiling ratio (DNBR), burn-out, maximum fuel rod cladding temperature,
fuel rod centreline temperature, etc., must be evaluated at local conditions i.e. in terms of asingle rod (pin) response rather than based on an assembly-wise response. Different
approximated “hot channel” or “hot assembly” models are used for approximatedpredictions of local safety parameters. Usually coupled thermal-hydraulic system andneutronics codes are further coupled with sub-channel codes [56-58], extending the multiphysics coupling to multi-scale modelling.
An example of validation of such multi-physics multi-scale code systems for licensing
applications is given in [58] for the RELAP5/PANTHER/COBRA code package. The
validation of this package depends on the separate validation of each major component of
the coupled package and on the validation of the coupling processes themselves (i.e. thedata exchange interfaces). The RELAP5, PANTHER and COBRA codes have beenvalidated separately as single physics codes for licensing applications. For the multiphysics multi-scale coupled code package the validation effort of the coupled code package
is mostly focused on the coupling processes themselves.
In addition, the validation of the coupling interfaces is performed followed by validationof the whole coupling process. Specific demonstrations that the interface tools achieved in
all cases the required data transfer is provided to show that:

the transfer of the variables between the different codes perform exactly as designed
and the code internal variables are consistently assigned to the required values;the coupled code solutions are identical to the separate stand-alone code solutions
with the same imposed boundary conditions;the coupled code solutions are not influenced by the time advancement techniqueschosen in the data transfer process.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  49

4. Uncertainty quantification approaches

This chapter discusses experimental uncertainty and the impact of data calibration forcomputational models.

4.1 Overview of uncertainty methodologies
One of the most important terms in reactor safety is the safety margin and the licensing of
nuclear power plants has been done by assuming conservative values for the computations.
Nowadays, it is possible to estimate certain parameters using non-conservative data withthe complement of uncertainty evaluation and these calculations can also be used forlicensing. The uncertainty methods within the nuclear field are an important step to help
increase production, while staying below the safety limits [59].
Several uncertainty methods have been developed for uncertainty evaluation of the system
codes including:
Adjoint sensitivity analysis procedure [60]: The theory behind the method is to getsensitivity responses for all uncertain parameters and then use them to assess theuncertainty by propagation of error formulae. The method works for both linear and nonlinear systems and this makes it appropriate for the case of a nuclear power plant. Thismethod requires significant modifications of the code.
Propagation of input errors (PIE) [61]: This approach represents statistical variation of the
input parameters, together with their uncertainties, to reveal the propagation of errors
through the code. The method relies on real code calculations, but instead of calculating
certain parameters with discrete values, it performs many times the same calculation using
different values for these parameters. This approach allows evaluation of correlation
coefficients between the uncertain parameters and the target output parameter, which canbe used to determine the most influential parameters. The drawback of this method is thatall uncertain parameters must be identified, and probability density functions must be
assigned to them, which sometimes has to be an engineering judgement.
Propagation of output errors [62]: This methodology focuses on extraction of the errors
from a suitable experimental database coming from integral test facilities and real plantdata. The results calculated by the system codes are extrapolated with the uncertainty band,to obtain the uncertainty in the code prediction. The benefit is that the method does not
require the identification of all uncertain input parameters. The drawback is that it relies on
non-physical extrapolation of errors coming from all available relevant data and producing
the final uncertainty band which is not necessarily directly related to the given case.
The second method (propagation of input errors) is the most used within the industry andit has been extensively verified. It provides the opportunity to identify the influence of eachparameter on the results, which is not possible in the third method (propagation of output
errors). Furthermore, the propagation of output errors requires extensive and relevant
experimental data to implement the method, which is not available for example, in the case

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

50  NEA/NSC/R(2019)7
of BWR stability studies. Moreover, the propagation of input errors method relies on theactual code calculations without using approximations such as fitted response surfaces.

4.2 Propagation of input errors methodology
The PIE method is based on statistical variation of input parameters and it does not require
modifications of the code (except for the closure models and correlations). In principle
there is no limitation on the number of uncertain parameters and range of variation of theperturbed parameters. In this method, every parameter is considered as a probability
distribution, not as a discrete value, as shown in Figure 4.1. A random set of discrete values
based on their probability density functions (PDFs), for all “suspicious” parameters needs
to be generated. The sources of PDFs are literature (e.g. scientific papers) or an expert
judgement. After all important parameters are identified and the random sample is set, N
model runs should be performed with these randomly sampled parameters as input. The
required number of runs (N) is given by the Wilks’ formula, Equations (1) and (2) [63,74]:
1 – (𝛼/100)𝑁 ≥ 𝛽/100(1)

𝛽% is the confidence level that the maximum code result will not be exceeded with theprobability 𝛼% of the corresponding output distribution, which is to be compared to theacceptance criterion. The formula for the two-sided statistical tolerance intervals is givenas:
1 – (𝛼/100)𝑁 – N (1 – (𝛼/100)) (𝛼/100)𝑁−1 ≥ 𝛽/100(2)Here, 𝛼 and 𝛽 have the same meaning as in the previous equation.
Depending on the requirements for the confidence interval, one can estimate how many
calculations are needed to obtain the desired results. The minimum number of runs can befound in Table 4.1. Usually (95%/95%) one- or two-sided tolerance limit is used, which istaken as a best-estimate prediction requirement [63].
Table 4.1. Minimum number N of calculations
One-sided statistical tolerance limits

Two-sided statistical tolerance limits

α

0.9

0.95

0.99

0.9

0.95

0.99

0.9

22

45

230

38

77

388

0.95

29

59

299

46

93

473

0.99

44

90

459

64

130

662

β

Source: Triola, 2007.

In the PIE calculation, all parameters are varied simultaneously. The PIE uncertainty
calculation provides the uncertainty of the output parameters which are to be examined.
Comparing this result with the measurements provides a measure of the confidence of thecode prediction.
The Spearman rank correlation coefficient between the output parameter and the input
parameter value can be used to identify the most influential parameters. This coefficient
provides a quantitative indication if the considered parameter (and its uncertainty,
represented by its PDF) has a significant impact for example on the stability of the reactor
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  51

(in case of BWR studies). The criterion for “importance” consists of the so-called criticalvalue of the Spearman correlation coefficient. If the absolute value of the estimatedcorrelation coefficient is higher than the critical value, then this parameter is determined tohave a statistically significant impact on the result. Otherwise, the parameter will beconsidered to have low or no importance. The critical values rs for the Spearman rankcorrelation coefficient can be evaluated using Equation (3) [65].
rs = ±

𝑧
√𝑛−1

(3)

For the confidence interval of 95%, z = 1.96 and n is the number of runs [65]. Anotherimportant outcome of the Spearman rank correlation is the sign of the correlation
coefficient; if it is positive then increasing the input parameter increases the output
parameter, if it is negative then increasing the input parameter decreases the output
parameter.

Sources of uncertainties
The propagation of input errors method relies on the availability of the input uncertainty
probability density functions. Without these functions PIE uncertainty calculation could
not be performed. Several classifications of uncertainties can be found. In general,
uncertainties can be divided in two broad categories.
The first category is called aleatory (or stochastic) uncertainty. This uncertainty comesfrom the random nature of how a system behaves. An example of this uncertainty can be
the measurement of the temperature in a room. Each time one measures, it will be different
within the limits of the measuring instrument. This uncertainty is irreducible (cannot be
reduced using the same technology) and is associated with the performance of anexperiment or a measurement [64,65].
The second category is the epistemic (or subjective) uncertainty. This category stands forthe lack of knowledge of a certain parameter. An example of this uncertainty can be acertain correlation in the analytical model, which might not be able to represent the reality
in some ranges of other parameters during the simulation. This uncertainty is reducible (itcould be reduced if new knowledge is provided) [64,65].
The three major sources of uncertainties in accident analysis have been identified in [64].


Code or model uncertainty: Uncertainty associated with the models andcorrelations, the solution scheme, model options, un-modelled processes, datalibraries and deficiencies of the computer program.



Representation or simulation uncertainty: Uncertainty in representing or idealising
the real plant, such as that due to the inability to model a complex geometry
accurately, three dimensional effects, scaling, control and system simplifications.



Plant uncertainty: Uncertainty in the measuring or monitoring of a real plant, suchas reference plant parameters, instrument error, set points or instrument response,
initial conditions.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

52  NEA/NSC/R(2019)7
Figure 4.1. Illustration of the PIE method and results interpretation

Source: Glaeser, 2008.

4.3 Summary of general frameworks for uncertainty quantification
This section provides a summary of common features and differences of the current
uncertainty quantification (UQ) frameworks in use.

Code scaling applicability and uncertainty (CSAU)In 1988, the United States Nuclear Regulatory Commission (US NRC) revised theemergency core cooling system (ECCS) rule that allowed the use of best-estimate codes in
performing safety analyses with the estimation of uncertainty [66,67]. This alternative
approach commonly is referred to as the Best Estimate Plus Uncertainty (BEPU) method.
In this approach, various sources of uncertainties are combined to predict the aggregate
uncertainty in the results of the analysis.
The US NRC developed the code scaling, applicability and uncertainty (CSAU) [67]evaluation methodology to illustrate, a systematic, auditable method of estimating
uncertainty in the prediction of safety parameters from best-estimate computer codes anddemonstrated the method by applying it to large break loss-of-coolant accidents(LBLOCAs) for a standard PWR. The methodology heavily relies on separate effects tests
and somewhat on integral effects tests. The recognition of scaling distortion of these tests
and their contributions to the overall uncertainty is an important aspect of the CSAU
methodology.
The CSAU methodology is a 14-step procedure arranged in three elements as shown in
Figure 4.2. Element 1 ̶ Requirement and Code Capability ̶ includes the first six steps.
These steps identify the scenario or transient, power plant and important phenomena basedon a phenomena identification and ranking table (PIRT) analysis. Then the code for safety

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  53

analysis is chosen for the transient. Based on the code manuals, the code applicability isdetermined. The code applicability is assessed in Step 6 of the CSAU methodology. The
PIRT identifies the important phenomena for the given scenario and plant. The code’s
formulation, model, and correlations are reviewed to assess if the code has themodel/correlations to simulate the important phenomena, and if the code has scale-up
capability. It is recognised that while the formulation may be general, the correlations orconstitutive relationships are empirical. The code’s scalability will depend on these
correlations and the underlying tests. If the tests scale to the plant for the phenomena ofinterest, then the correlation is applicable. However, if the correlation was derived fromtests that do not scale to the plant, the code may not scale up to the plant. Code validation
with scaled tests or counterpart tests is another way of assessing code’s ability to scale andapplicability.
Element 2 − Assessment and Ranging of Parameters ̶ includes four steps. An assessment
matrix is established based on a PIRT. Nuclear power plant nodalisation is established andsafety calculations are performed. Based on comparing the calculation results with separate
effect test (SET) and integral effect test (IET) data sets, the nodalisation is modified anditerated. Step 9 concerns estimating the code’s uncertainty based on experiments, and Step
10 is for determining the effect of scaling and the resulting bias and uncertainty. With them,
the uncertainties of code models are estimated and are input to Step 13 where all theuncertainties and biases are combined. In Step 9, the uncertainty in predicting phenomenon
is estimated by identifying the appropriate parameters to represent it and selecting theseparate effects tests. In cases where there are data for full-scale facilities with reactor
thermal-hydraulic conditions, modelling the tests with the proper nodalisation, comparing
the predictions with the data and accounting for instrument uncertainty, will suffice.
However, for small tests, a scaling study will be needed to find any scaling bias, orcounterpart tests must be used at different scales. Engineering judgement and sensitivity
studies are also used to estimate the uncertainty range in the designated parameters
representing the phenomenon, or the bias in the figure of merit (FOM).
There are two approaches for estimating uncertainties, based on using integral effects tests,
or separate effect tests. If there are well-scaled integral facilities for the transient of interest,a comparison of the predicted and measured values of the FOM can provide an estimate ofuncertainty. However, it is very difficult to design a single integral effects facility that canscale the transient for all phases. Also, as the uncertainty propagates with time, the initiallower-scale uncertainty will grow over time. Therefore, in general, the role of integral
facilities is to verify an estimate of the overall uncertainty in the FOM attained bypropagating uncertainties through the code and combining them. In this approach, the code
uncertainty is estimated through the combination of uncertainties and biases of different
phenomena as computed from separate effects tests. This overall uncertainty in the FOM
should always be higher than the uncertainty estimated from integral effects test facilities.
In them, the uncertainties of code models that are estimated and are input to Step 13 where
all the uncertainties and biases are combined.
Element 3 − Sensitivity and Uncertainty Analysis ̶ includes the remaining four steps. In
this group, the overall uncertainty of the predicted FOM is evaluated. In the CSAU
methodology, three uncertainty sources are quantified – the code and experiment accuracy(Step 9), the effect of scaling (Step 10) and the reactor input parameters and state (Step 11).
The model uncertainties and reactor condition uncertainties are in the form of bias anddistribution. These are propagated through code calculations for the plant and prediction of
the FOM. These contributions to the FOM or safety parameter are combined in Step 13.
The CSAU methodology ends with the estimation of total uncertainty.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

54  NEA/NSC/R(2019)7
Figure 4.2. The CSAU methodology flow chart

Source: Boyack. et al., 1989.

As an example, the CSAU methodology was applied to an LBLOCA in a standard PWR.
The PIRT development in Step 3 identified five important phenomena; break flow, stored
energy, pump performance under two-phase conditions, steam binding and the ECC
bypass. Some of these phenomena were further broken into sub-phenomena.
Uncertainty in break flow prediction was estimated for subcooled and saturated conditions
as the code has two different models. Marviken tests (Marviken Project, 1974) provided
full-scale data, and so the uncertainty estimate did not need any additional scaling bias. The
ranging of the parameter (surrogate for the critical flow model) and bias were estimated for
subcooled and saturated choking.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  55

The review of available counterpart tests for the primary pump showed scale-dependencies
for the two-phase performance or degradation of the head. The two-phase flow has its own
characteristic length, such as bubble size, and the larger the size of the flow channel, thesmaller is the effect on the degradation of performance. In addition, specific speed also
influences degradation. Larger specific speed-pumps have more axial flow and tend todegrade less. The largest size pump in the database is the Westinghouse pump at a 1/3 scale.
The specific speed is around 5 200 rpm and is the same as a Westinghouse PWR pump.
Comparing two-phase performances of the pumps at different scales, it was concluded thatlarger pumps experience smaller degradations. Therefore, Westinghouse’s 1/3 scale data
can be used for the nuclear power plant pump. It will have a slightly larger degradation
than the nuclear power plant pump. However, this will be conservative as the pump flow
will decrease sooner, at least for LBLOCA.
The stored energy in the fuel was rated very high in the PIRT. It was shown, throughsensitivity analyses that the stored energy is the most sensitive to four parameters, namely,
gap conductance, peaking factor, fuel thermal conductivity and convective-heat transfer.
There are no scaling issues with the first three parameters since they are either independentof the length scale (thermal conductivity) or boundary conditions (peaking factor) or no
scaled study is available (gap conductance). However, the convective-heat transfer was
further broken into two parameters, minimum film-boiling temperature and the convectiveheat transfer coefficient. It is recognised that a lower Tmin will lead to an earlier transition
to film boiling in the blowdown phase and delay the transitioning to subcooled boiling in
the reflood phase and leading to higher peak cladding temperatures (PCT). There were datafrom many tests both for the tubes and the rods and for a large range in pressure. The TRAC
models were compared with the available data. It was recommended that a homogenous
nucleation temperature be used as it gave a lower T min than that predicted by TRAC. The
spread of all the data was 360°F and half of this value (180°F) was recommended for therange. Here, the approach is to use a conservative mean value and the range. The othercontributor to heat transfer is the heat transfer coefficients. During the process of comparing
the data it was found that TRAC over-predicted the heat transfer coefficient at void
fractions greater than 0.75 and that the cause of this over prediction is the film-boiling heattransfer coefficient. In the actual code application, it is recommended that the ForslundRohsenow film-boiling correlation be removed and the uncertainty in the remaining heattransfer models is accounted for through a multiplier with a range of 0.75 to 1.25. The
scaling effect was included in the overall uncertainty in the heat transfer model.
Another important phenomenon that affects the PCT is steam binding in the hot leg and thesteam generator’s u-tube. The liquid carried in the steam generator vaporises and creates ahigh pressure in the upper plenum, delaying the onset of reflooding, leading to a higher
PCT. The sub-phenomena that control vapour-binding are the drops coming from the liquidinterface in the core and the entrainment of these drops into the hot leg and steam generator.
Modelling of two Slab Core Test Facility (SCTF ̶ 1/21 size) tests with TRAC indicatedthat the code under-predicts the entrainment flow to the hot leg. A set of multipliers weredeveloped to apply to TRAC models to increase the number of drops emanating from theinterface and also to increase interfacial shear-stress to match the entrainment to the hot leg
and the steam generator. The SCTF is a small facility. It is assumed that as correlations thatare being affected are localised; they are not dependent on the size of the facility.
The phenomenon of ECC bypass is the most important phenomenon as it determines therefill phase and the flooding and quenching of the core. It affects the reflood PCT. The
ECC injection is from the accumulator to the cold leg and to the down-comer. The steam
up flow in the down-comer towards a break impedes the ECC flow through interfacial
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

56  NEA/NSC/R(2019)7
shear, thus delaying the re-flooding phase. The other phenomenon that assists in refillingis the condensation of steam in the down-comer. When non-condensable gases are present
in the down-comer, condensation is reduced and the ECC bypass is increased. The filling
time of the lower plenum will depend on the combination of these phenomena. Many tests
for ECC bypass phenomenon are available at different scales, such as CREARE (1/5),
Battelle Columbus Laboratory (BCL) (2/15), and CREARE (1/15). A comparison of thepredicted and measured lower plenum filling rates indicated that TRAC over-predicted thefilling rates for small scale facilities and under-predicted them for a 1/5 scale facility. Data
from a full-scale facility, Upper Plenum Test Facility (UPTF) [69], also became available.
The TRAC code under-predicted the filling rate for UPTF [68]. As UPTF is full-scale test,
the model’s uncertainty and the corresponding bias in PCT were estimated directly. In
addition, the TRAC code did not have a model for non-condensable gases. A separate
analysis was performed to estimate the amount of non-condensable gas coming out of thecoolant and its impact on delaying the reflood phase and the corresponding bias in reflood
PCT.
In the CSAU methodology, many integral test facilities at different scales were reviewed;
these scales included semi-scale, loss-of-fluid test (LOFT) [70], cylindricalcore test facility (CCTF) [71], slab core test facility (SCTF) [71], and Primarkreislauf −large-scale test facility (PKL) [72]. It was concluded that all the blowdown PCTs were
within the 95% tolerance limits when plotted as a function of the linear heating-rate. Thisfinding indicated that this is the most dominant phenomenon in the core. However, there
are other system-effects such as pump break that determine the time at which the heattransfer has degraded enough so that temperature rise is only a function of decay heat. In
the case of reflood peak, the plots were made with the rise in core temperature as a functionof the reflooding rate. It indicated that cores were well scaled in different facilities varying
in size from Semi-scale (1/1700) to CCTF (1/21). However, the reflooding rate will begoverned by system performance and any scaling distortion there will affect the rate ofreflooding. The study showed, as expected, that temperature rise declines with an increasein the reflooding rate.
Facilities scale the blowdown phase well. However, the scaling effect becomes more
pronounced for the refill- and reflood-phase due to the interaction of many scaled
phenomena, such as the ECC bypass. To find scale similarities the PCT were plotted as afunction of flow in the core. PCTs from different facilities correlated well. As the core
thermal-hydraulic was decoupled from rest of the system in the refill/reflood phase in thisstudy, the plot suggested that core was scaled well in all the facilities. A bias was estimatedfor the scaling effect of the ECC bypass phenomenon [62].
In estimating the uncertainty or bias induced by scale distortion in an integral test facility,
the impacted parameter (e.g. peak cladding temperature in a LOCA) and the dominant
contributing parameter (e.g. linear rate of heat generation in LOCA) are plotted for all
available data points covering all scales. Then, the uncertainty in reaching the 95%
confidence level is used as a confirmation of the CSAU procedure of aggregating important
uncertainties in phenomena, resulting in total uncertainty in the FOM.
In evaluating the scale-up capabilities of correlations in the code, based on the PIRT result,one evaluates the processes that impact the parameter of interest e.g. the model of thereactor’s coolant pump, the critical two-phase flow through breaks and liquid entrainment
to steam generators in a LOCA. All available scaled data used to develop the correlation or
the models in the code are compiled to determine the uncertainty or bias to reach a 95%
confidence level.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  57

If the distortion and scale-up capability are satisfied, the range of the nuclear power plantdata needs to be compared to the conditions of the database experiments and the closure
correlations in the code. If the nuclear power plant range is not covered, additional biases
are needed. After this evaluation, all the uncertainties and biases are added together as thebias from scaling (Step 10). Both are combined with other sources of uncertainty from
Steps 9 and 11 into the total uncertainty or bias (Step 13) for sensitivity calculation (Step
14,) using the nuclear power plant nodalisation. This will require many runs with the TRAC
code. To avoid expensive runs, the TRAC code was replaced by a surrogate surface byrunning TRAC for different values of the uncertain parameters. Once a response surface isestablished, thousands of values of the FOM are estimated by sampling the uncertainty
range for each parameter representing important phenomena. The large number of values
is then plotted as a probability density function (distribution) and 95% values can beobtained. However, this approach of finding the 95% value using a response surface limitedby the number of phenomena that can be considered.

GRS Uncertainty quantification methodology
The GRS method [73] is a probabilistic method based on the concept of propagating the
input uncertainties. All relevant uncertain parameters including the code, representationand plant uncertainties are identified, any dependencies between uncertain parameters arequantified and ranges and/or PDFs for each uncertain parameter are determined. Expert
judgement and experience from code applications to separate and integral test and full plant
application are principal sources of information for uncertain parameters identification andquantification. Characteristics of the GRS method are:

The uncertainty space of input parameters (defined by their uncertainty ranges) is
sampled at random according to the combined “subjective” probability distributionof the uncertain parameters and code calculations are performed by sampled sets ofparameters.



The number of code calculations is determined by the requirement to estimate atolerance/confidence interval for the quantity of interest (such as peak clad
temperature). The Wilks formula [64,74] is used to determine the number ofcalculations needed for deriving the uncertainty bands.



Statistical evaluations are performed to determine the sensitivities of input
parameter uncertainties on the uncertainties of key results (parameter importanceanalysis).



There are no limits for the number of uncertain parameters to be considered in theanalysis and the calculated uncertainty has a well-established statistical basis.



The method relies only on actual code calculations without using approximations
like fitted response surfaces.

For the selected plant transient, the method is applied to an integral effects test simulating
the same scenario prior to the plant analysis. If experimental data are not bounded, the setof uncertain input parameters must be modified. Experts identify significant uncertainties
to be considered in the analysis, including the modelling uncertainties and the relatedparameters and identify and quantify dependencies between uncertain parameters.
Subjective probability density functions are used to quantify the state of knowledge of
uncertain parameters for the specific scenario. The term “subjective” is used here to

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

58  NEA/NSC/R(2019)7
distinguish uncertainty due to imprecise knowledge from uncertainty due to stochastic or
random variability.
Uncertainties of code model parameters are derived based on validation experience. The
scaling effect must be quantified as model uncertainty. Additional uncertain model
parameters can be included, or PDFs can be modified, accounting for results from theanalysis of SET facilities. Input parameter values are simultaneously varied by random
sampling according to the subjective PDF and dependencies. A set of parameters isprovided to perform the required number n of code runs. For example, the 95% fractile and95% confidence limit of the resulting subjective distribution of the selected output
quantities is directly obtained from the n code results, without assuming any specificdistribution. No response surface is used or needed.
Sensitivity measures by using regression or correlation techniques from the sets of input
parameters and from the corresponding output values allow the ranking of the uncertain
input parameters in relation to their contribution to output uncertainty. Therefore, theranking of parameters is a result of the analysis, not of prior expert judgement. The 95%
fractile, 95% confidence limit and sensitivity measures for continuous-valued output
parameters are provided.
Upper statistical tolerance limits are the upper β confidence for the chosen α fractile. The
fractile indicates the probability content of the probability distributions of the code results(e.g. α = 95% means that PCT is below the tolerance limit with at least α = 95% probability).
One can be 95% confident that at least 95% of the combined influence of all thecharacterised uncertainties are below the tolerance limit. The confidence level is specified
because the probability is not analytically determined. It accounts for the possible influenceof the sampling error since the statements are obtained from a random sample of limited
size. The smallest number n of code runs to be performed is given by the Wilks formula[74] (Equations 1 and 2). The minimum number n of calculations for both one-sided andtwo-sided can be found in Table 4.1. As a consequence, the number n of code runs is
independent of the number of selected input uncertain parameters, only depending on thepercentage of the fractile and on the desired confidence level percentage. The number ofcode runs for deriving sensitivity measures is also independent of the number ofparameters. As an example, a total number of 100 runs is typical for the application of the
GRS method. For regulatory purposes where the margin to licensing criteria is of primaryinterest, the one-sided tolerance limit may be applied, i.e. for a 95th/95th percentile 59
calculations should be performed. The FOM is arranged in decreasing order and the higher
value represents 95% percentile with 95% confidence. The Wilks formula has been
extended for more than one resulting output parameter in a recent work [75].

GRS Sampling-based uncertainty analysis codes
General method
For the investigation of the uncertainty of output quantities obtained by simulation
programmes, sampling-based uncertainty analyses are performed. With this approach,
input uncertainties are propagated through the calculation chain to output uncertainties forquantities of interest. At first, the uncertain input parameters must be identified.
Furthermore, the corresponding probability distribution functions for each uncertain input
parameter must be chosen (e.g. the normal distribution with a mean value and a standard
deviation) and possible correlations between input parameters must be identified. Based on
this information, a set of samples for each input parameter is generated such that new input

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  59

data sets are created. The simulations are then performed with these varied input data. For
the output quantity of interest, a set of results is obtained which can be statistically analysed
to determine a mean value and a standard deviation. The standard deviation corresponds tothe uncertainty of the respective quantity due to the perturbed input parameters.

XSUSA
The XSUSA (Cross Section Uncertainty and Sensitivity Analysis) method [76,77] is basedon the random sampling GRS method implemented in the code package SUSA “Softwarefor Uncertainty and Sensitivity Analysis”) [78]. The uncertain input parameters as
described above are the neutron cross-sections that serve as input to neutron transport
codes. When applying the random sampling method with neutron cross-section
uncertainties, many nuclear data libraries are generated in which the quantities withavailable uncertainties (essentially inelastic and elastic scattering, the (n,2n) and capture
cross-sections, the fission cross-section, the number of neutrons per fission and the fission
neutron spectrum) are varied at the same time for a large number of nuclides. The basis forgenerating the data variations is the SCALE 6.1 covariance data library which contains
uncertainties for relevant nuclides [79]. Correlations between the energy group data of eachnuclide/reaction combination are considered as well as cross-correlations between data ofdifferent reactions or even data for different nuclides. It is furthermore assumed that theneutron cross-sections follow normal distributions with mean values and standard
deviations from the cross-section library and the covariance data files, respectively. Any
quantity that is output from the criticality or depletion calculation with the considered
neutron transport code can be statistically analysed. These typically include themultiplication factor, reactivity coefficients, power distributions as well as homogenised
macroscopic cross-sections generated for use in nodal diffusion codes [80,81].

SUSA
The software tool SUSA (Software for Uncertainty and Sensitivity Analyses) [82-84]
guides through the main steps of a probabilistic uncertainty and sensitivity analysis. These
steps can be summarised as follows:

identification of all phenomena, modelling assumptions and parameters that arepotentially important contributors to the uncertainty of the computational result andrepresentation of all uncertainty sources by uncertain parameters;quantification of the state of knowledge on the uncertain parameters in terms ofprobability distributions and dependence measures;generation of a sample of values for the uncertain parameters according to a
multivariate probability distribution which satisfies the input given in Step 2;computer code executions for each set of values sampled for the uncertain
parameters ̶ random sample from the unknown probability distribution of thecomputational result;quantification of the uncertainty of the computational result based on the sample
resulting from Step 4;ranking of the parameters with respect to their contribution to the overall
uncertainty of the computational result ̶ sensitivity analysis;

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

60  NEA/NSC/R(2019)7


comprehensive documentation of the analysis steps for scrutinising the analysis
results.

The last released version SUSA 4.0 is a powerful tool for uncertainty and sensitivity
analyses of computational results and an important part of the GRS code system for nuclear
reactor safety analyses. It provides support to quantify input uncertainties in terms ofprobability distributions, correlations and other appropriate dependence structures. For
Monte Carlo simulation, the simple random and the Latin Hypercube sampling procedure
are available. To prepare and launch computer code runs, a selection of code interfaces is
implemented. In the field of nuclear reactor safety analyses many options exist forquantifying the uncertainty of a computational result. Options for performing a sensitivity
analysis are also implemented. SUSA 4.0 combines well-established methods fromprobability calculus and statistics with a graphical user interface (GUI). The GUI guides
through the main analysis steps, requests input data where necessary, checks for input
errors and performs all other actions up to the final representation of results.

Automated Statistical Treatment of Uncertainty Method (ASTRUM)
Transients in nuclear power plants are simulated with large system codes that include best
possible models and correlations for thermal hydraulics and neutronics. The predictions
from these codes are best estimates of figures of merit or parameters of interest. However,there are many contributors to uncertainty in these predictions; thermal-hydraulic models,
numerics and parameters of neutronics. The US NRC has allowed utilities and vendors touse best-estimate codes with an estimate of uncertainty with high confidence.
10 Code of Federal Regulations (CFR) 50.46 states that “uncertainty must be accounted
for, so that, when the calculated ECCS cooling performance is compared to the criteria set
forth in paragraph (b) of this section, there is a high level of probability that the criteria
would not be exceeded.” Paragraph (b) of 10 CFR 50.46 contains the list of the acceptancecriteria. Clarification as to the US NRC expectations on the acceptable implementation of
the “high probability” requirement is provided in Section 4 of Regulatory Guide 1.157(Best-estimate Calculations of Emergency Core Cooling System Performance) that states“a 95% probability is considered acceptable by the NRC staff [· · · ].”The CSAU methodology was the first to establish a systematic and auditable method toestimate uncertainty in best-estimate calculation with high level of confidence.
Westinghouse developed Automated Statistical Treatment of Uncertainty Method(ASTRUM) in line with the CSAU methodology and followed two of the elements of the
CSAU methodology. Reviews of ASTRUM have been provided by Frepoli [85] and for amethod like ASTRUM by Martin and O’Dell [86].
The CSAU methodology has two important steps, PIRT in the first element that identifies
important phenomena for the transient and the method of combining various contributors
to uncertainty in code prediction for identified FOM in the third element. the individual
uncertainties are in the form of distributions and biases. as the cost of each code run washigh and number of calculations needed for high confidence was large, the CSAU
methodology adopted a response surface approach. The response surface is a surrogate of
the code. Because of the level of effort, PIRT was important to reduce the number ofphenomena to a manageable number. The response surface approach restricted the method
to a smaller set of contributors to the uncertainty. As the response surface can be sampled
as many times as needed in a very short time, the results corresponding to the 95%
percentile value has almost 100% confidence.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  61

The ASTRUM uncertainty propagation method for uncertainty analyses is similar to the
CSAU methodology and considers the effect of the uncertainties of input parameters,
computer code models, the initial and boundary conditions, geometry, scale effect and otherapplication-specific input data and solution algorithms on the predicted results for the
FOM. ASTRUM differs in the way uncertainties are combined and the numbers ofcontributors to the uncertainty are considered. The most widely used method of this type isthe methodology proposed first by GRS and based on well-established concepts and tools
from probability and statistics given by Wilks. The main advantage to using these tools isthat the number of calculations is independent of the number of uncertain parameters to beconsidered. The necessary number of code calculations is given by the Wilks’ formula [74].
The number of calculations depends only on the chosen tolerance limits (confidence) orintervals of the uncertainty statements of the results. Wilks’ formula does not need any
assumption about the distribution of the result. The number code calculations, N, for given
confidence (tolerance) of 𝛾 and percentile of 𝛽, for one parameter uncertainty is given by
Wilk’s formula:𝛾 = 1 − 𝛽𝑁(6)In case of multiple parameters, p, that are independent, the number of calculations is higher,
and are given by following relationship [65]:𝑁−𝑝𝑁!

𝛽 = ∑𝑗=0 (𝑁−𝑗)!𝑗! 𝛾 𝑗 (1 − 𝛾)𝑁−𝑗(7)

For example, for a single parameter, the number of calculations needed for 95% confidence
and 95% percentile is 59. In the case of three parameters, the number of calculations
required for the same level of confidence and percentile increases to 124.
Modelling uncertainties are represented by specific parameters. These represent two
possibilities:adding on, or multiplying correlations by a corrective term;a set of alternative model formulations.

Then, the respective state of knowledge is quantified by approximate probability
distributions that includes the range and shape of the distributions. Such a distribution
expresses how well the appropriate value of an uncertain parameter of the code application
is known in the light of all available evidence. A state of knowledge based on minimum
information at the parameter level is expressed by uniform distributions (state of maximum
ignorance). The selection and quantification of these uncertain parameters are based on
experience gained from validating the computer code by comparisons between the model’s
predictions and test data of integral tests and separate effects tests for the model parameters.
These tests are based on phenomena identified in the PIRT. In addition, where the tests areat smaller scale, the uncertainty should be augmented to include the scale effect.
In the ASTRUM approach, values of uncertain parameters are randomly sampled and thecode is run with a set of values representing all the uncertainties considered. For every run,a new set is randomly selected. The minimum number of code calculations depends on therequested probability content and confidence level of the statistical tolerance limits used in
the uncertainty statements of the results. The required minimum number of thesecalculation runs is given by the Wilks’ formula as stated earlier. The set of predicted valuesof the parameter are arranged from small to large value. In the case of PCT, the highest
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

62  NEA/NSC/R(2019)7
value will represent the 95% percentile (that is 95% of values will be lower than this PCT)and 95% confidence. In the case of three parameters of interest, the set of values of threeparameters from 124 runs are arranged in order. The top third value for each parameter
represents 95/95 conditions.
It should be noted that the SET/IET matrix will be different for different transients. The
range and distribution of each parameter representing the model uncertainty will be specificto the transient of interest. The PIRT will also be different. While the ASTRUM approachis not limited by the number of parameters representing the uncertainty, still the largest
effort is in determining the range and distribution of these parameters. A PIRT helps in
making problem manageable.
Frepoli [83] has shown through examples that the safety parameters predicted for the 95%
percentile by CSAU-type approach and ASTRUM approaches are different and there islarger margin with ASTRUM predictions (lower PCT, lower maximum oxidation andlower core-wide oxidation).

Uncertainty Methodology based on Accuracy Extrapolation − UMAE
The UMAE [87] whose flow diagram is given in Figure 4.3, is the prototype method forthe description of “the propagation of code output errors” approach. The method focuses
not on the evaluation of individual parameter uncertainties but on the propagation of errors
from a suitable database calculating the final uncertainty by extrapolating the accuracy
from relevant integral experiments to full scale nuclear power plants.
Considering ITF of reference water cooled reactor and qualified computer codes based on
advanced models, the method relies on code capability, qualified by application to facilities
of increasing scale. Direct data extrapolation from small scale experiments to reactor scale
is difficult due to the imperfect scaling criteria adopted in the design of each scaled down
facility. So, only the accuracy (i.e. the difference between measured and calculated
quantities) is extrapolated. Experimental and calculated data in differently scaled facilities
are used to demonstrate that physical phenomena and code predictive capabilities ofimportant phenomena do not change when increasing the dimensions of the facilities (seeright loop FG in Figure 4.3).
Other basic assumptions are that phenomena and transient scenarios in larger scale facilities
are close enough to plant conditions. The influence of user and nodalisation upon the output
uncertainty is minimised in the methodology. However, user and nodalisation inadequacies
affect the comparison between measured and calculated trends; the error due to this isconsidered in the extrapolation process and gives a contribution to the overall uncertainty.
The method utilises a database from similar tests and counterpart tests performed in ITF
that are representative of plant conditions. The quantification of code accuracy (step “fˮ in
Figure 4.3) is carried out by using a procedure based on the Fast Fourier Transform Based
Method (FFTBM, [88] and [89]) characterising the discrepancies between code
calculations and experimental data in the frequency domain and defining FOM for theaccuracy of each calculation. Different requirements must be fulfilled in order toextrapolate the accuracy.
Calculations of both ITF experiments and nuclear power plant transients are used to attain
uncertainty from accuracy. Nodalisations are set up and qualified against experimental databy an iterative procedure, requiring that a reasonable level of accuracy is satisfied. Similar
criteria are adopted in developing plant nodalisation and in performing plant transient
calculations (see left loop FG in Figure 4.3). The demonstration of the similarity of the

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  63

phenomena exhibited in test facilities and in plant calculations, accounting for scaling laws
considerations (step “kˮ in Figure 4.3), leads to the analytical simulation model, i.e. aqualified nodalisation of the nuclear power plant.
The following three main differences between the CSAU methodology and the UMAE canbe identified:Only expert (or engineering) judgement can stop the process of getting uncertainty
in the case of the CSAU methodology, while a detailed comparison between
measured and calculated trends may give the same results for the UMAE (path FG
in Figure 4.3).



Several sensitivity calculations using a plant nodalisation approved by expert
judgements are necessary in the CSAU methodology to get uncertainty; one plantcalculation through a qualified nodalisation is necessary in the UMAE.



To get uncertainty from the UMAE, experimental data in ITF must be available andrelated to the assigned transient; this is not the case in the CSAU methodology.
Furthermore, the code must be able to predict the measured scenario.



Minor differences between the CSAU methodology and the UMAE are related to
the following:– User qualification: unqualified users presumably will not get acceptable resultsfrom the block “f” in Figure 4.3, while they can perform sensitivity calculationsin the CSAU methodology.
– Errors that may be present in the plant nodalisations of both the CSAU
methodology and the UMAE. The probability that this happens in the UMAE is
minimised because of the analysis at block “k” in Figure 4.3.
– The use of the response surface methodology is included in the CSAU
demonstration and not in the UMAE.
– The UMAE assumes that the ratio between experimental and calculated
quantities (YE/YC) is a statistical quantity.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

64  NEA/NSC/R(2019)7
Figure 4.3. UMAE flow diagram (also adopted within the process of development
and application of CIAU)
GeneralQualification
Process

b

Code
GI

i

Plant
nodalization

a

FG
h

Generic
experimental
data

ITF
Nodalizations

g
Nodalization anduser qualificationcSpecific
experimental data

d

NO
j

Stop of theprocessPlantcalculationNO

ITF
CalculationeFGkAccuracy
Quantification (°)

Demonstrationof Similarity (°)

f(Phenomena Analysis)(Scaling Laws)

Accuracy
Extrapolation (°)
l

YES
m

ASM
Calculation(°) Special methodology developed

LN (°)

UncertaintynSource: D’Auria et al., 1995.

Code with capability of IAU (CIAU) and code with capability of IAU for coupled
thermal-hydraulic/neutronics calculation (CIAU-TN)
All the uncertainty evaluation methods discussed in the previous section are affected by
two main limitations:the resources needed for their application may be very demanding, ranging up to
several man-months;



the achieved results may be strongly method/user dependent (the so-called
uncertainty-method-user effect).

The last item should be considered together with the code-user effect, widely studied in thepast and may threaten the usefulness or the practical applicability of the results achievedby an uncertainty method. Therefore, the Internal Assessment of Uncertainty (IAU) was
requested as the follow-up of international conferences ([89] and [90]). The CIAU and
CIAU-TN approaches have been developed with the objective of reducing the limitations
mentioned above.
The CIAU and CIAU-TN are extensively discussed in the available technical literature [9195] and tens of additional relevant papers that provide comprehensive details about themethod, can be found in the bibliography lists of the above references. Therefore, thepresent section supplies only “spot-informationˮ about the CIAU, giving emphasis mostly
to the idea behind the method and its main features.
The bases of the CIAU method can be summarised by the following steps:

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  65the use of a “systematic qualification processˮ [95] of thermal-hydraulic code
calculations related both to ITFs and to nuclear power plants;

the “accuracy extrapolationˮ, i.e. the accuracy is not a function of the volume
scaling ratio KV and thus it can be extrapolated from facilities to nuclear power
plants to get uncertainty;

the “nuclear power plant status approachˮ to identify “phase spacesˮ (i.e.
combinations of finite intervals of selected – driving – quantities) to which areassociated single uncertainty values for each of the selected – output – quantities(i.e. responses);

the separation and recombination of time and quantity “errorˮ to split the physical(i.e. phenomena based) statistical treatment of the uncertainty in two contributions
associated with the values of the selected – output – quantities (i.e. responses) andwith the time when those values are reached during the transient;

the “error filling processˮ and the “error extraction processˮ to generate theaccuracy database (error filling process) and to use the derived uncertainty database(error extraction process) for the uncertainty evaluation of the qualified (following[95]) nuclear power plant code calculation.

In the next sub-sections, the word “errorˮ has the same meaning of “accuracyˮ, i.e. it refers
to the discrepancy between experiment and code results.

The qualification process
The qualification process described in [95] involves the fulfilment of different “conditionsof acceptabilityˮ for demonstrating the achievement of qualified ITF and nuclear power
plant nodalisations and related code calculations (in this term it can be considered like the“engineˮ of the CIAU). Data coming from generic experiments in integral facilities and in
separate effect test facilities, other than counterpart and similar tests and nuclear power
plant transients can be processed. One condition for the application of the CIAU method is
the similarity between the concerned plant scenario, in relation to which the uncertainty
must be calculated, and the experimental/reference database originating the accuracy
database (see also Figure 4.4).

The accuracy extrapolation
This step implies the demonstration that the code predictive capabilities of important
physical phenomena in differently scaled facilities do not change when increasing thedimensions of the facilities. Figure 4.4 depicts the expected behaviour of the ratio between
YE and YC of a generic response Y (e.g. PCT, minimum core collapsed level, etc.) as afunction of KV where YE and YC are respectively the experimental and calculated valuesof Y. The expected behaviour of the ratio YE / YC is found, as shown in Figure 4.4 andconfirmed by the analysis of several experiments and associated code calculations [96,97]
derived from a database of similar tests and counterpart tests performed in ITF that are
representative of the plant conditions. It should be noted that code-user effect andnodalisation inadequacies affect the comparison between measured and calculated trends
and thus they are considered in the accuracy evaluation and consequently in the accuracy
extrapolation process providing a contribution to the overall uncertainty.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

66  NEA/NSC/R(2019)7

The nuclear power plant status approach
The usual characterisation of any transient or event occurring or calculated in a typical
LWR is through a number of time trends, e.g. pressures, levels, temperatures, mass flow
rates versus time. The event time, or the time elapsed since the event beginning, constitutes
the main way to characterise the transient together with the initial and boundary conditions.
In this case, which can be identified as “time-domainˮ, time is taken as the horizontal axis
in the graphical representation of the transient evolution. Therefore, in the area of
uncertainty evaluation, each transient becomes unique, thus requiring a specific evaluationof the uncertainty that characterises any of the time trends. This is true notwithstanding thepossibility to consider key phenomena [98] that are common to classes of transients.
A different way to look at the same transients involves the use of the “phase-spaceˮ. Thisapproach consists of selecting a fixed, small group of quantities (called “driving quantities”
Qd) and in describing any event taking place in a nuclear power plant not as a function oftime, but by the group of values assumed by the selected quantities: each group of theselected variables represents a status of the plant. This approach is actually utilised to
optimise the emergency procedures of nuclear power plants. In the graphicalrepresentation, any relevant quantity can be used in the vertical or horizontal axis.
Figure 4.4. Accuracy behaviour as a function of Kv

Source: Petruzzi, Giannotti and D’Auria 2004; Petruzzi and D’Auria, 2016; Bovalini et al., 1992.

Although the transients seem to be quite different in the time domain, when they are
analysed in the phase-space domain it is possible to identify some areas or
phenomenological regions through which the different kinds of transients pass through andin correspondence to which the same phenomena are expected to occur. The basic idea of
the CIAU and CIAU-TN methods is that at any of these phenomenological regions, into
which the “phase-spaceˮ is subdivided, one uncertainty value for the selected output
quantities (called “object quantities”, Y) can be assigned independently on the kind oftransient. In other words, the nuclear power plant status is a region of phase space where
the uncertainty in the code prediction is assumed to be “uniformˮ.
The same idea, referring to specific thermal-hydraulic phenomena, is discussed in [98-100].
Those papers show that phenomenological areas or regions in the “phase-spaceˮ aresuitable for use in scaling and extrapolation studies. Additional support for planning themethod comes from the characterisation of generic plant status for the actuation of accident

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  67

management countermeasures, as discussed in [101]. Finally, the pursued approach issimilar to what is proposed by D.C. Groeneveld and P. Kirillov [102]: in that case, pressure,quality and flow rate are entries to the “look-upˮ table that provides a suitable value for the
CHF. In the present case, proper “driving quantitiesˮ are entries to the matrix and vector
which provide quantity and time uncertainty values respectively.

The separation and recombination of time and quantity errors
The definition of time and quantity error can be drawn from Figure 4.5. The dotted line is
the result of a system code calculation: Y is a generic thermal-hydraulic code output plottedversus time. Each point value in the curve is affected by a time error (Et in Figure 4.5a) andby a quantity error (Eq in Figure 4.5b). The availability of experimental data (measured inappropriate nuclear power plant simulators, i.e. ITFs) allows quantifying those errors andto generate the so-called (in the CIAU nomenclature) time and quantity accuracy database.
Owing to the uncertainty affecting any thermal-hydraulic code calculation, each point valueof the nuclear power plant code result may take any value within the rectangle (Figure 4.5c)
identified by the time (Ut) and quantity (Uq) error (uncertainty).
The amount of the uncertainty value (i.e. each edge of the rectangle) can be defined inprobabilistic terms consistently with what is recommended by current licensing
approaches: e.g. a 95% probability level is considered acceptable to the US NRC staff forcomparison of best-estimate predictions of postulated transients to the licensing limits in
10 CFR Part 50. The method used to combine the rectangles at the end of the CIAU process
for generating the CIAU uncertainty bands can be seen in Figure 4.5d. The adopted processensures a higher (still not quantified) level of probability with respect to the 95%
probability usually associated with the edge of the rectangle.
Figure 4.5. Definition of quantity and time errors to be included into
the quantity and time uncertainty database
Et

Y

Y

Eq
Exp.

Exp.

Calc.

Calc.
Time

Time

a) only Time Error (Et) is presentb) only Quantity Error (Eq) is present

Y

Y
Y

Ut

Uq

c) Combination of Time (Ut) and
Quantity (Uq) Uncertainty

Time

Time
d) Derivation of Continuous Uncertainty Bands

Source: D’Auria and Giannotti, 2000; Petruzzi et al., 2005; Petruzzi and D’Auria, 2008; Petruzzi. D’Auria, Giannotti
and Ivanov, 2004; Petruzzi, Giannotti and F. D’Auria, 2004.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

68  NEA/NSC/R(2019)7

The error filling process and the error extraction process
Two processes are foreseen for the realisation of the CIAU and CIAU-TN methods: the“error fillingˮ process and the “error extractionˮ process (see Figure 4.6). The former isdealing with:

the selection (block a) of relevant experiments (ITF and SETF), i.e. of thoseexperiments whose geometrical properties of the facility and boundary and initialconditions are similar to those of the concerned plant scenarios;the code calculation results (block b) qualified following the qualification
procedure(block c) of [95];

the derivation of the separate time and quantity accuracy (error) database (block d);the identification of the nuclear power plant statuses (block e);the storing of the time and quantity accuracy (error) values inside the selected (bythe ITF and/or SETF experiment scenario) nuclear power plant statuses (block f).

After that a qualified nuclear power plant code calculation (block h) has been madeavailable following the qualification procedure (block c) [95], the “error extractionˮ
process (block i) is used to draw out from the selected (by the transient) nuclear power
plant statuses the uncertainty values to be associated with the nominal (best-estimate)values of the object quantities for the uncertainty evaluation (block j). It should be notedthat only one nuclear power plant best-estimate calculation per transient is sufficient for
performing the uncertainty analysis. Between the two processes, the step dealing with theaccuracy extrapolation (block g) ([96-98], see also Figure 4.6) is performed for passing
from the accuracy database (output of the “error fillingˮ process) to the uncertainty
database (input of the “error extractionˮ process).

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  69

Figure 4.6. The error filling process and the error extraction process

Source: Petruzzi and D’Auria, 2016; Bovalini et al., 1992; D’Auria and Galassi, 1998.

A key feature of the CIAU and CIAU-TN approaches is the full reference to theexperimental data. Accuracy from the comparison between experimental and calculated
data is extrapolated to obtain uncertainty. A solution to the issues constituted by the“scaling” and “the qualification” of the computational tools is embedded into the method[100,101] through the qualification process that constitutes the engine for the developmentof CIAU and for the creation of the accuracy database.
Assigned a point in the time domain, the accuracy in predicting the time of occurrence ofany point is distinguished from the accuracy that characterises the quantity value at thatpoint. Quantity and time accuracies are associated to inaccuracies-in-code-models anduncertainties-in-boundary-and-initial-conditions including the time sequence of events andthe geometrical model of the problem. Based on the above:The “transient-time-dependentˮ calculation by a code resembles a succession ofsteady-state values at each time step and is supported by the consideration that thecode is based on a number and a variety of empirical correlations qualified atsteady-state with assigned geometric discretisation. Therefore, quantity accuracy
can be associated primarily with inaccuracies -in-code-models.



Inaccuracy associated with the physical actuation of components (e.g. for a valve
the time when the equivalent full flow area for the flow passage is attained) or themodelling of the I&C system (e.g. the scram signal) or inadequate nodalisation, canintroduce time errors that cannot be associated to code model deficiencies.
Therefore, time accuracy can be associated primarily with uncertainties-inboundary-and-initial-conditions.

Once the time and quantity accuracy database has been derived, the overall accuracy andthen uncertainty is obtained by the geometric combination of the two accuracies (anduncertainties) values, i.e. time and quantity, in the two-dimensional space-time plane.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

70  NEA/NSC/R(2019)7

4.4 New methods
CASUALIDAD method
The predictive modelling methodology constitutes a third approach for performing
uncertainty analysis, different from propagation of input uncertainties or from propagationof code output accuracies.
The method is based upon powerful mathematical tools to perform sensitivity analysis andupon the Data Adjustment/Assimilation (DAA) methodology [103,104] by which
experimental observations are combined with code predictions and their respective errors,
to provide an improved estimate of the system state and of the associated uncertainty
considering all input parameters α that affect any prediction, being part either of the code
models or of the input deck.
Since in most problems of practical interest, the number of input parameters α exceeds thenumber of responses R, the method requires the most efficient deterministic method forcomputing local sensitivities S for large-scale systems, which is the Adjoint Sensitivity
Analysis Procedure (ASAP) [105,106]. However, being large-scale systems frequently
characterised by critical points like bifurcations, turning points, saddle points, response
extrema or cliff-edge effects, local sensitivity tools cannot be properly used and adeterministic global sensitivity method has to be applied. In this context the Global Adjoint
Sensitivity Analysis Procedure (GASAP, [105,106]) is one of the most advanced tools todeterministically compute the system’s critical points y in the combined phase space
formed by the parameters, forward-state variables and adjoint variables. Subsequently the
local sensitivities of the responses R located at critical points y are analysed by the ASAP.
Once the sensitivity matrix S of the responses R with respect to the parameters α isavailable, the moment propagation equation is adopted to obtain the computed covariancematrix CR of the responses starting from the covariance matrix Cα of the system parameters.
The technique, by which experimental observations are combined with code predictions
and their respective errors, to provide an improved estimate of the system state is known
as DAA [104] and it is based on a Bayesian inference process. The idea at the basis of DAA
can be made more specific as follows: the computed results R and the respective statisticalerrors CR predicted by mathematical models and based on a “priorˮ or “firstˮ guess PDF
for the input parameters (i.e. Cα) are combined with proper experimental observations M
of the states of a system and associated uncertainty CM to generate “adjustedˮ values forthe system parameters (αIE, where the suffix Improved Estimate values (IE) and therespective input covariance matrix (CαIE, or “posteriorˮ PDF). From this process, which
can be considered as improved estimate analysis of the system’s states, the responses RIE
and the respective covariance matrix (CRIE) are finally derived. A short description of amethod based on predictive modelling methodology is given hereafter.
The CASUALIDAD method (Code with the capability of Adjoint Sensitivity andUncertainty AnaLysis by Internal Data ADjustment and assimilation) [107-109] has beendeveloped as a fully deterministic method based on advanced mathematical tools for
performing internally to the thermal-hydraulic system code the sensitivity and theuncertainty analysis. The availability of a suitable database of experiments (SETF and/or
ITF) and related qualified code calculations constitutes a prerequisite for the development
and the following application of the methodology. A simplified flow chart of the method
is depicted in Figure 4.7.
Similar to the CIAU (see section above) method, the process to develop the database of
improved estimations of input values and related covariance matrix is distinguished from
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  71

the application process where the database is used to produce the improved uncertainty
scenario. Thus the information embedded in the set of experiments and related qualified
code calculations is elaborated and transformed into suitable information for performing
an improved (in the sense that is based on the experimental evidences) uncertainty
evaluation. The general principles of the methodology together with brief descriptions of
the main tools constituting the CASUALIDAD are given in the first sub-section below,
whereas the structure of the method (i.e. the development process and the application
process) is discussed in detail in the second sub-section below.
Figure 4.7. Uncertainty evaluation based on CASUALIDAD methodology

Source: Petruzzi, 2008; Petruzzi et al., 2010; Petruzzi and D’Auria, 2014.

The bases of the CASUALIDAD method
Establishing the range and probability distribution function of parameters is fairly easy forparameters which describe the condition of the plant (such as initial power or geometricaldata) or for parameters describing physical data (such as thermal conductivity of UO2). It
is more difficult for parameters relative to the constitutive relationships (such as interfacial
friction) because they cannot be directly measured in facilities. To this end, the results fromSETF together with their experimental uncertainties are used for establishing or assessingthese constitutive relationships. However, the models developed from SETF are applicable
to a certain range of parameters, when often the correlations are extrapolated to more
extreme conditions in correspondence of which the data is lacking. In order to justify these
extrapolations, the ITF data, whose ranges of parameters are much closer to typicalconditions, are used. However, the derivation of the PDFs of the input uncertainty
parameters is never an easy and objective task and a common or agreed methodology is
still not available to the nuclear community.
The fundamental principle of the methodology here proposed can be derived from
Figure 4.8. The initial uncertainty distribution obtained from SETFs is shown coloured in
red and represents the uncertainty in each of the model parameters (multiple model
parameter distributions). After the application of the Bayesian Theorem (BT) and of the
Principle of the Maximum Likelihood (PML), the uncertainty distributions (in blue), which
are prior estimations for the code-simulation, are obtained. The code simulations are further
constrained (using BT and PML) against the ITF for considering both the system behaviour
and the use of the correlations (developed starting from SETFs) outside their range of
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

72  NEA/NSC/R(2019)7
validity. At the end of this step, the key parameters, their contributions to the uncertaintyof the system responses and the posterior parameter distributions (green curve in
Figure 4.8) are obtained. It should be noted that if the extrapolation of the SETF conditions
to the ITF conditions were not required and the various physics models employed in the
ITF were truly uncoupled, then it is expected that the prior and posterior distributions
resulting from the application of the BT & PML are very similar.
The framework for which the CASUALIDAD method has been developed deals with thedashed area in Figure 4.8, or in other terms the methodology is system oriented having as
its main objective the improvements of the estimations of the system output responses andrelated covariance matrix (i.e. uncertainty) through the reduction and improvement of the
input parameter values and covariance. Key elements for the methodology are:


availability of a frozen qualified and internationally recognised thermal-hydraulic
code;



availability of a suitable large database of ITF containing different transient
scenarios and different scales of facility (e.g. the set of tests belonging to the CIAU
database [109,110]);

use of a robust and rigorous procedure (i.e. set of acceptability criteria to besatisfied) for the qualification and acceptance of code calculations [95];availability of adequate (i.e. exact and efficient from computational resource pointof view) methods for performing the local and global sensitivity analysis. The
powerful tool based on ASAP is implemented for the derivation of the sensitivity
matrix S containing the local derivatives of any response Rn (obtained by the code)with respect to any parameter αi ( Rn αi ). For large-scale systems, in which thenumber of system parameters and/or parameter variations to be considered exceeds
the number of responses of interest, the ASAP is an effective method to employ.
The global sensitivity analysis is performed through the implementation of the
GASAP method aimed at determining all of the system's critical points andsubsequently analyse them locally by ASAP. Thus, the strong non-linearities
characterising the nuclear power plant system can be efficiently considered during
the sensitivity analysis and taken into account for the following step dealing withthe uncertainty evaluation;

implementation of the DAA method based on the BT and the PML for updating thea priori PDF of the input parameters α and responses R with the available
experiments M (“likelihood observations”, see also Element 2 above) for getting
the “posterior” IE of the input parameters, responses and related covariancematrices;use of the concept of status approach for grouping together the “posterior” IE of the
input parameters, responses and related covariance matrices derived from similartransients, i.e. transients selecting the same path in the phase space of the selected
driving quantities. A database of improved estimations is then generated and theappropriate information stored inside (i.e. the one in the phase space selected by
the nuclear power plant calculation) can be used during the application process ofthe methodology.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  73

Figure 4.8. The basic idea and framework of the CASUALIDAD method

Source: Petruzzi, 2008; Petruzzi et al., 2010; Petruzzi and D’Auria, 2014.

In addition to the above, the following assumptions are used in the methodology, including
criteria and steps embedded in the methodology which itself allows either the detection
when the assumptions are violated (and thus when the methodology cannot be applied) or
the demonstration of their well-founded basis:Phenomena and transient scenarios in larger scale facilities will be close enough toplant conditions (see also similar discussion for the CIAU method in the Section
above) ̶ this assumption also supports Element 6 in the list above.

Given that the discrepancy between measured and calculated responses (and not
the absolute values) is used for evaluating the uncertainty, this difference israndomly dispersed around the zero-value independent of the volume scaling
factors of the facilities ([111], see also Figure 4.6). Moreover, the methodology
provides a consistent indicator 𝜒 2 that quantifies the degree of consistency between
calculation and experiment and allows neglecting the contributions of the related
discrepancies to the uncertainty estimation when the consistency is poor (i.e. 𝜒 2 farfrom unity value [112].

The influence of user and nodalisation upon the uncertainty of the response isminimised in the methodology by the use of robust qualification procedures (seeElement 3 of the previous list). The consistency indicator 𝜒 2 contributes tominimise this influence discharging the cases where the consistency between
calculation and experiment is poor.

The structure of the method
The CASUALIDAD methodology has been structured in two main steps. The former has
the aim to generate the database of improved estimations starting from the available set ofexperimental data and related qualified calculations; the latter is dealing with the use of theselected (from the obtained database) set of improved estimations for the uncertainty
evaluation of the predicted nuclear power plant transient scenario. More details about thetwo processes are given hereafter.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

74  NEA/NSC/R(2019)7

CASUALIDAD, Development process
The development of the method and related database implies the availability of qualified
experimental ITF data (block a in Figure 4.9), the use of a robust and consistent procedurefor qualifying code calculations (block b) and the availability of a frozen version of thecode (block c). These requirements constitute pre-requisites to the development of themethodology itself. Moreover, it is assumed that a steady-state (SS) condition is achieved
before running the transient scenario, and consequently the values of input parameterscharacterising the BIC of the problem have to be considered after SS achievement. Moreprecisely four different kinds of input parameters α are distinguished:BIC
the parameters α SS
(block d) which characterise the BIC, evaluated after SS
achievement (e.g. the distribution of temperatures inside a core channel or the core
inlet mass flow rates);

the parameters α PP (block e) used for specifying Properties and Phenomena (PP)of the system. They include for instance the geometrical properties of the system(e.g. the flow area of a pipe or the core active heat transfer surface area), thephysical properties of the materials (e.g. the value of heat capacity at a certain
temperature) and also parameters characterising complex thermal-hydraulicBIC
aspects, like the pressure drop distribution. With respect to α SS
, the α PP
parameters are those parameters whose values specified in the input deck do notchange when the SS has been achieved;the parameters α NOD (block f) which characterise the nodalisation (NOD), i.e.
those parameters that constitute the discretisation of the real system into an
ensemble of control volumes and junctions through the definition of the volume
lengths and elevations, the volume flow areas, etc.;the parameters α CODE (inside block c) that are part of the correlations inside thecode. With respect to the other parameters, the α CODE might not be directly
accessible (modifiable) from the input deck.

The selection of the responses of interest (block g) among the thousands available from acode calculation implies an engineering judgement that must consider the relevance of thesafety and licensing issues. No limitation exists when performing the sensitivity analysis(see blocks i and k). The situation is instead different when performing the uncertainty
analysis (see blocks l and o), where practically the constraints derive from the available
experimental responses (block a). Once the responses R have been selected (block g), they
are passed to the sensitivity tools (block i) together with all input parameters α . The
sensitivity analysis is then performed locally and globally in order respectively to obtain
exact local derivatives of any response Rn with respect to any input parameter( Rn αi ) and to determine the system's critical points   (bifurcations, turning
points, saddle points, response extrema) and their derivatives ( ym αi ) with respect toany input parameter α i (block j). The local sensitivity analysis is performed by ASAP
[105,106] or alternatively can be carried out by Forward Sensitivity Analysis
Procedure (FSAP) [105,106]. The global sensitivity analysis is performed by GASAP[105,106] and makes possible to efficiently deal with the non-linearities characterising thenuclear power plant system. Finally, the sensitivity matrix S (block k) is obtained and canbe used for the following purposes (among others):αiy α

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  75

to support the qualification process of the nodalisation through quantitative
parameters that allow the verification of possible “unstableˮ behaviours of thedeveloped nodalisation, i.e. if a small change in any of the input parameters α
causes a large variation in the selected relevant responses R. In fact, the matrix S
providing the derivatives of any selected response respect with any parameter (e.g.
the derivative of the core PCT with respect to the length of one node in the u-tubeof the primary side of a steam generator) constitutes a large source of information
for performing the qualification process;to understand the system by highlighting important data and eliminating theunimportant ones;

to design and optimise the system (e.g. maximise the availability/minimise themaintenance);to perform the uncertainty analysis (see blocks l and o).
Figure 4.9: Flow chart of CASUALIDAD method: Development process
ITF calc.
(b)
UMAE
(d)

BIC
α SS

(e)

α PP

(f)

(a)

(o)

(g)
Input Deck

Experimental
Data (ITF)BIC
 = { α SS
, α PP ,α CODE}

Selection of
Responses

M, CM

(c)

(h)

TH-CODECODE(( αCODE
)

α NOD

(n)

R
DAA

(m)
BIC
C = {Cα ,SS, CαPP, CCODE}αStandard
Process(i)

Prior inputs

ASAP

Prior
Calculations
PosteriorCalculations(j)Y() , SY

GASAP
(s)



C

BIC
α SS

BIC
 ,SS

Covariance
Calculated
Response

S(FSAP, DIRECT)

Database
IE

IE, α PP

, C

IE

PP

, α CODE
IE

(p)

(l)

(k)

, C

IE

CR = S C ST

(r)

α

C

R
BIC
SS

BIC
 ,SS

IE

IE

,α

IE

PP

, C

, CR

IE

PP

,α
IE

IE

CODE

IE

, C



CODE

IE



(q)



CODE

Improved Estimation

Driving
Quantities

Numerical
Tools

IE



TRANSIENTIDENTIFICATION &Similarity Check

Source: Petruzzi, 2008; Petruzzi et al., 2010; Petruzzi and D’Auria, 2014.

While the sensitivity analysis is able to consider all parameters α , the uncertainty
methodology here developed cannot deal “directlyˮ with the α NOD parameters. The reason
for such a limitation is due to the lack of information about the variance (and generally
speaking about the covariance matrix) of the parameters characterising the nodalisation(e.g. the length of a volume in a nodalisation). However, as deducible from the word
“directlyˮ, it should be emphasised that this limitation does not imply that the input
uncertainties arising from the development of the nodalisation do not contribute to the final
evaluation of the uncertainty of the responses. In fact, these input uncertainties affect thediscrepancy between the experimentally measured values and the calculated values of the
responses and thus finally affect the uncertainty evaluation.
BIC, CαPP , CαCODE (related
The a priori knowledge (block m) of the covariance matrices Cα,SS
to the BIC, PP and CODE parameters), deduced from SETFs, available literature or based

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

76  NEA/NSC/R(2019)7
on engineering judgement, is then used in the “propagation of moments” equation to obtainthe covariance matrix of the calculated responses (block l). The next step (block n) implies
the use of the DAA as a methodology based on the BT and PML for consistently
incorporating observed information into a predicting model. More specifically, the a priori
knowledge of the:


BIC
PP
CODE
input parameters αSS , α , α(block o);input covariance matrices (block m);calculated response R (block h);calculated covariance matrix CR (block l);

is combined in a statistically optimal way with “likelihood observations” (block a), i.e.:experimental measures of the responses M;covariance matrix of the measured responses CM ;to consistently improve and reduce the posterior estimations (block p) of the:

input parameters

BIC
α SS

IE

R

, α PP, α CODE

IE

;

IE



calculated responses



input covariance matrices



IE

;
BIC
Cα,SS

calculated covariance matrix

IE

CR

,CαPP

IE

,CαCODE

IE

;

IE

.

Moreover, the DAA method supplies a quantitative measure of the consistency of theprocess through an indicator factor (  2 ) that allows quantification of the degree of thediscrepancy between calculation and experiment and to not perform the adjustment andassimilation when the consistency is poor. As the CASUALIDAD methodology is systemoriented, the experimental data to be incorporated in the DAA process are mostly dealing
with ITF. However, SETF and also nuclear power plant operational transients can beprocessed if available.
The final step of the methodology has the goal to generate a database of (posterior)
improved and reduced estimations of the input parameters and related covariance matrices.
For each of the experimental test, the process up to the block p in Figure 4.9 has to berepeated for generating the posterior estimations. Considering that different kinds ofscenario can be processed and that the same phenomena can occur in different transients,
the concept of the “status approach” is adopted for generating the database. The idea here
is to group together the improved estimations of the input parameters αIE and related
covariance matrices CαIE deriving from “similarˮ transients, i.e. transients selecting thesame path in the phases space of the selected driving quantities. It is assumed that similar
tests generate similar values of the percentages of parameters’ variation between“posterior” and “priorˮ estimations (similarity check). Thus, if the similarity check is
positively passed, the transient identification process (block q), based on the phase-spacesubdivision characterised by the selected driving quantities (block r), allows for

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  77

identification of the location inside the database where to store the obtained posterior
estimations (block s).

CASUALIDAD, Application process
The application of the CASUALIDAD is straightforward once the database of the improved
estimations of the input parameters and related covariance matrices is available.
After the nuclear power plant transient calculation has been qualified (block b in Figure
BIC, α PP , α NOD , block4.10), the input parameters characterising both the input deck (i.e. αSSc) and the correlations inside the code ( α CODE , inside block b) are evaluated on the “firstguessˮ basis (i.e. values deduced from SETFs, available literature or based on engineering
judgement). The relevant responses from the safety and licensing point of view must be
selected(block d) in agreement with the selection process performed during the development of themethodology. The transient evolution, defined by the quantities R (block e), selects a pathin the phase-space subdivision (characterised by the selected driving quantities, (block f)
which can be identified through the transient identification process (block g). Thus, theposterior estimations of the input parameters (block h) and input covariance matrices(block i) associated to the selected path in the phase space can be drawn out from thedatabase.
A second thermal-hydraulic system code calculation is set up using the posterior
estimations of the parameters characterising both the input deck ( α SS

BIC

correlations inside the code ( α

CODE

IE

IE

, α PP

IE) and the) in order to get an improved estimation of theIE

responses R (block j). The improved nuclear power plant transient calculation still
undergoes the qualification process.
The sensitivity analysis is then performed locally and globally (block k) to derive thesystem's critical points (block l) and the derivatives of any response with respect to any
input parameter (block m). The sensitivity matrix S can be used for the same purposes
stated in the development process of the methodology having as reference the nuclear
power plant instead of the ITF systems. Lastly, the uncertainty evaluation is performed by
the “propagation of moments” equation where the improved estimations of the input
IE

IE

BIC
PP
CODE
covariance matrices Cα,SS , Cα , CαIE

are used (block i) to obtain the improved

estimation of the covariance matrix of the calculated responses CRIE (block n). The diagonalvalues on the CRIE matrix allow finally the generation of continuous uncertainty bands forthe selected responses RIE (block o). A demonstrative application of CASUALIDAD
method is given in [109].

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

78  NEA/NSC/R(2019)7
Figure 4.10. Flow chart of CASUALIDAD method: Application process
(m)

(k)

Covariance
Calculated
Response

Y() , SY

CRIE = S CIE ST

ASAP(FSAP, DIRECT)(d)

Cladding Temperatures

Selection of Responses
and Driving Quantities

1450.0Lower Band
Upper Band

1300.01150.0Temperature (K)

RIE

BIC
SS

(o)
1600.0(j)(c) Input Deck

α

(l)

GASAP

NPP calc.
(a)
UMAE

(n)

S

1000.0

850.0

700.0

α

(d)

TH-CODE
CODE))((CODE

NOD



BIC
α SS

IE

, α PPIE, α CODE

550.0

(e)

(f)

(g)

400.0

250.0
0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

Time (s)

R

IE



(h)

TRANSIENTIDENTIFICATION

α PP

(b)

(i)

C BIC IE , C PP IE   ,SS
IE C CODE

Source: Petruzzi, 2008; Petruzzi et al., 2010; Petruzzi and D’Auria, 2014.

IMTHUA−
Integrated
Uncertainty Analysis

Methodology

forThermal–Hydraulics

The basic idea behind IMTHUA [113] is to develop an uncertainty analysis methodology
for best-estimate thermal-hydraulic codes (e.g. RELAP5) using and integrating all
available evidence (i.e. experimental data, information and expert judgement). The
methodology is quantitative and uses the Bayesian approach for evaluating theuncertainties associated with the thermal-hydraulic calculations.
Taking into account the limitations of the existing methodologies, the following factors arecarefully considered in IMTHUA:

Screening procedure for thermal-hydraulic phenomena: considering the extremely
high number of phenomena that can contribute to the uncertainty, it is neither
possible nor practical to explicitly quantify the impact of all such phenomena on
output uncertainty. Consequently, a screening procedure is needed, but it creates
non-trivial problems about the influence of sources of uncertainty neglected. In
addition to this, it must be considered that different sources of uncertainty do nothave equal influence.



Use of experimental data: the available information should be carefully
manipulated. Inference about physical models from experiments, credibility
assessment and value of parameters and relevance of information strictly require
sophisticated methodologies such as Bayesian methods. Moreover, themethodology has to be general, applicable to a wide range of scenarios and nottightly linked to peculiarities of specific cases.

Assessment and propagation of uncertainties: the method for assessment andpropagation of uncertainties must be efficient considering the available resources.

Black box approach: using certain uncertainty procedures it is not possible toexplicitly consider internal sub-model uncertainty.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  79

Basic on the above, the idea of IMTHUA is to incorporate the best features of the existing
methodologies into a new, more comprehensive method. The objective is to create an
uncertainty method that:

is general and applicable to a wide range of scenarios;treats all the important sources of uncertainty:– input parameters;– models and sub-models;

considers an output adjustment based on integral performance information;

uses a white box approach;is efficient with respect to cost and resources.

The main characteristics of IMTHUA are summarised in the following paragraphs.
Modified PIRT: In thermal-hydraulic calculations the number of phenomena that cancontribute to the uncertainty is huge. Considering that it is not practical to explicitly
consider all of these phenomena, a decision-making tool able to identify and rank the mostimportant phenomena in certain scenario is necessary. This is the idea behind the PIRT.
The problem of the traditional PIRT is that it only considers the effects of an input
phenomenon on the magnitude of the output, without examining the credibility of themodels and correlations related to the considered phenomenon. In order to avoid thislimitation, a modified PIRT is used in IMTHUA. Phenomena are screened and ranked
considering both their thermal-hydraulic and uncertainty importance. An analyticalhierarchical process is used for thermal-hydraulic ranking while an expert judgement
procedure is used to estimate the degree of credibility of the models and correlations used
to represent the considered phenomena.

Input phase: Treatment of input parameters
The input data consists of model coefficients, boundary and initial conditions and it isnecessary to evaluate the uncertainty of such data. Depending on the type of data andinformation available, the quantification of input parameter uncertainty can be done by the
maximum entropy approach and by the expert judgement based on availability and type ofdata and information. A Bayesian methodology allows updating uncertainty distributions
taking into account any new evidence.

Input phase: White Box approach
In addition to the input parameters, boundary and initial conditions, an important source of
uncertainty that should be considered in the input phase is the uncertainty associated withthe code models and sub-models. In this sense, a key objective of IMTHUA is thequantification of uncertainty due to model form. Firstly, the models, their alternative
options and the structure of their engagement in the computation should be identified anddocumented. This is done by a method for measuring the contribution of model structural
uncertainty to output uncertainty in addition to parameter uncertainties. Identified sourcesof uncertainties in code models and correlations are quantified by assigning probability
distributions.
Efficient uncertainty propagation: Input parameter, model and sub-model uncertainties
have to be propagated through code computations. A modified Wilks’ tolerance limit
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

80  NEA/NSC/R(2019)7
criteria sampling is used for the core of uncertainty propagation in IMTHUA, in order toreduce the number of Monte Carlo iterations for a given required accuracy.
Output correction phase: The output correction phase is necessary to account for user
effects, numerical approximations and, in general, sources of uncertainty not considered in
the input phase. In order to correct the output distribution, a Bayesian statistical method isused.
Firstly, an output database should be set up including all available test data on the FOM
classified by applicability and degree of relevance. A Bayesian methodology can be
implemented in the code for output automatic updating by comparing paired calculationand experimental data.
A key point of this step is that the experimental data should be evaluated for their degreeof relevance to the conditions of the evaluated scenario. The data can be considered as:completely relevant: in this case the data is used directly for updating;

partially relevant: in this case a Bayesian weighting process is needed and resultsin the evaluation of a factor assigned to the data before the updating process. The
value of this factor is between 0 (non-applicable) and 1 (applicable) and should bequantified by an expert group for each test facility, considering the similarities anddifferences between it and nuclear power plant and the transient scenario.

This step, together with the treatment of input parameters and the white box approach,makes IMTHUA a hybrid input-output driven method.

SEC NRS Methodology
The Scientific and Engineering Centre (SEC) for Nuclear and Radiation Safety (NRS) in
Russia has developed a regulatory approach for uncertainty analysis in the safetyassessment of nuclear installations. Russian nuclear safety regulations require safety
analysis to be supported with an evaluation of errors and uncertainties of the obtained
results. SEC NRS is the technical and scientific support organisation of the Russian
regulatory authority (Rostechnadzor). For nuclear power plants, this requirement is definedin [114]. Computer code calculation errors are to be assessed in a code verification andvalidation report, the requirements to which are given in [115]. However, [115] does not
explain how exactly the code-calculated parameter errors should be evaluated; nor does itsuggest how the error values obtained as a result of code validation should be used in asafety analysis of a nuclear facility (NF).
Code calculation error implies code output deviation from the measurement data obtainedin the course of code validation experiments. Considering that results of any measurements
feature uncertainty, one of the code calculation error components is the uncertainty ofexperimental measurements. Meanwhile, [116] requires obligatory justification of thesufficiency of experimental data used to validate a code, in addition to evaluation of
measurement uncertainty. This aspect is discussed in detail in [117].
Another code calculation error component is uncertainties caused by the assumptions andsimplifications adopted to build the code calculation model, including the uncertainties
associated with model nodalisation (control volumes, finite elements, etc.), selection ofnumerical solution technique and integration step for the code’s set of equations, and theadverse effect of an unskilled code user. According to [116], analysis and minimisation ofsuch uncertainties is a prerequisite to demonstrate code model correctness. In fact, thedetailed discussion covers only the following error components of code output: uncertainty
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  81

of code calculation model parameters that have a statistical nature and uncertainty ofexperimental measurements. Code calculation uncertainty caused by the uncertainty ofcode calculation model parameters is defined as code output variation due to uncertainty ofmodel parameters having a statistical nature (e.g. physical and chemical properties ofmaterials, geometry, empirical equation coefficients underlying the code calculation
model, etc.). This uncertainty is presented as an interval (range) with certain probabilistic
characteristics.
Key stages in the assessment of the influence of code model parameter uncertainty on code
output are shown in Figure 4.12. For Stage 1 the PIRT methodology is utilised.
To obtain a sound value for code calculation error, code validation should use only thoseexperimental data whose measurement uncertainty has been evaluated. The estimation ofcode calculation error during code validation consists first of estimation of error of steadystate calculations, and then followed by estimation of the error of transient calculations.
During the code validation process, it is reasonable to use code output deviation from themeasurement data as parameter calculation error, taking into account uncertainty of bothcode model parameters and the measurement data. These uncertainties are presented asintervals with specified probabilistic characteristics. Code calculations performed in theframework of a safety analysis of a nuclear facility may adopt only those error values thathave been obtained in the experiments conducted at real nuclear facilities and integral
experimental facilities, considering scaling factors. The number of such experiments isquite limited; therefore, in the safety analysis, the demanded conservatism of code output
will be demonstrated with the help of one of the model parameter uncertainty evaluation
methods. Of all numerous methods intended for evaluation of code model parameter
uncertainty, the one that has found widest application in safety analysis of nuclear facilities
is the approach based on the Wilks’ formula. In order for uncertainty analysis (UA) to be
accepted by the Russian regulatory authority (Rostechnadzor) the following is needed:verified and validated computer codes, experimental data used for validation should beproperly assessed (measurement uncertainties, scaling applicability), safety objectives andUA methods should be well-defined and expert judgement should be minimised or at least
formalised.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

82  NEA/NSC/R(2019)7
Figure 4.11. Process for evaluation of code output error

Source: SEC NRS, 2019.

Figure 4.12. Key stages in assessment of model parameters uncertainty
Stage 1. Selection of code model parameters
significantly affecting the code output,
and their statistical characterisation

Stage 2. Analysis of code calculation sensitivity tovariance of code model parameters
2.1 Composing sensitivity matrix
2.2 Discarding model parameters to which code
output has low sensitivity

Stage 3. Analysis of effect of code model parameter
uncertainty on code output
3.1 Setting various values to model parameters, andtesting their mutual independence
3.2 Conducting variant calculations using the code,and statistical processing of their results

Source: SEC NRS, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  83

IBRAE RAN Methodology
In the IBRAE RAN, the methodology for uncertainty-based best-estimate computer codes
validation has been developed and implemented in the VARIA code [118,119]. It employs
an approach, similar to one used in GRS method.
VARIA (from “variation”) is a cross-platform application for mass computing that is used
for uncertainty and sensitivity assessment in conjunction with best-estimate codes formodelling various aspects of nuclear power plant accident scenarios. It performs variationof input data for BE codes with desired parameter values distributions, controls executionof code instances on wide range of systems – from personal computers to large clusters
consisting of multiple nodes and makes statistical analysis of the results. Due to themodular structure, it can be used with a variety of BE codes – adaptation for a new code
requires just creation of a simple parser for input decks of this code. Meanwhile, the alreadyexisting parsers are sufficient for the majority of codes. Additional modules can be attached
for advanced statistical analysis, visualisation of the arrays of results, etc.
Currently the VARIA functionality besides the basic statistical treatment of the results
includes construction of uncertainty bands as well as calculation of sensitivity measuresthat include normalised multidimensional linear regression coefficients, Pearson,
Spearman and Kendall correlation coefficients. These measures can be obtained forarbitrary time moments of the transients thus allowing observation of their temporal
evolution. The results of the sensitivity study can be presented graphically both as bar
charts and as scatter plots showing the dependence of the simulation results on given varied
parameter with its linear approximation.
The IBRAE RAN validation methodology aims to estimate the uncertainty of the system
response resulting from the (lack of) accuracy of the parameters of the system of interest
and of the models implemented in the code. The representation uncertainties are minimisedby the space and time step convergence testing. The uncertainties resulting from model
scaling and from the user qualification are not accounted for. The validation procedure isoutlined in Figure 4.12. It includes following main steps.


The code input parameters that are supposed to influence the system response mostare defined with their variation ranges and distributions. This is done by the experts;
afterwards the sensitivity assessment will show their actual influence.

Time and space convergence is tested to minimise the representation uncertainties.



Multivariate simulations are performed. The number of code instances for
uncertainty assessment is defined from Wilks formula [74] according to desired
tolerance interval. Generally, it is 93, which corresponds to 0.95/0.95 probability
for two-sided interval. For sensitivity study, the number on code instances N
depends on the sensitivity measure taken. E.g. for multidimensional linear
regression coefficients, it depends on the number of varied parameters n: N >> n2.
Uncertainty range of the response(s) of interest is found along with the mean value
and the sensitivity coefficients.



Uncertainty of the reference (experimental) data is estimated.

Comparison of the multivariate simulations results with reference data is
performed. Various measures can be used depending on the nature of the data
(single point, several points, temporal dependence). These measures can include
the uncertainty of the reference data and of the multivariate simulations results.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

84  NEA/NSC/R(2019)7
The overall code uncertainty for the validated models is obtained by combination of theuncertainties obtained for simulation of particular experiments.
This methodology is primarily aimed on the aleatory uncertainties. The question of
applicability of such methods to the epistemic uncertainties is still open due to thesubjectivity of its quantification, since it requires expert opinion [3,59]. However, themethodology authors argue that if sufficiently reliable estimates for epistemic uncertainties
can be obtained, they can also be accounted for.
Figure 4.13. Validation procedure according to IBRAE RAN methodology

Source: IBRAE RAN, 2019.

4.5 Traditional single physics uncertainty quantification (UQ)
Concepts
Phenomena Identification and Ranking Table (PIRT)The development and utilisation of PIRT is based on the following approaches:

use of sensitivity and uncertainty analyses;

use of expert opinion;

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  85use of non-dimensional parameters: benefits, cautions and consequences.

Calibration
The model form uncertainty is one of the most significant sources of uncertainties and thecalibration step is used to improve models and to decrease model uncertainty. It is
important to have a segregation of calibration data from validation data.

Experimental data uncertainties
The evaluation of experimental uncertainties and propagation of errors from “measured”
data is a very important procedure. The evaluation of experimental uncertainty is performedin two ways:

direct (e.g. measured flow rate) versus indirect (imaging of bubble formation in
turbulent flow) experimental observations;

propagation of error from “measured” data that is inferred from experimental
observations from instruments (e.g. uncertainties in electronic signals fromthermocouples or radiation detectors into “measured” quantities of interest).

Transposition and extrapolation
The transposition (interpolation within the validation domain) and extrapolation beyond
validation domain using uncertainty analysis require:estimating the uncertainty in model input data for the application of interest;estimating model form uncertainty at the application of interest;identifying and distinguishing aleatory and epistemic uncertainties.

Propagation of uncertainties in single physics calculations
Various international benchmarks on BEPU applications to safety analysis have beenorganised and conducted. In this section the main outcomes from two NEA projects are
summarised.

The BEMUSE benchmark
The BEMUSE (Best-Estimate Methods – Uncertainty and Sensitivity Evaluation)
Programme – promoted by the Working Group on Accident Management and Analysis(WGAMA) and endorsed by the NEA Committee on the Safety of Nuclear Installations(CSNI) – represented an important step towards reliable application of high-quality bestestimate and uncertainty and sensitivity evaluation methods [120-123].
The programme was divided into two main steps (each one consisting of three phases)
devoted to the uncertainty and sensitivity analysis related to an LBLOCA in the LOFT
facility (L2-5 test) and in the ZION Nuclear Power Plant respectively. The programme
started in January 2004 and ended in 2010.
The operational objectives included an assessment of the applicability of best-estimate anduncertainty and sensitivity methods to integral tests and their use in reactor applications.
Both classes of uncertainty methods, were considered and investigated. Before
summarising the lessons learnt, it might be useful to compare some of the final results of
the BEMUSE Project Phase III with a previous (actually the first) international benchmark
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

86  NEA/NSC/R(2019)7
on the uncertainty methods, the NEA Uncertainty Method Study (UMS, [124]), which
focused on a small break LOCA experiment (LSTF SB-CL-18 5%) in the Japanese Large
Scale Test Facility (LSTF). Figure 4.14 provides an idea about the magnitude of thediscrepancy among the results predicted by different participants to the NEA UMS and
BEMUSE ([124] and [122]) benchmark respectively, adopting very similar uncertainty
methods (mostly of them applied methods based on propagation of input uncertainty
parameters) and in some cases using also the same thermal-hydraulic code. It can be notedthat the spread among the size of the uncertainty bands (for cladding temperatures in
Figure 4.14) predicted by the UMS participants in 1995 was not reduced by the participants
to the BEMUSE benchmark which was held more than 10 years later. The reasons that
explain the differences between the results of users of the same code and (almost) same
uncertainty method are several and coincide with some of the main outcomes from
BEMUSE project [123]:differences between applications of statistical methods may mainly be due toprocedures adopted:– to select the input parameters;– to properly estimate input parameter uncertainties;– to properly estimate the probability distribution functions.



Some participants using statistical methods specified few important parameters or
too narrow uncertainty ranges for important input uncertainties based on expert
judgement and not on sufficient code validation experience. Therefore, skill,
experience and knowledge of the users about the applied suitable computer code as
well as the used uncertainty method are important for the quality of the results.



An increased number of calculations for the statistical methods may be advisable
because it decreases the dispersion of the tolerance limits.

Differences between the methods based upon propagation of input uncertainties
and methods based upon propagation of output errors may come from different
experimental data bases used for the analysis.
Figure 4.14. Spread of uncertainty bands for cladding temperatures
predicted by UMS and BEMUSE participants
CLADDING TEMPERATURE

500
UMS - AEAT
UMS -CEA
UMS -ENUSA
UMS - GRS
UMS - UMAE-R
UMS - UMAE-C

400
350

700
CEA
GRS
IRSN
KAERI
KINS
NRI-1
NRI-2
PSI
UNIPI
UPC

600

500

300

Temperature (K)Uncertainty Bands Wide [K]

450

250
200
150

400

300

200

100
100

50
0

0

0.0

200.0

400.0

600.0

Time (s)UMS Participants (1995)

800.0

1000.0

0

10

20

30

40

50

60

70

80

90

Time (s)BEMUSE Participants (2006)

Sources: NEA, 2007; Wickett et al., 1998.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2100

NEA/NSC/R(2019)7  87

The PREMIUM Benchmark
In system thermal-hydraulics only few code input parameter uncertainties are obtained
from experimental observations (mainly from separate effect test facilities), whereas forthe major part of them engineering judgement is adopted for deriving (“firstˮ) guess valuesof ranges and PDF. This fact constitutes the source of a new user influence on the BEPU
results (in addition to the code-user-effect) that can be named “uncertainty-method-user
effect” and which has been found to be one of the lessons learnt from the BEMUSE project(see Section above and [115]). In this context, a new NEA benchmark called PREMIUM(Post-BEMUSE Re-flood Model Input Uncertainty Methods [125]) was launched in 2011
and completed in 2015.
Its main goal was the development, study, comparison and application of methods forquantification of the uncertainty of the physical models contained in thermal-hydraulicsystem codes used in nuclear safety. The application of these methods had the final goal toreduce the dependency on expert judgement in the input uncertainty quantification.
The benchmark application was focused on the physical models involved in the predictionof the core reflood, which is a fundamental stage in the LOCA scenario.
The PREMIUM benchmark has been a valuable exercise on methods for uncertainty
quantification of physical computational models and their application to the models
involved in the reflooding prediction. Different quantification methods and thermalhydraulic codes have been used in the benchmark. Results have been found to be very
dependent on the quantification method, rather than on the code. Furthermore, the resultsof the quantification process have shown a strong dependence on topics such as:the selected responses used in the quantification;the selected parameters to be quantified;

the selected experimental database for the quantification;the code modelling and the numerical implementation;the quantified models, which, in general depend on the thermal-hydraulic code
being used.

It was concluded that the quantification methods used in PREMIUM showed a strong user
effect and the results of the quantified uncertainties had a large variability and discrepancy
among the participants.
As a final outcome, the benchmark has revealed the necessity of further work on thequantification methods and on the development of “best practice” guidelines for theevaluation of model input uncertainty parameters. This is actually the goal of a newlaunched NEA project named SAPIUM (Systematic APproach for Input Uncertainty
quantification Methodology), which started in 2017.

4.6 Traditional multi-physics uncertainty quantification (UQ)
Propagation of uncertainties in multi-physics calculations
The LWR Uncertainty Analysis in Modelling (UAM) benchmark
In recent years there has been an increasing demand from nuclear research, industry, safety
and regulation for best-estimate predictions to be provided with their confidence bounds.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

88  NEA/NSC/R(2019)7
In addition to LWR best-estimate calculations for design and safety analysis, the different
aspects of Uncertainty Analysis in Modelling (UAM) are to be further developed andvalidated on scientific grounds in support of its performance. There is a need for efficientand powerful analysis methods suitable for such complex coupled multi-physics and multiscale simulations. The benchmark sequence addresses this need by integrating the expertisein reactor physics, thermal-hydraulics and reactor system modelling as well as uncertainty
and sensitivity analysis and contributes to the development and assessment ofadvanced/optimised uncertainty methods for use in best-estimate reactor simulations.
In the comprehensive international LWR UAM benchmark activity different uncertainty
analysis (UA) methods for coupled codes are being compared and their value assessed
including the validation of the methodologies for uncertainty propagation. For the first time
the uncertainty propagation is estimated through the whole simulation process on a unified
benchmark framework to provide credible coupled code predictions with defensible
uncertainty estimations of safety margins at the full core/system level. The benchmark
allows not only comparing and assessing the current UA methods on representative
applications but also stimulating the further development of efficient and powerful UA
methods suitable for complex coupled code simulations and helps to formulate
recommendations and guidelines on how to utilise advanced and optimised sensitivity
analysis and UA methods in “best-estimate” coupled reactor simulations in licensing
practices.
The approach described above is based on the introduction of nine steps (Exercises), whichallows for development of a benchmark framework which mixes information from theavailable integral facility and nuclear power plant experimental data with analytical andnumerical benchmarking. Such an approach compares and assesses current and new
uncertainty methods on representative applications and simultaneously benefits fromdifferent methodologies to arrive at recommendations and guidelines. These nine steps(Exercises) are carried out in three phases as follows:
Phase I (Neutronics Phase):

Exercise 1 (I-1): “Cell Physics” focused on the derivation of the multi-group
microscopic cross-section libraries;

Exercise 2 (I-2): “Lattice Physics” focused on the derivation of the few-group
macroscopic cross-section libraries;

Exercise 3 (I-3): “Core Physics” focused on the core steady-state stand-alone
neutronics calculations.

Phase II (Core Phase):

Exercise II-1: Fuel thermal properties relevant for transient performance;

Exercise II-2: Neutron kinetics stand-alone performance (kinetics data, space-timedependence treatment, etc.);

Exercise II-3: Thermal-hydraulic fuel bundle performance.

Phase III (System Phase):

Exercise III-1: Coupled neutronics/thermal-hydraulics core performance (coupled
steady state, coupled depletion and coupled core transient with boundary
conditions);

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  89

Exercise III-2: Thermal-hydraulics system performance;

Exercise III-3: Coupled neutronics kinetics thermal-hydraulic core/thermalhydraulic system performance.

Separate specifications are prepared for each phase in order to allow participation in thefull phase or only in a subset of the exercises [45]. Boundary conditions and necessary input
information are provided by the benchmark team. The intention is to follow the calculation
scheme for coupled calculations for LWR design and safety analysis established in the
nuclear power generation industry and regulation.
One of the observations from analysis of the submitted results for Phase 1 [126] is theimpact of spectral changes to the magnitude of nuclear data propagated uncertainties inneutronics integral parameters and local distributions. There are two major contributors tospectral changes – material compositions and temperatures. The latest impact is shown in
Figure 4.15 for different LWR numerical test cases and the Kritz experimental test cases,
where cold means hot zero power and hot means hot full power state with corresponding
moderator/coolant and fuel temperatures.
Figure 4.15. Temperature effect on propagation of nuclear data
uncertainties to multiplication factor uncertainty

Source: Bratton et al, 2014.

The expected impact and benefits of the LWR UAM benchmark activity for LWR safety
and licensing are summarised in [45]. This benchmark project is challenging and responds
to needs for estimation of confidence bounds for results from simulations and analysis inreal applications. Among the expected results of this project are:

systematic identification of uncertainty sources;systematic consideration of uncertainty and sensitivity methods in all steps; thisapproach will generate a new level of accuracy and will improve transparency ofcomplex dependencies;all results will be represented by reference results and variances and suitable
tolerance limits;the dominant parameters will be identified for all physical processes;

support of the quantification of safety margins;

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

90  NEA/NSC/R(2019)7


the experiences of validation will be explicitly and quantitatively documented;

recommendations and guidelines for the application of the new methodologies willbe established.

The LWR UAM activity will establish an internationally accepted benchmark framework
to compare, assess and further develop different uncertainty analysis methods associatedwith the design, operation and safety of LWRs. As a result, the LWR UAM benchmark
will help to address current nuclear power generation industry and regulation needs andissues related to practical implementation of risk-informed regulation. The realistic
evaluation of consequences must be made with best-estimate coupled codes, but to bemeaningful, such results should be supplemented by an uncertainty analysis. The use of
coupled codes allows avoidance of unnecessary penalties due to incoherent approximationsin the traditional decoupled calculations and to obtain more accurate evaluation of margins
regarding licensing limit. This becomes important for licensing power upgrades, improved
fuel assembly and control rod designs, higher burn-up and others issues related to operating
LWRs as well as to the new Generation III + designs being licensed now (ESBWR, AP1000, EPR-1600 and etc.). As such, establishment of this internationally accepted LWR
UAM benchmark framework offers the possibility to accelerate the licensing process whenusing best-estimate methods and contributes to establishing a unified framework to
estimate safety margins, which would provide more realistic, complete and logicalmeasures of reactor safety.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  91

5. Summary of validation and uncertainty quantification state of practice

5.1 Summary of practices for validation and uncertainty quantification for single
physics codes
System analysis codes
A key feature of the activities performed in nuclear reactor safety technology is thenecessity to demonstrate the qualification level of each tool adopted within an assigned
process and of each step of the concerned process [127-129]. Therefore, the qualificationof best-estimate codes, models, “best modelling practices” and uncertainty methods must
be considered of great importance in order to ensure the validity of performed BEPU
system analysis. A consistent code assessment supported by a qualified experimental
database is an important step for developing a solid basis for the uncertainty evaluation in
the frame of a Best Estimate Plus Uncertainty (BEPU) approach. An example is The
International Experimental Thermal HYdraulics Systems database, TIETHYS (www.oecdnea.org/tiethysweb/).
The experiments available through the Nuclear Energy Agency (NEA) Committee on the
Safety of Nuclear Installations (CSNI) Integral Test Facility (ITF) [123] and Separate
Effect Test Facility (SETF) [124,125] matrix constitute a large experimental database forthe nuclear thermal-hydraulic community. These databases collect over thirty years ofexperiments: separate effects test for individual phenomena, integral tests for large breakloss-of-coolant accidents (LBLOCAs), small break LOCA, transients, beyond design basis
accidents and accident management in pressurised water reactors (PWRs), boiling water
reactors (BWRs) and water-water energetic reactor (VVER) types of reactor. The enormous
amount of information has been used for the code assessment in the framework ofverification and validation (V&V) activities. The availability of the experimental database
constitutes also the prerequisite for the creation of a qualified accuracy database of system
thermal-hydraulic responses to be used for the uncertainty evaluation in the methods basedon “extrapolation of code output accuracies” but can contribute also to the V&V process
of methods based on “propagation of input uncertainties”.
An example of practice of validation and uncertainty quantification (VUQ) for System
Analysis codes is the Standardised and Consolidated Calculated and Reference
Experimental Database (SCCRED) [130] methodology which has been developed togenerate a series of documents and tools to set up a qualified experimental and calculated
database for V&V purposes of BEPU applications, i.e. best-estimate computational codes
and associated uncertainty methodologies. Figure 5.1 depicts the SCCRED diagram: theinformation contained in the experimental reports together with the code input
nodalisations are the elements to be elaborated in a systematic way by a qualified databasemade up of the following documents:the reference data set for the selected test of the facility;

the qualification report;the engineering handbook.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

92  NEA/NSC/R(2019)7
The methodology has been developed in such a way as to allow for the collection,organisation, use and preservation of an exhaustive set of geometrical data, experimental
results and qualified code calculation responses, ensuring finally (see Figure 5.1):
a. an exhaustive consolidated information set;b.

the traceability of the information, including the decisions taken during theactivity;c. the standardisation of the format by which the information is collected;
d. the application of a systematic approach for qualifying code calculations andassociated nodalisation techniques;e. the availability of a database of experimental and associated qualified calculated
results (TIETHYS);
f. the availability of a database of accuracy (deriving from the differences between
experiment and calculation results) to be used for:– V&V of the codes and associated nodalisation techniques; and– V&V of the uncertainty methods either based on the quantification of input
uncertainty parameters or output responses.
More details about each step of the SCCRED methodology are provided in [130] andhereafter the focus is on the validation process of code calculations (see item d above).
The qualification process of a thermal-hydraulic system code calculation aims todemonstrate that the code results (obtained by the application of the code with thedeveloped nodalisation) are a realistic approximation of the reference plant behaviour (afull-size nuclear power plant or a facility).
A major issue in the use of a mathematical model is the model’s capability to reproduce
the plant or facility behaviour under steady-state and transient conditions. These aspects
are the two main verifications that must be passed in the qualification process. The first ofthem is related to the realisation of a nodalisation-schematisation of the reference plant or
facility; the second one is related to the capability to reproduce and analyse the transient
behaviour to derive the necessary information.
Figure 5.1. SCCRED flow chartA

Main Flow Direction
Corrective Actions
Support Documents

G

Y, z

Model
Qualification
ReportF

X

C

Reference
Data Set

Nodalization
techniques

Calculation
notes

Establishment ofcorrective actions

Drawings,
data,reports

B
E

X

X

Test Report

BIC
D

Code Input
Deck

y
H

y

X, y

Engineering
Handbook
X<Y<Z: indicate three levels of analyst groups

Source: Petruzzi and D’Auria, 2016

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  93

The qualification of the nodalisation is a mandatory process to take into account the effectof many different sources of approximations:the data of the reference plant or facility available to the code user are typically
non-exhaustive to reproduce a perfect nodalisation-schematisation of the reference
plant;

the code-user derives, from the available data, an approximated nodalisationschematisation of the plant or facility reducing the level of detail of the simulated
hardware;

the code capability to reproduce the hardware, the plant systems and the actuation
logic of the systems further reduce the level of detail of the nodalisationschematisation.

The needs for qualifying the code capability and the nodalisation features to perform thetransient analysis derive from the following statements:the code options must be adequate;the nodalisation-schematisation solutions must be adequate;the simulation of some systems can be tested only under transient conditions(e.g. emergency core cooling system [ECCS]) that are not involved in the normal
plant operation, i.e. during steady-state code condition;the capability of the “code-nodalisation” of the system to reproduce relevant TH
phenomena expected in the transient must be tested.

The qualification procedure has been developed including the necessary checks for thedifferent aspects mentioned above and the criteria adopted to produce a judgement about
the acceptability of the code analysis results.
The goal of the qualification procedure is to develop and obtain a qualified nodalisationconsidering the comparison with the hardware data, BIC and the time trends of relevant
quantities. The procedure distinguishes between three main processes:


the “geometrical fidelity”;

the “steady state”;

the “transient” level of qualification.

A scheme of the procedure can be seen in Figure 5.2 where the processes are represented
by the steps “f” and “j” which constitute the checking points of the procedure. If thechecking points are not fulfilled (paths “g” and “k”) the nodalisation must be improved(step “c”). It must be pointed out that many cycles “c-d-e-f-g” could be necessary to gain
the access to the step “h”. Similarly every time the path ”k” is activated the process goesagain from “c” to “f” and then from “h” up to “j” (the path “c-d-e-f-h-i-j-k-c” could be
actuated more than one time). The last step “l” is reached at the end of the procedure andthis implies the qualification of the developed nodalisation. The above qualification process
has been applied in several international benchmarks, including the NEA Best-estimate
methods – Uncertainty and sensitivity evaluation (BEMUSE) [131].

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

94  NEA/NSC/R(2019)7
Figure 5.2. Flow chart of the qualification procedure

Source: Petruzzi and D’Auria, 2016

Once the qualification procedure has been successfully passed, the quantification of theaccuracy can be performed for the selected responses and phenomena and a database of
accuracy values is then derived. The availability of such a qualified database constitutes
the final outcome of the SCCRED steps and can be used:

for the V&V demonstration of the codes and associated nodalisation techniques;

for the derivation of the uncertainty values of the responses of interest following
the methods based on propagation of accuracies;as a basis for deriving statements about the PDF and the ranges of variation of the
input uncertainty parameters for the methods based on propagation of input
uncertainties;

for the derivation of a database of (posterior) improved and reduced estimations of
the input parameters and related covariance matrices following the methods basedon the predictive modelling approach.

Neutronics (reactor physics) codes
A summary of practices for VUQ for neutronics codes is given by the example of the work
performed at Idaho National Laboratory (INL), which is focused on Transient Reactor Test
Facility (TREAT) analysis and is summarised below:

INL has employed the traditional Monte Carlo method to TREAT models [132] tocompute the 95th percentile with a confidence level of 95%, sensitivities, output
correlations and high-order statistical moments based on Wilks’ formula.
Regarding this method, both the nuclear data (e.g. multi-group cross-sections) andthe system parameters (e.g. boron concentration, initial fuel temperature) can be
perturbed based on given probability distributions. The TREAT models are thenexecuted at each perturbed state and the output variations are recorded.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  95

INL has also employed some advanced methods to efficiently perform the SA/UQ
for TREAT transient calculations, i.e. the stochastic colocations combined withgeneralised polynomial chaos (gPC) expansions and the high dimensional model
reduction method combined with sparse grid and gPC expansions. The responses
are represented via gPC expansions, while the gPC coefficients are determined via
the Smolyak-like sparse grid quadrature. These methods will converge very fastwith few model evaluations for a moderate number of input parameters (i.e. < 15).
These methods are already applied to TREAT transient test #15 [132].



INL has also developed an active subspace method to handle hundreds of input
parameters. The input parameter space is reduced via principal component analysis
and input-output correlation analysis. Once the input space is reduced, the methods
mentioned above can be used to perform the UQ in the reduced space. This method
has been applied for IAEA-2D PWR benchmark problem [133].

INL has also developed an adjoint method inside the Rattlesnake transport code
enabling use of generalised perturbation theory to perform SA/UQ. This method isvery efficient if the number of inputs is very large and the number of responses isrelatively small. For each given response, one needs to calculate the generalised
adjoint solution that can be used to compute the response sensitivities with respect
to all input parameters. Once the sensitivities are known, the uncertainties of inputs
can be easily propagated via the “sandwich” equations. This method has alreadybeen applied to models of TREAT [134]. Similar capabilities are available in the
NEWT code within the SCALE system.

Fuel performance codes
Over the past several years, an effort has been made for the evaluation of uncertainties
associated with modelling and prediction of fuel behaviour using the best-estimate plus
uncertainty approach [18-20]. There are several sources of uncertainty in fuel performance
analysis. The rod’s manufacturing parameters, system boundary conditions andexperimentally determined material properties are never accurate but introduce various
amounts of uncertainty into the system. These uncertainties will be propagated to model
outputs such as the fuel centreline temperature, internal pressure and gap conductance. The
primary parameter of interest is the fuel temperature, because most of the physicalphenomena occurring in nuclear fuel during its irradiation in reactor are driven bytemperature. In the simulation process, the response of the models describing these
phenomena is mainly conditioned by the confidence in the calculated temperature of thefuel.
In best-estimate analysis quantifying both the magnitude and the source of the output
uncertainties is necessary. The latter is known as the sensitivity analysis, where moststudies are focused on the first-order effect of the input parameters on the output. A
common approach is to evaluate, for example, the Spearman correlation coefficients or the
first order Sobol’ indices from the model output [18], although some studies have shownthat the higher-order interactions between the input variables should not be completely
neglected.
In the NEA Benchmark for Uncertainty Analysis in Best-Estimate Modelling (UAM) for
LWRs Exercise II-1 [135] was developed to identify and propagate input uncertainties
through fuel performance codes for PWR, BWR and VVER fuel designs. The example ofthe boundary condition uncertainties in terms of upper and lower bounds, as well as thetype of distribution are provided in Table 5.1.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

96  NEA/NSC/R(2019)7
Tables 5.2 and 5.3 show the uncertainty of the boundary condition and the material
properties, respectively that are currently used in the benchmark.
Table 5.1. Exercise II-1 core boundary condition variations
Parameter
System pressure (%)
Coolant flow rate (%)
System power (%)
Inlet fluid temperature (K)

BWR
± 1.0
± 5.0
± 5.0
± 3.0

PWR
± 1.0
± 5.0
± 5.0
± 3.0

VVER
± 2.0
± 5.0
± 5.0
± 3.0

Distribution
NormalNormalNormalUniform

Source: Hou et al., 2016

Table 5.2. Core boundary condition standard deviations
Parameter

BWR
0.00333
0.01667
0.01667

System pressure
Coolant flow rate
System power

Standard Deviations
PWR
VVER
0.00333
0.00667
0.01667
0.01667
0.01667
0.01667

Source: Hou et al., 2016

Table 5.3. Material properties recommended uncertainties
Parameter
Fuel thermal conductivity
Fuel thermal expansion
Cladding thermal conductivity
Cladding thermal expansion

BWR± 0.5 W/m-K± 15%± 5 W/m-K
± 30%

PWR
± 0.5 W/m-K± 15%± 5 W/m-K
± 30%

VVER
± 0.5 W/m-K± 15%± 5 W/m-K± 30%

Source: Ikonen and Tulkki, 2014; Hou et al., 2016.

A list of output parameters calculated by fuel performance codes have been identified in
the benchmark, as they are important to propagation to other types of codes. For example,
the gap conductance is often specified in the input for the simplified fuel rod models inthermal-hydraulics codes, therefore it is vital to understand how it varies according to theprediction of fuel performance analyses. The uncertainty parameters to be propagated fromthe numerical cases are:


gap conductance (W/m2/K);



fuel thermal conductivity (W/m/K);

radial rod dimensions (rod outer diameter, rod elongation);

gap width (µm);



for transient cases, the heat transfer coefficient between cladding and coolant is alsoa requested propagated output (W/m2/K).

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  97

As an example of uncertainty propagation in fuel performance calculations, a comparison
between participants’ results for the PWR depletion test case of Exercise II-1 for the LWR
UAM benchmark [141] is shown in Figure 5.3:
Figure 5.3. PWR depletion test case

Source: Avramova and Ivanov, 2017.

Core thermal-hydraulics codes
The validation activities for core thermal-hydraulics codes are usually combined with astatistical uncertainty analysis framework. The input parameters include boundary
conditions, geometry and modelling uncertainties, which are selected using a sensitivity
study and then defined based on expert judgement. As an example, the methodology
developed for the core thermal-hydraulic simulator CTF coupled via scripts with thestatistical analysis tool, Design Analysis Kit for Optimisation and Tera-scale Applications(DAKOTA) is discussed in [136]. A forward uncertainty quantification method with Latin
Hypercube sampling is used, where the sample size is based on available computational
resources. The means and standard deviations of thermal-hydraulic quantities of interest
are reported, as well as the Spearman rank correlation coefficients between the inputs andoutputs. This allows the same interface to be used for many different statistical studies andenables automatic parallelisation of analyses. External Python scripts are used to prepare
the CTF input deck with input parameters determined by DAKOTA and to extract
quantities of interest from the CTF output. This CTF-DAKOTA interface is also used forthe sensitivity studies [137]. The means and standard deviations are accompanied by theirrespective standard errors and the correlation coefficients are tested for statisticalsignificance. The new element in this methodology is that given a specified amount ofcomputational resources, it can be used to quantify statistical significance using
fundamental statistical analyses. This is in contrast with the prevailing methods in nuclear
engineering, which provide a sample size necessary to achieve a specified level of statisticalcertainty [139].
As mentioned above the LWR UAM benchmark is an ongoing project that is creating
standard nuclear engineering problems for the application of uncertainty
methods [135]. Although the goal of the UAM benchmark is to address uncertainties in
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

98  NEA/NSC/R(2019)7
multi-physics and multi-scale simulations, its two first phases are composed of single
physics problems. Phase II, Exercise 3 of the UAM benchmark provides 12 test cases for
uncertainty quantification of core thermal-hydraulic codes for nuclear engineering
applications. These cases are numerical and experimental cases for steady and transient
applications based on representative PWR, BWR and VVER designs.
An extensive set of preliminary sensitivity studies were used to eliminate possible input
parameters that have minimal influence on the quantities of interest. First, each potential
input was varied over a small perturbation about its nominal value; the maximum change
in each quantity of interest is a measure of sensitivity. These so-called “parameter studiesˮ
quantified the univariate effect of each input uncertainty on the outputs and gave one
indication of whether an input should be included in future analyses. The second sensitivity
study method used was Morris Screening, which is a valuable tool to provide information
about input interactions with relatively little computational expense [138]. Essentially,
Morris Screening creates a randomised set of finite differences throughout the parameter
space. These finite differences, called “elementary effectsˮ, were used to quantify both theunivariate and multivariate effects between the inputs and outputs.
Both parameter studies and Morris Screening were completed for all four LWR UAM test
cases for a large set of potential input parameters. The results are prohibitively large andare therefore excluded from this report; an overview can be found in [136]. Thresholds
were set for the parameter study and Morris Screening results and those input parametersthat did not exceed the threshold for at least one quantity of interest were excluded fromthe study. Based on the sensitivity studies, 21 input uncertainties were selected. Each input
uncertainty is either normal or uniform. Both normal and uniform distributions arecharacterised by their bounds. Uniform distribution bounds correspond to their minimum
and maximum value. The normal distributions are bounded to avoid unphysical input
parameter ranges; the bounds correspond to a perturbation of three standard deviations.
The validation metric is important for experimental test cases where computational results
are compared to experimental data. Two validation metrics are chosen, the first is a simple
difference between the mean code result and the reported experimental value. This metricmaintains the units of the original measurement and is therefore very intuitive, but it cannot
give any indication of similarity between two distributions. To compare code results withuncertainty to experimental results with measurement error, it is necessary to use avalidation metric which can compare two distributions. A large variety of such metrics areavailable and one of these metrics is the Kolmogorov/Smirnov (KS) Test. The KS test
measures the maximum vertical distance between the two cumulative density functions(CDFs) through dKS. The two CDFs represent the code and experimental distributions
when the KS test is used for validation.
Results are presented for the BFBT-based experimental steady-state test case (seeFigure 5.4) from Exercise II-3 of the LWR UAM benchmark. The calculated results asrequested by the benchmark include: (1) three global quantities of interest (maximum void
fraction, maximum cladding surface temperature and pressure drop), (2) axial distributionsof temperature and void fraction and (3) an exit void fraction distribution. The axial
distribution of void fraction in channel 25, the cladding surface temperature distribution ofrod 22 and the exit distribution of void fraction are shown in Figure 5.6. The dips in theexit void distribution for this case are due to the influence of the unheated water rod at thecentre of the assembly. The exit void fractions given in Figure 5.6(c) are compared to the
BFBT experimental results. The difference between the experiment and the mean code
results are shown in Figure 5.5(a), where darker colour channels indicate that the CTF

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  99

prediction is further from the experiment. The BFBT measurements are assumed to have anormal distribution with a mean of the experimental measurement and two standard
deviations equal to 3% void. This value is the measurement error of the X-ray CT scannerused in the BFBT Facility. The code distribution is compared to the experimental
distribution using the KS test and the results are shown in Figure 5.5(b). Lighter colours
indicate that the two distributions are closer together. Figure 5.6 presents quantified
validation results for the BFBT case. To draw further conclusions, this same data ispresented in Figure 5.7 with a distinction made between four different types of subchannel.
Corner subchannels are channels that contact the assembly wall on two sides, sidesubchannels contact the wall on one side, and unheated channels are those that are in directcontact with the centre water rod on one side. The rest of the channels are considered centre
subchannels. A colour-coded map is included in the figure. The absolute difference
between the prediction and measurement (x-axis in Figure 5.7) only indicates how closethe means of the two distributions are, whereas the KS metric (y-axis) gives an indicationof closeness for both the mean and the data spread. The code and experimental standard
deviations are of the same magnitude and vary only slightly, therefore the two metrics areclosely related. Figure 5.7 indicates that the unheated rods are less accurately modelled in
CTF compared to the other subchannels. This is likely because the CTF model used does
not explicitly account for the centre water region, so any thermal inertia of the water rod iscompletely neglected.
Figure 5.4. Exercise II-3 – BFBT experimental test case

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

100  NEA/NSC/R(2019)7
Figure 5.5. Quantified validation of exit void fraction distribution for the BFBT tests case

Source: NCSU, 2019.

Figure 5.6. Results for the BFBT test case

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  101

Figure 5.7. Relationship between the absolute difference (between experimental value
and code results) and dKS for different subchannel types

Source: NCSU, 2019.

5.2 Summary of practices for validation and uncertainty quantification for multiphysics codes
The qualification procedure of traditional coupled multi-physics code systems is based on
the qualification framework (V&V) of separate single physics models/codes and includes
in addition V&V of the coupling methodologies of the different physics models. The
extended V&V procedure involves testing the functionality, the data exchange between
different physics models and their coupling, which is designed to model combined effects
determined by the interaction of models. The extended validation procedure compares thepredictions from coupled multi-physics code systems to available measured data andreference results. It is important to emphasise that such validation should be based on amulti-level approach like the one utilised in validating coupled neutronics/thermalhydraulics codes in international standard problems. Appropriate benchmarks have beendeveloped in international co-operation led by the NEA that permits testing theneutronics/thermal-hydraulics coupling and verifying the capability of the coupled(traditional multi-physics) codes to analyse complex transients with coupled core/plantinteractions. These benchmarks provide a validation basis for the current generation of
coupled best-estimate codes.
The current tendencies in coupled code developments are towards systematic integrationof uncertainty and sensitivity analysis with simulations for safety analysis. Sensitivity anduncertainty analysis capabilities must be further developed for comprehensive coupledcode simulations with non-linear feedback mechanisms as well as tested for uncertainty
propagation through multi-physics multi-scale calculations on comprehensive benchmark
frameworks, such as the LWR UAM benchmark described previously in this report [141].
Phase III of the LWR UAM benchmark is focused on propagation of multiple uncertainties
in coupled multi-physics steady-state, cycle depletion and transient calculations. The

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

102  NEA/NSC/R(2019)7
interactions between Phase III and Phase II (which is focused on uncertainty propagationin single physics models which are components of the coupled multi-physics calculations)
are shown in Figure 5.8.
One of the LWR core test problems in Exercises III-1 of the LWR UAM benchmark isbased on the Three Mile Island Unit 1 (TMI-1) PWR. An uncertainty propagation
mechanism has been developed using a stochastic sampling method by considering theuncertainties from neutronics, fuel modelling and thermal-hydraulics modelling in the
coupled multi-physics simulations. The LWR UAM Phase III benchmark specification has
been modelled using the coupled neutronics/thermal-hydraulics code system
TRACE/PARCS, which consists of a 3D neutronics model and a 3D thermal-hydraulics
core model (with integrated simplified fuel rod model). The input uncertainties of theneutronics simulation include the few-group cross-sections and kinetics parameters,
generated using the Sampler/Polaris sequence of SCALE 6.2.1. The parametrised crosssection and other nodal parameters uncertainty libraries have been generated following themethodologies developed in Exercise II-2 (see Figure 5.9). The most important heattransfer related variables were considered as sources of input uncertainty in the simplified
fuel rod modelling, including the thermal conductivity of fuel and cladding and the gap
conductance. The calculations performed within the framework of Exercise II-1 with thehigher-fidelity fuel performance code FRAPCON as well as utilising the available
experimental data helped developed parametrised values and associated uncertainty bounds
for these three parameters to be used in the coupled multi-physics calculations in Exercise
III-1. For thermal-hydraulic parameters a mini-PIRT for the envisioned initial steady-state
and transient application has been performed to identify which parameters plus
uncertainties are to be propagated. For such propagation the experience accumulated in the
Exercise II-3 PWR test problems was utilised. The envisioned transient applications are the
TMI-1 PWR Rod Ejection Accident (REA) scenarios. DAKOTA was used to sample theresulting single physics input parameters to the coupled multi-physics code system and to
perform the uncertainty analysis. Two types of simulations have been conducted: steadystate HZP and hot full power (HFP) core conditions at BOC and EOC and transient core
behaviour initiated by the spatially asymmetric REA scenarios. Quantities of Interest (QoI)
for the steady-state calculation, which includes core multiplication factor and power
peaking factors, were calculated with associated uncertainties and analysed by normality
test. For transient calculations, best-estimate results of the time evolution of core reactivity,
core power and peak fuel and cladding temperature were generated with 95%/95%
tolerance limits.
Figure 5.10 illustrates values and statistical error (as a result of using a statistical sampling
method with 150 samples in this study) in predicting multiplication factors for different
initial steady states while Figure 5.11 through Figure 5.13 show the predicted axial andpower distributions with propagated uncertainties. Note that larger uncertainties wereobserved at BOC than EOC and the propagated uncertainties of HZP states are more
pronounced than those of the HFP states.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  103

Figure 5.8. Interactions of Phase II and Phase III of the LWR UAM benchmark

Source: NCSU, 2019.

Figure 5.9. Process of generation of parametrised cross-section libraries plus uncertainties

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

104  NEA/NSC/R(2019)7
Figure 5.10. Statistical errors of 150 samples mean keff compare to the unperturbed (nominal) keff

Source: NCSU, 2019.

Figure 5.11. Axial power distributions plus uncertainties at different states

Source: NCSU, 2019.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  105

Figure 5.12. Radial power distributions plus uncertainties for HFP states

HFP BOC state

HFP EOC state

Source: NCSU, 2019.

Figure 5.13. Radial power distributions plus uncertainties for HZP states

HZP BOC state

HZP EOC state

Source: NCSU, 2019.

5.3 Use of Best Estimate Plus Uncertainty (BEPU) by regulation in licensing
calculations
Realistic methods currently in use in industry and regulation are referred to as “bestestimate” calculations, implying that they use a set of data, correlations and methods
designed to represent the phenomena, using the best available techniques. The riskinformed regulations also require that the uncertainty in these calculations be evaluated.
These calculations are based on traditional multi-physics tools and are usually named as
best-estimate plus uncertainty.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

106  NEA/NSC/R(2019)7
The traditional multi-physics tools have been used in two different ways in current
licensing practices depending on the country and associated regulatory framework: bestestimate bounding and BEPU. The traditional multi-physics codes (for example coupledneutron kinetics/thermal-hydraulics codes) have been utilised in licensing applications in
Europe in a best-estimate approach for transient and accident conditions with relevantchanges of the power density distribution and a tight coupling of neutronics and thermalhydraulics effects. Examples of such conditions are REA in PWRs or Rod Drop in BWRs,
MSLB and boron dilution in PWRs, Anticipated Transient Without Scram (ATWS) cases,
etc. In the US nuclear power generating industry, best-estimate methodologies
supplemented with uncertainty analysis have been applied for thermal-hydraulics
LBLOCA analysis. The US NRC accepts coupled 3D kinetics/thermal-hydraulics
calculations for accidents with strong space-time effects such as PWR REA and MSLB forlicensing power uprates, but these calculations have been applied in a bounding manner.
The international trends in using traditional multi-physics (best-estimate or realistic) codesin licensing applications is summarised in Table 5.4.
Table 5.4. Trends in using traditional multi-physics codes in licensing applications
Applied codes
Conservative code
Best-estimate(realistic) code
Best-estimate code
+ uncertainty
Best-estimate code
+ uncertainty

Initial andboundary
conditions
Conservative
Conservative
Realistic +uncertainty; partly
most unfavourable
conditions
Realistic +uncertainty

System availability

ApproachApplication

Conservative
assumptions
Conservative
assumptions
Conservative
assumptions

Deterministic

Old approach

Deterministic

Current practice

Deterministic

New international
trend; approach
recommended by
IAEA
Risk-informed future
standard?

PSA-basedassumptions

Deterministic +probabilistic

Source: Petruzzi et al., 2016.

Recently, within the licensing process of the Atucha II Pressurized Heavy Water Reactor(PHWR) the BEPU approach has been selected for issuing of Chapter 15 of the Final Safety
Analysis Report (FSAR) [142]. The complexity of a nuclear power plant and of the accident
scenarios may be a challenge for a conservative analysis and may justify the choice of a
BEPU approach in the licensing process. This implies two main needs: (1) the need to adopt
and to prove (to the regulatory authority) an adequate quality for the computational toolsand (2) the need to account for the uncertainty. The BEPU process was aimed at thelicensing of the Atucha II (can-II) Nuclear Power Plant in Argentina operated by
Nucleoeléctrica Argentina (NA-SA). Among the general attributes of a methodology toperform accident analysis of a nuclear power plant for licensing purposes, the very first one
should be compliance with the established regulatory requirements. A second attributedeals with the adequacy and the completeness of the selected spectrum of events that should
consider the combined contributions of deterministic and probabilistic methods. The third
attribute is connected to the availability of qualified tools and analytical procedures suitable
for the analysis of accident conditions envisaged for the nuclear power plant of concern.
The execution of the overall analysis and the evaluation of results in relation to slightly
fewer than 100 scenarios revealed the wide safety margins available for the nuclear power
plant of concern, which was licensed in 2014.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  107

Phenomena identification and ranking table (PIRT) Process
Introduction
Computer codes are used to design reactor systems and to analyse abnormal situations andtransients. The process of system modelling consists of mathematical numerical methods
and computer programs. The final product is a computer code. A computer code consistsof formulations and numerical methods. The formulations consist of conservation
equations for mass, momentum and energy for fluid, energy for fuel and equations forneutronics. The formulations also include hundreds of constitutive relationships
characterising different transfer terms between gas, liquid and solid (wall, clad etc.). In
addition to traditional multi-physics codes, there are other issues such as the effect ofradiation on fuel composition, out-gassing, fuel-clad gap properties, corrosion and claddeformation leading to fuel-clad interaction. There are many issues related to normal andabnormal conditions for the reactor operations that challenge safety barriers and there are
surrogate parameters that determine the effectiveness of the barrier. Some of these are clad
temperature, clad oxidation and containment pressure. In addition, codes also address
design related issues such as performance and power uprate.
For computer codes to be credible, they should have physics-based models that haveproperly adjusted parameters based on appropriate tests and have undergone rigorous
validation, where code results are compared with the test data. These tests are separate
effect tests and different levels of integral effect tests. Codes are applied to a variety ofreactor problems and the sub-models of physical phenomena have different levels of impact
on the QoI. Therefore, requirements for models for phenomena and component will be
specific to an application. This leads to a need to prioritise the model development and tests
for each application. In addition, the uncertainty quantification methods must also beappropriately chosen to characterise important physical phenomena and their interactions.
This is important as there are hundreds of models and it is expensive to quantify uncertainty
for each of them. Therefore, a method of identifying important phenomena is essential andit was first developed under the US NRC programme of code scaling, applicability anduncertainty (CSAU) quantification [67], called phenomena identification and ranking table(PIRT). There are two parts to PIRT, first, identification of phenomena and components at
different levels. Second, ranking them based on their impact on the QoI or a FOM QoI.
In the past, phenomena identification and ranking tables (PIRTs) were developed on thebasis of expert opinion. However, this becomes subjective. There are other approaches
based on scaling or sensitivity analyses that quantify the impact of phenomena on the FOM;
these are described in the next section.

Description
As PIRT is application dependent, the first step is to identify the reactor and the problem.
The problem could be steady state or transient. In the case of a transient, the duration of thetransient is divided into periods based on important events such as valve opening or
operator action, or some significant change in important phenomenon.
Once the problem and the period of the transient have been established, the phenomena aredecomposed from a top-down approach. The reactor system is first decomposed into larger
components or sections with groups of phenomena. Each of these sections is then further
decomposed. Finally, the decomposition leads to single basic phenomena. This step assuresthat all the phenomena and their hierarchy has been accounted for. This step depends onthe expertise of the PIRT participants.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

108  NEA/NSC/R(2019)7
The next step in a PIRT is to rank the phenomena on the basis of their impact on the FOM.
There are two approaches for ranking. The first approach is based on expert opinion. Thiswas earlier developed in the CSAU methodology and applied to PWR LBLOCA events.
This approach requires broad expertise among the participants of the PIRT. However, it
has been adopted in different fields since it was introduced in the CSAU methodology. The
weakness of this approach is that it is subjective and depends on expert opinion andresulting consensus.
The second approach is based on the relative quantitative impact of phenomena/component
on the FOM; a so-called Quantified Phenomena Identification and Ranking Table (QPIRT).
There are two ways to quantify the impact of different phenomena on the FOM. The first
approach requires an order of magnitude analysis based on the conservation equations andreference values of the variables for each phase of the transient or for steady state.
Examples of this approach are the hierarchical two-tiered scaling (H2TS) and the fractional
scaling analyses (FSA). In this approach the initial or average values of reference quantities
are used to nondimensionalise the system level balance equations for each phase of thetransient. This method provides a quantitative approach for phenomena/component
ranking.
The second approach for QPIRT is based on sensitivity analyses using the computer codes.
In this approach the sensitivity of output parameters or the FOM to model parameters or
initial and boundary conditions are estimated. Models with higher sensitivity have a higher
magnitude of change or rate of change. However, the impact will be measured by net
change of the parameter of interest or FOM over the variation of the model or input
parameters during period of interest. The weakness of this approach is that sensitivity andranking will be based on the models in the code that are themselves the target of code
validation.
There is another addition to a traditional PIRT, which is inclusion of the status ofknowledge. Knowledge will include an appropriate database and basic models. The
combination of PIRT ranking and the status of knowledge sets the priorities for future
experiment and model development. This new information was first incorporated into a
PIRT for the advanced CANDU Reactor in Canada. It has now also been included in the
DOE Consortium for Advanced Simulations of LWRs (CASL) programme.

ExamplesThree examples representing the approaches described earlier are documented below.
The US NRC first applied the PIRT process in the CSAU for LBLOCA. A group of experts
with knowledge of PWR LBLOCA analyses and tests was assembled. First, the transient
was divided into periods based on either dominant phenomena or valve or operator action.
A list of phenomena was identified. In the second iteration the phenomena are ranked basedon their impact on an identified FOM. In the case of LBLOCA, peak clad temperature wasselected as the FOM. The PIRT requires decomposition of the transient into phases and of
the system into components and phenomena. An example of this is shown in the CSAU
report [143].
LBLOCA was also analysed with the FSA methodology [144]. The approach is similar to
the CSAU methodology. The system level balance equations were nondimensionalisedwith reference quantities estimated at the beginning of the phase of the transient. The net
rate of change of state variables is the sum of the contributions from the different agents ofchange (phenomena). This approach identifies the phenomena through the agent of change

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  109

and the magnitude of the contribution to the net rate of change (frequency) is the measureof the ranking of the phenomena (see Table 5.5).
Table 5.5. FSA applications to LOFT and semi-scale

Net break flow

Net heating of twophase mixture

Net heating (cooling)of liquid

 Q

 Q

Pumping powerω is in %/s

bk

2

1

Noncondensable
gases

P

N

PP

LOFT

-577.3

1.41

-0.63

0.63

N.A.

Semi-scale-602.6

1.41

-0.52

0.52

N.A.

Source: Wulff et al., 2009.

The second example is from the CASL programme. CASL has identified several challenge
problems. One of the problems is pellet-clad interaction (PCI). In this case the simulation
requires interaction of multiple physics, namely, neutronics, thermal hydraulics and fuelperformance as shown in Figure 5.14. The envisioned validation pyramid is shown in
Figure 5.15 while the level PIRT for this multi-physics problem is depicted in Figure 5.16.
Figure 5.17 illustrates the bottom-up multi-scale validation approach with the lowest level
phenomena associated to modelling of subcooled boiling.
Figure 5.14. Coupling of physical phenomena for PCI

Source: N.IN.E., 2018.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 22

110  NEA/NSC/R(2019)7
Figure 5.15. PCI validation pyramid showing different physics

Flux maps, isotopic
data
Flux measurement,
code comparisons,activation rates

PCI inPWR

Ramp Test
Programs

ailFail/No-F ns
soCompari

Separate Effects
Testing

mp…..
FGR, Te
ΔD, ΔZ,

Halden IFAs, Ramp
Tests, Commercial
Rods
Separate Effects
Testingata
havior D
l and Be
a
)
ri
te
…
,
a
p
M
C, Cree(TC, SCtion
g Valida
Modelin alysis
r
io
v
a
h
Fuel Be PCI Failure An
for
Source: CASL, 2018.

Figure 5.16. Decomposed level PIRT for PCI

Source: CASL, 2018.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  111

Figure 5.17. Bottom-up approach with lowest level phenomena
associated to modelling of subcooled boiling

Source: CASL, 2018.

Oregon State University has developed a QPIRT for a 2-inch break in an AP1000 reactor
cold leg [145]. The method is based on the H2TS approach with the help of RELAP5
analyses. The starting formulation was based on a RELAP5 two-fluid approach. It yielded
28 non-dimensional groups. A simplified drift flux formulation yielded a smaller batch ofeight non-dimensional groups. The study was applied to the core region for a 2-inch breakfor the Advanced Plant EXperiment (APEX) facility designed for the AP600 plant.
RELAP5 results of the transient were used to estimate values for 14 non-dimensionalgroups for two phases. As expected, the non-dimensional groups change during different
phases of a small break LOCA and so does their ranking. Table 5.6 shows that for thevapour phase, the pressure drop and interfacial heat transfer are the top two processes
during In-Containment Refuelling Water Storage Tank (IRWST) injection. Pressure drop
and interfacial drag are the top two processes for the sump injection phase. However, forthe liquid phase, gravity and pressure are top two process groups (see Table 5.7). The twofluid formulation based PIRT is useful for prioritisation for code development. However,this approach is not as useful for developing integral level tests. The alternate approach is
to use a mixture level formulation for each phase of the transient. This led to eight П groups,
two in mass balance, four in momentum balance and two in energy balance. Figure 5.18
shows the π group values for five phases of the transient. Ranking of different phenomena
can therefore be established.
In addition, the study compared the QPIRT with a standard PIRT. The emphasis is different
between the QPIRT, which looks at balance equations and the PIRT where the effect ofphenomena on a FOM is considered. It should be noted that H2TS could be structured to gofrom the balance equations to any FOM. This study showed that for system and component
levels there was similarity in the ranking between the two approaches, except for a few
phenomena.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

112  NEA/NSC/R(2019)7
Table 5.6. Vapour Phase П Groups for APEX facility 2-inch break∏ Groups∏ Group values in transient
IRWST injection phase

Sump injection phase

Vapour generation
Pressure difference
GravityWall friction
Interfacial phase change acceleration

∏gΓ
∏gP
∏gB
∏FWG
∏ΓIG

4 100 (sec)
0.21
813.0
19.5
2.58
5.58

9 300 (sec)
1.93
153.7
3.26
0.41
2.68

Interfacial dragVirtual mass
Vaporisation transientPressure work
Wall heat transfer
Interfacial heat transfer

∏FIG
∏VMG
∏GVT
∏PG
∏Qwg
∏Qig

165.9
20.8
0.00
0.01
0.15
244

83.4
5.85
0.00
0.00
0.00
55.8

Interfacial phase change acceleration
Wall phase change
Dissipation

∏Γig
∏Γwg
∏DISSg

3.13
2.91
0.00

0.04
2.10
0.00

Physical processes
Mass

Momentum

EnergySource: Luo, 2012.

Table 5.7. Liquid Phase П Groups APEX 2-inch break∏fΓ∏ Group Values in Transient
IRWST injection phase
4 100 (sec)0.00

Sump injection phase
9 300 (sec)
0.02

Pressure difference
GravityWall friction
Interfacial phase change acceleration
Interfacial dragVirtual mass

∏fP
∏fB
∏FWF
∏ΓIF
∏FIF
∏VMF

397
7 055
0.64
0.01
10.6
1.33

153
7 647
10.26
0.00
60.3
4.23

Vapourisation transientPressure work
Wall heat transfer
Interfacial heat transfer
Interfacial phase change acceleration
Wall phase change

∏FVT
∏PF
∏Qwf
∏Qif
∏Γif
∏Γwf

0.00
0.00
0.27
4.39
0.01
0.06

0.00
0.00
0.15
4.83
0.29
0.14

Dissipation∏DISSf0.00

0.00

Physical processes∏ Groups

Mass

Vapour generation

Momentum

EnergySource: Luo, 2012.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  113

Figure 5.18. Values of П groups for different phases of the transient

Source: Luo, 2012.

Code validation and uncertainty analyses in licensing
All countries with operating and/or planned nuclear power plants have regulatory bodies
that ensure that the reactor design is safe and is operated safely during its life. Safety
decisions are made based on simulation of the reactor under design basis accident scenarios.
Each of the countries has either their own codes or internationally recognised codes suchas RELAP5, TRACE, CATHARE, ATHLET, etc.
Nuclear reactors are regulated by the US NRC based on analyses or simulation of reactors
under hypothetical accident scenarios. The analyses predict safety parameters that are
essential in protecting the public from the release of radioactive materials from the fuel.
The safety criteria are related to ensuring that the fuel clad is not breached. The acceptancecriteria that will ensure the integrity of the fuel clad are documented in 10 CFR 50.46 andthey are: (1) peak cladding temperature (PCT) should be less than 2 200 F; (2) local
maximum oxidation (LMO) should be less than 17%; (3) core-wide oxidation (CWO)
should be less than 1% (to limit the maximum amount of hydrogen generated); (4) the coreshould maintain a coolable geometry; and (5) long-term cooling should be demonstrated.
The US NRC has developed guidelines for development of an evaluation model (EM) for
safety studies and it is called the evaluation model. The EM consists of a reactor model,
nodalisation and the codes. These guidelines are suggestions and are not requirements.
The principles of an evaluation model development and assessment (EMDAP) weredeveloped from a study on quantifying reactor safety margins, based on the CSAU
evaluation methodology for a large break LOCA.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

114  NEA/NSC/R(2019)7
The purpose of the CSAU methodology was to demonstrate a method that could be used toquantify uncertainties as required by the best-estimate option described in the NRC’s 1988
revision to the ECCS Rule (10 CFR 50.46) [146]. The code uncertainty evaluation will
involve the entire process of EMDAP. Since the publication of the CSAU report in
December 1989, the CSAU process has been applied in many examples.
EMDAP is a multiple-step procedure, as shown in Figure 5.19. It has four elements andeach element has multiple steps, giving 20 steps in total. The process addresses the questionof the adequacy of an evaluation model (code and nodalisation) to simulate the scenario ofinterest in predicting safety parameters or a FOM.
The four elements of EMDAP address four topics; requirements of evaluation model,
assessment base to support identified requirements, development of evaluation model andassessment of adequacy of EM.
The following description is summarised from US NRC regulatory guide 1.203 [148].

Evaluation Model Development and Assessment Process (EMDAP)
The basic elements of EMDAP are shown in Figure 5.19. This regulatory position addresses
the four elements and the adequacy decision as shown in Figure 5.19. Adherence to an
EMDAP for new applications or a completely new EM could involve significant iterations
within the process. However, the same process applies even if the new EM is the result ofrelatively simple modifications to an existing EM.

Element 1: Establish requirements for evaluation model capability
Figure 5.20 illustrates four steps within this element and is described here.
Step 1 is to specify the plant, scenario and the purpose of analyses. Specifying the purpose
is important because any given transient may be analysed for different objectives. A
SBLOCA may be analysed to assess the potential for pressurised thermal shock (PTS) or
compliance with 10 CFR 50.46.
A LOCA may be analysed for load on containment. The statement of purpose influences
the entire process of development, assessment and analysis for the prediction of FOM.
The evaluation model applicability is scenario-dependent because the dominant processes,
safety parameters and acceptance criteria change from one scenario to another. The
transient scenario, therefore, dictates the processes that must be addressed. A complete
scenario definition is plant-class-specific or sometimes even plant-specific because thedominant phenomena and their interactions differ in varying degrees with the reactor design
or a plant-specific configuration such as a specific fuel type or core loading.
The licensee or applicant and the EM developer are responsible for justification of the useof EM and their applicability to plants and plant types.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  115

Figure 5.19. The EMDAP process (US NRC 2005)

Source: NRC, 2005.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

116  NEA/NSC/R(2019)7
Figure 5.20. Steps in Element 1

Source: NRC, 2005.

Step 2 identifies the FOM based on the scenario and purpose of analyses. FOM are thosequantitative that can be used to define acceptable answers for a safety analysis. The
requirements for maintaining the reactor in a safe condition during normal operation andduring transients and accidents are translated in terms of quantitative fuel and reactor
system design limits (DNBR, fuel temperatures, etc.) for the events of interest. For ECCS
design, five specific criteria described in 10 CFR 50.46 must be met for LOCA analysis.
During evaluation model development and assessment, a temporary “surrogate” FOM may
be used in evaluating the importance of phenomena and processes. In the case of SBLOCA,
vessel inventory was deemed more valuable in defining and assessing code capability.
Justification for using a surrogate FOM should be provided.
Because compensating errors in the code can unintentionally lead to correct answers,
additional performance measures serve as physical tracking points and additional proof of
accuracy. While the code may calculate the correct PCT, for example, incorrect orphysically impossible parameter values could evolve in other areas of the calculation.
Step 3 identifies systems, components, phases, geometries, fields and processes that should
be modelled in the EM. Hierarchical system decomposition methods were used toinvestigate scaling in complex systems [149,150]. These methods can also be valuable inidentifying EM characteristics. The hierarchical level are as follows:

system: the entire system that must be analysed for the proposed application;subsystems: major components that must be considered in the analysis; for some
applications, these may include the primary system, secondary system andcontainment; for other applications, only the primary system would need to beconsidered;

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  117modules: physical components within the subsystem (i.e. reactor vessel, steam
generator, pressuriser, piping run);

constituents: chemical form of substance (e.g. water, nitrogen, air, boron);phases: solid, liquid or vapour;

geometrical configurations (phase topology or flow regime): the geometrical shapedefined for a given transfer process (e.g. pool, drop, bubble, film);fields: the properties that are being transported (i.e. mass, momentum and energy);transport processes: mechanisms that determine the transport of and interactions
between constituent phases throughout the system.

Ingredients at each hierarchical level can be decomposed into the ingredients at the nextlevel down. This process is described as follows:each system can be divided into interacting subsystems;each subsystem can be divided into interacting modules;each module can be divided into interacting constituents;each constituent can be divided into interacting phases;each phase can be characterised by one or more geometrical configurations (phase
topology or flow regime);each phase can be described by field equations (e.g. conservation equations for mass,energy and momentum);the evolution of each field can be affected by several transport processes.

By carefully defining the number and type of each ingredient at each level, the evaluation
model developer should be able to establish the basic characteristics of the EM. It should
be noted that a deficiency at a higher level, is usually not resolvable by fixing ingredientsat lower levels.
Step 4 describes the process of identification and ranking of phenomena and processes as
they affect the FOM.
The scenario can be divided into distinct phases based on the events taking place such as
valves, or operator action. Each of these phases will have a different set of importantcomponents and phenomena with respect to the FOM. It should be noted that the FOM
could be a surrogate for safety parameters and the PIRT will reflect that. The PIRT canfollow decomposition of the model starting from plant level, moving down throughcomponent and process levels.
The PIRT should be used to determine the requirements for physical model development,
scalability, validation and sensitivity studies. Ultimately, the PIRT is used to guide any
uncertainty analysis or in the assessment of overall EM adequacy.
The initial phases of the PIRT process described in this step can rely heavily on expert
opinion, which can be subjective. Therefore, it is important to validate the PIRT using
experimentation and analysis. Scaling methods can also be used to validate the PIRT.
Sensitivity studies can help determine the relative influence of phenomena identified earlyin the PIRT development and for final validation of the PIRT as the EMDAP is iterated.
Some examples of sensitivity studies used for this purpose are shown in [146,151-153].
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

118  NEA/NSC/R(2019)7
The highly ranked phenomena and processes require greater modelling fidelity. The role of
the PIRT process in experiments, code development, and code applications associated withreactor safety analysis are shown in [151,154].

Element 2: Develop assessment base
Element 2 and its associated five steps are shown in Figure 5.21. In this element thevalidation matrix for specific scenario is established. Scaling methods and evaluation ofscale distortions in different tests are also performed.
Step 5 identifies that the test matrix is consistent with the PIRT and with the requirements
in Element 1 for code validation. The database should include the following:

separate effects experiments needed to develop and assess empirical correlations andother closure models;

integral systems tests to assess system interactions and global code capability;benchmarks with other codes (optional);

plant transient data (if available);

simple test problems to illustrate fundamental calculation device capability.

It should be noted that code-to-code comparison and fundamental tests are only
supplements and cannot replace SET, IET and plant data.
Step 6 is an import step in determining the adequacy of the validation test matrix for
intended use. A scaling approach is developed in this step and non-dimensional groups are
identified.
Scaling analyses should be conducted to ensure that the data and the models based on thosedata, will be applicable to the full-scale analysis of the plant transient. All tests, SETS and
IETs are designed to simulate some phases of some scenarios. Even there, the tests include
compromises and scale distortion. In some cases, tests designed for one phase of a scenario
may be used for a different scenario and may incur additional scale distortions. The scaling
analyses identify any such scale distortions. Knowledge of scale distortions is important inmaking a judgement about scalability of the EM. These distortions will add to biases anduncertainty in models and correlations in Element 4.
The scaling analyses employ both top-down and bottom-up approaches [150]. The topdown scaling approach evaluates the global system behaviour and systems interactions
from integral test facilities that can be shown to represent the plant-specific design underconsideration. A top-down scaling methodology is developed and applied to achieve thefollowing purposes:derive the non-dimensional groups governing similitude between facilities;show that these groups scale the results among the experimental facilities;

determine whether the ranges of group values provided by the experiment set
encompass the corresponding plant- and transient-specific values.

The bottom-up scaling analyses address issues raised in the plant- and transient-specific
PIRT related to localised behaviour. These analyses are used to explain differences amongtests in different experimental facilities and to use these explanations to infer the expected

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  119

plant behaviour and determine whether the experiments provide adequate plant-specificrepresentation.
Figure 5.21. Steps in Element 2

Source: NRC, 2005.

In Step 7, actual relevant SETS, IETS and plant data are selected based on the PIRT
developed in Step 4. This set of tests is called the assessment matrix. It is important that
there be multiple tests for the same phenomenon and components to avoid the effect ofcode tuning or calibration. Ideally, both the data that will be used to develop the correlationand the data that will be used to assess the correlation should be identified. This would help
to ensure that the correlation is not tuned to a particular data set and that the data used toassess the correlation have not been deliberately selected to make the correlation appear to
be more accurate than it truly is.
The data used for development and assessment should cover the full range of conditions
for which the correlation will be used. For integral behaviour assessment, counterpart tests(similar scenarios and transient conditions) in different experimental facilities at different
scales should be selected. Assessments using such tests lead to information about scaling
capability of the codes.
In Step 8, effects of IET distortions and SET scale-up capability are evaluated (US NRC
Regulation guide 1.203 [148]):
IET Distortions: Distortions in the IET database may arise from scaling
compromises (missing or atypical phenomena) in subscale facilities or atypical initial

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

120  NEA/NSC/R(2019)7
and boundary conditions in all facilities. The effects of the distortions should be
evaluated in the context of the experimental objectives determined in Step 5. If theeffects are important and distortions are significant, new tests should be sought or
performed.


SET Scale-up: As noted in Step 7, correlations should be based on separate effects
tests (SETs) at various scales. In the case of poor scale-up capability, it may benecessary to return to Step 6.

In Step 9, the experimental uncertainties in the database are documented. These
uncertainties arise from measurement errors, experimental distortions and other aspects ofexperimentation. If the quantified experimental uncertainties are too large compared to therequirements for evaluation model assessment, the data set or correlation should be
rejected. A code should not be penalised due to bad data. These uncertainties are added tocode calculation uncertainties determined from these tests.

Element 3: Develop evaluation model
Figure 5.22 depicts Element 3 with the three steps for developing the desired EM. Thiselement reviews the documentation to determine the adequacy of the calculation devices
for formulation, closure relationships and coupling between different codes. No
calculations with codes are part of this element.
Figure 5.22. Steps in Element 3

Source: NRC, 2005.

Step 10 is needed to assure that the EM is developed under quality control and with proper
documentation. Based on the requirements established in Element 1, an EM development
plan should be devised. Such a plan should include development standards and proceduresthat will apply throughout the development activity and should address the following
specific areas of focus:

design specifications for the calculation device;
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  121documentation requirements;

programming standards and procedures;transportability requirements;

quality assurance procedures;

configuration control procedures.

In Step 11, the code is assessed for adequacy of models and coupling for the intended
application based on requirements established in Element 1. The EM includes the modelsof the individual component calculation devices, as well as the coupling of the calculation
devices.
The structure for an individual device or code consists of the following six ingredients
which are consistent with the scaling approach in Step 6:Systems and components: The EM structure should be able to analyse the behaviourof all systems and components that play a role in the targeted application.

Constituents and phases: The code structure should be able to analyse the behaviourof all constituents and phases relevant to the targeted application.



Field equations: Field equations are solved to determine the transport of thequantities of interest (usually mass, energy and momentum).

Closure relations: Closure relations are correlations and equations that help to model
the terms in the field equations by providing code capability to model and scaleparticular processes.



Numerics: Numerics provide code capability to perform efficient and reliablecalculations.



Additional features: This addresses code capability to model boundary conditions
and control systems.

The coupling of different codes in space and time should be described. Coupling should beconsistent with the requirements and decomposition described in Step 3. Since most codes
include user options, all selections made should be justified as appropriate for the EM.
Step 12 is for review of adequacy of closure models. Models or closure relationships that
describe a specific process are developed using SET data. This includes models that can beused in a stand-alone mode or correlations that can be incorporated in a calculation device(usually a computer code). On rare occasions, sufficient experimental detail may beavailable to develop correlations from IET experiments. The scalability and range ofapplicability of a correlation may not be known (a priori) the first time it is developed or
selected for use in this step. An iteration of the scale-up evaluation (Step 8) and adequacy
assessment (Element 4) may be needed to ensure correlation applicability.
The basis, range of applicability and accuracy of incorporated phenomenological models
should be known and traceable. Justification should be provided for extension of any
models beyond their original bases.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

122  NEA/NSC/R(2019)7

Element 4: Assess evaluation model adequacy
Evaluation model adequacy can be assessed after the previous elements have been
established and the EM capability has been documented. Figure 5.23 is a diagram of
Element 4 and its eight related steps.
The EM assessment is divided into two parts as shown in Figure 5.23. The first part
(Steps 13-15) pertains to the bottom-up evaluation of the closure relations for each code.
In the first part, important closure models and correlations are examined by considering
their pedigree, applicability, fidelity to appropriate SET data and scalability. The termbottom-up is used because the review focuses on the fundamental building blocks of thecode.
The second part (Steps 16-19) pertains to the top-down evaluations of code-governing
equations, numerics, the integrated performance of each code, coupling and the integrated
performance of the overall EM.
In the second part of the assessment, the EM is evaluated by examining the field equations,numerics, applicability, fidelity to component or integral effects data and scalability. Thispart of the assessment is called the top-down review because it focuses on capabilities andperformance of the EM.
The third group of data is the actual plant transient data and it provides confirmatory
supporting assessments. It is important as there are no scale issues. Plant data can be used
for code assessment if it can be demonstrated that the available instrumentation provides
measurements of adequate resolution to assess the code.
It is important to note that any changes to an EM should include at least a partial assessment
to ensure that these changes do not produce unintended results in the code’s predictive
capability.
Step 13 is needed to review the database behind correlations and component models in thecodes. The pedigree evaluation relates to the physical basis of a closure model, assumptions
and limitations attributed to the model and details of the adequacy characterisation at thetime the model was developed. The applicability evaluation relates to whether the model,
as implemented in the code, is consistent with its pedigree and can cover the range of theconditions expected during the transient. This step will indicate if further assessment with
SETs should be done.
In Step 14, the relevant SETs are modelled and predictions are compared with the data.
SET input for component devices used in the model assessment (usually computer codes)
should be prepared to represent the phenomena and test facility being modelled, as well as
the characteristics of the nuclear power plant design. Nodalisation and option selection
should be consistent between the experimental facility and similar components in the
nuclear power plant.
Convergence studies should be performed to the extent practicable in both the test facility
and plant models. Some models are essentially lumped parameter models and, in thosecases, a nodalisation convergence study cannot be performed. In such cases, care should betaken to ensure that the model is applicable to both the test facility and the plant. It should
be noted that for the most system codes, the predictions depend on nodalisations and code
options and for assessment results to be applicable to plant calculations, the modellingshould be consistent with the plant model.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  123

When the calculations of the SETs are completed, the differences between calculated
results and experimental data for important phenomena should be quantified for bias anddeviation.
Figure 5.23. Steps in Element 4

Source: NRC, 2005.

Step 15 determines the scalability of the specific model or correlation is appropriate forapplication to the configuration and conditions of the plant and transient under evaluation.
There are many references that illustrate this process [149,155-158]. In addition,
counterpart tests could also be used to assess scalability of the correlations and component
models.
Step 16 is part of the four steps that assess the ability of the codes to integrate different
phenomena and components. This step deals with review of the adequacy of the code
formulations and numerics for the intended application.
The numeric solution evaluation considers convergence, property conservation andstability of code calculations to solve the original equations when applied to the target
application. The objective of this evaluation is to summarise information regarding thedomain of applicability of the numerical techniques and user options that may impact theaccuracy, stability and convergence features of each component code.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

124  NEA/NSC/R(2019)7
This step like Step 13, will indicate if further calculations must be performed. A complete
assessment within this step can only be performed after completing a sufficient foundationof assessment analyses.
Step 17 addresses the review of code capability before the simulation of IETs. Thisapplicability evaluation considers whether the integrated code is capable of modelling theplant systems and components. Before performing integrated analyses, the various EM
options, special models and inputs should be determined to have the inherent capability tomodel the major systems and subsystems required for the particular application.
Step 18 prepares input and performs calculations to assess system interactions and global
capability.
Step 19 assesses scalability of integrated calculations and data distortions.
At the end Step 20 determines biases and uncertainties in the evaluation method.

Adequacy decision
EMDAP provides a logical and auditable method to assess the applicability of an EM to an
intended scenario. This approach also provides a method of quantifying the uncertainty inimportant phenomena and component models. These uncertainties can then be combined
to provide an aggregated estimate of the uncertainty in the FOM. All 20 steps will befollowed.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  125

6. Conclusions

This report summarises basic concepts of validation and uncertainty quantification for
single and multi-physics multi-scale codes. It includes a review of validation anduncertainty quantification (UQ) for single physics codes such as – core reactor physics, fuelperformance, core thermal hydraulics and system thermal hydraulics. In addition, it
addresses the validation approach for multi-physics codes consisting of thermal-hydraulics
codes coupled with neutronics codes and fuel codes coupled with neutronics code.
The uncertainty quantification methods identified the source of uncertainty in prediction as
initial condition, boundary condition and model form uncertainty. The majority of methods
propagate uncertainties in initial conditions, boundary conditions and model form
uncertainty through the codes to estimate the aggregate uncertainty in the quantity ofinterest. There are few methods that predict uncertainty in final quantity by propagation of
output uncertainty from simulation of integral facilities. One method is a hybrid of these
two approaches. In licensing space, the code scaling, applicability and uncertainty (CSAU),
evaluation model development and assessment (EMDAP) and Gesellschaft für Anlagenund Reaktorsicherheit gGmbH (GRS) methodologies are widely used.
The common elements of all the approaches for validation and uncertainty quantification
are as follows:

specification of plant and transient;frozen code with good documentation;

a phenomena identification and ranking table (PIRT) process based on experts,
scaling or sensitivity analyses;

a test matrix consisting of separate effects and integral effects tests that represent
the phenomena in the plant for the transient of interest;

definition of plant input based on user guidelines and feedback from models forseparate effect tests (SETs) and integral effect tests (IETs);

contributors to uncertainty in figures of merit (FOMs) are initial and boundary
conditions, approximation in representation of actual geometry, model form
uncertainties and numerical methods. Model form uncertainty evaluation should
account for scale distortion.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

126  NEA/NSC/R(2019)7

References[1]

Finck, P.,T. Valentine, J. Dyrda, J. Gulliford, J. Hudelot and U. Rohatgi (2016),“OECD/NEA Expert Group on Multi-Physics Experimental Data, Benchmarks andValidation”, PHYSOR 2016 Conference, Sun Valley, ID, 1-5 May 2016.

[2]

Oberkampf, W. and T. Trucano (2004), Applied Mechanics Reviews, Vol. 57, no. 5, pp. 345384.

[3]

Oberkampf, W. and C. Roy (2010), Verification and Validation in Scientific Computing,Cambridge University Press, Cambridge.

[4]

Smith, K. (2008), “Studsvik Scandpower’s s V&V approach for LWR reactor physics
software”, Verification and Validation for Nuclear Systems Analysis Workshop, 21-25 July
2008, Idaho Falls, Idaho, United States.

[5]

Briggs, B. (2008), “The anatomy of a benchmark – Status of ICSBER and IRPhEP”,
Verification and Validation for Nuclear Systems Analysis Workshop, 21-25 July 2008, Idaho
Falls, Idaho, United States.

[6]

Mayhue, L., R. Milanova, H. Huria, B. Zhang, F. Franceschini, M. Ouisloumen, E. Mueller
and P. Forslun Guimaraes (2008), “Qualification of NEXUS/ANC nuclear design system for
PWR analysis”, PHYSOR08: International Conference on Reactor Physics, Nuclear Power:
A Sustainable Resource, Interlaken, Switzerland, 14-19 September 2008.

[7]

NEA (2017), International Handbook of Evaluated Reactor Physics Benchmark
Experiments, OECD/NEA Nuclear Science, 2017 Edition.

[8]

NEA (2016), International Handbook of Evaluated Criticality Safety Benchmark
Experiments, OECD/NEA Nuclear Science, 2016 Edition.

[9]

Hutton, J., D.J. Powney, P.K.Hutt, M.P. Knight, P. Bryce, A. Goddard, C.R. Schneidesch,D. Vantroyen, S. Bosso and O. Ergo (2000), “Comparisons of WIMS/PARTNER
calculations with measurements of a range of operating PWRs”, Proceedings of PHYSOR2000 Conference.

[10] Cunningham, M., C. Beye, F. Panisko, P. Medvedev and G. Bema (2001), FRAPTRAN: A
Computer Code for the Transient Analysis of Oxide Fuel Rods − NUREG/CR-6739, PNNL13576, Pacific Northwest National Laboratory, Richland, WA.
[11] Moal (2010), “Advanced models for the simulation of post-DNB phenomena during
reactivity initiated accidents with SCANAIR”, LWR Fuel Performance/TopFuel/WRFPM,
Orlando.
[12] Lassmann, K. (1992), “TRANSURANUS: A fuel rod analysis code ready for use”, Journalof Nuclear Materials, Vol. 188 pp. 295-302.
[13] Veshchunov, M.S., A.l Boldyrev, V. Ozrin, V. Shestak and V. Tarasov (2011), “A new
mechanistic code SFPR for modeling of single fuel rod performance under various regimesof LWR operation”, Nucl. Eng. Design. 2011. Vol. 241. pp. 2822-2830.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  127[14] Veshchunov, M.S., V. Ozrin, V. Shestak, V. Tarasov, R. Dubourg and G. Nicaise (2006),“Development of the mechanistic code MFPR for modelling fission-product release from
irradiated UO2 fuel”, Nucl. Eng. Design. 2006. Vol. 236. pp. 179-200.
[15] Geelhood, K., W. Luscher and C. Beyer (2011), “FRAPCON-4.0: A computer code for the
calculation of steady-state”, Thermal-Mechanical Behavior of Oxide Fuel Rods for HighBurnup (NUREG/CR-7022, Vol. 1, PNNL-19418).
[16] NEA (2013), RIA Fuel Codes Benchmark, Vol. 1. No. NEA-CSNI-R-2013-7. OECD, 2013.
[17] Geelhood, K., W. Luscher, C. Beyer, J. Cuta and M. Flanagan (2011), “FRAPTRAN 1.4: A
computer code for the transient analysis of oxide fuel rods”, Pacific Northwest National
Laboratory (PNNL), Richland, WA, US, NUREG/CR-7023 1 (2011).
[18] Bouloré, A., C. Struzik and F. Gaudier (2012), “Uncertainty and sensitivity analysis of thenuclear fuel thermal behavior”, Nuclear Engineering and Design 253 (2012): 200-210.
[19] Ikonen, T. and V. Tulkki (2014), “The importance of input interactions in the uncertainty
and sensitivity analysis of nuclear fuel behavior”, Nuclear Engineering and Design 275(2014): 229-241.
[20] Geelhood, K., W. Luscher, C. Beyer, D. Senor, M. Cunningham, D. Lanning and H. Adkins(2009), “Predictive bias and sensitivity in NRC fuel performance codes”, NUREG/CR-7001,
PNNL-17644, PNNL, Richland, WA.
[21] Rashid, J., S. Yagnik and R. Montgomery (2011), “Light water reactor fuel performancemodeling and multi-dimensional simulation”, JOM 63.8 (2011): 81.
[22] Geelhood, K. and W. Luscher (2014), “FRAPCON-3.5: A computer code for the calculationof steady-state”, Thermal-Mechanical Behavior of Oxide Fuel Rods for High Burnup,
NUREG/CR-7022, Vol. 1, Rev. 1, PNNL-19418, Vol. 1, Rev. 1.
[23] Geelhood, K.J.and W.G. Luscher (2014), “FRAPCON-3.5: Integral assessment”,
NUREG/CR-7022, Vol. 2, Rev. 1, PNNL-19418, Vol. 2, Rev. 1.
[24] Geelhood, K.J., W.G. Luscher and J.M. Cuta (2014), “FRAPTRAN-1.5: A computer codefor the transient analysis of oxide fuel rods”, NUREG/CR-7023, Vol. 1, Rev. 1 PNNL19400, Vol. 1, Rev. 1.
[25] Geelhood, K.J. and W.G. Luscher (2014), “FRAPTRAN-1.5: Integral assessment”,
NUREG/CR-7023, Vol. 2, Rev. 1, PNNL-19400, Vol. 2, Rev. 1.
[26] Toptan, A., R. Salko, M. Avramova, D. Kropaczek, K. Clarno (2017), “Development andassessment of CTFFuel”, CASL-U-2017-1418-000.
[27] Avramova, M. (2003), COBRA-TF Development, Qualification, and Application to LWR
Analysis, MS Thesis, Pennsylvania State University.
[28] Salko, R. et al. (2016), CTF Validation and Verification, CASL-U-2016-1113-000.
[29] NEA (2006), NUPEC BWR Full-size Fine-mesh Bundle Test (BFBT) Benchmark Volume 1:
Specifications, ISBN 92-64-01088-2, NEA No. 6212.
[30] Lahey, R.T., Jr., B.S. Shirlakar and D.W. Radcliffe (1970), Two-Phase Flow and Heat
Transfer In Multi-rod Geometries: Subchannel and Pressure Drop Measurements in a NineRod Bundle for Diabatic and Adiabatic Conditions, Technical report, GEAP-13049, General
Electric.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

128  NEA/NSC/R(2019)7[31] NEA (2010), OECD/NRC Benchmark Based on NUPEC PWR Subchannel and Bundle Test
(PSBT), NEA/NSC/DOC(2012)1.
[32] Smith, K.A., A.J. Baratta and G.E. Robinson (1995), “Coupled RELAP5 and CONTAIN
accident analysis using PVMˮ, Nuclear Safety 36, No. 1., 94-108.
[33] Weaver, W. (2005), Programmers Manual for PVM Coupling Interface in RELAP5-3D
Code, INL-EXT-05-00203.
[34] Ofstun, R. and J. Scobel (2006), Westinghouse Containment Analysis Methodology, WCAP16608-NP.
[35] Murata, K., D. Williams, R. Griffith, R. Gido, E. Tadios, F. Davis, G. Martinez, K.
Washington and J. Tills (1997), Code Manual for CONTAIN 2.0: A Computer Code for
Nuclear Reactor Containment Analysis, NUREG/CR-6533, SAND97-1735, Sandia
National Laboratories.
[36] Hong, S., Y. Choo, S. Hwang, B. Lee and S. Ha (2015), Development of CAP Code for
Nuclear Power Plant Containment: Lumped Model, NED, 291(2015), pp. 47-63.
[37] NEA (1987), “Code Validation Matrix of Thermo-Hydraulic Codes for LWR LOCA andTransients”, CSNI Report 132, OECD Publishing, Paris, www.oecd-nea.org/jcms/pl_15784.
[38] NEA (1987), “Code Validation Matrix of Thermo-Hydraulic Codes for LWR LOCA andTransients”, CSNI Report 132, OECD Publishing, Paris, www.oecd-nea.org/jcms/pl_15784.
[39] Aksan, N. (2008), “International standard problems and small break loss-of-coolant accident(SBLOCA)”, Proceedings of “THICKET 2008: Seminar on Transfer of Competence,
Knowledge and Experience Gained through CSNI Activities in the field ofThermalhydraulics”, CSNI OECD/NEA, INSTN and IRSN, Pisa (Italy) 5-9 May 2008.
[40] IAEA (2001), Validation of Accident and Safety Analysis Methodology, International
Atomic Energy Agency, Internal Technical Report, Vienna.
[41] Zuber, N. et al. (1998), “An integrated structure and scaling methodology for severe accident
technical issue resolution: Development of methodology”, Nuclear Engineering and Design,Vol. 186, Issues 1-2, 1, pp. 1-21.
[42] D’Auria, F., G. Galassi and P. Gatta (1997), “Scaling in nuclear system thermal hydraulics:
A way to utilise the available database”, ASME 32nd National Heat Transfer Conference,
vol. 350, pp. 35, Baltimore, MD, United States.
[43] Ivanov, K., E. Sartori, E. Royer, S. Langenbuch and K. Velkov (2007), “Validation of
coupled thermal-hydraulic and neutronic codes for safety analysis by international
cooperation”, Nuclear Technology, Vol. 157, No. 2, pp. 177-195.
[44] Avramova, M. and K. Ivanov (2010), “Verification and validation and uncertainty
quantification in multi-physics modeling for nuclear reactor design”, Progress in Nuclear
Energy, 52 (2010) 601-614.
[45] Avramova, M., K. Ivanov, T. Kozlowski, I. Pasichnyk, W. Zwermann, K. Velkov, E. Royer,
A. Yamaji and J. Gulliford (2015), “Multi-physics and multi-scale benchmarking anduncertainty quantification within OECD/NEA framework”, Annals of Nuclear Energy,
Invited paper, Vol. 84, pp. 178-196.
[46] Finnemann, H. and A. Galati (1991), Final Specifications: NEACRP 3-D LWR Core
Transient Benchmark, NEACRP-L-335.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  129[47] NEA (1993), “NEA-NSC 3-D/1-D PWR Core Transient Benchmark Uncontrolled
Withdrawal of Control Rods at Zero Power”, NEA/NSC(93)9, OECD Publishing, Paris,
www.oecd-nea.org/jcms/pl_16310.
[48] Kozlowski, T. and T. Downar (2006), PWR MOX/UO2 Core Transient Benchmark, Final
Report, , NEA/NSC/DOC(2006)20, www.oecd-nea.org/jcms/pl_14074.
[49] Toptan, A., R., Salko and M. Avramova (2017), “Review of CTF’s fuel rod modeling using
FRAPCON-4.0’s centerline temperature predictions”, Proceedings: 2017 ANS Annual
Meeting, San Francisco, CA, US, 11-15 June 2017.
[50] Magedanz, J., M. Avramova, Y. Perin and A. Velkov (2015), “High-fidelity multi-physics
system TORT-TD/CTF/FRAPTRAN for light water reactor analysis”, Annals of Nuclear
Energy, Invited paper, Vol. 84, pp. 234-143.
[51] NEA (2004), CRISSUE-S − WP2 Neutronics/Thermal-hydraulics Coupling in LWR
Technology: State-of-the-art Report (REAC-SOAR), Technical Report, www.oecdnea.org/jcms/pl_13968.
[52] Grandi, G. and D. Hagrman (2007), Improvements to the INTERPIN Code for High Burnup
and MOX Fuel, Vol. 97, TANSAO 97, pp. 614-615.
[53] Grandi, G. and K. Smith (2007), SIMULATE-3K Explicit Fuel Pin Modeling in RIAs, Vol.
96, TANSAO 96, pp. 627-629.
[54] Turner, J., K. Clarno, M. Sieger, R. Bartlett, B. Collins, R. Pawlowski, R. Schmidt and R.
Summers (2016), “The virtual environment for reactor applications (VERA): Design andarchitecture”, Journal of Computational Physics 326 (2016), 544-568.
[55] Avramova, M., K. Ivanov (2017), “Multi-physics modeling of nuclear reactors”, Access
Science, McGraw-Hill Global Education Holdings, LLC.
[56] Ziabletsev, D., M. Avramova, K. Ivanov (2004), “Development of PWR integrated safety
analysis methodology using multi-level coupling algorithm”, Nuclear Science andEngineering, Vol. 148, No. 3, pp.414-425.
[57] Solis, J., M. Avramova, K. Ivanov (2004), “Multi-level methodology in parallel computing
environment for evaluating BWR safety parameters”, Nuclear Technology, Vol. 146, No. 3,
pp. 267-278.
[58] Schneidesch, C. and J. Zhang (2004), “Qualification of the coupled
RELAP5/PANTHER/COBRA code package for licensing applications”, Proceedings ofInternational Meeting on Updates in Best-Estimate Methods in Nuclear Installation Safety
Analysis (BE-2004), Washington, DC, 14-18 November 2004.
[59] IAEA (2008), “Best estimate safety analysis for nuclear power plants: Uncertainty
evaluation”, IAEA Safety Reports Series No. 52, IAEA, Vienna.
[60] Cacuci, D. (2003), “Sensitivity and uncertainty analysis Volume 1 Theory”, Chapman &Hall/CRC.
[61] Glaeser, H. (2008), “GRS method for uncertainty and sensitivity evaluation of code results
and applications”, Science and Technology of Nuclear Installations, Vol. 2008, Article ID:
798901.
[62] Petruzzi, A. and F. D’Auria (2008), “Approaches, relevant topics and internal method for
uncertainty evaluation in predictions of thermal-hydraulic system codes”, Science andTechnology of Nuclear Installations, Vol. 2008, Article ID 325071.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

130  NEA/NSC/R(2019)7[63] Triola, M. (2007), Elementary Statistics Using Excel, 3rd edition, Pearson Education, Boston
2007.
[64] Hofer, E., M. Kloos, B. Krykacz-Hausmann, J. Peschke and M. Woltereck (2002), “Anapproximate epistemic uncertainty analysis approach in the presence of epistemic andaleatory uncertainties”, Reliability Engineering and System Safety 77 (2002), 229-238.
[65] Helton, J. and F. Davis (2002), “Illustration of sampling-based methods for uncertainty andsensitivity analysis”, Risk Analysis, Vol. 22, No. 3.
[66] USNRC (1988), Emergency Core Cooling Systems, Revisions to Acceptance Criteria,
Federal Register 53, 180.
[67] Boyack, B. et al. (1989), Quantifying Reactor Safety Margins, NUREG/CR-5249, R4.
[68] Rohatgi, U., J. Jo, L.Y. Neymotin and W. Wulff (1989), Bias in Peak Clad Temperature
Predictions Due to Uncertainties in Modeling of ECC Bypass and Dissolved NonCondensable Gas Phenomena, NUREG/CR-5254, BNL-NUREG-52168.
[69] Weiss, P., R. Emmerling, R. Hertlein and J. Liebert (1994), “UPTF experiment refined PWR
LOCA thermal-hydraulic scenarios: Conclusions from a full-scale experimental program”,
Journal of Nuclear Engineering and Design, Vol. 149, Issues 1–3, 1 September 1994, pp.
335-347.
[70] Nalezny, C.L. (1983), Summary of Nuclear Regulatory Commission's LOFT Program
Experiments, NUREG/CR-3214, EGG-2248.
[71] Murao, Y., T. Iguchi, H. Akimoto and T. Iwamura (1993), “Large-scale multi-dimensional
phenomena found in CCTF and SCTF experiments”, Nuclear Engineering and Design, Vol.
145, Issues 1-2, 2 November 1993, pp. 85-95.
[72] Umminger, K. (2012), “Integral test facility PKL: Experimental PWR accident
investigation”, Review Article, Hindawi Publishing Corporation, Science and Technologyof Nuclear Installations, Vol. 2012, Article ID 891056, 16 pages,
DOI:10.1155/2012/891056.
[73] Hofer, E. (1993), Probabilistische Unsicherheitsanalyse von Ergebnissen umfangreicher
Rechenmodelle, GRS-A-2002.
[74] Wilks, S. (1941), “Determination of sample sizes for setting tolerance limits”, Ann. Math.
Statist., 12, pp. 91-96.
[75] Guba, M., M. Makai and L. Pai (2003), “Statistical aspects of best estimate methods”,
Reliability Engineering and System Safety, Vol. 80, No 3, pp. 217-232.
[76] Zwermann, W., B. Krzykacz-Hausmann, L. Gallner and A. Pautz (2009), “Influence of
nuclear covariance data on reactor core calculations”, Proc. WONDER 2009, Saint-PaulLez-Durance,
France,29
September−
2 October 2009.
[77] Williams, M., G. Ilas, M.A. Jessee, B.T. Rearden, D. Wiarda, W. Zwermann, L. Gallner, M.
Klein, B. Krzykacz-Hausmann and A. Pautz (2013), “A statistical sampling method for
uncertainty analysis with SCALE and XSUSA”, Nucl. Techn. 183, pp. 515-526.
[78] Krzykacz, B., E. Hofer and M. Kloos (1994), “A software system for probabilistic
uncertainty and sensitivity analysis of results from computer models”, Proceedings ofInternational Conference on Probabilistic Safety Assessment and Management (PSAM-II),San Diego, CA, US.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  131[79] ORNL (2011), SCALE: A Comprehensive Modeling and Simulation Suite for Nuclear Safety
Analysis and Design, ORNL/TM-2005/39, Version 6.1.
[80] Bostelmann, F., F.-P. Weiss, A. Aures, K. Velkov, W. Zwermann, B. Rearden, M.A. Jessee,
M. Williams, D. Wiarda and W. Wieselquist (2015), “Uncertainty and sensitivity analysis
in criticality calculations with perturbation theory and sampling”, Proceedings of Joint
International Conference and Mathematics and Computation (M&C), Supercomputing in
Nuclear Applications (SNA) and the Monte Carlo (MC) Method, Nashville, TN, US, 19-23
April 2015, on CD-ROM.
[81] Bostelmann, F., W. Zwermann, B. Krzykacz-Hausmann, L. Gallner, A. Aures and K. Velkov(2015), “Sampling-based nuclear data uncertainty analysis in criticality and depletioncalculations”, Proc. ANS MC2015, Nashville, TN, US, 19-23 April 2015.
[82] Hofer, E., M. Kloos, B. Krzykacz-Hausmann, J. Peschke and M. Woltereck (2002), “Anapproximate epistemic uncertainty analysis approach in the presence of epistemic andaleatory uncertainties”, Reliability Engineering and System Safety 77 (2002), 229-238.
[83] Glaeser, H. (2008), “GRS method for uncertainty and sensitivity evaluation of code results
and applications”, Science and Technology of Nuclear Installations, Vol. 2008, Article ID:
798901.
[84] Kloos, M. (2015), “Main features of the tool SUSA 4.0 for uncertainty and sensitivity
analyses”, ESREL 2015, European Safety and Reliability Conference, Zürich, Switzerland,
7-10 September 2015.
[85] Frepoli, C. (2007), A Review of Westinghouse Realistic Large Break LOCA Evaluation
Model, Hindawi Publication, August 2007.
[86] Martin, R., L. O’Dell (2007), Development Considerations of AREVA NP Inc’s Realistic
LBLOCA Analysis Methodology, Hindawi Publication, December 2007.
[87] D’Auria, F., G. Giorgio and N. Debrecin (1995), “Outline of the uncertainty methodology
based on accuracy extrapolation (UMAE)”, J. Nuclear Technology, 109, No 1, pp. 21-38.
[88] Ambrosini, W., R. Bovalini and F. D’Auria (1990), “Evaluation of accuracy of thermalhydraulic codes calculations”, J. Energia Nucleare, 7, No. 2.
[89] Petruzzi, A., F. D’Auria (2005), Accuracy Quantification: Description of the Fast Fourier
Transform Based Method (FFTBM), DIMNP NT 556(05).
[90] NEA (1997), Proceedings of the OECD/CSNI Workshop on Transient Thermal-hydraulicand Neutronic Codes Requirements, Annapolis NEA/CSNI R(97)4, Washington, DC, US,
www.oecd-nea.org/jcms/pl_16162.
[91] D’Auria, F., W. Giannotti (2000), “Development of code with capability of internal
assessment of uncertainty”, J. Nuclear Technology, 131, No. 1, pp. 159-196.
[92] Petruzzi, A., F. D’Auria, W. Giannotti and K. Ivanov (2005), “Methodology of internal
assessment of uncertainty and extension to neutron-kinetics/thermal-hydraulics coupledcodes”, Nuclear Science and Engineering, 149, pp. 1-26.
[93] Petruzzi, A., F. D’Auria (2008), “Thermal-hydraulic system codes in nuclear reactor safety
and qualification procedures”, Hindawi Publishing Corporation, Science and Technology of
Nuclear Installations, Vol. 2008, Article ID 460795.
[94] Petruzzi, A., F. D’Auria, W. Giannotti and K. Ivanov (2004), “Enlargement of theuncertainty database of the CIAU methodology”, PHYSOR 2004, Chicago, Illinois, US.
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

132  NEA/NSC/R(2019)7[95] Petruzzi, A., W. Giannotti, F. D’Auria (2004), “Development, qualification and use of acode with the capability of internal assessment of uncertainty”, CNS, Sixth International
Conference on Simulation Methods in Nuclear Engineering, Montreal, Canada.
[96] Petruzzi, A., F. D’Auria (2016), “Standardized consolidated calculated and reference
experimental database (SCCRED): A supporting tool for V&V and uncertainty evaluationof best-estimate system codes for licensing applications”, Nuclear Science and Engineering,
15-79.
[97] Bovalini, R., F. D'Auria, M. Leonardi (1992), Qualification of the Fast Fourier Transform
based Methodology for the Quantification of Thermal-hydraulic System Code Accuracy,
DCMN NT 194(92), Pisa.
[98] D’Auria F., G.M. Galassi (1998), “Code validation and uncertainties in system thermalhydraulics”, J. Progress in Nuclear Energy, 33, No. 1 and 2, pp. 175-216.
[99] NEA (1989), Thermo-hydraulics of Emergency Core Cooling in Light Water Reactors-A
State of the Art Report, NEA/CSNI 161, OECD Publishing, Paris, www.oecdnea.org/jcms/pl_15836.
[100] D’Auria, F., V. Faluomi, N. Aksan (1995), A Proposed Methodology for the Analysis of a
Phenomenon in Separate Effects and Integral Test Facilities, Kerntechnik, 60, 4, 166.
[101] D’Auria, F., V. Pellicoro (1995), “Scaling and uncertainty in BWR instability problem”, 2ndRegional Meeting, Nuclear Energy in Central Europe, Portoroz, Slovenia, 11-14 September
1995.
[102] Groeneveld, D.C., L.K.H. Leung, P.L. Kirillov, V.P. Bobkov, I.P. Smogalev, V.N.
Vinogradov, X.C. Huang and E. Royer (1996), “The 1995 look-up tables for critical heatflux
intubes”,
J. Nuclear Engineering and Design, 163, 1,1.
[103] Cacuci, D.G., M. Ionescu-Bujor (2010), “Sensitivity and uncertainty analysis, data
assimilation and predictive best-estimate model calibration”, Chapter 17 in Vol.3, pp. 19132051, Handbook of Nuclear Engineering, D.G. Cacuci, Editor, ISBN: 978-0-387-98150-5,
Springer New York/Berlin.
[104] Cacuci, D.G., M. Ionescu-Bujor (2010), “Model calibration and best-estimate prediction
through experimental data assimilation: I. Mathematical framework”, Nucl. Sci. Eng., 165,
18-44.
[105] Cacuci, D.G. (2003), “Sensitivity and uncertainty analysis”, Theory, Vol. I, Chapman &Hall/CRC, Boca Raton (Fl, US), ISBN 1-58488-115-1.
[106] Cacuci, D.G., M. Ionescu-Bujor, I.M. Navon (2005), “Sensitivity and uncertainty analysis,
application to large scale systems”, Vol. II, Chapman & Hall/CRC, Boca Raton (Fl, US),
ISBN-10 1-58488-116-X.
[107] Petruzzi, A. (2008), Development and Application of Methodologies for Sensitivity Analysis
and Uncertainty Evaluation of the Results of the Best Estimate System Codes Applied in
Nuclear Technology, PhD Thesis, University of Pisa.
[108] Petruzzi, A., D.G. Cacuci, F. D’Auria (2010), “Model calibration and best-estimate
prediction through experimental data assimilation: II. Application to a blowdown benchmark
experiment”, Nucl. Sci. Eng., 165, 45-100.
[109] Petruzzi, A., F. D’Auria (2014), “Uncertainties in predictions by system thermal-hydrauliccodes: The CASUALIDAD method”, 10th International Topical meeting on Nuclear
EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  133

Thermal-Hydraulics, Operation and Safety (NUTHOS-10), 14-18 December 2014 Okinawa,
Japan.
[110] Petruzzi, A., F. D’Auria (2005), “Development, qualification and use of the CIAU method
for internal assessment of uncertainty in system codes”, Nureth-11, Avignon, (F).
[111] Bovalini, R., F. D'Auria (1993), “Scaling of the accuracy of Relap5/mod2 code”, J. Nucl.
Eng. & Des. 139, pp. 187-204.
[112] Cacuci, D.G. (2006), “On the use of adjoint operators for global optimization and data
assimilation”, Lectures Notes, Workshop on Predictability, Observations, and Uncertainties
in Geosciences, Tallahassee (Fl, US), 13-15 March 2006.
[113] Pourgol-Mohammad, M., A. Mosleh, M. Modarres (2007), Integrated Methodology for
Thermal-Hydraulics Code Uncertainty Analysis, A. James Clark School of Engineering,
University of Maryland at College Park, March 2007.
[114] SEC NRS (2016), General Safety Provisions for Nuclear Power Plants, OPB-88/97; NP001-15, approved by Rostechnadzor on 17 December 2015.
[115] Gosatomnadzor (2000), Requirements to Composition and Contents of Report on
Verification and Justification of Software Applied for Nuclear Facilities Safety Analysis,RD-03-34-2000, approved by Gosatomnadzor of Russia on 28 December 2000.
[116] Bogdan, S.N. et al. (2016), “Updating of regulatory requirements to software applied atnuclear facilities safety analysis”, Rostechnadzor Council on software certification: 25 years
on safety service, Collection of publications, Works of SEC NRS. – М.: SEC NRS.
[117] Bogdan, S., O. Kovalevich, N. Kozlova, S. Shevchenko and D. Yashnikov (2017), “Aboutthe calculation error evaluation in the framework of the nuclear facilities safety analysis”,
Nuclear and Radiation Safety, No. 2 (84)-2017, Moscow.
[118] Moiseenko, E.V., A.S. Filippov (2011), “A methodology for multivariate simulation withmassively parallel computing systems for NPP safety assessment: VARIA code”, Journal ofEngineering Thermophysics, 2011, Vol. 20, No. 3, pp. 249-259.
[119] Moiseenko, E.V., A. Filippov, N. Drobyshevsky, V. Strizhov and A.E. Kisselev (2011),“VARIA − An application for mass computing and statistical analysis of the simulation
results in BEPU safety assessment”, Proceedings of ICONE-19, 24-25 October 2011, Osaka,
Japan, ICONE-43435.
[120] Petruzzi, A., F. D’Auria, J. Micaelli, A. De Crecy and J. Royen (2004), “The BEMUSE
programme (Best-Estimate Methods – Uncertainty and Sensitivity Evaluation)”,International Meeting on Best-Estimate Methods in Nuclear Installation Safety Analysis(BE-2004) IX, Washington D.C. (US), 14-18 November 2004.
[121] NEA (2006), BEMUSE Phase II Report: Re-Analysis of the ISP-13 Exercise, Post Test
Analysis of the LOFT L2-5 Test Calculation, Report NEA/CSNI/R(2006)2, www.oecdnea.org/jcms/pl_18346/.
[122] NEA (2007), BEMUSE Phase III Report, Uncertainty and Sensitivity Analysis of the LOFT
L2-5 Test, NEA/CSNI/R(2007)4, 11 October 2007, www.oecd-nea.org/jcms/pl_18442 .
[123] NEA (2011), BEMUSE Phase VI Report, Status Report on the Area, Classification of the
Methods, Conclusions and Recommendations, NEA/CSNI/R(2011)4, 28 March 2011,www.oecd-nea.org/jcms/pl_19020.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

134  NEA/NSC/R(2019)7[124] NEA (1998), Report of the Uncertainty Method Study for Advanced Best Estimate Thermalhydraulic Code Applications, OECD/NEA/CSNI R (97) 35, I and II.
[125] NEA (2016), “PREMIUM, a Benchmark on the quantification of the uncertainty of thephysical models in the system thermal-hydraulic codes: methodologies and data review”,
NEA/CSNI/R(2016)9, OECD Publishing, Paris, www.oecd-nea.org/jcms/pl_19720.
[126] Bratton, R., M. Avramova, K. Ivanov (2014), “Benchmark for uncertainty analysis inmodeling (UAM) for LWRs − Summary and discussion of neutronics cases (Phase I)”,
Nuclear Engineering and Technology (NET), Vol. 46 No. 3, pp. 313-349, Special Issue on
Neutronics Sensitivity and Uncertainty Analysis, Invited paper.
[127] NEA (1996), Integral Test Facility Validation Matrix for the Assessment of ThermalHydraulic Codes for LWR LOCA and Transients, OCDE/GD(97)12, NEA/CSNI/R(96)17,
www.oecd-nea.org/jcms/pl_16134.
[128] NEA (1994), Separate Effects Test Matrix for Thermal-Hydraulic Code Validation Volume
I, Phenomena Characterization and Selection of Facilities and Tests, OECD/GD(94)82,
OECD publishing, Paris, www.oecd-nea.org/jcms/pl_15968.
[129] NEA (1994), Separate Effects Test Matrix for Thermal-Hydraulic Code Validation Volume
II, Facilities and Experiment Characteristics; OECD/GD(94)83, OECD Publishing, Paris,
www.oecd-nea.org/jcms/pl_15968.
[130] Petruzzi, A., F. D’Auria (2016), “Standardized consolidated calculated and reference
experimental database (SCCRED): A supporting tool for V&V and uncertainty evaluationof best-estimate system codes for licensing applications”, Nuclear Science and Engineering,
182.
[131] NEA (2006), BEMUSE Phase II Report: Re-Analysis of the ISP-13 Exercise, Post Test
Analysis of the LOFT L2-5 Test Calculation, NEA/CSNI/R(2006)2, OECD Publishing,Paris, www.oecd-nea.org/jcms/pl_18346.
[132] Wang, C., S. Schunert, P. Talbot, B. Baker, Y. Wang, J. Ortensi, F. Gleicher, M. DeHart andC. Rabiti (2017), “Uncertainty and sensitivity analysis of TREAT transient test #15”,
Proceedings of the 2017 American Nuclear Society Annual meeting, San Francisco.
[133] Talbot, P., C. Wang, C. Rabiti and A. Prinja (2016), “Multistep input reduction for highdimensional uncertainty quantification in RAVEN code”, Proceeding of PHYSOR 2016.
[134] Schunert, S., C. Wang, Y. Wang, F. Kong, J. Ortensi, B. Baker, F. Gleicher, M. DeHart and
R. Martineau (2017), “A generalized perturbation theory solver in Rattlesnake based on
PETs with application to TREAT steady state uncertainty quantification”, submitted to M&C
2017.
[135] NEA (2016), Benchmark for Uncertainty Analysis in Modelling (UAM) for Design,
Operation, and Safety Analysis of LWRs, Volume 2: Specification and Support Data for theCore Cases (Phase II) − Version 2, NEA/NSC/DOC(2012)11, OECD Publishing, Paris,
www.oecd-nea.org/jcms/pl_71109.
[136] Porter, N. (2015), Sensitivity and Uncertainty Study of CTF Using the Uncertainty Analysis
in Modeling Benchmark, MS Thesis, Pennsylvania State University.
[137] Porter, N., M. Avramova, K. Ivanov (2015), “Uncertainty and sensitivity analysis of
COBRA-TF for the OECD LWR UAM benchmark using Dakota”, Proc. Int. Topical
Meeting on Nuclear Reactor Thermal Hydraulics (NURETH), Chicago, IL, 30 August ̶
4 September 2015, American Nuclear Society.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

NEA/NSC/R(2019)7  135[138] Morris, M. (1991), “Factorial sampling plans for preliminary computational experiments”,
Technometrics, 33, 161.
[139] Porter, N., M. Avramova (2016), “Uncertainty analysis of CTF for the uncertainty analysis
in modeling benchmark”, Proc. Winter Meeting and Nuc. Tech. Expo., Las Vegas, NV,
6-10 November 2016, American Nuclear Society.
[140] Porter, N., M. Avramova, V. Mousseau (2017), Uncertainty Quantification Study of CTF for
the Uncertainty Analysis in Modeling Benchmark, NCSU.
[141] Avramova, M., K. Ivanov (2017), “Approach for testing multi-physics uncertainty
propagation in reactor core calculations within the framework of OECD/NEA LWR UAM
benchmark”, Proceedings: 48th Annual Meeting on Nuclear Technology (AMNT 2017), 1617 May 2017, Berlin, Germany.
[142] Petruzzi, A., M. Cherubini, F. D’Auria, M. Lanfredini and O. Mazzantini (2016), “TheBEPU evaluation model with RELAP5-3D for the licensing of the Atucha-II NPP”, Nuclear
Technology Vol. 193, 2016 − Issue 1, pp. 113-160.
[143] Wilson, G., B. Boyack (1998), “The role of PIRT process in experiments, code development
and code applications associated with reactor safety analyses”, Nuclear Engineering and
Design, Vol. 186.
[144] Wulff, W., N. Zuber, U. Rohatgi, I. Catton (2009), “Application of Fractional Scaling
Analysis (FSA) to loss of coolant accidents (LOCA); system level scaling for systemdepressurization”, ASME Journal Fluids Engineering, August 2009, Vol. 131.
[145] Luo, H. (2012), Quantified PIRT and Uncertainty Quantification for Computer Code
Validation, Ph.D thesis, Oregon State University.
[146] Shaw, R.A., S.Z. Rouhani, T.K. Larson and R.A. Dimenna (1988), Development of a
Phenomena Identification and Ranking Table (PIRT) for Thermal-Hydraulic Phenomenaduring a PWR Large-Break LOCA, NUREG/CR-5074, USNRC, August 1988.
[147] NRC (1989), Best-Estimate Calculations of Emergency Core Cooling System Performance,
Regulatory Guide 1.157.
[148] NRC (2005), Transient and Accident Analysis Methods, Regulatory Guide 1.203.
[149] Boyack, B. et al. (1991), An Integrated Structure and Scaling Methodology for Severe
Accident Technical Issue Resolution, Draft NUREG/CR-5809, USNRC, November 1991.
[150] Zuber, N. et al. (1998), “An integrated structure and scaling methodology for severe accident
technical issue resolution: Development of methodology”, Nuclear Engineering and Design,
186 (pp. 1-21).
[151] Wilson, G., B.E. Boyack (1998), “The role of the PIRT process in experiments, codedevelopment and code applications associated with reactor safety analysis”, Nuclear
Engineering and Design, 186 (pp. 23-37).
[152] Wulff, W. (1994), Uncertainty Analysis of Suppression Pool Heating during an ATWS in a
BWR-5 Plant, NUREG/CR-6200, USNRC, March 1994.
[153] Watkins, J.C., L.S. Ghan (1992), AHP Version 5.1, User’s Manual, EGG-ERTP-10585,
Idaho National Engineering Laboratory, October 1992.
[154] Holmström, H., H. Glaeser, A. Wickett and G. Wilson (1994), “Status of code uncertainty
evaluation methodologies”, Proceedings of the International Conference on New Trends in

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2

136  NEA/NSC/R(2019)7
Nuclear System Thermohydraulics, Dipartimento di Costruzioni Meccaniche Nucleari, Pisa,
Italy.
[155] Reyes, J., L. Hochreiter (1998), “Scaling analysis for the OSU AP600 test facility (APEX)”,
Nuclear Engineering and Design, 186 (pp. 53-109), 1 November 1998.
[156] Banerjee, S., M. Ortiz, T. Larson and D. Reeder (1998), “Scaling in the safety of next
generation reactors”, Nuclear Engineering and Design, 186 (pp. 111-133), 1 November
1998.
[157] Ransom, V., W. Wang and M. Ishii (1998), “Use of an ideal scaled model for scalingevaluation”, Nuclear Engineering and Design, 186 (pp. 135-148), 1 November 1998.
[158] Ishii, M. et al. (1998), “The three-level scaling approach with application to the Purdue
University Multi-Dimensional Integral Test Assembly (PUMA)”, Nuclear Engineering and
Design, 186 (pp. 177-211), 1 November 1998.
[159] Fletcher, C.D. et al. (1997), Adequacy Evaluation of RELAP5/MOD3, Version 3.2.1.2 for
Simulating AP600 Small Break Loss-of-Coolant Accidents, INEL-96/0400 (nonproprietaryversion), April 1997.

EXISTING PRACTICES FOR MULTI-PHYSICS VALIDATION: REPORT ON SUB-TASK 3, TASK FORCE 2