CREDIBILITY ASSESSMENT 

FRAMEWORK FOR

CRITICAL BOILING

TRANSITION MODELS

A Generic Safety Case to Determine 

the Credibility of Critical Heat Flux 

and Critical Power Models

Draft Report for Comment

3

COMMENTS ON DRAFT REPORT

Any interested party may submit comments on this report for consideration by the NRC staff. 

Comments may be accompanied by additional relevant information or supporting data. Please 

specify the report number NUREG/KM-0013 in your comments, and send them by the end of the 

comment period specified in the Federal Register notice announcing the availability of this report. 

Addresses: You may submit comments by any one of the following methods. Please include 

Docket ID NRC-2019-0043 in the subject line of your comments. Comments submitted in writing 

or in electronic form will be posted on the NRC website and on the Federal rulemaking website 

http://www.regulations.gov. 

Federal Rulemaking Website: Go to http://www.regulations.gov and search for documents filed 

under Docket ID NRC-2019-0043. 

Mail comments to: Office of Administration, Mail Stop: TWFN-7-A60M, U.S. Nuclear 

Regulatory Commission, Washington, DC 20555-0001, ATTN: Program Management, 

Announcements and Editing Staff.

For any questions about the material in this report, please contact: Joshua Kaizer, Reactor 

Engineer, 301-415-1532 or by e-mail at Joshua.Kaizer@nrc.gov.

Please be aware that any comments that you submit to the NRC will be considered a public 

record and entered into the Agencywide Documents Access and Management System 

(ADAMS). Do not provide information you would not want to be publicly available.

NUREG/KM-0013

Credibility Assessment 

Framework for Critical Boiling 

Transition Models

A generic safety case to determine the 

credibility of critical heat flux and critical 

power models

Office of Nuclear Reactor Regulation

Manuscript Completed:

Date Published:

Prepared by:

J.S. Kaizer

R. Anzalone

E. Brown

M. Panicker

S. Haider

J. Gilmer

T. Drzewiecki

A. Attard (retired, unable to comment on final version)

U.S. Nuclear Regulatory Commission

iii

1 ABSTRACT

2 Critical boiling transition (CBT) occurs when a flow regime that has a higher heat transfer rate 

3 transitions to a flow regime that has a significantly lower heat transfer rate. Models that predict a

4 CBT are a necessary part of reactor safety analysis because they are used to determine plant 

5 safety limits. Therefore, the review of CBT models has been a focus of the U.S. Nuclear 

6 Regulatory Commission (NRC) since its inception in 1975. 

7 This work presents a generic safety case in the form of a credibility assessment framework that

8 combines aspects of goal structuring notation and maturity assessment. This framework is 

9 focused on the credibility assessment of CBT models with specific application to reactor safety 

10 analysis. The NRC has performed many such assessments and has generated this framework

11 based on the experience of current and former NRC staff, as well as previous staff reviews as 

12 summarized in staff evaluations. This document includes a survey of the important technical and 

13 regulatory literature; a detailed technical discussion of CBT models and their application; and a

14 suggested framework for CBT models. This NUREG/KM summarizes the knowledge the NRC

15 staff has developed over the course of 40 years of CBT model and analysis reviews.
v

1
vi

1 TABLE OF CONTENTS

2 ABSTRACT...............................................................................................................................iii

3 TABLE OF CONTENTS ............................................................................................................ v

4 LIST OF FIGURES ...................................................................................................................vii

5 LIST OF TABLES......................................................................................................................ix

6 ACKNOWLEDGMENTS............................................................................................................xi

7 ABBREVIATIONS AND ACRONYMS .....................................................................................xiii

8 1 INTRODUCTION................................................................................................................... 1

9 Why Use the Term “Critical Boiling Transition”?.................................................................2

10 What Is Credibility?.............................................................................................................2

11 What Is a Credibility Assessment Framework? ..................................................................3

12 Credibility Assessment Framework for Critical Boiling Transition Models ..........................6

13 2 BACKGROUND ON CRITICAL BOILING TRANSITION...................................................... 7

14 Literature Survey ...............................................................................................................7

15 2.1.1 Technical References ...........................................................................................7

16 2.1.2 Regulatory References .......................................................................................12

17 Critical Boiling Transition Phenomena.............................................................................15

18 2.2.1 Departure from Nucleate Boiling.........................................................................15

19 2.2.2 Dryout .................................................................................................................16

20 2.2.3 Other Flow Regimes and Transitions..................................................................16

21 Determining When Critical Boiling Transition Occurs ......................................................16

22 2.3.1 Critical Heat Flux Models ....................................................................................17

23 2.3.2 Critical Power Models .........................................................................................17

24 2.3.3 Semi-empirical Modeling.....................................................................................17

25 2.3.4 Conservative vs. Non-Conservative Predictions.................................................18

26 Applying a Critical Boiling Transition Model.....................................................................18

27 2.4.1 Applying a Critical Boiling Transition Model in a Pressurized-Water 

28 Reactor..............................................................................................................18

29 2.4.2 Applying a Critical Boiling Transition Model in a Boiling-Water Reactor.............20

30 2.4.3 Applying a Steady-State Model to Transient Conditions.....................................21

31 Addressing Uncertainties and Errors ...............................................................................21

32 3 CREDIBILITY ASSESSMENT FRAMEWORK ....................................................................23

33 G1—Experimental Data...................................................................................................24

34 3.1.1 G1.1—Credible Test Facility...............................................................................24

35 3.1.2 G1.2—Accurate Measurements .........................................................................27

36 3.1.3 G1.3—Reproduction of Local Conditions............................................................38

37 G2—Model Generation....................................................................................................46

38 3.2.1 G2.1—The Mathematical Form ..........................................................................47

39 3.2.2 G2.2—Method for Determining Coefficients .......................................................53

40 G3—Validation through Error Quantification ...................................................................56

41 3.3.1 G3.1—Calculating Validation Error.....................................................................57

42 3.3.2 G3.2—Data Distribution in the Application Domain ............................................59

43 3.3.3 G3.3—Inconsistency in the Validation Error .......................................................68

44 3.3.4 G3.4—Calculating Model Uncertainty.................................................................73

45 3.3.5 G3.5—Model Implementation .............................................................................77

46 4 SUMMARY AND CONCLUSION.........................................................................................83

47 5 REFERENCES ....................................................................................................................85

48 APPENDIX A LISTING OF ALL GOALS...........................................................................A-1

v
vii

1

2
viii

1 LIST OF FIGURES

2 Figure 1 Goals............................................................................................................................. 4

3 Figure 2 Framework .................................................................................................................... 5

4 Figure 3 Decomposition of G — Main Goal............................................................................... 23

5 Figure 4 Decomposition of G1—Experimental Data.................................................................. 24

6 Figure 5 Decomposition of G1.1—Credible Test Facility........................................................... 25

7 Figure 6 Decomposition of G1.2—Accurate Measurements ..................................................... 28

8 Figure 7 Decomposition of G1.3—Reproduction of Local Conditions ....................................... 38

9 Figure 8 Decomposition of G2—Model Generation................................................................... 47

10 Figure 9 Decomposition of G2.1—The Mathematical Form ...................................................... 47

11 Figure 10 Decomposition of G2.2—Method for Determining Coefficients................................... 53

12 Figure 11 Decomposition of G3—Validation through Error Quantification .................................. 57

13 Figure 12 Regions in the Application Domain ............................................................................. 60

14 Figure 13 Decomposition of G3.2—Data Distribution in the Application Domain........................ 62

15 Figure 14 Decomposition of G3.3—Inconsistencies in the Validation Error................................ 69

16 Figure 15 Decomposition of G3.4—Quantification of the Model’s Error...................................... 75

17 Figure 16 Decomposition of G3.5—Model Implementation......................................................... 78

18

vii
ix

1
x

1 LIST OF TABLES

2 Table 1 Key Textbooks for the Review of CBT Models ............................................................... 7

3 Table 2 Key Papers for the Review of CBT Models .................................................................... 8

4 Table 3 Industry Reports Associated with CBT Models for PWRs.............................................. 9

5 Table 4 Industry Reports Associated with CBT Models for BWRs............................................ 11

6 Table 5 Regulatory References Associated with CBT Models .................................................. 13

7 Table 6 Evidence for G1.1.1—Test Facility Description ............................................................ 25

8 Table 7 Evidence for G1.1.2—Test Facility Comparison........................................................... 27

9 Table 8 Experimental Parameters Measured or Controlled ...................................................... 28

10 Table 9 Evidence for G1.2.1—Test Facility QA Program.......................................................... 30

11 Table 10 Evidence for G1.2.2—Statistical Design of Experiment ............................................... 31

12 Table 11 Evidence for G1.2.3—Data Fidelity .............................................................................. 34

13 Table 12 Evidence for G1.2.4—Instrumentation Uncertainty Impact .......................................... 35

14 Table 13 Evidence for G1.2.5—Repeated Test Points................................................................ 36

15 Table 14 Evidence for G1.2.6—Quantified Heat Losses............................................................. 37

16 Table 15 Evidence for G1.3.1—Equivalent Geometric Dimensions............................................ 39

17 Table 16 Evidence for G1.3.2—Prototypical Grid Spacers ......................................................... 41

18 Table 17 Evidence for G1.3.3—Axial Power Shapes.................................................................. 43

19 Table 18 Evidence for G1.3.4—Radial Power Peaking (PWR)................................................... 44

20 Table 19 Evidence for G1.3.4—Radial Power Peaking (BWR)................................................... 45

21 Table 20 Evidence for G1.3.5—Differences in the Test Assembly.............................................. 46

22 Table 21 Evidence for G2.1.1—Necessary Parameters ............................................................. 51

23 Table 22 Evidence for G2.1.2—Reasoning for the Mathematical Form...................................... 52

24 Table 23 Evidence for G2.2.1—Identification of Training Data ................................................... 54

25 Table 24 Evidence for G2.2.2—Calculation of the Model’s Coefficients ..................................... 55

26 Table 25 Evidence for G2.2.3—Calculation of Model-Specific Factors and Constants............... 56

27 Table 26 Evidence for G3.1—Calculating Validation Error.......................................................... 59

28 Table 27 Evidence for G3.2.1—Identification of Validation Data................................................. 63

29 Table 28 Evidence for G3.2.2—Defining the Application Domain............................................... 64

30 Table 29 Evidence for G3.2.3—Understanding the Expected Domain ....................................... 65

31 Table 30 Evidence for G3.2.4—Validation Error Data Density in the Expected Domain............. 66

32 Table 31 Evidence for G3.2.5—Sparse Regions......................................................................... 67

33 Table 32 Evidence for G3.2.6—Restricted to the Application Domain........................................ 68

34 Table 33 Evidence for G3.3.1—Identifying Non-poolable Data Sets........................................... 71

35 Table 34 Evidence for G3.3.2—Identifying Non-conservative Subregions.................................. 72

36 Table 35 Evidence for G3.3.3—Appropriate Trends ................................................................... 73

37 Table 36 Evidence for G3.4.1—Error Database.......................................................................... 75

38 Table 37 Evidence for G3.4.2—Validation Error Statistics .......................................................... 76

39 Table 38 Evidence for G3.4.3—Model Uncertainty Bias ............................................................. 77

40 Table 39 Evidence for G3.5.1—Same Computer Code .............................................................. 78

41 Table 40 Evidence for G3.5.2—Same Evaluation Methodology ................................................. 79

42 Table 41 Evidence for G3.5.3—Transient Prediction .................................................................. 80

43

ix
x

1
xii

1 ACKNOWLEDGMENTS

2 Frameworks, such as presented in this paper, are the result of tremendous effort by numerous

3 individuals. While these individuals and their technical contributions are too numerous to list, the 

4 authors offer special thanks to Robert Weisman and Julie Ezell for their legal review and advice

5 which resulted in significant improvement to the document. 

xi
x

1
xiv

1 ABBREVIATIONS AND ACRONYMS

2 1-D one-dimensional

3 2-D two-dimensional

4 3-D three-dimensional

5 AOO anticipated operational occurrence

6 ASME American Society of Mechanical Engineers

7 BWR boiling-water reactor

8 CBT critical boiling transition

9 CFR Code of Federal Regulations

10 CHF critical heat flux

11 CP critical power

12 DNB departure from nucleate boiling

13 DNBR departure from nucleate boiling ratio

14 G Goal

15 GSN goal structuring notation

16 LOCA loss-of-coolant accident

17 M&S modeling and simulation

18 MDNBR minimum departure from nucleate boiling ratio

19 NRC U.S. Nuclear Regulatory Commission

20 PCT peak cladding temperature

21 PWR pressurized-water reactor

22 R- or K-factor relative power factor

23 SAFDL specified acceptable fuel design limit

24 SLMCPR safety limit minimum critical power ratio

25 SRP Standard Review Plan

26 SSC systems, structures, and components

27 V&V verification and validation

xiii

1

1 1 INTRODUCTION

Critical boiling transition1 2 (CBT) is defined as a transition from a boiling flow regime that has a 

3 higher heat transfer rate to a flow regime that has a significantly lower heat transfer rate. For

4 scenarios in which the heat transfer is controlled by the heat flux (such as in nuclear fuel 

5 assembly), the reduction in heat transfer rate caused by the CBT results in an increase in the 

6 surface temperature in order to maintain the heat flux. If the reduction in the heat transfer rate and 

7 resulting increase in surface temperature is large enough, the surface may weaken or melt. In a 

8 nuclear power plant, this cladding softening or melting is considered fuel damage. 

9 To ensure that the fuel is not damaged during normal operation or anticipated operational 

10 occurrences (AOOs), computer simulations of the fuel are performed to predict the 

11 thermal-hydraulic conditions that would occur in the fuel assemblies during various scenarios. The 

resulting thermal-hydraulic conditions are then input to a CBT model.

2 12 That CBT model predicts

13 the power which is required for a CBT to occur at the given thermal-hydraulic conditions. Hence 

14 the margin to CBT can be obtained by comparing the current power at the specific location in the 

15 fuel assembly to the power at which CBT occurs at the same thermal-hydraulic conditions. The 

16 U.S. Nuclear Regulatory Commission (NRC) has historically accepted that one way to 

17 demonstrate the avoidance of fuel damage during all normal operation and AOOs is to 

18 demonstrate that there is margin to a CBT. 

19 Because of the importance of CBT models, a major focus in reactor safety analysis is to 

20 determine whether the proposed models can correctly predict CBT. The NRC has reviewed many

21 CBT models over the years and has documented why each model was found acceptable (i.e., 

22 able to correctly predict CBT) in the corresponding safety evaluation. The authors of this 

23 document have used those safety evaluations along with their own expertise to produce a 

24 framework for assessing the credibility of CBT models. 

25 This document includes two main sections. The first section contains a brief background of 

26 literature relevant to the assessment of CBT models followed by a discussion of the CBT 

27 phenomena and how such phenomena are commonly modeled. The second section describes 

28 the development of the credibility assessment framework for CBT models and provides detailed 

aspects of that framework as well as the evidence3 29 commonly used to demonstrate that the 

30 criteria in the framework have been satisfied. In total, this document is meant to act as a textbook 

31 for those interested in the assessment of CBT models.

 1 Many terms have been used to describe these models, including critical heat flux, critical power, 

critical quality versus boiling length, departure from nucleate boiling, dryout, burnout, and flow 

boiling crisis. 

2 Historically, the models are commonly referred to as correlations because they correlate the CBT 

phenomenon to other variables in the flow field. However, the term “correlation” has a very 

specific meaning in statistics; therefore, this document will refer to them as “models.” 

3 Evidence as used throughout this document is not intended to mean the rules and legal principles 

that govern the proof of facts in a legal proceeding. Rather, as used in this document, “evidence” is 

the available body of facts or information indicating whether a belief or proposition is true or valid.
2

1 Why Use the Term “Critical Boiling Transition”?

2 Hewitt and Hall-Taylor (1970) discussed a wide range of terms used to describe the phenomenon 

3 associated with dryout and critical heat flux (CHF). They noted that the “large diversity of terms 

4 tends to be confusing and this diversity reflects a continuing search for a term which is both 

5 descriptive and scientifically accurate.” They analyzed the most common terms used (burnout, 

6 departure from nucleate boiling (DNB), dryout, and CHF); recognized that each term had its own 

7 inadequacies and merits; and chose “burnout” as the least unsatisfactory term. Unfortunately, the 

8 current literature on the subject does not reflect their choice, which seems to have settled mostly 

9 on the term “critical heat flux,” although dryout and DNB are still commonly used.

10 Although CHF is technically independent of any specific phenomena, it is very closely tied to the 

11 phenomena of DNB, which occurs when nucleate boiling becomes inadequate to transfer the heat 

12 at the fuel surface to the coolant. At that point, the boiling regime begins to depart from nucleate 

13 boiling and begins “transition boiling,” which is the boiling regime between nucleate boiling and 

14 film boiling. The close association between CHF and the phenomenon of DNB is likely due to the 

15 fact that CHF is the quantity used to determine whether DNB will occur in a pressurized-water 

16 reactor (PWR). However, CHF is typically not the quantity used to determine whether dryout (i.e., 

17 the drying out of the thin annular film in contact with the fuel cladding) has occurred in a 

18 boiling-water reactor (BWR). Additionally, the heat flux that causes a phenomenon to occur (i.e., 

19 the CHF) is different from the phenomenon itself. In technical discussions, the authors found it 

20 necessary to separate the phenomenon from any quantity associated with it.

21 Even considering all of these arguments, the authors of this document, like Hewitt and Hall-Taylor, 

22 were hesitant to introduce new terminology and initially decided to use the common term “critical 

23 heat flux.” However, as the discussion became more detailed and finer distinctions were 

24 necessary, the authors reluctantly decided that a different term was necessary and could not be 

25 avoided. Therefore, the authors chose to use the term “Critical boiling transition,” because it better 

26 describes the pertinent phenomena and allows for the necessary distinctions. Because CBT is a 

new term, we repeat its definition here: CBT4 27 is defined as a transition from a boiling flow regime 

28 that has a higher heat transfer rate to a flow regime that has a significantly lower heat transfer 

29 rate.

30 What Is Credibility?

31 The term “credibility” has seen wide application in the modeling and simulation (M&S) community, 

32 specifically in the areas focusing on Verification and Validation (V&V). However, the term is often 

33 left undefined. The American Society of Mechanical Engineering’s (ASME) V&V 10 Guide for 

34 Verification and Validation in Computational Solid Mechanics (2006) did not formally define the 

35 term, but did equate it to “trustworthiness.” Initially, NASA (2008) discussed the term, but 

36 purposefully chose not to define it and instead relied on “the usual sense of the English language.” 

37 Later, they defined the term as “the quality to elicit belief or trust in modeling and simulation

results” (NASA 2008B 38 ). Oberkampf and Roy (2010) do provide a definition for credibility of 

39 computational results – “results of an analysis that are worthy of belief or confidence,” but this 

40 definition is not much more detailed than ASME’s connection between credibility and 

41 trustworthiness. While credibility is intimately linked with trust, the component which is missing 

42 from these definitions is how much trust is needed in the specific use of the model. Therefore, the 

43 authors of this work have chosen to use a definition based on the work of Kaizer et al., (2015) 

 4 While CBTs can exist on other surfaces, this work is concerned only with fuel rods used in light water 

nuclear power plants. 
3

1 which captures the underlying link to trustworthiness, but maintains awareness of the necessity

2 to make a decision. 

3 Credibility is defined as the determination that an object (in this particular instance, a model) can 

4 be trusted for its intended purpose. As defined, this is a binary determination. Thus, an object is 

5 either deemed credible (i.e., can be trusted for its intended purpose) or not credible (i.e., cannot 

6 be trusted for its intended purpose). There are two interesting consequences from this definition of 

7 credibility. First, there is no middle ground, all objects must either be credible or not credible. 

8 Second, there is no “degree” of credibility. That is, by definition one object cannot be more 

9 credible than another. The authors fully acknowledge that some objects may certainly be more 

10 trusted than other objects. For example, one individual may have more experience and therefore 

11 be more trusted than another individual, or one simulation may be very well vetted and therefore 

12 be more trusted than another simulation. However, the credibility of those objects is defined to be 

13 binary (i.e., credible, not credible) because decisions themselves are binary (i.e., yes or no). 

14 What Is a Credibility Assessment Framework?

15 A credibility assessment framework provides a means to assess whether an object can be trusted 

16 for its intended purpose. Such a framework can be thought of as one form of a safety case. A 

17 safety case is defined as “a structured argument, supported by a body of evidence that provides a 

18 compelling, comprehensible, and valid case that a system is safe for a given application in a given 

operating environment.”5 19 Although various ways exist to provide a safety case (e.g., every safety 

20 evaluation produced by the NRC can be thought of as the documentation of a safety case or 

21 collection of safety cases), this document makes use of concepts formalized in goal structuring

22 notation (GSN). GSN (GSN Working Group, 2011) “is a graphical argumentation notation that 

23 can be used to document explicitly the individual elements of any argument (claims, evidence,

24 and contextual information) and, perhaps more significantly, the relationships that exist between 

25 these elements (i.e., how claims are supported by other claims, and ultimately by evidence, and 

26 the context that is defined for the argument).” See Denney et al. (2011) for an example of GSN.

27 The framework presented here combines the logic structure of GSN with the evaluation aspects 

28 of maturity assessment. Maturity assessment (Kaizer et al., 2015) is focused on measuring how 

29 “mature” an object is in specific attributes compared to its possible minimum and maximum 

30 amount of maturity in those attributes. Maturity assessment frameworks, such as the Predictive 

Capability Maturity Model (Oberkampf et al., 2007) and NASA-STD-7009 (NASA 2008B 31 ), focus 

32 on the evidence that is available and is a means to rank that evidence in a manner useful to a 

33 decision maker. For a more detailed description of a maturity assessment and its history, see 

34 Oberkampf and Roy (2010). 

35 The credibility assessment framework used in this document is unique in that it combines these 

36 two concepts by using the logical structure of goals from GSN and the evaluation of the possible 

37 evidence from maturity assessment. The framework is generated from a single main goal. That 

38 main goal is then logically decomposed into subgoals. By logical decomposition, we mean the 

39 act of generating a set of sub-goals which are logically equivalent to the original goal (i.e., 

 5 This document uses the definition provided by the United Kingdom’s Ministry of Defense (2007). Other 

U.S. government agencies which have made use of this concept include NASA (2015) and the FDA 

(2014). The authors’ use of the UK Ministry of Defense definition in this document does not imply 

USNRC approval of regulatory principles or approaches employed in the UK, nor should the use of the 

definition be understood to be an NRC endorsement of such principles or approaches as acceptable for 

use in the US. 
4

1 necessary and sufficient for the original goal to be met). This decomposition is expressed using 

2 GSN notation. Each subgoal can either be further logically decomposed into other subgoals, or 

3 if no further decomposition is deemed useful, the subgoal may be considered a base goal and 

4 evidence must be provided to demonstrate that the base goal is true. The evidence which is 

5 commonly provided is given in a maturity table, where it is ranked from least to most mature. A 

6 simple example to illustrate the logic is given below. 

7 The main goal (G) is written as a conclusion, such as “G - It is safe to drive over the bridge.” 

8 Notice that this goal is somewhat ambiguous. What is meant by “safe”? While there is common

9 agreement that it should be “safe” to drive over a bridge, there is disagreement as to what “safe”

10 means in this instance. Such ambiguity is often encountered, but frameworks such as the one 

11 provided in this document can be used to define what these ambiguous terms (such as “safe”) 

12 mean in practice. 

13 The main goal, G, is then logically decomposed into a set of sub-goals, where each sub-goal must 

14 be necessary (i.e., if the sub-goal is false, the main goal must also be false) and the set of sub15 goals must be sufficient (i.e., if the set of sub-goals is true, the main goal must also be true) to 

16 demonstrate that the main goal is true. This simple example has two subgoals: (1) “The bridge 

17 can withstand the weight of my car.” and (2) “There will not be a natural disaster while I am driving 

18 over the bridge.” These goals are given in Figure 1 below. 

19

20 Figure 1 Goals

21 Each subgoal (e.g., G1 and G2) must either be further decomposed into additional sub-goals, or 

22 evidence provided to determine if those sub-goals could be considered true. For this example, no 

23 further decomposition was considered. Potential levels of evidence that could be provided to 

24 demonstrate that each subgoal is true (i.e., has been met) are given in Figure 2 below. 
5

1

2 Figure 2 Framework

3 The evidence provided is the justification for concluding that the specific base goal is true (i.e., has 

4 been met). This evidence is ranked from least to most mature, or providing the least certain 

5 justification that the base goal is met to the most certain justification. With higher levels of 

6 evidence (e.g., level 3 as opposed to level 1), we can be more certain that the associated base 

7 goal is true. Thus, an individual driving over a bridge on his or her daily commute would likely 

8 require a very low level of evidence to determine the bridge is credible (i.e., safe to drive across). 

9 In all likelihood, the individual may not even consciously think about the credibility of the bridge, or 

10 if he or she did, the individual would likely rely on low levels of evidence. However, if the bridge 

11 were used to transport heavy haul freight (i.e., oversized loads), a much higher level of evidence 

12 would likely be required before the bridge was deemed credible. 

13 The specific pieces of evidence which are considered by this framework are given in Figure 2. If 

14 any other evidence (i.e., levels of G1 or G2) are used to demonstrate that the associated goal is 

15 true, that evidence should be placed in its appropriate rank in the table. Thus, one could argue 

16 that seeing another car drive over the bridge right before they do is evidence that the bridge can 

17 withstand the weight of their own car. If this evidence is going to be used, it should be ranked 

18 according to the other evidence already in the table (likely falling between levels 2 and 3 and 

19 requiring a re-numbering of the table). 

20 Notice that the ambiguity of the word “safe” in the main goal G has now been removed. That is, by 

21 saying “It is safe to drive over the bridge,” we have not only defined safe as meaning G1 and G2

22 are true, but we would also state what evidence was given (e.g., Level 3 for G1 and Level 2 for 

23 G2). Thus, the ambiguous word “safe” is explicitly defined using the framework. 

24 Additionally, anything not specified in the framework was not considered in determining credibility. 

25 Because the framework explicitly establishes the assumptions underlying an assessment, it can 

26 be helpful when identifying any areas that may need further consideration (that is, additional sub-
6

1 goals or evidence levels). For example, an individual could argue that our sample framework lacks 

2 a sub-goal that accounts for the driving ability of other drivers on the bridge. Another may argue 

3 that our first sub-goal should not only consider the weight of our car, but all other vehicles on the 

4 bridge at the same time. One of the largest advantages to these frameworks is that others can 

5 quickly and easily determine what was and what was not considered. Further, the framework 

6 could be updated quickly and easily to account for any omissions. 

7 Credibility Assessment Framework for Critical Boiling Transition Models

8 The credibility assessment framework presented in this work is focused on critical boiling 

9 transition models. While this framework was generated based on the NRC staff’s experience 

10 reviewing these models, the framework itself is more broadly applicable to any use of any CBT 

11 model. This includes the entire spectrum of possible uses from something as simple as a 

12 homework problem to something as significant as reactor safety analysis, and all uses in between.

13 It is important to remember that the appropriate evidence level will change based on the model’s 

14 intended use. Thus, the level of evidence appropriate for reactor safety analysis will likely be 

15 much higher than that which is appropriate for a homework problem. 

16 As this framework is applicable to any use of a CBT model (including, but not limited to, reactor 

17 safety analysis), the authors have chosen to use a broader terminology when describing the 

18 details of the framework as it can be applied to determining credibility. The process of determining 

credibility involves two distinct roles: the analyst and the assessor.

6 19 It is the role of the analyst to 

20 generate the model, gather the evidence, and present the argument that the model can be 

21 trusted. It is the role of the assessor to determine if the evidence presented is sufficient to justify

22 that the model can be trusted for its intended purpose. In regulatory environments, these roles are 

23 usually filled by separate individuals from different organizations, the analyst being the applicant 

24 and the assessor being the regulatory agency staff member (e.g., at the NRC, this role is typically 

25 called a reviewer). However, in other environments both roles could be performed by individuals 

26 from the same organization (i.e., internal peer review), and in some cases could be performed by 

27 the same individual (e.g., a homework problem). 

 6 The ‘assessor’ is not a reference to a specific role as defined by other national or international 

organizations. Instead, the word was chosen solely based on the fact that the person who applies the 

credibility assessment framework is making an assessment, and is therefore an assessor. 
7

1 2 BACKGROUND ON CRITICAL BOILING TRANSITION

2 Literature Survey

3 This section provides a literature survey of the references considered important for the NRC

4 review of CBT models. Many references associated with CBT phenomena exist; however, the 

5 following are of special interest because they are commonly cited in discussions of the models 

6 used in nuclear power reactors. For convenience, the references have been separated into 

7 technical references (i.e., textbooks, articles, and industry reports) and regulatory references.

8 2.1.1 Technical References

9 Tables 1, 2, 3, and 4 list the key technical references for CBT models. 

10 Table 1 Key Textbooks for the Review of CBT Models

Author Title Date

Hewitt and 

Hall-Taylor Annular Two-Phase Flow 1970

Tong Boiling Crisis and Critical Heat Flux 1972

Todreas 

and Kazimi Nuclear Systems I: Thermal Hydraulic Fundamentals 1990

Lahey and 

Moody

The Thermal Hydraulics of a Boiling Water Nuclear 

Reactor 1993

Tong and 

Tang Boiling Heat Transfer and Two-Phase Flow 1997

11
8

1 Table 2 Key Papers for the Review of CBT Models

Author Title Date

Leidenfrost On the Fixation of Water in Diverse Fire 1756 

(1966)

Tong et al. Influence of Axially Nonuniform Heat Flux on DNB 1965

Macbeth An Appraisal of Forced Convection Burnout Data 1965–

1966

Barnett

A Correlation of Burnout Data for Uniformly Heated 

Annuli and Its Uses for Predicting Burnout in 

Uniformly Heated Rod Bundles

1966

Healzer et

al.

Design Basis for Critical Heat Flux Condition in 

Boiling Water Reactors 1966

Tong Prediction of Departure from Nucleate Boiling for an 

Axially Non-Uniform Heat Flux Distribution 1967

Biasi et al.

Studies on Burnout: Part 3—A New Correlation for 

Round Ducts and Uniform Heating and Its 

Comparison with World Data

1967

Gellerstedt 

et al.

Correlation of Critical Heat Flux in a Bundle Cooled 

by Pressurized Water 1969

Hughes A Correlation of Rod Bundle Critical Heat Flux for 

Water in the Pressure Range 150 to 725 psia 1970

Piepel and 

Cuta

Statistical Concepts and Techniques for Developing, 

Evaluating, and Validating CHF Models and 

Corresponding Fuel Design Limits

1993

Groeneveld The 2006 CHF Look-Up Table 2007

Yang et al. Uniform versus Nonuniform Axial Power Distribution 

in Rod Bundle CHF Experiments 2014

Kaizer

Identification of Nonconservative Subregions in 

Empirical Models Demonstrated Using Critical Heat 

Flux Models

2015

Groeneveld CHF Data Used to Generate 2006 Groeneveld CHF 

Lookup Tables 2016

2
9

1 Table 3 Industry Reports Associated with CBT Models for PWRs

CBT Model Title Date

B&W-2 Correlation of Critical Heat Flux in a Bundle Cooled 

by Pressurized Water 1970

CE-1

C-E [Combustion Engineering] Critical Heat Flux: 

Critical Heat Flux Correlation for C-E Fuel 

Assemblies with Standard Spacer Grids, Part 1—

Uniform Axial Power Distribution

1976

XNB DNB Exxon Nuclear DNB Correlation for PWR Fuel 

Designs 1983

WRB-1

New Westinghouse Correlation WRB-1 for 

Predicting Critical Heat Flux in Rod Bundles with 

Mixing Vane Grids

1984

WRB-2 VANTAGE 5H Fuel Assembly 1985

CE-1 

(modified)

C-E Critical Heat Flux: Critical Heat Flux Correlation 

for C-E Fuel Assemblies with Standard Spacer 

Grids, Part 2—Non-Uniform Axial Power Distribution

1984

ANFP DNB Departure from Nucleate Boiling Correlation for High 

Thermal Performance Fuel 1990

BWU The BWU Critical Heat Flux Correlations 1996

WRB-2M

Modified WRB-2 Correlation, WRB-2M, for 

Predicting Critical Heat Flux in 17x17 Rod Bundles 

with Modified LPD Mixing Vane Grids

1999

BWU 

Addendum 1

The BWU Critical Heat FIux Correlations 

Applications to the Mark-B11 and Mark-BW17 

MSM Designs

2000

BWU 

Addendum 2

Application of BWU-Z CHF Correlation to the 

Mark-BW 17 Fuel Design with Mid-Span Mixing 

Grids

2002

ABB-NV and

ABB-TV

Addendum 1 to WCAP-1 4565-P-A Qualification of 

ABB Critical Heat Flux Correlations with VIPRE-01 

Code

2004

HTP Departure from Nucleate Boiling Correlation for High 

Thermal Performance Fuel 2005

BHTP BHTP DNB Correlation Applied with LYNXT 2005

BWU 

Addendum 3

The BWU-B11R CHF Correlation for the Mark-B11 

Spacer Grid 2005

WSSV and 

WSSV-T

Westinghouse Correlations WSSV and WSSV-T for 

Predicting Critical Heat Flux in Rod Bundles with 

Side Supported Mixing Vanes

2007

ACH-2 The ACH-2 CHF Correlation for the U.S. EPR 2007
10

CBT Model Title Date

ABB-NV 

(extended) 

and WLOP

Addendum 2 to WCAP-14565-P-A Extended 

Application of ABB-NV Correlation and Modified 

ABB-NV Correlation WLOP for PWR Low Pressure 

Applications

2008

WNG-1

Westinghouse Next Generation Correlation 

(WNG-1) for Predicting Critical Heat Flux in Rod 

Bundles with Split Vane Mixing Grids

2010

WRB-1 and 

WRB-2 Thermal Design Methodology 2013

KCE-1 KCE-1 Critical Heat Flux Correlation for PLUS7 

Thermal Design 2012

ORFEO The ORFEO-GAIA and ORFEO-NMGRID Critical 

Heat Flux Correlations 2016

1
11

1 Table 4 Industry Reports Associated with CBT Models for BWRs

CBT Model Title Date

GE transient

CHF

Loss-of-Coolant Accident and Emergency Core 

Cooling Models for General Electric Boiling 

Water Reactors

1971

GEXL General Electric Thermal Analysis Basis Data, 

Correlation and Design Application 1977

ANFB ANFB Critical Power Correlation 1990

R-Factors R-Factor Calculation Method for GE11, GE12, 

and GE13 Fuel 1999

D2 10x10 SVEA Fuel Critical Power Experiments 

and CPR Correlations: SVEA-96+ 1999

D1 10x10 SVEA Fuel Critical Power Experiments 

and CPR Correlations: SVEA-96 2000

GEXL96 GEXL96 Correlation for ATRIUM-9B Fuel 2001

GEXL10 GEXL10 Correlation for GE12 Fuel 2001

GEXL80 GEXL80 Correlation for SVEA96+ Fuel 2004

D4 10x10 SVEA Fuel Critical Power Experiments 

and CPR Correlation: SVEA-96 Optima2 2005

GEXL97 GEXL97 Correlation Applicable to ATRIUM-10 

Fuel 2008

D4 (Modified 

R-Factor)

SVEA-96 Optima2 CPR Correlation (D4): 

Modified R-factors for Part-Length Rods 2009

D4 (High and 

Low Flow)

SVEA-96 Optima2 CPR Correlation (D4): High 

and Low Flow Applications 2009

GEXL17 GEXL17 Correlation for GNF2 Fuel 2009

SPCB SPCB Critical Power Correlation 2009

GEXL14 GEXL14 Correlation for GE 14 Fuel 2011

ACE/ATRIUM-10 ACE/ATRIUM-10 Critical Power Correlation 2014

ACE/ATRIUM-10 

XM ACE/ATRIUM 10XM Critical Power Correlation 2014

D5

10x10 SVEA Fuel Critical Power Experiments 

and New CPR Correlation: D5 for SVEA-96 

Optima3

2013

ACE/ATRIUM-11 ACE/ATRIUM-11 Critical Power Correlation 2015

2
12

1 2.1.2 Regulatory References

2 The regulatory references are separated into the following types: 

3 • Regulations. The Code of Federal Regulations (CFR) sets forth regulations that licensees

4 must satisfy.

5 • Guidance. Following NRC guidance is one way to satisfy the corresponding regulations.

6 Such guidance can be found in NRC Regulatory Guides and NRC publications in specified 

7 NUREGS. In addition, the application regulations require an applicant to identify and 

8 describe all differences in design features, analytical techniques, and procedural 

9 measures proposed for a facility compared to those in NUREG-0800, “Standard Review 

10 Plan for the Review of Safety Analysis Reports for Nuclear Power Plants: LWR Edition” 

11 (SRP). Previous safety evaluations can also inform the staff’s review of an application.

12 • Generic Communications. The NRC may choose to send out a general communication on 

13 an issue for numerous reasons. Generic communications include administrative letters, 

14 bulletins, circulars, generic letters, information assessment team advisories, information 

15 notices, regulatory issue summaries, security advisories, and documents for comment. 

16 Table 5 lists the regulatory references associated with CBT models in reactor safety analyses.
13

1 Table 5 Regulatory References Associated with CBT Models

Type Title Date

Regulations

10 CFR Part 50, “Domestic Licensing of 

Production and Utilization Facilities,” Appendix A, 

“General Design Criteria for Nuclear Power 

Plants,” General Design Criterion 10, “Reactor 

Design”

N/A

Regulations 10 CFR 50.36, “Technical specifications” N/A

Regulations 10 CFR 50.34, “Contents of Applications; 

Technical Information” N/A

Regulations

10 CFR Part 50, Appendix B, “Quality Assurance 

Criteria for Nuclear Power Plants and Fuel 

Reprocessing Plants”

N/A

Guidance SRP Section 4.2, “Fuel System Design” 2007

Guidance SRP Section 4.4, “Thermal and Hydraulic Design” 2007

Generic 

Communication

Information Notice 2014-01, “Fuel Safety Limit 

Calculation Inputs Were Inconsistent with 

NRC-Approved Correlation Limit Values”

2014

Standard NQA-1 - Quality Assurance Requirements for 

Nuclear Facility Applications 2015

2 10 CFR Part 50, Appendix A, General Design Criterion 10

3 General Design Criterion (GDC) 10 in 10 CFR Part 50, Appendix A, is the principal regulation 

4 associated with a CBT. This criterion introduces the concept of specified acceptable fuel design 

5 limits (SAFDLs). In essence, SAFDLs are those limits placed on certain variables to ensure that 

6 the fuel does not fail. One such SAFDL is associated with CBT. Because the decrease in heat 

7 transfer following a CBT could result in fuel failure, a SAFDL is used to demonstrate that a CBT 

8 does not occur during normal operation and AOOs. Therefore, fuel failure is precluded during 

normal operation and AOOs.

7 9

10 SRP Section 4.4 includes the following two SAFDLs for use in accounting for the uncertainties 

11 involved in developing and using a CBT model (e.g., uncertainties in the values of process

12 parameters, core design parameters, calculation methods, instrumentation) and ensuring that

13 fuel failure is precluded:

 7 Experiencing such a transition may not immediately result in fuel failure. The decrease in heat transfer and 

subsequent increase in fuel temperature may not be enough to cause the cladding to weaken or melt. 

Therefore, the point of CBT is considered to be a conservative limit compared to the actual point of fuel 

damage.
14

1 (1) There should be a 95-percent probability at the 95-percent confidence level that the hot 

2 fuel rod in the core does not experience a CBT during normal operation or AOOs.

3

4 (2) At least 99.9 percent of the fuel rods in the core will not experience a CBT during normal 

5 operation or AOOs.

6 Typically, SAFDL No. 1 is associated with PWRs, and SAFDL No. 2 is associated with BWRs. 

7

8 Before May 21, 1971, when the GDC took effect, the Atomic Energy Commission (AEC), the 

9 predecessor to the NRC, approved construction permits for nuclear power plants based on plant10 specific Principal Design Criteria (PDC) that applicants proposed in their construction permit 

11 applications as required by the then-extant provisions of 10 CFR 50.34(a). The AEC published 

12 proposed General Design Criteria in the Federal Register (32 FR 10213) on July 11, 1967, 

13 sometimes referred to as the AEC Draft GDC, which were generally consistent with the PDC 

14 previously proposed in applications for construction permits. AEC Draft GDC 6 is the relevant draft 

15 GDC and is substantially similar to the current GDC 10. AEC Draft GDC 6 also calls for the 

16 reactor core to be designed with appropriate margin to specified limits that preclude fuel damage. 

17 10 CFR 50.36

18 The second regulation associated with a CBT is 10 CFR 50.36, part of which focuses on defining

19 technical specification safety limits. There are multiple limits that are associated with CBT models 

20 used during plant operation. These limits can be operating limits, alarms, analysis limits, and 

21 safety limits. Generally, only the safety limit and associated limiting conditions for operation 

22 (LCOs) and surveillance requirements (SRs) are included in the plant’s technical specifications. 

23 The safety limit associated with CBT is typically focused on an accurate quantification of the 

24 uncertainty of the CBT model and may also include the quantification of additional uncertainties as 

25 well.

26 10 CFR 50.34

27 The third regulation associated with a CBT is in 10 CFR 50.34, which focuses on defining the 

28 information that a licensee must present to ensure safe operation. Specifically, 

29 10 CFR 50.34(a)(4) requires that the Preliminary Safety Analysis Report (PSAR) include 

30 determination of the margins of safety during normal operation and AOOs. One of these is the

31 margin to CBT, which verifies that fuel failure is precluded during normal operation and AOOs 

32 through analysis. 

33 10 CFR Part 50, Appendix B

34 The fourth regulation associated with a CBT appears in 10 CFR Part 50, Appendix B. It requires 

35 licensees to include certain structures, systems, and components (SSCs) in a quality assurance 

36 program that satisfies specific criteria. Appendix B, Criterion III, requires that specified design 

37 control measure be applied to the design of safety-related SSCs and these measures apply to 

38 safety analyses for these SSCs. The CBT model is a key component of the safety analysis subject 

39 to 10 CFR Part 50, Appendix B. 

40 Other Regulations 

41 Both 10 CFR 50.46 and 10 CFR Part 50, Appendix K, Section I.C.4 focus on the modeling of a 

42 nuclear power plant during accident scenarios. While many of these scenarios involve the use of

43 CBT models, a different model may be used than is used to analyze SSC performance during 
15

1 AOOs. For example, the CBT models used during LOCAs are typically low pressure, conservative 

2 models, which are not necessarily fuel design specific. While these models are reviewed by the 

3 NRC as a part of any accident evaluation model, they typically are not a major focus during those 

4 reviews. 

5 Critical Boiling Transition Phenomena

6 A CBT occurs when a flow regime that has a higher heat transfer rate transitions to a flow regime 

7 that has a significantly lower heat transfer rate. In nuclear fuel rods, the heat flux from the fuel 

8 pellet to the fuel cladding is mostly independent of the heat transferred from the cladding surface 

9 to the coolant. As a result, the cladding temperatures will increase until the new heat transfer 

10 mechanisms can remove all of the heat from the pellet; the primary mechanism for post-CBT heat 

11 transfer will dictate the magnitude of the cladding surface temperature increase. Typically, the 

12 post-CBT heat transfer mechanism transfers heat at a much lower rate (i.e., it is less efficient) 

13 than the pre-CBT mechanism, and therefore causes a dramatic increase in clad temperature. The 

14 temperature increase resulting from CBT could cause the fuel rod cladding surface to weaken or 

15 melt and result in fuel failure, therefore it is considered a critical transition. Hence, the heat flux at 

16 which this transition occurs is known as the critical heat flux, the assembly power at which this 

17 transition occurs is known as the critical power, and the quality at which this transition occurs is 

18 known as the critical quality.

19 The difference in the rate of heat transfer associated with the flow regimes before and after the 

20 transition is a convenient way to understand the phenomena of CBT. The sections below discuss 

21 the two most common critical boiling transitions, DNB and dryout. 

22 2.2.1 Departure from Nucleate Boiling

23 Departure from nucleate boiling results from a change in the flow regime from nucleate boiling to 

24 film boiling and is chiefly a concern in PWRs. During nucleate boiling, the bulk coolant, which is 

25 mostly liquid with some vapor, is in intimate contact with the cladding. Vapor is generated as 

26 bubbles on the cladding surface at nucleation sites. These bubbles grow on the surface, detach, 

27 and flow into the bulk coolant stream. As each bubble leaves the surface, cooler liquid fills the 

28 space near the surface that was formerly occupied by the bubble, and the boiling process is 

29 repeated. The growth, transport, and collapse of the bubbles increases turbulence close to the 

30 wall and causes increased mixing in the thermal boundary layer. Ultimately, this boiling results in 

31 extremely high heat transfer rates; therefore, the cladding surface is able to support high heat 

32 fluxes at relatively low surface temperatures. 

33 Departure from nucleate boiling occurs when bulk liquid is prevented from coming into contact 

34 with the surface. The ultimate cause of the phenomenon is not fully understood but is believed to 

35 be bubble crowding that prevents liquid from contacting the surface. Once liquid coolant can no 

36 longer contact the surface, heat transfer to the liquid through convection is no longer possible, and

37 the only mechanisms that transfer heat to the bulk liquid coolant are conduction through the vapor 

38 and radiation from the surface. At normal cladding temperatures, both of these types of heat 

39 transfer mechanisms are relatively inefficient, and the surface’s temperature must dramatically 

40 increase to remove the heat generated in the pellet. This temperature increase is large enough to 

41 cause the surface to become unwettable, thus creating a dry patch. This dry patch may spread 

42 axially along the rod and blanket a large majority of the rod in vapor. Thus, the flow regime 

43 transitions to film boiling. This rapid increase in surface temperature may also result in fuel failure 

44 in a very short period of time.
16

1 2.2.2 Dryout

2 Dryout results from a change in the flow regime from annular flow around the fuel rods to 

3 dispersed flow and is mostly a concern in BWRs. In annular flow, a thin liquid film surrounds the 

4 cladding, and the bulk flow is mostly vapor with some liquid droplets. Convection transfers heat 

5 from the cladding to the annular film, causing some of the liquid in the annular film to evaporate 

6 from the film surface and thus adding more vapor to the bulk flow. It is currently believed that

7 evaporation is the only “boiling” that occurs in the annular film boiling regime; no vapor formation 

8 occurs at nucleation sites, and no bubbles are generated. 

9 As the coolant flows up the channel, it carries the liquid film up along the cladding surface. This 

10 results in entrainment of liquid droplets from the annular film into the bulk coolant, thus reducing 

11 the amount of liquid in the film. However, some of the droplets in the bulk coolant are also 

12 deposited back onto the film. This deposition will increase the amount of liquid in the film and is a 

13 chief concern in the design of grid spacers for BWR assemblies. In summary, as the liquid film 

14 flows up the cladding, evaporation and entrainment remove liquid from the film while deposition 

15 adds liquid to the film. 

16 Dryout occurs when the annular film disappears completely. Upon reaching dryout, the bulk fluid 

17 transitions from annular flow to dispersed flow. In dispersed flow, there is no continuous liquid film 

18 on the cladding, and the bulk flow consists of a mixture of vapor and dispersed liquid droplets. 

19 Convection occurs between the vapor and the fuel rod. The droplets also act as a heat sink, as 

20 they are in the heated vapor and may absorb heat from the vapor as well as impact the heated 

21 rod (assuming the rod is still wettable). Generally, radiation is not a significant mode of heat 

22 transfer until the surface temperature become much higher (at which point, the rod is typically 

23 unwettable). Although the heat transfer is less in dispersed flow than in annular flow, it is still 

24 substantial. As a result, the increase in cladding temperature is typically not as dramatic as that 

25 resulting from DNB. However, sustained time in dryout will eventually result in fuel failure.

26 2.2.3 Other Flow Regimes and Transitions

27 It is important to recognize that the flow regimes inside a reactor core are not precisely defined. 

28 Further, potential transitions occur between flow regimes that are not considered “critical” or do 

29 not result in a “crisis” because the transition would not significantly reduce heat transfer. For 

30 example, different portions of a PWR fuel assembly may be in subcooled nucleate boiling (i.e., 

31 boiling which occurs when the bulk of the liquid is sub-cooled and not at saturation), nucleate 

32 boiling, and annular flow. Although a shift from nucleate boiling to another regime is technically a 

33 departure, it is only considered DNB if the new regime has a significantly lower heat transfer rate. 

34 It is also important to note that the same CBT model will generally be applied in every flow regime 

35 in a given reactor type and is not associated with only a single flow regime. Thus, a specific DNB 

36 model used in a PWR will not only be used to predict whether the flow regime has transitioned 

37 from nucleate to film boiling, but will also be used to predict a transition from subcooled nucleate 

38 to film boiling or a transition from annular flow to film boiling, as all of those flow regimes can exist 

39 in a PWR assembly. 

40 Determining When Critical Boiling Transition Occurs

41 Given certain key parameter values (e.g., flow rate, power, pressure, temperature), a CBT model 

42 predicts either the CHF or the critical power (CP) of the assembly that would cause a CBT. This

43 predicted value is then compared to the current heat flux or assembly power to determine the 
17

1 margin to CBT. Typically, CHF models are used in PWRs, whereas CP models are used in 

2 BWRs. 

3 2.3.1 Critical Heat Flux Models

4 Critical heat flux is the cladding surface heat flux that causes a CBT for a given set of local 

5 conditions. It is chiefly associated with PWRs and the phenomenon of DNB; however, as stated 

6 earlier, CHF models can also predict other CBTs (e.g., the transition from annular flow to film 

7 boiling). CHF models are developed through experiments where, under a given set of inlet flow 

8 conditions, power is increased until CHF is observed. A computer code is used to calculate the 

9 local flow conditions from the boundary conditions of the experiment, and CHF is correlated to 

10 those local flow conditions. Thus, when a computer code is used to simulate an AOO, the CHF 

11 model can use the local conditions calculated at any location in the core to predict the critical heat 

12 flux at that location. The predicted CHF is then compared with the local heat flux to determine the 

13 margin to CBT at that location.

14 2.3.2 Critical Power Models

15 Critical power is the assembly power that causes a CBT. It is chiefly associated with BWRs and 

16 the phenomenon of dryout; however, as stated earlier, CP models can also predict other critical 

17 flow transitions. Further, the term “CP model” is something of a misnomer because these models

18 do not generally correlate CP to local conditions (as a CHF model does); instead, they correlate 

19 the critical quality (i.e., the quality that causes a CBT) to the boiling length (i.e., distance from the 

point of initiation of bulk boiling to the location of a CBT).

8 20 Thus, when a computer code is used to 

21 simulate an AOO, the inlet conditions (e.g., power, inlet flow) along with certain local conditions 

22 are used to calculate the quality at various axial elevations in the fuel. The quality at each axial 

23 elevation is compared to the critical quality at that elevation by assuming that the boiling length is 

24 the elevation of the location under consideration. Generally, the critical quality is much greater

25 than the predicted quality; therefore, the assembly power is increased until, at some axial 

26 elevation, the critical quality is equal to the predicted local quality. The lowest assembly power at 

27 which at least one location equals a quality greater than or equal to the critical quality is known as 

28 the CP.

29 2.3.3 Semi-empirical Modeling

30 Since 1970, tremendous strides have been made in the generation of CBT models; however,

31 these models are still predominantly semi-empirical (i.e., the models are based more on 

32 experimental data than on first-principle physics). Known physical behavior is often used to inform 

33 the model’s mathematical form, but empirical coefficients are still needed to ensure accurate 

34 model predictions. In effect, this means that, although the model may be informed by physics, 

35 they are not treated as theoretical models, but are treated as empirical or data-driven models in 

36 that they must be validated with experimental data and should not be used outside of the range of 

37 their validation database. 

 8 This is not exclusively true because other models are more mechanistic than critical 

quality/boiling length correlations. Regardless, even these mechanistic boiling transition models 

do not generally correlate CP directly to fluid conditions.
18

1 2.3.4 Conservative vs. Non-Conservative Predictions

2 For CBT models, “conservative” means that the model will predict a CBT before the actual 

3 occurrence of the phenomenon (e.g., at lower powers, at lower flow rates, at lower qualities). 

4 Conversely, “non-conservative” means that the model will predict a CBT after the actual 

5 occurrence of the phenomenon (e.g., at higher powers, at higher flow rates, at higher qualities).

6 Applying a Critical Boiling Transition Model

Unlike many closure models9 7 that are developed directly from experimental data, CBT models 

may10 8 call for input that typically cannot be measured directly. In such instances, a 

9 thermal-hydraulic computer code is used. This code will calculate the values of key variables 

10 needed by the CBT model (e.g., local quality, local mass flux) using some set of field equations 

11 and any necessary closure models. Thus, the development and, more importantly, the validation 

12 of a CBT model may highly depend on the thermal-hydraulic computer code used and the code 

13 options selected. For this reason, using such a CBT model in a different code or in the same code 

14 with substantially different code options (e.g., different two-phase closure models) would call for a 

15 complete re-validation using those new code or code options. 

16 The application of CBT models can be somewhat confusing. Many closure models, such as 

17 Dittus-Boelter (1930), operate as a simple function. The function takes in certain inputs and 

18 returns an output. Thus, the validation of such a function would ensure that, given the correct 

19 inputs, the function returns the correct output. Although CBT models follow a similar process, the 

20 models themselves cannot typically be used or validated in such a simple manner. For example, 

21 consider an experiment of a test assembly whose power is increased until a CBT occurs. For this 

22 experiment, the inlet flow rate and temperature, axial and radial power shapes, pressure, and 

23 assembly power have been measured. However, most CBT models do not use this measured 

24 data, but require a different set of data to make a prediction. Hence, the measured data from the 

25 experiment is input into a computer code and that code generates the required input data such 

26 that the CBT model can make a prediction. The following sections describe how this simple 

27 situation would be evaluated using methods commonly applied in a PWR and a BWR. 

28 2.4.1 Applying a Critical Boiling Transition Model in a Pressurized-Water Reactor

29 In a PWR assembly, a prediction of the CHF would be calculated for each subchannel at each 

30 axial elevation. Consider a 5x5 assembly that contains 25 rods, 16 internal subchannels (i.e., 

31 between the rods), and 20 external subchannels (i.e., between the rods and the channel wall). 

32 Assume that the assembly has a height of 3.65 meters (12 feet) and that the computer code uses 

33 an axial nodalization of 7.62 centimeters (3 inches). In total, each subchannel would have 48 axial 

34 elevations; given the 36 subchannels (internal plus external), that results in a total of 1,728 nodes, 

35 each of which would have its own CHF prediction from the CBT model. Which of the 1,728 

36 predictions should be compared with the single measured CHF from the experiment? 

 9 Closure models are those additional models needed in order to “close” the problem. They provide 

the additional relations needed for the number of equations to equal to the number of unknowns 

so the problem can be solved. They supplement the conservation equations. 

10 Typically, CHF models used in PWRs are subject to this restriction. The subchannel code 

provides detailed information about the local flow conditions that the CHF model uses to make a 

prediction. Dryout models are generally less affected because they do not need detailed 

information about the local flow conditions. 
19

1 At first glance, comparing the predicted CHF at the location where the CHF was indicated in the 

2 test would seem to be the best approach. Suppose that a thermocouple on one of the inside rods 

3 (i.e., a rod internal to the 5x5 array and not on the boundary) was the first thermocouple to 

4 indicate that a CHF occurred. Further, suppose that this thermocouple is located at an elevation of 

5 2.74 meters (9 feet). This rod would be a member of four subchannels; thus, there may be no way 

6 to determine in which of the four surrounding subchannels CHF actually occurred. This does 

7 seem to make the problem more tractable because, instead of considering 1,728 nodes, that 

8 number is reduced to 4 nodes. However, in such experiments, multiple rods will experience the 

temperature rise associated with a CBT 9 .11 While one rod at one axial elevation will achieve such a

10 temperature rise first, the temperature rise used to indicate CHF is somewhat arbitrary in that a 

11 slightly different criterion may result in a different CBT value. For example, changing the CBT

12 criterion from a rise of 16.67 degrees Celsius (C) (30 degrees Fahrenheit (F)) to a rise of 11.1 

13 degrees C (20 degrees F) may result in the selection of a different rod in CBT. Suppose there 

14 were five thermocouples indicating that CHF was very likely occurring at those locations. That 

15 would mean that of the 1728 nodes in the bundle, 20 would need to be considered for the CBT 

16 point. 

17 The measured heat flux at the time of CHF could be compared to each of the predicted CHF 

18 values in the 20 nodes; however, it is not clear how a single “predicted CHF” value could be 

19 objectively chosen. While a ratio of “measured CHF” to “predicted CHF” could be found at each 

20 point (the measured value from the experiment and the predicted value from the CBT model) 

21 which of these 20 values should be taken as the value from this test? The maximum value, the 

22 minimum value, the mean of all 20 values? The usual practice is described below.

23 It is important to remember that the overall goal of a CBT model is to determine whether a CBT

24 will occur. Thus, the validation process should focus on ensuring that the model appropriately

25 predicts a CBT, and not necessarily that the model predicts CBT at the correct location. Thus, 

26 when using the model to make predictions “measured CHF” data will not be available for the 

27 reactor assembly under normal operation and AOO conditions. Considering the 5x5 assembly, 

28 only 1,728 predictions of CHF for each time step of the scenario will be available. Therefore, those 

29 predicted values of CHF are typically compared to the local values of heat flux to determine which 

30 of the nodes is closest to CHF using the departure from nucleate boiling ratio (DNBR). The DNBR 

31 is defined as the ratio of the predicted CHF of a node to the current heat flux of a node. 

32 Equation 1 gives the DNBR.

DDDDDDDD = qqCCCCCC

′′

qqLLLLLLLLLL

′′ (1)

33 Notice that as long as the node is far from the conditions that cause CHF, the value of DNBR will 

34 be greater than 1. As the node approaches those conditions, the DNBR value approaches 1, and 

35 when heat flux in the node is equal to the CHF, the DNBR is equal to 1. Given that these 

36 simulations are used to demonstrate that CHF does not occur, the DNBR in all of the nodes 

37 should always be greater than 1. Further, the node with the smallest DNBR, commonly called 

38 minimum departure from nucleate boiling ratio (MDNBR), is the node closest to the conditions that 

39 cause CHF.

40 These concepts of DNBR and MDNBR are used to select a “predicted CHF” value to compare 

41 with the “measured CHF” value from the experiment. From the 1,728 nodes, the node that 

 11 The temperature rise selected is usually on the order of 11.1 to 27.78 C (20 to 50 degrees F) in under 

1 second.
20

1 contains the MDNBR could be used as the “predicted” node, and the CHF prediction at this node 

2 could be the “predicted CHF”. This may or may not be one of the 20 nodes discussed earlier, but 

3 using the CHF from this node as the “predicted CHF” results in a much more representative error 

4 of how the CBT model will be applied in practice. While the analyst may know which sub-channel 

5 and what elevation CHF occurred at in the experiment, this information is not known in the real6 world scenario. Thus, this information should not be used in determining the model’s error. 

7 Instead, the “predicted CHF” value should be determined using the same method that will be used 

8 when the model is applied in the real-world scenario. 

9 Note that the MDNBR location (and hence the “predicted” CHF value) may change during model 

10 development. Thus, as the model changes during its development, different nodal locations in 

11 different subchannels would likely be determined to be more the limiting node.

12 2.4.2 Applying a Critical Boiling Transition Model in a Boiling-Water Reactor

13 In a BWR assembly, the calculation of the predicted CP would consider each fuel rod in the 

14 assembly individually. Consider a 5x5 assembly that contains 25 rods that has a height of 3.65 

15 meters (12 feet) an axial nodalization of 7.62 centimeters (3 inches). Most BWR methods do not 

16 model all of the rods and subchannels; instead, they model only a single rod surrounded by a 

17 single subchannel of fluid. Modeling all of the subchannels is considered unnecessary because 

18 the fuel assembly is contained within a channel; therefore, the water cannot flow between 

19 assemblies. To account for the varying thermal-hydraulic conditions at the different locations in 

20 the assembly, two different factors are used to “convert” the results of the single rod analysis and 

21 make it applicable to the entire assembly. 

22 The first factor is a relative power factor, commonly called the R- or K-factor. The R- or K-factor

23 accounts for the power in a specific rod compared to the powers in the surrounding rods. In the 

24 above example, a different R- or K-factor would be calculated for each of the 25 rods depending 

25 on each rod’s individual power, which can change over the cycle. The second factor is a 

26 thermal-mixing factor, commonly called an additive constant. The thermal-mixing factor accounts 

27 for the thermal performance at that specific xy location in the assembly. In the above example, a 

28 different thermal-mixing factor would be calculated for each of the 25 rods depending on the xy 

29 location of each rod in the assembly; that factor would not change for that assembly design. 

30 Ideally, the local conditions calculated in the assembly could be directly correlated to the CP. 

31 However, this is not the case. A change in power has a dramatic impact on the entire flow field 

32 along the length of the assembly, and integral, not local, effects are commonly considered the 

cause of the CP and its associated phenomenon of dryout.

12 33 To determine the CP, the mass flow 

34 rate, axial and radial power shape, and pressure are fixed. The quality at a given elevation can 

35 then be compared to the predicted critical quality from the CBT model given the boiling length 

36 (i.e., the length from the start of boiling to the elevation of interest). The power input to the model

37 is increased or decreased until the calculated quality at that location is equal to the critical quality. 

38 The corresponding power is the CP. 

 12 This consideration of integral effects, as well as the concept of flow memory (Tong 1965), seems 

to be somewhat of a misnomer. Although what occurs upstream shapes the flow field, CBT 

occurs at a single location based on the conditions of the local fluid and the heat from the wall. If 

those local fluid conditions could be modeled perfectly, a consideration of integral effects would 

not be necessary. However, because of modeling limitations, many of the important parameters 

of that local fluid cannot be directly modeled; therefore, concepts such as flow memory are useful 

as modeling simplifications.
21

1 Because there are 25 rods, there could be 25 different CPs for each axial elevation. However, 

2 because many CBT models correlate the critical quality to the boiling length, it is not necessary to 

3 perform calculations below the boiling length. Additionally, it is not necessary to determine the 

4 power that would cause a CBT at a certain axial elevation. For example, suppose a CBT occurred 

5 on rod 17 at an axial elevation of 10 feet. If an analyst wanted to determine what power would 

6 cause a CBT at 8 feet, the power would need to be increased. However, increasing the power to 

7 cause a CBT at 8 feet would not make much sense because the goal is to avoid a CBT entirely, 

8 and at the current power level, a CBT has occurred. Thus, it is not the power that causes a CBT at 

9 every elevation that is important; instead, it is the lowest power that causes a CBT at any 

10 elevation at or below the top of the active fuel that is most important.

11 2.4.3 Applying a Steady-State Model to Transient Conditions

12 Generally, CBT models are generated with steady-state data (i.e., the test facility reaches a 

13 steady state and slowly increases the power until CBT occurs). Information from those data points 

14 is then used to generate CBT models. However, when the data are applied in a reactor safety 

15 analysis, the CBT model is applied to the transient (i.e., time-varying) conditions occurring during 

16 a scenario. Historically, this application of a correlation developed on steady-state data to 

17 transient conditions has been considered conservative, and often a few transient tests are 

18 performed to demonstrate that the prediction of a CBT model is conservative when it is applied in 

19 a transient fashion. 

20 Addressing Uncertainties and Errors

21 Many uncertainties and errors are associated with a CBT model. First and foremost, some of 

22 these uncertainties have specific meanings and should be defined. In this work, a distinction is 

23 made between an error and an uncertainty. The term “error” focuses on the difference between 

24 specific predicted values and their corresponding specific “actual” value. For example, the error in 

25 a single measurement (absolute error or relative error) is a comparison of the true value to the

26 measured value. The term “uncertainty” focuses on quantifying the variability of a set of values for 

27 future predictions. For example, while a prediction is generally a single value, it may be better to 

28 think of that prediction as a range of values where that range is defined by the uncertainty in the 

29 prediction. The various forms of uncertainties discussed throughout this document are defined as 

30 follows:

31 • Instrumentation uncertainty is associated with a specific instrument used in the 

32 experiment. This uncertainty is a result of the underlying precision of the instrument, and is 

33 typically provided by the manufacturer of the device in question. Examples include the 

34 ±0.50 degrees C (±0.90 degrees F) of a K-type thermocouple or the 1 percent uncertainty 

35 of a pressure transducer. Generally, instrumentation uncertainty (future behavior) is 

36 approximated through the instrumentation error (past behavior).
22

1 • Measurement uncertainty is the total uncertainty associated with recording the 

2 measurement from a piece of instrumentation. Although this is often considered to be 

3 simply the instrumentation uncertainty, that may be an oversimplification. Uncertainty is 

4 often associated with recording the value from the instrument. Data-logging systems 

5 typically read in voltages, but not all measurements are provided as a voltage, and these

6 values would need to be converted. Additionally, some uncertainty occurs in the voltage 

7 reading of the data-logging system itself. For example, pressure transducers often provide 

8 an output between 4 to 20 milliamperes. This output must be converted through a resistor 

9 before it can be measured as a voltage. The uncertainty of the resistance in that resistor 

10 should be accounted for in the measurement uncertainty because it may not have been 

11 accounted for in the instrumentation uncertainty.

12 • Experimental uncertainty is the total uncertainty associated with recording the value of 

13 quantity of interest from an experiment. In many instances, an instrument that measures

14 the quantity of interest may not be available, or even if one is available, that measurement 

15 may depend on multiple instruments. For example, the uncertainty associated with the 

16 CHF “measurement” would at least need to consider uncertainties associated with the 

17 measured power, the manufacturing tolerances of the heater rods (which influence the 

18 axial heat and heat flux shape), and the thermocouples used to determine when a CHF 

19 event occurs. 

20 • Model error is the difference between the model’s predicted CHF or CP and the actual 

21 CHF or CP. 

22 • Model application error is similar to model error, but it accounts for the fact that the CBT

23 model is not used as a standalone equation, but used in a larger calculational framework.

24 • Validation error is a sample from the population of the model application error. If we 

25 consider the model application error as a set which contains the entire population of all 

26 possible uses of the model, then the validation error is the sample from that population for 

27 which a CHF or CP value was measured in a particular experiment. 

28 • Model uncertainty is associated with the application of the CBT model in a future analysis. 

29 This may also be referred to as the predictive capability of the model. This uncertainty 

30 quantifies the difference (or ratio) between the power at which a model predicts CBT will

31 occur and the power at which CBT would actually occur. Note that it is not only the 

32 uncertainty of how the model predicted the experimental data (i.e., the validation error) but 

33 also includes how the model would have predicted other experimental data (i.e., other 

34 samples from the model application error) and how that experimental data relates to the 

35 real world system of interest of the fuel assembly in a nuclear power plant. 

36 • Plant parameter uncertainties are associated with specific plant parameters, such as flow, 

37 power, and pressures. Although these uncertainties do not generally affect the CBT model 

38 directly, they are used along with the CBT model to generate the safety limit. 
23

1 3 CREDIBILITY ASSESSMENT FRAMEWORK

2 This section discusses the development of a credibility assessment framework for CBT models. 

3 As described above, this framework is a generic safety case expressed using concepts from GSN

4 and maturity assessment. This framework was developed based on the experience of members of 

5 the NRC technical staff, documented safety evaluations from previous NRC reviews, and various 

6 documents found in the open literature. While it was the goal of the authors to have this 

7 framework be applicable to all uses of a CBT (i.e., from a homework problem to reactor safety 

8 analysis), much of the evidence is based on the evidence that has been historically used for CBT 

9 models applied in reactor safety analysis.

10 The purpose of the framework is summarized as the main goal, G - The CBT model can be 

11 trusted. Everything which follows is focused on demonstrating that this main goal is true and

12 defines exactly what is meant by the statement “The CBT model can be trusted”. The main goal is 

13 decomposed into the three subgoals in Figure 3 below.

14

15 Figure 3 Decomposition of G — Main Goal

16 As discussed above, the goals (G, G1, G2, G3) are intentionally ambiguous. While there may be 

17 no consensus on what is meant by the words “trusted,” “appropriate,” logical”, and “sufficient,”

18 most will agree that for a CBT model to be trusted, its experimental data must be appropriate, the 

19 model must be logical, and the validation must be sufficient. The further development of the 

20 framework through continued decomposition of each goal into sub-goals and specification of the 

21 possible levels of evidence, acts to more clearly define these ambiguous terms. 

The bulk of this section will focus on the decomposition of all sub-goals into base goals.

13 22 For 

23 each base goal, we provide a discussion of the levels of evidence used for demonstrating that the 

24 base goals are true and a discussion of the evidence levels that have been historically used for 

25 CBT models in reactor safety analysis. 

 13 A goal that is not decomposed further but is supported by evidence.
24

1 G1—Experimental Data

2 Experimental data are the cornerstone of a CBT model. The data are used to generate the 

3 coefficients of the model and validate the model. Additionally, previous experimental data often 

4 used influence the form of the model. Therefore, it is essential that experimental data are 

5 appropriate. The three subgoals in Figure 4 are used to demonstrate that the experimental data 

6 are appropriate. 

7

8

9 Figure 4 Decomposition of G1—Experimental Data

10

11 3.1.1 G1.1—Credible Test Facility

12 Test facilities that are used to measure CBT primarily focus on measuring key flow parameters 

13 that occur during the critical transition. Experimental data has been collected at multiple research 

14 facilities and universities over many years (Groeneveld 2007). However, because the time, effort, 

15 and resources needed to set up a reliable facility are quite significant, most CBT data used in the 

16 nuclear industry have historically come from one of the following facilities:

17 • Columbia University’s Heat Transfer Research Facility (closed in 2003)

18 • General Electric Company’s ATLAS test loop facility in San Jose, CA (closed)

19 • Stern Laboratories in Hamilton, Ontario (still in use)

20 • AREVA’s KATHY loop in Karlstein, Germany (still in use)

21 • Westinghouse Electric Corporation’s FRIGG and ODEN loops in Västerås, Sweden, for 

22 BWRs and PWRs, respectively (still in use) 
25

1 The two subgoals in Figure 5 are used to demonstrate the credibility of the test facility. 

2

3 Figure 5 Decomposition of G1.1—Credible Test Facility

4 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

5 discuss the evidence that could be used to demonstrate that these two base goals (G1.1.1 and 

6 G1.1.2) have been satisfied. Additionally, a discussion is provided on the evidence that has been 

7 historically used for CBT models applied in reactor safety analysis. 

8 G1.1.1—Test Facility Description

9 The test facility contains the test loop, the control equipment, interconnected piping, and 

10 instrumentation needed to perform the experiment. Test loops usually consist of a test section 

11 (which contains the simulated test assembly), pressurizer, heat exchangers, pumps, pressure 

12 transducers (both absolute and differential), flow meters, and thermocouples. The test assembly

13 contains the simulated fuel rods, which not only supply the power to the test section but also 

14 contain the thermocouples that indicate when a CBT occurs. 

15 The description of the test facility must enable the assessor to understand how the facility 

16 operates and how the data were obtained. For assessors familiar with CBT testing and for 

17 established test facilities, a reference that describes the facility is typically sufficient 

18 documentation. In the past, having the assessor visit the test facility and witness testing first hand 

19 has greatly increased the assessor’s understanding, reducing the total time needed for the 

20 assessment, particularly for new assessors and/or new test facilitates. Table 6 gives the evidence 

21 commonly provided to demonstrate that this goal has been satisfied.

22 Table 6 Evidence for G1.1.1—Test Facility Description

G1.1.1 The test facility is well understood. 

Level Evidence

1

A reference that describes the test facility in appropriate detail has 

been provided. At a minimum, the reference includes loop, test section, 

and heater rod descriptions.

2 The assessors have visited the test facility. Additionally, a reference 

that describes the test facility in appropriate detail has been provided. 
26

At a minimum, the reference includes loop, test section, and heater rod 

descriptions.

1

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 1 has been most commonly accepted by the NRC staff, but Level 2 has resulted in 

4 increased review efficiency. Because the goal of the reference describing the test facility is to 

5 allow the assessor to fully understand the function of the test facility including operation, control, 

6 and measurement capabilities, it has often been found to be convenient to have the assessor visit 

7 the test facility and witness testing. This is especially true for new assessors unfamiliar with a test 

8 facility, but also true for experienced assessors who have not reviewed data from a particular test 

9 facility for some period of time. Visiting a test facility and observing testing has been a much more 

10 efficient way for the assessor to gain an understanding of the test facility than by reading 

11 documentation alone. A significant portion of an assessor’s time is spent gaining an 

12 understanding of the test facility. The assessor must understand the facility to such an extent that 

13 he or she is able to fully understand a complete test run including how the various pieces of 

14 equipment interact. Thus, actually visiting the test facility greatly increases the rate of 

15 understanding, typically leading to a reduction in the time needed to perform the assessment and 

16 fewer questions. 

17 G1.1.2—Test Facility Comparison

18 The test facility description is used as an indicator to determine if the facility is capable of 

19 generating accurate data. However, another key piece of evidence is the validation of the test 

20 facility itself. One type of validation frequently used is a comparison of the measured CBT data to 

21 the results from another credible facility. The justification for the test facilities should be based on 

22 factors other than the test facility itself (e.g., comparison to a benchmark, reproduction of data

23 from another facility, or reproduction of known phenomena).

24 Most facilities in use today have been compared to their older counterparts (for example, many 

25 facilities have performed tests to compare to data collected at Columbia University). However, 

26 because of the proprietary nature of the test sections, it may be difficult to obtain comparisons to 

27 actual CBT data. Therefore, though a new facility would be under the greatest scrutiny in this 

28 framework, it may have difficulty meeting this criterion. When comparisons to actual CBT 

29 measurements are not possible, the assessor should compare the test facility under evaluation to 

30 measured quantities from other experiments (e.g., in the open literature) with similar phenomena. 

31 Table 7 gives the evidence commonly provided to demonstrate that this goal has been satisfied. 
27

1 Table 7 Evidence for G1.1.2—Test Facility Comparison

G1.1.2 The test facility has been verified by comparison to an outside source.

Level Evidence

1

The test facility has been verified by comparison of data obtained at the 

facility to some benchmarks or some known phenomenological 

behavior.

2

The test facility has been verified by comparison of data obtained from 

tests at the facility to data other than CBT data obtained from a credible 

facility. 

3 The test facility has been verified by comparison of CBT data obtained 

at the facility to CBT data obtained from a credible facility.

4

The test facility has been verified by comparison of CBT data obtained 

at the facility to CBT data obtained over the same application domain 

as that of the proposed model at a credible facility.

2

3 Historical Evidence Levels for Reactor Safety Analysis

4 Evidence at Level 2 and Level 3 have been most commonly accepted by the NRC staff. This is 

largely due to the fact that most test facilitates in operation today are 2nd 5 generation facilitates, and 

part of their initial testing program was to establish consistency with the data taken from 1st 6

7 generation facilities. When comparisons to actual CBT measurements are not possible, it is 

8 possible for the assessor to consider other measured quantities besides CBT from other 

9 experiments with similar phenomena. 

10 3.1.2 G1.2—Accurate Measurements

11 In order for the test data to be relied upon, the test facility needs to provide accurate 

12 measurements of all important experimental parameters, including the measurement of CHF or 

13 CP. It is important to note that neither CHF nor CP is a directly measured parameter (like flow rate 

14 or pressure); instead, the CHF or CP value is inferred from the assembly power, axial and radial

15 power peaking, and a thermocouple indication that signifies CBT has occurred in the test facility

16 and where in the test section CBT has occurred. 

Typically, five experimental parameters are directly measured or controlled.

14 17 The type of control 

18 used for the experimental parameters depends on the type of data being taken. Usually, the 

19 desired values are programmed into a computer, and the computer will maneuver the control 

20 equipment to the desired state point. Table 8 presents the methods used to measure and control 

21 each experimental parameter. 

 14 Although the axial heat flux shape is very important for obtaining the local power and may be changed 

through the exchange of test rods, it is not a measured value during the experiment and, therefore, will be 

treated in Section 3.1.3 on local conditions.
28

1 Table 8 Experimental Parameters Measured or Controlled

Parameter Method of Measurement Typical Method of Control

Pressure Absolute and differential pressure 

cells on the test section A pressurizer on the test loop

Power (including 

radial power peaking) Reading from rectifiers Rectifiers that supply power to the simulated fuel rods

Inlet Flow Rate Flow meter at the inlet Valve at the inlet or pump speed

Inlet Temperature Thermocouple at the inlet Heat exchanger or mixer at the 

inlet

Rod Temperature 

Change

Thermocouples inside the 

simulated fuel rods

N/A (the change in rod 

temperature is not controlled, but is 

a response quantity)

2 The six subgoals in Figure 6 are used to demonstrate the accuracy of the measurements. 

3

4 Figure 6 Decomposition of G1.2—Accurate Measurements

5 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

6 discuss the evidence that could be used to demonstrate that these six base goals have been 

7 satisfied. Additionally, a discussion is provided on the evidence that has been historically used for 

8 CBT models applied in reactor safety analysis.
29

1 G1.2.1— Test Facility Quality Assurance (QA) Program

2 A determination regarding the credibility of a test facility is often assessed by reviewing the quality 

3 assurance program applicable to the test facility. Typically, an assessment of a facility’s QA 

4 program involves determining its compliance with a standard (e.g., ASME’s NQA-1, “Quality 

5 Assurance Requirements for Nuclear Facility Applications). While different QA standards will have 

6 different elements, the following represent some of the issues that should be addressed:

7 • Calibrated instrumentation - Routine calibration of the instrumentation is necessary to 

8 ensure that an instrument is resulting in a precise measurement and to quantify any 

9 instrumentation error (i.e., accuracy and precision). Generally, the instrumentation’s 

10 calibration is checked on a routine basis, with the calibration interval set to account for 

11 instrument drift over time and drift due to operation. This check should be performed often

12 enough to avoid having to recalibrate the instrumentation after its use. If an instrument 

13 does need to be recalibrated after a test, it likely means that the last set of data points 

14 taken with that instrument were taken when the instrument was out of calibration. At a 

15 minimum, a calibration check should be performed at both the beginning and end of a test 

16 campaign. The general assumption is that, if an instrument is within its calibration 

17 specification at the beginning and end of a campaign, there is very little chance that it was 

18 out of its specification at any time during the campaign. Note that, contrary to the 

19 discussion above, the heater rod thermocouples used to detect a CBT are often not 

20 calibrated because the absolute value of the temperature is not used. Instead, as 

21 previously discussed, a change in temperature over a period of time is used as the 

22 criterion for determining that a CBT has occurred. However, the thermocouples used to 

23 determine fluid and wall temperatures elsewhere in the test loop should be calibrated. 

24 NQA-1, Requirement 12 “Control of Measuring and Test Equipment” provides more details 

25 on instrument calibration.

26

27 • Appropriate equipment – The experimental parameters measured in CBT experiments 

28 are provided in Table 8 above. Therefore, instrumentation should be employed to measure 

29 these parameters. However, as instrumentation may fail or provide anomalous

30 measurements, a common practice is to employ redundant and diverse instrumentation. 

31 Redundant instrumentation is necessary to ensure that (1) instrumentation remains in 

32 calibration, and (2) an instrument which suddenly becomes uncalibrated does not greatly 

33 impact the resulting experimental data. Further, diverse instrumentation (i.e., use of a 

34 different process to perform the measurement) helps achieve a higher degree of 

35 confidence that the final measurement is accurate because it reduces the potential for 

36 “common cause” failures that could result in inaccurate measurements.

37

38 • Trained personnel – There are many appropriate ways in which the data could be 

39 obtained. It is important that the personnel performing the tests have been trained on the 

40 test procedure and test equipment, and are able to follow the test procedure in order to 

41 ensure consistent experimental results. 

42

43 • Condition of test equipment and the item to be tested – The test equipment, including 

44 the instrumentation, the test section, and all connected piping, should be demonstrated to 

45 be in working order. Generally, the bulk of these activities is performed during the 

46 shakedown testing, which ensures the test facility is behaving as expected.

47
30

1 • Suitable environmental conditions – As CBT tests are often performed in state of the art 

2 experimental test facilities, the conditions for both the equipment and the personnel are 

3 generally suitable environments. 

4

5 • Provisions for data acquisition – As the data will be used to validate the CBT model, the 

6 acquisition of the data are of paramount importance. While there are multiple data 

7 acquisition systems that could be used, it is important for specific procedures to be 

8 developed and used in order to determine how the data are reduced to the final set of 

9 measured values. 

10

11 Table 9 gives the evidence commonly provided to demonstrate that this goal has been satisfied.

12

13 Table 9 Evidence for G1.2.1—Test Facility QA Program

G1.2.1 The test facility has an appropriate quality assurance program.

Level Evidence

1

A QA program exists that reflects the basic tenets of quality assurance 

as referenced by a widely accepted international quality organization

(e.g., NQA-1).

2

A QA program exists that reflects the basic tenets of quality assurance 

as referenced by a widely accepted international quality organization

(e.g., NQA-1). Documentation is provided that outlines how the design, 

construction, and test activities were conducted consistent with the QA 

Program. It is clear that the base expectations of QA were applied.

3

A QA program exists that reflects the basic tenets of quality assurance 

as referenced by a widely accepted international quality organization

(e.g., NQA-1). Documentation is provided that outlines how the design, 

construction, and test activities were conducted consistent with the QA 

Program. It is clear that the base expectations of QA were applied. 

Audit reports properly identify, track, and indicate correction of 

conditions adverse to quality and are available for inspection.

14 Historical Evidence Levels for Reactor Safety Analysis

15 Level 3 has been most commonly accepted by the NRC staff. While the CBT assessor does not 

16 typically examine the QA program in the same detail as a QA inspector, previous NRC reviews 

17 have shown that understanding the QA program helped the assessor gain an improved

18 understanding of how the data were taken, controlled, reduced, and then used to generate the 

19 model. It is important to note that most assessors have typically limited their review to confirming 

20 that some type of QA program was in place, rather than providing an extensive review of that 

21 program itself. 
31

1 G1.2.2—Statistical Design of Experiment

2 The goal of the statistical design of the experiment (Box, Hunter, and Hunter, 1978) is to ensure 

3 that the testing methods do not introduce any biases into the figure of merit (i.e., the CHF or CP 

4 value). Most of the statistical methods used to quantify the uncertainty treat all errors as random. 

5 This is equivalent to assuming that each measurement is taken at a randomly determined 

experimental state point15 6 that is completely independent of any measurements taken before or 

7 after. However, that is generally not possible for CBT experiments. First, large changes in 

8 pressure (and sometimes flow rate) can put tremendous stresses on the test section. Second, 

9 changes in flow cause the test section to reach a new thermal equilibrium, which may take a long 

10 time. As such, it is often not feasible to dramatically change the flow rate or pressure between test 

11 points. Because of these issues, the order in which the test points are taken is typically not 

12 random. Table 10 gives the evidence commonly provided to demonstrate that this goal has been 

13 satisfied.

14 Table 10 Evidence for G1.2.2—Statistical Design of Experiment

G1.2.2

The experiment has been appropriately statistically designed (i.e., the 

value of a system parameter from any test was completely 

independent from its value in the test before and after the test).

Level Evidence

1 One or more system parameters were randomized, but no 

consideration was given to other system parameters. 

2 One or more system parameters were randomized, and some 

consideration was given to all other system parameters.

3

One or more system parameters were randomized, and those 

parameters that were not randomized between tests were randomized 

in larger test blocks. 

4 All system parameters were completely randomized. 

15 Historical Evidence Levels for Reactor Safety Analysis

16 Level 3 has been most commonly accepted by the NRC staff. In general, the design of the 

17 experiment attempts to randomize the system parameters as much as possible between each 

18 test. Since testing is often split into groups (e.g., a set of tests at a single pressure and/or flow 

19 rate), parameters are often randomized between test groups. For example, if the pressure were

20 held constant during a group of tests, then the pressures from group to group should be 

21 randomized. As much as possible, flow rates are also randomized for a fixed pressure. Because 

22 randomization (i.e., independence) is a key assumption in all of the statistics performed on the 

23 data and because it is generally not possible to guarantee randomization through the design of 

24 the experiment, repeated test points have become a vital part of demonstrating that there are no 

25 biases in the test facility.

 15 By state point, we mean the value of each variable that completely determines the state of the system. 
32

1 G1.2.3—Data Fidelity

2 The method used to obtain CBT data should result in an accurate measurement of CBT. There 

3 are typically two different types of tests used in CBT experiments: (1) those used to obtain

4 steady-state data and (2) those used to obtain transient data. It is vital that assessors understand 

5 exactly what is occurring in each of these tests. Therefore a careful evaluation of the test 

6 constraints, input assumptions, and expected result ranges should be employed.

7 Measuring a Steady-State Data Point

8 For steady-state data, the objective is to determine the state point at which CBT occurs. A state 

9 point is a coordinate in an n-dimensional space defined by all of the parameters which make up 

10 the system. In general, there are two main types of state point: experimental state points and 

11 model state points. For an experimental state point, the parameters of interest are those 

12 parameters that influence the overall experiment (e.g., system pressure, total power (including 

13 radial and axial peaking), inlet flow rate, and inlet temperature). For a model state point, the 

14 parameters of interest are those parameters needed by the model to make a prediction of CHF or 

15 CP. Depending on the model itself, these generally include global as well as local parameters as 

16 well as parameters that are not measured in the experiment (e.g., local mass flux, local quality). 

17 For PWRs, the values of parameters that are not measured in the experiment are obtained using 

18 a subchannel code that predicts the local flow behavior in the subchannels using the experimental 

19 parameters as boundary conditions. In a sense, the subchannel code used can be thought of as 

20 the means by which the experimental state point is transformed into a model state point. 

21 It is important to note that a reactor almost never operates at a steady state, especially during an 

22 AOO. Because the models are based on steady-state data, the model effectively treats each AOO 

23 as if it were made up of a multitude of steady-state state points and determines the heat flux or 

24 assembly power that causes a CBT at those individual state points. Multiple previous applications 

25 of steady-state models have been demonstrated to be conservative (i.e., a model developed with 

26 steady-state data will generally underpredict the heat flux or assembly power that causes a CBT), 

27 and it is common for analysts to provide data demonstrating that this conservative assumption 

28 remains true for each individual CBT model. 

29 The following standard procedure is used to measure a steady-state data point:

30 (1) An experimental state point is chosen. As previously discussed, a single value of 

31 pressure, power, power shape, inlet flow rate, and inlet temperature is generally 

32 chosen. Usually, the initial power is chosen to be somewhat lower than that expected 

33 to cause a CBT.

34 (2) The experimental facility is driven to the state point. Generally, a computer operates 

35 the control system to allow for finer control. 

36 (3) Once the initial state point is reached, power is slowly increased while maintaining 

37 steady conditions on the other experimental parameters. Some variation in the values 

38 of the experimental parameters will exist, but this variation should be kept small and 

39 should be accounted for in test procedures. Although steady-state CBT data could be 

40 obtained by varying any one of the experimental parameters in an appropriate 

41 direction while keeping the others constant (e.g., decreasing the flow rate), such data 

42 are usually obtained by slowly increasing the power.
33

1 (4) As the power is slowly increased, the rod internal thermocouples are monitored. A 

2 CBT is assumed to have occurred if the temperature indicated by one of the 

3 thermocouples increases by a specified amount over a specified small period of time 

4 or if some maximum temperature is reached.

5 (5) Once a CBT occurs, power is reduced and the values of the parameters that make up 

6 the experimental state point are written to a file. These data, along with the known 

7 axial and radial power shape, can then be used to calculate either the CHF or the CP. 

8 Measuring a Transient Data Point

9 The objective for transient data are to determine the lowest power level at which a specific 

10 transient will cause a CBT. In this case, a specific transient is defined through specified 

11 time-dependent functions for each experimental parameter. Typically, a computer controls the 

12 experimental parameters to ensure that the test achieves the desired behavior of the 

13 time-dependent functions. However, not all experimental parameters will vary during the transient 

14 (e.g., pressure is almost never varied because of the strain this would place on the test loop). In 

15 this sense, steady state can be considered a special type of transient where all time-dependent 

16 functions are held constant. 

17 It is also important to note that each AOO is not directly mapped to a specific transient test. 

18 Although some AOOs can be mapped into a transient test (e.g., loss of flow), this is not possible 

19 with all AOOs. AOOs involving rapid changes in pressure are especially challenging because any 

20 rapid change in pressure in the test loop could put the loop at risk. Therefore, additional analysis 

21 is usually performed to determine how the transient testing bounds the AOOs.

22 One of the similarities between transient and steady-state testing is the objective of the test. In 

23 each case, the objective is to determine the minimum power at which a CBT will occur for some 

24 set of initial and boundary conditions. It is important that the focus is on obtaining the minimum

25 power at which a CBT occurs under some set of conditions. Simply finding any power which 

26 causes a CBT is not useful as one can always be caused by any sufficiently high power. For 

example, every conceivable transient will result in a CBT at Graham’s number16 27 of watts, or even

10 28 100 watts (much smaller than Graham’s number of watts). This does not mean that CBT will 

29 occur only at a power of Graham’s number because it will obviously occur at much lower powers. 

30 Therefore, the objective is to determine the minimum power at which a CBT occurs for those initial 

31 and boundary conditions. Thus, if those conditions (either steady state or transient) occur in a 

32 reactor and if that minimum power is not reached, a CBT would not occur.

33 The following standard procedure is commonly used to measure a transient data point: 

34 (1) A specific transient is chosen. As previously discussed, time-dependent functions of 

35 pressure, power, inlet flow rate, and inlet temperature are generally chosen.

36 (2) The experimental facility is driven to the state point. Generally, a computer operates 

37 the control system to allow for finer control. 

 16 Graham’s number is one of the largest numbers known in mathematics. It is many orders of magnitude 

larger than the total number of particles in the observable universe. 
34

1 (3) Once the initial condition state point is reached, the transient is started. The values of 

2 experimental parameters are defined as time-dependent functions that are controlled 

3 to within their desired magnitude by the control system. 

4 (4) The rod internal thermocouples are monitored during the transient. A CBT is assumed 

5 to have happened if the temperature indicated by one of the thermocouples increases 

6 by a specified amount over a specified small period of time.

7 (5) The magnitude of the initial power can be either increased or decreased, and the 

8 same transient can be run again to determine the minimum power at which a CBT 

9 occurs. Frequently, the same transient is performed multiple times to determine the 

10 minimum power.

11 (6) Once the minimum power at which a CBT occurs is known, power is reduced, and the 

12 values of parameters that make up the experimental state point are written to a file. 

13 These data, along with the known axial and radial power shape, can then be used to 

14 calculate either the CHF or the CP for that transient.

15 Table 11 gives the evidence commonly provided to demonstrate that this goal has been satisfied.

16 Table 11 Evidence for G1.2.3—Data Fidelity

G1.2.3 The method used to obtain critical boiling transition data results in an 

accurate measurement.

Level Evidence

1 A reference has been provided that describes the method used to 

obtain results from both steady-state and transient tests. 

2

A reference has been provided that describes the method used to 

obtain both steady-state and transient tests. The assessors have 

examined the reference and believe that it will result in accurate 

measurements of the CBT for both steady-state and transient tests. 

3

A reference has been provided that describes the method used to 

obtain both steady-state and transient tests. The assessors have 

examined the reference and believe that it will result in accurate 

measurements of the CBT for both steady-state and transient tests. 

Additionally, the assessors have observed the method in practice. 

17 Historical Evidence Levels for Reactor Safety Analysis

18 Levels 2 and 3 have been most commonly accepted by the NRC staff. An accurate measurement 

19 of CBT has three main focuses: (1) Ensuring the state point (i.e., pressure, mass flux, inlet 

20 subcooling, power) has been measured and maintained during the entire test run within some 

21 small uncertainty; (2) Ensuring that any CBT that would occur is captured in the data; (3) Ensuring 

22 that the power at which CBT was recorded was the lowest power that would cause a CBT at that 

23 state point. A large part of the review process is spent in gaining an understanding of how the 

24 data are taken, reduced, and then used to generate the model. To that end, observing the 

25 experiment has been one of the most efficient ways to gain this information.
35

1 G1.2.4—Instrumentation Uncertainty Impact

2 Accurate measurements are vital to the success of any experimental program. Therefore, the flow 

3 rates, temperatures, pressures, and powers must be measured accurately and precisely, and their 

4 associated instrumentation uncertainty must be kept low. Typically, the model’s uncertainty does 

5 not directly account for instrumentation uncertainties; instead, such uncertainties are treated as

6 part of the randomness of the data. If those uncertainties are reasonably low over the range for 

7 which the measurements are taken, this assumption is generally valid. Table 12 gives the 

8 evidence commonly provided to demonstrate that this goal has been satisfied.

9 Table 12 Evidence for G1.2.4—Instrumentation Uncertainty Impact

G1.2.4 The instrumentation uncertainties have been demonstrated to have a 

minimal impact on the measured CHF or CP. 

Level Evidence

1 The instrumentation uncertainties have been quantified.

2

The instrumentation uncertainties have been quantified and an 

analysis is used to demonstrate that the uncertainties result in a 

minimal impact on the measured CHF or CP.

OR

The instrumentation uncertainties have not been quantified, but 

repeated test points allow those uncertainties to be captured directly in 

the CHF or CP value.

3

The instrumentation uncertainties have been quantified and an 

analysis is used to demonstrate that the uncertainties result in a 

minimal impact on the measured CHF or CP. This has further been 

demonstrated by experiments (e.g., repeated test points). 

10 Historical Evidence Levels for Reactor Safety Analysis

11 Level 3 has been the most commonly accepted by the NRC. While a quantitative analysis of the 

12 instrumentation uncertainties on the measured CHF or CP values is possible, it is often more 

13 complicated than simply taking additional data points to measure the uncertainty directly. While 

14 such an analysis does assume that the instrumentation’s uncertainty remains constant over the 

15 course of the test, this can usually be confirmed by performing an additional test at the same state 

16 point to generate a repeat test point. 

17 G1.2.5—Repeated Test Points 

18 The instrumentation uncertainty may be obtained from the instrumentation manufacturer or during 

19 calibration. However, the uncertainty on the “measured” CHF or CP at the location of interest 

20 (i.e., the experimental uncertainty) cannot be obtained so easily. This uncertainty is a combination 

21 of the instrument uncertainty; uncertainties of other input parameters (e.g., axial power shape, 

22 selection of the subchannel of interest); and the method used to combine all of the parameters to 

23 generate a “measured” CHF or CP at the location of interest. 

24 Because the CHF or CP at the location of interest cannot be directly measured, the experimental 

25 uncertainty should be determined by obtaining a “measurement” of CHF or CP at the 
36

1 experimental state point multiple times over the entire test cycle and analyzing the variability in the 

2 results. Some variation in the input parameters will occur because obtaining the exact same 

3 experimental state point (i.e., pressure, flow rate, and inlet subcooling) is not possible, but this 

4 variability should be small compared to the uncertainty in the measured CHF or CP value. A 

5 number of repeated test points should be taken at multiple experimental state points and at 

6 various times during the test campaign to ensure that the behavior of the test facility has not 

7 changed and to provide a quantitative estimate of the uncertainty in the “measured” CHF or CP. 

8 The variability in the resulting CHF or CP values should be much lower than the quantified 

9 uncertainty of the model. If it is not, this is evidence that there is an error in determining the 

10 model’s uncertainty. Table 13 gives the evidence commonly provided to demonstrate that this 

11 goal has been satisfied.

12 Table 13 Evidence for G1.2.5—Repeated Test Points

G1.2.5 The uncertainty in the CHF or CP is quantified through repeated tests 

at the same state points.

Level Evidence

1 No repeat test points have been taken. 

2 One repeat test point was taken over the test campaign. The variability 

in the resulting CHF or CP value was reasonably low. 

3

Multiple repeat test points were taken over the test campaign at various 

input parameters. The variability in the resulting CHF or CP values was 

reasonably low. 

13 Historical Evidence Levels for Reactor Safety Analysis

14 Level 2 and Level 3 have been most commonly accepted by the NRC staff. Aside from satisfying 

15 this goal (G.1.2.5), multiple repeat test points (Level 3) can also be used as evidence that the 

16 behavior of the test assembly remains consistent over the time frame of the test. The repeated 

17 test points may become much more important if other aspects of the behavior of the test assembly 

18 are called into question. For example, if there is a geometry change during testing, then the 

19 impact of that change could be determined to be minimal if there are an adequate number of 

20 repeated test points. The variability from repeat test points is typically small compared to the 

21 uncertainty of the CBT model. Additionally, due to the limitations on the statistical design of the 

22 experiment, multiple repeat test points are one way to provide evidence that the errors are indeed 

23 random and that each experimental state point can be considered independent of every other 

24 state point. 

25 G1.2.6—Quantified Heat Losses 

26 Along with accurate flow, pressure, temperature, and power measurements, the test section heat 

27 losses should also be quantified. Because the CHF or CP is obtained from the power 

28 measurement, ignoring the heat losses would result in a “measured” CHF or CP higher than the 

29 “actual” CHF or CP value by the amount of heat loss. This would result in a non-conservative

30 measurement. 

31 Typically, test section heat losses are kept very low through active means. In many cases, the test 

32 section may sit in a heated water bath to ensure minimum heat loss through the walls. Generally, 
37

1 while the absolute value of the test section heat losses to the surroundings increases as the test 

2 assembly power increases, the percentage of the heat losses relative to the test assembly power 

3 actually decreases (i.e., the fraction of heat dissipated in the fluid in the test section is lower for 

4 higher powered tests). Therefore, the bounding heat losses are generally quantified through a test

5 conducted at a low assembly power. The assessor needs to establish whether the measured CHF

6 or CP data were corrected for the heat losses before the development of the CHF or CP model. If 

7 not, the assessor should consider the inherent non-conservatism. Table 14 gives the evidence 

8 commonly provided to demonstrate that this goal has been satisfied.

9 Table 14 Evidence for G1.2.6—Quantified Heat Losses

G1.2.6 The heat losses from the test section are quantified, appropriately low, 

and duly accounted for in the measured data.

Level Evidence

1 Heat losses have been quantified and are minimal, but they have not 

been removed from the power used to calculate the CHF or CP.

2 Heat losses have been quantified and have been removed from the 

power used to calculate the CHF or CP.

10 Historical Evidence Levels for Reactor Safety Analysis

11 Level 1 has been most commonly accepted by the NRC staff. Generally, the percentage of heat 

12 loss is calculated for each test. The percentage of heat loss is usually estimated to be greater 

13 than that of the actual heat loss measured, but it should still be very low compared to the overall 

14 power. Overestimating the heat loss is conservative for the reason given above. While it is 

15 generally desirable to minimize heat losses from the test section, it is not strictly necessary as 

16 long as the heat losses are measured and accounted for the in power measurement. 
38

1 3.1.3 G1.3—Reproduction of Local Conditions

2 The local conditions in the reactor fuel assembly should be reproduced in the test assembly to

3 ensure that experimental data taken in the laboratory apply to the reactor fuel assembly placed in 

4 the reactor. The five subgoals in Figure 7 are used to demonstrate the reproduction of local 

5 conditions. 

6

7 Figure 7 Decomposition of G1.3—Reproduction of Local Conditions

8 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

9 discuss the evidence that could be used to demonstrate that these five base goals have been 

10 satisfied. Additionally, a discussion is provided on the evidence that has been historically used for 

11 CBT models applied in reactor safety analysis.

12 G1.3.1—Equivalent Geometric Dimensions 

13 The test assembly provides the structure in which the flow field will be established. The flow field

14 details, many of which will not be measured or directly reproduced in the computer simulation, will 

15 directly affect the CBT. Therefore, the flow field in the test assembly should be as similar as 

16 possible to the flow field in the reactor fuel assembly. 

17 To ensure a similar flow field, the test assembly is manufactured as a prototypical fuel assembly. 

18 This includes the fuel rod pitch and diameter, guide tube rod location and diameter, part-length 

19 rod height and axial and radial locations, flow areas, number of grid spacers, distances between 

20 grid spacers, grid spacer heights relative to the bottom of the fuel assembly, and total assembly
39

1 height. Each of these dimensions should be within the design tolerances of the reactor 

2 assemblies.

3 For BWRs, the test assembly is typically full size (e.g., 8x8, 9x9, 10x10) or symmetric (5x5).

4 However, for PWRs, a full-size assembly (e.g., 15x15, 17x17) would require a substantial amount 

5 of power. Therefore, smaller 5x5 or 6x6 test assemblies are used. In the early days of CBT

6 testing, 4x4 or smaller assemblies were used, but the unheated channel wall surrounding the test 

7 assembly had too large an effect on the interior subchannels. Therefore, 4x4 (and smaller) 

assemblies are considered too small to provide an adequate representation.

17 8

9 Note that heater rods are potentially subject to large electromagnetic forces caused by the current 

10 flowing through them. These forces must be countered or the rods will bend and the subchannel 

11 flow area will change during testing. In indirectly heated rods, the direction of the current in 

12 adjacent rods can be reversed to counter the electromagnetic forces. However, this is not possible 

13 in directly heated rods because the electric potential must be the same in all rods at each grid 

14 spacer. Therefore, in order to maintain the sub-channel size in directly heated rod bundles, simple 

15 support grids are commonly used. These grids provide structural support and are designed to 

16 have minimal impact on the flow field. Often, the grids are only needed in sections of the 

17 assembly where there are large spans between mixing vane grids. Table 15 gives the evidence 

18 commonly provided to demonstrate that this goal has been satisfied.

19 Table 15 Evidence for G1.3.1—Equivalent Geometric Dimensions

G1.3.1

The test assembly used in the experiment should have geometric 

dimensions equivalent to those of the fuel assembly used in the reactor 

for all major components.

Level Evidence

1

Many of the components in the test assembly have geometric 

dimensions equivalent to those of fuel assemblies used in reactors and

are within the design tolerance of the fuel assemblies that will be used 

in the reactor. Any components that do not have equivalent geometric 

dimensions have dimensions that would result in a conservatively 

lower prediction of the power or heat flux that causes a CBT. 

2

The vast majority of the components in the test assembly have 

equivalent geometric dimensions that are within the design tolerance of 

the fuel assemblies that will be used in the reactor. The few

components that do not have equivalent geometric dimensions would 

have a minimal impact on CBT measurements.

3

All components in the test assembly have equivalent geometric 

dimensions that are within the design tolerance of the fuel assemblies

that will be used in the reactor. 

 17 This is not referred to as the “cold-wall effect,” even though it is due to the impact of the outer cold wall. The

term “cold-wall effect” is reserved for the effect of control rod guide tubes and instrument tubes on CHF 

performance. 
40

1 Historical Evidence Levels for Reactor Safety Analysis

2 Level 2 has been most commonly accepted by the NRC staff. For some older CBT models, the 

3 heated length was varied to cover a wider range of fuel. While this could be understand to be level 

4 1, it would strongly depend on the importance of the heated length in the CBT model. 

5 While there may be instances in which a CBT model may only achieve Level 1, the demonstration 

6 that level 1 is acceptable is challenging as it is difficult to prove that the CBT model would produce 

7 conservative predictions under all conditions. 

8 G1.3.2—Prototypical Grid Spacers 

9 One of the most important parts of the prototypical assembly is the grid spacer. The spacers 

10 ensure that the rods maintain the same pitch as the assembly used in the reactor. The spacers 

11 are also the major source of turbulence which acts to increase the heat transfer from the fuel rods. 

12 Grid spacers are specifically designed to increase the power or heat flux at which a CBT occurs. 

13 In BWRs, the grid spacer is typically designed to increase deposition by directing more of the 

14 water droplets entrained in the vapor flow back onto the liquid film. Great care is taken to ensure 

15 that the liquid film is not separated (stripped) from the fuel rod in the vicinity of the grid spacer. In 

16 PWRs, the grid spacer is typically designed to strip the bubble layer from near the fuel rod surface 

17 to reduce bubble crowding and to enhance turbulence and mixing in the subchannel. 

18 Arguably, the design of the grid spacer will have a larger impact on the CBT than any other input 

19 parameter. The grid spacers increase the margin to CBT through their increase in turbulence or 

20 increase in deposition on the fuel rod. However, the current generation of the computer 

21 simulations that make use of CBT models do not directly simulate the impact of the spacers; 

22 therefore, the CHF or CP model must capture the spacers’ impact. The number of mixing vanes, 

23 the shape of the vanes, the location of the vanes in the subchannel, the surface area of the vanes, 

24 the angle of the vanes, and the direction of swirl caused by the vanes can all affect the thermal 

25 mixing in the fuel assembly subchannel. Therefore, it is vital that the grid spacer used in the test 

26 assembly is prototypical when compared to the grid spacer used in the reactor core.

27 Unfortunately, it is not always possible to use prototypical grid spacers. Therefore, if such grid 

28 spacers cannot be used, the grid spacers used in the test section should result in conservative 

29 behavior compared to the grid spacers in the reactor core. However, it is very difficult to prove that 

30 the one grid spacer will result in conservative behavior under all conditions when compared with 

31 another grid spacer. Therefore, demonstrating conservative behavior can be a challenge. 

32 Additionally, fuel assemblies may be comprised of different grid spacer types at different axial 

33 elevations. Therefore, the same grid types should appear in the test assembly and at the same 

34 elevations as in reactor fuel. Table 16 gives the evidence commonly provided to demonstrate that

35 this goal has been satisfied.
41

1 Table 16 Evidence for G1.3.2—Prototypical Grid Spacers

G1.3.2 The grid spacers used in the test assembly should be prototypical of 

the grid spacers used in the reactor assembly.

Level Evidence

1

The grid spacers used in the test assembly will result in a conservative 

under-prediction of the true thermal mixing caused by the grid spacers 

in the reactor assembly. 

2 The grid spacers used are very similar to those that will be used in the 

reactor assembly but with some slight differences. 

3

The grid spacers used are identical to those that will be used in the 

reactor assembly except for the number of rods (e.g., a 6x6 cutout of a 

17x17 assembly). 

4

The grid spacers used are identical to those that will be used in the 

reactor assembly (either identical in size or a symmetric cut of the grid 

spacer). 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 3 has been most commonly accepted by the NRC staff for PWRs, and Level 4 has been

4 most commonly accepted for BWRs. PWRs typically operate at a higher linear power density, 

5 have more rods per assembly, and have fewer assemblies per core. Therefore, it is impractical to 

6 test an entire PWR assembly in a test facility because the power needed would be too high. 

7 Additionally, PWR methods use a true subchannel analysis and, therefore, model the grid

8 spacers’ impact on the local fluid quantities. On the other hand, BWR methods use a simplified 

9 subchannel analysis that considers only assembly-averaged flow parameters and, therefore, calls 

10 for experimental details on every fuel rod in the assembly. For this reason, BWR tests use 

11 full-sized assemblies or representative symmetric sub-assemblies.

12 Levels 1 and 2 are not common in reactor safety analyses, as even small changes in the grid 

13 spacer can have major impacts to the flow field. 

14 G1.3.3—Axial Power Shapes 

15 It is important to reproduce the local powers created by the reactor assembly in the test assembly. 

16 This is generally done by testing combinations of axial and radial power shapes. Although the fuel 

17 rods in the reactor can take on an almost infinite number of axial power shapes, generally only 

18 three shapes (cosine, up-skew, and down-skew) are used in testing for BWR models, and three 

19 shapes (uniform, cosine, and up-skew) are used in testing for PWR models. Additionally, because 

20 of the current experimental designs, the only way to change the axial power shape even in 

21 modern CBT testing is to replace the test rods, which is a major undertaking. Every test rod, 

22 regardless of whether it is directly or indirectly heated, is constructed to produce a specific axial 

23 power shape. 

24 In a directly heated rod, the rod is connected to a power source at the top and bottom, and 

25 electricity flowing through the rod itself generates the heat for the test. The axial power shape is 

26 manufactured into the rod by adjusting the rod’s wall thickness—this impacts the rod’s electrical 

27 resistance and hence the power produced at different elevations. The outside rod diameter is held 
42

1 constant, and the inside diameter is changed to make the rod’s cross-sectional area thicker or 

2 thinner. If the rod wall’s cross-sectional area is increased by making the wall thicker, the resistivity 

3 of that section will decrease, and the power produced per unit length will decrease. Conversely, if 

4 the rod wall’s cross-sectional area is decreased by making the wall thinner, the resistivity of that 

5 section will increase, and the power will increase. Because the highest rod power occurs at the 

6 thinnest areas, which are not easy to manufacture, the uncertainty on this peak power 

7 (i.e., thickness of the rod) was historically one of the largest uncertainties in the experiment.

8 In an indirectly heated rod, a heating coil is placed inside the rod and the power shape is 

9 controlled by modifying the dimensions of the coil. This coil is then slid into a clad, which acts as

10 the surface of the test rod. Because PWR testing calls for high heat fluxes, PWR testing generally 

11 uses directly heated rods. BWR testing may use either directly or indirectly heated rods. 

12 Although any number of axial power shapes could be prescribed in the manufacturing of the rods, 

13 typically the rods will have one of four shapes: (1) uniform, (2) cosine, (3) up-skew, or

14 (4) down-skew. Aside from the uniform power shape, each power shape represents a different 

15 situation or a different time in the core life. Historically, the uniform power shape was the first 

16 power shape used in testing because of the ease of manufacturing (i.e., tubes of a constant wall 

17 thickness). However, such a shape always results in a CBT at the very top of the assembly. This 

18 situation is considered unphysical (i.e., it does not occur in actual reactors), and questions have 

19 recently been raised (Yang et al., 2014) on the usefulness of such uniform test data.

20 Consequently, the uniform power shape has been used less frequently in modern CBT testing.

21 Because early CBT data were based on testing that assumed a uniform power shape, a method 

22 was needed to convert the model’s predictions so the models could be used for the nonuniform 

23 power shapes that occur in reactors. One method used was the Tong factor (Tong et al., 1965). 

24 Initially, the Tong factor was not a part of the CHF model. Instead, it was used to “correct” the 

25 prediction of the CHF model. The factor attempts to adjust the predicted CHF based on the given 

26 axial power shape, some information on local conditions, and the elevation under consideration. 

27 However, as CHF models have developed, this shape dependence has become more integrated 

28 into the model itself.

29 Ultimately, it is important to ensure that the axial power shapes tested bound all possible power 

30 shapes for which the CBT model will be used. One way to demonstrate this is by training a model

31 (i.e., statistically determining its coefficients using regression) with one axial power shape and 

32 validating it with another. Table 17 gives the evidence commonly provided to demonstrate that this 

33 goal has been satisfied.
43

1 Table 17 Evidence for G1.3.3—Axial Power Shapes

G1.3.3 The axial power shapes in the test assembly should reflect the 

expected or limiting axial power shapes in the reactor assembly.

Level Evidence

1

Only one axial power shape was used in the test assembly. However, 

a justification for why the single axial power shape was sufficient is 

provided. 

2

The commonly tested axial power shapes were used in the test 

assembly. Further, an explanation of why those shapes were 

appropriate was provided. 

3

A number of axial power shapes were used in the test assembly. 

Further, it was demonstrated that the CBT model was able to make 

accurate predictions of axial power shapes whose data were not used 

as training data for the model. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 and Level 3 have been most commonly accepted by the NRC staff. Generally, cosine, 

4 up-skew, and down-skew power shapes are used for BWR fuel testing, and cosine and up-skew 

5 (and maybe uniform) power shapes are used for PWR fuel testing. Level 1 has been used in the 

6 past to confirm a model’s behavior on similar fuel or to make a small modification to an existing 

7 model but not to qualify a new model. Level 3 is sometimes used as it is often easier to 

8 demonstrate through test data that the CBT model is insensitive to axial power shape than to 

9 provide other justification. 

10 G1.3.4—Radial Power Peaking 

11 It is important to reproduce the local powers experienced by the reactor assembly in the test 

12 assembly. Generally, this has been done by testing a combination of axial and radial power 

13 shapes. Varying the radial power shape (i.e., radial power peaking) is generally much easier than 

14 varying the axial power shape because it can be done by simply supplying more power to select 

15 rods in the test assembly and does not necessitate replacing the rods in the assembly. 

16 The importance of the radial power peaking is different for BWR and PWR testing. In PWR 

17 testing, the radial power peaking tends to be used to ensure that the CBT occurs away from the 

18 outside wall and near the central locations of the test assembly. Because the assembly is only a 

19 portion (e.g., 5x5, 6x6) of the entire assembly (e.g., 14x14, 17x17), there is a desire to ensure that 

20 the CBT occurs closer to the center of the test assembly and away from any edge effects of the 

21 wall, as such a boundary does not exist in an open lattice core. The model predicting a CBT is 

22 applied over every subchannel in a fuel assembly, and the resulting predicted CHF is compared to 

23 the heat flux from the fuel rods. Although the radial power peaking will affect the heat flux from the 

24 fuel rods and consequently the local fluid conditions, the computer code directly simulates all of 

25 those impacts. 

26 However, the radial peaking in BWR testing serves a different purpose as a result of how BWR 

27 CP correlations are applied. In the current generation of CP correlations, assembly-average 

28 thermal-hydraulic conditions and pin powers are used as inputs to the correlation. The margin to 

29 dryout in the assembly is then calculated based on the limiting R- or K-factor. R- or K-factors are 
44

1 calculated for each rod based on the pin power distribution of the surrounding rods and the rod 

2 additive constant, which is a correlated parameter developed for each rod. Radial power peaking 

3 in BWR testing is therefore used to drive different rods into dryout so an additive constant can be 

4 determined for each individual rod (or its symmetric partners). This constant accounts for the local 

5 thermal-hydraulic conditions in the fluid surrounding the rod in a way that is similar to the 

6 subchannel code used in PWR CHF analysis. The testing should be performed over the full range 

7 of R- and K-factors expected in the reactor so that the local thermal-hydraulic effects are properly 

8 captured in the additive constant.

9 Because of this difference between PWR and BWR CBT modeling, the criteria for BWRs and 

10 PWRs are different. Table 18 gives the evidence commonly provided to demonstrate that this 

11 criterion (PWR only) has been satisfied.

12 Table 18 Evidence for G1.3.4—Radial Power Peaking (PWR)

G1.3.4 The radial power peaking in the test assembly should reflect the 

expected or limiting radial powers in the reactor assembly.

Level Evidence

1 Radial power distributions are consistent with those peaking factors 

expected in reactor fuel.

2 Radial power distributions are higher than those peaking factors 

expected in reactor fuel.

3

Radial power distributions in the test rods result in a hot subchannel 

(i.e., a subchannel surrounded by peaked rods that have higher 

peaking factors than those normally expected in reactor fuel). 

13 Historical Evidence Levels for Reactor Safety Analysis

14 Level 3 has been most commonly accepted by the NRC staff for PWR fuel. Generally, the hot 

15 subchannels are designed toward the interior of the test assembly to ensure the CBT does not 

16 occur on an exterior rod, which may be influenced by the channel wall. 

17

18 Table 19 gives the evidence commonly provided to demonstrate that this criterion (BWR only) has 

19 been satisfied.
45

1 Table 19 Evidence for G1.3.4—Radial Power Peaking (BWR)

G1.3.4 The radial power peaking in the test assembly should reflect the 

expected or limiting radial powers in the reactor assembly.

Level Evidence

1 A wide range of radial power peaking was tested. 

2

The testing procedure ensured that each rod experienced dryout in 

multiple tests over multiple different radial power distributions, thus 

ensuring the thermal-hydraulic behavior captured in the R- or K-factor 

and any rod additive constant would be based on the appropriate rod 

behavior. 

3

The testing procedure ensured that each rod experienced dryout in 

multiple tests over multiple different radial power distributions, thus 

ensuring the thermal-hydraulic behavior captured in the R- or K-factor 

and any rod additive constant would be based on the appropriate rod 

behavior. Additionally, the radial power peaking tested bound the 

possible radial powers that could be observed during normal conditions 

and any transients. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff for BWR fuel. Generally, the tests 

4 are focused on peaking each rod in the assembly to ensure a sufficient database for calculating

5 the additive constant. Often, every single rod in the assembly does not need to be peaked

6 because there is some flow symmetry; therefore, only some locations need to be investigated, 

7 assuming the assembly behaves symmetrically. If the assembly does not behave symmetrically, 

8 more rods in the assembly would need to be peaked to obtain measurements of their 

9 performance. 

10 G1.3.5—Differences in the Test Assembly

11 The test assembly used in the experiment and the actual fuel assembly used in the reactor should 

12 have few differences, if any. Because much of the important flow behavior of the assembly is not 

13 modeled in the computer simulation but captured through the empirical CBT model, the test 

14 assembly used to generate that model must be very similar to the actual fuel assembly. However, 

15 the two assemblies will likely always have small differences that must be understood and 

16 demonstrated to have little-to-no impact. Table 20 gives the evidence commonly provided to 

17 demonstrate that this goal has been satisfied.
46

1 Table 20 Evidence for G1.3.5—Differences in the Test Assembly

G1.3.5

Any differences between the test assembly and the reactor assembly

should have a minimal impact on the flow field. This includes 

components that are not in the reactor assembly but are needed for 

testing purposes.

Level Evidence

1

The main flow features of the test assembly are the same as those of 

the fuel assembly, with analysis demonstrating that all differences are

small. 

2

The main flow features of the test assembly are the same as those of 

the fuel assembly, with experiment demonstrating that all differences 

are small. 

3 The test assembly is identical to a symmetric portion (e.g., 5x5) of the 

actual fuel assembly. 

4 The test assembly is identical to the actual fuel assembly. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 3 has been most commonly accepted by the NRC staff for PWR fuel because of the 

4 reduced fuel assembly size (a 17x17 reactor assembly is a 5x5 or 6x6 test assembly) and the use 

5 of support spacers. Level 3 or level 4 is most common for BWRs because the entire assembly (or 

6 a very large portion of it) can often be used in the test. Levels 1 and 2 are uncommon, as it is very 

7 difficult to justify the use of a CBT model on fuel which is very different from that which was tested. 

8 There are known issues that create deviations between the test assembly and the fuel assembly 

9 used in the reactor. For example, in BWR testing, the part-length rods can sometimes prove 

10 problematic; therefore, the test assembly may be very similar but not exactly identical to the actual 

11 fuel assembly. Because these experiments are very costly and very difficult, differences between 

12 the test and fuel assembly are not uncommon. In some past cases, data were discarded because 

13 of such differences, and additional testing had to be conducted. In other cases, the differences 

14 were small enough that the data were acceptable for use and additional testing was unnecessary. 

15 Much is left to the experience and engineering judgment of the assessor and the analyst.

16 G2—Model Generation

17 The statement “The model has been generated in a logical fashion” is intentionally broad because

18 the decision to rely on the model rests mostly on the validation data rather than its method of 

19 generation. Additionally, a model could be generated in many ways, and any or every one of 

20 those ways could be acceptable. Arguably, it would be possible to guess both the model form and 

21 coefficients. If such a model were appropriately validated, showed reasonable physical behavior 

22 over the range of its intended use, and had quantified uncertainty, there would be no reason to 

23 disallow the use of that model, even though it was based on a guess. 

24 Although any number of methods could be used to generate a CBT model, understanding what 

25 method was used and the reasoning behind that method is helpful to the assessor. Therefore, the 

26 criteria in this section are less focused on ensuring that a specific method was followed and more 

27 focused on ensuring that whatever method was followed is explained and is logical. 
47

1 The field of machine learning has addressed the general process used to generate a model (and 

2 many of the concerns in that process). Therefore, many of the concepts and terms used in that 

3 field will be used here. The two subgoals in Figure 8 are used to demonstrate that the model was 

4 generated in a logical fashion. 

5

6 Figure 8 Decomposition of G2—Model Generation

7 3.2.1 G2.1—The Mathematical Form

8 The mathematical form of the model must be appropriate in that all relevant parameters appear as 

9 variables in the model and the model form itself is reasonable. Typically, the mathematical form of 

10 the model is chosen based on an organization’s past experience. The two subgoals in Figure 9

11 are used to demonstrate that the mathematical form of the model is appropriate.

12

13 Figure 9 Decomposition of G2.1—The Mathematical Form

14 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

15 discuss the evidence that could be used to demonstrate that these two base goals have been 

16 satisfied. Additionally, a discussion is provided on the evidence which has been historically used 

17 for CBT models applied in reactor safety analysis.
48

1 G2.1.1—Necessary Parameters

2 CHF models are typically represented as a function of several (5 to 10) parameters, where each 

3 variable is generally based on a local parameter in the subchannel. The following are the most 

4 common parameters: 

5 • pressure

6 • local mass flux 

7 • local quality

8 • inlet enthalpy

9 • heated hydraulic diameter (to account for any cold-wall effect)

10 • grid spacing

11 • other flow or geometry parameters

12 CP models are also represented by functions of several variables but typically not by local 

13 parameters of the subchannel; instead, they are generally based on fuel assembly inlet 

14 parameters, including the following: 

15 • pressure 

16 • inlet mass flux

17 • inlet subcooling 

18 • R- or K-factor (related to local peaking)

19 • additive constant (related to the flow/enthalpy redistribution of a specific spacer design)

20 • other flow or geometry parameters

21 Pressure

22 Pressure can have a first-order impact on the fluid properties, the flow regime, and thus the 

23 predicted CBT. Most AOOs occur at pressures close to the system pressure. The major exception 

24 to this is the main steamline break in a PWR, which typically has the lowest pressure of any 

AOO18 25 . Because the pressure encountered during a main steamline break is usually much lower 

26 than the normal operating pressure, a specific low-pressure CHF model is often used. 

27 Mass Flux

28 For PWRs, a local mass flux is used in the calculation of the CHF. This local mass flux is obtained 

29 from a subchannel code because PWRs have an open lattice core, and significant mixing 

30 between fuel assemblies can occur. For example, it is a common practice in PWR safety analyses 

31 to conservatively model the flow entering the hot assembly by reducing it by a small percentage. 

32 Because it is an open lattice core, the flow redistributes rather quickly, and this impact is almost 

33 negligible after only a few grid spacers. However, at higher axial elevations, the hotter 

34 subchannels will generate increased vapor, thus increasing the pressure drop and driving fluid to 

35 other subchannels (and potentially into adjacent assemblies). Because the local mass flux 

36 calculated by the subchannel code can have a first-order effect on the prediction of the CBT, the 

37 code (and all of the selected modeling options) is considered part of the CHF model. Any change 

 18While a main steam line break is formally classified as an accident and not an AOO, many plants analyze them to 

the stricter standard of an AOO. Limited fuel failure is permitted in a postulated accidents where no fuel failure is 

permitted in an AOO. 
49

1 to the code or selection of any different modeling options would warrant revalidation of the CHF 

2 model with the new code or modeling options. 

3 For BWRs, the local mass flux is typically not necessary because the fuel assembly is bounded by 

4 its channel, and mixing between assemblies does not occur. Therefore, CBT can be correlated to 

5 the inlet mass flux. Although a mass exchange occurs between the vapor flow, the liquid droplets, 

6 and the fluid film, this exchange is modeled through the calculation of the quality, and the CP

7 model itself captures the entire process. 

8 Local Quality

9 For PWRs, the local quality has a first-order impact on the CBT. One thing which seems to have a 

10 large impact on the local quality is the power shape. Tong’s factor (or similar shape factors)

11 accounts for different axial power shapes by reducing (or increasing) the heat flux that is needed 

12 to predict a CHF. Tong’s factor is supposed to account for the “history” of the flow that would be 

13 affected by axial power shape. One theory is that the Tong factor accounts for the radial distance 

14 between the heated wall, the void location in the flow, and the void concentration. Although the 

15 quality calculated is the total quality of the subchannel, it is quality “near the wall” that would likely 

16 have the largest impact on CHF. Thus, a shape factor like Tong’s is used to account for this 

17 quality distribution in a specific cell of the subchannel. Voids closer to the wall may result in a 

18 lower CHF than would voids in the center of the channel because voids at the wall could influence 

19 bubble crowding and hence influence the CHF.

20 For BWRs, the local quality is more of a predictive parameter than a correlating parameter. Many 

21 CP models correlate the current boiling length to a critical quality. In such a model, ensuring that 

22 CP has not occurred is synonymous with ensuring that the current quality is lower than the critical 

23 quality. 

24 Inlet Enthalpy

25 The inlet enthalpy is used to determine how close the inlet flow conditions are to boiling (e.g., inlet 

26 subcooling). If the inlet subcooling is high, boiling will generally occur at higher axial elevations in 

27 the fuel assembly, and a higher power will be needed to cause a CBT. Although inlet subcooling 

28 can be low, some amount of inlet subcooling is typically necessary or else the start of boiling can 

29 occur outside of the fuel assembly and it is not possible to define a boiling length. Models that 

30 correlate boiling length to a critical quality inherently assume that the entire boiling length will be in 

31 the fuel assembly, which would therefore typically imply that the flow enters the assembly with 

32 some subcooling. Even if this assumption is not used, it is usually very difficult to test conditions 

33 with zero or negative inlet subcooling (i.e., flow is already boiling).

34 Inlet subcooling is not as relevant for CHF models as they focus more on local conditions. More 

35 generally, PWRs operate with inlet conditions that are much farther from saturation (i.e., more 

36 subcooled) than BWRs. However, experimental validation should be used to confirm that the flow 

37 at the inlet is subcooled if necessary.

38 Heated Hydraulic Diameter

39 Typically, for CHF models, the subchannel heated hydraulic diameter (or a ratio of the heated 

40 hydraulic diameter to the true hydraulic diameter) is used instead of the actual hydraulic diameter 

41 because of the difference between the behavior of a subchannel surrounded by four rods and the 

42 behavior of a subchannel that contains an unheated guide tube. The guide tube is considered a 
50

1 “cold wall”; therefore, its impact is known as the “cold-wall effect.” Although a guide tube may 

2 change the hydraulic diameter of a subchannel, some guide tubes are of similar size to a fuel rod

3 and, therefore, would have minimal impact on the hydraulic diameter of the channel. However, 

4 because the guide tube is unheated, it would have a large impact on the heated hydraulic 

5 diameter. 

6 Although it is important to explicitly account for the cold-wall effect in PWRs, it is not directly 

7 addressed in BWRs. Generally, the K- or R-factor and the additive constants would account for 

8 any impact from the water rods or channel box in a BWR. 

9 Grid Spacing

10 If the grid spacing (i.e., the distance between two grids) does not vary for a fuel design, obtaining

11 test data at multiple grid spacings is not necessary. However, if the grid spacing can change 

12 (e.g., intermediate flow mixers are positioned between some spacer grids), the effect of the 

13 distance between all possible combinations of the grids should be accounted for the CBT model.

14 Typically, CBT occurs just upstream of (i.e., below) a grid spacer. For PWRs, the turbulence is 

15 maximized just downstream of (i.e., above) a grid and decreases as the fluid travels further from 

16 the grid, reaching a minimum just upstream of the next grid. Therefore, longer spans between 

17 grids result in more reduction in turbulence and less mixing, thus increasing the potential for a 

18 CBT. For BWRs, the grids direct the droplets entrained in the vapor core to the liquid film on the 

19 fuel rod, thus increasing the liquid film thickness. However, as the flow moves downstream from 

20 the grid, the additional deposition caused by the grid decreases and the liquid film evaporates and 

21 is entrained by the vapor flow. If the deposition rate falls off too quickly or if evaporation or 

22 entrainment is too great, the film may dry out before it reaches the next grid where deposition will 

23 increase once again. 

24 Additionally, the grids themselves act as fins. Thus, while a CBT would be expected to occur just 

25 upstream of a grid, it would be highly unlikely to occur inside a grid because some amount of heat 

26 transfer occurs from the rod to the grid and the grid to the coolant. Additionally, the grids 

27 themselves are often covered in water, either from the continuous flow field in a PWR or from 

28 droplets in a BWR.

29 R- or K-Factor and Additive Constants (BWR only)

30 The R- or K-factors and additive constants account for the impacts of various phenomena on CP

31 predictions for each rod position. The additive constants are terms that account for the increase or 

32 decrease in mixing at some xy location in the grid assembly. These terms are obtained from 

33 experimental testing and generally stay fixed for a particular rod xy location. The R- or K-factors

34 include the impact of the various power levels of the surrounding rods on the rod in question. 

35 These factors and constants have been colloquially termed the “poor man’s subchannel code.” 

36 Instead of simulating a large number of subchannels in the hot assembly, a BWR analysis will 

37 simulate only a single rod surrounded by a single subchannel at assembly-averaged conditions. 

38 The R- or K-factors are then used, along with the additive constants, to determine the behavior of 

39 the rods at each xy location in the assembly. 

40 Other Parameters

41 CBT models may use other parameters. Historically, the heated length has been used, but recent 

42 work suggests that this is not the best length parameter to correlate against because the boiling 
51

1 length (i.e., distance from the start of boiling to the current location under consideration) has a 

2 larger impact on the CBT (Wieckhorst et al., 2013; Wieckhorst et al., 2015).

3 Table 21 gives the evidence commonly provided to demonstrate that this goal has been satisfied. 

4 Table 21 Evidence for G2.1.1—Necessary Parameters

G2.1.1 The mathematical form of the model contains all the necessary 

parameters.

Level Evidence

1 The model contains all the parameters measured in the experiment. 

2

The model parameters include those which have been commonly used 

in previous models and are considered to be the parameters that have 

the most significant impacts on a CBT. 

3 It is demonstrated from first principles that the model contains all the 

necessary parameters. 

5 Historical Evidence Levels for Reactor Safety Analysis

6 Level 2 has been most commonly accepted by the NRC staff. Typically, the CBT model includes a 

7 few parameters in addition to those measured in the experiment. Level 3 is considered an ideal 

8 situation, and the authors are not aware of a complete first-principle understanding of phenomena 

9 associated with a CBT. This is especially true for DNB, for which the phenomenon involved is 

10 much more complex than dryout because it involves multiple length scales and a strong 

11 dependence on turbulence. It is possible that a claim of thorough understanding of the first 

12 principles of CBT could be demonstrated by developing a correlation using very little training data 

13 and validating it against a wide variety of conditions.

14 G2.1.2—Reasoning for the Mathematical Form

15 Currently, there is no known “best” mathematical form for CBT models, which are expressed as

16 multivariate functions because a complete first-principle understanding of the underlying 

17 phenomena does not exist. Additionally, because of nonlinear behavior, it may be difficult to 

18 separate the impact of the chosen mathematical form and the impact of the chosen values for 

19 coefficients. Thus, even identical mathematical forms can behave much differently with different 

20 choices of coefficients. Although there is no single “correct” way to generate the mathematical 

21 form, the method behind generation of the form should be described to ensure that it is 

22 reasonable to the assessor. Additionally, because the validation process will quantify the model’s 

23 uncertainty, this criterion focuses on understanding how the mathematical form was generated 

24 rather than on ensuring that it was generated in a particular manner. Table 22 gives the evidence 

25 commonly provided to demonstrate that this goal has been satisfied.
52

1 Table 22 Evidence for G2.1.2—Reasoning for the Mathematical Form

G2.1.2 The reasoning for choosing the mathematical form of the model should 

be discussed and should be logical. 

Level Evidence

1 The basis of the model’s mathematical form is described. 

2

The basis of the model’s mathematical form is described. The 

description includes the development of the form and justification of the 

essential elements of the form.

3

A very thorough description of the origins of the mathematical form of 

the model is provided. This description includes the history of the form, 

the justifications for using the form, and the process for generating the 

form. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. In many cases, the development of 

4 the mathematical model has occurred over the course of many years and has been influenced by 

5 numerous factors. Although it is helpful for the assessor to understand this history, and it has 

6 previously increased the review efficiency, it is not strictly necessary. Thus, Level 3 and Level 1 

7 are not uncommon. 

8 In general, as long as the model has been validated with data that covers its expected range of 

9 use, contains all the necessary parameters, and has a logical form, then the specific form of the 

10 model would have a minor impact on model predictions. A model with a logical form will generate 

11 relevant predictions over the entire application domain. Trends between data points should be 

12 reasonable in that the model should not be discontinuous and the trends should be well-behaved 

13 mathematically. Because there are a large variety of mathematical forms that could be chosen, 

14 the specific form should not result in unreasonable predictions (e.g., very high, very low, negative, 

15 complex numbers) inside the expected domain. 
53

1 3.2.2 G2.2—Method for Determining Coefficients

2 The process for determining the values of the model’s coefficients should be appropriate. Again, 

3 the meaning of “appropriate” in terms of a model’s coefficients is vague. Although only a single set

4 of the coefficients would result in the lowest error, as judged by some norm (e.g., the Euclidian 

5 norm), minimizing this error is often not the most important criterion when determining the 

6 coefficient values. Instead, great care is usually taken to ensure that the model reflects actual

7 physical behavior rather than simply minimizing the error. Thus, many of the coefficients for a 

8 model are chosen to ensure that the model has certain desired trends. The three subgoals in 

9 Figure 10 are used to demonstrate that the method for determining the coefficients is appropriate. 

10

11

12

13 Figure 10 Decomposition of G2.2—Method for Determining Coefficients

14 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

15 discuss the evidence which could be used to demonstrate that these three base goals have been 

16 satisfied. Additionally, a discussion is provided on the evidence that has been historically used for 

17 CBT models applied in reactor safety analysis.

18 G2.2.1—Identification of Training Data

19 The training data are the experimental data used to generate the coefficients of the model. They

20 are distinguished from the validation data, which are the experimental data that are used in the 

21 validation process. Ideally, different data should be used for each role. Typically, some large 

22 percentage (usually between 70 and 100 percent) of the experimental data will be used as training 

23 data. Table 23 gives the evidence commonly provided to demonstrate that this goal has been 

24 satisfied.
54

1 Table 23 Evidence for G2.2.1—Identification of Training Data

G2.2.1 The training data (i.e., the data used to generate the coefficients of the 

model) should be identified. 

Level Evidence

1 100% of the experimental data are used as training data. 

2 Between 90–100% of the experimental data are used as training data. 

3 Between 80–90% of the experimental data are used as training data. 

4 Between 70–80% of the experimental data are used as training data. 

5 Between 60–70% of the experimental data are used as training data. 

6 Between 50–60% of the experimental data are used as training data. 

7 Between 40–50% of the experimental data are used as training data. 

8 Between 30–40% of the experimental data are used as training data. 

9 Between 20–30% of the experimental data are used as training data. 

10 Between 10–20% of the experimental data are used as training data. 

11 Between 0–10% of the experimental data are used as training data. 

12 None of the experimental data are used as training data. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Levels 1–3 have been most commonly accepted by the NRC staff. As there is no minimum or 

4 maximum portion of the data that should be used to train the model, this criterion focuses more on 

5 identifying what data are used to train the model rather than on ensuring that a certain amount is 

6 (or is not) training data. 

7 In general, all experimental data should be either training or validation data. Thus, if 70 percent of 

8 the data are training data, the remaining 30 percent could be used as validation data. Section 3.3

9 discusses the criteria on the amount of validation data. However, one way to demonstrate the 

10 power of a specific model is to have a very small percentage of training data and a very large 

11 percentage of validation data.

12 G2.2.2—Calculation of the Model’s Coefficients

13 Again, there is typically no single best way to calculate the model’s coefficients. For PWRs, 

14 because of the simplicity of the CHF model, the focus is typically on reducing overall error. 

15 However, the models for BWRs are generally more complex; therefore, the focus is typically on 

16 ensuring that the model has the desired behavior as a function of certain parameters. Whichever 

17 method is used, the assessors should understand that method. Table 24 gives the evidence 

18 commonly provided to demonstrate that this goal has been satisfied.
55

1 Table 24 Evidence for G2.2.2—Calculation of the Model’s Coefficients

G2.2.2 The method for calculating the model’s coefficients should be 

described. 

Level Evidence

1 A brief description of the method for calculating the model’s coefficients 

is provided. 

2 A detailed description of the method for calculating the model’s 

coefficients is provided. 

3

A very thorough description of the method for calculating the model’s

coefficients is provided. This includes the walkthrough for gathering the 

experimental data, the data reduction process, and the methods used 

to generate the coefficients. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. The method for calculating the 

4 model’s coefficients tends to be very detailed. The models are treated as strictly data driven 

5 models (i.e., empirical or semi-empirical) in that there is no assumption that the model form 

6 contains any ability to predict the physics besides that which it demonstrates through its validation.

7 While it is possible that the model form may be based on equations from first-principle physics, it 

8 is not assumed that the model contains any inherent ability to predict the underlying physical 

9 mechanisms of the CBT. Therefore, there is no “best practice” in terms of the manner in which the 

10 model’s coefficients are calculated. 

11 Because the model’s uncertainty will be quantified with validation data, choosing the model’s 

12 coefficients is mostly focused on reducing the model’s uncertainty. In the extreme case, the 

13 model’s coefficients could be guessed and, as long as the model’s uncertainty is quantified, the 

14 model would still be acceptable for use (all non-linear regressions require a guess of the model

15 coefficients as a starting point). Further, it is common for the model coefficients to be chosen to 

16 ensure some known behavior over specific ranges of the model, and not simply to ensure the 

17 smallest validation error. 

18 G2.2.3—Calculation of Model-Specific Factors and Constants (BWR Only)

19 The R- or K-factor and additive constants are part of the coefficients of the model itself. However, 

20 they are often treated separately from the calculation of other coefficients in the model. They are a 

21 very important part of BWR simulations because they allow local fuel rod behavior to be modeled 

22 without using detailed local conditions, so their generation should be well understood. Table 25

23 gives the evidence commonly provided to demonstrate that this goal has been satisfied.
56

1 Table 25 Evidence for G2.2.3—Calculation of Model-Specific Factors and 

2 Constants

G2.2.3

The method for calculating the R- or K-factor and the additive constants 

(for both full-length and part-length rods) should be described. Further, a 

description of how such values are calculated if dryout is not measured 

on the rod under consideration should be provided (BWRs only).

Level Evidence

1 A brief description of the method for calculating these values is provided.

2 A detailed description of the method for calculating these values is 

provided.

3

A very thorough description of the method for calculating these values is 

provided. This includes a walkthrough for gathering the experimental 

data, the data reduction process, and the methods used to generate 

these values. 

3 Historical Evidence Levels for Reactor Safety Analysis

4 Level 2 has been most commonly accepted by the NRC staff. The method for calculating the R- or 

5 K-factor and additive constants tends to be very detailed. It is important for the assessor to 

6 understand the process so that he or she can confirm that the behavior modeled in the R- or K7 factors and additive constants would result in a reasonable prediction of CBT in a BWR. 

8 G3—Validation through Error Quantification

9 Validation is the accumulation of evidence used to assess the claim that a model can predict a 

10 physical quantity (Oberkampf and Roy, 2010). Thus, validation is a never-ending process 

11 because more evidence can always be obtained to bolster this claim. However, at some point, 

12 when the accumulation of evidence is considered sufficient to make the judgment that the model 

13 can be trusted for its given purpose, the model is said to be validated. This is not to say that 

14 further validation would not be useful but rather that it is believed that the validation currently 

15 provided demonstrates that the model can be trusted for its specific use. The authors believe that 

16 Anderson and Bates were very wise to begin the first chapter of their book on validation 

17 (Anderson and Bates, 2001) with a quote from the National Research Council: “Absolute validity 

18 of a model is never determined” (National Research Council, 1990).

19 Because of the desire to ensure that the model’s prediction is conservative, any bias or 

20 uncertainty, or both, in the model’s prediction of CHF or CP should be adequately quantified such 

21 that safety analyses can account for it. This process is uncertainty quantification. The first step in 

22 this process is to use the experimental data (i.e., the validation data) along with the model’s 

23 prediction of that experimental data to calculate the validation error. If the validation error is 

24 appropriately distributed through the model’s application domain and if any inconsistencies in the 

25 validation error are accounted for, statistics from the validation error can be used to determine the 

26 model’s uncertainty. The five subgoals in Figure 11 are used to demonstrate that the model has 

27 sufficient validation through the quantification of its error.
57

1

2 Figure 11 Decomposition of G3—Validation through Error Quantification

3

4 3.3.1 G3.1—Calculating Validation Error

5 Typically, model error is thought of as the difference between the actual value that occurs in 

6 nature and the predicted value of the model. If the model is simple enough or if the experiment is 

complex enough, the measured value from the experiment can be used as the actual value.

19 7

8 Ideally, the error could be calculated from the measured value of the instrumentation and the 

9 model’s prediction under the same conditions. However, this is often oversimplification. Instead, 

10 the error of interest should not be the model error but the model application error (i.e., what is the 

11 error of using the model in the same manner as it will be applied in the safety analysis).

12 To clarify, one way to calculate the model error is to measure the heat flux or power at the location 

13 of a CBT and consider this the measured value and then use the CBT model along with the flow 

14 conditions at the time of the CBT to obtain a predicted value at that same location. However, a 

15 CBT model is generally not applied in this manner. First, it is typical for multiple rods to experience 

16 a CBT at the same time. Second, it is typical for the same rod to experience a CBT at different 

17 elevations at the same time. Third, the definition for a rod experiencing a CBT is somewhat 

18 variable. Generally, the criteria for determining the occurrence of a CBT is some specified change 

19 in temperature over a short time span. During testing, a number of thermocouples may register a 

20 change just under this amount; therefore, the rods are not considered to have experienced a CBT. 

21 However, under this definition it is possible that a CBT may still have occurred. These challenges 

22 could make determination of a single measured value from an experiment very difficult. 

23 Additionally, the objective is not to ensure that the CBT model can be trusted for predicting the 

24 behavior of an experiment for which the heat flux or power that causes a CBT are known; instead,

25 the objective is to determine whether the model can be trusted when applied in a reactor safety 

26 analysis where the heat flux or power will be unknown. The interest is not in the model error but in 

 19 This statement ignores any differences between the measured value of a quantity and the actual value of 

that quantity; the discussion on instrumentation uncertainties addresses these differences. 
58

1 the model application error. For this reason, the measured and the predicted values should be 

2 related to how a reactor safety analysis applies the model.

3 For example, the focus of PWR safety analysis is to determine which of the subchannels have the 

4 MDNBR value because this subchannel would be the “closest” one to experiencing a CBT. Thus, 

5 when a transient is simulated, the MDNBR is obtained, and if that value is greater than some 

6 safety limit, CBT is precluded. This method of analysis differs from the experiment in two main 

7 ways. First, the experiment determines which rod experienced a CBT, but the simulation 

8 determines which subchannel has the MDNBR. Second, because of how a CBT is defined in the 

9 experiment, it is common for more than one rod and even more than one location on the same rod 

10 to register as having experienced a CBT, but the simulation produces only one MDNBR value. 

11 Because the model is applied using the MDNBR, the measured and predicted values should be 

12 related to the DNBR value. 

13 The term “validation error” was chosen to represent the error of interest for two main reasons. The 

14 first reason is to distinguish it from the model error, which is commonly thought of as a difference 

15 between the model’s prediction and a measurement. Determining the measured and predicted

16 values is not as straightforward as many may consider. The second reason is that this error could 

17 have been called a “model application error,” but that term was not chosen for a different reason. 

18 The model application error is defined as the total population of error of the possible uses of the 

19 model inside the expected domain. If an experimental measurement of CBT could be obtained at 

20 every point in the expected domain (i.e., an infinite number of points), than that infinite set would 

21 be the actual model application error. The validation error is a sample from the model application 

22 error population. The validation error is based on the validation data, which only exist at a finite 

23 number of points in the expected domain. This distinction is important because one of the key 

24 assumptions is that the validation error is a representative sample of the model application error. 

25 Generally, the validation error for a model is either represented as an absolute error 

26 (i.e., measured – predicted), or as a relative error (e.g., (measured – predicted)/measured). CBT

27 models in particular use a form of the relative error—measured/predicted is commonly used for 

28 PWR validation and predicted/measured is commonly used for BWR validation. Thus, for PWRs, 

29 values that are below 1 are non-conservative (i.e., a CBT occurred at heat fluxes below the 

30 model’s prediction), and values that are above 1 are conservative (i.e., a CBT occurred at heat 

31 fluxes above the model’s prediction). Conversely, for BWRs, values that are below 1 are 

32 conservative (i.e., a CBT occurred at powers above the model’s prediction), and values that are 

33 above 1 are non-conservative (i.e., a CBT occurred at powers below the model’s prediction). 

34 Table 26 gives the evidence commonly provided to demonstrate that this goal has been satisfied.
59

1 Table 26 Evidence for G3.1—Calculating Validation Error

G3.1 The correct validation error has been calculated. 

Level Evidence

1 The validation error is a sample for the population of the model error. 

2 The validation error is a sample for the population of the model 

application error. 

3

The model is applied such that the populations of the model error and 

model application error are identical. The validation error is a sample 

from this population. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. While CBT models are often 

4 considered as stand-alone models, they are used as part of larger thermal-hydraulic

5 methodologies. Thus, the error in a CBT model is typically quantified as if it is being used inside 

6 the larger methodology (level 2) rather than used as a standalone model (level 1). Level 3 would 

7 be ideal as it would mean that model can be treated as a standalone equation. 

8 3.3.2 G3.2—Data Distribution in the Application Domain

9 The validation error data points should be appropriately distributed throughout the application 

10 domain. Consider each of the N input variables used by the model as a dimension (e.g., pressure, 

11 mass flux, inlet subcooling). The set of all inputs could be used to generate an N-dimensional 

12 “application space,” and the “application domain” is the domain in this space over which the model 

13 could be applied to predict CHF or CP. Typically, the application domain is defined as an 

14 n-orthotope which is a two-dimensional (2-D) rectangle, a three-dimensional (3-D) box, or a 

15 hyper-rectangle in dimensions greater than 3-D. This shape, the generalization of a rectangle to 

16 higher dimensions, is a simplification of the true shape of the application domain and is used 

17 because it can be easily defined by N inequalities (corresponding to the number of dimensions in 

18 the application space). Using this shape allows a computer program to easily determine whether 

19 the current location in the application space is inside or outside of the application domain. For 

20 example, the boundaries on the pressure are typically given as follows:

21 PPMMMMMM ≤ PP ≤ PPMM MM (2)

22 To ensure the model should be used to make a prediction, the computer code will check to ensure 

23 that the current pressure is between the minimum and maximum pressure of the application 

24 domain. 

25 Defining the application domain as a set of independent inequalities is computationally

26 convenient, but the model may not be valid over that entire domain. Consider the following 

27 simplified 2-D domain. Six types of subregions can be defined within the 2-D application space, 

28 depending on their proximity to validation error data points and their position relative to the 

29 application domain. These six types of subregions, shown in Figure 12, would also exist in 

30 application spaces of higher dimensions.
60

1

2 Figure 12 Regions in the Application Domain

3 Region 1—Well Covered

4 The first type of region is any region in the application domain that both contains data and is 

5 surrounded by data. In this region, the data are not sparse, and the region would be considered 

6 “well covered.” Although it is tempting to believe that the entire application domain is “well

7 covered,” this is only the ideal and is generally not true in practice. 

8 Region 2—Localized Hole

9 The second type of region is any region in the application domain that contains little to no data but 

10 is surrounded by data and thus forms a hole. As the number of dimensions of the application 

11 domain increases (i.e., Figure 1 shows a 2-D application domain, but it is common to have 

12 domains of six or more dimensions), it is not always clear whether the use of the model in such a 

13 region should be considered interpolation or extrapolation. In either circumstance, as long as the 

14 region itself is not too big, the use of the model in such regions is generally accepted as justified. 

15 Note that there will always be a “hole” between data points because the space is continuous, and 

16 the data exist only at discrete points. However, the assessor must exercise judgment about how 

17 far apart data are to constitute a localized hole.

18 Region 3—Edge

19 The third type of region is any region in the application domain that contains little to no data and is 

20 only partially surrounded by data and, therefore, is at an edge. Although uses of the model “near” 

21 the bulk of the data would seem reasonable, at some point the region of interest becomes 

22 sufficiently distant from the validation error data that the model cannot be considered validated 

23 and should not be used in the absence of other justification.
61

1 Region 4—Isolated Known Unknown

2 The fourth type of region is any region in the application domain that contains no data and is 

3 somewhat far from any region that does contain data; however, it is a region over which the model 

4 can be justified. For example, one common, conservative modeling assumption is to construct 

5 CBT models such that the predicted CHF or CP will be 0 at a mass flux of 0. In reality, as the 

6 mass flux goes to 0, the predicted CHF will go to a pool-boiling CHF value, which is much higher 

7 than 0. Thus, while the region may not have data, the use of the model in the region would be 

8 known to be very conservative. 

9 Region 5—Isolated Region

10 The fifth type of region is any region in the application domain that contains no data and is far from 

11 any region that does contain data. In other words, it is an isolated region. Moreover, it is an 

12 isolated region in which the model’s behavior is unknown. The application domain likely only 

13 includes such a region because the choice to represent the domain was a rectangle. The use of 

14 the model in such regions of the application domain should be precluded, but that would only be 

15 accomplished by defining a more complicated shape for the application domain. In 2-D, this could 

16 be easily done. However, many real models are in six or even more dimensions, and the 

17 representation of complex shapes in multiple dimensions is very difficult. Although the application 

domain will likely always be defined as a hyper-rectangle, the domain where the analyst expects20 18

19 to use the model is actually closer to a “hyper-jelly bean” (as described by one engineer). 

20 This is a concern with any higher dimensional model and is one reason why the application 

21 domain needs to contain validation error data that span the expected range of use. Although 

22 isolated unknown regions are always possible, the best way to ensure that the model will never be 

23 used in such a region is to ensure that all conceivable regions where the model will actually be 

24 used have data. 

25 It is important to realize that the model’s prediction in an isolated unknown region is suspect. On 

26 the one hand, the model may have been developed such that it happens to provide reasonable 

27 estimates of the CBT in that region. On the other hand, it may predict something completely 

28 unphysical in that region such as a negative heat flux or negative power. Model developers tend 

29 to understand where these regions exist, and apply the models only in the regions that contain 

30 data. However, a new user who is unfamiliar with the model’s development process could easily 

31 pick up the model, start using it, and wonder why it is making very strange predictions in certain 

32 regions.

33 Region 6—Outside Region 

34 The sixth type of region is any region that is outside the application domain. The computer code 

35 using the model will flag the use of the model as improper only in these outside regions. 

36 Considering these regions, the six subgoals in Figure 13 are used to demonstrate that the 

37 validation error data were appropriately distributed throughout the application domain. 

38

 20 Hence, this document separates the two domains. The Application Domain is the domain over which the model 

is applied, and is an n-dimensional rectangle. However, the Expected Domain is where the analysts would expect 

the model to be used, and is subset of the application domain, but generally a much more complex shape that 

cannot easily be well defined.
62

1

2

3 Figure 13 Decomposition of G3.2—Data Distribution in the Application Domain

4 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

5 discuss the evidence that could be used to demonstrate that these six base goals have been 

6 satisfied. Additionally, a discussion is provided on the evidence that has been historically used for 

7 CBT models applied in reactor safety analysis.

8 G3.2.1—Identification of Validation Data

9 The validation data are the experimentally measured values that are used to quantify the model’s 

10 error. Ideally, these data should be independent from the training data. The model will be used to 

11 make predictions about the CBT throughout the application domain. The focus of validation is to 

12 quantify the error of those predictions. Although it may seem that use of the training data would be 

13 appropriate, the model has already been “tuned” to that data. Thus, quantifying the error of the 

14 training data would provide an estimate of “how well the model can predict data that were used to 

15 generate the model.” This is different from “how well the model can predict data that were not 

16 used to generate the model.” Because substantially more data points appear in the application 

17 domain (an infinite number) than were used to generate the model and because these points are

18 the ones of most interest in future uses of the model, the focus should be on generating an 

19 estimate of the error over those points which were not used to generate the model. Thus, 

20 experimental data that have not been used to train the model should be held in reserve and used 

21 only to validate the model because the model’s behavior using these data are indicative of the 

22 type of predictions that will be made in its future uses.

23 However, in many instances, the validation data and the training data are one and the same. 

24 There are methods in machine learning that can be applied to determine whether the selection of 

25 the training data affects the resulting uncertainty, such as random subsamples and k-folds. In 

26 each of these methods, the data are randomly separated into subsets of training and validation 

27 data. The training data are used to develop the coefficients of the model, and the validation data 

28 are used to determine the overall uncertainty of the model. Then, the process is repeated with a 

29 different randomly-selected data set assigned to training and the remaining data assigned to 

30 validation. Processes like these can provide reasonable estimates of the impact of using the same 
63

1 training data as validation data. Even for well-formed models, using the same dataset for training

2 and validation can increase uncertainty by 2 to 3 percent. This increase is small but far from 

3 negligible, and it may be higher or lower depending on the circumstances. Table 27 gives the 

4 evidence commonly provided to demonstrate that this goal has been satisfied.

5 Table 27 Evidence for G3.2.1—Identification of Validation Data

G3.2.1 The validation data (i.e., the data used to quantify the model’s error) 

should be identified.

Level Evidence

1 Validation data have been identified, and they are the same as the 

training data. 

2

Validation data have been identified, and they are the same as the 

training data. To quantify this impact, a method such as k-folds or 

random subsamples has been used.

3 The validation data are independent from the training data. 

6 Historical Evidence Levels for Reactor Safety Analysis

7 Level 3 (specifically, a 70/30 or 80/20 split between the training and validation data) has been

8 most commonly accepted by the NRC staff. In a sense, this is similar to performing a single 

9 k-folds calculation or a single random calculation of subsamples. Level 2 has been used in the 

10 past, but the model’s error in predicting data that were not used to generate the model will almost 

11 always be greater than its error in predicting data that were used to generate the model. 

12 Therefore, using the same data for training data as validation data often involves additional work, 

13 such as k-folds or random subsamples. If only Level 1 is achieved, a bias may need to be added 

14 to account for the fact that the resulting error is likely lower than actually expected.

15 G3.2.2—Defining the Application Domain

16 The application domain should be defined such that the computer code applying the model is able 

17 to determine whether the model should be used for a given set of input parameters. Generally, 

18 this is done using inequalities such as those given in the following expressions: 

PPMMMMMM ≤ PP ≤ PPMM MM (3)

GGMMMMMM ≤ GG ≤ GGMM MM (4)

19 Defining the application domain in such a manner results in a hyper-rectangle, which contains 

20 many regions in which no data exist. Although a more accurate method of defining the application 

21 domain could be used to only specify the region which contains data, such alternative methods do 

22 not currently exist. Table 28 gives the evidence commonly provided to demonstrate that this goal 

23 has been satisfied.
64

1 Table 28 Evidence for G3.2.2—Defining the Application Domain

G3.2.2 The application domain of the model should be mathematically defined.

Level Evidence

1 The application domain has been mathematically defined as a 

hyper-rectangle. 

2 The application domain has been mathematically defined as a shape 

other than a hyper-rectangle to better capture its true shape. 

3 The application domain has been mathematically defined in terms of a 

maximum allowable distance from validation error data. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 1 has been most commonly accepted by the NRC staff. Because application domains 

4 defined as hyper-rectangles often contain many regions which are technically part of the 

5 application domain, but contain no data and are far from where the plant operates, there is 

6 generally a desire for the analyst to not only specify the application domain, but also to 

7 understand the expected domain. 

8 G3.2.3—Understanding the Expected Domain

9 The application domain is defined as the domain in the N-dimensional input space over which the 

10 model could be applied. However, that domain is different from the domain over which the model 

11 is expected to be applied. The “expected domain” is defined as the domain in the N-dimensional 

12 input space over which the model will likely be applied because it corresponds to state points that 

13 occur during normal operation or AOOs. Unlike the application domain, which is mathematically 

14 defined so that a computer can determine whether the model is being used outside of that 

15 domain, the expected domain is generally not formally defined due to its complex shape. For 

16 example, if the application domain is represented as a box in a series of 2-D plots of one input 

17 parameter versus another input parameter, the expected domain would be represented by some 

18 region in each box.

19 Ideally, as knowledge progresses, the application domain would become closer to the expected 

20 domain, and both domains would contain only regions with data. Table 29 gives the evidence 

21 commonly provided to demonstrate that this goal has been satisfied.
65

1 Table 29 Evidence for G3.2.3—Understanding the Expected Domain

G3.2.3 The expected domain of the model should be understood.

Level Evidence

1 Each parameter in the model is considered separately. 

2

2-D plots (parameter versus parameter) that contain the locations of 

the validation error data and the expected range of those parameters 

during normal operation and AOOs are provided. The expected ranges 

are well covered by validation error data (N parameters = N∙(N−1)

2 plots).

3 Another method that considers more than two parameters at a time 

(e.g., 3-D plots) is used. 

4 A method that considers all N parameters at the same time is used. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. Although used in the past, Level 1 

4 completely ignores any correlations between the input parameters themselves. For example, if 

5 there is only low-pressure data at low mass flows and high-pressure data at high mass flows, the 

6 model’s prediction in the region that has both low-pressure and high mass flow would not have 

7 any associated validation error data. However, to determine whether this situation exists, the data 

8 should be plotted with more than one input parameter at once (i.e., at least a 2-D plot). 

9 Just as Level 1 reduces the problem to a single dimension in the N-dimensional input space, 

10 Level 2 reduces the problem to two input dimensions. Both dimensional reductions cause the loss 

11 of information, but the information loss caused by reductions from N dimensions to 2-D is believed 

12 to be less significant. Ideally, all N dimensions could be considered at the same time, but the 

13 authors are not currently aware of a method for doing so.

14 G3.2.4—Validation Error Data Density in the Expected Domain

15 The expected domain should have adequate data density to ensure adequate coverage for future 

16 uses of the model. Typically, the regions with the most data will be those regions in which the 

17 plant will be close to normal operating conditions. However, these regions are not necessarily the 

18 same as the regions in which the plant would be closest to experiencing a CBT. Thus, although 

19 certain regions are expected to have a very high density of validation error data, the entire 

20 expected domain should be well covered. Note that the entire application domain will likely not be 

21 covered, due to the practice of representing the application domain as a hyper-rectangle. While it 

22 is not necessary for the entire application domain to be well covered with validation data, it is 

23 necessary for the expected domain to be well covered. Table 30 gives the evidence commonly 

24 provided to demonstrate that this goal has been satisfied. 
66

1 Table 30 Evidence for G3.2.4—Validation Error Data Density in the Expected Domain

G3.2.4 There should be adequate validation error data density throughout the 

expected and application domains.

Level Evidence

1

Each input parameter is considered independently from all others. 

Few regions have sparse data, and the model’s use in those regions 

can be justified.

Thus, the problem is treated as N number of 1-D spaces.

2

A set of two input parameters are considered in combination. The data 

density is sufficient, only a few regions of sparse data exist, and the 

model’s use in those regions can be justified. All possible combinations 

of sets of two input parameters are considered. 

Thus, the problem is treated as �

NN

2� number of 2-D spaces.

3

A set of three input parameters are considered in combination. The 

data density is sufficient, only a few regions of sparse data exist, and 

the model’s use in those regions can be justified. All possible 

combinations of sets of input parameters are considered. 

Thus, the problem is treated as �

NN

3� number of 3-D spaces.

4

All input parameters are considered in combination. The data density is 

sufficient, only a few regions of sparse data exist, and the model’s use 

in those regions can be justified.

Thus, the problem is treated as a single N-D space.

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. Again, Level 1 is considered

4 insufficient because it ignores any correlations between the input parameters themselves. Level 3 

5 would require the use of 3D plots, and such plots are difficult to represent in a 2D document (i.e., 

6 on a sheet of paper). For this reason, assessors have previously found it important to obtain the 

7 data used to correlate and validate the model—this data can be used to generate 3-D plots, which 

8 can be examined in detail on a computer. In addition, the number of dimensions observed at the 

9 same time can be increased to four dimensions by using a color gradient on the points. Higher 

10 dimensional plots are possible, but understanding such plots as the number of dimensions grows

11 becomes difficult. 

12 As of yet, there are no precise limits on data density. Even the concept of data density is difficult

13 to define precisely, as the volume over which the data density would be determined contains 

14 dimensions that cannot be easily combined in a meaningful way. Therefore, the density in each 

15 region is generally judged to be sufficient if it is similar to previous densities from past approved 

16 models. It is expected that there will be a large cluster of points around the normal operating 

17 conditions and fewer points at the extremes of the expected domain. 
67

1 Finally, levels 1-3 are graphical methods that rely on qualitative judgment. Level 4 considers some 

2 method that is quantitative, but the authors are not aware of any such method that currently 

3 exists. 

4 G3.2.5—Sparse Regions

5 As discussed above, there may be sparse regions in the application domain for a variety of 

6 reasons. Usually, sparse regions appear in the application domain because of the method chosen 

7 to describe the domain (e.g., as a hyper-rectangle). However, these regions in the application 

8 domain may not be a part of the expected domain. Such regions in the application domain but not 

9 in the expected domain should be identified, but further justification is not necessary, as the model 

10 is not expected to be used in those regions. However, any sparse region that lies within the 

11 expected domain would need further justification as the model would be expected to be used in 

12 that region. Table 31 gives the evidence commonly provided to demonstrate that this goal has 

13 been satisfied. 

14 Table 31 Evidence for G3.2.5—Sparse Regions

G3.2.5 Sparse regions (i.e., regions of low data density) in the expected and 

application domains should be identified and justified.

Level Evidence

1 There are many sparse regions in the expected domain. 

2 There may be some sparse regions in the application domain. There 

may be some sparse regions in the expected domain. 

3

There may be some sparse regions in the application domain. There 

may be some sparse regions in the expected domain, but the use of 

the CBT model in these regions is justified. 

4 There may be some sparse regions in the application domain. There 

are no sparse regions in the expected domain. 

5 There are no sparse regions in either the application or the expected 

domain.

15 Historical Evidence Levels for Reactor Safety Analysis

16 Level 3 has been most commonly accepted by the NRC staff. There may be sparse regions that 

17 are at the edges of the model’s intended use (e.g., low mass flux or high mass flux), though

18 additional justification is usually provided for these regions. As discussed above, there are 

19 numerous ways to justify the use of a model in a sparse region. The most common are: (1) 

20 demonstrating that the model is conservative in the region, (2) demonstrating that it is not possible 

21 for the fuel assembly to operate in the region, and (3) demonstrating that the region is not, in fact, 

22 a sparse region. However, there are often instances in which the model does need to be used in a 

23 region that is sparse (or at least has a very low data density). In these instances, a bias applied to

24 the model in the region in question may address the sparseness of the data without unnecessarily 

25 negatively impacting the model’s predictions in other parts of the application domain. In the higher 

26 dimensional spaces that are typical of most real application domains, the issue of sparse regions 

27 becomes more difficult to understand and define.
68

1 G3.2.6—Restricted to the Application Domain

2 Restricting the CBT model to its application domain is important. There are a variety of ways in 

3 which this restriction can be placed and upheld on the computer code using the CBT model. Table 

4 32 gives the evidence commonly provided to demonstrate that this goal has been satisfied. 

5 Table 32 Evidence for G3.2.6—Restricted to the Application Domain

G3.2.6 The model should be restricted to its application domain.

Level Evidence

1

The computer code does not check whether the model is being used 

outside of its application domain. Instead, the code analyst ensures

that the model was used only inside of its application domain when 

reviewing the code output. 

2

If the computer code attempts to use the model outside of its 

application domain, the code’s output marks it as a warning; however,

the simulation continues to run. 

3

If the computer code attempts to use the model outside of its 

application domain, the code’s output marks it as an error; however,

the simulation continues to run. 

4

If the computer code attempts to use the model outside of its 

application domain, the code’s output marks it as an error, and the 

simulation immediately quits running. 

6 Historical Evidence Levels for Reactor Safety Analysis

7 Levels 3 and 4 have been most commonly accepted by the NRC staff. Level 1 would present

8 human-factors issues and should not be used if more than a few simulations are needed in the 

9 particular application. Level 2 could also be present human-factors issues because users may not 

10 recognize the severity of the application domain violation. In general, appropriate evidence for 

11 G3.2.6 depends on the QA program the simulation is performed under and whether the restriction 

12 to the application domain is the responsibility of that QA program or of the computer code itself. 

13 3.3.3 G3.3—Inconsistency in the Validation Error

14 Statistics from the validation error will be used as estimates of parameters from the population of 

15 the model application error in order to quantify the uncertainty of the CBT model. This assumes 

16 that the model application error can be described as a single population with the same distribution 

17 and parameters (e.g., mean, variance) over the entire application domain and that the validation 

18 error is a representative sample of this distribution. 

19 As discussed by Box, Hunter, and Hunter (1978) one of the key assumptions in the data are the 

20 assumption of independence. If the model application error is dependent on its location in the 

21 application domain, it would be a collection of many populations, not a single population. Piepel 

22 and Cuta (1993) argue that the validation error would not likely be from a single population; 

23 instead, it would contain subregions in the application domain where the validation error would be 

24 from different populations. Although the authors agree that this is likely the case, the assumption 

25 of a single underlying population and independence should be reasonable as long as the 
69

validation error is consistent and no obvious non-conservatisms exist.

21 1 The three subgoals in 

2 Figure 14 are used to demonstrate that any inconsistencies in the validation error have been 

3 appropriately addressed.

4

5 Figure 14 Decomposition of G3.3—Inconsistencies in the Validation Error

6 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

7 discuss the evidence which could be used to demonstrate that the three base goals have been 

8 satisfied. Additionally, a discussion is provided on the evidence which has been historically used 

9 for CBT models applied in reactor safety analysis.

10 G3.3.1—Identifying Non-poolable Data Sets

11 The validation error is typically made up of multiple sets of data. The validation error data may be 

12 taken at low pressures, high flows, different axial power shapes, slightly different geometries, and 

13 so on. Analysts generally assume that all of this data are poolable, i.e., that all of the data can be 

14 treated as if they came from a single underlying population. If this is true, then the validation error, 

15 which is based on the validation data, may be a representative sample from this larger population, 

16 and therefore a good estimate of the behavior of the total population of model application error.

17 However, there are a number of reasons why the validation error may not be a representative 

18 sample of the overall population of the model application error. First and foremost, the validation 

19 error itself may represent several different populations. That is, pooling all of the validation errors 

20 from each data set into a single validation error may be incorrect. For example, the CBT model 

21 may make much better predictions at low pressures than at high mass fluxes. Pooling the data 

22 would obscure this difference. The assumption of poolability should be tested by identifying key 

23 data subsets in the validation error data set and by determining whether those data sets are

24 indeed from the same population. 

 21 In this sense, “non-conservative” means that the prediction of the CBT model over predicts the CHF or CP 

value by an amount greater than that accounted for by any uncertainty applied to the model. For CHF 

correlations, this would typically be the 95/95 value. 
70

1 Although statistical tests can be performed to determine whether two subsets are from the same 

2 population (i.e., have the same distribution shape, the same mean, the same variance), caution 

3 should be used. Incorrectly determining that the data sets are from different populations when, in 

4 fact, they are from the same population is common. This is known as a Type 1 error or a “false 

5 positive.” The probability of Type 1 errors increases with each additional test performed. For 

6 example, using the common significance value of 5 percent, the probability of obtaining a false 

7 positive after one test is only 5 percent. However, if 14 tests are performed with the same 

8 significance value, the probability of obtaining at least one false positive is over 50 percent. Thus, 

9 even if the data are from the same population, performing 14 tests will more than likely result in 

10 the conclusion that the data are from different populations. Therefore, these tests should be 

11 applied only when necessary. 

12 For PWRs, data should be separated (at a minimum) by axial power shape and by subchannel

13 type (rod and guide tube) as these are the main data sets that have been shown to be non14 poolable. If any of the sets are not poolable, the model’s uncertainty should be derived from the 

15 limiting data set. For BWRs, data should be separated (at a minimum) by axial power shape. 

16 Thus, it should be determined whether all power shapes are poolable data sets. If they are not 

17 poolable, the model’s uncertainty should be derived from the limiting data set. 

18 The following statistical tests are commonly used during this process:

19 • Analysis of variance, commonly known as ANOVA, for equality of means

20 • T-test, for equality of means

21 • F-test, for equality of variances

22 • Chi-square test, for equality of variances

23 • D’Agostino’s test, for normality

24 • Wilks-Shapiro test, for normality

25 • Anderson-Darling test, for normality

26

27 If the validation error is made from (i.e., contains) data sets that are not poolable, the most limiting 

28 or most conservative data set should be chosen if using a single value to quantify the model’s 

29 uncertainty. Table 33 gives the evidence commonly provided to demonstrate that this goal has 

30 been satisfied.
71

1 Table 33 Evidence for G3.3.1—Identifying Non-poolable Data Sets

G3.3.1

The validation error should be investigated to ensure that it does not 

contain any subgroups that are obviously not from the same population 

(i.e., non-poolable).

Level Evidence

1 No subgroups were analyzed for poolability. 

2

All relevant subgroups were investigated, and there was statistical 

evidence that the groups were from different populations. Therefore, 

the statistics from the limiting subgroup data set were used to 

determine the model’s uncertainty. 

3

All relevant subgroups were investigated, and there was no statistical 

evidence that the groups were from different populations. The statistics 

from the combined data sets were used to determine the model’s 

uncertainty.

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 2 has been most commonly accepted by the NRC staff. If Level 1 were presented, it would 

4 generally call for additional work and justification to be acceptable in reactor safety analysis, as it 

5 is very common for models to have different predictive behavior over their application domain. 

6 Level 3 is also common, but not as often achieved, as there is usually a subgroup which is slightly 

7 more limiting than the others. 

8 G3.3.2—Identifying Non-conservative Subregions

9 Another key assumption is the assumption of statistical independence of the data (i.e., 

10 independent and identically distributed or iid) in the expected domain. As Piepel and Cuta (1993) 

11 point out, the model’s uncertainty will likely vary over the expected domain. Therefore, an effort is 

12 made to determine whether any obvious non-conservative subregions can be identified in the 

13 validation error. The absence of such a subregion does not prove that statistical independence

14 exists; however, the authors are not aware of any other means to make such a determination. 

15 Historically, non-conservative subregions have been identified by reviewing plots of the validation 

16 error versus the various input parameters (e.g., pressure, mass flux, quality). The lack of a visual 

17 trend in these plots was the justification that the model’s uncertainty did not vary over the 

18 application domain. However, Kaizer (2015) points out that this visual one-dimensional (1-D)

19 plotting method ignores dependences among the various parameters and that non-conservative

20 subregions in the expected domain can be missed. Therefore, he proposed another method that 

21 can be used to analyze data in up to three dimensions at a time. Although this proposed method 

22 has a visual component to identify suspected non-conservative regions, it uses a statistical test to 

23 determine whether the subregion is, in fact, non-conservative. 

24 Because this method is limited to three dimensions, only the most important input parameters are 

25 typically investigated together. For PWRs, those three parameters are typically the mass flux, 

26 pressure, and local quality. For BWRs, those three parameters are typically the mass flux, 

27 pressure, and inlet temperature (or subcooling). Other combinations should also be investigated 

28 as necessary. 
72

1 Proving that non-conservative subregions do not exist is not the objective. Such proof would call 

2 for taking a very large number of data points. Given the limited data available in the validation 

3 error data set, the only statement that can be confirmed is that no obvious non-conservative

4 subregion has been identified. However, if a non-conservative subregion is found, the model 

5 uncertainty in that region would need to be increased to reflect the model’s predictive capability in 

6 that region.

7 Table 34 gives the evidence commonly provided to demonstrate that this goal has been satisfied.

8 Table 34 Evidence for G3.3.2—Identifying Non-conservative Subregions

G3.3.2

The expected domain should be investigated to determine if it contains 

any non-conservative subregions that would impact the predictive 

capability of the model.

Level Evidence

1

Plots of each model input parameter versus the validation error 

(i.e., predicted versus measured or measured versus predicted) are 

provided. This visual method (e.g., the 1-D method) demonstrates that 

there are no trends in the validation error with any input parameter. 

2

Plots of each model input parameter versus the validation error 

(i.e., predicted over measured or measured over predicted) are 

provided. This visual method (e.g., the 1-D method) demonstrates that 

there are no trends in the validation error with any input parameter. 

Additionally, a method similar to the one proposed by Kaizer (2015) is 

used to demonstrate that there are no obvious non-conservative

subregions in the application domain. 

3

A method further refined from the one proposed by Kaizer (2015) is 

used. Such a method is able to consider all N-dimensions at the same 

time and does not call for the user to visually identify any suspected 

non-conservative subregions.

9 Historical Evidence Levels for Reactor Safety Analysis

10 Level 1 has historically been most commonly accepted by the NRC staff. However, recent reviews 

11 have used Level 2. The method discussed in Level 2 has revealed multiple non-conservative 

12 subregions that required additional analysis or testing. Level 3 is ideal as it would be a completely 

13 objective; however, the authors are not currently aware of any such method. 

14 G3.3.3—Appropriate Trends

15 Certain trends common in CHF and CP models could be expected in future models. Generally, 

16 these trends can be seen by analyzing the plots of CHF or CP versus each of the various model 

17 parameters. This includes both an examination of all the data at once and an examination of only 

18 selected portions of the data (e.g., CHF at nominal pressures with decreasing mass flux). 

19 Depending on the situation, the measured and predicted CHF and CP may need to be analyzed 

20 separately. Table 35 gives the evidence commonly provided to demonstrate that this goal has 

21 been satisfied.
73

1 Table 35 Evidence for G3.3.3—Appropriate Trends

G3.3.3 The model’s predictions trend as expected in each of the various 

model parameters.

Level Evidence

1 Plots of the validation error (i.e., predicted over measured or measured 

over predicted) versus each model input parameter are provided. 

2 Plots of the measured or predicted CBT values versus each model 

input parameter are provided. All trends are as expected. 

3 Plots of the measured and predicted CBT values versus each model 

input parameter are provided. All trends are as expected.

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 1 has been most commonly accepted by the NRC staff. The trends should not only be 

4 smooth and continuous, but also conform to known behavior of the associated phenomena. It is 

5 often helpful to compare the trends of the current model with trends from previously approved 

6 models. Generally, further details of this criterion are investigated only if inconsistent behavior in 

7 the expected domain is suspected.

8 3.3.4 G3.4—Calculating Model Uncertainty

9 In CHF models used in PWRs and CP models used in BWRs, the model uncertainty is obtained 

10 from the validation error. However, the means of calculating the model uncertainty and its 

11 application to reactor safety analysis vary greatly. 

12 Departure from Nucleate Boiling Ratio Limit Used in Pressurized-Water Reactors

13 For CHF models used in PWRs, the model’s uncertainty is applied in the DNBR limit. That limit is 

14 used to ensure that there will be at least a 95-percent probability at the 95-percent confidence 

15 level that the hot fuel rod in the core does not experience a DNB or CBT condition during normal 

16 operation or AOOs. This DNBR limit is solely dependent on the CHF model’s performance and is 

17 independent of any conditions at the plant (e.g., the loading pattern). 

18 The DNBR limit is a statistical limit derived from the validation error. The validation error (usually 

19 represented as a ratio of the measured CHF to the model predicted CHF) is assumed to be a 

20 representative sample from the population of the model application error. Therefore, the 

95th 21 percentile of the population of the model application error is estimated using the 95/95 value 

from the validation error. In other words, the 95th 22 percentile of the validation error is estimated 

23 using a process that will overestimate the percentile 95 percent of the time (e.g., Owen’s method 

24 (Owen, 1963) and Wilks’ method (Wilks 1941; Wilks 1943)). This 95/95 value is then used as the 

25 DNBR limit and bounds the uncertainty of the CHF model. 

26 For example, if the measured versus predicted values are normally distributed, the DNBR limit 

27 could be determined to be the 95/95 value calculated, as prescribed by Owen, with the k-value 

28 obtained from Owen’s tables. Equation 5 is used to calculate the 95/95 value:

ll 29 95/95 = μμ − kk ∙ σσ (5)
74

Where ll 1 95/95 is the 95/95 value, μ is the mean of the measured to predicted values, k is a factor 

2 from Owen’s tables, and σ is the standard deviation of the measured to predicted values. 

3 Generally, the DNBR limit is simply the reciprocal of the 95/95 value; however, a conservative 

4 bias is usually added. Generally, this bias (bb) may simply come from rounding up the DNBR limit 

5 to a number with 3 significant figures (i.e., 1.133 becomes 1.14), but additional biases may also 

6 be added to account for other non-conservatism in the model (for more details see Information 

7 Notice 2014-01 (U.S. Nuclear Regulatory Commission, 2014)). Equation 6 gives the resulting 

8 DNBR limit:

DDDDDDDD LLLLLLLLLL = 1

95/95

9 + bb (6)

10 Safety Limit Minimum Critical Power Ratio Used in Boiling-Water Reactors

11 For CP models used in BWRs, the safety limit minimum critical power ratio (SLMCPR) reflects the 

12 model’s uncertainty. That limit is used to ensure at least 99.9 percent of the fuel rods in the core 

13 will not experience a CBT during normal operation or AOOs. Unlike the DNBR limit, the SLMCPR 

14 does not depend solely on the CP model’s performance, but instead also depends on some 

15 conditions at the plant (e.g., the core design). 

16 A separate methodology is used to determine the SLMCPR, and the uncertainty in the CP model 

17 is an input to that methodology. Usually, this uncertainty is represented by the standard deviation 

of the model’s prediction of the experimental data22 18 (i.e., the standard deviation of the validation 

19 error). This standard deviation is used to capture the model’s uncertainty. If the mean of the 

20 validation error is greater than one or if the sample is not normal, than the model’s uncertainty is 

21 increased by artificially increasing the standard deviation before it is used in the SLMCPR

22 methodology. Mean values of less than one are generally not credited in determining the 

23 SLMCPR.

24 Conservative Calculation of the Model Statistics

25 The model’s uncertainty is quantified using statistics from the validation error. Those statistics are

26 estimates of the parameters from the population of the model application error. Thus, the statistics 

27 of the validation error should be calculated in such a manner that they bound the true model 

28 application error. The three subgoals in Figure 15 are used to demonstrate that the validation 

29 error has been appropriately quantified.

 22 It should be noted that in PWRs, the validation error is given as the ratio of the measured value to the predicted 

value, but in BWRs the validation error is usually given as the ratio of the predicted value to the measured value. 
75

1

2 Figure 15 Decomposition of G3.4—Quantification of the Model’s Error

3 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

4 discuss the evidence which could be used to demonstrate that the three base goals have been 

5 satisfied. Additionally, a discussion is provided on the evidence which has been historically used 

6 for CBT models applied in reactor safety analysis.

7 G3.4.1—Error Database

8 It may not be appropriate to use the entire validation error database to calculate the model’s 

9 statistics, especially if the expected domain has non-poolable data sets or non-conservative

10 subregions. Therefore, the assessor should confirm that the statistics used to generate the 

11 validation error are from an appropriate sample of data. Table 36 gives the evidence commonly 

12 provided to demonstrate that this goal has been satisfied.

13 Table 36 Evidence for G3.4.1—Error Database

G3.4.1 The validation error statistics should be calculated from an appropriate 

database.

Level Evidence

1 The model’s uncertainty was calculated using the entire database of 

validation error. 

2 The model’s uncertainty was calculated using a subset of the validation 

error, which resulted in a more conservative calculation. 

3 The model’s uncertainty was calculated from the limiting subset of the 

validation error, which resulted in a more conservative calculation.

14 Historical Evidence Levels for Reactor Safety Analysis

15 Level 1 has been most commonly accepted by the NRC staff, but it generally assumes the data 

16 are poolable and does not contain any non-conservative subregions. Level 2 is often provided if it 

17 appears that a subset may be more limiting, but there is no definitive proof. Generally, if definitive 

18 proof exists that a specific subset if most limiting, then the uncertainty is often calculated from only 

19 the data in that subset (Level 3). 
76

1 G3.4.2—Validation Error Statistics

2 The method used to calculate the validation error statistics should be appropriate. This generally 

3 means ensuring that the assumptions of any method used are fulfilled (e.g., if Owen’s method is 

4 used to calculate the 95/95 value, the distribution of the validation error should be normally 

5 distributed). Statistical methods may call for the data (i.e., the validation error) to (1) have the 

6 same mean and variance (i.e., homoscedasticity), (2) be from the same distribution, (3) be from a 

7 normal distribution, and (4) be independent and identically distributed (i.e., iid data). If populations 

8 within the data do not have the same mean or variance, a conservative mean or variance can be 

9 chosen to bound the model uncertainty. If the data are not normally distributed, a nonparametric 

10 method (such as the Wilks method) can be used to calculate the model uncertainty. However, if 

11 the data are not independent and identically distributed, the model’s predictive capability would 

12 vary depending on the location in the application domain, and the model’s uncertainty would have 

13 to account for this variability. Table 37 gives the evidence commonly provided to demonstrate that

14 this goal has been satisfied.

15 Table 37 Evidence for G3.4.2—Validation Error Statistics

G3.4.2 The validation error statistics should be calculated using an appropriate 

method.

Level Evidence

1

The data used to calculate the model’s uncertainty appear to be 

independent and identically distributed. The method used to calculate 

the statistics is a parametric method. Although the necessary 

preconditions of such a method were not satisfied, assumptions could 

be made to ensure that the resulting uncertainty was conservative. 

2

The data used to calculate the model’s uncertainty appear to be 

independent and identically distributed, and one of the following 

applies:

• The method used to calculate the statistics is a parametric 

method. The assumptions of such a method were demonstrated 

to be true (i.e., there is no reason to believe they are false) 

through statistical testing.

• The method used to calculate the statistics is a nonparametric 

method. 

16 Historical Evidence Levels for Reactor Safety Analysis

17 Level 2 has been most commonly accepted by the NRC staff. However, Level 1 has been used 

18 when the resulting statistics could be justified to be conservative. 

19 G3.4.3—Model Uncertainty Bias

20 After the model’s uncertainty is calculated, it is commonly biased in a conservative direction. For 

21 example, a vendor may want to use a three-digit number as the DNBR limit. Thus, if the DNBR 

22 limit were calculated as 1.2301, it would be “rounded” up to 1.24 (which is equivalent to the 

23 addition of a bias of 0.0099). However, sometimes a bias is added to account for an uncertainty 

24 that the model does not address. Table 38 gives the evidence commonly provided to demonstrate

25 that this goal has been satisfied.
77

1 Table 38 Evidence for G3.4.3—Model Uncertainty Bias

G3.4.3 The model’s uncertainty should be appropriately biased.

Level Evidence

1 The model needed a large bias (> 1%). 

2 The model needed a small bias (< 1%). 

3 The model needed no bias, or the only biasing was due to rounding. 

2 Historical Evidence Levels for Reactor Safety Analysis

3 Level 3 has most commonly been accepted by the NRC staff, but Level 2 has also been 

4 commonly accepted. A larger bias (i.e., greater than 1 percent) indicates that some uncertainty in 

5 the model was not accounted for, which is generally avoided. In addition, because such biases 

6 are generally applied based on engineering judgment, not experimental data, the bias itself is 

7 subjective. Although situations arise that warrant the use of large biases, it is far from desirable 

8 because there is often little justification for choosing the specific bias instead of a larger or smaller 

9 value.

10 3.3.5 G3.5—Model Implementation

11 Once the model’s uncertainty has been quantified by experimental data, the model can be applied 

12 in a reactor safety analysis. However, the implementation of the model in the analysis should be 

13 consistent with its use during validation. 

14 For some CBT models, this may mean that the same computer code is used in both the validation 

15 and the application of the models. Although certain inputs to the CBT model (e.g., pressures, flow 

16 rates, power) would be expected to change depending on the situation in which the model was 

17 used, those inputs may depend less on the situation and more on which closure models were 

18 selected in the computer code that exercises the CBT model. In those situations, it may be 

19 possible to change the inputs to the CBT model without changing the inputs to the computer code 

20 itself (e.g., plant conditions) but merely by changing the closure models chosen. Therefore, it is 

21 important to ensure that if the inputs to the CBT model depend on closure models in the computer 

22 code that implements the CBT model, the same closure models are used in both the validation 

23 and the application of the CBT model. 

24 The reason for this is that the CBT model was validated with those closure models being applied, 

25 and the uncertainty was quantified using only that set of closure models. The CBT model could be 

26 used with another set of closure models, but the uncertainty would need to be quantified again 

27 (i.e., determine the new validation error with the new closure models). Re-validation of the model 

28 and re-quantification of the uncertainty is not necessarily a major exercise. The experimental data 

29 already exist, and a new data set of validation errors can be obtained using the changed model, 

30 code, or options. For the framework discussed here, only Criteria G3.2.1 and G3.2.2 would likely 

31 need to be confirmed because the evidence supplied to justify all other criteria would likely remain 

32 the same; however, this should be borne out by the analysis. 
78

1 If the model’s prediction in the changed code is similar to its prediction in the previous code, the 

2 evidence used to justify these two criteria may even remain the same. The three subgoals in 

3 Figure 16 are used to demonstrate that the model has been correctly implemented. 

4

5 Figure 16 Decomposition of G3.5—Model Implementation

6 No further decompositions of the subgoals were deemed useful. Therefore, the sections below 

7 discuss the evidence which could be used to demonstrate that these three base goals have been 

8 satisfied. Additionally, a discussion is provided on the evidence which has been historically used 

9 for CBT models applied in reactor safety analysis.

10 G3.5.1—Same Computer Code

11 The computer code and the options used to specify the closure models and any other functionality 

12 of that computer code should be the same. This is a much larger concern in PWRs because a 

13 subchannel simulation contains many more uses of the field equations and closure models. The 

14 direct modeling of the thermal-hydraulic response of the assembly should be consistent from the 

15 validation to the application of a CBT model. Table 39 gives the evidence commonly provided to 

16 demonstrate that this goal has been satisfied.

17 Table 39 Evidence for G3.5.1—Same Computer Code

G3.5.1 The model has been implemented in the same computer code that was 

used to generate the validation error.

Level Evidence

1 The model has been implemented in a computer code very similar to 

the one that was used to generate the validation error. 

2

The same computer code with the same closure models and code 

options that was used to generate the validation error will be used to 

perform any reactor safety analysis. 
79

1 Historical Evidence Levels for Reactor Safety Analysis

2 Level 1 has been most commonly accepted by the NRC staff for BWRs and Level 2 for PWRs. 

3 Many CBT models used for BWRs calculate the critical power of an assembly, and therefore the 

4 computer code used generally does not calculate complex local thermal-hydraulic phenomena, 

5 but rather more general parameters like assembly quality. As these formulations rely on few 

6 closure models, it is possible to use the same CBT model in multiple BWR analysis codes. 

7 However, because PWR analysis is performed at the sub-channel level, there are multiple closure 

8 models used. Those closure models calculate the local parameters that are used by the CBT 

9 model. Because the CBT model predictions could be changed by changing the closure models, 

10 changing a computer code generally involves re-analyzing the validation data with the new code 

11 for the PWRs. 

12 G3.5.2—Same Evaluation Methodology

13 It is important not only to use the same computer code to implement the CBT model but also to 

14 implement the model in the same manner. Although the comparison of the “measured” values to 

15 the “predicted” values is the basis for the validation, as discussed above, this comparison is 

16 generally not as simple as comparing the CHF or CP at the location in the test assembly that 

17 experienced a CBT to the predicated value at that location; therefore, a distinction was drawn 

18 between model error and model application error. Another way of referring to the same evaluation 

19 methodology is to ensure that the manner in which the model will be used (i.e., model application 

20 error), is consistent with how the validation error was determined.

21 Section 3.3, which defines validation error, discusses the reasoning for this. Here, the authors will 

22 only reiterate that the goal of using the CBT model is to ensure that a CBT does not occur, not to 

23 ensure that, if a CBT does occur, the model predicts the exact location where it occurs. If the 

24 model is able to identify the location, that can be evidence that the model is well correlated with 

25 the physics of the assembly, but it is not a requirement and, moreover, may not be useful when 

26 determining whether the model is appropriate. 

27 Table 40 gives the evidence commonly provided to demonstrate that this goal has been satisfied.

28 Table 40 Evidence for G3.5.2—Same Evaluation Methodology

G3.5.2

The model’s prediction of the CBT is being applied using the same 

evaluation methodology used to predict the validation data set for 

determining the validation error.

Level Evidence

1 The model is implemented using a very similar evaluation 

methodology. 

2 The model is implemented using the same evaluation methodology. 

29 Historical Evidence Levels for Reactor Safety Analysis

30 Level 2 has been most commonly accepted by the NRC staff. Level 1 could be used if an analysis 

31 demonstrated that the changes would not affect the model’s uncertainty. 
80

1 G3.5.3—Transient Prediction

2 Like many other thermal-hydraulic models, many CBT models are developed using data taken 

under steady-state conditions but applied in a transient23 3 simulation. Although this is a common 

4 practice, it should be justified, especially for models that contain integrals over space or time. This 

5 is generally more of a focus in BWRs than it is in PWRs and is ultimately demonstrated through 

6 transient tests. Those tests generally use time-varying inputs for power, flow, subcooling, or a 

7 combination of these parameters. The goal is to demonstrate that there are no transients at which 

8 a CBT occurred but was not predicted and, secondarily, to demonstrate that there were no tests in 

9 which CBT was predicted (i.e., should have occurred) but did not occur. 

10 Again, the goal of these tests is to demonstrate how well the model predicts whether CBT will 

11 occur; therefore, these transient tests should be conducted close to conditions that cause a CBT. 

12 Tests that are run too far from those conditions in either direction (i.e., either a test that was very 

13 far from a CBT actually occurring such as a very low-power test or a test in which a CBT must 

14 occur such as a very high-power test) would not be useful. 

15 Table 41 gives the evidence commonly provided to demonstrate that this goal has been satisfied.

16 Table 41 Evidence for G3.5.3—Transient Prediction

G3.5.3 The model results in an accurate or conservative prediction when it is 

used to predict transient behavior. 

Level Evidence

1 No experimental justification is provided.

2 Some experimental justification is provided.

3 Statistically significant experimental justification is provided. 

17 Historical Evidence Levels for Reactor Safety Analysis

18 Level 1 or Level 2 have been commonly accepted by the NRC staff for PWRs, whereas Level 2 or 

19 Level 3 have been commonly accepted for BWRs. The reason for the additional testing is likely 

20 due to the manner in which CBT is modeled differently in BWRs and PWRs. In PWRs, the CBT is 

21 based on local sub-channel parameters. Historically, it has been shown that CBT models which 

22 are generated with steady state data will accurately or conservatively predict CBT during a 

23 transient. This same assumption about CBT models being made with steady state data being 

24 conservative for transient data are also made for models used in BWRs. However, while it is 

25 possible to confirm this assumption through testing on a BWR test assembly, such testing would 

26 be very difficult on a PWR test assembly.

27 For Level 3, by “statistically significant,” the authors mean that there were enough conservative 

28 predictions from transients (i.e., those in which the CBT model was correct) to account for any 

29 situations in which the CBT model may have been non-conservative. For example, if the CBT

30 model was non-conservative in a single test but conservative in only eight tests, its predictive 

 23 In this context, transient means “time varying.”
81

1 capability would be in question, as eight tests is generally considered to be too small a number to 

2 determine any statistical significance. 

83

1 4 SUMMARY AND CONCLUSION

2 This work presents a generic safety case that can be used to determine the credibility of CBT 

3 models. This safety case was generated through the experience of many experts at the NRC, 

4 previously written safety evaluations, and documents in the open literature. This document 

5 captures the knowledge and experience of multiple NRC staff members over many years. The 

6 document presents a background on CBT including a literature survey, a description of the 

7 underlying phenomena, and how those phenomena are commonly modeled. The document also 

8 presents a credibility assessment framework, which combines the structure from GSN with the 

9 capability of maturity assessment. The elements of the framework provided in this document have

10 been applied in multiple reviews at the NRC and have decreased total review time, increased 

11 review consistency, and increased review efficiency. 

85

1 5 REFERENCES

2 [1] Hewitt, G.F., and N.S. Hall-Taylor, Annular Two-Phase Flow, Pergamon Press, Oxford, 

3 United Kingdom, 1970.

4 [2] Tong, L.S., and Y.S. Tang, Boiling Heat Transfer and Two-Phase Flow, Second Edition, 

5 Taylor & Francis Group, Washington, DC, 1997.

6 [3] ASME. Guide for verification and validation in computational solid mechanics. ASME V&V 

7 10-2006, New York, NY, American Society of Mechanical Engineers, 2006.

[4] NASA. “Towards a Credibility Assessment of Models and Simulations,” 49th 8

9 AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. 

10 AIAA-2008-2156, 2008.

[5] NASAB 11 . Standard for Models and Simulations. NASA-STD-7009, Washington, DC, 

12 National Aeronautics and Space Administration, 2008.

13 [6] Oberkampf, W.L., and C.J. Roy, Verification and Validation in Scientific Computing, 

14 Cambridge University Press, Cambridge, United Kingdom, 2010.

15 [7] Kaizer, J.S., A.K. Heller, and W.L. Oberkampf, “Scientific Computer Simulation Review,” 

16 Reliability Engineering and System Safety, 138:210–218, 2015.

17 [8] Ministry of Defense, Defense Standard 00-56, “Safety Management Requirements for 

18 Defense Systems, Part 1: Requirements,” Issue 4, 2007.

19 [9] Denney, E.W., and Pai, G.J., “Safety Case Patterns: Theory and Applications”, NASA/TM–

20 2015–218492, Feburary 2015.

21 [10] Food and Drug Administration, “Infusion Pumps Total Product Life Cycle - Guidance for 

22 Industry and FDA Staff”, December 2, 2014.

23 [11] Goal Structure Notation (GSN) Working Group, GSN Community Standard Version 1, 

24 Origin Consulting Limited, York, United Kingdom, 2011.

25 [12] Denney, E., G. Pai, and I. Habli, “Towards Measurement of Confidence in Safety 

Cases,” Proceedings of the 5th 26 International Symposium on Empirical Software

27 Engineering and Measurement, pp. 380–383, September 2011.

28 [13] Oberkampf, W.L., M. Pilch, and T.G. Trucano, Predictive Capability Maturity Model for 

29 Computational Modeling and Simulation. SAND2007-5948, Albuquerque, NM, Sandia 

30 National Laboratories, 2007.

31 [14] Tong, L.S., Boiling Crisis and Critical Heat Flux, U.S. Atomic Energy Commission, 1972.

32 [15] Todreas, N.E., and M.S. Kazimi, Nuclear Systems I: Thermal Hydraulic Fundamentals, 

33 Taylor & Francis Group, Washington, DC, 1990.

34 [16] Lahey, R.T., and F.J. Moody, The Thermal Hydraulics of a Boiling Water Nuclear 

35 Reactor—Second Edition, American Nuclear Society, La Grange Park, IL, 1993.

36 [17] Leidenfrost, J.G., “On the Fixation of Water in Diverse Fire,” A Tract about some qualtiies 

37 of common water, 1756, translated, Wares, C., International Journal of Heat Mass 

38 Transfer, 9:1153, 1966.

39 [18] Tong, L.S., H.B. Currin, P.S. Larsen, and O.G. Smith, “Influence of Axially Nonuniform 

40 Heat Flux on DNB,” AIChE Chemical Engineering Symposium Series, 62(64):35–40, 

41 1965.
86

1 [19] Macbeth, R.V., “An Appraisal of Forced Convection Burnout Data,” Proceedings of the

2 Institute of Mechanical Engineers, 1965–1966.

3 [20] Barnett, P.G., “A Correlation of Burnout Data for Uniformly Heated Annuli and Its Use for 

4 Predicting Burnout in Uniformly Heated Rod Bundles,” Atomic Energy Establishment 

5 Winfrith (AEEW-R) 463, 1966.

6 [21] Healzer, J.M., J.E. Hench, E. Janssen, and S. Levy, “Design Basis for Critical Heat Flux 

7 Condition in Boiling Water Reactors,” APED-5, September 1966.

8 [22] Tong, L.S., “Prediction of Departure from Nucleate Boiling for an Axially Non-Uniform Heat 

9 Flux Distribution,” Journal of Nuclear Energy, 21:241–248, 1967.

10 [23] Biasi, L., G.S. Clerici, S. Garribba, R. Sala, and A. Tozzi, “Studies on Burnout: Part 3—A 

11 New Correlation for Round Ducts and Uniform Heating and Its Comparison with World 

12 Data,” Energia Nucleare 14:530–536, 1967.

13 [24] Gellerstedt, J.S., R.A. Lee, W.J. Oberjohn, R.H. Wilson, and L.J. Stanek, “Correlation of 

14 Critical Heat Flux in a Bundle Cooled by Pressurized Water,” Two-Phase Flow and Heat

15 Transfer in Rod Bundles, American Society of Mechanical Engineers, New York, NY,

16 1969.

17 [25] Hughes, E.D., “A Correlation of Rod Bundle Critical Heat Flux for Water in the Pressure 

18 Range 150 to 725 psia,” IN-1412, Idaho Nuclear Corporation, Idaho Falls, ID, 1970.

19 [26] Piepel, G.F., and J. M. Cuta, “Statistical Concepts and Techniques for Developing, 

20 Evaluating, and Validating CHF Models and Corresponding Fuel Design Limits,” SKI 

21 Technical Report, 93:46, 1993.

22 [27] Groeneveld, D.C., J. Shan, A.Z. Vasic, L.K.H. Leung, A. Durmayaz, J. Yang, S.C. Cheng, 

23 and A. Tanase, “The 2006 CHF Look-Up Table,” Nuclear Engineering and Design, 

24 237(15):1909–1922, 2007.

25 [28] Yang, B., J. Shan, J. Gou, H. Zhang, A. Liu, and H. Mao, “Uniform versus Nonuniform 

26 Axial Power Distribution in Rod Bundle CHF Experiments,” Science and Technology of 

27 Nuclear Installations, Volume 2014, 2014.

28 [29] Kaizer, J.S., “Identification of Nonconservative Subregions in Empirical Models 

29 Demonstrated Using Critical Heat Flux Models,” Nuclear Technology, 190:65–71, 2015.

30 [30] Groeneveld, D.C., “CHF Data Used to Generate 2006 Groeneveld CHF Lookup Tables,”

31 NUREG/KM-0011, U.S. Nuclear Regulatory Commission, Washington, DC, 2017.

32 [31] Babcock & Wilcox, “Correlation of Critical Heat Flux in a Bundle Cooled by Pressurized 

33 Water,” BAW-10000, Babcock & Wilcox, Lynchburg, VA, March 1970, Agencywide 

34 Documents Access and Management System (ADAMS) Accession No. ML082490748 

35 (Proprietary Information, Nonpublicly Available).

36 [32] Combustion Engineering (C-E), “C-E Critical Heat Flux: Critical Heat Flux Correlation for 

37 C-E Fuel Assemblies with Standard Spacer Grids, Part 1—Uniform Axial Power 

38 Distribution,” CENPD-162-P-A, C-E, Stamford, CT, September 1976, ADAMS Accession 

39 No. ML083010357 (Proprietary Information, Nonpublicly Available).

40 [33] Exxon Nuclear Company, “Exxon Nuclear DNB Correlation for PWR fuel Designs,” 

41 XN-NF-621(P)(A), Revision 1, Richland, WA, September 1983, ADAMS Accession 

42 No. ML16265A315 (Proprietary Information, Nonpublicly Available).

43 [34] Motley, F.E., K.W. Hill, F.F. Cadek, and J. Shefcheck, “New Westinghouse Correlation 

44 WRB-1 for Predicting Critical Heat Flux in Rod Bundles with Mixing Vane Grids,” 
87

1 WCAP-8762-P-A, Westinghouse Electric Company, Pittsburgh, PA, July 1984, ADAMS 

2 Accession No. ML080630433 (Proprietary Information, Nonpublicly Available).

3 [35] Westinghouse Electric Company, “VANTAGE 5H Fuel Assembly,” WCAP-10444-P-A, 

4 Pittsburgh, PA, September 1985, ADAMS Accession No. ML080650257 (Proprietary 

5 Information, Nonpublicly Available).

6 [36] C-E, “C-E Critical Heat Flux: Critical Heat Flux Correlation for C-E Fuel Assemblies with 

7 Standard Spacer Grids, Part 2—Non-Uniform Axial Power Distribution,” CENPD-207-P-A, 

8 Stamford, CT, December 1984, ADAMS Accession No. ML16260A362 (Proprietary 

9 Information, Nonpublicly Available).

10 [37] AREVA, “Departure from Nucleate Boiling Correlation for High Thermal Performance 

11 Fuel,” ANF-1224(P)(A), and Supplement 1 to ANF-1224(P)(A), Lynchburg, VA, April 1990.

12 [38] Farnswoth, D.A., and G.A. Meyer, “The BWU Critical Heat Flux Correlations,” 

13 BAW-10199P-A, Revision 0, Framatome Technologies, Lynchburg, VA, February 1996.

14 [39] Smith III, L.D., M.W Lloyd, Y.X. Sung, and W.J. Leech, “Modified WRB-2 Correlation, 

15 WRB-2M, for Predicting Critical Heat Flux in 17x17 Rod Bundles with Modified LPD Mixing 

16 Vane Grids,” WCAP-15025-P-A, Westinghouse Electric Company, Pittsburgh, PA, 

17 April 1999, ADAMS Accession No. ML081610106 (Proprietary Information, Nonpublicly 

18 Available).

19 [40] Farnsworth, D.A., and G.A. Meyer, “The BWU Critical Heat FIux Correlations Applications 

20 to the Mark-B11 and Mark-BW17 MSM Designs,” BAW-10199P-A, Addendum 1,

21 Framatome Cogema Fuels, Lynchburg, VA, December 2000, ADAMS Accession 

22 No. ML003777245 (Proprietary Version, Nonpublicly Available).

23 [41] Farnsworth, D.A., and G.A. Meyer, “Application of BWU-Z CHF Correlation to the 

24 Mark-BW 17 Fuel Design with Mid-Span Mixing Grids,” BAW-10199P-A, Addendum 2, 

25 Framatome ANP, Lynchburg, VA, September 2002, ADAMS Accession 

26 Nos. ML022560552 (Proprietary Version, Nonpublicly Available) and ML022560550

27 (Nonproprietary Version, Publicly Available).

28 [42] Sung, Y.X., P.F. Joffre, and P.A. Hilton, “Addendum 1 to WCAP-1 4565-P-A Qualification 

29 of ABB Critical Heat Flux Correlations with VIPRE-01 Code,” WCAP-14565-P-A 

30 Addendum 1-A, Westinghouse Electric Company, Pittsburgh, PA, August 2004, ADAMS 

31 Accession Nos. ML042610371 (Proprietary Version, Nonpublicly Available) and 

32 ML042610368 (Nonproprietary Version, Publicly Available).

33 [43] AREVA, “Departure from Nucleate Boiling Correlation for High Thermal Performance 

34 Fuel,” EMF-92-153(P)(A), Revision 1, January 2005, ADAMS Accession 

35 Nos. ML051020019 (Proprietary Version, Nonpublicly Available) and ML051020017

36 (Nonproprietary Version, Publicly Available).

37 [44] Farnsworth, D., and K.R. Greene, “BHTP DNB Correlation Applied with LYNXT,” 

38 BAW-10241(P)(A), Revision 1, AREVA, Lynchburg, VA, July 2005, ADAMS Accession 

39 Nos. ML052500092 (Proprietary Version, Nonpublicly Available) and ML052500075

40 (Nonproprietary Version, Publicly Available).

41 [45] Farnsworth, D., “The BWU-B11R CHF Correlation for the Mark-B11 Spacer Grid,” 

42 BAW-10199P-A, Addendum 3, Framatome ANP, Lynchburg, VA, November 2005, 

43 ADAMS Accession No. ML070170690 (Proprietary Version, Nonpublicly Available)

44 ML042990354 (Nonproprietary Version, Publicly Available).
88

1 [46] Joffre, P.F., Y.R. Chang, R. Kapoor, Y.X. Sung, L.D. Smith III, and P.A. Hilton, 

2 “Westinghouse Correlations WSSV and WSSV-T for Predicting Critical Heat Flux in Rod 

3 Bundles with Side Supported Mixing Vanes,” WCAP-16523-P-A, Westinghouse Electric 

4 Company, Pittsburgh, PA, August 2007, ADAMS Accession Nos. ML072570633 

5 (Proprietary Version, Nonpublicly Available) and ML072570327 (Nonproprietary Version,

6 Publicly Available).

7 [47] Farnsworth, D.A., and R.L. Harne, “The ACH-2 CHF Correlation for the U.S. EPR,” 

8 ANP-10269P-A, AREVA, Lynchburg, VA, December 2007, ADAMS Accession 

9 Nos. ML080790191 (Proprietary Version, Nonpublicly Available) and ML080790193

10 (Nonproprietary Version, Publicly Available).

11 [48] Joffre, P.F., R. Kapoor, Y.X. Sung, and P.A. Hilton, “Addendum 2 to WCAP-14565-P-A 

12 Extended Application of ABB-NV Correlation and Modified ABB-NV Correlation WLOP for 

13 PWR Low Pressure Applications,” WCAP-14565-P-A, Addendum 2-P-A, Westinghouse 

14 Electric Company, Pittsburgh, PA, April 2008, ADAMS Accession Nos. ML081280713 

15 (Proprietary Version, Nonpublicly Available) and ML081280712 (Nonproprietary Version,

16 Publicly Available).

17 [49] Joffre, P.F., Y.X. Sung, R. Mathur, L.D. Smith III, and P.A. Hilton, “Westinghouse Next 

18 Generation Correlation (WNG-1) for Predicting Critical Heat Flux in Rod Bundles with Split 

19 Vane Mixing Grids,” WCAP-16766-P-A, Westinghouse Electric Company, Pittsburgh, PA, 

20 February 2010, ADAMS Accession Nos. ML100850532 (Proprietary Version, Nonpublicly 

21 Available) and ML100850528 (Nonproprietary Version, Publicly Available).

22 [50] Mitsubishi Heavy Industries, Ltd., “Thermal Design Methodology,” MUAP-07009-P-A, 

23 Tokyo, Japan, August 2013, ADAMS Accession Nos. ML13284A072 (Proprietary Version,

24 Nonpublicly Available) and ML13284A069 (Nonproprietary Version, Publicly Available).

25 [51] Korea Hydro & Nuclear Power Company, “KCE-1 Critical Heat Flux Correlation for PLUS7 

26 Thermal Design,” APR1400-F-C-TR-12002-P, Gyeongju, South Korea, November 2012, 

27 ADAMS Accession Nos. ML13018A158 (Proprietary Version, Nonpublicly Available) and

28 ML13018A147 (Nonproprietary Version, Publicly Available).

29 [52] AREVA, “The ORFEO-GAIA and ORFEO-NMGRID Critical Heat Flux Correlations,” 

30 ANP-10341(P), Lynchburg, VA, August 2016, ADAMS Accession Nos. ML16238A076 

31 (Proprietary Version, Nonpublicly Available) and ML16238A078 (Nonproprietary Version,

32 Publicly Available).

33 [53] Slifer, B.C., and J.E. Hench, “Loss-of-Coolant Accident and Emergency Core Cooling 

34 Models for General Electric Boiling Water Reactors,” NEDO-10329, Equation C-32, 

35 General Electric Company, San Jose, CA, April 1971.

36 [54] General Electric Company, “General Electric BWR Thermal Analysis Basis (GETAB): 

37 Data, Correlation and Design Application,” NEDO-10958-PA, San Jose, CA, 

38 January 1977, ADAMS Accession Nos. ML092820214 (Proprietary Version, Nonpublicly 

39 Available) and ML102290144 (Nonproprietary Version, Publicly Available).

40 [55] AREVA, “ANFB Critical Power Correlation,” ANF-1125(P)(A), and Supplements 1 and 2, 

41 Richalnd, WA, April 1990, ADAMS Accession No. ML081820434 (Proprietary Information,

42 Nonpublicly Available).

43 [56] General Electric Hitachi Nuclear Energy, “R-Factor Calculation Method for GE11, GE12, 

44 and GE13 Fuel,” NEDC-32505P-A, Revision 1, Wilmington, NC, July 1999, ADAMS 

45 Accession No. ML060520637 (Proprietary Information, Nonpublicly Available).
89

1 [57] Harris, W.R., and Y.Y. Yung, “10x10 SVEA Fuel Critical Power Experiments and CPR 

2 Correlations: SVEA-96+,” CENPD-389-P-A, ABB Combustion Engineering Nuclear Power, 

3 Inc., Windsor, CT, September 1999, ADAMS Accession Nos. ML993470286 (Proprietary 

4 Version, Nonpublicly Available) and ML993420024 (Nonproprietary Version, Publicly 

5 Available).

6 [58] Harris, W.R., and Y.Y. Yung, “10x10 SVEA Fuel Critical Power Experiments and CPR 

7 Correlations: SVEA-96,” CENPD-392-P-A, Revision 00, CE Nuclear Power, LLC, 

8 Monroeville, PA, September 2000, ADAMS Accession Nos. ML003767392 (Proprietary 

9 Version, Nonpublicly Available) and ML003767366 (Nonproprietary Version, Publicly 

10 Available).

11 [59] Harrington, R., and J.G.M. Anderson, “GEXL96 Correlation for ATRIUM-9B Fuel,” 

12 NEDC-32981P, Global Nuclear Fuel, Wilmington, NC, September 2001, ADAMS 

13 Accession Nos. ML003755947 (Proprietary Version, Nonpublicly Available), 

14 ML012490537 (Safety Evaluation, Proprietary Version, Nonpublicly Available), and 

15 ML012670193 (Nonproprietary Version, Publicly Available).

16 [60] Harrington, R., and J.G.M. Anderson, “GEXL10 Correlation for GE12 Fuel,” 

17 NEDC-32464P, Revision 2, Global Nuclear Fuel, Wilmington, NC, September 2001, 

18 ADAMS Accession No. ML012760512 (Proprietary Information, Nonpublicly Available).

19 [61] Harrington, R., “GEXL80 Correlation for SVEA96+ Fuel,” NEDC-33107P-A, Revision 1, 

20 Global Nuclear Fuel, Wilmington, NC, October 2004, ADAMS Accession 

21 Nos. ML043210062 (Proprietary Version, Nonpublicly Available) and ML043210058 

22 (Nonproprietary Version, Publicly Available).

23 [62] Harris, W., M. Majed, G. Norback, and Y.Y. Yung, “10x10 SVEA Fuel Critical Power 

24 Experiments and CPR Correlation: SVEA-96 Optima2,” WCAP-16081-P-A, Westinghouse 

25 Electric Company, Pittsburgh, PA, March 2005, ADAMS Accession No. ML051260213 

26 (Proprietary Information, Nonpublicly Available) ML003676083 (Nonproprietary Version, 

27 Publicly Available).

28 [63] Global Nuclear Fuel, “GEXL97 Correlation Applicable to ATRIUM-10 Fuel,” 

29 NEDC-33383P, Revision 1, Wilmington, NC, June 2008, ADAMS Accession 

30 No. ML082070090 (Proprietary Information, Nonpublicly Available) and ML082070088

31 (Nonproprietary Version, Publicly Available).

32 [64] Norback, G., and W. Harris, “SVEA-96 Optima2 CPR Correlation (D4): Modified R-factors 

33 for Part-Length Rods,” WCAP-16081-P-A, Addendum 2-A, Westinghouse Electric 

34 Company, Pittsburgh, PA, February 2009, ADAMS Accession Nos. ML072200243 

35 (Proprietary Version, Nonpublicly Available) and ML072200242 (Nonproprietary Version,

36 Publicly Available).

37 [65] Norback, G., and W. Harris, “SVEA-96 Optima2 CPR Correlation (D4): High and Low Flow 

38 Applications,” WCAP-16081-P-A, Addendum 1-A, Westinghouse Electric Company, 

39 Pittsburgh, PA, March 2009, ADAMS Accession Nos. ML091060144 (Proprietary Version,

40 Nonpublicly Available) and ML091060143 (Nonproprietary Version, Publicly Available).

41 [66] Global Nuclear Fuel, “GEXL17 Correlation for GNF2 Fuel,” NEDC-33292P, Revision 3,

42 Wilmington, NC, June 2009, ADAMS Accession No. ML091830641 (Proprietary 

43 Information, Nonpublicly Available) and ML091830624 (Nonproprietary Version, Publicly 

44 Available).

45 [67] AREVA, “SPCB Critical Power Correlation,” EMF-2209(NP)(A), Revision 3, Richland, WA, 

46 September 2009, ADAMS Accession Nos. ML093650230 (Proprietary Version,
90

1 Nonpublicly Available) and ML093650235 (Nonproprietary Version, Publicly Available) and 

2 ML111290532 (Nonproprietary Version, Publicly Available).

3 [68] Global Nuclear Fuel, “GEXL14 Correlation for GE 14 Fuel,” NEDC-32851P-A, Revision 5, 

4 Wilmington, NC, April 2011, ADAMS Accession No. ML111290535 (Proprietary 

5 Information, Nonpublicly Available).

6 [69] AREVA, “ACE/ATRIUM-10 Critical Power Correlation,” ANP-10249(P)(A), Revision 2,

7 Richland, WA, March 2014, ADAMS Accession Nos. ML14175A226 (Part 1 of 3,

8 Proprietary Version, Nonpublicly Available); ML14175A227 (Part 2 of 3, Proprietary 

9 Version, Nonpublicly Available); ML14175A228 (Part 3 of 3, Proprietary Version,

10 Nonpublicly Available), and ML14175A229 (Nonproprietary Version, Publicly Available).

11 [70] AREVA, “ACE/ATRIUM 10XM Critical Power Correlation,” ANP-10298(P)(A), Revision 1, 

12 Richland, WA, March 2014, ADAMS Accession Nos. ML14183A739 (Part 1 of 3, 

13 Proprietary Version, Nonpublicly Available); ML14183A743 (Part 2 of 3, Proprietary 

14 Version, Nonpublicly Available); ML14183A748 (Part 3 of 3, Proprietary Version,

15 Nonpublicly Available); and ML14183A734 (Nonproprietary Version, Publicly Available).

16 [71] Bergmann, U., M. Hemlin, K. Bergman, and J-M. Le Corre, “10x10 SVEA Fuel Critical 

17 Power Experiments and New CPR Correlation: D5 for SVEA-96 Optima3,” 

18 WCAP-17794-P, November 2013, ADAMS Accession Nos. ML13333A276 (Proprietary 

19 Version, Nonpublicly Available) and ML13333A275 (Nonproprietary Version, Publicly 

20 Available).

21 [72] AREVA, “ACE/ATRIUM-11 Critical Power Correlation,” ANP-10335PP, Richland, WA, 

22 February 2015, ADAMS Accession Nos. ML15062A552 (Part 1 of 2, Proprietary Version,

23 Nonpublicly Available); ML15062A555 (Part 2 of 2, Proprietary Version, Nonpublicly 

24 Available); and ML15062A554 (Nonproprietary Version, Publicly Available).

25 [73] U.S. Code of Federal Regulations, “General Design Criteria for Nuclear Power Plants,”

26 Appendix A to “Domestic Licensing of Production and Utilization Facilities,” Part 50, 

27 Chapter I, Title 10, “Energy.”

28 [74] U.S. Code of Federal Regulations, “ECCS Evaluation Models,” Appendix K to “Domestic 

29 Licensing of Production and Utilization Facilities,” Part 50, Chapter I, Title 10, “Energy.”

30 [75] U.S. Code of Federal Regulations, “Contents of Applications; Technical Information,”

31 Section 34 of “Domestic Licensing of Production and Utilization Facilities,” Part 50, 

32 Chapter I, Title 10, “Energy.”

33 [76] U.S. Code of Federal Regulations, “Quality Assurance Criteria for Nuclear Power Plants 

34 and Fuel Reporcessing Plants,” Appendix B to “Domestic Licensing of Production and 

35 Utilization Facilities,” Part 50, Chapter I, Title 10, “Energy.”

36 [77] U.S. Nuclear Regulatory Commission, “Fuel System Design,” Section 4.2 of 

37 NUREG-0800, “Standard Review Plan for the Review of Safety Analysis Reports for

38 Nuclear Power Plants: LWR Edition,” Revision 3, March 2007, ADAMS Accession 

39 No. ML070740002.

40 [78] U.S. Nuclear Regulatory Commission, “Thermal and Hydraulic Design,” Section 4.4 of 

41 NUREG-0800, “Standard Review Plan for the Review of Safety Analysis Reports for

42 Nuclear Power Plants: LWR Edition,” Revision 2, March 2007, ADAMS Accession 

43 No. ML070550060.
91

1 [79] U.S. Nuclear Regulatory Commission, “Fuel Safety Limit Calculation Inputs Were 

2 Inconsistent with NRC-Approved Correlation Limit Values,” Information Notice 2014-01, 

3 February 21, 2014, ADAMS Accession No. ML13325A966.

4 [80] ANSI/ASME NQA-1, “Quality Assurance Program Requirements for Nuclear Power 

5 Plants,” American National Standards Institute/American Society of Mechanical Engineers, 

6 New York, NY, 2015 

7 [81] Dittus, F.W., and L.M.K. Boelter, “Heat Transfer in Automobile Radiators of the Tublar 

8 Type”, University of California Publications in Engineering, 2, pp 443-461, 1930.

9 [82] Box, G.E.P., W.G. Hunter, and J.S. Hunter, Statistics for Experimenters, John Wiley & 

10 Sons, Ltd., New York, NY, 1978.

11 [83] Wieckhorst, O., S. Opel, R. Harne, and F. Filhol, “Challenges in CHF Correlation 

12 Development,” LWR Fuel Performance Meeting (TopFuel 2013), September 15–19, 2013.

13 [84] Wieckhorst, O., H. Gabriel, R. Harne, M. Anghelescu, and O. Martinie, “ORFEO—A CHF 

Correlation for GAIA, AREVA’s Advanced PWR Fuel Assembly Design,” 16th 14 International 

15 Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-16), Chicago, IL, 

16 August 30–September 4, 2015.

17 [85] Anderson, M.G., and P.D. Bates (Eds.), Model Validation: Perspectives in Hydrological 

18 Science. John Wiley & Sons, Ltd., Chichester, United Kingdom, 2001.

19 [86] National Research Council, Ground Water Models; Scientific and Regulatory Applications,

20 The National Academies Press, Washington, DC, p. 303, 1990.

21 [87] Owen, D.B., “Factors from One-Sided Tolerance Limits and the Variables Sampling 

22 Plans,” SCR-607, Sandia Corporation, Albuquerque, NM, 1963.

23 [88] Wilks, S.S., “Determination of Sample Sizes for Setting Tolerance Limits,” The Annals of 

24 Mathematical Statistics, 12(1), 91–96, 1941.

25 [89] Wald, A., “An Extension of Wilks’ Method for Setting Tolerance Limits,” The Annals of 

26 Mathematical Statistics, 14(1), 43–55, 1943.
92
A-1

1 APPENDIX A LISTING OF ALL GOALS 

GOAL The critical boiling transition model can be trusted.

G1 The experimental data supporting the critical boiling transition 

model are appropriate. 

G1.1 The experimental data have been collected at a credible test 

facility.

G1.1.1 The test facility is well understood.

G1.1.2 The test facility has been verified by comparison to an outside 

source.

G1.2 The experimental data have been accurately measured.

G1.2.1 The test facility has an appropriate quality assurance program.

G1.2.2

The experiment has been appropriately statistically designed 

(i.e., the value of a system parameter from any test was 

completely independent from its value in the test before and 

after the test).

G1.2.3 The method used to obtain critical boiling transition data 

results in an accurate measurement.

G1.2.4

The instrumentation uncertainties have been demonstrated to 

have a minimal impact on the measured critical heat flux or 

critical power.

G1.2.5 The uncertainty in the critical heat flux or critical power is 

quantified through repeated tests at the same state points.

G1.2.6

The heat losses from the test section are quantified, 

appropriately low, and duly accounted for in the measured 

data.

G1.3 The test assembly reproduced the local conditions in the 

reactor fuel assembly.

G1.3.1

The test assembly used in the experiment should have 

geometric dimensions equivalent to those of the fuel assembly

used in the reactor for all major components.

G1.3.2 The grid spacers used in the test assembly should be 

prototypical of the grid spacers used in the reactor assembly. 

G1.3.3

The axial power shapes in the test assembly should reflect the 

expected or limiting axial power shapes in the reactor 

assembly.

G1.3.4 The radial power peaking in the test assembly should reflect 

the expected or limiting radial powers in the reactor assembly.

G1.3.5

Any differences between the test assembly and the reactor 

assembly should have a minimal impact on the flow field. This 

includes components that are not in the reactor assembly but 

that are needed for testing purposes.

G2 The model was generated in a logical fashion. 
A-2

G2.1 The mathematical form of the model is appropriate. 

G2.1.1 The mathematical form of the model contains all the 

necessary parameters. 

G2.1.2 The reasoning for choosing the mathematical form of the 

model should be discussed and should be logical.

G2.2 The process for determining the model’s coefficients was 

appropriate. 

G2.2.1 The training data (i.e., the data used to generate the 

coefficients of the model) should be identified. 

G2.2.2 The method for calculating the model’s coefficients should be 

described.

G2.2.3

The method for calculating the R- or K-factor and the additive 

constants (for both full-length and part-length rods) should be 

described. Further, a description of how such values are 

calculated if dryout is not measured on the rod under 

consideration should be provided (boiling-water reactors only).

G3 The model has sufficient validation as demonstrated through 

appropriate quantification of its error.

G3.1 The correct validation error has been calculated. 

G3.2 The validation error is appropriately distributed throughout the 

application domain.

G3.2.1 The validation data (i.e., the data used to quantify the model’s 

error) should be identified.

G3.2.2 The application domain of the model should be 

mathematically defined.

G3.2.3 The expected domain of the model should be understood.

G3.2.4 There should be adequate validation error data density 

throughout the expected and application domains.

G3.2.5

Sparse regions (i.e., regions of low data density) in the 

expected and application domains should be identified and 

justified.

G3.2.6 The model should be restricted to its application domain.

G3.3 Any inconsistencies in the validation error have been 

accounted for appropriately. 

G3.3.1

The validation error should be investigated to ensure that it 

does not contain any subgroups that are obviously not from 

the same population (i.e., non-poolable).

G3.3.2

The expected domain should be investigated to determine if it 

contains any non-conservative subregions that would impact 

the predictive capability of the model.

G3.3.3 The model’s predictions trend as expected in each of the 

various model parameters.
A-3

G3.4 The model’s uncertainty has been appropriately calculated 

from the validation error.

G3.4.1 The validation error statistics should be calculated from an 

appropriate database. 

G3.4.2 The validation error statistics should be calculated using an 

appropriate method. 

G3.4.3 The model’s uncertainty should be appropriately biased. 

G3.5 The model has been correctly implemented.

G3.5.1 The model has been implemented in the same computer code 

that was used to generate the validation error. 

G3.5.2

The model’s prediction of the CBT is being applied using the 

same evaluation methodology used to predict the validation 

data set for determining the validation error.

G3.5.3 The model results in an accurate or conservative prediction 

when it is used to predict transient behavior.

1



Technical 

Division of Safety Systems

Office of Nuclear Reactor Regulation

U.S. Nuclear Regulatory Commission

Washington, DC 20555-0001

Same as above

Critical boiling transition (CBT) occurs when a flow regime that has a higher heat transfer rate transitions to a flow 

regime that has a significantly lower heat transfer rate. Models that predict a CBT are a necessary part of reactor safety 

analysis because they are used to determine plant safety limits. Therefore, the review of CBT models has been a focus 

of the U.S. Nuclear Regulatory Commission (NRC) since its inception in 1975. 

This work presents a generic safety case in the form of a credibility assessment framework that combines aspects of 

goal structuring notation and maturity assessment. This framework is focused on the credibility assessment of CBT 

models with specific application to reactor safety analysis. The NRC has performed many such assessments and has 

generated this framework based on the experience of current and former NRC staff, as well as previous staff reviews as 

summarized in staff evaluations. This document includes a survey of the important technical and regulatory literature; a 

detailed technical discussion of CBT models and their application; and a suggested framework for CBT models. This 

NUREG/KM summarizes the knowledge the NRC staff has developed over the course of 40 years of CBT model and 

analysis reviews. 

Critical heat flux, critical power, departure from nucleate boiling, critical quality, 

boiling crisis, burnout, dryout, critical boiling transition 


United States 

Nuclear Regulatory Commission 

NUREG/KM-0013 

Draft

March 2019