Technical Report

NREL/TP-550-47502 

August 2010

Example Procedures for 

Developing Acceptance-Range 

Criteria for BESTEST-EX

Ron Judkoff, Ben Polly, and Marcus Bianchi

National Renewable Energy Laboratory

Joel Neymark

J. Neymark & Associates

Link to Accompanying Zipped Data Files (938 KB)
 

Technical Report Example Procedures for NREL/TP-550-47502 Developing Acceptance-Range August 2010 

Criteria for BESTEST-EX 

Ron Judkoff, Ben Polly, and Marcus Bianchi 

National Renewable Energy Laboratory 

Joel Neymark 

J. Neymark & Associates 

Prepared under Task No. ARRB.1000 

National Renewable Energy Laboratory

1617 Cole Boulevard, Golden, Colorado 80401-3393 

303-275-3000 • www.nrel.gov 

NREL is a national laboratory of the U.S. Department of Energy 

Office of Energy Efficiency and Renewable Energy 

Operated by the Alliance for Sustainable Energy, LLC 

Contract No. DE-AC36-08-GO28308 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

NOTICE 

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, nor any of their 

contributors, makes any warranty, express or implied, or assumes any legal liability or responsibility for the 

accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or 

represents that its use would not infringe privately owned rights. Reference herein to any specific commercial 

product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily 

constitute or imply its endorsement, recommendation, or favoring by the United States government or any 

agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those 

of the United States government or any agency thereof. 

Available electronically at http://www.osti.gov/bridge 

Available for a processing fee to U.S. Department of Energy 

and its contractors, in paper, from: 

U.S. Department of Energy 

Office of Scientific and Technical Information 

P.O. Box 62 

Oak Ridge, TN 37831-0062 

phone: 865.576.8401 

fax: 865.576.5728 

email: mailto:reports@adonis.osti.gov 

Available for sale to the public, in paper, from: 

U.S. Department of Commerce 

National Technical Information Service 

5285 Port Royal Road 

Springfield, VA 22161 

phone: 800.553.6847 

fax: 703.605.6900 

email: orders@ntis.fedworld.gov 

online ordering: http://www.ntis.gov/ordering.htm 

Printed on paper containing at least 50% wastepaper, including 20% postconsumer waste 
 

Acknowledgments 

For feedback and comments on the test procedure provided by BESTEST-EX Working Group 

participants during preliminary simulation trials, we wish to thank the following companies and 

individuals that ran simulations and responded to questionnaires: Apogee Interactive Inc.: Joel Gilbert, 

John Laun, Eric Shewbridge, Lei Wang; Architectural Energy Corporation: Brian Christensen, Rob 

Salcido; Conservation Services Group: Bruce Harley, Mick Rookwood, David Weitz, Michael Blasnik 

(consultant); Florida Solar Energy Center: Philip Fairey; ICF International: Brian Dean, Mike L’Ecuyer; 

Oak Ridge National Laboratory: Mark Ternes; Performance Systems Development LLC: Chris Balbach, 

Ethan MacCormick, John McCartney, Greg Thomas. 

Paul Norton of the National Renewable Energy Laboratory (NREL) provided initial organizational and 

management support during project startup. Ren Anderson of NREL also provided managerial support. 

Nick Long of NREL provided assistance with automating reference simulations. 

We appreciate the support and guidance of Ed Pollock, Terry Logee, and Lew Pratsch, of the 

U.S. Department of Energy. 

iii 
 

 

 

 

 

Nomenclature 

Avg. average 

BESTEST Building Energy Simulation Test 

Conf confidence interval 

DOE U.S. Department of Energy 

DOE-2.1E DOE-2.1E version JJ Hirsch PC 2.1En136 

E+ EnergyPlus version 3.1 

EIA Energy Information Administration 

HERS Home Energy Rating System 

Ins. insulation 

Max maximum 

Min minimum 

NREL National Renewable Energy Laboratory 

Ref reference simulation result 

sol_abs solar absorptance 

Sqrt square root 

Stdv standard deviation (“N−1” [sample] type) 

SUNREL SUNREL version 1.14 

“-C” calibrated energy savings test cases 

“-P” building physics test cases 

iv 
 

 

Contents 

Acknowledgments .............................................................................................. iii 

Nomenclature .................................................................................................... iv 

Accompanying Files ........................................................................................... vi 

Introduction ...................................................................................................... vi 

1 Establishing Acceptance Ranges ..................................................................... 1 

2 BESTEST-EX Acceptance Criteria Overview .................................................... 1 

3 Example of Procedure for Developing Acceptance Ranges ..................................... 2 

4 Additional Criteria ..................................................................................... 5 

5 Acceptance Criteria as Applied to “-P” Test Cases ................................................ 5 

References ....................................................................................................... 9 

Figures 

Figure 1. Building physics heating tests: Reference simulation results and acceptance criteria ..... 7 

Figure 2. Building physics cooling tests: Reference simulation results and acceptance criteria ..... 7 

Tables 

Table 1. Example Range Criteria Using Fictitious Reference Results .................................... 3 

Table 2. Sample Student’s t Confidence Coefficients (tc) ................................................... 4 

Table 3. BESTEST-EX Example Range-Setting Procedure: Building Physics Heating Tests ........ 8 

Table 4. BESTEST-EX Example Range-Setting Procedure: Building Physics Cooling Tests ........ 8 

v 
 

 

 

Accompanying Files (Electronic Media Contents) 

The following file provided within B-EX-Phase-1-Ref-P-Results+Example-acceptance-criteria.zip 

applies as it is called out in this document: 

B-EX-Phase-1-Ref-P-Results+Example-Acceptance-Criteria.xls: Spreadsheet that contains 

reference simulation results presented in Judkoff et al. 2010, Appendix G. The example acceptance 

criteria presented in this document are applied to the physics results in the accompanying spreadsheet. 

Introduction 

A certifying or accrediting agency may develop acceptance-range setting criteria to suit particular needs. 

Neither DOE, NREL, nor the authors of this document may be held responsible for any misfortunes that 

occur from use of these example acceptance criteria in a certification program. 

This document provides an example procedure for establishing acceptance-range criteria to assess results 

from software undergoing BESTEST-EX (Judkoff et al. 2010). This example method for BESTEST-EX 

is a modified version of the method described in HERS BESTEST (Judkoff and Neymark 1995). 

vi 
 

 

 

 

 

1 Establishing Acceptance Ranges 

In choosing algorithms for determining acceptance ranges, it is important to consider the following: 

1.	 Establishing a buffer range around reference results is desirable for the following reasons: 

•	 Minor differences have minor energy cost impacts; therefore, a result just outside the range of 

reference results should be acceptable. 

•	 Where confidence interval ranges are very narrow, it is advisable to have alternative 

“economic threshold” buffer zone range expansion criteria so software is not eliminated 

because of relatively insignificant differences in energy consumption or energy costs. 

•	 Allow some bias for cautious (conservative) energy savings predictions, but limit the allowed 

overprediction of energy savings. 

2.	 The use of statistical confidence intervals (Spiegel 1961) provides a theoretical basis for developing 

acceptance ranges. The 95% confidence level was chosen for the example presented here because a 

97.5% confidence interval would widen the acceptance range to a point where the test cases lack 

meaning (are too easy to pass). In HERS BESTEST it was determined empirically that for most cases, 

confidence coefficients corresponding to confidence intervals in the range of 80%–95% yield 

reasonable acceptance ranges. 

3.	 Where reference results are very close together, such that the confidence interval maximum or 

minimum values could fall very close to the reference results maximum or minimum values, a value 

of $6/month (about 5% of annual heating consumption and 10% of annual cooling consumption for 

the building physics base case [L200EX-P]) is applied to the range. This results in range expansion 

values of ± 5.72 million Btu/yr and ± 621 kWh/yr assuming $12.58/million Btu and $0.116/kWh 

average 2006–2008 residential retail (delivered) heating season gas and cooling season electricity 

prices, respectively (EIA 2009a, 2009b). These values are taken as a reasonable threshold of 

economic uncertainty. That is, any software disagreements within ± 5.72 million Btu or ± 621 kWh of 

the reference results extremes for a given case, including difference (or “delta”) cases, would result in 

relatively insignificant utility cost disagreements and therefore should not be cause for eliminating a 

given software tool, even if it falls outside confidence limits based on the chosen confidence interval. 

Depending on fuel prices, climate, mortgage lending policies, and other circumstances in specific 

regions, it may make sense to adjust these criteria. 

4.	 To limit allowed overprediction of energy savings, the confidence interval and economic threshold 

criteria for extending the maximum range are reduced to 90%, and 3.88 million Btu/yr or 421 

kWh/yr, respectively. This is discussed further in the development of the acceptance range setting 

equations (see Section 3). 

5.	 Some cases may deserve stricter acceptance criteria than would be generated using the range setting 

procedure described above. A possible example would be cases with higher absolute loads or higher 

load differences. In these cases, where the percentage variation among reference results can be 

roughly consistent with those for lower load cases, the higher values may produce an unreasonably 

large extension of the acceptance range in terms of estimated fuel costs. Acceptance ranges may be 

narrowed by altering the confidence interval or the economic threshold buffer. However, the 

acceptance range must always include the maximum and minimum values of the reference results. 

2 BESTEST-EX Acceptance Criteria Overview 

Within BESTEST-EX the building physics (“-P”) cases are specified differently than the calibrated 

energy savings (“-C”) cases. The “-P” cases provide explicit inputs for all cases. The “-C” cases provide 

approximate input ranges for key inputs to account for uncertainty associated with audit information and 

measurements, occupant behavior, etc. For the “-C” cases, explicit inputs are randomly selected within 

the approximate input ranges to generate utility bills using the reference simulation programs; tested 

1
 
 

 

 

 

 

 

software tools are allowed to apply calibration given the reference utility billing data and approximate 

input ranges (selected explicit inputs used for the reference simulations remain hidden to allow for blind 

testing). Because the “-C” cases apply approximate input ranges (known uncertainty) for selected inputs, 

and because some base-case scenarios (see Judkoff et al. 2010, Section 1.3.1.2) can have randomly 

selected reference explicit inputs that are more difficult to estimate from calibration than others, the 

acceptance criteria for the “-C” cases should be less strict than that for the “-P” cases. Therefore, the 

following example acceptance criteria are provided: 

•	 “-P” case acceptance 

o	 Programs must pass all designated cases 

 “-P” reference results are provided with the test procedure 

o	 Compare all energy savings case results 

o	 Compare annual usage only for the base case (L200EX-P) 

•	 “-C” case acceptance 

o Programs must pass a reasonable fraction (example: 80%) of the designated cases 

 “-C” energy savings reference results are not provided with the test procedure 

o	 Compare all energy savings case results only 

 Base-case annual usage results are calibrated to reference bills. 

3 Example of Procedure for Developing Acceptance Ranges 

Table 1 presents example fictitious results and acceptance range limits that result from the example 

procedure described here. A step-by-step description of the procedures used to arrive at each element is 

also included. Values indicated by bold font in Table 1 are the resulting acceptance range limit values for 

the fictitious results set, as determined using the example range setting criteria described below. This is a 

modified version of the example originally presented in HERS BESTEST Volume 1, Appendix H. A 

notable difference between the HERS BESTEST example and that in BESTEST-EX is that the example 

in BESTEST-EX focuses on energy savings sensitivity (or “delta”) cases; only one building physics case 

(L200EX-P) is examined in an annual usage (“absolute”) context for developing BESTEST-EX 

acceptance criteria. 

1.	 Using Reference Results 1, 2, and 3 from Table 1, determine the maximum reference result, the 

minimum reference result, the sample mean (average) of the reference results, and the sample 

standard deviation (using N−1 method) of the reference results. These quantities are shown in 

Table 1 as “Ref Max,” “Ref Min,” “Ref Mean,” and “Ref Stdv,” respectively. 

2
 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1. Example Range Criteria Using Fictitious Reference Results 

Description Case 1 

(106 Btu) 

Case 2 

(106 Btu) 

Delta Case 1 −

Case 2 

(106 Btu) 

Reference Result 1 73.00 46.00 27.00 

Reference Result 2 70.00 45.00 25.00 

Reference Result 3 82.00 49.00 33.00 

Ref Max 82.00 49.00 33.00 

Ref Min 70.00 45.00 25.00 

Ref Mean 75.00 46.67 28.33 

Ref Stdv (“N−1” [sample] type) 6.24 2.08 4.16 

Ref 95% Conf Max 90.51 51.84 

Ref 90% Conf Max 35.35 

Ref 95% Conf Min 59.49 41.50 17.99 

Ref Max + 5.72 million Btu (6.035 GJ)* 87.72 54.72 

Ref Max + 3.88 million Btu (4.094 GJ) 36.88 

Ref Min − 5.72 million Btu (6.035 GJ) 64.28 39.28 19.28 

Example Range Max 90.51 54.72 36.88 

Example Range Min 59.49 39.28 17.99 

*	 1 million Btu = 1.055056 GJ = 0.2930711 MWh. 

2.	 Calculate the confidence interval for the population sample mean assuming a Student’s t 

distribution (Spiegel 1961) based on the reference results. The extremes (confidence limits) of the 

confidence interval for the population mean are determined from: 

L (Eq. 1) a	 = X 

N 

stn c )( +

N 

stc )( L − (Eq. 2) b = X 

Where: 

La = maximum confidence limit for the confidence interval 

Lb = minimum confidence limit for the confidence interval 

X = sample mean 

tc = confidence coefficient, see below 

s = sample standard deviation = SQRT{SUM[(xj − AVG(xj))2

]/(N−1)}, 

for j = 1 to N
 

N = number of samples
 

n = upper limit reduction factor to limit overprediction of energy
 

savings; n = 1 for annual usage (“absolute”) results. 

The confidence coefficient (tc) is determined by the sample size and the desired confidence 

interval. For this example, with a sample size of three (N = 3) and a desired confidence interval 

of 95%: 

3
 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

tc = 4.303 (see Table 2 for other confidence coefficients) (Eq. 3) 

To limit allowed overprediction of energy savings, for energy savings sensitivity (“delta”) results 

“n” was selected such that the upper acceptance range limit results from a 90% confidence 

interval. This results in an asymmetric acceptance range that implicitly allows bias for cautious 

(conservative) energy savings predictions. Equation 3a solves for “n”, applying values of Table 2, 

as: 

n = 2.920 / 4.303 = 0.6786	 (Eq. 3a) 

Plugging values from above into Equations 1 and 2 gives: 

3

)303.4( s L + (for “absolute” maxima)		 (Eq. 4) a95 = X 

(0.6786)(4.303)s (for “delta” maxima) (Eq. 5) 

3 

(4.303)s 

La90 = X +

Lb = X − (for “absolute” and “delta” minima) (Eq. 6) 

3 

The resulting confidence limits are shown in Table 1 as “Ref 95% Conf Max,” “Ref 90% Conf 

Max,” and “Ref 95% Conf Min.” 

Table 2 provides a limited set of Student’s t confidence coefficients that may be used for other 

sample sizes and confidence intervals. Additional tables for other confidence limits and sample 

sizes are available in many statistics text books. 

Table 2. Sample Student’s t Confidence Coefficients (tc) 

Sample Size 

(N) 

Desired Confidence Interval 

80% 90% 95% 

2 3.078 6.314 12.706 

3 1.886 2.920 4.303 

4 1.638 2.353 3.182 

5 1.533 2.132 2.776 

6 1.476 2.015 2.571 

7 1.440 1.943 2.447 

8 1.415 1.895 2.363 

3.	 Similarly, calculate: 

Lc = (Ref Max) + n × 5.72 million Btu (6.035 GJ) (Eq. 7) 

Ld = (Ref Min) − 5.72 million Btu (6.035 GJ) (Eq. 8) 

The results of these calculations are shown in Table 1 as “Ref Max + 5.72 million Btu (6.035 

GJ)” (applying n = 1 in Equation 7), “Ref Max + 3.88 million Btu (4.094 GJ)” (applying n = 

0.6786 in Equation 7), and “Ref Min − 5.72 million Btu (6.035 GJ)” (applying Equation 8). 

4.	 The example acceptance range (“Range Max,” “Range Min”) is then determined for the delta 

results by taking the maximum of “Ref 90% Conf Max” and “Ref Max + 3.88 million Btu (4.094 

4
 
 

 

 

 

 

 

 

GJ)” as “Range Max” and the minimum of “Ref 95% Conf Min” and “Ref Min − 5.72 million 

Btu (6.035 GJ)” as “Range Min.” Similarly, the “Range Max” for absolute results is determined 

by taking the maximum of “Ref 95% Conf Max” and “Ref Max + 5.72 million Btu (6.035 GJ)”; 

“Range Min” for the absolute results is the same as that for the delta results. Using Table 1, a 

software tool passes a case if its test result falls within the “Example Range Max” and “Example 

Range Min” for that case. In Table 1, fictitious sets of results are used, such that the confidence 

interval ranges and the economic threshold ranges set the range extremes for Case 1 and Case 2, 

respectively. It is also possible to have results where one range-setting method sets one extreme 

and the other range-setting method sets the other extreme, as shown in the “Delta Case 1 − Case 

2” result of Table 1. 

For this example, a software tool would “pass” a particular test case if its result for that test case falls 

within the acceptance range represented by “Example Range Max” and “Example Range Min” in the 

bottom portion of Table 1. Similarly, a software tool would pass a test suite if its results for all “-P” test 

cases and a satisfactory fraction of “-C” test cases in the given test suite fall within all acceptance ranges. 

4 Additional Criteria 

For the building physics test cases the above criteria allow zero and opposite sensitivities to pass some of 

the cases. This occurs where the sensitivity of the retrofit is relatively small. As a result of preliminary 

simulation trials, additional criteria were added so that zero and opposite sensitivities are not allowed to 

pass for the following cases: 

•	 L260−L265EXPC (electricity consumption decrease by cool roof for the building physics tests 

with space cooling) 

•	 L200−L265EXCnC (electricity consumption decrease by cool roof for the calibrated energy 

savings tests with space cooling) 

•	 L200−L240EXCnC (electricity consumption decrease by thermostat “setup” for the calibrated 

energy savings test with space cooling). 

This rule was applied for these cases only. Of the building physics cooling test case results that would 

allow zero and opposite sensitivities to pass before applying the additional criteria, the cool roof (Case 

L265EX-P) has the largest mean sensitivity among reference results. Without the additional criteria, the 

minimum range boundary for the “-P” thermostat setup sensitivity (versus the base case) is just above 

zero; however, many “-C” results for thermostat setup sensitivity would be allowed to have zero and 

opposite sensitivities. Other cases either had enough sensitivity so that applying the additional criteria was 

not necessary, or had too little sensitivity in the reference simulation results for the building physics cases 

to justify applying the additional criteria. 

5 Acceptance Criteria as Applied to “-P” Test Cases 

An example of applying this procedure to the BESTEST-EX reference results for the “-P” cases follows. 

Reference results were developed using: 

•	 DOE-2.1E version JJ Hirsch PC 2.1En136 (DOE-2 Reference Manual 1981, DOE-2 Supplement 

1994) 

•	 EnergyPlus version 3.1 (EnergyPlus Input Output Reference 2009) 

•	 SUNREL version 1.14 (Deru et al. 2002) 

In Figures 1 and 2 the acceptance range maxima and minima are indicated by “range” bars. The 

statistically based acceptance ranges are shown with blue range bars; the economic threshold based ranges 

5
 
 

 

 

are shown with green range bars. A tested tool passes a case if its result for that case falls within the 

greatest maximum and least minimum defined by the blue and green range bars. 

The example acceptance ranges for the BESTEST-EX “-P” cases are developed as shown in Tables 3 and 

4. An electronic version of the calculations is provided with B-EX-Phase-1-Ref-P-Results+ExampleAcceptance-Criteria.xls included with the accompanying electronic files. Cell addresses for finding data 

in the xls file are given in small font below the tables. 

Only the results and acceptance ranges for the building physics (“-P”) test cases are shown in the figures 

and the tables. For the calibrated energy savings (“-C”) test cases, reference simulation results and 

randomly selected explicit inputs used in the reference simulations are intentionally not given for blind 

testing. 

6
 
 

 

Annual Gas Usage or Savings 

Buildings Physics Heating Tests 

150 

Gas Usage or Savings (million Btu)

125 

100 

75 

50 

25 

0 

-25 

L200EX-PH L200 - L200 - L200 - L200 - L200 - L260 - L200 - L200 ­

base-case L210EXPH L220EXPH L225EXPH L240EXPH L250EXPH L265EXPH L270EXPH L300EXPH 

air_seal attic_ins. wall_ins. setback windows sol_abs shading combined 

Case 

E+ 

SUNREL 

DOE2.1E 

Figure 1. Building physics heating tests: Reference simulation results and acceptance criteria 

Annual Electricity Usage or Savings 

Buildings Physics Cooling Tests 

-1000 

1000 

3000 

5000 

7000 

9000 

11000 

13000 

15000 

17000 

Electricity Usage or Savings (kWh)

E+ 

SUNREL 

DOE2.1E 

L200EX-PC L200 - L200 - L200 - L200 - L200 - L260 - L200 - L200 ­

base-case L210EXPC L220EXPC L225EXPC L240EXPC L250EXPC L265EXPC L270EXPC L300EXPC 

air_seal attic_ins. wall_ins. setback windows sol_abs shading combined 

Case 

Figure 2. Building physics cooling tests: Reference simulation results and acceptance criteria 

7 
 

 

 

 

 

Table 3. BESTEST-EX Example Range-Setting Procedure: Building Physics Heating Tests 

Total Annual Gas Consumption and Savings (million Btu/y) 

Ref Conf Conf Range Range 

Case EnergyPlus SUNREL DOE2.1E MEAN Max Min $ Max $ Min Max Min 

L200EX-PH base-case 119.01 134.68 119.32 124.34 146.59 102.08 140.40 113.29 146.59 102.08 

L200 - L210EXPH air_seal 17.14 15.88 15.33 16.12 17.68 13.81 21.02 9.61 21.02 9.61 

L200 - L220EXPH attic_ins. 14.27 15.74 14.34 14.78 16.19 12.72 19.63 8.55 19.63 8.55 

L200 - L225EXPH wall_ins. 19.10 25.00 18.69 20.93 26.88 12.16 28.88 12.97 28.88 12.16 

L200 - L240EXPH setback 10.91 11.42 10.56 10.96 11.69 9.89 15.30 4.84 15.30 4.84 

L200 - L250EXPH windows 10.86 17.50 9.92 12.76 19.73 2.49 21.38 4.20 21.38 2.49 

L260 - L265EXPH sol_abs -4.08 -2.74 -2.58 -3.13 -1.74 -5.19 1.31 -9.81 1.31 -9.81 

L200 - L270EXPH shading -9.27 -11.66 -9.65 -10.19 -8.03 -13.38 -5.38 -17.38 -5.38 -17.38 

L200 - L300EXPH combined 66.38 77.81 65.34 69.85 81.51 52.66 81.70 59.62 81.70 52.66

 B-EX-Phase-1-Ref-P-Results+Example-Accpetance-Criteria.xls: GasHtgData! A264:L277 7-May-2010 

Table 4. BESTEST-EX Example Range-Setting Procedure: Building Physics Cooling Tests 

Total Annual Electricity Consumption and Savings (kWh/y) 

Ref Conf Conf Range Range 

Case EnergyPlus SUNREL DOE2.1E MEAN Max Min $ Max $ Min Max Min 

L200EX-PC base-case 10664 11966 10622 11084 12982 9186 12587 10001 12982 9186 

L200 - L210EXPC air_seal 140 103 156 133 178 67 577 -517 577 -517 

L200 - L220EXPC attic_ins. 405 596 428 476 653 216 1018 -216 1018 -216 

L200 - L225EXPC wall_ins. 454 656 259 456 792 -38 1078 -362 1078 -362 

L200 - L240EXPC setback 671 765 700 712 793 593 1186 50 1186 50 

L200 - L250EXPC windows 1310 1840 1234 1461 2017 642 2261 613 2261 613 

L260 - L265EXPC sol_abs 821 609 586 672 890 350 1242 >0 1242 >0 

L200 - L270EXPC shading 1247 1508 1325 1360 1585 1028 1929 626 1929 626 

L200 - L300EXPC combined 3235 4161 3330 3575 4435 2309 4583 2614 4583 2309

 B-EX-Phase-1-Ref-P-Results+Example-Acceptance-Criteria.XLS: ElecClgData! A275:L288 7-May-2010 

Abbreviations used in tables: 

Conf = value determined from confidence interval equations (see Equations 1 through 6) 

Max = maximum 

Min = minimum 

Ref = reference simulations 

$ = economic threshold criteria (see Equations 7 and 8) 

8
 
 

 

 

 

 

 

 

 

 

References 

Deru, M.; Judkoff, R.; Torcellini, P. (2002). SUNREL Technical Reference Manual. NREL/BK-550­

30193. Golden, CO: National Renewable Energy Laboratory. 

DOE-2 Reference Manual (Version 2.1A) Part 1. (1981). D. York and C. Cappiello, eds. Berkeley, CA: 

Lawrence Berkeley Laboratory. 

DOE-2 Supplement (Version 2.1E). (1994). Berkeley, CA: Lawrence Berkeley Laboratory. 

EnergyPlus Input Output Reference. (2009). Urbana, IL: University of Illinois, and Berkeley, CA: 

Lawrence Berkeley National Laboratory. http://apps1.eere.energy.gov/buildings/energyplus/pdfs/ 

InputOutputReference.pdf/. Last accessed January 2010. 

EIA. (2009a). “Annual U.S. Price of Natural Gas Delivered to Consumers.” Washington, D.C: Energy 

Information Administration. http://tonto.eia.doe.gov/dnav/ng/hist/n3010us3m.htm. Last accessed 

January 2010. 

EIA. (2009b). “Average Retail Price of Electricity to Ultimate Customers: Total by End-Use Sector.” 

Washington, D.C: Energy Information Administration. www.eia.doe.gov/cneaf/electricity/epm/ 

table5_3.html. Last accessed January 2010. 

Judkoff, R.; Neymark, J. (1995). Home Energy Rating System Building Energy Simulation Test (HERS 

BESTEST). NREL/TP-472-7332a and TP-472-7332b. Golden, CO: National Renewable Energy 

Laboratory. 

Judkoff, R.; Neymark, J.; Polly, B.; Bianchi, M. (2010). Building Energy Simulation Test for Existing 

Homes (BESTEST-EX) Phase 1 Test Procedure: Building Thermal Fabric Cases. NREL/TP-550-47427. 

Golden, CO: National Renewable Energy Laboratory. 

Spiegel, M.R. (1961). Schaum’s Outline of Theory and Problems of Statistics. New York, NY: 

McGraw-Hill. 

9
 
F1147-E(10/2008) 

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing the burden, to Department of Defense, Executive Services and Communications Directorate (0704-0188). Respondents 

should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a 

currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ORGANIZATION.

1. REPORT DATE (DD-MM-YYYY)

August 2010

2. REPORT TYPE

Technical Report

3. DATES COVERED (From - To)

 

4. TITLE AND SUBTITLE

Example Procedures for Developing Acceptance-Range Criteria for 

BESTEST-EX

5a. CONTRACT NUMBER

DE-AC36-08-GO28308

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

R. Judkoff, B. Polly, M. Bianchi, and J. Neymark

5d. PROJECT NUMBER

NREL/TP-550-47502 

5e. TASK NUMBER

ARRB.1000

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

National Renewable Energy Laboratory

1617 Cole Blvd.

Golden, CO 80401-3393

8. PERFORMING ORGANIZATION

REPORT NUMBER

NREL/TP-550-47502 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

NREL

11. SPONSORING/MONITORING

AGENCY REPORT NUMBER

12. DISTRIBUTION AVAILABILITY STATEMENT

National Technical Information Service

U.S. Department of Commerce

5285 Port Royal Road

Springfield, VA 22161

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 Words)

This document provides an example procedure for establishing acceptance-range criteria to assess results from 

software undergoing BESTEST-EX. This example method for BESTEST-EX is a modified version of the method 

described in HERS BESTEST. 

15. SUBJECT TERMS

bestest; retrofit energy savings; building energy simulation; acceptance criteria

16. SECURITY CLASSIFICATION OF: 17. LIMITATION 

OF ABSTRACT

UL

18. NUMBER 

OF PAGES

19a. NAME OF RESPONSIBLE PERSON

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified 19b. TELEPHONE NUMBER (Include area code)

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18