NREL is a national laboratory of the U.S. Department of Energy, Office of Energy 

Efficiency & Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

Contract No. DE-AC36-08GO28308

Building Energy Simulation Test for 

Existing Homes (BESTEST-EX): 

Instructions for Implementing the 

Test Procedure, Calibration Test 

Reference Results, and Example 

Acceptance-Range Criteria

Ron Judkoff, Ben Polly, and Marcus Bianchi

National Renewable Energy Laboratory

Joel Neymark

J. Neymark & Associates

Mike Kennedy

Mike D. Kennedy, Inc. 

Link to Accompanying Zipped Data Files (3.9 MB)

This document is intended for use with the following documents:

Building Energy Simulation Test for Existing Homes (BESTEST-EX), 

NREL/TP-550-47427

Example Procedures for Developing Acceptance-Range Criteria for BESTEST-EX, 

NREL/TP-550-47502

Technical Report

NREL/TP-5500-52414

August 2011
NREL is a national laboratory of the U.S. Department of Energy, Office of Energy 

Efficiency & Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

National Renewable Energy Laboratory

1617 Cole Boulevard

Golden, Colorado 80401

303-275-3000 • www.nrel.gov

Contract No. DE-AC36-08GO28308

Building Energy Simulation Test for 

Existing Homes (BESTEST-EX): 

Instructions for Implementing the 

Test Procedure, Calibration Test 

Reference Results, and Example 

Acceptance-Range Criteria

Ron Judkoff, Ben Polly, and Marcus Bianchi

National Renewable Energy Laboratory

Joel Neymark

J. Neymark & Associates

Mike Kennedy

Mike D. Kennedy, Inc. 

Prepared under Task No. ARRB.2302

Technical Report

NREL/TP-5500-52414

August 2011
NOTICE

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, 

express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of 

any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately 

owned rights. Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, 

or favoring by the United States government or any agency thereof. The views and opinions of authors 

expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.

Available electronically at http://www.osti.gov/bridge

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/help/ordermethods.aspx

Cover Photos: (left to right) PIX 16416, PIX 17423, PIX 16560, PIX 17613, PIX 17436, PIX 17721

Printed on paper containing at least 50% wastepaper, including 10% post consumer waste. 
iii

Acknowledgments

For feedback and comments on the test procedure provided by BESTEST-EX Working Group 

participants during preliminary simulation trials, we wish to thank the following companies and 

individuals that ran simulations and responded to questionnaires: Apogee Interactive Inc.: Joel Gilbert, 

Eric Shewbridge, Lei Wang; Architectural Energy Corporation: Brian Christensen, Rob Salcido; 

Conservation Services Group: Bruce Harley, Mick Rookwood, David Weitz, Michael Blasnik 

(consultant); Florida Solar Energy Center: Philip Fairey; ICF International: Brian Dean, Mike L’Ecuyer; 

Oak Ridge National Laboratory: Mark Ternes; Performance Systems Development LLC: Chris Balbach, 

Ethan MacCormick, John McCartney, Greg Thomas.

Ren Anderson of the National Renewable Energy Laboratory (NREL) provided managerial support. Nick 

Long of NREL provided assistance with automating reference simulations. Paul Norton formerly of 

NREL (now a private consultant) provided initial organizational and management support during project 

startup.

We appreciate the support and guidance of David Lee and Chris Early of the U.S. Department of Energy 

(DOE), and former DOE personnel (recently retired) including Ed Pollock, Terry Logee, and Lew 

Pratsch. 
iv

Nomenclature

Italicized terms are defined in Appendix A.

air_seal air seal retrofit

Approx. approximate

APR April

attic_ins. attic insulation retrofit

Avg. average

BESTEST Building Energy Simulation Test

BESTEST-EX Building Energy Simulation Test for Existing Homes

combined combined air seal, insulation, low-emissivity window and programmable 

thermostat retrofit (as used in Figures 1 to 14 and Tables 1 to 14)

Conf. value determined from confidence interval equations 

COP coefficient of performance

C1C targeted high space cooling consumption scenario

C2C targeted low space cooling consumption scenario

C3C fully random, near-nominal space cooling consumption scenario

C4C fully random, high space cooling consumption scenario 

C5C fully random, low space cooling consumption scenario

C6C fully random, mid-high space cooling consumption scenario

C7C fully random, mid-low space cooling consumption scenario

C1H targeted high space heating consumption scenario

C2H targeted low space heating consumption scenario

C3H fully random, near-nominal space heating consumption scenario

C4H fully random, high space heating consumption scenario

C5H fully random, low space heating consumption scenario

C6H fully random, mid-high space heating consumption scenario

C7H fully random, mid-low space heating consumption scenario

DOE U.S. Department of Energy

DOE-2.1E DOE-2.1E version JJ Hirsch PC 2.1En136

Econ. economic 

EnergyPlus EnergyPlus version 3.1

MAR March

Max maximum

Min minimum

Nom nominal input

NOV November

NREL National Renewable Energy Laboratory

OCT October

Ref reference simulation result

sol_abs solar absorptance (as used in Figures 1 to 14 and Tables 1 to 14)

SEP September

setback thermostat setback (as used in Figures 1 to 14 and Tables 1 to 14)

Sqrt square root

Stdv standard deviation (“N−1” [sample] type)

SUNREL SUNREL version 1.14

wall_ins. wall insulation retrofit

windows low-emissivity window retrofit (as used in Figures 1 to 14 and Tables 1 to 14)

“-C” calibration test cases

“-P” building physics test cases

$ economic threshold criteria 
v

Contents

Acknowledgments .............................................................................................. iii

Nomenclature .................................................................................................... iv

Accompanying Files ........................................................................................... v

Introduction ...................................................................................................... vii

1 Instructions for Implementing Physics Tests and Calibration Tests ........................... 1

2 Calibration Test Reference Results and Example Acceptance Criteria ......................... 2

3 Randomly Selected Explicit Inputs ................................................................ 14

References ....................................................................................................... 17

Appendix A Definitions ...................................................................................... 18

Appendix B Clarifications to Existing Documents ...................................................... 19

Accompanying Files (Electronic Media Contents)

The following files, provided within B-EX-Phase-1-Ref-C-Results+Example-acceptance-criteria.zip,

apply as they are called out in this document (see README-BESTEST-EX, included with the 

accompanying electronic files): 

• B-EX-Calibration-BaseCase-Utility-Data.xls: Electronic version of utility energy consumption 

data for use with the calibration test cases, presented in Judkoff et al. (2010a), Section 1.3.1.2.

• B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: Spreadsheet that contains 

reference simulation results for the utility bill calibration tests; the building physics test case 

results are also included here for convenience. The example acceptance criteria presented in 

Judkoff et al. (2010b) are applied to the physics and calibration tests results. Use BESTEST-EXPhase-1-Output.xls (Judkoff et al. 2010a) to enter simulation results for the program being tested.

The following reference simulation input files are provided for informative use: 

The subfolder B-EX-Ref-Simulation-Calibration-Input-Files contains reference simulation input files 

developed by NREL for the BESTEST-EX calibration (“-C”) test cases. Simulation input files are 

organized in subfolders by calibration scenario. 

Subfolder Description

\CnC Cooling tests input files (C1C through C7C)

 \CnH Heating tests input files (C1H through C7H)

 \B-EX-Ref-Simulation-Weather-Files Weather data files 

Reference simulation input files are described further within README-BESTEST-EX-Calibration-TestFiles.doc, included with the accompanying electronic files. Reference simulation input files for the 

building physics tests are included with Judkoff et al. (2010b).
vi

Figures

Figure 1. Calibration tests, scenario C1H: reference simulation results and acceptance criteria 3 

Figure 2. Calibration tests, scenario C2H: reference simulation results and acceptance criteria 3

Figure 3. Calibration tests, scenario C3H: reference simulation results and acceptance criteria 4

Figure 4. Calibration tests, scenario C4H: reference simulation results and acceptance criteria 4

Figure 5. Calibration tests, scenario C5H: reference simulation results and acceptance criteria 5

Figure 6. Calibration tests, scenario C6H: reference simulation results and acceptance criteria 5

Figure 7. Calibration tests, scenario C7H: reference simulation results and acceptance criteria 6 

Figure 8. Calibration tests, scenario C1C: reference simulation results and acceptance criteria 6

Figure 9. Calibration tests, scenario C2C: reference simulation results and acceptance criteria 7

Figure 10. Calibration tests, scenario C3C: reference simulation results and acceptance criteria 7 

Figure 11. Calibration tests, scenario C4C: reference simulation results and acceptance criteria 8

Figure 12. Calibration tests, scenario C5C: reference simulation results and acceptance criteria 8

Figure 13. Calibration tests, scenario C6C: reference simulation results and acceptance criteria 9

Figure 14. Calibration tests, scenario C7C: reference simulation results and acceptance criteria 9

Tables

Table 1. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C1H 10

Table 2. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C2H 10

Table 3. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C3H 10

Table 4. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C4H 10

Table 5. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C5H 11

Table 6. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C6H 11

Table 7. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C7H 11

Table 8. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C1C 11

Table 9. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C2C 12

Table 10. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C3C 12

Table 11. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C4C 12

Table 12. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C5C 13

Table 13. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C6C 13

Table 14. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C7C 13

Table 15. Explicit Inputs for Utility Bill Calibration Tests with Space Heating 15

Table 16. Explicit Inputs for Utility Bill Calibration Tests with Space Cooling 16
vii

Introduction

BESTEST-EX was initially published (Judkoff et al., 2010a, 2010b) with the intention that the calibration 

test results needed to be blind, so a certifying agency could use it for certifying calibration methods. After 

publication, it became apparent that the residential energy retrofit modeling industry wanted to have the 

ability to self-test their software, and we realized that this ability– consistent with self-testing capability 

provided for the BESTEST-EX building physics tests – would be beneficial to both software developers 

and users. Therefore, this document was developed to include previously withheld calibration test results, 

along with instructions for coordinating the use of this document with the previously published 

BESTEST-EX documents. Should the need arise in the future for a blind certification test, additional 

calibration test data would have to be generated for that purpose. 

This document provides:

• Instructions for implementing/running the BESTEST-EX physics and calibration tests (Judkoff et 

al. 2010a); see Section 1 of this document. 

• Electronic version of calibration utility data given in the test specification (Judkoff et al. 2010a, 

Sec 1.3.1.2.); see accompanying files. 

• Calibration test reference program results and example acceptance ranges; see Section 2. A 

certifying or accrediting agency may develop acceptance-range setting criteria to suit particular 

needs. Neither DOE, NREL, nor the authors of this document may be held responsible for any 

misfortunes that occur from use of these example acceptance ranges in a certification program.

• Randomly selected explicit inputs for the calibration tests; see Section 3. 

• Reference simulation input files for the calibration tests; see accompanying files. 

• Clarifications to Judkoff et al. (2010a, 2010b) related to publication of calibration test results for 

non-blind testing, and other clarifications; see Appendix B of this document. There are no 

substantive changes to the test specification.
1

1 Instructions for Implementing Physics Tests and Calibration 

Tests

The following documents are needed to implement the BESTEST-EX physics and calibration tests:

• Building Energy Simulation Test for Existing Homes (BESTEST-EX), Phase 1 Test Procedure: 

Building Thermal Fabric Test Cases, NREL/TP-550-47427. (Judkoff et al. 2010a)

• Example Procedures for Developing Acceptance-Range Criteria, NREL/TP-550-47502. (Judkoff 

et al. 2010b)

• This document. 

Clarifications to Judkoff et al. (2010a, 2010b) related to publication of calibration test results for nonblind testing, and other clarifications are included in Appendix B of this document; there are no 

substantive changes to the test specification. For definitions of terms used here see Appendix A of this 

document. 

Physics Tests

The physics test cases are to be implemented as described in the test specification (Judkoff et al. 2010a) 

and compared with results and example acceptance criteria included in Judkoff et al. (2010b). For 

convenience an electronic version of the physics test results is included in B-EX-Phase-1-Ref-CResults+Example-Acceptance-Criteria.xls accompanying this document.

Calibration Tests

The calibration test cases are to be implemented as described in the test specification (Judkoff et al. 

2010a). For convenience, an electronic version of the utility data specified in Judkoff et al. (2010a), 

Section 1.3.1.2 is included in B-EX-Calibration-BaseCase-Utility-Data.xls accompanying this document. 

Base-case and energy savings results for the tested program can be compared with results and example 

acceptance criteria included in Section 2. Calibrated base-case inputs for the tested program can be

compared with the randomly selected base-case explicit inputs used to develop the reference results; the 

reference program base-case explicit inputs are tabulated in Section 3. The Section 3 explicit inputs are 

implemented in the reference programs as shown in the simulation input files included with the 

accompanying electronic media.

Publication of calibration test results allows self-assessment of automated, semi-automated, and manual 

calibration methods. 

Note to certifying entities: The calibration test results published here are best applied for testing 

automated calibration methods directly integrated with the tested software. For testing semi-automated 

and manual calibration methods it is better to apply a separate set of results for blind third party testing. 

Results for additional calibration scenarios needed for blind third party testing have not been developed.
2

 

2 Calibration Test Reference Results and Example Acceptance 

Criteria

The concepts applied for establishing example acceptance ranges are described in Sections 1, 3, and 4 of 

Judkoff et al. (2010b). As described in Judkoff et al. (2010b), Section 2, programs must pass 100% of the 

physics test cases, and programs must pass a reasonable fraction (example: 80%) of the calibration test 

cases. 

An example of applying this procedure to the BESTEST-EX reference results for the utility bill 

calibration (“-C”) cases follows. Reference results were developed using:

• DOE-2.1E version JJ Hirsch PC 2.1En136 (DOE-2 Reference Manual 1981, DOE-2 Supplement 

1994)

• EnergyPlus version 3.1 (EnergyPlus Input Output Reference 2009)

• SUNREL version 1.14 (Deru et al. 2002)

In Figures 1 through 14, the example acceptance range maxima and minima are indicated by “range” bars. 

The statistically based acceptance ranges are shown with blue range bars; the economic threshold based 

ranges are shown with green range bars. A tested tool passes a case if its result for that case falls within 

the greatest maximum and least minimum defined by the blue and green range bars. 

The example acceptance ranges for the BESTEST-EX “-C” cases are developed as shown in Tables 1 

through 14. An electronic version of the calculations is provided with B-EX-Phase-1-Ref-CResults+Example-Acceptance-Criteria.xls included with the accompanying electronic files. Cell 

addresses for finding data in the xls file are given in small font below the tables. 

Abbreviations used in the figures and tables are included are with the Nomenclature (see the front matter 

of this document).
3

Figure 1. Calibration tests, scenario C1H: reference simulation results and acceptance criteria

Figure 2. Calibration tests, scenario C2H: reference simulation results and acceptance criteria

0

20

40

60

80

100

120

140

160

180

200

L200EX-C1H

base-case

L200 -

L210EXC1H

air_seal

L200 -

L220EXC1H

attic_ins.

L200 -

L225EXC1H

wall_ins.

L200 -

L240EXC1H

setback

L200 -

L250EXC1H

windows

L200 -

L300EXC1H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C1H - Targeted High Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

0

25

50

75

100

125

L200EX-C2H

base-case

L200 -

L210EXC2H

air_seal

L200 -

L220EXC2H

attic_ins.

L200 -

L225EXC2H

wall_ins.

L200 -

L240EXC2H

setback

L200 -

L250EXC2H

windows

L200 -

L300EXC2H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C2H - Targeted Low Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
4

Figure 3. Calibration tests, scenario C3H: reference simulation results and acceptance criteria

Figure 4. Calibration tests, scenario C4H: reference simulation results and acceptance criteria

0

25

50

75

100

125

150

175

L200EX-C3H

base-case

L200 -

L210EXC3H

air_seal

L200 -

L220EXC3H

attic_ins.

L200 -

L225EXC3H

wall_ins.

L200 -

L240EXC3H

setback

L200 -

L250EXC3H

windows

L200 -

L300EXC3H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C3H - Fully Random, Near-Nominal Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

0

20

40

60

80

100

120

140

160

180

200

L200EX-C4H

base-case

L200 -

L210EXC4H

air_seal

L200 -

L220EXC4H

attic_ins.

L200 -

L225EXC4H

wall_ins.

L200 -

L240EXC4H

setback

L200 -

L250EXC4H

windows

L200 -

L300EXC4H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C4H - Fully Random, High Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
5

Figure 5. Calibration tests, scenario C5H: reference simulation results and acceptance criteria

Figure 6. Calibration tests, scenario C6H: reference simulation results and acceptance criteria

0

20

40

60

80

100

120

L200EX-C5H

base-case

L200 -

L210EXC5H

air_seal

L200 -

L220EXC5H

attic_ins.

L200 -

L225EXC5H

wall_ins.

L200 -

L240EXC5H

setback

L200 -

L250EXC5H

windows

L200 -

L300EXC5H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C5H - Fully Random, Low Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

0

25

50

75

100

125

150

175

L200EX-C6H

base-case

L200 -

L210EXC6H

air_seal

L200 -

L220EXC6H

attic_ins.

L200 -

L225EXC6H

wall_ins.

L200 -

L240EXC6H

setback

L200 -

L250EXC6H

windows

L200 -

L300EXC6H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C6H - Fully Random, Mid-High Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
6

Figure 7. Calibration tests, scenario C7H: reference simulation results and acceptance criteria

Figure 8. Calibration tests, scenario C1C: reference simulation results and acceptance criteria

0

25

50

75

100

125

L200EX-C7H

base-case

L200 -

L210EXC7H

air_seal

L200 -

L220EXC7H

attic_ins.

L200 -

L225EXC7H

wall_ins.

L200 -

L240EXC7H

setback

L200 -

L250EXC7H

windows

L200 -

L300EXC7H

combined

Gas Use or Savings (million Btu)

Annual Gas Use or Savings 

C7H - Fully Random, Mid-Low Space Heating Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

-1000

1000

3000

5000

7000

9000

11000

13000

15000

17000

19000

L200EX-C1C

base-case

L200 -

L210EXC1C

air_seal

L200 -

L220EXC1C

attic_ins.

L200 -

L225EXC1C

wall_ins.

L200 -

L240EXC1C

setback

L200 -

L250EXC1C

windows

L200 -

L265EXC1C

sol_abs

L200 -

L300EXC1C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings 

C1C - Targeted High Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
7

Figure 9. Calibration tests, scenario C2C: reference simulation results and acceptance criteria

Figure 10. Calibration tests, scenario C3C: reference simulation results and acceptance criteria

-1000

1000

3000

5000

7000

9000

11000

13000

L200EX-C2C

base-case

L200 -

L210EXC2C

air_seal

L200 -

L220EXC2C

attic_ins.

L200 -

L225EXC2C

wall_ins.

L200 -

L240EXC2C

setback

L200 -

L250EXC2C

windows

L200 -

L265EXC2C

sol_abs

L200 -

L300EXC2C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings

C2C - Targeted Low Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

-1000

1000

3000

5000

7000

9000

11000

13000

15000

L200EX-C3C

base-case

L200 -

L210EXC3C

air_seal

L200 -

L220EXC3C

attic_ins.

L200 -

L225EXC3C

wall_ins.

L200 -

L240EXC3C

setback

L200 -

L250EXC3C

windows

L200 -

L265EXC3C

sol_abs

L200 -

L300EXC3C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings 

C3C - Fully Random, Near-Nominal Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
8

Figure 11. Calibration tests, scenario C4C: reference simulation results and acceptance criteria

Figure 12. Calibration tests, scenario C5C: reference simulation results and acceptance criteria

-1000

1000

3000

5000

7000

9000

11000

13000

15000

17000

19000

L200EX-C4C

base-case

L200 -

L210EXC4C

air_seal

L200 -

L220EXC4C

attic_ins.

L200 -

L225EXC4C

wall_ins.

L200 -

L240EXC4C

setback

L200 -

L250EXC4C

windows

L200 -

L265EXC4C

sol_abs

L200 -

L300EXC4C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings 

C4C - Fully Random, High Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

-1000

1000

3000

5000

7000

9000

11000

L200EX-C5C

base-case

L200 -

L210EXC5C

air_seal

L200 -

L220EXC5C

attic_ins.

L200 -

L225EXC5H

wall_ins.

L200 -

L240EXC5C

setback

L200 -

L250EXC5C

windows

L200 -

L265EXC5C

sol_abs

L200 -

L300EXC5C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings 

C5C - Fully Random, Low Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
9

Figure 13. Calibration tests, scenario C6C: reference simulation results and acceptance criteria

Figure 14. Calibration tests, scenario C7C: reference simulation results and acceptance criteria

-1000

1000

3000

5000

7000

9000

11000

13000

15000

17000

19000

L200EX-C6C

base-case

L200 -

L210EXC6C

air_seal

L200 -

L220EXC6C

attic_ins.

L200 -

L225EXC6C

wall_ins.

L200 -

L240EXC6C

setback

L200 -

L250EXC6C

windows

L200 -

L265EXC6C

sol_abs

L200 -

L300EXC6C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings

C6C - Fully Random, Mid-High Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E

-1000

1000

3000

5000

7000

9000

11000

13000

15000

17000

L200EX-C7C

base-case

L200 -

L210EXC7C

air_seal

L200 -

L220EXC7C

attic_ins.

L200 -

L225EXC7C

wall_ins.

L200 -

L240EXC7C

setback

L200 -

L250EXC7C

windows

L200 -

L265EXC7C

sol_abs

L200 -

L300EXC7C

combined

Electricity Use or Savings (kWh)

Annual Electricity Use or Savings

C7C - Fully Random, Mid-Low Space Cooling Consumption Base Case

EnergyPlus SUNREL DOE-2.1E
10

Table 1. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C1H

Table 2. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C2H 

Table 3. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C3H 

Table 4. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C4H 

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C1H base-case 149.47 168.00 149.01 155.49 182.40 128.58 173.72 143.29 182.40 128.58

L200 - L210EXC1H air_seal 20.28 18.93 18.15 19.12 20.94 16.45 24.16 12.43 24.16 12.43

L200 - L220EXC1H attic_ins. 30.82 33.27 30.47 31.52 34.09 27.74 37.15 24.75 37.15 24.75

L200 - L225EXC1H wall_ins. 21.99 29.27 21.18 24.14 31.65 13.08 33.15 15.45 33.15 13.08

L200 - L240EXC1H setback 14.07 14.74 13.51 14.11 15.15 12.57 18.63 7.78 18.63 7.78

L200 - L250EXC1H windows 12.29 20.28 10.83 14.47 23.04 1.83 24.16 5.11 24.16 1.83

L200 - L300EXC1H combined 91.77 105.72 89.98 95.82 110.35 74.42 109.60 84.26 110.35 74.42

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! AB266:AL277 7/13/2011

Conf. Bounds

Statisical Econ. Threshold Acceptance

Bounds Range Bounds

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C2H base-case 91.28 102.34 91.54 95.06 110.74 79.38 108.07 85.56 110.74 79.38

L200 - L210EXC2H air_seal 12.96 12.16 11.54 12.22 13.41 10.46 16.84 5.82 16.84 5.82

L200 - L220EXC2H attic_ins. 9.03 9.94 9.03 9.33 10.22 8.03 13.83 3.31 13.83 3.31

L200 - L225EXC2H wall_ins. 14.19 18.38 14.06 15.54 19.69 9.44 22.26 8.33 22.26 8.33

L200 - L240EXC2H setback 7.82 8.10 7.54 7.82 8.29 7.12 11.98 1.81 11.98 1.81

L200 - L250EXC2H windows 8.44 13.40 7.92 9.92 15.02 2.41 17.28 2.20 17.28 2.20

L200 - L300EXC2H combined 48.02 56.35 47.35 50.57 59.03 38.11 60.23 41.62 60.23 38.11

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! AS266:BC277 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C3H base-case 122.34 137.85 123.06 127.75 149.50 106.00 143.57 116.61 149.50 106.00

L200 - L210EXC3H air_seal 18.36 16.85 16.41 17.21 18.93 14.67 22.24 10.69 22.24 10.69

L200 - L220EXC3H attic_ins. 16.74 18.57 16.99 17.43 19.10 14.97 22.45 11.02 22.45 11.02

L200 - L225EXC3H wall_ins. 15.77 20.89 15.51 17.39 22.50 9.86 24.77 9.79 24.77 9.79

L200 - L240EXC3H setback 10.23 10.57 9.85 10.22 10.83 9.32 14.45 4.13 14.45 4.13

L200 - L250EXC3H windows 11.79 18.47 10.74 13.66 20.74 3.25 22.35 5.01 22.35 3.25

L200 - L300EXC3H combined 67.38 78.22 66.34 70.65 81.74 54.30 82.11 60.61 82.11 54.30

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! BJ266:BT277 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C4H base-case 152.72 171.44 152.55 158.90 185.87 131.93 177.16 146.83 185.87 131.93

L200 - L210EXC4H air_seal 19.49 18.20 17.46 18.39 20.12 15.83 23.38 11.74 23.38 11.74

L200 - L220EXC4H attic_ins. 28.62 31.07 28.33 29.34 31.88 25.60 34.95 22.61 34.95 22.61

L200 - L225EXC4H wall_ins. 22.08 29.53 21.19 24.26 31.98 12.88 33.41 15.46 33.41 12.88

L200 - L240EXC4H setback 13.44 14.19 12.90 13.51 14.60 11.91 18.07 7.18 18.07 7.18

L200 - L250EXC4H windows 11.95 19.60 10.47 14.01 22.27 1.83 23.48 4.74 23.48 1.83

L200 - L300EXC4H combined 87.99 102.05 86.30 92.11 106.69 70.64 105.93 80.57 106.69 70.64

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! CA266:CK277 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds
11

Table 5. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C5H 

Table 6. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C6H 

Table 7. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C7H 

Table 8. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C1C 

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C5H base-case 87.80 98.59 88.69 91.69 106.57 76.82 104.31 82.08 106.57 76.82

L200 - L210EXC5H air_seal 12.27 11.85 10.97 11.69 12.82 10.04 16.15 5.24 16.15 5.24

L200 - L220EXC5H attic_ins. 10.38 11.24 10.38 10.67 11.51 9.43 15.13 4.66 15.13 4.66

L200 - L225EXC5H wall_ins. 12.93 16.54 12.99 14.15 17.64 9.01 20.43 7.21 20.43 7.21

L200 - L240EXC5H setback 8.11 8.62 8.12 8.28 8.77 7.56 12.50 2.39 12.50 2.39

L200 - L250EXC5H windows 7.76 12.76 7.50 9.34 14.34 1.97 16.64 1.77 16.64 1.77

L200 - L300EXC5H combined 46.31 54.45 46.24 49.00 56.95 37.27 58.33 40.51 58.33 37.27

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! CR266:DB277 7/13/2011

Statisical

Conf. Bounds Bounds Range Bounds

Econ. Threshold Acceptance

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C6H base-case 129.78 146.57 129.86 135.40 159.42 111.38 152.29 124.05 159.42 111.38

L200 - L210EXC6H air_seal 20.31 18.68 18.16 19.05 20.93 16.27 24.19 12.44 24.19 12.44

L200 - L220EXC6H attic_ins. 13.89 15.41 14.01 14.44 15.86 12.34 19.30 8.17 19.30 8.17

L200 - L225EXC6H wall_ins. 21.30 27.71 20.88 23.30 29.76 13.78 31.60 15.16 31.60 13.78

L200 - L240EXC6H setback 11.57 12.00 11.13 11.56 12.30 10.48 15.88 5.40 15.88 5.40

L200 - L250EXC6H windows 12.94 20.26 11.76 14.99 22.75 3.55 24.15 6.04 24.15 3.55

L200 - L300EXC6H combined 73.81 86.09 72.32 77.41 90.15 58.63 89.98 66.60 90.15 58.63

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! DI266:DS277 7/13/2011

Econ. Threshold Acceptance

Bounds Range Bounds

Statisical

Conf. Bounds

Total Annual Gas Consumption and Savings (million Btu/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C7H base-case 99.42 112.22 99.88 103.84 121.87 85.80 117.94 93.69 121.87 85.80

L200 - L210EXC7H air_seal 15.56 14.70 13.92 14.72 16.11 12.69 19.44 8.19 19.44 8.19

L200 - L220EXC7H attic_ins. 9.77 10.74 9.82 10.11 11.03 8.76 14.62 4.05 14.62 4.05

L200 - L225EXC7H wall_ins. 17.25 22.14 17.20 18.86 23.65 11.80 26.03 11.47 26.03 11.47

L200 - L240EXC7H setback 9.76 10.24 9.65 9.88 10.42 9.10 14.13 3.93 14.13 3.93

L200 - L250EXC7H windows 10.10 15.94 9.48 11.84 17.85 2.99 19.82 3.76 19.82 2.99

L200 - L300EXC7H combined 56.44 66.22 55.95 59.53 69.30 45.14 70.10 50.23 70.10 45.14

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: GasHtgData! DZ266:EJ277 7/13/2011

Acceptance

Conf. Bounds Bounds Range Bounds

Statisical Econ. Threshold

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C1C base-case 14989 16649 14952 15530 17938 13123 17270 14331 17938 13123

L200 - L210EXC1C air_seal 168 121 188 159 216 74 609 -499 609 -499

L200 - L220EXC1C attic_ins. 421 677 476 525 752 190 1098 -200 1098 -200

L200 - L225EXC1C wall_ins. 474 734 234 480 902 -141 1155 -387 1155 -387

L200 - L240EXC1C setback 804 907 833 848 938 716 1328 183 1328 183

L200 - L250EXC1C windows 1536 2163 1442 1714 2375 739 2584 821 2584 739

L200 - L265EXC1C sol_abs 604 432 426 487 658 236 1025 >0 1025 >0

L200 - L300EXC1C combined 3140 4128 2962 3410 4469 1849 4550 2341 4550 1849

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! AF276:AP289 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds
12

Table 9. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C2C

Table 10. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C3C

Table 11. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C4C

Table 12. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C5C

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C2C base-case 9257 10397 9135 9596 11326 7866 11018 8514 11326 7866

L200 - L210EXC2C air_seal 124 92 141 119 161 57 563 -528 563 -528

L200 - L220EXC2C attic_ins. 476 621 464 520 667 303 1042 -157 1042 -157

L200 - L225EXC2C wall_ins. 542 708 334 528 843 63 1129 -287 1129 -287

L200 - L240EXC2C setback 646 734 669 683 759 570 1155 25 1155 25

L200 - L250EXC2C windows 1263 1773 1185 1407 1946 613 2194 564 2194 564

L200 - L265EXC2C sol_abs 687 508 492 562 745 293 1109 >0 1109 >0

L200 - L300EXC2C combined 2919 3601 2698 3073 3866 1904 4022 2077 4022 1904

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! AW276:BG289 7/13/2011

Conf. Bounds Bounds Range Bounds

Statisical Econ. Threshold Acceptance

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C3C base-case 11969 13184 11892 12348 14150 10547 13805 11271 14150 10547

L200 - L210EXC3C air_seal 145 109 159 138 181 73 580 -512 580 -512

L200 - L220EXC3C attic_ins. 526 736 538 600 799 307 1157 -94 1157 -94

L200 - L225EXC3C wall_ins. 422 608 239 423 734 -36 1029 -382 1029 -382

L200 - L240EXC3C setback 602 673 616 630 694 536 1095 >0 1095 >0

L200 - L250EXC3C windows 1141 1622 1077 1280 1782 540 2043 457 2043 457

L200 - L265EXC3C sol_abs 667 476 464 536 727 253 1088 >0 1088 >0

L200 - L300EXC3C combined 2696 3432 2518 2882 3699 1678 3853 1897 3853 1678

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! BN276:BX289 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C4C base-case 14455 16075 14407 14979 17338 12620 16696 13786 17338 12620

L200 - L210EXC4C air_seal 160 116 178 151 206 71 600 -505 600 -505

L200 - L220EXC4C attic_ins. 588 883 632 701 969 305 1304 -33 1304 -33

L200 - L225EXC4C wall_ins. 506 754 259 506 923 -108 1175 -362 1175 -362

L200 - L240EXC4C setback 819 927 851 866 959 728 1348 198 1348 198

L200 - L250EXC4C windows 1492 2108 1409 1670 2313 722 2529 789 2529 722

L200 - L265EXC4C sol_abs 773 551 543 622 842 299 1194 >0 1194 >0

L200 - L300EXC4C combined 3301 4288 3115 3568 4631 2002 4709 2494 4709 2002

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! CE276:CO289 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C5C base-case 8770 9820 8787 9126 10620 7631 10441 8149 10620 7631

L200 - L210EXC5C air_seal 61 39 76 59 89 13 497 -581 497 -581

L200 - L220EXC5C attic_ins. 200 323 225 249 359 88 744 -421 744 -421

L200 - L225EXC5H wall_ins. 276 420 133 276 518 -80 841 -488 841 -488

L200 - L240EXC5C setback 534 640 573 582 672 450 1061 >0 1061 >0

L200 - L250EXC5C windows 1071 1505 1021 1199 1647 539 1926 401 1926 401

L200 - L265EXC5C sol_abs 346 258 254 286 374 156 768 >0 768 >0

L200 - L300EXC5C combined 1943 2575 1863 2127 2785 1158 2997 1242 2997 1158

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! CV276:DF289 7/13/2011

Range Bounds

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds
13

Table 13. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C6C

Table 14. BESTEST-EX Example Range-Setting Procedure: Calibration Tests, Scenario C7C

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C6C base-case 15004 16395 14971 15457 17476 13438 17016 14351 17476 13438

L200 - L210EXC6C air_seal 112 80 132 108 152 43 553 -541 553 -541

L200 - L220EXC6C attic_ins. 415 599 435 483 653 232 1020 -206 1020 -206

L200 - L225EXC6C wall_ins. 405 584 216 402 712 -57 1006 -405 1006 -405

L200 - L240EXC6C setback 714 817 752 761 849 632 1238 94 1238 94

L200 - L250EXC6C windows 1378 1949 1312 1546 2136 677 2370 691 2370 677

L200 - L265EXC6C sol_abs 649 473 464 529 704 270 1070 >0 1070 >0

L200 - L300EXC6C combined 2836 3642 2681 3053 3923 1771 4063 2060 4063 1771

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! DM276:DW289 7/13/2011

Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds

Statisical

Total Annual Electricity Consumption and Savings (kWh/y)

Ref

Case EnergyPlus SUNREL DOE-2.1E Mean Max Min Max Min Max Min

L200EX-C7C base-case 14759 15840 14674 15091 16705 13477 16460 14054 16705 13477

L200 - L210EXC7C air_seal 43 27 61 44 72 2 482 -593 482 -593

L200 - L220EXC7C attic_ins. 250 349 256 285 379 146 771 -371 771 -371

L200 - L225EXC7C wall_ins. 370 512 178 353 636 -63 933 -442 933 -442

L200 - L240EXC7C setback 579 671 603 618 698 499 1092 >0 1092 >0

L200 - L250EXC7C windows 1063 1517 1001 1194 1669 493 1939 380 1939 380

L200 - L265EXC7C sol_abs 471 355 343 389 509 214 892 >0 892 >0

L200 - L300EXC7C combined 2135 2746 1965 2282 2975 1261 3167 1344 3167 1261

 B-EX-Phase-1-Ref-C-Results+Example-Acceptance-Criteria.xls: ElecClgData! ED276:EN289 7/13/2011

Statisical Econ. Threshold Acceptance

Conf. Bounds Bounds Range Bounds
14

3 Randomly Selected Explicit Inputs

Explicit inputs used in the reference simulations are provided in Tables 3-1 and 3-2 for the space heating 

and space cooling cases, respectively. In these tables the explicit inputs for each calibration scenario are 

provided in the column headed by the name of the given calibration scenario (e.g., “C1H”). The 

approximate input range from which the explicit inputs were randomly selected is described in the group 

of columns headed by “Approx. Input Range”. Information about how the explicit inputs are applied in 

the test procedure and about how they are randomly selected from within the approximate input ranges is 

given in Judkoff et al. (2010a), Section 1 and Appendix F, respectively. Abbreviations used in both tables 

are included with the Nomenclature (see the front matter of this document). 

In Tables 3-1 and 3-2 related groups of inputs are separated by solid lines; dashed/lighter lines separate 

relevant inputs within a group. Italicized values in Tables 3-1 and 3-2 are equivalent or supplemental 

approximate input range minimum, maximum and nominal values and explicit input values, which are

calculated based on primary approximate input range minimum, maximum and nominal values and 

randomly selected explicit input values (shown in the tables with normal font). For example, for each 

calibration scenario, siding thickness explicit inputs are provided that supplement the randomly selected 

exterior wall R-values, and these values are within the corresponding listed external siding approximate 

input range minimum and maximum values. An example of the relationship between the external siding 

thickness and the exterior wall R-value is shown in Judkoff et al. (2010a), Table 1-4.
15

Table 15. Explicit Inputs for Utility Bill Calibration Tests with Space Heating

HEATING

Approximate Input C1H C2H C3H C4H C5H C6H C7H Min Nom Max

Exterior Wall R-Value 

(h·ft2

·°F/Btu)

5.078 5.141 5.848 4.892 5.495 5.209 5.185 4.500 5.091 6.200

Siding Thickness (in.) c 0.4293 0.4686 0.9132 0.3129 0.6908 0.5111 0.4960 0.0685 0.4400 1.1355

Attic R-Value (h·ft2·°F/Btu) 8.511 15.489 12.125 8.699 14.056 15.308 16.933 7.100 13.673 19.300

Fiberglass Batt Thickness 

(in.) 1.6487 4.1563 2.9422 1.7155 3.6381 4.0909 4.6794 1.1490 3.5000 5.5450

Joist Thickness (in.) d 1.6487 4.1563 2.9422 1.7155 3.6381 4.0909 4.6794 1.1490 3.5000 5.5000

Effective Leakage Area at 

4 Pa (in 2 )

207.73 195.98 177.55 197.13 195.13 188.32 207.55 137.39 196.27 215.89

Equivalent CFM at 50 Pa (see Judkoff et al. [2010a], Table 1-8a)

Heating, Colo. Spr. (CFM) 4234 3994 3619 4018 3977 3838 4230 2800 4000 4400

Equivalent ACH (see Judkoff et al. [2010a], Table 1-8b) e

Heating, Colo. Spr. (ACH) 0.808 0.762 0.691 0.767 0.759 0.733 0.807 0.534 0.760 0.835

Occupant Sensible Loads 

(Btu/day) 7037 11375 5208 10365 8196 10945 12628 4347 8694 13041

Electric Sensible Loads 

(Btu/day) 40354 66143 34129 29957 60764 44102 61779 18234 36468 80000

Non-HVAC Electricity to 

Internal Gains (%) 64.36 80.57 81.38 85.36 73.66 66.16 71.93 60.00 75.00 90.00

Resulting Annual Non-HVAC 

Electricity Use (kWh/y) 6707 8782 4486 3754 8824 7131 9188 2167 5201 14263

Gas Sensible Loads 

(Btu/day) 12650 15358 17194 16810 19721 13702 15584 7464 14928 22392

Non-HVAC Gas Use to 

Internal Gains (%) 27.51 23.40 27.48 23.06 29.72 24.23 30.72 20.00 27.50 35.00

Resulting Annual Non-HVAC 

Gas Use (MBtu/y) 16.78 23.96 22.84 26.61 24.22 20.64 18.52 7.78 19.81 40.87

Exterior Solar Absorptance 0.576 0.592 0.716 0.598 0.520 0.608 0.560 0.500 0.600 0.800

Heating Set Point (°F) 69.02 65.44 69.79 69.31 62.95 70.06 66.89 60.00 68.00 75.00

Season (% of total annual 

space conditioning load) 95.54 91.41 93.13 95.97 95.19 94.92 96.60 90.00 95.00 99.00

Space Conditioning Seasons

 Start Month SEP OCT OCT SEP OCT OCT OCT OCT

 Start Day 24 29 13 22 25 7 9 7

 End Month MAY APR MAY MAY MAY MAY MAY MAY

 End Day 23 23 11 26 7 17 18 16

Furnace Efficiency (%) 64.51 77.08 69.92 68.27 75.52 65.40 74.22 60.00 70.00 80.00

a

 Categories of inputs are separated by solid lines; dashed/lighter lines separate relevant inputs within a group. 

 See Judkoff et al (2010a) for details about how the inputs are applied.

b

 Italicized values indicate equivalent range values and explicit inputs that were calculated from basis range 

 values and randomly selected values (shown with normal font).

c

 Wall R-value may be adjusted by varying the modeled siding thickness.

d

 Modeled joist thickness is the same as for insulation, but not greater than 5.5 inches.

e

 This input is for programs that do not use more detailed methods.

Approx. Input Rangea,b Explicit Inputs for the Calibration Scenariosa,b
16

Table 16. Explicit Inputs for Utility Bill Calibration Tests with Space Cooling

COOLING

Approximate Input C1C C2C C3C C4C C5C C6C C7C Min Nom Max

Exterior Wall R-Value 

(h·ft2

·°F/Btu)

5.056 5.136 5.224 4.969 5.126 5.611 4.980 4.500 5.091 6.200

Siding Thickness (in.) c 0.4157 0.4657 0.5209 0.3609 0.4595 0.7640 0.3681 0.0685 0.4400 1.1355

Attic R-Value (h·ft2

·°F/Btu) 13.294 14.500 11.076 10.840 16.828 14.346 18.363 7.100 13.673 19.300

Fiberglass Batt Thickness 

(in.) 3.3633 3.7985 2.5649 2.4805 4.6412 3.7430 5.1983 1.1490 3.5000 5.5450

Joist Thickness (in.) d 3.3633 3.7985 2.5649 2.4805 4.6412 3.7430 5.1983 1.1490 3.5000 5.5000

Effective Leakage Area at 

4 Pa (in 2 )

202.63 177.18 188.70 202.38 156.69 200.23 175.57 137.39 196.27 215.89

Equivalent CFM at 50 Pa (see Judkoff et al. [2010a], Table 1-8a)

 Cooling, Las Vegas (CFM) 3834 3353 3571 3830 2965 3789 3322 2600 3714 4085

Equivalent ACH (see Judkoff et al. [2010a], Table 1-8b) e

Cooling, Las Vegas (ACH) 0.509 0.445 0.474 0.508 0.393 0.503 0.441 0.345 0.492 0.540

Occupant Sensible Loads 

(Btu/day) 11557 6183 9803 11737 10158 7965 12353 4347 8694 13041

Electric Sensible Loads 

(Btu/day) 57952 28400 52158 50419 30702 56850 63840 18234 36468 80000

Non-HVAC Electricity to 

Internal Gains (%) 76.59 79.95 84.80 69.95 66.48 67.60 68.13 60.00 75.00 90.00

Resulting Annual Non-HVAC 

Electricity Use (kWh/y) 8094 3800 6579 7710 4940 8996 10024 2167 5201 14263

Gas Sensible Loads 

(Btu/day) 20877 20407 13574 19012 12866 15915 14577 7464 14928 22392

Non-HVAC Gas Use to 

Internal Gains (%) 32.63 23.72 27.42 27.38 29.73 24.92 26.49 20.00 27.50 35.00

Resulting Annual Non-HVAC 

Gas Use (MBtu/y) 23.35 31.40 18.07 25.34 15.80 23.31 20.09 7.78 19.81 40.87

Exterior Solar Absorptance 0.554 0.745 0.639 0.577 0.549 0.639 0.703 0.500 0.600 0.800

Cooling Set Point (°F) 77.11 78.44 77.20 77.07 81.88 79.84 82.63 71.00 78.00 86.00

Season (% of total annual 

space conditioning load) 95.35 94.13 91.60 95.97 97.52 94.20 93.43 90.00 95.00 99.00

Space Conditioning Seasons

 Start Month MAR MAR APR MAR MAR APR APR MAR

 Start Day 20 26 3 15 29 2 6 28

 End Month OCT OCT OCT NOV OCT OCT OCT OCT

 End Day 30 28 22 6 30 26 23 28

COP 2.662 3.093 3.348 2.786 3.259 2.622 3.019 2.500 3.000 3.500

a

 Categories of inputs are separated by solid lines; dashed/lighter lines separate relevant inputs within a group. 

 See Judkoff et al (2010a) for details about how the inputs are applied.

b

 Italicized values indicate equivalent range values and explicit inputs that were calculated from basis range 

 values and randomly selected values (shown with normal font).

c

 Wall R-value may be adjusted by varying the modeled siding thickness.

d

 Modeled joist thickness is the same as for insulation, but not greater than 5.5 inches.

e

 This input is for programs that do not use more detailed methods.

Approx. Input Rangea,b Explicit Inputs for the Calibration Scenariosa,b
17

References

Deru, M.; Judkoff, R.; Torcellini, P. (2002). SUNREL Technical Reference Manual. NREL/BK-550-

30193. Golden, CO: National Renewable Energy Laboratory.

DOE-2 Reference Manual (Version 2.1A) Part 1. (1981). D. York and C. Cappiello, eds. Berkeley, CA: 

Lawrence Berkeley Laboratory.

DOE-2 Supplement (Version 2.1E). (1994). Berkeley, CA: Lawrence Berkeley Laboratory.

EnergyPlus Input Output Reference. (2009). Urbana, IL: University of Illinois, and Berkeley, CA:

Lawrence Berkeley National Laboratory. http://apps1.eere.energy.gov/buildings/energyplus/pdfs/ 

InputOutputReference.pdf/. Last accessed January 2010.

Judkoff, R.; Neymark, J.; Polly, B.; Bianchi, M. (2010a). Building Energy Simulation Test for Existing 

Homes (BESTEST-EX) Phase 1 Test Procedure: Building Thermal Fabric Cases. NREL/TP-550-47427. 

Golden, CO: National Renewable Energy Laboratory. http://www.nrel.gov/docs/fy10osti/47427.pdf. 

Judkoff, R.; Neymark, J.; Polly, B.; Bianchi, M. (2010b). Example Procedures for Developing 

Acceptance-Range Criteria for BESTEST-EX. NREL/TP-550-47502. Golden, CO: National Renewable 

Energy Laboratory. http://www.nrel.gov/docs/fy10osti/47502.pdf. 
18

Appendix A. Definitions 

approximate input: an input for which an approximate input range has been defined; see listing in

Judkoff et al. [2010a], Section 1.3.1.2. Also see approximate input range.

approximate input range: the specified range of possible values for an approximate input that forms the 

basis uncertainty range for selecting calibrated approximate inputs for the tested programs (see Judkoff et 

al. [2010a], Section 1.3.1.2), and from which explicit inputs are randomly selected in accordance with the 

process described in Judkoff et al. (2010a), Appendix F. Also see calibrated input and explicit input.

calibrated input or calibrated approximate input: inputs for tested programs that are determined based 

on specified approximate input ranges and nominal input values using calibration to obtain agreement 

with base-case reference utility billing data. Also see approximate input range and nominal input.

explicit input: inputs for simulations used to develop reference utility billing data that are randomly 

selected from within specified approximate input ranges according to the process described in Judkoff et 

al. (2010a), Appendix F. Also see approximate input range.

fully random: Calibration scenario where explicit inputs are selected randomly from the entire

approximate input range (see Judkoff et al. [2010a], Appendix F). Also see approximate input range and 

explicit input.

nominal input: an input value as specified for the building physics base case (Case L200EX-P, see 

Judkoff et al. [2010a], Section 1.2.1).

targeted high: Calibration scenario where explicit inputs are selected randomly from the portion of the 

approximate input range (upper or lower) that leads to increased space conditioning consumption versus 

nominal input values (see Judkoff et al. [2010a], Appendix F). Also see approximate input range, explicit 

input and nominal input.

targeted low: Calibration scenario where explicit inputs are selected randomly from the portion of the 

approximate input range (upper or lower) that leads to decreased space conditioning consumption versus 

nominal input values (see Judkoff et al. [2010a], Appendix F). Also see approximate input range, explicit 

input, and nominal input.
19

Appendix B. Clarifications to Existing Documents

Clarifications to specific portions of text are indicated with underline/strikethrough. These clarifications

are related to publication of calibration test results for non-blind testing and other clarifications. There are 

no substantive changes to the test specification. The original documents (Judkoff et al. 2010a, 2010b) 

have not been updated.

Test Specification (NREL/TP-550-47427) – related to going from blind to non-blind testing

Introduction, p. xiii, subparagraph: “Calibrated energy savings test cases ....”

Calibrated energy savings test cases with specified base-case monthly utility bill data and 

uncertainty ranges for selected inputs: A given audit model (and associated calibration method) is 

tested by comparing utility-bill-calibrated energy savings predictions to results from the reference 

programs listed above. Reference results for the calibrated energy-savings tests are not published with 

the test procedure so that both automated and manual calibration methods are tested blind, without 

access to the reference results (answers). Practical application of this procedure requires that testedprogram results are compared to reference results by a third-party. The calibrated energy savings tests 

represent a new methodological development, further described under “Methodology” below.

1.1.9.2, p. 2, delete 2nd paragraph 

Reference results for the calibrated energy-savings tests are not published with the test procedure, so that 

both automated and manual calibration methods may be tested blind, without access to the reference 

results. Practical application of this procedure requires that tested-program results are compared to 

reference results by a third-party.

1.3.1.2, p. 48, last paragraph

Reference energy use data provided in Tables 1-24a through 1-24g and Tables 1-25a through 1-25g are 

the average of the results for the reference simulation models using EnergyPlus, SUNREL, and

DOE-2.1E. The reference simulations apply explicit inputs randomly selected from within the given 

approximate input ranges (see Appendix F). All reference simulation explicit inputs are selected 

independently for each space-heating and space-cooling base-case scenario, except heating thermostat 

settings/schedule and furnace efficiency are only selected for space heating cases, and cooling thermostat 

settings/schedule and cooling COP are only selected for space cooling cases. For the purpose of running 

the calibration tests, Tthe reference simulation explicit inputs are intended to be unknown for the software 

being tested and are not given in the test specification. The reference simulation explicit inputs are 

included with the calibration test results in a separate document [editor’s note: see Section 3 of this 

document]. To properly implement the test procedure, do not compare tested program calibrated inputs

with reference simulation explicit inputs until after modeling the calibration tests, i.e., run the test cases as 

if the reference simulation explicit inputs are unknown. Thirteen months of base-case energy use data are 

provided as recommended by the BESTEST-EX Working Group (2009). In Tables 1-24a through 1-24g 

and Tables 1-25a through 1-25g, gas use is for the furnace and DHW only; electricity use is for space 

cooling equipment and all other appliances (except DHW); and HVAC fan electricity is zero, as specified 

in Section 1.2.1.15. 
20

References, p. 64.

Judkoff, R.; Neymark, J.; Polly, B.; Bianchi, M. (2010). Example Procedures for Developing Acceptance 

Range Criteria for BESTEST-EX. NREL/TP-550-4750247542. Golden, CO: National Renewable Energy 

Laboratory.

Appendix F, p. 101, 3rd paragraph, delete last sentence. 

All explicit inputs and the calculated values for dependent inputs are not known by the participants 

testing software.

Appendix G, Section G.1, p. 104, last paragraph

Only the results for the “-P” test cases are shown in the figures and the tables. For the calibrated energy 

savings (“-C”) test cases, reference simulation results and randomly selected explicit inputs used in the 

reference simulations are intentionally not given for blind testing.

I.3.2, p. 123, delete 1st bullet

• Consider developing a version of the procedure for testing programs with automated calibration; 

such a version would have calibrated savings results available so that results would not have to 

be reviewed by a third party.

Physics Results and Example Acceptance Criteria (NREL/TP-550-47502) – related to going 

from blind to non-blind testing

Section 2, pp. 1-2 

Within BESTEST-EX the building physics (“-P”) cases are specified differently than the calibrated 

energy savings (“-C”) cases. The “-P” cases provide explicit inputs for all cases. The “-C” cases provide 

approximate input ranges for key inputs to account for uncertainty associated with audit information and 

measurements, occupant behavior, etc. For the “-C” cases, explicit inputs are randomly selected within 

the approximate input ranges to generate utility bills using the reference simulation programs; tested 

software tools are allowed to apply calibration given the reference utility billing data and approximate 

input ranges (selected explicit inputs used for the reference simulations remain hidden to allow for blind 

testing). Because the “-C” cases apply approximate input ranges (known uncertainty) for selected inputs, 

and because some base-case scenarios (see Judkoff et al. 2010, Section 1.3.1.2) can have randomly 

selected reference explicit inputs that are more difficult to estimate from calibration than others, the 

acceptance criteria for the “-C” cases should be less strict than that for the “-P” cases. Therefore, the 

following example acceptance criteria are provided:

• “-P” case acceptance

o Programs must pass all designated cases

 “-P” reference results are provided with the test procedure 

o Compare all energy savings case results

o Compare annual usage only for the base case (L200EX-P)

• “-C” case acceptance

o Programs must pass a reasonable fraction (example: 80%) of the designated cases

o “-C” energy savings reference results are not provided. with the test procedure. [Editor’s 

note: See Section 2 of this document.]
21

o Compare all energy savings case results only

 Base-case annual usage results are calibrated to reference bills.

Section 5, p. 6, last paragraph

Only tThe results and acceptance ranges for the building physics (“-P”) test cases are shown in the figures 

and the tables. For the calibrationed energy savings (“-C”) test cases, reference simulation results and 

randomly selected explicit inputs used in the reference simulations are intentionally not given for blind 

testing provided. [Editor’s note: See Sections 2 and 3 of this document.]

Test Specification (NREL/TP-550-47427) – related to BESTEST-EX Methodology

Introduction, pp. xiv, xv: 

The building physics test cases described in the preceding section are a direct application of software-tosoftware comparative test methods. The calibrated energy savings tests required NREL to make a 

methodological advancement to existing comparative test methods, as follows. 

1. Introduce input uncertainty into the test specification (this represents uncertainty 

associated with developing inputs from audit survey data):

a. Perform sensitivity tests on inputs with potentially high uncertainty to determine their 

relative effect on outputs; select the inputs that have the greatest effect on outputs as 

approximate inputs. 

b. Specify uncertainty ranges (approximate input ranges) for the each approximate inputs. 

2. Develop reference simulation results (this is done by the test developers): 

a. Generate base-case synthetic utility bill data using the same state-of-the-art reference 

simulation programs as used in the building physics test cases.

i. For the reference simulations, inputs that are randomly selected from within the 

specified approximate input ranges are designated as explicit inputs.; the 

reference simulation explicit inputs are not included in the test specification (kept 

secret)

ii. All reference simulations use the same or equivalent explicit inputs for a given 

calibration scenario. 

iii. The synthetic utility bill data are taken as the average of the reference simulation 

results.

b. Generate reference energy savings results by adjusting appropriate base-case inputs 

(including explicit inputs) as specified for each retrofit case.

3. Develop tested program results (this is done by the test takers): 

a. Develop the preliminary non-calibrated base-case model for a given calibration scenario.

b. Predict energy savings by either: via one of the following: 

i. Calibrateing the base-case model inputs using the synthetic utility bills (described 

in 2a above), and then applying the specified retrofit cases to the calibrated 

model., or 
22

ii. Applying the specified retrofit to the non-uncalibrated base-case model and then 

calibrateing or correcting energy savings predictions using the synthetic utility 

bills (without adjustment to base-case model inputs), or e.g., as (calibrated 

savings) = (predicted savings) × (base-case actual bills)/(base-case predicted 

bills).

iii. Other calibration methods. The test cases make no recommendation about how to 

perform calibrations.

4. Compare results of tested programs (and their calibration techniques) versus reference 

simulation base-case usage and retrofit energy savings projections:

a. Example acceptance criteria (Judkoff et al. 2010b) may be used to facilitate the 

comparison.

The conceptual framework for this method was first proposed by Judkoff (2008) with important 

refinements contributed by others (Neymark and Norton 2009; Neymark et al. 2009). Development of the 

method was facilitated by convening a technical committee of software producers (the “BESTEST-EX 

Working Group”) to provide help with quantifying estimate approximate input ranges and developing

tested program results (see Step 1b and Step 3, respectively, above). The test procedure and example 

acceptance range criteria was were developed in an iterative process that allowed improvement of the test 

specification during the simulation trials and helped simulation trial participants to improve their 

software. 

5. Performing Calibration Tests Without Using Reference Programs

In its purest form, the calibration test would be implemented without using the reference simulation 

programs. Instead, synthetic utility billing data would be generated with the tested program itself. Such a 

pure calibration test requires a) automated calibration or b) that the modeler running the calibration test 

does not know the explicit inputs used to develop the synthetic utility bills, implying that an additional 

modeler is needed, and there is a “firewall” between the two modelers. Either method is acceptable, but 

the latter is impractical for certifying entities.

One process for self-testing a calibration method applied for a given program is described below (this can 

be done in a number of ways, as long as the modeler does not know the explicit inputs used to generate

synthetic utility bills for any given case):

• Develop uncalibrated results

o Develop an uncalibrated base-case model using nominal input values listed in Judkoff et 

al. (2010a), Appendix F, Table F-1.

 Equivalent and supplemental nominal inputs related to these values are provided 

in the tables of Judkoff et al. (2010a), Section 1.2.1. [Editor’s note: Also see 

Section 3 of this document.]

o Apply specified inputs for retrofit measures to the uncalibrated base-case model to obtain 

uncalibrated energy savings results.

• Develop reference base-case utility billing data

o Randomly select explicit inputs from within the given approximate input ranges, as 

described in Judkoff et al. (2010a), Appendix F. Replace the nominal inputs in the 

uncalibrated base-case model with the randomly selected explicit inputs to create a 

reference base-case model for the calibration scenario.

 In the original work, the randomly selected space conditioning season (% of total 

annual space conditioning load) was converted to equivalent start and end dates
23

by A) running an initial full-year annual simulation using the randomly selected 

explicit inputs for a given scenario and then B) analyzing the annual output to 

determine the start and end dates that would evenly crop the season to the 

randomly selected percentage. For example, if a 95% heating season was 

chosen, the date at which 2.5% of the annual heating load was reached was used 

as the start date, while the date at which 97.5% of the annual heating load was 

reached was used as the end date. This process can be implemented in selftesting, or as a simplification, annual simulations can be conducted and the 

annual space conditioning results can be multiplied by the season percentage 

expressed as a fraction (e.g., 0.95 corresponds to 95% season).

o Repeat the above step to create multiple calibration scenarios (each having its own 

reference base-case model) such that randomly selected explicit inputs are difficult to 

recall without looking at the input files. That is, adhere to the principle that the person 

implementing the calibration tests does not know, or cannot recall, the explicit inputs 

associated with each test case (i.e., be self-blinded). 

 For programs with purely automated calibration methods, self-blinding is not 

necessary and the process of Appendix F may be followed as described.

o Run each reference base-case model and extract reference utility billing data from the 

output files for each scenario.

 Do not review input data that are included with the output.

• Develop reference energy savings results

o For each selected calibration scenario, develop reference energy savings results by 

applying specified inputs for the retrofit measures to the reference base-case models 

containing randomly selected explicit inputs.

• Develop calibrated base-case and energy savings results

o For each selected calibration scenario, develop calibrated base-case models beginning 

with nominal inputs (i.e., using the uncalibrated base-case input file), and calibrate that 

model (i.e., develop calibrated inputs) to the reference base-case utility billing data for 

the selected scenario.

o Apply specified inputs for retrofit measures to the calibrated base-case model to obtain 

calibrated energy savings results.

o The above two bullets demonstrate one of multiple ways that utility data may be used for 

calibration. Another possible use of utility data is to calibrate energy savings directly. For 

example, this may be accomplished by applying the specified retrofit to the uncalibrated 

base-case model and then revising energy savings predictions using the synthetic utility 

bills (without adjustment to the base-case model inputs). For example, as (calibrated 

savings) = (predicted savings) × (base-case actual bills)/(base-case predicted bills).

• For each calibration scenario compare calibrated model energy savings results and calibrated 

base-case inputs to reference energy savings results and reference base-case explicit inputs. 

o The benefit of calibration for the self-tested program may be estimated by comparing 

energy savings predictions of the calibrated models versus the uncalibrated models versus 

the reference results. This is analogous to comparing “WG-CAL”, “WG-UNCAL”, and 

“REF” in Judkoff et al. (2010a), Appendix G, Section G.2. 
24

o A software developer cannot apply the example acceptance criteria of Judkoff et al.

(2010b) for their new calibration scenarios for only one set of reference program results.