NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

Improving the Accuracy of Software-Based 

Energy Analysis for Residential Buildings

Ben Polly

The Energy & 

Environmental Building 

Alliance (EEBA) Conference

September 14, 2011

Las Vegas, Nevada

NREL/PR-5500-52745
 NATIONAL RENEWABLE ENERGY LABORATORY

Learning Objectives

1. Understand the basic components of software-based 

energy analysis for residential buildings

2. Explore the concepts of “error” and “accuracy” when 

analysis predictions are compared to measured data

3. Learn how NREL is working to continuously improve 

the accuracy of energy analysis methods

4. Calibrate expectations for the accuracy of residential 

energy analysis methods

2 
 NATIONAL RENEWABLE ENERGY LABORATORY

Presentation Outline

A. Motivation and Goals

B. Basics and Terminology

C. NREL Efforts To Improve Accuracy

1. Building America House Simulation Protocols

2. National Residential Efficiency Measures Database

3. Field and Laboratory Systems Test Support

4. BESTEST 

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis

D. Final Remarks

3 

Focus for this talk
 NATIONAL RENEWABLE ENERGY LABORATORY

Motivation and Goals

4 
 NATIONAL RENEWABLE ENERGY LABORATORY

Motivation and Goals

5 

Pre-Retrofit Energy

Consumption

Post-Retrofit Energy

Consumption

Energy Savings

Example Energy Savings

Over-prediction

Predicted

Measured

Y 

Y > Z

Z 
 NATIONAL RENEWABLE ENERGY LABORATORY

Motivation and Goals

Motivation:

• Stakeholders rely on accurate 

predictions

• Previous research indicates that 

analysis methods tend to 

overpredict energy use for poorly 

insulated leaky homes 

 (see Appendix A of report →) 

• Inaccuracy Risk

6 
 NATIONAL RENEWABLE ENERGY LABORATORY

Motivation and Goals

Goals of NREL Analysis Accuracy R&D:

• Provide industry with the tools and technical 

information needed to improve the accuracy and 

consistency of analysis methods

• Reduce the risks associated with purchasing, financing, 

and selling energy efficiency upgrades

• Enhance software and input collection methods

considering impacts on accuracy, cost, and time of 

energy assessments

7 
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

8 
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

– Analysis methods and tools are available at various “levels”

• Single component

• Whole-building

• Community

– Whole-building energy analysis is used in the residential sector 

for many purposes:

• Design energy-efficient homes

• Produce labels, scores, and ratings

• Predict energy and cost savings from energy efficiency upgrades

• Determine cost and performance criteria for new energy-efficiency 

technologies

• Provide quantitative analysis and data to support programmatic and 

policy-related decisions.

9 
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics & Terminology

10

Field Data Collection 

Procedures: Published 

recommendations for 

collecting data from a house 

through measurement and 

observation.
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics & Terminology

11

Simulation Protocols: 

Published defaults and 

assumptions for simulation 

inputs that are not directly 

measured or observed in 

the field.
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics & Terminology

12

Software Test Suites: 

Published suites of tests 

through which commercial 

vendors can compare their 

software results to those of 

reference software, 

analytical solutions, and 

empirical data.
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics & Terminology

13
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

A range of perspectives:

Operational Analysis: Specific occupants are considered.

Asset Analysis: Standard occupants are considered.

One Home Many Homes

• House-specific

• Unique occupant behavior

• “Operational” mode

• Average

• Standard occupant behavior

• “Asset” mode

14
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

Compare predicted energy use to measured energy use:

error""= predicted − measured

Simulated energy use Measured energy use based 

on utility billing data

15
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

Many Metrics for Accuracy!

• Mean error

• Median error

• Mean percent error

• Median percent error

• Mean absolute error

• Median absolute error

• Mean absolute percent error

• Median absolute percent error

• Standard deviation of the errors

• Root mean squared error

• Realization rate

• And more..

16
 NATIONAL RENEWABLE ENERGY LABORATORY

Basics and Terminology

How accurate should a software tool be for a single home 

(let’s say, yours)?

– Example: 30% predicted utility bill savings; what range of actual 

savings would you accept?

• 29%–31%?

• 25%–35%?

• 20%–40%?

• 0%–60%?

On average for a large number of homes? 

17
 NATIONAL RENEWABLE ENERGY LABORATORY

NREL Efforts To Improve Accuracy

18
 NATIONAL RENEWABLE ENERGY LABORATORY

NREL Efforts To Improve Accuracy

19
 NATIONAL RENEWABLE ENERGY LABORATORY 20

NREL Efforts To Improve Accuracy

1. Building America House Simulation 

Protocols

2. National Residential Efficiency 

Measures Database

3. Field and Laboratory Test Support

4. BESTEST

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis
 NATIONAL RENEWABLE ENERGY LABORATORY

1. Building America House Simulation Protocols

21

Objective: Improve inputs for asset (non-operational) assessments.

• Provides detailed, standardized 

operational information

• Based on 15+ years of Building America 

research

• Used in BEopt optimization software

• Available to broader energy assessment 

community
 NATIONAL RENEWABLE ENERGY LABORATORY

2. National Residential Efficiency Measures Database

22

Objective: Improve inputs.

• Provides detailed and standardized 

resource of information for residential 

energy efficiency measures

• Measures list

• Performance parameters (U-

(U-value, SEER, etc.)

• Costs

• Standardized format

• Improve technical consistency and 

accuracy 

• Support external building science R&D

• Enhance public transparency

Building Retrofit 

Measure Database 

NREL DB Admin

LBNL/HES

ORNL/NEAT

NREL/BEopt

Public UI

NREL Firewall

XML Feeds

Public 

Comments/Input 

Database Private-Sector 

Tools

Data from 

Retrofit 

Programs
 NATIONAL RENEWABLE ENERGY LABORATORY

3. Field and Laboratory Systems Test Support

23

Objective: Improve inputs and models.

• Building America and BetterBuildings

• Empirical validation of systems models

• Bench testing in laboratory

• Full range of operation

• In situ testing in home

• Installed performance

• Interaction 

Source: NREL TP-5500-49881

Source: NREL TP-5500-47215

PIX #11459
 NATIONAL RENEWABLE ENERGY LABORATORY

4. BESTEST

24

Objective: Provide tools for developers to improve the accuracy of their 

software.

• Test suites can be comparative, analytical, 

or empirical

• Current focus is on developing empirical 

test suites

• Classes of empirical data include:

• Laboratory and field data for isolated 

building systems

• Building characteristics and utility bill 

data

• Test suites should be diagnostic
 NATIONAL RENEWABLE ENERGY LABORATORY

5. BEopt Diagnostic Test Suite

25

Objective: Identify and understand differences between commonly 

used building energy simulation engines.

• BEopt has many categories and 

options within each category

• Test suite automatically sweeps 

through many options/categories 

using each simulation engine

• Goal is to provide equivalent inputs

• Output can be compared to identify 

bugs and physical modeling 

differences

0

10

20

30

40

50

60

70

Energy Use

Ducts

Engine A Engine B
 NATIONAL RENEWABLE ENERGY LABORATORY

6. Addressing Potential Issues

26

Objective: Identify, investigate, and correct input and software issues.

• Developed initial list of potential issues (e.g., 

“temperature variation room-to-room,” 

“performance of uninsulated assemblies”)

• Conducting focused research to investigate and 

resolve issues

• Impact of improvements can be tracked by 

simulating houses in Field Data Repository
 NATIONAL RENEWABLE ENERGY LABORATORY

NREL Efforts To Improve Accuracy

27

1. Building America House Simulation Protocols

2. National Residential Efficiency Measures Database

3. Field and Laboratory Test Support

4. BESTEST 

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis
 NATIONAL RENEWABLE ENERGY LABORATORY

7. Field Data Repository

28

Objective: Develop infrastructure for comparing software predictions to 

measured data.

• Collect, organize, and leverage historical and emerging robust 

datasets of residential building characteristics and measured energy 

use

• Develop capabilities that facilitate connecting database to reference 

and industry software to automate process of simulating homes

• Develop visualization and statistical tools for comparing predictions 

to measured data

 
 NATIONAL RENEWABLE ENERGY LABORATORY

7. Field Data Repository: Process

29
 NATIONAL RENEWABLE ENERGY LABORATORY

7. Field Data Repository: Data Collection

In hand (as of 9/14/11)

In process

Full characterization and utility bills for: 

• 950 newer homes in Florida

• Up to 18,000 older homes in the Northeast

Planned

Collection of assessment tool input files (full characterization) from a subset of 

BetterBuildings grantees with facilitation from assessment tool providers

• These will augment data collected in BBIS

• Potentially hundreds of thousands of homes

30

Number of homes Characteristics Billing Post Upgrade Locations

2970 Full Full Yes CO, IA

1295 Full Full No MN, NC, NY, OR, TX, WI

2200 Full None No VT

1670 Limited Full Limited CA

14276 Limited Limited No CO
 NATIONAL RENEWABLE ENERGY LABORATORY

NREL Efforts to Improve Accuracy

31

1. Building America House Simulation Protocols

2. National Residential Efficiency Measures Database

3. Field and Laboratory Test Support

4. BESTEST 

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

32

Objective: Improve inputs and models.

• Utilize Field Data Repository

• Investigate differences and possible 

explanatory factors

• Review methods and assumptions

• Test impacts by parametrically 

“toggling” inputs

• Look for “drivers” affecting error

Measured

Predicted
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

Automated, comparative analysis for large populations of homes

• Graphical

• Statistical

• Tool-to-measured (utility bill)

• Tool-to-tool 

Linear fit through data

How good is agreement?

Perfect agreement

Measured

Predicted

Mean Measured Use (MMBtu) 112.7

Mean Predicted Use (MMBtu) 111.0

Mean Error (MMBtu) -1.7

Mean Absolute Error (MMBtu) 15.8

Mean Absolute Percent Error (%) 15.2

Root Mean Squared Error 20.7

% of Homes Less than +/- 25% Error 82

% of Homes Less than +/- 50% Error 100

33
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

Investigate differences and possible explanatory factors

• Review methods and assumptions

• Test impacts by parametrically “toggling” inputs

• Look for “drivers” affecting error

“Out of box” predictions

Perfect agreement

Measured

Predicted

Impact of changing an assumption 

or methodology, using or ignoring 

data from measurement or 

observation, etc.

34
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

Measured

Predicted

More or fewer data?

35

Example: with versus without blower door

PIX #15073
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

HDD

% Absolute Relative Error

What’s driving the “error”?

Enclosure Uo

% Absolute Relative Error

Leakage (ACH50)

% Absolute Relative Error

Window Area (sf)

% Absolute Relative Error

Strong driver Weak driver

36

*Examples Only
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

0

50

100

150

200

250

300

350

400

450

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280

Predicted Source Energy Use (MMBtu/yr)

Measured Source Energy Use (MMBtu/yr)

Predicted Energy Use for Each Software vs Measured Energy Use

Software A

Software B

Software C

Software D

Agreement

EXAMPLE DATA, NOT REAL SOFTWARERESULTS

37
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

Impact of assessment tool improvements

38

Measured

Predicted

v1.0

v2.0
 NATIONAL RENEWABLE ENERGY LABORATORY

8. Comparative Analysis

• NREL researchers are working to connect various 

software tools to the Field Data Repository

• Preliminary comparative analysis has helped improve 

DOE’s Home Energy Saver software

• Data transfer standard will improve accessibility and 

usefulness

39
 NATIONAL RENEWABLE ENERGY LABORATORY

NREL Efforts to Improve Accuracy

40

1. Building America House Simulation Protocols

2. National Residential Efficiency Measures Database

3. Field and Laboratory Test Support

4. BESTEST 

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

41

Objective: Estimate the uncertainty in software predictions due to 

uncertainty in input information.

• Requires estimates of uncertainty in input 

parameters

• Monte Carlo simulations performed using 

BEopt to generate distribution of 

simulation output

• Provides estimates of range of expected 

disagreement between predicted and 

actual energy uses

• Helps identify most important information 

to collect in energy assessments
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

• Whether an input is defaulted, estimated, or measured, there is some 

uncertainty as to how closely the input value matches the “true” value

– e.g., heating set point = 68 ± X °F 

• Errors in input propagate through a building energy simulation 

program to produce errors in output 

– e.g., predicted annual source energy use = 250 ± Y MMBtu

Research Goal: Establish uncertainty ranges around software 

predictions.

42
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

Types of Input Error:

• Building Inputs. Building geometry, material physical properties, and 

characteristics of mechanical equipment.

• Occupant Inputs. Descriptions of occupant behavior and occupantcontrolled equipment settings.

• Site Inputs. Local weather, soil properties, and adjacent 

structures/vegetation.

Research Question: In an asset evaluation, where specific 

occupant information is NOT collected, what is the uncertainty in 

energy use predictions due to uncertainty and variability in 

occupancy? 

43
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

44

Category House A House B

Orientation Front of house faces north Front of house faces north

Dimensions North and south walls = 57 ft 

East and west walls = 27 ft

One story

North and south walls = 50 ft 

East and west walls = 25 ft

Two story

Garage None None

Bedrooms/Baths 3/2 3/2

Neighbors 15 ft away on east and west 15 ft away on east and west

Eave Depth 2 ft 2 ft

Vented Crawlspace 2.0 ACH, R-11 between joists 2.0 ACH, R-19 between joists 

Exterior Walls 2 × 4, 16”o.c., no cavity insulation 

(R-1.2 air gap)

2 × 6, 21”o.c., R-19 cavity 

insulation

Unfinished Attic 2 × 6 joists, R-11 insulation 2 × 6 joists, R-49 insulation

Window Type Single pane, 

U = 0.774, SHGC = 0.679

Low-e double pane, 

U = 0.318 , SHGC = 0.302 

Window 

Area/Distribution

20% of exterior wall area, 20% of 

total window area on front, 40% on 

back, and 20% on each side

20% of exterior wall area, 20% 

of total window area on front, 40% 

on back, and 20% on each side

Gas Furnace 78% AFUE 92.5% AFUE

Air Conditioner SEER 10 (9.31 EER) SEER 13 (11.09 EER)

Ducts Uninsulated, in crawlspace, 

leakage fraction = 0.300

R-6 insulation, in crawlspace, 

leakage fraction = 0.075

Living Space SLA 0.000886 0.000360

Mechanical Ventilation Spot vent only Spot vent and ASHRAE 62.2 

(including infiltration credit)

Water Heater Gas, energy factor = 0.59, 

in living space

Gas, energy factor = 0.67, 

in living space

Major Appliances Refrigerator, electric range, 

dishwasher, clothes washer, 

electric clothes dryer.

Refrigerator, electric range, 

dishwasher, clothes washer, 

electric clothes dryer. 

House A: 1,539 ft2, one 

story, inefficient home, 

representative of 1960s-era 

construction

House B: 2,500 ft2, two 

story, more efficient 

home, representative of 

new construction
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

Technical Approach:

1. Identify operational inputs accessible in the BEopt input file and 

select a subset of key inputs for variation

2. Define probability distributions representing the uncertainty ranges 

for each key input

3. Randomly select values from probability distributions for each key 

input and simulate energy use for that “realization” of inputs

4. Repeat Step 3 for many realizations to generate a range of 

simulation output using BEopt/EnergyPlus

5. Analyze the distributions of simulated energy uses.

45
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

Technical Approach:

For preliminary analysis, assume inputs are either independent or (1:1)

• e.g., select heating set point and refrigerator multiplier independently

• e.g., use same selected value for clothes washer/dryer multipliers

Select inputs from triangular probability distributions

• Require estimates of minimum, nominal, and 

maximum values

• Allow for non-symmetric distributions

46

PIX # 07726
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

47

Input Minimum Nominal Maximum

Space Heating Set Point (°F) 60 68 75

Space Cooling Set Point (°F) 71 78 86

Miscellaneous Electric Loads Multiplier 0.2 0.8 2.0

Miscellaneous Gas Loads Multiplier 0.2 0.8 2.0

Miscellaneous Hot Water Loads Multiplier 0.2 0.8 2.0

Interior Shading Multiplier 0.5 0.6 1.0

Interior Lighting Multiplier 0.2 0.8 2.0

Exterior Lighting Multiplier 0.2 0.8 2.0

Furniture Conductivity (Btu-in./hr-ft2-°F) 0.64 0.80 0.96

Furniture Density (lb/ft3) 32 40 48

Furniture Specific Heat (Btu/lb-°F) 0.232 0.290 0.348

Furniture Area Fraction 0.1 0.3 0.5

Furniture Weight (lb/ft2) 2 8 14

Furniture Solar Absorptance 0.4 0.6 0.8

Refrigerator Multiplier 0.7 1.0 1.3

Range/Dishwasher Multiplier 0.2 0.8 2.0

Clothes Washer/Dryer Multiplier 0.2 0.8 2.0

Fraction of Total Window Area Open 0.00 0.04 0.14

Water Heater Set Point (°F) 110 125 140

Operational Inputs Perturbed in Analysis
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

48

Example Preliminary Results: Existing home annual source energy use by climate.
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

49

 HOUSE A HOUSE B

 MEAN SD SD/MEAN MEAN SD SD/MEAN

Atlanta, GA 212.2 27.4 0.13 146.5 19.8 0.14

Chicago, IL 327.1 34.8 0.11 193.8 20.4 0.11

Houston, TX 192.5 26.8 0.14 142.9 21.2 0.15

Los Angeles, CA 129.5 22.9 0.18 111.4 18.1 0.16

Phoenix, AZ 228.7 30.5 0.13 157.6 22.4 0.14

Seattle, WA 233.6 37.5 0.16 148.7 19.4 0.13

City Average 220.6 30.0 0.14 150.1 20.2 0.14

Summary of Preliminary Results: Annual source energy by climate.

Preliminary Conclusions:

• Variability in occupant behavior causes considerable uncertainty in 

predictions for an asset evaluation

• Approximately ± 20%-35% for total source energy use, may be more if 

input correlation is considered
 NATIONAL RENEWABLE ENERGY LABORATORY

9. Operational Uncertainty Analysis

0

0.05

0.1

0.15

0.2

0.25

0.3

Total Observed Variability (Predicted - Actual)

Operation Uncertainty Estimated from Monte Carlo

Explained by Operational Uncertainty

(+/- 2 Standard Deviations )

+/- 2 Standard Deviations for 

Observed Variability (Predicted - Actual)

50

Example Only
 NATIONAL RENEWABLE ENERGY LABORATORY

Final Remarks

51
 NATIONAL RENEWABLE ENERGY LABORATORY

Final Remarks

52

• Stakeholders rely on accurate predictions

• Analysis methods tend to over-predict energy use for poorly 

insulated, leaky homes

• NREL is following a well-defined process to identify, investigate, and 

resolve potential issues

• Various projects support overall effort

1. Building America House Simulation Protocols

2. National Residential Efficiency Measures Database

3. Field and Laboratory Test Support

4. BESTEST 

5. BEopt Diagnostic Test Suite

6. Addressing Potential Issues

7. Field Data Repository

8. Comparative Analysis

9. Uncertainty Analysis

Focus for this talk
 NATIONAL RENEWABLE ENERGY LABORATORY

References

53

 

Hendron, R.; Engebrecht, C. (2010). Building America House Simulation Protocols. 

Golden, CO: National Renewable Energy Laboratory. NREL/TP-550-49246. 

 

Judkoff, R.; Neymark, J. (2006). “Model Validation and Testing: The 

Methodological Foundation of ASHRAE Standard 140.” ASHRAE Transactions

112(2):367–376. Atlanta, GA: American Society of Heating, Refrigerating and Air- Conditioning Engineers.

 

Judkoff, R.; Wortman, D.; O’Doherty, B.; Burch, J. (2008). A Methodology for 

Validating Building Energy Analysis Simulations. Golden, CO: National Renewable 

Energy Laboratory. NREL/TP-550-42059. Based on unpublished report of 1983 with 

same authors and title, previously referenced as SERI/TR-254-1508.

 
 NATIONAL RENEWABLE ENERGY LABORATORY

More Information

54

• More information can be found in a recently published report:

 Polly, B.; Kruis, N.; Roberts, D. 2011. Assessing and Improving the Accuracy of 

Energy Analysis Methods for Residential Buildings. Golden, CO: National Renewable 

Energy Laboratory. DOE/GO-102011-3243. 

• Questions?