### Study Design

Evaluations must rely on either experimental or quasi-experimental design to determine savings. In experimental design, customers will be placed in either a treatment or control group based on a randomized control trial (RCT) approach. Quasi-experimental designs include methods such as the regression discontinuity method (selecting a comparison group based on customers that meet similar eligibility requirements) and the variation in adoption approach (selecting participants that enroll later in a program as a control group for those that participated in a prior period), but both of these introduce additional error into the analysis.

### Length of Study and Baseline Period

Reporting deadlines and data availability may limit the length of the study or baseline periods. For example, many annual energy efficiency reports are due in the first or second quarter of the year, which may put pressure on evaluators to use a truncated study period. In other cases, pulling billing system data from more than a year or two back may require accessing archive data, which may be cost- prohibitive.

### Annualizing Savings

Most jurisdictions in the U.S. report first year annualized savings, meaning that full year annualized savings are reported even for measures that get installed for only part of the year. Evaluations of behavior-program evaluations, however, often report monthly energy savings (since the models are typically based on monthly billing data). A utility annualizing savings from behavior-based programs could report significantly greater savings than a utility that limits the reporting to participant months (i.e., the number of participants for each month that received energy reports).

### Panel vs. Aggregated Data

Analysis models can either use energy data that are aggregated across time for both the pre- and post-program periods (e.g., average energy use for the period prior to and during the program year) or panel data (also called time series of cross-sectional data), which typically are data from multiple time points for pre- and post-program periods (e.g., monthly energy use for each month of the program and for the twelve months preceding the program).

### Comparison of Energy Usage vs. Change in Energy Usage

Analysis models can be specified to estimate program savings by either comparing the energy saved by the treatment group to the energy saved by the control group (a “difference in differences” approach), or comparing the energy use of the treatment group to the control group during the program (a post-treatment comparison).

### Equivalency Check

The degree to which a savings estimate is unbiased depends on how similar the control group is to the treatment group. Some studies determine if the households in the treatment group have characteristics that are statistically similar to those in the control group (e.g., compare the monthly or yearly pre-program energy use and the distribution of pre-program energy use between the treatment and control group), while others do not.

### Treatment of Households that Opt Out or Close Accounts

The treatment of households that opt out may vary between studies, with some studies leaving these homes in and others taking them out. This could potentially bias the results (e.g., households that drop out may take less actions to reduce energy than those that continue to receive the home energy

2013 International Energy Program Evaluation Conference, Chicago