Discrepancy , What Discrepancy ? Reasons for the Difference between Claimed and Evaluated Savings

Program cycle-to-program cycle, evaluators find a discrepancy between claimed savings from energy efficiency programs and their verified savings. These differences are caused by a number of factors, but which are most important? Which reasons for discrepancy affect the claimed energy savings to a large degree, to a smaller degree, and in which direction? The recent 2010 to 2012 California IOU (investor owned utility) custom impact evaluation involved the analysis of over 400 energy efficiency projects, including gross impact evaluation of savings claims, representing the IOU-wide custom portfolio. This paper quantifies the energy impact of the main reasons for discrepancy that were found from the evaluation’s commercial building energy efficiency measures, disaggregated by measure type for greater insights into trends by energy efficiency project. In evaluating the gross energy savings claims, we found that the discrepancy for each project could be due to a number of factors, including, for example, eligibility requirements, changes in operating hours, or the applicable baseline. By investigating and quantifying the reasons for the discrepancy, the evaluation community will assist the IOUs and their implementers in better assessing projects as they are being developed with respect to the highest risk factors. The expectation that the evaluated savings will better match the claimed savings, at least for factors the IOUs can control, will improve the effectiveness of custom programs in any jurisdiction where they are in use. Introduction The evaluation of energy efficiency programs serves an important regulatory and advisory function. Evaluation reports will generally provide a final realization rate for a program (or a portfolio of programs) and offer recommendations to increase program effectiveness. A deeper investigation into the reasons for discrepancies between claimed and evaluated savings, through careful standardized analysis of project results, and the reasons behind these results, can be used to inform the evaluation effort and minimize future discrepancies. To truly understand energy efficiency project performance, we investigate the discrepancy factors leading to that performance. This is particularly important for custom programs and markets, given the wide variability in measures types and project size. Custom projects are among the largest projects in an IOU’s energy efficiency portfolio. Understanding energy performance for custom projects from an evaluation perspective is important in improving project and program performance, and in meeting portfolio goals. The diversity of measures and programs across different subsectors, along with the multiple project drivers and market actors, make this investigation into discrepancy factors particularly important and instructive in determining what aspects of program design and project implementation to concentrate on when developing programs and projects – and when claiming and verifying energy savings. By considering the effects of discrepancy factors, both overall and by measure type, the utilities and their implementers can 251 4©2014 ACEEE Summer Study on Energy Efficiency in Buildings identify and act on areas for improvement and ways to reduce the discrepancies in the claimed and evaluated savings, achieving realization rates closer to unity. Eight discrepancy factors and seven measure groups are included in this analysis. A number of charts were constructed among factors and groups to facilitate comparison. The effects of the discrepancy factors on all commercial projects is shown first. Following this, charts showing the effects of the discrepancy factors on measure groups are displayed. The custom impact evaluation report authorized by the California Public Utility Commission (CPUC) Energy Division (ED) for the 2010-2012 evaluation cycle (Itron et al. 2013) is the source of data used in this analysis. This evaluation report is publicly available at www.calmac. org. The evaluation effort involved site-specific gross impact activities for 429 custom energy efficiency projects for the commercial, agricultural and industrial sectors. This paper focuses on the 157 commercial sector projects in that sample and measure groups relevant to that sector. Each project was evaluated and a reason for the discrepancy (and the percentage of the discrepancy) was provided by the evaluating engineer. For some projects, more than one reason was cited. When there was more than one factor, the discrepancy was allocated between factors. Discrepancies were explained in detail in the final site report for each energy efficiency project. By designing the evaluation to capture that information at the outset, the collection of discrepancy determinations and the percentages allocated to each factor was greatly facilitated. It is important to realize that there are both gross impact and net impact components. These components lead to gross realization rates, net to gross ratios, and net realization rates combining those factors. This paper addresses the gross energy savings and gross realization rate; discrepancies causing net impact differences have not been addressed. Gross realization rates, ex-post gross savings divided by ex-ante savings claimed, were calculated and are shown in this paper for different measure types based on site-specific energy project realization rates. Description of Discrepancy Factors The evaluation identified eight important discrepancy factors leading to adjustments to the ex-ante claims stemming from ex-post gross impact estimates in the M&V sample. The eight factors are: operating conditions, inappropriate baseline, calculation method, ineligible measure, equipment specification, tracking database discrepancy, program rule compliance, and unquantified fuel impacts. These factors are described in more detail in this section.  Differences in operating conditions (for example, hours of operation, VSD speeds, reversions to original operation, temperature setpoints that changed from the forecast setpoint, operations based on forecasts instead of the most current observed / measured conditions, changed production levels due to market conditions, etc.)  Inappropriate baselines or baseline conditions used for ex-ante savings estimation (for example, rejected early (or normal) replacement claims, new equipment that do not exceed code-required efficiency levels, new equipment that do not exceed industry standard efficiency levels, inaccurate baseline or pre-retrofit operating hours, etc.)  Calculation methods used for ex-post savings estimation were revised from those used to estimate ex-ante savings (for example, simulation models that utilize Title 24 compliance schedules as opposed to actual building operation; switching from Energy Pro to eQuest 1 Uninstalled measures, measure count variations, and inoperable measures were investigated at a project level but not found to be significant discrepancy factors. 252 4©2014 ACEEE Summer Study on Energy Efficiency in Buildings building simulation software; use of or different weather normalization techniques or weather files; different spreadsheet calculation approaches based on post-retrofit or postconstruction data availability; expanded spreadsheet approaches to account for varying versus constant loads, different resolution, and interactive effects; shifts to or from the use of billing analyses and interval data, particularly for peak demand impacts; use of calculation inputs defining initial or ex-post operating conditions; etc.)  Ineligible measures (measures or repairs not allowed by program rules or a replacement of an existing system with a system with identical performance).  Equipment specifications (changes from the final ex-ante estimates to the actual equipment installed in the field as verified by the evaluators)  Tracking database discrepancies (errors based on clear project documentation or duplication, as when ex-ante savings calculation results from project documentation do not match the final claimed ex-ante savings as reported in utility tracking data)  Program rule compliance (program rule violation that does not involve ineligible measures)  Unquantified fuel impacts (the analysis did not consider all fuel impacts; projects may involve electric HVAC measures that have thermal effects and projects where cogeneration systems are a source of electrical and thermal energy) Measure Group Definition Seven measure groups were selected based on the frequency on which they occurred in the custom sample. Projects were generally grouped into the primary measure type which accounted for the majority of savings; the exception is the whole building group which generally covered a range of measures in an individual project. The measure groups studied are as follows:  Chillers – any new or rehabilitated water-cooled or air-cooled chillers, and associated cooling towers  Boilers new or upgraded boilers and their ancillary systems, including boiler controls  HVAC Controls – includes sensors, control equipment, and EMS systems  HVAC Other – includes air handling units, economizers, fans, and pumps  Variable Speed Drives – any variable speed drive controlling motors (usually HVAC motors)  Commissioning / Retro-commissioning – adjustments to new or existing control systems  Whole Building Projects – comprehensive packages of energy efficiency measures, typically in new construction projects that utilize building simulation models such as Energy Pro Overall and Measure Specific Discrepancy Factors Explaining the Gap between the Claimed and Evaluated Savings Gross realization rates are presented for all evaluated commercial projects and for the measure groups in Table 1. The realization rates were calculated as an un-weighted simple average of the site specific project gross realization rates (including and excluding points with 253 4©2014 ACEEE Summer Study on Energy Efficiency in Buildings