From: Frey, Nathan J. <Nathan_J._Frey@omb.eop.gov> 

Sent: Thursday, July 23, 2015 6:42 PM 

To: Culligan, Kevin; Harvey, Reid; Vasu, Amy 

Cc: Szabo, Aaron 

Subject: Summary of Interagency Comments Under EO 12866 and 13563 -- Draft EM&V 

guidance for the CPP 

Attached please find a summary of additional interagency comments under EO 12866 and 13563 

on the EPA draft EM&V guidance for the CPP. 

 

Thanks and please let me know if you have any questions. 

Nathan 

<< SummaryofinteragencycommentsunderEO12866_13563_EO 12866_EM&V 

Guidance_7212015.docx >> 

<< SummaryofinteragencycommentsunderEO12866_13563_EO 12866_EM&V 

Guidance7232015(2).docx >> 

 
Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

1

Recommendations

1) The proposal preamble text stated “we intend to establish guidance for acceptable

quantification, monitoring, and verification of RE and demand‐side EE measures for an

approvable EM&V plan...[and] in developing guidance, the agency does not intend to limit the

types of RE and demand‐side EE measures and programs that can be included in a state plan,

provided that supporting EM&V is rigorous, complete, and consistent with EPA’s guidance.”

This appears to flag for the public that EPA’s guidance would be used as approvability criteria,

and reiterates that consistency with EPA’s future guidance would be treated as a minimum

requirement. And some language in the final preamble appears to be consistent with this

approach (e.g., Section VIII.8.k(4)b of the revised preamble states that CPB is required and that

EE providers can refer to EPA’s EM&V guidance for descriptions of the appropriate CPBs to apply

in specific contexts).

If EPA is finalizing this proposed use of the guidance, this is not clear and should be clarified in

the preamble, regulatory text, and guidance. We strongly support this approach, which allows

guidance to be updated over time. If EPA is not finalizing this proposed use of guidance, it

should clearly explain how it will use the guidance differently, and the rationale for departing

from the proposal.

2) Absent this approach, the requirements in the preamble and regulatory text itself are often

vague—e.g., “savings values must be determined on a regular basis;” and “savings...must be

quantified and verified using applicable, credible, and rigorous industry‐standard practices...”

Without specific definitions or clauses to the effect of “as determined by the Administrator,”

these requirements are not enforceable and likely only to cause confusion. Given the need for

flexibility in a rapidly evolving field, we strongly recommend that EPA caveat all such

requirements with “as determined by the Administrator,” or a requirement that states

“demonstrate to EPA’s satisfaction” that EM&V plans meet the standard for “applicable,

credible, and rigorous practices.”

3) The preamble describes a number of specific requirements that are not stipulated in the

regulatory text. For example, the requirement to use the CPB and minimum intervals for

determining savings depending on type of program. We recommend adding these specific

requirements to the regulatory text for clarity.

4) Although the draft guidance indicates that methods should be updated to reflect advancements

in best practice techniques and the availability of information, this is not reflected in the

regulatory requirements. It would be reasonable to expect that states using deemed savings

values use values that are empirically verified, if available. Since this approach allows states that

use deemed savings to continue doing so—just substituting verified values if available—it would

not be disruptive to existing EM&V programs. Rather than be overly restrictive in a field that is

rapidly advancing, this change would merely ensure that those rapid advancements are

reflected in state practice for ERC creation.
Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

2

Recommended regulatory text: All savings values assessed using deemed savings must use

deemed savings values that have been empirically verified by a rigorous study involving

randomized control trials and quasi‐experimental design, if such values are available and

reasonably applicable for the measures in question and the relevant conditions in which those

measures are applied. If such verified deemed savings values are not available at the time of

measure installation, but become available before the end of the measure EUL, savings values

must be adjusted prospectively to reflect the verified deemed savings values.

5) It may be reasonable to expect that measures used most commonly—and that are responsible

for the largest share of total EE ERC generation—meet a higher threshold for rigor. This

approach would also reflect a balance between rigor and cost, by requiring more rigor only for

the measures that present the greatest risk of overestimation. Given the value of ERCs for

measures that are most commonly employed, the additional costs of directly measuring and

verifying savings for a representative sample should not be excessive.

Recommended regulatory text: For measures with aggregate annual savings in excess of XX

MWh [threshold TBD], if deemed savings values are not empirically verified by a rigorous study

involving randomized control trials or quasi‐experimental design [because such values are

unavailable], then savings must either be directly and independently measured and verified for

at least a representative sample of measures for which deemed savings values are used to assess

savings, or studies using a RCT or quasi‐experimental design must be conducted to verify deemed

savings values. Deemed savings values must be adjusted prospectively to reflect findings from

direct measurement and verification or studies with a RCT or quasi‐experimental design.  

6) For individual measures that are estimated to generate outsized savings (e.g., large industrial

equipment upgrades...etc.) over a specified threshold (TBD), direct measurement and

verification or RCT/quasi‐experimental studies should be required (rather than deemed savings).   

Recommended regulatory text: For individual measures with annual savings in excess of XX

MWh [threshold TBD], savings must be directly measured and verified or assessed using a study

with RCT or quasi‐experimental design.

7) A couple of the key requirements are stated only as documentation requirements—the

requirement to address independent factors and assess rigor of savings values. These are critical

requirements and should be stated not only as documentation requirements, but as

implementation requirements. In other words, states should not only be required to document

how they address independent factors, but also directly required to address independent factors.

Without the latter, couldn’t a state satisfy the requirement by saying it doesn’t, so long as it

provides that documentation?  

We recommend that EPA restate these requirements as direct implementation requirements,

and then require that the state document how it has satisfied each requirement.

8) Some commenters suggest that section 3 and appendix C (which is very helpful) could be

expanded to add additional summaries and links to detailed examples or case studies for
Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

3

common efficiency programs. This could link to topic‐specific EM&V guidance to supplement

the general resources in table 3‐1. For example, behavioral efficiency programs

accompanied with EM&V done through randomized control trial is a widely practiced

approach that is not specifically documented here.

9) Some commenters suggest that this document could do more to explicitly support

continued improvement in EM&V. We appreciate the desire to allow a wide variety of

EM&V approaches, including the most commonly used today, but advanced EM&V

techniques such as randomized control trials and more real‐time measurement are likely to

yield better savings and allow program improvement, and should be encouraged.  

Possible action: add clarification in the introduction that states are encouraged to seek out

the most advanced EM&V approach that is viable, and that they may take the available

quality of EM&V into account when selecting what form of an efficiency program to adopt.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

Evaluation Measurement and Verification (EM&V) Guidance

for Demand‐Side Energy Efficiency (EE)

U.S. Environmental Protection Agency

DRAFT

7/09/15
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

iii

Contents

Acknowledgements ............................................................................................................................. v

Citing of this Document ....................................................................................................................... v

For More Information ............................................................................... Error! Bookmark not defined.

Acronyms and Abbreviations.............................................................................................................. vi

1. Introduction..................................................................................................................................... 1

1.1 Experience with EM&V............................................................................................................... 1

1.2 Basic Challenge with EM&V........................................................................................................ 2

1.3 Approach to EM&V in the Emission Guidelines........................................................................... 3

1.4 Applicability and Scope............................................................................................................... 4

1.5 Intended Audience ..................................................................................................................... 6

1.6 Guidance Updates ...................................................................................................................... 6

2. EM&V Guidance Applicable to All Demand‐Side EE .......................................................................... 7

2.1 EM&V Methods.......................................................................................................................... 8

2.1.1. Discussion.................................................................................................................................8

2.1.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.2 Electricity Savings Metrics and Baselines .................................................................................... 9

2.2.1. Discussion.................................................................................................................................9

2.2.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.3 Reporting Incremental and Cumulative Savings  ......................................................................... 9

2.3.1. Discussion............................................................................................................................... 12

2.3.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.4 Deemed Savings....................................................................................................................... 13

2.4.1. Discussion............................................................................................................................... 13

2.4.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.5 Independent Factors Affecting Energy Consumption and Savings ............................................. 15

2.5.1. Discussion............................................................................................................................... 15

2.5.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.6 Uncertainty .............................................................................................................................. 16

2.6.1. Discussion............................................................................................................................... 17

2.6.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.7 Avoiding Double Counting ........................................................................................................ 18

2.7.1. Discussion............................................................................................................................... 18

2.7.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.8 Effective Useful Life and Persistence of Savings........................................................................ 20

2.8.1. Discussion............................................................................................................................... 20

2.8.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.9 Savings Quantification and Verification Cycles.......................................................................... 23

2.9.1. Discussion............................................................................................................................... 23

2.9.2. Applicable Guidance................................................................... Error! Bookmark not defined.

2.10 Transmission and Distribution System Savings Adders............................................................ 24

Deleted: iv

Deleted: iv

Deleted: iv

Deleted: v

Deleted: 9

Deleted: 10

Deleted: 10

Deleted: 12

Deleted: 12

Deleted: 13

Deleted: 15

Deleted: 15

Deleted: 16

Deleted: 16

Deleted: 18

Deleted: 19

Deleted: 19

Deleted: 19

Deleted: 20

Deleted: 20

Deleted: 22

Deleted: 22

Deleted: 24

Deleted: 24

Deleted: 26

Deleted: 26

Deleted: 28
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

iv

2.10.1. Discussion............................................................................................................................. 24

2.10.2. Applicable Guidance................................................................. Error! Bookmark not defined.

2.11 Addressing Interactive Effects................................................................................................. 25

2.11.1. Discussion............................................................................................................................. 25

2.11.2. Applicable Guidance................................................................. Error! Bookmark not defined.

2.12 Use of EE EM&V Protocols and Guidelines.............................................................................. 25

2.12.1. Discussion............................................................................................................................. 25

2.12.2. Applicable Guidance................................................................. Error! Bookmark not defined.

3. Supplemental Guidance for Specific Categories of EE ....................................................................... 8

3.1 Demand‐Side Management Programs ...................................................................................... 30

3.1.1. Program Description............................................................................................................... 31

3.1.2. Applicable Guidance – Direct Action DSM .............................................................................. 32

3.1.3. Applicable Guidance – Indirect Action DSM ........................................................................... 32

3.2 Project‐Based Energy Efficiency ................................................................................................ 34

3.2.1. Description ............................................................................................................................. 34

3.2.2. Applicable Guidance – Project‐Based Energy Efficiency .........................................................34

3.2.3. Description ............................................................................................................................. 37

3.2.4. Applicable Guidance – State and Local Government Building Energy Code and

Compliance Programs ............................................................................................................ 38

3.3 State and Local Government Incremental Product Energy Standards........................................ 41

3.3.1. Description ............................................................................................................................. 41

3.3.2. Applicable Guidance – State and Local Government Incremental Product Energy

Standards............................................................................................................................... 41

Appendix A. Optional EE and RE EM&V Plan Templates....................................................................A‐1

Appendix B. Glossary of Terms..........................................................................................................B‐3

Appendix C. EM&V Best Practices and Resources for Common EE Technologies................................ C‐1

Deleted: 28

Deleted: 29

Deleted: 29

Deleted: 29

Deleted: 29

Deleted: 34

Deleted: 35

Deleted: 35

Deleted: 36

Deleted: 36

Deleted: 38

Deleted: 38

Deleted: 38

Deleted: 41

Deleted: 42

Deleted: 45

Deleted: 45

Deleted: 45

Deleted: B‐4

Commented [A1]: Lots of Error Bookmark not defined. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

v

Acknowledgements

Citing of this Document

Commented [A2]: ? 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

vi

Acronyms and Abbreviations

ANSI ‐ American National Standards Institute

ASHRAE ‐ American Society of Heating, Refrigerating, and Air‐Conditioning Engineers  

ASME – American Society of Mechanical Engineers

CHP – Combined Heat and Power (electricity generation systems)

C&S ‐ Building Energy Code and Product Energy Standard  

CPP – Clean Power Plan  

CPB ‐ Common Practice Baseline

CSF – Control Savings Factor (for lighting controls)

EGU ‐ Electricity Generation Units

EE – Energy efficiency

EERS – Energy Efficiency Resource Standard

ESCO – Energy services company

EFLH – Equivalent full load hours  

EM&V – Evaluation, Measurement, and Verification

EPA – (United States) Environmental Protection Agency

EUL – Effective Useful Life

FEMP – (U.S. Department of Energy) Federal Energy Management Program  

HVAC ‐ Heating, Ventilating, and Air‐Conditioning  

HHV – High Heating Value

IE – Independent third‐party evaluator

IPMVP ‐ International Performance Measurement and Verification Protocol  

LED–Light emitting diode

LHV – Low Heating Value

M&V – Measurement and verification

MMBtu – million British Thermal Units

MFG ‐ Multi‐Fuel Generator  

MWh – Megawatt‐hour

RCT ‐ Randomized Control Trial

RTF – (Northwest Power and Conservation Council Northwest) Regional Technical Forum

RE – Renewable energy

REC – Renewable Energy Certificate

RGU – Renewable generating unit

SEE ACTION – State and Local Energy Efficiency Action Network

T&D ‐ Transmission and distribution (system)

UMP – (United States Department of Energy) Uniform Method Project

WHP ‐ Waste heat to power (electricity generation systems)
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

1

1. Introduction  

This document is a supplemental resource for state air and energy officials, affected electric utility

generating units (EGUs), and energy efficiency (EE) providers to use as they plan for, implement, and

report on the MWh results of their EE programs and projects, consistent with the U.S. Environmental

Protection Agency’s (EPA) final carbon pollution guidelines. In the final emission guidelines, the EPA

defines several categories of zero‐ and low‐emitting measures – such as demand‐side energy efficiency

(EE) and renewable energy (RE) – that can be used to generate emission rate credits (ERC) or otherwise

adjust a CO2 emission rate for the purpose of achieving state emission goals and emission performance

standards that apply to affected EGUs. To ensure that these measures1 are quantifiable and verifiable,

the EPA has established regulatory requirements for evaluation, measurement, and verification (EM&V).

These regulatory requirements define the specific methods, technical assumptions, and approaches that

must be applied to quantify and verify mega‐watt hour (MWh) savings and generation from eligible

resources. The full list of applicable EM&V requirements is discussed in Sections VIII.G.3‐5 of the

emission guidelines.

For demand‐side EE, the EPA has determined that additional guidance may help EE providers develop

EM&V plans and monitoring and verification reports that satisfy the applicable regulatory requirements.

This guidance addresses key definitions, EM&V methods, the appropriate use of protocols and

guidelines, and other topics for successfully quantifying and verifying savings for purposes of generating

ERCs and adjusting an emission rate. However, this document is not intended as a primer on EM&V, nor

does it replace the need for a thorough review of the EPA’s regulatory requirements.

Note that, for RE and other zero‐ and low‐emitting measures that generate electricity, the EPA has

determined that EM&V relies primarily on direct measurements using revenue‐quality meters and other

industry‐standard equipment, and is therefore relatively straightforward. In these cases, all applicable

requirements for quantification and verification are provided in the emission guidelines, and no further

guidance is offered by the EPA at this time. Further, this document does not apply to states submitting

mass‐based plans. In these cases, compliance is determined solely based on measured CO2 emissions at

the stack.2   

1.1 Experience with EM&V for Demand‐Side EE

From the time that demand‐side EE emerged as an important energy strategy in the 1970s,3 efforts to

quantify and verify the MWh savings of these actions have been critical to their success, credibility, and

                                                            

1 The emission guidelines use the term “measure” to refer to an eligible action that results in MWh that can be

used to adjust a rate for compliance purposes.  In the context of demand‐side EE, “measure” refers to an

installation or modification of equipment or systems at an end‐use energy consumer facility – or strategy intended

to affect consumer energy use behaviors – that reduces the amount of energy that would otherwise have been

used to deliver an equivalent or improved level of end‐use service. 2 However, states with EE programs and projects in place EM&V will likely continue to require EM&V for purposes

of ensuring that electricity‐customer funds are spent prudently and cost‐effectively.    3 National Energy Program Fact Sheet on the President’s Program, April 20, 1977. Available at

http://www.presidency.ucsb.edu/ws/?pid=7373.  

Commented [A3]: General question – do GeoThermal 

heat pumps, especially in new construction, fit in here, or 

warrant a heads up section?
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

2

expansion. The earliest such efforts involved measurement and verification (M&V) of individual projects;

this was followed by an evolution and improvement in practices for a broad range of EE program

strategies and delivery mechanisms across sectors.4 Today, these EM&V practices are applied by

utilities, energy service companies (ESCOs), and other EE providers.  They are also backed up by several

well‐established protocols and guidelines, and overseen by Public Utility Commissions (PUCs) and other

implementing agencies and authorities. The EM&V industry is now comprised of many large firms and

hundreds of individual practitioners, and is supported by training and certification programs, as well as a

rich library of published reports and publicly available data and technical resources.

The EM&V approaches in wide use today – including budget levels, oversight procedures, and preferred

methods – are derived from PUC5 requirements for consumer‐funded EE programs, as well as the

Federal Energy Management Program’s (FEMP) requirements for ESCO projects. While the level of

review varies somewhat by state and program, these oversight mechanisms have generated the

majority of the protocols and best practices for quantifying savings in the industry. The specific EM&V

approach that is applied to a given program or project depends on the type of EE, overall policy

objectives, available budgets, and other factors.  For consumer‐funded EE programs and ESCO projects,

savings are typically quantified on an annual basis.  

With this evolution in the accuracy and reliability of EM&V, many states and utilities now routinely rely

on EE to achieve their energy (MWh) and capacity (MW) goals. In addition, two Independent System

Operators (ISOs) – ISO New England (ISO‐NE) and PJM – have established forward capacity markets

(FCMs)6 that pay suppliers of EE programs and other demand‐side resources to compete directly with

electric generators to meet regional capacity needs. The oversight and quality control of EE resources

that are bid into the market is governed by EM&V rules and requirements defined in evaluation manuals

established by ISO‐NE and PJM.  

Despite improvements in EM&V over time, quantification knowledge is more robust for some EE

program and policy types than for others.  Additionally, there is limited experience applying EM&V

procedures and protocols to emissions trading programs, where each MWh of replaced generation

becomes a commodity that can be bought and sold by a regulated source. Environmental regulators,

including the U.S. EPA and most state environmental agencies, have limited experience quantifying and

verifying EE in this regulatory context.  As a result, the EPA’s final emission guidelines include a number

of safeguards and quality control features that are intended to ensure the accuracy and reliability of the

claimed EE savings.

1.2 Determining Baselines   

The key challenge with quantifying EE savings is the identification of an accurate baseline from which to

measure MWh results. Regardless of the protocols and procedures applied, all savings values are

determined by comparing energy use after an EE program or project is installed (the reporting period)

                                                            

4 EM&V is defined to mean the set of procedures, methods, and analytic approaches used to quantify the MWh

from demand‐side EE and thereby ensure that the resulting savings and generation are quantifiable and verifiable.    5 These government entities are also referred to as a public service commission or a board of public utilities. 6 Forward capacity markets are intended to ensure sufficient electric generating capacity is available to meet

future peak electricity demand.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

3

with the best estimate of the likely energy use in the absence of the project or measure (the

“counterfactual” scenario, or baseline). Figure 1‐1 illustrates this concept. In general, the entity

overseeing or managing EE activities determines the applicable baseline prior to EE implementation on

the basis of policy objectives and various programmatic considerations.  MWh savings are then

quantified relative to this baseline using EM&V methods determined by program type, available

budgets, the magnitude of expected impact, and other factors.  

1.3 Approach to EM&V in the Emission Guidelines

In June 2014, the EPA proposed carbon pollution emission guidelines for certain existing EGUs, as well as

a “State Plans Considerations” technical support document (TSD) 7 that outlined a general approach to

establishing EM&V requirements and guidance. The TSD proposed that the EPA’s final EM&V

requirements could leverage the industry‐standard practices, protocols, and methods currently utilized

by the majority of states implementing demand‐side EE and RE programs. The EPA further noted that

many state PUCs, and other regulatory bodies and program management authorities, already have

significant EM&V infrastructure in place, and some have been applying, refining, and enhancing their

approaches for over 30 years.  

Figure 1‐1. Energy Use Before, During, and After an EE Project is Installed8

Most public comments on this topic affirmed that the protocols and procedures already in place and

widely used should indeed be leveraged in the EPA’s final emission guidelines. Commenters also

                                                            

7 See discussion beginning on p. 34 of the State Plan Considerations TSD for the Clean Power Plan Proposed Rule:

http://www2.epa.gov/carbon‐pollution‐standards/clean‐power‐plan‐proposed‐rule‐state‐plan‐considerations. 8 Ibid.

Deleted: 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

4

provided numerous examples of well‐regarded oversight approaches, quality‐control practices, and

quantification and verification methodologies that are currently applied (in addition to pointing out

needed improvements to EM&V practices, especially relating to cross‐jurisdictional inconsistencies in

the way EM&V is applied).   

In response to this feedback, the EPA is finalizing EM&V requirements that are consistent with and

leverage existing EM&V infrastructure.  These requirements also reflect several overarching objectives

and principles that were emphasized by the commenters. One of these is the importance of striking a

reasonable balance between EM&V rigor and accuracy on the one hand, and evaluation costs and effort

on the other. Other objectives for the EPA’s

requirements are to avoid excessive interference

with EM&V practices that are already robust,

transparent,9 and working well, and to support

the continued evolution of credible EM&V

practices into the future.

Finally, in establishing its requirements, the EPA

also considered the fact that demand‐side EE is

not a component of the BSER underlying the CO2

emission performance rates, but instead provides

an additional compliance option.  Therefore, the

requirements – as well as this draft guidance –

are consistent with “best practice” levels of

EM&V, as necessary to meet the criteria set out

for EPA approval of a state’s rate‐based emission

limitation plan.  

1.4 Applicability and Scope of this Guidance

Document  

This document establishes guidance to help

states, affected EGUs, and EE providers (including

the firms they hire to quantify and verify savings)

implement the EPA’s requirements for

quantifying and verifying MWh savings. The

document is therefore applicable only in the

context of rate‐based state plans—including

emission standards and state measures plan

types—that explicitly credit MWh savings in the

form of ERCs or other denominator rate

adjustments.  

                                                            

9 E.g., providing an opportunity for public involvement in planning activities, internet access to EM&V plans and

reports, and disclosure of relevant inputs, methods, and assumptions.

What is Not Covered

This document provides guidance on the best

practices, technical assumptions, and industry‐

standard protocols and procedures that can be

used to quantify and verify MWh savings from

demand‐side EE.  

Therefore, this document does not:

• Apply to zero‐ or low‐emitting measures other

than EE.

• Prescribe which demand‐side EE measures are

eligible.

• Provide criteria for projecting the impacts of

such measures.

• Offer information or resources about

calculating displaced CO2 emissions from EE.

• Address the accounting procedures for how a

quantified and verified MWh can be used to

adjust a rate, or for how cross‐jurisdictional

issues are addressed.

• Provide information on other state plan

requirements, such as the operation of ERC

tracking or emission trading systems.

Information about these aspects of state plan

submissions is provided in Section VIII of the

emission guidelines.

Deleted: <object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

5

This EM&V guidance does not apply to states electing to submit mass‐based plans in which compliance

is based on monitored emissions at the stack10. Therefore, mass‐based plans do not require explicit

adjustments to emission rates based on demand‐side EE.

1.5 How to Use this Document

The document provides guidance for addressing many – but not each – of the full set of EM&V

regulatory requirements (i.e., some of the requirements are self‐explanatory and do not call for further

guidance). As such, this document is intended to supplement, rather than replace, the need for a

thorough review of the requirements. The complete list of requirements for quantifying and verifying

MWh savings from demand‐side EE is provided in Section [X] of the emission guidelines.

This guidance is organized and intended to be used, as follows:

• Section 2 provides a high‐level discussion of 12 key EM&V topics and establishes guidance to help EE

providers develop EM&V plans that document how the applicable regulatory requirements will be

addressed. The guidance in this section applies to all demand‐side EE used to generate ERCs and

otherwise adjust an emission rate.  

• Section 3 establishes additional EM&V guidance—beyond the general guidance provided in

Section 2—for several common EE program and project types.  The specific program and project

types addressed include:  

- Programs implemented using utility customer funds (“utility EE programs”)

- Individual or aggregated EE projects, such as those implemented by ESCOs or at industrial

facilities (e.g., water treatment plants)

- Building energy codes

- Appliance energy standards

• Appendix A provides a brief glossary of key terms used in this document. The definitions are

consistent with industry‐standard protocols and reference documents, and can be supplemented

with the more complete glossary provided in the SEE Action Energy Efficiency Impact Evaluation

Guide.11  

• Appendix B provides templates for program and project EM&V plans. These templates can be used

(but are not required) to indicate how the relevant approaches, methods, and parameters required

by the emission guidelines will be applied over the course of program and project implementation.

As described below, EM&V plans are a required component of a qualification package used to

generate ERCs from eligible resources.  

• Appendix C provides an illustrative example of how the high‐level guidance on measurement and

verification in Section 2.1 can be applied in practice for several common technology types. Appendix

                                                            

10 It should be noted that states with mass‐based plans may have PUC‐led EM&V requirements to ensure that

consumer‐funded EE programs are cost‐effective and achieve incremental savings. 11 State and Local Energy Efficiency Action Network. 2012. Energy Efficiency Program Impact Evaluation Guide.

Prepared by Steven R. Schiller, Schiller Consulting, Inc., www.seeaction.energy.gov.

Commented [A4]: X needs to be defined, and it would be 

good to see this document “emission guidelines” and be able 

to cite to it. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

6

C addresses the scope, analytic steps, and best practices associated with measurement and

verification for these EE actions.  

While this document is not an EM&V primer, it provides discussion and references to several industry‐

standard protocols, guidelines, and other resources such as the State and Local Energy Efficiency Action

Network (SEE Action) EM&V Web portal12 and the EPA’s emission guidelines page.13

1.6 Intended Audience  

This guidance is intended for state officials, providers of demand‐side EE, and private firms (e.g.,

evaluators and verifiers) hired to help execute the regulatory requirements for quantification and

verification of EE savings. For state air regulators, lay readers, and the public, the high‐level descriptions

of the 12 topics addressed in Section 2 may be helpful.  These descriptions are followed by a discussion

of applicable guidance that EE providers, evaluators, and verifiers can use as they develop EM&V plans

and implement the relevant regulatory requirements. The guidance can also be used by non‐

government organizations, the public, and other parties interested in better understanding the EPA’s

EM&V requirements for purposes of participating in state plan development and similar processes.  

1.7 Evolving EM&V Practices and Guidance Updates  

The EPA recognizes that the best‐practice EM&V approaches, protocols, and procedures that are now

used by states, EE providers, and others – and upon which this guidance is largely based – will evolve

and improve over time as new technologies emerge and the EE marketplace changes. To ensure that

this guidance continues to be relevant and helpful over time, the EPA intends to periodically provide

updates that reflect changes to what is considered best practice. The agency may also release

complementary guidance on how “top‐down” (econometric) EM&V methods can be applied in the

context of state measures plans that incorporate demand‐side EE outside of an ERC trading program.

Such guidance would define an aggregate verification approach that demonstrates how states could

continue to make strategic EM&V investments while ensuring that all EM&V‐related regulatory

requirements in the emission guidelines are satisfied.14   

   

                                                            

12 See: https://www4.eere.energy.gov/seeaction/evaluation‐measurement‐and‐verification‐resource‐portal 13 See: http://www2.epa.gov/carbon‐pollution‐standards   14 This guidance would apply to states that elect to incorporate demand‐side EE outside of an ERC trading program,

and that manage their EE policies, programs and projects as a portfolio. EPA anticipates that a combination of

elements will be required to demonstrate adequate stringency of verification at the portfolio level and will be

issuing further guidance
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

7

2. EM&V Guidance Applicable to All Demand‐Side EE

This section includes EM&V guidance for 12 key topics that is applicable to all demand‐side EE that is

used to generate ERCs or otherwise adjust an emission rate.  It includes high‐level background

information and provides guidance and definitions to help implement these requirements. It also

provides a set of best practices, technical assumptions, and links to key industry‐standard protocols and

resources to assist states, EE providers, and private firms hired to help quantify and verify savings. As

previously described, the guidance provided here leverages and is consistent with current EM&V best

practices, protocols, and procedures that are already used by the majority of states across the country.

This guidance therefore balances the need for rigor and accuracy with the effort and cost associated

with quantification and verification.  

This guidance considers a best‐practice method to be one that either (1) is consistent with established

protocols or guidelines that have been developed with expert and stakeholder input; or (2) uses

alternative methods that are well documented and have been subjected to and endorsed by peer

review, but are not documented in established protocols or guidelines. The best practice methods

recommended here generally fall into the first category.

This section of the guidance is supplemented with additional details and discussion of best‐practice

approaches and technical assumptions applicable to common EE program categories (Section 3) and an

illustration of how these approaches should be applied to certain EE project and measure types

(Appendix C). For the 12 EM&V topics addressed below, the following information is provided:

• Discussion: This section includes a high‐level overview, relevant background information, and

applicability to the EPA’s emissions guidelines.

• Applicable Guidance: This information is intended to help states, EE providers, and private firms

hired to quantify and verify EE savings implement the regulatory requirements defined in the

emission guidelines, as reiterated here. These approaches and assumptions expand upon, illustrate,

and provide practical clarification of the preamble and emission guidelines and are based on

industry best practices.  

• Key Definitions: Text boxes are provided with definitions of key EM&V terms and concepts,

consistent with industry‐standard usage.

The EM&V topics covered in this section include:

1. EM&V methods  

2. Electricity savings metrics and baselines

3. Reporting timeframes and considerations

4. Deemed savings

5. Independent variables

6. Uncertainty  

7. Avoiding double counting Deleted: of
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

8

8. Effective useful life and persistence of savings

9. Savings quantification and verification  

10. Transmission and Distribution System Savings Adders

11. Interactive effects

12. Use of EE EM&V Protocols and Guidelines

2.1 EM&V Methods  

2.1.1. Discussion

Three general categories of EM&V methods for quantifying the MWh savings include deemed savings;

measurement and verification; and, comparison group methods,. Each of these methods is defined in

industry‐standard protocols and commonly applied by EE providers, management bodies, and

evaluation firms. The decision of which method to apply involves consideration of factors such as

objectives of the EE activity being evaluated, the scale of the activity, and evaluation budget and

resources.  Measurement and verification and deemed savings are commonly used for determining

savings from individual EE measures and projects. By

contrast, comparison‐group methods are usually

only used to estimate savings from EE programs, but

the use of such methods could be expanded further.  

A variant on large‐scale consumption data analysis is

the approach of using macroeconomic models,

sometimes known as “top‐down” EM&V. Top‐down

EM&V methods to evaluate portfolios of EE

programs involves using (1) aggregate (e.g., utility

service area, county, census block, etc.) energy use

or per‐unit energy consumption indices (e.g., energy

consumption per‐unit of output or per capita) and

(2) energy‐use driver data (e.g., income, prices,

population) to determine savings from portfolios of

programs. While this approach is not currently

recommended for use in generating tradable ERCs

from EE, it may provide a useful tool for verifying

savings that are calculated based on other

approaches.  It could also be useful for mass‐based

states that want to ensure they are getting good

overall impact out of their efficiency efforts.

2.1.2. Applicable Guidance

The regulatory requirements for EM&V methods are

described in Section [X] of the final emission

Key Definitions

 Deemed savings values are estimates of electricity savings for a

single unit of an installed EE measure that (1) has been

developed from data sources (such as prior metering studies)

and analytical methods that are widely considered acceptable

for the measure and purpose, and (2) is applicable to the

situation under which the measure is being implemented.

Common sources of deemed savings values are previous

evaluations and studies that involved actual measurements and

analyses. With deemed savings, the per‐unit MWh values are

determined and agreed to by parties prior to EE

implementation.  

 Measurement and verification is the process of determining

savings from an individual EE measure or project (versus an EE

program). The IPMVP, described in Section 3.12, defines two

retrofit isolation and two whole‐facility M&V options used in

the EE industry:

- Retrofit isolation – assessing savings from each EE

measure individually (IPMVP Options A & B).  

- Whole facility – analyzing savings from each EE measure

in a project/facility collectively (IPMVP Option C, review

of energy bills or Option D, calibrated simulation).

- Some combination of the above.  

 Comparison group EM&V methods determine program savings

by taking the difference between a comparison group’s

electricity consumption and the consumption of program

participants. Comparison group approaches include

randomized control trials (RCTs) and quasi‐experimental

methods using nonparticipants. These methods are generally

used to estimate program‐level savings versus facility‐ or

project‐level savings.

Deleted: <object>

Deleted: and

Deleted: ,

Deleted: and deemed savings

Commented [A5]: To track the more logical ordering in 

the text box 

Deleted: states pursuing a rate‐based state measures plan

that does not rely on trading

Deleted: may find it useful as tool for verifying savings

Deleted: 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

9

guidelines.  The following guidance is provided to help EE providers develop EM&V plans that document

how these requirements will be implemented.   

 Determine which of the above allowable methods to apply based on the type of program, expected

savings magnitude, and the program’s delivery mechanism when selecting the appropriate EM&V

method for quantifying savings.

• Wherever feasible and practical, apply quasi‐experimental design or randomized control trial (RCT)

approaches that make use of comparison groups to develop accurate and unbiased savings

estimates. For these approaches, apply best‐practice protocols and guidelines such as those

published by the SEE Action Network and the U.S. DOE’s Uniform Methods Project.

• Follow the guidance provided in Section 2.4 below on Deemed Savings, which provides definitions,

suggested approaches, and technical assumptions, and substitute empirical approaches wherever

feasible.

• Apply the guidance in Section 2.12 below on Use of EE EM&V Protocols and Guidelines, which

presents additional guidance on applying and referencing industry‐standard protocols and

guidelines.

 Refer to Appendix C of this document as an example of how these methods can be applied in

practice.  

 When applying measurement and verification methods, use one or more best‐practice protocols

and guidelines.  Examples include:

- International Performance Measurement and Verification Protocol (IPMVP, an international

M&V guidance document),15  

- Federal Energy Management Program (FEMP) protocols and guidelines,16

- American Society of Heating, Refrigeration, and Air‐Conditioning Engineers (ASHRAE) protocols

and guidelines17, and  

- U.S. DOE, The Uniform Methods Project (UMP): Methods for Determining Energy Efficiency18  

2.2 Electricity Savings Metrics and Baselines

2.2.1. Discussion

Electricity savings from an EE activity is the difference between observed usage with EE in place and the

usage that would have occurred in the absence of EE during the same time period, as shown in Figure 1‐

1 above. Defining what would have happened without the EE program or project (i.e., the

“counterfactual baseline conditions”) represents the fundamental challenge to quantifying savings, and

therefore warrants a comparatively detailed discussion.  Baselines are typically defined by the installed

equipment at the time of retrofit, or some other standard alternative to the installed EE technology. The

EPA, for purposes of the emission guidelines, defines EE savings as the difference between observed

                                                            

15 http://www.evo‐world.org 16 http://www1.eere.energy.gov/femp/pdfs/mv_guidelines.pdf 17 https://www.ashrae.org/home 18 http://www.energy.gov/eere/about‐us/ump‐protocols (version as of August 2015)

Commented [A8]: Label to citation, as this get more 

credibility as an EPA-led action than what is implied as a 

public advocacy group. 

Commented [A9]: For cases where multiple protocols are 

available, can we indicate which of these is preferable? For 

example, should we require UMP where available? 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

10

electricity usage and an appropriate “common practice baseline” (CPB).  Such CPBs are specified below

with respect to different EM&V methods and EE activities.   

One benefit of using the CPB is that it inherently adjusts the baseline over time to reflect market

conditions and naturally occurring improvements in efficiency over time.  By adjusting the baseline and

thereby capturing naturally‐occurring EE improvements, a well‐defined and consistently applied CPB

avoids “over‐crediting” EE program and project activity.  It also means that, for consumer‐funded EE

programs, the EPA can specify the reporting of “gross savings,” which is a familiar and readily‐available

reporting metric.  Reporting gross savings with

respect to a CPB, however, is sometimes referred

to as “adjusted gross savings”.  This is because the

application of a CPB inherently captures the

naturally occurring (business as usual) EE savings

that is only partially accounted for with the “gross

savings” metric typically used to quantify

consumer‐funded EE programs. 19

2.2.2. Applicable Guidance

The regulatory requirements for electricity savings

metrics and baselines are described in Section [X]

of the final emission guidelines.  The following

guidance is provided to help EE providers develop

EM&V plans that document how these

requirements will be implemented.   

• When a well‐designed comparison group

method is used, use the control group (with

randomized control trials) or the

comparisoncomparisoncomparisson group of

non‐participants (with quasi‐experimental

approaches) to quantify the CPB electricity

consumption.  

- In these cases, separately determining the

CPB efficiency of individual pieces of

equipment is unnecessary.20  

                                                            

19 Most states PUCs apply a “net‐to‐gross (NTG) analysis” to their consumer‐funded EE programs to ensure that

consumer funds are prudently expended, and to determine whether a particular EE program caused savings at a

particular residence, building, or facility to occur.  For discussion and guidance on determination of net and gross

savings, see the Uniform Methods Project Estimating Net Energy Saving: Methods and Practices.  Net savings

attempts to account for free ridership, spillover, and market effects (see Key Definitions text box). 20 The comparison and control groups, in most cases, intrinsically account for free ridership and participant

spillover, but do not account for nonparticipant spillover and long‐term market effects.

Key Definitions

 Electricity savings: change in electricity use resulting

from the implementation of EE measures and

practices, reported in units of megawatt‐hours (MWh)  

 Baseline: conditions (including electricity consumption)

that would have existed without implementation of the

subject EE activity. Baseline is used to estimate

measure‐, project‐ and program‐related savings.

 Common practice baseline (CPB): default technology

or condition that would have been in place at the time

of project implementation without a decision to install

a more efficient system or measure, for example, the

system(s) or measure(s) a typical consumer/building

owner would have continued with or installed at the

time of project implementation.  

 Free ridership: portion of a program’s supported

savings that would have occurred in the absence of the

program.  

 Spillover: additional reductions in energy consumption

that are due to an EE program’s influences beyond

those directly associated with the installed EE

measures—these include participant and

nonparticipant spillover.  

 Market effects: change in the supply chain for energy

efficient equipment or the behavior of participants and

market actors resulting from market intervention(s).  

Deleted: <object>

Commented [A10]: Under key definitions, I would expect 

to see “rebound effect” 

Commented [A11]: ? X 

Commented [A12]: In both cases, they are a control 

group, doesn’t matter if it’s an RCT or a quasi-experiment 

Deleted: compassion

Deleted: s

Deleted: i

Deleted: s

Deleted: i
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

11

- Design the comparison group and analysis approach in a way that represents what the

participants would have done absent the program or absent the EE intervention.  

- With comparison group specifications, describe how the comparison group was selected and

what it is intended to represent.  

• For the three EM&V methods addressed in Section 3.1, use the following CPB guidance:

- For equipment that is replaced on failure, define the baseline as:  

 The federal standard or the market average industry/consumer practice at the time of

implementation, whichever results in more conservative savings value. This approach

recognizes the dynamic nature of baselines in the context of changing market conditions..

 For states (or EE providers operating in those states) that have product standards that are

more stringent than the federal standard or market average, where the state is counting the

savings increment due to the more stringent product standard as a state measure, use the

state product standard as the CPB for EE at higher efficiency than the state product

standard. (See double counting Section 3.8)  

- For early replacement activities, with strong evidence that replacement of functioning

equipment is due to program influence, a dual baseline is applicable:  

 Use existing conditions for defining the baseline for the remaining useful lifetime21 (RUL) of

the replaced equipment or system.  Use the CPB that would apply to new construction or

replacement on failure for the remainder of the new equipment EUL.

- For building shell improvements of existing buildings, define the baseline as:

 The existing condition of the building shell unless renovations are extensive enough to

trigger new construction code compliance, in which case the following new construction

guidance applies

- For new construction, the baseline is defined as:  

 For commercial buildings, the more conservative of the applicable state or local building

code in effect as of January 1, 2013, the market industry average practice in the state, or

ASHRAE 90.1‐2007/2009 IECC. This baseline recognizes the dynamic nature of baselines in

the context of changing market conditions.22

 For residential buildings, the more conservative of the applicable state or local building code

in effect as of January 1, 2013, the market industry average practice in the state, or ECC

2009. This baseline recognizes the dynamic nature of baselines in the context of changing

market conditions.

                                                            

21 Remaining useful life cannot exceed the effective useful life of the replacement measure. 22 Implementation of the EE project itself should not be a determinant for code or standard applicability. For

example, if the project is a lighting retrofit and the lighting retrofit itself triggers a code requirement for upgraded

lighting, the code would not necessarily be an applicable basis for a CPB. In this situation, the code would only be

applicable as a CPB, if other, substantial, actions to the subject building (e.g., a major remodel) triggered the

subject code.

Commented [A14]: This paragraph is a bit confusing to 

me. Perhaps elaborate? 

Commented [A13]: These sentences seem contradictory. 

Isn’t the first sentence prescribing a static baseline as 

opposed to a dynamic (and more realistic) baseline? 

Commented [A15]: This program runs through 2030 so 

there will be multiple rounds of future federal code 

determinations by then. As these future updates are made, 

suggest using the most current federal code as the CPB. Of 

course, states have the option to decide how quickly to adopt 

new codes once DOE determines they are cost effective, but 

there is no reason that states should receiving EE credits for 

simply adopting codes that DOE has already determined are 

cost effective based on direct energy cost savings alone. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

12

 For states (or EE providers in those states) that are increasing the stringency of their state

building codes, use the updated state code as a baseline if the state has included and is

already counting those savings separately. (See double counting discussion in Section 3.8)

Table 7.1 of the 2012 SEE Action Impact Guide can be referred to for defining general approaches to

determining baselines for specific EE activities.23

2.3 Reporting Timeframes and Considerations  

2.3.1. Discussion

Current industry practices for reporting

annual electricity savings varies across states

and EE providers with regard to specific

content requirements, definitions, and timing.

Local policy objectives, the breadth of EE

activities, and other factors typically drive

decisions about such reporting attributes.   

The reporting process for consumer‐funded

EE programs typically involves projecting and

reporting savings as part of the planning

phase, as well as ongoing annual reporting as

savings accrue over time.  This “ex‐post”

reporting is typically based on savings derived

by applying one of the EM&V methods

described above in section 3.1.  In almost all

cases, ongoing annual savings values are reported on an incremental and cumulative basis (see text‐box

definitions).  

The regulatory requirements for EE reporting, as defined in the EPA’s emissions guidelines, are intended

to support greater consistency across jurisdictions and EE providers in how savings data are quantified

and collected. In addition, the requirement to report both incremental and cumulative savings, and the

associated timing provisions, are consistent with the Energy Information Agency’s (EIA) approach to

collecting EE program data via Form 86124.  

2.3.2. Applicable Guidance

The regulatory requirements for reporting timeframes and considerations are described in Section [X] of

the final emission guidelines.  The following guidance is provided to help EE providers develop EM&V

plans that document how these requirements will be implemented.   

                                                            

23 State and Local Energy Efficiency Action Network. 2012. Energy Efficiency Program Impact Evaluation Guide.

Prepared by Steven R. Schiller, Schiller Consulting, Inc., www.seeaction.energy.gov. (Or current version.) 24 See: http://www.eia.gov/electricity/data/eia861/

Key Definitions

 Reporting‐year incremental savings: the

electricity savings quantified and verified as a

result of EE activities operating for the first time

in the reporting year; reported in annual

megawatt hours (MWh).  

 Reporting‐year cumulative savings: the

electricity savings quantified and verified as a

result of EE activities in the reporting year from

both (a) EE activities operating for the first time in

the reporting year and (b) EE activities initiated in

prior years and still in effect; reported in

megawatt hours (MWh).

Deleted: <object>

Deleted: breath
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

13

• When determining the appropriate interval for applying EM&V methods (per Section 2.1), consider

the following: the complexity of the program or project, the variability of the savings, the EM&V

method being applied, and the relative scale and magnitude of savings.

• When using stipulated effective useful life (EUL) values (see subsection on Effective Useful Life and

Persistence of Savings) for reporting savings, report savings pro rata based the day the EE activity

began saving energy during the reporting year.  

• Report current year incremental and cumulative savings values on the basis of the best available

information at the time of reporting. Incorporate new information that becomes available during

the course of conducting EM&V over the project or measure lifetime, but only do so on a going‐

forward basis.  

- See text box that illustrates how forward adjustments can be applied.

2.4 Deemed Savings  

2.4.1. Discussion

As described in section 3.1 above, one common method for quantifying savings is the application of

“deemed savings values”. Deemed values are only appropriately applied to relatively simple, well‐

defined efficiency measures such as light bulbs or equipment for which the performance characteristics

and location‐specific conditions are well known. In these instances, a deemed savings approach can

provide reasonable reliable and accurate savings values, on average.  

Because these stipulated values are agreed upon in advance, deemed savings can help alleviate some of

the guesswork in program planning and design. Deemed savings can also result in a form of risk, in

which overestimates or underestimates of savings can occur when EE projects or measures do not

perform as expected, or if deemed values are applied in the wrong circumstances.  The verification

Example of Forward Adjustments to EE Savings  

If a program is determined to save 10,000 MWh per year for 10 years starting in Year X, 10,000 MWh may

be used as the amount of incremental savings in Year X and 10,000 MWh may be used for reporting

cumulative savings in Years X + 1 through X + 9. If during Year X + 3 a better annual savings value for that

program is determined to be 9,500 MWh, 9,500 MWh should be used for reporting cumulative savings

starting in Years X + 3 through X + 9. Examples of significant new information that could cause an update in

savings values include:

 Savings are adjusted because of verification activities; for example, savings might be zeroed for EE

actions in facilities that have been shut down or savings are reduced if high level of savings

degradation/performance is determined.  

 Savings are adjusted because of new cycles of savings determination (see subsection on verification

and evaluation cycles); for example, savings are updated when an applicable deemed savings value is

modified/updated in an approved database.

 Savings values are updated when errors are found in prior‐year savings calculations.  

Deleted: “annualized”

Deleted: as if they started accruing on January 1 of the

reporting year and continued in the program for a full year

(irrespective of which

Deleted: )

Deleted: ).

Deleted: <object>

Commented [A16]: Can we say something along the lines 

of that, as time goes on and EM&V progresses as a field, we 

expect that deemed savings will be replaced by more 

accurate techniques and will not represent “best practice” in 

many areas? 

Deleted: most

Commented [A17]: I think we need to be careful here. In 

many cases, even well-known cases, the deemed savings 

results are different than results based on actual energy bills. 

I strongly suggest including some wording here that 

indicates that deemed savings values should be adjusted 

when they are confirmed by experimental and quasiexperimental studies. 

This recognizes that evaluation is a rapidly developing field. 

Deleted: ; in effect, they minimize one type of risk by

providing certainty
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

14

activities commonly required by state PUCs and other

oversight and management bodies are essential to alleviate

this risk25.   

For more complex efficiency projects or for projects with

significant savings variability, measurement‐based methods

(i.e., measurement and verification, and large‐scale

consumption data analyses) are more appropriate than

deemed savings. However, deemed values can in some cases

be properly applied to these projects if one or more key

parameters is directly monitored over time at the project site.

For example, in a high‐efficiency motor program, actual

operating hours could be monitored over a full work cycle26.  

Deemed savings values are usually documented in resources

such as a document or database in formats that can vary from

a spreadsheet to an online searchable database. The term of

art for such resources is technical reference manuals (TRMs)27.

As of this document’s publication, approximately 20 TRMs are

in use across the United States at the state and regional level.28

TRMs can be valuable, but methodologies for estimating savings and the actual values vary widely. Some

TRMs include information based on prior‐year evaluations including, in some cases, rigorous metering,

and analysis. Many others have values based on computersimulations or engineering algorithms. Ongoing

and new state, regional, and federal efforts to improve the quality and documentation of TRMs are

encouraged and can support higher‐quality savings values for compliance with the EPA’s emissions

guidelines and reduced EM&V costs.

                                                            

25 The emission guidelines include regulatory requirements for verification, as described in section 3.9 of this

document.

26 Deemed savings can also be used together with monitoring of one or two key parameters in an engineering

calculation. For example, in a high‐efficiency motor program, actual operating hours could be monitored over a full

work cycle. This approach is consistent with the definition of IPMVP Option A.  However using deemed savings

alone is not considered a measurement and verification method, per se.

27 TRMs generally include information about the technology or measure itself, the savings value, measure life, peak

load impacts, engineering algorithms to calculate savings, source documentation, specific assumptions, and other

relevant material to support the calculation of savings in particular applications.

28 Many of these TRMs are listed in the following document: Scoping Study to Evaluate Feasibility of National

Databases for EM&V Documents and Measure Savings. SEE Action Network. Prepared by Jayaweera, T.; Haeri, H.;

Lee, A.; Bergen, S.; Kan, C.; Velonis, A.; Gurin, C.; Visser, M.; Grant, A.; Buckman, A.; The Cadmus Group Inc.

www.eere.energy. gov/seeaction/pdfs/emvscoping__databasefeasibility.pdf. June 2011.

Key Definitions

 Deemed savings values: estimates of

average annual electricity savings for a single

unit of an installed EE measure that (a) has

been developed from data sources and

analytical methods widely considered

acceptable for the measure and (b) is

applicable to the situation and conditions in

which the measure is implemented.

Individual parameters or calculation methods

also can be deemed, including EUL values.

Deemed savings values are applicable for

specific EE measures. A single deemed

savings value may not be used for a program

as a whole, nor for a multi‐measure project,

because of the degree of variation in how

systems are used in different building types

or market segments. Deemed savings are

also called stipulated savings.

Deleted: can

Deleted: the exclusive use of

Deleted: <object>

Deleted: analyses (e.g., using

Deleted: )
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

15

2.4.2. Applicable Guidance

The regulatory requirements for deemed savings are described in Section [X] of the final emission

guidelines.  The following guidance is provided to help EE providers develop EM&V plans that document

how these requirements will be implemented.   

• Each deemed savings value must indicate the conditions for which the value is applicable (e.g.,

climate, building type, end use, and measure implementation mechanism).  

• Ensure that deemed savings values:

- Are based on EE measure definition, applicability conditions, assumptions, calculations, and

references that are well documented in work papers that are publicly available;

- Are quantified as the most likely averages of electricity savings and other factors that determine

such values over the lifetime of the EE measure, such as average occupancy, typical weather,

typical operating hours and EUL;  

- Are developed by independent, third parties, and, whenever possible, are based on empirical

techniques such as RCTs and quasi‐experimental design.  

• Apply deemed savings values only for the specific EE measures, program or project delivery

mechanism (e.g., direct install versus point‐of‐sale rebates), end‐uses (e.g., single‐family versus

multifamily residential), operating conditions (e.g., climate, operating hours), and other assumptions

for which the values were developed.

• Update deemed savings values on a going‐forward basis; see subsection on Reporting Incremental

and Cumulative Savings.

2.5 Independent Factors Affecting Energy Consumption and Savings

2.5.1. Discussion

Observed changes in electricity consumption

are the result of a variety of independent

factors and influences. These range from the

outdoor temperature to occupancy levels in a

building to production levels at a factors. To

isolate the electricity savings that result from

the EE activity in question, each of these

independent factors must be taken into account.   

Determining the influence of these factors is critical to the credibility of savings estimates, and

distinguishes properly determined savings from a simple and unreliable comparison of electricity use

before and after implementation of a program, project, or measure. Each of the three EM&V methods

described in Section 3.1 has a mechanism for accounting for independent factors. For deemed savings

values, independent factors are implicitly quantified through the associated applicability conditions (see

Section 3.4); for project‐based measurement and verification, these factors are considered via the use of

regression analyses, computer simulation modeling, or engineering calculation adjustments; and for

Key Definition

 Independent factors: the factors (e.g., weather or

occupancy) that affect the variable under study

(e.g., electricity consumption) and can vary

independently of the variable under study.

Key Definition

 Independent factors: the factors (e.g., weather or

occupancy) that affect the variable under study

(e.g., electricity consumption) and can vary

independently of the variable under study.

Deleted: Document the

Deleted: must

Deleted: be

Commented [A18]: Here is where we might mention that 

we should encourage verification of deemed savings values 

with quasi-experimental and experimental methods, where 

practical and feasible. 

Deleted: (retroactive updating is not required

Deleted: )

Deleted: <object><object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

16

comparison group methods, through large‐scale consumption data analyses that often include

regression analyses.  

2.5.2. Applicable Guidance

The regulatory requirements for independent factors affecting energy consumption and savings are

described in Section [X] of the final emission guidelines.  The following guidance is provided to help EE

providers develop EM&V plans that document how these requirements will be implemented.   

• Within a single EE program, quantify savings for the constituent EE projects or measures using the

same assumptions for independent factors.  

• Quantify EE savings on the basis of independent factors, as follows:

- Actual conditions that exist over the period when EE savings occur (i.e., real‐time conditions

during the EUL).  

 With this approach, adjust baseline electricity‐consumption data to reflect actual

independent factors observed via real‐time savings quantification.  

- Normalized or standardized (typical) conditions that can be reasonably expected to occur

throughout the EUL.  

 With this approach, both baseline and performance period data on electricity consumption

must be normalized to data on the independent factors, where reasonable and appropriate.

Examples of normalized independent factors are:

o Typical weather conditions for a residential heating efficiency project,

o Typical occupancy rates for a commercial building lighting efficiency project, or

o Typical product production rates for an industrial efficiency project.

• Where first‐year savings values – derived by applying first‐year independent factors – are used to

represent annual savings for the EUL of the project or measure, provide a justification for why this is

a reasonable assumption (i.e., justify why first‐year independent factors can be shown to represent

standard/typical conditions).
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

17

2.6 Accuracy and Reliability of Quantified Savings

2.6.1. Discussion

EM&V plans should include information on how the accuracy and reliability of EE savings will be

determined based on the selected EM&V methods, as discussed in section 3.1. These level of accuracy

and reliability of quantified and verified MWh

savings values are determined by the rigor of

the EM&V methods applied, and the

associated magnitude of two types of error:  

• Systematic error: inaccuracies in the

same direction, causing savings values to

be consistently either overstated or

understated. Systematic errors may

result from incorrect assumptions, a

methodological issue, or a flawed

reporting system.

Random errors: errors occurring by

“chance” that may cause the savings

estimates to be inconsistently overstated

or understated. Random errors include

an observed change in energy use due to

unaccounted‐for factors that affect

energy use. The magnitude of random

error can be quantified based on the

variations observed across different

units.  

By contrast, systematic error or bias is not quantifiable based on the observations. For this reason, it

is important to report the quantifiable random error, and to describe the steps that have been taken

to minimize the potential for systematic error, and provide a subjective assessment of the potential

effects of such error.  

2.6.2. Applicable Guidance

The regulatory requirements for accuracy and reliability of quantified savings are described in Section

[X] of the final emission guidelines.  The following guidance is provided to help EE providers develop

EM&V plans that document how these requirements will be implemented.   

 Quantify savings using input assumptions that produce the “most likely” MWh values that are

neithernot overly optimistic.  

• When using sampling for quantifying savings values, ensure that:

- The population estimates derived from sampling achieve 10 percent relative precision with no

less than 80 percent confidence intervals.

Key Definition

 Rigor: the level of effort expended to minimize

uncertainty in savings estimates such as sampling

error and bias. The higher the level of rigor, the more

confident one is that the results of the EM&V

activities are both accurate and precise. Rigor may be

quantified or described with metrics for reliability and

certainty, such as statistical confidence and precision.

Key Definition

 Accuracy: A concept that refers to the relationship

between the true value of a variable and an estimate

of the value. The term can also be used in reference

to a model or a set of measured data, or to describe a

measuring instrument's capability.  

 Reliability: The quality of a measurement process that

would produce similar results on (1) repeated

observations of the same condition or event, or (2)

multiple observations of the same condition or event

by different observers. Reliability refers to the

likelihood that the observations can be replicated.  

 Rigor: the level of effort expended to minimize

uncertainty in savings estimates such as sampling

error and bias. The higher the level of rigor, the more

confident one is that the results of the EM&V

activities are both accurate and precise. Rigor may be

quantified or described with metrics for reliability and

certainty, such as statistical confidence and precision.

Deleted: <object><object>

Deleted: is

Deleted: not

Deleted: otnot overly conservative

Deleted: rnor

Commented [A20]: There should be a bias towards 

conservatism here. 

Commented [A21]: Could these ranges be tightened? E.g. 

to 5% precision and a 90% CI? 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

18

- Actual confidence/precision values are applied, versus ex‐ante estimates of

confidence/precision.

• Apply and cite applicable industry‐standard protocols and guidance documents for sampling. Best

practices for statistical sampling are described in:

- ISO New England Manual for Measurement and Verification of Demand Reduction Value from

Demand Resources.29  

- The California Evaluation Protocols.  

- Uniform Methods Project Sample Design Cross‐Cutting Protocol (Chapter 11).  

• For deemed savings:  

- Quantify random errors if applicable

- Include the statistical precision of any EM&V parameters determined using sampling

• If sampling is used to determine savings for a project or program, report the statistical precision of

the associated estimates. Examples include:

- Measurement and verification of the program savings, based on verification for a sample of

project and installed measures.   

- A program‐level estimate of savings or of key parameters used to calculated savings (e.g.

lighting operating hours) based on quantification conducted for a random sample of units.

- For savings determined by comparison group methods, the statistical precision of the program

effect measured by the comparison group analysis should be reported. Applicable guidance

documents include:  

 SEE Action Guide on evaluating behavior programs,30

 Uniform Methods Project Sample Design Cross‐Cutting Protocol (Section 11)31, and  

 Uniform Methods Project Whole‐Building Retrofit Evaluation Protocol (Section 8)32

• For all EM&V methods, discuss potential risks and biases (and associated quality‐control measures)

in EM&V plans and monitoring and verification reports.  

2.7 Avoiding Double Counting

2.7.1. Discussion

Double counting occurs when the MWh savings from a single EE program, project, or measure are

counted more than once under the same regulation or program. This type of error must be prevented to

                                                            

29 Current version is manual M‐MVDR, revision 6, dated June 1, 2014 30 State and Local Energy Efficiency Action Network. (May2012). Evaluation, Measurement, and Verification

(EM&V) of Residential Behavior‐Based Energy Efficiency Programs: Issues and Recommendations. Prepared by

Todd, A.; Stuart, E.; Schiller, S.; Goldman, C.; Lawrence Berkeley National Laboratory. www1.eere

energy.gov/seeaction/pdfs/emv_behaviorbased_eeprograms.pdf

31 See: http://www.energy.gov/sites/prod/files/2013/11/f5/53827‐11.pdf   32 See: http://www.energy.gov/sites/prod/files/2013/11/f5/53827‐11.pdf  

Commented [A22]: It would be helpful to define this. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

19

maintain programmatic integrity and credibility, and to ensure that EE results in real and permanent

reductions in CO2 emissions. Tracking, accounting, and quality checks are steps that are routinely

undertaken in states and regions across the country to avoid double counting. The purpose of these

steps is to avoid the following circumstances:

• Savings from a single EE program or project being claimed by more than one EGU or state. For

example:

- Two EGUs claiming the savings from the same lighting retrofit program, or an EGU claiming the

same savings that a state is claiming.

• Two or more EE programs or projects operating at the same time claiming savings from the same

MWh savings. For example:

- Some or all savings from the same retrofit being claimed by a residential behavior‐based

program and a retailer point‐of‐sale incentive program.33  

Savings from a single retrofit project being claimed by a utility incentive program and the ESCO

that implemented the retrofit.

• Two or more programs operating during different years claiming savings from the same projects or

measures, for example:

- A 2020 program incenting an LED lamp operating for 10 years, with the lamp failing after 2 years

and being replaced by a new LED lamp that receives an incentive from another program.

• Inconsistent baselines across a portfolio of programs, for example:

- A state claiming (1) in one program savings from enacting a C&S with 100‐percent compliance

that results in savings above a prior C&S or common practice, and (2) in another program

claiming savings with a baseline defined below the new C&S (e.g., a baseline defined by a prior

C&S) for the same types of EE projects or measures.

• States claiming credit for federal actions such as building code determinations or appliance

standards.

2.7.2. Applicable Guidance

The regulatory requirements for avoiding double counting are described in Section [X] of the final

emission guidelines.  The following guidance is provided to help EE providers develop EM&V plans that

document how these requirements will be implemented.   

• Implement systematic tracking and accounting procedures, including the use of well‐structured and

well‐maintained tracking and reporting systems such as those already being used by many states

and EE providers.  

• Implement the following procedures to avoid or correct for double counting:

                                                            

33 This potential for double counting is particularly important in the context of randomized encouragement

programs, where part of the savings seen in treatment‐control differences is due to increased participation in

general offering programs.

Commented [A23]: Unclear what “randomized 

encouragement program” means. Is this a typo that should 

read “randomized control trial”? 

Formatted: Bullet Level 1
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

20

- For programs and projects with identified consumers, track EE actions (type and number of

measures implemented) at the utility‐customer level using customer name, address, account

number, and date of actions for each program.  

- For programs without identified consumers, such as point‐of‐sale rebates and retailer or

manufacturer incentive programs, track applicable vendor, retailer, and manufacturer data.

Include the appropriate specifications and quantities of program‐incented units sold or shipped,

with as much granularity as possible.

- Use the consumer‐level data to identify and correct for duplicate EE activity records across

programs with “trackable” consumers and across non‐program projects such as private‐sector

transactions for projects sponsored by an ESCO.

- Identify and correct for duplicate EE activity records across programs and non‐program projects

such as private‐sector transactions for projects sponsored by an ESCO.

- Identify instances where tracked consumer activity is likely to be double counted with upstream

activity, and subtract the estimated overlap from one or the other’s savings claims.

• For programs with identified consumers but without identified EE actions (e.g., an information or

behavioral program) that apply comparison‐group EM&V, use the comparison group to determine

the incremental tracked activity in other programs.  

- The savings for this activity is reflected both in the total tracked savings for the other programs

and in the comparison‐group savings for the informational program, and should be subtracted

from one or the other total.  

- Use the SEE Action Guide or the UMP protocol for evaluating behavior programs for further

guidance on applying this method34.  

2.8 Effective Useful Life and Persistence of Savings

2.8.1. Discussion

While the full cost of an EE project or

measure is typically borne at the time of

purchase and installation, the resulting MWh

savings accrue for the full duration of time for

which the EE is in place.  The typical practice

of state PUCs (for consumer‐funded EE

programs) and ESCOs is to apply one or more

EM&V methods to measure and report EE

savings during the first year, and then to

assume a constant level of savings in each

                                                            

34 See: https://www4.eere.energy.gov/seeaction/publication/evaluation‐measurement‐and‐verification‐emv‐

residential‐behavior‐based‐energy‐efficiency

Key Definition

 Effective useful life (EUL): an estimate of the

duration of savings from individual EE measures,

reported in years. Values are typically specific to

individual EE measures or projects but also may be

specified by program. EUL is defined through various

means, including median number of years that the

measures/projects installed under a program are still

in place and operable.

Deleted: <object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

21

year for the full lifetime of the project or measure (or, alternatively, for a specified time horizon such as

10 years). This period of time is referred to as the effective useful life (EUL).35   

EUL is a function of the expected life of the equipment and the number of years that the measure

actually persists, as determined using field measurements of on‐the‐ground conditions. The methods for

determining EUL of a given measure vary somewhat across the country and by EE provider based on

observable conditions (such as weather and climate), input assumptions (such as rates of measure

failure), and other factors.36 For commonly‐implemented measure types, it is typical for state, regional,

or national entities to collaborate on mutually beneficial research to refine and improve EUL

measurements, especially in light of ongoing industry trends and technology evolution over time.   

2.8.2. Applicable Guidance

The regulatory requirements for effective useful life and persistence of savings are described in Section

[X] of the final emission guidelines.  The following guidance is provided to help EE providers develop

EM&V plans that document how these requirements will be implemented.   

                                                            

35 See: Energy Savings Lifetimes and Persistence: Practices, Issues and Data, Ian M. Hoffman, Steven R. Schiller,

Annika Todd, Megan A. Billingsley, Charles A. Goldman, Lisa C. Schwartz, Lawrence Berkeley National Laboratory,

February 2015. 36 For example, technical measure life or equipment life usually is defined as the median number of years that a

measure is installed or initiated and is operational. Less commonly, it is defined as the mean number of years to

failure. Median value means the time at which half of the measures are removed from service or are otherwise no

longer operating as assumed, and half remain operating as assumed.  

Commented [A24]: This guidance should clarify that EE 

savings should account for product innovation trends and 

related lock-in problems. For example, programs that 

encouraged swapping from incandescents to long-lived CFLs 

save energy initially, but may also have prevented certain 

consumers from realizing even deeper savings from LEDs 

that subsequently became available. 

Commented [A25]: Big space below 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

22

 Use Section 3.9 (Savings Quantification and Verification Cycles) of this document to inform first‐year

verification of whether measures are functioning and generating savings.  

 Determine deemed EUL values as the most likely averages (i.e., EULs should not be overly

conservative nor overly optimistic) using either or both:

- Documented best practices that are

determined by independent

entities and publically documented

per the requirements indicated

below in the subsection on Deemed

Savings.  

- Persistence studies conducted by

independent entities at least once

every 5 years to determine values.  

 If deemed EUL values are used, report

incremental and cumulative savings

with the assumption that savings revert

to zero at the end of the applied EUL

(after a period of time equal to the EUL

has elapsed). This implies that project

or program participants do not

continue the EE practice beyond

“measure life,” and that if a new

measure is implemented at the end of

the prior measure’s EUL, a

reassessment of the appropriate CPB is

needed.  

 Consider using regular field verifications

to supplement the use of stipulated EUL

values for measure lifetime and

persistence of savings.

- If field verification is applied on an ongoing basis, update verified savings values accordingly (i.e.,

for as long as verification indicates that savings persist).

 If stipulated EULs for specific measures are used, apply persistence and/or degradation studies to

establish average values (i.e., in lieu of conducting regular interval inspections to determine if

measures are functioning and generating savings).

Key Definitions

 Verification (project or measure): an assessment by an

independent entity to ensure that the EE measures

have been installed correctly and could generate the

predicted savings. Verification may include assessing

baseline conditions and confirming that the EE

measures are operating according to their design

intent. Site inspections, phone and mail surveys, and

desk review of program documentation are typical

verification activities.  

 Inspections: visits to facilities at which an EE project or

measure was implemented. Inspections document the

existence, characteristics, and operation of baseline or

EE project equipment and systems, and the factors

that affect energy use. Inspections may include review

of commissioning or retro‐commissioning

documentation.

 Realization rate: the ratio of electricity savings an

independent third party determines divided by the

savings expected or claimed for the measure, project,

or program provider(s). The ratio’s deviation from 1.0

can be due to (1) adjustments for data errors, (2)

differences in implemented measure counts or other

claimed information because of verification activities,

or (3) other performance differences revealed through

the M&V activities.

Deleted: <object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

23

 Participate in collaborative and joint research to improve the breadth and quality of EUL values

(several such research activities are ongoing in states around the country).

2.9 Savings Quantification and Verification Cycles

2.9.1. Discussion

Determining MWh savings from an EE activity is typically a two‐step process: (1) verifying that a

measure or project has been installed as an action that would not have occurred without the

programand (2) quantifying savings.  

To illustrate the difference between the two activities, consider a project involving replacement of

100‐watt incandescent lamps with 13‐watt LED lamps.

Verification involves confirming that the replaced lamps are 100 watts, that the new ones are 13 watts,

and that the lamps are installed and working. Verification confirms that the measure has the potential to

save electricity, with the savings depending on how many hours the lamps operate. The number of

hours the lamps operate might be determined by using a (1) deemed savings value for 13‐watt lamps

that replace 100‐watt lamps or (2) verified based on metering at a sample of projects, using the IPMVP

Option A or B.  

2.9.2. Applicable Guidance

The regulatory requirements for savings quantification and verification cycles are described in Section

[X] of the final emission guidelines.  The following guidance is provided to help EE providers develop

EM&V plans that document how these requirements will be implemented.   

• Verify a sample of projects to represent the full program population in cases where many projects

are implemented in a program (in cases where verifying every project is not feasible or practical).  

Determine sample sizes as defined in Section 2.6 on Uncertainty.

• Use verification findings to adjust annual electricity savings on a going forward basis.

• Apply the following verification strategies for these common EE activities:

- For EE retrofits, verification can include confirming (1) installation rates of the indicated EE

measures, (2) that these installations are above and beyond CPB and meet reasonable quality

standards, and (3) that the measures are operating correctly per design intent and can generate

the predicted savings.  

- For EE new construction projects, verification can include reviewing and confirming

commissioning documentation.

- For EE point‐of‐sale rebate or distributor incentive programs, verification can include

confirming the sales data used for determining electricity savings and verification with a sample

of end users.  

• Use EM&V and the derived savings realization rates to adjust claimed annual electricity savings.

Deleted: breath

Deleted: or a

Deleted: ‐driven action has occurred

Commented [A26]: Need closer attention to free riders 

issue here 

Deleted: 

Deleted: the installation

Deleted: s

Deleted: e

Deleted: distributer
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

24

- If savings quantification is conducted separately from savings verification (e.g., deemed savings

and measurement and verification methods), the savings quantification may determine savings

per verified unit.  This value can then be applied to the verified units.

- When comparison‐group methods are used, the analysis may be considered to provide a

combination of savings quantification and verification. That is, the analysis yields both what was

actually installed and operating, as well as the operational practices that affect savings. Separate

verification activities are therefore not necessary.  

 To continue to count savings after the first year with comparison group methods, establish

an EUL as described in Section 3.8.3 or continue the comparison group analysis in each year

for which savings continue to be counted.  

 If the comparison group analysis is continued in successive years, include a discussion of the

basis on which the comparison group remains appropriate and valid.  

2.10 Transmission and Distribution (T&D) Savings Adders

2.10.1. Discussion

The difference between the electricity generated (busbar value) and consumed (meter value) is due to

losses in the transmission and distribution (T&D) system. U.S. Energy Information Agency (EIA) data

indicates that national, annual T&D electricity losses average about 6 percent of the electricity that is

transmitted in the U.S.37 Every unit of electricity consumption avoided at an end‐use site also avoids

losses that would have occurred as that electricity was delivered through the T&D system. Requirements

and guidance on how T&D savings can be added to end‐use electricty savings values follow.

2.10.2. Applicable Guidance

The regulatory requirements for T&D savings adders are described in Section [X] of the final emission

guidelines.  The following guidance is provided to help EE providers develop EM&V plans that document

how these requirements will be implemented.   

 Adjust MWh savings results from EE programs, projects, and measures by using a T&D line‐loss

adjustment factor (adder) based on the lesser of 6 percent of the site‐level savings or the calculated

statewide annual average T&D loss rate (total electric supply minus direct electricity use divided by

direct electricity use)38 expressed as a percentage and based on values in the most recent year and

published in the U.S. EIA State Electricity Profile.   

                                                            

37 The national average loss from 1990 to 2012 is six percent. See:

http://www.eia.gov/tools/faqs/faq.cfm?id=105&t=3 38 Estimated losses in MWh, total electric supply, and direct electricity use values are available in the U.S. EIA’s

State Electricity Profiles. See table on Supply and Disposition of Electricity (currently Table 10). Direct electricity

use refers to the electricity generated at facilities that is not put onto the electricity grid, and therefore does not

contribute to T&D losses.

Commented [A27]: ? X 

Deleted: )
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

25

2.11 Interactive Effects

2.11.1. Discussion

EE measures often have indirect impacts on

electricity and fossil‐fuel use in systems not

directly affected by the subject measures. For

example:  

 Installing efficient lighting in a building’s

cooled and heated space can decrease the

electricity use of cooling systems and/or

increase natural gas consumption in

heating systems.

 Increasing insulation (or taking other

steps to reduce thermal losses from a building) can result in decreased natural gas consumption in

heating systems.  

This section provides guidance on how indirect electricity impacts, called interactive effects, should be

addressed.

2.11.2. Applicable Guidance

The regulatory requirements for interactive effects are described in Section [X] of the final emission

guidelines.  The following guidance is provided to help EE providers develop EM&V plans that document

how these requirements will be implemented.   

• Address quantification of interactive effects of electricity consumption in EM&V plans and

referenced in monitoring and verification reports.

• Use the U.S. DOE’s Uniform Methods Project (UMP)39 or other applicable protocols and methods to

estimate interactive electricity effects.

• It is not necessary to quantify the interactive effects of end‐use fossil fuel use (i.e., non‐electricity

fuels such as natural gas) for the purpose of the EPA’s emissions guidelines for affected electric

utility generating units.  

- However, it should be noted that that these cross‐fuel effects have implications for overall

greenhouse gas (GHG) emissions. Evaluating such effects is a best practice in many states.  

2.12 Use of EE EM&V Protocols and Guidelines

2.12.1. Discussion

A key aspect of industry best‐practices with EE savings quantification is the use of one or more EM&V

protocols and guidelines. Several such documents define, provide instructions for use, and generally

govern the application of the three EM&V methods outlined in Section 3.1 above. Examples of these

                                                            

39 See chapter 2: http://energy.gov/sites/prod/files/2013/07/f2/53827_complete.pdf (Version as of August 2015)

Key Definition

 Interactive effects: increases or decreases in the use of

electricity or fossil fuels that occur outside of the end

uses targeted by a specific EE measure, project, or

program. For example, reduction in lighting loads

through an energy‐efficient lighting retrofit can reduce

buildings’ air conditioning and increase heating

requirements because less heat is generated by

energy‐efficient lighting systems compared with less

efficient lighting systems.  

Key Definition

 Interactive effects: increases or decreases in the use of

electricity or fossil fuels that occur outside of the end

uses targeted by a specific EE measure, project, or

program. For example, reduction in lighting loads

through an energy‐efficient lighting retrofit can reduce

buildings’ air conditioning and increase heating

requirements because less heat is generated by

energy‐efficient lighting systems compared with less

efficient lighting systems.  

Deleted: <object><object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

26

protocols and guidelines are provided in Table 2‐1. EPA supports the use of these resources in state

plans, and encourages their further development for demand‐side EE savings as they relate to the

requirements of the CPP.  

As noted in Section 2 above, the development of these national, regional, and state‐by‐state protocols

and guidelines is primarily an outgrowth of the utility‐administered EE programs, as well as public‐ and

private‐sector ESCO projects, in place in most states throughout the country.

Table 2‐1: Examples of Industry‐Standard EE EM&V Protocols and Guidelines40  

Protocol/Guideline

Sponsor

Website Summary

Uniform Methods Project (UMP)

U.S. Department of Energy (DOE)

http://www.energy.gov/eere/abo

ut‐us/ump‐protocols  

Provides protocols for measurement and

verification of savings for many common EE

measures and technologies based on

accepted methods for a core set of widely

deployed EE measures (version as of August

2015).  

Energy Efficiency Program

Impact Evaluation Guide

SEE Action ‐ U.S.DOE and U.S.

Environmental Protection

Agency (EPA)

http://www.epa.gov/eeactionplan Describes common terminology, structures,

and approaches used for determining

energy savings and avoided emissions and

other non‐energy benefits resulting from

facility (non‐transportation) EE programs. It

provides context, planning guidance, and

discussion of issues that determine the most

appropriate evaluation objectives and best

practices approaches for different efficiency

portfolios.

Roadmap for Incorporating

Energy Efficiency/Renewable

Energy Policies and Programs

into State and Tribal

Implementation Plans

U.S. EPA

http://www.epa.gov/airquality/ee

re/pdfs/EEREmanual.pdf

Provides guidance for incorporating EE and

renewable energy policies and programs

into State and tribal implementation plans.

NEEP Regional‐Common EM&V

Methods and Savings

Assumptions Guidelines

Northeast Energy Efficiency

Partnership

http://www.neep.org/regional‐

emv‐methods‐and‐savings‐

assumptions‐guidelines‐2010

Provides methods for determining gross

energy and demand savings and savings

assumptions for a priority set of EE

program/project types or measures.

                                                            

40 Table 2‐1 lists of some of the protocols and guidelines that can be considered “established” and which are

publically available and have been promulgated and/or adopted by a State, regional, national, or international

organization. Stakeholders may request that EPA to recognize additional protocols and guidelines as they emerge

over time.  

Formatted: Font: Bold

Formatted: Font: Bold, Do not check spelling or

grammar

Deleted: Table 3‐1

Formatted Table

Deleted: Table 3‐1
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

27

Protocol/Guideline

Sponsor

Website Summary

California Energy Efficiency

Evaluation Protocols: Technical,

Methodological, and Reporting

Requirements for Evaluation

Professionals

California Public Utility

Commission

http://www.calmac.org/publicatio

ns/EvaluatorsProtocols%5FFinal%5

FAdoptedviaRuling%5F06%2D19%

2D2006%2Epdf  

Guides the efforts associated with evaluating

California’s EE programs and program

portfolios.

California Evaluation Framework

(2004)

California Public Utility

Commission

http://www.calmac.org/publicatio

ns/California%5FEvaluation%5FFra

mework%5FJune%5F2004%2Epdf  

Provides a consistent, systematized, cyclic

approach for planning and conducting

evaluations of California’s EE programs.

International Performance

Measurement and Verification

Protocol (IPMVP)

Efficiency Evaluation

Organization

http://www.evo‐world.org Provides an overview of current best

practices for determining savings from EE

projects and measures; provides a

framework and definitions that can help

practitioners develop M&V plans for their

projects.

FEMP M&V Guidelines

U.S. DOE Federal Energy

Management Program

http://www.mnv.lbl.gov/keyMnVD

ocs  

Provides guidelines and methods for

documenting and verifying the savings

associated with federal agency performance

contracts; contains procedures and

guidelines for quantifying the savings

resulting from energy efficiency.

ASHRAE Guideline 14,

Measurement of Energy and

Demand Savings

American Society of Heating,

Refrigerating, and Air‐

Conditioning Engineers  

http://www.ashrae.org ASHRAE is the professional engineering

society that has been most involved in

writing guidelines and standards associated

with EE; compared with the FEMP M&V

Guidelines and the IPMVP, Guideline 14 is

technically more detailed and addresses the

analyses, statistics, and physical

measurement of energy use for determining

energy savings.  

Regional Technical Forum (RTF)

Northwest Power and

Conservation Council

http://www.rtf.nwcouncil.org An advisory committee established to

develop standards to verify and evaluate the

savings from a wide range of EE and

conservation measures; maintains an

extensive and well documented database of

deemed savings values.

Formatted Table

Deleted: systemized
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

28

Protocol/Guideline

Sponsor

Website Summary

ISO‐NE Measurement and

Verification of Demand

Reduction Value from Demand

Resources – Manual M‐MVDR

Independent System Operator –

New England

http://www.iso‐

ne.com/participate/rules‐

procedures/manuals

Provides guidance and required criteria for

measuring and verifying performance of

demand resources participating in the

wholesale electric markets administered by

ISO.

PJM Manual 18B: Energy

Efficiency Measurement &

Verification

PJM Interconnection

https://www.google.com/url?sa=t

&rct=j&q=&esrc=s&source=web&c

d=1&ved=0CB8QFjAA&url=https%

3A%2F%2Fwww.pjm.com%2F~%2

Fmedia%2Fdocuments%2Fmanual

s%2Fm18b.ashx&ei=m9xHVceiEM

vJtQXviYDoDw&usg=AFQjCNEQb0

Z65Y_2ESjjdAP10sPjZb94Mw&sig2

=Ydqecugs2PPnuJTwxmtPIw&bvm

=bv.92291466,d.b2w

Provides guidance on measuring and

verifying the demand reduction value of EE

resources.

EE EM&V activities related to EPA’s emissions guidelines can benefit from applying established

guidelines and protocols. It should be noted, that while EPA encourages the use of well‐established

EM&V guidelines and protocols, the agency recognizes that these resources do not provide a step‐by‐

step “recipe” for quantifying savings. This is largely because:

• Approaches vary from state to state due to a variety of factors. These factors include the program,

project, or measure being evaluated, whether savings values have been established in the past for

the subject EE activity, and the overall magnitude of program impacts.  

• Guidelines and protocols per se provide flexibility with respect to key EM&V issues, such as selection

of methodologies and key parameters, the timeframe addressed by EM&V (short‐term, first‐year, or

lifetime data collection and analyses of savings), and the desired accuracy of savings.  

In this way, EPA recognized that the application of EM&V guidelines and protocols requires professional

judgment and assessment of the EE activities to determine the appropriate EM&V method,

assumptions, other factors to apply, and applicability within the context of EPA emissions guidelines,

calculations and other elements.

2.12.2. Applicable Guidance

The regulatory requirements for the use of EE EM&V protocols and guidelines are described in Section

[X] of the final emission guidelines.  The following guidance is provided to help EE providers develop

EM&V plans that document how these requirements will be implemented.   

Formatted Table
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

29

 If EM&V plans and monitoring and verification specify the use of protocols or guidelines, provide a

list of applicable minimum requirements and description of applicable sections, methods, and

portions of the protocol or guideline.41

                                                            

41 For example, simply referencing the IMPVP is not sufficient since it allows the use of four different options with

flexibility with regards to a number of savings calculation assumptions
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

30

3. Supplemental Guidance for Specific Categories of EE  

Section 4 provides EM&V guidance that is specific to several common categories of demand‐side EE

programs, projects, and measures. The EPA is providing guidance on these particular EE activities

because they are commonly implemented (at scale) in all regions of the country, represent the

significant majority of all EE savings currently being achieved, and because such guidance was requested

by stakeholders.  Each of these EE activities can also be characterized as having some form of state

agency oversight or dedicated EM&V protocols and procedures.   

The guidance in this section is intended to supplement Section 2, which applies to all demand‐side EE

savings (MWh). The best‐practice approaches and technical assumptions provided below are not

intended to provide a “recipe” for how EM&V for should be implemented. Rather, this guidance is

intended to inform decisions regarding certain EM&V parameters that are fundamental to the programs

described. EE providers can apply the approaches and assumptions in Section 4 as they prepare EM&V

plans for the specific program and project categories addressed.42 As above, this section leverages and is

consistent with prevailing industry best practices and assumptions currently being applied by state PUCs

and other oversight and sponsoring entities.  It applies only to rate‐based state plans that allow the use

of EE to generate ERCs or adjust an emission rate to comply with the emission guidelines.    

The final emission guidelines indicated that states and EE providers may undertake any EE activity that

meets the eligibility criteria described in Section VIII.x of the preamble and that is quantified, verified,

and documented according to the EM&V criteria in Section VIII.G. The specific categories of demand‐

side EE covered include:

• Demand‐Side Management (DSM) programs

• Project‐based EE (evaluated site‐by‐site)  

• State and local government building energy code and compliance programs

• State and local government product standards

By providing this additional guidance, it is EPA’s intention to clarify the EE activities that states may

implement.

                                                            

42 Should conflicts arise between the guidance in this section and the guidance in Section 2, this guidance has

precedence (since it is tailored to the specific programs and projects listed here). When preparing EM&V plans and

monitoring and verification reports, the applicable industry protocols and guidelines listed in this document should

be referenced.  

Deleted: 3

Commented [A28]: Would this also apply to state 

measures plans? 

Moved down [1]: By providing this additional guidance, it

is not the EPA’s intention to limit the EE activities that states

may implement.

Moved (insertion) [1]

Deleted: not the

Deleted: limit

Deleted: 3
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

31

3.1 Demand‐Side Management Programs

3.1.1. Program Description

DSM refers to a wide range of activities to support the efficient usage of electricity. These activities may

be funded from different sources through different mechanisms and be administered by various

entities, including investor‐owned utilities (IOU), public utilities, state energy offices (SEO), and non‐

profit entities sanctioned by state legislators and public utility commissions. Examples of DSM activities

and programs range from marketing, education, and outreach to the direct installation of efficient

appliances in low‐income households and complex industrial upgrades. In 2013, utility EE programs

alone were reported to have gross incremental43 electricity savings of approximately 24,000 GWh44 with

an estimated 16 million metric tons of avoided CO2.

DSM can include direct and indirect action programs:

• Direct action programs are those that result in the direct, explicit implementation at end‐use energy

consumer facilities, of installed pieces of equipment or systems, as well as modifications of

equipment, systems, or operations that reduce the amount of electricity that would otherwise have

been used to deliver an equivalent or improved level of end‐use service. Examples include:  

- Consumer product rebates,  

- Prescriptive and custom HVAC and lighting retrofit incentives,  

- Incentives or technical assistance for construction of new energy efficient buildings,  

- Whole‐house retrofits,  

- Street lighting retrofits,  

- Commissioning and retrocommissioning, and  

-

• Indirect action programs are those intended to facilitate or indirectly result in implementation, at

end‐use energy consumer facilities, of installed pieces of equipment or systems, as well as

modifications of equipment, systems, or operations that reduce the amount of electricity that would

otherwise have been used to deliver an equivalent or improved level of end‐use service. Examples

include:  

- Consumer behavior programs,  

- Marketing education and outreach programs,  

- Workforce education and training programs,  

                                                            

43 Incremental savings include all energy savings that accumulated in 2013 from new 2013 participation in existing

energy efficiency programs and all participants in new 2013 programs.   44 CEE Annual Industry Report, 2014 State of the Efficiency Program Industry, May 2015,

http://www.library.cee1.org/sites/default/files/library/12193/CEE_2014_Annual_Industry_Report.pdf

Commented [A29]: According to 

http://library.cee1.org/sites/default/files/library/12193/CEE_

2014_Annual_Industry_Report.pdf, US electric utility DSM 

was about $6B in 2013. $6B over 24 billion kwh implies 

$0.25/kWh for these savings. How do we reconcile that with 

other lower estimates of EE costs in the RIA? 

Moved down [2]: Programs directed toward low‐

income households and disadvantaged communities.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

32

- Financing programs,45 and  

- Energy audit programs.  

Note that both direct and indirect programs can be specifically directed toward low‐income households

and disadvantaged communities.  Specific EM&V guidance is organized by whether the subject program

relies on direct or indirect actions, as described above.

3.1.2. Applicable Guidance – Direct Action DSM  

The following EM&V guidance reflects industry best practices and technical assumptions, and is

intended to assist EE providers as they implement direct action DSM.

• EM&V methods – The savings from direct action DSM can be determined using the full suite of

EM&V methods defined in Section 2.1: measurement and verification, comparison group approaches

(including control groups and also called large scale consumption data analysis), and deemed

savings.  

• Baselines – CPBs are applicable for direct action DSM and should be defined consistently within a

given action, as described in Section 2.2.  

• Independent factors – To ensure that electricity savings are calculated consistently, each project in

a single program should apply the same approach to independent factor conditions, as described in

Section 2.5.  

• Accuracy and reliability of reported savings values – As indicated in Section 2.6, reported accuracy

and reliability should address both random error, quantified in terms of statistical precision, and

systematic bias, addressed via discussion of potential sources of bias and their likely effects.

• Evaluation cycles – Section 2.9 describes the requirements, including frequency, for verification to

confirm measures were installed and are operating to produce savings, and for quantification of

savings, and for their combination via comparison group methods..  

• Avoiding of double counting – Procedures to avoid double counting are described in Section 2.7.

3.1.3. Applicable Guidance – Indirect Action DSM  

• EM&V methods – The EM&V methods that can be used for indirect action DSM depend on the

program structure and delivery, as follows.

- Comparison group methods using RCT approaches are widely considered reliable and rigorous,

and are encouraged by the EPA. However, use of these methods require that the program can

be delivered to certain customers while excluding others from its influence, with random

assignment of who receives the program incentive and who does not.  

                                                            

45 Some direct action programs have financing components and the savings of those programs can be assessed as

described for direct action programs. However, for programs that only offer financing, their ability to generate

incremental savings can be assessed using the indirect program suggested approaches and assumptions provided

herein.

Moved (insertion) [2]

Deleted: P
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

33

- When applying comparison group methods based on quasi‐experimental design, the EE provider

should justify the validity of the quasi‐experimental comparison group as a representation of the

participants in the absence of the EE intervention. The approach to ensuring validity should also

be described.  

• Baselines  

- When direct comparison is used to quantify savings using comparison group methods, the MWh

usage of the control group of non‐participants defines the equivalent of a CPB. When regression

methods are used, the regression estimate that corresponds to the participant condition absent

the EE intervention represents the CPB. Direct comparison is commonly used with RCT, while

regression is more commonly used with quasi‐experimental design, but either analysis method

may be used with either design. The rationale for the design and analysis method used should

be reported. In particular, the report should describe how the specified comparison group, or

the regression analysis using this group, represents the participants absent the intervention.  

- When deemed or M&V methods are used, baselines for indirect action programs should be

defined consistently with baselines for similar EE actions in direct programs.

• Independent factors – Comparison group methods can be applied to produce savings for the

particular conditions of the consumption data included in the analysis, or on a weather‐normalized

basis.  

• Accuracy and reliability of reported savings values – As indicated in Section 2.6, reported accuracy

and reliability should address both random error, quantified in terms of statistical precision, and

systematic bias, addressed via discussion of potential sources of bias and their likely effects.

• Evaluation cycles – Section 2.9 describes the requirements, including frequency, for verification to

confirm measures were installed and are operating to produce savings, and for quantification of

savings, and for their combination via comparison group methods.  

• Verification – When comparison groups are the basis for measurement and verification, verification

for indirect DSM is the same as for direct DSM. That is, the savings determined by the comparison

group analysis implicitly reflects both what actions were taken and quantification of associated

savings.

- The following alternate verification approaches, however, are suggested for comparison and

control group approaches:

 With quasi‐experimental approaches, verification activities should be undertaken to ensure

implementation of the program. Examples include confirming the number of students that

took a training class and the curriculum of the class for a workforce education and training

program, or confirming that marketing or educational program materials were actually

distributed for a marketing program. The recommended frequency of verification depends

on program specific considerations, but should be between once per year and no less than

every 3 years.

 With RCT control groups, verification should confirm the proper (random) selection of

treatment and control group members.



Deleted: a recommended practice is to conduct at least

some

Deleted: With deemed or M&V approaches,

verification differs from that for direct action DSM. For

indirect DSM it is necessary to identify and characterize

the actions taken via the verification surveys, rather

than confirming already documented activity.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

34

• Effective useful life and persistence of savings – With indirect action programs using comparison

group methods, the persistence of savings should be determined each year (or as discussed above

possibly every other or third year). Savings are simply quantified until they are shown to fall to a

zero or near‐zero. For deemed or M&V methods, a value for the EUL of program savings should be

quantified for that period.

• Avoiding double counting – Applicable procedures for avoiding double counting are described in

Section 2.7.  

3.2 Project‐Based Energy Efficiency

3.2.1. Description

Project‐based EE involves implementation of one or more EE projects, retrofit or new construction, by

private firms (including ESCOs, contractors, or site owners/developers) at public or privately owned

facilities in any market sector (e.g., residential, multifamily, commercial). Project examples include

retrofits at a school or university campus, retrofits of individual office or apartment buildings, and

retrofits at industrial facilities; many projects are completed for the municipal, schools and hospitals

(MUSH) market, including water treatment plants.  This category can also include projects such as direct

load control and EE projects at water treatment plants.

A distinguishing characteristic of project‐based EE is that it applies project‐ or measure‐based

measurement and verification, or measure‐based deemed savings values, instead of program‐based

EM&V methods. That is, savings are quantified for each project. By contrast, program‐level savings

quantify and verify savings only for a sample of projects (to represent all projects in the full program), or

by conducting participant or market studies to determine factors that apply to the program as a whole.46  

During 2012, ESCO projects were estimated to have achieved gross incremental electricity savings of

about 2,500 GWh and 34,000 GWh of ongoing project savings. It should be noted that some of these

savings could overlap with DSM savings, and care should be taken to avoid double counting between

these two categories.47

3.2.2. Applicable Guidance – Project‐Based Energy Efficiency  

• EM&V methods – This EM&V guidance provides for the use of any of the EM&V methods defined in

Section 2.1 (i.e., measurement and verification, comparison group methods, and deemed savings).

In practice, however, comparison group methods are typically not practical for individual projects.

This is due to the difficulty in finding a proper comparison, or control, group for a single project or a

small group of projects.48  

                                                            

46 This is because the savings from individual projects are assigned to the responsible entity, whether a contractor

or consumer, and not to a program administrator as in the case with DSM programs. 47 Estimating Customer Electricity Savings From Projects Installed by the U.S. ESCO Industry, Juan Pablo Carvallo,

Peter H. Larsen, Charles A. Goldman, Lawrence Berkeley National Laboratory , LBNL‐6877E, May 2015. 48 For example, finding the right comparison facility, occupants, and owner for retrofits at a college campus, office
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

35

- Therefore, the recommended methods are M&V and deemed savings, as described Section 3.4.

In practice, M&V is often used in combination with deemed savings approaches.49 In practice,

the suggested and more common measurement and verification methods for retrofit projects

are IPMVP Options A, B and C and, for new construction or replace on failure, IPMVP Option D.

• Baselines – As indicated in Section 2.2, CPBs are the regulatory requirement for quantifying savings.

These baselines should be defined consistently across a portfolio of individual projects. Although the

individual projects might apply differing baselines (e.g., because of the applicability of a code or

standard, or the condition of replaced equipment), the requirements for defining baselines should

be consistent.  

For private EE contracts with consumers such as ESCO contracts, the baselines for savings

calculation may be set according to the needs and preference of the customer. However, for states

reporting savings to EPA, savings from these projects must be calculated using CPBs as defined in

Section 2.2. Thus, for state reporting of privately implemented project savings, a second set of

calculations may be required in addition to those used to establish and administer the contract

between the provider and its customer.

Commonly, the baseline for calculating savings for such contracts is the existing equipment. As

indicated in Section 2.2, the CPB for calculating savings corresponds to existing equipment only in

the case of retrofit, and then only for the remaining useful life (RUL) of the equipment. For replace‐

on‐failure or new construction, and for early replacement after the RUL, the CPB is standard

efficiency or market common practice equipment. Since it will not ordinarily be possible to take

measurements under standard efficiency or market practice conditions, an adjusted calculation

should be used to determine savings relative to the CPB as required.

If deemed methods are used to calculate savings for purposes of the private contract, an alternative

deemed calculation using the CPB as defined in Section 2.2 should be used to calculate savings for

purposes of reporting savings to the EPA.  

If M&V is the basis for the savings determined for the private contract, the measured values should

be used also in an alternative calculation of determine savings relative to the CPB. This means that

the CPB efficiency level should be substituted for the existing equipment efficiency in the savings

calculations or simulations developed from the measured data. If the M&V method is based on

measured differences between the prior existing and new equipment, the savings relative to the

CPB can be calculated from the savings relative to prior equipment as follows:

Savings|CPB  

= Savings|Existing x (1/EFFCPB ‐ 1/EFFNEW)/ (1/EFFEX ‐ 1/EFFNEW)

                                                            

building, or factory. With M&V, one could say that the “comparison group” is the affected facility before the

efficiency retrofit. This approach is known as pre‐post analysis. However, because of how the comparison group is

defined, it does not have the benefits of more rigorously implemented quasi‐experimental methods. 49 For example with IPMVP Option A, some information used to determine savings is gathered through metering

and other information is based on deemed savings values. In contrast to the application of deemed savings alone,

IPMVP Option A involves the application of facility‐specific data and parameters in the quantification of savings.

Advances in metering and analyses technologies and practices (such as advances in the use of data available from

smart meters) are lowering the cost and increasing the reliability of M&V savings quantification.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

36

where

Savings|CPB = Savings relative to the CPB

Savings|Existing = Savings relative to the

existing equipment

EFFCPB = efficiency of the CPB equipment

EFFNEW = efficiency of the new equipment

EFFEX = = efficiency of the existing

equipment.

Alternative methods to calculate savings

relative to the CPB may also be used.

• Independent factors. With M&V methods,

the savings typically should be based on

actual conditions. In contrast, normalized

conditions should be the basis for calculating

deemed savings values. Caution should be

used when combining M&V methods with

deemed savings methods within a single

project to ensure that independent factors

are considered consistently in savings

quantification.

• Accuracy and reliability of reported savings

values. As discussed in Section 3.6, the

certainty of reported savings values should be

described in monitoring and verification

reports via a written discussion of the M&V or

deemed savings method(s) used, potential

risks and biases, and related quality‐control

measures utilized. It is particularly important

to indicate the sources of any deemed savings

values, and to specify how the metering

(what is metered, for how long, etc.) provides

representative data. If sampling is used to

determine savings, the confidence and

precision of the sample’s metric (e.g., lighting

operating hours) must be reported. If the savings relative to the CPB are calculated as an adjustment

to savings relative to a different baseline used for the private contract, the method and rationale for

the determination of savings relative to the CPB should be described.

• Evaluation cycles. Section 3.9.2 describes the requirements, including frequency, for verification to

confirm measures were installed and are operating to produce savings, and for quantification of

savings.  

Industry‐Standard EM&V protocols

The following are industry best practices M&V

guidelines and protocols for project‐based EE

 IPMVP – International Performance

Measurement and Verification Protocol:

Concepts and Options for Determining Energy

and Water Savings Volume I, EVO‐10000 ‐

1.2007, Efficiency Valuation Organization.

www.evo‐world.com

 FEMP – M&V Guidelines: Measurement and

Verification for Federal Energy Projects,

Version 4.0, Prepared for the U.S. Department

of Energy Federal Energy Management

Program.

 ASHRAE – Guideline 14‐2002: Measurement of

Energy and Demand Savings, American Society

of Heating, Refrigerating, and Air‐Conditioning

Engineers

 U.S. DOE UMP – Uniform Method Project:

http://www.energy.gov/eere/downloads/unif

orm‐methods‐project‐methods‐determining‐

energy‐efficiency‐savings‐specific  

 For deemed savings practices, the operating

procedures of well‐established deemed

savings databases, such as the one operated

by the Northwest Regional Technical Forum.

http://www.rtf.nwcouncil.org ve contains a list

of protocols that can be applied, with

appropriate explanation in EM&V plans and

reports.

Industry‐Standard EM&V protocols

The following are industry best practices M&V

guidelines and protocols for project‐based EE

 IPMVP – International Performance

Measurement and Verification Protocol:

Concepts and Options for Determining Energy

and Water Savings Volume I, EVO‐10000 ‐

1.2007, Efficiency Valuation Organization.

www.evo‐world.com

 FEMP – M&V Guidelines: Measurement and

Verification for Federal Energy Projects,

Version 4.0, Prepared for the U.S. Department

of Energy Federal Energy Management

Program.

 ASHRAE – Guideline 14‐2002: Measurement of

Energy and Demand Savings, American Society

of Heating, Refrigerating, and Air‐Conditioning

Engineers

 U.S. DOE UMP – Uniform Method Project:

http://www.energy.gov/eere/downloads/unif

orm‐methods‐project‐methods‐determining‐

energy‐efficiency‐savings‐specific  

 For deemed savings practices, the operating

procedures of well‐established deemed

savings databases, such as the one operated

by the Northwest Regional Technical Forum.

http://www.rtf.nwcouncil.org ve contains a list

of protocols that can be applied, with

appropriate explanation in EM&V plans and

reports.

Commented [A31]: Big space below 

Deleted: <object><object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

37

• Effective useful life and persistence of savings. On a project‐by‐project basis, the persistence of

savings (i.e., the number of years after initial installation that savings are counted) can be

determined with deemed EUL values. Persistence of savings, however, is typically based on periodic

verification of all projects or a sample of representative projects. Verification, involving site

inspections to ensure that EE measures are still operating correctly, can occur each year for a subset

of projects on a rotating cycle, which should occur every 2 or 3 years. These verification results

should be used to adjust annual savings.  

• Avoiding double counting. Procedures to

avoid double counting are described in

Section 3.7.3. Particular caution should be

used to avoid double counting if the

project was also part of a direct action

DSM program or might be influenced by

an indirect action DSM program that

induces a consumer to save energy (e.g.,

a behavior program). This could

otherwise result in an outcome where

some of the same savings are claimed

twice.  If a project also participates in an

direct action program, the appropriate

steps include coordinating across

initiatives and adequately documenting

savings claims.

3.3 Building Energy Codes

3.2.3. Description

Building energy codes are legally enforceable

EE requirements that apply to the design and construction of buildings.50According to the U.S. DOE,

                                                            

50 Energy codes are adopted by State and local governments and govern the design and construction of new

residential and commercial structures in their jurisdictions, as well as in some cases the retrofitting of existing

structures. Energy codes reference areas of construction such as wall and ceiling insulation, window and door

specifications, HVAC equipment efficiency, and lighting fixtures. Usually, there are two methods for compliance.

The most common that codes take is the prescriptive approach, in which the code stipulates the stringency of the

materials and equipment the builder must use. The other method is the performance approach, the code allocates

a total allowable energy use for a building, and the builder can choose the materials and equipment that will meet

this target. State‐level policies on building energy codes vary widely across the 50 States, the District of Columbia,

and the five U.S. territories, and sometimes even within States, e.g. home rule States. Two foundational model

building energy codes may be adopted by States and local jurisdictions to regulate the design and construction of

new buildings: the International Energy Conservation Code® (IECC) and the ANSI/ASHRAE/IESNA Standard 90.1

Energy Standard for Buildings except Low‐Rise Residential Buildings.

50 The IECC addresses all residential and

commercial buildings. ASHRAE 90.1 covers commercial buildings, defined as buildings other than single‐family

dwellings and multi‐family buildings three stories or less above grade.  

Key Building Energy Code Definitions

 Code: Specifically, legal EE requirements that apply to

the design and construction of buildings, usually for

new buildings and for renovations and additions

applying to existing buildings.

 Adoption: The process and actions required to put a

code in place formally, such as a rulemaking process.

 Compliance: The process of meeting the code

requirements and demonstrating that these

requirements have been satisfied. Compliance is the

responsibility of the builder or contractor.

 Enforcement: The process of verifying that a building

meets the code. This process is typically conducted by a

building code official.

 Naturally Occurring Market Adoption (NOMAD): The

proportion of savings or application of measures

equivalent to the code that would have taken place in

the market even if the code had not been adopted.

Source: Attributing Building Energy Code Savings to Energy

Efficiency Programs, IEE/IMT/NEEP Report, February 2013, The

Cadmus Group, Inc., Energy Futures Group NMR Group, Inc.

Optimal Energy

Deleted: <object>
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

38

“model building energy codes and standards have the potential to save U.S. consumers an estimated

$330 billion by 2040. This equates to nearly 80 quads of cumulative full‐fuel‐cycle energy savings and

over 6.2 billion metric tons of avoided carbon dioxide emissions.”51

Specific building energy code actions that states and local governments may take include:

• Adoption of new energy codes with greater EE requirements than codes that have already been

determined by the federal government to be cost effective, and  

•

Ex‐ante estimates of building code impacts are completed regularly as codes are developed and

adopted. In contrast, ex‐post quantification of energy savings from building energy code adoption and

compliance activities is not as common or well established. The primary code adoption and compliance

impact evaluation work to‐date has been completed in six states (Arizona, California, New York, Oregon,

Rhode Island, and Washington) and at Pacific Northwest National Laboratory (PNNL)52 for U.S. DOE.

These states have regulatory structures that define acceptable procedures for quantifying savings from

code programs and attribute code program savings to EE program administrators.53

3.2.4. Applicable Guidance – State and Local Government Building Energy Code and Compliance

Programs

• Electricity Savings Metrics and Baselines – Electricity savings from building energy codes and

compliance enhancement programs should be determined with respect to a common practice

baseline (CPB). With building energy codes, the CPB should be determined with consideration of

naturally occurring market adoption (NOMAD) of efficiency building practices in the absence of an

energy code or a prior energy code. (See text box for definition of NOMAD and related terms).

The baseline needs to be estimated with consideration of building practices assumed to have

occurred in the absence of the code. Baseline building efficiency levels may be below, at, or even

above the efficiency of the code‐mandated efficiency

• EM&V methods – Achieved electricity savings from energy codes may be determined using a

bottom‐up approach that uses this equation, or a structurally similar analysis:

Annual electricity savings = (affected construction and retrofit area per year) × (electricity savings per

unit of area) × (compliance rate) = square footage per year × (MWH savings/square foot) x

(percentage)

For determining electricity savings per unit of area (for code adoption programs) or compliance

rates and energy savings (for compliance rate programs), acceptable options are direct empirical and

indirect estimation:

- Direct Empirical: These approaches are based on data collection for a representative sample of

existing buildings that have been built or renovated subject to the subject code.  

                                                            

51 https://www.energycodes.gov/status‐state‐energy‐code‐adoption ‐ accessed June 12, 2015 52 http://www.energycodes.gov/resource‐center   53 Attributing Building Energy Code Savings to Energy Efficiency Programs, IEE/IMT/NEEP Report, February 2013,

Prepared by: The Cadmus Group, Inc., Energy Futures Group NMR Group, Inc. Optimal Energy

Deleted: existing

Deleted: Actions to increase compliance with existing

codes.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

39

• With the modeling version of this approach, a building energy simulation model is used with typical

weather data and actual characteristics for the sample buildings. With the billing/energy data

analysis approach, energy bills from buildings built or renovated under the new code are compared

with energy bills of buildings constructed prior to the new code. The models or billing analyses are

used for:

- Electricity savings (net of compliance) – estimated by comparing the modeled energy use of an

as‐built building to modeled use of the same configuration building if it were built to meet the

prior code exactly, or comparing buildings built under the new code to similar buildings built

prior to the new code.  

- Compliance rates – estimated by comparing the modeled energy use of an as‐built building to

modeled use of the same building if it were built to meet the current code exactly.54

- )

• Accuracy and reliability of reported savings values – As discussed in Section 3.6, the certainty of

reported savings values should be described in monitoring and verification reports with a discussion

of the EM&V method(s) used, potential risks and biases, and related quality control measures

utilized. It is particularly important to specify the source(s) of construction and renovation data, per‐

unit savings estimates, compliance rates, and NOMAD. If sampling is used to determine savings, the

confidence and precision of the sample metric (e.g., compliance rates) should be reported.

• Evaluation cycles – Construction/renovation square‐footage data should be updated at least

biennially and per‐unit electricity savings values updated at least once every 4 years or whenever

new building energy codes are put in place for the building type(s), whichever is more often. The

principle metrics that should be updated at least every 4 years are baselines with respect to:

- Any new state or local building energy codes that have come into force, and

- NOMAD – as construction practices, with respect to efficiency, would be expected to change

over time, in the absence of codes, so that the CPB in, for example, 2029 might be much more

”efficient” than in 2025.

                                                            

54 The preferred way to determine compliance rates is with modeling or billing analyses of actual buildings that

were built or renovated and which were subject to the code. However, such analyses can be enhanced by, or

possibly replaced with interviews with key trade allies and/or experts in the building standard development

process.

Commented [A33]: There needs to be a requirement that 

any modeling approach uses a model that has undergone 

empirical verification. The recent Greenstone et al study 

showing a model estimate that was 40% off shows that 

purely modeling approaches without validation are not 

acceptable in this context. 

Deleted: ,

Deleted: <#>Indirect Estimation: This approach is

associated primarily with assessing code compliance rates

using secondary information, such as documents filed with

building departments, building audits, or surveys of code

officials. The DOE/PNNL checklist, developed under the

American Reinvestment and Recovery Act (ARRA), is an

example of a method that provides such an approach for

assessing code compliance.55 ¶

The recommended approach for determining electricity

savings involves accounting for naturally occurring market

adoption (NOMAD) of measures that are required by the

code. NOMAD may be estimated through collection of

market data or through expert judgment techniques.

Acceptable approaches include conducting literature

searches on the penetration rates of similar technologies

with similar product characteristics, the use of expert

opinions (e.g., via a Delphi process) on the expected

penetration rates in the absence of a code requirement,

and reviewing relevant market data. Use of multiple

approaches is encouraged.¶

Data on the amount of construction and retrofits that

would be affected by the new code each year (i.e., how

many square feet of each building type will be built or

renovated in a given year) are available from various

resources such as:¶

Industry statistics published by construction trade

organizations ¶

Publicly available building and renovation permit reports ¶

Data purchased from market research firms¶

U.S. Census data (used to scale national numbers to state‐

specific values

Commented [A34]: See preceding comment 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

40

• Effective useful life and persistence of savings – At a minimum, code savings can be assumed to

continue until at least the time at which a new state or local code is put into place. More

realistically, however, savings from structural measures in a building built under a new code may be

assumed to continue up to the time of the next natural major renovation of the building.

Avoiding double counting – Procedures to avoid double counting particularly between code initiatives

and programs or projects are described in Section 3.7.3. To avoid double counting among multiple code

efforts, a simple way may be to assume the baseline for the new code is the prior code, assuming

NOMAD has not moved building efficiency practices beyond the prior code.

Select EM&V Resources for Quantifying Savings from Building Codes

The following studies reflect the state‐of‐the‐art regarding ex‐post impact evaluation for building codes:  

• Building Energy Codes Program: National Benefits Assessment, 1992–2040. 2014. Pacific Northwest

National Laboratory  

• Measuring State Energy Code Compliance. 2010. Pacific Northwest National Laboratory  

• Residential Energy Code Compliance Evaluation: Guidance for Project Teams, Draft. November 2014.

Pacific Northwest National Laboratory. PNNL‐23538

• California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting

Requirements for Evaluation Professionals, Codes And Standards And Compliance Enhancement

Evaluation Protocol, April 2006, Prepared for the California Public Utilities Commission by The

TecMarket Works Team

• Attributing Building Energy Code Savings to Energy Efficiency Programs, Prepared by The Cadmus Group,

Inc., Energy Futures Group, NMR, and Optimal Energy for the Northeast Energy Efficiency Partnerships

(NEEP) and its funding partners, the Institute for Market Transformation (IMT) and IEE. February 2013

• Statewide Codes and Standards Program Impact Evaluation Report for Program Years 2010‐2012. August

2014. Prepared for: California Public Utilities Commission Prepared by Cadmus, Energy Services Division

and DNV GL

• Resource Guide for Policy Makers. June 2011. U.S. DOE Building Energy Codes Program

The following are reports that use or discuss “top down” (econometric) evaluation of codes:

• The Impact of State Level Building Codes on Residential Electricity Consumption, Anin

Aroonruengsawat,* Maximilian Auffhammer,** and Alan H. Sanstad, Energy Journal 2012, Volume 33,

Number 1

• Are Building Codes Effective at Saving Energy? Evidence From Residential Billing Data In Florida, Grant D.

Jacobsen and Matthew J. Kotchen*, The Review of Economics and Statistics, March 2013, 95(1): 34–49

• Analysis of Energy Saving Impacts of New Residential Energy Codes for the Gulf Coast. Lucas, R., ‘’’ The

Pacific Northwest National Laboratory contract DE‐AC05‐76RL01830 (2007)

• Why Has California’s Residential Electricity Consumption Been So Flat Since the 1980s? A Micro‐

econometric Approach, Costa, D.L., and M.E. Kahn, NBER working paper 15978 (2010)  

Select EM&V Resources for Quantifying Savings from Building Codes

The following studies reflect the state‐of‐the‐art regarding ex‐post impact evaluation for building codes:  

• Building Energy Codes Program: National Benefits Assessment, 1992–2040. 2014. Pacific Northwest

National Laboratory  

• Measuring State Energy Code Compliance. 2010. Pacific Northwest National Laboratory  

• Residential Energy Code Compliance Evaluation: Guidance for Project Teams, Draft. November 2014.

Pacific Northwest National Laboratory. PNNL‐23538

• California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting

Requirements for Evaluation Professionals, Codes And Standards And Compliance Enhancement

Evaluation Protocol, April 2006, Prepared for the California Public Utilities Commission by The

TecMarket Works Team

• Attributing Building Energy Code Savings to Energy Efficiency Programs, Prepared by The Cadmus Group,

Inc., Energy Futures Group, NMR, and Optimal Energy for the Northeast Energy Efficiency Partnerships

(NEEP) and its funding partners, the Institute for Market Transformation (IMT) and IEE. February 2013

• Statewide Codes and Standards Program Impact Evaluation Report for Program Years 2010‐2012. August

2014. Prepared for: California Public Utilities Commission Prepared by Cadmus, Energy Services Division

and DNV GL

• Resource Guide for Policy Makers. June 2011. U.S. DOE Building Energy Codes Program

The following are reports that use or discuss “top down” (econometric) evaluation of codes:

• The Impact of State Level Building Codes on Residential Electricity Consumption, Anin

Aroonruengsawat,* Maximilian Auffhammer,** and Alan H. Sanstad, Energy Journal 2012, Volume 33,

Number 1

• Are Building Codes Effective at Saving Energy? Evidence From Residential Billing Data In Florida, Grant D.

Jacobsen and Matthew J. Kotchen*, The Review of Economics and Statistics, March 2013, 95(1): 34–49

• Analysis of Energy Saving Impacts of New Residential Energy Codes for the Gulf Coast. Lucas, R., ‘’’ The

Pacific Northwest National Laboratory contract DE‐AC05‐76RL01830 (2007)

• Why Has California’s Residential Electricity Consumption Been So Flat Since the 1980s? A Micro‐

econometric Approach, Costa, D.L., and M.E. Kahn, NBER working paper 15978 (2010)  

Deleted: <object><object>

Commented [A35]: In the box below, last bullet, is a 

working paper considered adequately peer reviewed and 

final? If not, deletion would not hurt. 

Indeed, more broadly, why does the document select out 

these particular citations under discussion of “top down” 

approach – are they considered authoritative? If not, suggest 

deletion of some. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

41

3.3 State and Local Government Incremental Product Energy Standards

3.3.1. Description  

Appliance and equipment (product) energy standards are legally enforceable EE requirements that

specify the minimum efficiency levels of specific products. Federal standards56 currently apply to about

55 categories of appliances and equipment sold in the United States. 57 For products not subject to

existing national standards (and thus not subject to federal pre‐emption58), states may adopt their own

product standards for sales within their borders. Within the past decade, states have set standards for

products ranging from DVD players to swimming pool pumps to water dispensers. Historically, California

has taken the lead in setting state standards with other states following suit. Since 2001, Arizona,

Connecticut, Maryland, Massachusetts, New York, Oregon, Rhode Island, and Washington have each

passed several rounds of state product standards.59,60

In setting the actual product standards, the U.S. DOE and state energy offices adhere to an extensive

and robust analytical process. This typically involves conducting detailed energy savings (and economic

impact) potential studies of the anticipated future effects of these standards. In contrast, only a limited

number of ex‐post energy saving studies have been completed. California is among the farthest along in

EM&V protocol development, and has conducted three cycles of energy code and appliance standard

evaluations for the statewide Codes and Standards Program (implemented by four utility program

administrators). The evaluation approach used in California is based on bottom‐up analyses and is the

primary basis for the following suggested EM&V approaches.

3.3.2. Applicable Guidance – State and Local Government Incremental Product Energy Standards

                                                            

56 Electricity savings associated with federal product standards or adopting codes that the federal government has

already determined to be cost‐effective cannot be used for compliance with EPA’s emissions guidelines. 57 Standards usually include the test procedures needed to measure the efficiency of these products and some

guidance on how equipment that meets these standards needs to be labeled or included in directories. In terms of

savings, a recent study indicated that existing Federal standards alone will produce annual electricity savings in

several other states in 2035 of about 720 TWh, saving about 14 percent of what the projected annual electricity

consumption in that year would have been without standards and annual emissions reductions in 2035 of around

470 million metric tons of CO2. From: The Efficiency Boom: Cashing In on the Savings from Appliance Standards,

Updated from and supersedes report ASAP‐7/ACEEE‐A091, Amanda Lowenberger, Joanna Mauer, Andrew deLaski,

Marianne DiMascio, Jennifer Amann, and Steven Nadel, March 2012. Report Number ASAP‐8/ACEEE‐A123. For a

list of States that have adopted national standards, see: http://www.appliance‐standards.org/federal and

http://www.energy.gov/eere/buildings/appliance‐and‐equipment‐standards‐program   58 Federal regulation becomes the law and supersedes any State regulation. Once the Federal government

establishes an energy‐efficiency standard, no State may have a regulation different from the Federal standard. This

is referred to as Federal pre‐emption. 59 Quoted directly from http://www.appliance‐standards.org/standard‐basics‐DOE‐state‐legislature‐product‐

requirements accessed 12/30/14.   60 For list of State standards see http://www.appliance‐standards.org/states  

Deleted: ,

Commented [A36]: What does “several other states” mean 

in the footnote? 

Also, in footnote, “... been without standards and annual 

emissions reductions ...” Needs a comma and “... been 

without standards, along with annual emissions reductions 

...” 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

42

• Electricity Savings Metrics and Baselines – Determine savings from product energy standards with

respect to a CPB. For product standards, determine CPB with consideration of NOMAD of higher

efficiency products in the absence of a standard.

- NOMAD is a projection of what the annual sales,61 over time, of products meeting the standards

would have been if the standards had not been adopted.

- Once the standard is in effect, the natural market no longer exists or can exist in reality.

Methods for determining NOMAD are discussed in the next item on EM&V methods.

• EM&V methods – Achieved electricity savings from product standards should be determined on a

product‐by‐product basis using a bottom‐up approach and this equation:

Annual electricity savings =  

(products sales volume per year) × (per unit electricity savings of sold products subject to standard)  

There are two approaches that can be  applied, each with a different metric for sales volume:

- The first approach is based on the view that a standard moves the entire market, with:  

- Product sales volume = total applicable market sales

- Per unit electricity savings = the difference in sales‐weighted average energy use of products

sold under NOMAD vs after the new standard.  

- The second approach is based on the view that the standard eliminates the least efficient units,

with:  

 Product sales volume = applicable market sales currently below the new standard

 Per unit electricity savings = difference between the new standard and sales‐weighted

average energy use of products that would not meet the new standard under NOMAD.

Sources of product sales data are:

- Industry statistics published by product‐manufacturing trade organizations  

- Publicly available market characterization reports  

- Data purchased from market research firms, such a point‐of‐sale data and surveys of vendors,

retailers, and contractors

- U.S. DOE Technical Support Documents for appliance and equipment standards  

- U.S. Census data (used to scale national numbers to state‐specific values)

Caution should be used when applying these data as they can be based on limited market surveys or

might be missing certain products or models subject to the standard, but not necessarily tracked.  

Per‐unit electricity savings should be based on:

- Established laboratory‐based test methods to determine average unit energy savings. A source

of laboratory‐testing requirements could be those established by industry groups, DOE, and the

                                                            

61 Not only sales, but also installation of the sold equipment should be assessed, but for the purposes of this

guidance document, only sales data are described as involved in determining electricity savings.

Commented [A37]: Please clarify

Deleted: used to

Deleted: y

Commented [A38]: While neither of these approaches will 

be completely true, this second approach seems to be 

significantly less true. 

Deleted: or model‐based
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

43

California Energy Commission.62 The baseline should include assumptions about NOMAD to

calculate a conservative correction factor for the unit‐energy savings.

- Use of field measurements with NOMAD adjustments.

At this time, NOMAD estimates are

typically determined by using a Delphi

technique. These NOMAD estimates

should include consideration of effects of

prior‐year standards and standards

support programs.  

• Accuracy and reliability of reported

savings values – As discussed in Section

2.6, the certainty of reported savings

values should be described in monitoring

and verification reports with a discussion

of the EM&V method(s) used, potential

risks and biases, and related quality

control measures that were used.

Particularly important to indicate are the

source(s) of product sales data, per‐unit

savings estimates, and NOMAD. If

sampling is used in the process of

determining savings, however, the

confidence and precision of the sample’s

metric (e.g., field measured electricity

consumption) can and should be reported.

• Evaluation cycles – Electricity savings

values, which are reported for each year,

are based on per‐unit electricity savings

values and sales volume data. Sales

volume data should be updated at least

biennially and per‐unit electricity savings

values updated at least once every 4 years

or whenever new standards are put in

place for the same product(s), whichever

is more often. The principal metrics that

likely would be updated at least every 4

years are baselines with respect to:

- Any new state or federal product

standards that have come into force.

                                                            

62 Singh, Harinder, Ken Rider, Betty Chrisman, and Jared Babula. 2014. 2014 Appliance Efficiency Regulations.

California Energy Commission. Publication Number: CEC‐400‐2014‐009‐CMF  

Relevant EM&V protocols

 Following are a protocol and a guidebook that

can be consulted for guidance on product

standard EM&V:

- California Energy Efficiency Evaluation

Protocols: Technical, Methodological, and

Reporting Requirements for Evaluation

Professionals, Codes And Standards And

Compliance Enhancement Evaluation

Protocol, April 2006, Prepared for the

California Public Utilities Commission by The

TecMarket Works Team

- Section 9 of this document: Energy‐Efficiency

Labels and Standards: A Guidebook For

Appliances, Equipment, And Lighting 2nd

Edition, Stephen Wiel and James E. McMahon,

Collaborative Labeling And Appliance

Standards Program (CLASP) Washington,

D.C. USA, February 2005. LBNL Report

Number: LBNL‐45387(A)

- Related EM&V reports that can provide

valuable information are:

- Statewide Codes and Standards Program

Impact Evaluation Report for Program Years

2010–2012. August 2014. California Public

Utilities Commission Prepared by Cadmus,

Energy Services Division, and DNV GL.

- Realized and Prospective Impacts of U.S.

Energy Efficiency Standards for Residential

Appliances, Stephen Meyers, James

McMahon, Michael McNeil, Xiaomin Liu,

Environmental Energy Technologies Division

Lawrence Berkeley National Laboratory, LBNL‐

49504, June 2002.

Deleted: <object>

Commented [A39]: The insert box Relevant EM&V 

protocols: The last reference is to 2002 and seems to be 

dated datawise. Should it be here? 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

44

- NOMAD – it is expected that product efficiencies will change over time, in the absence of the

standard, so that the CPB in, for example, 2029 might be much more “efficient” than in 2025.

• Effective useful life and persistence of savings – At a minimum, savings should be assumed to

continue until at least a new state or local standard is put into place or until a federal standard that

meets or exceeds the state or local standard is put into force. More realistically, however, savings

from sale of products that meet a new standard should last through the operational life of the

product.  

• Avoiding double counting – Procedures to avoid double counting particularly between code

initiatives and programs or projects are described in Section 3.7. Updated product standards should

be considered so that prior and new state or local standards for the same product(s) do not double

count the same increments of savings. The simplest approach for avoiding this outcome is to assume

the baseline for the new standard is the prior standard, provided that NOMAD has not moved

product efficiency levels beyond the prior standard.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

A‐1

Appendix A. Optional EE EM&V Plan Template

The requirements for EM&V plans are discussed in Section VIII.x of the preamble to the emission

guidelines.  To help EE providers implement the associated requirements, the EPA is providing the

following EM&V plan templates.  Using these templates offer one way that EE providers can clearly

document the EM&V activities that will be carried out to properly quantify and verify savings.  

Completion of the template results in a stand‐alone EM&V plan that contains the information needed to

understand the EM&V to be undertaken, why, when, and how.  It also documents the suggested

qualifications, independence, and selection process to be used for selecting the evaluator. The following

is suggested as a template for such plans.

EM&V plans should be completed for each individual EE program or project to be used to generate ERCs

or otherwise adjust an emission rate.  For program‐level EM&V plans, any one or more of the EM&V

methods addressed in Section 2.1 above may be applied.  Plans for EE projects must apply measurement

and verification, as defined and described in that section.   

A.1. EM&V Plan Template

Part A. Document Information

1. Program/project name and jurisdiction (e.g., state where the program will be implemented)

2. EE provider and EM&V plan author

3. Contact person for information about the EM&V plan

4. Date and version of the EM&V plan

Part B. Program/Project Background

5. Short description of the program or project being evaluated

a. EE technologies offered or strategies implemented  

6. Entities

a. EE provider, including associated implementation contractors (if any)

b. State agency or other entity with oversight or implementing responsibility

c. Target market sector and delivery mechanism (for EE programs only)  

7. Implementation schedule

8. Projections for annual electricity savings over the effective useful life (EUL) of the EE

program/project63

9. Program/project budget

Part C. EM&V Overview

10. Savings quantification

a. Metric definition – i.e., annual MWh of adjusted gross savings based on a CPB

b. EM&V method – describe the value of the metric(s); include summary description of

data collection approach, sample design, analyses methods, and significant

                                                            

63 Program life may be estimated as the average life of the measures being offered under the program.  
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

A‐2

assumptions; indicate the industry best‐practices, protocols (e.g. IPMVP and the specific

option) being applied

c. Persistence of savings – if stipulated EULs are utilized, indicate source of such values and

why they are applicable to the subject program; describe any other approaches utilized.

d. Independent factors – list and describe to be considered and supporting data that will

be used to determine these variables’ value; include how these factors are analyzed to

quantify savings.

e. Description of whether savings will be based on conditions during performance period

or normalized to standard/typical conditions (or other conditions); define standard

conditions if applicable.

f. Deemed savings – if used, describe the version of the deemed savings database to be

applied and/or the source of deemed savings; and how correct application of deemed

savings values is ensured.

g. Sampling – if used, describe the sampling design and targeted confidence and precision

for resulting savings

h. T&D losses – if applied to savings values, specify the value of the loss factor, its source,

and the applicable methodology.

i. Interactive effects – describe how interactive effects are calculated, their values, data

sources and how the values are factored into measure saving values; if interactive

effects are not included, explain why.  

11. Verification activities – indicate intended verification activities, including:

a. When they will occur and who will conduct

b. How they will be used to adjust savings values as needed

12. Avoiding double counting – indicate sources of potential double counting of savings and how it

will be avoided including the tracking and accounting procedures that will be applied.  

13. Accuracy and reliability of reported savings values – indicate expectations for overall certainty of

savings estimates, how the certainty of quantified MWh savings values will be assessed, and the

methods to be used to control the types of error inherent to the applied EM&V methods,

including:  

a. Statistical indicators of certainty such as confidence and precision  

b. Discussion of the threats to validity and potential risks, bias or error inherent to the

applied EM&V methods

c. Methods for minimizing bias and other threats to validity

d. Quality control procedures  
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

B‐3

Appendix B. Glossary of Terms  

This glossary is not intended as a full list of applicable EM&V terms. Readers are encouraged to consult

SEE Action EM&V Guide glossary64 for additional definitions.

Baseline: conditions, including the level of electricity consumption, that would have occurred

without implementation of the subject EE activity. Baseline conditions are used to calculate

the baseline electricity use needed to estimate measure‐, project‐ and program‐related

savings.

Common practice baseline (CPB): default technology or condition that would have been in

place at the time of project implementation without a decision to install a more efficient

system or measure, for example, the system(s) or measure(s) a typical consumer/building

owner would have continued with or installed at the time of project implementation.

Comparison group EM&V methods involve determining program savings as the difference

between (the changes in) a comparison group’s electricity consumption and the (changes in)

electricity consumption of program participants. Comparison group approaches include RCTs

and quasi‐experimental methods using nonparticipants.65 These methods are applied to

program‐level savings, but ordinarily not for facility‐ or project‐level savings.  

Deemed savings values, also called stipulated savings values, are estimates of electricity

savings for a single unit of an installed EE measure that (1) has been developed from data

sources (such as prior metering studies) and analytical methods that are widely considered

acceptable for the measure and purpose, and (2) is applicable to the situation under which

the measure is being implemented.  

Individual parameters or calculation methods can also be deemed; for example, effective

useful life of a measure or the annual operating hours of light fixtures in an elementary school

classroom. Common sources of deemed savings values are previous evaluations and studies

that involved actual measurements and analyses.

                                                            

64State and Local Energy Efficiency Action Network. 2012. Energy Efficiency Program Impact Evaluation Guide.

Prepared by Steven R. Schiller, Schiller Consulting, Inc., www.seeaction.energy.gov. 65 With M&V, one could say that the “comparison group” is the affected facility before the efficiency retrofit. This

approach is known as pre‐post analysis and some people categorize it as a quasi‐experimental method. However,

because of how the comparison group is defined, it does not have the benefits of more rigorously implemented

quasi‐experimental methods and is not considered a comparison group method for the purposes of this guidance.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

B‐4

Effective useful life (EUL): an estimate of the duration of savings from individual EE measures,

reported in years. Values are typically specific to individual EE measures or projects but also

may be specified by program. EUL is defined through various means, including median

number of years that the measures/projects installed under a program are still in place and

operable.

Electricity savings: change in electricity use resulting from the implementation of EE

measures and practices, reported in units of megawatt‐hours (MWh).  

Evaluation measurement and verification plans:  

Free ridership: portion of a program’s supported savings that would have occurred in the

absence of the program.  

Independent verifier:

Independent variables: the explanatory factors (e.g., weather or occupancy) that affect the

variable under study (e.g., electricity consumption) and can vary independently of the variable

under study.

Inspections: site visits to facilities at which an EE project or measure was implemented.

Inspections document the existence, characteristics, and operation of baseline or EE project

equipment and systems, and the factors that affect energy use. Inspections can or might not

include review of commissioning or retro‐commissioning documentation.

Interactive effects: increases or decreases in the use of electricity or fossil fuels that occur

outside of the end uses targeted by a specific EE measure, project, or program. For example,

reduction in lighting loads through an energy‐efficient lighting retrofit can reduce buildings’

air conditioning and increase heating requirements because less heat is generated by energy‐

efficient lighting systems compared with less efficient lighting systems.

Market effects: change in the structure of a market or the behavior of participants and

market actors in a market that reflects an increase in the adoption of EE products, services, or

practices and is causally related to market intervention(s).  

Commented [A40]: Lots of roll-over space. 

Commented [A41]: Incomplete 

Commented [A42]: Incomplete 

Commented [A43]: Big space below 

Commented [A44]: Big space below 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

B‐5

Measurement and verification (M&V) is the process of using measurements to determine

electricity savings associated with individual projects or measures reliably. M&V can be

applied to a census or sample of projects (measures) in a program (project) to determine

average savings for the program (project). With M&V, baseline electricity consumption is

compared with post‐project (or post‐measure) electricity consumption measurements, along

with analyses and documented assumptions to quantify savings. The IPMVP defines two

retrofit isolation and two whole‐facility M&V options used in the efficiency industry:  

Retrofit isolation – assessing savings from each EE measure individually (IPMVP Options A

& B). Although Option A allows some stipulation of factors, Option A differs from the use

of deemed saving values in that all M&V options require some project site

measurements.

Whole facility – analyzing savings from each EE measure in a project/facility collectively –

(IPMVP Option C, review of energy bills or Option D, calibrated simulation).

Some combination of the above.  

See Chapter 5 for more information on the IPMVP options.

Monitoring and verification (M&V) report:  

NOMAD: ...

Realization rate: the ratio of electricity savings an independent third party determines

divided by the savings claimed by the measure, project, or program provider(s). The ratio’s

deviation from 1.0 can be due to (1) adjustments for data errors, (2) differences in

implemented measure counts or other claimed information because of verification activities,

or (3) other differences revealed through the EM&V process.

Reporting year cumulative savings: defined as the amount of electricity savings quantified

and verified as a result of EE activities in the reporting year from both (a) EE activities

operating for the first time in the reporting year and (b) EE activities initiated in prior years

and still in effect; reported in annual megawatt hours, MWh.  

Reporting year incremental savings: defined as the amount of electricity savings quantified

and verified because of EE activities operating for the first time in the reporting year;

reported in annual megawatt hours, MWh.  

Rigor: defined as the level of effort expended to minimize uncertainty due to factors such as

sampling error and bias. The higher the level of rigor, the more confident one is that the

results of the EM&V activities are both accurate and precise. Rigor may be quantified or

described with metrics for reliability and certainty, such as in terms of reported value

confidence and precision.

Commented [A45]: Incomplete 

Commented [A46]: Big space below 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

B‐6

Spillover: additional reductions in energy consumption that are due to program influences

beyond those directly associated with participation—these include participant and

nonparticipant spillover.  

Verification: an independent assessment to ensure that the EE measures have been installed

correctly and could generate the predicted savings. Verification may include assessing

baseline conditions and confirming that the EE measures are operating according to their

design intent. Project site inspections, participant phone and mail surveys, and review of

implementer and program, project, and measure documentation are typical verification

activities.  
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐1

Appendix C. Measurement and Verification for Common EE Technologies: Best

Practices and Resources

C.1. Introduction

Appendix C describes best practices and resources for measurement and verification of common EE

technologies.  As discussed in Section 3.1, measurement and verification is a set of procedures for

quantifying savings from an individual EE measure or project, versus an EE program. This appendix

illustrates how many of the technical topics introduced in Section 3 are typically applied in practice to

quantify and verify savings, with relevance for EM&V planning, execution, and documentation. It

describes the measurement and verification approaches frequently applied to quantify savings for

specific technologies at the project and measure level. The purpose of this section is to provide

examples of how savings are quantified at the project or measure level, and to document the

measurement challenges and listing key lessons from experience gained over the last several decades.  

Much of this information can be applied to measures and technologies that are not specifically covered.  

This section does not provide guidance on whether measurement and verification (as discussed here) or

using deemed savings values from a technical resource manual (TRM) is the preferred approach for

quantifying savings from the project or measure in question. In practice, the former involves collecting

primary field data on usage and efficiency to determine savings at the customer site, while the latter is

typically coupled with verification of measure installations.  As discussed in Section 3.1, tradeoffs need

to be considered between cost, accuracy, and value of the resulting information in light of available

budgets and desired levels of accuracy and reliability.  

C.2. Residential Lighting

Definition  

These projects involve installing efficient lighting products in residential dwelling, such as dimmable

lights and Light Emitting Diode (LED) lamps. In all cases, the measure installed is designed to promote

lamps with higher efficacy (lumens/Watt) and the same or better color resolution index. This guidance is

applicable to at least the following four common program design approaches, for both new construction

and retrofit applications of residential lighting measures:

• Direct installation of lighting products, including Energy Service Company (ESCO) projects

• Consumer incentives (e.g., rebates, coupons)

• Giveaways (installed for free during audits or mailed to homeowners)

• Upstream/Mid‐stream/Downstream buy downs of retail lamp/fixture prices (e.g., point of sale

rebates)

Key Challenges in Estimating Energy Savings

The primary measurement challenges for estimating lighting technology savings evaluation are:

Commented [A47]: May need to update section references

Formatted: Font: 11 pt

Formatted: Font: 11 pt

Formatted: Font: 11 pt

Formatted: Font: 11 pt

Commented [A48]: If M&V only applies to individual 

measures/projects then need to clarify this earlier in the 

document and use more consistently throughout. 

Deleted: compact fluorescent lamps (CFLs), both standard

“twister” bulbs and specialty CFLs such as reflectors, A‐

Lamps, globes, and
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐2

1. Operating hours – Determining the operating hours for the lighting systems (baseline and

project operating hours) where in many cases the exact location, building type, and fixture type

are not known or cannot be easily found.

2. Baseline wattage – Determining the baseline wattage of lamps being replaced by more efficient

lamps (baseline wattage) when in many cases the lumen output of the baseline bulb is not

known and it is not known if the more efficient lamps provides lower, higher or equivalent

lumens output. In addition, for upstream programs it is not known if the more efficient bulb

simply replaced a bulb of similar efficiency or efficacy.  

3. Indirect savings or additional electricity use resulting from interaction with HVAC systems –

Estimating the indirect effects of installing more efficient lamps on the energy use of existing

heating, ventilation and cooling systems.

4. Free Riders and Technical Progress– Estimating underlying market adoption rates is challenging.  

Particularly in lighting where product innovation is occurring a rapid pace.  Consequently,

programs that incent switching to current long‐lived technology (e.g. a CFLs installed in 2010

persist through 2020 in a light use application) may yield short term energy savings but also

reduce longer‐term savings that would have occurred if the consumer had later voluntarily

switched to more efficient options (e.g. shifting directly from incandescent to a higher efficiency

LED in 2015).  

The most common evaluation methods for dealing with each challenge are discussed below:

A. Estimating Hours of Operation (HOU)66

M&V Approach 67 Typical Application

Self‐reports from homeowner Sites where trained auditors/contractors have

installed more efficient bulbs as part of residential

audit or direct install program

Deemed hours from previous evaluations or

TRM

Evaluation of savings from upstream programs where

retailers or distributors have received rebates to buy

down incremental cost of bulbs, exact location of

bulb installation is not known and the evaluation

budget is not sufficient to cover cost of metering.

Install lighting loggers for short‐term (from

weeks to months) measurement and to

extrapolate result to annual usage

Most common application in states where lighting

programs have been operated for many years,

lighting represents a significant portion of portfolio

savings and results can be used by multiple program

administrators

B. Estimating Baseline wattage

                                                            

66 Note the range of recently evaluated residential lamp operating hours range from 1.7 to 3.0 hours per day.   67 Note all of these approaches assume that hours of use for the new efficient bulbs is equivalent to the hours of

use for the baseline bulbs. This assumption may need to be tested as saturation of LED’s increase.  

Deleted: I

Deleted: Indirect

Formatted: Font: Not Bold

Deleted: <#>¶
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐3

M&V Approach Typical Application

In‐home Inspection of a sample of baseline

bulbs

Direct‐install programs, Residential Audit Programs

where LED’s are installed for low cost or free

Use multiplier factors taken from TRM or

previous evaluations to directly convert

tracking data on energy efficient bulb wattage

to estimated delta watts68  

Upstream lighting programs where location of bulb

installation is not known, tracking systems are not

completely developed and program is in first few

years of operation  

Use Energy Star Lumen Equivalence Method

with adjustment for EISA standards to estimate

delta watts.69

Upstream lighting programs where tracking systems

have matured and shelf data exists to adjust baseline

usage to account for EISA standard impact and

possible sell through of existing product

C. Estimating Interactive effects from efficient lighting on HVAC system usage

M&V Approach Typical Application

Estimate indirect savings impacts using Waste Heat

factors taken from previous evaluation studies or

TRMs from adjacent states or regions.

Lighting Programs or Direct Project installations

where efficiency of existing HVAC systems is

not known

Use simulation modeling   Lighting Programs have been in effect for many

years and the evaluation budget is sufficient to

gather data on the system efficiency of HVAC

systems where lamps have been installed

D. Other important evaluation issues

Finally, it is important to note that as lighting efficiency programs mature and the saturation of CFLs and

LED increase, there will be a higher possibility that efficient lighting products will be used to replace in

situ or existing efficient lighting products previously purchased. This possibility can be assessed using

survey of homeowner purchase patterns or limited on site visits to a sample of participants.

Relevant protocols or evaluation studies for more information

1. Uniform Methods Project; Chapter 21: Residential Lighting Protocol (December 2014)

http://www.energy.gov/sites/prod/files/2015/02/f19/UMPChapter21‐residential‐lighting‐

evaluation‐protocol.pdf  

2. KEMA, Inc.; Cadmus Group, Inc. (February 8, 2010). Final Evaluation Report: Upstream Lighting

Program. Prepared for California Public Utilities Commission, Energy Division.

                                                            

68 Savings=multiplier * efficient bulb wattage. This multiplier is usually based on a bin match of the lumens

produced by specific baseline and more efficient bulbs 69 Examples of equivalent wattage tables can be found in Massachusetts Technical Reference Manual 2011

(http://www.ma‐eeac.org/index.htm), the New York Standard Approach for Estimating Energy Savings from Energy

Efficiency Programs 2010 (http://www.dps.ny.gov/TechManualNYRevised10‐15‐10.pdf ), and the Datasase for

Energy Efficiency Resources (http://www.deeresources.com )

Deleted: CFL or
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐4

http://www.calmac.org/publications/FinalUpstreamLightingEvaluationReport_Vol2_C

ALMAC.pdf.

3. Navigant Consulting; Itron; Opinion Dynamics; Michaels Energy (October 23, 2013). Residential

Energy Star Lighting PY5 Evaluation Report. Presented to Commonwealth Edison Company.

http://www.ilsag.info/comed_eval_reports.html.  

4. Nexus Market Research, Inc.; RLW Analytics, Inc. GDS Associates (January 20, 2009). Residential

Lighting Markdown Impact Evaluation. Prepared for Markdown and Buydown Program Sponsors

in Connecticut, Massachusetts, Rhode Island, and Vermont.

http://www.env.state.ma.us/dpu/docs/electric/09‐64/12409nstrd2ae.pdf

C.3. Small Commercial and Residential Unitary and Split Cooling Systems

Definition

These projects involve installing efficient cooling systems in residential and small commercial

applications. Typical projects involve packaged systems with capacities between three and 20 tons and

split systems with capacities <=65,000 BTU/hour (5.4 tons). This guidance is applicable to at least the

following common program design approaches, for both new construction and retrofit applications of

small commercial and residential unitary and split systems measures:

• Direct installation of cooling systems, including ESCO projects

• Consumer incentives (e.g., rebates, coupons)

• Upstream/Mid‐stream/Downstream buy downs of retail cooling system prices (e.g., point of sale

rebates)

Key Challenges in Estimating Energy Savings

The two most difficult or primary measurement challenges for this technology are:

(1) Determining baseline efficiency conditions and  

(2) Determining Equivalent Full Load Hours (EFLH) for cooling system use.  

The most common evaluation methods for dealing with each challenge are discussed below:

A. Determining Baseline Efficiency Conditions or System Ratings

M&V Approach Typical Application

Gather baseline efficiency ratings from HVAC

system equipment nameplates from a sample of

participating homeowners or survey HVAC

contractors to determine the CPB

For situations where contractor or homeowner

reports cooling system is being replaced early or

before the end of its useful life.  

Use minimum efficiency level required by building

or appliance code to estimate savings

For situations where the cooling system is being

installed because equipment is failing or burned

out or as part of a new construction project
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐5

B. Determining Equivalent Full Load Hours for Cooling System Use

M&V Approach Typical Application

Simulate hours of HVAC system use using building

simulation models based on prototypes of typical

homes and climate conditions

Situations where size, climate conditions and

layout of homes is known with reasonably

accuracy based on current or previous data

collection

Conduct short‐term metering of system use during

cooling season for a short period of time, at least one

month, during typical occupancy conditions),  

Situations where accurate estimates of impacts

on peak use are needed and there is sufficient

evaluation budget to support the metering.  

Use deemed hours of cooling use from neighboring

state, or regional TRMs  

Situations where programs are just starting out

and or it is not practical to measure or meter

system hours of use

Relevant Protocols and Sources for readers interested in more detailed information

1. Uniform Methods Project, Chapter 4:Small Commercial and Residential Unitary and Split Cooling

Systems protocol (April 2014)

2. ASHRAE. (2010). Performance Measurement Protocols for Commercial Buildings.  

3. Consortium for Energy Efficiency (CEE). (2009). Commercial Unitary AC and HP Specifications,

Unitary Air Conditioning Specification. Effective January 16, 2009.

www.cee1.org/com/hecac/hecactiers.pdf. ISO‐New England, Inc. (June 2010).  

4. ISO New England Manual for Measurement and Verification of Demand Reduction Value from

Demand Resources Manual (M‐MVDR).

5. KEMA. (February 10, 2010). Evaluation Measurement and Verification of the California Public

Utilities Commission HVAC High Impact Measures and Specialized Commercial Contract Group

Programs 2006‐2008 Program Year.

www.calmac.org/publications/Vol_1_HVAC_Spec_Comm_Report_02‐10‐10.pdf.

C.4. Residential Whole Building Retrofits

Definition  

These projects involve installing multiple EE measures, such as insulation, weather stripping, set‐back

thermostats, efficient lighting and/or efficient heating and cooling systems in residential dwellings. The

savings from individual measures usually interact with each other, so it is not appropriate to estimate

the savings from individual measures and sum them.  

This guidance is applicable to common program‐design approaches such as Home Performance with

Energy Star (HPwES), residential audit programs, and any other program that seeks to reduce overall

energy use in residential dwellings using measures that address both building shell and building system

improvements.
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐6

Key Measurement Challenges

The key challenge in estimating energy savings from this program type is to isolate the electricity savings

from the installation of multiple measures in the participating houses from market, weather, economic

and other factors that affect household electricity consumption. Three different evaluation methods to

deal with this challenge and their typical applications are described in the tables below.  

M&V Approach Typical Application

Randomized Control Trials applied to a sample of

participants and non‐participants in the program70

Programs or projects where customers can be

randomly assigned the opportunity to

participate in the program and do not have to

be recruited.

Quasi Experimental Design to Select Participant and

Control Groups and control for self‐selection bias

using regression and pooled techniques.71 Two

approaches include:

1. Two‐stage72 regression approach using past and

future participants as the comparison or control

group

2. Pooled fixed effects regression approach if

there is insufficient data on past or future

participants to create the comparison group

Most common program situation where

customers must be recruited to join programs

and install measures, Billing data is available

and evaluation budget is sufficient to gather

detailed data on home characteristics.  

Building simulation models to estimate savings using

Building Characteristic Data Gathered by Program

contractors or implementers  

Situations where it is difficult to access

sufficient billing data and or administrator has

confidence in implementer’s building

simulation models because it has been

calibrated to billing energy usage.  

Relevant Protocols for More Information

1. Uniform Methods Project, Chapter 8: Whole‐Building Retrofit with Consumption Data Analysis

Evaluation Protocol reference. http://www.energy.gov/sites/prod/files/2013/11/f5/53827‐8.pdf

2. Fels, M.F., ed. (February/May 1986). Energy and Buildings: Special Issue Devoted to Measuring

Energy Savings: The Scorekeeping Approach. (9:1&2).  

                                                            

70 State and Local Energy Efficiency Action Network. 2012. Evaluation, Measurement, and Verification (EM&V) of

Residential Behavior‐Based Energy Efficiency Programs: Issues and Recommendations. Prepared by A. Todd, E.

Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National Laboratory. http://www.behavioranalytics.lbl.gov.   71 Uniform Methods Project, Chapter 8: Whole‐Building Retrofit with Consumption ( Data Analysis Evaluation

Protocol (Prepared by Mimi Rosenberg and Ken Agnew for the National Renewable Energy Laboratory NREL, April

2013) http://www.energy.gov/sites/prod/files/2013/11/f5/53827‐8.pdf 72 Two stages in this regression approach refers to the first stage of the regression analysis where changes in usage

or savings are estimated using past or previous participating customers and then in a second stage of analysis

where changes in usage are estimated for the current participant sample. Savings are estimated using a difference

of differences approach across the entire group. Use of past or future participants as a comparison group helps

avoid biases in savings estimates introduced by customer self‐selection into the program.  
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐7

3. Fels, M.F.; Kissock, K; Marean, M.A.; and Reynolds, C. (January 1995). PRISM (Advance Version

1.0) Users’ Guide. Center for Energy and Environment Studies. Princeton, New Jersey

C.5. Commercial Lighting

Definition  

These projects involve installing efficient lighting products (ballasts, lamps, and fixtures) in

nonresidential buildings such as offices, grocery stores, restaurants, warehouses, retail, and multifamily

housing units. Example projects include installing T5/T8 fluorescent lighting systems for general area

lighting and more focused LEDs for task lighting. There are three common program delivery approaches:

• Direct installation of lighting products, including ESCO projects

• Consumer incentives (e.g., rebates)

• Upstream/Mid‐stream/Downstream buy downs of lamp and fixture prices (e.g., distributor

incentives).

Key Measurement Challenges

Key measurement challenges include: 1) Ensuring lighting logger readings are carefully reviewed and

screened to ensure accurate estimates of hours of use in rooms where daylighting and occupancy

fluctuations can cause flicker problems. 2) Accurately weighting these use area specific HOU results into

an average hours of use for the entire building 3) Determining baseline wattage of lighting systems

where the precise physical location is unknown and inventories of lighting systems that do not meet

current code are still available for sale.  

The most common evaluation methods for dealing with each challenge are discussed below:

A. Determining Average HOU by Room Type and Whole Building

M&V Approach Typical Application

Collect average hours of usage by area use type73

from a representative sample of buildings using self‐

reports from building managers and or HOU

estimates from forecasting models.

Project or program evaluations of new lighting

projects or measures where there is insufficient

time or resources to install lighting loggers and

collect usage data over a 6 to 12 week period.  

Retrieve hours of lighting system or fixture usage by

room type from building energy management

systems or time clocks

Evaluations at sites where sophisticated energy

management systems log lighting usage and

schedules.  

Install lighting loggers in a representative sample of

area use types within a sample of participating

buildings.  

Evaluations where savings from lighting

projects constitute the majority of portfolio

savings and no recent studies of lighting HOU

                                                            

73Examples of area use types include conference room, offices, halls, bathrooms, storage rooms, or retail display.

Commented [A49]: Dated references. Are these 

documents still relevant? 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐8

have been conducted in available TRMS or

recent evaluations

B. Determining Baseline Lighting System Wattage

M&V Approach Typical Application

Collect detailed fixture wattage inventory by area

usage type before the lighting retrofits are installed.  

For accelerated or early retirement of existing

lighting systems before the projected or

anticipated end of their useful lives from

building manager’s perspective.  

Determine the CPB conditions using most recent

state or federal product standards for equivalent

lumens or trade ally input if market baseline is more

efficient than standards. Potentially make

adjustments for non‐qualifying equipment that is still

available in distributor warehouses.

For new construction, normal replacement, or

replace on burnout projects where the baseline

replacement system is most likely to be a

lighting system that just meets minimum

efficiency requirements

Install lighting loggers in a representative sample of

room usage types within a sample of participating

buildings. Develop a lighting logger quality control

protocol to ensure HOU estimates are accurate.

Evaluations where savings from lighting

projects constitute the majority of portfolio

savings and no recent studies of lighting HOU

have been conducted in available TRMS or

recent evaluations

Relevant Protocols for readers interested in more detailed information

1. Uniform Methods Project, Chapter 2: Commercial and Industrial Lighting Evaluation Protocol

(April 2013)  

2. California Public Utilities Commission (CPUC). (2008). http://www.deeresources.com.  

3. Federal Energy Management Program (FEMP). (2008). M&V Guidelines: Measurement and

Verification for Federal Energy Projects Version 3.0.

http://www1.eere.energy.gov/femp/pdfs/mv_guidelines.pdf

4. Massachusetts Program Administrators. (October 2011). Massachusetts Technical Reference

Manual for Estimating Savings from Energy Efficiency Measures 2012 Program Year–Plan

Version. www.masssave.com.

5. Itron, Nonresidential Downstream Lighting Impact Evaluation Report, Appendix D and E

available at:

http://www.calmac.org/publications/Nonresidential_Downstream_Lighting_Impact_Eval_Repor

t AppendicesA‐J.pdf Commented [A50]: I couldn’t find this link. 
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐9

C.6. Commercial Chiller Systems

Definition  

These projects involve installing or retrofitting electricity consuming chillers and/or associated auxiliary

equipment. Chillers provide mechanical cooling for commercial, institutional, multiunit residential, and

industrial facilities. Cooling may be required for facility heating, ventilation, and air conditioning (HVAC)

systems or for process cooling loads (e.g., data centers, manufacturing process cooling).  

Key Challenges in Quantifying Energy Savings

Key measurement challenges include: (1) Determining annual electricity consumption of the existing

chiller system, (2) Determining the length of the appropriate monitoring periods for collection of system

data, and (3) Normalizing savings to performance period independent variables such as weather and

changes in occupancy.

Baseline Energy Consumption

Methods for determining baseline electricity consumption of Chiller Systems are outlined in the table

below.

M&V Approach Typical Application

IPMVP Option A – Retrofit Isolation

Key parameter measurements include cooling load,

outside air temperature and part‐load efficiency

factors from manufacturer testing

Project level evaluation where expected

variation in occupancy and chiller loads is likely

to be relatively low over a typical year

IPMVP Option C – Whole Building Billing Analysis  

Pre and post installation over at least a year period

For early replacement projects where

estimated costs of metering are high and the

estimated savings are large compared to

normal variations in total building energy usage

IPMVP Option D – Calibrated Building Simulation

Modeling with potential adjustments for impact of

new product standards and or changes in occupancy

For replace on burnout or new construction

projects where baseline condition is likely to be

a chiller system that meets the minimum

efficiency standards

All of the methods above require the evaluator to collect data on the condition of the existing

equipment (with the exception of new construction) and make an estimate of the remaining useful life

of the equipment for use in developing savings estimates using the dual baseline approach. To support

use of existing usage conditions for early replacement projects, evaluators must present evidence that

the program has likely induced an early replacement of the existing chiller equipment. Data sources for

condition and remaining useful life of in‐situ equipment/systems include interviews with facility

managers, equipment logs, inspection of equipment, and manufacturer’s data.  

Determining the Length of the Monitoring Period
*** E.O. 12866 Review – Draft – Do Not Cite, Quote or Release During Review ***

Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

C‐10

The frequency and length of the monitoring period (for electricity consumption, cooling loads, etc.)

associated with chiller equipment depends on the type of load and frequency of equipment operation.

Continuous monitoring for large systems with variable loads (such as chillers service process loads) is

recommended for at least three months, while spot measurements of electricity consumption can be

sufficient for systems with constant loads and known hours of operation.  

Normalizing Savings to Account for Independent Variables  

In many cases, it will be necessary to run regression equations to separate out the impacts of chiller

measures from independent variables such as operating temperatures or process throughput. The

model’s predictive power should be tested using R2 values or T statistics on the estimated coefficient for

the independent variable of interest to ensure accuracy. The Commercial Chiller protocol74 from the

Uniform Methods Project recommends a model fit thresholds of >0.75 for R2 and T statistics of greater

than 2.0 to determine if savings estimates are statistically significant.

Relevant Protocols and Evaluation Sources for More Information

1. Uniform Methods Project, Chapter 14: Chiller Evaluation Protocol, Chapter reference ( Prepared

by Alex Tiessen, Posterity Group for the National Renewable Energy Laboratory, September

2014). Reference http://www.energy.gov/sites/prod/files/2014/10/f18/Chapter%2014‐‐

Chiller%20Evaluation%20Protocol.pdf

2. EVO. (January 2012). International Performance Measurement and Verification Protocol –

Concepts and Options for Determining Energy and Water Savings Volume 1. Efficiency Valuation

Organization.

3. Fagan, F., Bradley, K., Lutz, A. (2011). Strategies for Improving the Accuracy of Industrial

Program Savings Estimates and Increasing Overall Program Influence. Prepared for the

International Energy Program Evaluation Conference.

4. Ridge, R., Jacobs, P., Tress, H., Hall, N., Evans, B. (2011). One Solution to Capturing the Benefits

of Early Replacement: When Approximately Correct is Good Enough. Prepared for the

International Energy Program Evaluation Conference.

5. US DOE FEMP. (April 2008). M&V Guidelines: Measurement and Verification for Federal Energy

Projects—Version 3.0. U.S. Department of Energy Federal Energy Management

                                                            

74 Table 6 page 13 in the Chiller Evaluation protocol at

http://energy.gov/sites/prod/files/2014/10/f18/Chapter%2014‐‐Chiller%20Evaluation%20Protocol.pdf