Scoping Study to Evaluate Feasibility of 

National Databases for EM&V Documents 

and Measure Savings

Tina Jayaweera, Hossein Haeri, Allen Lee, Scott Bergen, 

Cynthia Kan, Aquila Velonis, Christy Gurin, Michael Visser, 

Andrew Grant, and Ashley Buckman

The Cadmus Group Inc. / Energy Services

720 SW Washington Street, Suite 400

Portland, OR 97205

503.467.7100

Evaluation, Measurement and Verification Working Group 

June 2011
June 2011 www.seeaction.energy.gov ii

Disclaimer

This document was prepared as an account of work sponsored by the United States Government. While this 

document is believed to contain correct information, neither the United States Government nor any agency thereof, 

nor The Regents of the University of California, nor any of their employees, makes any warranty, express or implied, 

or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, 

product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein 

to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, 

does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States 

Government or any agency thereof, or The Regents of the University of California. The views and opinions of 

authors expressed herein do not necessarily state or reflect those of the United States Government or any agency 

thereof, or The Regents of the University of California.
June 2011 www.seeaction.energy.gov iii

Acknowledgements

This report was prepared for Lawrence Berkeley National Laboratory (LBNL) and funded by the Permitting, Siting 

and Analysis Division of the U.S. Department of Energy’s Office of Electricity Delivery and Energy Reliability under 

Lawrence Berkeley National Laboratory Contract No. DE-AC02-05CH11231. The project was managed by Steven R. 

Schiller, of Schiller Consulting, Inc. and Charles A. Goldman of LBNL. We appreciate the ongoing support and 

guidance they provided throughout the course of this study. The information that forms the basis for the findings 

and recommendations of this study was gathered through interviews with the national experts listed below. The 

Cadmus Group Inc. thanks them for their invaluable insights and contributions, while remaining solely responsible 

for any errors in this report.Dan Barbieri (KEMA); Ben Bronfman (The Cadmus Group Inc); Peter Carvalho (CGI); Tim 

Caulfield (California Measurement Advisory Council); Pete Cleff (PPL Electric Utilities); Kevin Cooney (Navigant

Consulting); Dean Creed (Santee Cooper); Tom Eckman (Northwest Power and Conservation Council); Carla Frisch

(U.S. Department of Energy); Terry Fry (Nexant, Inc).; Lauren Gage (Bonneville Power Administration); Donald 

Gilligan (National Association of Energy Service Companies); Fred Gordon (Energy Trust of Oregon); Sarah Griffith

(Consortium for Energy Efficiency); Dennis Hartline (Maryland Energy Administration); George Hernandez (Pacific 

Northwest National Laboratory); Sami Khawaja (The Cadmus Group Inc); Steve Kromer (Efficiency Valuation 

Organization); Noah Long (Natural Resources Defense Council); Steven Long (Southern California Edison); Beatrice 

Mayo (Pacific Gas & Electric Company); Cecily McChalicher (Northeast Energy Efficiency Partnerships); Jennifer 

Meissner (New York State Energy Research and Development Authority); Eric Merkt (Navigant Consulting); Julie 

Michals (Northeast Energy Efficiency Partnerships); Bill Miller (Sentech Inc.); Monica Nevius (Consortium for Energy 

Efficiency); Larry Philbin (Santee Cooper); Deborah Reynolds (Washington Utilities and Transportation Commission); 

Shahana Samiullah (Southern California Edison); John Stoops (KEMA); David Sumi (The Cadmus Group Inc); Elizabeth 

Titus (Northeast Energy Efficiency Partnerships); P. S. (Vish) Vishwanath (V-TECH, Inc); Bryan Ward (The Cadmus

Group Inc); Tory Webber (Southern California Edison); and Holly Wiggins (Dayton Power & Light).
June 2011 www.seeaction.energy.gov iv

List of Acronyms and Abbreviations

ACEEE American Council for an Energy-Efficient Economy

AFUE Annual Fuel Utilization Efficiency 

Btu British thermal unit

CALMAC California Measurement Advisory Council

CEE Consortium for Energy Efficiency

CFL compact fluorescent light bulb

CMS content management system 

Council Northwest Power and Conservation Council

CPA conservation potential assessment

CPUC California Public Utilities Commission

DEER Database for Energy Efficient Resources 

DOE U.S. Department of Energy 

DS&A Deemed Savings and Algorithm 

DWH domestic water heating

ECM electronically commutated motor

EF energy factor

EM&V evaluation, measurement, and verification

ETO Energy Trust of Oregon

FTE full-time equivalence

HE high efficiency

HP horsepower

HVAC heating, ventilation, and air conditioning

kWh kilowatt hour

LBNL Lawrence Berkeley National Laboratory

M&V Measurement and verification

MAPE Market Assessment and Program Evaluation 

MMBtu 1 million British thermal units

NEEA Northwest Energy Efficiency Alliance

NEEP Northeast Energy Efficiency Partnership

NLCBT National Laboratory Collaborative on Building Technologies

NREL National Renewable Energy Laboratory

NTG nitroglycerin 

NY DPS New York Department of Public Service

NYSERDA New York State Energy Research and Development Authority

PUC Public Utility Commission

RFP Request for Proposals

RTF Regional Technical Forum

SEE Action State and Local Energy Efficiency Action Network

TRM Technical Reference Manual

W watt
June 2011 www.seeaction.energy.gov v

Definitions

TRM (technical reference manual)* is a resource document that includes information used in program planning and 

reporting of energy-efficiency programs. It can include savings values for measures, engineering algorithms to 

calculate savings, impact factors to be applied to calculated savings (e.g., net-to-gross values), source 

documentation, specified assumptions, and other relevant material to support the calculation of measure and 

program savings. A TRM may be in the form of a document or an electronic database. TRMs are currently in effect 

in 21 states. 

EM&V documents database is a Web-based repository of information on evaluation, measurement, and verification 

(EM&V) plans and reports. 

Energy efficiency measure* is an installed piece of equipment or system, or modification of equipment, systems, or 

operations on end-use customer facilities that reduces the total amount of electrical or gas energy and capacity that 

would otherwise have been needed to deliver an equivalent or improved level of end-use service.

Ex ante savings values* are forecasted savings used for program and portfolio planning purposes.

Ex post savings values* are savings estimates reported by an evaluator after the energy impact evaluation has been 

completed. 

EM&V reports are formal documents describing the methodology, sources of data, and results of energy-efficiency 

program impact evaluations or project measurement and verification activities. EM&V reports are often prepared 

by third-party, independent contractors. 

EM&V plans establish the general procedures and methods for determining the energy savings and other impacts 

of energy efficiency measures, projects, programs, and/or portfolios. EM&V plans may be specific to a particular 

evaluation activity, or come in the form of a general protocol for evaluation of all programs. 

Market studies* are analyses that provide an assessment of how and how well a specific market or market segment 

is functioning with respect to the definition of well-functioning markets or with respect to other specific policy 

objectives. Common types of such studies are market characterizations, appliance and equipment saturation 

studies, conservation potential studies, and benchmarking and best-practice studies. 

Process evaluation reports* are a systematic assessment of an energy efficiency program for the purposes of 

documenting program operations at the time of the examination and identifying and recommending improvements 

to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of 

participant satisfaction.

* Adapted from the EM&V Forum Glossary of Terms and Acronyms, Version 2, March 2011.
June 2011 www.seeaction.energy.gov vi

Table of Contents

Executive Summary.................................................................................................................................viii

Purpose and Approach ...............................................................................................................................viii

Current Situation .......................................................................................................................................... ix

Summary of Findings ................................................................................................................................... ix

EM&V Documents...................................................................................................................................................................................... ix

National DS&A Database .........................................................................................................................................................................x

Recommended Solutions ............................................................................................................................. xi

Introduction .................................................................................................................................................1

Report Organization ......................................................................................................................................1

Approach......................................................................................................................................................1

Review of Existing Resources.......................................................................................................................2

Internal Expert Interviews..............................................................................................................................2

Interviews with Industry Professionals ..........................................................................................................2

Review of Existing Resources...................................................................................................................3

Existing EM&V State, Regional, and National Documents Resources and Databases ...............................4

Existing National or Regional EM&V Documents Databases.................................................................................................... 5

States and Other Organizations’ EM&V Documents Databases .............................................................................................. 5

CALMAC.......................................................................................................................................................................................................... 5

CEE MAPE...................................................................................................................................................................................................... 6

Existing State, Regional, and National TRM Resources ..............................................................................6

TRM Review.................................................................................................................................................................................................. 9

Measure Review .........................................................................................................................................12

Findings .....................................................................................................................................................19

EM&V Documents Database ......................................................................................................................19

Needs.............................................................................................................................................................................................................19

Feasibility ....................................................................................................................................................................................................19

Content and Functionality ....................................................................................................................................................................20

National (or Regional) Deemed Savings and Algorithm Database.............................................................22

Needs.............................................................................................................................................................................................................22

Feasibility ....................................................................................................................................................................................................23

Content and Functionality ....................................................................................................................................................................24

Quality Control ..........................................................................................................................................................................................27

Comparison of National DS&A Database to Existing TRM Resources.................................................................................27

Cross-Database (DS&A and EM&V Document Database) Considerations ...............................................28

Hosting..........................................................................................................................................................................................................28

Funding Sources........................................................................................................................................................................................28

Criteria for Best Practices.....................................................................................................................................................................29

Links Between EM&V Documents and DS&A Databases ..........................................................................................................29

Recommendations ....................................................................................................................................30

EM&V Document Database—Focused.......................................................................................................33

Content .........................................................................................................................................................................................................33

Technology..................................................................................................................................................................................................34

Cost.................................................................................................................................................................................................................34

Implementation.........................................................................................................................................................................................35

DS&A Database—Focused.........................................................................................................................36

Content .........................................................................................................................................................................................................36

Technology..................................................................................................................................................................................................36

Costs...............................................................................................................................................................................................................36

Implementation.........................................................................................................................................................................................36

EM&V Document Database—Comprehensive ...........................................................................................36
June 2011 www.seeaction.energy.gov vii

Content .........................................................................................................................................................................................................36

Technology..................................................................................................................................................................................................37

Cost.................................................................................................................................................................................................................38

Implementation.........................................................................................................................................................................................40

DS&A Database—Comprehensive .............................................................................................................40

Content .........................................................................................................................................................................................................40

Technology..................................................................................................................................................................................................42

Cost.................................................................................................................................................................................................................42

Implementation.........................................................................................................................................................................................44

Conclusion.................................................................................................................................................44

List of Tables

Table 1. External Interview Respondents3

Table 2. Sources for EM&V Plans, Reports, and Market Studies4

Table 3. TRM Resources*7

Table 4. Results of Measure Ranking Process11

Table 5. List of 10 Measures for Analysis and Review, Each in the Residential and Commercial Sectors12

Table 6. Residential Measure Comparison Findings and Recommendations Summary14

Table 7. Commercial Measure Comparison Findings and Recommendations Summary16

Table 7. Commercial Measure Comparison Findings and Recommendations Summary18

Table 8. Recommendations of Possible Candidates for Database Hosting28

Table 9. Resource Options31

Table 10. Estimated Development Labor and Cost (Focused EM&V database)34

Table 11. Annual Recurring Labor and Costs (Focused EM&V Database)35

Table 12. Estimated Development Labor and Cost (Comprehensive EM&V Database)39

Table 13. Annual Recurring Labor and Costs (Comprehensive EM&V Database)40

Table 14. Estimated Development Labor and Cost (Comprehensive DS&A Database)43

Table 15. Annual Recurring Labor and Costs (Comprehensive DS&A Database)44
June 2011 www.seeaction.energy.gov viii

Executive Summary

This report presents the results of a scoping study to assess the need for national databases that can support best 

practices in energy-efficiency program evaluation, measurement, and verification (EM&V) and to identify options 

for meeting those needs. The study was commissioned by Lawrence Berkeley National Laboratory on behalf of the 

State and Local Energy Efficiency Action Network’s (SEE Action) EM&V Working Group.

Purpose and Approach

The SEE Action Network’s EM&V Working Group recognized the need for developing a set of databases to support 

best practices in energy efficiency program design and EM&V as a high-priority project. These databases are 

expected to facilitate energy efficiency program design and evaluation, particularly in states with nascent energy 

efficiency initiatives. 

The working group expressed interest in two categories of publicly accessible, national databases:

1. National Deemed Savings and Algorithm (DS&A) Database: A database of standardized, regionspecific, ex ante algorithms and associated savings estimates for conventional electric and natural gas 

energy efficiency measures

2. EM&V Document Database: A database to provide access to available existing EM&V plans, EM&V 

reports, market studies, and other supplemental documents. 

The focus for these databases is to support ratepayer-funded programs administered by utilities or third-party 

agencies and overseen by regulators.

The principal objectives of this scoping study were to: 

 Assess the need for these databases through surveys of energy efficiency experts and potential users

 Assess the consistency of existing technical reference manuals (TRMs) and other sources of deemed 

savings, and issues associated with developing a national DS&A database by comparing assumptions, 

savings values, and algorithms for 20 common energy efficiency measures, as reported in various 

existing state and regional databases

 Assess existing EM&V document database resources and the issues associated with developing a 

national EM&V documents database

 Evaluate the feasibility and costs of alternative options for meeting the identified database needs 

 Recommend practical options for the development, contents, and maintenance of a national DS&A 

database and EM&V document database that effectively satisfies the broadest set of needs 

identified.

The Cadmus Group Inc. compiled data for this study from secondary and primary sources. We began the study with 

a review of the publicly available sources for EM&V plans and reports, TRMs, and market studies. We compiled 

information on current needs and available options through 29 semi-formal, structured interview sessions with 35 

energy efficiency professionals, including members of the SEE Action Network EM&V Working Group and industry 

experts representing utilities, non-governmental regulators, consulting firms, and information technology 

professionals. 

The interviews covered a broad range of topics regarding the databases, with a focus on perceived needs, feasibility

of developing various database options, desirable content and functionalities, potential administrative structures, 

and funding options. 
June 2011 www.seeaction.energy.gov ix

Current Situation

A number of sources for EM&V documents and TRMs are currently available online. In our research for this study, 

we identified 19 publicly available sources providing EM&V documents. These sources are diverse in content, 

structure, and functionality. They also range in sophistication from simply providing links to reports housed 

elsewhere to fully-searchable databases, allowing filtering by sector, report type, and other criteria. Most EM&V 

reports available online are housed on national or regional association websites, or on websites of state-level 

entities, such as the California Measurement Advisory Council. 

However, a nationwide comprehensive database of EM&V documents does not exist. The Consortium for Energy 

Efficiency website appears to be the most comprehensive source for such documents, although its scope is limited 

to submissions only from its members. 

EM&V plans are far less prevalent than EM&V reports. In this study, we identified only three sources of publicly

available EM&V plans, consisting of those jurisdictions—California, Pennsylvania, and New York—where statewide 

EM&V protocols are in place. 

We also identified 17 sources with ex ante measure savings values, covering 21 states and the District of Columbia. 

These sources differ in geographic coverage, but are largely regional and are administered by state regulatory 

commissions, state advisory committees, non-profit organizations, or utilities. The research, development, and 

maintenance of these sources are generally contracted out to consultants. The only sources with a national scope 

are those available through the ENERGY STAR®program, which provides measure-specific savings calculators, 

generally in a Microsoft Excel®format. The other sources vary in information presented, from tables of ex ante

stipulated savings estimates to algorithms.

Summary of Findings

EM&V Documents

The information we gathered from interviews provides overwhelming evidence of the need for an EM&V document 

database. Nearly all respondents agreed that priority should be given to the needs of new EM&V practitioners, with 

a focus on an audience of efficiency program administrator (primarily utility) staff and state regulators, although the 

database should be potentially useful to anyone. Nearly every interviewee agreed that some form of a centralized 

resource could help expand best practices in EM&V, create greater transparency and consistency of savings 

estimates, facilitate more widespread exchange of information and data-sharing among practitioners, and improve 

the overall quality of EM&V throughout the United States. A particular value is providing case studies/examples for 

those regulators and program administrators just starting to establish EM&V requirements and procedures.

The information obtained also points to concerns about the feasibility of such an undertaking, particularly with 

establishment and funding needs, procedures for cataloguing documents, structures for administration and ongoing 

maintenance as well as ensuring long-term consistency and quality. Interviewees generally agreed that it would be 

important, but difficult, to create an institutional framework and infrastructure that encourages voluntary submittal 

of materials as they become available. Many thought continuous and consistent voluntary participation could prove 

to be the real barrier for the national EM&V document database.

Perspectives varied with respect to the content and structure of a new, national EM&V documents database. The 

main question raised by interviewees was whether the database should be a comprehensive repository of all EM&V 

documents, or a more selective sample of documents that, according to certain criteria, distinguish them as 

exemplary or best practice. While arguably a selective database might prove more useful to new practitioners, there 

was little agreement as to what screening criteria should be included and how documents should be selected. 

Interviewees made several suggestions about how rigorous the screening process should be. These ranged from a 

simple check to determine whether the document had been accepted or approved formally by regulators, to a 
June 2011 www.seeaction.energy.gov x

requirement that documents be subjected to stringent review and ranking processes by experts, based on 

predetermined criteria or perhaps just their judgment. In addition, the criteria of “best practices” can be quite 

problematic, as it is a relative versus absolute term.

A frequent suggestion was to create a database to serve as a “starter kit” or a set of examples and templates for 

state regulators and program administrators new to energy efficiency. This focused database, as we call it in this 

report, would guide this audience toward best and/or common practice approaches for developing EM&V 

requirements for their energy efficiency initiatives. 

Regardless of the scope, the database search capabilities will likely prove to be one of its most important functions. 

Interviewees made a number of suggestions for searching and filtering capabilities, including key words, program 

type, sample size, year of publication, methodologies, location, fuel type, and some measure of the study 

complexity. 

National DS&A Database

There is a widespread recognition of the usefulness of national, or at least regional, databases of deemed savings 

and algorithms. However, based on our interviews, many barriers would need to be overcome before developing 

such a resource that meets the needs of states. In general, the notion of creating a national resource was not 

considered readily possible, largely due to vast regional variations in baselines, weather, economics, demographics, 

equipment stocks, measure costs, program structures, evaluation needs, and, in particular, regulatory 

requirements. However, developing several regional databases is possible and in total could represent a 

national deemed savings and algorithms.

The major concerns raised by many respondents were: (a) the feasibility of creating such a database and of 

practitioners coming to a consensus (in a given state or region, let alone nationally) on the appropriate methods, 

algorithms, and assumptions; and (b) having individual states’ regulatory agencies, which oversee efficiency 

programs, adopting such a database when they have either existing TRMs or processes in place to develop ex ante

savings values, algorithms, etc. Transparency in measure assumptions is critical to gaining acceptance, and a 

consistent methodology for collecting measure data is needed. To create a resource that could be used to compare 

savings across programs or states, a high level of detail and guidance would be required for each measure to ensure 

transparency and reduce the likelihood of human errors. Another substantial challenge would be the constant 

updates required to adjust measure savings values based on ongoing research, changes in baselines, and new data. 

Interviewees differed widely on the optimal structure, content, and functionalities of one or more regional (and/or 

national) DS&A databases. In general, respondents reported there is a greater need for a database of saving 

algorithms rather than values. The possibility of providing additional functionality of a dynamic database with 

embedded algorithms (that is, allowing the user to solve for the algorithm through the interface) was also 

discussed. Although it may offer greater flexibility and be useful to a broader range of users, the development and 

ongoing maintenance required may prove too difficult to overcome. A few respondents also indicated that the 

energy efficiency focus should be on whole-building savings and not measure-specific values, decreasing the 

usefulness of a national DS&A database, as that type of resource focuses on per unit savings. Finally, the large 

majority of experts reported that a national DS&A database would not be used consistently unless mandated by 

regulators, and most opposed such a mandate, particularly in those states where a TRM has already been 

developed. 

Our detailed review of 20 measures across 17 TRMs confirmed the legitimacy of these concerns. The TRMs had a 

wide variation in saving methodologies, technical assumptions, and input variables for estimating savings. Even 

where algorithms are similar, the input units and baseline assumptions are typically different. In addition, a number 

of the resources we reviewed had errors, highlighting the difficulty in creating such a resource. These all point to 

the challenges of creating a regional or national DS&A database, and the importance of consistent assumptions, 

quality control, and long-term and substantial funding. 
June 2011 www.seeaction.energy.gov xi

Recommended Solutions

We began the scoping study by considering a broad range of solutions for the two databases (EM&V documents and 

DS&A), based on the perceived needs of both new and experienced practitioners, the desirable features and 

functionalities, and technology options. The information we gathered in the course of interviews and from our more 

detailed review of existing resources and research on available technologies helped to narrow the options into two 

possible configurations for each of the two databases, differentiated primarily in terms of the overall complexity 

and scope. We identify these as focused and comprehensive options (see Table ES-1). These two options vary 

primarily by their levels of user interface and how static the data are (particularly for the DS&A database). The 

focused resources are primarily targeted for new practitioners; the comprehensive resources are for expert users. 

New practitioners may find the comprehensive resources overwhelming, while expert users may find the focused 

resource lacking. This report includes estimates of resources required, both in employee-years (2,080 annual hours) 

or full-time equivalent (FTE) employee hours and dollars for both the development of the database as well as 

ongoing administration and maintenance. 

Table ES-1. Resource Options

Option Targeted 

User

EM&V Documents DS&A

Focused New Practitioner –

individual or 

organization 

seeking 

introductory 

information

Content:

 Links to existing resources

 Guidelines on best practices

 Exemplar documents (based on 

expert input) 

Content:

 Links to existing TRMs and other 

existing measure resources

 White papers on technologies

 Example algorithms with details on 

parameters used 

Cost:

 Development 

Employee-years: 1.7 – 2.5

Cost: $520k-$775k

 Annual Maintenance

FTE: 0.6 – 0.9

Cost: $200k-$300k

Cost:

 Development 

Employee-years: 1.0 – 1.5

Cost: $325k-$500k

 Annual Maintenance

FTE: 0.4 – 0.6

Cost: $125k-$185k

Comprehensive Expert User –

individual or 

organization 

seeking 

supplemental or 

comparative data 

Content:

 Large collection of diverse 

documents

 Sophisticated search and tagging

 Regular additions and updates

 Data collected during evaluation

Content:

National (or regional) database(s) of 

savings estimates based on either:

 A database of verified past results 

and/or modeling data

Or

 An interactive calculation engine 

based on a set of predefined 

algorithms (i.e. an online TRM)

Cost:

 Development Cost

Employee-years: 2.7 – 4.0

Cost: $825k-$1.25M

 Annual Maintenance

FTE: 1.0 – 1.5

Cost: $325k-$500k

Cost:

 Development

Employee-years: 3.7 – 5.5

Cost: $1.1M-$1.7M

 Annual Maintenance

FTE: 1.8 – 2.6

Cost: $560k-$850k
June 2011 www.seeaction.energy.gov xii

If the two focused database options were combined into a robust starter kit resource, an integrated solution could 

likely be developed and deployed closer to the cost of the focused EM&V document resource ($520k–$775k) with 

an additional incremental cost of approximately $30k for development and management of the additional content 

required.

Although the results of this study indicate interest in both types of databases, a comprehensive EM&V document 

database was considered by interviewees to be more viable than a DS&A database. We recommend initially 

developing the starter kit, combining the features of the focused DS&A and EM&V documents databases described 

above. This would provide a database of documents and links to ex ante savings values and algorithms within a 

single website and create a summary resource for those new to energy efficiency. 

The focused solution could serve as a sound starting point, which may evolve into a more comprehensive solution 

over time, if built using an open source content management system that allows for expansion. By starting with a 

smaller-scale option, it would be possible to better gauge users’ needs and use of the databases, gather feedback, 

and expand the solution as appropriate. In terms of developers and administrators of these databases, the findings 

of the study indicate that the U.S. Department of Energy, either directly or through an agent, would be the entity 

best suited to spearhead and fund the development and deployment efforts as well as provide credence to its 

validity and value.
June 2011 www.seeaction.energy.gov 1

Introduction 

Lawrence Berkeley National Laboratory (LBNL) provides research support and technical assistance to the State and 

Local Energy Efficiency Action Network (SEE Action) on selected energy efficiency policies, programs, and technical 

issues. 

The SEE Action Network EM&V Working Group has recognized the need to develop a set of databases to support 

energy efficiency evaluation, measurement, and verification (EM&V) as a high-priority project. These databases 

would help standardize ex ante measure savings, and would provide easy access to available best practice EM&V 

plans, reports, market studies, and other related studies. The databases are expected to facilitate energy efficiency 

program design and evaluation, particularly in states with nascent energy efficiency initiatives. The focus of these 

databases is to support ratepayer-funded programs, administered by utilities or other third-party entities, and 

overseen by regulators.

Under subcontract with LBNL, The Cadmus Group Inc. (Cadmus) conducted this scoping study to research the 

various options for the content, design, and administration of such databases, and to provide estimates of their 

associated costs. The EM&V Working Group envisions the development of two or three types of publicly accessible 

databases:

 National Deemed Savings and Algorithm (DS&A) Database. A national database of standardized,

region-specific, ex ante algorithms and associated stipulated savings estimates for conventional 

electric and natural gas energy efficiency measures. 

 EM&V Document Database. One or more national databases that provide access to the energy

efficiency EM&V plans and reports available in various state jurisdictions and utilities.

This report presents the results of the scoping study. We begin with a review of currently available resources, and 

then assess the need for such databases according to the views of energy efficiency industry experts and new 

practitioners. Next, we determine gaps between the two groups (experts and new practitioners) in terms of 

information and data needs, and provide a framework for database systems and a supporting infrastructure to fill 

this gap.

Report Organization

This report is organized into five parts. This introduction is considered Section 1. Section 2 describes the overall 

approach of the study. Section 3 presents the results of our existing resources review. Section 4 summarizes 

findings from the interviews and from our review of ex ante measure savings in available databases. Section 5 offers 

recommendations for alternative database options and their associated costs to build and maintain. Finally, we 

report detailed technical specifications of measures from available databases in Appendices A and B (see separate 

document). These appendices document the results of our review of the approaches to calculating savings for 20 

measures across various technical reference manuals (TRMs). Appendix C lists the websites that contain EM&V 

documents.

Approach

Cadmus compiled the data for this study from secondary and primary sources. The secondary data collection effort 

included a review of available resources for EM&V plans and reports, TRMs, and market studies from a variety of 

sources, including the Internet. We collected primary data through structured and informal interviews with industry 

experts, SEE Action EM&V Working Group members, and several potential future users of the database.
June 2011 www.seeaction.energy.gov 2

Review of Existing Resources

The first stage of the scoping study involved gathering information on various existing databases at the national, 

regional, state, utility, and state regulatory commission levels. For the initial search and review, we developed a list 

of entities with a broad scope, such as the Efficiency Valuation Organization and the International Energy Program 

Evaluation Conference, and identified states and entities most experienced in EM&V. We later expanded this list 

based on feedback from the industry experts interviewed. 

To facilitate data collection, Cadmus tracked whether each entity’s website included EM&V plans, reports, and 

market studies, and other types of reports. When documents could not be readily found, we applied key word 

searches throughout the website content for terms such as evaluation, report, study, and plan. Due to time and 

budget constraints, if we could not locate documents on a website within 30 to 45 minutes, we assumed they were 

not included online or were not readily accessible. In a few cases, during our interviews we received additional 

information on the availability of these data. In addition, where readily accessible online, we recorded information 

about the database administration method, data storage format, search capabilities, and other database features.

We made a parallel search for sources with data on ex ante measure savings algorithms and/or values. These 

included existing databases such as the Northwest Regional Technical Forum (RTF), California’s Database for Energy 

Efficient Resources (DEER), ENERGY STAR®calculators, and TRMs adopted in various states. 

As part of this effort, Cadmus compiled a comprehensive list of specific energy efficiency measures with savings 

algorithms or values from these sources. As described in Section 3 of this report, we then ranked these measures 

based on criteria such as prevalence, sector diversity, and magnitude of potential energy savings. We then selected 

20 measures for further comparison of their baselines, assumptions, calculation procedures, and reported saving 

values as found in existing databases. This comparison and review of existing databases provided insights on the 

feasibility of creating a new national database, or multiple regional databases, of ex ante measure savings 

algorithms and values, based on commonalities among savings algorithms or values, and extrapolation of those 

algorithms to the national level. 

Internal Expert Interviews

After our initial review of existing sources, we conducted interviews with four Cadmus staff members who are 

energy efficiency EM&V industry experts1to elicit information on state, national, and regional level entities that 

have undertaken a significant number of evaluations over the last five years. This allowed us to gauge the 

completeness of our existing resources list and determine if we should expand our research. These internal experts 

also provided insights into potential topics to include in our interviews with respondents outside Cadmus, and they 

furnished a list of potential candidates for interviews. These internal interviewees also provided information on 

possible criteria for determining whether a report, plan, or market study represented best practices.

Interviews with Industry Professionals

Next, we conducted structured, informal interviews with industry experts outside Cadmus. Through these 

interviews, we collected information on the evaluation plans and reports database, in addition to the measures 

database. We determined the discussion topics for these interviews based on the overall project objectives and on 

the insights gained through our Cadmus expert interviews. Our interviews with industry professionals focused on 

the following for both types of databases:

 

1

Cadmus experts interviewed were M. Sami Khawaja, Ben Bronfman, Bryan Ward, and David Sumi.
June 2011 www.seeaction.energy.gov 3

 Perceived needs

 Feasibility

 Content

 Functionality

 Structure/organization

 Data sources

 Funding and administration.

Our first key interview questions addressed whether respondents believe there is a need for the databases and 

what specific needs each type of database might address. We asked about their concerns regarding the feasibility of 

these databases to identify potential problems that could occur in database creation and upkeep. Cadmus 

interviewers also asked about the desired content and functionality of each database, specifically in terms of the 

types of information that should be included and the desirable search options and functional capabilities. We 

discussed the database structure and organization with interviewees to determine their database layout 

preferences. To obtain feedback on interviewees’ experience with other databases and their preferences, we asked 

for their observations of each database we mentioned. The data we gathered on functionality and features helped 

frame other research areas, such as cost, necessary components, selection processes, and ongoing administration. 

Moreover, we used this opportunity to elicit suggestions for funding sources and database administration options. 

We conducted 29 telephone interviews. The interviewees represented utilities, regional non-governmental 

organizations, regulators, and consultants. In several cases, more than one respondent participated in the call, and 

we were able to solicit input from 35 individuals. Interviewees included industry experts, personnel from three 

utilities that are relatively new to energy efficiency and EM&V (referred to as nascent practitioners in this report), 

and members of the SEE Action EM&V Working Group (see Table 1). 

Table 1. External Interview Respondents

Respondent Type Number of Interviews

Industry Experts 16

Nascent Practitioners 3

SEE Action Network EM&V Working Group Members 7

Database Experts and Administrators 3

Total 29

We conducted additional interviews with three database administrators, one of which is a database developer, to 

gain insights into specific database management details. We also conducted an interview with a U.S. Department of 

Energy (DOE) staff member who is also a SEE Action facilitator. 

Review of Existing Resources

This section presents the results of our existing resources review. The first subsection presents results from our 

review of EM&V resources, followed by the results of our review of existing TRM resources and other sources of ex 

ante measure savings values and algorithms.
June 2011 www.seeaction.energy.gov 4

Existing EM&V State, Regional, and National Documents Resources and Databases

Existing EM&V resources and online databases are diverse in their content, structure, and functionality. They also 

range in sophistication from simply providing links to reports housed elsewhere to having fully searchable databases 

that allow filtering by sector, report type, and other criteria. Most EM&V reports available online are housed on 

national or regional association websites or on websites of state-level entities. An all-inclusive national database of 

EM&V reports, plans, and/or market studies does not currently exist. The Consortium for Energy Efficiency (CEE)

website appears to be the most complete resource for such reports, although its scope is limited to submissions 

from only its members. 

Entities providing online sources of EM&V plans, reports, and market studies are listed in 

Table 2, indicating the types of documents available from each. A complete summary of findings, with Web 

addresses, for all resources we researched is in Appendix C. 

Table 2. Sources for EM&V Plans, Reports, and Market Studies

Scope/State

Organization 

Type Organization Name

EM&V 

Plans

EM&V 

Reports

Market 

Studies

National Nonprofit Consortium for Energy Efficiency 

(CEE)

√ √

Nonprofit American Council for an Energy 

Efficient Economy (ACEEE)

√

Regional NE Nonprofit Northeast Energy Efficiency 

Partnerships EM&V Forum

(NEEP)

√ √

Regional 

PNW

Nonprofit Northwest Energy Efficiency 

Alliance (NEEA)

√ √

Regional Public Organization/ 

Power Producer

Bonneville Power Administration 

(BPA)

√ √

California Public University University of California -

California Institute for Energy and 

Environment (UC CIEE)

√ √

Public Utilities 

Commission (PUC)

California Public Utilities 

Commission (CPUC)

√ √ √

PUC California Measurement Advisory 

Council (CALMAC)

√ √

Connecticut State Utility 

Partnership

Connecticut Energy Efficiency 

Fund

√ √

Maine Program 

Administrator

Efficiency Maine √ √

Missouri Utility Ameren √ √

New York Third-Party 

Administrator

New York State Energy Research 

and Development Authority 

(NYSERDA)

√ √

State Department New York Department of Public 

Service (NY DPS)

√ √

Oregon Third-Party 

Administrator

Energy Trust of Oregon (ETO) √ √

Pennsylvania PUC Public Utility Commission √

Texas Other Electric Utility Marketing 

Managers of Texas

√ √

PUC Texas PUC √ √
June 2011 www.seeaction.energy.gov 5

Table 2. Sources for EM&V Plans, Reports, and Market Studies

Scope/State

Organization 

Type Organization Name

EM&V 

Plans

EM&V 

Reports

Market 

Studies

Washington Utility Seattle City Light √ √

Wisconsin Utility Organization Wisconsin Focus on Energy √ √

Existing National or Regional EM&V Documents Databases

The national and regional websites that contain repositories of EM&V reports were those for CEE, Northeast Energy 

Efficiency Partnerships (NEEP) EM&V Forum, and the Northwest Energy Efficiency Alliance (NEEA). The availability of 

these reports varied across the existing databases. While all the databases we reviewed contained EM&V reports, 

EM&V plans were only available from three entities, consisting of the jurisdictions in California, Pennsylvania, and 

New York. This is primarily due to the fact that, in most cases, the plans are not required to be part of the public 

record, tend to change during the evaluation process, and are summarized as part of the final EM&V report. 

Our expert interviewees confirmed issues that will likely limit the availability of EM&V plans, including concerns 

about the proprietary nature of methods and techniques, particularly for site-specific measurement and verification 

(M&V). Another issue is that even in cases where statewide EM&V protocols exist, specific evaluation methods tend 

to be described in the evaluation reports: the plans themselves may not be required. Nearly all of the sites we 

reviewed included some market studies. These national and regional sites varied with respect to salient 

characteristics, such as functionality and search capability. 

States and Other Organizations’ EM&V Documents Databases

In our review of state utility commission and individual utility sources online, we found considerable inconsistency 

in the availability of EM&V reports and market studies. Approximately one-third of state commissions and energy 

efficiency organizations provide these resources online. The databases ranged from very simple (consisting of links 

to reports and studies housed elsewhere) to extensive repositories with search capabilities. State-level databases 

varied in complexity, but most simply provided lists of reports with links to embedded documents. Several of the 

websites we reviewed, specifically from the Energy Trust of Oregon (ETO) and the California Measurement Advisory 

Council (CALMAC), provided complex databases with search features. Most sources only posted reports for studies 

and evaluations conducted in their geographic areas (usually by state). Only the California Public Utilities 

Commission (CPUC) and the New York Department of Public Service (NY DPS) websites provided EM&V plans.

Two of the identified resources have been widely used as repositories for EM&V reports and other documents, and 

both are also maintained: CALMAC’s searchable database and CEE’s Market Assessment and Program Evaluation 

(MAPE) Clearinghouse. As these resources could serve as a basis for a national database, we describe each in 

further detail.

CALMAC

The most widely known and used EM&V document resource is CALMAC. In 2000, CALMAC was started as a 

partnership between three participating utilities that wanted to share and make the documents they generated 

publicly available. CALMAC now consists of the three original utilities (San Diego Gas and Electric, Pacific Gas and 

Electric, and Southern California Edison), as well as the California Energy Commission, and the CPUC. CALMAC’s 

online database contains over 1,000 documents related to energy efficiency, demand-response, low-income, and 

self-generation programs, with the first two categories having the most representation. Document categories 

included evaluation guidelines, program evaluations, and market studies. Documents were searchable, based on 
June 2011 www.seeaction.energy.gov 6

text searches of key fields and document abstracts, and results could be filtered by program type, category, author, 

sector, or date.

When initially populated, CALMAC posted over 500 documents to the site, dating back to 1990. Most of these were 

scans of original paper documents. Once CALMAC became well established and highly utilized, the CPUC mandated 

submission of all California energy efficiency and demand-response documents to the site.

The site’s technology remains essentially the same as when it was created. The search capabilities do not include 

full text searches of documents. Rather, submissions must include abstracts containing specific information, 

including a summary of results and any relevant key words or phrases. CALMAC relies on document authors or 

sponsors to submit documents to the site, and does not actively gather new materials. Submissions are made via an 

online form with attached documents, and the CALMAC administrator screens submissions for completeness before 

final posting.

CEE MAPE

Another online document resource is the MAPE Clearinghouse, administered by the CEE. CEE membership is open

to organizations that have a regulatory or legislative mandate to administer energy efficiency programs as well as to 

other public stakeholders in such programs. Such administrators may be utilities, or statewide and regional 

organizations or agencies. MAPE contains over 1,000 documents, submitted for posting by CEE members. 

Documents are accessed by using a simple text search field. Results contain short document summaries and links to 

PDFs of executive summaries and full documents. While currently publicly accessible, the CEE is working on a more 

sophisticated online database, which will be accessible to CEE members exclusively. The updated site will be built 

using the DrupalTM Content Management System (CMS) platform. 

Existing State, Regional, and National TRM Resources 

We identified 17 resources for measure savings values and algorithms, covering 21 states and the District of 

Columbia (see Table 3, identified as of February 2011). These resources differed in coverage from state to region to

national levels. Most of the resources we identified are administered by a state commission or agency. The 

remaining resources are administered by advisory committees, non-profit organizations, trusts, or public utilities. 

Many of these administrative groups contract out the research, development of calculations, and creation of the 

resources to consulting firms. Table 3 includes an indication of the measure savings information contained within 

the resource: ex ante savings values or algorithms. For the resources that have ex ante savings estimates, a 

stipulated (deemed) value is given for the savings. For the resources primarily composed of algorithms, a formula is 

given for where the input parameters are stipulated or determined based on specific conditions (e.g., a lookup table 

for hours-of-use based on building type). For the resources that include both algorithms and ex ante savings values, 

the primary format for measure savings is algorithms in nearly all cases. Massachusetts is the exception, where 

algorithms and values are roughly equally represented. These resources primarily contained energy savings and 

measure costs; few include non-energy benefits or market transformation metrics. 
June 2011 www.seeaction.energy.gov 7

Table 3. TRM Resources*

Scope Resource Name Format

Information 

Included Administrator

National ENERGY STAR Online 

Calculators

Ex ante savings

based on algorithms

Agency

Regional –

Northwest

Regional Technical Forum (RTF) Deemed 

Measures

Online 

Database

Ex ante savings

based on algorithms

Advisory 

Committee

Regional –

Midatlantic

Mid-Atlantic TRM PDF Algorithms and 

Ex ante savings

Non-Profit 

Organization

Arkansas Arkansas Deemed Savings Quick Start 

Programs

PDF Algorithms Public Utility

California Database for Energy Efficient Resources 

(DEER)

Software 

Program

Ex ante savings State Commission

Connecticut Connecticut Light & Power and United 

Illuminating Company Program Savings 

Documentation

PDF Algorithms and 

Ex ante savings

Public Utility

Hawaii Hawaii Energy Efficiency Program TRM PDF Algorithms and

Ex ante savings

State Commission

Maine Efficiency Maine TRM PDF Algorithms and

Ex ante savings

Trust

Massachusetts Massachusetts Statewide TRM for 

Estimating Savings from Energy Efficiency 

Measures

PDF Algorithms and

Ex ante savings

Agency

Michigan Michigan Energy Measures Database Excel 

Database

Ex ante savings State Commission

New Jersey New Jersey Clean Energy Program 

Protocols to Measure Resource Savings

PDF Algorithms and

Ex ante savings

Agency

New York New York Standard Approach for 

Estimating Energy Savings from Energy 

Efficiency Programs

PDF Algorithms and

Ex ante savings

Agency

Ohio Ohio TRM Online 

Database

Algorithms and

Ex ante savings

State Commission

Pennsylvania Pennsylvania TRM DOC Algorithms and

Ex ante savings

State Commission

Texas Deemed Savings, Installation, and 

Efficiency Standards

PDF Algorithms and

Ex ante savings

State Commission

Vermont Efficiency Vermont Technical Reference 

User Manual

PDF Algorithms and

Ex ante savings

Non-Profit 

Organization

Wisconsin Focus on Energy Evaluation Business 

Programs: Deemed Savings Manual V1.0

PDF Algorithms and

Ex ante savings

State Commission

* These resources were obtained in February 2011. Any updates and amendments since that date were not reviewed.

The only nationally applicable resource, available through the ENERGY STAR program, contains measure-specific 

savings calculators in Microsoft Excel®. These calculators are the only interactive resource available, allowing user 

modification of certain parameters, such as usage hours, number of units, territorial region, equipment size, and 

fuel type. Only measures qualifying as ENERGY STAR have calculators; however, not all ENERGY STAR measures 

currently have calculators.

At the regional level, we reviewed two resources in depth: the Northwest RTF’s online tool and database and the 

Mid-Atlantic TRM. These regional resources are used by the states within each region. The Northwest RTF’s savings 

estimates are used in Idaho, Montana, Oregon, and Washington. The Mid-Atlantic TRM is used in Delaware, 

Maryland, and the District of Columbia. 
June 2011 www.seeaction.energy.gov 8

The remaining 14 resources listed in Table 3 are classified as state resources. They typically address areas with less 

diversity in climate, baseline conditions, and structural characteristics. Most of these resources are available as 

electronic documents in PDF or Microsoft Word®formats. The remaining state resources are Microsoft Excel 

workbooks or custom programs (such as California’s DEER), requiring the user to download specialized software to 

access the database. 

Additional TRM and Related Resources

Three additional resources not listed in Table 3 are available: the National Renewable Energy Laboratory (NREL) 

National Residential Efficiency Measures Database; conservation potential assessment (CPA) reports, including the 

Northwest Power and Conservation Council’s (Council’s) 6th Power Plan; and the DOE Buildings Performance 

Database. 

The format of the NREL database is useful for an online measure and cost database. This resource’s focus is to 

provide cost information for a wide range of residential measures; savings information is sparse. Consequently, we 

did not include this database in our detailed, measure-level analysis.

CPA studies are an additional source of detailed information on measure costs and savings. However, we did not 

include CPAs in this study for several reasons. First, detailed, measure-level information is not always provided in 

publicly available potential study reports. Second, detailed data, where available, can be difficult to extract in a 

useful format, since CPAs tend to be in PDF format, which have no formulas for calculations. Even in rare cases 

where the detailed data are available in a spreadsheet format, the data tend to be hard-coded values or protected, 

and assumptions in the calculations are not known. Moreover, different CPAs report savings in different units (e.g., 

per square foot or per unit of measure), making it difficult to compare across studies. The exception is the 

conservation potential analysis of the Council’s 6th Power Plan, a regional study and a common point of reference 

for the development of potential studies throughout the Northwest. This analysis has relatively transparent and 

readily available calculations of savings, based primarily on data available from the Northwest RTF.

Finally, the National Laboratory Collaborative on Building Technologies (NLCBT), recently launched by DOE, is 

developing the DOE Buildings Performance Database. According to information provided by the DOE project 

personnel2, this data warehouse is designed to store building descriptions, including facility characteristics, such as 

size, functional use, equipment type, location (metadata), and time-series energy consumption data (not saving

values). The DOE Buildings Performance Database will also provide a layer of “tools” for analysis. A data 

specification has been developed to standardize the type of data in the warehouse. It is expected that this 

specification will also support other DOE building data initiatives. Initial data are coming from government buildings

or federal stimulus funded projects (residential and commercial) for which the NLCBT can access consumption 

records. Consumption data may be available in monthly, hourly or intervals as short as every 15 minutes. The “tool” 

layer will initially consist of a savings tool, designed to compute differences in consumption between user-selected 

data points, and a financial analysis tool designed to show the distribution of cash flows arising from those savings 

results. 

NLCBT plans to expand the type of data in the data layer, and to provide an Application Program Interface tool, with 

which users may access and analyze the data. Thus, although the DOE Buildings Performance Database may not 

serve directly as a source for measure-specific information, one can use the data of available consumption and 

facility characteristics in conjunction with various statistical and financial analytic techniques to develop estimates 

of measure-level savings or cost-effectiveness. As the amount of data and range of tools increase, the DOE Buildings 

Performance Database may provide an alternative to estimating savings, based on measured energy usage as 

compared to the traditional, calculation-based approaches now available.

 

2

Personnel communication with Dr. William Miller, Senior Advisor, Sentech, Inc.
June 2011 www.seeaction.energy.gov 9

TRM Review

To inform the feasibility of creating a national DS&A database, Cadmus compared measure savings values and 

algorithms across the resources listed above. We reviewed and compared 20 measures—10 in the residential and 

10 in the commercial sector—and included both electric and gas impacts across the 17 identified resources.3 We 

began our process for selecting the 20 measures by categorizing measures available in the databases by end use, 

fuel type, and sector, then ranked them according to three criteria, as described below. 

End Use

We used the following end-use categories to group measures for this scoping study:

 Appliances  Motor/Pump

 Building Package  New Construction

 Cooking  Other

 Envelope  Plug Load

 HVAC  Refrigeration

 Industrial  Ventilation

 Lighting  Water Heating

Fuel Type

Electricity and natural gas were the primary fuel types we considered. Gas measures typically reported savings in 

MMBtu and, less commonly, in therms. Natural gas is usually the assumed fuel; however, for a small number of 

measures, oil or propane was included. The majority of gas measures produce space- or water-heating savings. 

Measures in the electric category typically reported savings in kilowatt hours (kWh). Energy savings from electric 

measures are mainly attributable to electricity savings, but they may produce gas savings as well. For example, an 

efficient dishwasher in a home with a gas hot water heater may save electricity and gas. 

Measures typically thought of as electric-only may have small but quantifiable effects on gas consumption through 

interactive effects. For example, inefficient lights, such as incandescent light bulbs, waste energy through heat 

generation. That heat affects ambient temperatures, leading to increased conditioned-space cooling loads and 

decreased heating loads. When more efficient and cooler lights are retrofitted into spaces, secondary energy 

savings result from less need for cooling, but heating energy consumption may increase due to the reduced waste 

heat emanating from the lighting. 

Fuel-switching (or conversion) measures can have a large effect on both electricity and gas usage. For example, 

switching from a conventional electric water heater to a high-efficiency gas water heater shifts the energy load from 

the electric grid to natural gas suppliers. We did not consider fuel-switching measures in this review and comparison 

of measures due to their complexity and because regulatory questions exist regarding their appropriateness as an 

energy efficiency measure in many jurisdictions.

Sector

Ex ante savings values and algorithms for measures may also be categorized by the market sector(s) to which they 

apply, such as residential, commercial, or industrial. In some cases, the resources identified the applicable sector(s); 

 

3

This review was also informed by a recent KEMA report: Regional EM&V Forum Report: Common EM&V Methods and Savings 

Assumptions Project, May 2010.
June 2011 www.seeaction.energy.gov 10

however, for several of the resources we reviewed, it was not always apparent to which sector(s) a measure’s 

values or algorithms applied. This ambiguity could sometimes be resolved by examining algorithm inputs such as 

hours-of-use and total operating hours. 

Measures are most commonly reported for the residential or commercial sector. Some resources identified a few 

measures as being applicable to commercial and industrial sectors, such as air compressors and pumps. Most nonlighting measures for the industrial sector (such as process improvements) were custom, and deemed or calculation 

algorithm procedures are not suitable for estimating their savings. 

Ranking Measures

We began with 31 measures in the ranking—14 residential and 17 commercial. These measures were commonly 

listed in the sources we reviewed, and presented a broad selection of end uses while covering both fuel types and 

both sectors. We ranked these measures based on the following criteria to determine their suitability for further 

review and analysis:

 Savings impacts: We gave measures a higher score if they had the potential to produce larger 

savings. We ranked impacts based on our experience with conservation potential studies on a scale 

from 1 to 5, with 1 producing very small savings and 5 producing very large savings.

 Prevalence: We scored measures for prevalence based on the number of TRMs, protocols, or 

databases in which the measure appeared. We ranked prevalence on a scale from 1 to 5, with 1 being 

not at all prevalent and 5 being very prevalent.

 Diversity of calculation methodology: We gave measures with multiple methods for calculating 

savings a higher score to allow alternative methods to be represented. We ranked diversity of 

calculation methodologies according to the following scoring scheme: 

o 1 = Differing units (e.g., lighting savings per house versus per bulb)

o 2 = Differing in two or more adjustment factors

o 3 = Differing treatments for interactive effects

o 4 = Differing treatments for interactive effects and other adjustment factors

o 5 = Completely different approaches used (e.g., deemed vs. equation).

 Fuel diversity: We assigned a value of 1 to measures that produce both gas and electricity savings, 

and assigned a value of 0 to measures that affect only one fuel source. For example, we gave 

measures with an impact on heating loads a score of 1 because they could potentially reduce gas or 

electricity consumption, depending on the heating fuel type.

 Sector diversity: We based the sector diversity score on whether a measure applied to one or 

multiple sectors. We assigned a value of 1 to measures found in both commercial and residential 

sectors, and assigned a value of 0 to measures only found in one sector.

We further weighted the assigned values by importance on a scale of 1 to 5, as shown in Table 4. We then added 

the five weighted rank values to determine the overall ranking of each measure. The 20 highest-ranking measures in 

the residential and commercial sectors are listed in Table 5.
June 2011 www.seeaction.energy.gov 11

Table 4. Results of Measure Ranking Process

Rank Measure Fuel* Combination of 

Weighted Factors Impact Prevalence Diversity Calculation Methodology Fuel Diversity Sector Diversity

Weighting Factor 5 4 3 2 1

Residential

1 Lighting (CFLs) Electric 61 5 5 5 0 1

2 Refrigerator Recycling or 

Retirement

Electric 56 5 4 5 0 0

3 HE Furnace Gas 49 5 2 5 0 1

4 Low-Flow Showerhead Either 46 5 3 2 1 1

5 Ceiling/Attic Insulation Both 45 3 3 5 1 1

6 HE Air-Source Heat Pump Electric 42 4 3 3 0 1

7 ENERGY STAR®

Refrigerator

Electric 40 2 5 3 0 1

7 HE DWH Storage Either 40 2 3 5 1 1

7 Furnace Motor ECM Electric 38 3 2 5 0 0

10 HE Clothes Washer Both 37 1 5 3 1 1

11 HE Room AC Electric 36 2 4 3 0 1

12 Lighting Controls Electric 25 1 1 5 0 1

13 ENERGY STAR Ceiling Fan Electric 22 1 2 3 0 0

Commercial

1 Lighting (linear fluorescents) Electric 55 5 5 3 0 1

2 HE Furnace Gas 53 5 3 5 0 1

3 HE DWH Storage Either 45 3 3 5 1 1

4 Lighting Controls Electric 43 4 4 2 0 1

5 HE Motor Electric 41 3 5 2 0 0

6 HE Air-Source Heat Pump Electric 37 3 3 3 0 1

6 Roof Insulation Both 37 1 3 5 5 5

8 HE Clothes Washer Both 35 2 3 5 5 3

8 Air Compressor Electric 35 3 4 0 0 1

10 Refrigeration ECM Electric 33 3 3 0 0 2

11 ENERGY STAR Refrigerator Electric 32 3 2 5 0 3

11 HE Cooking Gas 32 2 3 0 0 3

13 HE Room AC Electric 28 2 2 5 0 3

14 Vending Machine Electric 27 2 2 0 0 3

15 Traffic Signals Electric 26 2 3 0 0 1

16 Low-Flow Showerhead Either 20 1 2 5 5 1

* Fuel type represents primary fuel saved. “Both” indicates a measure that can save heating and cooling usage, and thus could save both gas and electric fuels. 

“Either” indicates the fuel source for the measure could be electric or gas, so the savings could be electric or gas.
June 2011 www.seeaction.energy.gov 12

Table 5. List of 10 Measures for Analysis and Review, Each in the Residential and 

Commercial Sectors

Residential Sector Measures Fuel Type

1 Lighting – CFLs Electric

2 Refrigerator Recycling / Retirement Electric

3 HE Furnace Gas

4 Low-Flow Showerhead Either

5 Ceiling/Attic Insulation Both

6 HE Air-Source Heat Pump Electric

7 ENERGY STAR Refrigerator Electric

8 HE DWH Storage Either

9 Furnace Motor ECM Electric

10 HE Clothes Washer Both

Commercial Sector Measures Fuel Type

1 Lighting– Linear Fluorescents Electric

2 HE Furnace Gas

3 HE DWH Storage Either

4 Lighting Controls Electric

5 HE Motor Electric

6 HE Air-Source Heat Pump Electric

7 Roof Insulation Both

8 HE Clothes Washer Both

9 Air Compressor Electric

10 Refrigeration ECM Electric

Measure Review

Our detailed review of the 20 measures across various TRMs resulted in three general themes: consistency in 

methods and assumptions, transparency and clarity, and accuracy. We compared the methods for savings 

calculations and the results for each of the 20 measures across five (at most) TRMs. We also included in our review 

an indication if data around measure costs or secondary benefits were provided.

The results of our review indicated a wide variation in methodologies for estimating savings among the 17 TRMs. 

There were exceptions where TRMs were relatively consistent in their algorithms and input assumptions, such as for 

high-efficiency (HE) motors and commercial air compressors. For the most part, however, there was little 

consistency in which variables and other inputs were used. Some TRMs, for example, included waste heat factors, 

in-service rates, and partial load factors, while other TRMs did not. Given that we weighted our selected measures 

based on the diversity of savings, this is perhaps not wholly surprising. However, the wide range of methodologies is 

still notable. 

Methodological approaches also varied among TRMs. For example, the roof insulation measure in some TRMs 

based savings on building energy simulations, while others based savings on engineering algorithms. Even TRMs 

with similar algorithm methodologies used different input assumptions, such as variable names (e.g., Btu/hr vs. 

tons) and baseline assumptions (e.g., federal code vs. current practice). The magnitude of these impacts can vary

based on these assumptions; for example, the variable names or units should not change the fundamental 

approach, while the baseline assumptions can significantly impact the savings estimates. These inconsistencies have 

implications for creating standardized approaches in a national database. 
June 2011 www.seeaction.energy.gov 13

The transparency and level of detail regarding methods and assumptions ranged from being well-sourced to being 

minimal. None of the TRMs referenced all their sources completely. In several cases, it was not clear what the input 

assumptions were based on, especially with fully deemed values. In other cases, input assumptions were not clearly 

defined, making it difficult to determine the appropriateness of different values used in the algorithm. TRMs with an 

adequate level of transparency provided multiple references and outlined the intermediate steps showing how the 

calculations were made and the sources for the data. 

Our review indicated that algorithms used in savings calculations were generally correct and based on accepted 

methodologies. However, our review also showed considerable errors, ranging from obvious typographical errors, 

to missing default values for certain assumptions, to minor calculation errors in a few cases. In several cases, data 

tables were mislabeled and footnotes and references were incorrectly numbered. In several cases, variable names 

were missing entirely. 

Tables 6 and 7 summarize our findings and recommendations for each of the 20 measures we reviewed as part of 

the measure comparison. We provided the range in gross energy savings where possible, and the key factors 

contributing to the spread. For those measures that can provide gas and/or electric savings, the savings spread is 

provided in the most common unit across the TRMs reviewed. For example, if three of the five TRMs represent 

savings for high-efficiency water heaters in kWh, and the remaining two provide savings in Btus or therms, the 

spread from only those three TRMs is provided. Details are provided in Appendices A and B for the residential and 

commercial measures, respectively. 
June 2011 www.seeaction.energy.gov 14

Table 6. Residential Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

CFL 27 – 49 kWh Findings: Lighting is a well-studied measure subject to many adjustment factors, depending on installation location (general house vs. 

living room), application (interior vs. exterior), HVAC system, and delivery mechanism (retail, direct install, socket count, etc.). Storage 

and removal factors vary and must be obtained through regional studies. Additional variation arises from hours-of-use assumptions. 

Recommendation: Residential lighting on a single fixture level is a relatively straightforward measure that can be included in a 

national DS&A database with a supporting wattage table. However, inclusion of the aforementioned adjustment factors will be 

challenging. 

Refrigerator 

Early 

Replacement

724 – 1,728 kWh 

(gross)

Findings: This measure relies on deemed savings obtained from studies. The average existing refrigerator energy consumption can 

also be determined by accessing the appliance recycling contractor database. Variations in the calculation of deemed values include 

accounting for degradation over time, lab-to-field true-up factors, replacement rate, usage hours, and energy usage of replacement 

appliance. NTG factors were included in a few of the sources we reviewed.

Recommendation: Refrigerator recycling/retirement can be adapted into a national DS&A database relatively easily. Measure savings 

should be based on evaluation studies that account for the energy consumption of the existing residential refrigerator stock. However, 

it may be challenging to come to an agreement about which study to base this information on and what factors to include. 

Furnace 5.54 – 12.7 MMBtu Findings: Most savings are calculated based on the hours-of-use; however, variations include the annual heating load by vintage and 

floor area. These inputs vary regionally and even within the same state. Another approach is based on building simulations. Energy 

savings units were most commonly in MMBtu; however, one TRM assumed the use of an ECM furnace fan, and yielding electricity 

savings.

Recommendation: As the algorithm is generally quite uniform, it can be adapted into a national DS&A database. As a weathersensitive measure, the usage would vary by location and application, making the development of reference tables potentially 

challenging. 

Showerhead 130 – 583 kWh Findings: Electric and gas savings follow similar algorithms. At the most general level, the prevailing savings algorithm breaks down 

into two parts: the water savings and savings from the energy required to heat that amount of water. Annual water savings are

calculated in two primary ways: the first is based on the number of showers per day, the second approach is on based on the number 

of people per household and the average number of showers per household. The energy that would have been required to heat the 

amount of water saved is the most straightforward approach, and is based on the volumetric heat capacity of water and the 

temperature delta. 

Recommendation: Showerheads could be included in a standardized national DS&A database. The measure is ideally accompanied 

by regionally tabulated values for the water temperature delta and usage. The efficiency of the water heater (by fuel type) should be 

well defined in the TRM, either on a national or regional basis. 

Roof Insulation Not comparable Findings: Energy savings are calculated on a square-foot basis, either through use of simulation software or algorithms. The unit 

savings are then scaled up through the effective area of increased insulation for each project. We found multiple algorithms that varied 

by HVAC system type and fuel type. The baseline and efficient conditions vary based on R-values.

Recommendation: This measure would be challenging to standardize, and savings are best determined through building simulation 

modeling for regional building stocks.
June 2011 www.seeaction.energy.gov 15

Table 6. Residential Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

Air-Source 

Heat Pump

164 – 951 kWh Findings: The sources we reviewed used both calculations and energy modeling to determine savings. Baseline conditions are often 

based on energy code requirements, with the efficient condition defined by the nameplate efficiency exceeding the baseline. Variations 

arise due to the efficiency increment between baseline and efficient condition, as well as by hours-of-use assumptions. 

Recommendation: Heat pumps are an important measure in many utility programs and, given the relatively low variation in calculation 

approaches and the well-defined baseline and efficient conditions, could be included in a national DS&A database. The challenge will 

be to develop weather-sensitive hours-of-use by climate zone, building type, and other influential characteristics.

Storage Water 

Heater

99 – 347 kWh Findings: Energy savings are generally calculated using a variation of the prevailing algorithm, and using the volume of hot water 

consumed (gallons per day), the temperature differential of the water in vs. out of the water heater, and the equipment efficiency. The 

only assumptions for this measure that may be regional are the water main temperature, which varies by city, and the ambient room 

temperature. The baseline and efficient energy factor (EF) values depend on fuel type. 

Recommendation: Considering the minimal weather sensitivity of water heaters (except for the potential HVAC interaction and 

variations in water main temperatures) and the algorithm’s ease of use, efficient water heaters are a good candidate for inclusion in a 

national DS&A database. Separate algorithms are recommended for gas and electric water heaters. National or regional average water 

usage will need to be tabulated. 

ECM for 

Furnace Fan

419 – 733 kWh Findings: Energy savings are deemed on a per-furnace basis, with values originating from various technical studies. Some sources 

use a heating degree day adjustment factor, while others use regional variations in AFUE or hours-of-use based on studies. 

Recommendation: Adapting the residential ECM furnace fan measure into a national DS&A database could be intensive, as it would 

be with other weather-sensitive measures. The difficulty in adapting reported savings for use in a national DS&A database would be 

accommodating differences in building configuration and weather impacts. Some of the variation in this measure can be 

accommodated by load hour or degree day adjustments for different regions. Savings should be broken out by state or climate zone, 

by fuel type, and by furnaces with and without a central air conditioner.

ENERGY 

STAR 

Refrigerator

45 – 106 kWh Findings: Savings are deemed, per refrigerator, based on appliance characteristics (e.g., ice through door, freezer configuration, 

freezer and refrigerator volume, efficiency level such as CEE vs. ENERGY STAR). Two primary approaches were used to determine 

savings: (1) define maximum consumption limits for baseline and efficient appliances; and (2) use average region-specific appliance 

data. Some sources include an HVAC adjustment factor to account for refrigerators in a conditioned space interacting with the HVAC 

system. 

Recommendation: This measure is ideal for inclusion in a national DS&A database as a set of deemed savings by appliance 

configuration and efficiency level.
June 2011 www.seeaction.energy.gov 16

Table 6. Residential Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

Clothes 

Washer

127 – 258 kWh Findings: Deemed savings are tabulated by domestic water heating (DWH) and dryer fuel, and efficiency level (CEE vs. ENERGY 

STAR). The clothes washer is not a standalone measure. Savings from DWH and from the dryer (due to less moisture in the clothes) 

are usually implicit. One of the sources calculates weighted savings values over the entire fuel mix.

Recommendation: Residential clothes washers are ideal for inclusion in a national DS&A database as a set of deemed savings by 

DWH and dryer fuel type, as well as by efficiency level. This will provide deemed savings based on easily observable attributes. 

Clothes washers in multifamily settings may require separate calculations to account for in-unit washers and washers in a laundry

center (common area). Savings for all fuel types should be claimed for each unit in a national DS&A database.

Table 7. Commercial Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

Indoor Lighting 34 – 80* kWh Findings: The savings calculation requires the user to supply actual values or use the tabulated wattages, hours-of-use, HVAC 

interaction factor, and quantity. Many TRMs contained reference wattage tables; however, these tables were inconsistent. Variations in 

variable definitions and other input values contribute to the spread in the savings value.

Recommendation: Lighting is currently a significant component of utility programs nationwide and, on a single fixture replacement 

level, is a good measure for the national DS&A database. Developing a universal/standard wattage table will be challenging. 

Furnace 1.1 – 30 MMBtu Findings: Most of the calculations rely on the heating full load hours; however, another approach uses heating degree days. Another 

variation is the assumption of whether an ECM furnace fan motor is or is not installed. The greatest contribution to the measure 

savings spread appears to be from the variation in usage. Most baseline efficiencies are based on the federal standard of 0.78 AFUE. 

Recommendation: Although weather sensitive, this measure is recommended for inclusion in a national DS&A database because of 

the relatively small variation in the calculation methodologies observed and the small number of inputs. The biggest challenge will be in 

developing full load hours based on location and application (building types) across the nation. 

Storage Water 

Heater

1.5 – 7.6 MMBtu Findings: As with many other measures, the savings calculation approach multiplies a percent of savings (based on equipment 

efficiency) by the normal consumption. Differing approaches are used to determine the annual hot water consumption; additional 

differences in the calculation depend on tank size and fuel types. 

Recommendation: Accommodating this measure in a national DS&A database would be best achieved by breaking out the savings 

calculation by fuel type and tank size. Since some units are rated by their energy factor and others by thermal efficiency, different 

algorithms will be needed to appropriately determine savings. The annual hot water consumption should be provided based on 

important characteristics such as facility type, size, occupancy, etc.

Lighting Controls 180 – 535 kWh Findings: Savings are calculated on a per controlled load basis, similar to the indoor lighting measure. The algorithm contains an 
June 2011 www.seeaction.energy.gov 17

Table 7. Commercial Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

energy savings factor that is similar to the percent savings and is based on a certain type of control (occupancy sensor, time clock, 

etc). Variations in the algorithm include use of the in-service rate adjustment factor and the HVAC interaction factor. 

Recommendation: This measure is ideal for inclusion in a national DS&A database because all methodologies follow mathematically 

similar formulas. Reference tables are needed to cover a range of control types and referenced operating hours.

Motor 

(1-200 HP)

387 – 662 kWh Findings: Per motor savings are calculated using baseline and efficient conditions, as defined by federal standards (based on motor 

type and HP), with the main variability attributed to hours-of-use and the rated load factor.

Recommendation: Motors are ideal for inclusion in a national DS&A database because there is little variability in the algorithm used to 

calculate energy savings. A standardized algorithm should also come with reference tables of baseline efficiencies, operating hours,

and rated load factors, as well as instructions on how to pick an appropriate input value. Developing a universal/standard operating 

hours table may be challenging, and this challenge is the main cause for the range of savings in the TRMs reviewed.
June 2011 www.seeaction.energy.gov 18

Table 8. Commercial Measure Comparison Findings and Recommendations Summary

Measure

Gross Annual 

Energy Savings 

Spread

Main Findings and Recommendations

Air-Source Heat 

Pump

109 – 3,031 kWh Findings: The TRMs we reviewed used both calculations and energy modeling to determine savings. The range in savings is due to 

variations in hours-of-use assumptions, climate zone, vintage, and/or building type. 

Recommendation: Heat pumps are an important measure in many utility programs and, given the relatively low variation in 

calculation approaches, could be included in a national DS&A database. The challenge will be to develop weather-sensitive hours-ofuse by climate zone, building type, and other influential characteristics. 

Roof Insulation Not comparable Findings: Savings are commonly developed from outputs of building simulations and are tabulated by a host of characteristics. Some 

TRMs used an engineering calculation. Measure baselines and efficiency requirements vary regionally.

Recommendation: This measure would be difficult to standardize and is not recommended for initial inclusion in a national DS&A 

database, which would require capturing local building practices, building stocks, code requirements, and other influences.

Clothes Washer 224 – 921 kWh Findings: Savings from a clothes washer also include domestic water heating (DWH) and dryer savings (due to less moisture in the 

clothes). Energy savings per load are tabulated based on the water heater and dryer fuel(s) (e.g., gas, electric, line dry). The annual 

clothes washer usage is multiplied by the savings per load to calculate the annual savings. Variation in the number of loads per year 

(from 271 to 2,190) is the strongest contributor to the spread in energy savings.

Recommendation: Commercial clothes washers are a good candidate for the national DS&A database since many of the calculation 

approaches within the TRMs are similar. This measure should be broken out by the water heater and dryer fuel type(s). Default values 

for different use scenarios would take additional resources to develop. 

Air Compressor 11 – 23 kWh Findings: Energy savings are calculated per compressor on a retrofit basis. The baseline condition is typically a modulating air 

compressor with blow down, and the efficient condition varied by equipment control type. Most commonly, the control type is a 

variable speed drive. All algorithms depend on an energy savings factor that is provided in the supporting measure documentation. 

Variability arises primarily from these tabulated energy saving factors.

Recommendation: The small variation in air compressor energy savings methodologies makes it an ideal measure for inclusion in a 

national DS&A database. To properly support this measure, a table of operating hours will be needed, possibly broken down by usage 

scenario (e.g., 3-shift).

ECM for 

Refrigeration

264 – 2,033 kWh Findings: Savings approaches vary from using deemed energy savings values to employing calculations requiring multiple inputs. 

Multiple calculations are provided based on application (e.g., freezer vs. cooler, walk-in vs. other systems). Most sources account for 

the refrigeration system efficiency, while one provides deemed energy savings per rated equipment watt, which is then scaled by the 

system wattage. 

Recommendation: This measure can be included in a national DS&A database, but will first need to be broken out by application, 

motor size, and equipment configuration (ECM or ECM with controller; and PSC or shaded-pole). 

* Original value was 239 kWh for a three-lamp fixture; we divided by three to calculate a one-lamp fixture equivalent.
Month Year www.seeaction.energy.gov 19

Findings

This section presents the results of our scoping study findings for the EM&V documents and National (or regional) 

Deemed Savings and Algorithm database in terms of: the needs they might address, their desirable features and 

functionality, their feasibility in light of issues affecting their development and maintenance, and costs. 

EM&V Documents Database

Needs

Experts in the industry and those newer to the energy efficiency field expressed an overwhelming need for a 

national database of EM&V resources. Nearly every interviewee agreed that a national database of EM&V reports, 

plans, market studies, and other relevant documents would be a very useful resource. One respondent commented 

that a centralized resource providing such documents is needed, but cautioned that a database might not be the

best format. 

Interviewees cited several benefits of having a national EM&V documents database, particularly savings in the time 

and effort required to locate useful materials. Although many such EM&V reports and studies are publicly available, 

searching and compiling relevant materials would be time-consuming, as they are currently spread across several 

disparate sources. Being able to access EM&V documents from across the country in a single database could greatly 

reduce the time required to locate relevant sources, while increasing the availability of quality information and 

most importantly case studies/examples—which are perhaps the most common request of those new to the EM&V 

field. This is important for those new to energy efficiency—seven of the 19 industry professionals we interviewed 

indicated that a national database would help nascent states begin program planning and evaluation.

Interviewees also indicated that a national EM&V documents database could fill the need for greater transparency 

and consistency than is currently possible from the existing resources. They also stated that a national EM&V 

documents database could greatly facilitate comparisons across the country, moving the evaluation community 

towards a more formal data sharing process.

Feasibility

Although respondents reported that a national database of EM&V documents has benefits, they raised several 

issues that would have to be resolved before a valuable resource, providing benefits beyond those currently 

available, could be developed. These concerns relate to the feasibility of the database itself and included 

development of the database and maintenance infrastructure, procedures for cataloguing documents, and 

development and maintenance costs.

Interviewees expressed concerns about the complexities of database development and maintenance, especially 

with respect to the tasks of gathering and properly cataloguing existing EM&V reports and resources. As noted, the 

EM&V plans, reports, and market studies that are available online are spread across multiple websites, and many 

more documents are not available online. Respondents noted that the initial push to gather these documents would 

be a massive undertaking, but they reported that overall it would be worth the time and effort. 

Ensuring consistent categorization and tagging of compiled documents was another concern related to the 

feasibility of such a database. Several interviewees indicated that an individual with a library sciences background 

would be required for development and ongoing maintenance. In addition, depending on the final product, it might 

be necessary to develop data formatting and reporting requirements to maintain consistency, especially if the 

database permitted comparisons across the country.
June 2011 www.seeaction.energy.gov 20

The ability to acquire EM&V reports and plans, market studies, and other relevant documents presented another 

feasibility issue, applying to initial development stages and to continued database upkeep. After the initial task of 

compiling existing resources, stakeholders would have to decide whether to mandate document submittal or 

operate on a voluntary basis. Industry professionals noted that while persuading people or organizations to 

voluntarily submit documents could be a challenge, mandatory submission could prove burdensome and difficult to 

implement, especially if individuals and organizations were required to submit documents to more than one group.

The costs to create and maintain a database on this scale could be large, as several interviewees noted. However, 

no one expressed a concern that costs alone should prevent the project from moving forward. Although these

feasibility issues should be considered before moving forward, they did not appear to render a national database of 

EM&V resources unachievable. 

Content and Functionality

Interviewees indicated a wide range of document types that should be considered for inclusion in the database, 

including process and impact evaluation plans and reports, market studies, and other relevant studies. A diverse set 

of functionality options was also identified by the respondents. To create this type of database, up-front decisions 

would need to be made about the types of documents to include, and an appropriate database structure would 

need to be determined to present documents in a readily accessible manner. 

The original scope of this study envisioned a database comprised of EM&V plans, EM&V reports, and market 

characterization studies. EM&V plans can include several different types of documents; a recent study suggests four 

types: EM&V frameworks, portfolio plans, detailed research plans, and site-specific evaluation plans.4Including all 

four of these types of documents would be necessary to provide a comprehensive resource. However, as noted 

earlier, evaluation plans are not readily available publicly. Three of the five interviewees that provided input on this 

subject expressed concerns about proprietary information barriers, which would make publishing difficult. Although 

many respondents said that including EM&V plans would be useful, their reservations about the availability of such 

documents suggested that it might not be feasible to include them. Often, EM&V reports include a detailed 

description of the evaluation research plan; therefore, including the original plans may not be critically important. 

After interviewing industry experts and representatives from states with newly developing energy efficiency 

initiatives, it became clear that expanding the contents to include the following additional types of documents and 

supporting data would be useful:

 Common evaluation methodologies (e.g., billing analysis, metering, modeling)

 Evaluation framework documents and protocols

 Technical/white papers

 Documents relating to market transformation or other resource acquisition strategies

 Examples of requests for proposals (RFPs)

 Links to regulatory filings and orders on energy-efficiency matters

 Dictionary/glossary of industry terms 

 Raw data used in evaluations.

 

4

Schiller, Goldman, and Galawish. National Energy Efficiency Evaluation, Measurement and Verification (EM&V) Standard: 

Scoping Study of Issues and Implementation Requirements, LBNL-4265E. April 2011.
June 2011 www.seeaction.energy.gov 21

Several interviewees noted that many of the listed documents would need to be accompanied by sufficient 

explanation and context, including program and regulatory background information. Interviewees also recognized 

that raw data, though highly desirable, would be difficult to obtain due to confidentiality concerns. However, these

types of documents currently have limited public availability; so an active search and selection process would need 

to happen before including them in a national database.

The respondents generally reported that the needs of professionals new to the industry should be given priority in 

determining the scope and coverage of the EM&V documents database. Development of a resource that provides 

value to more experienced industry professionals poses additional challenges. Respondents were evenly split (six 

and six) about whether the database should be a comprehensive repository of all known EM&V reports and market 

studies, or be a representative sample based on particular screening criteria. The remaining interviewees were 

undecided or did not provide input on this topic.

Those respondents leaning toward a comprehensive database suggested that instead of limiting the number of 

reports, the database should include filters, descriptors, or flags to guide users new to energy efficiency. They said 

this would make the database less overwhelming, while providing value for more experienced users. 

Interviewees that indicated interest in having a selective database made several suggestions about how rigorous the 

screening process should be. These ranged from a simple check to determine whether the document had been 

accepted or formally approved by regulators, to a requirement that they be subjected to stringent review and 

ranking processes by experts, based on predetermined criteria or their expert judgment.

Interviewees also noted that, while the concept of a database designed to inform users of best practices was 

attractive, it presents several additional feasibility issues. As several interviewees noted, screening reports and 

market studies based on best practices criteria could be problematic, given that best practices are a relative concept 

that depend on the goals and budget of a given program and/or evaluation. The experience or budget of nascent 

states may not support employing those best practices that would be defined by more experienced states, such as 

California. If the database were limited to best practices most effective in experienced states, nascent states might 

not be able to identify studies or methods most suitable for their specific situations. As a possible resolution of 

these potentially conflicting needs, interviewees suggested incorporating a tiered approach for identifying best 

practices in the database to achieve a better balance of rigor and cost, allowing entities to choose approaches 

fitting their specific situations and regulatory environments from a larger pool of approaches. 

A frequent suggestion from interviewees was to create a database that serves as a “starter kit” for states new to 

energy efficiency, guiding them toward developing energy efficiency initiatives or complying with energy efficiency 

standards or requirements. This starter kit would provide sample documents and guide the user through their 

applicability and relevance.

Regardless of whether the database is comprehensive or limited to a screened sample of documents, the databases’ 

search capabilities will likely be one of its most important functions. Interviewees made a number of suggestions for 

search or filtering capabilities, including the following:

 Key word(s)

 Program implementation type and goal (e.g., direct install, point of sale rebates, financing programs, 

market transformation, emerging technology)

 Programs market (e.g., commercial, residential, low income)

 Technologies and measures (e.g., CFLs, LEDs, HVAC)

 Sample size

 Evaluation history (whether similar studies have been completed)
June 2011 www.seeaction.energy.gov 22

 Date (years) of EM&V activity

 Methodological approach (billing analysis, metering, etc.)

 Region 

 Fuel type

 Complexity of study.

Finally, interviewees mentioned that a help desk feature would be an extremely valuable function of the database, 

especially for those new to energy efficiency. As one respondent noted, much quality control and decision making 

occurs behind the scenes, and having an option to talk to individuals making these decisions (or someone 

knowledgeable about the decisions) would be very useful. Having a common answer source for questions arising as 

people utilize information in the database would be beneficial for improving consistency and decreasing the 

chances of data being applied incorrectly or out-of-context. 

National (or Regional) Deemed Savings and Algorithm Database

Needs

Respondents by and large agreed that a national DS&A database (or at least multiple, regional databases) would be 

useful, but their opinions differed widely on the optimal structure, content, and functionalities of such a resource. 

For example, a measures database was less of a priority for respondents in regions such as the Northwest, where 

such a database already exists. 

The majority of respondents reported that a national DS&A database would help states and utilities new to the 

industry with planning, implementing, and evaluating energy efficiency initiatives. Four interviewees stated the 

resource would facilitate comparison and lead to consistency across the country, and three respondents suggested 

it would provide more transparency in numbers and algorithms used. Most interviewees said the resource would 

have to account for regional differences to truly address user requirements. The interviews, however, were 

designed to address the need and feasibility of a database of savings estimates. Metrics around non-energy 

benefits, such as job creation, or market transformation were not discussed, as that was beyond the focus of this 

study. 

Two interviewees raised a potential concern with having a national DS&A database: the market’s possible transition 

toward more custom or whole-building savings approaches. One interviewee, a utility representative, mentioned 

that the market has moved more toward custom measures, and cautioned against creating a costly prescriptive 

measures database due to declining shares of prescriptive measures in total portfolio savings. Reportedly, the 

majority of savings in this utility’s service territory have derived from whole-facility retrofits, design improvements, 

and custom measures, making prescriptive savings estimates less useful to this particular utility. No representatives 

from the other seven utilities we spoke with mentioned this market transition, making it difficult to determine 

whether the trend is widespread or varies by utility area, state, or region. However, a non-utility respondent voiced 

a similar concern: that the focus moving forward in the energy efficiency industry should be on whole-building 

savings and not measure-specific values. This respondent cautioned that creating a national measures database 

would diminish the level of rigor in estimating energy savings.

Overall, most respondents reported that there is a greater need for a database of algorithms than for a database 

providing only specified ex ante savings values. However, all respondents new to the field reported struggling in the 

planning stages to determine the best algorithm to use for each measure, given the variety of existing TRMs. 

Respondents from two nascent utilities stated that algorithms were helpful as guidelines, but that for savings 

estimates to be useful, it is critical for them to understand the inputs and assumptions resulting from determining 

that savings estimate. The third nascent utility, on the other hand, would prefer to be provided with a deemed 
June 2011 www.seeaction.energy.gov 23

value that would remain the same from the planning stage through evaluation. This was based on their experience 

collecting the data needed for complicated savings algorithms—much of it customer data that are difficult to 

obtain. 

The idea of national, standardized measure savings algorithms or values was generally not considered feasible at 

this time (or in the near future) by interviewees, largely due to the widely varying regulatory requirements, 

baseline(s), program structures, and evaluation needs. Almost all interviewees saw the value in having national 

savings estimates that are region or climate-zone specific; in other words, a collection of regional DS&A databases 

would compose the national database. However, most stated that a national database would not be used 

consistently unless mandated by regulators, and most opposed a mandate to use the values or algorithms in the 

database. Reportedly, requiring national savings values would cause a large amount of pushback from states and 

entities with established protocols and TRMs. One respondent said that national standards are needed, but 

acknowledged such standardization would be difficult to achieve and that the market is not ready for such a change. 

Three respondents noted that in order to provide the transparency needed for savings estimates, consistent 

measure data collection requirements would be needed, and none are currently in place. 

The general consensus was a desire for a national DS&A database to be available for use at will, but without 

mandated nationwide standardization on savings values. One utility representative noted he would oppose creation 

of the database if its use would be required, given that his state already has a TRM and it would not be in his utility’s

best interest to have a national database. Several other utilities expressed anxiety regarding national 

standardization, primarily if it involved replacing the current deemed savings estimates they had worked hard to 

create. Other respondents, however, noted that a nationally required measures database would add consistency 

and make the data more accessible.

Despite the lack of support for national standardized savings values or algorithms, interviewees thought the DS&A 

database would be useful for states and utilities with little energy efficiency program experience, as long as the 

feasibility issues could be overcome. They said a database compiling the different algorithms and stipulated savings 

estimates would be helpful, as long as individual entities and regulatory divisions remained entitled to use their own 

methods. They stated that such a resource would allow for a comparison of methods used nationwide, which two 

interviewees said might naturally lead to more consistent practices over time. If such a resource were accepted by 

regulators and industry actors, there would need to be opportunities for stakeholder inputs and transparency into 

what the numbers represent. This would include clearly explaining inputs and assumptions, either in arriving at the 

savings estimates or as imbedded information in the algorithms supplied. 

Feasibility 

Several types of concerns were raised about the feasibility of creating and maintaining a national DS&A database. 

Although most interviewees thought a measure database resource would be useful, they also acknowledged it 

would take a significant amount of effort for the database to be sufficiently comprehensive enough to be widely 

useful. Respondents also thought it would be difficult to keep the resource up-to-date. Five interviewees thought it 

was simply not feasible. Others stated that creating the national DS&A database is potentially feasible, depending 

on multiple contingencies. One interviewee warned that the scope of this project could easily become 

unmanageable due to extreme differences in measure savings estimates. 

The interviewees identified multiple concerns that would need to be addressed for a national DS&A database to be 

widely useful. One of the main concerns is the vast regional variation in factors such as baselines (codes, standards, 

standard practice, and when different baselines are applied), weather, economics, demographics, vintage, and 

measure costs that need to be reflected. This may, in part, be alleviated through multiple regional databases. 

However, even then, there may still be wide variation within a designated region. Only one interviewee thought it 

was a good idea to replace regional and state-mandated TRMs with national standardized deemed values, but this 

interviewee also recognized these changes would have to be region-specific. 
June 2011 www.seeaction.energy.gov 24

Another substantial challenge facing development of the national DS&A database is the constant updates required 

to adjust measure savings values based on ongoing research and studies. Six interviewees mentioned that the 

upkeep needed to maintain the usefulness of a measures database would be extremely costly and time intensive. A 

concern voiced by two respondents was that if savings estimates were changed in the middle of a program cycle, 

the values used in planning the programs might not be the same as those used for their evaluation, making it 

difficult to meet savings goals. This could be addressed at the state level by coordinating updates and program 

cycles, but would be difficult to address at the national level because utility program cycles are not on a consistent 

schedule nationwide. 

In addition, two respondents were concerned about the large variance in ex ante savings numbers across TRMs and 

regions, and said that could create skepticism about energy savings values. Both respondents stated it would be 

difficult to develop a database that is sufficiently accurate and reliable enough to ensure the industry’s confidence.

If a national database produces different savings estimates in different locations without explanation, those 

differences would be highlighted and could raise doubts about their accuracy. If differences are evident but the 

reasons are explained, it is possible that the credibility of the database would be enhanced and users would better 

understand what affects energy savings. 

Content and Functionality 

Respondents offered many unique, often conflicting ideas regarding the desirable content and functionality of the 

DS&A database. All respondents addressing this topic stated that a high transparency level for the savings 

calculations would be necessary. 

Respondents commented that deemed savings values would be more suitable for use by regions where evaluation 

data are available to support the deemed values. However, it would be difficult to find sufficient data in many 

locations, and it would be labor intensive to develop the data needed to support a national DS&A database for 

many measures reviewed. Deemed savings are most suitable for complex measures (with multiple interactive 

effects) that are typically analyzed using building simulation software. Such measures would include building 

envelope measures and large HVAC systems. In addition, a naming protocol would need to be developed to 

communicate the scope and applicability of each measure. The disadvantage of this approach is the large number of 

permutations (measures).5

However, most respondents favor a resource that includes algorithms over simply reporting savings values. 

Algorithms would need to be clearly presented, along with applicability conditions and documentation of underlying 

assumptions used to derive savings values, including:

 Measure definition, program delivery mechanism, customer characteristics and use of the measure 

and/or the process/equipment with which it is associated 

 Regional characteristics

 Assumptions made

 Inputs used

 Program specifications

 Regulatory requirements

 Whether values are gross or net.

 

5

DEER is a good example of this: the 2005 version has 361 unique measures and 129,165 permutations.
June 2011 www.seeaction.energy.gov 25

A dynamic database, with embedded algorithms based on engineering calculations, may offer flexibility in 

aggregation levels and appeal to a wider range of users; however, the support level required may be equally, if not 

more, demanding than the stipulated savings approach. The database methodology would need to account for 

many of the same considerations applicable to a stipulated savings approach. Based on the interviews and findings 

from our review of existing TRMs, we suggest the DS&A database include:

 A preferred algorithm, accommodating a range of scenarios while remaining user friendly (inputs 

should be based on easily observable and quantifiable measure characteristics, such as the tank size 

for a water heater).

o The algorithm should include a description detailing aggregation levels. For example, a specific 

motor retrofit (one motor type, RPM, and horsepower [HP] combination) or the averaged motor 

retrofit (savings vary by HP only, algorithm already weighted by type and RPM). 

o The baseline efficiency and efficient conditions must be clearly described.

 Guidance on use of actual vs. default input values (e.g., actual hours-of-use or values provided in the 

database). The actual policy that might be applied in a specific jurisdiction for using a mixture of 

actual and default values would likely be established by the authority responsible for that jurisdiction. 

As observed in many measure comparisons, different TRMs, approved by different regulators, have 

differing approaches. Some example approaches include:

o Supporting reference tables and default values should be provided for ease of 

use. (Default values must be regularly adjusted to accommodate codes and standards updates.)

o For users from regions where data are not readily available, it may be preferable to use actual 

input values (and to collect these data during program operations). 

o Some adjustment factors vary greatly from region to region, such as the in-service rate, end-use 

interaction, and coincidence factor. A possible way to handle these variations is to include a 

placeholder “master” adjustment factor in the algorithm, and set the default value to ‘1’. Regions 

where multiple adjustment factors are required can determine their own effective adjustment 

value for use in the calculation. 

 Definitions of input variables to ensure transparency and reduction in human errors.

o Input units will always need to be defined (e.g., HVAC capacity in tons vs. Btu/hr), along with the 

appropriate number of significant figures for the chosen algorithm (e.g., 3 tons vs. 2.75 tons). 

o Including conversion factors, or unit-flexible dynamic algorithms, would add a desirable database 

feature, though be more labor intensive. This would require the database or program perform 

dimensional analysis to ensure a valid result. 

o Variables, even those with the same symbol and definition, were used inconsistently in 

mathematically equivalent algorithms. A good example of this is the waste heat factor for 

lighting-HVAC interactions. Many TRMs referred to this interaction as ‘WHFe’; however, usage 

diverges when this interaction is based on a negative number, such as in equation A, or when the 

calculation yields a number greater than one, such as in equation B. 

 Equation A:

 

 

 

 
June 2011 www.seeaction.energy.gov 26

 Equation B:

 

 

 

 

 Adequately detailed information for each algorithm and user-adjustable input, ideally with an 

example calculation. For instance, in the case where a 60 watt (W) incandescent lamp would be 

replaced by a 15 W CFL:

 

 

 

 

 

 

 

 

 

 

 

 

 

This level of detail would be necessary to allow other entities to compare the database values to their own 

programs and account for differences. The need to address regional differences was cited as a critical functional 

requirement by a majority of interviewees. Without clear information on how to account for regional differences, 

data and calculations may be used out-of-context, fostering distrust in the accuracy of the resource. 

One respondent noted, however, that states new to energy efficiency programs may be overwhelmed by algorithms 

in TRMs, and suggested a calculator to reach this target audience. Although this option better addresses the needs 

of those new to the field, professionals interested in a more robust tool may struggle with a calculator’s inherent 

limitations. Whatever is decided, the tool would need to clearly define all algorithms used in the calculator 

functions and provide flexibility in inputs to be widely useful for both expert and non-expert user groups. 

Another respondent offered a similar idea, suggesting the DS&A database could have a methodology filter that 

would allow the user to select defaults or to input data for their own market. They also mentioned that data 

sources need to be clearly cited and identified. As we found during our TRM research, circular references from 

these sources were not uncommon. 

Several additional common themes on content emerged from the interviews. Most respondents agreed the 

database would need ongoing improvement and upkeep to remain current. Two respondents suggested including a 

date stamp on values, and one suggested “sunset” provisions, such as those used by the RTF (where values would 

become obsolete after a given amount of time). Generally, respondents stated that tracking the age of values would 

facilitate their staying current with recent studies. Another suggestion, noted several times, is the need to have raw 

data included in the database for greater transparency.

Opinions varied regarding the content necessary for the DS&A database to be useful. While some reported the 

database would need to cover a vast assortment of measures, an almost equal number said the opposite, preferring 

a database more narrowly focused on certain common measures with significant potential. Those favoring a 

narrower focus thought common measures would serve as a good starting point, and the database could be 

expanded as needed. Almost all interviewees preferring the more comprehensive version noted there would be 

tremendous feasibility issues in actually proceeding with this option.

Although the interviewees generally agreed that most challenges facing the creation of a national DS&A database 

could probably be overcome, they reported one aspect that would be nearly impossible to develop: estimating peak 

load impacts. One reason is that larger states with multiple climate zones and high load diversity could have utilities 

with significantly different peak demand patterns. In addition, peak periods tend to be unique to particular utilities. 

Thus, creating a standard set of coincidence factors would have to be based on gross approximations, and the 

resulting values would have limited use.
June 2011 www.seeaction.energy.gov 27

Given the more urgent needs of those new to energy efficiency programs and the likely difficulties inherent in 

developing a national DS&A database, interviewees suggested alternative, intermediate resources to meet these 

needs. One suggested a series of white papers on different approaches and algorithms used for some common 

measures. This could easily be expanded as more white papers are developed, and would be useful to those new to 

energy efficiency programs because greater detail would be provided in a format easier to read than a resource that 

simply provides savings values and algorithms without more complete background information. These white papers 

could later be integrated with a measures database and serve as background material. Other suggestions included 

incorporating the results of potential studies and baseline estimates in the database. 

Quality Control 

Several respondents mentioned quality control as being critical to the development of a national DS&A database. 

One interviewee suggested including values accepted by a regulatory body in the database. Another respondent 

suggested having stakeholders agree on methods and criteria used to screen information and data entered into the 

database. Quality assurance for data in the DS&A database would be especially important to users new to energy 

efficiency programs who want to use this resource as a guide, since they might not have the knowledge to recognize 

errors.

Comparison of National DS&A Database to Existing TRM Resources

Most respondents were familiar with DEER, and some had experience with other states’ TRM resources and the 

Northwest RTF. Knowledge of these databases allowed respondents to provide insights into the shortcomings of 

current measure savings sources. 

DEER

Although California-specific, DEER has been a useful resource for users in many different locations and with a wide 

range of backgrounds. All interviewees new to the industry had used DEER at some point for planning and 

developing programs. However, one respondent indicated DEER was “overwhelming to a newcomer.” Another 

noted, as it is California-specific, that much of its information was not relevant to their area. In general, more 

experienced DEER users valued its comprehensiveness, with both savings and cost data for a wide array of 

measures. One respondent also indicated the portability of the database was very convenient. However, those 

interviewed felt the data sources need to be more transparent and include date stamping, and the database could 

be difficult to navigate due to its volume and spreadsheet format.

In general, TRMs are customized to particular locations. However, they are often used across geographic areas by 

simply altering assumption about climate. Interviewees noted this may be an inappropriate use of the TRMs. One 

industry expert elaborated by saying: “States should consider other adjustments besides weather, and a challenge 

with a national TRM is understanding all those parameters.” TRMs may also not be being applied properly when 

algorithms are very complicated and have lengthy calculations with a large number of inputs. 

Northwest RTF

Those familiar with the Northwest RTF spoke very favorably of its capabilities. The RTF is a regional forum with an 

online repository, providing deemed values for a comprehensive list of measures with supporting calculations and 

documentation. Several interviewees commented that part of what makes the RTF database robust and credible is 

it uses evaluation data to inform the ex ante savings estimates. One interviewee even suggested the national 

database would be most useful if it consists of several regional forums and databases similar to the RTF. Another 

interviewee from the Northwest praised the RTF’s forum approach, allowing interested parties to weigh in, which 

this individual thought would be a good feature for a national database. The only improvement suggested for the 

RTF concerned its search capabilities, which users said are difficult to use for those unfamiliar with the site. 
June 2011 www.seeaction.energy.gov 28

Cross-Database (DS&A and EM&V Document Database) Considerations

The previous subsections discussed issues specific to each of the two databases under consideration. This 

subsection covers topics, such as hosting and funding, that are common to both databases, and outlines the 

possible linkages between the two databases.

Hosting

During interviews, Cadmus asked for feedback on appropriate entities for developing, maintaining, and managing 

both the databases. Industry professionals most often suggested DOE as the single entity that would be most suited 

to hosting such databases, and the suggestion of DOE in conjunction with a third-party and an independent 

administrative entity was mentioned almost as frequently. Other suggestions included: a regional infrastructure or 

collection of regional entities; a forum similar to the Northwest RTF; the CEE; and a community-based, non-federal 

entity (see Table 9).

Table 9. Recommendations of Possible Candidates 

for Database Hosting

Hosting Recommendation Frequency

DOE 5

DOE Manages Contract with Third-Party Administrator 4

Independent Entity 4

Regional Infrastructure or Collection of Regional Entities 3

Forum Like Northwest RTF 1

CEE 1

Note: Multiple responses allowed.

Determining the best entity for hosting the databases must take into account several factors, and will require 

thorough review and careful consideration. Interviewees emphasized the importance of having a database owner 

and host that is trustworthy to all stakeholders (utilities, regulators, third-party implementation contractors, and 

EM&V professionals). They noted the administrative function would be complicated, since it requires expertise in 

multiple disciplines such as library science (cataloguing and archiving), industry knowledge, engineering, and 

information technology. 

Funding Sources

Most interviewees stated DOE would be the most appropriate funding source, at least during early development 

and implementation stages. After the databases have been developed and are operational, most of the funding 

responsibility may be transferred to other entities, such as states, ratepayers, current sources of evaluation funding, 

and national utility associations, among others. 

Eight out of nine responding interviewees said they would be willing to pay a subscription fee for access to the 

databases, depending on the quality of the databases and the added value they offer. Some interviewees suggested 

imposing fees on a sliding scale; so states or entities with smaller evaluation or energy efficiency budgets would not 

be overburdened by costs. A subscription fee may only be possible after the databases are proven to be valuable 

resources. Two interviewees noted they would be more willing to pay a subscription fee if they could provide 

feedback on database content and updates. Regardless of the databases’ hosts or funding source, several 

interviewees commented the involvement of multiple stakeholders in the planning and implementation process

would strengthen and improve the final products. 
June 2011 www.seeaction.energy.gov 29

Criteria for Best Practices

Promoting the adoption of standard, best practices in savings calculations and EM&V practices should be a primary 

objective in developing the national databases. However, determination of what constitutes a best practice requires 

establishing clear criteria and using them consistently to rate and screen TRMs and EM&V reports. In addition, the 

criteria of best practices are relative, depending on efficiency program and EM&V goals as well as budgets and 

resources available to conduct the EM&V. We asked the industry experts to identify potential criteria to be 

considered when evaluating whether a plan, report, or market study serves as an example of best practices. Criteria 

identified included:

 The plan/report clearly describes:

o Assumptions

o Sample design and sample sizes

o Certainty of results

o Clear descriptions of context and objectives

o Source of data (primary versus secondary) 

o Variance between ex ante and ex post savings

 The report includes actionable recommendations

 The plan/report uses plain wording, not excessive jargon or engineering language

 The plan/report discusses threats to validity and controls for alternate hypotheses.

Respondents generally expressed concern about the many obstacles to defining a best practice objectively and that 

such a determination would vary by situation. These barriers may make it more feasible to address best practices on 

the actual product, such as a final EM&V report, than to judge whether best practices were actually employed in 

conducting the evaluation or study. For example, criteria such as involvement of independent third-party 

evaluators, approval by regulators, or study vintage could serve as objective and consistent measures that might be 

used. 

The key criterion for a national DS&A database is transparency. Nearly all respondents indicated that for such a 

national database to be accepted, the approach must be clear and apparent. Ideally, the savings estimates must be 

based on evaluation results (similar to the RTF), though recognizing those data may not be available, particularly in 

states where little or no EM&V has occurred.

Links Between EM&V Documents and DS&A Databases

Several industry professionals expressed an interest in establishing a link between the EM&V documents database 

and the DS&A database. Specifically, they envisioned the evaluation results from reports and studies in the 

documents database used for informing data in the measures database. Although developing this capability would 

be challenging, a link between databases could improve the transparency and consistency of documentation, and, 

as one respondent noted, this would support public information sharing. The Northwest RTF was mentioned several 

times as a solid model for a national database, and it was noted the RTF relies on evaluation findings as the basis for 

regularly updating measure savings. Some respondents viewed the RTF as more of a source for ex post than ex ante

savings estimates.
June 2011 www.seeaction.energy.gov 30

Recommendations

In our interviews with potential EM&V document and DS&A database users, we identified two basic needs. 

The first is to provide new practitioners with available resources that are easy to use and have a limited, focused 

content. New practitioners need information of all types to support their work, including EM&V plans and reports 

and measure ex ante savings information, as well as supplemental information, such as EM&V protocols, market 

studies, measure-specific technical studies, TRMs, and RFP examples. To facilitate locating and understanding these 

documents, an online resource should have clear navigation and search capabilities, and avoid clutter that might 

overwhelm the user. For ex ante measure savings, new practitioners need resources to allow them to produce 

savings estimates applicable to their particular geographic region and other unique circumstances as easily as 

possible. 

As new practitioners gain expertise in energy efficiency program planning, administration, and EM&V, their needs 

change. Instead of general, how-to information, they are more likely to need greater detail and access to a broader 

collection of available work. This information will allow them to broaden their knowledge and compare their 

methods and results with others. 

The second basic need our interviews exposed was to provide experts with a single-source location to search for 

and retrieve information. Our interviewees were generally aware of resources readily accessible online, but were 

unhappy with the limited nature of these resources and the time and effort required to perform a thorough search 

across resources. An online resource for EM&V documents that would satisfy experts would necessarily be more 

comprehensive than one satisfying the needs of new practitioners, and the methods for populating the database 

would be significantly different and more resource intensive. 

In regard to the DS&A database, experienced practitioners expressed the desire for a resource allowing them to 

easily find savings estimates or supporting documents and studies across many different regions and programs. The 

purpose of such a resource would be to better understand different underlying methodologies and to compare their 

findings.

Recognizing different needs between new and experienced practitioners, there are two different database 

options—focused and comprehensive—for each of the two resources: an EM&V database and a database of 

measure savings. The four options and their primary features are outlined in Table 10. 
June 2011 www.seeaction.energy.gov 31

*Savings based on past evaluation results would need to be updated regularly. The two comprehensive measure savings options 

are mutually exclusive as described below.

The two focused options could together comprise a starter kit resource, providing a single resource of documents 

and links to EM&V documents and DS&A resources. Given that SEE Action’s EM&V Working Group originally 

envisioned these databases as primarily serving nascent states, our primary recommendation is to concentrate 

initially on developing the starter kit rather than the more comprehensive options.

The scope of this study included examining whether the database containing EM&V documents should also include 

EM&V plans, or if a separate database should be created for plans. Cadmus recommends using a single database 

solution for the following reasons:

 It provides a single location for practitioners to find the information they require

 No technological barriers exist for implementing a combined database, as the database structure and 

site functionality will be identical regardless of document type.

 Our interviewees expressed a strong desire to not limit the types of documents collected on the site, 

but rather to have a variety of content, as described above.

We developed the cost estimates for each resource configuration in two steps. First, we organized the development 

and administration functions of the databases into specific tasks. Second, we assigned cost estimates to each task 

according to its anticipated required level of effort. 

Level of effort is expressed in employee-years, based on 2,080 annual hours for development, or FTE for annual 

maintenance, and an average labor rate of $150/hour. This rate can be adjusted to account for varying labor costs, 

depending on the organization ultimately conducting the work. We provided low and high estimates, in addition to 

the likely estimate, to show the potential range of costs. The low estimate is 25% less than the likely estimate, and 

the high estimate is 50% more.

Table 10. Resource Options

Targeted User EM&V Documents Database

Deemed Savings and 

Algorithm Database

New Practitioner - individual or 

organization seeking 

introductory information

Focused

 Links to existing resources

 Guidelines on best practices

 Exemplar documents included based 

on expert input 

Focused

 Links to TRMs and other existing 

measure databases

 White papers on technologies

 Example algorithms with details on 

parameters used 

Expert User - individual or 

organization seeking 

supplemental or comparative 

data 

Comprehensive

 Large collection of diverse 

documents

 Sophisticated search and tagging

 Regular additions and updates

 Data collected during evaluation

Comprehensive

National or regional database(s) with 

savings estimates base on either:

 A database of verified past results* or 

modeling data

Or

 An interactive calculation engine built on 

a set of predefined algorithms (i.e. an 

online TRM)
June 2011 www.seeaction.energy.gov 32

We considered initial development costs and annual recurring costs to maintain and enhance the resources. The 

task areas we considered and their supporting subtasks include:

Development Costs

 Requirements Gathering

o Client requirements

o User/stakeholder interviews

 Specification and Design

o Specification and use case development (a use case is a description of steps or actions in a 

software system that lead the user towards a desired result)

o Design and design documentation

o Cost estimation

 Technical Implementation

o Technology selection

o Template setup

o Initial content creation/loading 

o Searching for and evaluating existing documents to identify those representing best practices

o New content development, compiling and loading documents and descriptive text

 Content Development

o Determine topics to cover

o Design and arrange sections (pages)

o Site navigation

o Determine the amount of documents to include

o Determine methods for identifying and evaluating documents

 Testing

o Internal testing

o Client acceptance testing

o Modifications

 Release and Maintenance

o Deployment

o Marketing

o Link/search placement
June 2011 www.seeaction.energy.gov 33

Annual Recurring Costs

 Content Management

o New content development and identifying new source documents

o Site contents updates

o Ongoing quality assurance/quality control 

 Site Administration

o Hosting

o Site management

o User support

 Site Enhancement

o Change request tracking

o Site updates and testing.

The main features, possible content, information technology, and costs for each of the four options identified in 

Table 10 are described below. 

EM&V Document Database—Focused

Content

A resource that would address the needs of new EM&V practitioners would be best accomplished through a Web 

content management system (CMS), with original page content, documents, navigation elements, and search 

capabilities.

The website would include original page content describing EM&V practices and processes, and have links to other 

online resources, including those identified as part of this study. Compared to a strict document repository with 

only search capabilities, this type of site would permit navigation to specific topic areas and content, which could 

include links to supporting documents. Documents included on the site would be vetted by an expert or expert

panel, and would be judged on their usefulness to new practitioners and their representation of industry best 

practices. Potential topic areas and supporting documents may include, but not necessarily be limited to:

 EM&V plans

 EM&V protocols

 EM&V reports

 Market study reports

 Conservation potential studies

 Process evaluation reports

 Example procurement documents such as RFPs, Requests for Qualifications, and lists of EM&V 

practitioners.
June 2011 www.seeaction.energy.gov 34

Although the content in this focused resource would initially be limited, the CMS framework allows expansion of 

topic areas and document types without additional investment in the underlying technology.

Technology

CMS provides a popular tool for website creation, and many proprietary and open-source products are available for 

use. Microsoft SharePoint®is the most widely recognized proprietary software. Open-source products include 

Joomla!®, WordPress, DrupalTM, and many others. CMS popularity is driven by its ease of use and low cost to set up 

and administer. CMS does not require extensive programming or database setups. Development work primarily 

occurs in template setups and content loading.

Cost

The expected level of effort and associated costs for developing and maintaining the focused EM&V database 

option are summarized in Table 11 and 11. As explained above, we based the cost estimates on the expected level 

of effort, expressed in employee-years (for development) or FTE (for maintenance), assuming 2,080 annual hours at 

a labor cost of $150/hr. We expect the total costs for development and launch of this option to range between 

approximately $520,000 and $775,000, with between $200,000 to $300,000 required for ongoing annual 

administration and maintenance (Table 11 and Table 12). 

Table 11. Estimated Development Labor and Cost (Focused EM&V 

database)

Task

Estimated 

Employee-years

Requirements Gathering 0.20

Specification and Design 0.125

Technical Implementation 0.20

Content Development 0.75

Testing 0.125

Release and Maintenance 0.25

Total Development Employee-Years 1.65

Total Development Cost ($150/hr)

Most Likely $514,800

Low $386,100

High $772,200
June 2011 www.seeaction.energy.gov 35

Table 12. Annual Recurring Labor and Costs 

(Focused EM&V Database)

Task Estimated FTE

Content Management 0.25

Site Administration 0.25

Site Enhancement 0.125

Total Recurring FTE 0.625

Hosting (third-party) $6,000

Total Development Cost ($150/hr)

Most Likely $201,000

Low $150,750

High $301,500

Content development represents the highest cost element, and is also the potential source of the most variability in 

the total cost, specifically as it relates to searching for and evaluating documents and creating new content for the 

site. The site’s exact content—including topics covered, design and arrangement of sections (pages), site navigation, 

the amount of documents to include, and methods for identifying and evaluating documents—would need to be 

better specified before a more accurate estimate could be calculated. 

The hosting costs would vary depending on the configuration (internal versus third-party) and on the software 

platform chosen for the site. We have assumed a third-party hosting configuration, based on feedback from 

interviewees that the site would be best managed by an impartial entity. We further assumed an open-source 

solution would suffice, and there would not be significant software costs involved.

Implementation

Cadmus recommends a CMS-based, focused content website, largely because of its relative ease to create and 

launch. The implementation issues that need to be resolved include:

 What content should be included?

 Who would create and administer the site?

 What amount of ongoing content generation would be necessary?

 If expert evaluation of documents proves to be required, which experts would be used, how they 

would be selected, what evaluation process they would use, and how much they would cost?
June 2011 www.seeaction.energy.gov 36

DS&A Database—Focused

Similarly to the EM&V documents providing a focused database resource, a focused DS&A database resource would 

serve as an online resource, targeted toward practitioners new to the energy efficiency field.

Content

A focused resource, assisting new practitioners with generating ex ante measure savings for program planning and 

EM&V, should provide information on calculation methodologies and best practices. The content would be similar 

to that presented in TRMs adopted by state utility regulators for planning and evaluation purposes. The resource 

would not, however, directly supply ex ante savings algorithms or values or other measure-specific data; instead, it 

would provide links to existing TRMs or other measure savings values resources. In the comprehensive resource 

subsection below, we discuss in greater detail the options and issues with having a resource providing measure 

savings.

Technology

Cadmus suggests including the focused measure resources into the focused document resource as a specific 

information category. This portion of the database could include existing TRMs, primary source materials on 

savings, and other technical documents related to measure savings.

Costs

If the measures information is combined with the documents resource, the only additional costs would be the 

incremental hours spent collecting and loading measure-related documents. We estimate this time at roughly 200

hours initially, and between 100 and 200 hours each subsequent year. The incremental cost would be around 

$30,000 in the first year, and between $15,000 and $30,000 for every year thereafter. 

Implementation

The only additional implementation issue with including the measures information in the documents resource is 

determining what content to include, given variations in the computational methodologies and assumptions found 

in existing TRMs.

EM&V Document Database—Comprehensive

Many experts expressed a desire to have a comprehensive, national repository of EM&V documents, including 

evaluation reports, plans, and the other types of documents listed above. As described, the two existing primary 

EM&V document sources (CALMAC and the CEE site) have limitations in their completeness, usability, and 

accessibility, leaving the need for a fully comprehensive, national-scope document repository unmet. 

Content

A national repository could cover the same topic areas as the focused option. The goal would be to compile as many 

documents as possible in one site to remove the need for practitioners to search multiple sites. A sophisticated 

search and filtering user interface would provide easy access to the documents and, ideally, would limit the number 

of results returned. In this way, users could find what they seek while reviewing only a small number of search 

results. Cadmus recommends each document in the repository include:

 A descriptive title

 A five to 10 line summary
June 2011 www.seeaction.energy.gov 37

 An abstract

 A list of keywords, selected from a predefined master list to ensure consistency

 A format allowing full-text indexing for searches

 As much descriptive metadata as possible, which could include:

o Author name(s)

o Author firm

o Sponsoring agency(ies)

o State(s)

o Report date

o Program year

o Program type (energy efficiency, demand response, low-income, etc.)

o Document type (evaluation report, protocol, market study, etc.)

o Sector(s)

o Specific measure types included

o Specific analytical methods used.

Both CALMAC and CEE use a relatively passive process for gathering content. While the CPUC requires utilities to 

post certain documents to CALMAC, the onus is on document authors or sponsors to make submissions and fill out 

online forms. CEE relies on a voluntary submission process with its members.

Passive approaches and more active approaches to content gathering each have advantages and disadvantages. A 

passive approach takes less effort, but at the expense of incomplete coverage and of being potentially biased 

toward the most diligent contributors. An active approach would be more time-consuming, but would lead to a 

more populated and robust site. Active methods for gathering documents include: developing and maintaining a list 

of all known online sites where documents are posted, and regularly checking for new material; and soliciting 

documents directly from utilities, commissions, consultants, and other sources.

Several experts discussed the possibility of a national effort, spearheaded by DOE or another agency, to establish a 

framework and infrastructure for collecting public material for inclusion in the database. This raises several issues 

related to appropriate monitoring and compliance, which would need to be further researched. Regardless of the 

chosen collection method, implementers will need to perform a librarian function to monitor and follow up on 

submissions. Each submitted document would have to be vetted for appropriateness, quality, and data validation. 

Technology

A CMS similar to that suggested for the focused option and based on the same software would likely meet the 

requirements of a more comprehensive resource. By basing this database on the same software framework, the 

comprehensive database could evolve from the focused database. 

For usability, Cadmus suggests having both simple and advanced search options available to users, as commonly 

occurs on many websites. Simple search interfaces typically provide a single text box for entering a search term, and 

returns results matching the term across multiple fields (document title, abstract, keywords, etc.). An advanced 

search and filtering interface would also allow searching for specific terms, while providing searches in specific 
June 2011 www.seeaction.energy.gov 38

fields, multi-term searches, and limited results based on filters applied to any of the metadata fields. For example, 

the CALMAC search page provides filtering capabilities for the limited amount of metadata collected (but does not 

allow field-specific text searches).

Open source CMS software often includes a core search module with basic functionality, allowing the developer 

community to create add-on modules to provide enhanced features. Modules exist for the Drupal platform, for 

example, to meet advanced and full-text search features, but it may be necessary for the implementing organization 

to create custom modules specifically for this application. The cost estimates below assumes some custom work will 

be required.

Cost

The total costs for development and launch of the comprehensive EM&V documents database are expected to 

range between approximately $825,000 and $1.25 million, with between $320,000 and $480,000 required for 

ongoing annual administration and maintenance (Table 13 Table 14). Again, these estimates are based on the 

expected level of effort, expressed in employee-years or FTE, assuming 2,080 annual hours at a labor cost of 

$150/hr.
June 2011 www.seeaction.energy.gov 39

Table 13. Estimated Development Labor and Cost (Comprehensive 

EM&V Database)

Task Estimated 

Employee-Years

Requirements Gathering 0.20

Specification and Design 0.25

Technical Implementation 0.25

Content Development 1.50

Testing 0.20

Release and Maintenance 0.25

Total Development Employee-Years 2.65

Total Development Cost ($150/hr)

Most Likely $826,800

Low $620,100

High $1,240,200
June 2011 www.seeaction.energy.gov 40

Table 14. Annual Recurring Labor and Costs 

(Comprehensive EM&V Database)

Task Estimated FTE

Content Management 0.50

Site Administration 0.25

Site Enhancement 0.25

Total Recurring FTE 1.00

Hosting (third-party) $9,000

Total Recurring Cost ($150/hr)

Most Likely $321,000

Low $240,750

High $481,500

Implementation

Implementation issues to be addressed for creating a comprehensive, national EM&V document resource include:

 Collecting content. Though the cost estimate above assumes an active approach to content 

collection, it may prove to be more time consuming to find, evaluate, and post documents than our 

estimate allows.

 Generating keywords and abstracts. Under an active collection process, implementers would review 

each document and generate an abstract, list of keywords, and other metadata to fit the site format, 

allowing for sophisticated filtering.

 Permission to post. The implementer may find certain entities would not grant permission for their 

documents to be posted on the site, particularly if doing so might reduce traffic to a similar website, 

such as CALMAC. Without a national mandate to submit documents, the question of having rights to 

reproduce content from other sources requires further research.

 Site creation and administration. The responsibility for creating and administering ongoing 

maintenance of the site needs to be determined.

DS&A Database—Comprehensive

Content

Determining a resource that would serve as a comprehensive national DS&A database proved to be the most 

challenging aspect of this scoping study. Ideally, we envision this resource as a database of measure savings values

that a practitioner could access by selecting the desired measure, configuration, baseline, region/weather, and 
June 2011 www.seeaction.energy.gov 41

other parameters, and that the database would provide feedback on specific savings and, perhaps, cost 

information. Savings values would be based on actual measured data or be calculated based on a universally 

accepted set of algorithms and regionally specific parameters made applicable to the entire country. Significant 

resources would be needed to determine the appropriate values or algorithms to use.

Unfortunately, it would be extremely difficult to create such a resource. The data required for a data-driven 

resource do not exist in sufficient quantities or qualities to populate such a tool, and no universal agreement exists 

across existing TRMs as to how measure savings should be calculated (see Section 0, Review of Existing Resources). 

It would be more feasible, although still difficult, to create individual regional DS&A databases. Depending on the 

region, data may be more available and consistent. However, the primary obstacle of determining appropriate 

methods, algorithms, and assumptions to use would remain.

Above, we alluded to the issue of there being two distinct methods available for arriving at an ex ante savings value: 

a data-driven model approach, based on prior observation or building simulations; and an algorithmic approach. 

These methods would require different types of software development, although efforts and costs would be similar. 

We describe each approach individually, and then provide a cost estimate applying to either method.

Data-driven Model

DOE’s original intent was for Cadmus to investigate the feasibility of creating a national DS&A database. To compile 

such a data-driven resource, data to populate the database would need to exist, be readily accessible, and be 

described sufficiently to allow its conversion to a standard format. Certainly, some of these data exist in various 

forms as products of ongoing energy efficiency work. These data could also be determined from building 

simulations, conducted for a large variety of building types and climate zones (as done for DEER). 

Some existing resources compile this type of information, most notably the Council’s RTF, which maintains an online 

database of measure information, including Microsoft Excel spreadsheets with savings data based on a combination 

of algorithmic approaches and ex post savings. Data are specific to the Northwest region, and go through a rigorous 

review process before inclusion on the site. The RTF’s annual budget is approximately $1.5 million, and 

approximately 25 voting members volunteer their time and expertise to support the RTF.

Algorithmic Approach

Early in our work on this study, Cadmus began to explore the option of an algorithm-based tool, simply because the 

majority of TRMs in use across the country are based on these. We discussed this internally and with expert 

practitioners to determine what would be required to create an online tool of algorithms, where users could select 

desired measure parameters and have a calculated ex ante value returned. Such a tool would require development 

of the following:

 A set of universally accepted algorithms for all measures included in the database

 Regionally specific parameter data to serve as inputs to the calculations

 An engine to perform the calculations.

Developing each of these items would require substantial work, and, in the case of a national DS&A database, 

would require high levels of cooperation and acceptance across the various states, utilities, and other organizations 

serving as stakeholders in the process. From the interviews, we learned stakeholder input and transparency would 

be critical for entities to accept, trust, and adopt the resource.

As discussed earlier, consistent approaches and assumptions have not been provided for calculating ex ante savings 

in the TRMs we reviewed. The process by which a national DS&A database would be created is not clear, and,

without a federal requirement to do so, it is likely many states would opt to not adopt a national standard. Without
June 2011 www.seeaction.energy.gov 42

a national requirement, the usefulness and value of the database would be diminished. Cadmus considered 

incorporating all available TRMs, allowing users to select which approach best meets their needs. We suggest, 

however, that in addition to a significant per-TRM increase in costs, the inclusion of many different options would 

confuse users, and lead to validity questions about each option and the general algorithmic approach.

In regard to parameters, TRMs often specify statewide values, such as hours-of-operation, baseline condition, and 

heating/cooling degree days. These values vary by TRM, and may not yet exist for states without a TRM or a history 

of energy efficiency programs. The method for reconciling existing parameters values between TRMs and the new 

values created is not clear. Another approach is to have all parameters be user-specified. This, however, puts the 

burden on the user, and could lead to misuse of the tool.

Finally, the implementer would need to develop the underlying calculation engine. Consulting firms we interviewed 

as experts in this field already have calculation engines or the capability to create them. This is the most 

straightforward design aspect, but would still be expensive.

Technology

Either of the two model options above would require a custom-developed software tool, and could not be 

implemented with existing solutions. Development and hosting of the site could be conducted in any number of 

what are referred to as solutions stacks. Solutions stacks represent combinations of compatible document servers, 

Web servers, applications, and database software used for Web-based software development. Examples of popular 

solutions stacks include:

 LAMP (Linux, Apache, MySQL, Perl/PHP/Python)

 WINS (Windows, IIS, .NET, MSSQL)

 Oracle (Oracle Integrated Stack).

The specific technology chosen would depend more on the specialty of the selected implementer rather than the 

software’s specific cost or capabilities.

Either of the savings tool models will require custom database development to store information. In the data-driven 

model, these would include database tables to store savings values and all relevant metadata. In the algorithmic 

approach, tables would store formula information and all parameter data available for user selection. 

Administrative interfaces for loading, managing, and updating the database would be required in either case.

Cost

We expect the total costs for development and launch of a comprehensive measure resource to range between 

approximately $1.1 million and $1.7 million, with between $560,000 and $840,000 for ongoing annual 

administration and maintenance (Table 15 and Table 16). These estimates assume a database tool would be 

developed from scratch, independently of any particular consulting/software development firm’s existing software 

that might be adapted. The content creation component also assumes roughly two employee-years. These costs are 

for the development of a national database. If regional databases were developed instead, the technological costs 

for the first database would be the same, but subsequent databases would have fewer resource needs for 

requirements gathering and specification and design. The content development would be less, roughly 0.5 

employee-years per region.
June 2011 www.seeaction.energy.gov 43

Table 15. Estimated Development Labor and Cost 

(Comprehensive DS&A Database)

Task Estimated 

Employee-Years

Requirements Gathering 0.20

Specification and Design 0.25

Technical Implementation 0.75

Content Development 2.0

Testing 0.20

Release and Maintenance 0.25

Total Development Employee-Years 3.65

Total Development Cost ($150/hr)

Most Likely $1,138,800

Low $854,100

High

$1,708,200
June 2011 www.seeaction.energy.gov 44

Table 16. Annual Recurring Labor and Costs 

(Comprehensive DS&A Database)

Task Estimated FTE

Content Management 0.75

Site Administration 0.50

Site Enhancement 0.50

Total Recurring FTE 1.75

Hosting (third-party) $12,000

Total Recurring Cost ($150/hr)

Most Likely $558,000

Low $418,500

High $837,000

Implementation

Implementation issues regarding available ex ante savings values and the possibility of creating a national database

were discussed above. Another significant issue with the algorithms and ex ante savings values is the frequency with 

which the underlying data can change, largely due to updates in codes and standards. Many experts we interviewed 

reported this would pose a significant challenge. A strategy for providing updates would need to be determined,

with consideration to keeping past versions available, since multiyear programs and evaluation efforts require a 

consistent savings resource. Maintaining multiple, concurrent versions of the same data would be difficult, costly, 

and potentially confusing to users.

A barrier to using the data-driven approach is in providing sufficient documentation on the sources of each 

datapoint to gain user trust that data are accurate. A separate document database may need to exist along with 

savings data to house the original source material.

Conclusion

This report presents the results of a scoping study to research the various options for the content development, 

design, and administration of two potential national databases. The first is a national database of standardized, 

region-specific, ex ante algorithms, and associated stipulated savings estimates for conventional electric and natural

gas energy efficiency measures (a national DS&A database). The second is a document database of EM&V 

documents. 

Cadmus reviewed currently available resources and interviewed several stakeholders to assess the needs and 

challenges for these databases, according to the views of energy efficiency industry experts and new practitioners. 

The study results indicate a national DS&A database would be very difficult to produce, with significant regulatory 

and other barriers. An EM&V documents database would be more tractable and valuable to the energy efficiency 

community. However, ongoing maintenance and upkeep of such a database could require significant resources. 
June 2011 www.seeaction.energy.gov 45

Cadmus recommends initially developing a focused resource (or starter kit), providing links and including key 

documents to guide practitioners new to the energy efficiency field. This focused resource could be built upon to 

develop a comprehensive resource of EM&V documents if sufficient interest and funding remains available.
This information was developed as a product of the State and Local Energy Efficiency Action Network (SEE Action), facilitated by the U.S. 

Department of Energy/U.S. Environmental Protection Agency. Content does not imply an endorsement by the individuals or organizations that 

are part of SEE Action working groups, or reflect the views, policies, or otherwise of the federal government.