# Scoping Study to Evaluate Feasibility of

National Databases for EM&V Documents and Measure Savings

Tina Jayaweera, Hossein Haeri, Allen Lee, Scott Bergen, Cynthia Kan, Aquila Velonis, Christy Gurin, Michael Visser, Andrew Grant, and Ashley Buckman
The Cadmus Group Inc. / Energy Services 720 SW Washington Street, Suite 400 Portland, OR 97205 503.467.7100

Evaluation, Measurement and Verification Working Group

June 2011June 2011 www.seeaction.energy.gov ii



## Disclaimer

This docume nt was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor The Regents of the University of California, n or any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or The Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, or The Regents of t he University of California.June 2011 www.seeaction.energy.gov iii



## Acknowledgements

This report was prepared for Lawrence Berkeley National Laboratory (LBNL) and funded by the Permitting, Siting and Analysis Division of the U.S. Department of Energy’s Office of Electricity Delivery and Energy Reliability under Lawrence Berkeley National Laboratory Contract No. DE

- AC02

- 05CH11231. The project was managed by Steven R. Schiller, of Schiller Consulting, Inc. and Charles A. Goldman of LBNL. We appreciate the ongoing support and guidance they provide d throughout the course of this study. The information that forms the basis for the findings and recommendations of this study was gathered through interviews with the national experts l i sted below. The Cadmus Group Inc. thanks them for their invaluable in sights and contributions, while remaining solely responsible for any errors in this report. Dan Barbieri ( KEMA ); Ben Bronfman ( The Cadmus Group Inc ); Peter Carvalho ( CGI ); Tim
Caulfield ( C alifornia Measurement Advisory Council); Pete Cleff ( PPL Electric Uti lities); Kevin Cooney ( Navigant Consulting); Dean Creed ( Santee Cooper ); Tom Eckman ( Northwest Power and Conservation Council); Carla Frisch (U.S. D epartment of Energy); Terry Fry ( Nexant , Inc).; Lauren Gage ( B onneville P ower A dministration); Donald Gillig an ( N ational A ssociation of E nergy Service C ompanies); Fred Gordon ( E nergy T rust of Oregon); Sarah Griffith ( C onsortium for E nergy E fficiency); Dennis Hartline ( Maryland Energy Administration ); George Hernandez ( Pacific Northwest National Laboratory ); Sami Khawaja (The Cadmus Group Inc); Steve Kromer (Efficiency Valuation Organization); Noah Long ( N atural R esources D efense C ouncil); Steven Long ( S outhern C alifornia E dison); Beatrice Mayo ( P acific G as & E lectric Company); Cecily McChalicher ( N ortheast E nergy E fficiency P artnerships); Jennifer

Meissner ( N ew Y ork S tate E nergy R esearch and D evelopment A uthority); Eric Merkt ( Navigant Consulting); Julie Michals (Northeast Energy Efficiency Partnerships); Bill Miller (Sentech Inc.); Monica Nevius ( C onsortium for E nergy
E fficiency); Larry Philbin ( Santee Cooper ); Deborah Reynolds ( W ashington Utilities and Transportation Commission);
Shahana Samiullah ( S outhern C alifornia E dison); John Stoops ( KEMA ); David Sumi (The Cadmus Group Inc); Elizabeth Titus ( N ortheast E nerg y E fficiency P artnerships); P

. S

. (Vish) Vishwanath (V

- TECH, I nc ); Bryan Ward (The Cadmus Group Inc); Tory Webber (Southern California Edison); and Holly Wiggins ( Dayton P ower & L ight).June 2011 www.seeaction.energy.gov iv



## List of Acronyms and Abbreviations

ACEEE American Council for an Ene rgy

- Efficient Economy

AFUE Annual Fuel Utilization Efficiency

Btu British thermal unit

CALMAC California Measurement Advisory Council

CEE Consortium for Energy Efficiency

CFL compact fluorescent light bulb

CMS c ontent management system
Council Northwest Power and Conservation Council

CPA c onservation potential assessment

CPUC California Public Utilities Commission

DEER Database for Energy Efficient Resources

DOE U.S. Department of Energy

DS&A Deemed Savings and Algorithm

DWH domestic water heating

ECM electronically commutated motor

EF energy factor

EM&V e valuation, measurement, and verification

ETO Energy Trust of Oregon

FTE f ull

- t ime e quivalence

HE h igh efficiency

HP horsepower

HVAC heating, ventilation, and air conditioning

kWh kilowatt hour

LBNL L awrence Berkeley National Laboratory

M&V Measurement and verification

MAPE Market Assessment and Program Evaluation

MMBtu 1 million British thermal units

NEEA Northwest Energy Efficiency Alliance

NEEP Northeast Energy Efficiency Partnership

NLCBT National Laboratory Collaborative on Building Technologies

NREL National Renewable Energy Laboratory

NTG nitroglycerin

NY DPS New York Department of Public Service

NYSERDA New York State Energy Research and Development Authority

PUC Public Utility Commission

RFP Request for Proposals

RTF Regional Technical Forum

SEE Action State and Local Energy Efficiency Action Network

TRM Technical Reference Manual

W wattJune 2011 www.seeaction.energy.gov v



## Definitions

TRM (technical reference manual) * is a resource document that includes information used in pr ogram p lanning and reporting of energy

- efficiency programs. It can include savings values for measures, engineering algorithms to calculate savings, impact factors to be applied to calculated savings (e.g., net

- to

- gross values), source documentation, speci fied assumptions, and other relevant material to support the calculation of measure and program savings. A TRM may be in the form of a document or an electronic database. TRMs are currently in effect

in 21 states.

EM&V d ocuments d atabase is a Web

- based re pository of information on evaluation, measurement, and verification (EM&V) plans and reports.

Energy efficiency measure * is an installed piece of equipment or system, or modification of equipment, systems, or operations on end

- use customer facilities tha t reduces the total amount of electrical or gas energy and capacity that would otherwise have been needed to deliver an equivalent or improved level of end

- use service

.

Ex ante savings values * are f orecasted savings used for program and portfolio planning purposes.

Ex post savings values * are s avings estimate s reported by an evaluator after the energy impact evaluation has been completed

.

EM&V reports are formal documents describing the methodology, sources of data, and results of energy

- efficiency

progra m impact evaluations or project measurement and verification activities. EM&V reports are often prepared by third

- party, independent contractors.

EM&V plans establish the general procedures and methods for determining the energy savings and other impacts of energy efficiency measures, projects, programs, and/or portfolios. EM&V plans may be specific to a particular evaluation activity, or come in the form of a general protocol for evaluation of all programs.

Market studies * are analyses that provide an as sessment of how and how well a specific market or market segment is functioning with respect to the definition of well

- functioning markets or with respect to other specific policy objectives. Common types of such studies are market characterizations, appli ance and equipment saturation studies, conservation potential studies, and benchmarking and best

- practice studies.

Process evaluation reports * are a systematic assessment of an energy efficiency program for the purposes of documenting program operations a t the time of the examination and identifying and recommending improvements to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction.

* Adapted from t he EM&V Forum Gloss ary of Terms and Acronyms, Version 2, March 2011

.June 2011 www.seeaction.energy.gov vi



## Table of Contents

Executive Summary

................................ ................................ ................................ ................................ . viii Purpos e and Approach

................................ ................................ ................................ ............................... viii Current Situation

................................ ................................ ................................ ................................ .......... ix Summary of Findings

................................ ................................ ................................ ................................ ... ix
EM&V Docume nts

................................ ................................ ................................ ................................ ................................ ...................... ix National DS&A Database

................................ ................................ ................................ ................................ ................................ ......... x Recommended Solutions

................................ ................................ ................................ ............................. xi

Introduction

................................ ................................ ................................ ................................ ................. 1 Report Organization

................................ ................................ ................................ ................................ ...... 1
Approach

................................ ................................ ................................ ................................ ...................... 1 Review of Existing Resources

................................ ................................ ................................ ....................... 2 Internal Expert Interviews

................................ ................................ ................................ .............................. 2 Interviews with Industry Professionals

................................ ................................ ................................ .......... 2

Review of Ex isting Resources

................................ ................................ ................................ ................... 3 Existing EM&V State, Regional, and National Documents Resources and Databases

............................... 4 Existing National or Regional EM&V Documents Databases

................................ ................................ ................................ .... 5 States and Other Organizations’ EM&V Documents Databases

................................ ................................ .............................. 5
CALMAC

................................ ................................ ................................ ................................ ................................ ................................ .......... 5
CEE MAPE

................................ ................................ ................................ ................................ ................................ ................................ ...... 6 Existing State, Regional, and National TRM Resources

................................ ................................ .............. 6
TRM Review

................................ ................................ ................................ ................................ ................................ ................................ .. 9 Measure Review

................................ ................................ ................................ ................................ ......... 12
Findings

................................ ................................ ................................ ................................ ..................... 19 EM&V Documents Database

................................ ................................ ................................ ...................... 19
Needs

................................ ................................ ................................ ................................ ................................ ................................ ............. 19 Feasibility

................................ ................................ ................................ ................................ ................................ ................................ .... 19 Content and Functionality

................................ ................................ ................................ ................................ ................................ .... 20 National (or Region al) Deemed Savings and Algorithm Database

................................ ............................. 22
Needs

................................ ................................ ................................ ................................ ................................ ................................ ............. 22 Feasibility

................................ ................................ ................................ ................................ ................................ ................................ .... 23
Content and Functionality

................................ ................................ ................................ ................................ ................................ .... 24 Quality Control

................................ ................................ ................................ ................................ ................................ .......................... 27 Comparison of National DS&A Database to Existing TRM Resources

................................ ................................ ................. 27 Cross

- Database (DS&A and EM&V Document Database) Considerations

................................ ............... 28 Hosting

................................ ................................ ................................ ................................ ................................ ................................ .......... 28 Funding Sources

................................ ................................ ................................ ................................ ................................ ........................ 28 Criteria for Best Practices

................................ ................................ ................................ ................................ ................................ ..... 29 Links Between EM&V Documents and DS&A Databases

................................ ................................ ................................ .......... 29

Re commendations

................................ ................................ ................................ ................................ .... 30 EM&V Document Database

— Focused

................................ ................................ ................................ ....... 33
Content

................................ ................................ ................................ ................................ ................................ ................................ ......... 33 Technology

................................ ................................ ................................ ................................ ................................ ................................ .. 34
Cost

................................ ................................ ................................ ................................ ................................ ................................ ................. 34 Implementation

................................ ................................ ................................ ................................ ................................ ......................... 35 DS&A Database

— Focused

................................ ................................ ................................ ......................... 36
Content

................................ ................................ ................................ ................................ ................................ ................................ ......... 36 Technology

................................ ................................ ................................ ................................ ................................ ................................ .. 36
Costs

................................ ................................ ................................ ................................ ................................ ................................ ............... 36 Impleme ntation

................................ ................................ ................................ ................................ ................................ ......................... 36 EM&V Document Database

— Comprehensive

................................ ................................ ........................... 36June 2011 www.seeaction.energy.gov vii

Content

................................ ................................ ................................ ................................ ................................ ................................ ......... 36 Technology

................................ ................................ ................................ ................................ ................................ ................................ .. 37
Cost

................................ ................................ ................................ ................................ ................................ ................................ ................. 38 Implementation

................................ ................................ ................................ ................................ ................................ ......................... 40 DS&A Database

— Comprehensive

................................ ................................ ................................ ............. 40
Content

................................ ................................ ................................ ................................ ................................ ................................ ......... 40 Technology

................................ ................................ ................................ ................................ ................................ ................................ .. 42
Cost

................................ ................................ ................................ ................................ ................................ ................................ ................. 42
Im plementation

................................ ................................ ................................ ................................ ................................ ......................... 44

Conclusion

................................ ................................ ................................ ................................ ................. 44



## List of Tables

Table 1. External Interview Respondents 3 Table 2. Sources for EM&V Plan s, Reports, and Market Studies 4 Table 3. TRM Resources* 7 Table 4. Results of Measure Ranking Process 11 Table 5. List of 10 Measures for Analysis and Review, Each in the Residential and Commercial Sectors 12 Table 6. Residential Measure Comparison Findings and Recommendations Summary 14 Table 7. Commercial Measure Comparison Findings and Recommendations Summary 16 Table 7. Commercial Measure Comparison Findings and Recommendations Summary 18 Table 8. Recommendations of Possible Candidates for Database Hosting 28 Table 9. Resou rce Options 31 Table 10. Estimated Development Labor and Cost (Focused EM&V database) 34 Table 11. Annual Recurring Labor and Costs (Focused EM&V Database) 35 Table 12. Estimated Development Labor and Cost (Comprehensive EM&V Database) 39 Table 13. Annual Recurring Labor and Costs (Comprehensive EM&V Database) 40 Table 14. Estimated Development Labor and Cost (Co mprehensive DS&A Database) 43 Table 15. Annual Recurring Labor and Costs (Comprehensive DS&A Database) 44June 2011 www.seeaction.energy.gov viii



## Executive Summary

This report presents the results of a scoping study to assess the need for national databases that can support best practices in energy

- efficiency program evaluation, measurement, and verification (EM&V) and to identify options for meeting those needs. The study was commissioned by L awrence Berkeley National Laboratory on be half of the

State and Local Energy Efficiency Action Network’s (SEE Action) EM&V Working Group.



### Purpose and Approach

The SEE Action Network ’s EM&V Work ing Group recognized the need for develop ing a set of databases to support best practices in energy effic iency program design and EM&V as a high

- priority project. These databases are expected to facilitate energy efficiency program design and evaluation, particularly in states with nascent energy efficiency initiatives.

The working group expressed interest i n two categories of publicly accessible, national databases:

1. National Deemed Savings and Algorithm (DS&A) Database: A database of standardized, region

specific, ex ante algorithms and associated savings estimates for conventional electric and natural gas

e nergy efficiency measures

2. EM&V Document Database: A database to provide access to available existing EM&V plans, EM&V reports, market studies, and other supplemental documents.

The focus for these databases is to support ratepayer

- funded programs administ ered by utilities or third

- party agencies and overseen by regulators.

The principal objectives of th is scoping study were to:

 Assess the need for these databases through surveys of energy efficiency experts and potential users

 Assess the consistency of ex isting technical reference manuals (TRMs) and other sources of deemed savings, and issues associated with developing a national DS&A database by comparing assumptions, savings values, and algorithms for 20 common energy efficiency measures, as reported in various existing state and regional databases

 Assess existing EM&V document database resources and the issues associated with developing a
national EM&V documents database

 Evaluate the feasibility and costs of alternative options for meeting the identified database needs

 Recommend practical options for the development, contents, and maintenance of a national DS&A database and EM&V document database that effectively satisfies the broadest set of needs
identified.

The Cadmus Group Inc. compiled data for this study from secondary and primary sources. We began the study with
a review of the publicly available sources for EM&V plans and reports, TRMs , and market studies. We compiled information on current needs and available options through 29 semi

- formal, struc tured interview sessions with 35

energy efficiency professionals, including mem bers of the SEE Action Network EM&V Working Group and industry experts representing utilities, non

- governmental regulators, consulting firms, and information technology professi onals.

The interviews covered a broad range of topics regarding the databases, with a focus on perceived needs, feasibility of developing various database options , desirable content and functionalities, potential administrative structures, and funding opt ions.June 2011 www.seeaction.energy.gov ix



### Current Situation

A number of sources for EM&V documents and TRMs are currently available online . In our research for this study,
we identified 19 publicly available sources providing EM&V documents . These sources are diverse in content,
structure, and functionality. They also range in sophistication from simply providing links to reports housed elsewhere to fully

- searchable databases, allowing filtering by sector, report type, and other criteria. Most EM&V reports available online are housed on nati onal or regional association w ebsites

, or on w ebsites of state

- level entities, such as the California Measurement Advisory Council.

However, a nationwide comprehensive database of EM&V documents does not exist

. T he Consortium for Energy Efficiency w ebsite appears to be the most comprehensive source for such documents , although its scope is limited to submissions only from its members.

EM&V plans are far less prevalent than EM&V reports. In this study, we identified only three sources of publicly
available EM&V plans, consisting of those jurisdictions

— C alifornia, Pennsylvania, and New York

— w here statewide EM&V protocols are in place.

We also identified 17 source s with ex ante measure savings values , covering 21 states and the District of Columbia.
These so urce s differ in geographic coverage, but are largely regional and are administered by state regulatory commissions, state advisory committees, non

- profit organizations, or utilities. The research, development, and
maintenance of these source s are generally contracted out to consultants. The only source s with a national scope are those available through the ENERGY STAR ® program, which provides measure

- specific savings calculators, generally in a Microsoft Excel ® format. The other sources vary in information presented, from tables of ex ante stipulated savings estimates to algorithms

.


### Summary of Findings

EM&V Documents

The information we gathered from interviews provides overwhelming evidence of the need for an EM&V document database. Nearly all respondents ag reed that priority should be given to the needs of new EM&V practitioners, with a focus on an audience of efficiency program administrator (primarily utility) staff and state regulators, although the database should be potentially useful to anyone. Nearly every interviewee agreed that some form of a centralized resource could help expand best practices in EM&V, create greater transparency and consistency of savings estimates, facilitate more widespread exchange of information and data

- sharing among practiti oners, and improve the overall quality of EM&V throughout the United States. A particular value is providing case studies/examples for those regulators and program administrators just starting to establish EM&V requirements and procedures.

The information obtained also points to concerns about the feasibility of such an undertaking, particularly with establishment and funding needs, procedures for cataloguing documents, structures for administration and ongoing maintenance as well as ensuring long

- term c ons isten cy and quality. Interviewees generally agreed that it would be important, but difficult, to create an institutional framework and infrastructure that encourages voluntary submittal of materials as they become available. Many thought continuous and con sistent voluntary participation could prove
to be the real barrier for the national EM&V document database.

Perspectives varied with respect to the content and structure of a new, national EM&V documents database. The main question raised by interviewees w as whether the database should be a comprehensive repository of all EM&V documents, or a more selective sample of documents that, according to certain criteria, distinguish them as exemplary or best practice. While arguably a selective database might prove more useful to new practitioners, there was little agreement as to what screening criteria should be included and how documents should be selected. Interviewees made several suggestions about how rigorous the screening process should be. These ranged from a simple check to determine whether the document had been accepted or approved formally by regulators, to aJune 2011 www.seeaction.energy.gov x

requirement that documents be subjected to stringent review and ranking processes by experts, based on predetermined criteria or perhaps just their judgment. In addition, the criteria of “best practices” can be quite problematic, as it is a relative versus absolute term.

A frequent suggestion was to create a database to serve as a “starter kit” or a set of examples and templates for state regulators and program administrators new to energy efficiency. This focused database, as we call it in this report, would guide this audience toward best and/or common practice approaches for developing EM&V requirements for their energy efficiency initiatives.

Reg ardless of the scope, the database search capabilities will likely prove to be one of its most important function s

. Interviewees made a number of suggestions for search ing and filtering capabilities, including k ey word s, program type, sample size, year of publication, methodologies, location, fuel type, and some measure of the study complexity.

National DS&A Database

There is a widespread recognition of the useful ness of national, or at least regional, databases of deemed savings and algorithms. However, b ased on our interviews, many barriers would need to be overcome before developing such a resource that meets the needs of states. In general, the notion of creating a national resource was not considered readily possible, largely due to vast regional varia tions in baselines, weather, economic s , demographic s , equipment stocks

, measure costs , program structures, evaluation needs, and, in particular, regulatory requirements. However, developing several regional databases is possible and in total could represen t a national deemed savings and algorithms.

The major concerns raised by many respondents were: (a) the feasibility of creating such a database and of practitioners coming to a consensus (in a given state or region, let alone nationally) on the appropriate methods, algorithms, and assumptions ; and (b) having individual states’ regulatory agencies, which oversee efficiency programs, adopting such a database when they have either existing TRMs or processes in place to develop ex ante savings values, algorithm s, etc

. Transparency in measure assumptions is critical to gaining acceptance, and a consistent methodology for collecting measure data is needed. To create a resource that could be used to compare savings across programs or states, a high level of detail and guidance would be required for each measure to ensure transparency and reduce the likelihood of human errors. Another substantial challenge would be the constant updates required to adjust measure savings values based on ongoing research, changes in ba selines, and new data.

Interviewees differed widely on the optimal structure, content, and functionalities of one or more regional (and/or national) DS&A databases. In general, respondents reported there is a greater need for a database of saving algorith ms rather than value s . The possibility of providing additional functionality of a dynamic database with embedded algorithms (that is, allowing the user to solve for the algorithm through the interface) was also discussed. Although it may offer greater flex ibility and be useful to a broader range of users

, the development and ongoing maintenance required may prove too difficult to overcome. A few respondents also indicated that the energy efficiency focus should be on whole

- building savings and not measure

- s pecific values, decreasing the usefulness of a national DS&A database, as that type of resource focuses on per unit savings. Finally, the large majority of experts reported that a national DS&A database would not be used consistently unless mandated by

reg ulators, and most opposed such a mandate, particularly in those states where a TRM has already been developed.

Our detailed review of 20 measures across 17 TRMs confirmed the legitimacy of these concerns. The TRMs had a wide variation in saving methodolog ies, technical assumptions, and input variables for estimating savings. Even where algorithms are similar, the input units and baseline assumptions are typically different. In addition, a number of the resources we reviewed had errors, highlighting the dif ficulty in creating such a resource. These all point to the challenges of creating a regional or national DS&A database, and the importance of consistent assumptions, quality control, and long

- term and substantial funding.June 2011 www.seeaction.energy.gov xi



### Recommended Solutions

We began t he scoping study by considering a broad range of solutions for the two databases (EM&V documents and DS&A), based on the perceived needs of both new and experienced practitioners, the desirable features and functionalities, and technology options. The info rmation we gathered in the course of interviews and from our more detailed review of existing resources and research on available technologies helped to narrow the options into two possible configurations for each of the two databases, differentiated prima rily in terms of the overall complexity and scope. We identify these as focused and comprehensive options (see Table ES

- 1). These two options vary primarily by their levels of user interface and how static the data are (particularly for the DS&A database). The focused resources are primarily targeted for new practitioners; the comprehensive resources are for expert users. New practitioners may find the comprehensive resources overwhelming, while expert users may find the focused resource lacking. This repor t includes estimates of resources required, both in employee

- years (2,080 annual hours)
or full

- time equivalent (FTE) employee hours and dollars for both the development of the database as well as ongoing administration and maintenance.



### Table ES

- 1. Resou rce Options

Op tion Targeted

User EM&V Documents DS&A

Focused New Practitioner

–

individual or organization seeking introductory
information Content:

 Links to existing resources

 Guidelines on best practices

 Exemplar documents (based on expert input) Co ntent :

 Links to existing TRMs and other existing measure resources

 White papers on technologies

 Example algorithms with details on parameters used Cost:

 Development Employee

- years: 1.7

– 2.5 Cost: $520k

- $775k

 Annual Maintenance FTE: 0.6

– 0.9 Cost: $2 00k

- $300k Cost:

 Development Employee

- years: 1.0

– 1.5 Cost: $ 325 k

- $ 500 k

 Annual Maintenance FTE: 0.4

– 0.6 Cost: $ 125 k

- $ 185 k Comprehensive Expert User

–

individual or organization seeking supplemental or comparative data Content:

 Large collection of div erse documents

 Sophisticated search and tagging

 Regular additions and updates

 Data collected during evaluation Content: National (or regional) database(s) of savings estimates based on either:

 A database of verified past results and/or modeling data Or

 A n interactive calculation engine based on a set of predefined algorithms (i.e. an online TRM) Cost:

 Development Cost Employee

- years : 2.7

– 4.0 Cost: $825k

- $1.25M

 Annual Maintenance FTE: 1.0

– 1.5 Cost: $325k

- $500k Cost:

 Development Employee

- years: 3.7

– 5.5 Cost: $1.1M

- $1.7M

 Annual Maintenance FTE: 1.8

– 2.6 Cost: $560k

- $850kJune 2011 www.seeaction.energy.gov xii

If the two focused database options were combined into a robust starter kit resource, an integrated solution could likely be developed and deployed closer to the cost of the focus ed EM&V document resource ($520k

– $775k) with an additional incremental cost of approximately $30k for development and management of the additional content required.

Although the results of this study indicate interest in both types of databases, a comprehe nsive EM&V document database was considered by interviewees to be more viable than a DS&A database. We recommend initially developing the starter kit, combining the features of the focused DS&A and EM&V documents databases described above. This would provi de a database of documents and links to ex ante savings values and algorithms within a single website and create a summary resource for those new to energy efficiency.

The focused solution could serve as a sound starting point, which may evolve into a mor e comprehensive solution

over time , if built using an open source content management system that allows for expansion

. By starting with a smaller

- scale option, it would be possible to better gauge users’ needs and use of the databases , gather feedback,
and expand the solution as appropriate. In terms of developers and administrators of these databases, the findings of the study indicate that the U.S. Department of Energy, either directly or through an agent, would be the entity best suited to spearhead and fund the development and deployment efforts as well as provide credence to its validity and value.J u ne 2011 www.seeaction.energy.gov 1



## Introduction

Lawrence Berkeley National Laboratory (LBNL) provides research support and technical assistance to the State and Local Energy Efficiency Act ion Network (SEE Action) on selected energy efficiency policies, programs, and technical

issues.

The SEE Action Network EM&V Working Group has recognized the need to develop a set of databases to support energy efficiency e valuation, m easurement, and v eri fication (EM&V) as a high

- priority project. These databases would help standardize ex ante measure savings, and would provide easy access to available best practice EM&V plans, reports, market studies, and other related studies. The databases are expected to facilitate energy efficiency program design and evaluation, particularly in states with nascent energy efficiency initiatives. The focus of these databases is to support ratepayer

- funded programs, administered by utilities or other third

- party entities, and overseen by regulators.

Under subcontract with LBNL, The Cadmus Group Inc. (Cadmus) conducted this scoping study to research the various options for the content, design, and administration of such databases, and to provide estimates of their

associate d costs. The EM&V Working Group envisions the development of two or three types of publicly accessible
databases:

 National Deemed Savings and Algorithm (DS&A) Database. A national database of standardized

, region

- specific, ex ante algorithms and associated stipulated savings estimates for conventional
electric and natural gas energy efficiency measures

.

 EM&V Document Database. One or more n ational databases that provide access to the energy efficiency EM&V plans and reports available in various state juris dictions and utilities

.

This report presents the results of the scoping study. We begin with a review of currently available resources, and then assess the need for such databases according to the views of energy efficiency industry experts and new practi tioners. Next, we determine gaps between the two groups (experts and new practitioners) in terms of information and data needs, and provide a framework for database systems and a supporting infrastructure to fill this gap.



### Report Organization

This report i s organized into five parts. This introduction is considered Section 1. Section 2 describes the overall approach of the study. Section 3 presents the results of our existing resources review. Section 4 summarizes findings from the interviews and from our r eview of ex ante measure savings in available databases. Section 5 offers recommendations for alternative database options and their associated costs to build and maintain. Finally, we report detailed technical specifications of measure s from available dat abases in Appendices A and B (see separate document). These appendices document the results of our review of the approaches to calculating savings for 20 measures across various technical reference manuals (TRMs). Appendix C lists the websites that contain EM&V
documents

. 

## Approach

Cadmus compiled the data for this study from secondary and primary sources. The secondary data collection effort included a review of available resources for EM&V plans and reports, TRMs, and market studies from a variety of

sourc es, including the Internet. We collected primary data through structured and informal interviews with industry experts, SEE Action EM&V Working Group members, and several potential future users of the database.June 2011 www.seeaction.energy.gov 2



### Review of Existing Resources

The first stage of the scoping study involved gathering information on various existing databases at the national, regional, state, utility, and state regulatory commission levels. For the initial search and review, we developed a list of entities with a broad scope, suc h as the Efficiency Valuation Organization and the International Energy Program Evaluation Conference, and identified states and entities most experienced in EM&V. We later expanded this list based on feedback from the industry experts interviewed.

To fac ilitate data collection, Cadmus tracked whether each entity’s website included EM&V plans, reports, and market studies, and other types of reports. When documents could not be readily found, we applied key word searches throughout the website content for t erms such as evaluation, report, study, and plan. Due to time and budget constraints, if we could not locate documents on a website within 30 to 45 minutes, we assumed they were not included online or were not readily accessible. In a few cases, during our interviews we received additional information on the availability of these data. In addition, where readily accessible online, we recorded information about the database administration method, data storage format, search capabilities, and other database f eatures.

We made a parallel search for sources with data on ex ante measure savings algorithms and/or values. These included existing databases such as the Northwest Regional Technical Forum (RTF), California’s Database for Energy Efficient Resources (DEER ), ENERGY STAR ® calculators, and TRMs adopted in various states.

As part of this effort, Cadmus compiled a comprehensive list of specific energy efficiency measures with savings algorithms or values from these sources. As described in Section 3 of this re port, we then ranked these measures based on criteria such as prevalence, sector diversity, and magnitude of potential energy savings. We then selected 20 measures for further comparison of their baselines, assumptions, calculation procedures, and reported saving values as found in existing databases. This comparison and review of existing databases provided insights on the feasibility of creating a new national database, or multiple regional databases, of ex ante measure savings algorithms and values, base d on commonalities among savings algorithms or values, and extrapolation of those algorithms to the national level.



### Internal Expert Interviews

After our initial review of existing sources, we conducted interviews with four Cadmus staff members who are

ene rgy efficiency EM&V industry experts 1 to elicit information on state, national, and regional level entities that have undertaken a significant number of evaluations over the last five years. This allowed us to gauge the complete ness of our existing resourc es list and determine if we should expand our research. The se internal experts
also provided insight s into potential topics to include in our interviews with respondents outside Cadmus

, and they furnished a list of potential candidates for interviews

. Thes e internal interviewees also provided information on possible criteria for determining whether a report, plan, or market study represent ed best practices.



### Interviews with Industry Professionals

Next, we conducted structured, informal interviews with indust ry experts outside Cadmus. Through these

interviews

, we collected information on the evaluation plan s and report s database

, in addition to the measures
database. We determined t he discussion topics for these interviews based on the overall project objectiv es and on the insights gained through our Cadmus expert interviews

. Our interviews with industry professionals focused on the following for both types of databases :

1 Cadmus experts interviewed were M. Sami Khawaja, Ben Bronfman, Bryan Ward, and David Sumi.June 2011 www.seeaction.energy.gov 3  Perceived need s

 Feasibility

 Content

 Functionality

 Structure/organization

 Data sources

 Fun ding and administration.

Our first key interview questions addressed whether respondents believe there is a need for the databases and what specific needs each type of database might address. We asked about their concerns regarding the feasibility of
these databases to identify potential problems that could occur in database creation and upkeep. Cadmus interviewers also asked about the desired content and functionality of each database, specifically in terms of the types of information that should be includ ed and the desirable search options and functional capabilities. We discussed the database structure and organization with interviewees to determine their database layout preferences. To obtain feedback on interviewees’ experience with other databases and their preferences, we asked for their observations of each database we mentioned. The data we gathered on functionality and features helped frame other research areas, such as cost, necessary components, selection processes, and ongoing administration.

Mor eover, we used this opportunity to elicit suggestions for funding sources and database administration options.

We conducted 29 telephone interviews. The interviewees represented utilities, regional non

- governmental organizations, regulators, and consultan ts. In several cases, more than one respondent participated in the call, and we were able to solicit input from 35 individuals. Interviewees included industry experts, personnel from three utilities that are relatively new to energy efficiency and EM&V (re ferred to as nascent practitioners in this report), and members of the SEE Action EM&V Working Group (see Table 1 ).



### Table 1 . External Interview Respondents

Respondent Type Number of Inter views Industry Expert s 16 Nascent Practitioners 3 SEE Action Network EM&V Working Group Members 7 Database Experts and Administrators 3
Total 29

We conducted additional interviews with three database administrators, one of which is a database develo per, to gain insights into specific database management details. We also conducted an interview with a U.S. Department of Energy (DOE) staff member who is also a SEE Action facilitator.



## Review of Existing Resources

This section presents the results of our existing resources review. The first subsection presents results from our review of EM&V resources, followed by the results of our review of existing TRM resources and other sources of ex

ante measure savings values and algorithms.June 2011 www.seeaction.energy.gov 4



### Existing EM&V State, Regional, and National Documents Resources and Databases

Existing EM&V resources and online databases are diverse in their content, structure, and functionality. They also range in sophistication from simply providing links to reports housed elsewhere to h aving fully searchable databases that allow filtering by sector, report type, and other criteria. Most EM&V reports available online are housed on national or regional association w ebsites or on w ebsites of state

- level entities. An all

- inclusive national d atabase of EM&V reports, plans, and/or market studies does not currently exist. The Consortium for Energy Efficiency ( CEE )

w ebsite appears to be the most complete resource for such reports, although its scope is limited to submissions from only its members

.

Entities providing online sources of EM&V plans, reports, and market studies are listed in
Table 2 , indicating the types of documents available from each. A complete summary of findings, with Web addresses, for all resources we researched is in Appendix C.



### Table 2 . Sources for EM&V Plans, Reports, and Market Studies

Scope/State Organization

Type Organization Name EM&V Plans EM&V

Reports Market

Studies
National Nonprofit Consortium for Energy Efficiency (CEE) √ √

Nonprofit American Council for an Energy

Efficient Economy (ACEEE) √

Regional NE Nonprofit Northeast Energy Efficiency Partnership s EM&V Forum (NEEP) √ √

Regional PNW Nonprofit Northwest Energy Efficiency Alliance (NEEA) √ √

Regional Public Or ganization/

Power Producer Bonneville Power Administration

(BPA) √ √

California Public University University of California

- California Institute for Energy and Environment ( UC CIEE ) √ √

Public Utilities Commission ( PUC ) California Public Utilities

Co mmission (CPUC) √ √ √

PUC California Measurement Advisory

Council (CALMAC) √ √

Connecticut State Utility

Partnership Connecticut Energy Efficiency

Fund √ √

Maine Program

Administrator Efficiency Maine √ √

Missouri Utility Ameren √ √

New York Thir d

- Party

Administrator New York State Energy R esearch

and Development Authority (NYSERDA) √ √

State Department New York Department of Public

Service (NY DPS) √ √

Oregon Third

- Party

Administrator Energy Trust of Oregon (ETO) √ √

Pennsylvania PUC Publi c Utility Commission √

Texas Other Electric Utility Marketing Managers of Texas √ √

PUC Texas PUC √ √June 2011 www.seeaction.energy.gov 5 

### Table 2 . Sources for EM&V Plans, Reports, and Market Studies

Scope/State Organization

Type Organization Name EM&V Plans EM&V

Reports Market

Studies Washington Utility Seattle City Light √ √

Wisconsin Utility Organization Wisconsin Focus on Energy √ √

Existing National or Regional EM&V Do cuments Databases

The national and regional websites that contain repositories of EM&V reports were those for CEE, Northeast Energy Efficiency Partnership s (NEEP) EM&V Forum, and the Northwest Energy Efficiency Alliance (NEEA). The availability of
these re ports varied across the existing databases. While all the databases we reviewed contained EM&V reports, EM&V plans were only available from three entities, consisting of the jurisdictions in California, Pennsylvania, and New York. This is primarily due to the fact that, in most cases, the plans are not required to be part of the public record, tend to change during the evaluation process, and are summarized as part of the final EM&V report.

Our expert interviewees confirmed issues that will likely limit th e availability of EM&V plans, including concerns about the proprietary nature of methods and techniques, particularly for site

- specific measurement and verification (M&V). Another issue is that even in cases where statewide EM&V protocols exist, specific e valuation methods tend to be described in the evaluation reports: the plans themselves may not be required. Nearly all of the sites we reviewed included some market studies. These national and regional sites varied with respect to salient characteristics, such as functionality and search capability.

States and Other Organizations’ EM&V Documents Databases

In our review of state utility commission and individual utility sources online, we found considerable inconsistency in the availability of EM&V reports and market studies. Approximately one

- third of state commissions and energy efficiency organizations provide these resources online. The databases ranged from very simple (consisting of links to reports and studies housed elsewhere) to extensive repositori es with search capabilities. State

- level databases varied in complexity, but most simply provided lists of reports with links to embedded documents. Several of the
websites we reviewed, specifically from the Energy Trust of Oregon (ETO) and the California Measurement Advisory
Council ( CALMAC), provided complex databases with search features. Most sources only posted reports for studies and evaluations conducted in their geographic areas (usually by state). Only the California Public Utilities Commission (CP UC) and the New York Department of Public Service (NY DPS) websites provided EM&V plans.

Two of the identified resources have been widely used as repositories for EM&V reports and other documents, and
both are also maintained: CALMAC’s searchable database and CEE’s Market Assessment and Program Evaluation (MAPE) Clearinghouse. As these resources could serve as a basis for a national database, we describe each in
further detail.

CALMAC

The most widely known and used EM&V document resource is CALMAC. In 2000 , CALMAC was started as a partnership between three participating utilities that wanted to share and make the documents they generated publicly available. CALMAC now consists of the three original utilities (San Diego Gas and Electric, Pacific Gas and
Elec tric, and Southern California Edison), as well as the California Energy Commission, and the CPUC. CALMAC’s online database contains over 1,000 documents related to energy efficiency, demand

- response, low

- income, and
self

- generation programs, with the first two categories having the most representation. Document categories included evaluation guidelines, program evaluations, and market studies. Documents were searchable, based onJune 2011 www.seeaction.energy.gov 6

text searches of key fields and document abstracts, and results could be filter ed by program type, category, author, sector, or date.

When initially populated, CALMAC posted over 500 documents to the site, dating back to 1990. Most of these were scans of original paper documents. Once CALMAC became well established and highly utilize d, the CPUC mandated submission of all California energy efficiency and demand

- response documents to the site.

The site’s technology remains essentially the same as when it was created. The search capabilities do not include
full text searches of documents . Rather, submissions must include abstracts containing specific information, including a summary of results and any relevant key words or phrases. CALMAC relies on document authors or sponsors to submit documents to the site, and does not actively gather new materials. Submissions are made via an online form with attached documents, and the CALMAC administrator screens submissions for completeness before final posting.

CEE MAPE

Another online document resource is the MAPE Clearinghouse

, administered by the CEE. CEE membership is open to organizations that have a regulatory or legislative mandate to administer energy efficiency programs as well as to other public stakeholders in such programs. Such administrators may be utilities

, or statewide and regional

o rganizations or agencies. MAPE contains over 1,000 documents, submitted for posting by CEE members. Documents are accessed by using a simple text search field. Results contain short document summaries and links to
PDFs of executive summaries and full docum ents. While currently publicly accessible, the CEE is working on a more sophisticated online database, which will be accessible to CEE members exclusively. The updated site will be built using the Drupal TM Content Management System (CMS) platform.



### Existin g State, Regional, and National TRM Resources

We identified 17 resources for measure savings values and algorithms, covering 21 states and the District of Columbia ( see Table 3 , identified as of February 2011 ). Th ese resources differ ed in coverage from state to region to national levels. Most of the resources we identified are administered by a state commission or agency. The remaining resources are administered by advisory committees, non

- profit organizations, tru sts, or public utilities. Many of these administrative groups contract out the research, development of calculations, and creation of the resources to consulting firms. Table 3 includes an indication of the measure savings information contained within
the resource: ex ante savings values or algorithms. For the resources that have ex ante savings estimates, a stipulated (deemed) value is given for the savings. For the resources primarily composed of algorithms, a for mula is given for where the input parameters are stipulated or determined based on specific conditions (e.g., a lookup table
for hours

- of

- use based on building type). For the resources that include both algorithms and ex ante savings values, the primary fo rmat for measure savings is algorithms in nearly all cases. Massachusetts is the exception, where algorithms and values are roughly equally represented. These resources primarily contained energy savings and measure costs; few include non

- energy benefits o r market transformation metrics.June 2011 www.seeaction.energy.gov 7 

### Table 3 . TRM Resources*

Scope Resource Name Format Information Included Administrator
National ENERGY STAR Online Calculators Ex ante savings

based on algorithms Agency

Regional

– Northwest Re gional Technical Forum (RTF) Deemed

Measures Online Database Ex ante savings

based on algorithms Advisory

Committee Regional

– Midatlantic Mid

- Atlantic TRM PDF Algorithms and Ex ante savings Non

- Profit

Organization
Arkansas Arkansas Deemed Savings Quick Start

Programs PDF Algorithms Public Utility

California Database for Energy Efficient Resources

(DEER) Software Program Ex ante savings State Commission

Connecticut Connecticut Light & Power and United

Illuminating Company Program Savings Documentation P DF Algorithms and Ex ante savings Public Utility

Hawaii Hawaii Energy Efficiency Program TRM PDF Algorithms and Ex ante savings State Commission

Maine Efficiency Maine TRM PDF Algorithms and Ex ante savings Trust

Massachusetts Massachusetts Statewide TRM for

Estimating Savings from Energy Efficiency Measures PDF Algorithms and Ex ante savings Agency

Michigan Michigan Energy Measures Database Excel Database Ex ante savings State Commission

New Jersey New Jersey Clean Energy Program

Protocols to Meas ure Resource Savings PDF Algorithms and Ex ante savings Agency

New York New York Standard Approach for

Estimating Energy Savings from Energy Efficiency Programs PDF Algorithms and Ex ante savings Agency

Ohio Ohio TRM Online Database Algorithms and

Ex ante savings State Commission

Pennsylvania Pennsylvania TRM DOC Algorithms and Ex ante savings State Commission

Texas Deemed Savings, Installation, and

Efficiency Standards PDF Algorithms and Ex ante savings State Commission

Vermont Efficiency Vermont Technical Reference

User Manual PDF Algorithms and Ex ante savings Non

- Profit

Organization
Wisconsin Focus on Energy Evaluation Business

Programs: Deemed Savings Manual V1.0 PDF Algorithms and Ex ante savings State Commission

* These resources were ob tained in February 2011. Any u pdates and amendments since that date were not reviewed.

The only nationally applicable resource, available through the ENERGY STAR program, contains measure

- specific savings calculators in Microsoft Excel ® . These calculators are the only interactive resource available, allowing user modification of certain parameters, such as usage hours, number of units, territorial region, equipment size, and fuel type. Only measures qualifying as ENERGY STAR have calculators; however, not a ll ENERGY STAR measures currently have calculators.

At the regional level, we reviewed two resources in depth: the Northwest RTF’s online tool and database and the
Mid

- Atlantic TRM. These regional resources are used by the states within each region. The No rthwest RTF’s savings
estimates are used in Idaho, Montana, Oregon, and Washington. The Mid

- Atlantic TRM is used in Delaware, Maryland, and the District of Columbia.June 2011 www.seeaction.energy.gov 8

The remaining 14 resources listed in Table 3 ar e classified as state resources. They typically address areas with less diversity in climate, baseline conditions, and structural characteristics. Most of these resources are available as
electronic documents in PDF or Microsoft Word ® formats. The remainin g state resources are Microsoft Excel workbooks or custom programs (such as California’s DEER), requiring the user to download specialized software to
access the database.

Additional TRM and Related Resources

Three additional resources not listed in Table 3 are available: the National Renewable Energy Laboratory (NREL) National Residential Efficiency Measures Database; conservation potential assessment (CPA) reports, including the
Northwest Power and Conservation C ouncil’s (Council’s) 6 th Power Plan; and the DOE Buildings Performance
Database.

The format of the NREL database is useful for an online measure and cost database. This resource’s focus is to provide cost information for a wide range of residential measur es; savings information is sparse. Consequently, we did not include this database in our detailed, measure

- level analysis.

CPA studies are an additional source of detailed information on measure costs and savings. However, we did not
include CPAs in this study for several reasons. First, detailed, measure

- level information is not always provided in publicly available potential study reports. Second, detailed data, where available, can be difficult to extract in a useful format, since CPAs tend to be in PDF format, which have no formulas for calculations. Even in rare cases where the detailed data are available in a spreadsheet format, the data tend to be hard

- coded values or protected, and assumptions in the calculations are not known. Moreover, different C PAs report savings in different units (e.g., per square foot or per unit of measure), making it difficult to compare across studies. The exception is the conservation potential analysis of the Council’s 6 th Power Plan, a regional study and a common point o f reference for the development of potential studies throughout the Northwest. This analysis has relatively transparent and readily available calculations of savings, based primarily on data available from the Northwest RTF.

Finally, t he National Laborator y Collaborative on Building Technologies (NLCBT) , recently launched by DOE, is developing the DOE Buildings Performance Database

. According to information provided by the DOE project personnel 2

, th is data warehouse is designed to store building description s, including facility characteristics, such as size, functional use, equipment type, location (metadata), and time

- series energy consumption data ( not saving
values ). The DOE Buildings Performance Database will also provide a layer of “tools” for analysis. A data specification has been developed to standardize the type of data in the warehouse. It is expected that this specification will also support other DOE building data initiatives. Initial data are coming from government buildings
or federal stimulus f unded projects (residential and commercial) for which the NLCBT can access consumption
records. Consumption data may be available in monthly, hourly or intervals as short as every 15 minutes

. The “tool” layer will initially consist of a savings tool, desig ned to compute differences in consumption between user

- selected data points, and a financial analysis tool designed to show the distribution of cash flows arising from those savings
results.

NLCBT plans to expand the type of data in the data layer, and to provide an Application Program Interface tool , with
which users may access and analyze the data. Thus, although the DOE Buildings Performance Database may not serve directly as a source for measure

- specific information, one can use the data of available c onsumption and facility characteristics in conjunction with various statistical and financial analytic techniques to develop estimates
of measure

- level savings or cost

- effectiveness. As the amount of data and range of tools increase, the DOE Buildings
Perf ormance Database may provide an alternative to estimating savings, based on measured energy usage as compared to the traditional, calculation

- based approaches now available.

2 Personnel communication with Dr. William Miller, Senior Advisor, Sentech, Inc.June 2011 www.seeaction.energy.gov 9

TRM Review

To inform the feasibility of creating a national DS&A database, Cadmus compared measure savings values and algorithms across the resources listed above. We reviewed and compared 20 measures

— 10 in the residential and
10 in the commercial sector

— and included both electric and gas impacts across the 17 identified resources. 3 We began our process for selecting the 20 measures by categorizing measures available in the databases by end use, fuel type, and sector, then ranked them according to three criteria, as described below.

End Use

We used the following end

- use categories to gr oup measures for this scoping study:

 Appliances  Motor/Pump

 Building Package  New Construction

 Cooking  Other

 Envelope  Plug Load

 HVAC  Refrigeration

 Industrial  Ventilation

 Lighting  Water Heating

Fuel Type

Electricity and natural gas were the prim ary fuel types we considered. Gas measures typically reported savings in MMBtu and, less commonly, in therms. Natural gas is usually the assumed fuel; however, for a small number of measures, oil or propane was included. The majority of gas measures produc e space

- or water

- heating savings.

Measures in the electric category typically reported savings in kilowatt hours (kWh). Energy savings from electric measures are mainly attributable to electricity savings, but they may produce gas savings as well. For ex ample, an efficient dishwasher in a home with a gas hot water heater may save electricity and gas.

Measures typically thought of as electric

- only may have small but quantifiable effects on gas consumption through interactive effects. For example, ineffici ent lights, such as incandescent light bulbs, waste energy through heat generation. That heat affects ambient temperatures, leading to increased conditioned

- space cooling loads and decreased heating loads. When more efficient and cooler lights are retrofit ted into spaces, secondary energy savings result from less need for cooling, but heating energy consumption may increase due to the reduced waste heat emanating from the lighting.

Fuel

- switching (or conversion) measures can have a large effect on both ele ctricity and gas usage. For example, switching from a conventional electric water heater to a high

- efficiency gas water heater shifts the energy load from the electric grid to natural gas suppliers. We did not consider fuel

- switching measures in this revie w and comparison of measures due to their complexity and because regulatory questions exist regarding their appropriateness as an energy efficiency measure in many jurisdictions.

Sector

Ex ante savings values and algorithms for measures may also be categor ized by the market sector(s) to which they apply, such as residential, commercial, or industrial. In some cases, the resources identified the applicable sector(s);

3 This review was also informed by a recent KEMA report: Regional EM&V Forum Report: Common EM&V Methods and Savings Assumptions Project , May 2010.June 2011 www.seeaction.energy.gov 10

however, for several of the resources we reviewed, it was not always apparent to which secto r(s) a measure’s values or algorithms applied. This ambiguity could sometimes be resolved by examining algorithm inputs such as
hours

- of

- use and total operating hours.

Measures are most commonly reported for the residential or commercial sector. Some reso urces identified a few measures as being applicable to commercial and industrial sectors, such as air compressors and pumps. Most non

lighting measures for the industrial sector (such as process improvements) were custom, and deemed or calculation algorith m procedures are not suitable for estimating their savings.

R anking M easures

We began with 31 measures in the ranking

— 14 residential and 17 commercial. These measures were commonly listed in the sources we reviewed, and presented a broad selection of end uses while covering both fuel types and both sectors. We ranked these measures based on the following criteria to determine their suitability for further review and analysis:

 Savings impacts: We gave measures a higher score if they had the potential to pro duce larger savings. We ranked impacts based on our experience with conservation potential studies on a scale from 1 to 5, with 1 producing very small savings and 5 producing very large savings.

 Prevalence: We scored measures for prevalence based on the nu mber of TRMs, protocols, or databases in which the measure appeared. We ranked prevalence on a scale from 1 to 5, with 1 being not at all prevalent and 5 being very prevalent.

 Diversity of calculation methodology: We gave measures with multiple methods fo r calculating savings a higher score to allow alternative methods to be represented. We ranked diversity of calculation methodologies according to the following scoring scheme:

o 1 = Differing units (e.g., lighting savings per house versus per bulb)

o 2 = Dif fering in two or more adjustment factors

o 3 = Differing treatments for interactive effects

o 4 = Differing treatments for interactive effects and other adjustment factors

o 5 = Completely different approaches used (e.g., deemed vs. equation).

 Fuel diversity: We assigned a value of 1 to measures that produce both gas and electricity savings, and assigned a value of 0 to measures that affect only one fuel source. For example, we gave measures with an impact on heating loads a score of 1 because they could potentia lly reduce gas or electricity consumption, depending on the heating fuel type.

 Sector diversity: We based the sector diversity score on whether a measure applied to one or multiple sectors. We assigned a value of 1 to measures found in both commercial and residential sectors, and assigned a value of 0 to measures only found in one sector.

We further weighted the assigned values by importance on a scale of 1 to 5, as shown in Table 4 . We then added the five weighted rank values to determine the overall ranking of each measure. The 20 highest

- ranking measures in
the residential and commercial sectors are listed in Table 5

.June 2011 www.seeaction.energy.gov 11 Table 4 . Results of Measure Ra nking Process

Rank Measure Fuel* Combination of Weighted Factors Impact Prevalence Diversity Calculation Methodology Fuel Diversity Sector Diversity

Weighting Factor 5 4 3 2 1
Residential 1 Lighting (CFLs) Electric 61 5 5 5 0 1
2 Refrigerator Rec ycling or Retirement Electric 56 5 4 5 0 0

3 HE Furnace Gas 49 5 2 5 0 1
4 Low

- Flow Showerhead Either 46 5 3 2 1 1
5 Ceiling/Attic Insulation Both 45 3 3 5 1 1
6 HE Air

- Source Heat Pump Electric 42 4 3 3 0 1
7 ENERGY STAR ® Refrigerator Electric 4 0 2 5 3 0 1

7 HE DWH Storage Either 40 2 3 5 1 1
7 Furnace Motor ECM Electric 38 3 2 5 0 0
10 HE Clothes Washer Both 37 1 5 3 1 1
11 HE Room AC Electric 36 2 4 3 0 1
12 Lighting Controls Electric 25 1 1 5 0 1
13 ENERGY STAR Ceiling Fan Electric 22 1 2 3 0 0
Commercial 1 Lighting (linear fluorescents) Electric 55 5 5 3 0 1
2 HE Furnace Gas 53 5 3 5 0 1
3 HE DWH Storage Either 45 3 3 5 1 1
4 Lighting Controls Electric 43 4 4 2 0 1
5 HE Motor Electric 41 3 5 2 0 0
6 HE Air

- Source Heat Pu mp Electric 37 3 3 3 0 1
6 Roof Insulation Both 37 1 3 5 5 5
8 HE Clothes Washer Both 35 2 3 5 5 3
8 Air Compressor Electric 35 3 4 0 0 1
10 Refrigeration ECM Electric 33 3 3 0 0 2
11 ENERGY STAR Refrigerator Electric 32 3 2 5 0 3
11 HE Cooking Gas 32 2 3 0 0 3
13 HE Room AC Electric 28 2 2 5 0 3
14 Vending Machine Electric 27 2 2 0 0 3
15 Traffic Signals Electric 26 2 3 0 0 1
16 Low

- Flow Showerhead Either 20 1 2 5 5 1 * Fuel type represents primary fuel saved. “Both” indicates a measur e that can save heating and cooling usage, and thus could save both gas and electric fuels. “Either” indicates the fuel source for the measure could be electric or gas, so the savings could be electric or gas.June 2011 www.seeaction.energy.gov 12 

### Table 5 . List of 10 M ea sures for Analysis and Review, Each in the Residential and
Commercial Sectors

Residential Sector Measures Fuel Type

1 Light ing

– CFL s Electric
2 Refrigerator Recycling / Retirement Electric
3 HE Furnace Gas
4 Low

- Flow Showerhead Either
5 Ceiling/A ttic Insulation Both
6 HE A ir

- S ource H eat P ump Electric
7 E NERGY S TAR Refrigerator Electric
8 HE D W H Storage Either
9 Furnace M otor ECM Electric
10 HE Clothes Washer Both



### Commercial Sector Measures Fuel Type

1 Light ing

– Linear Fluorescent s Electric
2 HE Furnace Gas
3 HE DWH Storage Either
4 Lighting Controls Electric
5 HE Motor Electric
6 HE A ir

- S ource H eat Pump Electric
7 Roof Insulation Both
8 HE Clothes Washer Both
9 Air Compressor Electric
10 Refrigeration ECM Electric



### Measure Review

O ur detailed review of the 20 measures across various TRMs resulted in three general themes: consistency in methods and assumptions, transparency and clarity, and accuracy. We compared the methods for savings
calculations and the results for each of the 20 measures across five (at most) TRMs. We also included in our review an indication if data around measure costs or secondary benefits were provided.

The results of our review indicated a wide variation in methodologies for estimating savings among the 17 TR Ms. There were exceptions where TRMs were relatively consistent in their algorithms and input assumptions, such as for high

- efficiency (HE) motors and commercial air compressors. For the most part, however, there was little consistency in which variables a nd other inputs were used. Some TRMs, for example, included waste heat factors,

in

- service rates, and partial load factors, while other TRMs did not. Given that we weighted our selected measures based on the diversity of savings, this is perhaps not wholly surprising. However, the wide range of methodologies is
still notable.

Methodological approaches also varied among TRMs. For example, the roof insulation measure in some TRMs based savings on building energy simulations, while others based savings on eng ineering algorithms. Even TRMs with similar algorithm methodologies used different input assumptions, such as variable names (e.g., Btu/hr vs. tons) and baseline assumptions (e.g., federal code vs. current practice). The magnitude of these impacts can vary based on these assumptions; for example, the variable names or units should not change the fundamental approach, while the baseline assumptions can significantly impact the savings estimates. These inconsistencies have implications for creating standardiz ed approaches in a national database.June 2011 www.seeaction.energy.gov 13

The transparency and level of detail regarding methods and assumptions ranged from being well

- sourced to being minimal. None of the TRMs referenced all their sources completely. In several cases, it was not clear what the input assumptions were based on, especially with fully deemed values. In other cases, input assumptions were not clearly defined, making it difficult to determine the appropriateness of different values used in the algorithm. TRMs with an adequate lev el of transparency provided multiple references and outlined the intermediate steps showing how the
calculations were made and the sources for the data.

Our review indicated that algorithms used in savings calculations were generally correct and based on accepted methodologies. However, our review also showed considerable errors, ranging from obvious typographical errors, to missing default values for certain assumptions, to minor calculation errors in a few cases. In several cases, data
tables were mislab eled and footnotes and references were incorrectly numbered. In several cases, variable names were missing entirely.

Tables 6 and 7 summarize our findings and recommendations for each of the 20 measures we reviewed as part of the measure comparison. We pr ovided the range in gross energy savings where possible, and the key factors contributing to the spread. For those measures that can provide gas and/or electric savings, the savings spread is provided in the most common unit across the TRMs reviewed. For e xample, if three of the five TRMs represent savings for high

- efficiency water heaters in kWh, and the remaining two provide savings in Btus or therms, the spread from only those three TRMs is provided. Details are provided in Appendices A and B for the res idential and commercial measures, respectively.June 2011 www.seeaction.energy.gov 14 

### Table 6 . Residential Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Spread Main Findings and Recommendations

CFL 27

– 49 kWh Findings : Lighting is a well

- studied measure subject to many adjustment factors, depending on installation location (general house vs. living room), application (interior vs. exterior), HVAC system, and delivery mechanism (retail, direct install, socket count , etc. ) . Storage and removal factors vary and must be obtained through regional studies. Additional variation arises from hours

- of

- use assumptions. Recommendation : Residential lighting on a single fixture level is a relatively straightforward measure that can be included in a national DS&A database with a supporting wattage table. However, inclusion of the aforementioned adjustment factors will be challenging. Refrigerator Early Replacement 724

– 1

, 728 kWh

(gross) Findings : This measure relies on deemed savings obtained from studies. The average existing refrigerator energy consumption can

also be determined by accessing the appliance recycling contractor database. Variations in the calculation of deemed values include accounting for degradation over time, lab

- t o

- field true

- up factors, replacement rate, usage hours, and energy usage of replacement appliance. NTG factors were included in a few of the sources we reviewed. Recommendation : Refrigerator recycling/retirement can be adapted into a national DS&A database relatively easily. Measure savings should be based on evaluation studies that account for the energy consumption of the existing residential refrigerator stock. However, it may be challenging to come to an agreement about which study to base this informat ion on and what factors to include. Furnace 5.54

– 12.7 MMB tu Findings : Most savings are calculated based on the hours

- of

- use; however, variations include the annual heating load by vintage and floor area. These inputs vary regionally and even within the same state. Another approach is based on building simulations. Energy savings units were most commonly in MMB tu ; however, one TRM assumed the use of an ECM furnace fan, and yielding electricity savings. Recommendation : As the algorithm is generally quite uniform, it can be adapted into a national DS&A database . As a weather

sensitive measure, the usage would vary by location and application, making the development of reference tables potentially challenging. Showerhead 130

– 583 kWh Findings : E lectric an d gas savings follow similar algorithms. At the most general level, the prevailing savings algorithm breaks down into two parts: the water savings and savings from the energy required to heat that amount of water. A nnual water savings are calculated in two primary ways : the first is based on the number of showers per day, the second approach is on based on the number of people per household and the average number of showers per household. The energy that would have been required to heat the amount of water saved is the most straightforward approach

, and is based on the volumetric heat capacity of water and the temperature delta. Recommendation : Showerheads c ould be included in a standardized national DS&A database . The measure is ideally accompanied by regi onally tabulated values for the water temperature delta and usage. The efficiency of the water heater (by fuel type) should b e well defined in the TRM, either on a national or regional basis. Roof Insulation Not comparable Findings : Energy savings are ca lculated on a square

- foot basis, either through use of simulation software or algorithms. The unit savings are then scaled up through the effective area of increased insulation for each project. We found m ultiple algorithms that varied by HVAC system type and fuel type. The baseline and efficient conditions var y based on R

- values. Recommendation : This measure would be challenging to standardize

, and savings are best determined through building simulation modeling for regional building stocks.June 2011 www.seeaction.energy.gov 15 

### Table 6 . Residential Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Spread Main Findings and Recommendations

Air

- Source He at Pump 164

– 951 kWh Findings : The sources we reviewed used both calculations and energy modeling to determine savings. Baseline conditions are often based on energy code requirements, with the efficient condition defined by the nameplate efficiency excee ding the baseline. Variations arise due to the efficiency increment between baseline and efficient condition , as well as by hours

- of

- use assumptions. Recommendation : Heat pumps are an important measure in many utility programs and, given the relatively lo w variation in calculation approaches and the well

- defined basel ine and efficient conditions, c ould be included in a national DS&A database . The challenge will be to develop weather

- sensitive hours

- of

- use by climate zone, building type, and other influenti al characteristics. Storage Water Heater 99

– 347 kWh Findings : Energy savings are generally calculated using a variation of the prevailing algorithm, and using the volume of hot water consumed (gallons per day), the temperature differential of the water in vs. out of the water heater, and the equipment efficiency. The only assumptions for this measure that may be regional are the water main temperature, which varies by city, and the ambient room temperature. The baseline and efficient energy factor (EF) v alues depend on fuel type. Recommendation : Considering the minimal weather sensitivity of water heaters (except for the potential HVAC interaction and variations in water main temperatures) and the algorithm’s ease of use, efficient water heaters are a go od candidate for inclusion in a national DS&A database . Separate algorithms are recommended for gas and electric water heaters. National or regional average water usage will need to be tabulated. ECM for Furnace Fan 419

– 733 kWh Findings : Energy savings are deemed on a per

- furnace basis, with values originating from various technical studies. Some sources use a heating degree day adjustment factor

, while others use regional variations in AFUE or hours

- of

- use based on studies. Recommendation : Adapting th e residential ECM furnace fan measure into a national DS&A database could be intensive, as it would be with other weather

- sensitive measures. The difficulty in adapting reported savings for use in a national DS&A database would be accommodating differences in building configuration and weather impacts. Some of the variation in this measure can be accommodated by load hour or degree day adjustments for different regions. Savings should be broken out by state or climate z one, by fuel type, and by furnaces wit h and without a central air conditioner. ENERGY STAR Refrigerator 45

– 106 kWh Findings : Savings are deemed, per refrigerator, based on appliance characteristics ( e.g., ice through door, freezer configuration, freezer and refrigerator volume, efficiency level such as CEE vs. ENERGY STAR). Two primary approaches were used to determine savings : (1) define maximum consumption limits for baseline and efficient appliances ; and (2) use average region

- specific appliance data. Some sources include an HVAC adjustm ent factor to account for refrigerators in a conditioned space interacting with the HVAC system. Recommendation : This measure is ideal for inclusion in a national DS&A database as a set of deemed savings by appliance configuration and efficiency level.June 2011 www.seeaction.energy.gov 16 

### Table 6 . Residential Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Spread Main Findings and Recommendations

C lothes Washer 127

– 258 kWh Findings : Deemed savings are tabulated by domestic water heating (DWH) and dryer fuel, and efficiency level (CEE vs. ENERGY STAR). The clothes washer is not a standalone measure. Savings from DWH and from the dryer (due to less moisture in the clothes) are usually implicit. One of the sources calculates weighted savings values over the entire fuel mix

. Recommendation : Residential clothes washers are ideal for inclusion in a national DS&A database as a set of deemed savings by DW H and dryer fuel type, as well as by efficiency level. This will provide deemed savings based on easily observable attributes. Clothes washers in multifamily settings may require separate calculations to account for in

- unit washers and washers in a laundry center (common area). S avings for all fuel type s should be claimed for each unit in a national DS&A database

.



### Table 7 . Commercial Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Spread Main Findings and Recommendations

Indoor Lighting 34

– 80* kWh Findings : The savings calculation requires the user to supply actual values or use the tabulated wattages, hours

- of

- use, HVAC interaction factor, and quantity. Many TRMs contained reference w attage tables; however, the s e tables were inconsisten t . Variations in variable definitions and other input values contribute to the spread in the savings value. Recommendation : Lighting is currently a significant component of utility programs nationwide an d, on a single fixture replacement level, is a good measure for the national DS&A database . Developing a universal/standard wattage table will be challeng ing

. Furnace 1.1

– 30 MMB tu Findings : Most of the calculations rely on the heating full load hours; however, another approach uses heating degree days. Another variation is the assumption of whether an ECM furnace fan motor is or is not installed. The greatest contribution to the meas ure savings spread appears to be from the variation in usage. Most bas eline efficiencies are based on the federal standard of 0.78 AFUE. Recommendation : Although weather sensitive, this measure is recommended for inclusion in a national DS&A database because of the relatively small variation in the calculation methodologies observed and the small number of inputs. The biggest challenge will be in developing full load hours based on location and application (building types) across the nation. Storage Water Heater 1.5

– 7.6 MMB tu Findings : As with many other measures, the sav ings calculation approach multiplies a percent of savings (based on equipment efficiency) by the normal consumption. Differing approaches are used to determine the annual hot water consumption; additiona l differences in the calculation depend on tank size and fuel types. Recommendation : Accommodating this measure in a national DS&A database would be best achieved by breaking out the savings calculation by fuel type and tank size. Since some units are rated by their energy factor and others by thermal effic iency, different algorithms will be need ed to appropriately determine savings. The annual hot water consumption should be provided based on important characteristics such as facility type, size, occupancy, etc. Lighting Controls 180

– 535 kWh Findings : S avings are calculated on a per controlled load basis

, similar to the indoor lighting measure. The algorithm contains anJune 2011 www.seeaction.energy.gov 17 

### Table 7 . Commercial Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Spread Main Findings and Recommendations

energy savings factor that is similar to the percent savings and is based on a certain type of control (occupancy sensor, tim e clock, et c). Variations in the algorithm include use of the in

- service rate adjustment factor and the HVAC interaction factor. Recommendation : This measure is ideal for inclusion in a national DS&A database because all methodologies follow mathematically similar f ormulas. Reference tables are needed to cover a range of control types and referenced operating hours.
Motor (1

- 200 HP) 387

– 662 kWh Findings : Per motor savings are calculated using baseline and efficient conditions

, as defined by federal standards (bas ed on motor type and HP), with the main variability attributed to hours

- of

- use and the rated load factor. Recommendation : Motors are ideal for inclusion in a national DS&A database because there is little variability in the algorithm used to calculate ener gy savings. A standardized algorithm should also come with reference tables of baseline efficiencies, operating hours

, and rated load factors, as well as instructions on how to pick an appropriate input value. Developing a universal/standard op erating hour s table may be challeng ing, and this challenge is the main cause for the range of savings in the TRMs reviewed.June 2011 www.seeaction.energy.gov 18 

### Table 8 . Commercial Measure Comparison Findings and Recommendations Summary

Measure Gross Annual

Energy Savings Sp read Main Findings and Recommendations

Air

- Source Heat Pump 109

– 3,031 kWh Findings : The TRMs we reviewed used both calculations and energy modeling to determine savings. The range in savings is due to variations in hours

- of

- use assumptions, climate zone , vintage, and/or building type. Recommendation : Heat pumps are an important measure in many utility programs and, given the relatively low variati on in calculation approaches, c ould be included in a national DS&A database . The challenge will be to develo p weather

- sensitive hours

- of

use by climate zone, building type, and other influential characteristics. Roof Insulation Not comparable Findings : Savings are commonly developed from outputs of building simulations and are tabulated by a host of characteri stics. Some TRMs used a n engineering calculation. Measure baselines and efficiency requirements vary regionally. Recommendation : This measure would be difficult to standardize and is not recommended for initial inclu sion in a national DS&A database, which would require capturing local building practices, building stocks, code requirements, and other influences. Clothes Washer 224

– 921 kWh Findings : Savings from a clothes washer also include domestic water heating (DWH) and dryer savings (due to less moist ure in the clothes). Energy savings per load are tabulated based on the water heater and dryer fuel(s) ( e.g., gas, electric, line dry). The annual clothes washer usage is multiplied by the savings per load to calculate the annual savings. Variation in the number of loads per year ( from 271 to 2,190) is the strongest contributor to the spread in energy savings. Recommendation : Commercial clothes washers are a good candidate for the national DS&A database since many of the calculation approaches within the T RMs are similar. This measure should be broken out by the water heater and dryer fuel type(s) . Default values for different use scenarios would take additional resources to develop. Air Compressor 11

– 23 kWh Findings : Energy savings are calculated per c ompressor on a retrofit basis. The baseline condition is typically a modulating air compressor with blow down

, and the efficient condition varied by equipment control type. Most commonly, the control type is a variable speed drive. All algorithms depend on an energy savings factor that is provided in the supporting measure documentation. Variability arises primarily from these tabulated energy saving factors. Recommendation : The small variation in air compressor energy savings methodologies makes it an idea l measure for inclusion in a national DS&A database . To properly support this measure, a table of operating hours will be needed, possibly broken down by usage scenario (e.g.

, 3

- shift)

. ECM for Refrigeration 264

– 2,033 kWh Findings : Savings approaches v ary from using deemed energy savings values to employing calculations requiring multiple inputs. Multiple calculations are provided based on application ( e.g., freezer vs. cooler, walk

- in vs. other systems). Most sources account for the refrigeration syste m efficiency, while one provides deemed energy savings per rated equipment watt, which is then scaled by the system wattage. Recommendation : This measure can be included in a national DS&A database, but will first need to be broken out by application, mot or size, and equipment configuration (ECM or ECM with controller; and PSC or shaded

- pole).

* Original value was 239 kWh for a three

- lamp fixture; we divided by three to calculate a one

- lamp fixture equivalen t.Month Year www.seeaction.e nergy.gov 19



## Findings

This section presents the results of our scoping study findings for the EM&V documents and National (or regional) Deemed Savings and Algorithm database in terms of: the needs they might address, their desirable features and functionality, their feasibility in light of issues affecting thei r development and maintenance, and costs.



### EM&V Documents Database

Needs

Experts in the industry and those newer to the energy efficiency field expressed an overwhelming need for a national database of EM&V resources. Nearly every interviewee agreed that a national database of EM&V reports, plans, market studies, and other relevant documents would be a very useful resource. One respondent commented that a centralized resource providing such documents is needed, but cautioned that a database might not be the
best format.

Interviewees cited several benefits of having a national EM&V documents database, particularly savings in the time and effort required to locate useful materials. Although many such EM&V reports and studies are publicly available, searching and compiling relevant materials would be time

- consuming, as they are currently spread across several disparate sources. Being able to access EM&V documents from across the country in a single database could greatly reduce the time required to locate relev ant sources, while increasing the availability of quality information and most importantly case studies/examples

— which are perhaps the most common request of those new to the EM&V field. This is important for those new to energy efficiency

— seven of the 19 industry professionals we interviewed indicated that a national database would help nascent states begin program planning and evaluation.

Interviewees also indicated that a national EM&V documents database could fill the need for greater transparency
and c onsistency than is currently possible from the existing resources. They also stated that a national EM&V documents database could greatly facilitate comparisons across the country, moving the evaluation community towards a more formal data sharing process.

Feasibility

Although respondents reported that a national database of EM&V documents has benefits

, they raised several

issues that would have to be resolved before a valuable resource, providing benefits beyond those currently available, could be develop ed. These concerns relate to the feasibility of the database itself and included development of the database and maintenance infrastructure, procedures for cataloguing documents, and development and maintenance costs.

Interviewees expressed concerns about the complexities of database development and maintenance, especially with respect to the tasks of gathering and properly cataloguing existing EM&V reports and resources

. As noted, the EM&V plans, reports, and market studies that are available online are s pread across multiple w ebsites , and many more documents are not available online. Respondents noted that t he initial push to gather these documents would be a massive undertaking, but they reported that overall it would be worth the time and effort.

Ensur ing c onsistent categorization and tagging of compiled documents was another concern related to the feasibility of such a database. Several interviewees indicated that an individual with a library sciences background would be required for development and on going maintenance. In addition, depending on the final product, it might be necessary to develop data formatting and reporting requirements to maintain consistency, especially if the database permitted comparisons across the country

.J une 2011 www.seeaction.energy.gov 20

The ability to acquir e EM&V reports and plans , market studies, and other relevant documents present ed another feasibility issue, appl ying to initial development stages and to continued database up keep. After the initial task of compiling existing resources, stakeholder s would have to decide whether to mandate document submittal or operate on a voluntary basis. Industry professionals noted that while persuading people or organizations to voluntarily submit documents could be a challenge, mandatory submission could prove burdenso me and difficult to implement , especially if individuals and organizations were required to submit documents to more than one group.

The costs to create and maintain a database on this scale could be large, as several interviewees noted. However,

no one ex pressed a concern that costs alone should prevent the project from moving forward. Although these feasibility issues should be considered before moving forward , they did not appear to render a national database of
EM&V resources unachievable.

Content and Functionality

Interviewees indicated a wide range of document types that should be considered for inclusion in the database, including process and impact evaluation plans and reports, market studies, and other relevant studies. A diverse set
of functionali ty options was also identified by the respondents. To create this type of database, up

- front decisions would need to be made about the types of documents to include, and an appropriate database structure would need to be determined to present documents in a readily accessible manner.

The original scope of this study envisioned a database comprised of EM&V plans, EM&V reports, and market characterization studies. EM&V plans can include several different types of documents; a recent study suggests four
types : EM&V frameworks, portfolio plans, detailed research plans, and site

- specific evaluation plans. 4 Including all four of these types of documents would be necessary to provide a comprehensive resource. However, as noted earlier, evaluation plans are not rea dily available publicly. Three of the five interviewees that provided input on this subject expressed concerns about proprietary information barriers, which would make publishing difficult. Although many respondents said that including EM&V plans would be useful, their reservations about the availability of such documents suggested that it might not be feasible to include them. Often, EM&V reports include a detailed description of the evaluation research plan; therefore, including the original plans may not be critically important.

After interviewing industry experts and representatives from states with newly developing energy efficiency initiatives, it became clear that expanding the contents to include the following additional types of documents and

suppo rting data would be useful:

 Common evaluation methodologies (e.g., billing analysis, metering, modeling)

 Evaluation framework documents and protocols

 Technical/white papers

 Documents relating to market transformation or other resource acquisition strategie s

 Example s of requests for proposals (RFPs)

 Links to regulatory filings and orders on energy

- efficiency matters

 Dictionary/glossary of industry terms

 Raw data used in evaluations.

4 Schiller, Goldman, and Galawish

. National Energy Efficiency Evaluation, Measurement and Verification (EM&V) Standard: Scoping Study of Issues and Implementation Requirements

, LBNL

- 4265E. April 2011.J une 2011 www.seeaction.energy.gov 21

Several interviewees noted that many of the listed documents would need to be accompanied by sufficient explanation and context, including program and regulatory background information. Interviewees also recognized that raw data, though highly desirable, would be difficult to obtain due to confidentiality concerns. However, these types of documents currently have limited public availability; so an active search and selection process would need to happen before including them in a national database.

The respondents generally reported that the needs of professionals new to the indus try should be given priority in determining the scope and coverage of the EM&V documents database. Development of a resource that provides value to more experienced industry professionals poses additional challenges. Respondents were evenly split (six
and six) about whether the database should be a comprehensive repository of all known EM&V reports and market studies, or be a representative sample based on particular screening criteria. The remaining interviewees were undecided or did not provide input on t his topic.

Those respondents leaning toward a comprehensive database suggested that instead of limiting the number of reports, the database should include filters, descriptors, or flags to guide users new to energy efficiency. They said
this would make the database less overwhelming, while providing value for more experienced users.

Interviewees that indicated interest in having a selective database made several suggestions about how rigorous the screening process should be. These ranged from a simple chec k to determine whether the document had been accepted or formally approved by regulators, to a requirement that they be subjected to stringent review and ranking processes by experts, based on predetermined criteria or their expert judgment.

Interviewees a lso noted that, while the concept of a database designed to inform users of best practices was attractive, it presents several additional feasibility issues. As several interviewees noted, screening reports and market studies based on best practices criter ia could be problematic, given that best practices are a relative concept that depend on the goals and budget of a given program and/or evaluation. The experience or budget of nascent states may not support employing those best practices that would be defi ned by more experienced states, such as California. If the database were limited to best practices most effective in experienced states, nascent states might not be able to identify studies or methods most suitable for their specific situations. As a possi ble resolution of these potentially conflicting needs, interviewees suggested incorporating a tiered approach for identifying best practices in the database to achieve a better balance of rigor and cost, allowing entities to choose approaches fitting their specific situation s and regulatory environment s from a larger pool of approaches

.

A frequent suggestion from interviewees was to create a database that serves as a “starter kit” for states new to energy efficiency, guiding them toward developing energy e fficiency initiatives or complying with energy efficiency standards or requirements. This starter kit would provide sample documents and guide the user through their applicability and relevance.

Regardless of whether the database is comprehensive or limite d to a screened sample of documents

, the databases’ search capabilities will likely be one of its most important function s . Interviewees made a number of suggestions for search or filtering capabilities, including the following:

 Key word (s)

 Program impleme ntation type and goal (e.g., direct install, point of sale rebates, financing programs, market transformation, emerging technology)

 Programs market (e.g., commercial, residential, low income)

 Technologies and measures (e.g., CFLs, LEDs, HVAC)

 Sample size

 E valuation history (whether similar studies have been completed)J une 2011 www.seeaction.energy.gov 22  Date (years) of EM&V activity

 Methodological approach (billing analysis, metering, etc.)

 Region

 Fuel type

 Complexity of study.

Finally, interviewees mentioned that a help desk feature would b e an extremely valuable function of the database

, especially for those new to energy efficiency. As one respondent noted, much quality control and decision making
occurs behind the scenes

, and having an option to talk to individual s making these decisions ( or someone knowledgeable about the decisions ) would be very useful. Having a common answer source for q uestions aris ing as people utilize information in the database would be beneficial for improving consistency and decreasing the chances of data being ap plied incorrectly or out

- of

- context.



### National (or Regional) Deemed Savings and Algorithm Database

Needs

Respondents by and large agreed that a national DS&A database (or at least multiple, regional databases) would be useful, but their opinions differed w idely on the optimal structure, content, and functionalities of such a resource. For example, a measures database was less of a priority for respondents in regions such as the Northwest, where such a database already exists.

The majority of respondents re ported that a national DS&A database would help states and utilities new to the industry with planning, implementing, and evaluating energy efficiency initiatives. Four interviewees stated the

resource would facilitate comparison and lead to consistency ac ross the country

, and three respondents suggested it would provide more transparency in numbers and algorithms used. Most interviewees said the resource would
have to account for regional differences to truly address user requirements

. The interviews, howe ver, were designed to address the need and feasibility of a database of savings estimates. Metrics around non

- energy benefits, such as job creation, or market transformation were not discussed, as that was beyond the focus of this study.

Two interviewees raised a potential concern with having a national DS&A database: the market’s possible transition
toward more custom or whole

- building savings approaches. One interviewee , a utility representative, mentioned
that the market has moved more toward custom mea sures , and cautioned against creating a costly prescriptive
measures database due to declining shares of prescriptive measures in total portfolio savings. Reportedly, t he majority of savings i n this utility’s service territory have derived from whole

- facil ity retrofits, design improvements

, and custom measures, mak ing prescriptive savings estimates less useful to this particular utility. No representatives
from the other seven utilities we spoke with mentioned this market transition , making it difficult to determine
whether the trend is widespread or varies by utility area, state, or region. However, a non

- utility respondent voiced a similar concern: that the focus moving forward in the energy efficiency industry should be on whole

- building savings and not m easure

- specific values. This respondent cautioned that creating a national measures database would diminish the level of rigor in estimating energy savings.

Overall, most respondents reported that there is a greater need for a database of algorithms than f or a database providing only specified ex ante savings values. However, all respondents new to the field reported struggling in the planning stages to determine the best algorithm to use for each measure , given the variety of existing TRMs.

R espondents fro m two nascent utilities stated that algorithms were helpful as guidelines , but that for savings estimates to be useful, it is critical for them to understand the inputs and assumptions resulting from determining that savings estimate. The third nascent uti lity , on the other hand, would prefer to be provided with a deemedJ une 2011 www.seeaction.energy.gov 23

value that would remain the same from the planning stage through evaluation. This was based on their experience collecting the data needed for complicated savings algorithms

— much of it cust omer data that are difficult to
obtain

.

The idea of national, standardized measure savings algorithms or values was generally not considered feasible at this time (or in the near future) by interviewees, largely due to the widely varying regulatory requir ements, baseline(s), program structures, and evaluation needs. Almost all interviewees saw the value in having national savings estimates that are region or climate

- zone specific ; in other words, a collection of regional DS&A databases would compose the na tional database. However, most stated that a national database would not be used consistently unless mandated by regulators, and most opposed a mandate to use the values or algorithms in the
database. Reportedly, requiring national savings values would cau se a large amount of pushback from states and entities with established protocols and TRMs. One respondent said that national standards are needed, but acknowledged such standardization would be difficult to achieve and that the market is not ready for suc h a change. Three respondents noted that in order to provide the transparency needed for savings estimates, consistent measure data collection requirements would be needed, and none are currently in place.

The general consensus was a desire for a national DS&A database to be available for use at will, but without mandated nationwide standardization on savings values. One utility representative noted he would oppose creation of the database if its use would be required, given that his state already has a TR M and it would n o t be in his utility’s
best interest to have a national database

. S everal other utilit ies expressed anxiety regarding national
standard ization , primarily if it involved replac ing the current deemed savings estimates they had worked hard to

create. Other respondents, however, noted that a nationally required measures database would add consistency
and make the data more accessible.

Despite the lack of support for national standardized savings values or algorithms, interviewees thought the DS& A database would be useful for states and utilities with little energy efficiency program experience, as long as the feasibility issues could be overcome. They said a database compiling the different algorithms and stipulated savings
estimates would be hel pful, as long as individual entities and regulatory divisions remained entitled to use their own methods. They stated that such a resource would allow for a comparison of methods used nationwide, which two interviewees said might naturally lead to more con sistent practices over time. If such a resource were accepted by regulators and industry actors, there would need to be opportunities for stakeholder inputs and transparency into what the numbers represent. This would include clearly explaining inputs and assumptions, either in arriving at the savings estimates or as imbedded information in the algorithms supplied.

Feasibility

Several types of concerns were raised about the f easibility of creating and maintaining a national DS&A database

. Although most in terviewees thought a measure database resource would be useful, they also acknowledged it would take a significant amount of effort for the database to be sufficiently comprehensive enough to be widely
useful. Respondents also thought it would be difficult to keep the resource up

- to

- date . Five interviewees thought it was simply not feasible. Others stated that creating the national DS&A database is potentially feasible, depending

on multiple contingencies. One interviewee warned that the scope of this proje ct could easily become unmanageable due to extreme differences in measure savings estimates.

The interviewees identified multiple concerns that would need to be addressed for a national DS&A database to be widely useful. One of the main concerns i s the va st regional variation in factors such as baselines (codes, standards, standard practice, and when different baselines are applied), weather, economic s , demographic s , vintage, and

measure costs that need to be reflected. This may, in part, be alleviated thr ough multiple regional databases. However, even then, there may still be wide variation within a designated region. Only one interviewee thought it was a good idea to replace regional and state

- mandated TRMs with national standardized deemed values, but th is interviewee also recognized these changes would have to be region

- specific.J une 2011 www.seeaction.energy.gov 24

Another substantial challenge facing development of the national DS&A database is the constant updates required to adjust measure savings values based on ongoing research and s tudies. Six interviewees mentioned that the upkeep needed to maintain the usefulness of a measures database would be extremely costly and time intensive. A concern voiced by two respondents was that if savings estimates were changed in the middle of a prog ram cycle, the values used in planning the programs might not be the same as those used for their evaluation, making it difficult to meet savings goals. This could be addressed at the state level by coordinating updates and program cycles, but would be dif ficult to address at the national level because utility program cycles are not on a consistent
schedule nationwide.

In addition, t wo respondents were concerned about the large variance in ex ante savings numbers across TRMs and regions, and said that coul d create skepticism about energy savings values. Both respondents stated it would be
difficult to develop a database that is sufficiently accurate and reliable enough to ensure the industr y’s confidence. If a national database produces different savings es timates in different locations without explanation, those differences would be highlighted and could raise doubts about their accuracy. If differences are evident but the reasons are explained, it is possible that the credibility of the database would be e nhanced and users would better understand what affects energy savings.

Content and Functionality

Respondents offered many unique, often conflicting ideas regarding the desirable content and functionality of the DS&A database. All respondents addressing t his topic stated that a high transparency level for the savings calculations would be necessary.

Respondents commented that d eemed savings values would be more suitable for use by regions where evaluation data are available to support the deemed values. H owever, it would be difficult to find sufficient data in many locations, and it would be labor intensive to develop the data needed to support a national DS&A database for many measures reviewed. Deemed savings are most suitable for complex measures (with multiple interactive effects) that are typically analyzed using building simulation software. Such measures would include building envelope measures and large HVAC systems. In addition, a naming protocol would need to be developed to communicate the scope and applicability of each measure. The disadvantage of this approach is the large number of permutations (measures). 5

However, most respondents favor a resource that includes algorithms over simply reporting savings values. Algorithms would need to be clea rly presented, along with applicability conditions and documentation of underlying assumptions used to derive savings values, including:

 Measure definition, program delivery mechanism, customer characteristics and use of the measure and/or the process/equi pment with which it is associated

 Regional characteristics

 A ssumptions made

 I nputs used

 P rogram specifications

 R egulatory requirements

 W hether values are gross or net.

5 DEER is a good example of this: the 2005 version has 361 unique measures and 129,165 permutations.J une 2011 www.seeaction.energy.gov 25

A dynamic database, with embedded algorithms based on engineering calculations, may of fer flexibility in aggregation levels and appeal to a wider range of users; however, the support level required may be equally, if not more, demanding than the stipulated savings approach. The database methodology would need to account for many of the same considerations applicable to a stipulated savings approach. Based on the interviews and findings from our review of existing TRMs, we suggest the DS&A database include:

 A preferred algorithm, accommodating a range of scenarios while remaining user friendl y (inputs should be based on easily observable and quantifiable measure characteristics, such as the tank size for a water heater).

o The algorithm should include a description detailing aggregation levels. For example, a specific motor retrofit (one motor t ype, RPM, and horsepower [HP] combination) or the averaged motor retrofit (savings vary by HP only, algorithm already weighted by type and RPM).

o The baseline efficiency and efficient conditions must be clearly described.

 Guidance on use of actual vs. defa ult input values (e.g., actual hours

- of

- use or values provided in the database). The actual policy that might be applied in a specific jurisdiction for using a mixture of actual and default values would likely be established by the authority responsible fo r that jurisdiction. As observed in many measure comparisons, different TRMs, approved by different regulators, have differing approaches. Some example approaches include:

o Supporting reference tables and default values should be provided for ease of

use. (Default values must be regularly adjusted to accommodate codes and standards updates.)

o For users from regions where data are not readily available, it may be preferable to use actual input values (and to collect these data during program operations).

o Som e adjustment factors vary greatly from region to region, such as the in

- service rate, end

- use interaction, and coincidence factor. A possible way to handle these variations is to include a placeholder “master” adjustment factor in the algorithm, and set th e default value to ‘1’. Regions where multiple adjustment factors are required can determine their own effective adjustment
value for use in the calculation.

 Definitions of input variables to ensure transparency and reduction in human errors.

o Input units will always need to be defined (e.g., HVAC capacity in tons vs. Btu/hr), along with the appropriate number of significant figures for the chosen algorithm (e.g., 3 tons vs. 2.75 tons).

o Including conversion factors, or unit

- flexible dynamic algorithms, wou ld add a desirable database feature, though be more labor intensive. This would require the database or program perform dimensional analysis to ensure a valid result.

o Variables, even those with the same symbol and definition, were used inconsistently in

m athematically equivalent algorithms. A good example of this is the waste heat factor for lighting

- HVAC interactions. Many TRMs referred to this interaction as ‘WHF
e ’; however, usage diverges when this interaction is based on a negative number, such as in e quation A, or when the calculation yields a number greater than one, such as in equation B.



###  Equation A:J une 2011 www.seeaction.energy.gov 26 

###  Equation B :

 Adequately detailed information for e ach algorithm and user

- adjustable input

, ideally with an example calculation. For instance, in the case w here a 60 watt ( W ) incandescent lamp would be replaced by a 15 W CFL:

This level of detail would be necessary to allow other entities to compar e the database values to their own programs and account for differences. The need to address regional differences was cited as a critical functional requirement by a majority of interviewees. Without clear information on how to account for regional differe nces, data and calculations may be used out

- of

- context, fostering distrust in the accuracy of the resource.

One respondent noted, however, that states new to energy efficiency programs may be overwhelmed by algorithms in TRMs, and suggested a calculator t o reach this target audience. Although this option better addresses the needs of those new to the field, professionals interested in a more robust tool may struggle with a calculator’s inherent limitations. Whatever is decided, the tool would need to clear ly define all algorithms used in the calculator functions and provide flexibility in inputs to be widely useful for both expert and non

- expert user groups.

Another respondent offered a similar idea, suggesting the DS&A database could have a methodology fi lter that
would allow the user to select default s or to input data for their own market . They also mentioned that data sources need to be clearly cited and identified. As we found during our TRM research, circular references from
these sources were not unc ommon.

Several additional common themes on content emerged from the interviews. Most respondents agreed the database would need ongoing improvement and upkeep to remain current. Two respondents suggested including a date stamp on values, and one suggested “sunset” provisions , such as those used by the RTF (where values would become obsolete after a given amount of time)

. G eneral ly, respondents stated that tracking the age of values would
facilitate their staying current with recent studies. Another suggest ion

, noted several times

, i s the need to have raw data included in the database for greater transparency.

Opinions varied regarding the content necessary for the DS&A database to be useful. While some reported the
database would need to cover a vast assor tment of measures, an almost equal number said the opposite, preferring a database more narrowly focused on certain common measures with significant potential. Those favoring a narrower focus thought common measures would serve as a good starting point, an d the database could be expanded as needed. Almost all interviewees preferring the more comprehensive version noted there would be tremendous feasibility issues in actually proceeding with this option.

Although the interviewees generally agreed that most c hallenges facing the creation of a national DS&A database could probably be overcome, they reported one aspect that would be nearly impossible to develop: estimating peak load impacts. One reason is that larger states with multiple climate zones and high l oad diversity could have utilities with significantly different peak demand patterns. In addition, peak periods tend to be unique to particular utilities. Thus, creating a standard set of coincidence factors would have to be based on gross approximations, and the resulting values would have limited use.J une 2011 www.seeaction.energy.gov 27

Given the more urgent needs of those new to energy efficiency programs and the likely difficulties inherent in developing a national DS&A database, interviewees suggested alternative, intermediate resources to meet these needs. One suggested a series of white papers on different approaches and algorithms used for some common measures. This could easily be expanded as more white papers are developed, and would be useful to those new to energy efficiency progra ms because greater detail would be provided in a format easier to read than a resource that simply provid es savings values and algorithms without more complete background information. These white papers could later be integrated with a measures database an d serve as background material. Other suggestions included incorporating the results of potential studies and baseline estimates in the database

.

Quality Control

S everal respondents mentioned q uality control as being critical to the development of a nati onal DS&A database. One interviewee suggested including values accepted by a regulatory body in the database. Another respondent suggested having stakeholders agree on methods and criteria used to screen information and data entered into the database. Qual ity assurance for data in the DS&A database would be especially important to users new to energy efficiency programs who want to use this resource as a guide , since they might not have the knowledge to recognize

errors

.

Comparison of National DS&A Databas e to Existing TRM Resources

Most respondents were familiar with DEER, and some had experience with other states’ TRM resources and the Northwest RTF. Knowledge of these databases allowed respondents to provide insights into the shortcomings of

current meas ure savings sources.

DEER

Although California

- specific, DEER has been a useful resource for users in many different locations and with a wide range of backgrounds. All interviewees new to the industry had used DEER at some point for planning and developin g programs. However, one respondent indicated DEER was “ overwhelming to a newcomer .” Another noted, as it is California

- specific, that much of its information was not relevant to their area. In general, more experienced DEER users valued its comprehensiven ess, with both savings and cost data for a wide array of measures. One respondent also indicated the portability of the database was very convenient. However, those interviewed felt the data sources need to be more transparent and include date stamping, an d the database could be difficult to navigate due to its volume and spreadsheet format.

In general, TRMs are customized to particular locations. However, they are often used across geographic areas by simply altering assumption about climate. Interviewees noted this may be an inappropriate use of the TRMs. One industry expert elaborated by saying: “ States should consider other adjustments besides weather, and a challenge with a national TRM is understanding all those parameters

. ” TRMs may also not be being applied properly when algorithms are very complicated and have lengthy calculations with a large number of inputs.

Northwest RTF

Those familiar with the Northwest RTF spoke very favorably of its capabilities. The RTF is a regional forum with an online rep ository, providing deemed values for a comprehensive list of measures with supporting calculations and documentation. Several interviewees commented that part of what makes the RTF database robust and credible is
it uses evaluation data to inform the ex an te savings estimates. One interviewee even suggested the national database would be most useful if it consists of several regional forums and databases similar to the RTF. Another interviewee from the Northwest praised the RTF’s forum approach, allowing in terested parties to weigh in, which this individual thought would be a good feature for a national database. The only improvement suggested for the RTF concerned its search capabilities, which users said are difficult to use for those unfamiliar with the s ite.J une 2011 www.seeaction.energy.gov 28



### Cross

- Database (DS&A and EM&V Document Database) Considerations

The previous sub sections discussed issues specific to each of the two database s under consideration. This

s ubs ection covers topics, such as hosting and funding, that are common to both d atabases , and outlines the possible linkages between the two databases.

Hosting

During interviews, Cadmus asked for feedback on appropriate entities for developing, maintaining, and managing both the databases. Industry professionals most often suggested DOE as the single entity that would be most suited to hosting such databases, and the suggestion of DOE in conjunction with a third

- party and an independent administrative entity was mentioned almost as frequently. Other suggestions included: a regional in frastructure or collection of regional entities; a forum similar to the Northwest RTF; the CEE; and a community

- based, non

- federal entity (see Table 9 ).



### Table 9 . Recommendations of Possible Candidates for Database Hosting

Hosting Recommendation Frequency

DOE 5
DOE Manages Contract with Third

- Party Administrator 4 Independent Entity 4 Reg ional Infrastructure or Collection of Regional Entities 3

Forum Like Northwest RTF 1

CEE 1
Note: M ultiple responses allowed

.

Determining the best entity for hosting the database s must take into account several factors , and will require thorough review and careful consideration. Interviewees emphasized the importance of having a database owner
and host that is trustworthy to all stakeholders (utilities, regulators, third

- party implementation contractors, and EM&V professionals). They noted the administrative function would be complicated, since it requires expertise in multiple disciplines such as librar y science (cataloguing and archiving), industry knowledge, engineering, and information technology.

Funding Sources

Most interviewees stated DOE would be the most appropriate funding source, at least during early development and implementation stages. Aft er the databases have been developed and are operational, most of the funding responsibility may be transferred to other entities, such as states, ratepayers, current sources of evaluation funding, and national utility associations, among others.

Eight ou t of nine responding interviewees said they would be willing to pay a subscription fee for access to the databases, depending on the quality of the databases and the added value they offer. Some interviewees suggested imposing fees on a sliding scale; so s tates or entities with smaller evaluation or energy efficiency budgets would not be overburdened by costs. A subscription fee may only be possible after the databases are proven to be valuable resources. Two interviewees noted they would be more willing to pay a subscription fee if they could provide feedback on database content and updates. Regardless of the databases’ hosts or funding source, several interviewees commented the involvement of multiple stakeholders in the planning and implementation process would strengthen and improve the final products.J une 2011 www.seeaction.energy.gov 29

Criteria for Best Practices

Promoting the adoption of standard, best practices in savings calculations and EM&V practices should be a primary objective in developing the national databases. However, determ ination of what constitutes a best practice requires establishing clear criteria and using them consistently to rate and screen TRMs and EM&V reports. In addition, the criteria of best practices are relative, depending on efficiency program and EM&V goals as well as budgets and resources available to conduct the EM&V. We asked the industry experts to identify potential criteria to be considered when evaluating whether a plan, report, or market study serves as an example of best practices. Criteria
identifie d included:

 The plan/report clearly describes:

o Assumptions

o Sample design and sample sizes

o Certainty of results

o Clear descriptions of context and objectives

o Source of data (primary versus secondary)

o Variance between ex ante and ex post savings

 The report i ncludes actionable recommendations

 The plan/report uses plain wording, not excessive jargon or engineering language

 The plan/report discusses threats to validity and controls for alternate hypotheses.

R espondent s generally expressed concern about the many obstacles to defining a best practice objectively and that such a determination would vary by situation. These barriers may make it more feasible to address best practices on the actual product, such as a final EM&V report

, than to judg e whether best pract ices were actually employed in conducting the evaluation or study. For example, criteria such as involvement of independent third

- party evaluators, approval by regulators, or study vintage could serve as objective and consistent measures that might be
used

.

The key criterion for a national DS&A database is transparency. Nearly all respondents indicated that for such a national database to be accepted, the approach must be clear and apparent. Ideally, the savings estimates must be
based on evaluation result s (similar to the RTF), though recognizing those data may not be available, particularly in
states where little or no EM&V has occurred.

Link s B etween EM&V Documents and DS&A Databases

Several i ndustry professionals expressed an interest in establishing a link between the EM&V document s database
and the DS&A database . Specifically, they envision ed the evaluation results from reports and studies in the
documents database used for informing data in the measures database. Although developing this capability wo uld be challenging, a link between databases could improve the transparency and consistency of documentation, and, as one respondent noted, this would support public information sharing. The Northwest RTF was mentioned several
times as a solid model for a national database, and it was noted the RTF relies on evaluation findings as the basis for regularly updating measure savings. Some respondents viewed the RTF as more of a source for ex post than ex ante savings estimates.J une 2011 www.seeaction.energy.gov 30



## Recommendations

In our interview s with potential EM&V document and DS&A database users, we identified two basic needs.

The first is to provide new practitioners with available resources that are easy to use and have a limited, focused content. New practitioners need information of all t ypes to support their work, including EM&V plans and reports
and measure ex ante savings information, as well as supplemental information, such as EM&V protocols, market studies, measure

- specific technical studies, TRMs, and RFP examples. To facilitate loc ating and understanding these documents, an online resource should have clear navigation and search capabilities, and avoid clutter that might
overwhelm the user. For ex ante measure savings, new practitioners need resources to allow them to produce saving s estimates applicable to their particular geographic region and other unique circumstances as easily as possible.

As new practitioners gain expertise in energy efficiency program planning, administration, and EM&V, their needs change. Instead of general, how

- to information, they are more likely to need greater detail and access to a broader collection of available work. This information will allow them to broaden their knowledge and compare their
methods and results with others.

The second basic need our interviews exposed was to provide experts with a single

- source location to search for and retrieve information. Our interviewees were generally aware of resources readily accessible online, but were unhappy with the limited nature of these resources and t he time and effort required to perform a thorough search across resources. An online resource for EM&V documents that would satisfy experts would necessarily be more comprehensive than one satisfying the needs of new practitioners, and the methods for popu lating the database would be significantly different and more resource intensive.

In regard to the DS&A database, experienced practitioners expressed the desire for a resource allowing them to easily find savings estimates or supporting documents and stud ies across many different regions and programs. The purpose of such a resource would be to better understand different underlying methodologies and to compare their findings.

Recognizing different needs between new and experienced practitioners, there are two different database options

— focused and comprehensive

— for each of the two resources: an EM&V database and a database of measure savings. The four options and their primary features are outlined in Table 10

.J une 2011 www.seeaction.energy.gov 31

* Savings based on past evaluation results would need to be updated regularly. The two comprehensive measure savings options are mutually exclusive as described below.

The two focused options could together comprise a starter kit resource, providing a single resource of documents
and links to EM&V documents and DS&A resources. Given that S EE Action’s EM&V Working Group originally envisioned these databases as primarily serving nascent states, our primary recommendation is to concentrate initially on developing the starter kit rather than the more comprehensive options.

The scope of this stu dy included examining whether the database containing EM&V documents should also include EM&V plans, or if a separate database should be created for plans. Cadmus recommends using a single database solution for the following reasons:

 It provides a single l ocation for practitioners to find the information they require

 N o technological barriers exist for implementing a combined database, as the database structure and site functionality will be identical regardless of document type

.

 Our interviewees expressed a strong desire to not limit the type s of documents collected on the site, but rather to have a variety of content

, as described above

.

We developed the cost estimates for each resource configuration in two steps. First, we organized the development
and ad ministration functions of the databases into specific tasks. Second, we assigned cost estimates to each task according to its anticipated required level of effort.

Level of effort is expressed in employee

- years, based on 2,080 annual hours for development , or FTE for annual maintenance, and an average labor rate of $150/hour. This rate can be adjusted to account for varying labor costs, depending on the organization ultimately conducting the work. We provided low and high estimates, in addition to
the like ly estimate, to show the potential range of costs. The low estimate is 25% less than the likely estimate, and the high estimate is 50% more. 

### Table 10 . Resource Options

Targeted User EM&V Documents Database Deemed Savings and Algorithm Database

New Practitioner

- individual or organization seeking introductory information Focused

 Links to existing resources

 Guideline s on best practices

 Exemplar documents included based on expert input Focused

 Links to TRMs and other existing measure databases

 White papers on technologies

 Example algorithms with details on parameters used Expert User

- individual or organization see king supplemental or comparative
data Comprehensive

 Large collection of diverse documents

 Sophisticated search and tagging

 Regular additions and updates

 Data collected during evaluation Comprehensive
National or regional database(s) with savings estimate s base on either:

 A database of verified past results * or modeling data Or

 An interactive calculation engine built on a set of predefined algorithms ( i.e. an online TRM)J une 2011 www.seeaction.energy.gov 32

We considered initial development costs and annual recurring costs to maintain and enhance the resources. The
task areas we considered and their supporting subtasks include:



### Development Costs

 Requirements Gathering

o Client requirements

o User/stakeholder interviews

 Specification and Design

o Specification and use case development (a use case is a description of steps or act ions in a software system that lead the user towards a desired result)

o Design and design documentation

o Cost estimation

 Technical Implementation

o Technology selection

o Template setup

o Initial content creation/loading

o Searching for and evaluating existing docu ments to identify those representing best practices

o New content development, compiling and loading documents and descriptive text

 Content Development

o Determine topics to cover

o Design and arrange sections (pages)

o Site navigation

o Determine the amount of docu ments to include

o Determine methods for identifying and evaluating documents

 Testing

o Internal testing

o Client acceptance testing

o Modifications

 Release and Maintenance

o Deployment

o Marketing

o Link/search placementJ une 2011 www.seeaction.energy.gov 33



### Annual Recurring Costs

 Content Management

o New c ontent development and identifying new source documents

o Site contents updates

o Ongoing quality assurance/quality control

 Site Administration

o Hosting

o Site management

o User support

 Site Enhancement

o Change request tracking

o Site updates and testing.

The main fe atures, possible content, information technology, and costs for each of the four options identified in
Table 10 are described below.



### EM&V Document Database

— Focused

Content

A resource that would address the needs o f new EM&V practitioners would be best accomplished through a Web content management system (CMS), with original page content, documents, navigation elements, and search capabilities.

The website would include original page content describing EM&V practice s and processes, and have links to other online resources, including those identified as part of this study. Compared to a strict document repository with only search capabilities, this type of site would permit navigation to specific topic areas and conte nt, which could include links to supporting documents. Documents included on the site would be vetted by an expert or expert panel, and would be judged on their usefulness to new practitioners and their representation of industry best practices. Potential topic areas and supporting documents may include, but not necessarily be limited to:

 EM&V plans

 EM&V protocols

 EM&V reports

 Market study reports

 Conservation p otential studies

 Process evaluation reports

 Example procurement documents such as RFPs, Requests for Qualifications, and lists of EM&V practitioners.J une 2011 www.seeaction.energy.gov 34

Although the content in this focused resource would initially be limited, the CMS framework allows expansion of topic areas and document types without additional investment in the underlying technology.

Technology

CMS provides a popular tool for website creation, and many proprietary and open

- source products are available for
use. Microsoft SharePoint ® is the most widely recognized proprietary software. Open

- source products include
Joomla! ® , WordPress, D rupal TM , and many others. CMS popularity is driven by its ease of use and low cost to set up and administer. CMS does not require extensive programming or database setups. Development work primarily occurs in template setups and content loading.

Cost

The e xpected level of effort and associated costs for developing and maintaining the focused EM&V database option are summarized in Table 11 and 11. As explained above, we based the cost estimates on the expected level of effort, expressed in employee

- years (for development) or FTE (for maintenance), assuming 2,080 annual hours at a labor cost of $150/hr. We expect the total costs for development and launch of this option to range between approximately $520,000 and $775, 000, with between $200,000 to $300,000 required for ongoing annual administration and maintenance ( Table 11 and Table 12 ).



### Table 11 . Estimated Deve lopment Labor and Cost (Focused EM&V database)

Task Estimated Employee

- years


### Requirements Gathering 0.20

Specification and Design 0.125

Technical Implementation 0.20

Content Development 0.75

Testing 0.125

Release and Maintenance 0.25

Total Develop ment Employee

- Years 1.65

Total Development Cost ($150/hr)

Most Likely $514,800

Low $386,100

High $772,200J une 2011 www.seeaction.energy.gov 35 

### Table 12 . Annual Recurring Labor and Costs (Focused EM&V Database)

Task Estimated FTE



### Content Management 0.25

S ite Administration 0.25

Site Enhancement 0.125

Total Recurring FTE 0.625

Hosting ( third

- party) $6,000

Total Development Cost ($150/hr)

Most Likely $201,000

Low $150,750

High $301,500

Content development represents the highest cost element, and is also the potential source of the most variability in the total cost, specifically as it relates to searching for and evaluating documents and creating new content for the
site. The site’s exact content

— including topics covered, design and arrangement of s ections (pages), site navigation, the amount of documents to include, and methods for identifying and evaluating documents

— would need to be better specified before a more accurate estimate could be calculated.

The hosting costs would vary depending on the configuration (internal versus third

- party) and on the software platform chosen for the site. We have assumed a third

- party hosting configuration, based on feedback from interviewees that the site would be best managed by an impartial entity. We further a ssumed an open

- source solution would suffice, and there would not be significant software costs involved.

Implementation

Cadmus recommends a CMS

- based, focused content w ebsite

, largely because of its relative ease to create and
launch. The implementation i ssues that need to be resolved include:

 What content should be include d?

 Who would create and administer the site ?

 What amount of ongoing content generation would be necessary?

 If expert evaluation of documents proves to be required, which experts would be used , how they
would be selected, what evaluation process they would use, and how much they would cost ?J une 2011 www.seeaction.energy.gov 36



### DS&A Database

— Focused

Similarly to the EM&V documents providing a focused database resource, a focused DS&A database resource would
serve as an online resource, targeted toward practitioners new to the energy efficiency field.

Content

A focused resource

, assist ing new practitioners with generating ex ante measure savings for program planning and
EM&V

, sh ould provide information on calculation methodologi es and best practices. The c ontent would be similar to that presented in TRMs adopted by state utility regulators for planning and evaluation purposes. The resource would not, however, directly supply ex ante savings algorithms or values or other measure

- s pecific data; instead, it would provide links to existing TRMs or other measure savings values resources. In the comprehensive resource subsection below, w e discuss in greater detail the options and issues with having a resource providing measure savings.

Technology

Cadmus suggests including the focused measure resources into the focused document resource as a specific
information category. This portion of the database could include e xisting TRM s , primary source materials on savings

, and other technical doc uments related to measure savings.

Costs

If the measures information is combined with the document s resource , the only additional costs would be the incremental hours spent collecting and loading measure

- related documents. We estimate this time at roughly 200 hours initially, and between 100 and 200 hours each subsequent year. The incremental cost would be around $30,000 in the first year, and between $15,000 and $30,000 for every year thereafter.

Implementation

The only additional implementation issue wit h including the measure s information in the documents resource is determining what content to include

, given variations in the computational methodologies and assumptions found in existing TRMs.



### EM&V Document Database

— Comprehensive

Many experts expressed a desire to have a comprehensive, national repository of EM&V documents, including evaluation reports, plans, and the other types of documents listed above. As described, the two existing primary EM&V document sources (CALMAC and the CEE site) have limitati ons in their completeness, usability, and accessibility, leaving the need for a fully comprehensive, national

- scope document repository unmet.

Content

A national repository could cover the same topic areas as the focused option. The goal would be to compi le as many documents as possible in one site to remove the need for practitioners to search multiple sites. A sophisticated search and filtering user interface would provide easy access to the documents and, ideally, would limit the number
of results retur ned. In this way, users could find what they seek while reviewing only a small number of search
results. Cadmus recommends each document in the repository include :

 A descriptive title

 A five to 10 line summaryJ une 2011 www.seeaction.energy.gov 37  An abstract

 A list of keywords, selected from a predefined master list to ensure consistency

 A format allowing full

- text indexing for searches

 As much descriptive metadata as possible, which could include:

o Author name(s)

o Author firm

o Sponsoring agency(ies)

o State(s)

o Report date

o Program year

o Program type (energy efficiency, demand response, low

- income, etc.)

o Document type (evaluation report, protocol, market study, etc.)

o Sector(s)

o Specific measure types included

o Specific analytical methods used.

Both CALMAC and CEE use a relatively passive process for gat hering content. While the CPUC requires utilities to post certain documents to CALMAC, the onus is on document authors or sponsors to make submissions and fill out
online form s

. CEE relies on a voluntary submission process with its members.

P assive approac h es and more active approach es to content gathering each have advantages and disadvantages

. A passive approach takes less effort , but at the expense of incomplete coverage and of being potentially biased toward the most diligent contributors. An active app roach would be more time

- consuming

, but would lead to a

more populated and robust site. Active methods for gathering documents include : developing and maintaining a list of all known online sites where documents are posted

, and regularly checking for new m aterial ; and soliciting documents directly from utilities, commissions, consultants, and other sources.

S everal experts discussed the possibility of a national effort, spearheaded by DOE or another agency

, to establish a
framework and infrastructure for co llecting public material for inclusion in the database

. This raises several issues related to appropriate monitoring and compliance , which would need to be further researched. Regardless of the chosen collection method, implementer s will need to perform a librarian function to monitor and follow up on
submissions. Each submitted document would have to be vetted for appropriateness , quality, and data validation.

Technology

A CMS similar to that suggested for the focused option and based on the same software would likely meet the requirements of a more comprehensive resource. By basing this database on the same software framework, the comprehensive database could evolve from the focused database.

For usability, Cadmus suggests having both simple and advanced search options available to users, as commonly occurs on many websites. Simple search interfaces typically provide a single text box for entering a search term, and returns results matching the term across multiple fields (document title, abstract, keywor ds, etc.). An advanced search and filtering interface would also allow searching for specific terms, while providing searches in specificJ une 2011 www.seeaction.energy.gov 38

fields, multi

- term searches, and limited results based on filters applied to any of the metadata fields. For example, the CALMAC search page provides filtering capabilities for the limited amount of metadata collected (but does not
allow field

- specific text searches).

Open source CMS software often includes a core search module with basic functionality, allowing the devel oper community to create add

- on modules to provide enhanced features. Modules exist for the Drupal platform, for example, to meet advanced and full

- text search features, but it may be necessary for the implementing organization to create custom modules spe cifically for this application. The cost estimates below assumes some custom work will be required.

Cost

The total costs for development and launch of the comprehensive EM&V documents database are expected to range between approximately $825,000 and $1 .25 million , with between $32 0 ,000 and $ 480 ,000 required for ongoing annual administration and maintenance ( Table 13 Table 14 ). Again, these estimates are based on the
expecte d level of effort, expressed in employee

- years or FTE, assuming 2,080 annual hours at a labor cost of $150/hr.J une 2011 www.seeaction.energy.gov 39 

### Table 13 . Estimated Development Labor and Cost (Comprehensive EM&V Database)

Task Estimated Employee

- Years



### Requirem ents Gathering 0.20

Specification and Design 0.25

Technical Implementation 0.25

Content Development 1.50

Testing 0.20

Release and Maintenance 0.25

Total Development Employee

- Years 2.65

Total Development Cost ($150/hr)

Most Likely $826,800

Low $62 0,100

High $1,240,200J une 2011 www.seeaction.energy.gov 40 

### Table 14 . Annual Recurring Labor and Costs (Comprehensive EM&V Database)

Task Estimated FTE



### Content Management 0.50

Site Administration 0.25

Site Enhancement 0.25

Total Recurring FTE 1.00

Hostin g ( third

- party) $9,000

Total Recurring Cost ($150/hr)

Most Likely $321,000

Low $240,750

High $481,500

Implementation

Implementation issues to be addressed for creating a comprehensive, national EM&V document resource include:

 Collecting content. Thou gh the cost estimate above assumes an active approach to content collection, it may prove to be more time consuming to find, evaluate, and post documents than our
estimate allows.

 Generating keywords and abstracts. Under an active collection process, imple menters would review each document and generate an abstract, list of keywords, and other metadata to fit the site format, allowing for sophisticated filtering.

 Permission to post. The implementer may find certain entities would not grant permission for the ir documents to be posted on the site, particularly if doing so might reduce traffic to a similar w ebsite, such as CALMAC. Without a national mandate to submit documents, the question of having rights to reproduce content from other sources requires furthe r research.

 Site creation and administration. The responsibility for creating and administering ongoing
maintenance of the site needs to be determined.



### DS&A Database

— Comprehensive

Content

Determining a resource that would serve as a comprehensive national DS&A database proved to be the most challenging aspect of this scoping study. Ideally, we envision this resource as a database of measure savings values that a practitioner could access by selecting the desired measure, configuration, baseline, region/weat her, andJ une 2011 www.seeaction.energy.gov 41

other parameters, and that the database would provide feedback on specific savings and, perhaps, cost information. Savings values would be based on actual measured data or be calculated based on a universally accepted set of algorithms and regiona lly specific parameters made applicable to the entire country. Significant resources would be needed to determine the appropriate values or algorithms to use.

Unfortunately, it would be extremely difficult to create such a resource. The data required for a data

- driven resource do not exist in sufficient quantities or qualities to populate such a tool, and no universal agreement exists across existing TRMs as to how measure savings should be calculated (see Section 0

, Review of Existing Resources ). It would be more feasible, although still difficult, to create individual regional DS&A databases. Depending on the region, data may be more available and consistent. However, the primary obstacle of determining appropri ate methods, algorithms, and assumptions to use would remain.

Above, we alluded to the issue of there being two distinct methods available for arriving at an ex ante savings value:
a data

- driven model approach, based on prior observation or building simula tions; and an algorithmic approach. These methods would require different types of software development, although efforts and costs would be similar. We describe each approach individually, and then provide a cost estimate applying to either method.

Data

- d riven Model

DOE’s original intent was for Cadmus to investigate the feasibility of creating a national DS&A database. To compile
such a data

- driven resource, data to populate the database would need to exist, be readily accessible, and be
described suffici ently to allow its conversion to a standard format. Certainly, some of these data exist in various forms as products of ongoing energy efficiency work. These data could also be determined from building simulations, conducted for a large variety of building types and climate zones (as done for DEER).

Some existing resources compile this type of information, most notably the Council’s RTF, which maintains an online
database of measure information

, including Microsoft Excel spreadsheets with savings data base d on a combination of algorithmic approaches and ex post savings. D ata are specific to the Northwest region

, and go through a rigorous review process before inclusion on the site. The RTF’s annual budget is approximately $1.5 million , and approximately 25 voting members volunteer their time and expertise to support the RTF

.

Algorithmic Approach

Early in our work on this study, Cadmus began to explore the option of an algorithm

- based tool

, simply because the majority of TRMs in use across the country are ba sed on these

. We discussed this internally and with expert practitioners to determine what would be required to create an online tool of algorithms, where users could select desired measure parameters and have a calculated ex ante value returned. Such a to ol would require development
of the following :

 A set of universally accepted algorithms for all measures included in the database

 Regionally specific parameter data to serve as input s to the calculations

 An engine to perform the calculations

.

Developing e a ch of these items would require substantial work

, and, in the case of a national DS&A database

, would require high levels of cooperation and acceptance across the various states, utilities, and other organizations serving as stakeholders in the process. Fr om the interviews, we learned stakeholder input and transparency would
be critical for entities to accept , trust, and adopt the resource.

As discussed earlier , consistent approaches and assumptions have not been provided for calculating ex ante savings

in the TRMs we reviewed. The process by which a national DS&A database would be created is not clear

, and

, without a federal requirement to do so, it is likely many states would opt to not adopt a national standard. WithoutJ une 2011 www.seeaction.energy.gov 42

a national requirement, the usefuln ess and value of the database would be diminished. Cadmus considered incorporating all available TRMs, allowing users to select which approach best meets their needs. We suggest, however, that in addition to a significant per

- TRM increase in co st s , the inc lusion of many different options would
confuse users

, and lead to validity questions about each option and the general algorithmic approach.

In regard to parameters, TRMs often specify statewide values

, such as hours

- of

- operation, baseline condition, and
h eating/cooling degree days. These values vary by TRM

, and may not yet exist for states without a TRM or a history of energy efficiency programs

. The method for reconciling existing parameters values between TRMs and the new values created is not clear. Ano ther approach is to have all parameters be user

- specified. This , however, puts the burden on the user, and could lead to misuse of the tool.

Finally, the implementer would need to develop the underlying calculation engine. Consulting firms we interviewed

a s experts in th is field already have calculation engines or the capability to create them. This is the most straightforward design aspect

, but would still be expensive.

Technology

Either of the two model options above would require a custom

- developed soft ware tool

, and could not be implemented with existing solutions. D evelopment and hosting of the site could be conducted in any number of
what are referred to as solutions stacks

. Solutions stacks represent combination s of compatible document server s , W eb s erver s , application s , and database software used for W eb

- based software development. Examples of popular
solution s stacks include:

 LAMP (Linux, Apache, MySQL, Perl/PHP/Python)

 WINS (Windows, IIS, .NET, MSSQL)

 Oracle (Oracle Integrated Stack)

.

The specific technology chosen would depend more on the specialty of the selected implementer rather than the software’s specific cost or capabilities.

Either of the savings tool models will require custom database development to store information. In the data

- driven

m odel, these would include database tables to store savings values and all relevant metadata. In the algorithmic approach, tables would store formula information and all parameter data available for user selection. Administrative interfaces for loading, man aging, and updating the database would be required in either case.

Cost

We expect the total costs for development and launch of a comprehensive measure resource to range between approximately $1.1 million and $1.7 million, with between $560,000 and $840,00 0 for ongoing annual administration and maintenance ( Table 15 and Table 16 ). These estimates assume a database tool would be developed from scratch, independently of any pa rticular consulting/software development firm’s existing software that might be adapted. The content creation component also assumes roughly two employee

- years. These costs are for the development of a national database. If regional databases were develope d instead, the technological costs for the first database would be the same, but subsequent databases would have fewer resource needs for requirements gathering and specification and design. The content development would be less, roughly 0.5 employee

- years per region.J une 2011 www.seeaction.energy.gov 43 

### Table 15 . Estimated Development Labor and Cost (Comprehensive DS&A Database)

Task Estimated Employee

- Years



### Requirements Gathering 0.20

Specification and Design 0.25

Technical Implementation 0.75

Content Develop ment 2. 0

Testing 0.20

Release and Maintenance 0.25

Total Development Employee

- Years 3.6 5
Total Development Cost ($150/hr)

Most Likely $ 1,138 ,800

Low $ 854 ,100

High $1, 708,200J une 2011 www.seeaction.energy.gov 44 

### Table 16 . Annual Recurring Labor and Costs ( Comprehensive DS&A Database)
T ask Estimated FTE



### Content Management 0.75

Site Administration 0.50

Site Enhancement 0.50

Total Recurring FTE 1.75

Hosting ( third

- party) $12,000

Total Recurring Cost ($150/hr)

Most Likely $558,000

Low $418,500

High $ 837,000

Implementation

Implementation issues regard ing available ex ante savings values and the possibility of creating a national database
were discussed above. Another significant issue with the algorithms and ex ante savings values is the frequency wit h
which the underlying data can change, largely due to updates in codes and standards. Many experts we interviewed reported this would pose a significant challenge. A strategy for providing updates would need to be determined,
with consideration to keeping past versions available

, since multiyear programs and evaluation efforts require a consistent savings resource. Maintaining multiple, concurrent versions of the same data would be difficult, costly, and potentially confusing to users.

A barrier to using t he data

- driven approach is in providing sufficient documentation on the sources of each datapoint to gain user trust that data are accurate. A separate document database may need to exist along with savings data to house the original source material.



## Concl usion

This report presents the results of a scoping study to research the various options for the content development, design, and administration of two potential national databases. The first is a national database of standardized, region

- specific, ex ant e algorithms, and associated stipulated savings estimates for conventional electric and natural

gas energy efficiency measures (a national DS&A database). The second is a document database of EM&V
documents.

Cadmus reviewed currently available resources a nd interviewed several stakeholders to assess the needs and challenges for these databases, according to the views of energy efficiency industry experts and new practitioners. The study results indicate a national DS&A database would be very difficult to p roduce, with significant regulatory and other barriers. An EM&V documents database would be more tractable and valuable to the energy efficiency community. However, ongoing maintenance and upkeep of such a database could require significant resources.J une 2011 www.seeaction.energy.gov 45

Cad mus recommends initially developing a focused resource (or starter kit), providing links and including key documents to guide practitioners new to the energy efficiency field. This focused resource could be built upon to develop a comprehensive resource of EM&V documents if sufficient interest and funding remains available.This information was developed as a p roduct of the State and Local Energy Efficien cy Action Network (SEE Action), facilitated by the U.S.

Department of Energy/U.S. Environmental Protection Agency. Content does not imply an endorsement by the individuals or organizations that are part of SEE A ction working groups, or reflect the views, policies, or otherwise of the federal government.