# STATE OF CALIFORNIA

PUBLIC UTILITIES COMMISSION

505 Van Ness Avenue San Francisco, California 94102



### APRIL 2006

Prepared for the 

### California Public Utilities Commission

by 

### The TecMarket Works Team

Under Contract with and Directed by the CPUC’s Energy Division, and with guidance from Joint Staf f 

# California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements fo r Evaluation Professionals

# California Energy Efficiency Evaluation Protocols: Technical, Methodological and Reporting Requirements for Evaluation Professionals

{a.k.a. Evaluators’ Protocols}

Prepared under direction of the Energy Division , with the guidance by Joint Staff , for the 

# California Public Utilities Commission

APRIL 2006

Submitted by



### Nick Hall, Johna Roth, Carmen Best
TecMarket Works

TecMarket Business Center 165 West Netherwood Road, Second Floor, Suite A Oregon, WI 53575 NPHall@TecMarket.net
608 835 8855

And sub-contractors


### Sharyn Barata
Opinion Dynamics, Irvine, California



### Pete Jacobs
BuildingMetrics, Inc., Boulder, Colorado


### Ken Keating, Ph.D.
Energy Program Consultant, Portland, Oregon



### Steve Kromer

RCx Services, Oakland, California


### Lori Megdal, Ph.D.
Megdal & Associates, Acton, Massachusetts



### Jane Peters, Ph.D.
Research Into Action, Portland, Oregon


### Richard Ridge, Ph.D.
Ridge & Associates, Alameda, California



### Francis Trottier
Francis Trottier Consulting, Penn Valley, California



### Ed Vine, Ph.D.
Energy Program Consultant, Berkeley, CaliforniaEvaluators’ Protocols AcknowledgementsEvaluators’ Protocols Acknowledgements

CPUC i TecMarket Works Team 

## Acknowledgements

The authors wish to acknowledge and express our appreciation to th e many individuals who contributed to the development of the California Evaluation Prot ocols. Without the support and
assistance of these individuals this effo rt would not have been possible.

The Joint Staff (California Public Utilities Commission and the Califor nia Energy Commission) provided considerable Protocol development guidance and conducted multiple rounds of reviews of all sections of the Protocols. These individuals and their affiliations are the following:

• Ariana Merlino, Project Manager, Ener gy Division, California Public Utilities
Commission

• Mike Messenger, Calif ornia Energy Commission

Appreciation is also extended to the Administrative Law Judge, Meg Gottstein, who ordered the development of the Protocols and who provided instructive guidance and policy direction along the way.

In addition to the oversight and guidance provided by the above i ndividuals, others within the Energy Division of the California Public Ut ilities Commission and the California Energy Commission provided valuable c ontributions and support. For these efforts we thank the following individuals:

• Nancy Jenkins, California Energy Commission

• Tim Drew, Energy Division, Californi a Public Utilities Commission

• Zenaida Tapawan-Conway, Energy Division, California Public Utilities Commission

• Peter Lai, Energy Division, Califor nia Public Utilities Commission

• Nora Gatchalian, Energy Division, Calif ornia Public Utilities Commission

• Jeorge Tagnipes, Energy Division, Ca lifornia Public Ut ilities Commission

• Sylvia Bender, California Energy Commission

We also wish to thank the Calif ornia investor owned utilities a nd their program management and evaluation staff who have attended workshops an d provided both written and verbal comments during the Protocol development process. And we wish to thank the public representatives who attended the workshops and provided verbal a nd written comments. All of these combined efforts helped move the development of the Prot ocols to a successful comp letion in a very short period of time.

Lastly, we wish to thank the TecMarket Works Protocol Project Team who under direction from the ALJ and the Joint Staff, and with useful comments from the IOUs and the public, took the Protocols from concept to completion under the oversight of Joint Staff. This team was made up of the following individuals:

• Nick Hall, Johna Roth, Carmen Best, TecMarket WorksEvaluators’ Protocols Acknowledgements

CPUC ii TecMarket Works Team • Sharyn Barata, Opinion Dynamics Corporation

• Pete Jacobs, Building Metrics Inc.

• Ken Keating, Ken Keating and Associates

• Steve Kromer, RCx Services

• Lori Megdal, Megdal & Associates

• Jane Peters, and Marjorie Mc Rae, Research Into Action

• Rick Ridge, Richard Ridge and Associates

• Ed Vine, Edward Vine and Associates

• Francis Trottier, Francis Trottier ConsultingEvaluators’ Protocols Table of Contents

CPUC iii TecMarket Works Team 

## Table of Contents

ACKNOWLEDGEMENTS .......................................................................................................... I

TABLE OF TABLES................................................................................................................

.. IX

TABLE OF FIGURES...............................................................................................................

. XI

CALIFORNIA ENERGY EFFICIENCY EV ALUATION PROTOCOLS: TECHNICAL, METHODOLOGICAL AND REPORTING REQUIREMENTS FOR EVALUATION PROFESSIONALS {A.K.A. EV ALUATORS’ PROTOCOLS}............................................... 1

I NTRODUCTION

......................................................................................................................... 1

H OW T HE P ROTOCOLS W ERE D EVELOPED

............................................................................... 5

H OW THE P ROTOCOLS W ORK T OGETHER

................................................................................. 6

H OW THE P ROTOCOLS M EET CPUC G OALS

............................................................................. 7

U SE OF THE E VALUATION R ESULTS TO D OCUMENT E NERGY S AVINGS AND D EMAND I MPACTS 9

T HE E VALUATION I DENTIFICATION AND P LANNING P ROCESS

.................................................. 9

E VALUATION R IGOR AND B UDGETS

....................................................................................... 13 Impact Evaluations.............................................................................................................

........................... 13

Process Ev aluatio ns ............................................................................................................

.......................... 14 Market Effect s Evalua tions .....................................................................................................

...................... 14 Codes and Standards and Emerging Technology Prog ram Evaluations ...................................................... 14
Evaluation Budgets .............................................................................................................

.......................... 15

R ECOMMENDATIONS FOR U SING THE P ROTOCOLS

.................................................................. 15

T HE D ETAILED E VALUATION W ORK P LAN

............................................................................. 15

C ONFIDENTIALITY I SSUES

....................................................................................................... 17

C ONTACTING THE C USTOMER

................................................................................................. 18
IMPACT EVALUATION PROTOCOL .................................................................................. 19

I NTRODUCTION

....................................................................................................................... 19

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................... 21

O VERVIEW OF THE P ROTOCOL

................................................................................................ 21 Protocol Types.................................................................................................................

.............................. 21 Rigor..........................................................................................................................

.................................... 22 Key Metrics, Inputs and Outputs................................................................................................

................... 23

E NERGY AND D EMAND I MPACT P ROTOCOLS

.......................................................................... 25
Gross Energy Im pact Protocol...................................................................................................

................... 25 Gross Demand Impact Protocol...................................................................................................

................. 32 Participant Net Im pact Protocol ................................................................................................

................... 36

I NDIRECT I MPACT E VALUATION P ROTOCOL

........................................................................... 40

G UIDANCE ON S KILLS R EQUIRED TO C ONDUCT I MPACT E VALUATIONS

................................. 45

S UMMARY OF P ROTOCOL -D RIVEN I MPACT E VALUATION A CTIVITIES

.................................... 46 MEASUREMENT AND VERIFICATION (M&V) PROTOCOL ........................................ 49

I NTRODUCTION

....................................................................................................................... 49

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................... 50Evaluators’ Protocols Table of Contents

CPUC iv TecMarket Works Team O VERVIEW OF THE P ROTOCOL

................................................................................................ 50

M&V Framework & Language .......................................................................................................

.............. 51 Relationship of the M&V Prot ocol to Othe r Protocols ............................................................................

..... 51 Key Metrics, Inputs and Outputs................................................................................................

................... 52

S ITE -S PECIFIC M&V P LAN

..................................................................................................... 54

M&V R IGOR L EVELS

............................................................................................................. 56

M EASURE I NSTALLATION V ERIFICATION

................................................................................ 56

Measure Existence..............................................................................................................

........................... 56
Installatio n Qua lity ...........................................................................................................

............................ 56 Correct Operation and Poten tial to Genera te Savings ............................................................................

..... 57

M&V P ROTOCOL FOR B ASIC L EVEL OF R IGOR

....................................................................... 57
IPMVP Option...................................................................................................................

............................ 57 Sources of Stip ulated Data.....................................................................................................

....................... 58
Baseline De finitio n............................................................................................................

............................ 58 Monitoring Strate gy and Duration...............................................................................................

................. 58 Weather Adjustments............................................................................................................

......................... 58

M&V P ROTOCOL FOR E NHANCED L EVEL OF R IGOR

............................................................... 58
IPMVP Option...................................................................................................................

............................ 59 Sources of Stip ulated Data.....................................................................................................

....................... 59
Baseline De finitio n............................................................................................................

............................ 60 Monitoring Strate gy and Duration...............................................................................................

................. 60 Weather Adjustments............................................................................................................

......................... 60
Calibration Targets............................................................................................................

........................... 60
Additional Pr ovisions..........................................................................................................

.......................... 61

M&V A PPROACH E XAMPLES

................................................................................................. 61

O VERALL R ESULTS R EPORTING

.............................................................................................. 62

S AMPLING S TRATEGIES

.......................................................................................................... 63

S KILLS R EQUIRED FOR M&V ................................................................................................. 63

S UMMARY OF P ROTOCOL -D RIVEN M&V A CTIVITIES

............................................................ 63
EMERGING TECHNOLOGIES PROTOCOL ...................................................................... 65

I NTRODUCTION

....................................................................................................................... 65 Audience and Resp onsible Actors ................................................................................................

................. 66 Key Metrics, Inputs , and Outputs...............................................................................................

................... 66 Evaluation Planning............................................................................................................

.......................... 67 A Sample of Available ETP Evaluatio n Method s ...................................................................................

....... 68
Protocols Re quirem ents .........................................................................................................

....................... 69 Integration of Re sults .........................................................................................................

........................... 75 Reporting of Results ...........................................................................................................

........................... 75 Summary........................................................................................................................

................................ 76 Summary of Protocol-Driven Emergi ng Technology Evaluation Activities .................................................. 76 References .....................................................................................................................

................................ 77 CODES AND STANDARDS AND COMP LIANCE ENHANCEMENT EVALUATION
PROTOCOL.......................................................................................................................

......... 81

I NTRODUCTION

....................................................................................................................... 81 Audience and Resp onsible Actors ................................................................................................

................. 82 Key Inputs, and Outputs ........................................................................................................

........................ 83
Evaluation Methods.............................................................................................................

.......................... 84 Evaluation Planning............................................................................................................

.......................... 84 Technology-Specific Code and Standard Change Theory............................................................................

. 85
Evaluation Approach............................................................................................................

......................... 88Evaluators’ Protocols Table of Contents

CPUC v TecMarket Works Team Summary........................................................................................................................

................................ 98 Summary of Protocol-Driven Codes and Standards Eval uation Activities ................................................... 99

C ODE C OMPLIANCE E NHANCEMENT P ROGRAMS

.................................................................. 100 Definition of a Code Complia nce Enhancemen t Program .......................................................................... 100
What this Protocol is Designed To Do ...........................................................................................

............. 100 Joint Staff Re sponsib ilities

................................................................................................... ....................... 100

D RAFT E VALUATION P LAN

................................................................................................... 101 Program Theory Revi ew and Assessment...........................................................................................

......... 101 Pre-Program Co mpliance Rate....................................................................................................

............... 101 Post-Program Compliance Rate...................................................................................................

............... 102 Adjustment For Naturally Oc curring Compliance Change......................................................................... 102 Net Program-Induced Compliance Change ..........................................................................................

...... 102 Assessment of En ergy Savings...................................................................................................

.................. 103 Recommendations for Program Changes............................................................................................

........ 103 Cost Effectiven ess Assessment..................................................................................................

................... 103

R EPORTING OF E VALUATION R ESULTS

................................................................................. 103

R EFERENCES

......................................................................................................................... 104
EFFECTIVE USEFUL LIFE EVALUATI ON PROTOCOL (RETENTION AND DEGRADATION)...................................................................................................................

.. 105

I NTRODUCTION

..................................................................................................................... 105

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................. 107

O VERVIEW OF THE P ROTOCOL

.............................................................................................. 108 Protocol Types.................................................................................................................

............................ 108 Rigor..........................................................................................................................

.................................. 109 Key Metrics, Inputs , and Outputs...............................................................................................

................. 110

R ETENTION S TUDY P ROTOCOL

............................................................................................. 111

D EGRADATION S TUDY P ROTOCOL

........................................................................................ 116

E FFECTIVE U SEFUL L IFE A NALYSIS P ROTOCOL

.................................................................... 119

R EPORTING R EQUIREMENTS

................................................................................................. 126

S TUDY S ELECTION AND T IMING

........................................................................................... 127

G UIDANCE ON S KILLS R EQUIRED TO C ONDUCT R ETENTION , EUL, AND T ECHNICAL
D EGRADATION E VALUATIONS

.............................................................................................. 128 Summary of Protocol-Driven Impact Evaluat ion Activ ities ........................................................................ 130 PROCESS EVALUATI ON PROTOCOL .............................................................................. 131

I NTRODUCTION

..................................................................................................................... 131

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................. 132

O VERVIEW OF THE P ROTOCOL

.............................................................................................. 132

P ROCESS E VALUATION P LANNING

........................................................................................ 133

A NNUAL P ROCESS E VALUATION P LANNING M EETING

......................................................... 134

R ECOMMENDATIONS FOR C HANGE

....................................................................................... 135

K EY I SSUES AND I NFORMATION C OVERED

........................................................................... 135

P ROCESS E VALUATION E FFORTS

.......................................................................................... 136 Program-Specific Proce ss Evaluatio n Plans ......................................................................................

........ 137
Data Collection and A ssessment Efforts .........................................................................................

............ 137 Conducting Investi gative Efforts ...............................................................................................

.................. 138Evaluators’ Protocols Table of Contents

CPUC vi TecMarket Works Team Independence...................................................................................................................

............................ 140 Selection of Evalua tion Contra ctors ............................................................................................

............... 140

S KILLS R EQUIRED FOR C ONDUCTING P ROCESS E VALUATIONS

............................................. 140
MARKET EFFECTS EVAL UATION PROTOCOL............................................................ 143

I NTRODUCTION

..................................................................................................................... 143 Overview of the Mar ket Effects Protocol ........................................................................................

............ 143 The Market Effects Protoc ol and Other Protocols................................................................................

...... 146 Key Market Effects I nputs and Outputs..........................................................................................

............. 147

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................. 148

S TEPS IN C ONDUCTING M ARKET E FFECTS E VALUATIONS

.................................................... 148 Scoping Study ..................................................................................................................

............................ 149 Collection of Baseline and Longitudinal Indicators.............................................................................

....... 154 Analysis of Ma rket E ffect s.....................................................................................................

...................... 155 Reporting......................................................................................................................

............................... 158

G UIDANCE ON S KILLS R EQUIRED TO C ONDUCT M ARKET E FFECTS E VALUATIONS

............... 159

C ONSIDERATIONS FOR C ONDUCTING A M ARKET E FFECTS E VALUATION

.............................. 159

S UMMARY OF P ROTOCOL -D RIVEN M ARKET E FFECTS E VALUATION A CTIVITIES

.................. 160
SAMPLING AND UNCERTAINTY PROTOCOL............................................................... 163

I NTRODUCTION

..................................................................................................................... 163

P RECISION : G ROSS AND N ET I MPACT , M EASUREMENT AND V ERIFICATION

, AND V ERIFICATION A CTIVITIES

........................................................................................................................... 164

D EVELOPMENT OF THE E VALUATION S TUDY W ORK P LAN

................................................... 167

P ROCESS E VALUATIONS

....................................................................................................... 169

M ARKET E FFECTS

................................................................................................................. 170

S YSTEM L EARNING

............................................................................................................... 170

A CCEPTABLE S AMPLING M ETHODS

...................................................................................... 171

S KILLS R EQUIRED FOR S AMPLING & U NCERTAINTY

............................................................ 171

A UDIENCE AND R ESPONSIBLE A CTORS

................................................................................. 171

K EY M ETRICS AND I NFORMATION C OVERED

........................................................................ 172 Sample Size and Preci sion ......................................................................................................

.................... 172 Validity and Rese arch Design ...................................................................................................

.................. 172 Accuracy.......................................................................................................................

............................... 173 Summary of Sampling and Uncertainty Protocol...................................................................................

..... 175 EVALUATION REPORTING PROTOCOL ........................................................................ 177

I NTRODUCTION

..................................................................................................................... 177 Report Deli very Da tes..........................................................................................................

....................... 177 Common Evaluation Repo rting Requi rements .......................................................................................

..... 177 Evaluation Type Specific Reporting Re quirements ................................................................................

..... 183

S AMPLE R EPORTING T ABLES

................................................................................................ 195
EVALUATION SUPPORT INFORMATION NEEDED FROM ADMINISTRATORS.. 205

Storage and Disposal of Customer In formation Used in the Eval uation .................................................... 211 APPENDIX A. MEASURE-LEVEL M&V RESULTS REPORTING REQUIREMENTS

............................................................................................................................... ...................... 213

APPENDIX B. GLOSSARY OF TERMS ............................................................................. 217

APPENDIX C. PERFORM ANCE BASIS METRICS ......................................................... 247Evaluators’ Protocols Table of Contents

CPUC vii TecMarket Works Team APPENDIX D. A PRIMER FOR USING POWER ANALYSIS TO DETERMINE
SAMPLE SIZES ...................................................................................................................

.... 249

B ASICS OF P OWER A NALYSIS AND THE P ROTOCOLS

............................................................. 250

E XAMPLE OF V ARYING P ARAMETERS AND E STIMATING R EQUIRED S AMPLE S IZE FOR S URVIVAL A NALYSIS THROUGH P OWER A NALYSIS

.............................................................. 251

R EFERENCES

......................................................................................................................... 254
APPENDIX E. SUMMARY TABLES FOR ALL PROTOCOLS ....................................... 255

Summary of Protocol-Driven Impact Evaluat ion Activ ities ........................................................................ 256 Summary of Protocol-D riven M&V Activitie s......................................................................................

....... 262 Emerging Technology ............................................................................................................

..................... 264
Codes and St andards............................................................................................................

....................... 265 Effective Us eful Life ..........................................................................................................

.......................... 267 Summary of Protocol-Driven Market Effects Evalua tion Acti vities............................................................ 270 Sampling and Un certainty.......................................................................................................

.................... 274Evaluators’ Protocols Table of Contents

CPUC viii TecMarket Works TeamEvaluators’ Protocols Table of Contents

CPUC ix TecMarket Works Team 

## Table of Tables

Table 1. Required Protocols for Gross Energy Evaluation.......................................................... 26 Table 2. Required Protocols for Gross Demand Evaluation........................................................ 33 Table 3. Required Protocols for Pa rticipant Net Impact Evaluation ........................................... 36 Table 4. Required Protocols for Indirect Impact Evaluation ....................................................... 41 Table 5. Summary of M&V Protocol for Basic Level of Rigor .................................................. 57 Table 6. Summary of M&V Protocol for Enhanced Level of Rigor ........................................... 59 Table 7. Model Calibration Targets............................................................................................

. 60 Table 8. Programs Compliant with ASHR AE Standard 140-2001 (Partial List) ........................ 61 Table 9. Example IPMVP Options by Measure Type and Rigor Level ...................................... 61 Table 10. Sample of Availa ble ETP Evaluation Methods ........................................................... 68 Table 11. Required Protocols for Measure Retention Study ..................................................... 113 Table 12. Required Protocols for Degradation Study................................................................ 118 Table 13. Required Protocols for EUL Analysis Studies .......................................................... 122 Table 14. Required Protocols for Market Effects Evaluation Scoping Studies......................... 150 Table 15. Required Protocol for Market Theory and Logic Models ......................................... 151 Table 16. Types of Market Interventions and Associated Possible Indicators .......................... 153 Table 17. Required Protocol for Market Effects Evaluation Indicator Studies ......................... 155 Table 18. Required Protocol for Preponderance of Evidence Approach to Causal Attribution
Estimation .....................................................................................................................

...... 156 Table 19. Required Protocols for Gross Impacts....................................................................... 165 Table 20. Required Protocols for Net Impacts........................................................................... 166 Table 21. Required Protocols for Measur e-level Measurement and Verification ..................... 167 Table 22. Required Protocols for Samp ling of Measures Within a Site.................................... 167 Table 23. Required Protocols for Verification........................................................................... 167 Table 24. Measure-Level Imp act Reporting Requirements....................................................... 213 Table 25. Sample Sizes as a Function of Alpha and Power....................................................... 253Evaluators’ Protocols Table of Contents

CPUC x TecMarket Works TeamEvaluators’ Protocols Table of Contents

CPUC xi TecMarket Works Team 

## Table of Figures

Figure 1. Operational Overview of How the Protocols Relate to Each Other............................... 7 Figure 2. The Program Evaluation Planning Pr ocess for Programs, Program Components and Program-Covered Technologies ........................................................................................... 12 Figure 3. The Market Effects Evaluation Planning Process ........................................................ 13 Figure 4. Required Protocols for Direct Im pact and Indirect Impact Evaluations ...................... 23 Figure 5. Potential Alternative Behavioral Impact Paths............................................................. 43 Figure 6. Measurement & Verification Information Flow Diagram............................................ 53 Figure 7. An Example of How Findings Acro ss the Three Types of Studies Would Work Together for Persistence Evaluations.................................................................................. 109 Figure 8. Protocols and Rigor Levels for EUL Evaluations ...................................................... 110 Figure 9. Cumulative Logistic Function .................................................................................... 123 Figure 10. Logistic Survival Function with EUL=15 and b=0.2 ................................................ 124 Figure 11. Sources of Energy Effi ciency Changes in the Market.............................................. 145Evaluators’ Protocols Introduction

CPUC xii TecMarket Works TeamEvaluators’ Protocols Introduction

CPUC 1 TecMarket Works Team 

## California Energy Efficiency Evaluation Protocols: Technical, Methodological and Reporting Requirements for Evaluation Professionals

{a.k.a. Evaluators’ Protocols}



### Introduction

This chapter presents and describes the Calif ornia Energy Efficiency Evaluation Protocols: Technical, Methodological and Reporting Requirements for Ev aluation Professionals (a.k.a. Evaluators’ Protocols, referred to hereafter collectively as the Protocols and individually as Protocol) that are designed to meet California’s evaluation objectives.

This document is to be used to guide the effo rts associated with conducting evaluations of California’s energy efficiency programs and program portfolios launched after December 31, 2005. The Protocols are the primar y guidance tools policy makers wi ll use to plan and structure
evaluation efforts and that staff of the Californi a Public Utilities Comm ission’s Energy Division (CPUC-ED) and the California Ener gy Commission (CEC) (collectively the Joint Staff), and the portfolio (or program) administrators (Adminis trators) will use to plan and oversee the completion of evaluation efforts. The Protoc ols are also the primary guidance documents evaluation contractors will use to design and conduct evaluations for programs implemented after December 31, 2005. This chapter provides an introduction to, and overall guidance for, the use of specific Protocols presented in later chapters of this document.

The Protocols are significantly grounded in the California Evaluation Framework of June 2004 1
( Evaluation Framework) . The Protocols reference the Evaluation Framework and other documents that provide exampl es of applicable methods. The requirements for conducting evaluation studies, however, are always those stated in the Protocols, which take precedence over other evaluation guidance documents, unless otherw ise approved or required by the CPUC. That is, these Protocols are the primary ev aluation guidance documents for all types of evaluations presented in these Pr otocols, however this is not to be construed as limiting the ability of the CPUC or the Joint Staff to evalua te items in addition to or beyond those identified in these Protocols or to use evaluation processe s and procedures beyond those presented in these
Protocols. While these Protoc ols are the key guiding documents for the program evaluation efforts, the CPUC and the Joint Staff reserve the right to utilize additional methodologies or approach if they better meet the CPUC’s evaluation objectives and when it serves to provide reliable evaluation results using the most cost-efficient approaches available. In addition, the
Protocols should be considered a “living” document that may n eed to be updated and revised from time to time as standard evaluation approaches evolve and as Joint St aff and Administrators gain experience using the Protocols. The CPUC will determine when an update is necessary and what process will be used to complete any update s that the agency deems necessary. Protocol users should always confirm that they are refe rring to the most recently CPUC-approved and - adopted version, which can be found on the CPUC website.

1 TecMarket Works, The California Evaluation Framework (Southern California Edison Company, 2004). The report can be obtained on the CALMAC Web site at: http://www. calmac.org/search.asp. Enter “California Evaluation Framework” and download the 500-page reference document as an Adobe .pdf file.Evaluators’ Protocols Introduction

CPUC 2 TecMarket Works Team Most of the Protocols are designe d to function within an evaluati on planning process that focuses on the evaluation needs within a given program cycl e. This planning pro cess is described in a other documents adopted by the ALJ and the CPUC, and most directly at part of what are known
as the Process Protocols.

The Protocols cover several types of evaluation efforts. The evaluation types covered include the following: direct and indirect impact {incl uding the associated measurement and verification approaches (M&V)}, market effects, emergi ng technology, codes and standards and process evaluations. In addition, the Prot ocols provide specific guidelines for conducting effective useful life studies and how evaluation samples should be selected. The primary goal of this document is to specify minimum acceptable evaluation approaches a nd the operational environments in which evaluations are conducted. The primary purpos e of the Protocols is to establish a uniform approach for:

• Conducting robust and cost-efficient en ergy efficiency evaluation studies;

• Documenting ex-post evaluation-confirmed (i .e. realized) energy efficiency program and portfolio effects;

• Supporting the performance bases for j udging energy efficiency program and portfolio achievements; and

• Providing data to support en ergy efficiency program and portfolio cost-effectiveness

assessments.

The Protocols may have other uses such as providing support for improving ex-ante energy and demand savings estimates.

This document includes a separate Protocol for each of the following categories:

• Impact Evaluation - Direct and Indirect Effects

• Measurement and Verification

• Process Evaluation

• Market Effects Evaluation

• Codes and Standards Program Evaluation

• Emerging Technology Program Evaluation

• Sampling and Uncertainty Protocol (for use in determining evaluation sampling approaches) Reporting Protocol (to guide ev aluation data collec tion and reporting)

• Effective Useful Life Prot ocol (used to establish th e period over which energy savings can be relied upon)

The Protocols also include information on the ty pe of evaluation-related information and support needed from program administrators and implemente rs in order to conduct the evaluation efforts. The purpose of each of the listed Protocols is described below.

Impact Evaluation Protocol: The Impact Evaluation Protoc ol prescribes the minimum allowable methods to meet a specified level of ri gor that will be used to measure and document the program or program component impacts achieved as a result of implementing energy efficiency programs and program portfolios. Impact evaluations estimate net changes inEvaluators’ Protocols Introduction

CPUC 3 TecMarket Works Team electricity usage, electricity demand, therm usage a nd/or behavioral impacts that are expected to produce changes in energy use and demand. Impact evaluations are limited to addressing the direct or indirect energy impacts of the program on participants, including participant spillover impacts. However, while the Protocols provide for the assessment of partic ipant spillover, these results are not to be counted toward program or portfolio energy savings goal accomplishments, and as such are to be distinctly and se parately identified in any impact reporting. 2 The impact evaluation studies are also not expected to docum ent program influences on the operations of a market or the program's impacts on non-particip ants. Program-induced changes that affect nonparticipants or the way a market operates ar e addressed in the Market Effects Evaluation Protocol. Results from the impact evaluations will support a cost-e ffectiveness assessment at the program and portfolio level.

Measurement and Verification (M&V) Protocol: The M&V Protocol is designed to prescribe
how field measurements and data collection wi ll be conducted to support impact evaluations, updates to ex-ante measure savings es timates and process evaluations.

Process Evaluation Protocol: The Process Evaluation Protoc ol is designed to support Administrator (i.e. Investor Owned Utility or IOU) efforts to conduct evaluations that both document program operations and provide the basis for improving the operations or costeffectiveness of the programs offered within the portfolio.

Market Effects Evaluation Protocol: The Market Effects Evaluati on Protocol is designed to guide evaluations conducted to document the vari ous market changes that affect the way energy
is used within a market and estimate the en ergy and demand savings associated with those changes that are induced by se ts of program or portfolio interventions in a market.

Codes and Standards Program Evaluation Protocol : The Codes and Standards Program Evaluation Protocol is designe d to guide evaluation approaches for codes and standards

programs.

Emerging Technology Program Evaluation Protocol : The Emerging Technology Program Evaluation Protocol is designe d to guide evaluation approaches for emerging technology

programs.

Effective Useful Life Protocol : The Effective Useful Life Pr otocol is designed to guide evaluation approaches for establishing the effectiv e useful life of progr am measures, including approaches for evaluating measure retention and technical degradation of measure performance.
The effective useful life of a m easure is the period of time over which program-induced energy impacts can be relied upon.

2 The Protocols prescribe minimum requirements for how to conduct and report evaluations. The Performance Basis

Protocol takes precedence with regard to including savi ngs toward program or portfolio goals and performance measurement. The most recent CPUC decision will always take precedence and be used for the interpretation and application of t he Protocols.Evaluators’ Protocols Introduction

CPUC 4 TecMarket Works Team Sampling and Uncertainty Protocol : The Sampling and Uncertainty Protocol is designed to prescribe the approach for sele cting samples and conducting rese arch design and analysis in order to identify, mitigate and minimize bias in support of the Protocols identified above.

Reporting Protocols: The Reporting Protocol prescribes the way in which evaluation reports are to be delivered and the way information is to be presented in those reports.

Evaluation Support Information Needed from Administrators: The Protocol document also includes a chapter on the types of information Administrators shall provide to contractors conducting evaluation studies covered by the Protocols.

The four primary types of Evalua tion Protocols that cover the ma jority of California’s program offerings are the Impact, M&V, Market Effects and Process Protocols. These are supported by the Sampling and Uncertainty Protocol. Howeve r, there are two types of programs that are different enough in their scope and intended resu lts that they require a separate Evaluation Protocol (Codes and Standards and Emerging Tec hnology). As such, two Evaluation Protocols are directed to a specific t ype of program (Codes and Sta ndards and Emerging Technology), while the remaining Protocols eith er operate to establish a minimum set of allowable methods for a specific type of evaluation or in support thereof. Any program, program component or set of programs could be included within each of these types of evaluations. Th e difference lies not in which programs are eligible for which types of evaluations, but in the purpose of and outputs from each of these evaluation types.

The outputs from an impact (and its associated M&V efforts) evaluation are program or program component net energy, demand or behavioral impact s from program participation. Those from a market effects evaluation are energy and demand impacts created by market changes caused by a program or set of programs. While a proce ss evaluation produces the documentation and assessment of program processes, and recomme ndations to improve them. A program could easily be included in all three types of evalua tions. For example, a single program of great significance with respect to the overall portfolio might be dire ctly assessed using impact and process evaluations and also be included in a market effects evaluation for all programs operating in a given ma rket sector.

While it is important to know what is in these Pr otocols (above), it is also important to know
what is not included in these Pr otocols. These Protocols do not cover the evaluati on or research approaches for the following types of programs, efforts or activities:

• Low-income program evaluations;

• Market research for program design, planning or operations;

• Technical, market or other types of potentials studies;

• Meta-evaluations or comparative st udies using evaluation study results;

• Demand response programs;

• Renewable energy programs;

• On-site or distributed generation or combined heat and power programs;

• Green house gas or pollution reduction studies;

• Cost-effectiveness methods, approaches or procedures;Evaluators’ Protocols Introduction

CPUC 5 TecMarket Works Team • Forecasting methods, approaches or procedures; and

• Public Interest Energy Research (PIER) evaluation efforts.

While it is expected that the Pr otocols will need to be updated from time to time, it is also expected that new Protocols may need to be adde d to this document as the need for different types of information evolves. For example, California may need to establish Protocols for crediting greenhouse gas reductions resulting from the energy effi ciency program portfolios or for addressing demand response programs that are currently outside the sc ope of the Protocols.



### How The Protocols Were Developed

The Protocols were develope d over two different but overl apping three-month timelines involving a number of activities, including presentations to the pub lic and the rece ipt of public comments and recommendations. The Impact, M& V, Process, Market Effects, Sampling and Reporting Protocols were develope d first, and followed by the development of the Codes and Standard, Emerging Technology, and Effective Useful Life Protocols. All of the Protocols were developed using the following approach:

1. The consulting team that the CPUC-ED cont racted to develop th e Protocols (TecMarket Team) assembled and reviewed comments from previous Protocol an d performance basis workshops and comments received during the development of the Evaluation
Framework ;
2. Using the Evaluation Framework , previous comments and discussions with the Joint Staff, draft concept Protocol outlines were developed. These concepts were then discussed within a series of meetings with th e Joint Staff leading to the development of a set of draft concept Protocols;

3. The draft concept Protocols were presented in public workshops. During the workshops, the attending public was requested to comment on the draft concept Protocols. These
comments were recorded and summarized in wo rkshop notes and used to inform Protocol development. At this time, the draft con cept Protocols were also placed on the CPUC website for additional public review. An announcement was sent to the CPUC Energy Efficiency service lists a dvising the public of the work shops and the draft concept Protocol postings. These efforts allowe d both attendees and non-attendees of the workshop to review the draft concept Protocols and provide comments;

4. Following the workshop, the TecMarket Team collected comments from both workshop
attendees and non-attendees. These comments were distributed to and reviewed by the
Joint Staff and the TecMarket Team and used to guide the draft Protocol development efforts;

5. The TecMarket Team developed a set of draf t Protocols under the direction of CPUC-ED
staff and in consultation with the Joint Staff. The draft Protocols were provided to the
Joint Staff for review and comment in order to identify concerns and is sues that needed to
be addressed in the final draft Protocols. Upon reviewing the draft Protocols, the Joint Staff requested modifications to the Protocols;Evaluators’ Protocols Introduction

CPUC 6 TecMarket Works Team 6. The TecMarket Team modified the draft Prot ocols consistent with direction provided by CPUC-ED staff, in consultation with Joint Staff, and provided them to the CPUC-ED project manager for final review and editing;

7. The CPUC-ED project manager submitted the dr aft Protocols to the ALJ for review and
acceptance;

8. The ALJ, in consultation with the CPUC-ED project manager and Joint Staff, reviewed and accepted the final Protocols.

9. The ALJ adopted these Protocols via a Ru ling, per the authority delegated her by the

CPUC.

In addition to the process outlined above, the first set of Protocols developed (Impact, M&V, Process, Market Effects, Sampling and Reportin g) went through an additional round of public review and comment, Joint Staff review a nd commentary, and CPUC-ED project manager approval and editing process before they were provided in final form to the ALJ for review and
acceptance.



### How the Protocols Work Together

The Protocols are designed to support the need for public accountability and oversight, the need for program improvements (especially cost-eff ectiveness improvements) and the documentation of effects from publicly funded or rate-payer funded energy e fficiency programs provided in
California. The individual Prot ocols are designed to work toge ther to achieve these goals.

The Impact Evaluation Protocol is meant to guide the design of evaluations that provide reliable ex-post participant-focused net program imp acts. These net impacts include peak demand (kilowatts (kW) of electricity), energy (kilowatt-hours (kWh) of electricity and therms of natural gas) and behavioral impacts. The Protocol is focused such th at program level impacts can be summed to estimate impacts at the Administrator portfolio level. The Protocol also allows for impact estimates at the program component delivery level (e.g., direct instal l, participant rebate and information distribution) or at the technology level (e.g., CFLs, motors, HVAC tune-up and refrigerators) when the specific evaluati on is meant to acquire these metrics.

The Impact Evaluation Protocol do es not operate in isolation fr om the other Protocols. The M&V Protocol supports impact eval uations and can often serve in a feedback or support role for process evaluations if coordinate d to do so. Similarly, the Sampli ng and Uncertainty Protocol is designed to support impact evaluations, as we ll as M&V, and process and market effects evaluations by assuring that th e sampling designs provide unb iased estimates based on the
information needs associated with each evaluation effort. Fina lly, the Reporting Protocol is designed to support all of the ev aluation activities by detailing th e information that must be reported for each type of evaluation. The entire evaluation process is facilitated by the additional
identification of the information Administrators need to provide the evaluation contractors.

The Protocols, and the evaluations conducted unde r them, support several efforts. For example, many of the evaluation results, es pecially the impact evaluation results and the verification aspects of the M&V Protocol, are designed to support program performance assessment,Evaluators’ Protocols Introduction

CPUC 7 TecMarket Works Team including the performance-based metrics associated with ex-pos t energy savings and verification
of installed measures.

The following diagram provides an overview of how the Protocols work in relationship to each other and the organizations that are responsible for using the Protocols to conduct evaluation
research.

Figure 1. Operational Overview of How the Protocols Relate to Each Other

Note: The Process Evaluation Protocol is a guidance document and is less instructive than the other Protocols that are more prescriptive in design. While the Process Eval uation Protocol does contain re quired reporting and planning activities, it designates that the key decisions on what, when and how to evaluate are the responsibility of the
Administrators.



### How the Protocols Meet CPUC Goals

The primary evaluation-related goal of the CP UC is to assess net program-specific energy impacts or the market level impact s of the portfolio of energy effi ciency services and to compare these results with the assigned energy savings goal s. Similarly, the CPUC must be assured that when an evaluation is conducted it can rely on the findings of that research to accurately reflect the energy benefits available to the citizens of Calif ornia in exchange for the resources spent. As a result, the following goals are incorporated into the operations of the Protocols:

• To identify the annual energy and peak demand impacts associated with each program offered, for which there are exp ected savings, over the period of time the program measures are projected to provide net participant energy impacts. This will almost always be for a longer period of time than the program funding cycle;Evaluators’ Protocols Introduction

CPUC 8 TecMarket Works Team • To identify the annual energy and peak demand impacts associated with major program delivery mechanisms (e.g., direct in stall approaches, incentive and rebate approaches, and education, marketing and outreach programs) over the period of time the program measures are projected to provide net participant energy impacts;

• To estimate the annual energy and peak demand impacts associ ated with each Administrator’s portfolio projected over th e period of time the program services are expected to provide net energy impacts;

• To compare the evaluation results across programs, types of programs (program groups) and program portfolios to assess their relative performance and costeffectiveness;

• To identify under-performing program or program components , so they may be improved or withdrawn from the portfolio of services;

• To understand the potential of programs and program services to cost-effectively increase the supply of energy re sources for California citizens;

• To understand how programs or program operations can be modified to improve their performance and the overall perf ormance of the portfolios;

• To inform future updates to ex-ante ener gy and peak demand savings estimates for program planning purposes;

• Provide timely information to improve program design and selection for future program cycles;

• To be able to tailor the evaluation appr oaches and budgets to meet the need for reliable energy impact and market effect s information while minimizing evaluation costs and reducing risks of making poor efficiency supply decisions; and

• To use an objective and transparent evalua tion process that assesses the impacts from all types of programs that are expected to provide efficiency res ources in California.

The Energy Action Plan, the Energy Efficien cy Policy Manual and other related CPUC documents have established aggressive goals fo r energy efficiency in California. Throughout these guidance documents, it is explicitly recognized that investments in energy savings are uncertain and, hence, carry some risk. Th e guidance documents emphasize the need for “reliable” savings estimates. Efforts to define “reliable” lead to quantif ication. To quantify and manage these risks, one must in clude all relevant and cost-effective sources of information on the performance of the investment and the underlying uncertainty in these data.

To the greatest extent possible, the Joint Staff will seek to allocate evaluation resources to reduce uncertainty in the estimates and evaluations of achieved gross and net savi ngs. The criteria for allocating evaluation resources wi ll be influenced by risk consid erations associated with a program’s designs and operational characteristic s, the expected energy savings, the need to minimize uncertainty in the assessment process and th e cost to quantify and manage these risks. The overarching theme in the management of th e evaluation effort should follow the IQM risk principle: Identify, Quantify and Manage. This principle is based on the recognition that all estimated savings from energy efficiency a nd conservation programs (as well as estimated energy and capacity from traditional supply-si de resources) include some uncertainty and,Evaluators’ Protocols Introduction

CPUC 9 TecMarket Works Team consequently, risk. In the past , planners, evaluators and other staff have often relied on singlepoint savings calculations (e.g., average kWh savi ngs) that were subsequently discounted, based on professional judgment. Risk was not quant ified, therefore it coul d not be effectively managed. By explicitly identifyi ng factors that induce or affect uncertainty and by taking steps towards quantifying that risk, the Joint Staff can make more informed decisions on how to effectively manage the evaluation efforts and reduce the overall risk associated with the efficiency portfolio.



### Use of the Evaluation Results to Document Energy Savings and Demand Impacts

There are several Protocol-guided evaluations that provide net ener gy impacts that will be used to understand program, portfolio and/or statewide energy savings. These are:

• The direct program impact evaluations that document the energy savings associated with the actions taken through program partic ipation, such as when a rebated motor is installed or when a high-efficien cy cooling system is upgraded;

• The indirect program impact evaluations that document the behavioral change, and in some cases the energy savings associated with the behavioral changes made as a result of program activities, such as when training is provided to customers. For example, when a customer installs an en ergy-efficient technol ogy due to exposure to a training program and without any other progra m assistance;

• Evaluations conducted according to the Codes and Standards Program Evaluation Protocol that provide the net energy impacts associated with a code or standard change; and

• The market effects evaluations that document the net effects of one or more programs on the operations of a market and applies energy savings estimates to these programinduced market changes.

All of these impacts will be a ssessed for statewide energy and demand impacts. However, for the purposes of crediting individual programs or Administrator program portfolios with energy impacts, only the first three ca tegories of net energy impacts documented in the evaluations will be counted, and not those from market effects ev aluations. The evaluations in the first three categories will derive program-specific net energy impacts and will be used to sum up to the investor-owned utility (IOU) portfolio impacts and used to derive the statewide program impacts.



### The Evaluation Identification and Planning Process

The program evaluation planning process shall begi n with a high-level assessment of the need to evaluate a program or program component. This assessment will consider, among other factors, the importance of the savings to the portfolio a nd the uncertainty regard ing the ex-ante savings estimates. Based on this assessment, the Join t Staff will decide whether each program or program strategy must comply with the Protocols or whether it will be re quired to comply only with the CPUC’s program reporting requirements.Evaluators’ Protocols Introduction

CPUC 10 TecMarket Works Team For those programs that will receive a Protocol-g uided evaluation, the next series of issues
should be addressed to determine if Protocols that cover multiple types of programs or a program-specific Protocol should be used. Thes e focus on specific types or characteristics of programs. If the program is focused on emer ging technologies, then the Emerging Technology Program Evaluation Protocol must guide the evaluation. If it is a Codes and Standards Program, then the evaluation must be guided by the Code s and Standards Program Evaluation Protocol. Other types of program evaluations will be guid ed by the Protocols designed for a wide variety
of resource and non-re source programs.

The next question to address is whether the prog ram or program strategy is expected to obtain
direct energy or demand savings. Producing savings directly means that the link between the program activity and the savings is clear, straight forward and relatively fast. These types of programs are often referred to as resource or resource acquisition programs. An example of such a program is an incentive program, such as a single -family rebate program, that offers incentives
to residential customers to inst all efficient equipment. For each participant who receives an incentive, there is the clear expectation that ther e will be savings based upon the program’s direct results in obtaining equipment installations. Information and education programs are examples of programs that do not provide such direct impact s. For these programs, there is a more tenuous link between the program activities and any eventual savings. That is, a training program may or may not result in any savings and the savings that are achieved ar e not direct. Savings obtained from providing training services depend upon that program inducing some form of behavior change (such as purchase and installation behavior or participation in a mo re direct efficiency program). This would be indirect savings. If a program is one that provides savings indirectly

, then its evaluation must be guided by the Indirect Impact Evaluation Protocol that explicitly addresses the need to link progr am-induced behavioral changes to eventual energy and demand impacts. Some programs may intend to produce energy savings by providing behavior change information or education for which an impact ev aluation of energy savings is not needed by the
CPUC. These evaluations would follow the Indir ect Impact Evaluation Protocol and quantify behaviors changed or actions ta ken, but not move to the step of allocating energy savings to
those efforts. Joint Staff will determine which Evaluation Protocols to apply to which programs as part of their eval uation planning efforts.

If the program is defined as one that directly produces energy and demand impacts, it must be determined whether it will be guided by the Impact Evaluation Protocol, 3 the M&V Protocol or both. Programs assigned to the M&V Protocol only (not assigned an impact evaluation) will be those for which savings are expected to be relatively small and certain (reliable).

A program with a combination of large and/or uncertain savings must be guided by the Impact
Evaluation Protocol. If such programs do not cover any measures that should be specifically evaluated in order to upd ate the Database for Energy Efficiency Resources (DEER) an impact evaluation at the program or progr am-strategy level (rather than at the technology level) must be planned. However, if the program or program st rategy covers measures that should be evaluated

3 The Impact Evaluation Protocol contains the Indirect Im pact Protocol and three others related to estimation of

direct savings: the Gross Energy Impact, Gross Demand Impact and Participant Net Impact Protocols. The Impact
Evaluation Protocol also often “calls for” the M&V Protocol that provides requirements for M&V-related activities within the impact evaluation methods.Evaluators’ Protocols Introduction

CPUC 11 TecMarket Works Team in order to update DEER, it must determined whether there is a sufficient number of these
measures on which to base a te chnology-level assessment. If s o, evaluators shall develop a measure-level plan to evaluate th ese technologies, as well as plan an impact evaluation at the program or subprogram level.

If there is an insufficient number of a particular measure wi thin a single program, a determination
needs to be made whether there is a suffici ent number of the measure across the program strategies being addresse d within a program group to allow for an evaluation. If so, the evaluator shall develop a measure-level plan to evaluate th ese technologies. Note that measure-level plans should always be nested within the overall impact evaluation for the program or program strategy. Ultimately, the eval uator must account for all the energy and demand impacts for a given program or program strategy.

Figure 2 illustrates th e high-level overview of the program evaluation planning process for programs, program strategies and measures.Evaluators’ Protocols Introduction

CPUC 12 TecMarket Works Team Figure 2. The Program Evaluation Planning Process for Programs, Program Components and Program-Covered Technologies

The procedure is much less structured for dete rmining when to conduct a market effects study. Figure 3 provides a diagram of the related decision process. In this process, the Joint Staff will examine the mix of programs and strategies with in the Administrator portfolios and the markets in which they are operating. Markets will be se lected for Market Effects Evaluation when theEvaluators’ Protocols Introduction

CPUC 13 TecMarket Works Team Joint Staff finds that such an evaluation woul d provide valuable information for directing program improvements and/or for better assessing the complete impacts from the portfolio of programs. Markets may be selected for a Ma rket Effects Evaluation due to a preliminary
assessment that there are substantial investments in that market across programs where potential market effects (including non-participant spillove r) could be measured or need to be tracked
and/or assessed. Markets can also be selected for a Market Effects Evaluation when one or more programs operating in that market are best evaluated at the market -level due to their overlapping nature or overlapping goals to change how a market operates (sometimes called market transformation goals).

Figure 3. The Market Effects Evaluation Planning Process



### Evaluation Rigor and Budgets

The process of setting evaluation pr iorities and budgets for each type of evaluation effort is as
follows:



### Impact Evaluations

For impact studies, the Joint Staff will review the Administrator’s portfolios and programs and establish evaluation groupings. These groupi ngs will consist of multiple programs having common characteristics that provi de evaluation efficiencies in the contracting, supervision and implementation of the evaluation efforts. The gr oupings will typically include similar types of programs (e.g., residential rebate s, commercial rebates, in formation and education, and marketing and outreach) or markets, so that th e evaluation contracts will focus on similar types of programs and program evaluation efforts.

Once the evaluation groups are structured, the Joint Staff will decide which programs (or program components) will receive ve rification-only analysis, direct im pact evaluation or indirect impact evaluation. 4 Each of these will be assigned mi nimum rigor level requirements along with a budget based on a number of factors listed in the Evaluation Framework including:

• The amount of savings expected from each program in the group;

• Whether the programs are expected to grow or shrink in the future;

4 See the Impact Evaluation Protocol her ein for further description of these di fferent types of evaluations and the

various protocols and rigor levels within them.Evaluators’ Protocols Introduction

CPUC 14 TecMarket Works Team • The uncertainty about expected savings and the risk programs pose to achieving portfolio savings goals; and

• How long it has been since the last evaluati on and how much the pr ogram has changed in
the interim.

In setting the level of rigor and the evaluation budgets for th e program groups and the individual programs within each group, the Joint Staff w ill conduct an evaluation needs assessment to assign a level of evaluation rigor to each program or program co mponent. Based on the analysis and criteria listed above, the Jo int Staff will establish appropri ate evaluation budgets across the program evaluation groups. These budget levels w ill be used in the development of Request for Proposals (RFPs) to conduct the evaluation efforts. They will also serve to communicate to
evaluation contractors how evaluation efforts will be structured.

From this effort, the Joint Staff will provide a high-level evaluation plan that presents the overall evaluation goals and approaches selected for the program groups. The plan will be updated annually as the evaluations proceed, as the need for information changes and as adjustments to the evaluation rigor or approach are identified. The plans will be presented to the public for review and comment each year prior to their implementation in a public workshop to solicit
comments and recommendations from interested st akeholders. Once public comments have been obtained, the plan will be fina lized and used to support the ev aluation bidding and contracting

process.

Once an evaluation is launched, the Joint Staff w ill monitor evaluation efforts and their progress to ensure that evaluation approaches meet or exceed the evaluation rigor assigned, in order to
obtain the most reliable evaluation re sults within the available budgets.



### Process Evaluations

For process evaluations, Administ rators are responsible for setti ng evaluation priorities, budgets, evaluation timing and conducting th e evaluation effort. These ac tivities are presented to the CPUC-ED, the CEC and the public via an annual portfolio/program evalua tion plan and a public workshop. See the Process Evaluation Protocol for additional details.



### Market Effects Evaluations

The Joint Staff is responsible for identifying mark ets for which market effects evaluations will be conducted. These studies will be planned and budgeted individually in accordance with the information and data reliability needs of the Joint Staff.



### Codes and Standards and Emerging Technology Program Evaluations

These two program types require evaluations di fferent enough in their goals and objectives, approaches for accomplishing goals and operational characteristics that this document contains Protocols specifically designed for them. While these two types of programs will be evaluated per their respective Protocols, th ey may also have other types of evaluation efforts applied, such as process or market effects evaluations.Evaluators’ Protocols Introduction

CPUC 15 TecMarket Works Team 

### Evaluation Budgets

Each program group evaluation will have a budget cap within which to carry out a variety of
evaluation activities. Efforts to maximize reliabi lity will be carried out within the budget constraints and inevitably invol ve a number of tradeoffs rega rding precision and identifying, mitigating and minimizing potential bias. Addi tional information and guidance on establishing evaluation budgets is provided in the Evaluation Framework 5

. 

### Recommendations for Using the Protocols

The Protocols provide guidance and requirement s for planning and conducting California’s energy efficiency program evaluations. The Pr otocols should be used by the Joint Staff and
Administrators to structure the evaluation process and associated activities. Joint Staff involved in program evaluation efforts should have an ex pert understanding of the Protocols. Evaluation staff within Administrator orga nizations should have the same level of understanding of the Protocols as appropriate to ac tivities in which they have responsibility. All evaluation contractors should be required to have an expert understanding of the Prot ocols that will directly
affect the studies and the methodol ogical approaches they must conduc t. It is also recommended
that all of these involved parties have a working knowle dge of the contents of the Evaluation
Framework as applicable for the areas in which they work.

When a conflict exists between the Evaluation Framework or other reference documents and the Protocols, the Protocols will take preceden ce unless otherwise approved by the CPUC-ED.



### The Detailed Evaluation Work Plan

All program evaluations are required to have a de tailed evaluation work plan. In many cases the program evaluation work plans will be clustere d within evaluation groupings. However, even within these groupings, there must be a detailed ev aluation work plan stru ctured at the program (and in some cases at the program component) level that identifies how the program will be evaluated and the steps to be taken to conduct th e evaluation. The eval uation work plan shall include the following components to support an assessment of the ad equacy and approach of the
evaluation effort:

• Cover page containing the names of the pr ogram(s), Administrators and evaluation contractors, date of the evaluation work plan and the program tracking number(s) for program(s) covered in the plan;

• Table of Contents;

• High-level summary overview of the pr ograms and the evaluation efforts;

• Brief description of the program(s) being eval uated including a high level presentation of the program theory. If the program does not have a formal program theory, the evaluation plan should incorporate a brie f presentation of th e evaluation-assumed program theory so that the Joint Staff ma y understand the sequence of events leading from program actions and activities to desired results (direct or indirect energy impacts);

5 TecMarket Works, 74-79.Evaluators’ Protocols Introduction

CPUC 16 TecMarket Works Team • Presentation of the evaluation goals and the deta iled researchable issu es to be addressed in the evaluation. (These will al so be presented and discussed in the evaluation reports;)

• Description of how the evaluation addresse s the researchable issues, including a description of the evaluation priorities and th e use of assigned rigor levels to address these priorities;

• A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity and sources of bi as and a short descrip tion of the approaches planned to reduce threats, reduce bias and increase the reliability of the findings and minimize bias and uncertainty;

• Task descriptions of the evaluation efforts;

• Description of the analys is activities and approaches to be taken:

o For energy acquisition and procurement pr ograms, include a de scription of the approach that will be used to estimate kW, kWh and therm impacts for each year over the EUL of program-covered measures, including a description of the approach to be used to adjust the exp ected impacts for the persistence of the impacts;

o For information or education programs, in clude a discussion of the approach that
will be used to estimate the actions or behaviors taken and/or knowledge gained that is expected to lead to energy impacts;

o For process or operational assessments, incl ude a description of the approach used to identify changes that can be expected to improve the cost-effectiveness of or participant satisfaction with the program;

• Description of the M&V efforts (i mpact evaluations only) including:

o Reference to International Performance Measurement and Verification Protocol (IPMVP) option 6 , if used;

o Detailed description of the option-specific approach; and

o Description of any deviations from the IPMVP option, if any;

• Description of the sampling rationale , methods and needed sample sizes.

• Discussion of the specific Performance Basis Metrics that will be reported in the draft and final evaluation plan;

• A definition of the terms “participant” a nd “non-participant” as it applies to the evaluation being conducted;

• Detailed description of the information that will be needed from the IOUs or from the program-reporting database maintained at the CPUC-ED in order to conduct the
evaluation and an estimate of the date that the information will be needed. This same information will be included in evaluation-related data requests;

6 More information on the IPMVP can be found in the Evaluation Framework (148-149), or at the IPMVP Web site at www.ipmvp.org.Evaluators’ Protocols Introduction

CPUC 17 TecMarket Works Team • Evaluation activities timeline for the program cycle, including identification of
deliverables and deliverable due dates. This should also include early, mid-stream and late cycle feedback deliverables and deliverab le dates. (These date s must be coordinated
with the information needs of the Joint St aff and their program-portfolio assessment needs schedule;)

• Total program budget, total ev aluation budget and a task-lev el evaluation budget for the study; and

• Contact information for the lead Administra tor, lead program manager and evaluation manager, including addresses, telephone num bers, fax numbers and e-mail addresses.

The evaluation work plan should be written in a style and with enough de tail that it can be clearly understood by Administrators, policy makers and evaluation professi onals, and replicated by other evaluation contractors.



### Confidentiality Issues

Confidentiality is an essential part of the evalua tion process and is included in this section to set a baseline for how information will be treated within the evaluation efforts. The following aspects of confidentially are incorporated into all evaluations conducte d under the guidance of
the Protocols.

1. All evaluation contractors will be required to sign confidentiality agreements in order to conduct evaluations funded through the Protocols. These agreements will be incorporated into all evaluation contracts. For impact, market effects, codes and standards, emerging technology and M&V studies, the agreements will be incorporated into contracts awarded by th e CPUC or the CEC as appropriate. For process evaluations, the indi vidual Administrators issu ing the process evaluation contracts are responsible for incorporati ng confidentiality agreements. However, evaluation information, including customer-sp ecific information, can be shared across
evaluation contractors within the same eval uation team and across teams. However, this data is to be protected from e xposure beyond the evaluation teams and all contractors must sign confidentiality agreem ents prior to the receipt of customerspecific information.

2. All customer-specific information will be treated as confidential and safeguarded from public disclosure. Evaluation contract ors are granted access to participant and customer specific information maintained by the Administrators as needed to conduct the evaluation efforts, however, no evaluati on contractor will allow participant or customer specific information to be releas ed to individuals or organizations beyond their research team, unless specifically permitted in writing by each customer for
which information is to be released. A ll memoranda, letters, e-mails, reports and databases that are developed or used in th e evaluation efforts that contain participantspecific or customer-specific information, whet her an individual, a firm or business or an organization, are covered by this confidentiality requirement.Evaluators’ Protocols Introduction

CPUC 18 TecMarket Works Team 

### Contacting the Customer

A critical component to the success of any eval uation effort is the maintenance of a supportive relationship between the customer and the many di fferent types of organizations that influence the evaluation effort. IOU representatives, CP UC-ED, CEC, evaluation contractors and others
involved in the evaluation efforts need to be diligent in making sure that customers and participants are not over-contac ted in support of them. Whenev er possible, customer contact
initiatives should be coordinated to avoid over-contact. Customer requests to be excluded from evaluation efforts should be respected. Customer co mplaints associated with evaluation efforts should be reported to the CPUC-ED and the associ ated Administrator within 48 hours of receipt.

Before customers are contacted by evaluation contra ctors, their representati ves or subcontractors, the prime evaluation contractor will notify the Admi nistrators of the need to do so and work to agree on an approach and timeline that may ch ange from study to study. All final customer contact approaches and contact Protocols shoul d specify customers to be contacted (as an attachment), reasons for the contact, information to be collected, the method of contact and the
associated timeline.

Administrators will inform the appropriate indivi duals within their organizations of any related

customer contact.Evaluators’ Protocols Impact Evaluation

CPUC 19 TecMarket Works Team 

## Impact Evaluation Protocol

Introduction

The Impact Evaluation Protocol is applicab le for all programs or program components designated by the Joint Sta ff for a direct or indirect 7 impact evaluation, especially for those programs claiming energy or demand savings an d for those programs that are expected to influence energy-related behavior s and can be linked to energy and/or demand savings. This Protocol is designed to relia bly estimate program impacts. Information, education and advertising efforts determined by the Joint Staff to have an indirect impact evaluation are expected, at a minimum, to measure the program -induced behavioral changes, often leading to energy and demand savings estimates. 8

The Impact Evaluation Protocol is established to ensure that a ll evaluations of program-specific energy and demand savings, and program-speci fic impacts are conducted using evaluation methods deemed acceptable based on the assigne d level of rigor for that evaluation. The Protocol’s list of allowable methods is one compone nt that helps ensure gr eater reliability in the energy and demand savings estimates from Califor nia’s energy efficiency efforts. The Joint Staff can assign different levels of rigor to each program, thus allowing the flexibility to allocate evaluation resources according to the needs of th e Portfolio given uncertainty in the expected savings, the size of expected savings, the program budget and other criteria. The Joint Staff will instruct evaluation contractors to use specific rigor levels based on its application of the Protocol’s decision criteria, and mix of ev aluation choices and re source allocations.

Rigor is defined as the level of expected reliabi lity. The higher the le vel of rigor, the more
confident we are that the results of the evaluation are both accurate and precise, i.e., reliable. That is, reliability and rigor are synonymous. Reliability is discussed in the Sampling and Uncertainty Protocol and in the Evaluation Framework where it is noted that sampling precision does not equate to accuracy. Both are important components in reliability, as used by the CPUC. Each program will be assigned a specific evaluation rigor level for its primary evaluation objectives to guarantee that a mi nimum standard is met.

“Impact evaluation” refers here to all program-specific evaluations designed to measure program impacts. Impact evaluations attempt to estimate net changes in electricity usage, electricity demand, usage of therms and/or behavioral imp acts that are expected to produce changes in energy use. Evaluations conducted according to the Impact Evalua tion Protocol are expected to obtain energy or demand savings estimates wherev er possible. Impact evaluations of programs or program components designed to directly achieve energy and demand savings should follow the Direct Impact Evaluation Protoc ol to measure these savings.

7 The term “indirect impact evaluation” refers to those pr ogram-specific evaluations desig ned to measure the specific

program goals that create an impact that is expected to eventually lead to energy and/or demand savings but where these savings cannot be directly estimated.

8 This is the minimum expectation. The evaluation re search design, however, could surpass this through an

experimental or quasi-experimental design t hat estimates energy and demand savings.Evaluators’ Protocols Impact Evaluation

CPUC 20 TecMarket Works Team The Indirect Impact Evaluation Protocol should be used for evaluations of those programs or program components primarily designed to obtain be havior changes that will eventually lead to energy and demand savings but do not directly do so within the program. The Indirect Impact
Evaluation Protocol is intended for those programs where the primary uncertainty lies in the program’s ability to obtain the behavior change (s) targeted by the pr ogram. Indirect impact evaluations will, therefore, be linked wherever possible to previously measured energy or demand savings estimates that would yield savings estimates with the same rigor required by the Basic rigor level for impact eval uations described below. This link to reliable stipulated or engineering calculated energy and demand savings estimates is not always possible for behavioral program efforts assigned to receive an impact evaluation. In th ese cases, an indirect impact evaluation shall be conducted at the Ba sic rigor level to measure the program-induced impacts, as described below.

This Protocol often refers to a “program or program component.” A program component, as defined by this Protocol, is any identifiable portion of a program . This could be a measure, a delivery mechanism, a set of delivery mechanisms or measures, or a set of delivery mechanisms or measures that follow a chain from an activity depicted in a program logic model. The Joint Staff may desire a direct or indirect impact evaluation of a progr am as well as a separate analysis for the impact evaluation for one of its progra m components. This might occur, for example,
when more detailed evaluation information is n eeded for a measure for future program planning or to support an update of DEER, a new measure is being piloted or expanded in its use, or a new delivery mechanism has been added.

Impact evaluations are limited to addressing the direct impacts of the program on participants and estimating participant spillover impacts. 9 These studies do not include documenting program influences on the operations of a market or the program's impacts on non-participants. Program-induced changes on the way a market opera tes or on non-participan ts are addressed in
the Market Effects Evaluation Protocol.

The Impact Evaluation Protocol de scribes the metrics to be produced from an impact evaluation. This includes the target parameters that must be used as part of devel oping the evaluation design in order to produce these metrics. This Protocol also presents an overview of how the Impact Evaluation Protocol is integr ated with the M&V, Sampling, Market Effects and Reporting Protocols for the implementation of a direct impact or indirect impact ev aluation and within the overall system to produce reliable portfolio level evaluated savings estimates. This systematic Protocol-linked process is designed to be part of a proposal selection and evaluation plan review process, which is followed by ongoing management of the evaluation a nd evaluation reporting.

It is expected that evaluation contractors will respond to requests for proposals (RFPs) for impact evaluations with proposals th at meet the standards contained in th e Protocols. It is expected that generally accepted statistical methods as published in textbooks used at accredited universities or articles in peer-reviewed journa ls will be used for parameter estimation from sample data in

9 For a thorough evaluation, impact evaluations should estimate direct program savings and participant spillover

savings. These estimates need to be distinct estimate s and not a combined estimate across the two whenever possible. Current CPUC policy, as the Protocols are bei ng developed, states that only program savings and not participant spillover will be counted towards pr ogram and administrator goals and performance.Evaluators’ Protocols Impact Evaluation

CPUC 21 TecMarket Works Team regression-based approaches a nd for moving from sample estim ation to program or populationbased estimates. Engineering methods are expe cted to meet the requirements in the M&V Protocol and follow generally accepted practices as published in engineering textbooks used at accredited universities or articles in peer-rev iewed journals along with generally accepted statistical methods (as described above). Ev aluation contractors may propose optional methods in addition to Protocol-compliant methods, if the optional methods provide at least as much rigor and accuracy as the Protocol-covered approach.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• Joint Staff Evaluation Planners will use the Protocol to determine when a direct impact or indirect impact evaluation is appropriate and to as sign the level of rigor expected for the study, as input into the evaluation RFPs for imp act evaluation contractors, and as background and criteria for use in reviewing impact eval uation plans, managing the impact evaluations and reviewing impact evaluation reports and results;

• The Evaluation Project Team will use the Protocol to ensure that their detailed direct impact or indirect impact evaluation plan(s ) address(es) key requirements for each program or program component based upon the level(s) of rigor designated by the Joint Staff. They
will also use the Protocol to double-check that the Protocol requirements have been met as they conduct, complete and report the impact evaluations;

• Administrators will use the Protocol to understand how the impact evaluation will be conducted on their programs and to understand the evaluation data needs to support the impact evaluation. In addition, the Protocol provides background for the Administrator’s use to determine when to intervene in the program design and implementation efforts to achieve continued and/or greate r efficiency gains;

• Program Implementers will use the Protocol to understand the impact evaluation that will be conducted of their programs and program co mponents. Often, they will be required to provide data to support the impact evaluation. The Protocol will also provide background for
their use to understand when to intervene to achieve continued and/or greater efficiency gains; and

• ISO / System planners will use savings and uncertainty estimates for load forecasting and system planning.



### Overview of the Protocol

Protocol Types

The overall Impact Evaluation Prot ocol contains one subset of 3 Protocols for estimating direct energy and demand impacts and one fo r estimating indirect impacts.

Direct Impact Eval uation Protocols:

• The Gross Energy Impact Protocol has two levels of rigor (Basic and Enhanced) for developing gross energy estimates;Evaluators’ Protocols Impact Evaluation

CPUC 22 TecMarket Works Team • The Gross Demand Impact Protocol has two levels of rigor (Basic and Enhanced) for developing gross demand estimates; and

• The Participant Net Impact Protocol has three levels of rigor for developing net impact estimates (Basic, Standard and Enhanced).

The Indirect Impact Evaluation Protocol has three levels of rigor (Basic, Standard and Enhanced). The Basic Rigor level is reserved for those programs or program components that cannot be linked to energy savings but where net behavior changes need to be estimated to measure program impacts. This Protocol include s the requirement that the measured impacts are net impacts (i.e., program-induced).



### Rigor

The general rules for how often evaluations need to be conduct ed are determined by the Joint Staff. The Joint Staff will decide, for each re levant program, if and when the program will receive an impact evaluation. The Joint Staff may choose not to have an impact evaluation conducted for a particular program or program component. When the Joint Staff decides a program will receive an impact evaluation, it also selects whether a direct impact evaluation or indirect impact evaluation is most appropriate and the level of evaluation rigor required. The Impact Evaluation Protocol then establishes the methods appropriate for the given type of impact evaluation and assigned level of ev aluation rigor. In this way, th e Protocols establish a minimum level of evaluation rigor in order to ensure that the savings estimates produced are at the level of reliability needed to support the overall reliability of the savings in the Administrator’s Portfolio
and the statewide Portfolio. 10

Each level of rigor provides a class of allowabl e methods in order to offer flexibility for the potential evaluation contractors to assess and propose the most accurate and cost-effective
methods that meet the Joint Staff’s needs. The principle is to provide minimum specifications for a set of options at each rigor level and yet en courage evaluation contractors to use both the art and science of evaluation to develop affordable and quality evaluations that produce reliable savings estimates.

The Joint Staff may assign one rigor level for a pr ogram and a different level of rigor for one or more of its program components. When this happ ens, the evaluation must meet the level of rigor for that program component as assigned (to include meeting the Sampling and Uncertainty requirements) as well as the rigor level for the program as a whole.

The various Protocols and associat ed rigor levels required for dire ct impact and indirect impact
evaluations are illustrated in Figure 4.

10 Savings for programs with expected savings could be included in the Portfolio savings estimates based upon an

accounting effort that multiplies the number, or verified number, of in stallations times the latest evaluated savings estimates or deemed savings, as dete rmined more appropriate by Joint Staff. The verifi ed number of installations is the number of program installations based upon th e reported number combined with the results of any verification activities required by Joint Staff. (See the M&V Protocol for the description of the Verification Protocol.)Evaluators’ Protocols Impact Evaluation

CPUC 23 TecMarket Works Team Figure 4. Required Protocols for Direct Impact and Indirect Impact Evaluations



### Key Metrics, Inputs and Outputs

Impact evaluations will draw upon data from pr ogram databases, program descriptions, DEER databases, work papers developed during program planning, utility demand metering and consumption data for participants and non-partic ipants, utility-, state government- or local government-collected weather data, on-site meas urement, monitoring and observational data, survey and interview data collection, and other pr ior study data and reports. These will be used Direct Impact

Evaluation Basic
Basic Basic

Enhanced Standard

Enhanced

Enhanced

Basic

Standard

Enhanced CPUC Assigns Required Rigor Levels CPUC Assigns Required Rigor Levels

Indirect Impact Protocol Gross Energy Impact
Protocol Gross Demand Impact
Protocol Participant Net Impact
Protocol

Indirect Impact

EvaluationEvaluators’ Protocols Impact Evaluation

CPUC 24 TecMarket Works Team with the Impact Evaluation Protocol-allowabl e methods to produce program, program strategy and program component (measure-level, as reque sted) impact evaluations. These must be conducted using the Joint Staff-approved evaluation plans.

The Impact Evaluation Protocol will guide the estimation of ev aluation-adjusted gross and net savings for energy (kWh) and demand (kW) fo r electricity-using equi pment (and behaviors related to electricity-using equipment) and ne t therm savings for gas-using equipment (and behaviors related to gas-using e quipment). The kWh, kW and therm impacts are required to be reported separately for the first year and for each year thereafter over the pe riod of time in which net program-induced savings are expected. Th e programs’ expected savings from program plans, reported savings and the evaluation’s esti mate of savings will be reported in these annual savings tables. The Reporting Protocol, which all direct impact and indi rect impact evaluation reporting must follow, provides furthe r description and table examples.

Because impact evaluations must follow the Sampling and Uncertainty Protocol, evaluators must also assess, minimize and mitigate potential bi as and present the achieved level of precision (including relative precision, erro r bounds, coefficient of variations, standard deviations and error ratios) for interpreting in formation, summarizing savings and its precision across programs, and providing the information necessary for fu ture evaluation planning. Where precision is calculated from the chaining or pooling of ev aluation study efforts, the above precision information should be provided for each study effort as well as the combined result.

When requested by the Joint Staff, impact ev aluations must produce the required metrics by delivery mechanism (e.g., rebates and direct instal l). Where delivery mechanisms differ within a program, this Protocol requires that the impact evaluation be designed, conducted and reported to provide the energy and demand metrics (along with the precision information) for each delivery mechanism, when the Joint Staff identifies deliv ery method-associated im pacts as an evaluation goal.

Evaluations conducted according to the Gross Impact Protocol, Gro ss Demand Protocol and Participant Net Impact Protocol will produce gross and net kWh, kW and therm impacts. The evaluation analysis results must be used with program database, verification, standard weather information, and other participant and non-partic ipant data, as necessary, to produce program energy and demand savings estimates. Measure e ffective useful life (EUL) from DEER or as otherwise approved by the Joint Staff will be us ed to create the required energy and demand impacts for first year and for each year thereaf ter over the period of time in which savings are expected based upon measure EUL. Any evaluati on findings that might call into question the EULs being used must be presented to the Jo int Staff when discovered and discussed in the evaluation report. Further description and exampl es of the required tables are provided in the Reporting Protocol.

All direct impact and indirect impact evaluations are expect ed to assess and discuss the
differences between the ex-ante estimates and th e evaluation produced ex- post estimates. To the extent that the data gathered and evaluation analyses conducted can explain the causes for these differences, this must be presented and discusse d. Cases in which explaining these differences due to lack of data or problems of interpre tation should be noted in the evaluation report.Evaluators’ Protocols Impact Evaluation

CPUC 25 TecMarket Works Team 

### Energy and Demand Impact Protocols

These are minimum standard Protocols. All met hods in a higher class of rigor are allowable as they exceed the minimum criteria. For example, if the program has a Joint Staff assigned rigor of Basic or Standard and the method proposed by the evaluation contractor is an option under a rigor of Enhanced (but is not listed under Basi c or Standard), this method is acceptable for meeting the Protocol.



### Gross Energy Impact Protocol

The Gross Energy Impact Protocol is summarized in Table 1. Further de scription, additional requirements, clarification and examples of this Protocol are presented after the table. The methods used and the way in which they are used and reported must meet all the requirements discussed within this section (not just those within the summary ta ble or those within the text) to provide unbiased reliable estimates of program le vel gross energy impacts in order to comply with the Gross Energy Impact Protocol. The Protocols sometimes reference the Evaluation
Framework or other documents which provide ex amples of applicable methods. The requirements, however, are always those stated in the Protocols, which take precedence over all other protocols and evaluation guidance documen ts in all circumstances, unless otherwise approved or required by the CPUC.

All M&V referred to in the Impact Evaluation Pr otocol must be planned, conducted and reported according to the M&V Protocol. M&V may be c onducted at a higher level of rigor, with more inputs measured or metered, or with greater precision than the minimum shown within the Impact Evaluation Protocol, but not with a lower le vel of rigor. The M&V Protocol can also be required by the Joint Staff or used by evaluators to enhance other evaluation efforts. For example, an evaluator proposing a Statistically Adjusted Engin eering (SAE) regression model may use (or the Joint Staff may require the use of) the M&V Protocol to conduct field measurements on the sample of participants to be included in the SAE model to improve the engineering estimates. This may involve conducting measurement/metering and utilizing IPMVP Option A.

The overall goal of the Direct Impact Evaluation Protocol (whi ch includes the Gross Energy Impact Protocol) is to obtain unbiased reliab le estimates of program-level net energy and demand savings over the life of the expected net impact.Evaluators’ Protocols Impact Evaluation

CPUC 26 TecMarket Works Team Table 1. Required Protocols for Gross Energy Evaluation

Rigor Level Minimum Allowable Methods for Gross Energy Evaluation

Basic 1. Simple Engineering Model (SEM) with M&V equal to IPMVP Option A and meeting all requirements in the M&V Protocol for this method. Sampling according to the Sampling and Uncertainty Protocol.

2. Normalized Annual Consumption (N AC) using pre- and post-program participation consumption from utility bills from the appropriate meters related to the measures undertaken, normalized for weather, using identified weather data to normalize for heating and/or cooling as is appropriate to measures included. Tw elve (12) months pre-retrofit and twelve (12) months post-retrofit consumption data is required. Sampling must be according to the Sampling and Uncertainty Protocol.

Enhanced 1. A fully specified regression analys is of consumption information from utility bills with inclusion/adjustment for changes and background variables over the time period of analysis that could potentially be correlated with the gross energy savings being measured. Twelve (12) months post-retrofit consumption data are required. Twelve (12) months pre-retrofit consumption data are required, unless program design does not allow pre-retrofit billing data, such as in new constr uction. In these cases, well-matched control group s and post-retrofit consumption analysis is allowable. 11 Sampling must be according to the Sampling and Uncertainty Protocol utilizing power analysis as an input to determining required sample size(s).

2. Building energy simulation models t hat are calibrated as described in IPMVP Option D requirements in the M&V Protocols. If appropriate, may alternatively use a process-engineering model (e.g., AirMaster+) with calibration as described in the M&V Protocols. Sampling according to the Sampling and Uncertainty Protocol.

3. Retrofit Isolation engineering models as described in IPMVP Option B requirements in the M&V Protocols. Sampling according to the Sampling and Uncertainty Protocol.

4. Experimental design established within the program implementation process, designed to obtain reliable net energy savings based upondifferences between energy consumption between treatment and nontreatment groups from consumption data. 12 Sampling must be according to the Sampling and Uncertainty Protocol.

11 Post-retrofit only billing collapses the analysis from cross-sectional time-series to cross-sectional. Given this, even

more care and examination is expected with regard to cont rolling for cross-sectional issu es that could potentially bias the savings estimate.

12 The overall goal of the Direct Impact Pr otocols is to obtain reliable net energy and demand savings estimates. If

the methodology directly estimates net savings at the same or better rigor than the required level of rigor, then a gross savings and participant net impact analys is is not required to be shown separately.Evaluators’ Protocols Impact Evaluation

CPUC 27 TecMarket Works Team Basic Rigor

There are two classes of evalua tion methods that set the minimum allowable methods for the Gross Energy Impact Protoc ol Basic rigor level.

Simple Engineering Model (SEM)

The first class of allowable methods is the si mple engineering model (SEM). An SEM is equivalent to IPMVP Option A and must be conduc ted as described in the M&V Protocol. This
method is described and a few re ferences are presented in the Evaluation Framework 13 . These types of models can be straightforward algorithms for calculating energy impacts for nonweather dependent measures such as energy-effi cient lighting, applian ces, motors and cooking equipment. Exceptions to this requirement are programs offering comprehensive measure packages with significant measure interac tions, to include co mmissioning and retrocommissioning programs, and new construction pr ograms. Evaluations of these programs conducted using engineering methods must use building energy simulation modeling under IPMVP Option D as described in the Enhan ced rigor level of the Impact Protocol.

Sampling for the M&V used in the SEM must be conducted as prescribed in the Sampling and Uncertainty Protocol which incl udes developing the sample to ta rget a minimum of 30 percent precision at a 90 percent confiden ce level. Knowledge of the components of the SEM and the propagation of error method must be used to determ ine what needs to be measured in the SEM to meet this requirement. (See the M&V Protocol for more detail on the related requirements.) In both the evaluation plan and the evaluation report, the inputs sele cted and the methods selected for the measurement/monitoring must be justified in terms of why they are the factors that provide the most likely unbiased and reliable gross energy impact estimates for the evaluation study being conducted.

Normalized Annual Consumption (NAC)

The second class of allowable methods is normali zed annual consumption (NAC) analysis. This is a regression-based method that analyzes mont hly kWh or therm consumption data provided by
utilities. This method and a few references are presented in the Evaluation Framework 14 . The NAC analysis can be conducted us ing statistical soft ware, such as the Princeton Scorekeeping Method (PRISM), and other statistically base d approaches using SAS or SPSS. The NAC method, often using PRISM, has been most ofte n used to estimate energy impacts produced by whole house retrofit programs.

To comply with this Protocol, NAC must nor malize consumption for weather effects using a generally accepted set of weather data (from utility weather monitoring stations, National Oceanic & Atmospheric Administra tion (NOAA) weather monitoring st ations or others as used by California energy forecasting and supply analysts). Weather data must be used to normalize for heating and/or cooling as a ppropriate to the measures include d. Final savings estimates must also use weather data to report bo th actual savings from the weathe r data used in the analysis and expected annual savings fitted to the CEC c limate thermal zone (CTZ) long-term average
weather data.

13 TecMarket Works, 123-129.

14 Ibid, 105-106.Evaluators’ Protocols Impact Evaluation

CPUC 28 TecMarket Works Team A minimum of twelve months pre -retrofit and twelve months post- retrofit consumption data is required. However, there might be a number of participants who are excluded from the analysis because they do not have the required minimum of twelve months of pre- and post-consumption data. For example, some populations, because they are more mobile (e.g., rental populations and particularly low-income households), will be less likely to have the required amount of pre- and post-consumption data. An examination should be made on whether the in clusion or exclusion of such participants could pot entially bias the results.

Often, a census approach is undertaken for NAC. Where sampling is used, it must follow the Sampling and Uncertainty Protocol.

Enhanced Rigor

There are four classes of allo wable methods to meet the minimum requirements for the Gross Energy Impact Protocol Enhanced rigor level. One of thes e is regression analysis of consumption with specific modeling requirements, two are different engineering-based methods with specific M&V and model calibration requireme nts, and the fourth is experimental design
established within the program implementation process to sp ecifically obtain unbiased reliable estimates of net energy and demand savings.

Regression Analysis
The first class of allowable methods is regressi on analysis of consumption data provided by utilities that statistically adju sts for key variables that change over time and are potentially correlated with gross or net energy savings. As a way of capturing the influence of weather, evaluators may incorporate weather-normalized consumption as the dependent variable or include heating- and cooling-degree days directly in the model. Other variables that change over time that are often correlated with gross and ne t energy savings include, among others, the state of the economy (recession, recovery, economic growth), 15 fuel prices, occupancy changes, behavior changes (set-point ch anges, schedules, usage freque ncy), changes in operation and changes in schedule. The evaluator is free to select the most appropriate additional variables to
include.

The modeler is also free to select the functiona l form of the model (a variety of linear and nonlinear forms) as well as the type of model. A wide variety of model types may be used, including Statistically Adjusted Engineering (S AE) models, Analysis of Covariance (ANCOVA) or fixed-effects models and other regression models. The Evaluation Framework presents the SAE model and a few references, 16 and ANCOVA with a few references. 17 These types of impact evaluations have been conducted for residential whole-house, heating and cooling retrofit, refrigerator and water he ating replacement, and small and large commercial programs. The Enhanced Gross Impact regression option is not limited to these two types of models. Finally, the testing of alternative specifications is encouraged.

15 See the discussion on page 118 in the Evaluation Framework and the article cited in its footnote 82 for more information and an example.

16 TecMarket Works, 108-109.

17 Ibid, 109-111.Evaluators’ Protocols Impact Evaluation

CPUC 29 TecMarket Works Team Power analysis, 18 results from prior studies on similar programs, and profes sional judgment are to be used to determine the required sample size. Sampling and analysis and mitigation for uncertainty must be planned and conducted acco rding to the Sampling and Uncertainty Protocol.

The primary consideration in the use of regressi on analysis to meet the Enhanced Gross Energy Impact Protocol is that the anal ysis must be designed to obtain re liable energy savings. In order for regression to begin to meet the unbiased elem ent in this requirement, the regression analysis must incorporate and control for background and ch ange variables that mi ght otherwise bias the measurement of the energy savings. There are se veral ways in which this can be accomplished. One common method is to include participant an d non-participant analyses . If this method is selected, particular care and ju stification must be made for th e non-participant group selected and its appropriateness for the program and partic ipant population being analyzed. Secondly, research design and analysis needs to consider whether the analysis is providing gross impact, net impact or something in between that must then be adjusted or analyzed in a second step to produce, at a minimum, reliable unbiased ne t of free-ridership savings estimates.

Alternatively, surveys of participants and the cr eation of change variables can be created and incorporated into the regression analysis. A nother example would be to create or obtain participant or non-participant cha nge variables from secondary or other aggregate or individual studies of similar/matched populatio ns for inclusion within the regr ession analysis. The specific method and research design to accomplish this re quirement is not specified, but the evaluation plan, analysis and evaluation re port must present, justify, disc uss and analyze the method and data utilized to accomplish this requirement.

A minimum of twelve months pre -retrofit and twelve months post- retrofit consumption data is required. However, there might be a number of participants who are excluded from the analysis because they do not have the required minimum of 12 months of pre- and post-consumption data. For example, some populations, because they are more mobile (e.g., rental populations and particularly low-income households), will be less likely to have the required amount of pre- and post-consumption data. An examination should be made on whether the in clusion or exclusion of such participants could potentially bias the results.

Twelve months pre-retrofit bil ling data are required unless the program design does not allow pre-retrofit billing data, such as in new construction. In these cases, well-matched control groups and post-retrofit billing analysis is allowable. Post-retrofit onl y billing collapses the analysis
from cross-sectional time-series to cross-sectiona l. Given this, even more care and examination is expected with regard to controlling for cross- sectional issues that could potentially bias the savings estimate.

Final savings estimates must report both actual savings from the weather data used in the analysis and expected annual savings fitted to the CEC CTZ long-term average weather data.

18 Power analysis is a statistical technique to determine sa mple size requirements to ensure statistical significance

can be found. There are several software packages and calculation Web sites that conduct the power analysis calculation. See the Sampling and Uncert ainty Protocol for more discussion and reference. Power analysis is only being required in the Protocol for determining required samp le sizes. Appendix D provides further detail on using power analysis for developing sample size requirements.Evaluators’ Protocols Impact Evaluation

CPUC 30 TecMarket Works Team The regression-based methods must use power analys is to plan their sample size (unless census samples are being used). Regression-based me thods must also meet the requirements of the Sampling and Uncertainty Protocol. Many of th e requirements in the Sampling and Uncertainty Protocol require specific actions and documentation regarding data cleaning, model specification, testing and reporti ng for regression-based methods.

Engineering Models

The second class of allowable methods is bui lding energy simulation programs calibrated as described in the Option D require ments in the M&V Protocols. This method is described and a few references are presented in the Evaluation Framework. 19

The engineering models that meet the Opti on D requirements are generally building energy simulation models, as described in the Evaluation Framework. 20 This can be applicable to many types of programs that influence commercial, institutional, residential and other buildings where the measures impact the heating, ventilation or air conditioning (HVAC) e nd-use. This method is often used for new construction programs and building, heating/coo ling or shell measure
retrofits in commercial and residential programs.

In addition, industrial efforts can include changes in process opera tions and the appropriate type of model could be a process-engineering model. These are specialized engineering models and software that conduct engineering analysis for industry-specific industria l processes. Where these types of models are more appropriate, the Gross Energy Impact Protocol allows the use of a process engineering model with calibration as described in th e M&V Protocols to meet the Enhanced rigor level.

Sampling must be conducted according to the Sampling and Uncertainty Protocol.

Retrofit Isolation Measurements

The third class of allowable met hods is the retrofit isolation measurements as described in Option B requirements in the M&V Protocols. This method is us ed in cases where full field measurement of all parameters for the energy us e for the system where the efficiency measure was applied are feasible and can provide the most reliable results in a co st-efficient evaluation. An overview of this method is provided in the Evaluation Framework. 21 Applying a variable frequency drive to a constant speed pump in a variable flow pumping application would be a typical example of when this method would likely be used.

Sampling must be conducted according to the Sampling and Un certainty Protocol.

Experimental Design
The fourth class of allowable methods is experi mental design. Experimental design with energy and demand measurement (either consumption da ta comparison or engineering-based with M&V) comparisons between the treatment and non-treatment groups meets the Enhanced Gross

19 TecMarket Works, 129-133 and 176-181.

20 Ibid, 176-181.

21 Ibid, 166-169.Evaluators’ Protocols Impact Evaluation

CPUC 31 TecMarket Works Team Energy Impact Protocol rigor level. Experimental design will normally measure net energy and demand impacts and meet the criteria for equal or better rigor for the overall net savings and demand estimates such that the Gross Impact Prot ocol and the Participant Net Impact Protocol requirements are met. Currently, experimental desi gn has not been widely us ed within efficiency
evaluation. See the Evaluation Framework 22 for a description and some examples of potential experimental designs within en ergy efficiency efforts. Samp ling conducted as part of the experimental design must be conducted according to the Sampling and Uncertainty Protocol.

All Gross Energy Impact Methods All impact evaluations should employ a research design that has properly identified participants made available from the program database(s). The regression methods of pre- and postconsumption and the calibrated engineering model equivalent to Option D could yield results not restricted to the program being evaluated if participation in mu ltiple programs occurs around the same time period or overlaps in influence. This could contribute to double counting at the portfolio level. To avoid this possibility, a ll Administrators are requi red to provide data on participation in other programs for all progr am participants, including when participation occurred. Evaluators are requi red to ensure that their met hodologies and analysis account for any overlap in program participation and measur es that could potential ly bias the program
evaluation results.

All impact evaluations must meet the requireme nts of the Sampling and Uncertainty Protocol. Regression analysis of consumption data requi res addressing outliers, missing data, weather adjustment, selection bias, background variables, data screens, heterogeneity of customers, autocorrelation, truncation, erro r in measuring variables, m odel specification and omitted variable error, heteroscedasticity, collinearity a nd influential data points. Engineering analysis and M&V-based methods are required to addres s sources of uncertainty in parameters, construction of baseline, guarding against meas urement error, site se lection and non-response bias, engineering model bias, modeler bias, deemed parameter bias, meter bias, sensor placement bias and non-random selection of equipm ent or circuits to monitor.

Each item in these lists above must be addressed as they all have the potential to bias the savings estimates. Bias is the greatest threat to the reliability of savings estimates. The primary difference between the Basic and Enhanced rigor levels is that the minimum allowable methods in the Enhanced rigor level dir ectly address or control for the more likely sources of potential bias in that class of methods (e .g., regression-based versus engin eering-based). This means that
the minimum allowable methods in the Enhanced rigor level are expect ed to provide more reliable savings estimates than the minimum allowable methods in the Basic rigor level.

All impact evaluations must meet the rigor level assigned. If ri gor is assigned for a measure or program component, the rigor leve l must be met for analysis of that measure or program component. If measure-level analyses are condu cted and no rigor level has been assigned for these measures, they may be conducted at either th e Basic or Enhanced rigor level as long as the impact evaluation of the program as a whole is designed to achieve its overall target precision

22 Ibid, 104-105.Evaluators’ Protocols Impact Evaluation

CPUC 32 TecMarket Works Team level and addresses all of the potential bias issu es listed above and described in the Sampling and Uncertainty Protocol.

Experience in energy efficiency program evalua tion has shown that ther e are cases where some methods are more likely to yield defensible resu lts than others for certain sectors or program designs. The Impact Evaluation Protocol does not restrict the methods used to those that have been successfully used previously. However, th e Joint Staff will consider this factor in both
contractor selection and in th e review and approval process of the evaluation plan. Methods proposed that do not have a successful track record must have thorough documentation on how the methods, techniques or data that will be us ed can be expected to produce reliable savings estimates and how the key personnel conducting this effort are qualified to do so. For example, experience to date in energy efficiency imp act program evaluation has generally shown the following:

• NAC methods are most applicable to resident ial and small commercial efforts where the expected energy savings are at least 10 percent of pre-in stallation usage;

• NAC methods are not well suited to handle si gnificant issues with heteroscedasticity, truncation, self-selection or changes in bac kground issues (e.g., si gnificant change in economic conditions-large recession, r ecovery or economic growth);

• SEM methods are not well suited for whole build ing measures with interactive effects or commissioning/retro-co mmissioning efforts;

• The heterogeneity and multitude of background variable issues for industrial customers and unique commercial (e.g., ski resorts and am usement parks/faciliti es) or institutional (e.g., water/wastewater and prisons) customer s make the use of any regression-based consumption analysis difficult and potentia lly less reliable than engineering-based methods;

• Regression-based consumption analyses are less likely to be able to obtain definitive energy savings estimates where the expected en ergy savings are not at least 10 percent of pre-installati on usage; and

• Regression-based consumption analysis is quite difficult for new construction programs due to the lack of pre-retrof it consumption data and the cons equential greater burden for controlling for cross-sectional issues for co mparing participants and non-participants (and self-selection bias, particul arly if the non-participants ar e any form of rejecters of program participation). New construction program impact evaluations are generally conducted using engineering models (such as those described in IPMVP Option D).



### Gross Demand Impact Protocol

The Gross Demand Impact Prot ocols are summarized in Table 2. Further de scription, additional requirements, clarification and examples of these Protocols follow the table. For an evaluation to be in compliance with the Gross Demand Impact Protocol, the methods used and the way in which data are used and reported must meet all the requirements discussed within this section. The intent is to provide unbiased reliable estim ates of program level demand impacts for those programs that are expected to reduce electricit y demand. The Protocols sometimes reference the
Evaluation Framework which provides examples of appli cable methods. The requirements,Evaluators’ Protocols Impact Evaluation

CPUC 33 TecMarket Works Team however, are always those stated in the Protocol s, which take precedence over all other protocols and evaluation guidance documents in all circum stances unless otherwise approved or required by the CPUC.

Table 2. Required Protocols for Gross Demand Evaluation

Rigor Level Minimum Allowable Methods for Gross Demand Evaluation

Basic Reliance upon secondary data for estimating demand impacts as a function of energy savings. End-use savings load shapes or end-use load shapes from one of the following will be used to estimate demand impacts:

1. End-use savings load shapes, end-use load shapes or allocation factors
from simulations conducted for DEER

2. Allocation factors from CEC foreca sting models or utility forecasting models with approval through t he evaluation plan review process

3. Allocation based on end-use savings load shapes or end-use load shapes from other studies for related programs/similar markets with approval through the evalua tion plan review process

Enhanced Primary demand impact data must be collected during the peak hour during the peak month for each utility system peak. Estimation of demand impact estimates based on these data is requi red. If the methodology and data used can readily provide 8,760-hour outpu t, these should also be provided. 23 Sampling requirements can be met at the program level but reporting must be by climate zone (according to CEC’ s climate zone classification).

1. If interval or time-of-use consumption data are available for participants through utility bills, these data c an be used for regression analysis, accounting for weather, day type and other pertinent change variables, to determine demand impact estimates. Pre- and post-retrofit billing periods must contain peak periods. Requires using power analysis, evaluations of similar programs, and professional judgment to determine sample size requirements for planni ng the evaluation. Needs to meet the requirements of the Sampli ng and Uncertainty Protocol.

2. Spot or continuous metering/measurement of peak pre and post-retrofit during the peak hour of the peak mo nth for the utility system peak to be used with full measurement Option B or calibrated engineering model Option D meeting all requirements as provided in the M&V Protocol. Pre-retrofit data must be adjust ed for weather and other pertinent change variables. Must meet t he Sampling and Uncertainty Protocol with a program target of 10% precision at a 90% confidence level.

3. Experimental design established within the program implementation process, designed to obtain reliable net demand savings based upon differences between energy consumption during peak demand periods between treatment and non-treatment groups from consumption data or spot or continuous metering. 24 Sampling must be according to the Sampling and Uncertainty Protocol.

23 This includes the use of 15-minute interval data or Building Energy Simulation models whose output is 8,760 hourly

data.

24 The overall goal of the Impact Protoc ols is to obtain reliable net energy and demand savings estimates. If the

methodology directly estimates net savings at the same or better rigor than the required level of rigor, then a gross savings and participant net impact analysis is not required to be shown separately.Evaluators’ Protocols Impact Evaluation

CPUC 34 TecMarket Works Team All M&V referred to in the Impact Evaluation Pr otocol must be planned, conducted and reported according to the M&V Protocol. M&V may be c onducted at a higher level of rigor, with more inputs measured or metered, or with greater precision than the minimum shown within the Impact Evaluation Protocol, but not with a lower le vel of rigor. The M&V Protocol can also be required by the Joint Staff or used by evaluato rs to enhance other evaluation efforts.

For the purposes of the Gross Demand Impact Pr otocol, demand impacts must be reported as energy savings estimates for six time periods for each of four months as follows: noon-1 p.m., 1- 2 p.m., 2-3 p.m., 3-4 p.m., 4-5 p.m. and 5-6 p.m. for June, July, August and September for each climate zone in which there are program particip ants. These demand savings are to be estimated using the Typical Meteorol ogical Year from the National Oceanic & Atmospheric Administration (NOAA), the CEC CTZ long-term average weather data, the Administrator’s long-term average weather year or th e CEC’s rolling average weather year.

The Joint Staff may require that specific studi es have additional reporting requirements to include reporting at th e 8,760-hour level or specific report ing for targeted transmission or
distribution areas. These will be decided on a ca se-by-case basis as part of the work scoping process or during the evaluation planning process. Identification of these requirements and how they will be met will be incorporated into th e evaluation plan and will be conducted and reported as approved within the evaluation planning process.

The Gross Demand Impact Protocol has two rigor levels: Basic and Enhanced. The Basic rigor level uses secondary data to al locate gross energy savings to determine demand savings. The Enhanced level requires primar y data collection either throug h field measurement according to the M&V Protocols or using regression analysis of demand or interval consumption data.

Basic Rigor The Basic rigor level for the Gross Demand Impact Protocol prescribes that at a minimum, onpeak demand savings are estimated based on allo cation of gross energy savings through the use of allocation factors, end-use load shapes or en d-use savings load shapes. These secondary data can be from DEER, the CEC forecasting model util ity end-use load shape data or other prior studies, with those in the la tter two categories needing re view and approval through the evaluation planning review process.

Enhanced Rigor The Enhanced rigor level for the Gross Demand Imp act Protocol requires primary data from the program participants. This could be interval -metered data, time-of-use (TOU) consumption billing data, from field measurement or from b illing demand data. (This latter is only allowable if the issues of when buildings peak versus de mand ratchets and peak periods are addressed in the analysis.) Estimation of peak demand savi ngs estimates is required. If the methodology and data used can readily provide 8,760-hour out put, these should be provided. Sampling requirements can be met at the program level but reporting must be by climate zone (according to CEC’s climate zone classification). The Join t Staff may require a program evaluation to use the Gross Demand Impact Protocol for transmi ssion and distribution (T&D) demand savings as they deem necessary. Demand evaluation require ments and the methods being employed to meet them need to be clear in the evaluation plan s and agreed upon through the evaluation planning review process.Evaluators’ Protocols Impact Evaluation

CPUC 35 TecMarket Works Team A regression model specified to measure program impacts for peak time periods (via analysis of interval data) or TOU/demand 25 consumption metering can be used to estimate program gross demand. This regression analysis must properly account for weathe r influences that are specific to the demand estimation and other pertinent ch ange variables (e.g., day-type and hours of occupancy). Regression analysis with interv al data should focus on obtaining direct demand impacts. If demand consumption data are used, a methodology to estimate demand savings based upon the demand regression analysis must be detailed in the evaluation plan and approved through the evaluation planning revi ew process. Pre- and post-r etrofit billing periods must contain peak periods within this analysis. A pow er analysis in combination with evaluations of similar program and professional judgment must be used to select and justify the proposed sample sizes. 26 The evaluation planning, analysis and re porting must meet th e requirements of the Sampling and Uncertainty Protocol.

The second class of primary data collection for the Enhanced Gross Demand Impact Protocol is to conduct field measurement of peak impacts with in the evaluation effort. Spot or continuous metering/measurement at peak pre- and post-re trofit will be conducted during the peak hour in the peak month for the utility system peak. Thes e data will be used with one of two engineering modeling approaches: (1) full measurement Option B or (2) calibrated engineering model Option D, where the modeling approach must meet all requirements as provided in the M&V Protocol.
An overview of the full measurement Option B method is provided in the Evaluation
Framework. 27 The calibrated engineering model Op tion D method is described and a few references are presented in the Evaluation Framework. 28 Further information and the specific requirements for the Protocols ar e provided in the M&V Protocol. Both of these engineering methods need to be designed to a program ta rget of 10 percent prec ision at a 90 percent confidence level and must meet the requirements of the Sampling and Uncertainty Protocol.

The third class of allowable methods is experi mental design with prim ary data collection. Experimental design with demand measuremen t comparisons between customers randomly assigned to the treatment and non-treatment groups meets the Enhanced Gross Demand Protocol rigor level. Experimental desi gn will need to measure energy sa vings during peak periods either through interval data or spot or continuous monitoring of comparison treatment and nontreatment groups to calculate demand savings esti mates. Currently, experimental design has not been widely used within efficiency evaluation. The Evaluation Framework provides a description and some examples of potential ex perimental designs within energy efficiency
efforts. 29 Sampling conducted as part of the expe rimental design must be conducted according to the Sampling and Uncertainty Protocol.

25 If demand billing is used, the research design must address the issues of building demand versus time period for

peak and issues with demand ratchets and how the evalua tion can reliably provide demand savings estimates.

26 Power analysis is a statistical technique that can be used (among other things) to determine sample size

requirements to ensure statistical si gnificance can be found. There are several software packages and calculation Web sites that conduct the power analysis calculation. Power analysis is onl y being required in the Protocol for determining required sample sizes. One of many possible references includes: Cohen, Jacob (1989) Statistical Power Analysis for the Behavioral Sciences , Lawrence Erlbaum Associates, Inc. Appendix D provides further detail on using power analysis for developing sample size requirements.

27 TecMarket Works, 166-169.

28 Ibid, 129-133 and 176-181.

29 Ibid, 104-105.Evaluators’ Protocols Impact Evaluation

CPUC 36 TecMarket Works Team 

### Participant Net Impact Protocol

The Participant Net Impact Protocols are summarized in Table 3. Further description, additional requirements, clarification and examples of these Protocols are presented be low the table. Being in compliance with the Particip ant Net Protocol means that th e methods used, and the way in which they are used and reported, meet all the re quirements discussed within this section. The intent is to provide reliable estimates of program level net energy and demand impacts when combined with the results from work complying with the Gross Energy Impact Protocol and the Gross Demand Impact Protocol. The Protocols sometimes reference the Evaluation Framework which provides examples of applicable methods. The requirements, however, are always those stated in the Protocols, which take precedence over all other protocols and evaluation guidance documents in all circumstances, unless othe rwise approved or required by the CPUC.

All M&V referred to in the Impact Evaluation Pr otocol must be planned, conducted and reported according to the M&V Protocol. M&V may be c onducted at a higher level of rigor, with more inputs measured or metered, or with greater precision than the minimum shown within the Impact Evaluation Protocol, but not with a lower le vel of rigor. The M&V Protocol can also be required by the Joint Staff or used by evaluato rs to enhance other evaluation efforts.

Table 3. Required Protocols for Participant Net Impact Evaluation

Rigor Level Minimum Allowable Methods for Participant Net Impact Evaluation

Basic 1. Participant self-report.

Standard 1. Participant and non-participant analys is of utility consumption data that
addresses the issue of self-selection.

2. Enhanced self-report method using other data sources relevant to the decision to install/adopt. These could include, for example, record/business policy and paper review, examination of other similar decisions, interviews with multiple actors at end-user, interviews with midstream and upstream market actors, Title 24 review of typically built buildings by builders and/ or stocking practices.

3. Econometric or discrete choice 30 with participant and non-participant comparison addressing the issue of self-selection.

Enhanced 1. “Triangulation” using more than on e of the methods in the Standard Rigor Level. This must include analysis and justification for the method for deriving the triangulation estimate from the estimates obtained.

All participant net impact analysis must be design ed to estimate the propor tion of savings that is program-induced and net of free-ridership estimates (not including spillover savings estimates).
This means that it is net of what would have o ccurred in the absence of the program. The degree to which the research design, selected method, survey instrument design, question wording and model specification can reliably capture this und erlying construct is th e evaluation’s construct

30 The instrumental-decomposition (ID) me thod described and referenced in the Evaluation Framework (page 145) is an allowable method that falls into this category. A propensity score methodology is also an allowable method in this category as described in: Itzhak Yanovitzky, Elaine Zanu tto and Robert Hornik,, “Estimating causal effects of public health education campaigns using propensity score methodology.” Evaluation and Program Planning 28 (2005): 209–220.Evaluators’ Protocols Impact Evaluation

CPUC 37 TecMarket Works Team validity. 31 These elements must work together a nd must be justified based upon how well they address construct validity.

Participant net impact analysis must address the following issues: Probability that the participant would have adopted the technology or behavior in the absence of the program (participa nt free-ridership);

• If adopted in the absence of the program, the probability or proportion (partial freeridership) of expected savings induced by the program given its ability to:

o Increase the efficiency of wh at would have been adopted;

o Make the adoption occur earlier than when it would have occurred; and

o Increase the quantity of efficient equi pment that would have been adopted.

• The estimation of participant net is c onsistent with decision-making behavior;

• Consistency is assessed to ensure that other fo rms of bias, such as, centrality bias, are not introduced;

• If survey methods are used, ensuring that survey questions (instrumentation) and techniques are employed to minimize social desirability bias;

• Results that include only free-ridership adjustment are clearly labeled as such;

• Report participant free-ridership and participant spillover separately where the methodologies selected allow this to be done;

• If at least some portion of participant spillover may be embedded within the gross savings estimates cannot be separated out using the estimation method chosen (e.g., a regression approach is used and the spillover behavior is simultaneous with program participation), clearly present why participan t spillover may be present within these estimates and a qualitative assessment of whether these might be expected to be significant or not compared to the program savings estimate; and

• If only participant free-ridership is pres ented in the report wi thout a reporting of participant spillover savings, clearly discuss that this presents a downwardly biased presentation of overall true net savings.

A general discussion of the net-to-gross principa ls, methods and a few references are presented
in the Evaluation Framework

. 32

The research design, selected method, survey in strument design or modeling specification(s) must also address participant self-selection bi as(es). Overall sample size targets can be by program. However, all survey or interview inqu iries concerning particip ant net (free-ridership and spillover, and application to gross impacts to obtain net sa vings) need to be conducted and measured by measure or end-use. Considera tions of uncertainty should guide the sample stratification plan.

31 Construct validity refers to the extent to which the operating variable/instrument/survey question accurately taps

and properly measures the underlying concept/abs tract idea that is designed to be measured.

32 TecMarket Works, 133-146.Evaluators’ Protocols Impact Evaluation

CPUC 38 TecMarket Works Team Basic Rigor – Self Reports Participant self-reports is the minimum allowable Basic rigor level method in the Participant Net Impact Protocol. The development of the survey instrument, scoring for responses, and handling of missing data and inconsistent responses need s to address those issues presented above and according to the Sampling and Uncertainty Protocol . A discussion of these issues can be found
in the Evaluation Framework

. 33

Like the other approaches to estimating the ne t-to-gross ratio (NTGR), there is no precision target when using the self-report method. However, unlike the estimation of the required sample sizes when using the regression and discrete choi ce approaches to estimating the NTGR, the selfreport approach poses a unique se t of challenges to estimating required sample sizes. These challenges stem from the fact that the self-re port methods for estimati ng free-ridership involve greater issues with construct validity and of ten include a variety of layered measurements involving the collection of both qualitative and quantitative data from various actors involved in the decision to install the efficient equipment. Such a situation makes it difficult to arrive at a prior estimate of the expected variance needed to estimate the sample size. Thus, in order to ensure consistency and comp arability, and eliminate potential gaming, this Protocol establishes a minimum sample size for the participant se lf-report method of 300 participant decision-makers for at least 300 par ticipant sites (where d ecision-makers may cover more than one site) or a census attempt, whichever is smaller. 34 An estimate of the achieved precision for net savings mu st be reported as well as a deta iled description of the method used
for its estimation.

Standard Rigor

There are three classes of allo wable methods to meet the minimum requirement for Participant Net Impact Protocol Standard rigor level.

Participant / Non-participant Comparison
The first of these is a compar ison of participant and non-partic ipant energy consumption that addresses participant self-selection bias. Some of the potential methods to be used are described
in the Evaluation Framework. 35 The evaluation plan and report need to include an analysis and explanation of why the select ed research design, methodology and actual model specification were selected. A power analysis in combina tion with evaluations of similar program and professional judgment must be used to sele ct and justify the proposed sample sizes. 36

33 Ibid, 136-140.

34 This is considered the best feasible a pproach at the time of the creation of this Protocol. Alternative proposals and

the support and justifications that address all of the issues discussed her e on the aggregation of variance for the proposed self-report method may be submitted to Joint Staf f as an additional option (but not instead of the Protocol requirements) in impact evaluation RFPs and in Evaluation Plans. Joint Staff may elect to approve an Evaluation Plan with a well justified alternative.

35 TecMarket Works, 142-145.

36 Power analysis is a statistical technique to determine sa mple size requirements to ensure statistical significance

can be found. There are several software packages and calculation Web sites that conduct the power analysis calculation. See the Sampling and Uncert ainty Protocol for more discussion and reference. Power analysis is only being required in the Protocol for determining required samp le sizes. Appendix D provides further detail on using power analysis for developing sample size requirements.Evaluators’ Protocols Impact Evaluation

CPUC 39 TecMarket Works Team Program-Specific Enhanced Self Reports
The second allowable method is a program-speci fic enhanced self-report one that draws upon multiple data sources concerning the decision to in stall/adopt. These could include, for example, record/business policy and paper review, examinati on of other similar decisions, interviews with multiple actors at the end-user, interviews with mid-stream and upstream market actors, Title 24 review of typical buildings by builders and st ocking practices. For comm ercial/industrial entities multiple decision makers within a firm/corporation could be interviewed, as well as reviews of records and policy documents, and inquiries in to decision-making. It also could draw upon either primary data collection or secondary data collection if available on the same California market (from market assessment studies or mark et effects studies recently completed). The
enhanced method could also incl ude engineering components to assist in determining what would have occurred in the absence of the program. Data collected from such multiple sources would be used to triangulate on an estimate of the participant free- ridership and spillover rate for that program. A brief di scussion of some of these types of methods and examples is provided in
the Evaluation Framework

. 37

Like the other approaches to estimating the NTGR , there is no precision target when using the self-report method. However, un like the estimation of the require d sample sizes when using the regression and discrete choice approaches, the self-report appr oach poses a unique set of challenges to estimating required sample sizes. Thes e challenges stem from th e fact that the selfreport methods for estimating free-ridership involv e greater issues with construct validity and often include a variety of layered measurements involving the collection of both qualitative and quantitative data from various acto rs involved in the decision to in stall the efficient equipment. Such a situation makes it difficult to arrive at a pr ior estimate of the expected variance needed to estimate the sample size. This Protocol, instea d, establishes a minimum sample size for end-use participants: a sample of 300 part icipant decision-makers for at le ast 300 participan t sites (where decision-makers may cover more than one site) or a census attempt, whichever is smaller. Sample sizes of other actors, engineering work or record review need to be described in the evaluation plan and approved through the evaluation planning review process.

Econometric or Discrete-Choice Analysis
The third allowable method in the Standard rigor le vel is econometric or di screte-choice analysis of participant and non-part icipants that addresses participant self-selection bias. An overview of
some of these methods and a few references can be found in the Evaluation Framework

. 38 The evaluation plan and report need to include an analysis and explanati on of why the selected research design, methodology and actual model specifi cation were selected. A power analysis in combination with evaluations of similar progra ms and professional judgment must be used to select and justify the proposed sample sizes.

Two of the Standard rigor level methods require comparisons with non-participants. It is important that care be taken for selecting th e appropriate comparison group. There is not a single rule about what constitutes an appropria te comparison group, since the selection of the group depends on such factors as type of market transaction, methodology or comparison

37 TecMarket Works, 141-142.

38 Ibid, 142-145.Evaluators’ Protocols Impact Evaluation

CPUC 40 TecMarket Works Team purpose. Yet, this should be carefully consid ered and the proposed non-participant comparison group and the criteria used in se lecting this group should be discu ssed in the evaluation plan, and reviewed and approved through the ev aluation planning review process.

Enhanced Rigor – Comparison of Multiple Approaches One of the primary concerns with measurements of participant net is of construct validity. Given this, the Enhanced rigor level requires the use of at least two of the Stan dard rigor level methods to triangulate 39 on an estimate of participant net. This must include analysis and justification for the method for deriving the triangulation estimate, not solely on averages, from the estimates
obtained.

Participant net savings evaluation includes the evaluation of free-ridership and participant spillover. Presenting both yields a more accurate picture of what the participant would have done in the absence of the program and the fu ll impacts of the program. The evaluation plan, analysis and report must address how the met hods were selected and how the analysis was conducted. Net of free-ridership (N et of FR) estimates must be pr ovided in the evaluation report. Current CPUC policy, as the Protocols are being developed, is that only program savings and not participant spillover will be counted towards prog ram and Administrator goals and performance.
These are the Net of FR estimates.



### Indirect Impact Evaluation Protocol

The Indirect Impact Evaluation Protocol is th e minimum standard Protocol for programs that seek to change the behavior of consumers and for which some level of gross energy and demand savings is expected. These programs are typica lly information, education, marketing, promotion, outreach or other types that may not have specified energy savings goals, but are still expected to provide energy impacts within their target markets. The Protocol has multiple levels of rigor that
can be used to conduct the evaluations. On ce a minimum rigor level is assigned for an evaluation, all methods in a highe r class of rigor are allowable, as they exceed the minimum criteria. For example, if the program has an assigned the Stan dard rigor level and the method selected for implementation is an option under the Enhanced rigor level (but is not listed under the Standard rigor level), this method is acceptable for meeting the Protocol.

The Indirect Impact Evaluati on Protocol is summarized in Table 4. A discussion of behavioral impact evaluation and selected references are provided in the Evaluation Framework

. 40 Further description, additional requirements, clarification and examples of th is Protocol follow the table. In order to comply with the Indi rect Impact Evaluation Protocol the methods used and the way in which they are used and reported must meet all the requirements discussed within this section. The intent is to provide reliable estimates of program level impacts and, when required, gross energy and demand impacts. The Protocols sometimes reference the Evaluation Framework and

39 A strict dictionary definition of triangulation would m ean incorporating three measurements. The term is used here

to mean a process of analysis that examines at least tw o measurements and assesses what their differences might
mean. Then the best estimate derived from this exam is determined to properly represent the underlying construct to meet construct validity issues to obtain the most re liable estimate from the multiple analyses conducted.

40 TecMarket Works, 234-242.Evaluators’ Protocols Impact Evaluation

CPUC 41 TecMarket Works Team other documents which provide examples of a pplicable methods. The requirements, however, are always those stated in the Protocols, which take precedence over all other protocols and evaluation guidance documents in all circumstances, unless otherwise approved or required by
the CPUC.

All M&V referred to in the Impact Evaluation Pr otocol must be planned, conducted and reported according to the M&V Protocol. M&V may be c onducted at a higher level of rigor, with more inputs measured or metered, or with greater precision than the minimum shown within the Impact Evaluation Protocol, but not with a lower le vel of rigor. The M&V Protocol can also be required by the Joint Staff or used by evalua tors to enhance other evaluation efforts.

Table 4. Required Protocols for Indirect Impact Evaluation

Rigor Level Minimum Allowable Methods for Indirect Impact Evaluation

Basic An evaluation to estimate the progra m’s net changes on the behavior of the participants is required; the impact of the program on participant behavior.

Standard A two-stage analysis is required that will produce energy and demand savings. The first stage is to conduct an evaluation to estimate the program’s net changes on the behavior of the participa nts/targeted-customers. The second is to link the behaviors identified to estimates of energy and demand savings based upon prior studies (as approved through the evaluation planning or evaluation review process).

Enhanced A three-stage analysis is required that will produce energy and demand savings. The first stage is to conduct an evaluation to estimate the program’s net impact on the behavior changes of the participants. The second stage is to link the behavioral changes to estimates of energy and demand savings based upon prior studies (as approved through the evaluation planning or evaluation review process). The third stage is to conduct field observation/testing to verify that the occurrence of the level of net behavioral changes.

Basic Rigor In this Protocol, programs or program compone nts are assigned by Joint Staff to receive an Indirect Impact Evaluation if th e program’s primary goal is to pr oduce behavioral changes. The primary uncertainty within the logic chain of obtaining energy and demand savings from these types of programs is the estimation of th e program-induced impact on the behavior of participants. Therefore, the prim ary focus of the Indirect Impact Evaluation is in evaluating and estimating the program’s net impact on behavioral change. This is the primary component for the evaluation research design.

There are several types of resear ch design that could be used fo r conducting an Indirect Impact Evaluation. There are many social science me thodologies that could apply depending upon the program goals, logic, program design and mark et operation. Guidance for these types of
evaluations can be found in the Evaluation Framework

. 41

Indirect impact evaluation de sign, analysis and reporting must address the following issues:

41 Ibid, 234-242. Much of the guidance provided from the Evaluation Framework chapter on Market Transformation Evaluation (pages 245-268) can also provide useful insights and references.Evaluators’ Protocols Impact Evaluation

CPUC 42 TecMarket Works Team • Expected impacts and the target audience for these impacts;

• How the expected impacts will be measured;

• Identification and measurement of baseline (a nd where baseline would have been in the absence of the program, i.e., forecasted, dynami c baseline or estimated counter-factual from research design) or identification a nd measurement of well-matched non-treatment comparison group over time;

• Extent of exposure/treatment and how this is being measured in the evaluation; and

• Self-selection bias and how this is being cont rolled for to obtain an unbiased estimate of the program-induced impact.

The assessment or development of a program theory and logic model (PT/LM) is
recommended. 42 The PT/LM could be particularly usef ul if expanded to include the expected
interactions with the market or the use of behavioral change mode ls. These can be valuable as a foundation for the evaluation research design, res earchable questions a nd basis for developing survey/interview questions. Though a PT/LM is not required, it is an impor tant tool to ensure
that the evaluation research desi gn can measure the program’s beha vioral impacts. A detailed evaluation research design for the Indirect Impact Evaluation is required and must be reviewed and approved through the evaluati on planning review process.

All sampling must be done in accordance with the Sampling and Uncertainty Protocol. Any sampling for regression analysis must use power analysis in combination with evaluations of similar program and professional judgment to determine required sample sizes.

Methodologies using a treatment/non-treatment group comparison that include controlling for self-selection are encouraged. Methods coul d include the enhancem ent of those methods
described in the Evaluation Framework

. 43 There are also many other methods used in other evaluation fields that could be found to be equa lly or more valid. One possible example is the use of the propensity scoring method that has been used to evaluate public health campaigns and control for the selectivity bias in treatment levels. 44 The evaluation plan and report need to include an analysis and explanation of why the research design, methodology and actual model specification were selected.

Standard and Enhanced Rigor Levels In the Standard and Enhanced rigor levels , evaluation studies are conducted to link net behavioral impacts to energy and demand saving impacts based upon prior studies. These prior studies do not need to be previously completed eval uations (however this is preferred if they are available). For example, linking net behavior change savings estimates using DEER will meet the Indirect Impact Evaluation Prot ocol. Linking savings estimates to past evaluations of similar programs, new engineering models for savings es timates or other studies must be approved by the Joint Staff through the ev aluation review process.

42 Ibid, 30-38 and 45-48.

43 Ibid, 142-145.

44 Yanovitzky, Zanutto and Hornik, 209 – 220. Also included (with additional references) in a review of possible

net/causality methods for energy efficiency evaluation in: Lisa Skumatz, Dan Violette, and Rose Woods, “Successful Techniques for Identifying, Measuring, and Attributing Causality in Efficiency and Transformation Programs.” Proceedings of the 2004 ACEEE Summer Study in Buildings (2004): 2.260 – 2.273.Evaluators’ Protocols Impact Evaluation

CPUC 43 TecMarket Works Team A behavioral impact program (through informati on, education, training, advertising or other nonmonetary incentive efforts) may be part of a portfolio to lead customer/market actors into other programs. This program/program component could be assigned an Indirect Impact Evaluation to determine the impact the program(s) is having on the portfolio and to provide input for the process evaluation of the program. An assignment of the Standard rigor le vel requires that an impact evaluation be conducted and linked to en ergy and demand savings estimates. The energy and demand savings, however, woul d not, in this case, be added to the portfolio level savings unless a method is used and approved by the Joint Staff to ensure that these savings are not
double counted with those attrib uted to other programs.

Four types of impacts from a behavi oral change program are shown in Figure 5. Inducing customers into other programs is shown as Path A. Savings from this path are not direct savings due to the information, educati on, training or advertising prog ram under study. The savings are those obtained through the direct program. However, documenting the impacts of this effort is important to estimate the various components that contri bute to generating a portfolio’s savings and to aid in making investment decisions. An example might be customers who participate and obtain high-efficiency room air c onditioners through a re bate program due to behavioral impacts from the program being evaluated.

Install measure through direct

program Induced into Direct Program Behavioral Program Effect

Behavior change with no equipment

re placement/added

Observable/
measurable behavior change Non

- verifiable behavior change, too small to measure effects, intermittent change, or very long

- term Induced to directly install measure without other program participation A

B

C D

Figure 5. Potential Alternative Behavioral Impact Paths

Programs or program components that directly influence customer behavior to purchase high efficiency replacement equipment or add equipmen t that can save energy (e.g., timers) are shown as Path B. If assigned an Indire ct Impact Evaluation with a Sta ndard or Enhanced rigor level, these programs would be expected to undertake similar evaluation de signs to those in Path A. The energy and demand savings for these, however, are directly attributable to the program effort being evaluated. The research design may need to estimate a nd find the proportion of customersEvaluators’ Protocols Impact Evaluation

CPUC 44 TecMarket Works Team that take these actions outside of other program s. An example might be customers who purchase high efficiency room air-conditi oning due only to this progra m and who did not receive any
financial incentives from othe r portfolio efforts to do so.

Path C refers to those program-induced behavioral changes that can be observed or measured but are not tied to equipment replacem ent or the addition of equipment. This could include such changes as those to business policies regarding energy efficiency, archite cts’ decisions on when to test daylighting alternatives , and/or plant managers’ operati ng and maintenance schedules.

Path D represents behavioral changes that are too sm all, long-term or intermittent to be costefficiently verified through obs ervation, field-testing or survey ing with enough reliability to measure any energy and demand impacts. Depe nding on the level of investment and the
advances made in the evaluation of behavioral change, the programs or program components that fall into this category could vary over time. Pa th D examples include residential behavior of turning off lights, educating children through sc hool programs to cha nging their energy-use behavior when they are adults, and changes in re sidential thermostat set points. The Joint Staff will only assign a Basic rigor level for this categor y if meeting a higher rigor level would not be possible. This could occur because a specific estimate of the degree of the impact cannot be obtained cost-effectively or th e link and translation to ener gy and demand savings is not available or cost-effective to develop.

Every program evaluation is required to dem onstrate that the program is accomplishing its primary goals of affecting behavioral change, as stated in its PT/LM.

It is expected that the Indirect Impact Evalua tion for paths A, B and C will be assigned either a Standard or Enhanced rigor le vel depending upon the size of res ources being invested and the importance of the anticipated outcomes to the ove rall success of the portfolio. The indirect impact evaluation for an Enhanced rigor level is distinguished from a Sta ndard rigor level by the requirement to conduct field observa tion/testing to verify net change s in behavior. For Path D it is expected that only a Basic rigor level will usually be assigned. The evaluation design for each path is briefly described below.

Path A : The evaluation design to veri fy these actions is most straightforward for Path A. Verification through program par ticipation is sufficient given these programs are conducting their own verification and impact evaluation.

Path B : The evaluation design for Path B require s the additional step of finding effected customers. This step would have to be pa rt of the evaluation de sign when estimating the proportion affected in the impact evaluation. The evaluation plan must propose the research design to accomplish this and be approved within the evaluation planning review process.

Path C : The evaluation research design needed to accomplish an Enhanced rigor indirect impact evaluation following Path C is more challenging. A Path C evaluation plan needs to be presented in enough detail for its logic and potential reliability to be revi ewed as part of the evaluation planning review process. Examples of Path C activities include review of pre- and post-program architectural plans, review of government policy, pl anning and hearing documentsEvaluators’ Protocols Impact Evaluation

CPUC 45 TecMarket Works Team and their dates of adoption along with interv iew support, examination of business policy manuals, and review of business programs cr eated due to education efforts and testing subsequent employee knowledge and reported actions.

Path D : For path D, the Basic level rigor indirect impact evaluation must be used to demonstrate that the program has carried out specific activit ies that are designed to produce behavioral change.



### Guidance on Skills Required to Conduct Impact Evaluations

The Impact Evaluation Protocol includes gross energy and demand impact Protocols, Protocols for participant net impacts and a Protocol for indirect impact evalua tion. There are multiple
methods within these various Protocols that cr eate the need for different skills depending upon the method that is being used. The method em ployed determines the skills and experience requirements for that method. The senior, advi sory and leadership personnel for an impact evaluation effort must have the sp ecific skills and experience fo r the method they are leading and the time budgeted for responsible project task leadership and quali ty control. The degree of
involvement needed from senior skilled staff is dependent upon the skill and experience of the mid-level personnel conducting much of the analysis work.

Several of the energy, demand and participant ne t methods use statistical/econometric methods. These are used with utility demand metering and consumption data, and with data gathered for decision analysis (in the case of discrete choice). The use of statistical/econometric methods requires personnel trained in these methods and/or with significant expe rience in using them. This experience and/or training must include testing alternative speci fications, testing and correcting for violations of regression assumpti on violations, and using them within the context of program evaluation.

Another class of methods relies on engineering type methods that draw upon the rules of physics
to calculate estimates of ener gy and demand savings. Simple engineering equations can be understood and used by most people with a general science backgr ound. Yet, to ensure reliable use of the principles, impact evaluations using the simple engineering model should still use personnel with experience in this area, Certified Energy Managers 45 or personnel with training in mechanical or architectural engineering. Building energy simulation models and process engineering models generally require personnel with a college degree in mechanical or architectural engineering or si gnificant related, equivalent ex perience. Process engineering models may also require specific engineering ex perience or research re garding the industrial process or facilities being studied.

There are methods within the Gross Energy Im pact Protocol (e.g., e nhanced gross energy regression-based (enhanced 1.)) th at could employ significant primar y survey or interview data collection. The participant net impact methods that employ the self-report and enhanced selfreport approaches require simila r experience and training. The ev aluators using these methods should have sufficient experience implementing surv eys, interviews, group interviews and other types of primary data collecti on activities as are being reco mmended. They need to have

45 The Association of Energy Engineers (AEE) offers course s and a certificate for a Certif ied Energy Manager (CEM).Evaluators’ Protocols Impact Evaluation

CPUC 46 TecMarket Works Team experience in energy efficiency markets, the social sciences, and interview and survey instrument design, implementation and analysis.

Indirect impact evaluation me thods could be based upon survey and interview analysis methods
and/or statistical/econometric methods. The evaluators must be trained and experienced in conducting social science resear ch with a strong understanding of assessing and testing causal relationships between exposur e to the program and possibl e outcomes. An important requirement for these evaluators is to have a st rong foundation in research design and the ability to create research designs to test for net behavioral impacts of energy efficiency programs.



### Summary of Protocol-Driven Impact Evaluation Activities

1 The Joint Staff identifies which programs and program components will receive an impact evaluation and identify the type of impact evaluation(s) to be conduct ed and at what rigor level.

2 The Joint Staff determines any special needs on a case -by-case basis that will be required from particular program or program component evaluations. CPUC-ED issues request for proposals for impact evaluations, selects evaluation contract ors and establishes scope(s) of work.

3 Program theory and logic models (PT/LM), if available, must be reviewed/assessed as needed to properly identify impacts and evaluation elem ents required to assess net program impacts. Research design and sampling plan developed to m eet Protocol requirements at a program or program component basis as designated by the Joint Staff rigor level assignments. This includes meeting requirements from the Sampling and Unce rtainty Protocol, M&V Protocol and Reporting Protocol, as are applicable given Impact Evaluati on Protocol requirements. Research design and sampling must be designed to meet any of the Joint Staff requirements for additional analyses including, but not limited to, the estimation of net impacts by delivery mechanism, the estimation of transmission and/or distribution benefits, or other ar eas designated of specific concern by the Joint Staff. Develop Evaluation Plan, submit it to the CPUC-ED and revise as necessary to have an approved Evaluation Plan that meets th e Impact Evaluation Protocols.

4 All impact evaluation teams must be staffed so as to meet the skills required for the research design, sampling, appropriate and selected impact evaluation method, uncertainty analysis, and reporting being planned and conducted.

5 Develop precise definitions of participants, non-participants and comparison groups. Obtain
concurrence with the CPUC-ED on these definit ions which are to be used in developing the research design and sampling plans.

6 All impact evaluations must meet the requirem ents of the Sampling and Uncertainty Protocol.

6.a There are 2 primary sampling considerations for regression-based consumption analysis.

(1) Unless a census is utilized, conduct a power analysis to estimate the required sample size. One may also consider prior evaluations for similar programs and professional judgment (must use all of t hese for the Enhanced level of rigor); and

(2) Must use a minimum of 12 months pre a nd post-retrofit consum ption data, except when program approach does not allow pre-retrofit data (e.g., new construction).

6.b All engineering-based methods must:Evaluators’ Protocols Impact Evaluation

CPUC 47 TecMarket Works Team (1) Estimate the uncertainty in all deemed and measured input parameters and consider propagation of error when determining measured quantities and sample sizes to meet the required error tolerance levels; and

(2) Use a combination of deemed and measured data sources with sufficient sample sizes designed to meet a 30% error toleranc e level in the reported value at a 90% confidence level to meet the Basic rigor level and a 10% error tolerance level at a 90% confidence level for the Enhanced rigor level.

6.c Participant and non-participant comparisons and econometric/discrete-choice methods for Participant Net Impact evaluation will use power analysis combined with examinations of prior evaluation studies for similar programs to derive required sample sizes.

6.d Self-report and Enhanced self-report methods for Participant Net Impact evaluations must at a program level have a minimum sample size of 300 participant decision-makers for at least 300 participant sites (where decision-makers may cover more than one site) or a census attempt, whichever is smaller, (while investigation will be at a measure or end-use level).

7 All impact evaluations must be planned, conduct ed, analyzed and reported to minimize potential bias in the estimates, justify the methods selected for doing this and report all analysis of potential bias issues as described in the Sampling and Uncertainty Protocol, Impact Evaluation Protocol and M&V Protocol. Primary considerations that must be addressed (based upon method employed)
are as follows:

7.a Regression-based consumpt ion analysis must incorporate:

(1) Addressing the influence of weather when weather sensitive measures have been included in the program evaluation;

(2) Assessing potential bias given inclusion/ exclusion issues due to the 12 month preand post-retrofit consumption minimum requirement;

(3) For the Enhanced rigor level, assess , plan, measure and incorporate background and change variables that might be expect ed to be correlated with gross and net energy and/or demand savings;

(4) Comparison groups must be carefully select ed with justification of the criteria for selection of the comparison group and di scussion of any potential bias and how the selected comparison group provides the best available minimization of any potential bias; and

(5) Interval or TOU consumption data for demand impact analysis must contain the peak period for the utility system peak. If demand billing data is used for demand impact analysis, the research design must address the issues of building demand versus time period for peak and issues with demand ratchets and how the evaluation can reliably provide demand savings estimates. Demand savings must be reported by CTZ.

7.b Engineering-based meth ods must incorporate:

(1) Addressing the influence of weather when weather sensitive measures have been included in the program evaluation;

( 2 ) Meetin g all the re q uirements in the M&V Protocol includin g issues of baselineEvaluators’ Protocols Impact Evaluation

CPUC 48 TecMarket Works Team determination; and

(3) For the Enhanced rigor level of demand impact analysis using spot or continuous metering/measurement pre- and post-retrofit for the peak hour of the peak month for the utility system peak. Demand savings must be reported by CTZ.

7.c Experimental design must use spot or c ontinuous metering/measurement pre and postretrofit for the peak hour of the peak mont h for the utility system peak for determining demand impacts. Demand savings must be reported by CTZ.

7.d Indirect impact analys is must incorporate:

(1) Description of expected impacts (direct behavioral and indirect energy and demand impacts) and how they will be measured;

(2) Discussion of identification and measurement of baseline;

(3) Extent of exposure/treatment and its measurement;

(4) Comparison groups must be carefully select ed with justification of the criteria for selection of the comparison group and disc ussion of any potential issues of bias and how the selected comparison group provides the best available minimization of potential bias; and

(5) Assessing, planning for and analyzing to control for self-selection bias.

8 Regression analysis of consumption data must address outliers, missing data, weather adjustment, selection bias, background variables, data screens, au tocorrelation, truncation, error in measuring variables, model specification and omitted vari able error, heteroscedasticity, collinearity and influential data points. These areas must be addr essed and reported in accordance with the Sampling and Uncertainty Protocol.

9 Engineering analysis and M&V based methods are r equired to address sources of uncertainty in parameters, construction of baseline, guarding against measurement error, site selection and nonresponse bias, engineering model bias, modeler bias, deemed parameter bias, meter bias, sensor placement bias and non-random selection of equipment or circuits to monitor. These areas must be addressed and reported in accordance with the Sampling and Uncertainty Protocol.

10 Develop draft evaluation report to include meeti ng all requirements in the Reporting Protocol and incorporating the program’s performance metrics.

11 Develop final evaluation report in accordance with guidance provided by the Joint Staff. Submit final evaluation report to the CPUC-ED.

12 Once accepted by the CPUC-E D, develop abstracts and post them and report on CALMAC Web site following the CALMAC posting instructions.

Note: The steps included in this evaluation summary table must comply with all the requirements within the Impact Evaluation Protocol.Evaluators’ Protocols Measurement and Verification

CPUC 49 TecMarket Works Team 

## Measurement and Verification (M&V) Protocol

Introduction

When, in the course of conducting evaluations, it becomes necessary or advisable to collect physical evidence from field installations of ener gy efficiency technologies, the evaluator must design, document and implement a measurement and verification (M&V) project. M&V will typically be used to support impact studies by providing measured quantitative data from the field. One of the primary uses is to reduce unc ertainty in baselines, e ngineering calculations, equipment performance and operational parameters. However, M&V can be used in process and market effects evaluations as well, when such data are useful for understanding issues such as measure quality and suitability for particular ap plications, installation practices and quality, baseline equipment efficiency and operation prac tices, and other issues identified by the process and/or market effects evaluation plan. For the purposes of this Protocol, M&V will cover all
field activities dedicated to coll ecting site engineering informati on. This includes such activities as measure counts, observations of field conditio ns, building occupant or operator interviews conducted in-person, measurements of parameters, and metering and monitoring.

How M&V differs from impact evaluation : M&V refers to data collection, monitoring and analysis activities associated with the calculation of gross energy and peak demand savings from individual customer sites or projects. Gross a nd net impacts at the progr am level will be guided by the Impact Evaluation Protoc ol, where results from M&V st udies conducted on a sample of
sites will be combined with other information to develop an overall estimate of savings by
program or program component. 46

Sources of uncertainty in engineering estimates : Engineering estimates are based on the application of the basic laws of physics to the calculation of energy consumption and energy savings resulting from the implementation of energy-efficient equipment and systems. Engineering models range from simple one-line algorithms to systems of complex engineering equations contained within a building energy simu lation program such as DOE-2. Uncertainty in engineering estimates stems from uncertainty in the inputs to an engineering model and the uncertainty in the ability of the algorithms to predict savings.

Uncertainty analysis and M&V planning: Energy efficiency programs utilize a wide range of
technical and behavioral tools and concepts as “measures.” The likelihood of success of the measure depends on a large number assumptions , many of which can be verified through
measurement. Measured data from field studie s are used to quantify and reduce the uncertainty in energy and peak demand impact calculations. Wh ile this Protocol is written to support the overall goal of creating more re liable savings estimates and fo recasts, we recognize that M&V activities must be planned and resources must be allocated to reduce these uncertainties. Uncertainty analysis conducte d during the planning phase sha ll be used to identify the assumptions that have the greate st contribution to th e overall savings uncertainty and allocate resources in an appropriate manner to address these uncertainties.

46 It is possible that some impact evaluations will not requ ire M&V. See the Impact Evaluation Protocol herein for

more information.Evaluators’ Protocols Measurement and Verification

CPUC 50 TecMarket Works Team The development of this Protocol is driven by the desire to create and implement a rational framework to identify and conduct a wide range of M&V activities. As the Joint Staff recognizes that precision is a key requirement in forecasting and reporting, it will seek to allocate
resources such that the value of the M&V activi ties is applied to iden tify, quantify and manage risk associated with the uncertainty in the e xpected savings from meas ures and programs. The Protocol supports the overal l M&V goals and priorities established by the Joint Staff:

• Improve reliability of savings estimates;

• Determine whether energy and peak demand savings goals have been met;

• Improve DEER estimates of energy and peak demand savings; and

• Inform future program planning and selection processes.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• Joint Staff Evaluation Planners should understand the uncertainty in the overall energy and peak demand savings calculations and identify the degree to which field measurements can reduce that uncertainty (at appropriate cost);

• The Evaluation Project Team will use field measurements to calculate gross savings estimates and answer specific process and market effects evaluation questions;

• Administrators and IOUs will use M&V project results to refine unit savings estimates and/or engineering parameters used in futu re program planning, and utilize early and midstream M&V findings to adjust program priorities within the portfolio;

• Program Implementers will use early M&V project results to revise program delivery approaches and measures;

• Site Owners should allow access to site for field meas urements and may have an interest in the energy savings resulting from efficiency upgrades subject to the M&V effort; and

• DEER Planners will use field data to develop, ca librate and generally improve DEER energy and demand savings estimates.



### Overview of the Protocol

This M&V Protocol is intende d to set guidelines for conduc ting and reporting field data
collection activities in support of energy efficiency program evaluations. The M&V Protocol covers the following issues:

• M&V framework;

• Requirements for installation verification;

• M&V requirements;

• M&V approach examples;Evaluators’ Protocols Measurement and Verification

CPUC 51 TecMarket Works Team • Project reporting and docu mentation requirements;

• Sampling strategies; and

• Skills required for conducting M&V activities.

For more information on conducting M&V studies, please refer to the Evaluation Framework

. 47



### M&V Framework & Language

M&V projects conducted under this Prot ocol shall adhere to the IPMVP, 48 with additional criteria specified herein. The IPMVP is a flexible framework th at allows users to craft M&V plans for specific projects with consideration of:

• The type of contractual arrangement in force;

• The types and quantities of uncertainty in the project savings estimate; and

• The cost to create the M&V plan and conduc t all activities in th e plan, including:

o Meter and sensor placement;

o Data collection; and

o Data analysis and reporting.

Whereas field measurements are an important co mponent of program impact estimation and the IPMVP is written to allow users flexibility, it s application requires a thorough knowledge of measure performance characteristics and data ac quisition techniques. Building and energy using facilities in general tend to vary widely in term s of the electrical and mechanical infrastructure that supplies the energy commodity. A measuremen t strategy that is simple and cheap in one building (such as measuring lightin g energy at a main panel) may be much more expensive in a similar building that is wired differently. For this reason, M&V resources , costs and benefits must be called upon and allocated consid ering site-specific characteristics.



### Relationship of the M&V Pr otocol to Other Protocols

The M&V Protocol is a subset of the Impact Evaluation, Process Evaluation and Market Effects
Protocols. M&V activities described within this Protocol are initiated by these three Protocols. Not every evaluation study will require M&V. When M&V is indicated, the M&V Protocol provides the requirements for meeting the variou s levels of required M&V and points to the applicable pages of the Evaluation Framework for more guiding information and references.

Sampling activities conducted within the M&V projec ts prescribed within this Protocol shall be conducted in accordance with the Sampling Protoc ol. Impact and process evaluation studies calling for M&V data will include a site selection sampling Protocol.

47 TecMarket Works, pages 147-204.

48 The IPMVP provides four options for conducting M&V studies. Option C – Whole Facility, is very close in concept

to a statistical billing analysis and it is covered under the Impact Evaluation Protocol to avoid confusion.Evaluators’ Protocols Measurement and Verification

CPUC 52 TecMarket Works Team 

### Key Metrics, Inputs and Outputs

M&V studies, since they are directed by the Imp act Evaluation and/or th e Process or Market Effects Protocols, will draw upon the same data sources, such as data from program databases, program descriptions, DEER, work papers provided by program implementers, utility demand metering and consumption data for both particip ants and non-participants, utility weather data, on-site measurement, monitoring and observationa l data, survey and interview data collection, and other prior study data and reports. These will be used as directed by the M&V Protocol to produce measure-level energy and peak demand sa vings for sampled sites as directed by the Impact Evaluation Protocol. The overall inform ation inputs and outputs to the M&V process are
shown in Figure 6.

Because M&V studies are required to follow the Sampling and Uncertainty Protocol, evaluators must also assess, minimize and mitigate potential bias and present the achieved level of precision including relative precision, e rror bounds on M&V results in s upport of the impact evaluation
effort.

All M&V reporting must also follow the Reporting Protocol. Verificat ion-only output metrics
are defined as the fraction of installed measures that meet the provisions of the M&V Protocol.Evaluators’ Protocols Measurement and Verification

CPUC 53 TecMarket Works Team Figure 6. Measurement & Verification Information Flow DiagramEvaluators’ Protocols Measurement and Verification

CPUC 54 TecMarket Works Team 

### Site-Specific M&V Plan

In requiring the adherence to the IPMVP, this Protocol requires submittal of an M&V plan for each field measurement project undertaken that do cuments the project procedures and rationale
such that the results can be audited for accur acy and repeatability. Within the guidelines established by the IPMVP and the Pr otocols, there is considerable latitude for the practitioner in developing a site-specific M&V plan and impl ementing the plan in the field. The M&V
contractor shall evaluate the uncertainty in the desired data product and develop a site-specific M&V plan that manages the uncertainty in the most cost-effective manner.

Initial estimates of engineering parameter uncertain ties should be used to provide an estimate of the overall uncertainty in the savings calculations. Assumptions used to create initial estimates of parameter uncertainty values should be docum ented. The contribution of specific engineering parameters to the overall uncertainty in the savi ngs calculations should be identified and used to guide the development of the M&V plan.

The M&V plan must include the following sections: 49

1. Goals and Objectives;
2. Building Characteristics;
3. Data Products and Project Output;
4. M&V Option;
5. Data Analysis Procedures and Algorithms;
6. Field Monitoring Data Points;
7. Data Product Accuracy (including data acquisi tion system accuracy and sensor placement issues);
8. Verification and Quality Assurance Proce dures (including sensor calibration); and
9. Recording and Data Exchange Format.

The content of each of these se ctions is described below.

Identify Goals and Objectives: The goals and objectives of th e M&V project should be stated explicitly in the M&V plan.

Specify Site Characteristics: Site characteristics should be documented in the plan to help
future users of the data unders tand the context of the monitored data. The site characteristics description should include:

• General building configuration and envelope ch aracteristics, such as building floor area, conditioned floor area, number of building floors, opaque wall area and U-value; window area, U-value and solar heat gain coefficient;

• Building occupant information, such as number of occupants, occupancy schedule, building activities;

• Internal loads, such as lighting power de nsity, appliances, plug and process loads;

49 See the Evaluation Framework , pages 147-153.Evaluators’ Protocols Measurement and Verification

CPUC 55 TecMarket Works Team • Type and quantity and nominal efficien cy of heating and cooling systems;

• Important HVAC system control set points;

• Changes in building occupancy or operation dur ing the monitoring period that may affect results; and

• Description of the energy conservation measures at the site and thei r respective projected savings.

Specify Data Products and Project Output: The end products of the M&V activity should be specified. These data products should be referen ced to the goals and ob jectives on the project and include a specification of the data formats and engineering units, wi th reference to the Reporting Protocol Appendix A.

Specify M&V Option : The M&V option chosen for the proj ect should be specified according
to the IPMVP consistent with the M&V Protocol.

Specify Data Analysis Pr ocedures and Algorithms : Engineering equations and stipulated values as applicable shall be identified and re ferenced within the M&V plan. Documentation supporting baseline assumptions shall be provided.

This is a key component of the M&V plan. Often, data are collected without a clear understanding of the later use for the data. This can result in e ither extraneous data collection and/or missing data during the data analysis step. Fully specifyi ng the data analysis procedures will help ensure that an efficient and comprehensive M&V plan is presented.

Specify Field Monitoring Data Points: The actual field measurements planned should be specified, including the sensor type, locati on and engineering units. For example:

• For measuring the run-time of a boiler, th e field data point de scription would be: “Accumulated run-time of draft fan servi ng boiler number 1, usi ng an inductive run-time logger mounted on the draft fan motor.”

• For measuring air conditioner s upply air temperature, the fiel d data point description would
be: Duct air temperature (in degrees Fahrenheit) using a sheathed thermistor sensor located in the supply duct three f eet downstream from AC-1.

• For measuring chilled water temperature, th e field data point de scription would be: “Chilled water supply temperature measured with a probe- type thermistor inserted in a thermowell.”

Estimate Data Product Accuracy: All measurement systems have error, expressed in terms of the accuracy of the sensor and the recording devi ce. The combined errors should be estimated using a propagation of error analysis and the e xpected final data product accuracy described.

Specify Verification and Quality Assurance Procedures: Data analysis procedures to identify invalid data and treatment of missing da ta and/or outliers must be provided.Evaluators’ Protocols Measurement and Verification

CPUC 56 TecMarket Works Team Specify Recording and Data Exchange Formats : Data formats compliant with the data reporting Protocol should be described.



### M&V Rigor Levels

Rigor is defined as the level of expected reliabi lity. The higher the le vel of rigor, the more
confident we are the results of th e evaluation are both accurate and pr ecise, i.e., reliable. That is, reliability and rigor are treated as synonymous. Reliability is discussed in the Sampling and Uncertainty Protocol and in the Evaluation Framework 50 where it is noted that sampling precision does not equate to accuracy. Both are important components in reliability, as used by
the CPUC.

In accordance with the Impact Evaluation Protoc ol, M&V requirements are set according to two levels of rigor. The Joint Staff will set rigor levels for each program according to their overall planning priorities as described in the Impact Evaluation Protocol . Each rigor level provides a set of allowable methods that offers flexibility for the M&V contractor to propose the most costeffective method considering the conditions prevaili ng at each sampled site. The principle is to establish a minimum level of evaluation rigor. The M&V contractor is free to propose options providing greater rigor than the minimu m specified in this Protocol.



### Measure Installation Verification

The objectives of measure installa tion verification are to confirm th at the measures were actually installed, the installation meets reasonable qualit y standards, and the measures are operating correctly and have the potential to generate the predicted savings. Instal lation verification shall be conducted at all sites claiming energy or pe ak demand impacts where M&V is conducted. Installation verification activities may also be specified by the Process or Market Effects
Protocols.



### Measure Existence

Measure existence shall be verified through on-site inspections of facilities. Measure, make and
model number data shall be collect ed and compared to participant program records as applicable. Sampling may be employed at large facilities w ith numerous measures installed. As-built construction documents may be used to verify m easures such as wall insulation where access is difficult or impossible. Spot m easurements may be used to suppl ement visual inspections, such
as solar transmission measurements and low-e coa ting detection instruments to verify the optical properties of windows and glazing systems.



### Installation Quality

Measure installation inspections shall note the qual ity of measure installa tion, including the level of workmanship employed by installing contractor toward the measure installation and repairs to existing infrastructure affected by measure installation, and physical appearance and
attractiveness of the measure in its installed condition. Installa tion quality guidelines developed by program implementer shall be used to assess installation quality. If such guidelines are not available, they shall be devel oped by the M&V contractor and a pproved by the Joint Staff prior

50 TecMarket Works, pages 287-314.Evaluators’ Protocols Measurement and Verification

CPUC 57 TecMarket Works Team to conducting any verification activities. Inst allation quality shall be determined from the perspective of the customer.



### Correct Operation and Potent ial to Genera te Savings

Correct measure application and measure operation shall be observed and compared to project design intent. For example, CFL applications in seldom used areas or occupancy sensors in spaces with frequent occupancy shall be noted during measure verification activities. At enhanced rigor sites, commissioni ng reports (as applicable) sha ll be obtained and reviewed to verify proper operation of installed systems. If measures have not been commissioned, measure design intent shall be established from program records and/or construction documents; and functional performance testing sh all be conducted to verify operation of systems in accordance with design intent.



### M&V Protocol for Basic Level of Rigor

The M&V Protocols for the Basic le vel of rigor are summarized in Table 5. Further explanations of the provisions of this Protocol follow the tabl e. The M&V contractor is free to propose more rigorous M&V activities during eval uation planning or as directed by the Joint Staff evaluation

managers.

Table 5. Summary of M&V Protocol for Basic Level of Rigor

Provision Requirement Verification Physical inspection of installation to verify correct measure installation and installation quality IPMVP Option Option A 51
Source of Stipulated Data DEER assumptions, program work papers, engineering references, manufacturers catalog data, on-site survey data Baseline Definition Consistent with program baseline definition. May include federal or Title 20 appliance standards effective at date of equipment manufacture, Title 24 building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the program
Monitoring Strategy and Duration Spot or short-term meas urements depending on measure type Weather Adjustments Weather dependent measur es: normalize to long-term average weather data as directed by the Im pact Evaluation Protocol Calibration Criteria Not applicable

Additional Provisions None



### IPMVP Option

The standard M&V Protocol shall conform to IPMVP Option A - Partially Measured Retrofit
Isolation. 52 Savings under Option A are determined by partial field measurement of the energy

51 Exceptions to this provision are programs offering comprehensive measure packages with significant measure

interactions; commissioning, and retrocommissioning program s; and new construction programs. Evaluation of measure savings within these programs conducted using engineering methods must follow the Enhanced rigor M&V Protocol and use building energy simulation modeling under IPMVP Option D.

52 See the Evaluation Framework , pages 165-166.Evaluators’ Protocols Measurement and Verification

CPUC 58 TecMarket Works Team use of the system(s) to which an energy conser vation measure (ECM) was applied separate from the energy use of the rest of the facility. Measur ements may be either short-term or continuous. Partial measurement means that some parameter( s) affecting the building’s energy use may be stipulated, if the total impact of possible stipulation error(s) is not significant to the resultant savings. Savings are estimated from engineerin g calculations based on stipulated values and spot, short-term and/or continuous post-retro fit measurements. Fi eld-verified measure installation counts applied to deemed savings es timates do not meet the requirements of this
Protocol.



### Sources of Stipulated Data

Stipulated data may be taken from DEER unit energy savings analysis assumptions, efficiency program work-papers, secondary research, engi neering references, manufacturers’ catalog data, and/or on-site survey data as applicable. Valu es and sources for stipulated values must be documented in the M&V plan.



### Baseline Definition

The baseline used for M&V activities shall be consistent with the baseline definition used by the program. This may include applicable state and/or Federal efficiency standards for appliance or building energy efficiency, existing equipment efficiency or common replacement or design practices as defined by the program evaluated.



### Monitoring Strategy and Duration

Spot or short-term measurements may be use d, provided the measurement strategy and duration is sufficient to allow calculation of energy a nd peak demand savings within the uncertainty bounds prescribed by the Impact Evaluation Prot ocol. Pre-installation monitoring may be required in some cases to meet the applicable uncertainty requirements. 53 The Evaluation
Framework provides more information on monitoring strategy and duration. 54



### Weather Adjustments

Impacts of weather-dependent measures shall be normalized to long-term average weather data as directed by the Impact Evaluation Protoc ol. Weather conditions prevailing during the monitoring period must be reported. Weather data may be obtained from the nearest representative NOAA or utility weather station or collected on-site. Techniques used to perform the weather adjustment s must be documented.



### M&V Protocol for Enhanced Level of Rigor

The M&V Protocols for the Enhanced level of rigor are summarized in Table 6. Further explanations of the provisions of this Protocol follow the table. The M&V contractor is free to propose more rigorous M&V activi ties during evaluation planning or as directed by the Joint Staff evaluation managers.

53 Specific requirements for pre-installation monitoring are not stated in this Protocol, but are a consequence of the

uncertainty analysis conducted during M&V planning.

54 TecMarket Works, 182-188.Evaluators’ Protocols Measurement and Verification

CPUC 59 TecMarket Works Team Table 6. Summary of M&V Protocol for Enhanced Level of Rigor

Provision Requirement

Verification Physical inspection of installation to verify correct measure installation and installation quality. Review of commissi oning reports or functional performance testing to verify correct operation

IPMVP Option Option B or Option D

Source of Stipulated Data DEER assumptions, program work papers, engineering references, manufacturers catalog data, on-site survey data

Baseline Definition Consistent with program baseline definition. May include federal or Title 20 appliance standards effective at dat e of equipment manufacture, Title 24 building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the

program

Monitoring Duration Sufficient to capture all operational modes and seasons

Weather Adjustments Weather dependent measures: normalize to long-term average weather data as directed by the Impact Evaluation Protocol

Calibration Criteria Option D building energy simulation mo dels calibrated to monthly billing or interval demand data. Optional calibration to end-use metered data

Additional Provisions Hourly building energy simulation pr ogram compliant with ASHRAE Standard
140-2001



### IPMVP Option

The Enhanced rigor M&V Protocol shall conf orm to IPMVP Option B - Retrofit Isolation 55 or IPMVP Option D - Calibrated Simulation. 56 Under Option B, savings are determined by field measurement of the energy use of the systems to which the ECM was applied separate from the energy use of the rest of the facility. Savi ngs are estimated directly from measurements. Stipulated values are not allowed. Under Opti on D, savings are determined through simulation of the energy use of components or the whol e facility. Simulation routines should be demonstrated to adequately model actual energy pe rformance measured in the facility. Savings are estimated from energy use simulation, calibrate d with hourly or monthly utility billing data, and/or end-use metering.



### Sources of Stipulated Data

Stipulations are not allowed unde r IPMVP Option B. Under IPMVP Option D, stipulated values used to define the energy simulation model are a llowed. Sources of stipulated data may include DEER unit energy savings analysis assumptions, efficiency program work papers, secondary research, engineering references, simulation progr am default values, manufacturers’ catalog data and/or on-site survey data as appr opriate. It is impractical to li st and reference all data used to define a simulation model. However, model inpu t assumptions that are highly influential in predicting energy and/or peak demand savings sh all be identified and documented within the

55 See the Evaluation Framework , pages 166-168.

56 See the Evaluation Framework , pages 176-182.Evaluators’ Protocols Measurement and Verification

CPUC 60 TecMarket Works Team M&V plan. Simulation program name, full ve rsion number including applicable release information, and input files sha ll be provided as documentation.



### Baseline Definition

The baseline used for the M&V activities shall be consistent with the baseline definition used by the program. This may include applicable state a nd/or federal efficiency standards for appliance or building energy efficiency, existing equipmen t efficiency or common replacement or design practices as defined by the program evaluated.



### Monitoring Strategy and Duration

Monitoring shall be sufficient to capture all ope rational modes and seasons applicable to measure performance. Pre-installation monitoring may be required in some cases to meet the applicable uncertainty requirements. 57 The Evaluation Framework provides more information on monitoring strategy and duration. 58



### Weather Adjustments

Impacts of weather-dependent measures estimat ed under Option B shall be normalized to longterm average weather data for CEC CTZ in wh ich the site is located. Weather conditions prevailing during the monitoring period must be reported. Weather data may be obtained from the nearest representative NOAA or utility weather station or collect ed on-site. Techniques used to perform the weather adjustments must be documented. Simulation analysis under Option D shall be conducted using long-term average weat her data for CEC CTZ in which the site is
located.



### Calibration Targets

Building energy simulation models developed u nder Option D shall be calibrated to monthly energy consumption data. If interval demand data ar e available, these data shall be used in lieu of monthly energy consumption data. If the modeled floor space area does not match the metered floor space area within ± 20 percent, model calibration is not required. Modelers shall make reasonable attempts to meet the calibration targets listed in Table 7 below. In some cases, forcing a model to meet a particular calibration target may introduce biases in the energy savings estimates. Models not meeting the calibration target s shall be identified a nd reasons why it is not reasonable to meet these targets must be doc umented. The Joint Staff may impose additional requirements for short-term end-use monitoring of systems affected by the energy conservation measure during evaluation plan development and review.

Table 7. Model Calibration Targets

Data Interval Maximum Root Mean Square (RMS) Error Maximum Mean Bias Error Monthly ± 15% ± 5% Hourly ± 30% ± 10%

57 Specific requirements for pre-installation monitoring are not stated in this Protocol, but are a consequence of the

uncertainty analysis conducted during M&V planning.

58 TecMarket Works, 182-188.Evaluators’ Protocols Measurement and Verification

CPUC 61 TecMarket Works Team 

### Additional Provisions

Building energy simulation programs used under Option D shall be compliant with ASHRAE
Standard 140-2001. 59 For example, a partial list of progr ams compliant with the Standard is
shown in Table 8 below:

Table 8. Programs Compliant with ASHRAE Standard 140-2001 (Partial List)

Program Sector(s) Micropas Residential
DOE-2 Residential and Commercial EnergyPlus Residential and Commercial

Software using any ASHRAE Standard 140-complain t program as a calculation engine shall be in compliance with this provision of the Protocol.



### M&V Approach Examples

This section provides examples of M&V approaches as they apply to specific measure types and rigor levels. The examples are provided for general guidance; M&V contractors are free to proposed M&V plans that are compliant with the Protocols and make sense for the specific site conditions. Example IPMVP options by meas ure type and rigor level are shown in Table 9
below:

Table 9. Example IPMVP Options by Measure Type and Rigor Level

Measure Type Basic Rigor Level Enhanced Rigor Level
Appliances A B

Commissioning and O&M programs D D
Comprehensive D D
Envelope D D

Food Service A B

HVAC Controls D D HVAC Equipment Efficiency A D

Lighting Controls A B

Lighting Efficiency A B

New Construction D D

Non-HVAC Motor Controls A B

Non-HVAC Motor Efficiency A B

Process A B

Refrigeration A D

Water Heating A B

Water Pumping/Treatment A B

59 Programs used for non-HVAC simulation, such as industrial processes or refrigeration, do not need to comply with

this provision of the Protocol.Evaluators’ Protocols Measurement and Verification

CPUC 62 TecMarket Works Team 

### Overall Results Reporting

For each M&V project conducted, th e M&V contractor must submit a site-specific M&V report. This report is an addendum to the M&V plan submitted prior to conducting field activities and covers the site-specific M&V resu lts and final uncertainty analys is. In addition to the sitespecific M&V reports, an overall M&V report shall be filed for each program where M&V
activities were conducted within the scope of an individual M&V project contract. The overall M&V report shall include a discus sion on the potential sources of bias in the results and steps taken to control and minimize bias, as discusse d in the Sampling and Uncertainty Protocol.

Results shall be reported according to the Repor ting Protocol and shall conform to the DEER
database format as shown inEvaluators’ Protocols Measurement and Verification

CPUC 63 TecMarket Works Team APPENDIX A. Measure-Level M&V Results Reporting Requirements

. Energy and peak demand savings resulting from weather dependent measures shall be reported under weather conditions prevailing during the course of the M& V project. These weat her conditions shall be reported along with the energy and peak demand impact information. The impacts shall be
normalized to standard weather conditions as directed by the Impact Evaluation Protocol.



### Sampling Strategies

M&V projects will be conducted on a sample of program participants and non-participants, as directed by the Impact Evaluation Protocol. Samples drawn for M&V projects shall be congruent with the impact evaluation sample or be nested within the impact evaluation sample where possible. Justification for drawing samp les for M&V projects independently from the impact evaluation sample must be provided.

Early scheduling of M&V studies to provide feedback to the program implementer shall be considered in the sample design process. Partic ipant samples for M&V activities may need to be drawn in stages, before the full participant popula tion is established. If problems are identified in early M&V activities and corrected by the impl ementer, follow-up surveys on a sub-sample of sites may be required to verify that the pr ogram delivery modifications are effective.

Samples of measures selected for monitoring at a particular site shall be representative of all
measures at the site and shall be selected at random. Measures with in a building are often grouped according to similar usage patterns, thus reducing the expected variability in the measured quantity within each usage group. W ithin each usage group, the sample unit may be the individual measure, a particular circuit or point of control as designated by the M&V plan. Sample units shall be selected at random. System atic sampling with random start is acceptable. The sampling strategy shall address all measures pr esent at the site that are subject to the M&V study. Target uncertainties for sample designs are specified in the Sampling and Uncertainty
Protocol.



### Skills Required for M&V

Simple engineering equations and simple instrumentation such as run-time data loggers can be understood and used by people with a gene ral science or engineering background. 60 Specific training in the use of building energy simula tion programs and instrumentation systems is advised but not required.



### Summary of Protocol-Driven M&V Activities

1 Receive input from impact evaluation plan. Re ceive M&V site selection and expected rigor level from the impact evaluation plan.

60 The Association of Energy Engineers (AEE) offers a ce rtificate for a Certified E nergy Manager (CEM) and a

Certified Measurement and Verification professional (CMVP). The material covered in the CEM program is good background for understanding energy engineering concep ts addressed by measurement and verification. The CMVP program provides additional training and certification specific to M&V projects.Evaluators’ Protocols Measurement and Verification

CPUC 64 TecMarket Works Team 2 Develop overall M&V plan. The M&V option for each site shall be established according to the rigor assignment and allowable options under the Impact Evaluation Protocol. Project baseline definition with justification shall be reported. Overall M&V planning shall consider the needs of process evaluation studies for measure insta llation verification and measure performance information. The overall M&V plan shall be su bmitted for approval to the evaluation project manager as designated by the CPUC-ED. 3 Assess data sources. For each sampled site, t he data resources for the engineering analysis must be identified and reviewed. Data sources may include program descriptions, program databases, DEER estimates and underlying documentation, program work papers and on-site surveys. Uncertainties associated with engineering parameter s must be estimated. Baseline uncertainties, where not explicitly documented elsewhere, ma y be informed by professional judgment. 4 Conduct uncertainty analysis. The uncertainty in the estimated savings must be estimated using a propagation of error analysis. The parameters having the greatest influence on the uncertainty must be identified from the propagation of error analysis. 5 Develop site-specific M&V plan according to the outline in the M&V Protocols. The M&V plan must
address data collection conducted to reduce uncert ainty in the engineering estimates of savings. Sampling of measures within a particular site shall be done in accordance with the Sampling and Uncertainty Protocol. The site-specific M&V plan shall be submitted for review and approval to the evaluation project manager designated by the CPUC -ED prior to commencing field data collection. 6 Conduct pre- and/or post-installation monitoring as indicated by M&V plan. Data collection must be conducted in accordance with the site-specific M &V plan. Changes to the M&V plan resulting from unanticipated field conditions shall be document ed and submitted to the evaluation project manager designated by the CPUC-ED. 7 Conduct data analysis and estimate site-specific savings. Conduct analysis of field data and estimate site savings in accordance with site -specific M&V plan. Energy savings estimates for weather-dependent measures shall be normalized to long-term average weather conditions as directed by the Impact Evaluation Protocol. 8 Prepare site-specific M&V report. Prepare a site -specific M&V report for each site used in the analysis that includes the site-specific M&V plan, data collection, data analysis, calculation of measured engineering parameters and overall savi ngs estimates. Calculate the uncertainties associated with energy savings estimates and meas urement-derived engineer ing parameters. The site-specific uncertainty analysis shall include an estimate of the sampling error associated with individual measure sampling within the site, measur ement error associated with field data collection and uncertainties associated with any non-measur ed (deemed) parameters. Potential sources of bias associated with the measurements and engi neering analysis shall be identified and steps to minimize the bias shall be reported in accor dance with the Sampling and Uncertainty Protocol. 9 Prepare draft overall M&V report. A draft over all M&V project report shall be submitted to the CPUC-ED that meets all the requirements of t he Reporting Protocol, demonstrates compliance with the overall M&V plan developed in step 2 and summarizes the results from each site. Site-specific M&V reports shall be included as an Appendix. Raw field data and data analysis results shall be supplied electronically in accordance with the Reporting Protocol. 10 Prepare final overall M&V report. Prepare final overall M&V report in accordance with review comments provided by the Joint Staff. 11 Submit final M&V report. Submit final M&V r eport and associated datas ets to the CPUC-ED. 12 Post final M&V report on the CALMAC Web site . Once accepted by the CPUC-ED, develop abstracts and post them and final M&V report on the CALMAC Web site following the CALMAC posting instructions.Evaluators’ Protocols Emerging Technology

CPUC 65 TecMarket Works Team 

## Emerging Technologies Protocol

Introduction

The Statewide Emerging Technologies Program (ETP ) is an information-only program that seeks
to accelerate the introduction of innovative energy efficient technologies, applications and analytical tools that are not widely adopted in California. The overall objective of the ET Program is to verify the performance of ne w energy efficiency i nnovations which can be transferred directly into the marketplace and/or in tegrated into utility portfolios in support of resource acquisition goals for en ergy efficiency. Emerging tec hnologies may include hardware, software, design tools, strategies and services. Finally, it is recognized th at such programs are expected to have a number of failures 61 (technologies that do not perf orm as expected) given the
inherent risks 62 associated with the technologies selected for in vestigation.

Because of the absence of energy and demand goals and the longer lead time required to introduce new technologies directly into the market and/or into utility energy efficiency programs, a separate Protocol has been prepar ed to guide the ETP ev aluation. The evaluation approach in this Protocol is th eory-driven and is based on monitoring the full range of activities, outputs, and immediate, intermediate and long- range outcomes. This approach explicitly recognizes that while many, if not all, of these outputs and outcomes are difficult, if not impossible, to monetize, they can be documented and monitored over time to assess whether the program is on track to achieve the ultimate impacts 63

.

Because the ETP and other similar programs will e volve over time, the ETP Protocol is designed to be flexible so that the evaluation, measurem ent, and verification (E M&V) requirements will apply not only to the 2006-2008 ETP but to future ETP designs as well. Of course, the ETP Protocols will also evolve as evaluators ga in experience in evaluating such programs.

This Protocol insures a minimum level of evaluation rigor in order to ensure stakeholders that the performance of the emerging technology program s is on-track to achieve their longer-term

61 There are two types of failure: 1) failure of the technol ogy to perform as expected (note: such failures can provide

valuable information to members of the various target audien ces), and 2) the failure of the utility to select promising technologies such that a reasonable number of new tech nologies are not being funneled into utility energy efficiency programs. This Protocol will address both types of failure.

62 Risk involves the exposure to a chance of injury or lo ss (Random House, 1966). Hardware, software, design tools,

strategies and services (products) have varying levels of uncertainty as to whether they will perform as expected. Thus, investing in these products assumes varying levels of risk that the return on t hese investments might not be fully realized (i.e., there will be a loss).

63 Unlike the methods identified in the Impact Protocol, the me thods for evaluating the benefit s of public investment in

RD&D and related emerging technology programs are not nearly as advanced. However, it has been recognized by many that stakeholders should not have to wait three to five to ten years before discovering whether projects with relatively long times are successf ul (Lee, Russell, Gretchen Jordan, Paul Leiby, Brandon Owens, James Wolf (2003); Link, Albert N. (1996); Ruegg, Rosalie and Irwin Fe ller (2003); Shipp, Stephani e, Aaron Kirtley, and Shawn McKay (2004); U.S. Department of Commerce, Advanced Te chnology Program, National Institute of Standards and Technology, Technology Administration (2001); U.S. De partment of Commerce, Advanced Technology Program, National Institute of Standards and Technology, Technolog y Administration. (2001)). There is agreement among many researchers that one should be able to identify imm ediate and intermediate indicators that can reassure
stakeholders that the efforts are on tra ck to achieve such objectives as successful deployment of new technologies into utility energy efficiency programs and the bridging of the “chasm”, leading eventually to significant energy and demand impacts.Evaluators’ Protocols Emerging Technology

CPUC 66 TecMarket Works Team objectives. This Protocol also provides a wide array of allowable met hods in order to offer flexibility for the potential evaluation contractor s to propose the most reliable and cost-effective
methods that meet the Joint Staff’s need s for a given set of evaluation objectives.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• Joint Staff evaluation planners – will use the Protocol (1) as input into the ETP evaluation RFPs, and (2) as background and criteria fo r use in reviewing ETP evaluation plans, managing the ETP evaluations, and reviewing ETP evaluation reports and results.

• Evaluation project team – will use the Protocol to ensure that their detailed ETP evaluationplan(s) meets the requirements in the Protocol. They will also use the Protocol to doublecheck that the Protocol requirements have been met as they conduct, complete and report the
ETP evaluations.

• Portfolio administrators – will use the Protocol to understa nd how the ETP evaluation will be
conducted and to understand the ev aluation data needs to sup port the ETP evaluation. In addition, the Protocol provides background for the administrator’s use to determine when to intervene in the program design and implement ation efforts to achieve continued and/or greater efficiency gains.

• Program implementers – will use the Protocol to understa nd the ETP evaluation that will be conducted on their programs and program compon ents. Often, they will be required to provide data to suppor t the evaluation.

• PIER Program administrators – will use the Protocol to understand the ETP evaluation
because the activities of the PIER are linked to the activities of the ETP. In some cases, they may be required to provide da ta to support the evaluation.



### Key Metrics, Inputs, and Outputs

ETP evaluations will rely on both secondary and primary data related to various indicators associated with program inputs (e.g., budgets and staff), outputs (e.g., techni cal reports, articles published, and software) and outcomes (e.g., change in awareness, reduction of performance uncertainty and an increase in adopti on rates in the targeted population). Secondary data can include, among others, data from program da tabases, program descriptions, Emerging Technologies Coordination Council (ETCC) databa ses, work papers developed during program planning, technical reports, white papers, conference papers, on-site measurement and monitoring, and other prior study data and reports. Primary data can include, among others, observational data (e.g., on-site vis its to demonstration sites), su rveys and in-depth interviews with members of the various target populati ons as well as those who host a demonstration project. Peer reviews can also be conducted using independent experts. Energy and demand impacts are not performance indicators for the ETP since it is an information-only Program

. These longer-term energy and demand impacts ar e more appropriately the focus of impact
evaluations which will be conduc ted for utility resource acquis ition and market transformation programs after the “new” ETP technologies are de ployed in these programs. A more complete listing of possible indicators is pr ovided later in this Protocol. Fi nally, which data to collect andEvaluators’ Protocols Emerging Technology

CPUC 67 TecMarket Works Team what to report are contingent on the size of the evaluation budget, the indicators identified in the program theory and logic model as being the most important, and the chosen methods.

These data will be used within the ETP Protocol’s selected methods, a more detailed sample of which is presented later in this Protocol, and conducted through a Joint Staff approved evaluation plan. Unlike resource acquisition programs which are focused on net energy and demand impacts, the performance of the ETP will be ba sed on the preponderance of evidence associated with the analysis of a relatively la rge number of diverse indicators.

The actual information included in a given repor t will vary depending on the methods chosen. The specific information to be repor ted from each study must be de termined by the Joint Staff in
close collaboration with th e independent evaluator.



### Evaluation Planning

Once an independent evaluator is hired, the eval uator must prepare a final detailed evaluation work plan that allocates the study’s finite resources to maximize the value and use of the information collected while taking into account the requirements of th e ETP Protocol. As part of this plan, the evaluator must sp ecifically address the various sour ces of potential error that are relevant and explain how the resources allocated to each will mitigate the error 64 . The evaluation should also focus on gathering information on specific project and program goals and expectations early in the program cycle from the administrators so that plans can be made to insure that the necessary data are collected.

When samples are used, the ETP evaluation must follow the Sampling and Uncertainty Protocols. Evaluators must assess, minimize, and mitigate potential bias and present, when relevant, the achieved level of precision (incl uding relative precision, error bounds, coefficient of variations, and standard deviations) for interpreti ng information. It is exp ected that the aggregate analysis, described later in this Protocol, of all ETP projects must first be conducted in order to inform the sampling plan (e.g., the aggregate an alysis should shed some light on useful stratification schemes).

The Joint Staff, and other outside stakeholders as deemed appropriate by the CPUC, will review the evaluation plan submitted and discuss with the independent evaluator any tradeoffs they deem necessary to maximize the reliability of th e ETP performance assessment. For example, if surveys are conducted of various target audiences, Joint Staff can decide to increase the sample sizes in order to increase pr ecision, recognizing that other sour ces of error will receive fewer
resources or that additional reso urces may be required to support the change. Or, Joint Staff can decide to reduce the sample sizes and settle for lo wer precision in exchange for a greater effort to reduce, for example, non-response bias. In the fi nal plan, evaluation resour ces will be allocated in a way that is consistent with cost-efficient evaluation, i.e., where evaluation resources are set
and allocated at levels that maximize the value received from these resources.

64 In the pre-1998 Protocols, there was no requirement to address these sources of error in the research plan.

Evaluators only had to describe in the final report whether th ey had to address these various errors and, if so, what they did to mitigate their effects. See Chapter 12 of the California Evaluation Framework for further details.Evaluators’ Protocols Emerging Technology

CPUC 68 TecMarket Works Team 

### A Sample of Available ETP Evaluation Methods

One of the goals of the ETP Protocol is to co mbine progress measures for different types of projects in such a way that provides a meani ngful assessment of the e ffectiveness of the ETP program in reaching portfolio level goals like accelerating the introduction of new technologies into utility energy efficiency programs and/or di rectly into the marketpl ace. A review of the
evaluation literature reveals a numbe r of approaches that could be applied to the ETP evaluation. The following table lists and briefly di scusses a number of these methods.

Table 10. Sample of Available ETP Evaluation Methods

Method Brief Description Example of Use

Analytical/conceptual modeling of underlying theory Investigating underlying concepts and developing models to advance understanding of some aspect of a program, project, or phenomenon. To describe conceptually the paths through which projects evolve or through which spillover effects may occur and validate the underlying theory.

Survey Asking multiple parties a uniform set of questions about activities, plans, relationships, accomplishments, value, or other topics, which can be statistically analyzed. To find out how many members of a given target audience have been informed about a given technology through the
dissemination efforts of the ETP.

Case study - descriptive Using single-case or multiple-case designs with single or multiple units of analysis for investigating in-depth a program or project, a technology, or a facility, describing and explaining how and why developments of interest have
occurred. To recount how a particular joint venture (e.g., between the ETP and a customer who hosts a technology demonstration; between the ETP and a manufacturer) was formed, how parties shared research tasks, and why the collaboration was successful
or unsuccessful.

Sociometric and
social network analysis Identifying and study ing the structure of relationships by direct observation, survey, and statistical analysis of secondary databases to increase understanding of social/organizational behavior and
related economic outcomes. To learn how projects can be structured to increase the diffusion of resulting knowledge.

Bibliometrics -

counts Tracking the quantity of research outputs. To find how many publications per applied research dollar a technology assessment generated.

Bibliometrics -
citations Assessing the frequency with which others cite publications or patents and noting who is doing the citing. To learn the extent and pattern of dissemination of a technology assessment’s publications and patents.

Bibliometrics - content analysis Extracting content information fromtext using techniques such as coword analysis, database tomography, and textual data mining, supplemented by visualization techniques. To identify a project’s contribution, and the timing of that contribution, to the evolution of a technology.Evaluators’ Protocols Emerging Technology

CPUC 69 TecMarket Works Team

Historical tracing Tracing forward from research to a
future outcome or backward from an outcome to precursor contributing developments. To identify apparent linkages between a ratepayer-funded applied research project and something of significance that happens later or has already occurred.

Expert judgment/Peer
Review Using informed judgments to make

assessments. Experts can be called upon to give their opinions about the technical quality and effectiveness of a technology assessment. The experts generally render their verdict after reviewing written or orally presented
evidence.

Source: Adapted from Ruegg and Feller (2003)



### Protocols Requirements

There is only one level of rigor for the ETP Protocols which has eight required components.

Verification of Basic Achievements In their 2006-2008 program implementation plans, each utility has established three basic goals
that are framed in terms of:

• achieving a certain number of emergi ng technology application assessments 65

,

• updating the Emerging Technology Database, and

• conducting a certain number of meetings annually of the Em erging Technologies Coordinating Council.

A straightforward verification of whether each ut ility has met these goals must be conducted.
The 2006-2008 ETP verification should include:

• obtaining all relevant documentation of t echnology assessments launched during the program period 66

,

• comparing the contents of the Emerging Tec hnology Database before the program period and at the conclusion of the program period, and

• documenting the meetings of the Emer ging Technologies Coordinating Council.

Beyond 2008, it is assumed that the utilities will c ontinue have a set of basic goals that are amenable to such simple verification. Howe ver, independent evaluators must go beyond the simple verification of whether utilities have ach ieved these basic goals. The remainder of this
Protocol describes those activit ies that must be conducted as a part of a rigorous and comprehensive evaluation of the ETP.

65 The technology application assessments may consist of di verse project types including: feasibility studies,

simulation analyses, field demonstrations, controlled env ironment tests, commercial product development, design methodologies and tool development. Some assessm ents may take up to four years to complete.

66 Evaluation consultant contracts will include conf identiality and non-disclosure agreements to cover applicable

documents.Evaluators’ Protocols Emerging Technology

CPUC 70 TecMarket Works Team Program Theory and Logic Model Prior to the identification and quantification of performance indicators, the ETP program theory and logic model must be developed. The California Evaluation Framework of June 2004 defines program theory and makes an important di stinction between a program theory and a logic
model:

A program theory is a presentation of the goals of a program, incorporated with a detailed presentation of the activities that the program will use to accomplish those goals and the identification of the causal relationships between the activities and the program’s effects. The program th eory describes, in detail, the expected causal relationships between program goals and program activities in a way that allows the reader to understand why the proposed program activities are expected to result in the accomplishment of the pr ogram goals. A well-developed program theory can (and should) also describe th e barriers that will be overcome in order to accomplish the goals and clearly desc ribe how the program activities are expected to overcome those barriers. A program theory may also indicate (from the developers pers pective) what program progre ss and goal attainment metrics should be tracked in order to assess program effects.

Program theories (PT) are sometimes cal led the program logic model (LM). A
stricter definition would be to differentiate the program theory as the textual description while the logic model is the graphical representation of the program theory showing the flow between activiti es, their outputs, and subsequent shortterm, intermediate, and long-term outcomes. Often the logic model is displayed
with these elements in boxes and the causa l flow being shown by arrows from one to the others in the program logic. It can also be displayed as a table with the linear relationship presented by the rows in the table. The interactions between activities, outputs, and outcomes are cri tical to understandi ng the program logic and argue for the need to have, or constr uct, both a program theory and a program logic model. (p. 31)

A more thorough discussion of program theory a nd logic models can be found in Chapter 4 of
the California Evaluation Framework

.

Describing the various ETP activities and how th ese activities interrelate to produce immediate, intermediate, and long-term outputs and outcomes is a necessary first step. These outputs and outcomes can be considered additional objectives beyond the three basic objectives describe above. Once described, the underlying theory must be explicated, i.e., w hy are these activities expected to achieve these outputs and outcomes. As part of this process, immediate, intermediate, and long-term indi cators of progress toward the u ltimate goals will be identified. Some of these indicators are easily quantifiabl e (number of papers and patents, amount of additional investment) and others are somewhat more difficult to quantify (changes in behavior, changes in procedures). While the indicators pursued by th e independent evaluator should be guided by the logic model, there might be other indica tors that the CPUC wishes to pursue that are related to objectives other than thos e explicitly noted in the logic model.Evaluators’ Protocols Emerging Technology

CPUC 71 TecMarket Works Team As a part of the development of the program l ogic model, the various target audiences for the ETP activities must be identified. Once the program theory and logic model have been developed, future evaluation efforts must review the logic model and theory to determine if changes are needed. Finally, it is recognized that , while there will be a statewide ETP theory and logic model, it is possible that utility-specific pr ogram theories and logic models will be required if each utility’s ETP deviates in important ways from the statewide theory and logic model.

Aggregate Level of Analysis The aggregate analysis is desi gned to achieve two objectives:

• To describe, for each utility, the basic components or elements that make up the ETP and provide the necessary broader context for assessing the performance of the ETP (e.g., budgets, FTEs, types of technology assessm ents, average duration of projects,
collaboration with other inst itutions/agencies, etc), and

• To determine, for each utility, the extent to which the overarching program and policy objectives have been met (e.g., addressing the needs of all customer sectors, assuming acceptable levels of risk, etc.).

The aggregate analysis involves the analys is of a variety of data collected for all of the projects in each utility’s ETP portfolio. Such a level of analysis provides a statistical overview of the ETP portfolio (e.g., frequencies, cross tabulations, means etc. ) across multiple projects and participants in order to achieve the two objectives listed above. The analysis of these aggregate
data will allow one to address a number of c ontextual, program and policy questions, such as:

1. What are the various sources of funding, (PGC, academic institutions, manufacturers, government agencies, etc.), by type of technology assessment? 2. How many full-time equivalent ETP employ ees are involved by type of technology

assessment? 3. How does PGC funding and co-funding vary by type of technology assessment by sector
over time? 4. How does PGC funding and co-funding vary by end use and/or by sector over time? 5. What is the frequency of the various type s of technology assessments, by end use, over
time? 6. How is risk being balanced (e.g., measures th at do not perform as expected versus those that do)? 7. What is the average durati on of a technology assessment? 8. Are the technology assessments proportionately focused on sectors and end uses in which there are the greatest expected po tential energy and demand benefits? 9. How many technology assessmen ts are launched annually? 10. How many technology assessments are currently active? 11. What percent of the technologies sponsored by the ETP have been de ployed into utility energy efficiency program and/or directly into the marketplace? 12. Are there imbalances in the types of projects funded?
13. Are the needs of all the sector s being adequately addressed?Evaluators’ Protocols Emerging Technology

CPUC 72 TecMarket Works Team Data for ETP assessment can be collected using a survey of key ETP staff along with extracts from the program database or ETCC database. Examples of data that could be collected for the aggregate analysis include:

1. Funding by the PGC and by other entities (a uthorized budget, invoiced and committed) 2. Stage of development for each technology 3. Specific technologies and end uses 4. Expected long-term energy and demand bene fits from each project (provided by ETP program staff and/or the ETCC database) a nd the possible timeline of those forecast. 5. Project initiation and completion (dat e on which all work has ceased) dates 6. Failures (technologies that do not perform as expected based on ETP analysis) as a percent of all projects 7. Subjective assessment of risk 8. Targeted sectors and populat ion(s) within that sector, 9. Whether the technology has been deployed in to a utility energy efficiency program and/or directly into the marketplace.

The eventual list of key variable s will be determined in close collaboration with the CPUC-ED, the independent evaluator and ETP staff.

Implementation Analysis The final task is to conduct a program- and u tility-specific analysis to determine whether there have been any deviations from the program im plementation plan, as described in the program theory and logic model. Any deviations from the plan and implementation problems must be explained. This analysis must focus on such issues as the selecti on process used by ETP managers to select “promising” projects, collaboration between PIER, the ETP, and utility program staff, and unanticipated problems and thei r resolution. This analysis must be initiated early in the program period so that any nece ssary corrective guidan ce can be provided to program administrators on an on-going basis. Independent evaluato rs should look for opportunities to collaborate with utilities, which are respons ible for condu cting process evaluations of the ETP.

Measure Tracking Those technologies that have been deployed to utility energy efficiency programs must be tracked over time to determine their adoption rates 67 and resulting energy and demand impacts. Adoption rates and energy and demand impacts are useful indicators of how well the ETP screened promising technologies an d developed strategies, in clos e collaboration with the utilitysponsored energy efficiency programs, to cross the “chasm”. The goal of this component of the
Protocol is not to attribute these savings directly to ETP as a resource, but to show a clear trail of which ETP technologies are being accelerated into utility energy efficiency programs. Only by planning for this type of tracking can an evaluatio n adequately answer the future questions posed by key stakeholders regarding the ul timate impacts of ETP activities.

67 Adoption rates (e.g. the number of measures adopted on an annual basis) for various measures installed through

utility resource acquisition programs and associated energy and demand impacts will be obtained from utility program tracking databases. This is generally considered as distinct from a market penet ration rate or a saturation

rate.Evaluators’ Protocols Emerging Technology

CPUC 73 TecMarket Works Team While the previous five components are focu sed on the entire ETP, including all of the technology assessments, the next three com ponents focus on samples of projects.

Detailed Analysis of Key Performance Indicators This component involves the collection of additional data that address a number of areas, such as: 1) knowledge creation, 2) knowledge dissemi nation 3) technical progress, 4) progress towards commercialization, and 5) the deployment of new measures to utility-sponsored energy efficiency programs. Specifyi ng the indicator variables for th e ETP should be guided by the ETP logic model, which identifies short, intermed iate, and long-term outcomes associated with diverse projects. Some examples of project-level indicators fo r which data could be collected

are:

• knowledge created

o technical papers

o articles published

o technical reports

o conference presentations

o fact sheets

o brochures

• knowledge disseminated

o technical reports distributed and to whom

o number and content of workshops and professional forums

o conference presentations, topics and dates and estimated size of audience

o number of fact sheets distributed and to whom

o brochures distributed and to whom

o websites created (includes hits on the websites and downloads)

o bibliometric counts

• number of demonstration projects

• performance data collected at demonstration sites

• technical and market barriers overcome, technical milestones met, and significant knowledge gained

• remaining technical and market barriers

• prototypes developed and protot ypes passing performance tests

• patents (both filed and granted)

• licenses

• awards for excellence

• interviews with those hosti ng the demonstrat ion projects

• collaboration with manufacturers

• the number and description of new measur es being deployed directly into the marketplace and/or into utility programs.

Depending on the nature of the project, one could also examine the extent to which the project has attracted capital for advancing commerciali zation objectives, includi ng resources provided by any funding partners.Evaluators’ Protocols Emerging Technology

CPUC 74 TecMarket Works Team Finally, for each selected project, the reasons why it was selected must be discussed in terms of the selection criteria. Such topics as th e technology’s technical and economic energy and demand potential, description of th e targeted populations, the iden tified risk factors, market barriers, the existence of known delivery channels, and the eviden ce that there was a need to need for a bridging function could be discussed.

All data must be systematically analyzed so that an overall assessment of each utility’s ETP with respect to its specific objectives can be c onducted by the independent evaluator. These objectives must be determined early in the program cycle, as part of the development of the logic model, so that a plan to gather the necessary data can be designed.

If there are fewer than 30 projects 68 within a given utility during the program period, a census of all projects must be conducted. If there are more than 30 proj ects, then a random sample of projects must be evaluated. The size of the sample must be determined by the independent
evaluator in close collaboration with the Joint St aff. The sample design must be informed by the aggregate analysis. In addition, th e sample of projects for each util ity should be stratified by size of budget, the level of uncertainty regarding succe ss, or the magnitude of expected benefits. The
stratification variable will be sel ected after the aggregate analysis.

This next two components have two objectives: 1) to conduct a more rigorous assessment of the technical achievements of selected ETP proj ects through the use of a peer review panel 69 , and 2) to provide a more definitive assessment of th e extent to which the “chasm”, defined as a discontinuity in the product life cycle that occu rs from early adopter to the mass market (Moore, 2002) 70 , has been bridged. Projects selected fo r these next two components should be nested within the sample of those selected for Deta iled Analysis of Key Performance Indicators.

Peer Review A random sample of the ETP projects for each utility must be subject to a technical review using the peer review process. For example, such pr ojects as the laboratory testing of refrigeration measures could be subjected to a technical review in order to eval uate the quality of the research process and output (e.g., whether the design of the study was sound, whether the project provided any new insights on the assessed technology). The focus s hould be on those projects in the highest strata (i.e., those w ith the largest budgets, the greate st uncertainty regarding success, or the greatest expected benefits id entified in the previous component, Detailed Analysis of Key Performance Indicators. The number of projects that are pe er reviewed for each utility and the
extent of each review must be determined base d on the size and complexity of projects and the size of the evaluation budget.

68 A project can cover a variety of activities associated wi th a technology application assessment including feasibility

studies, simulation analyses, field demonstrations, contro lled environment tests, commercial product development, design methodologies and tool development. Some asse ssments may take up to four years to complete.

69 See the Peer Review Guide prepared by the U.S. Depar tment of Energy’s Office of Energy Efficiency and Renewable Energy (EERE) Peer Review Task Force, 2004.

70 The chasm separates the early adopters from the early majority. Crossing the chasm re quires that those in the

early majority receive something that the early adopters do not need, the needed assura nces from trusted sources regarding new technologies. Many new products fail because they are not able to cross the chasm in terms of new product design and marketing strategy, from the early market (early adopter) to the mass market (early majority).Evaluators’ Protocols Emerging Technology

CPUC 75 TecMarket Works Team Peer reviewers will be selected by the evalua tion contractor in clos e collaboration with the CPUC-ED with input from the utilities. Each po tential reviewer will be asked to identify any areas related to this project where a conflict or appearance of conflict co uld exist and explain the nature of that conflict. A key resource rega rding the use of peer reviewers is the “PEER REVIEW GUIDE: Based on a Survey of Best Pract ices for In-Progress Peer Review.” This document was prepared in 2004 by the Office of Energy Efficiency and Renewable Energy (EERE) Peer Review Task Force for U.S. Department of Energy, Energy Efficiency and Renewable Energy.

Target Audience Surveys To assess the extent to which the chasm is being crossed, surveys 71 of members of the various target audiences (end users and those upstream from the end users including those who request materials, download materials, are directly se nt materials, visited de monstration sites, and attended conferences and workshops) must be co nducted in order to determine the impact of knowledge dissemination on the targ eted populations with respect to any reductions in key market barriers and any subsequent increases in the adoption of ETP t echnologies. Of course, this requires that in the development of the progr am logic model the various target audiences for
the ETP activities must be identified and that ba selines are established so that progress can be
measured.



### Integration of Results

The results for each utility must be aggregated acr oss the projects examined so that, based on the preponderance of the evidence, conclusions regardi ng a utility’s performan ce with respect to its entire ETP portfolio can be assess ed. These results must then be aggregated across utilities so that the performance of the statewide ETP, based on the preponderance of the evidence, can also be assessed. Various approaches to aggregat ing performance indicator s are available including Keeney and Raiffa (1993), Reugg and Felle r (2003), and Shipp et al. (2004).



### Reporting of Results

The Emerging Technology Program Evaluation will be reported consistent with the requirements for all evaluation reports described in the Repor ting Protocol in the s ection entitled “Common Evaluation Reporting Requirements.” In addition, the following elements should be included in the evaluation reports under the Methods heading.

• Program Theory and Logic Model

• Goal Verification

• Aggregate-Level Analysis

• Implementation Analysis

• Measure Tracking

• Detailed Analysis of Key Performance Indicators

• Peer Review

• Target Audience Surveys

71 Whenever surveys are based on samples, the Sampling and Uncertainty Protocols apply.Evaluators’ Protocols Emerging Technology

CPUC 76 TecMarket Works Team These presentations must be provided in enough detail that the differe nces (if any) in the methodological approach across di fferent technologies and uti lities can be understood by the reader. Finally, one must describe the approach for integrating the st udy results so that the overall performance of the ETP can be assessed.

The Reporting Protocols includes a requirement that all evaluation reports include a presentation of the detailed study findings. This presentation must be provid ed in enough detail that the different results or findings (i f any) can be understood for each technology assessment covered in the study. The report should present the results of each of the required eight components
contained in the ETP Protocol. Re ports will be provided consistent with the Reporting Protocol.



### Summary

The following table provides a summary of the Prot ocol that can be used to guide the evaluation
efforts once the detailed contents of the Protocol are well understood.



### Summary of Protocol-Driven Emergi ng Technology Evaluation Activities

1 Joint staff selects an evaluation contra ctor to implement the Emerging Technology Program evaluation.

2 The ETP managers, in collaboration with t he evaluation contractor and the CPUC-ED, develop logic models and program theories to inform the evaluation plan.

3 The contractor works with the CPUC-ED on the development of the draft evaluation plan (with possible input from the program implement er) consistent with the ETP Protocol. As necessary, the plan must comply with the ot her Protocols (Impact Evaluation Protocol, Process Evaluation Protocol, Market Effects Protocols, the Sampling and Uncertainty Protocol and the Reporting Protocol) in the development of the evaluation plan and in the implementation and reporting efforts.

4 The CPUC-ED works with the evaluation contractor to finalize and approve an evaluation plan from which the contra ctor can begin the evaluation effort.

5 The contractor carries out all eight of t he required Protocol requirements in order to measures key short, intermediate, and long–ra nge performance indicators identified in the logic model. 6 The contractor reports the results of the final evaluation to the CPUC-ED and Joint Staff consistent with the provisions in the Reporting Protocol.

7 Once the report is accepted by the CP UC-ED, the contactor develops abstracts and posts the report on CALMAC web site follo wing the CALMAC posting instructions.

Note: the steps included in this evaluation summary table must comply with all the requirements within the Emerging Technology Protocol in order to be in compliance. Any deviations from the Protocol must be agreed to by Joint Staff and fully documented within the evaluation plan and in the evaluation report.Evaluators’ Protocols Emerging Technology

CPUC 77 TecMarket Works Team 

### References

Ausubel, Jesse H. and H. Dale Langford, 1997, Technological Trajectories and the Human

Environment , New York: National Academy Press.

Boardman, Anthony E., David H. Greenberg, Aidan R. Vining, and David L. Weimer, 1996, Cost-Benefit Analysis : Concepts and Practice, Upper Saddle River, N.J.: Prentice Hall.

Breland, H. (1983). The direct assessment of wri ting skill: A measurement review (Tech. Rep. No. 83-6). Princeton, N.J.: Colle ge Entrance Examination Board.

California Public Utilities Commission, 2001, California Standard Practice Manual: Economic Analysis of Demand-Side Programs and Projects

.

Cohen, J.: “A Coefficient of Agreement for Nominal Scales.” Educational Psychological Measurement, 20, 37-46, 1960.

Edwards, Ward and J. Robert Newman, 1982, Multiattribute Evaluation , Thousand Oaks, CA: Sage Publications.

Edwards, Ward, Marcia Guttentag and Kurt Snapper, 1975, “A Deci sion Theoretic Approach to Evaluation Research,” In Elmer L Stru ening and Marcia Guttentag (Eds.), Handbook of Evaluation Research, Beverly Hills, CA: Sage Publications, Inc.

Fleiss, Joseph L. Statistical Me thods for Rates and Proportions 2 nd Edition. New York: John Wiley & Sons, 1981.

Law, J. Randolph and Paul J. Sherman. “Do Rate rs Agree? Assessing Inter-Rater Agreement in the Evaluation of Air Crew Resource Management Skills.” In Proceedings of the Eighth International Symposium on Aviation Psychology , Columbus, Ohio: Ohio State University,

2001.

Lee, Russell, Gretchen Jordan, Paul Leiby, Brandon Owens, James Wolf, 2003, "Estimating the Benefits of Government-Sponsored Energy R&D," Research Evaluation , Vol. 12 No. 3, pp. 189-
195.

Mahajan, Vijay, and Robert A. Peterson, 1985, Models for Innovation Diffusion , Newbury Park,
CA: SAGE Publications.

Mahajan, Vijay, Eitan Muller, and Rajendra K. Srivastava, 1990, “Determination of Adopter Categories by Using Inn ovation Diffusion Models,” Journal of Marketing Research , Vol. XXVII, pp. 37-50.

Mahajan, Vijay, Eitan Muller, and Yoram Wind, 2000, New-Product Diffusion Models , Boston:
Kluwer Academic Press.Evaluators’ Protocols Emerging Technology

CPUC 78 TecMarket Works Team McGraw, K. O. and S. P. Wong. (1966). Forming inferences about some intraclass correlation
coefficients. Psychological Methods , Vol. 1, No. 1, 30-46.

Random House, 1966. The Random House Dictionary of the English Language, ed. Jess Stein.
New York: Random House.

RER, 1999, “Suggestions for Incorporating Multi period Market Transformation Effects and Energy Center Informational Impacts in the Public Purpose Tests for Some 2000 Programs,” Prepared by Regional Economic Research fo r the Southern California Edison Company.

Ridge, Richard, 2000, “Evaluation of Southe rn California Edison’s Emerging Technology Program,” Submitted to the Southern California Edison.

Ridge, Richard, 2003, “Evaluation of the 2002 Ca lifornia Emerging Technologies Program,”
Submitted to the Southern California Edison.

Rogers, Everett M., 1995, Diffusion of innovations, New York: The Free Press. Rogers, Everett M. and F. Floyd Shoemaker, 1971, Communication of innovations: A crosscultural approach, New York: Free Press.

Ruegg, Rosalie and Irwin Feller, 2003, A Toolkit for Evaluating Public R&D Investment: Models, Methods, and Findings from ATP’s First Decade . Prepared for the Economic Assessment Office, Advanced Technology Progr am, National Institute of Standards and Technology, Gaithersburg, MD.

Saaty, T.L., 1980, The Analytic Hierarchy Process, New York: McGraw-Hill.

Sebold, Frederick D., Alan Fields, Lisa Skumatz, Shel Feldman, Miriam Goldberg, Ken Keating, and Jane Peters, 2001, A Framework for Planning and Asse ssing Publicly Funded Energy Efficiency (Study ID PG&E-SW040) , Prepared for Pacific Gas & Electric.

Shipp, Stephanie, Aaron Ki rtley, and Shawn McKay, 2004, Evaluating Individual Projects and the Portfolio of Completed Projects in the Advanced Technology Program . Unpublished manuscript.

Soft, Steven, 2002, Power System Economics: Desi gning Market for Electricity, Piscataway, N.J.: IEEE Press/Wiley-Interscience, John-Wiley & Sons, Inc.

Sultan, Fareena, John U. Farley, and Donald R. Lehmann, 1990, “A meta-analysis of applications of diffusion models,” Journal of Marketing Research , Vol. XXVII, pp. 70-

“The California Evaluation Framew ork.” (June 2004). Prepared for the California P ublic Utilities Commission and the Project Advisory Group by TecMarket Works, Megdal & Associates, Architectural Energy Corporation, RLW Analytic s, Resource Insight, B & B Resources, Ken Keating and Associates, Ed Vine and Associat es, American Council for an Energy Efficient Economy, Ralph Prahl and Asso ciates, and Innovologie.Evaluators’ Protocols Emerging Technology

CPUC 79 TecMarket Works Team U.S. Department of Commerce, Advanced Tech nology Program, National Institute of Standards and Technology, Technology Administration. “Per formance of 50 Completed ATP Projects: Status Report - Number 2 (NIST Special Public ation 950-2).” U.S. Gove rnment Printing Office, Washington, D.C., December 2001.

U.S. Department of Commerce, Advanced Tech nology Program, National Institute of Standards and Technology, Technology Administration. (20 01). “Measuring ATP Impact: 2004 Report of Economic Progress.” U.S. Government Printing Office, Washington, D.C.

U.S. Department of Energy, Energy Efficiency and Renewable Energy. (2004). “PEER REVIEW GUIDE: Based on a Survey of Best Practices for In-Progress Peer Review.” Prepared by the Office of Energy Efficiency and Renewable En ergy (EERE) Peer Review Task Force.Evaluators’ Protocols Codes and Standards

CPUC 80 TecMarket Works TeamEvaluators’ Protocols Codes and Standards

CPUC 81 TecMarket Works Team 

## Codes and Standards and Compliance Enhancement
Evaluation Protocol

Introduction

This Protocol covers approaches for evaluating codes and standa rds programs, and for evaluating code compliance enhancement programs. The primar y focus of this Protocol is to present the approach for documenting savings from the Calif ornia Codes and Standards Program and the evaluation of Code Compliance Programs yet to be developed and implemented. The Code Compliance Enhancement Protocol is being added at this time because the IOUs are considering the addition of compliance enhancement programs in to their energy efficiency program portfolio. The Compliance Enhancement Program Evaluation Pr otocol is new and has never before been applied within the evaluation community. As a re sult it is designed to be flexible, allowing a wide range of approaches to be conducted once they are approved by the Joint Staff.

This Protocol describes how gross and net ener gy savings will be estimated for programs that change or contribute to a change in building codes or appliance standards that are expected to result in energy savings and progr ams that are implemented to increase the level of compliance with code requirements. It does not cover process evalua tions or other types of evaluations that may address additional research goals. Other sec tions of the Protocols provide instructions on
these studies. This Protocol identifies a series of evaluation-related activities that produce estimates of gross and net energy saving from Codes and Standards Programs and net energy savings from Code Compliance Programs. In a ddition, this Protocol id entifies the audience and responsible actors associated with these evaluation efforts, the ke y metrics to be produced from the evaluations, the change theories and the logi c models that need to detail the assumed causal relationships for achieving the savings, and the eval uation approach that is to be used to estimate gross and net program impacts. Th ese issues are discussed below.

We note early in the Protocols that codes and stan dards evaluations that follow this Protocol are best contracted prior to and launched at th e same time that the CEC is assessing which technologies should be considered for the next ro und of codes or standards changes. This effort is launched approximately three years before a change begins produci ng energy savings. The evaluations of the Code Compliance Enhancem ent Programs should be launched at the same time the programs are first launched so that base line compliance assessment can be compared to post-implementation changes in compliance.

The evaluation contractor sele cted to conduct the evaluation of the Codes and Standards Programs will need to realize that the change theories and logic models developed by the program will be adjusted and expanded or contracted from time to time as new change-related causal relationships are identified and as program activities are modified to meet the program’s objectives. These conditions will require a multi-year evaluation effort that is timed to the code program’s change process rather than the program implementation cycles, so that the evaluation contractor can be charged with the responsibility to evaluate a specific set of assigned code or standard changes. As additional code change s are developed over time, additional evaluation
contracts will be awarded to c over the code changes not includ ed in the previous group ofEvaluators’ Protocols Codes and Standards

CPUC 82 TecMarket Works Team evaluated changes. This means that there will be periods of time in which multiple evaluation contracts may be in force to evaluate the program , but these studies will focus on a different set of changes.

The evaluation activities conducted under this Codes and Standa rds Protocol are established to be both prospective and retrospe ctive. They are designed to a ssess events and conditions that occur in the future, such as the projected energy savings to be achieved. However, they are also designed to be retrospective, wi th true-up efforts that look back over time and adjust evaluation findings to reflect actual market conditions. As such the evaluati ons may be contracted in two phases, with the first phase be ing the assessment and projection of current and future savings, followed by true-up studies that look back and ad just the projected findings and energy savings to reflect actual construction, re trofit, and purchase patterns.

The evaluations conducted under the Codes and Sta ndards Protocol will need to be staffed and managed to be adaptive to the different stages associated with the different activities of the change process that will occur at different times. The evaluation contractor must be aware that they will need to coordinate with the program administrators to be able to respond to the different efforts and activities with the righ t evaluation activities at the right time.

Finally, Both the Codes and Standards Protoc ol and the Compliance Enhancement Protocol
included at the end of the Protocol is new to the evaluation industry. As they are used and tested over the next few program cycles it will need to be updated to reflect the experiences of the first
sets of evaluations conducted unde r these Protocols. Likewise, al l Protocols need to be updated periodically as new methods and approaches ar e developed and as the evaluation reporting needs change.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• Joint Staff Evaluation Planners – will use the Protocol to develop evaluation RFPs for the impact evaluation contracts to review and s upervise the evaluation contractors to assure adherence to the Protocol, to describe the ev aluation’s focus and approach to the evaluation stakeholders and information consumers, and to meet other needs identified by the Joint
Staff.

• Evaluation Contractors – will use the Protocol to develop their detailed evaluation plan in accordance with Protocol requirements and provide unbiased, objective, and independent evaluation results. They will use the Protocol to guide the evaluation effort and to ensure that the Protocol requirements have been met a nd the evaluation report provides the required
information.

• Portfolio Administrators – will use the Protocol to unders tand how the evaluation will be
conducted and what evaluation data needs and ef forts are needed to support the evaluation. In addition, the Protocol provide s background that administrators can use to determine when to intervene in the program design and implem entation efforts to achieve continued and/or greater efficiency gains.Evaluators’ Protocols Codes and Standards

CPUC 83 TecMarket Works Team • Program Implementers – will use the Protocol to unders tand the evaluation that will be conducted on their programs and program compon ents. Often, they will be required to provide data to support the evaluation. The Protocol wi ll also provide background information that implementers can use to unders tand when to intervene to achieve continued and/or greater efficiency gains.



### Key Inputs, and Outputs

There are several key Evaluation Protocol related inputs and output s including the energy impacts caused by the program-induced changes. This section of the Protocol lists the key information inputs that are needed to conduct th e evaluation and the key outputs that will be provided as a result of the evaluation.

Key Inputs The major evaluation input metrics needed to conduct the evalua tion efforts include:

1. Codes and Standards Program Theory and Logic Models, 2. Codes and Standards change descriptions, 3. Technology descriptions, 4. Program activity descriptions, 5. Identification of key codes and standards stakeholders, 6. Identification of the jurisdictions cove red by the codes and standards changes, 7. Estimate of pre codes and standards technology adoption or penetration rates before changes to the code are made.

Key Outputs The major outputs from the evaluation efforts include:

1. A listing of the technologies or practices influenced by the program that experienced an energy efficient code or standard change. 2. A listing of the code and standard changes that will be addressed in the evaluation. (Items 1 and 2 may be the same, but also may be different if the evaluation is addressing a subset of the changes.) 3. An estimate of the influence of the program on the code and standard changes for each technology or practice incl uded in the evaluation. 4. An estimate of the naturally occurring market adoption rates for each technology or practice included in the evaluation.
5. An estimate of the date when each code or standard change would have occurred without the program for each technology or be havior included in the evaluation. 6. An estimate of the level of non-compliance expected for the tec hnologies and practices covered in the evaluation over the peri od of time that savings are projected. 7. An estimate of gross and net market-level energy impacts for the program as a whole and for each technology and practice covered in th e program and for each utility territory funding the program. This estimate of imp acts should not exceed a 30-year effects
lifetime.Evaluators’ Protocols Codes and Standards

CPUC 84 TecMarket Works Team 

### Evaluation Methods

The evaluation of Codes and Standards programs re quires an Evaluation Prot ocol that is guided by the Impact Evaluation Protocol. The primary approach to establishing an energy savings value for the Codes and Standards Program is to assess the energy impacts of the market adoption and decision changes caused by the code or standard change, and then adjust those savings to account for what would have occurr ed if the program neve r existed. The evaluation must identify the net energy impacts that can be di rectly attributed to the program’s actions that
would not have occurred over the course of th e normal non-program influenced operations of the
market.

The end result of the application of this Protocol is the identification of the net ex-post energy savings achieved from code and standard cha nges above and beyond what would naturally occur in the market through normal non-code/standard driven technology a doption behavior and through the normal cycle of codes and standards updating activities. The resulting net programinduced energy savings are the savings that are caused by the program’s efforts.

The following sections of this Protocol descri be the required efforts for evaluating these programs. We note that the evaluation of the Codes and Standards Program can be accomplished in a single multi-year study incorporating an as sessment of the gross energy impacts from the code or standard changes, followed by the applica tion of net adjustment approaches described in this Protocol to produce net effects. These two efforts can also be structured independently, as conducted in the 2005 study by the Heschong Mahone Group. That study relied on the energy impact estimates from previously conducted energy impact studies.



### Evaluation Planning

Once an independent evaluator is hired, the evaluator must prepare a detailed evaluation plan that allocates the study’s finite resources to maximize the value and use of the information collected. The plan must provide detailed task-l evel information and fully describe the data collection and analysis approaches that are to be conducted. Th e plan must be provided in enough detail that it can be replicat ed to achieve the same conclusions . As part of this plan, the evaluator must specifically addr ess the various sources of releva nt potential error and explain how the error will be mitigated 72

.

The Impact Evaluation Protocol w ill guide the gross market-level energy impact estimates, and the Sampling and Uncertainty Protocol will guide the gross market-level energy impact estimates and the approaches for identifying net adjustme nts to the gross savings . In conducting this evaluation, evaluators must assess, minimize, a nd mitigate potential bias and, when relevant, present the achieved level of pr ecision (including relative precis ion, error bounds, coefficients of variation, standard deviations, and erro r ratios) for interpreting the data.

72 In the pre-1998 Protocols, there was no requirement to address these sources of error in the research plan.

Evaluators only had to describe in the final report whether th ey had to address these various errors and, if so, what they did to mitigate their effects.Evaluators’ Protocols Codes and Standards

CPUC 85 TecMarket Works Team It is expected that a technology and behavior-specific code and standard application potential analysis will first be conducted to establish the population characteristics needed to inform the sampling plans associated with the evaluation. The potential analysis is an assessment that
describes the current saturation and penetration of the specific technologies or behaviors that may be evaluated. The study then identifies the re maining potential that will be captured via a code or standard change. Alternatively, the ev aluation will use market size estimates prepared by the Codes and Standards Program that have already projected the po tential applications remaining in the market. If the program-devel oped potentials analysis is used, the evaluation contractor must first assess the methodological ap proach used by the program to determine the suitability for use in the evaluation effort and identify weaknesses in the projections that can influence the accuracy of the evaluation findings. If the assessment is found to be unreliable, the
evaluation contractor will work with the Joint Staff to establish a methodology for estimating the market application potential and the characteristics of the market s needed to inform the study’s sampling plans. This will help ensure that the gr oss and net energy impact estimates for the code or standard change is representa tive of the market in which th e changes are to be measured.

The Joint Staff, and other outside stakeholders as deemed appropriate by the CPUC-ED, will review the evaluation plan submitted and discuss with the independent evaluator tradeoffs that are deemed necessary to maximize the reliability of the impact estimates. The Joint Staff can decide to modify the approach as necessary in order to increase precision or to improve the reliability of the study findings, or to have the plan meet budget or timeline considerations.

The evaluation plan will also identify any info rmation that will need to be supplied by the utilities so that they will have advanced notice of what will be requested in an official data request once the study is launched.



### Technology-Specific Code and Standard Change Theory

The first step in the evaluation process is to revi ew the codes or standard s change theories. The change theory is similar to a pr ogram theory, but it focuses on the measures included in the code or standard change, and the theo retical approach that the progra m is using to bring about the change. The change theory should present a story of how the program moves from the development of a change concept (for example, th e need to change the co de covering residential sidewall insulation in single family homes) to th e completion of the code or standard change and a description of the savings expect ed. It should also include an es timate of the difference in the penetration of the code or st andard-covered technologies between the pre-code adoption market and the post-code adoption market. The change theory should iden tify the activities that the program undertakes in its efforts to move from a change concept to a successful code or standard change. A code or standard change theory shoul d be developed for each code or standard being changed. For example, if the code change focu ses on duct sealing, there should be a duct sealing code change theory that describe s the activities that will be used to bring about the change in duct sealing practice. The code and sta ndard change theory should include:

1. A description of the technol ogies and measures affected by the change and the change being made.Evaluators’ Protocols Codes and Standards

CPUC 86 TecMarket Works Team 2. A description of the program activities, effo rts, and events associated with the change making process.

3. Identification of the key stakeholders the pr ogram needs to work w ith to influence the change, including their names, titles, organizations, addresses, phone numbers and where possible, their e-mail addresses 73 . These should all be key market actors that are (or are expected to be) instrumental in bringing about or helping to bring about the change. These individuals should be grouped by their roles in the change making process (program management and implem entation, code review and assessment, case study development, economic impact assessment, environmental impact assessment, market impact analysis, tec hnology availability assessment, supply chain analysis, lobbying, policy review and development, skills an alysis, etc.). By providing these examples, we are not suggesting that these classifications be used, but rather demonstrate that some form of responsibil ity classification be used so that the
evaluation contractor understand s their individual roles in th e change process. Lists of individuals involved in the change effort s and their roles should be maintained throughout the program’s implementation e fforts and program managers should be ready to provide these lists to the evalua tion contractor on request. These interviews will be conducted over the pre- and post-change period.

4. The outputs, products, efforts and activities from the program that are used to cause the change, identifying how they are used to affect or support the change.

5. The incremental and final outcomes from th e program’s change efforts and activities that have been or are planned to be accomplished.

6. The timelines associated with the progra m’s change efforts, including the adoption dates of each change and the date the change is to apply. We expect that the program change timeline will be multi-year, because code or standard change efforts are launched at least two years before a formal adoption takes place, and at least three years will pass before they become effective in the market.

7. A description of the code and standard that has changed (after the official adoption), and an electronic or hard copy of the parts of the code or standard that are changed,
with code or standard reference numbers to allow independent confirmation of the change.

8. A description of the jurisdictions covere d and not covered by the code or standard changed, and any conditions that would ex empt or prohibit a jurisdiction from implementing the code or standard. This should identify all th e significant reasons why a code or standard may not be fully a dopted within the jurisdictions affected by
the code or standard.

9. A pre-code and standard change descri ption of the penetrat ion levels of the technologies covered in the code or standard in the targeted mark ets and a description of the expected penetration levels followi ng adoption of the program-influenced code or standard. These penetrations should be provided for each of the markets being targeted for the code or standard.

73 The names, addresses, and contact in formation of the people the program works with should be considered

confidential information and pr otected from disclosure.Evaluators’ Protocols Codes and Standards

CPUC 87 TecMarket Works Team For assistance in understanding the nature of a pr ogram theory and the associated logic models,
see the California Evaluation Framework , page 30. While the Framework does not detail what is included in a change theory, the codes and st andards change theory should be similar to a program theory and the supporting logic model. However, the focus of the evaluation of the Codes and Standards Program is not at the program summary level, but in stead is developed for each technology/measure targeted by the Codes and St andards Program. It is expected that the change theory for codes and standards program s will include theories on technologies/measures that are being successfully moved or have moved to a code or standard change. This condition
allows the evaluation contractor to understand the full nature of the program operations and focus, including the approaches for technologies and measures that move from the concept stage to the code or standard change stage.

The code and standard change theory will be a key document used to guide the evaluation effort. Without the code and standard change theory, the evaluation contractor cannot fully understand the efforts, events, and key individuals that must be considered to develop the evaluation plan. This Protocol recognizes that the change theori es will be developed and modified over time, as the program moves through the im plementation process. The change theories developed early in the process are expected to be less specific and less “ fleshed-out ” than the theories developed mid-stream and during the final adoptions processes. The eval uation contractor will need to make sure that the change theories used to guide the evaluation efforts are the most recent theories. These Protocols require the program ad ministrators to provide updated program change
theories to the evaluation cont ractors immediately after they are developed or modified. However, the evaluation contract or should also confirm that th ey are planning the evaluation using the most up-to-date change theories.

Each code and standard change theory shoul d be accompanied by a code and standard logic model that graphically displays each theory. The logic model will include the resources used by the program, the activities of the program, the outputs from the program activities, and the outcomes expected from the changed codes and standards.

These documents will be instrumental in estimating the level of influence of the program on the adoption of the specific codes and standards changes.

The evaluation contractor will request the program theory and logic models from the program administrator(s) immediately after the evaluation contract is negotia ted. If the program staff has not developed the theory, the evaluation contractor will notify the program administrator(s) and the CPUC-ED that the code change theory is not available to guide the evaluation planning process and the evaluation planning efforts canno t proceed. At this time, the CPUC-ED will
instruct the administrators of th e program to develop the code a nd standard change theories and supportive logic models. The program administrato r(s) will then devel op the theories and the supporting logic models for the covered technologies.

If the program theories and logi c models are not available at th e time of the evaluation request, the administrator my elect to hire contractors to develop or help develop these materials. These
materials must be delivered to the CPUC-ED with in 40 days of the notice and be used to guide the development of the evaluation plan. The de velopment of the evaluation plan should beEvaluators’ Protocols Codes and Standards

CPUC 88 TecMarket Works Team launched immediately after the eval uation contractor is hired, but not finalized until the program theories and logic models have been delivered an d used to guide the evaluation planning efforts. Because program theories and logic models are “living documents” that change as program designs and objectives change, it is important th at the most updated theories and models guide the evaluation plan. Alternatively, the CPUC-ED can instruct the evaluation contractor to work with the program managers to develop the cha nge theories and the supporting logic models to guide the evaluation effort. If this step is taken, the Program Administrator must “sign-off” on the accuracy of the theories and the supporting models before they are used to guide the
evaluation efforts.



### Evaluation Approach

Identify the Evaluation-Covered Codes & Standards In this effort the evaluation contractor, in coor dination with the program administrators and Joint Staff, will identify the specific codes and standards that have been, in some way, influenced by the program’s activities, and identi fy those that will be incorporat ed into the evaluation effort. This assessment will use the code change theori es, logic models and market actor information provided above, in addition to cons ultations with the pr ogram administrators and Joint Staff. Typically, the impact evaluation will focus on 5 to 25 changed portions of applicable codes and/or standards, depending on the number of code or standard changes that have been adopted, however, the actual number may be mo re or less than this range.

Not all energy-related code or standard change s are caused by or influenced by the Codes and Standards Program(s). These non-program changed codes or standards are not included in the impact evaluation. Similarly, not all codes and standards changes target ed by the program make it into a new code or standard, how ever the costs of these efforts should be included in the cost
effectiveness evaluation of the c odes and standards program, even if they have not yet become adopted by one or more jurisdictions.

The codes and standard changes that can be incl uded in the impact evaluation plan and assessed
in the evaluation are those for which:

1. The program has developed a code or sta ndard change theory and supportive logic model,

2. The program-covered change has been adopted, or is expected to be adopted by at least one public jurisdiction (c ity, county, or state) who has made the code or standard a required or voluntary practice, and

3. The change theory provides a reasonable cause and effect relationship leading from a concept stage to an adopted code or sta ndard, indicating that the program’s actions can be expected to have a positive influe nce on the adoption process. If there is disagreement on what constitutes “ a reasonable cause and effect relationship ,” Joint Staff will make the decision with advice from the program administrator and the
evaluation contractor.Evaluators’ Protocols Codes and Standards

CPUC 89 TecMarket Works Team When these conditions exist, an assessment of th e impacts of that technology or practice change will be included in the impact evaluation. Howeve r, the Joint Staff, after consulting with the program administrators and working in concert with the evaluation contractor, may elect to modify the code and standard changes addressed in the study as a result of expected or projected program actions.

Conduct a Codes and Standards Gross Market-Level Energy Impact Assessment

The evaluation contactor will conduc t a load impact evaluation of the savings (kWh, kW, and therms of natural gas) expected from the technologies that are covered by the code and standard changes. This study is a gross market-level a ssessment that focuses on the total amount of savings that can be expected by the changes, rega rdless of the cause of those changes. However, this study only focuses on those changes that ar e targeted by the program and for which the code change theory explicitly identifies as being affected by the program’s efforts.

In conducting this study the evaluation contractor will follow the Impact Evaluation Protocol to estimate savings from the technologies affected by the code or standard change. The “Basic Level of Rigor” for estimating gross energy impact s, as identified in the Impact Evaluation Protocol, is to be applied to a ssessing the gross market-level energy impacts. However, the Joint Staff can stipulate either more or less rigorous methods during th e evaluation planning process if there is a need for more accurate savings estimate s, if budget or timeline re straints requires a less rigorous approach, or if Protoc ol-covered evaluation findings that have already estimated the energy impacts for a given technology can be used to estimate market-lev el gross savings. The goal in establishing this requirement is to have flexibility in the eval uation design process to
meet unforeseen barriers to the ev aluation, but still establish a defa ult level of rigor for which the
estimates can be based. The evaluation contractor will work with the Joint Staff to set rigor levels consistent with the needs of the study, the study timeline and the evaluation resources.

The evaluation contractor may not need to conduct an impact evaluation assessment on a particular technology or practice if that technology or practice has already been evaluated using a reliable impact assessment approach similar to the approaches covered in the Impact Evaluation Protocol (2006). When previous evaluation findings can be direc tly used or modeled (simulated) to reflect the use and application conditions asso ciated with the changed codes and standards, that approach should be used if it results in a reliable energy savings estimate. Likewise, the evaluation contractor may not need to conduct an impact evaluation on a particular technology or practice if a review of the program’s estimates of energy savings, and the supporting documentation and case studies, are found to be relia ble. In this case th e evaluation contractor should review the program’s estimated savings and, in consultation with Joint Staff and the program administrators, discuss th e threats to validity associated with the estimation approach and determine if the approach is reliable enough that the eval uation contractor can use the estimates, or if they can be made more relia ble through additional engineering adjustments, modeling or modeling changes, a dditional field M&V, or appli cation testing efforts. The purpose of allowing the use of pr evious evaluation results and of the program’s energy saving estimates is to not expend evaluation resources if reliable energy savi ngs projections can be constructed by using previous work.Evaluators’ Protocols Codes and Standards

CPUC 90 TecMarket Works Team If there are no previous impact evaluation studies associated with a specific code or standard change that can be used, or adju sted and used, and if the progra m’s energy savings estimates are
found to be unreliable or have si gnificant threats to validity ma king them unreliable even with addition modeling, M&V efforts, or field testing, th e evaluation contractor is to develop a plan to assess the energy savings for that technology or practice using th e Impact Evaluation Protocols to develop the evaluation approach.

It is expected that as the En ergy Impact Protocol (2006) is adopted and used, more and more technologies will have been evaluated under the Pr otocols in which the results can be used or adjusted to reflect expected code and standard application conditions, th ereby reducing the need for new technology evaluations to feed the code s and standards gross market-level impact
estimates.

As noted earlier, the default approach for conduc ting the market-level energy impact assessment
is set at the Basic Level of Ri gor as specified in the Impact Evaluation Protocol for estimating gross program impacts unless the Joint Staff or the CPUC-ED has assigned a different level of rigor for a given technology. In making the rigor assignments, the Joint Staff will consider past evaluations and their energy savi ngs estimates for covered technol ogies and the potential to use these study results, the need for different leve ls of accuracy in the market-level energy assessment for individual technologies, the av ailable budget to support the assessment and the timeline for the evaluation, in addition to other criteria. These requirements mean that at a
minimum:

1. Simple engineering model estimation approaches, or

2. Normalized annual consumption approaches will be used, unless

3. The CPUC-ED or Joint Staff have approve d an alternative approach based on the
criteria discussed above.

The results of this assessment will be an annual energy savings estimate covering the first year of code or standard adoption for each technology or be havior change covered in the change theory. This estimate will be based on the expected penetration rate associated with each change across
the market sectors for which the code or standard change applies, assuming that it would impact all installations covered by the change. In assessing the saving s it will be necessary for the
evaluation contractor to estimate the increase in adoption of each technology or behavior change resulting from the code or standard change. Th is assessment will most likely involve the use of projected construction le vels grounded on historic constructi on patterns, estimated retrofits and change-outs driven by normal market forces, a nd other estimates of change for each of the changes. This annual savings will then be projected into the futu re to construct a time-sensitive estimate of gross savings.

In assessing the gross market-level energy sa vings it will be important for the evaluation
contractor to understand that th e code or standard changes su pported by the program’s efforts may not be consistent with the newly adopted changes. That is, the program may focus its efforts on a more aggressive or less aggressive energy efficient ch ange to the code or standard than what is actually adopted. As a result, th e gross energy savings assessment must focus on the changes made to the adopted codes and standards that were influenced by the program, rather than the changes recommended by the program. Likewise, the ev aluation contractor must checkEvaluators’ Protocols Codes and Standards

CPUC 91 TecMarket Works Team to see that the program-influenced changes are still in force. Just as codes can change to be more energy efficient, they can also change to be less energy efficient.

In assessing the gross market savings the eval uation process should disaggregate the savings assessment efforts into the specific installation, construction or purchase changes being made as a result of the code or standard change. This may mean disaggregating the savings analysis into measure groups or small clusters of measure gr oups rather than aggregating multiple measures and practices into large groups 74

.

Once the gross market-level energy impacts ar e identified, the following approach will be applied to develop an estimate of net program effects.

Estimate the Program’s Influence on the Adoption of Codes & Standards Once the gross market-level energy savings estimate s are established, they must be adjusted to
account for the influence of the Codes and Standa rds Program on the code or standard change. The program may be only minimally responsib le for a given change, or may have had a significant influence on the code and standard adoption process 75 . For each technology or behavior, the evaluation contractor must establish a percent attribu tion factor for the savings that can be attributed to the program. These per centages can range from no influence (0% if the program had no tangible influence on the cha nge) to a significant influence potentially approaching 100 percent (if the prog ram was the primary influencing factor driving the change).

A stakeholder interview-based preponderance of the evidence approach will be used for this process. This process will identify key stakeholde rs and conduct multiple interviews with these stakeholders at different point s in time along the adop tion path, during both the pre-adoption and post-adoption period.

The evaluation contractor will c onduct interviews with a repr esentative sample of the key
stakeholders identified earlier (see item 3 in the Technology-Speci fic Code and Standard Change Theory above) and use the result s of these interviews, along with reviews of program materials and documents (including lobbying documents, staff reports, case st udies, and staff and stakeholder correspondence as ava ilable) and attendance at progr am meetings and key events associated with the adoption process (to the ex tent possible and practic al) to assign causation percentages for the change to various change agents identifie d by the stakeholders, including
direct or indirect efforts of the program. In making these attribution assignme nts, the evaluation contractor will want to consider the potential bias of the individuals interviewed and of the information reviewed, and cross-check stated opinions with applicable documents and the opinion of other stakeholders, in order to test the causal rela tionships between actions and
results. The evaluation contractor should make as objective an assignment as possible. The evaluation contractor will assign weights to the opin ions of the stakeholders based on a review of all available information (noted above). The c ontractor will assign highe r weights to those who

74 Note: a previous study aggregated the assessment into one change assessment cluster that represented 66% of

the savings even though the change re presented different measures, approac hes and technologies. The study should disaggregate the assessment to the extent possible and practical given the evaluation needs and

resources.
75 The assignment of attribution of cause is to assess energy savings via the evaluation approach. It is not placed in

this Protocol to establish the program’s NTG valu es or to change the ex ante projected savings.Evaluators’ Protocols Codes and Standards

CPUC 92 TecMarket Works Team are most likely to have a complete understanding of the change efforts and processes relative to a specific technology or set of technologies and who are more likely able to accurately judge the relative causes of the adoption of the new codes or standards. This will allow the attribution of change to be more informed by those who are in a position to best judge the reasons for the change. Utility and other program staff and cont ractors hired by the program should be included in the sampling approach and be interviewed. As with these and other individuals interviewed, the evaluation contractor will keep in mind the pote ntial biases that may be associated with any single individual. In the weigh ting process, significant weights should be applied to the opinions of non-program stakeholders who are instrument al in the statewide jurisdictional decision processes to adopt a code or sta ndard change and to advisors or key stakeholders informing this process. In selecting a sample of interviewees, the Sampling and Uncertainty Protocol should be followed, with the sampling method determined at th e individual code or st andard change level. The interview process should be structured to conduct both pre-change and post-change
interviews.

The interview protocol and the interview guide should be designe d to be objective and rely on the opinions of the key stakeholders. The interv iew guide should be a prompted guide, so that the interviewee is not placed in the position of tr ying to identify all the different causes for the change. The evaluation contractor will devel op a list of program and non-program associated change agents/causes based on a review of the ch ange theories and interviews with a small but adequate sample of evaluation contractor-sel ected program and non-program stakeholders. The sample selection for these interviews does not have to follow the Sampling and Uncertainty
Protocol.

The evaluation contractor will plan the sample selection for the stakeholder interviews to focus on the program-identified stakeholders containe d in the change theory documents or other associated documents. However, the evaluation contractor will use a “snowball” sampling approach in which the sampled interviewees will be asked to identify additions to the sample of
individuals the interviewee indicat es were instrumental in the ch ange consideration or decision process. The evaluation contractor will target an additional 20 percent of the interview sample points to interviewing stakeholders recommended by the interviewees who are not on the change theory stakeholder list. If the ev aluation contractor is unable to obtain an additional 20 percent, the contractor will conduct as many of the addi tional interviews as possible and state in the evaluation report that they were unable to iden tify or interview an additional 20 percent.

The results from the interviews will be aggreg ated and used to assign technology and behavior change attribution of the changes caused by the Codes and Standards program. The results of this process will be a percentage distribu tion of the causes for each change across the
stakeholder-identified reasons for the success of th e newly adopted code or standard change for each of the technologies or behaviors covered.

It is expected that there will be significant levels of interview ove rlap across the technologies and behaviors so that a single interv iew may cover several technologi es or behaviors related to a code change or changes. This sampling process assures that adequate sa mples will be selected for each technology or behavior-associated change and that the attribution will be based on program-identified and stakeholder-identified change agents.Evaluators’ Protocols Codes and Standards

CPUC 93 TecMarket Works Team Once the attributions have been es tablished at the technology level, the evaluation contractor will multiply the energy savings for each technology or be havior by the attribution score to identify the gross market-level energy impacts that were caused by the Codes and Standards Program. This savings estimate will be further adjusted to account for net program effects (see below).

The timing of the estimation of the program’s influence is critical to the success of the
evaluation. The attribution assessment must be started very early in the Codes and Standards Program cycle, but not completed until the adoption process has been completed for the changes being evaluated. The technology or behavior change selections and the associated code and standard development efforts for the 2008 codes and standards began in the fall of 2005 and will continue through early 2006. In order for the a ttribution efforts to be based on recent knowledge, the interviews must be conducted during the technology selecti on and demonstration development process (as appropriate) and again when the adoption process is complete. This means that the attribution assessment may need to be launched years before the program experiences its first code or standard associated savings 76

.

In assessing the program’s influence on the adopt ion process, the evalua tion contractor should consider a number of program and market cond itions and activities that influence the adoption process and the associated adoption decisions rela tive to the individual cha nges. In considering these changes the Protocols references the Codes and Standards white paper 77 in which different adoption influence weights were used to assign attribution. While this white paper should be examined in the evaluation planning process, the evaluation contractor sh ould be careful not to select program or market condi tion weighting criteria that corre lates with or overlaps among the weighting metrics so that the weighting appro ach acts to double-count ad option influence across more than one of the weighting criteria.

Estimate Net Program Induced Energy Impact The gross market-level energy impacts that we re caused by the Codes and Standards Program must be adjusted to account for naturally occurr ing market adoption changes, normally occurring codes and standards revisions, and non-complian ce with the new codes a nd standards. These adjustments are discussed below and need to be ma de in the order prescribed in this Protocol.

Naturally-Occurring Market Adoption The first adjustment to the gross energy savings estimate identified above is an adjustment to account for the naturally occurring market adopt ion rates. New energy efficient products are likely to penetrate and be adopted by at least a po rtion of the market even without the Codes and Standards Program. As a result, the projecte d naturally occurring adoption and penetration, which would occur without the program, needs to be subtracted from the program’s gross energy impacts.

76 This means that evaluations of codes and standards pr ograms conducted in the first years following the issuance

of this Protocol will be operating in a “catch-up” mode because the program will have already launched the change
efforts on which the first evaluation will focus.

77 Codes and Standards Program Savings Estimate, Augus t 1, 2005 (or most recent revision), Heschong Mahone

Group, page 8. CALMAC SCE0241.01.Evaluators’ Protocols Codes and Standards

CPUC 94 TecMarket Works Team Naturally occurring adoption rates for premium energy efficient pr oducts typically occur in an

“ S” shape pattern that never reaches 100 percent penetration as long as there are alternative technologies in the market. This is especially true when the alternatives are lower cost technologies. Some energy efficient technologies may never capture a majority of the market share without a mandatory code or standard. Othe rs may move to capture the majority of the market without a code or standard. However, ther e is likely to always be some level of increased penetration of a superior product that delivers benefits to a user, up to a point of product demand saturation, based on the characteristics of the pro duct and the alternative c hoices in the market. Similarly, some customers never adopt a new produc t regardless of the benefits of the product. These customers are typically labeled as “laggard s” within the technolo gy adoption literature.

This step requires the evaluati on contractor to establish expe cted adoption curves for each technology included in the impact assessment. The evaluation contractor will use a range of approaches to establish the estimated penetrati on curves, including conducti ng literature searches on the penetration rates of sim ilar technologies with similar pr oduct characteristics, the use of expert opinions on the expected penetration rates in the absence of a requirement to use the technology, relevant market data and other approaches as deemed appropriate in the evaluation planning effort.

The evaluation contactor will then adjust the projected savings to account for the naturally occurring adoption for each techno logy covered in the assessment.

Non-Compliance Adjustment The second adjustment to gross savings is an adjustment for non-compliance. Since not all buildings or appliance decision makers will fu lly comply with the newly adopted codes or standards, these lost savings must be subtracted from the gross estimate.

In the real world, there is often a range of app liances or measures present in the market, some falling below the standard and some above the st andard in their energy efficiency levels. Similarly, technologies that do not comply with th e new code or standard are often stocked and sold in the market regardless of the requireme nts adopted. For example, while programmable thermostats are now required in California for most space heating and cooling applications, it is easy to acquire and install non-compliant thermost ats because of the stocking and sales patterns of a wide variety of wholesale and retail outlets , including internet sales. In some cases, if permits are not required or obtained, the c odes and standards enforcement mechanisms associated with the building inspection process may not be applied, enabling non-approved installations to occur. Likewise, it is difficult to inspect code-covered ap plications of measures such as insulation once the construction is comp leted to enforce code compliance, making this measure difficult to inspect and enforce.

In order to comply with the Evaluation Prot ocol, the evaluation cont ractor must estimate noncompliance across the technologies being assessed a nd adjust the anticipated savings for the net non-compliance rate over time. Fo r technologies that do not compl y, but are easily available in the market, the non-compliance rate may be high . However, for other technologies that are typically inspected as part of th e construction or retrofit process, the non-compliance rate may be
low.Evaluators’ Protocols Codes and Standards

CPUC 95 TecMarket Works Team To establish the rate of non-comp liance the evaluation contractor will conduct inte rviews with a set of building architects, e ngineers, contractors, product wholesalers and retailers and
installation contractors. The evaluation contactor will design a sample plan consistent with the Sampling and Uncertainty Protocol to matc h the technologies being assessed. Because compliance is measure-specific, samples will be set at the technology level within each code or standard changed. In developing this adjustment the contractor will need to be sensitive to differences in compliance rates across the stat e and over time. As a result, the evaluation
contractor should consider approaches for adjusting for local differences in compliance rates, such as establishing and using compliance assessment jurisdictions. These approaches will be coordinated with and approved by Joint Staff before they are implemented.

The evaluation contractor will also assess the avai lability of non-compliant technologies in the market by examining the stocking practices of selected suppliers of the technologies. For example, if a building products supplier stocks 30 percent non -compliant technologies, the noncompliance rate for that technology can be assumed to be 30 percent for their customer market, unless there is evidence to the contrary collected during the interview efforts. The evaluation contactor will suggest ways to conduct the stoc king assessment and can include such approaches as visits to suppliers to examine the stocking mix or interviews with suppliers to estimate their stocking mix.

The evaluation contractor will then assess the re sults of the interviews, the examinations and other assessment approaches suggested by the evaluation contractor and approved by the Joint Staff and estimate the rate of compliance for each technology or behavior change. The estimate will not be a single fixed level, but will be tim e-adjusted, so that the expected rate of noncompliance will change over time. To arrive at the time-adjusted compliance estimate the
evaluation contractor should rely on projections provide d by the interv iewees.

It is important for the evaluati on contractor to focus on identi fying net compliance adjustments during this assessment and take into account the pre-change comp liance rate for a given change condition. There may be substantial portions of the market that are not in compliance before the change and are not in complian ce after the change. Likewise, a non-compliant rate before the program may have the same non-compliant rate af ter the change. The ev aluation contractor is expected to develop plans that provide for net compliance cha nges over time. The contractor
will coordinate with Joint Staff in this effort.

The evaluation contactor will then adjust the projected savings to account for the estimated levels of non-compliance.

Normally-Occurring Standards Adoption Next an adjustment to the gross savings need s to account for the norma lly occurring codes and standards change process. A primary effect of the Codes and Standards Program is to accelerate the time it takes for the CEC and other jurisdicti onal organizations to up date current codes and standards or adopt new codes or standards. The CEC employs a th ree-year update cycle, keeping the standards up-to-date and cost-e ffective as market c onditions change. Ho wever, without the Codes and Standards Program resources, the updates might not encompass the same type of technology analysis and change cons iderations. It is reasonable to assume, therefore, that theEvaluators’ Protocols Codes and Standards

CPUC 96 TecMarket Works Team standards adopted by the CEC or other jurisdicti ons would have been adopted in the normal course of events, but over a much longer period of time. The energy savings from the Codes and Standards Program should only include the savings from the codes and standards implemented as a result of the program’s efforts for the period of time that they would not be covered by a
revised code or standard during the normal course of the update cycle.

In order to establish the estimated time at wh ich the CEC and other jurisdictions would have adopted or created a code or standard without the program, th e evaluation contractor must establish a panel of experts who are familiar with and involved in the code change efforts. This panel will consist of CEC program staff, CEC code and standard update staff, code and standard public officials within other ju risdictions, and other experts as deemed appropriate by the evaluation contractor and approved by the CPUC -ED or the Joint Staff. The evaluation contractor will then conduct a minimum two-round Delphi 78 assessment with this expert panel to arrive at a projected date that the CEC would be expected to imp lement a new code or standard in the absence of program initiatives. This pr ocess should cover each technology or behavior in the assessment. It is expected that the size of this panel will be between 10 and 20 experts.

Once the estimated timeline for each code or standa rd change is established, the energy savings for the technologies and behaviors changed as a result of the code and standard changes will not be counted beyond that projected date, but in no event will the savings be counted beyond a 30- year period. This step sets an end-date for th e period of time that savings can be counted for each code or standard change.

Actual Construction and Retrofit True-Up The energy savings estimates produced from this Protocol are based on a single assessment of the gross energy savings for a singl e year projected into the future . However, not all years are the same. The economy and other changes (interes t rates, unemployment, consumer confidence, etc.) affect the rate at which technologies ar e adopted and used, and thereby influence energy savings. As a result, it is necessary that the CPUC-ED may elect to periodically issue a new RFP to conduct an update of the projected savings to account for actual savings. When the CPUC-ED requests an update, the evaluati on contractor will assess the market and update the savings projections to account for actua l construction and adoption.

It is not possible to accurately estimate sa vings without knowing how much construction was actually accomplished following a code or standard change. There are severa l ways to adjust the energy savings projections to account for actual construction and a prefe rred approach is not specified in this Protocol. Howe ver, a true-up of actual constructi on is needed to help increase the accuracy of the savings estimate over time. If the true-up evaluation is conducted in the 5 th
year following the code change, then the true- up should contain estimate s of actual construction for the first 4 years of which permitting and building records could be assessed. Once the evaluation has a history of actua l construction, the new projecti on of future construction (to estimate future savings) can be based on the hi storical construction. Once the projection is

78 Delphi assessment is an iterative pr ocess that involves repeated rounds of information gathering across a selected

group of experts. Responses to one round are summariz ed and developed to feed the next round of information gathering. The purpose of the Delphi is to seek agreement across the group of experts.Evaluators’ Protocols Codes and Standards

CPUC 97 TecMarket Works Team established, the savings can be pr ojected and the adjustments can then be subtracted (or added, depending on actual construction data) from these or iginal projections to obt ain net realized past savings and the updated projected future savings based on the updated estimate. When an update is requested the CPUC-ED or the Joint Staff will wo rk with the evaluation contractor to identify an approach to be used. This approach may be based on construction industry statistics (e.g., annual real estate construction estimates), building construc tion databases (e.g., the Dodge
database and/or Construction I ndustry Research Board (CIRB) re views of building permits for a set of representative jurisdictions ), assessments of sales data if the data can be reliably obtained (a historic problem for sales data collection) or other approaches.

Multiple-Counting of Ener gy Savings Adjustment To make sure that the savings from code cha nge covered measures, practices and purchases are not counted more than once, no energy effici ency or demand management/response programs that offers code or standard change covered m easures, equipment or practices is permitted to count the savings from these meas ures, practices or purchases to ward their energy savings goals
once the codes and standards evaluation has docum ented the savings from these efforts, unless those savings are from Code Compliance E nhancement programs. The Code Compliance Enhancement Program evaluation will then document savings beyond what is achieved as a result of the code and standards change 79

.

Measure Life Adjustments This Protocol excludes an adjustment for measur e life. It is assumed that once a measure is adopted as a result of a code or standard change , the behavior will be repe ated until that code or standard is eliminated or updated. However, ev en if the code change is updated, the savings from the measures are still provided. Likewise, new evaluations will document the increased efficiency of the updated codes or standards. In addition, the inclusion of normal market adoption rate adjustments and normal code and sta ndard change revisions will act to significantly reduce the savings over time. However, the energy savings provided via the use of this Protocol shall not be projected beyond 30 years.

Impacts by Utility Service Territory Once the statewide estimates of adjusted net savi ngs have been estimated, an allocation of the savings to the utility service territories can be ma de. This assignment of savings will be based on assigning savings to a utility for me asures that are actually installed within their service territory.
The allocation will be based on the distribut ion of new home construction, nonresidential construction square footage, and appliance sales fo recasts within each service territory such that the total savings across the territories equals 100% of the adjusted net savings estimated from the program less the savings from the local jurisdictions that had impl emented or were in the process of implementing the adoption of changes covered in the scope of the evaluation. This Protocol
condition means that the evaluation contractor wi ll need to conduct a surv ey of at minimum the 20 most populated (or preferably an d provided the data is available the 20 jurisdictions with the highest numbers of building-starts) local jurisdictions within each IOU service territory to assess if that jurisdiction was substantially in the process of convert ing their code or standards to the

79 See Protocol steps for assessing Compliance Enhancem ent Programs located at the end of this Protocol.Evaluators’ Protocols Codes and Standards

CPUC 98 TecMarket Works Team covered changes or had completed this effort at the time the program was advocating for the change. The evaluation contractor must obtain approval from the Joint Staff on the jurisdictions to be surveyed for each code or standard change. It is expected that there will be substantial overlap among the identified juri sdictions and that most of the targeted jurisdictions will be surveyed for more than one of the program’ s covered changes. In selecting the local jurisdictions to survey, the eval uation contractor will survey enough jurisdictions to be able to reliably measure the program’s net effects.

Because the construction, retrofit and sales mark ets change over time, this assessment and the adjustment approach will need to be trued-up pe riodically. These refinements will be specified by the Joint Staff, or the CPUC, in order to allocate savings over time based on market
conditions.

Reporting The evaluation report will be provided in complian ce with the Reporting Protocol. A draft report will be provided for review and comment to the stakeholders (see Reporting Protocol). Once comments are provided on the draf t report the evaluation contract or will work with the Joint Staff to finalize the report. Once the final report is accepted by the Joint Staff, the evaluation
contractor will construct an abstract consistent the instructions contained on the CALMAC.org web site and post the report.



### Summary

This Protocol describes a way to estimate the gross and net energy impacts from the Codes and Standards Program. It begins with the review of the program change th eory and logic models and the development of an evaluation plan. The imp lementation of the plan consists of estimates of gross market-level impacts for each technolog y and behavior adjusted to account for naturally occurring market changes, non-program induced code and standard revisions, and code compliance rates for each technology. The eval uation delivers net impacts for each technology and for the program as a whole, and then distri butes the energy impacts to the participating utility companies.

This Protocol is prescriptive in nature, but allows for the use of new techniques or approaches when approved by the CPUC-ED or Joint Staff. As a result, it does not impede the evolution of evaluation approaches.

The Codes and Standards Program Evaluation Pr otocol is guided by the Impact Evaluation Protocol, the Sampling and Uncertainty Pr otocol and the Reporting Protocol.

The following table provides a summary of the Prot ocol that can be used to guide the evaluation
efforts once the detailed contents of the Protocol are well understood.Evaluators’ Protocols Codes and Standards

CPUC 99 TecMarket Works Team 

### Summary of Protocol-Driven Codes an d Standards Evaluation Activities

1 Joint staff selects an evaluation contra ctor to implement the Codes and Standards Program evaluation.

2 The evaluation contractor reviews the pr ogram change theories and the program logic models, identifies the technologies or behaviors that can be evaluated via the Protocol, constructs a draft evaluation plan and submit s the plan for approval to the CPUC-ED.
The contractor works with the CPUC-ED on the development of the draft evaluation plan and rigor levels. The plan must use the Impact Evaluation Protocol, the Sampling and Uncertainty Protocol and the Reporting Prot ocol in the development of the evaluation plan and in the implementation and reporting efforts.

3 The CPUC-ED works with the evaluation contractor to finalize and approve an evaluation plan from which the contra ctor can begin the evaluation effort.

4 The contractor conducts an assessment of the gross market-level energy impacts for each code and standard covered technology or behavior being evaluated consistent with the rigor level assignments.

5 The contractor determines the influence of the program on the adoption of each code and standard covered in the study and alloca tes adoption attribution. The assessment uses an interview approach for this assessm ent. This assessment is accomplished as early in the code change cycle as possible but preferably in the technology selection and demonstration phase of the cycle.

6 The contractor estimates naturally occurr ing code and standard covered technology or behavior adoption rates based on literature reviews and interviews with experts.

7 The contractor adjusts the gross market le vel energy savings estimates to account for the net adjustment factors for naturally occurring technology adoption, naturally occurring code change, and non-compliance. This approach nets out the influence of non-program-induced impacts from the gross mark et-level impacts for each technology.

8 The contractor estimates the timeline asso ciated with adoption of a code and standard without the program, using a Delphi approach with an expert panel.

9 The program administrators remove savings estimates from their programs for codecovered measures.

10 The evaluation contractor assesses the cons truction and sales efforts for each utility company service territory and allocates savings by IOU based on the construction and
sales estimates.

11 The contractor reports the results of t he evaluation to the CPUC-ED and Joint Staff consistent with the provisions in the Reporting Protocol.

12 Once the report is accepted by the CP UC-ED, the contactor develops abstracts and posts the report on the CALMAC web site following the CALMAC posting instructions.

13 As needed, the CPUC-ED or the Joint Staff can request the evaluation contractor to update and report the actual energy savings ov er time consistent with the Protocol.
U p dates can be conducted with a different evaluation contractor than those doin g theEvaluators’ Protocols Codes and Standards

CPUC 100 TecMarket Works Team original assessment.

Note: the steps included in this evaluation su mmary table must be accomplished in accordance with all the requirements within th e Codes and Standards Protocol in order to be in compliance.



### Code Compliance Enhancement Programs

To conduct energy impact evaluations of programs designed to influence the rate of compliance
of code-covered measures the evaluation contra ctor should not follow the Codes and Standards Program Evaluation Protocol pres ented above, but should follow this Protocol specifically designed to estimate the energy savings from these programs.

Because the California IOU por tfolios have not included code compliance improvement programs in their portfolios, and because these pr ograms have yet to be evaluated to the extent that a standard evaluation approach can be reliably identified, this Protocol allows a wide variety of methods and approaches for assessing the savings from these efforts. After several of these
evaluations have been conducte d and the success of the appr oaches documented, a standard approach may be developed and added to the current Protocols. Until that time the following guidance will be used to st ructure and implement the evaluation of code compliance enhancement programs.



### Definition of a Code Complia nce Enhancement Program

A Code Compliance Enhancement Program (CEP) is any energy efficiency, demand reduction or demand management program whose primary purpose is to increase the level of customers complying with a code requirement that saves energy (kWh, kW, therms).



### What this Protocol is Designed To Do

This Protocol establishes a framework under whic h CEP programs are to be evaluated to assess energy impacts. This Protocol does not establis h program designs, program design criteria or program development approaches.

Code compliance enhancement programs can be incentive programs that are designed to increase compliance by providing incentives to customers to do what is required, educational programs to
make customers or trade allies aware of the co de and the need for compliance, training programs to train customers or trade allies how to comp ly, enforcement programs that take enforcement actions against non-compiling property owners, or other types of program designs. These programs may also involve more than one type of delivery strategy.



### Joint Staff Responsibilities

Because CEP are not (at this time) part of the suite of energy program services delivered in California, the Joint Staff are responsible for de termining when to evaluate a CEP program and how that evaluation should be conducted and repo rted. However, that evaluation must employ the following approaches unless other approaches are requested and approved by the Joint Staff.Evaluators’ Protocols Codes and Standards

CPUC 101 TecMarket Works Team 

### Draft Evaluation Plan

The evaluation contractor will prepare a draft de tailed evaluation plan and submit that plan for review and approval to the Joint Staff. The plan should provide for a time-series measurement approach that can be replicated at different times over the im plementation period. Joint Staff will review and comment on the evaluation plan a nd will work with the evaluation contractor to focus the plan on the evaluation objectives of the CPUC. This plan will serve as the approval process for launching the detailed evaluation planning efforts.



### Program Theory Review and Assessment

The evaluation contractor will review and asse ss the program theory provided by the program administrator. This review will be focused on understanding the approach the program is taking to effect a compliance change and the activitie s that are employed to accomplish the program’s objectives. Once the program theory has been asse ssed the evaluation contractor will modify the draft evaluation plan and submit the plan to Join t Staff for review and approval. One purpose of the program theory review is to allow for the examination or the program theory to feed the evaluation planning process so that the evalua tion contractor can id entify key measurement points on which the program needs to focus. Once the draft evaluation plan has been updated from the program theory review effort and appr oved by the Joint Staff the evaluation efforts can
be launched.



### Pre-Program Compliance Rate

The evaluation contractor will work with the Join t Staff to develop an approach for measuring the pre-program compliance rate for the measur es covered by the program. This approach should focus on assessing the condition of the ma rket and taking measurements that allow the
evaluation contractor to iden tify the level of pre-program non-compliance within the geographical areas targeted by the program. The Codes and Standard s Evaluation Protocol incorporates instructions on assessing compliance rates for evaluating Codes and Standards programs. These instructions are incorporated in to this Protocol as a guidance resource for identifying non-compliance rates during the pr e-program period. The Joint Staff and the
evaluation contractor are free to develop other methods if, in the opinion of the Joint Staff, the alternative approach can be expect ed to be more or equally reliab le to the approach presented in
the Codes and Standards Protocol.

The purpose of this activity is to establish th e baseline from which post-program changes in construction practice or measured installed can be assessed. It is expected that this assessment
will need to be sensitive to local jurisdictions and changes in compliance within the local jurisdictions. The outcome of this effort will be the identification of the level of compliance for each program-targeted code change within the market sectors and jurisdictions on which the program’s efforts are focused.

It is expected that the pre-program compliance rate s will be set at some level of detail that will allow the evaluation to identify jurisdictional differences in compliance rates.Evaluators’ Protocols Codes and Standards

CPUC 102 TecMarket Works Team 

### Post-Program Compliance Rate

At a period of time to be determined by the Join t Staff the evaluation cont ractor will again apply the same strategy used to assess pre-program compliance rates within the geographical areas targeted by the programs. These areas on which the evaluation will focus must also be the same areas of the state that the pre-program comp liance assessment was focused so that the jurisdictions examined in the pre-program assessment match the jurisdictions examined in the post-program assessment. The primary purpose of th is effort is to document the compliance rate after the program has been implemented long enough for an expected change in compliance to be
measurable.

The time periods for the post-program compliance rate assessment will be set periodically over the program implementation period to allow re sults to be tracked over time and reported consistent with the reporting needs of the CPUC. For some measures and programs this may mean an assessment every six months, for others the assessment can be done annually, while for others the assessment may be needed every few y ears. The evaluation cont ractor in coordination with the Joint Staff will identify the periods in which the post-program compliance should be
assessed.



### Adjustment For Naturally O ccurring Compliance Change

The natural compliance rate for most code re quirements will change over time. Normally, compliance is expected to be lower on the date th e change first applies. This is then followed by a period in which compliance rates increase and begin to stabilize as the change is structured into
the market and local code officials and trade allie s change their approaches to comply with the new code. Because the compliance rates cha nge as a normal course within the market operations, the normal compliance rate that woul d have occurred without the program must be adjusted out of the calculation for net program compliance changes. The evaluation contractor will work with the Joint Staff to identify an a pproach for identifying normal compliance change rates. Because these programs have not been impl emented or evaluated in the past, a prescribed approach for identifying the rate of natura lly occurring compliance is excluded from this Protocol. Joint Staff may wish to employ trade ally surveys, expert panels, code official interviews, measure sales tracking approaches or comparison areas where the program services are not offered. Each of these approaches has th eir own strengths and we aknesses that should be assessed and considered in the planning process. Joint Staff or the evaluation contractor may wish to suggest other approaches for considerat ion. However, the Joint Staff must approve the procedures for identifying natura lly occurring compliance change be fore the effort is launched.



### Net Program-Induced Compliance Change

Once the pre-program and at least one round of post-program compliance assessments is conducted, the evaluation contractor will asse ss the net change in compliance across the jurisdictions targeted by the program. The evaluati on contractor will work with the Joint Staff to identify the approach to be taken in this assessmen t, however it is expected that the approach will be a simple jurisdiction controlled net change assessment.Evaluators’ Protocols Codes and Standards

CPUC 103 TecMarket Works Team 

### Assessment of En ergy Savings

Once the net assessment of change is identifie d, the evaluation contractor will use the savings estimates provided from the codes and standards program evaluation for the same code covered
measures included in the CEP. If the codes and standards evaluation effort is not completed in time for the CEP evaluation, the Joint Staff will decide to delay the completion of the CEP evaluation or launch the impact assessment appr oach prescribed in the Codes and Standards Evaluation Protocol for assessing net energy impacts from the code covered changes. However
the Joint Staff can consider the use of other approaches, such as the assignment of DEER estimated savings for the covered measures, or an engineering-based assessment to estimate the probable energy savings, if these approaches are considered reliable predictors of the savings associated with a specific change. Other approach es can be applied at the request or approval of the Joint Staff, but these methods should focus on obtaining reliable savings estimates consistent with the available evaluation budge t and the study timelines.

The net assessment procedures used must take in to account the measures that would have been
installed or constructed wit hout the CEP program and the energy consumption difference
between what would have been installed or constr ucted compared to the code-required efficiency
levels.



### Recommendations for Program Changes

The evaluation contractor is also to provide recommendations for program changes that can be developed as a result of the examination of the program theories, and the implementation of the
evaluation assessment efforts. While this eval uation is not a process evaluation, the evaluation contractor may be able to provide valuable ch ange recommendations that can be considered by the program administrator.



### Cost Effectiveness Assessment

The evaluation contractor will c onduct a cost effectiveness assessment using the program cost data reported to the CPUC in the monthly or quarterly program progress tracking cost reports submitted by the administrator to the CPUC. Th e evaluation contractor will conduct a TRC and a PAC test consistent with the approach provide d in the Standard Practic e Manual. The results will be reported in the evaluation report.



### Reporting of Evaluation Results

The evaluation report should follow the Evaluation Reporting Protocol to meet the timelines and deliverable dates specified in the approved evalua tion plan. The deliverable dates will take into consideration the reporting n eeds of the CPUC across the mu lti-year program implementation period.Evaluators’ Protocols Codes and Standards

CPUC 104 TecMarket Works Team 

### References

Belzer, D., K. Cort, D. Winiarski, E. Richman, and M. Friedrich, (2002) Analysis of Potential Benefits and Costs of Adopting ASHRAE Standar d 90.1-1999 as a Building Energy Code in
Illinois Jurisdictions. Report prepared for U.S. Departme nt of Energy by Pacific Northwest National Laboratory.

Cort, K. and D. Belzer, (2002) Statewide Savings Projections from the Adoption of a Commercial Building Energy Code in Illinois . Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory.

Cort, K., D. Belzer, D. Wini arski, and E. Richman (2002) Analysis of Potential Benefits and Costs of Updating the Commercial Building Energy Code in Iowa . Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory

Cort, K., D. Belzer, M. Halverson, E. Richman, and D. Winiarski (2002) Analysis of Potential Benefits and Costs of Adopting ASHRAE Standar d 90.1-1999 as a Commercial Building Energy Code in Michigan . Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory.

Jacobs, P., and D. Roberts. (2001) “Estimated Impacts of the Non-Residential and Appliance
Standards” in CA IOU Codes and Standards Earnings Claim Framework Final Report. San Francisco, CA: Pacific Gas and Electric Company.

Heschong Mahone Group. (2005) Codes and Standards White Paper on Methods for Estimating Savings , prepared for Southern California Edison Company.

Heschong Mahone Groups. (2005) Codes and Standards Program Savings Estimate , Prepared for Southern California Edison Company.

Heschong Mahone Group. (2001) CA IOU Codes and Standards Earnings Claims Framework, Final Report. Prepared for Pacific Gas and Electric Company.

Lucas, R. (2001) Assessment of Impacts from Updating Idaho’s Residential Energy Code to Comply with the 2000 International Energy Conservation Code. Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory.

Lucas, R. (2001) Assessment of Impacts from Updating Ke ntucky’s Residential Energy Code to Comply with the 2000 International Energy Conservation Code. Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory.

Lucas, R. (2002) Assessment of Impacts from Adop ting the 2000 International Energy Conservation Code for Residentia l Buildings in Illinois (Draft). Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory.

Lucas, R. (2002) Assessment of Impacts from Updating Ne w Mexico’s Residential Energy Code to Comply with the 2000 Interna tional Energy Conservation Code. Report prepared for U.S. Department of Energy by Pacific Northwest National Laboratory. Nittler, K., Wilcox, B. (2001) “Estimated Impacts of the Residential and A ppliance Standards” in CA IOU Codes and Standards Earnings Claim Framework Final Report. San Francisco, CA: Pacific Gas and Electric Company.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 105 TecMarket Works Team 

## Effective Useful Life Evaluation Protocol (Retention and Degradation)

Introduction

One of the most important evaluation issues is how long energy savings are expected to last (persist) once an energy efficiency measure has b een installed. The Effective Useful Life (EUL) Evaluation Protocol was develope d to address this i ssue and should be used to establish the period of time over which energy savings will be c ounted or credited for all measures that have claimed savings. This Protocol contains requir ements for the allowable methods for three types of evaluation studies: retention, degrad ation, and EUL analysis studies.

A persistence study measures changes in the net impacts that are achieved through installation/adoption of program-covered measures over time. These changes include retention and performance degradation. The definition of retention as used in this Protocol is the proportion of measures retained in place and that are operable. Effec tive useful life (EUL) is the estimate of the median number of years that the measures installed under th e program ar e still in place and operable (retained). 80

The primary purpose of this Protocol is to provid e ex-post estimates of effective useful life and performance degradation for those measures whos e estimates are either highly uncertain and/or have not been covered in studies over the past 5 years. These results will be used to make prospective adjustments to the measure leve l EUL estimates and performance degradation estimates for Program Years 2009 and beyond, but will not be used for retroactive adjustments of the performance of the 2006-2008 portfolios.

The Effective Useful Life Evaluati on Protocol is established to en sure that all persistence-related evaluations are conducted using evaluation me thods deemed acceptable based upon the assigned level of rigor for that evaluati on. The identification of allowable methods is one component of this Evaluation Protocol that helps ensure grea ter reliability in the energy and demand savings estimates from California’s energy efficiency effort s. The Joint Staff can assign different levels of rigor for each measure in any study under this Protocol, thus allowing the Joint Staff the flexibility to allocate evaluation resources according to the needs of the Portfolio given uncertainties in the expected savings, the size of expected savings, the program budget and other
criteria. The Joint Staff will instruct the evaluati on contractors to use specific rigor levels based upon the Joint Staff’s application of the decisi on criteria contained in this Protocol and
evaluation resource allocations.

Rigor is defined as the level of expected reliabi lity. The higher the le vel of rigor, the more
confident we are that the results of the evaluation are both accurate and precise, i.e., reliable.

80 These definitions are as in the Glossary. Some are the same and some have been modified from what was used

in the prior M&E Protocols. There ar e, however, inconsistencies across diffe rent states in how these terms are used. Evaluators conducting EUL evaluations in Ca lifornia need to be familiar with the current California
definitions.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 106 TecMarket Works Team That is, reliability and rigor ar e synonymous. Reliability is di scussed in the Impact Evaluation Protocol, the Sampling and Uncer tainty Protocol, and in the Evaluation Framework where it is noted that sampling precision does not equate to accuracy. Both precision and accuracy are important components in reliability, as used by the CPUC. Each evaluation study will be assigned a specific evaluation rigor level for its primary evaluation objectives to guarantee that a
minimum standard is met.

Past experience presents a few important notes of caution. Many past pe rsistence studies were unable to provide results that were significantly different (statisti cally) from the ex-ante results, so that most of the current ex-post EULs are th e same as the ex-ante estimates. Besides finding relatively high retention rates in mo st cases, a consistent and importa nt finding in these studies is that a longer period of time is needed for conduc ting these studies, so that larger samples of failures are available, and so th at technology failure and removal ra tes can be better documented
and used to make more accurate assessments of fa ilure rate functions. The selection of what to measure, when the measurements should be la unched, and how often they should be conducted are critical study planning considerations that Join t Staff will direct to ensure reliable results are
achieved.

Performance degradation includes both (1) techni cal operational characteristics of the measures, including operating conditions and product design, and (2) human interaction components and
behavioral measures. This Prot ocol refers to these two differe nt components of performance degradation as technical degrad ation and behavioral degradati on, respectively. (Performance degradation studies are also referr ed to in this Protocol more simply as degradation studies.)

Performance degradation accounts for both time-re lated and use-related change in the energy savings from an energy efficient measure or practic e relative to a standard efficiency measure or practice. It is important to note that the ener gy savings over time is a difference rather than a straight measurement of the progr am equipment/behavior. It is the difference over time, between the energy usage of the efficient equipment/beh avior and the standard equipment/behavior it replaced that is the focus of the measurement. Energy efficiency in both standard and high efficiency equipment often decreases over time. The energy savings over time is the difference between these two curves. The technical degradati on factor is a set of ra tios for each year after installation/adoption as the proportion of savings obt ained in that year comp ared to the first-year savings estimate, regardless of th e retention estimate or EUL (whi ch is applied separately to obtain overall savings persisted). The technical (or behavioral) degradation factor could be 1.0 for each year in the forecast (often 20-year tech nical degradation factors are estimated) if the energy efficiency decreases (energy usage increa ses) by the same percentage each year as the standard equipment. This is the case where t echnical degradation rates are the same for both types of equipment. The technical (or behavior al) degradation factor would be higher if the efficient equipment holds its level of efficien cy longer/better than the standard equipment 81 and lower if there is more relative degradation.

Technical degradation studies may not be rou tinely required in the 2006-2008 round of EM&V studies because the incremental level of this type of degradation measured in five persistence

81 This was found to be the case in 3 of the 25 measures studied in the five persistence studies conducted under the

prior M&E Protocols: residential d/x air-conditioning, residential refrigerators, and agricultural pumps.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 107 TecMarket Works Team studies from 1995 to 2000 was found to be in significant for over 95% of measures. 82 Joint Staff, however, may require a technical degradation study at their disc retion. These may be needed based upon comments and findings within impact eval uations that discover potential issues with technical degradation, technologies not assessed in the five prior studies, changes in technology for the efficient or standard equipment, or for other reasons. For example, a technical degradation study may be desired for duct sealin g which has not been previously studied.

The prior persistence studies included human inte raction/behavior in the assessments made for the 25 measures examined. The importance of this may have been most prominent in the assessments of daylighting and energy managementsystems (EMS). The contribution of longerterm behavioral impacts on energy savings expect ations over time could be an issue that still needs further examination, particularly given th e greater emphasis being seen on more recent measures that include larger be havioral interaction components. The large human influence in the degradation factors found in EMS may also s uggest periodic re-assessment or more fieldbased measurement studies. Measures that may need to be considered for behavioral degradation studies also include, but are not limited to, commissioning, retr o-commissioning, operations and maintenance (O&M) efficiency efforts, programma ble thermostats, and specific behavior-based
initiatives. Joint Staff will select which m easures will receive what types of studies and determine the scope of those evaluation st udies focusing on behavioral degradation.

It is expected that evaluation contractors will respond to requests for proposals (RFPs) for retention, EUL, and performance degradation eval uations with proposals that meet the standards
contained in this Protocol. Th e minimum allowable methods, sample criteria, and data collection criteria for these types of evaluations are provided later in this EUL Evalua tion Protocol. In their proposal, evaluation contractors may propose (i n addition to Protocol compliant methods) optional methods, if the contractor can clearly demonstr ate that the optional methods provide at least as much rigor and accuracy as the Protocol-covered approach.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• Joint Staff evaluation planners – will use the Protocol to dete rmine: (1) when special studies are needed for evaluating the retention, EUL, an d degradation of particular measures, (2) as input into the RFPs for the evaluation of retent ion, EUL, and degradation, and (3) as background and criteria for use in reviewing retention, EUL, and de gradation evaluation plans, managing the retention, EUL, and degr adation evaluations, and reviewing retention, EUL, and degradation evalua tion reports and results.

• Evaluation project team – will use the Protocol to make sure that the evaluations of the retention, EUL, and degradation of particular measures are based upon the level of rigor(s) designated by the Joint Staff. They will also use the Protocol to double-check that the

82 These five persistence studies conducted during the prio r M&E Protocols and under supervision of the CADMAC

Persistence Subcommittee are referred to as Persistenc e 1 (P1), Persistence 2 (P2), Persistence 3A (P3A), Persistence 3B (P3B), and Neg-TDF Supplement (PNg). These studies covered 25 measures and can be found in the CALMAC searchable database: 2023.pdf, 1998051 4CAD0006MR.pdf, 2028.pdf, 19990223CAD0003MR.pdf, and 19990223CAD0004MR.pdf.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 108 TecMarket Works Team Protocol requirements have been met as they conduct, complete, a nd report the retention, EUL, and degradation evaluations.

• Portfolio administrators – will use the Protocol to unders tand how the retention, EUL, and degradation evaluations will be conducte d on their programs, and to understand the evaluation data needs to support the retent ion, EUL, and degradation evaluations.

• Program Implementers – will use the Protocol to unders tand how the retention, EUL, and degradation evaluations will be conducted on thei r programs. Often, they will be required to provide data to support the retention, EUL, and degradation evaluations.

• ISO / System planners – will utilize retention, EUL, and degradation estimates and uncertainty estimates for load forecasting and system planning.



### Overview of the Protocol

This section briefly describes th e three Protocols contained with in the EUL Evaluation Protocol, Protocol rigor levels, key metric s assessed, and assessment inputs a nd outputs. This section is followed by sections that present the three Protocols that describe the allowable minimum methods for retention, degradation, and EUL st udies. The reporting requirements for studies
conducted within this Protocol are provided. This is followe d by a short section providing guidance on the skills required by evaluators to conduct the type of studies described in this Protocol. The last section provi des a brief summary list of the steps needed to comply with the
EUL Evaluation Protocol.



### Protocol Types

The Effective Useful Life Evalua tion Protocol contains three Protocols, each providing the minimum requirements for: (1) retention studies, (2) degradation studies, and (3) EUL analysis studies. Each Protocol has two levels of rigor (Basic and Enhanced).

For each study, Joint Staff and their evaluato rs should examine opportunities for coordination (e.g., sampling and identifying and marking measures for further study) with impact studies: for example, it may be possible to conduct an analysis of retention for some measures in an impact study, to use the same sample, or use the same sample in a later study. But there are limitations in coordination: e.g., budget, appropriate sample , issues with ensuring random sampling across different objectives, etc. Strate gically, it may be best to examin e coordination for three different types of coordination: concurre nt studies (e.g., examining current program impacts and retention from earlier participation at the same large commercial/industrial si tes), past studies (i.e., using samples and information from prior impact studies for later retention or de gradation studies), and future studies (e.g., collecting placement/location in formation or tagging in an impact study in
order to assist a futu re retention study).

An example of how findings from the three types of persistence evaluations would work together to provide a measure’s overall persis tence is presented graphically in Figure 7.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 109 TecMarket Works Team Figure 7. An Example of How Findings Across the Three Types of Studies Would Work Together for Persistence Evaluations



### Rigor

When the Joint Staff decides a measure will receive a retention, EUL, and/or degradation evaluation, it also selects the level of evaluation ri gor that is required. Th e Effective Useful Life
Evaluation Protocol establishes the methods appropriate for the type of retention, EUL, and degradation evaluation designated to be conducted for the assigned level of evaluation rigor. In this way, the Protocol establishes a minimum level of evaluation rigor in order to ensure that the retention estimates, degradation factors, a nd EUL estimates produced are at the level of reliability needed to support the overall reliability of the savings in the administrator’s Portfolio
and the statewide Portfolio. Retention Protocol Study Retention Study:
In-Place & Operable

EUL Analysis Protocol EUL Analysis

Adequate time since adoption for enough failures

and large sample for EUL analysis?

Retention study is treated as an early verification study. A later retention study is needed that is designed to meet EUL analysis needs. A later retention study
is conducted. No Yes

Degradation Study Protocol Technical Degradation Study

Behavioral Degradation Study

All prior relevant studies plus current studies are used to assess best persistence estimate

and EUL and degradation factors for future program planning.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 110 TecMarket Works Team The level of rigor provides a class of allowable methods in order to offer flexibility for the potential evaluation contractors to assess, a nd propose the most accurate and cost-effective
methods that meet the Joint Staff’s needs. The principle is to provide minimum specifications for allowable methods, sample size criteria, and minimum data collection specifications and yet encourage evaluation contractors to utilize both the art and sc ience of evaluation to develop affordable and quality evaluations that produce re liable savings estimates. There are two levels of rigor for each of the three Protocol s: Basic and Enhanced, as shown in Figure 8. (The requirements for these rigor leve ls are described below).

Figure 8. Protocols and Rigor Levels for EUL Evaluations

Joint Staff may assign rigor levels for evaluation studies covered in this Protocol for a measure or group of measures, for a measure within a delive ry strategy, sector, or application. Separate retention estimates, degradation factors, and EULs may be required for measures by delivery strategy, sector, and application as assigned by Jo int Staff. (Further discussion is provided below.)



### Key Metrics, Inputs, and Outputs

Retention, EUL, and degradation evaluations will draw upon relevant data obtained from program databases, program descriptions, D EER database, work papers developed during program planning, on-site measurements, obser vational data, survey and interview data collection, manufacturers’ studi es, ASHRAE studies, laboratory studies, and other prior study data and reports. The use of these resources to support the planning and implementation of retention, EUL and degradation studies will he lp produce more reliable retention, degradation
and EUL estimates.

Retention studies will provide the percent of the measures retained, along with clear descriptions of the methods used to determine measure-specific retention rates. In a ddition, these studies will provide complete definitions of what is considered an “ operable condition ” that constitutes a Persistence
Evaluation Basic Basic

Enhanced Enhanced CPUC Assigns Required Rigo r Levels for All Types of Studies by Measure and Strategy/Application

Retention Study Protocol Degradation Study Protocol EUL Analysis Protocol

Basic

EnhancedEvaluators’ Protocols Effective Useful Life Protocol

CPUC 111 TecMarket Works Team retained status, and describe the testing criteria used to determ ine the operable status. Reporting of the retention estimates and degradation factor s will also include a clear description of the methods employed and any adjustments made to ensu re that the estimates appropriately represent all program installed/adopted measures without bi as associated with changes in occupancy or
location. The location where the me asure was originally located need s to be maintained within a sample, regardless of occupant status, in cases where measures are not moved. For measures/behaviors that can be portable or easi ly moved, the study will verify the location and
use of the measures and determine if they are still being used in a way that provides the projected savings. For the purposes of this Protocol, loss of retention is assumed when participants have moved out and taken the measures with them, unl ess the study provides re liable installation and energy savings use verification for the retaining of measures within the same utility service territory. (Finding and tracking movers for this purpose, however, is not required.)

Because EUL evaluations must follow the Sampli ng and Uncertainty Protocol, evaluators must also assess, minimize, and mitigat e potential bias and present the achieved level of precision (including relative precision, erro r bounds, coefficients of varia tion, standard deviations, and error ratios) for interpreting in formation, summarizing retention es timates, degradation factors, and EULs and their precision by measure and stra tegy/application, and providing the information necessary for future evaluation planning. Where precision is calculated fro m chaining or pooling of evaluation study efforts, the above precision information should be provided for each study
effort as well as the combined result.

All studies and evaluations c onducted under this Protocol must comply with the reporting requirements contained in this Protocol.



### Retention Study Protocol

These are minimum standard Protocols. All me thods with higher rigor are allowable as they exceed the minimum criteria. For example, if th e measure has a Joint Staff-assigned Basic Rigor and the method proposed by the evaluation contr actor is an option under the Enhanced Rigor level, this method will be acceptable for mee ting the Protocol if it meets budget and timing constraints. The Enhanced Rigor approach is th e preferred approach for all retention studies. The Basic Rigor level may be assigned where this is more reasonable given the technology involved and budget constraints.

The Retention Study Protoc ol is summarized in Table 11. Further description, additional requirements, clarification, and examples of this Pr otocol are presented afte r the table. Being in compliance with the Retention Study Protocol means that the methods used and the way in which they are utilized and reported meet all the requirements discussed with in this section (not just those within the summary table or those within the text) to provide unbiased reliable
retention estimates. These Protocols someti mes reference other documents that provide examples of applicable methods. However, the operative requirements are only those stated in these Protocols, and not in the other references.

Measure retention studies collect data to determine the proportion of measures that are in place and operational. The primary evaluation compone nts of a measure retention study are research design, survey-site visit instrument design, esta blishing the definition of an operable statusEvaluators’ Protocols Effective Useful Life Protocol

CPUC 112 TecMarket Works Team condition, identifying how this condition will be measured, and establishing the data collection and analysis approach. The meas ure retention estimate can be a straightforward calculation from the data collected. The key planning document a ssociated with the study is the evaluation plan, which presents and discusses the methods for thes e components, as well as describing the data collection field efforts to be employed to s upport the data collection approach, and the study reporting to be delivered.

Joint Staff will decide which measures must rece ive retention studies and whether these studies must be conducted by delivery strategy, sector or other segmentation scheme in order to obtain
reliable EUL estimates that can be used as a basis for future program planning. The evaluation contractor is expected to assess these instructions and work with Joint Staff to ensure that the most appropriate and cost-effec tive retention evaluation design is developed. This should be done as part of the initial eval uation planning and be completed prior to the completion of the final approved Evaluation Plan.

All retention evaluations are required to have a detailed ev aluation plan. The evaluation plan needs to include a number of components to support an assessment of the adequacy and approach of the evaluation effort. Th ese include the following components:

• Cover page containing the measures and deliver y strategies or appli cations included in the retention evaluation, program names in th e portfolios that include these, program administrators for these programs and thei r program tracking number(s), evaluation contractor, and the date of evaluation plan.

• Table of Contents.

• High-level summary overview of the measures and delivery strategies or applications included in the retention evaluation, the progr ams affected, and the evaluation efforts.

• Presentation of the evaluation goals and resear chable issues addressed in the evaluation.

• Description of how the evaluation addresse s the researchable issues, including a description of the evaluation priorities and th e use of assigned rigor levels to address these priorities.

• A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity, sources of bias, and a short de scription of the approaches planned to reduce threats, bias, and uncertainty.

• Task descriptions of the evaluation efforts.

• Review of any related retent ion and EUL study planning effort s prepared for Joint Staff to include prior estimation of failure sample size requirements, panel retention data needs and availability, and data tagging and co llection efforts for these measures.

• Detailed description of the sampling rationale, methods, and sample sizes.

• Detailed description of the definition and methods for determining an operational
condition.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 113 TecMarket Works Team • Detailed description of the informati on that will be needed from the Program
Administrators in order to conduct the eval uation that will be included in evaluationrelated data requests, including an estimate of date that the information will be needed or for which accessibility to the data is needed.

• Total evaluation budget and a task-level evaluation budget for the study; and

• Contact information for th e evaluation manager, including, mail address, telephone numbers, fax numbers and e-mail address.

The evaluation plan should be written in a styl e and with enough detail that it can be clearly understood by program administrators, policy makers, and evaluation professionals.

Table 11. Required Protocols for Measure Retention Study

Rigor Level Retention Evaluation Allowable Methods

Basic 1. In-place and operable status assessment based upon on-site inspections. Sampling must meet the Basic Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirements for the subsequent EUL study. See below specification.)

2. Non-site methods (such as tel ephone surveys/interviews, analysis of consumption data, or use of other dat a, e.g. from EMS systems) may be proposed but must be explicitly appr oved by Joint Staff through the evaluation planning process. Samplin g must meet the Basic Rigor Level requirements discussed in this Protoc ol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirements for the subsequent EUL study. See below specification.)

Enhanced 1. In-place and operable status assessment based upon on-site inspections. Sampling must meet the Enhanced Rigor Level requirements discussed in this Protoc ol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirement for the subsequent EUL study. See below specification.)

The analysis of the retention data in either a Basic Rigor or Enhanced Rigor level study must include reporting the retenti on estimate as found in the study. Nevertheless, an assessment
should be included as to whether one model or one brand showed a strong affect on the retention
estimate where the exclusion of this model or brand from programs would change the expected resulting EUL by more than 25 percent (25%). If this is suspected prior to completion of the Evaluation Plan, the retention st udy sampling design and sample sizes may need to be conducted to produce retention estimates for this model or brand separately from that of other
models/brands where both retention estimates me et the sampling and precision criteria required for the assigned level of rigor fo r that retention study. Joint Sta ff and their evaluators may want
to consider whether the re tention study sample should be part of an impact evaluation sample, or have separate samples for retention and impacts studies.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 114 TecMarket Works Team Basic Rigor In-place and operable status assessment based upon on-site inspections is considered the default requirement for retention studies.

The in-place assessment shall be verified through on-site inspections of facilities. Measure,
make and model number data shall be collected a nd compared to particip ant program records as applicable. As-built construction documents may be used to verify selected measures where access is difficult or impossible (such as wall insu lation). Spot measurements may be used to supplement visual inspections, such as solar transmission measurements and low-e coating detection instruments, to ve rify the optical properties of windows and glazing systems.

Correct measure operation shall be observed and co mpared to project design intent. Often this observation is a simple test of whether the equi pment is running or can be turned on. This can also include, however, changes in application or sector such that the operational nature of the equipment no longer meets project design intent. For example, working gas-cooking equipment
that had been installed in a restaurant but is now installed in the restaurant owner’s home is most likely no longer generating the expected ener gy savings and would not be counted as an program-induced operable condition.

Non-site methods (telephone survey s/interviews, analysis of cons umption data, or use of other data, e.g. from EMS systems) may be proposed al ong with a detailed description as to why the proposed method(s) would be reliable for the stu dy measures in their st rategies/applications based upon theoretical and past st udy justifications. All methods, however, must be assessed and approved by Joint Staff through the evaluation planning process and e xplicit acceptance of proposed non-site methods are require d prior to their being used.

The reasons for lack of retention, and the ra tes of non-retention, should be gathered when feasible for use in developing EUL study designs and future retention studies.

In most cases, there will be a sample size require ment for an EUL study that will be used to determine the sample size requirement for a corresponding retention study since a survival analysis will be based on data collected earlie r in a corresponding rete ntion study. Thus, for a given retention time period under study, the sample size for a retention study must meet any prior sample size requirements determined for EU L studies on the proposed measures in these strategies/applications and must meet the re quirements of the Sampling and Uncertainty Protocol. However, there are two conditions th at could arise that s hould be addressed:

• If there is no EUL

- determined required sample size for the retention time period under study and the study retenti on period is within 30% of the expected EUL, then the sample size required for an EUL study must be calcul ated in order to determine the retention study sample size requirement. This includes using power analysis at a power of at least

0.7 for the Basic Rigor level to determine th e sample size required at a 90% confidence level (alpha set at 0.10), and then deriving the required retention sample size based upon the proportion of the original pool expected to be found in-place and operable (the exante EUL). (See the EUL Analysis Protoc ol and Appendix D for more information concerning the use of power analysis for determining sample size requirements.)Evaluators’ Protocols Effective Useful Life Protocol

CPUC 115 TecMarket Works Team • If there is no EUL- determined required sample size for the retention time period under study and the study retention period is not within 30% of the expected EUL, then the retention study sample size requirement should be based upon the coefficient of variation, standard deviation, and other available estima tes of variance for the percent of equipment that is in place and operable from prior studi es. The sample size s hould be large enough to attain a minimum of 30 % precis ion at the 90% confidence level.

Enhanced Rigor The in-place assessment shall be verified through on-site inspections of facilities. Measure,
make and model number data shall be collected a nd compared to particip ant program records as applicable. As-built construction documents may be used to verify measures such as wall insulation where access is difficult or impossibl e. Spot measurements may be used to supplement visual inspections, such as solar transmission measurements and low-e coating detection instruments, to ve rify the optical properties of windows and glazing systems.

Correct measure operation shall be observed and compared to project design intent. Commissioning reports (as applicable) shall be ob tained and reviewed to verify proper operation of installed systems. If measures have not b een commissioned, measure design intent shall be established from program records and/or cons truction documents; and functional performance testing shall be conducted to verify operation of systems in accordance with design intent.
This must also include as assessment of whether changes in application or sector are such that the operational nature of the equipment no longe r meets project design intent. For example, working gas-cooking equipment that had been in a restaurant but is now in the restaurant owner’s home is no longer meeting project design in tent and is no longer generating the expected energy savings.

Analysis of consumption data or use of data from EMS systems may be proposed along with a detailed description as to why the proposed met hod(s) would be reliable for the study measures in their strategies/applications based upon theore tical and past study justifications. (Telephone surveying or interviewing techniques are not presented as an allowed approach within the Enhanced Rigor level for retention studies.) All methods, however, must be assessed and approved by Joint Staff through the evaluation planning process and e xplicit acceptance of proposed non-site methods are require d prior to their being used.

The reasons for lack of retention, and the rates of these, should be gathered when feasible for use in developing EUL study designs and future reten tion studies. For example, in one study, the removal rate of refrigerators dur ing the first five years was f ound to be higher for locations
where the consumer moved. 83 It could be expected that as th e refrigerator ages the probability for older refrigerators being m oved with consumers may decrease. As a result, an improved
EUL function would reflect the risk of participants moving with their refrigerators.

83 This hypothesis, its testing, and consequences were ex amined in a study by Quantum Consulting and Megdal &

Associates in the Retention Study of Pacific Gas and Electric Company’s 1996 and 1997 Appliance Energy Efficiency Programs , Study ID 373 1R1, March 2001.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 116 TecMarket Works Team In most cases, there will be a sample size require ment for an EUL study that will be used to determine the sample size requirement for a corresponding retention study since a survival analysis will be based on data collected earlie r in a corresponding rete ntion study. Thus, for a given retention time period under study, the sample size for a retention study must meet any prior sample size requirements determined for EU L studies on the proposed measures in these strategies/applications and must meet the re quirements of the Sampling and Uncertainty Protocol. However, there are two conditions th at could arise that s hould be addressed::

• If there is no EUL- determined required sample size for the retention time period under study and the study retenti on period is within 30% of the expected EUL, then the sample size required for an EUL study must be calcul ated in order to determine the retention study sample size requirement. This includes using power analysis at a power of at least

0.8 for the Enhanced Rigor level to determine the failure sample size required at a 90% confidence level (alpha set at 0.10) and then deriving the required retention sample size based upon the proportion of the original pool expected to be found in-place and operational. (See the EUL Analysis Protoc ol and Appendix D for more information concerning the use of power analysis for determining sample size requirements.)

If there is no EUL -determined required sample size fo r the retention time period under study and the study retention period is not within 30% of the expected EUL, then the retention study sample size requirement should be determin ed based upon the coefficient of variation,
standard deviation and other available estimates of variance for the percent of equipment that is in place and operable from prior studies. Th e sample size should be large enough to attain a minimum of 10 % precision at the 90% confidence level.



### Degradation Study Protocol

These are minimum standard Protocols. All me thods with higher rigor are allowable as they exceed the minimum criteria. For example, if the measure has been assigned a Basic level of rigor by the Joint Staff and the method proposed by the evaluation contractor is an option under Enhanced, this method will be acceptable for meeting the Protocol. The Enhanced Rigor approach is the preferred approach for all re tention studies. The Ba sic Rigor level may be assigned as is reasonable given the te chnology involved and budget constraints.

The Degradation Study Protoc ol is summarized in Table 12. Further de scription, additional requirements, clarification, and examples of this Pr otocol are presented afte r the table. Being in compliance with the Degradation Study Protocol means that the methods used and the way in which they are utilized and reported meet all the requirements discussed with in this section (not just those within the summary table or those within the text) to provide unbiased reliable
estimates of the technical de gradation factor. The Protoc ols sometimes reference other documents that provide examples of applicable methods. The re quirements, however, are always those stated in these Protocols, which take pr ecedence over all others in all circumstances.

Performance degradation studies produce a factor that is a multiplier used to account for both time-related and use-related change in the ener gy savings of a high efficiency measure or practice relative to a standard effi ciency measure or practice. It is important to note that the degradation study is a relative difference measurement between the high efficiencyEvaluators’ Protocols Effective Useful Life Protocol

CPUC 117 TecMarket Works Team equipment/behavior and the non-high efficiency equipment/behavior over time (and not the relative level of retention). Said in a differe nt way, it is the difference over time between the energy usage of the energy efficient equipment/b ehavior and the standard equipment/behavior it replaced. Appropriate standard m easure comparisons are critical.

Studies must designate and clearly describe all the elements that will be analyzed for the degradation factor produced. If only a technical degradation f actor is produced, or only a behavioral degradation factor is produced for a measure that contai ns both (i.e. when a technical degradation factor and a behavior al degradation factor is associ ated with the same piece of equipment), this must be clearly noted. If th e equipment has both a technical and behavioral degradation factor, and one of these is to come from a previous study or another source, and the other is being addressed in th e Protocol-covered study, both factor s must be presented in the report, and the analysis must produc e a combined factor for that equipment. If only one factor is being produced, the study must clea rly describe whether this covers the full degradation factor to
be used for that measure or indicate if additi onal research is needed to establish a more complete/reliable factor that in cludes both technical a nd behavioral degradation. For example, a measure that is purely behavioral (e.g., mainte nance behavior schedules ) would only receive a behavioral degradation factor a nd this should be clearly describe d, however, a measure such as a programmable thermostat would need a combin ed performance degradation factor that incorporates the technical degr adation factor for the measure and the behavioral degradation
factor associated with the use of th e measure as an energy saving device.

Since the degradation factor is the difference be tween the standard equipment/behavior and that of the program measures, the over time, changes in usage for sta ndard and efficiency must be clearly assessed/measured and reported or the co mponent differences and their changes in usage over time must be explained and assessed/measured.

All degradation evaluations are required to have a detailed ev aluation plan. The evaluation plan needs to include a number of components to support an assessment of the adequacy and approach of the evaluation effort. Th ese include the following components:

• Cover page containing the measures and deliver y strategies or appli cations included in the degradation evaluation, program names in the portfolios that include these, program administrators for these programs and thei r program tracking number(s), evaluation contractor, and the date of evaluation plan.

• Table of Contents.

• High-level summary overview of the measures and delivery strategies or applications included in the degradation evaluation, the pr ograms affected, and the evaluation efforts.

• Presentation of the evaluation goals and resear chable issues addressed in the evaluation.

• Description of how the evaluation addresse s the researchable issues, including a description of the evaluation priorities and th e use of assigned rigor levels to address these priorities.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 118 TecMarket Works Team • A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity, sources of bias, and a short de scription of the approaches planned to reduce threats, bias, and uncertainty.

• Task descriptions of the evaluation efforts.

• Detailed description of the sampling rationale, methods, and sample sizes.

• Detailed description of the methodology to be used for the assessment for both the standard equipment/behavior and that for the efficient equipment/behavior, and the approach to be used for quantifying the di fference between these two conditions. This condition applies regardless if the study is determin ing the EUL by assessing the equipment as a whole unit, or if the assessment is conducte d for a key component of the equipment, or if the assessment is based on engineering assumptions about the expected performance or performance life of a component of the equipment.

• Detailed description of the informati on that will be needed from the Program
Administrators in order to conduct the eval uation that will be included in evaluationrelated data requests, including an estimate of date that the information will be needed or for which accessibility to the data is needed.

• Total evaluation budget and a task-level evaluation budget for the study; and

• Contact information for th e evaluation manager, including, mail address, telephone numbers, fax numbers and e-mail address.

The evaluation plan should be written in a styl e and with enough detail that it can be clearly understood by program administrators, policy makers, and evaluation professionals.

Table 12. Required Protocols for Degradation Study

Rigor Level Allowable Methods for Degradation Studies

Basic 1. Literature review required for technical degradation studies across a range of engineering-based literature, to include but not limited to manufacturer’s studies, ASHRAE studies, and laboratory studies. Review of technology assessments. Assessments using simple engineering models for technology components and which examine key input variables and uncertainty factor s affecting technical degradation.

2. Telephone surveys/interviews with a research design that meets accepted social science behavioral re search expectations for behavioral degradation.

Enhanced 1. For technical degradation: field measurement testing.

2. For behavioral degradation: fiel d observations and measurement.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 119 TecMarket Works Team Basic Rigor Technical degradation studies re quire a literature review for the measures under study. The literature search should include journal ar ticles, conference proceedings, manufacturer publications, publications of engineering societ ies (e.g., ASHRAE), natio nal laboratories, and government agencies, and the gray literature (i.e ., studies that are not widely published but are available upon request). In a ddition, technology assessments usi ng simple engineering models for technology components will be conducted. Studi es will be conducted that examine key input variables and uncertainty factors affecting technical degradation.

Laboratory testing may be used to determine th e technical degradation factor(s) for the Basic Rigor level. Laboratory testing involves the measurement of energy use of both energy efficient and standard equipment over time, but in unoccupied facilities. Laboratory testing must account for the operational conditions expected for inst allations obtained through the California programs that incorporate the measur e in their service mix.

Telephone surveys/interviews with a research design that meets accepted social science behavioral research expectations for behavioral degradation. The use of the term acceptable, in this case, means that the approach and the da ta collection methods would pass a peer review process using highly experienced professional social science rese archers, in such a way that peers would support and defend the approach as bei ng objective and reliable within the ability of the approach selected. The types of questions asked will focus on whether the energy efficient and standard equipment are being used/operated as designed and the reasons for their non-use or changes in use.

Enhanced Rigor Technical degradation studies at the Enhanced Rigor level requi re field measurement. Field measurement involves the measurement of energy use for both the energy efficient and the standard equipment over time – these measures woul d be located in occupied facilities. These measurements must be designed to collect data on the equipment or equipment components in order to reduce the greatest uncerta inties associated with the degrad ation factor estimates and be conducted with sample sizes to allow for 90% confidence and 30% precision in these

measurements.

Behavioral degradation st udies focus on the observation (and m easurement, if applicable) of the use of energy efficient and standard equipment in facilities. The studies can be short term (onetime site visits) or long term (periodic site visi ts) to assess if the measures are being used as designed. The types of questions asked will focu s on whether the energy efficient and standard equipment are being used/operated as designed a nd the reasons for their non-use or changes in use. The self-reports will be matched with observational data for confirmation. Measurement studies must be designed to collect data to reduce the greatest uncertainties associated with the degradation factor estimates a nd be conducted on sample sizes to allow for 90% confidence and 30% precision in these measurements.



### Effective Useful Life Analysis Protocol

These are minimum standard Effective Useful Li fe (EUL) Protocols. All methods with higher rigor are allowable as they exceed the minimum crit eria. For example, if the measure has a Joint Staff assigned rigor of Basic a nd the method proposed by the evalua tion contractor is an option under Enhanced, this method will be accep table for meeting the Protocol.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 120 TecMarket Works Team The EUL Analysis Protocol is summarized in Table 13. Further description, additional requirements, clarification, and examples of this Pr otocol are presented afte r the table. Being in compliance with the EUL Analysis Protocol mean s that the methods used and the way in which they are utilized and reported meet all the requir ements discussed within this section (not just those within the summary table or those within th e text) to provide unbiase d reliable estimates of
EUL. The Protocols sometimes reference other doc uments that provide ex amples of applicable methods. The requirements, however, are always those stated in these Protocols, which take precedence over all others in all circumstances.

The objective of the EUL analysis studies is to es timate the ex-post EUL, defined as the estimate of the median number of years that the measures installed under the program are still in place, operable, and providing savings. Evaluators are expected to develop a plan for estimating
survival functions for measures included in the sc ope of their work. The plan should incorporate an assessment of the study’s ex-post EUL compar ed to the findings of other studies on this measure for this delivery strategy/application and to the EUL contained in the DEER database (if one exists). The study should also provide re commendations for whether the new EUL should be substituted for the existing ex-ante EUL for future program planning or if another DEER category for the measure should be developed. Th e EUL studies are also required to report the findings from the most recent degrada tion studies related to the EUL study measures/applications, so that the EUL report is a depository for all current persistence studies for the study measures/applications. This will as sist Joint Staff and future evaluators find all relevant persistence information for a measure/applicati on in one location.

Joint Staff will decide which measures must rece ive EUL studies and whether these studies must be conducted by delivery strategy, sector, or othe r segmentation scheme in order to obtain reliable EUL estimates that can be appropriate ly used for various future program planning
alternatives. The evaluation contr actor is expected to assess these instructions and work with Joint Staff to ensure that the most appropriate and cost-effective evaluation design is developed,
so reliable EUL estimates are obtained to meet this purpose. This shoul d be done as part of initial evaluation planning and be completed prior to completion of the Evaluation Plan.

As part of the evaluation pla n, the evaluator should propose a method for estimating a survival
function and a survival rate of measures inst alled over time using standard techniques (see below). This should include an identification of factors that might lead to lower survival rates
and a discussion of how the confidence interval s for the survival functions derived will be
estimated. This should also in clude a discussion of potential s ources of bias in the methods proposed and how these sources of bias will be mitigated.

All EUL analysis evaluations are required to have a detaile d evaluation plan. The evaluation plan needs to include a number of components to support an assessment of the adequacy and approach of the evaluation effort. Th ese include the following components:

• Cover page containing the measures and deliver y strategies or appli cations included in the EUL analysis evaluation, program names in the current and past (up to 10 years previous programs) portfolios that include these, program administrators for theseEvaluators’ Protocols Effective Useful Life Protocol

CPUC 121 TecMarket Works Team programs and their program tracking number(s) if applicable, evaluation contractor for those programs, and the date of evaluation plan.

• Table of Contents.

• High-level summary overview of the measures and delivery strategies or applications included in the evaluation, the programs affect ed, and the evaluation efforts conducted.

• Presentation of the evaluation goals and resear chable issues addressed in the current EUL evaluation being planned.

• Description of how the evaluation addresse s the researchable issues, including a description of the evaluation priorities and th e use of assigned rigor levels to address these priorities.

• A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity, sources of bias, and a short de scription of the approaches planned to reduce threats, bias, and uncertainty.

• Task descriptions of the evaluation efforts.

• Review of any related EUL study planning effort s prepared for Joint Staff to include prior estimation of failure sample size requir ements, panel retent ion data needs and availability, and data tagging and co llection efforts for these measures.

• Detailed examination of related retention st udies, assessment of prior and concurrent
retention studies and recommendation of what additional data must be collected and, if so, why. If additional data collection is proposed, then a detailed description of the sampling rationale, methods, and sa mple sizes must be included.

• Detailed description of the informati on that will be needed from the Program
Administrators in order to conduct the eval uation that will be included in evaluationrelated data requests, including an estimate of date that the information will be needed or for which accessibility to the data is needed.

• Total evaluation budget and a task-level evaluation budget for the study; and

• Contact information for th e evaluation manager, including, mail address, telephone numbers, fax numbers and e-mail address.

The evaluation plan should be written in a styl e and with enough detail that it can be clearly understood by program administrators, policy makers, and evaluation professionals.

As noted below, we rely on power analysis fo r helping to differentiate Basic Rigor from Enhanced Rigor (see Appendix D for more discussi on and references). Statistical power is the probability that statistical significan ce will be attained, given that there is a measurable treatment effect. Power analysis is a st atistical technique th at can be used (among other things) to determine sample size requirements to ensure th at statistical significance can be found. Power analysis is a required component in the Protocol to assist in de termining required sample sizes.

The Basic level of rigor in the EUL Protocols re quires that a 0.70 level of power be planned at
the 90% level of confidence. While the Enhanced level of rigor require s that a 0.80 level of power be planned also at the 90 % level of confidence. In de termining sample sizes in theEvaluators’ Protocols Effective Useful Life Protocol

CPUC 122 TecMarket Works Team research planning process, values for key paramete rs can be varied in an attempt to balance a level of statistical power, the alph a, the duration of the study, and the effect size, all determined with an eye on the budget constraints. The values will probably be unique to each measure selected for study. The differing power requirements between the Basic and Enhanced level of rigor drives up the required sample size to meet the Enhanced Rigor level versus that needed to meet the Basic Rigor level. The results of the power analysis will be combined with professional judgment and past studies to arrive at the required samp le sizes. The selected sample size, the results of the power analysis, and the justification for the sample size proposed must be included in the evaluation plan. This evaluation plan must be approved by Joint Staff prior to sample data
collection.

Table 13. Required Protocols for EUL Analysis Studies

Rigor Level Allowable Methods for EUL Analysis Studies

Basic 1. Classic survival analysis (defined bel ow) or other analysis methods that specifically control for right-censored data (those cases of failure that might take place some time after data are collected) must be attempted. For methods not accounting for right-censored data, the functional form of the model used to estimate EUL (“model functional form”) must be justified and theoretica lly supported. Sampling must meet the Basic Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. Sample size requirements will be determined through the use of power analysis, results from prior studies on si milar programs, and professional judgment. Power analysis used to determine the required sample size must be calculated by setting power to at least at 0.7 to determine the sample size required at a 90% confidence level (alpha set at 0.10). Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regression-based approaches and a 90% confidence level with 30% precision is to be used for non-regression components.

Enhanced 1. Classic survival analysis (defined bel ow) or other analysis methods that specifically control for right-censored data (those cases of failure that might take place some time after data are collected) must be attempted. The functional form of the model used to estimate EUL (“model functional form”) must be justified an d theoretically supported. Sampling must meet the Enhanced Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. Sample size requirements will be determined through the use of power analysis, results from prior studies on similar programs, and professional judgment. Power analysis used will set power to at least to 0.8 to determine the sample size required at a 90% confidence level (alpha set at 0.10). Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regression-based approaches and a 90% confidence level with 10% precision is to be used for non-regression components.

Basic Rigor Current ex-ante EULs were developed using e ngineering experience and assumptions, past M&V-related evaluation efforts, and past EU L studies. Engineering analysis and M&VEvaluators’ Protocols Effective Useful Life Protocol

CPUC 123 TecMarket Works Team observations suggest that energy efficiency measur es generally last a certain average length of time and then rapidly move out of use as the measur es reach their end of life service. However, these approaches have generally not consider ed retention and behavi oral degradation in establishing the EUL estimates. Similarly, a few measures may continue to last significantly beyond their expected lifetime.

An initial approximation for most types of EUL fo recasts efforts involve some form of a linear estimate, even if the estimate is not linear during the first years of use, or during the later years. This typically involves trying to fit a line to the observed data and use this to predict EUL estimates. Yet, the engineering experience for e fficiency measures suggests that a linear model may not represent the survival function of many energy efficiency measures.

Common alternative models include logistic and exponential models. A variation of the logistic function can be used to describe a pattern of litt le loss in the early years with increasing loss as the measure approaches its expected life, w ith a flattening loss occurring thereafter.

The standard cumulative logis tic probability function is:

Pi = F(Z i ) = F( α + β X i ) = 1/(1 + e -( α + β X i ) )

The logistic model is generally used to measure a nd predict probabilities that an event will occur. This model limits the end points to zero and one. The cumulative logistic, the logistic model,
looks like the curve shown in Figure 9.

Probability

1.0

0.50

0.0
-3 -1 0 1 3
Value of Function Figure 9. Cumulative Logistic Function

The logistic function that best fits the engineering observations described above relies upon a logistic function of time for id entifying the EUL. This is:

F(Z i ) = 1-[1/(1 + e -(t + EUL) b )

With the survival function as in Figure 10.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 124 TecMarket Works Team Figure 10. Logistic Survival Function with EUL=15 and b=0.2

Many energy efficiency retention studies examine energy efficiency equipment as being either
there or not. This dichotomous scale allows the possibility of using cla ssical survival analysis techniques. These outcomes are dichotomous, that is, they either occur or not and can be
measured as zero or one events.

Classic survival anal ysis is specifically design ed to account for the fact th at “failures” might take place some time after when data are measured. In th e statistical lite rature, these cases are said to be “right censored” – their failures are not incl uded in the analysis, because the time period was not long enough to include their eventual failure. As a result, estimating the mean or median when data are right censored can provide a biased estimate. Classic survival analysis techniques have been developed that account for this right censorship in th e data and are able to provide unbiased estimates. Thus, given enough data, many functional forms of survival analysis models (“model functional forms”) can be tested with ava ilable survival analysis statistical programs. The regression techniques availa ble allow considerati on of right censored data and can handle continuous time data, discrete tim e data, and other types of data. 84

84 Multiple statistical modeling packages (SAS ® , Stata ® , SPSS ® , R ® , S+ ® , and others) provide survival analysis

programs. There are several commercia l and graduate textbooks in biostatisti cs that are excellent references for classic survival analysis. One of these used as reference for some of the prior EUL studies in California is the
SAS ® statistical package and the reference Survival Analysis Using the SAS ®

System: A Practical Guide by Dr. Paul D. Allison, SAS ® Institute, 1995. Several model functional forms are available and should be considered for testing. These forms include logistic, logistic with durati on squared (to fit expected pattern of inflection point slowing of retention losses), log normal, expo nential, Weibull, and gamma. A few of many possible references include: The Statistical Analysis of Failure Time Data (Wiley Series in Probability and Statistics) by John D. Kalbfleisch, Ross L. Prentice, Wiley, 2003; Survival Analysis: A Self-Learning Textby David G. Kleinbaum, Mitchel Klein, SpringerVerlag New York, LLC, 2005; Survival Analysis by David Machin, Wiley, 2006; and Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence by Judith D. Singer, John B. Willett, Oxford University Press,
2003. 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.0013579111315171921

Years SurvivedProportion

MedianEvaluators’ Protocols Effective Useful Life Protocol

CPUC 125 TecMarket Works Team Given the advantages of classic survival analysis for producing unbiased estimates of the EUL, both levels of rigor require attempting this method where applicable. The high demand for
failure data and the need for differentiated data over time to best a pproximate a reasonable
functional form have often meant that these mode ls either did not provi de algorithm convergence or produce reasonable results. A ccordingly, other methods that sp ecifically control for right censored data not yet well defined or explained must be attempted in its place, if they are appropriate to the hypothesized hazard functional form.

Where a specific method that controls for the i ssues associated with right censored data cannot be made workable or to produce reasonable results that can be justified, then the evaluator may
resort to other methods to estimate the EUL. Nevertheless, the EU L estimate from other methods must either adjust or , at a minimum, discuss the lik ely potential bias in the EUL estimate given the inability to contro l for right censored data issues.

Sample size requirements will be determined th rough the use of power analysis, results from prior studies on similar programs, and professional judgment. Power analysis will set power to
at least to 0.7 to determine the sample size required at a 90% confidence level (alpha set at 0.10) and then derive the required retention sample size based upon the proporti on of the original pool expected to be found in-place and operable (e x-ante EUL) and the desired effect size as determine by Joint Staff. Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regressionbased approaches and a 90% conf idence level with 30% precision is to be used for nonregression components. Sampling and reporting of sampling and uncerta inty must meet the requirements of the Sampling and Uncertainty Protocol.

Joint Staff may want to consider whether the EUL study sample should be part of an impact evaluation sample, or have separate samples for EUL and impacts studies. Joint Staff may assign separate retention and EUL analysis studies or joint studies as they find appropriate for the timing of the evaluations and efficiencies between studies.

Enhanced Rigor All of the analysis requirements for the Basic Rigor level apply to the Enhanced Rigor level.

Sample size requirements will be determined th rough the use of power analysis, results from prior studies on similar programs, and professional judgment. Power analysis will set power to
at least to 0.8 to determine the sample size required at a 90% confidence level (alpha set at 0.10) and then deriving the required retention sample size based upon the propo rtion of the original pool expected to be found in-place and operable (e x-ante EUL) and the desired effect size as determine by Joint Staff. Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regressionbased approaches and a 90% conf idence level with 10% precision is to be used for nonregression components. Sampling and reporting of sampling and uncerta inty must meet the requirements of the Sampling and Uncertainty Protocol.

Joint Staff may want to consider whether the EUL study sample should be part of an impact evaluation sample, or have separate samples for EUL and impacts studies. Joint Staff mayEvaluators’ Protocols Effective Useful Life Protocol

CPUC 126 TecMarket Works Team assign separate retention and EUL analysis studies or joint studies as they find appropriate for the timing of the evaluations and efficiencies between studies.



### Reporting Requirements

All EUL evaluations are expected to assess and discuss the differences between the (a) ex-ante EUL estimates from DEER or as otherwise appr oved by the Joint Staff a nd (b) the ex-post EUL estimates produced by the EUL evaluation study(ies). To the extent that the data gathered and evaluation analyses conducted can explain the causes for thes e differences, this must be presented and discussed. The evaluation report sh ould note situations in which explanations are not possible due to lack of suff icient data or problems with in terpretation. The EUL evaluation report must also include a recommendation of the EUL for the measure and delivery strategy/application that should be used for future program planning. This recommendation may take the form of recommending the replacement of a DEER EUL or the establishment of a new DEER category.

The EUL studies are also required to report the fi ndings from the most recent degradation studies related to the EUL study measures /applications, so that the EUL report is a depository for all current persistence studies for the study measures/a pplications. This will assist Joint Staff and future evaluators find all relevant persisten ce information for a meas ure/application in one
location.

All reporting under this Protocol should include the following:

1. Cover page containing the measures and delivery strategies or applicat ions included in the retention evaluation, program name s in the portfolios over the last 5 years that include these, program administrators for these programs a nd their program tracking number(s), evaluation contractor, and the date of evaluation plan.

2. Table of Contents.

3. High-level summary overview of the measures and delivery strategies or applications included in the evaluation, the programs affected, and the evaluation efforts.

4. Presentation of the evaluation goals and resear chable issues addressed in the evaluation.

5. Description of how the evalua tion addresses the researchable issues, including a description of the evaluation priorities and the use of assi gned rigor levels to a ddress these priorities.

6. Detailed description of the data collection and an alysis methodology.

7. Current and prior retention results for select ed measures given delivery strategy/application and their precision levels at a 90% confidence interval.

8. Retention, degradation, and EUL findings as is appropriate for the study assigned.

9. A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity, sources of bias, and a short de scription of the approaches planned to reduce threats, bias, and uncertainty.

10. Contact information for the evaluation manager, including, ma il address, telephone numbers,
fax numbers and e-mail address.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 127 TecMarket Works Team In addition to the above, retention studies must also include the following:

• Description of initial and final samp le of measures still surviving.

• Describe any findings on fact ors leading to the higher or lower retention rates.

• Description of removal reasons, their dist ribution, and potential issues created by
different removal reasons and the research design and functional forms that should be investigated in future EUL studies for these measures.

In addition to the overall EUL study reporting requirements, degradation studies must also include the following:

• Describe any findings on factor s leading to the relative de gradation rates and absolute degradation rates, if available.

• Describe the impact of de gradation on energy savings

In addition to the overall EUL study reporting requirements, EUL analysis studies must also include the following:

• Specific equations for survival functions and estimated precision of curve fit.

• Analysis of the ex-post EUL compared to the ex-ante EUL and comparison of to the methods and results from any prior retent ion, degradation, or EU L studies available for that measure (to include comparisons by delivery strategy and application).

• Recommended EUL for the measure and deliv ery strategy/application that should be used for future program planning.



### Study Selection and Timing

A significant amount of funding has been spent in California conducting EUL studies under the prior M&E Protocols. These studies, complete d through 2004, have been reviewed in a recent study (Skumatz et al. 2005). 85 Those measures with useful lives that have been confirmed in the last five years are less likely in need of additional study in the 2006- 2008 study period. Early work by Joint Staff in the EUL planning effort may include an initial study of required EUL sample sizes, review of prior EUL studies and their data collection methods, and an assessment of which measures should be prioritized for whic h types of studies. Important questions for early EUL planning efforts include the following:

• Which measures should obtain earl y panel data collection plans?

• Where can data be collected in the future through periodic rete ntion studies that incorporate estimated removal dates prior to the retention study date (prospective studies that use retrospective methods in their site or telephone surv eying)? For example, it may

85 Revised/Updated EULs based on retention and persistence studies results , Revised Report, Lisa Skumatz, John

Gardner, Skumatz Economic Research Associates, Inc., S uperior, CO, 2005. It should be noted that several ninthyear studies have been or are being conducte d since the summary report was produced.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 128 TecMarket Works Team be possible to conduct studies on data collected from past years on measures promoted in previous programs for conducti ng a retention study, or one can simply conduct retention studies on measures installed in the current programs.

• Which long-life measures should have a retros pective analysis conduc ted for obtaining an
EUL estimate? 86 For example, a measure that wa s included in a previous program currently has an ex-ante EUL of over 7 years, so that obtai ning a reliable ex-post EUL estimate in the next three years from a pros pective approach woul d be highly unlikely. However, an ex-post EUL estimate may be obtainable by conducting retrospective analyses on the prior program sites.

The analysis could also include an assessment of the costs, benefits, and removal risks for tagging 87 and/or using radio frequency identificatio n (RFID) chips by program implementers to simplify future retention studies. This informati on can be used to estimate study costs, timing of study RFPs, and as input into a ny risk analysis used to allo cate resources for EUL studies.

The general rules for how often and what EUL/re tention/degradation evaluations need to be conducted are determined by the Joint Staff. The Joint Staff will decide for each measure, if and when the measure will receive a retention study , a degradation study, and/or an EUL evaluation. They will also decide whether the studies need to be conducted for the measure in a single classification or segregated by de livery strategy or application and whether degradation studies will be overall or technology-based or behavior-b ased. A few examples of when this might occur are: (1) where early removal is a risk due to performance, comfort or aesthetic concerns, (2) when more detailed evaluation information is needed for a measure for future program planning, (3) to support an update of DEER, or (4) a new measure is being piloted or expanded in its use. Joint Staff may assess these situati ons through risk analysis to determine which types
of EUL studies to undertake and when.



### Guidance on Skills Required to Conduct Retention, EUL, and Technical Degradation Evaluations

The senior, advisory and leadership personnel for EUL analysis evaluation efforts need to have the specific skills and experience in regression and statistics provi ng an ability to be able to conduct classic survival analysis and handle EUL functional form and issue analysis, as well as the time budgeted for responsible project ta sk leadership and quality control.

86 The retention/EUL studies conducted under the prior M&E Protocols inclu ded panel studies, one-retention point

site visit studies, site and telephone surveys that had fiel d observation estimate, and consumer estimates on prior removal dates. The one-point retention st udies generally provided only two time points for the retention analysis. This does not allow for information to help determine t he appropriate functional form or for inflection points in
removal rates. Panel studies offer the most reliable in formation but are quite expensive. Periodic site and telephone surveys with consumer estimates of removal date s offer the most cost-effectiv e data if their reliability is
sufficient for accurate EUL estimation . Some measures can have retentio n (in place and operational) reliably measured through telephone surveys (e.g., attic and wall insulation seeking remodeling occurrences) while others may require site visit verification and measurement.

87 A 1994 study by ASW Engineering and KVD Research C onsulting concluded that t agging equipment might not be

viable due to retention issues with the tags.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 129 TecMarket Works Team There are methods that could em ploy significant primary survey or interview data collection. The evaluators using these methods should have sufficient experience implementing surveys, interviews, group interviews, and other types of primary data collection activities as are being
recommended.

Engineering or audit skills are needed for site visit operable testing fo r some measures. The extent of the experience and expertise needed va ries with the sophistic ation of the operational testing. Verification of make, model number, a nd likelihood that the piece of equipment is the original program installed one can be made by auditors or engineers with experience/training with regard to identification of the type of equi pment being examined. Op erable verification that uses commissioning reports, energy management system reports, or similar reporting must be reviewed by engineers with experience/training that allows a quality verification effort.

Telephone surveys and interviews need to be co nducted by experienced pe rsonnel. These studies and their instruments must be designed with personnel with experience in energy efficiency markets, the social sciences, and interview a nd survey instrument design, implementation and analysis.

Technical degradation studies requi re senior experienced engineer s that are quite familiar with the equipment to be studied, its standard c ounter-part, and the components, operations, and effects of changes in the operational conditi ons on the components and function of the equipment. Senior personnel must also have the time budgeted for signi ficant input and review, for responsible project task lead ership and quality control. The degree of involvement needed from senior skilled staff is dependent upon th e skill and experience of the mid-level personnel conducting much of the analysis work.

Methods for conducting behavioral degradati on could be based upon survey and interview analysis methods and/or statis tical/econometric methods. Th e personnel to conduct the work need to have the skills and expe rience for the method being propose d. The evaluators need to be trained and experienced in conduc ting social science research with a strong understanding of assessing and testing causal re lationships between exposure to the program and possible outcomes. An important requirement is for th ese evaluators to have a strong foundation in research design and the ability to create research designs to test for ne t behavioral impacts of energy efficiency programs.Evaluators’ Protocols Effective Useful Life Protocol

CPUC 130 TecMarket Works Team 

### Summary of Protocol-Driven Im pact Evaluation Activities

1 Joint Staff will review retention, EUL, and degradation planning information, perhaps through an initial study of (1) prior retention, EUL, and degradation studies and methods, (2) required retention, EUL, and degradation sample sizes, an d (3) assessment of data collection methods for the prioritized measure and delivery strategy/ application needs. Along with any risk analysis information, Joint Staff will identify which measur es by delivery strategy /application will receive which type of retention, EUL, and degradation evaluation, when, and at what rigor level.

Joint Staff will determine any special needs on a case-by-case basis that will be required for particular retention, EUL, and degradation evaluat ions. Joint Staff will develop preliminary RFPs for groups of studies based upon timing of the neede d data collection or analysis, similar sectors or issues to be addressed, and requiring similar skill sets. CPUC-ED will issue RFPs for retention, EUL, and degradation evaluations, select evaluation contractors, and establ ish scope(s) of work.

2 Evaluators will develop a research design and samp ling plan to meet Protocol requirements as designated by the Joint Staff rigor level assignments. This includes meeting requirements from the Sampling and Uncertainty Protocol, as are applic able given Effective Useful Life Evaluation Protocol requirements. Research design and samp ling must be designed to meet any of the Joint Staff requirements for additional analyses to include but not limited to areas designated of specific concern by the Joint Staff. Evaluators will devel op and submit an Evaluation Plan to Joint Staff, and the plan will be revised as necessary to have an approved Evaluation Plan that meets the
Effective Useful Life Evaluation Protocol.

3 All retention, EUL, and degradation study evaluat ion teams (including panel data collection teams) will make sure their teams are ap propriately staffed, in order to meet the skills required for the research design, sampling, and selected retent ion, EUL, and degradation evaluation method, uncertainty analysis, and reporting being planned and conducted.

4 All retention, EUL, and degradation study evaluat ions will be planned, conducted, and analyzed to minimize potential bias in the estimates (showing the methods for doing th is), and evaluators will report all analyses of potential bias issues as de scribed in the Sampling and Uncertainty Protocol.

5 All retention, EUL, and degradation evaluations will be conducted according to the Evaluation Plan and appropriate Protocols.

6 Evaluators will develop the draft ev aluation report in accordance to guidance provided by the Joint Staff and reporting requirements in this Protocol.

7 Final evaluation report will be developed in accord ance to guidance provided by the Joint Staff, and
then submitted to Joint Staff.

8 Once accepted by Joint Staff, abstracts w ill be developed, and a report will be posted on the CALMAC web site following the CALMAC posting instructions.

Note: the steps included in this evaluation summar y table must comply with all the requirements
within the Effective Useful Life Evaluation Protocol in order to be in compliance.Evaluators’ Protocols Process Evaluation

CPUC 131 TecMarket Works Team 

## Process Evaluation Protocol

Introduction

The Process evaluation is not a required evaluatio n activity in California. It is, however, often critical to the successful implementation of co st-effective and cost-efficient energy efficiency programs. Process evaluations identify improveme nts or modifications to a group of programs, individual programs or program co mponents, that directly or indi rectly acquire or help acquire, energy savings in the short-term (resource acqui sition programs) or the longer-term (education, information, advertising, promotion and market effects or market transformation efforts).

The primary purpose of the process evaluation is an in-depth investigation and assessment of one or more program-related characteristics in order to provide speci fic and highly detailed recommendations for program changes. Typicall y, recommendations are de signed to affect one or more areas of the program’s operational pr actices. Process evalua tions are a significant undertaking designed to produce improved a nd more cost-effective programs.

This Protocol identifies how pr ocess evaluations for Californi a energy efficiency programs, products or services placed in to the market during and afte r the 2006 program year will be
conducted. The Evaluation Framework is incorporated into this Protocol as a key guidance document for conducting process evaluations. Before applying it, users of this Protocol should have a working knowledge of Chapter 8, “Process Evaluations,” of the Evaluation Framework. Key references to the process evalua tion literature are also found in the Evaluation Framework

. 88

In addition, all users of the Process Evaluation Protocol should be familiar with the sampling guidance provided in the Sampling Protocol. Wh ile the use of the Sampling Protocol is not required for planning and conducting process ev aluations, it provides guidance for process evaluation sampling and sample selection criteria . The Reporting Protocol contained in this document contains the process evaluation reporti ng requirements and is a part of the Process Evaluation Protocol. The Reportin g Protocol helps assure that the reports generated from the process evaluation provide comparable resu lts and recommendations on which program management can act, and, at the same time, meet the CPUC’s reporting requirements.

A process evaluation is defined as a systematic assessment of an energy efficiency program, product or service, or a component of an energy efficiency program, product or service, for the purposes of identifying and recommending improvements that can be made to the program to increase the its efficiency or effectiveness in acquiring energy resource s while maintaining high levels of participant satisfac tion and documenting program oper ations at the time of the examination. The primary goal of the process evaluation is the development that improve program efficiency or effectiveness that when im plemented can be expected, in some direct or indirect way, to improve the cost effectiven ess of the program. Th is definition updates the definition provided in the Evaluation Framework

. 89

88 TecMarket Works, page 205.

89 Ibid, page 207.Evaluators’ Protocols Process Evaluation

CPUC 132 TecMarket Works Team Process Evaluation Responsibilities While the CPUC- ED must approve all process ev aluation contractors, th e process evaluations themselves are to be planned, budgeted, designed, implemented and report ed under the direction of the Administrators, following th e guidance laid out in this Prot ocol. The Administrators are responsible for the process eval uations for their statewide prog rams, the Administrator-specific programs implemented within thei r services areas, the programs conducted by third parties under contract to the Administrators and the programs or services that are pr ocured via a bidding or other contractual processes. The Administrato rs are responsible for developing the Annual
Process Evaluation Plan and obt aining related comments and recommendations from the Joint
Staff.

Objectives of the Process Evaluation The process evaluation’s primary objective is to help program de signers and managers structure their programs to achieve cost-effective savings while maintaining high levels of customer satisfaction. The process evalua tion helps accomplish this goal by providing recommendations for changing the program’s structure, manage ment, administration, design, delivery, operations or targets. Consequently, Administrators ofte n want early process evaluation feedback. The process evaluation also provides ongoing feedback to the Administrators that allow them to make timely program changes or to follow the pr ogress of the study or to review early findings. Where appropriate, the process eval uation should test for the use of best practices and determine if specific best practices should be incorporated. It is expected that process evaluations will be needed both in the early stages of the program design and deployment efforts to provide timely feedback to the IOUs on them, and over the lif e of the program as issues are identified.



### Audience and Responsible Actors

This Protocol is to be used by Administrators and their evaluation contra ctors to conduct process evaluations and by the CPUC-ED to provide ong oing guidance and oversight to the process
evaluation activities. The Protocol allows considerable flexibility and judgment by the Administrators to determine when a process ev aluation is needed and the issues on which the process evaluation should focus.

The Administrators are responsible for program design, operation and goal attainment for the programs and services funded through their implem entation and contracting efforts. They must structure their process evaluation efforts to support these responsibilities.

Other stakeholders should be familiar with the intent and scope of the Protocol to obtain an adequate understanding of the purpose and scope of the process evaluation efforts and how these
studies are to be conducted.



### Overview of the Protocol

As mentioned previously, this Pr otocol is specifically designed to work in conjunction with the
Evaluation Framework . The chapter on process evaluation in the Evaluation Framework is a key advisory component of this Protocol. 90

90 Ibid, page 205.Evaluators’ Protocols Process Evaluation

CPUC 133 TecMarket Works Team This Protocol provides guidance to Administrators on the criteria us ed to determine if and when to conduct a process evaluation and on the re searchable issues targeted by the study. 91 However, Administrators are free to identify additional or different decision criteria.

Because a process evaluation is not a CPUC-requi red activity, the Administrators will determine if one is to be conducted, when it will occur a nd the investigative areas on which the evaluation shall focus. As a result, there is no waiver process associated with the Process Evaluation Protocol. However, the Protoc ols suggest several ke y investigative areas on which the process
evaluation can focus.

Finally, the Protocol presents the types of investigative tools and a pproaches that can be used to conduct the process ev aluation efforts.

As discussed earlier, the Proce ss Evaluation Protocol is linked with other of the Protocols, including the Sampling and Uncertainty Protocol and the Reporting Protocol. These two latter
Protocols should be considered sub-components of the Process Evaluation Protocol. However, the Sampling and Uncertainty Protocol is adviso ry, while the Reporting Pr otocol is a required component of the Process Evaluation Protocol.

Because there can be overlap between the informa tion collected in the process, market effects and impact evaluations and associated M&V effo rts, the process evaluation efforts should be
structured to coordinate with ot her evaluation efforts to the exte nt practical. This may require cross-organizational coordination. Such coordi nation minimizes customer contact, maximizes data collection efforts and improves evaluation effi ciency. This does not mean that these studies must be inter-linked or consolidated, but it do es mean that there will be times when the information collected in one study will be valuable to other studies and times when studies will
benefit from a coordinated effort.

In these cases, there will need to be cl ose coordination between the CPUC-ED, the
Administrators and the evaluation contractors for the related eval uation efforts. This Protocol does not specify how this coordination should be structured or conducted, but does identify the need for it. However, the Protocol also rec ognizes that the sk ill sets needed for conducting a process evaluation are often different than those needed for an impact or market effects study and these differences may limit the ex tent of the coordination efforts.



### Process Evaluation Planning

There are several key issues that should be cons idered when planning a pr ocess evaluation. It is anticipated that most programs will have at leas t one in-depth comprehensive process evaluation within each program funding cycle (e.g., 2006-2 008), but a program may have more or less studies depending on the issues th at the IOUs need to research , the timing of the information needed and the importance of those i ssues within the program cycle.

The process evaluation decision road map in the Evaluation Framework 92 identifies several operational conditions that can be considered by the IOUs for targeting a process evaluation.

91 Ibid, page 220.Evaluators’ Protocols Process Evaluation

CPUC 134 TecMarket Works Team The Administrators should assess these criteria annually and other criteria as appropriate to determine if a process evaluati on is needed. This annual assessment should be conducted by the IOUs administering the program and reported in an Annual Proce ss Evaluation Plan delivered to the CPUC-ED no later than the first of December, before the start of each program year or the month before the start of a new program when th e program does not start at the beginning of a

program year.

This annual planning requirement applies to all programs being administered or funded via the Public Goods Charge (PGC) or Procurement pr ogram funds for the upcoming program year. While the detailed process planning efforts w ill not be fully known at the beginning of the program cycle, the plan should present the stru cture and operations of the detailed planning efforts, identify the key decision criteria to be us ed to determine if and when a process evaluation will be planned and launched, and present the pro cess evaluation efforts planned at that time. The annual plans developed following the first ye ar would then present the process evaluations planned and launched during the previous year and present th e known process evaluation needs for the upcoming year.



### Annual Process Evaluation Planning Meeting

For each year of the program cycle, the Administ rators shall hold at least one process evaluation planning meeting to review and discuss their process evaluation plans and obtain Joint Staff input to help guide planning efforts. The meeti ng shall be held between July and November of the year preceding the evaluation period. The Join t Staff will be notified of these meetings at least two weeks in advance of the meeting. Th e meeting dates, times and locations will be
coordinated with the Joint Staff to maximize the attendance potentia l of the interested parties. During the meeting, the Administrators will pres ent their process evaluation plans and solicit comments. Within two weeks following the meeting, but no later than December 22, the Administrators will finalize their process evaluation plans and submit a final plan to the Joint Staff for its review. The plan does not have to be approved by the CPUC-ED.

The decision for determining if a process evalua tion is needed for all programs rests with the Administrators. However, the evaluation pla nning process must be conducted annually, incorporate the use of the planning meetings disc ussed above and lead to the submission of an
Annual Portfolio Process Evaluation Pl an to the Joint Staff for review.

Program managers may want the process evaluati on to supplement the program’s quality control or quality assurance components, to confirm the installation pr actice and/or to conduct program reviews and develop recommendations for improv ements. Administrators should consider the
different functions and benefits of the process eval uation under different potential program grouping scenarios and weigh the associated pros and cons wh en structuring their process evaluation plans.

The timing of the process evaluatio n is an important component of the planning process. In some cases, Administrators will want to launch their process evaluations early to help ensure that programs are well designed, are achieving savi ngs shortly after laun ch and are providing

92 Ibid, page 222.Evaluators’ Protocols Process Evaluation

CPUC 135 TecMarket Works Team effective services. In other cases, Administ rators may give the programs time to become established in the market and adopt more rou tine operational approaches before launching the process evaluation. In still other cases, Administra tors may wish to establish an ongoing process evaluation effort so that the pr ogram is periodically evaluated ove r its three-year cycle. This third condition may be more applicable to ne w programs or programs being provided by a new vendor than to more established programs that have demonstrated thei r cost-effec tiveness and operational efficiencies in earlier studies.

The Annual Process Evaluation Plan submitted to the Joint Staff should indicate the level of resources and budgets devoted to the process evaluation efforts.

The goal of this Protocol is not to require unneces sary process evaluation efforts, but to provide
tools to Administrators for the consideration of key decision criteria typi cally associated with process evaluation efforts and related implemen tation. Administrators should only plan and launch process evaluations when they are expected to serve as an effective program management tool. The Annual Process Evaluation Plan helps to provide the Joint Staff with a minimum level of assurance that program change s are being effectively assessed and managed and that process evaluations are considered when they can be effectively employed. The decision to conduct a process evaluation is ultimately the Administrators’. These deci sions should be conveyed to the
CPUC-Ed in the Annual Pr ocess Evaluation Plan.



### Recommendations for Change

The primary purpose of process evaluation is to develop recommendations for program design or operation changes that can be expected to cost -effectively improve the issues, conditions or problems being investigated. The primary deliver able of all process evaluations is a process evaluation report that presents the study findi ngs and the associated recommendations for program changes (see Reporting Protocol).



### Key Issues and Information Covered

Administrators and their need for operational in formation to improve programs guide the process
evaluation effort. This necessari ly covers a very wide range of investigative issues that the process evaluation can address. The process ev aluation may take on the challenge of evaluating most, if not all, aspects associated with the desi gn or operations of a program in order to improve the energy resources acquired (dir ectly or indirectly) by that pr ogram. The process evaluation plan can also address issues applicable to th e programs under review over a single year or over multiple years and can examine a wide range of issues, including:

Program Design

• Program design, design characteri stics and design process;

• Program mission, vision and goa l setting and its process;

• Assessment or development of program and market operations theories and supportive logic models, theory assumptions an d key theory relations hips - especially their causal relationships; and

• Use of new practices or best practices.Evaluators’ Protocols Process Evaluation

CPUC 136 TecMarket Works Team Program Administration

• Program oversight and improvement process;

• Program staffing allocation and requirements;

• Management and staff skill and training needs;

• Program information and information support systems; and

• Reporting and the relationship between eff ective tracking and management, including both operational and financial management.

Program Implementation and Delivery

• Description and assessment of the program implementation and delivery process;

• Quality control methods and operational issues;

• Program management and manage ment’s operational practices;

• Program delivery systems, components and implementation practices;

• Program targeting, marketing and outreach efforts;

• Program goal attainment and goal-associat ed implementation processes and results;

• Program timing, timelines and time-sensitive accomplishments; and

• Quality control procedures and processes.

Market Response

• Customer interaction and sa tisfaction (both overall satisf action and satisfaction with key program components and including sa tisfaction with key customer-productprovider relationships and support services);

• Customer or participant energy efficiency or load reduction need s and the ability of the program to provide for those needs;

• Market allies interaction and satisfaction;

• Low participation rates or associated energy savings;

• Market allies needs and the ability of the program to provide for those needs;

• Reasons for overly high free-riders or too lo w a level of market effects, free-drivers or spillover; and

• Intended or unanticipated market effects.



### Process Evaluation Efforts

One of the primary purposes of the Process Evalua tion Protocol is to en sure an appropriately broad consideration of potential process ev aluation issues for each program within the Administrator portfolio and a fram ework that provides critical ev aluation thinking to produce the best overall process evaluation efforts for the portfolio.Evaluators’ Protocols Process Evaluation

CPUC 137 TecMarket Works Team 

### Program-Specific Proces s Evaluation Plans

In addition to the Administrator’s Annual Portfolio Process Evaluation Plan, each process evaluation should have a program-specific or pr ogram-group-specific detailed process evaluation plan to guide the evaluation efforts. These deta iled plans should include the process evaluation approach, identification of progr am-specific or program group-specific focus of the evaluation efforts, detailed researchable issues to be addr essed, activity timing issues and the resources to be used. However, it is the Admini strator’s responsibility to specif ically determine the content and focus of such plans. The detailed program-sp ecific or program-group-sp ecific detailed process evaluation plans do not need to be submitted to the Joint Staff for review.

The Process Evaluation Protocol is designed to balance a llowing the CPUC-ED and other
stakeholders a level of assurance that there is a minimum set of standards for process evaluations across the portfolios and allowing the necessa ry flexibility and control for program administration and process evaluation management.



### Data Collection and Assessment Efforts

Process evaluation efforts can include a wide ra nge of data collection and assessment efforts,
such as:

• Interviews and surveys with Administrato rs, designers, managers and implementation staff (including contractors, s ub-contractors and field staff);

• Interviews and surveys with trade allies, contractors, suppliers, manufacturers and other market actors and stakeholders;

• Interviews and surveys with par ticipants and non-participants;

• Interviews and surveys with technology users;

• Interviews and surveys with key policy make rs and public goods charge stakeholders;

• Observations of operations and field effort s, including field tests and investigative efforts;

• Unannounced participation in the progra m to test operations and operational practices, processes and interactions;

• Operational observations and field-testi ng, including process related measurement and verification efforts. These can be announced or unannounced;

• Workflow, production and productivity measurements;

• Reviews, assessments and testing of record s, databases, program-related materials and tools used;

• Collection and analysis of relevant data or databases from third-party sources (e.g., equipment vendors, trade allies and stakeh olders and market data suppliers); and

• Focus groups with participants , non-participants, trade al lies and other key market actors associated with the program or the market in which the program operates.Evaluators’ Protocols Process Evaluation

CPUC 138 TecMarket Works Team This list of activities is not meant to be exhaustive, but illustrative. Administrators are free to specify other data collection and assessment e fforts beyond those listed above. However, in selecting the evaluation approaches to be used, a key consideration is the level of reliability of the study approach and the accur acy of the study findings. All studies are expected to be structured in a way that provides reliable findings on which accurate and reliable recommendations can be developed.



### Conducting Investigative Efforts

This section of the Protocol provides guidance on conducting specific investigative efforts typically associated with the process evaluation.

Interviews Professional process evaluation in terviewers should conduct proce ss evaluation inte rviews. The
Evaluation Framework provides guidance on the type of training and experience needed by process evaluation staff that conduct interviews. 93 In-depth interviews can be conducted inperson (off-site or on-site) or by telephone. In-pers on interviews enable the interviewer to gain a deeper understanding of the expe rience of the intervie wee and can lead to more reliable and more comprehensive information gathering than phone interviews. P hone interviews do not allow for the observation of key body signals that serve as clues for the probing process. However, both approaches are equally valid if the questions are well designed and the interviewer is skilled in interviewing techniques. If in-person interviews are not possible given the nature of the study or the location of the inte rviewee, then telephone in terviews can be used. Regardless of the type, interviews should be sc heduled in advance and should last an hour or more. Detailed comprehensive process evaluatio n interviews may last several hours. E-mail interviews are rarely used unless the evaluation professional can easily guide the interview and
move it in directions that need additional information or inves tigative probes. In addition, as technologies that can be used to support the in terview effort evolve, there may be additional approaches that can be considered or used, such as web-conferencing or web-interviews. However, in assessing the applicability of th ese technologies the primary focus should be on allowing the interviewer to be able to manage a nd focus the interview as it proceeds so that indepth probes and ancillary follow-up questions can be placed into the interview at the time they
are needed.

Group Interviews The group interview may be used to obtain info rmation from a group of individuals typically having one or more similar characteristics. Th e focus group, one of the more familiar types of group interviewing techniques, is used to focus on the response to a limited set of issues – such as in product development research. The use of other types of group interviews can be appropriate for evaluation in a limited nu mber of circumstances, for example:

• Obtaining feedback from a group of installers or outreach coordinators who can “focus” on the specific issues of their job or their experience with end-users; and

• Qualitatively investigating issues that will be further explored in quantitative surveys.

Experienced professional evaluation expert s must conduct group interviews. All group interviews should be guided by sk illed moderators and documented in a way that allows for a

93 Ibid, page 206.Evaluators’ Protocols Process Evaluation

CPUC 139 TecMarket Works Team review of the moderator’s instru ctions to the attendees, the m oderator’s approach to managing the group and the moderator’s instructions, questi ons and involvement, as well as a detailed understanding of comments prov ided by all attendees.

A summary report of each group interview containing the above- listed information should be included in the evaluation report. The group interview report shoul d also include a professional interpretation of the results discussing how they will or can be used and a discussion of how the group results will be confirmed or test ed quantitatively, if required.

Group interviews are a reliable data collection approach but they do not provide results that can be generalized to a population except when the participants in the group interview are a statistically representative sample of that population. Thus, a focus group (because the participants do not typically consti tute a statistically representative sample of the population) is not an acceptable means to qua ntitatively assess programs, but can, in some circumstances, provide supportive information that can be used in a process evaluation finding. For this reason, they are included as an approved data collection effort within this Protocol when accompanied by other assessment approaches that can quantitatively test their results.

Surveys Survey efforts that are used to support proces s evaluations are typica lly conducted via telephone interviews. However, there will be occasions wh en other approaches are preferred. For instance, when there are large numbers of participants or non-participants or when the inquiry will benefit from the respondent seeing an illustration, survey techniques could include mail, e-mail or Webbased approaches and other types of surveys. Si milarly, small targeted surveys with trade allies or program participants who have provided e- mail addresses for this purpose may be most efficiently conducted using an e-mail/Internet co mbined survey. There is a great variety of survey techniques and they should be selected according to specific requirements of the data
collection effort.

In all cases, professional process evaluation survey designers should construct and test the survey questionnaires to avoid unnecessary bias in question topic or st ructure, or in the responses received. The questions in the surveys should follow construction practices that result in objectively worded questions with provisions for recording all e xpected responses. Questions
should be structured so that th ey are single-subject, focused questions. Questions that are typically referred as “double- barreled” questions (containing more than one subject-verb relationship) should be avoided as they bias the information collected.

Most important in implementing any type of surv ey is to follow the principles of good survey design and implementation such as those developed by Don Dillman. 94 Whether the survey is implemented using the telephone, mail, e-mail or Internet there are a specific methods that should be applied to ensure valid and reliable results. These include repeated contacts with the sample as well as carefully structured invitations to participate and questions.

94 Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York, NY: John Wiley & Sons.

Dillman, Don A. 1978. Mail and telephone surveys: The Total Design Method . New York, NY: John Wiley & Sons.Evaluators’ Protocols Process Evaluation

CPUC 140 TecMarket Works Team Observations and Field Testing Field-testing and observations shou ld be done in a way that allows the observation or testing of the program as it would be opera ting in the absence of the eval uation professional. Observers are to instruct program staff that they are to conduct themselves exactly as they would if the observer were not present. The observing evaluati on professional is not to engage in activities that act to change the way the activity would ha ve occurred if the evaluation professional were not there. All key observations and measuremen ts should be documented at the time of the observation or testing.

Unannounced Participation In some cases, it may be appropriate for the evaluati on contractor to enroll in the program to test the program’s operations and delivery aspects. Wh en this is designed as part of the evaluation plan, program management is not to be informed of who will be particip ating, how they will be participating or when that partic ipation will occur. Participation is to be unannounced and field
observations and measurements will be conducted without the knowledge of the program staff to the extent practical. This approach can be used for a wide range of programs in which unannounced participation by an evaluation professi onal can allow the evalua tion expert to view the program from the perspectiv e of a typical participant.



### Independence

The organization conducting the pr ocess evaluation should be inde pendent of the organizations involved in the program design, management and implementation efforts. The evaluation should be conducted as an “arms-length ” assessment, such that the pr ocess evaluation professionals have no financial stake in the programs or program components bei ng evaluated beyond the contracted evaluation efforts. Similarly, pr ocess evaluation profe ssionals should have no financial or financially related in terest in the study results or from efforts resulting from the implementation of evaluation recommendations.



### Selection of Evaluation Contractors

Administrators are charged with the responsibility to plan, contra ct, manage and administratively oversee the implementation of the process evaluati on efforts consistent with this Protocol.
Administrators should focus their co ntractor selection efforts, so that only profe ssional, skilled process evaluation contractors are solicited for conduc ting the process evaluations. The CPUCED must approve the selection of the eval uation contractors to conduct the studies. The contractor approval process will be structured by the CPUC-ED consistent with the ALJ’s decision. This process will be developed outside of this Protocol. Approval by the CPUC-ED will be based on the qualif ications of the firms or individuals considered for conducting the
studies.



### Skills Required for Conducting Process Evaluations

The investigative processes associated with designing, managing and conducting process evaluations focus on a wide range of research able issues. These issues can range from evaluating the ability of a program’s data manage ment system to support the informational needs of the program to assessing if the program is well-designed, managed, targeted, marketed and operated. As a result, the sk ills needed to conduct proces s evaluations are varied.Evaluators’ Protocols Process Evaluation

CPUC 141 TecMarket Works Team Evaluations that focus on the de sign and operation of program info rmation systems, for example,
need evaluators that understand how informa tion management and information availability influence a program’s management, operations, productivity and results. However, the evaluators should also be skilled at designing, developing and implementing information systems in order to recommend changes to improve the pr ogram’s ability to cost -effectively achieve its goals. Process evaluators who assess program sa tisfaction levels need to have the skills to identify and analyze different program characteris tics that influence satisfaction and be able to identify those characteristics that can be change d to improve satisfaction scores. In the process evaluation, measuring satisfaction is not enough, the study should assess the reasons for the satisfaction scores and identify how to impr ove these scores without harming the costeffectiveness of the program.

Similarly, evaluators who focus on assessing program targeting, marketing and promotional operations need to have skills necessary to assess informati on flow, content and presentation
effects as well as the skills associated with understanding how markets and market segments operate and can be influenced through differe nt outreach and promotional efforts. These examples demonstrate the need to match the skil ls of the process evalua tor with the research goals of the specific process evaluation.

It is equally important that pr ocess evaluation managers be trai ned and/or experienced with the tools used in the process evalua tion. For example, if a telephone survey is needed, evaluators need to be knowledgeable and experienced in the field of survey research and instrument design. If focus groups are needed, eval uators should be knowledgeable a nd experienced in the field of focus group design and operation, as well as asse ssing and applying the results from the focus

group.

Because of the diversity of researchable issues associated with conducting process evaluations and the diversity of skills needed to address these issues, it is diffi cult to define a specific set of skills needed to conduct these evaluations. Instead, this Protocol recognizes that a diverse set of program assessment and information analysis ski lls are needed across th e various investigative issues on which these evaluations typically focus. However, in general, the process evaluator should have the following knowledge and skills:

• Expert knowledge of a wide range of energy efficiency programs and a strong understanding of their operational designs, management practices and program goals;

• Expert knowledge of different process evaluation data collection methods and approaches, and a working knowledge of th e process evaluation literature and how evaluation approaches have been applied in the energy efficiency program field;

• Strong analysis capabilities and an expert unde rstanding of cause-and-effect relationships that impact the ability of en ergy efficiency programs to cost-effectively accomplish their goals, including experience in program theory and l ogic model construction and

assessment;

• Strong understanding of statis tical analysis approaches and analytical procedures appropriate for the process ev aluation research goals;Evaluators’ Protocols Process Evaluation

CPUC 142 TecMarket Works Team • Strong understanding of sampling methods and approaches and the ability to identify potential biases in a sampling approach and to develop control strategies for mitigating levels of bias and improving the relia bility of evaluation results; and

• High level of past experience in conducting process evaluations of energy efficiency programs and in reporting the results of these studies.Evaluators’ Protocols Market Effects

CPUC 143 TecMarket Works Team 

## Market Effects Evaluation Protocol

Introduction

The Market Effects Protocol is designed to meas ure net market effects at a market level when
one or more of the Protocol-cov ered energy efficiency funded progr am efforts target a market.
Net market effects are those effects that are induced by Protocol-covered energy efficiency programs and are net of market activities i nduced by non-energy efficiency programs including normal market changes.

The application of the Market Effects Protocol sh ould result in an estimate of the energy (kWh), peak (kW) or therm impacts associated with th e net market effects resulting from Protocolcovered energy efficiency program interventions. These net energy market effects are referred to

in A Framework for Planning and Assessing Publicly Funded Energy Efficiency ( 2001 Framework Study) as “ultimate market effects” or “ul timate indicators” because they are the
desired indicator of whether net energy efficiency changes are occurring in the market. 95 The Market Effects Protocol is design ed, therefore, to facilitate not just the estimate of net market effects but also, and primarily, the estimate of net energy market effects. Th at is, a market effects study both quantifies the changes occurring in the market caused by the energy efficiency programs and provides an estimate of the energy impacts associated with them.

The Market Effects Protocol doe s not apply to the measuremen t of individual program-level market effects or direct program savings typi cally used for program-level cost-effectiveness
assessments and refinement decision s. Rather the focus of the mark et effects evaluation is at a market level in which may diffe rent energy efficiency programs can operate. Yet, the Protocol applies to program-induced market changes that could be missed or double counted if measured program by program. As a result, the use of th e Market Effects Protoc ol should focus on the effects of groups of programs within a market over multiple program cycles.



### Overview of the Market Effects Protocol

This Protocol applies when net market effects are to be estimated at a market rather than program
level. Market effects are defined in the Evaluation Framework as “[a] change in the structure or functioning of a market or the beha vior of participants in a market that result from one or more program efforts. Typically these efforts are de signed to increase the adoption of energy efficient products, services, or practice and are cau sally related to market interventions.” 96 This definition, however, was created within the context of guidance for conducting program evaluation of a market transformation style progr am. A market transfor mation program is one that is specifically designed and fielded for th e purpose of changing the way a market operates so that energy savings are achieved at a market leve l. That is, these type s of programs are designed
to focus at the market level. A more effectiv e definition for the Market Effects Protocol for assessing the market effects from multiple progra ms that may or may not be designed to change market operations is that in A Scoping Study on Energy-Efficiency Market Transformation by

95 Frederick D. Sebold et al. A Framework for Planning and Assessing Publicly Funded Energy Efficiency . (Pacific Gas and Electric Company. 2001): 6-4.

96 TecMarket Works, 429.Evaluators’ Protocols Market Effects

CPUC 144 TecMarket Works Team California Utility DSM Programs (the Scoping Study) : “A change in the structure of a market or the behavior of participants in a market that is reflective of an increase in the adoption of energyefficient products, services, or practices and is causally related to market intervention(s).” 97 This definition stresses the market rather than the pr ogram nature of market effects, and is the working definition for this Protocol.

The Evaluation Framework states that “there are no universa lly accepted definitions within the energy efficiency industry pertaining to what constitutes a program’s market.” 98 A review of various dictionaries demonstrates that it has multiple meanings. “Market” as used in this Protocol refers to the commercial activity (manufacturing, distributing, buying and selling) associated with products and services that aff ect energy usage. The specific market focus of each evaluation should be defined as an early ac tivity in scoping each market effects evaluation.
The Evaluation Framework provides guidance for defining a market and where multiple programs are operating in the same market, again, the primary focus of this Protocol. 99

Market effects include both short- term and long-term effects. The long-term effects are the most difficult to capture at a program level because they broadly affect a market not just the specific participants in a program or in a grouping of programs. This Protocol targets those long-term
effects.

A market-level evaluation effort is recommende d when there are multiple statewide or local
interventions in a market such as those of Ca lifornia’s energy efficiency programs and where other efforts are also acting to change that mark et. Other efforts can be associated with the normal operations of the market or when othe r non-California energy e fficiency efforts are changing markets, such as with the national E NERGY S TAR ® program, manufacturer promotions
and retail sales efforts. A market level effort is also appropriate when a single large and particularly effective program is expected to ha ve broad and long-term market effects in a single
market.

Figure 11 shows the relationship between program -induced market effects, program market spillover and normal energy efficiency trends in the market. Effects driven by interventions by other organization, as well as the market itself, are all assumed to be within the normal energy efficiency trends of the market.

97 Joe Eto, Ralph Prahl, and Jeff Schlegel. A Scoping Study on Energy-Efficiency Market Transformation by California

Utility DSM Programs . (Berkeley, CA: Ernest Orlando Lawrence Berkeley National Laboratory, 1996). LBNL-39059 UC-1322, 9.

98 TecMarket Works, 250.

99 Ibid, 250-251.Evaluators’ Protocols Market Effects

CPUC 145 TecMarket Works Team Figure 11. Sources of Energy Efficiency Changes in the Market

There are two types of market effects discusse d in the energy efficiency industry. There are those that are occurring now as a result of how programs are changing markets. And there are those that are forecasted to occur later (after the program has been discontinued) due to the changes established or put into motion by the program. The Protocol recognizes that the methodologies to estimate each of these types of market effects can differ and that potential issues of bias that must be identified, mitigat ed and minimized are also different. The Market Effect Protocol is designed to measure only the current market effects and not those forecasted to occur at some future point.

A great deal of effort has been expended over the past 10-15 years to estimate market effects, yet
most of these efforts did not estimate net energy market effects, but concentrated on measurement of indicators such as awareness, sa les and changes in practices by market actors. Evaluations estimating net market effects with ener gy estimates, the focus of this Protocol, are at an early stage of development. A variety of studies have been c onducted, but only a limited number at the highest levels of rigor. However, this is a critica lly important field of research
since the market effects of en ergy savings caused by California’s energy efficiency programs are likely to be substantial once documented. Given th e early stage of develo pment of methods, it is important that this Protocol encourage the cont inued advancement of the field and not prescribe or limit methodological approaches. Resource Acquisition

Program-Induced Energy Savings Information, Education, Marketing,Promotion ProgramInduced Energy Savings Program-Induced Market Spillove r Energy Savings Non-Program Induced Energy SavingsEvaluators’ Protocols Market Effects

CPUC 146 TecMarket Works Team Key to a successful market effects evaluati on will be the initial scoping study. The scoping study will define the market to be studied, develop a market theory to test in the analysis, assess data availability for the market effects study, develop a methodology for additional data
collection and recommend an anal ysis approach. For programs th at are specifically designed to change the way a market operates, the program th eory should also be considered in developing the initial scoping study. However, for standa rd programs that are not designed to change market operations, the program theory is not a significant consideration in the development of the scoping study.

Because market effects evaluation is still evolvi ng there are a limited, but clearly defined, set of
activities that should be considered. Market effects eval uations should be developed using experimental or quasi-experimental designs whenev er possible and the approach should be peer reviewed prior to implementing the study to ensure th at it will provide valid and reliable results. Triangulation of data and analys is approaches is preferred wh en possible and teaming with industry organizations and professi onals can be benefici al. The studies shoul d also take into account regional differences within the market be ing studied and will at times need to move beyond California boundaries to the regional or national level to co llect data. Finally, allocation to utility service territory will be a challenge and dependent on data availability, but should be an important consideration in the scoping study.



### The Market Effects Protoc ol and Other Protocols

Often the individual Protocols overlap and are supported by other Protocol s. There are three primary output Protocols: Impact, Market Eff ects and Process Evaluation. There are three
Protocols that can be called on to support or provide additio nal requirements for all the Protocols. These are the Sampling and Uncer tainty, Measurement & Verification (M&V) and Reporting Protocols. The guidance provided by the Impact Evaluation, Market Effects and Process Evaluation Reporting Protocol s applies to all types of effi ciency program evaluations in California. The Sampling and Uncertainty Protocol provides further deli neation of sampling and uncertainty assessment and reporting requirements. The M&V Protocol is similarly a reference and supporting Protocol for the Impact Evaluatio n Protocol and can be used to inform and provide input for a Process Evaluation on issues relating to measure installation and performance.

The Impact Evaluation and M&V Protocols are supporting Protocols to the Market Effects Protocol as related to estimating net energy market effects. At the same time while the Impact
Evaluation Protocol addresses net energy effects through esti mation of free-ridership andparticipant spillover, it does not include measurement of non-participant spillover. Nonparticipant spillover specifically refers to changes in the market that result from program influences and this is appropriately estimat ed as part of the net market effects.

There is another important integration aspect for the individual Prot ocols. A complete measurement of program impacts involves comb ining the results from the market effects evaluations and the program impact and indirect impact studies. The market effects net of program impacts would generally represent the program market level impacts and nonparticipant spillover. Yet, differences in methodologies and the multitude of programEvaluators’ Protocols Market Effects

CPUC 147 TecMarket Works Team evaluations require careful thought and analysis to ensure that when the impacts are combined adjustments are made to ensure no double countin g occurs. At the same time, there are bound to be some programs for which program impacts have been evaluated but their particular market has not (e.g., in cases of niche markets or uniqu e program elements). This would need to be added to reach an estimate of the overall portf olio expected energy a nd demand savings to be reported to system planners. Familiarity with the Impact Evaluation, M&V, Sampling and Reporting Protocols is recommended to conduc t a Protocol-compliant Market Effects
Evaluation.

Finally, the Process Evaluation Pr otocol outlines types of data collection methodologies that
should also be considered when conducting primary data collection fo r estimating market effects.
The Evaluation Framework provides further detail on data co llection methodologies and issues to consider when examining markets, familiarity with which is recommended for designing or conducting a Market Effects Evaluation. 100



### Key Market Effects Inputs and Outputs

Inputs for a market effects evaluation incl ude but are not limited to the following:

• Names and contact information for program staff for the programs identified as targeting the defined market;

• Names and contact information for mid-stream or upstream market actors identified by the Administrator as opera ting in the defined market;

• Evaluations and market research conducte d by the utilities for the defined market during the previous five years;

• Market and program theory documents de veloped by the Administrators for the programs identified as targeting the defined market;

• Names and contact information for key info rmants consulted during the development of the programs identified as ta rgeting the defined market; and

• National data on the market from sources such as the US Census Bureau, Energy Information Administration, Consortium for Energy Efficiency, Department of Energy and Environmental Protection Agency.

The Market Effects Protocol will generate an estimate of net energy market effects in kWh and kW in markets for electricity-usi ng equipment and therms in markets for gas-using equipment. These metrics at times will require that the mark et effects estimates link to the results of the M&V or Impact Evaluation Protoc ol to provide estimates of energy impacts based on the market
effects measured. The market to which these estimates apply will be defined in terms of location, the utilities involved, the equipment and s ector, and the program years of interest.

This approach requires that the evaluator estim ate what changes would have occurred in the market without the energy efficiency effort s provided by the programs. Because of the uncertainty in the evaluation process, the estimat e will likely be a range of probable effects, rather than a point estimate (e.g., confidence inte rvals). These studies should always include a
clear statement of the uncertain ty around the range estimate. The Reporting Protocol discusses this issue in greater detail.

100 TecMarket Works, Chapters 4, 8, 9, 10, 12, and 13.Evaluators’ Protocols Market Effects

CPUC 148 TecMarket Works Team The outputs will be used to inform the program pl anning process for the next program cycle, to structure the program planning efforts to ma ximize net market effects of statewide and Administrator's portfolios, while also maximi zing net program-induced energy impacts. The
market effects evaluations shoul d be structured to provide ma rket effects impact information prior to June before the end of the program cycl e so that the results can be considered in the planning efforts for the next program cycle.

The market effects study results can also be used in comparison with the results of the program evaluation efforts to identify net market en ergy impacts that are be yond the direct programinduced effects. In addition, results will serv e as an estimation range check for the savings projected from the program evaluations. In al l cases the total market effects should be a summation of the direct program-induced effects, the normal market changes to become more energy efficient and the non-partic ipant spillover effects. In no cases should the programinduced impacts be greater than the net market effects identified in the market effects
evaluations. Given the different methodologies employed, the analys is of the different sets of results, the methodologies used, a nd their relative advantages a nd weaknesses must be carefully
conducted in order to obtain the most reliable estimates of overall impacts from the energy efficiency programs.



### Audience and Responsible Actors

The audience and responsible actors for this Protocol include the following:

• CPUC-ED and CEC will use the Protocol to determine when a market effects study is appropriate and to guide the research approach;

• The Evaluation Contractor Team will use the Protocol to ensu re that their market effects evaluation plan and its conduct address key requirements for a market effects study;

• Administrators will use the Protocol to understand the market interactions that are occurring as a result of efforts within a given market and, in part, to determ ine when interventions
should be modified to achieve co ntinued efficiency gains; and

• Program Implementers will use the results to assess the reach and success of their program efforts and in part to determine when interven tions should be modified to achieve continued efficiency gains.



### Steps in Conducting Market Effects Evaluations

The following five primary activities should be c onducted in any market effects evaluation. This
section describes in some detail what is entailed in each step.

1. Conduct a scoping study to determine optimum data collection and analysis approach for the evaluation;

2. Select a contractor and deve lop a detailed evaluation plan;

3. Collect baseline and longitudinal indicators;

4. Analyze market effects; andEvaluators’ Protocols Market Effects

CPUC 149 TecMarket Works Team 5. Produce the Market Effects Report.



### Scoping Study

The appropriate approach for a market effects study cannot be readily determined without a scoping study to define the market to be studied, de velop a market theory to test in the analysis, assess data availability for the market effects st udy, specify a model of ma rket change, develop a methodology for data collection and r ecommend an analysis approach.

Scoping studies will require differe nt levels of effort depending on the complexity of the market of interest and the number and types of program interventions in that market. Scoping studies
can also be used to determine which markets show promise for reliable and valid market effects
evaluation.

The evaluation contractor will be expected to review past studie s conducted of the market by the California utilities and other energy organizations. It will not be enough to simply look at the programs being offered during the program year s of interest. A thorough review of the CALMAC database for applicable studies and report s as well as interviews with contacts at each utility and program managers will be important. Access to market assessment studies conducted by the utilities will be important to provide a sound understanding of the market conditions prior to program implementation and to support an under standing of market prog ress and/or contribute to the preponderance of evidence for causality/attr ibution. The evaluation contractor should also review the potential valu e of national and regional data sets including data colle cted by the US Census Bureau and the Energy Information Ad ministration and organizations such as the Consortium for Energy Efficiency. No potentia l source should be i gnored and it is very important that the utilities be cooperative in this effort.

The key activities of the scoping study include de fining the scope range and limits of the subject market. The Joint Staff will make an initial determination of the definition of the market by location, utilities involv ed, sector, and likely equipment and behavior to be included. The scoping study contractor, however, will assess this initial definition and ultimately determine the definition that will yield the mo st reliable and meaningful resu lts about net market effects of
interest. This is a critical fi rst step and requires a full understa nding of how the interventions of interest were designed to operate and how the ma rket in which they were launched is perceived
to operate.

This process will provide the framework for the development of the market theory and conducting a logic analysis of the interventions, wh ich will be used to guide the market effects evaluation. The logic model and ma rket theory will then be used to develop a list of indicators for tracking market effects. These indicators co uld be model specifications, a detailed list of
indicators to be tracked thr ough baseline and longitudinal data collection efforts or both. The end result of the scoping study is an evaluation plan that detail s the strategy for the market
effects evaluation. Table 14 displays the Pr otocol for scoping studies for market effects
evaluations.Evaluators’ Protocols Market Effects

CPUC 150 TecMarket Works Team Table 14. Required Protocols for Market Effects Evaluation Scoping Studies

Level of Rigor Scoping Study Requirements

Basic Define the market by its location, the utilities involved, the equipment, behaviors, sector and the program years of interest. Develop market theory. Identify available secondary data and potential sources for primary data. Outline data collection and analysis approaches

Enhanced Define the market by its location, the utilities involved, the equipment, behaviors, sector and the program years of interest. Develop market theory and logic model. Detail indicato rs. Identify available secondary data and primary data that can be used to track changes in indicators. Outline data collection approach. Recommend hy potheses to test in the market effects study. Recommend the analysis approach most likely to be
effective.

Market Theory and Logic Models The assessment, refinement and/or development of a market theory with logic models are key
activities of the scoping study. The 2001 Framework Study 101 and the Evaluation Framework 102
both address the value and proce ss of developing a program or ma rket theory. The evaluation contractor will need to articulate a market theory in order to proceed with baseline measurement for market effects evaluation. At a minimum, this market theory shall de scribe how the market operates and articulate market a ssumptions and associated research questions. This must be done at a level of detail sufficient to develop data collection instruments for baseline measurement. If the assessment includes programs that are design ed specifically to change the way a market operates the program theory should also be consis tent with and embedded in the theory of how the market operates. 103

Market-level evaluations seek to document the ch anges in adoption behavi or that cause changes in energy savings. 104 It is important, therefore, to clearly articulate the assumed changes in the market, so they can be measured for the market e ffects study. If this is done properly the market effects evaluation can document changes in adop tion, efficiency and provide an estimate of savings. This process also facilitates model specification.

A higher level of rigor is achieved when the market theory can be describe d in a narrative and/or a graphic logic model. A narrative or gra phic logic model permits a greater depth of understanding of the indicators driv ing anticipated market outcomes. It can also help to identify
the various sources of influence on market effects outside of the program efforts. The simplest approach to a logic diagram is to view the boxes as potential measurem ent indicators and the arrows as a hint to questions regarding causa l links, program implemen tation theory, where to examine underlying behavioral change assumpti ons, and areas for researchable questions.

101 Sebold et al., pages 4-2 – 4-6.

102 TecMarket Works, pages 30-37.

103 Nicholas P. Hall & John Reed. “Merging Program-Theory and Market-Theory in the Evaluation Planning

Process.” Proceedings of the International E nergy Program Evaluation Conference (2001).

104 Sebold et al., page 6-9, Figure 6-2.Evaluators’ Protocols Market Effects

CPUC 151 TecMarket Works Team Interviews or workshops should be used to de velop the program theory. These should include program managers who understand the program purpose and can articulate the assumptions about how the program will change the market. 105

The key issues that should emerge from the work shops are program activities, identification of key market actors, assumed market barriers, expected outputs, outcomes and likely indicators of change, alternative hypotheses for change, external influe nces and causal links within anticipated
timelines for achievement. Table 15 displays the Protocol for market theory and logic models.

An important distinction for the program theory /logic model development for a study under this Protocol is that the theory/logic model needs to be for a set of programs and capture both how the individual programs aim/hypothesize to affect the market as well as how they interact and support one another for market changes. The interaction, the degree to which and how to measure their ability to mutually support one anot her and how they interactively operate within a market are important analysis points and co mplications for a market level evaluation. Articulating the many assumptions this presents a nd then examining which are the most critical and how to test them are key to the degree to which the final market effects study will be comprehensive and defensible. A detailed under standing of how the market operates and how the various program interventions change or suppor t that is a fundamental starting point for being able to attribute market changes to a group of programs, i.e., market effects.

Table 15. Required Protocol for Market Theory and Logic Models

Level of Rigor Market Theory and Logic Model Requirements

Basic Identification of assumptions about ant icipated changes in the market and associated research questions. Market theory should include market operations and conditions, external influences, and as sumptions about changes in the market (which could include market operational theory, market structure and function studies, and product and communication flows). Develop program theory and logic models across programs in that market. Analyze across both of these to examine program interventions, external influences and associated research questions. Theories and logic models should be generated through interviews with program staff and a sample of market actors.

Enhanced Articulate market theory and, if reas onable, develop graphical model of market theory. Market theory should include market operations and conditions, and changes occurring in the market (could in clude market operational theory, market structure and function studies, and product and communication flows). Develop multiple program theory and logic model s for those programs intervening in the market. Integrate the market theory and program theory/logic models to examine external and programmatic influences, as sumptions about changes in the market and associated research questions. Theories and logic models should be generated through interviews or workshops with program staff from each of the programs and a sample of a wide variety of market actors. Use a literature review and other studies of these markets and iteration with program staff to ensure thoroughness in measuring the critical parameters for both market development
from external influences and market effects.

105 TecMarket Works, 30-38, 45-49, and 245-254. These sections of the Evaluation Framework (and the references provided for both chapters) will help in understandi ng the goals of program theory and logic models.Evaluators’ Protocols Market Effects

CPUC 152 TecMarket Works Team Determination of Indicators The scoping study will determine what indicators s hould be used to assess market effects. The process emerges from the analysis of the market theory and program logic models but must also include an assessment of available data and prim ary data collection options. The use of smaller experimental designs imbedded into the program operations or quasi-experimental design is encouraged as a way to improve rigor without si gnificantly increasing da ta collection costs over what is already required within this Protocol.

The market effects study should estimate what changes would have occurred in the market without program efforts. The indicators are used to draw conclusions about these changes. The
focus should be on ultimate market indicators (the indicato rs of energy changes in the market). In developing the indicators there will be trade-offs that result in a level of rigor for the estimates. The scoping study should, therefore, address the level of rigor for the evaluation. The key considerations for the rigor of market effects estimates are the accuracy of the estimates of energy impacts and the accuracy of the attributio n of market effects. The limitations of the
market effects evaluation shoul d be clearly articulated relativ e to these two issues and the scoping study should detail how the r ecommended approach addresses each.

The scoping study should also state the market assu mptions and associated research questions to be addressed by the market effects study. At a mark et level, there are a variety of interventions that might occur and a variety of approaches that might be used to track and measure change in
those interventions and their effect on the market. Table 16 indicates th e general types of
interventions that can influen ce change in a market and, therefore, suggests the types of indicators that might be track ed in market effects studies. The scoping study should clearly
describe the relevance of the indicators to the market theory and how these indicators can be interpreted to indicate market effects.Evaluators’ Protocols Market Effects

CPUC 153 TecMarket Works Team Table 16. Types of Market Interventions and Associated Possible Indicators

Intervention Type Ultimate Market Indicator Other Indicators

Advertising/Outreach/ Branding Value of energy savings from
sales and/or market share changes for targeted efficient

measures Awareness, source of awareness, intention to purchase, amount of exposure to
intervention

Upstream Vendor
Incentives Value of energy savings from
sales and/or market share changes for targeted efficient

measures Stocking practices, product availability, price

Trade Ally Training Value of energy savings from
sales and/or market share changes for targeted efficient
measures or market share of efficient buildings Surveys of practi ces, willingness to implement changes in installation or purchase, recommendation practices

End-user Training Value of energy savings from
sales and/or market share changes for targeted efficient
measures or market share of efficient O&M practice Surveys of practi ces, willingness to implement changes in operation or purchase

Downstream Incentives Value of energy savings from sales and/or market share changes for targeted efficient

measures Non-participant awareness of the program, non-participant awareness of program participant experience

Detailed Market Effects Evaluation Plan Once the scoping study is completed, the CPUC-ED w ill contract with an ev aluator to implement
the market effects evaluation. The first task fo r the evaluator will be to develop a detailed evaluation plan to implement the re commendations in the scoping study.

The Evaluation Framework discusses the need and value of an evaluation plan in some detail. 106
It is important that the Market Effects Evalua tion Plan clearly documen ts the results of the scoping study and details the approach that shou ld be taken to conduct the evaluation. The scoping study defines the market, details the market theory and logic and identifies the indicators to be used for tracking market effects. The evaluation plan captures th ese findings and details the process by which the indicator data will be co llected or generated and describes the analysis approach to be used to estimate gross and net ma rket effects and the resulting net energy market
effects.

The evaluation plan should include a detailed description of the data collection approach including how indicators will be measured, popul ation estimates and sampling targets. There
should be a clear discussion of the analysis strategies and model sp ecification if appropriate, and how the analysis plan will be developed. Ther e should also be a sche dule of milestones and deliverables and clear delineation of what information and data sources will be required from the
utilities and other California entities.

106 Ibid, 56-58.Evaluators’ Protocols Market Effects

CPUC 154 TecMarket Works Team 

### Collection of Baseline and Lo ngitudinal Indicators

Baseline studies are addressed in the 2001 Framework Study 107 and the Evaluation
Framework. 108 There are a variety of indicators that might be chosen to track market progress
and thus determine whether market effects have occurred. Primary and secondary data are used for indicator studies. Primary data collection mu st be done carefully and samples used should be determined using the Sampling and Uncertainty Pr otocol. In those cases where secondary data exist, care should also be taken to understand the manner in which the data were collected to be certain of its appropriateness for market effect s estimation. Where available, secondary data often provide a source for estimating market shar e for both efficient and non-efficient equipment
sold in a market and can be the most effective way to obtain data for non-program affected areas.

Primary data collection involves co llecting data (such as sales data ) directly from actors in the market of interest. These types of studies vary in complexity, but at a minimum, the sample must be representative of the population of ma rket actors. When su rveying retailers and distributors, effort also needs to be made to adjust for double counting and to weight sales reports to account for total share of the market (see below). It is also possible to establish baselines for behaviors or energy using equipm ent by surveying end-users or market actors targeted for training or information services. 109 These types of studies all require that the questions asked enable the anal yst to differentiate between sa les of efficient and standard equipment or behaviors that improve efficiency over standard practice. Alternatives that provide potentially less biased or more readily accessible or controllable data should be examined. For example, changes in saturation over time might be a worthwhile alternative to sales data in some

cases.

A higher level of rigor for primary data collectio n is achieved by carefully designed studies. For example, the California market share tracking st udies for residential equipment are carefully designed to have a panel of par ticipating retailers and distributors whose data can be weighted appropriately to estimate market share. To im plement such a data collection effort requires establishing long-term relationships with retailers and distributo rs to provide sales data on a regular basis. Such studies requi re that the sample be carefully selected so that reported sales can be properly weighted to account for differentia l roles in the market by different retailers and
distributors. Double coun ting also has to be avoided since dist ributors supply retailers. Highest levels of rigor are achieved by using multiple da ta sources to triangulate on the estimate of market share caused by the program efforts. Table 17 shows the Protocol for indicator studies.

107 Sebold et al., pages 5-2 and 7-1 to 7-36.

108 TecMarket Works, 254-262.

109 Appliance sales have been tracked biennially in Wisconsin since 1993 (Energy Center of Wisconsin, 2004) by

asking end-users about their purchases. The 2003 Ap pliance Sales Tracking Study is available at http://www.ecw.org . The Northwest Energy Efficiency Alliance tracks changes in behaviors for many of their programs by surveying representative samples of end-users and trade allies.Evaluators’ Protocols Market Effects

CPUC 155 TecMarket Works Team Table 17. Required Protocol for Market Effects Evaluation Indicator Studies

Level of Rigor Indicator Study Requirements

Basic Select appropriate market actor group for each indicator, survey representative samples of market actors able to report on each indicator from market experience. A baseline study must be conducted as early as possible. On-going tracking provides the basis for comparisons.

Enhanced Select appropriate market actor group for each indicator. Conduct longitudinal study of representativ e samples of market actors able to report on each indicator from market experience. Samples weighted to represent known parameters in the population of interest. A baseline study must be conducted as early as possible, on-going tracking provides the basis for comparisons.



### Analysis of Market Effects

The analysis of market effects has several components. Fi rst it should be determined if the indicators demonstrate any change in the market at all. This would be the estimation of gross market changes. Causality/attribution (which resu lts in net market effects), sustainability and net energy impacts should then be estimated.

Gross Market Effects Once the indicators have been collected for time one and time two, the analyst must determine the change in indicators across th e time periods. For indicators such as market share and sales, it is reasonable to make direct comparisons. A vari ety of studies have shown that market share can be tracked directly overtime and these co mparisons are fairly straightforward.

For other indicators such as awar eness and knowledge, it is possibl e to make direct comparisons of indicators across time periods, bu t it is common that the directi on and intensity of change in indicators will vary. One method that has been found to be effective in this type of situation is a
binomial test. 110

Estimating Causal Attribution Causality should be examined to estimate net mark et effects. The goal of the activity is to estimate the proportion of market changes that can be attributed to program interventions using PGC and procurement funds, as versus those na turally occurring in the market or from interventions using non-PGC and non-procuremen t funds to arrive at market effects.

There are two primary approaches for estimating causal attribution, one uses a preponderance of evidence approach and the other uses a modeling approach. The ultimate goal for assessment of causal attribution is to avoid retrospective analys is in which contacts are asked to judge what efforts had effects on the market. Retrospective a pproaches have great potential for bias because
contacts are themselves influenced and cannot maintain objective perspectives.

110 Richard F Spellman, Bruce Johnson, Lori Megdal and Shel Feldman, “Measuring Market Transformation Progress

& the Binomial Test: Recent Expe rience at Boston Gas Company.” Proceedings, ACEEE 2000 Summer Study on Energy Efficiency in Buildings . (Washington, DC: American Council for an Energy-Efficient Economy, 2000).Evaluators’ Protocols Market Effects

CPUC 156 TecMarket Works Team Preponderance of Evidence Approach for Attribution
In some cases, it is best to use a “preponderance of evidence” approach to assess the attribution of market effects. In this approach the analys t relies on triangulation from multiple data sources to draw conclusions about the presence and attrib ution of market effects. This approach is accomplished by interviewing and surveying knowle dgeable market actors. Program staff, utility staff and trade allies provi de useful information for unders tanding the context of sales and counts of behavior. Over time, these views provi de much of the information needed to draw
conclusions about attribution and su stainability. Systematic sampli ng is very important to ensure
that bias is minimized.

A minimum level of rigor requires that samples of trade allies be included in the sampling plan, as they provide a less biased perspective due to their market-centric rather than energy efficiency-centric view. Rigor improves with mo re comprehensive samples of trade allies and other market actors. A variety of approaches can be used including choice and ranking surveys, focus groups, Delphi surveys and others. 111

The preponderance of evidence appro ach is inherently a qualitative analysis process in which the analyst uses multiple points of view to estimat e the proportion of market effects that can be attributed to the program interventions. As noted above, the estimate will likely be a range, due in part to the qualitative nature of the analysis, but also to the di fficulty in fully specifying all the factors that influence markets. The highest le vel of rigor relies on informants from multiple perspectives enabling the analyst to triangulate on the market effects. Table 18 shows the Protocol for the preponderance of eviden ce approach to attribution estimation.

Table 18. Required Protocol for Preponderance of Evidence Approach to Causal
Attribution Estimation

Level of Rigor Preponderance of Evidence Approach Requirements

Basic A representative sample of market actors surveyed or interviewed to provide self-reports on perceived changes in the market, attribution and the sustainability of those changes.

Enhanced Quasi-experimental or experimental design with comparison groups using a representative sample of mark et actors surveyed or interviewed to provide self-reports on perceived changes in the market, attribution and the sustainability of those changes.

Net Market Effects Modeling for Causation
The alternative to a preponderance of evidence appr oach is to use net effects modeling to control for non-PGC and non-procurement funded activities . In this approach the analyst uses multivariate models or simultaneous modeling system s to estimate net market effects. A variety
of methods can be used. Some of thes e are discussed in Chapter 7 of the 2001 Framework Study,
in which it is suggested that the use of dynamic baseline s in which a forecast of market changes are made in time one, using time one data, and then in time two the forecast is tested against the

111 Sebold et al., 6-23 - 6-25 and 7-5 - 7-7.Evaluators’ Protocols Market Effects

CPUC 157 TecMarket Works Team actual conditions of time two. 112 Additional methods are being e xplored and hold a great deal of promise for clarifying the extent of market effects caused by ener gy efficiency program efforts.

A modeling approach permits the an alyst to specify a model of th e program theory and to test that model with data gathered in time one and time two. This is a growing area of investigation with a limited number of studies having been completed as of 2005. In constructing such a model, it is important that the model specifications reflect the complexity of the market. This is the greatest challenge for this approach. It is likely that such an approach will require multiple equations to model the various activities that occur in a market and the various points of intervention that energy efficien cy programs exert on a market.

Given the early stage of development for this ty pe of approach, it is no t possible to determine levels of rigor. Advancements on these methods are being developed as it appears this approach could offer a greater level of rigor, quan tification and testing than prior methods.

Estimating Sustainability Sustainability is the degree to which one can expect the market changes to last into the future. Sustainability is not readily estimated using ne t effects modeling therefor e the preponderance of evidence approach is the most frequently used for estimating sustainability. As with attribution a minimum level of rigor requires th at samples of trade allies be included, as they provide a less biased perspective due to their market-centric ra ther than energy efficiency-centric view. Rigor improves with more comprehensive samples of trad e allies and other market actors. A variety of approaches can be used including choice a nd ranking surveys, focus groups and Delphi

surveys. 113 Another valid approach to estimate sustai nability is identifying changes in market structure and operations, and how the changed mark et contains mechanisms to sustain them. This could include examining profitability analys es for important support businesses or business operations and how these are maintained without continued program intervention.

As noted previously, the preponderance of evidence approach is inherently a qualitative analysis process in which the analyst uses multiple points of view to estimate whether the market effects attributed to the program interven tions can be expected to continue into the future. The highest level of rigor relies on informants and analyses from multiple perspectives enabling the analyst to triangulate on sustainability. A market with mu ltiple support areas for continued sustainability will have a greater likelihood of having the changed market operation be sustainable.

The result of the estimation of sustainability is a statement on the likelihood of the market effects continuing without the energy effi ciency program intervention or with reduced interventions.
Given California’s current interest in market eff ects that have recently occurred rather than those forecasted to occur (the focus of market effects es timation efforts in several other states), there is significantly less need for measures of sustainabi lity. This issue, howev er, is a critical one
whenever forecasts of market effect s are the dominant evaluation concern.

Market Effects Metrics and Energy Savings When the net effects modeling or preponderance of evidence approach is used to estimate net market effects, the analysis will result in an estimate of market share for the sales or counts of

112 Sebold et al., pages 7-1 to 7-36.

113 Ibid, 6-23 to 6-25 and 7-5 to 7-7.Evaluators’ Protocols Market Effects

CPUC 158 TecMarket Works Team behavior, or other indicator(s) attr ibuted to the program. The net market effects must then be linked to an estimate of energy savings. The sa les and counts of behavi or or other indicators
used to estimate market effects are linked to th e energy savings for those measures or behaviors estimated through the M&V or Impact Evaluation Pr otocols, or in DEER. Savings estimates are directly applied to net changes in sales, count s of behavior and the like by multiplying the savings term by the associated amount of energy us age for the equipment or behavior of interest.

In some cases when net market effects modeling is used, it is anticipated that energy will be a term in the equation. Therefore, rather than linking the estimated net market effect to a savings estimate, the analyst will use the energy term as the dependent variable th at is being modeled. The indicators will be the independent vari ables specified to explain the energy term.



### Reporting

The evaluation report should also ad dress the level of rigor for th e study. The key considerations for the rigor of market effects estimates are the accuracy of th e energy impact estimates and the accuracy of the attributi on of market effects. The limitations of the market effects evaluation should be clearly articulated relative to these two issues.

In addition to estimating net energy market effect s, the market effects evaluation report should clearly state the market assumptions and associat ed research questions addressed by the market effects study. The market eff ects evaluation report should clearl y articulate the logic of the approach - whether using a prepon derance of evidence approach to justify net market effects or a regression-based modeling approach. 114 Both approaches should build on the market theory as a hypothesis that was developed ea rlier in the scoping study.

Market effects evaluations will result in a re port documenting the evaluation and its findings. The Reporting Protocol describes the content of the market effe cts evaluation report. The key aspects of that report include the following:

• Documentation of the market theory and the program theory/logic model as developed in the scoping study including an assessment of the initial market theory and program theory based on the results of the evaluation, and recommendations for a revised market theory/progr am theory, if needed;

• Documentation of the data collection and anal ysis process used for the market change indicators, whether the data used were primary data or secondary data. What indicators were used, how the data were assembled, collected and analyzed and the
results of the various indicators studies;

• Documentation of the estimation of gross mark et effects that result from an analysis of the indicators, regression mode ling or triangulation of the two;

• Documentation of the process used and results obtained for estimating causal attribution and sustainability and the resulting estimate of net market effects; and

114 Modeling the market processes and change processes, some sequential and some simultaneous, is encouraged

as an enhancement for a regression-based modeling approach over a single-equation model. Any use of a singleequation model must justify the model s pecification and its ability to capture t he critical evaluation elements seen in the market theory and program theory/logic models.Evaluators’ Protocols Market Effects

CPUC 159 TecMarket Works Team • Documentation of the process used and re sults obtained when estimating net energy market effects. What energy data were used and how they were linked to the estimate
of net market effects.



### Guidance on Skills Required to Co nduct Market Effects Evaluations

This Protocol suggests that there are two prim ary strategies for conducting net market energy effects evaluations, each of which can be considered rigorous when well-executed.

A preponderance of evidence approach, in which th e analyst relies on triangulation from multiple data sources, is used to draw conclusions about the presence of market ef fects. While secondary
data can be used in this a pproach, significant primary data collection is expected. The preponderance of evidence approach, therefore, requires skills in designing and implementing survey and interview instruments to collect indi cators that correspond to the theory and reflect how the market is thought to operate. The evaluators should have sufficient experience to implement surveys, interviews, group interviews and other types of primary data collection activities. Since energy savings are drawn from impact evalua tion results, the firms conducting
market effects evaluations should have vast experience in energy e fficiency markets, the social sciences. and interview and survey instru ment design, implementation and analysis.

The net market effects modeling approach, in wh ich the analyst uses multivariate models to estimate net market effects, can use primary or secondary data, although the use of secondary data has been most common. This type of a pproach is largely depe ndent on professional evaluators experienced in regression-based a nd multivariate modeling. The evaluator must be able to specify a model of the market during th e scoping study and then populate the model with secondary and primary data. One approach is to develop a forecast of the market in time one and then test it in time two. Another approach is to take a retrospective approach using secondary data over a multi-year period. Other approaches are still emerging. The major limitation of the net market effect modeling appro ach is the availability of suff icient data to meet the model requirements. Modeling systems and/or specif ication that can mirror market operations and program theory interventions have still to be developed. Modeling the market processes and change processes, some of which are sequential while some are simultaneous, is an enhancement to a regression-based modeling approach and is encouraged over a sing le-equation model Any use of a single-equation model must justify the model specific ation and its ability to capture the
critical evaluation elements seen in the market theory and program theory/logic models. Thus, a scoping study should be used to determine if such an approach is warranted and can be expected
to be successful.



### Considerations for Conducting a Market Effects Evaluation

The key consideration for conducting a market ef fects evaluation is determining whether market level effects are expected. As noted previously, programs that operate within a market have ripple effects on other programs operating in that market. Obvious examples are how the United States Department of Energy and the Environmental Protection Agency’s E NERGY S TAR efforts interact with the California energy efficiency program activities to en courage the adoption of energy-efficient appliances in the residential sector. All states are showing increased adoption of

E NERGY S TAR appliances, but the question remains as to what part of this market change isEvaluators’ Protocols Market Effects

CPUC 160 TecMarket Works Team induced by the California programs. Another exam ple concerns new residential and commercial construction activities that are affected by the implementation of California’s building codes (Title 24) and by California’s pr ogram activities designed to cha nge construction behaviors and code-covered practices.

A market effects evaluation is appropriate when ne t market effects are used to justify a program or group of programs, 115 or when net market effects may be of interest to the Joint Staff for a set of programs operating in the same market. At the same time, the Impact Evaluation Protocol does not measure non-participant spillover due to th e assessment that these are best measured by
market effects evaluations. This means that th e full effect of Californi a’s investments in energy efficiency programs may not be obtained through th e sum of individual program evaluations but, instead, through an analytic deriva tion from the program-level evaluations and the market effects evaluations. (See the discussion in the Impact Evaluation Protocol on interaction with the Market Effects Protocol.) Market effects evaluati ons are needed, then, to have the information to derive the full impacts of the Ca lifornia efforts and investment.

As noted previously, determining the “market” is an important step in the scoping study. The
Joint Staff will make recommendations for markets in which they expect market effects to be measured. Markets of possible interest to the Joint Staff include, but are not limited to:

• Residential appliance;

• Mass marketing campaigns;

• Residential construction;

• Nonresidential construction;

• Agricultural services;

• Commercial lighting;

• Residential lighting;

• Education (general public and ta rgeted groups, e.g., contractors);

• Training programs; and

• Technical assistance programs.



### Summary of Protocol-Driven Ma rket Effects Evaluation Activities

1 Joint staff identifies the markets or market sect ors (and the associated se t of programs) that will
receive a market effects evaluation and identif ies the potential approach and rigor level for the scoping study.

2 Joint staff identifies market- or market sector-specific study needs that will be assessed (including program-specific or program group spec ific study needs) from the evaluation. CPUCED issues request for proposals for market effects scoping study, selects the scoping study contractor and establishes a scope(s) of work.

115 TecMarket Works, 247, Figure 10.1.Evaluators’ Protocols Market Effects

CPUC 161 TecMarket Works Team 3 Evaluation contractor develops scoping study. A scoping study will more finely define the market boundaries for the study, including its location, t he utilities involved, the equipment or behaviors to be assessed and the program-influenced years of interest. The scoping study will develop a market theory and a logic model; identify the market change indicators to track; and the available primary and secondary dat a sources. The study will al so identify the hypotheses to test and the data collection approach, and provide a recommended analysis approach and model specification (if appropriate).

4 A market change theory and logic model (MCT/LM) should be developed to identify assumed
direction of effects and indicators for measur ing effects. The market theory should include market operations and conditions, and changes occu rring in the market (could include a market operations theory, market st ructure and function scenarios, and product and communication flows) The theory and logic model should be generated through interviews or workshops with program staff from each of the programs that are expected to influence the market being assessed and a sample of a wide variety of mark et actors and should incorporate a literature
review.

5 Joint staff reviews the scoping study and deter mines how to proceed with the Market Effects Evaluation. CPUC-ED issues r equest for proposals for evaluat ion contractors, selects the contractor, establishes a final scope(s) of work and negotiates the contract.

6 All market effects evaluation t eams must be staffed to meet the skills required for the research design, sampling, appropriate and selected ev aluation method, uncertainty analysis and reporting requirements.

7 A research design and sampling plan should be developed to meet Protocol requirements at the market level to meet the Joint Staff assigned study rigor level. This includes meeting requirements from the Sampling and Uncertaint y Protocol and the Reporting Protocol, as applicable. The evaluation contractor will develop an Evaluation Plan, submit it to the CPUC-ED and revise as necessary.

8 Indicators studies conducted as part of t he Market Effects Evaluation should be based on the results of the scoping study, addr ess the appropriate market actor group(s) for each indicator.

9 All Market Effects Evaluations must meet t he requirements of the Sampling and Uncertainty Protocol. The 90/10 level of precision is a mini mum precision target for the most important data collection efforts on its most important variables. Which data collection efforts and variables are considered to be the most important will be deter mined in close collaboration with the CPUC-ED

10 The gross market effects and the estimate of energy savings associated with the market effects should be estimated. Estimation of gross market effects can be as simple as comparing
indicators between time one and time two and t hen multiplying the energy value derived in an M&V supported impact assessment or from DE ER, or using a CPUC-ED-approved net energy
effects model.

11 Attribution or causality should be address ed to estimate net effects using either a preponderance of evidence approach or a net effects modeling approach.

a. For a preponderance of evidence approach a determination of attribution should use quasi-experimental or experimental design with comparison groups using a representative sample of market actors. This may include interviews to provide selfreports on perceived changes in the marke t, attribution and the sustainability of those changes as well as direct observation or ot her data to support changes resulting from the program.Evaluators’ Protocols Market Effects

CPUC 162 TecMarket Works Team b. For a net effects modeling approach to estima te causality, the model specifications must reflect the complexity of the market. It is likely that such an approach will require multiple equations to model the various activi ties that occur in a market and the various points of intervention that energy efficiency programs exert on a market.

12 Sustainability should be addressed using a preponderance of evidence approach.

13 Develop draft evaluation report to include meet ing all requirements in the Reporting Protocol and incorporating the program’s performance metrics.

14 Develop final evaluation report in accordanc e to guidance provided by Joint Staff.

15 Submit final evaluation report to the CPUC-ED.

16 Once the report is accepted by the CPUC-ED, develop abstracts and post them and the report on CALMAC Web site following the CALMAC posting instructions

Note: The steps included in this evaluation summary table must comply with all the requirements
of the Market Effects Evaluation Protocol.Evaluators’ Protocols Sampling and Uncertainty

CPUC 163 TecMarket Works Team 

## Sampling and Uncertainty Protocol

Introduction

There are some important sim ilarities between the pre-1998 pr otocols and the 2006 Protocols related to impact and M&V studies. Both sets of protocols focus on obtaining reliable estimates of energy and demand impacts. Reliable esti mates are interpreted as estimates that are reasonably accurate and precise, th at is, they contain a minimal am ount of error from a variety ofsources such as sampling error, measurement error, and model misspecification error. The pre1998 protocols concern the same issues listed in the Evaluation Framework : 116

• Non-response and other forms of selection bias;

• Measurement error;

• Erroneous specification of the statistical model;

• Choosing an inappropriate baseline;

• Self-selection of program participants;

• Misinterpretation of association as causal effects;

• Construct validity;

• Statistical validity;

• Internal validity; and

• External validity.

However, the two protocols also have differences , the two primary of whic h relate to the number of study types and the degree of precision require d for energy-use estimates. The 2006 Protocols
must address an additional set of studies that include proce ss evaluations, indirect impact evaluations for education, traini ng and advertising programs, and market effects evaluations. The reliability of information produced by thes e studies is equally im portant and must be
addressed in the 2006 Protocols.

The pre-1998 protocols require 90/10 precision for estimates of annual energy use while the 2006 Protocols set precision targets 117 whenever possible for a variety of parameters including savings. 118 Precision targets are set rather than required since, as discussed in the Evaluation
Framework and its cited study of this issue by Sonnenblick and Eto, bias could be much more

116 See Evaluation Framework , 292-294 for examples and definitions of the terms listed here, along with citations to

reference documents.

117 A precision target is a goal establis hed at the beginning of an evaluation based in large part on initial estimates of

uncertainty. If an evaluator fails to actually achieve the ta rgeted level of precision, there will be no penalties since the assumptions underlying the sample sizes proposed in each evaluation plan will have been clearly presented and carefully documented . A failure to meet the precision target for a given program will only require an adjustment of the input assumptions prio r to the next evaluation cycle and, if necessary, a reallocation of evaluation dollars to support in creased sample sizes.

118 The Evaluation Framework proposed no precision targets or requirement s for savings or for any other parameters associated with such studies as proce ss and market effect s evaluations.Evaluators’ Protocols Sampling and Uncertainty

CPUC 164 TecMarket Works Team important than precision for the reliability of the savings estimates or the cost-effectiveness
calculations. 119 In addition, as any evaluation study pr oceeds, the data collected could contain much more error than originally thought, requirin g more resources to be devoted to reducing this
bias and fewer resources devoted to achieving the required st atistical precision. Or, the variability in the savings could be so great th at it would be impossibl e to meet the precision requirement. The evaluator must have the flexibility to respond to data issues as they arise in order to maximize the reliability of the savings. Therefore, focusing on sample error, while giving relatively little attention to these other sources of error, would compromise the CPUC’s objective of obtaining reliable estimates of kWh and kW impacts.

Finally, the guidelines regarding sampling and uncertainty must be followed for each utility service territory. For example, precision targets, when specified for a particular level of rigor,
must be set for each utility service territory.



### Precision: Gross and Net Impact, Measurement and Verification, and
Verification Activities

There are a number of impact-re lated activities concerning precisi on addressed in this section:

• Estimation of gross impacts (including M&V);

• Estimation of net impacts;

• M&V in support of specific measure studies; and

• Verification studies in support of non-Impact Evaluation Protocol gross and net impacts.

The issue of precision for each of these type s of analytical studies is addressed in Table 19 through Table 23.

119 California Framework, p. 296.Evaluators’ Protocols Sampling and Uncertainty

CPUC 165 TecMarket Works Team Table 19. Required Protocols for Gross Impacts 120

Rigor
Level Gross Impact Options

Simplified Engineering Models : The relative precision is 90/30 121 . The sampling unit is the premise. The sample size selected mu st be justified in the evaluation plan and approved as part of the ev aluation planning process.

Basic Normalized Annual Consumption (NAC) Models : There are no targets for relative precision. This is due to the fact that NAC models are typically estimated for all participants with an adequate amount of pre- and post-billing data. Thus, there is no sampling error. However, if sampling is conducted, either a power analysis 122 or justification based upon prior evaluations of similar programs must be used to determine sample sizes. The sample size selected must be justified in the evaluati on plan and approved as part of the evaluation planning process.

Regression : There are no relative precision tar gets for regression models that estimate gross energy or demand impacts. Evaluators are expected to conduct, at a minimum, a statistical power analysis as a way of in itially estimating the required sample size. 123 Other
information can be taken into account such as professional judgment and prior evaluations of similar programs. The sample size select ed must be justified in the evaluation plan and approved as part of the ev aluation planning process. Enhanced

Engineering Models : The target relative precision for gross energy and demand impacts is 90/10. The sampling unit is the premise. The sample size selected must be justified in the evaluation plan and approved as part of the evaluation planning process.

120 See the Impact Evaluation Protocol for a descrip tion of methods and page references in the Evaluation

Framework for further information and examples.

121 Also of interest, in addition to the relative precision, are the actual kW h, kW, and therm bounds of the interval.

122 Statistical power is the probab ility that statistical significance will be atta ined, given that there really is a treatment

effect. Power analysis is a statistical technique that ca n be used (among other things) to determine sample size requirements to ensure statistical sign ificance can be found. Power analysis is only being required in the Protocol for determining required sample sizes. There are se veral software packages and calculation Web sites that conduct the power analysis calculatio n. One of many possible references includes: Cohen, Jacob (1989) Statistical Power Analysis for the Behavioral Sciences , Lawrence Erlbaum Associates, Inc.

123 Ibid.Evaluators’ Protocols Sampling and Uncertainty

CPUC 166 TecMarket Works Team Table 20. Required Protocols for Net Impacts

Rigor Level Net Impacts Options

Basic For the self-report approach (Option Basic.1) , given the greater issues with construct validity and variety of layered measurements involved in estimating participant NTGRs, no relative precision target has been established. 124 To ensure consistency and comparability a minimum sample size of 300 sites (or decision-makers in cases where decision-makers cover multiple sites) or a census 125 , whichever is smaller, is required.

Standard If the method used for estimating net energy and demand impacts is regression-based, there are no relative precision targets. If the method used for estimating NTGRs is regression-based (discrete choice), there are no re lative precision targets. In either case, evaluators are expected to conduct, at a minimu m, a statistical power analysis as a way of initially estimating the required sample size. 126 Other information can be taken into account such as professional judgment and pr ior evaluations of similar programs.

For the self-report approach (Option Standard.2 ), there are no precision targets since the estimated NTGR will typically be estimated us ing information collected from multiple decision-makers involving a mix of quantitat ive and qualitative information around which a
standard error cannot be constructed. Thus to ensure consistency and comparability, for such studies, a minimum sample size of 300 sites (or decision-makers in cases where decision-makers cover multiple sites) or a census, whichever is smaller, is required.

Enhanced The requirements described for Enhanced apply depending on the methods chosen.

124 This is considered the best feasible approach at the time of the creatio n of this Protocol. Like the other

approaches to estimating the net-to-gross ratio (NTGR), t here is no precision target when using the self-report method. However, unlike the estimati on of the required sample sizes when using the regression and discrete choice approaches, the self-report approach poses a unique set of challenges to estimating required sample sizes. These challenges stem from the fact that the self-report methods for esti mating free-ridership involve greater issues with construct validity, and often include a variety of layered measurements involv ing the collection of both qualitative and quantitative data fr om various actors involved in the decision to install the efficient equipment. Such a situation makes it difficult to arrive at a prior estima te of the expected variance needed to estimate the sample
size. Alternative proposals and the sup port and justifications that address all of the issues discussed here on the

aggregation of variance for the proposed self-report method may be submitted to Joint Staff as an additional option (but not instead of the Protocol requ irements) in impact evaluation RFPs and in Evaluation Plans. Joint Staff may elect to approve an Evaluation Plan wi th a well-justified alternative.

125 A census is rarely achieved. Rather, one attempts to conduct a census, recognizing that there will nearly always be some sites, participants or non-participants who drop out fo r a variety of reasons such as refusals or insufficient
data.

126 Statistical power is the probab ility that statistical significance will be atta ined, given that there really is a treatment

effect. Power analysis is a statistical technique that ca n be used (among other things) to determine sample size requirements to ensure statistical sign ificance can be found. Power analysis is only being required in the Protocol for determining required sample sizes. There are se veral software packages and calculation Web sites that conduct the power analysis calculation.Evaluators’ Protocols Sampling and Uncertainty

CPUC 167 TecMarket Works Team Table 21. Required Protocols for Measure-level Measurement and Verification

Rigor
Level M&V Options

Basic Simplified Engineering Models : The target relative precision for gross energy and demand impacts is 90/30. The sample unit may be the individual measure, a particular circuit or point of control as designated by the M&V plan.

Enhanced Direct Measurement and Energy Simulation Models : The target relative precision for gross energy and demand impacts is 90/10. The sample unit may be the individual measure, a particular circuit or point of control as designated by the M&V plan.

Table 22. Required Protocols for Sampling of Measures Within a Site

The target relative precision is 90/20 for each measure selected for investigation. The sampling unit (measure, circuit, control point) shall be designated by the M&V plan. The initial assumption regarding the coefficient of variation for determining sample size is 0.5.

Table 23. Required Protocols for Verification

Rigor Level Verification Options

Basic The target relative precision is 90/10. The key parameter upon which the variability for the sample size calculation is based is binar y (i.e., Is it meeting the basic verification criteria specified in the M&V Protocol?).

Enhanced The target relative precision is 90/10. The key parameter upon which the variability for the sample size calculation is based is binar y (i.e., Is it meeting the enhanced verification criteria specified in the M&V Protocol?).

Of course, when sampling from any population it should always be assumed that there will be
some attrition due to such factors as refusals to participate in a telephone survey or an on-site inspection, or insufficient data. As a result, a larger sample than is actually needed should always be drawn based on the best estimate of expected attrition.



### Development of the Evaluation Study Work Plan

For each study in the evaluator’s defined set of studies, the evaluator must prepare a detailed evaluation work plan (plan) th at allocates resources to maximize reliability for the program group and takes into account that the level of ri gor will likely vary by program. In many cases, the evaluator will be required to develop a separa te work plan for each program in the study set. In some cases, a draft plan will be required as part of the initial proposal package, in others the evaluator may be required to deve lop this work plan after the hiri ng process is complete. As part of this plan, the evaluator must specifically addre ss the various sources of error that are relevant and explain how the resources allocated to each will mitigate the error 127 . They must also estimate the statistical precision that the planned evaluation will achieve. It is also recognized

127 In the pre-1998 M&E Protocols, there was no requirement to address these sources of error in the research plan.

Evaluators only had to describe in the final report whether th ey had to address these various errors and, if so, what they did to mitigate their effects.Evaluators’ Protocols Sampling and Uncertainty

CPUC 168 TecMarket Works Team that the targeted precision at the program level must be allowed to vary in ways that produce the greatest precision at the program group level . For example, in some cases accepting a lower level of precision for programs with small savi ngs might allow for the allocation of greater resources to programs with larger savings, thus increasing the achieved pr ecision for the program

group. 128

The Joint Staff and other outside resources as deemed appropriate by the CPUC will review the evaluation plan submitted and discuss with the independent evaluator any changes they deem necessary to maximize the reliability of the savings estimates at the program group level. 129 The Joint Staff might decide to increase the sample size in order to incr ease precision, recognizing
that other sources of error will receive fewer re sources, or they might decide to reduce the sample and settle for lower precision in exch ange for a greater effort to reduce non-response bias. In the final plan, the evaluators and Join t Staff will endeavor to allocate their available evaluation resources in such a way as to maximi ze the reliability of th e savings and the value
received from the evaluation efforts. In orde r to more adequately address accuracy and/or precision, once evaluation studies are underway, Joint Staff ma y adjust the allocation of resources that were initially dedicated to th e evaluation of a given program, program group, or study set.

The level of rigor assigned to each program will vary depending on the evaluation priorities and budgets discussed above. However, because each program is somewhat unique with respect to the various sources of bias, there is no specific set of required methods and level of effort for minimizing bias that can be assigned based on the level of evaluation rigor assignment.

At the same time, every impact and indirect imp act evaluation plan, analysis and report is expected to seriously address, at a minimum, each and every one of the ten sources of uncertainty listed in the introduction of this se ction. The assessment of the potential issues, testing, minimization approaches and mitigation effo rts are to be discussed in the evaluation plan and carried forward through the evaluation a nd evaluation reporting. This assessment and reporting needs to include the ju stification based on prior evaluations, ev aluation science and other research (with appropriate citation) that support the eval uation research design decisions made in the evaluation plan and the handling of the issues through the analyses. The reporting should include specific data collection, measurem ent and handling of each issue at a level of detail that allows the study results to be replic ated. Results from tests of alternative methods of data handling should be included. For example, if outliers are dropped from the analysis, the reporting should include the methods used to identify outliers, anal ysis results with and without outliers, and the justification us ed in deciding to remove some or all of the outliers. Data cleaning methods and decision rules should be supplied with at least some testing of the analysis impacts produced by varying the primary paramete rs in these decision ru les. Similarly, any sampling and site selection parameters need to be examined for potential bias with appropriate research questions and tests be ing conducted on key parameters.

128 See California Framework , pp. 305-313 for a description and some examples of how to allocate resources and sample sizes to obtain the sm allest possible error bound for a group of programs.

129 Ibid, 298-300 for a description of calculating error bounds and precision levels for different types of evaluation

study integrations.Evaluators’ Protocols Sampling and Uncertainty

CPUC 169 TecMarket Works Team 

### Process Evaluations

For process evaluations, the focus is on reliabi lity at the program level, with the level of evaluation rigor varying as a function of evalua tion priorities and budgets. However, because each program is somewhat unique, with respect to the data being collected and the various sources of bias, there is no specific set of re quired methods and level of effort for minimizing bias that can be assigned to a program that has b een assigned a given leve l of evaluation rigor.

Requiring 90/10 precision, for example, for all inquiries is very likely infeasible and not costefficient because budgets are limited, there is of ten a large set of evaluation questions to be addressed (i.e., many different questions and para meters for which some level of precision could be desired), not all of which ar e quantitative, and the informa tion sought from different survey and interview groups might not be equally valuab le. For example, one might want to field a small survey to get a sense of the motivation of a particular market actor. Again, it is important for the evaluator to have the flexibility to maxi mize the reliability of th eir findings. However, the 90/10 level of precision should be adopted as a minimum precision target for the most important data collection efforts on its most impo rtant variables. Which data collection efforts and variables are considered to be the most impo rtant for process evaluations will be determined by the independent evaluator in close co llaboration with utility EM&V staff.

There are circumstances when it might be desirabl e to use M&V as input to the analysis of a problem being investigated in a process evaluati on. If M&V is not conducted by the Joint Staff evaluations, utility evaluation st aff may chose to specify M&V activities within the process
evaluation RFP. 130 If the M&V Protocol is used for purpos es outside impact, in direct impact and verification analysis, a target precision should, at a minimum, be 30 percent precision at a 90 percent confidence level (or 90/30 precision).

The evaluator must prepare a detailed plan th at allocates resources in order to maximize reliability for the findings and for key parameter estimates for each program in the group. As part of this plan, the evaluator must specifically address the various sources of error that are relevant and explain how the resources allocated to each will minimize and/or mitigate the

error. 131 They must also estimate the statistical precision that the planned evaluation will achieve on selected primary quantitative measurements.

The Joint Staff and other outside resources as deemed appropriate by the CPUC will review the evaluation plan submitted and discuss with the independent evaluator any changes they deem necessary to maximize the reliability of the findi ngs at the program level. The evaluation staff might decide to increase the sample size in orde r to increase precision, re cognizing that the other
sources of error will receive fewer resources. Or it might decide to reduce the sample size and settle for lower precision in exchange for a gr eater effort to reduce non-response bias. In the final plan, evaluation resources wi ll be allocated in a way that maximizes the reliability of the findings for each program.

130 Coordination of M&V studies for process and impact pur poses is a key issue that must be addressed by the

evaluation plans for both process and impact evaluation.

131 In the pre-1998 Protocols, there was no requirement to address these sources of error in the research plan.

Evaluators only had to describe in the final report whether th ey had to address these various errors and, if so, what they did to mitigate their effects.Evaluators’ Protocols Sampling and Uncertainty

CPUC 170 TecMarket Works Team 

### Market Effects

The focus is on the market level for market effect s evaluations. The level of rigor assigned to a particular market effects study will depend on the evaluation prio rities and budgets. However, because each market effects study will be some what unique with respect to the data being
collected and the various sources of bias, there is no sp ecific set of required methods and level of effort for minimizing bias that can be assigned to a given market effects study.

Requiring 90/10 precision for all estimates, for exam ple, is very likely in feasible and not costefficient because budgets are limited, there ar e often a large set of evaluation questions, outcomes and causal mechanisms to be assessed in a market effects evaluation (i.e., many different questions and parameters for which some level of preci sion could be desired), and the information sought from different survey, inte rview groups and data sources might not be equally valuable. For example, one might want to field a small survey to roughly estimate the number of HVAC contractors who actively prom ote energy-efficient air conditioners. Again, it is important for the evaluator to have the flexibil ity to maximize the reliab ility of their findings. However, the 90/10 level of precision should be adopted as a minimum pr ecision target for the most important data collection efforts on its most important variables. Which data collection efforts and variables are considered to be the most important will be determined by the independent evaluator in clos e collaboration with the CPUC.

The evaluator must prepare a detailed evaluatio n plan that allocates resources in order to maximize reliability of market-level estimates. As part of this pl an, the evaluator must specifically address the various so urces of error that are relevant and explain how the resources allocated to each will minimize and/or mitigate the error (e.g., non-response bias, measurement error, and self-selection bias). 132 They must also estimate the statistical precision that the planned evaluation will achieve on key estimates a nd for the overall estimate of market effects (to include the propa gation of error).

The Joint Staff and other outside resources as deemed appropriate by the CPUC will review the evaluation plan submitted and discuss with the independent evaluator any changes they deem necessary to maximize the reliability of the estimates at the market level. For example, The Joint Staff might decide to increase th e sample size or budget in order to increase precision for specific parameters or study elements, recognizing that the other sources of error will receive fewer resources. Or it might decide to reduce the samp le and settle for lower precision in exchange for a greater effort to reduce non-response bias. In the final plan, evaluation resources will be allocated in a way that maximizes the re liability of the mark et-level estimates.



### System Learning

The hallmark of any learning system is that f eedback is processed and any necessary course corrections are made. Once a par ticular evaluation is launched, it’s certainly possible that midcourse adjustments will be made to the initial pl an to maximize savings reliability. For example,

132 In the pre-1998 Protocols, there was no requirement to address these sources of error in the research plan.

Evaluators only had to describe in the final report whether th ey had to address these various errors and, if so, what they did to mitigate their effects.Evaluators’ Protocols Sampling and Uncertainty

CPUC 171 TecMarket Works Team the coefficients of variation (CVs) 133 for certain key parameters, m easures, end-uses or programs might actually be smaller than anticipated or the random and/or systematic measurement error might be worse. As data are collected and assessed, decisions can be made regarding the
reallocation of resources.

Once a particular study is completed or all the studies within a given group are completed, the CPUC-ED, utility EM&V staff a nd the independent evaluators can review the achieved precision
and the results of efforts to minimize bias a nd recommend how evaluati on resources can be reallocated for the next evaluation cycle.



### Acceptable Sampling Methods

It is rarely possible, for a vari ety of different reasons, to con duct a census of a ny population (e.g., program participants, programs non-pa rticipants or lighting vendors). 134 Especially in a state the size of California, this is due la rgely to the fact that many of the populations are quite large and the cost of attempting a census study would be pr ohibitive. Instead, random samples drawn from these populations are almost always used as a wa y to estimate various ch aracteristics of these populations. The specific approaches to maximi zing precision are left up to the independent evaluator. For example, one can choose from a variety of sample proced ures recognized in the statistical literature, such as sequential sampling, cluster sampling, multi-stage sampling and stratified sampling with regr ession estimation. There are many available books on sampling techniques that can be used as reference. 135



### Skills Required for Sa mpling & Uncertainty

Population database work and simple random sa mpling (or census) do not require an advanced statistics background. Other more complex sa mple designs require basic training and/or experience in statistics to ensure that the me thods are understood and a pplied correctly. Those conducting and reviewing this work should have at least basic gra duate statistics or equivalent experience with a mentor in this area. The skills required for addressing the uncertainty
associated with the various methods for estimati ng the gross and net energy and demand impacts as well as the net impacts are describe d as part of the Impact Protocols.



### Audience and Responsible Actors

• Program Evaluators should understand and implement this Pr otocol. They also need to be able to correctly estimate the expected precision and accuracy. Based on the achieved precision and accuracy, they must recommend an y reallocation of evaluation resources going forward;

• CPUC-ED CEC and Utility EM&V Staff should understand this Protocol and be able to correctly interpret the expected and achieved levels of preci sion and accuracy in order to

133 The sample standard deviation divided by the sample mean. See page 320 of the Evaluation Framework

. 134 In process evaluations, a census is possible in some more limited populations such as staff and program

contractors.

135 The two cited in the Evaluation Framework are 1) Cochran, William G. Sampling Techniques . New York: John Wiley & Sons, 1977 and 2) Sarndal, Carl -Eric, Bengt Swensson and Jan Wretman. Model Assisted Survey Sampling . New York: Springer-Verlag, 1992.Evaluators’ Protocols Sampling and Uncertainty

CPUC 172 TecMarket Works Team accept or reject any recommendations regardi ng the reallocation of evaluation resources going forward; and

• Utility System Planners should be able to understand the achieved precision and accuracy and the overall reliability of the savings in order to assess their resource value.



### Key Metrics and Information Covered

All evaluation reports must contain a variety of information regarding the sample design and implementation as well as a variety of inform ation regarding the vari ous sources of bias encountered and efforts to mitigate them. These are outlined below.



### Sample Size and Precision

Whenever estimates are based on a sample in any evaluation, the following information, as appropriate, must be reported:

• The definition of the population from which the sample was drawn;

• The sample design (e.g., simple random, stratified random and two-stage);

• The assumptions and related documentation upon which the initial sample size calculations were based (e.g., CV for key i nputs in an engineeri ng algorithm, CV for proportion of audit participants who adopt recommendations, the specified statistical power, effect size, confiden ce level and alpha level);

• The details of how the initial sample sizes were calculated to ach ieve the agreed upon level(s) of precision;

• The achieved precision around program-level gross and net kWh, kW, and therm impacts, key process evaluation measurements, and other program impacts such as attitude change and knowledge gains;

• The confidence intervals specified in te rms of the kWh, kW, and therm impacts;

• The details of how the achieved sample size was used to calculate the precision; and

• Response rate and attrition and any suspected non-response bias and efforts to address
it. 136



### Validity and Research Design

• Discuss threats to internal validity (the exte nt to which alternative potential causes of the
measured effect have been ru led out within the analysis); 137

• Discuss threats to exte rnal validity (the extent to wh ich the analysis results found for a sample are true for the popul ation and the program overall); 138 and

136 See “Definitions of Response, Refusal, and Cooperat ion Rates” prepared by the Council for Marketing and

Opinion Research and the “CASRO Guidelines for Survey Re search Quality,” prepared by the Council of American Survey Research Organizations (www.casro.org).

137 See Evaluation Framework , pp. 292-295 and 425.

138 Ibid, 292-295 and 421.Evaluators’ Protocols Sampling and Uncertainty

CPUC 173 TecMarket Works Team • Discuss assessment of construct validity a nd potential remaining issues of construct validity for the primary evaluation outputs (t he extent to which the measurement (and instrumentation, such as survey wordi ng) captures the underlying abstract idea). 139



### Accuracy

Regression models:

• Describe procedures used for the treatment of outliers, missing data points and weather adjustment;

• Describe what was done to control for selection bias, if suspected;

• Describe what was done to control for the effects of background va riables, such as economic and political activity that may account for any increase or decrease in consumption in addition to the program itself;

• Describe procedures used to screen data for inclusion into the final analysis dataset. Show how many customers, installations or observations were eliminated with each
screen. The reviewer should be able to clearly follow the development of the final analysis dataset;

• Regression statistics: For all final models, pr ovide standard regre ssion statistics in a tabular form;

• Specification: Refer to the section(s) of the study that present the initial and final model specifications that were used, the rationale for each, and the documentation for the major
alternative models used. In addition, the presentation of th e specification should address, at a minimum, the following issues:

o Describe how the model specification a nd estimation procedures recognize and address heterogeneity of customers (i.e., cross-sectional variation);

o Describe how the model specification a nd estimation procedures recognize and address changes in factors that affect consumption over time (i.e., time series variation), apart from program effects;

o Describe how the model specification a nd estimation procedures recognize and address the fact that participants self-selec t into that status, an d discuss the effects
of self-selection on model estimates whet her or not self-selection is treated explicitly;

o Describe how truncation within the data and regression towards the mean within the participant population (e.g., within low-income populations) is tested for, the results of this test, and how model specification and estimation procedures recognize and addre ss these issues;

o Discuss the factors, and their associat ed measures, that are omitted from the analysis, and any tests, reasoning or sp ecial circumstances that justify their omission; and

139 Ibid, 292-298 and 414.Evaluators’ Protocols Sampling and Uncertainty

CPUC 174 TecMarket Works Team o Describe how the model specification can be interpreted to yield the measurement of program impacts.

• Error in measuring variables: Describe wh ether and how this issue was addressed, and what was done to minimize the problem;

• Autocorrelation: Describe any autocorrelati on problems and the solutions specifically taken to address the problem. Specific id entification and mitigation diagnostics should be presented, including differing treatment for sub-groups, if any;

• Heteroscedasticity: Describe the diagnostics car ried out, the solutions attempted and their
effects. If left unt reated, explain why;

• Collinearity: Describe procedures used to address the problem of collinearity, and the reasons for either not treating it or treating it to the level that it was;

• Influential data points: Descri be the influential data diagno stics that were used, and how the identified outliers were treated;

• Missing data: Describe the methods used for handling missing data during the analysis phase of the study; and

• Precision: Present the methods for the calculati on of standard errors for key parameters such as gross impacts, net impacts, NT GRs, and key process and market effects

measurements.
Engineering Models Including M&V

• Describe the primary sources of uncertainty in deemed and measured parameters used in engineering models;

• Describe the construction of the baseline. Include assessment and de scription of how the
selection of baseline affect s the development of gross impacts versus net impacts. Baseline definitions shall be consistent with those used in the net analysis;

• Discuss efforts to guard against measurement error associated with the various M&V data collection efforts;

• Discuss site selection and pot ential non-response bias, any tests performed to assess potential bias across and within site measur ements, and potential effects of any remaining concerns in this area;

• Describe any potential measur ement or bias issues associated with the measurement approaches and tools used as they apply to specific program parameters and estimates:

o Engineering model bias – systematic unde r- or over-prediction of effects of a measure by an engineering model;

o Modeler bias – the systematic under- or over-prediction of eff ects of a measure by a building energy simulation (e.g., DOE-2) modeler. Also includes the random under- or over-prediction of effects of a measure by a building energy simulation (e.g., DOE-2) modeler;

o Deemed parameter bias – systematic deviat ion in a deemed parameter used in an engineering model;Evaluators’ Protocols Sampling and Uncertainty

CPUC 175 TecMarket Works Team o Meter bias – systematic error in meter and/or sensor;

o Sensor placement bias – systematic ov er- or under-predic tion of measured quantity due to sensor placement (could be combined with above); and

o Non-random selection of equipment and/or circuits to monitor.



### Summary of Sampling and Uncertainty Protocol

A summary of these Protocols is not provided here. Rather, in the summaries provided at the end of the other Protocols (Impact, M&V, Emerging Technology, Codes and Standards, Effective Useful Life and Market Effects), the relevant elements of the Sampling and Uncertainty Protocols are discussed.Evaluators’ Protocols Reporting

CPUC 176 TecMarket Works TeamEvaluators’ Protocols Reporting

CPUC 177 TecMarket Works Team 

## Evaluation Reporting Protocol

Introduction

The Evaluation Reporting Protocol identifies the information that must be incorporated in the different types of evaluation reports and specifies how it is to be reported. This is accomplished by first identifying the common information requi red across all evaluation reports. Then the
Protocol describes additional information and presentation formats for each of the types of evaluation reports.

The reporting information contained in this Protocol is that which support the program
evaluation efforts. There are other reporting requirements asso ciated with program status, progress and financial reporting no t covered in this Protocol for which Administrators are responsible. For information relating to program status, progress and financial reporting the
reader is referred to the CPUC-ED.



### Report Delivery Dates

The delivery dates for each evaluation report must be identified in each program evaluation plan. Both the report delivery dates a nd changes to these dates must be approved by the CPUC-ED. The scheduling of the all draft and final evalua tion reports must consider the timing of the information needs of the key stakeholders in cluding the CPUC-ED, the CEC and the portfolio Administrators, so that the evaluation results can be provided in time to use the results to support program “performance basis” assessments and to support future program design and evaluation planning. This requirement does not imply that onl y two reports (one draft and one final) will be required from the evaluation contractor. It is expected that each evaluation will have multiple reporting periods across the multi-year study pe riod. Each evaluation plan will detail the deliverables to be provided with in the study scope and the due da tes for each deliverable. Once the final reports are approved by the Joint Sta ff, the evaluation contractor will deliver the electronic and hard copy reports and post the fi nal evaluation report on the CALMAC Web site
consistent with the instructions detailed in this Protocol.



### Common Evaluation Reporting Requirements

This section of the Reporting Protocol pres ents the reporting requi rements specifying the information that must be reported in the various types of draft, draft-fi nal and final evaluation reports. Typically these requirements apply to the evaluation contractor s conducting the studies and preparing the reports.

The present Reporting Protocol is different than previous Califo rnia reporting protocols. In addition to new evaluation repor ting requirements, there are al so performance basis reporting metrics that need to be reported when applicable . The evaluation contractors are responsible for knowing what information is required in th eir evaluation reports and for conducting the evaluation efforts in a way that provides the requi red information. The evaluation contractor will coordinate with the CPUC-ED to identify the perf ormance basis reporting metrics to be included
in each evaluation and structure th e evaluation plan to meet those requirements. Final negotiated study-specific evaluation budgets will be struct ured to meet this Protocol requirement.Evaluators’ Protocols Reporting

CPUC 178 TecMarket Works Team The following reporting requirements apply to all evaluation reports produced from studies of California’s energy efficiency programs incl uding process, impact and market effects
evaluations.

The reporting requirements included in this Protoc ol are minimum requirements. Each program evaluation may have additional reporting require ments that are specified in the approved evaluation plan. For example, an evaluation plan may require that the evaluation report provide “designated units of measure” reporting at the pr ogram level. These units may include items such as kWh savings/square foot of commercial building served or kW savings/square foot of home served. These may also be structured so that the reporting require ments are more defined, such as kWh savings/square feet of commercial building conditioned space served, or kW savings/square foot of occupied space, heat ed space, cooled space, or other criteria.

1. Draft reports are to be provi ded in electronic formats . Draft and final-draft energy and load impact reports, M&V reports, codes and standards reports, emerging technology reports, effective useful life reports and draft market effects evaluation reports will be provided to the
Joint Staff in electronic file formats consistent with the file format requirements provided in this Protocol for final reports (see below). Draft process evaluation reports will be provided in formats determined by the Administrators requesting the studies.

2. At least 10 copies of all final evaluation rep orts must be submitted in bound hard copy format on recycled paper using double-side d printing to minimize the use of paper. No less than four hard copies shoul d be provided to the CPUC-ED, two hard copies to the CEC, three hard copies to the Administrator(s) fo r the program(s) being evaluated and one hard copy to the program implementation manager (w hether a contractor or employee of the Administrator) of the program being evaluated. The Administrator and the Joint Staff can request that evaluation contractor s provide additional copies as appropriate or can advise the
evaluation contractors that fewe r hard copies are needed. Th is requirement serves as the
minimum deliverable of the final evaluati on reports in bound hard copy format unless specified differently for an individual study.

3. All final reports will be provided to the CP UC-ED, the CEC and the Administrators in unprotected (no password restrictions) electro nic formats and protected formats that can be made available to the public. The electronic formats must be provided in two software versions with each report provided in a single electronic file. The unprotected electronic reports must be provided in Microsoft Word ® . The protected formats should be provided in Adobe ® formats in a version that is lo adable/readable by the organization contracting for the study. The electronic file s must be named in a way that allows the recipients to understand the program or th e group of programs on which the evaluation reports. Examples of acceptable file names include the following:

a. 06 PG&E Mass Market Process Eval.pdf

b. 06-08 SCE Res Programs Impact Eval.doc

c. 06 SCE Appliance Recy cling Process Eval.pdf

d. 06-07 Statewide Multi-Family Programs Impact Eval.docEvaluators’ Protocols Reporting

CPUC 179 TecMarket Works Team Evaluation contractors conducting en ergy impact studies will also provide Microsoft Excel files presenting the energy savings (kW, kWh, therms) from direct or indirect impact, codes and standards, or market effects studies as described in this Protocol (see Sample Reporting Tables at the end of this Protocol).

4. Within five days of the subm ission and acceptance of the final evaluation report, the organization providing the report must post it and its abstract on the CALMAC Web site using the posting instructions provi ded by CALMAC at the time of posting. The abstract posted on the CALMAC site should be the one included within the final evaluation report located just after the title page. Care should be ta ken in developing the abstract to allow the CALMAC search engines to easily find the report when system users conduct keyword searches. Upon posting, CALMAC will distribute an e-mail announcement of the availability of the report to the CPUC’s en ergy efficiency docket list-serve and to the
CALMAC distribution list.

5. All evaluation reports must contain the fo llowing information on the report cover of both the electronic and hard copy files.

a. Report title that reflects the type(s) of evaluation(s) being conducted (e.g., Energy and Demand Impact Evaluation, Process Evaluation, Effective Useful Life Evaluation, Codes and Standards Program Evaluation, Market Effects Evaluation, or Market Effects Evaluation);

b. Official name of the program(s) as recorded in the CPUC’s program tracking system (EEGA), including the progr am cycle identifier (e.g., 2006-2008, 2009- 2011);

c. Official CPUC/EEGA tracking number(s ) of the program(s) being evaluated;

d. Date of the evaluation report;

e. Name of the organization conducting the evaluation;

f. Name of the organization administering the evaluation;

g. Name of the organization administering the program; and

h. Name of the organization implementing the program.

6. The title page of both hard copy and el ectronic formats must include the following
information:

a. The same information provided on the report cover, plus the following:

b. Name of the organization conducting the evaluation and full contact information for the evaluation lead(s) responsible for the study;

c. Name of the organization administ ering the evaluation and full contact information for the lead Administrator; andEvaluators’ Protocols Reporting

CPUC 180 TecMarket Works Team d. Name of the organization implementing the program and full contact information for the lead program director or manager.

(Contact information should include indivi dual’s name, address, phone number, fax number and e-mail address.)

7. Abstract. Following the title page, the report will include a report abstract. The abstract should be developed consistent with the “Re port Summary” developm ent instructions for posting on the CALMAC Web site. The abstract should be less than 200 words (or consistent with current CALMAC guidance ) and include important key words that allow CALMAC’s Web site’s search engines to locate the report during r outine searches.

8. Evaluation reports should include, at a minimum, the following sections:

a. Cover
b. Title Page

c. Abstract

d. Table of Contents
e. Executive Summary - this section should very briefly present a review of the evaluation findings and the study’s reco mmendations for program change, this should typically be no more than 1-3 pa ges. The findings and recommendations included in the summary should reference the primary text location within the report where each finding or recomme ndation is analyzed and presented.
f. Introduction and Purpose of the Study - this section should give a summary
overview of the evaluation and the evaluati on objectives and re searchable issues.
This section should discuss if each of th e researchable issues presented in the evaluation plan was addressed in the eval uation report and identify if any issues were not addressed and pr ovide the reason why not.
g. Description of Programs Covered in Study - this section should provide a description of the program(s) being eval uated in enough detail that readers can understand the program(s) and have an understanding of the program and program components that delivered the eval uation identified effects. The program description should also include the counts of the number of participants at the end of each program year for each program, a nd estimates of the technical potential (measure counts) for each measure covere d by the program. This market potential
should estimate the number of units that c ould be installed by the program if the technical potential was achieved for each measure covered by the program within the program’s target market. The techni cal potential should be provided by the program Administrator and should be include d in the data request delivered to the
Administrators. If the Administrator doe s not provide the data, the report should so stipulate, identifying the data reques ted and the reason why the data could not be provided. If the Admini strator cannot provide the re quested technical potential data, the report may not be able to disc uss the technical potential and the fraction of this potential achieved by the program.Evaluators’ Protocols Reporting

CPUC 181 TecMarket Works Team h. Study Methodology - this section should describe the evaluation approach in enough detail to allow a repetition of the study in a way that would produce identical or similar findings. See additional content information below.
i. Reliability Assessment of the Study Findi ngs – this should include a discussion of the threats to validity and sources of bias and the a pproaches used to reduce threats, reduce bias and increase the reliabi lity of the findings, and a discussion of study findings precision levels.
j. Detailed Study Findings - this section presents the study findings in detail.
k. Recommendations for Program Changes - this section shou ld be a detailed
identification and discussion of the recommended changes, including the anticipated cost of the recommended change and the expected effect of the change on the operations and cost-effec tiveness of the program(s).
l. Appendix A - appendix A should be a pr esentation of the performance metrics identified by the CPUC-ED that apply to the types of programs being evaluated and a presentation of the evaluation’s assessment of the performance of the program for each of the performance metr ics covered in the evaluation plan.
m. Appendix B - appendix B should present a nd discuss the success and timing of the data requests provided to the Administra tors and the amount of time between the response and the receipt of the requested data. This section should discuss the success in obtaining the information n eeded to conduct th e evaluations and identify any request made that were not provided in accordance with the provisions in this Protocol. If inform ation was requested and not provided, the appendix should discuss the implications of not obtaining the data on the accuracy and reliability of the study findings. (Infor mation that is maintained in the CPUCED program-reporting database can be obt ained from the CPUC-ED and does not need to be collected from the IOUs.)

The Study Methodology section must include the following:
a. Overview of the approach;
b. Questions addressed in the evaluation;
c. The Protocols and rigor leve ls assigned to the study;
d. Description of the study methodology;
e. How the study meets or exceeds Protocol requirements;
f. How the study addresses issues presente d in the Protocols regarding the methods;
g. Sampling methodology;
h. Expected precision or power analysis results (as required by the Sampling & Uncertainty Protocol);
i. Sample descriptions (including populati on characteristics, contact information availability and sample disposition rates);
j. Description of the baseline;

k. Sources of baseline data;
l. Description of measures; andEvaluators’ Protocols Reporting

CPUC 182 TecMarket Works Team m. Assumptions on measure performa nce (including data sources).

The Reliability Assessment section of the report shoul d focus its presentation and discussion on the targeted and achieved precis ion levels for the key findings presented, the sources of uncertainty in the approaches used and in the key findings presented, and a
discussion of how the evaluation was structured and managed to reduc e or control for the sources of uncertainty. All potential threat s to validity given the methodology used, as presented in the Sampling & Uncertainty Protoc ol, must be assessed and discussed. This
section should also discuss the evaluator’s opinion of how the types and levels of uncertainty affect the study findings. Findings also need to include information for estimation of required sample sizes for fu ture evaluations and recommendations on evaluation method improvements to increase re liability, reduce or te st for potential bias and increase cost efficiency in the evaluation study(ies).

The Recommendations for Program Changes section on need only be added when changes have been identified during the evalua tion process. In general, impact evaluation studies will have the fewest program change recommendations. Market effects evaluations should provide reco mmendations that the evaluati on contractor thinks will improve the ability of the program(s) to in fluence market change. Process evaluations will typically have recommendations, as generating recommendations that increase the cost-effectiveness of the pr ogram(s) is the primary purpose of conducting the process
evaluation.

The evaluation reports should generally be written for a wide range of individuals, including individuals not familiar with evaluation approaches or the field’s specialized terminology. Technical information needed to report methodologies used for research design, sampling, impact analysis, M&V effort s, regression and engineering analysis, bias detection, bias correction and other tec hnical areas must be reported and should not be avoided to ensure readability by a wide r range of audience. A summary of the methodology, findings and decisions covering these issues shoul d be written for a wider audience, however the more te chnical details relating to th ese reporting categories must also be provided.

9. Databases and analysis datasets are the property of the State of California and should be provided to the CPUC-ED within 10 work ing days of the acceptance of the final evaluation report. Database and analysis datasets sh all be delivered in commonly accepted formats, such as SPSS ® , SAS ® , ASCII formatted or defined fields, tab or comma delimited, ASCII text, Microsoft Excel ® , Microsoft Access ® , dBase TM or other similarly commonly available formats. Non-common proprietary data bases are not acceptable deliverable formats. Database suppliers should ne gotiate with the CPUC on a fo rmat structured during the evaluation planning process. Data bases and analysis datasets s hould be provided in electronic
formats with data dictionaries that describe th e fields and field formats. The databases and analysis databases should be named so that th ey can be linked to the program being evaluated and the evaluation report presen ting the findings. They should be provided so that the CPUC or their consultants can duplicate the analysis effort. If the data in the database or in theEvaluators’ Protocols Reporting

CPUC 183 TecMarket Works Team analysis datasets is modified from the data that was collected the modifications should be
disclosed.

Performance Basis Eval uation Reporting Metrics In addition to the above-identified common repor ting requirements, each evaluation should also report, in a table format, those metrics asso ciated with the CPUC-ED’s performance basis reporting requirements that are collected during th e evaluation effort. While not all evaluations will collect and report all of the CPUC-ED perf ormance basis metrics needed by the CPUC-ED,
those metrics that are collected or assessed within the evaluation e ffort should be reported in the draft and final evaluation reports. The perfor mance basis metrics that the evaluation should report, if collected or assessed as part of the evaluation effort are listed in Appendix C. Each evaluation contractor should identify the performance basis metrics that will be collected and assessed during the evaluation planning effort and id entify those metrics that will be reported in the draft and final evaluation reports



### Evaluation Type Specific Reporting Requirements

The following reporting requirements are in addi tion to the reporting requ irements noted above and are presented for each type of evaluation and evaluation effort.

Energy Impact Evaluations and Supporting M&V Efforts The energy impact evaluation report must focus on reporting the gross and net achieved energy savings and demand reduction that can be expected as a result of the program’s efforts for each program year and for the program at the end of the program cycle in accordance with the progress of the evaluation within the program cy cle. The impacts should be reported for each full calendar year (2006, 2007, 2008 and totaled fo r all years within a program cycle (2006- 2008) over the effective useful lif e (EUL) of the measures instal led or the behaviors changed. The reporting should assume a full year of measur e use for the year in which the measures are installed. This avoids partial ye ar reporting during the year of inst allation and at the end of the EUL. That is, a program that installs measures that have an effective measure life of 10 years installed in 2006, would report 10 year s of savings for that measure with the first full year being 2006, regardless of the date that the measures were installed during 2006. For programs that have a mix of measures with different EULs, th e savings projections will reflect the end-of-EUL drop-offs so that the projected sa vings represent only those savings that are expected in a specific year. When the CPUC-ED specifies that an ev aluation will assess measure-level savings, the assessment should target the measures approve d by the CPUC. In some cases this will encompass all of the measures included in the program and, in other cases, it will include only some specific measures.

The reported savings need also be net of interactive eff ects. For example, if lighting measures are installed there may be a correspond ing decrease or increase in HVAC costs. Or if there are therm savings that produce an increas e in electric consumption, these conditions need to be incorporated into the net effects estimate.

Savings also need to be reported by the CEC’ s five Climate Thermal Zones (CTZ) used for assessing Title 24 compliance, within the zone s that have evaluation- study-covered programEvaluators’ Protocols Reporting

CPUC 184 TecMarket Works Team participants. 140 This does not mean that M&V sampling needs to be conducted at the CTZ level, but that impact and supporting M&V results must be modeled so that the impacts are reported for each of the climate zones in which participants appear. However, the Joint Staff may request specific studies report impacts by each of the 16 climate zones in which participants appear if this requirement is in the approved evaluation plan. The CEC will provide the CTZ maps, address and geo-code matches to each climat e zone and weather data to the evaluation contractors on request. Reporti ng also must be provided for each IOU when a program is provided in more than on e IOU service territory.

Every energy impact evaluation report should include the following information:

1. CPUC approved program ex-ante net and gr oss, kW, kWh and therm savings goals recorded at the beginning of the program fundi ng cycle and any modifications to these goals made during the funding cycle. These should be the energy savings targets for the programs included in the Administrator’s portfolio fili ngs approved by the CPUC and any changes to these goals resulting from adjustments made. If the goals have changed during the funding cycle, a brief discussion of the reasons for th e change should be repor ted also. Goals should be reported for each calendar year in which impacts are projected.

2. The Administrator-generated annual gross kW, kWh and therm savings. These should be the energy and demand savings estimates th at the Administrator reports to the CPUC-ED as achieved against the CPUC-approved goals.

3. Evaluation projected annual gross and net MW (megawatt) impacts measured for each calendar year for each year over the EUL of the measures instal led or behaviors taken. Gross and net demand savings must be reported for si x time periods over each of four months as follows: noon–1 p.m., 1–2 p.m., 2- 3 p.m., 3-4 p.m., 4-5 p.m. a nd 5-6 p.m. for June, July, August and September, for each climate zone for which there are program participants. These demand savings are to be estimated us ing the CEC’s five CTZs used for assessing Title 24 compliance. This metric represents the evaluation contractor’s best estimate of the gross and net program-induced par ticipant-based MW impacts. This metric is to be reported separately for total program savings and broke n out by program-induced direct and indirect (as appropriate to each study) impacts and for pa rticipant spillover eff ects, if any. If the evaluation is designed to deliv er measure-level kW (reported as MW) savings, the savings will be reported for each measure included in the measure-level assessment. In addition, the effects are to be reported for the measure as a whole and for both direct and indirect program effects and participant spillover effects. In addition to these reporting requirements the Joint Staff may identify additional kW reporting re quirements for specific studies during the program evaluation planning process. The demand impacts are those that can be documented at the time of the evaluation and they are not to include projected im pacts as a result of actions not yet taken. These impacts are not to include market effect s or non-participant spillover kW effects, but instead focus only on the impacts from participants who take

140 California Climate Zones, The climate zones used for this purpose are the California Energy Commission’s five

Climate Thermal Zones used for assessing Title-24 complian ce unless specified differently by the CPUC-ED during the program planning process. In some case it may be necessary to require reporting by each of the 16 SCE Climate Thermal Zones, also referred to as the 16 Title 24 climate zones.Evaluators’ Protocols Reporting

CPUC 185 TecMarket Works Team advantage of the program’s offeri ngs and who may replicate those actions in their facilities.
If the evaluation contactor determines that MW impacts increase or degrade over time, the annual projections of impacts must incorporat e that increase or de gradation factor and explain the cause, the reliability and the meas urement approach for documenting the increase or degradation rate. (See tables below for ex ample of reporting formats.) Participant spillover is to be reported in the evaluation, but will not be credited for the purposes of goal accomplishment at this time.

4. Evaluation projected annual MWh (megawatt-hours) gross and net savings measured for each calendar year for each year over the EU L of the measures installed or behaviors taken. Savings should be reported for the program as a whole and for each of the CEC’s five CTZs used for assessing Title 24 compliance in which the program operates. This metric represents the evaluation contra ctor’s best estimate of the en ergy savings that will occur because of the actions of the program. There ar e three reporting metrics associated with this requirement. The annual MWh savings are to be reported for the program as a whole and separately, for both program participation-ba sed direct and indire ct savings, and for participant-spillover-based savings. If the evaluation is designed to deliver measure-level savings, the savings will be reported for each measure included in the measure-level assessment. The savings are those that can be documented at the time of the evaluation and they are not to include projected savings as a result of actions not ye t taken. These savings are not to include market effects or non-part icipant spillover savings, but instead focus only on the savings from participants (direct and sp illover) that take adva ntage of the program’s offerings. If the evaluation contactor determines that savings increase or degrade over time, the annual projections of savings must incorpor ate that increase or degradation factor and explain the cause, the reliability and the meas urement approach for documenting the increase or degradation rate. (See ta bles below for example of re porting formats.) Participant spillover is to be reported in the evaluation, bu t will not be credited for the purposes of goal accomplishment.

5. Evaluation-projected annual gross and net therms (100,000 BTU/therm or 100 cubic feet of methane) of natural gas savings measured for each calendar year for each year over
the EUL of the measures installed or behaviors taken. This metric represents the evaluation
contractor’s best estimate of th e energy savings that will occur because of the actions of the program. The annual therm savings are to be reported separately to include program participation-based savings plus participant sp illover based savings, if any and totaled for program savings. However, participant spillove r will not be counted toward the program or portfolio goal achievements. Th e savings are those that can be documented at the time of the evaluation and they are not to in clude projected savings as a re sult of actions not yet taken. If the evaluation is designed to estimate measur e-level savings, the savings will be reported
for each measure included in the measure-level assessment. These savings are not to include market effects, participant or non-participan t spillover savings estimates, but instead focus only on the savings from participants that take advantage of the program ’s offerings. If the evaluation contactor determines that savings increase or degrade over time, the annual projections of savings must in corporate that increase or degr adation factor and explain the cause, the reliability and the measurement approach for documenting the increase or degradation rate. (See tabl es below for example of reporting formats.)Evaluators’ Protocols Reporting

CPUC 186 TecMarket Works Team The Energy Impacts Protocol requires the eval uation contractor to estimate annual gross and net impacts over the EUL of the installed t echnologies or the be havior change-induced actions. For measures like CFLs, the expected life of the impacts may only be a couple of years, while for building design changes, the impacts may be over 30 years or more if the evaluation determines that the changes woul d not have occurred in the absence of the program offerings. It is the re sponsibility of the ev aluation contractor to establish evaluation designs and approaches that allow these metrics to be reported in the evaluation report. One of the primary reasons that these metrics are re quired is so that por tfolio energy and load impact curves can be generated for each pr ogram, for each IOU and for the portfolio as a whole. These savings are not to include non-participant savings that may have been influenced by the program’s operations or the spillover caused in the non-participant population as a result of the program. They are to include participant spillover or participant action replications that result as a function of program participation. However, participant spillover savings are not to be counted towa rd program or portfolio goal achievements.

6. Measure counts per participant. This metric is incorporated into the reporting criteria so that the evaluation report provi des a presentation of the t ypes of measures taken by the program participants and the number of those actions taken per participant. This metric is to be retrospective and report only the actions taken as a result of the program at the time of the evaluation. However, the evaluation should true up these metrics at the end of each program year so that they can be reported for each prog ram year (see sample reporting sheet at the end of this Protocol). The assessment shoul d be based upon tracking system reviews and informed by the impact evaluation and the suppo rtive M&V efforts. It can also be supported by the process evaluation efforts, if there is coordination among the evaluation efforts. The evaluation study can also separate ly report projected actions to be taken, if approved in the evaluation plan.

7. Measure counts versus program goals . This metric is incorporated into the reporting criteria so that the evaluation report provide s a presentation of the evaluation verified program accomplishments relative to measure in stallation goals. This metric is to be retrospective and report only the actions taken as a result of the program at the time of the
evaluation. It should be base d upon tracking system reviews and informed by the impact evaluation and the supportive M&V efforts. It can also be supported by the process evaluation efforts, if there is coordination am ong the evaluation efforts. The evaluation study can also report projected actions to be taken, if approved in the evaluation plan.

8. Measure-level savings . If the evaluation plan is struct ured to provide measure-level or behavior-level savings estimates, these metrics should be reported for the covered measures. Not all program evaluation plans will be focused at the measure or behavior level. However, for those that are, as a result of the Joint St aff evaluation prioritizati on efforts, the savings should be reported at the measure or behavior level. In thes e cases, the program evaluation report should specify the program offering and design conditions that lead to the measurelevel savings and the measure us e conditions that affect the savings. The purpose of this requirement is to be able to update the DEER database estimates by changing current estimates, as new data are developed and a dd new measure classifications to the DEEREvaluators’ Protocols Reporting

CPUC 187 TecMarket Works Team database when it is apparent that the progr am design and operational conditions affect the level of energy and demand savings.

9. Measurement reliability metrics. Results and all measurement reliability information must be reported at the program level, program group level and for any program component or delivery mechanism with a designated separate le vel of rigor or as de signed in the approved evaluation plan. In addition, the following data reliability metrics should be reported for the energy impact estimates provide d in the evaluation report.

a. Precision level at the 90% confidence level of the direct participation energy savings (kWh/MWh);

b. Precision level at the 90% confidence level of the participant spillover energy impacts (kWh/MWh) (if available separately given the methodology selected);

c. Precision level at the 90% confidence level of the direct demand energy impact (kW/MW);

d. Precision level at the 90% confidence level of the participant spillover demand impacts (kW/MW) (if available separa tely given the methodology selected);

e. Coefficient of variation (CV) or st andard deviations (S D) and means on the realization rate(s) for the program’s en ergy effects and for all strata in any stratified sampling effort; and

f. P values for all energy impact estimates (kW, kWh, therms).

10. Savings comparison . The report should includ e a presentation and discussion of the CPUC approved program goals compared to the estim ated realized savings from the evaluation findings (this should be expanded in the Appendix described below).

11. Appendix C . Appendix C should present, assess and di scuss the similarities and differences between Administrator savings assumptions and projections, and the results of the evaluation findings. This discussion should identify wh at assumptions were confirmed and not confirmed, and identify recommended changes to the assumptions that Administrators use to project savings.

12. Appendix D. Appendix D should present the weather data used to con duct the evaluation, including the heating and cooling degree-days used in the study, if any.

Note: See end of Reporting Protoc ol chapter for examples of energy impact reporting tables.

Measurement and Verification (M&V)

1. M&V plan and reporting requirements . For impact evaluati ons that are supported by measurement and verification (M&V) efforts, the evaluation report should present the program-specific M&V plan in enough detail th at the plan can be replicated. The plan
should describe and/or discuss:

a. How the M&V samples were identified and selected;Evaluators’ Protocols Reporting

CPUC 188 TecMarket Works Team b. How the M&V activities were used to support the impact assessment; c. Any disagreement between the sampli ng plan and the sampling approach used,
and how the difference influences the reliability of the study findings; d. Sampling and measurement bias issues a nd how these biases can be expected to influence the impact estimates; e. How the biases were controlled or mitigated in the M&V efforts and what statistical or measurement approaches we re used to adjust the M&V data to inform the impact estimates; and
f. How the M&V results were used to estimate net program energy impacts.

Justification for the identification and selection of the baseline is required. An assessment
and discussion of the baseline se lected and its consistency of use for gross and net impacts
must be included.

Site-specific M&V plans prepared during the c ourse of the study shall be provided in an Appendix to the impact evaluation report. Th e site-specific M&V plan shall include all topics specified in the M&V Pr otocol, including assumptions used for stipulated parameters, the source of the assumptions and uncertainti es associated with the M&V study results. Measure-level M&V results shall be reported according to the applicable DEER-compatible format listing in Appendix A of the M&V Protocol. Energy and peak demand savings resulting from weather dependent measures shall be reported unde r weather conditions prevailing during the course of the M&V project. These weathe r conditions shall be reported along with the energy and peak demand impact information. The impacts shall be
normalized to standard weather conditions cons istent with the CEC CTZ long-term average
weather conditions for the climate zone in which the site is located.

2. M&V analysis database. The M&V analysis database(s) will be provided to the CPUC-ED upon delivery of the evaluation report. Site -specific M&V results shall be reported electronically according to database formats es tablished by the Joint Staff compatible with
EEGA and DEER databases. Field data shall be supplied in a non-proprietary format, such as ASCII text, Microsoft Excel, Microsoft Acces s, dBase or XML for inclusion in an M&V data warehouse. Proprietary databases are not acceptable deliverable formats. Building
characteristics data collecte d during on-site surveys shall be reported according to the International Alliance for Interoperability (I AI) Industry Foundation Class (IFC), ifcXML or aecXML formats; Green Building XML (gbXML) fo rmat or other electronic data formats as designated by the Joint Staff. The Joint Staff w ill establish procedures to submit, receive and store M&V database(s) within a data repository. Because the databases will, by their very nature consist of customer-specific information, they will be secured and safeguarded against public release. Evaluation contractors will be informed of these instructions during the evaluation planning process.

Emerging Technology Program Evaluations The Emerging Technology Program Evaluation will be reported consistent with the requirements for all reports described in this Protocol under Common Evaluation Re porting Requirements (above). In addition, the following elements shoul d be included in the evaluation reports under the Methods heading.Evaluators’ Protocols Reporting

CPUC 189 TecMarket Works Team • Program Theory and Logic Model

• Goal Verification

• Aggregate-Level Analysis

• Implementation Analysis

• Measure Tracking

• Detailed Analysis of Key Performance Indicators

• Peer Review

• Target Audience Surveys

These presentations must be provided in enough detail that the differe nces (if any) in the methodological approach across di fferent technologies and uti lities can be understood by the reader. Finally, one must describe the approach for integrating the st udy results so that the overall performance of the ETP can be assessed.

The Reporting Protocols includes a requirement that all evaluation reports include a presentation of the detailed study findings. This presentation must be provid ed in enough detail that the different results or findings (i f any) can be understood for each technology assessment covered in the study. The report should present the results of each of the required eight components
contained in the ETP Protocol. Re ports will be provided consistent with the Reporting Protocol.

Codes and Standards Program Evaluations The Codes and Standards Program Evaluation will be reported consistent with the requirements for all reports described above ( Common Evaluation Reporting Re quirements) and shall also present the following information.

1. Change theories. The report should present each of the code or standard change theories in an appendix to allow the reader to unde rstand the theory behind the change achieved. The report should include a brief summary of the change theories in th e text of the report.

2. Change timelines. The report should present a timeline associated with the program’s efforts employed to influence changes for each code or standard change influenced by the program. The timeline should begin with th e time at which the code targeting and
selection effort was launched and end with the code adoption date. The code adoption date should be followed by the date that the change takes effect.

3. Overview of the program activiti es that caused the change. The report should provide
a discussion of the activities a nd events that are wholly or in part responsible for the program-induced changes.

4. Summary code or standard changes. The report should present a summary of the change to the code or standard caused by the actions of the program. The summary should be detailed enough for the reader to understand the change that occurred and the significance of the change to th e level of savings predicted.

5. Jurisdictions. The report should discuss the jurisdic tions covered and not covered by the changes for each change included in the study.Evaluators’ Protocols Reporting

CPUC 190 TecMarket Works Team 6. Listing of meetings, events , activities and documents. The report should present a listing of each of the meetings, events, and activities attended or monitored by the evaluation contractor to suppor t the evaluation effort and a listing of the documents reviewed to support the study.

7. Interviewees. The report should provided the titl es of all interviewees providing information used in the analysis. The contractor should report the names of the interviewees in an accompanying memora ndum, but not place the names of these individuals in the final public document. Note: individual interview results should be
treated as confid ential information.

8. Pre-change penetration rates. The report should present the pre-change technology adoption or penetration rates reported by the program.

9. Naturally occurring market adoption. The report should describe the approach for estimating the naturally occurring code and st andard changes and the results of applying the approach for each change.

10. Attribution approach and results. The report should discuss the program attribution analysis approach and results of that approach for each change covered in the study.

11. Gross market-level energy impacts. The report should present the approach for estimating the gross market-level energy imp acts for each change and provide the results of that analysis.

12. Net market-level energy impacts. The reports should presen t the net energy impact adjustment approach and the results of that approach so that the reader can understand the influence of each adjustment on the resulti ng net savings. The contractor should report net savings for each change and report the resulting net savings on the reporting spreadsheets (see Sample Reporting Tables).

Market Effects Evaluations The reporting for the market effects evalua tion must include the following information.

1. Market theory integrated with program theory. The report should clearly present and describe the market theory and, if constructe d, the market logic model. The program and market theory should be integrat ed so that the anticipated ne t market effects (those market effects induced by program interventions) can be more readily perceived within the context of the market theory. This will provide a more comprehensive framework from which to
conduct the market effects evaluation. The mark et theory and program theory(ies) should provide the following information:

a. The market theory should be described in detail, in cluding how the markets operates, its

structure and scope, and how the various energy efficiency interventions are expected to change the market. The market theory shoul d include how other mark et actors, activities and interventions are functioning to change th e market that may work in sync or in opposition to the energy efficiency programs. The market theory should present a comprehensive view of how the market opera tes and address how, when, where, why and
under what conditions market e ffects are expected to occur. The report should identifyEvaluators’ Protocols Reporting

CPUC 191 TecMarket Works Team the specific changes in the market that can be observed and measured if the market is being changed. The market theory should de scribe the individual questions that were
asked and the indicators or metrics monitore d to assess when, how and to what degree the market is being changed and it should iden tify key market conditions that influence market change and the rates of change.

b. Program theory. If one or more of the programs expected to cause changes to the operations of a market is desi gned specifically to cause a ma rket effect (change in the operations of a market defined for the evalua tion), the report should present and describe the program theory(ies). Th e evaluation report should pres ent and discuss the program theory used by program managers to structur e their change efforts and explain how the program theory was used to focus the evalua tion efforts. If a program logic model is developed it should be presented. The progr am theory should present how the program’s operations lead to observable and measurable market effects. It should show the resources placed into the market, how they are placed, the resulting planned activities and the anticipated outcomes (the end effects) fr om those activities. The program theory should identify the key market metrics or meas urement points that are expected to change as a result of the program’s efforts and actions.

2. Assessment of gross measure or behavior change. The report should present and describe how the gross level of m easure and/or behavior change in the market is being measured and confirmed through data collection, change measurem ents or change verification efforts. The report should identify any primary or secondary data used to estimate baseline or current
condition measure or behavior use status. The report should descri be how both baseline market conditions and current conditions are qua ntified and how gross measure or behavioruse conditions are being estimat ed. The report should discus s the reliability of these
estimation methods and the various threats to the validity and accuracy of the estimation approach. The report should pres ent the results of a Monte Carl o or other risk assessment approach that examines the difference in repo rt conclusions that would occur if the key assumptions in the establishment of the gro ss measure or behavior change vary within reasonable levels of variance. This activity s hould result in the assi gnment of gross measure use or behavior change conditions that have resulted in the ma rket as a result of all market effects, including program induced and non-progra m induced effects, and the presentation of the degree of variance that c ould be expected within those measurement conditions.

3. Gross and net market change attribution assignments. The report should explain the rationale behind the study’s approach for identifying and allocating causal actions and activities across the market change metrics and change indicators and identify how net program-induced market change will be identifie d. The report should present the sources of market change and describe how allocation of the cause of the change is being proportioned across the various change agents (program and n on-program influenced), so that all observed changes in the market are assigned to one or more reasons for the observed change. The report should present and discuss the proporti oning approach and any data weighting or assignment systems used, and justify why th e assignment approach is reliable and representative of how the market works. The report should discuss any inconsistencies between the allocation approach and th e market theory discussed earlier.Evaluators’ Protocols Reporting

CPUC 192 TecMarket Works Team 4. Net market change . The report should present the re sults of applying the attribution assessment with the gross measure or behavior change assessment (previous two reporting activities) to identify the net program-induced measure or behavior changes and to identify the programs or program even ts that are included in the program-related attribution assignments. This assessment will identify the proportion of the market change that is caused by the program(s).

5. Assignment of energy impacts . The report is to present th e results of the assignment of energy impacts resulting from applying energy and demand impacts associated with the net measure changes in the market that are cause d by the program’s efforts. The accepted practice for this assignment is to use th e energy and demand savings for the covered technologies or behaviors reported in the latest DEER update. For measures not included in the DEER update, the evaluation will report th e best engineering assessment estimation approach for the technologies or behaviors not included in the DEER da tabase as guided by the results of the most recent evaluations of those technologies, if any, with appropriate references and justification for their applicabil ity to this analysis. If a net effects modeling approach is used the steps of the process must be clearly described in a manner consistent to permit the analysis to be repeated by other researchers.

Process Evaluations The process evaluation report sha ll include the following reporti ng requirements in addition to the common evaluation reporting re quirements presented earlier:

1. Detailed program description. While all evaluation reports are to have a description of the program(s) covered in the evaluations, the pro cess evaluation report must present a detailed operational description of the program that focuses on the program components being evaluated. Use of a program fl ow model is highly recommended. The reader of the report must be able to understand the operations of the program being evaluated in significant enough detail that they can understand the componen ts of the program that would be affected by the program change recommendations.

2. Program theory. The process evaluation should in clude a presentation of the program theory. The program theory should, when possibl e, be the theory developed or approved by
the Administrators. If the Administrators ha ve not developed a program theory, they should be provided with the opportunity to develop the theory for inclus ion in the evaluation report. If the detailed program theory is not available or cannot be provided in time for the evaluation report due date, the evaluator should include a summary program theory built from the evaluation team’s program knowledge. This theory does not have to be approved or reviewed by the Administrator to be included in the evaluatio n report. The purpose of this requirement is to have a complete program de scription in the evaluation report and provide the Administrators the opportunity to provide the included program theory, but not to burden the Administrator with the development of the pr ogram theory or logic models if they have not already been developed. If the evaluation co ntractor develops the program theory or the associated logic model, it shoul d be noted as such and complete enough for the reader toEvaluators’ Protocols Reporting

CPUC 193 TecMarket Works Team understand the environment in which the program recommendations are to be placed, but does not need to be a finely detailed program theory or logic model.

3. Support for recommended program changes . While all evaluation re ports are expected to have a section on recommended program change s, identifying these recommendations is one of the primary purposes of the process evaluation report. All recommendations need to be adequately supported, per the Protocol requ irements. Each recommendation should be included in the Executive Summary and then pr esented in the Findings text along with the analysis conducted and the theoretical basi s for making the recommendation. The Findings
section should include a desc ription on how the recommendati on is expected to help the program, including the expected effect implementing the change will have on the operations of the program. The Findings section shoul d include a discussion on how the recommended change can be made, who should be responsible for making the ch ange and the expected cost and benefits of the change. If the informati on to conduct a cost-benef it forecast/prediction for the recommended changes is collected as pa rt of the approved evaluation plan, the report
should include a cost-benefit assessment of th e recommendation so that the cost of the change can be compared to the expected benefits.

4. Detailed presentation of findings. A detailed presentation of the findings from the study is required. The Findings should convey the cond itions of the program being evaluated and should be presented in enough detail that any re ader can understand them and the associated implications to the cost-effective operations of the program. (See 3 above for more details on content requirements of the Findings section.)

Effective Useful Life Evaluations The Effective Useful Life Evaluation will be reported consistent with the requirements for all reports described above ( Common Evaluation Reporting Requirem ents) and shall also present the following information.

All EUL evaluations are expected to assess and discuss the differences between (a) the ex-ante EUL estimates from DEER or as otherwise appr oved by the Joint Staff a nd (b) the ex-post EUL estimates produced by the EUL evaluation study(ies). To the extent that the data gathered and evaluation analyses conducted can explain the causes for thes e differences, this must be presented and discussed. The evaluation report sh ould note situations in which explanations are not possible due to lack of suff icient data or problems with in terpretation. The EUL evaluation report must also include a recommendation of the EUL for the measure and delivery strategy/application that should be used for future program planning. This recommendation may take the form of recommending the replacement of a DEER EUL or the establishment of a new DEER category.

The EUL studies are also required to report the fi ndings from the most recent degradation studies related to the EUL study measures /applications, so that the EUL report is a depository for all current persistence studies for the study measures/a pplications. This will assist Joint Staff and future evaluators in finding all relevant persistence information for a measure/application in one
location.Evaluators’ Protocols Reporting

CPUC 194 TecMarket Works Team All reporting under the Effective Useful Life Protocol should include the following:

11. Cover page containing the measures and delivery strategies or applicat ions included in the retention evaluation, program name s in the portfolios over the last 5 years that include these, program administrators for these programs a nd their program tracking number(s), evaluation contractor, and the date of evaluation plan.

12. Table of Contents.

13. High-level summary overview of the measures and delivery strategies or applications included in the evaluation, the programs affected, and the evaluation efforts.

14. Presentation of the evaluation goals and resear chable issues addresse d in the evaluation.

15. Description of how the evalua tion addresses the researchable issues, including a description of the evaluation priorities and the use of assi gned rigor levels to a ddress these priorities.

16. Detailed description of the data collection and an alysis methodology.

17. Current and prior retention results for select ed measures given delivery strategy/application and their precision levels at a 90% confidence interval.

18. Retention, degradation, and EUL findings as is appropriate for the study assigned.

19. A discussion of the reliability assessment to be conducted, including a discussion of the expected threats to validity, sources of bias, and a short de scription of the approaches planned to reduce threats, bias, and uncertainty.

20. Contact information for the evaluation manager, including ad dress, telephone numbers, fax
number and e-mail address.

In addition to the above requirements, retention studies must also include the following:

2. Description of initial and final sa mple of measures still surviving.

3. Description of any findings on factors leading to the highe r or lower retention rates.

4. Description of removal reasons , their distribution, and potentia l issues created by different removal reasons and the research design and func tional forms that should be investigated in
future EUL studies for these measures.

In addition to the overall EUL study reporting requirements, degradation studies must also include the following:

1. Describe any findings on factor s leading to the relative de gradation rates and absolute degradation rates, if available.

2. Describe the impact of de gradation on energy savings

In addition to the overall EUL study reporting requirements, EUL analysis studies must also include the following:

1. Specific equations for survival functions and estimated precision of curve fit.Evaluators’ Protocols Reporting

CPUC 195 TecMarket Works Team 2. Analysis of the ex-post EUL compared to th e ex-ante EUL and comparison of to the methods and results from any prior retention, degradatio n, or EUL studies available for that measure (to include comparisons by deliv ery strategy and application). 3. Recommended EUL for the measure and delivery strategy/application that should be used for future program planning.

Additional Information

1. All evaluation reports are public property and are owned by the State of California.

2. No reports or other delivera bles will contain information that allows examiners of the final delivered reports and databases to be able to identify individual residential or
non-residential customers or their ener gy consumption, energy demand or energy costs. Individual customer or participant in formation is to be treated as confidential information and protected by confidentiality agreements. Customer-specific information will be safeguarded from public access.

3. Customer information developed in the ev aluation efforts or used to support the
evaluation efforts will be maintained for a limited period of time consistent with the
needs of the evaluation efforts and to s upport time-series or time-sensitive analysis, but will not be indefinably maintained. All customer-specific information will be maintained and protected from disclosure fo r as long as there is an evaluation plan covering the use of the data to support an evaluation effort. Once there is no evaluation plan covering the use of the cu stomer-specific data, it will be deleted or discarded in the following ways:

a. Electronic data files will be dele ted from electronic storage systems;

b. Hard copy data files will be shredde d and recycled or discarded if they cannot be recycled;

c. Electronic data file medium (e .g., DVDs, CD, electronic tape) will be shredded and recycled or discarde d if it cannot be recycled; and

d. Other materials containing customer-s pecific information will be rendered unreadable and recycled or discarded.

4. The CPUC-ED will develop a data arch ive plan for the housing, maintenance, supervision and protection of evaluation-related data. Th e data archival plan shall support the segregation of data by the t ype of the data stored (e.g., program participation data, market description information, customer metered data and evaluation analysis data). These examples are provided to be exemplary only and are not intended to identify or define th e categories within the archive plan.



### Sample Reporting Tables

The following section of the Protocols provides the reporting tables that are to be completed by the evaluation contractors and provided in hard copy format in an appendix, and electronic format with the evaluation reports. A more comple te set of tables will be provided by the Joint
Staff in Microsoft Excel formats for evaluati on contractors to use to report study resultsEvaluators’ Protocols Reporting

CPUC 196 TecMarket Works Team following the approval and distribution of the Re porting Protocol. Energy impacts should be reported for each program. In addition, energy impact s should be reported as specified in each approved evaluation plan. In some cases this will require energy impact reporting by delivery strategy or other approach. It is important that the evaluation study reports the total savings for the programs being evaluated so th at full credit for the energy sa vings impacts can be recorded for each program.Evaluators’ Protocols ReportingCPUC 197 TecMarket Works TeamExample Megawatt Reporting Table(1) – Program Wide MW savings across all climate thermal zonesThis table reports the total program participation megawatts saved across all climate zones inwhich the program was offered.If theprogram covers more than one IOU territory, tables for each IOU should also be reported. Separate MW savings tables should beprepared for the months of June, July, August and September.See Reporting Protocol for additional information. Table reportstheAdministrator-forecasted MW savings (gross and net), the Administrator-reported MW savings (gross and net) and the evaluationreported gross, net of free-riders and spillover MW savings.Administratorforecasted exante MW(CPUCapproved)Administratorforecasted exante MW(CPUCapproved)Administratorreported exante MWAdministratorreported exanti MWGrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeriders1 20062 20073 20084 20095 20106 20117 20128 20139 201410 201511 201612 201713 201814 201915 202016 202117 202218 202319 202420 2025Noon-1PM 1PM-2PM 2PM-3PM 3PM-4PMPG&E Residential Direct Install Widget ProgramProgram ID #: 1234-06Climate Zone: All Program Covered Climate ZonesProgramYearAdd more rows if approved in evaluation planCalendarYearGrossNet ofFreeriders GrossNet ofFreeridersSavings TimelineAll Program MeasuresTotal Program MW SavingsJune of 2006 Evaluation Projected Demand Impacts(MW average weekday across periods)Evaluators’ Protocols ReportingCPUC 198 TecMarket Works TeamExample MW Reporting Table (2) – ProgramMW savings for a specific climate zone.This table provides an example of a CEC thermal climate zone(CTZ)-specific MW savings reporting table. (CEC CTZ-1).The savings reported in this table are only those that occur within thermal climate zone (CTZ) 1. These tables are for reportingprogram MW impacts in specific CTZs for programs offered and with participants in more than one CTZ.Administratorforecasted exante MW(CPUCapproved)Administratorforecasted exante MW(CPUCapproved)Administratorreported exante MWAdministratorreported exanti MWGrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover1 20062 20073 20084 20095 20106 20117 20128 20139 201410 201511 201612 201713 201814 201915 202016 202117 202218 202319 202420 20252PM-3PM 3PM-4PMAdd more rows if approved in evaluation planAll Program MeasuresJune of 2006 Evaluation Projected Demand Impacts(MW average weekday across periods)GrossNet ofFreeridersNoon-1PM 1PM-2PMProgramYearCalendarYearGrossNet ofFreeridersPG&E Residential Direct Install Widget ProgramProgram ID #: 1234-06Climate Zone: CEC CTZ 1Savings TimelineTotal Program MW SavingsEvaluators’ Protocols ReportingCPUC 199 TecMarket Works TeamExample MW Reporting Table (3) – Measure-specific, program wide MW – all climate zonesThis table provides an example of the reporting requirements for individual measures assessed inthe evaluationstudy if the evaluationplan is approved to address measure assessments. Itpresents the measure-specific program savings for “Measure X” for all climatezones. In this example the measure-specific impacts are provided atthe program level, not at the climate zone level. In some cases, ifthe evaluation plan specifies it, impacts may be required to be reported at the climate zone level.Administratorforecastedex-ante MW(CPUCapproved)Administratorforecastedex-ante MW(CPUCapproved)Administratorreported exante MWAdministratorreported exanti MWGrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover GrossNet ofFreeridersParticipantSpillover1 20062 20073 20084 20095 20106 20117 20128 20139 201410 201511 201612 201713 201814 201915 202016 202117 202218 202319 202420 20252PM-3PM 3PM-4PMAdd more rows if approved in evaluation planMeasure XJune of 2006 Evaluation Projected Demand Impacts(MW average weekday across periods)GrossNet ofFreeridersNoon-1PM 1PM-2PMProgramYearCalendarYearGrossNet ofFreeridersPG&E Residential Direct Install Widget ProgramProgram ID #: 1234-06Climate Zone: All Program Covered Climate ZonesSavings TimelineMeasure Specific MW SavingsEvaluators’ Protocols ReportingCPUC 200 TecMarket Works TeamExample Megawatt hours Reporting Table(4) – Program Wide annual MWh impactsThis table provides an example of the tableto be used for reporting program wide MWhsavings. In this case the table is for allprogram measures across all CEC CTZ covered by the sample program.Gross Net of Freeriders Goss Net of Freeriders Gross FreeridersParticipantSpillover Net of FreeridersProgram Year Calendar Year1 20062 20073 20084 20095 20106 20117 20128 20139 201410 201511 201612 201713 201814 201915 202016 202117 202218 202319 202420 2025Add more rows if approved in the evaluation plan.Program ID#: 1234-06Savings TimelineTotal Program MWh SavingsAdministratorforecasted ex-anteMWh savings(CPUC approved)Administratorforecasted ex-anteMWh savings (CPUCapproved)Administratorreported ex-anteMWh savingsAdministratorreported ex-antiMWh savingsEvaluationEstimatedAnnual MWhsavingsEvaluationEstimatedAnnual MWhsavingsEvaluationEstimatedAnnual MWhsavingsEvaluationEstimatedAnnual MWhsavingsPG&E Residential Direct Install Widget ProgramAll Program MeasuresClimate Zone: All Program Covered Climate ZonesEvaluators’ Protocols ReportingCPUC 201 TecMarket Works TeamExample Natural Gas Reporting Table (5) – Programwide annual Therms of natural gas savingsThis sample table is used to report the total program energy savings in therms of natural gas for a sample program.Gross Net of Freeriders Goss Net of Freeriders Gross FreeridersParticipantSpillover Net of FreeridersProgramYear Calendar Year1 20062 20073 20084 20095 20106 20117 20128 20139 201410 201511 201612 201713 201814 201915 202016 202117 202218 202319 202420 2025Savings TimelineAdd more rows if approved in the evaluation plan.Total Program Therm SavingsAdministratorforecasted exante thermsavings (CPUCapproved)Administratorforecasted exante thermsavings (CPUCapproved)Administratorreported ex-antetherm savingsAdministratorreported ex-antitherm savingsEvaluationestimatedtherm savingsEvaluationestimatedtherm savingsEvaluationEstimatedtherm savingsPG&E Residential Direct Install Widget ProgramProgram ID#: 1234-06All Program MeasuresClimate Zone: All Program Covered Climate ZonesEvaluationEstimatedtherm savingsEvaluators’ Protocols ReportingCPUC 202 TecMarket Works TeamExample Measure Count Reporting Table (6) – Units installed by program yearThis table provides an example of the tableto be used to report the number of measures the Administrator projects to be installed overthe program cycle and the number ofmeasures estimated via evaluation efforts tobe installed. As theevaluation contractor conductsthe evaluation, the results of interviews,surveys, monitoring and verification efforts will allow the contractor to estimate the numberof units to be installed for each of the measures offered throughthe program. This table is to be provided in each of the impactevaluation reports to the extent it can becompleted at the time of the evaluation effort. In addition, after the last year ofthe programcycle, the evaluation contractor is to complete this table foreach year in the program cycle,summing the total evaluation estimatedinstalls for the program cycle and assess the difference betweenthe Administrator-projected installs and the evaluation-estimatedinstalls.PG&E Residential Direct Install Widget ProgramProgram ID#: 1234-05(use CPUC approved standard measure descriptions)Measure AMeasure BMeasure CMeasure DMeasure EMeasure FMeasure GMeasure HMeasure IMeasure JMeasure KMeasure LMeasure MMeasure NEtc.Etc.Add more lines if needed.Measure Counts ReportingCPUC-ED Approved Units of MeasureTotal 2006-2008programadministratorprojected units tobe installed2006 Evaluationestimated totalunits installed2006 Evaluationestimatedaverage unitsper participant2007 Evaluationestimated totalunits installed2007 Evaluationestimatedaverage unitsper participant2008 Evaluationestimated totalunits installed2008 Evaluationestimatedaverage unitsper participantTotal 2006-2008 Evaluationestimated unitsinstalledEvaluators’ Protocols Protocols Reporting

CPUC 203 TecMarket Works Team The above tables are examples of the energy imp act reporting tables to be provided from the evaluation efforts. A final, more comprehensive set of tables will be developed by the Joint Staff and distributed by the CPUC-ED once the Protocols are approved. The tables will be developed in Microsoft Excel so that they can be populated by the evaluation contractors and reported in the evaluation reports and de livered as separate Excel files.Evaluators’ Protocols Evaluation Support Information

CPUC 204 TecMarket Works TeamEvaluators’ Protocols Evaluation Support Information

CPUC 205 TecMarket Works Team 

## Evaluation Support Information Needed From
Administrators

This section of the Protocols presents the types of information that the Administrators will need to provide in support of the evaluation efforts. Not all of the information listed below will be needed for all evaluations. Rather, each evaluation will need a somewhat different set of information from the types of information presente d in this chapter. In some specific cases the evaluation contractor may need to request inform ation that is not detailed below in order to
conduct an evaluation. As noted in the above in dividual Protocols, each evaluation plan will describe the type of information that the eval uation contractors need to complete their study.
This allows the Administrators to have an advan ced notice of the type of information that will be requested via a formal data request and an unders tanding of how the data is to be used in the evaluation study. In requesting data from the Ad ministrators the evalua tion contractors should
first determine if the data needed is availabl e in the CPUC-ED’s progr am tracking and reporting database. If the required information is av ailable from the CPUC-ED’s database, evaluation
contractors should obtain it di rectly from the CPUC-ED.

The information needed from the Administrators is considered basic program and participant tracking information. However the Joint Staff rea lizes that there may be circumstances when the requested information is not avai lable from the Administrators or may be not be available in
electronic formats. If an Administrator is una ble to provide the requested information for a specific program, the Administrator will advise the organization requesting the information and
the Joint Staff that the informati on is not available and explain th e reasons why. If the requested information is available only in hard copy records, the Administra tor will inform the requesting evaluation contractor and the Joint Staff and they will come to an agreement on the what information should be provided and in what format.

It is expected that the Administrators will re spond to all evaluation data requests within 30 working days by providing as much of the reque sted information covered in this Reporting Protocol as possible, in form ats agreed upon by the Administra tors and the evaluation team leads. It is expected that inform ation not covered in this Reporti ng Protocol, but that is necessary to conduct the evaluation in accordance with appr oved evaluation plans, will also be provided within 30 working days of the r eceipt of a data request. If th is timeline cannot be met by the Administrators, the Administrators will provid e the requesting organization and the CPUC-ED with an explanation of why the timeline cannot be met and work with the CPUC-ED and the
evaluation contractor to establis h a mutually agreed upon delivery timeline. The Administrator will inform the CPUC-ED’s evaluation manager wh en the requested data has been provided to
the evaluation contractor.

It is the Administrator’s respons ibility to establish and maintain program-tracking systems that are capable of supporting the evalua tion efforts and of meeting the re quirements specified in this
Protocol. Joint Staff

All evaluation-related data request s will be provided to the appropriate Administrator(s) and the
CPUC-ED at the same time. No evaluation contr actor will contact customers or participants forEvaluators’ Protocols Evaluation Support Information

CPUC 206 TecMarket Works Team evaluation information without the approval of the CPUC-ED a nd at least 15 days advanced notification provided to the appr opriate Administrator(s).

The Joint Staff understands that in some cases th e data requested may not be available from the
Administrator and it will the re sponsibility of the Administrator, the CPUC-ED, the CEC or the
evaluation contractor to collect th e needed data. When the needed data is not available from the Administrators, the evaluation pl an should address how the data will be collected. Again, the
evaluation contractors should c onsider the information availa ble in the CPUC-ED’s program tracking database to determine what data needs to be collected from the Administrators and what data can be collected directly from th e CPUC-ED’s program tracking database.

The evaluation contractor will limit all data requests to information critical to the success of the evaluation. Information requests will be for e nough data to successfully conduct the study at the needed population sample sizes, but will not over-request sample points beyond what is needed to conduct the evaluation. Evaluation data requests will need to plan for sample erosion due to a wide variety of conditions.

All measure information must be reported by the Administrators and the evaluation contractor by the CPUC-ED-approved measure desc ription list so that identical measures are described using
the same terms and definitions. If no list exists at the time of this Protocol, the Joint Staff will develop a uniform measure descrip tion list that all parties will us e (IOUs, contractors, third-party providers, CPUC-ED, CEC and other stakeholders ) and distribute that list via the CPUC-ED’s web page and the energy efficiency list serve. The descriptions will include an official identifier
and an abbreviated term that can be used in tr acking systems, tables and charts where space is
limited.

Most Administrators will find th at the reporting requirements are consistent with the type of information that evaluation contractors have requ ested in the past. However, there are some examples of information detailed below that ma y not be routinely maintained for each program or updated on a regular basis. As a result, there may be some specific parts of a data request that the Administrators will be unable to provide. In general, the Administrato rs are responsible for providing the Administrator-coll ected or implementer-collected information requested by the
evaluation contractors to allow the evaluations to be conducted consistent with the evaluation plans approved by the Joint Staff or by the CPUC-ED.

The following data should be readily available from the Ad ministrators.

Program Information

1. Full program descriptions, in cluding operational or proced ures manuals and activities descriptions and description of implementation territories;

2. Detailed descriptions of tracking syst em and tracking system operations, including data dictionaries;

3. Program management and staff names, titles, work locations, phone numbers, fax numbers, e-mail addresses;Evaluators’ Protocols Evaluation Support Information

CPUC 207 TecMarket Works Team 4. Program theories and associated logic models if developed. If not developed a statement that they have not been develope d with a projected date of delivery of the completed theories and logic models;

5. Market operations theories describing the operations of the markets in which the program operates and, if availa ble, a description of how th e program is to change the operations of the market;

6. A description of the size of the market targeted by the program, and a description of
the baseline conditions at the measure/beha vior level and a discussion of how the program is expected to change baseline measure/behavior conditions, if available;

7. A description of the pre-program technical potential at the m easure/behavior level and a projection of the remaining technical potential at the end of the program cycle, if available; and

8. When the program relies on key market acto rs, trade allies and ot her stakeholders to deliver or support the program in order to reach the energy saving or outreach goals, the Administrator should provi de a listing, description of and contact information for these individuals/organizations.

Participant Data For the purposes of this Protocol a participant is defined as an individual or an organization that receives a program service or financial incentiv e. For most programs, pa rticipants are clearly defined in the program tracking systems. Howeve r, there are times when a participant is not clearly defined or is not easily identified. The CPUC-ED expect s that the Administrators will focus efforts on collecting participant information to the extent possible and practical for various types of programs or program services. Particip ants in resource programs are generally easy to identify as they directly receive a service or a financial incentive. Participants in other programs, such as marketing and outreach programs can be harder to identify and report. This Protocol does not act to require all programs to identify all participants. However when participant information is collected by the Administrators or their subcontractor, much of this information
will be of value to the evaluation efforts. It is to the responsibility of the Administrators to work
with their subcontractors to assure that when possible and practical th e following information
should be collected and maintained.

The following participant data should be availabl e in electronic form with supporting database
dictionaries to the evalua tion teams on request.

Non-residential program data requests for end-user focused programs
1. Name of program(s) or program component(s);
2. Name of firms participating in program or program component;
3. Service turn on date;
4. Primary and secondary NAIC codes associated with the participan ts if available;
5. Extent to which customer is a repeat pa rticipant or a participant in other programs over the previous five years, if available or accessible;
6. Pre-participation measure and measure-us e information, descriptions and conditions;
7. Address(es) of the participating firm s or key participation decision makers;Evaluators’ Protocols Evaluation Support Information

CPUC 208 TecMarket Works Team 8. Address(es) where program-related action is taken or for the services received;
9. Listing or description of actions taken or services received for each location by measure and end-use according to sta ndard measure and end-use definitions established herein. These lists and descrip tions should, to the extent possible, be standardized so that all Administrators use the same term for the same measure;
10. Individual participation contact in formation for each location to include:

a. First and last name;

b. Address;

c. Phone number;
d. Fax number (if collected); and
e. E-mail address (if collected).
11. Dates of key action/activity/installation step s associated with program participation:
a. Program enrollment date(s);
b. Rebate or incentive payment date(s);

c. Measure install dates;
d. Date of training received; and
e. Post-installation measure inspection dates.
12. Financial assistance amounts paid to pa rticipant by measure or action taken;
13. Project description information;
14. Estimated savings for actions taken;
15. Summary characteristics of building on which actions ar e taken or the operational
environment in which measures are installed if collected;
16. Account and meter numbers and consumpti on histories from utility bills from all
relevant meters for at least twelve mont hs prior to program enrollment date and through to current period. Note: The evaluati on contractor will work with the IOUs to
understand what metered data is availabl e for which types of customers and the formats and time intervals associated with the metered data;

17. Rate classification; and
18. The size and operational characteristics of the market in which the program is to operate including the number of covered technologies operating in the market and their expected normal failure, ch ange-out or replacement rates.

Residential program data request s for end-user focused programs
1. Name of program(s) or program component(s) of th e participation;
2. Type of building or structure associated with the participant or the participation;
3. Pre-participation measure and measure us e information, descriptions and conditions;

4. Service turn on date;
5. Name of individual enrolling in the program or receiving service;
6. Address of the participant;
7. Extent to which customer is a repeat pa rticipant or a participant in other programs over the previous five years, if available or accessible;Evaluators’ Protocols Evaluation Support Information

CPUC 209 TecMarket Works Team 8. Address where action is taken or for the services received;
9. Listing or description of actions taken or services received according to standard measure and end-use definitions;
10. Individual participation c ontact information to include:

a. First and last name;

b. Address;

c. Phone number;
d. Fax number;(if available and collected); and
e. E-mail address (if available and collected).
11. Dates of key action/activity/installation step s associated with program participation:
a. Program enrollment date(s);
b. Rebate or incentive payment date(s);

c. Measure install dates;
d. Date of training received; and
e. Post-installation inspection dates.
12. Financial assistance amounts paid to pa rticipant by measure or action taken;
13. Project description information;
14. Estimated savings for actions taken;

15. Account numbers and meter numbers and cons umption histories from utility bills for
all relevant meters for at least twelve m onths prior to program enrollment date and through to current. Note: The evaluation c ontractor will work with the IOUs to
understand what metered data is availabl e for which types of customers and the formats and time intervals associated with the metered data;

16. Rate classification; and
17. The size and operational characteristics of the market in which the program is to operate including the number of covered technologies operating in the market and their expected normal failure, ch ange-out or replacement rates.

Non-participant or rejecter data for end-user focused programs
1. Description of program services offered to customer;
2. Date of offering or contact;

3. Method of contact;

4. Name of contact;

5. Address of contact;

6. Phone number of c ontact (if known); and
7. E-mail of contact (if known).

Program data for mid-stream and upstream focused programs
1. Name of program(s) or program component(s);
2. Name of firms participating in program or program component; 3. Primary and secondary NAIC codes associated with the participan ts if available;Evaluators’ Protocols Evaluation Support Information

CPUC 210 TecMarket Works Team 4. Extent to which customer is a repeat pa rticipant or a participant in other programs over the previous five years, if available or accessible;
5. Pre participation/measure and measure us e information, descriptions and conditions;
6. Address of the participating firms or key participation decision makers;
7. Address(es) where action is taken or for the services received;
8. Listing or description of actions taken or services received for each location;
9. Individual participation c ontact information to include:
a. First and last name (if known);

b. Address;

c. Phone number;
d. Fax number (if collected); and
e. E-mail address (if collected).
10. Dates of key action/activity/installation step s associated with program participation:
a. Program enrollment date(s);
b. Rebate or incentive payment date(s);
c. Date of training received; and
d. Dates, numbers and type s of material received.
11. Financial assistance amounts paid to participant by action taken;
12. End-user information as is made available to the program;
13. The size and operational characteristics of the market in which the program is to operate including the number of covered technologies operating in the market and their expected normal failure, chan ge-out or replacement rates; and
14. Names and copies of previous evaluations and market research efforts used by the program to plan and structure program offerings and implementation efforts.

Program data for information, educat ion and advertising-focused programs
1. Name of program(s) or program component(s);
2. Target population descripti on, size, source of identifying information and lists of population members used in outreach act ivities. The size and operational
characteristics of the market in which the program is to operate including the number of covered technologies operating in the ma rket and their expected normal failure, change-out or replacement rates;
3. Contact information where individual pa rticipants are identified to include:
a. First and last name of key c ontacts for each location (if known);

b. Address of individual contacts;

c. Phone number of individual contacts;

d. Fax number of individua ls (if collected); and
e. E-mail address of individuals (if collected).
4. Marketing materials by numb ers, types and distribution;
5. Education or Media plan as is appropriate;Evaluators’ Protocols Evaluation Support Information

CPUC 211 TecMarket Works Team 6. Execution records for training held; inform ation venues used; program participation agreements, commitments or other simila r agreements; post-buy analysis; and other documentation of actual output;
7. Records for dates, number, location, target audience and attendance of events held, Web site hits, call-in numbers and rates, reach, frequency, Gross Rating Points, impressions, click through rate, composition, coverage, earned medi a, value of public service announcements, and other tracking and monitoring information the program maintains, as appropriate to the effort and for each wave, campaign and targeted
effort. Include definitions and calculation methods for monitoring statistics used;
8. End-user information available to the program; and
9. Names and copies of previous evaluations and market research efforts used by the program to plan and structure program offerings and implementation efforts.



### Storage and Disposal of Customer In formation Used in the Evaluation

Customer information received to support the evaluation efforts wi ll be maintained for a limited period of time consistent with the needs of the evaluation effort s and to support time-series or time-sensitive analysis, but will not be mainta ined indefinitely. All customer-specific information will be maintained and protected from disclosure for as long as there is an evaluation plan covering the use of the data to support an evaluation effort. Once there is no evaluation plan covering the use of the customer-specific data , it will be deleted or discarded within 3 years in the following ways:

1. Electronic files will be deleted from electronic storage systems;

2. Hard copy files will be shredded and recycl ed or discarded if it cannot be recycled;

3. Electronic medium (e.g., DVDs, CD, electr onic tape) will be sh redded and recycled
or discarded if it ca nnot be recycled; and

4. Other materials containing customer-speci fic information will be rendered unreadable and recycled or discarded.Evaluators’ Protocols Appendix A

CPUC 212 TecMarket Works TeamEvaluators’ Protocols Appendix A

CPUC 213 TecMarket Works Team 

## APPENDIX A. Measure-Level M&V Results Reporting Requirements

Measure-level results from M&V studies shall be reported as unit savings estimates (kWh/unit, kW/unit, therm/unit) normalized in terms consistent with DEER as described below or as amended by CPUC-approved revision of this Appendix.

Table 24. Measure-Level Impact Reporting Requirements

Measure Category Measure Subcategory Normalization
Units Measures included

Nozzle Low pressure nozzles Irrigation
Acre of land Micro irrigation conversion Agricultural

Greenhouse Sq ft of glazing Heat curtain, IR film
Clothes Dryer Efficient dryer Dryer Efficient clothes dryers

Commercial Cooking Equipment Equipment Griddles, fryers, warming cabinets,

steamers Design evaporator

tons Floating head pressure and suction pressure controls Case linear feet Case lighting controls Controls

Motor Case and cooler fan controls

1000 SF of sales

area Refrigerant holdback valves

Design compressor

tons VSD on compressor

Design evaporator

tons Oversized condensers, sub cooling, compressor and condenser change outs Case linear feet Case covers, reach-in conversions, case replacements Motor Efficient evaporator fan motors

Cooler Door closers

Door Anti-sweat heater elimination Equipment

Freezer Door closers Commercial Refrigeration

Maintenance Design evaporator

tons Refrigeration system maintenance

Circulation Pump 1000 SF of building Hot water recirculation pumps and pump
controls

Clothes washer Clothes washer Efficient clothes washers

Dishwasher Dishwasher Efficient dishwashers

Faucet aerators House hold Faucet aerators
Heat pump water
heater Water heater Heat pump water heaters

Efficient water heater
- residential Water heater Storage type water heaters

Efficient water heater
- commercial 1000 SF of building Storage type water heaters Hot Water

Low flow showerhead Showerhead Low flow showerheadsEvaluators’ Protocols Appendix A

CPUC 214 TecMarket Works Team Measure Category Measure Subcategory Normalization
Units Measures included

Pipe wrap Household Hot water pipe insulation and heat traps

Point of use water
heater 1000 SF of building Instantaneous water heaters

Tons Economizer retrofit and repair Controls
1000 SF of building Programmable thermostats, energy management systems, time clocks, space heating hot water and chilled water loop temperature control 1000 SF of building Ventilation rate changes, evaporative coolers, air to air heat recovery, commercial
furnaces and boilers

kBtu/hr of furnace capacity Residential furnaces

Nameplate motor hp Efficient motors in HVAC applications, 3 way to 2 way valve conversions on chilled water and space heating hot water coils, variable frequency drives HVAC

Equipment

Tons High efficiency packaged AC and heat pumps, high efficiency chillers, watersideeconomizers, evaporative ventilation air precoolers Ballast Fixture Dimming ballasts
CFL lamps Lamp Screw-in and hardwire compact fluorescent lamps De-lamp Fixture All interior lighting fixture types
Exit sign Exit sign LED exit signs
Exterior lighting Lamp HID lamps for exterior lighting applications
Linear fluorescent Fixture High effici ency fluorescent lighting fixtures with T-8 or T-5, linear or U-Tube lamps Metal halide Lamp High efficiency metal halide lamps
Occupancy sensor Sensor Occupanc y sensors for interior lighting applications

Photocell Photocell Photocell co ntrols for exterior lighting applications

Time clock Time clock Time clock controls for interior or exterior lighting applications Lighting controls - general kW controlled Other general purpose interior lighting control systems Lighting

LPD reduction kW reduced Efficient lighting design providing reduced lighting power density Copy machine Copy machine Efficient copy machine Interior Plug
Loads Equipment kW reduced Use of efficient office equipment resulting in equipment power density reduction

Motors Motor Efficient non-HVAC motors Miscellaneous
Vending machine Machine Efficient vending machines and vending
machine controllers Pools Pool pump Pump Efficient pool pumps and pool pump
controllersEvaluators’ Protocols Appendix A

CPUC 215 TecMarket Works Team Measure Category Measure Subcategory Normalization
Units Measures included

Refrigerator Refrigerator Effici ent residential refrigerators or refrigerator/freezers Residential Refrigeration

Freezer Freezer Efficient residential freezers
Shell 1000 SF of building Weatherization, air leakage sealing
Fenestration 100 SF of window High performance windows, skylights and glazing systems

Insulation 1000 SF of
insulation Insulation, cool roofs Shell

Equipment 1000 SF of building Whole-house fansEvaluators’ Protocols Glossary of Terms

CPUC 216 TecMarket Works TeamEvaluators’ Protocols Glossary of Terms

CPUC 217 TecMarket Works Team 

## APPENDIX B. Glossary of Terms 141

ACCURACY - An indication of how close a value is to the true value of the quantity in question.
The term could also be used in reference to a m odel or a set of measured data, or to describe a measuring instrument’s capability.

ACHIEVABLE POTENTIAL - The am ount of savings that can be achieved due to specific program designs and delivery approaches, including program funding and measure incentive levels. Achievable potential studies are sometim es referred to as Market Potential studies.

ADMINISTRATOR - A person, company, partnership, corporation, associati on or other entity selected by the CPUC and any subcon tractor that is retained by an aforesaid entity to contract for and administer energy efficiency programs funded in whole or in part from electric or gas Public Goods Charge (PGC) funds. For purposes of implementing PU Code Section 381.1, an “administrator” is any party that receives funding for and implements energy efficiency programs pursuant to PU Code Section 381. Similarly, a person, company or other entity
selected to contract and admi nister energy efficiency programs funded by procurement funds.

ANALYSIS OF COVARIANCE (ANCOVA) MODELS - A type of regression model also
referred to as a “fixed effects” model. This model allows each individu al to act as its own control. The unique effects of the stable, but un measured characteristics of each customer are
their “fixed effects” from which this method takes its name. These fixed effects are held

constant.

ASHRAE - American Society of Heating, Re frigerating and Air- Conditioning Engineers

AUTOCORRELATION - The breakdown in the assumptions that the errors in regression analysis are uncorrelated due to correlation in th e error term across observations in a time-series or cross-series, the erro r in one time period is directly corr elated to the error in another time period or cross-sectional category. First-order serial correlation is when that correlation is with the error in the subsequent/preceding time period. The correlation can be positive or negative.

BASELINE DATA - The measurements and fact s describing facility operations and design during the baseline period. This will include en ergy use or demand and parameters of facility operation that govern energy use or demand.

BASELINE FORECAST - A predic tion of future energy needs wh ich does not take into account the likely effects of new efficiency pr ograms that have not yet been started.

BASELINE MODEL - The set of ar ithmetic factors, equations or data used to describe the relationship between energy use or demand and ot her baseline data. A model may also be a simulation process involving a specified si mulation engine and set of input data.

141 Terms defined as used herein and within the context of energy efficiency evaluation.Evaluators’ Protocols Glossary of Terms

CPUC 218 TecMarket Works Team BASELINE PERIOD - The period of time selected as representative of facility operations before
retrofit.

BEHAVIORAL DEGRADATION FACT OR - A multiplier used to account for time-related change in the energy savings of a high efficiency measure or practice relative to a standard efficiency measure or practice due to changes in behavior in relation to the measure or practice.

BILLING DATA - Has multiple meanings. Metered data obtained from the electric or gas meter used to bill the customer for energy used in a particular billing period. Meters used for this purpose typically conform to regulat ory standards established for eac h customer class. Also used to describe the data representing the bills cust omers receive from the energy provider and also used to describe the customer billing and paymen t streams associated with customer accounts.
This term is used to describe both consum ption and demand, and account billing and payment
information.

BILLING DEMAND - The demand used to calculate th e demand charge cost. This is very often the monthly peak demand of the customer, but it may have a floor of some percentage of the highest monthly peak of the pr evious several months (a dema nd “ratchet”). May have other meanings associated with cust omer account billing practices.

BRITISH THERMAL UNIT (Btu or BTU) - The sta ndard measure of heat energy. It takes one Btu to raise the temperature of one pound of wa ter by one degree Fahrenhe it at sea level. For example, it takes about 1,000 BTUs to make a pot of coffee. One Btu is equivalent to 252 calories, 778 foot-pounds, 1055 joules and 0.293 watt- hours. Note: the abbrev iation is seen as “Btu” or “BTU” interchangeably.

BUILDING COMMISSIONING - Building commissi oning provides documented confirmation that building systems as constructed function in accordance with the intent of the building designers and satisfy the owner’s operational needs.

BUILDING ENERGY EFFICIENCY STANDARDS - Ca lifornia Code of Regulations, Title 24, Part 2, Chapter 2-53; regulating the energy effici ency of buildings constructed in California.

BUILDING ENERGY SIMULATION MODEL - Computer models based on physical engineering principals and/or standards used to estimate ener gy usage and/or savings. These models do not make use of billing or metered data , but usually incorporate site-specific data on customers and physical systems. Building Simulati on Models usually requir e such site-specific data as square footage, weather, surface orient ations, elevations, space volumes, construction materials, equipment use, lighting and build ing occupancy. Building simulation models can usually account for interactive effects between end-uses (e.g. , lighting and HVAC), part-load efficiencies and changes in external and inte rnal heat gains/losses. Examples of building simulation models include ADM2, BLAST and DOE-2.

BUILDING ENVELOPE - The assembly of exteri or partitions of a building that enclose conditioned spaces, through which thermal energy may be transferred to or from the exterior,Evaluators’ Protocols Glossary of Terms

CPUC 219 TecMarket Works Team unconditioned spaces or the ground. (See Californi a Code of Regulations, Title 24, Section 2- 5302.)

CADMAC - See CALIFORNIA DEMAND-SIDE MANAGEMENT MEASUREMENT
ADVISORY COUNCIL.

CALIFORNIA MEASUREMENT ADVISORY COUNCIL (CAL MAC) - An informal committee made up of representatives of th e California IOUs, CPUC, CEC and NRDC. CALMAC provides a forum for the development, implementation, presentation, discussion and review of regional and statewid e market assessment and evaluati on studies for California energy efficiency programs conducted by member orga nizations using Public Goods Charge funds.

CALIFORNIA DEMAND-SIDE MANAGEMENT MEASUREMENT ADVISORY COUNCIL (CADMAC) - An informal committee made up of u tility representatives, the Office of Ratepayer Advocates and the CEC. The purpose of the comm ittee is to: provide a forum for presentations,
discussions and review of De mand Side Management (DSM) program measurement studies underway or completed; to coordinate the de velopment and implement ation of measurement studies common to all or most of the utilities; and to facilitate the development of effective,
state-of-the-art Protocols fo r measuring and evaluating the impacts of DSM programs.

CALIFORNIA ENERGY COMMISSION (CEC) - Th e state agency established by the WarrenAlquist State Energy Resources Conservation an d Development Act in 1974 (Public Resources Code, Sections 25000 et seq.) re sponsible for energy policy. F unding for the CEC’s activities comes from the Energy Resources Program A ccount, Federal Petroleu m Violation Escrow
Account and other sources. The CEC has st atewide power plant si ting, supply and demand forecasting, as well as multiple types of en ergy policy and analysis responsibilities.

CALIFORNIA PUBLIC UTILITIES COMMISSION (CPUC) - A state agency created by
constitutional amendment in 1911 to regulate the rates and services of privately owned utilities and transportation companies. The CPUC is an administrative agency that exercises both legislative and judicial powers; its decisions and orders may be appealed only to the California Supreme Court. The major duties of the CPUC are to regulate privately owned utilities, securing adequate service to the public at rates that are just and reasonable both to customers and shareholders of the utilities; including rates, electricity transmission lines and natural gas pipelines. The CPUC also provides electricit y and natural gas forecasting, and analysis and planning of energy supply and resources. Its headquarters are in San Francisco.

CALMAC – See CALIFORNIA MANAGEMENT MEASUREMENT ADVISORY COUNCIL.

CAPACITY - The amount of electric power for which a generating unit, generating station or other electrical apparatus is rated either by the us er or manufacturer. The term is also used for the total volume of natural ga s that can flow through a pipeline over a given amount of time, considering such factors as compression and pipeline size.

CEC - See CALIFORNIA ENERGY COMMISSION.Evaluators’ Protocols Glossary of Terms

CPUC 220 TecMarket Works Team CHANGE MODEL - A type of bil ling analysis designed to explai n changes in energy usage. This can take the form of having the change in energy consumption (pre versus post) as the dependent variable (e.g., December pre-retrofit us age – December post-retr ofit usage) or having consumption as the dependent variable and pre- retrofit consumption as one of the independent
variables.

CLIMATE THERMAL ZONE (CTZ) – A geographical area in the state that has particular weather patterns. These zones ar e used to determine the type of building standards that are required by law.

CLTD – See COOLING LOAD TEMPERATURE DIFFERENCE.

COEFFICIENT OF VARIATION - Th e sample standard deviation divided by the sample mean (cv = sd/y). See page 320 of the Evaluation Framework

.

COINCIDENT DEMAND - The metered demand of a de vice, circuit or building that occurs at the same time as the peak demand of the building or facility or at the same time as some other peak of interest, such as a utility’s system lo ad. This should properly be expressed so as to indicate the peak of inte rest, e.g., “demand coincident with the bu ilding peak.”

COMMERCIALIZATION - Programs or activities that increase the va lue or decrease the cost of integrating new products or services into the electricity sector.

COMMISSIONING - See BUILDING COMMISSIONING.

COMPARISON GROUP - A group of customers who did not participate in a given program during the program year and who share as many ch aracteristics as possible with the participant

group.

COMPREHENSIVE - A program or project desi gned to achieve all cost-effective energy efficiency activities in individual buildings, usually including multiple energy efficiency

measures.

CONDITIONAL DEMAND ANALYSIS (CDA) - A type of billing analysis in which observed energy consumption is estimated as a function of major end-uses, often portrayed as dummy
variables for their existence at th e customer residence/facility.

CONDITIONED FLOOR AREA - The floor area of enclosed conditione d spaces on all floors
measured from the interior surfaces of exterior partitions for nonresidential buildings and from the exterior surfaces of exterior partitions fo r residential buildings. (See California Code of Regulations, Title 24, Section 2-5302.)

CONDITIONED SPACE - Enclosed space that is eith er directly or indir ectly conditioned. (See California Code of Regulati ons, Title 24, Section 2-5302.)Evaluators’ Protocols Glossary of Terms

CPUC 221 TecMarket Works Team CONDITIONED SPACE, DIRECTLY - An enclos ed space that is provided with heating equipment that has a capac ity exceeding 10 Btus/(hr-ft 2 ) or with cooling equipment that has a capacity exceeding 10 Btus/(hr-ft 2 ). An exception is if the hea ting and cooling equipment is designed and thermostatically c ontrolled to maintain a process environment temperature less than 65° F or greater than 85° F for the whole space the equipment serves. (See California Code of Regulations, Title 24, Section 2- 5302.)

CONDITIONED SPACE, INDIRECTLY - Enclosed sp ace that: (1) has a greater area weighted heat transfer coefficient (u-value) between it a nd directly conditioned spaces than between it and the outdoors or unconditioned space; (2) has ai r transferred from directly conditioned space moving through it at a rate exce eding three air changes/hour.

CONSERVATION - Steps taken to cause less energy to be used than would otherwise be the case. These steps may involve, for example, im proved efficiency, avoi dance of waste, and reduced consumption. Related activities include, for example, installing equipment (such as a computer to ensure efficient energy use), modi fying equipment (such as making a boiler more efficient), adding insulation a nd changing behavior patterns.

CONSTRUCT VALIDITY - The extent to which an operating variable/instrument accurately taps an underlying concept/hypothe sis, properly measuring an abstract quality or idea.

CONTENT VALIDITY - The extent to which an operating measure taps all the separate subconcepts of a complicated concept.

CONVERGENT VALIDITY - When two instruments/questions/measurement methods obtain similar results when measuring the sa me underlying construct with varying questions/approaches.

COOLING DEGREE DAYS - The cumulative number of degrees in a month or year by which the mean temperature is above 18.3°C /65° F.

CORRELATION COEFFICIENT - A measure of the linear associ ation between two variables, calculated as the square root of the R2 obtained by regressi ng one variable on the other and signed to indicate whether the rela tionship is positive or negative.

COST-EFFECTIVENESS - An indicat or of the relative performan ce or economic attractiveness of any energy efficiency invest ment or practice when compared to the costs of energy produced
and delivered in the absence of such an investme nt. In the energy efficiency field, the present
value of the estimated benefits produced by an energy efficiency program as compared to the estimated total program’s costs, from the perspectiv e of either society as a whole or of individual customers, to determine if the proposed investme nt or measure is desirable from a variety of perspectives, e.g., whether the estimated benefits exceed the estimated costs. See also TOTAL

RESOURCE COST TEST – SOCIETAL VE RSION and PARTICIPANT COST TEST.

CPUC - See CALIFORNIA PUBL IC UTILITIES COMMISSION.Evaluators’ Protocols Glossary of Terms

CPUC 222 TecMarket Works Team CTZ – See CLIMATE THERMAL ZONE.

CUSTOMER - Any person or entity responsible for payment of an electric and/or gas bill to and with an active meter serviced by a utility company (refers to IOU customers herein).

CUSTOMER INFORMATION - Non-public informati on and data specific to a utility customer that the utility acquired or developed in the course of its provision of utility services.

CV – See COEFFICIENT OF VARIATION.

DATABASE FOR ENERGY-EFFICIENT RESOURCE S (DEER) – A database sponsored by the CEC and CPUC designed to provide well-documented estimates of energy and peak demand savings values, measure costs, and effective useful life (EUL) all with one data source. The users of the data are intended to be program planners , regulatory reviewers an d planners, utility and regulatory forecasters, and consultants supporting utility and regulatory research and evaluation efforts. DEER has been designated by the CPUC as its source for deemed and impact costs for program planning.

DAYLIGHTING - The use of sunlight to s upplement or replace electric lighting.

DEER – See DATABASE FOR ENERGY-EFFICIENT RESOURCES.

DEMAND - The time rate of energy flow. Dema nd usually refers to electric power and is measured in kW (equals kWh/h) but can also refe r to natural gas, usuall y as Btu/hr, kBtu/hr, therms/day or ccf/day.

DEMAND (Utility) - The rate or leve l at which electricity or natural gas is delivered to users at a given point in time. Electric demand is expres sed in kilowatts (kW). Demand should not be confused with load, which is the amount of power delivered or required at any specified point or points on a system.

DEMAND BILLING - The electric capacity requirem ent for which a large user pays. It may be based on the customer’s peak demand during the contract year, on a previous maximum or on an agreed minimum. Measured in kilowatts.

DEMAND CHARGE - The su m to be paid by a large electric ity consumer for its peak usage
level.

DEMAND RESPONSIVENESS - Also sometimes refe rred to as load shifting. Activities or equipment that induce consumers to use energy at different (lower cost) times of day or to interrupt energy use for certain equipment tempor arily, usually in direct response to a price signal. Examples include interruptible rates, doing laundry after 7 p.m., and air conditioner recycling programs.

DEMAND SAVINGS - The reduction in the demand from the pre- retrofit baseline to the postretrofit demand, once independent va riables (such as weather or o ccupancy) have been adjustedEvaluators’ Protocols Glossary of Terms

CPUC 223 TecMarket Works Team for. This term is usually applied to billing dema nd, to calculate cost savings or to peak demand, for equipment sizing purposes.

DEMAND SIDE MANAGEMENT (DSM) - The methods used to manage energy demand including energy efficiency, load management, fuel substitution and load building. See LOAD

MANAGEMENT.

DIRECT ENERGY SAVINGS (DIRECT PROG RAM ENERGY SAVINGS) - The use of the words “direct savings” or “direct program savings” refers to the savings from programs that are responsible for the achievement of specific ener gy efficiency goals. Typically these are thought of as resource acquisition programs or programs that install or expedite the installation of energy-efficient equipment and which directly cause or help to cause energy efficiency to be achieved. Rebate, incentive or direct install programs provi de direct energy savings.

DIRECT INSTALL or DIRECT INSTALLATI ON PROGRAMS - These types of programs provide free energy efficiency measures and their installation for qualified customers. Typical measures distributed by these pr ograms include low flow showerheads and compact fluorescent
bulbs.

DIRECTLY COOLED SPACE is an enclosed space that is provided with a space-cooling system that has a capacity exceeding 5 Btu/(hr×ft2), unl ess the space-cooling system is designed and thermostatically controlled to maintain a space temperature less than 55°F or to maintain a space temperature greater than 90°F for the whole space that the system serves.

DIRECTLY HEATED SPACE is an enclosed spa ce that is provided with wood heating or is provided with a space-heating system that has a capacity exceeding 10 Btu/(hr×ft2) unless the space-heating system is designed and thermostatical ly controlled to maintain a space temperature less than 55°F or to maintain a space temperatur e greater than 90°F for the whole space that the
system serves.

DISTRIBUTED GENERATI ON - A distributed generation sy stem involves small amounts of generation located on a utility’s distribution system for the purpos e of meeting lo cal (substation level) peak loads and/or displacing the need to build additional (or upgrade) local distribution
lines.

DOUBLE-BARRELED QUESTIONS - A poorly word ed questionnaire item, which actually asks two questions at the same time, thereby not allowing unique and accurate interpretation of
the results.

DRY-BULB TEMPERATURE - A measure of the sensible temperature of air.

DSM - See DEMAND SIDE MANAGEMENT.

ECM – Energy Conservation Measure. See MEASURE and ENERGY EFFICIENCY

MEASURE.Evaluators’ Protocols Glossary of Terms

CPUC 224 TecMarket Works Team EDUCATION PROGRAMS - Programs primarily in tended to educate customers about energyefficient technologies or behavi ors or provide information a bout programs that offer energy efficiency or load reduction information or services.

EFFECTIVE USEFUL LIFE (EUL) - An estimate of the median number of years that the measures installed under a program are still in place and operable.

EFFICIENCY - The ratio of the useful ener gy delivered by a dynamic system (such as a machine, engine or motor) to the energy supplied to it over the same period or cycle of operation. The ratio is usually determined under specific test conditions.

ELECTRIC PUBLIC GOODS CHAR GE (PGC) - Per Assembly Bill (AB) 1890, a universal charge applied to each electric utility custome r’s bill to support the provision of public goods. Public goods covered by California’s electric PGC include public purpose energy efficiency programs, low-income services, renewables, a nd energy-related research and development.

EM&V - Evaluation, Measurement, Monitoring and Verification.

EMISSIVITY - The property of emitting radiati on; possessed by all materials to a varying

extent.

EMITTANCE - The emissivity of a material, e xpressed as a fraction. Emittance values range from 0.05 for brightly polished metals to 0.96 for flat black paint.

END-USE (MEASURES/GROUPS) - Refers to a broa d or sometimes narrower category that the program is concentrating efforts upon. Examples of end-uses include refrigeration, food service, HVAC, appliances, envelope and lighting.

ENERGY CONSUMPTION - The amount of energy consumed in the form in which it is acquired by the user. The term excludes elec trical generation and distribution losses.

ENERGY COST - The total cost for energy, including such charge s as base charges, demand charges, customer charges, power fact or charges and miscellaneous charges.

ENERGY EFFICIENCY - Using le ss energy to perform the same function. Programs designed to use energy more efficiently - doing the same with less. For the purpose of this paper, energy efficiency programs are distinguished from DSM pr ograms in that the latter are utility-sponsored and financed, while the former is a broader term not limited to any particular sponsor or funding source. “Energy conservation” is a term that ha s also been used but it has the connotation of doing without in order to save energy rather than using less energy to perform the same function and so is not used as much today. Many people use these terms interchangeably.

ENERGY EFFICIENCY IMPROVEMENT - Reduced energy use for a comparable level of service, resulting from the instal lation of an energy efficiency measure or the adoption of an energy efficiency practice. Level of service may be expressed in such ways as the volume of aEvaluators’ Protocols Glossary of Terms

CPUC 225 TecMarket Works Team refrigerator, temperature levels, production ou tput of a manufacturing facility or lighting level/square foot.

ENERGY EFFICIENCY MEASURE - In stallation of equipment, subsystems or systems, or modification of equipment, subsystems, systems or operations on the customer side of the meter, for the purpose of reducing energy and/or demand (and, hence, energy and/or demand costs) at a comparable level of service.

ENERGY EFFICIENCY OF A MEASURE - A measure of the energy used to provide a specific service or to accomplish a specific amount of wo rk (e.g., kWh/cubic foot of a refrigerator, therms/gallon of hot water).

ENERGY EFFICIENCY OF EQUIPMENT - The per centage of gross en ergy input that is realized as useful energy output of a piece of equipment.

ENERGY EFFICIENCY PRACTICE - The use of high-efficiency products, services and practices or an energy-using appliance or pi ece of equipment, to reduce energy usage while maintaining a comparable level of service when in stalled or applied on the customer side of the meter. Energy efficiency activities typically require permanent replacement of energy-using equipment with more efficient models. Exampl es: refrigerator replacement, light fixture replacement, cooling equipment upgrades.

ENERGY MANAGEMENT SYSTEM - A control sy stem (often computerized) designed to regulate the energy consumption of a building by controlling the operation of energy consuming systems, such as the heating, ventilation a nd air conditioning (HVAC), lighting and water heating systems.

ENERGY RESOURCES PROGRAM ACCOUNT (ERPA) - The state law that directs California electric utility companies to gather a stat e energy surcharge/kilowa tt-hour of electricity consumed by a customer. These funds are used for operation of the CEC. As of January 1, 2004, the surcharge is set at $0.0003/kWh.

ENERGY SAVINGS - The reduction in use of ener gy from the pre-retrofit baseline to the postretrofit energy use, once independent variables (such as weather or occupancy) have been adjusted for.

ENGINEERING APPROACHES - Methods using engineering algorithms or models to estimate energy and/or demand use.

ENGINEERING USEFUL LIFE - An engineering esti mate of the number of years that a piece of equipment will operate if properly maintained.

ERPA - See ENERGY RESOU RCES PROGRAM ACCOUNT.

ERROR - Deviation of measurements from the true value.Evaluators’ Protocols Glossary of Terms

CPUC 226 TecMarket Works Team EUL - See EFFECTIVE USEFUL LIFE.

EVALUATION - The performance of studies and activ ities aimed at determining the effects of a program; any of a wide range of assessment activities associated with understanding or documenting program performance or potential pe rformance, assessing program or programrelated markets and market operations; any of a wide range of evalua tive efforts including assessing program-induced changes in energy effi ciency markets, levels of demand or energy savings and program cost-effectiveness.

EX-ANTE SAVINGS ESTIMATE – Administrator-forecasted savings used for program and portfolio planning purposes as filed with the CPUC, from the Latin for “beforehand.”

EX-POST EVALUATION ESTIMATED SAVINGS - Savings estimates reported by the independent evaluator after the energy impact evaluation and the associated M&V efforts have been completed. If only the term “ex-post savings” is used, it will be assumed that it is referring to the ex-post evaluation estimate, the most co mmon usage, from the Latin for “from something
done afterward.”

EX-POST (PROGRAM) ADMINISTRATOR-ESTIM ATED SAVINGS - Savings estimates reported by the Administrator after program im plementation has begun (Administrator-reported ex post), from the Latin for “f rom something done afterward.”

EX-POST (PROGRAM) ADMINISTRATOR-FORECASTED SAVINGS – Savings estimates forecasted by the Administrator during the progr am and portfolio planning process, from the Latin for “from something done afterward.”

EXTERNAL VALIDITY - The extent to which the a ssociation between an independent variable and a dependent variable that is demonstrated wi thin a research setting also holds true in the general environment.

FREE-DRIVER - A non-participant who adopted a pa rticular efficiency m easure or practice as a result of a utility program. See SPI LLOVER EFFECTS for aggregate impacts.

FREE-RIDER - A program participant who would have implemented the program measure or practice in the absence of the program.

GAS PUBLIC GOODS CHARGE (PGC) - Cr eated by AB1002 in 2000, an unbundled rate component included on gas customer bills to fu nd public purpose programs including those for energy efficiency, low-income, and research and development.

GIGAWATT (GW) - One thousand megawatts (1,000 MW), one million kilowatts (1,000,000 kW) or one billion watts (1,000,000,000 watts) of el ectricity. One gigawattis enough to supply
the electric demand of about one million average California homes.

GIGAWATT-HOUR (GWH) - One million k ilowatt-hours of electric power.Evaluators’ Protocols Glossary of Terms

CPUC 227 TecMarket Works Team GLAZING - A covering of transparent or translucent material (typically glass or plastic) used for admitting light.

GROSS AREA - The area of a surface including areas not belonging to that surface (such as windows and doors in a wall).

GROSS LOAD IMPACT - The change in energy consumption and/or demand that results directly from program-related actions taken by pa rticipants in a DSM program, regardless of why they participated. Related to Gross En ergy Impact and Gross Demand Protocols.

HEAT CAPACITY - The amount of heat necessary to raise the te mperature of a given mass one degree. Heat capacity may be calculated by multiplying the mass by the specific heat.

HEAT GAIN - An increase in the amount of heat c ontained in a space, resu lting from direct solar radiation, heat flow through walls, windows and ot her building surfaces, and the heat given off by people, lights, equipment and other sources.

HEAT LOSS - A decrease in the amount of heat contained in a space, resulting from heat flow through walls, windows, roof and other building su rfaces and from exfiltr ation of warm air.

HEAT PUMP - An air conditioning unit which is cap able of heating by refr igeration, transferring heat from one (often cooler) medium to anothe r (often warmer) medium and which may or may not include a capability for coo ling. This reverse-cycle air cond itioner usually provides cooling in summer and heating in winter.

HEAT TRANSFER - Flow of heat energy induced by a temperature difference. Heat flow through a building envelope typicall y flows from a heated or hot ar ea, to a cooled or cold area.

HETEROSCEDASTICITY – Unequa l error variance. In statistics, a sequence or a vector of
random variables is heteroscedasti c if the random variab les in the sequence or vector may have
different variances. This violates the re gression assumption of constant variance (the variance of
the errors is constant across observations or hom oscedastic). Typically, re siduals are plotted to assess this assumption. Standard estimation me thods are inefficient when the errors are heteroscedastic. A common example is when varian ce is expected to be greater on a variable measurement for larger firms than for smaller firms.

HOMOSCEDASTIC (HOMOSCEDASTICITY) - Consta nt error variance, an assumption of classical regression analysis. See also HETEROSCEDASTICITY.

HORSEPOWER (HP) - A unit for measuring the rate of doing work. One horsepower equals about three-fourths of a kilowatt (745.7 watts).

HVAC - Heating Ventilation and Air Conditioning.

HVAC SYSTEM - The equipment, distribution network and termin als that provides either collectively or individually the pr ocesses of heating, ventilating or air conditioning to a building.Evaluators’ Protocols Glossary of Terms

CPUC 228 TecMarket Works Team IMPACT EVALUATION - Used to measure the program-specific induced changes in energy and/or demand usage (such kWh, kW and therms) and/ or behavior attributed to energy efficiency and demand response programs.

IMPACT YEAR - Depending on the context, impact year means either (a ) the twelve months subsequent to program participation used to repr esent program costs or load impacts occurring in that year, or (b) any calendar year after th e program year in which impacts may occur.

IMPLEMENTATION THEORY - A theory describi ng how a program should be structured and implemented and the theoretical rationale supporting the reasons for the program structure and the implementation approach.

IMPLEMENTER - An entity or person selected a nd contracted with or qualified by a program Administrator or by the CPUC to receive PGC funds for providing products and services to

customers.

INCENTIVES - Financial support (e.g., rebates, low- interest loans) to install energy efficiency measures. The incentives are solicited by the cu stomer and based on the customer’s billing history and/or customer -specific information.

INDEPENDENT VARIABLES - The factors that affect the energy and demand used in a building but cannot be controlled (e.g., weather or occupancy).

INDIRECT ENERGY SAVINGS (INDIRECT PROGRAM ENERGY SAVINGS) - The use of the words “indirect savings” or “i ndirect program savings” refers to programs that are typically information, education, marketing or outreach programs in which the program’s actions are expected to result in energy sa vings achieved through the actions of the customers exposed to the program’s efforts, without direct enrollment in an program that has energy savings goals.

INFORMATION PROGRAMS - Programs primarily intended to provide customers with information regarding generic (not customer-specific) cons ervation and energy efficiency opportunities. For these programs, the informati on may be unsolicited by the customer. Programs that provide incentives in the form of unsolicited coupons for di scount on low cost measures are
also included.

INSULATION, THERMAL - A materi al having a relatively high re sistance of heat flow and used principally to retard heat flow. See R-VALUE.

INTEGRATED PART-LOAD VALUE (IPLV) - A single number figure of merit based on partload EER or COP expressing part-load efficiency for air conditioning and heat pump equipment on the basis of weighted ope ration at various load cap acities for the equipment.

INTERNAL VALIDITY - The extent to which altern ative explanations can be eliminated as
causes for an observed association between inde pendent and dependent variable(s) within a research setting/sample.Evaluators’ Protocols Glossary of Terms

CPUC 229 TecMarket Works Team INTERNATIONAL PERFORMANCE MEASUREM ENT AND VERIFICATION PROTOCOL (IPMVP) – The IPMVP provides an overview of cu rrent best practice t echniques available for verifying results of energy efficiency, water efficiency, and renewabl e energy projects in
commercial and industrial facilities . It may also be used by faci lity operators to assess and improve facility performance. The IPMVP is the leading international standard in M&V protocols. It has been translat ed into 10 languages and is used in more than 40 countries.

INVESTOR-OWNED UTILITY (IOU) - A private company that provides a utility, such as water, natural gas or electricity, to a specific serv ice area. California investor-owned utilities are regulated by the CPUC.

IPMVP – See INTERNATIONAL PERFORMANCE MEASUREMENT AND
VERIFICATION PROTOCOL.

JOULE - A unit of work or energy equal to the amount of work done when the point of application of force of 1 Newton is displaced 1 meter in the dir ection of the force. It takes 1,055 joules to equal a British thermal unit. It takes about 1 million joules to make a pot of coffee.

kBtu – One thousand (1,000) British Thermal Units (Btu). See also BRITISH THERMAL

UNIT.

KILOWATT (kW) - One thousand (1,000) watts. A un it of measure of the amount of electricity needed to operate given equipment. On a hot su mmer afternoon a typical home with central air conditioning and other equipment in use mi ght have a demand of four kW each hour.

KILOWATT-HOUR (kWh) - The most commonly us ed unit of measure indicating the amount of electricity consumed over time; one kilo watt of electricity supplied for one hour.

LEVEL OF SERVICES - The utilit y received by a customer fr om energy-using equipment. Level of service may be expressed, for example, as the volume of a refrigerator, an indoor temperature level, the production output of a manufacturing faci lity, or lighting levels/square
foot.

LOAD - The amount of electric power supplied to meet one or more end-user’s needs. The amount of electric power delivered or required at any specified point or points on a system. Load originates primarily at the pow er-consuming equipment of the customer. Load should not be confused with demand, which is the rate at which power is delivered to or by a system, part of a system, or a piece of equipment.

LOAD DIVERSITY - The condition that exists when the peak demands of a variety of electric
customers occur at different times. The diffe rence between the peak of coincident and
noncoincident demands of two or more individual loads. This is the objective of “load molding” strategies, ultimately curbing the tota l capacity requirements of a utility.

LOAD FACTOR - The ratio of the amount of elect ricity a consumer used during a given time span and the amount that would have been used if the usage had stayed at the consumer’s highestEvaluators’ Protocols Glossary of Terms

CPUC 230 TecMarket Works Team demand level during the whole time. The term also is used to mean the pe rcentage of capacity of an energy facility - such as a pow er plant or gas pipeline - that is utilized in a given period of time. The ratio of the average load to p eak load during a specified time interval.

LOAD IMPACT - Changes in electric energy use, electric peak demand or natural gas use.

LOAD MANAGEMENT - Steps taken to reduce power demand at peak load times or to shift some power demand to off-peak times to better meet the utility system capability for a given hour, day, week, season, or year. Load manageme nt may be obtained by persuading consumers to modify behavior, by using equipment that regu lates or controls electric consumption or by
other means.

LOAD SHAPE - The time-of-use pattern of customer or equipment energy use. This pattern can be over a day (24 hours) or over a year (8760 hours).

LOAD SHAPE IMPACTS - Changes in load shape induced by a program.

LOGIC MODEL - The graphical representation of the program theory showing the flow between activities, their outputs and subsequent short-term, intermedia te and long-term outcomes. Often the logic model is displayed with these elemen ts in boxes and the causal flow being shown by arrows from one to the others in the program logic. It can also be displayed as a table with the linear relationship presented by the rows in the table.

LOW-E - A special coating that reduces the emissivity of a wi ndow assembly, thereby reducing the heat transfer through the assembly.

LUMEN - A measure of the amount of light availabl e from a light source equivalent to the light emitted by one candle.

LUMENS/WATT - A measure of the efficacy of a light fixture; the number of lumens output/watt of power consumed.

MAIN METER - The meter that measures the energy used for the whole facili ty. There is at least one meter for each energy source and possibly more than one per source for large facilities. Typically, utility meters are used, but dataloggers may also be used as long as they isolate the load for the facility being studied. When more than one meter per energy source exists for a facility, the main meter may be considered the accumulation of all the meters involved.

MARKET - The commercial activity (manufacturing, distributing, buyi ng and selling) associated with products and services that affect energy usage.

MARKET ACTORS - Individuals and organizations in the pr oduction, distribution and/or delivery chain of energy efficiency products, servi ces and practices. This may include, but is not limited to, manufacturers, distri butors, wholesalers, retailers, vendors, deal ers, contractors, developers, builders, financ ial institutions, and real estate brokers and agents.Evaluators’ Protocols Glossary of Terms

CPUC 231 TecMarket Works Team MARKET ASSESSMENT - An analysis function th at provides an assessment of how and how well a specific market or market segment is f unctioning with respect to the definition of wellfunctioning markets or with resp ect to other specific policy ob jectives. Generally includes a characterization or description of the specific market or market segments, including a description of the types and number of buyers and sellers in the market, the key actors that influence the market, the type and number of tr ansactions that occur on an annua l basis and the extent to which energy efficiency is considered an important part of these transactions by market participants. This analysis may also include an assessment of whether or not a market has been sufficiently transformed to justify a reduction or elimina tion of specific program interventions. Market assessment can be blended with strategic pla nning analysis to produ ce recommended program designs or budgets. One particular kind of market assessment effo rt is a baseline study, or the
characterization of a market before the commence ment of a specific inte rvention in the market, for the purpose of guiding the intervention an d/or assessing its effectiveness later.

MARKET BARRIER - Any char acteristic of the market for an energy-related product, service or practice that helps to explain th e gap between the actua l level of investment in, or practice of, energy efficiency and an increased level that woul d appear to be cost-beneficial to the consumer.

MARKET EFFECT - A change in th e structure or functioning of a market or the behavior of participants in a market that re sult from one or more program effo rts. Typically these efforts are designed to increase in the adoption of energy-e fficient products, services or practices and are causally related to market interventions.

MARKET EVENT - The broader circ umstances under which a customer considers adopting an energy efficiency product, servi ce or practice. Types of mark et events include, but are not necessarily limited to: (a) new construction, or th e construction of a new bu ilding or facility; (b) renovation, or the updating of an ex isting building or facility; (c) remodeling, or a change in an existing building; (d) replacement, or the replaceme nt of equipment, either as a result of an emergency such as equipment failure or as part of a broader planned even t; and, (e) retrofit, or the early replacement of equipment or refitting of a building or facility while equipment is still functioning, often as a result of an intervention into energy efficiency markets.

MARKET PARTICIPANTS - The indi viduals and organizations par ticipating in transactions with one another within an energy efficiency mark et or markets, including customers and market

actors.

MARKET POTENTIAL - See ACHIEVABLE POTENTIAL.

MARKET SECTORS - General types of markets that a program may target or in which a service offering may be placed. Market sectors include categories such as Agricultural, Commercial, Industrial, Government and Institutional. Ma rket sectors help the CPUC assess how well its portfolio of programs is addressing the variety of markets for energy efficiency products and
services in the state.

MARKET SEGMENTS - A part of a market sector that can be grouped together as a result of a characteristic similar to the group. Within the re sidential sector are market segments such asEvaluators’ Protocols Glossary of Terms

CPUC 232 TecMarket Works Team renters, owners, multi-family and single-family. These market segments help the CPUC assess how well its portfolio of programs is addressing the variety of segments within the markets
served.

MARKET THEORY - A theore tical description of how a market operates relative to a specific program or set of programs designed to influence th at market. Market theo ries typically include
the identification of ke y market actors, information flows and product flows through the market, relative to a program designed to change the way the market ope rates. Market theories are typically grounded upon the information provided from a market assessment but can also be
based on other information. Market theories ofte n describe how a program intervention can take advantage of the structure and func tion of a market to transform th e market. Market theories can also describe the key barriers and benefits associ ated with a market and describe how a program can exploit the benefits a nd overcome the barriers.

MARKET TRANSFORMATION - A reduction in mark et barriers resulting from a market intervention, as evidenced by a set of market eff ects, that lasts after the intervention has been withdrawn, reduced or changed.

MEASURE (noun) - A product whose installation and operation at a customer’s premises results
in a reduction in the customer’s on-site energy use, compared to what would have happened
otherwise. See also ENERGY EFFICIENCY MEASURE.

MEASURE (verb) - Use of an in strument to assess a physical qua ntity or use of a computer simulation to estimate a physical quantity.

MEASURE RETENTION – The degree to which meas ures are retained in use after they are
installed. Measure retention studies assess the length of time the measure(s) installed during the program year are maintained in operating condition and the extent to which there has been a significant reduction in the effec tiveness of the measure(s).

MEASURED SAVINGS - Savings or reductions in billing determinants, which are determined using engineering analysis in combination with measured data or through billing analysis.

MEGAWATT (MW) - One thousand kilowatts (1,000 kW) or one million (1,000,000) watts. One megawatt is enough energy to power 1,000 average California homes.

MEGAWATT HOUR (MWh) - One thousand kilowatt- hours. This amount of electricity would supply the monthly power needs of 1,000 typical hom es in the Western U.S. (This is a rounding up to 8,760 kWh/year/home based on an average of 8,549 kWh used/household/year. (U.S. DOE EIA, 1997 annual/capita electricity consumption figures.))

METER - A device used to measure some quan tity, for example, electrical demand, electrical energy, temperature and flow. A device for measuri ng levels and volumes of a customer’s gas or electricity use.Evaluators’ Protocols Glossary of Terms

CPUC 233 TecMarket Works Team METERED DATA - Data collected at customer premises over time through a meter for a specific end-use or energy-using system (e.g., lig hting and HVAC), or location (e.g., floors of a building or a whole premise). Metered data may be collected over a vari ety of time intervals. Usually refers to electricity or gas data.

METERED DEMAND - The average time rate of energy flow over a period of time recorded by a utility meter.

METERING - The collection of energy consumpti on data over time at customer premises through the use of meters. These meters may collect information about kWh, kW or therms, with respect to an end-use, a ci rcuit, a piece or equipment or a w hole building (or facility). Shortterm metering generally refers to data collection for no more than a few weeks. End-use metering refers specifically to separate data collection for one or more end-uses in a building, such as lighting, air conditioning or refrig eration. What is called “spot mete ring” is not metering in this sense, but is an instantaneous measurement (rather than over time) of volts, amps, watts or power factor to determine equipm ent size and/or power draw.

METRIC - A point of measurement. Any point of measurement that can be defined, quantified
and assessed.

MODEL - A mathematical representation or calcula tion procedure that is used to predict the energy use and demand in a buildi ng or facility or to estimate efficiency program savings estimates. Models may be based on equations that specifically represent th e physical processes or may be the result of statistical analysis of energy use data.

MONITORING (equipment or system) - Gathering of relevant measurement data over time to evaluate equipment or system performance, e.g., chiller electric de mand, inlet evaporator temperature and flow, outlet evaporator temperat ure, condenser inlet temperature, and ambient dry-bulb temperature and relative humidity or wet-bulb temperat ure, for use in developing a chiller performance map (e.g., kW /ton vs. cooling load and vs. condenser inlet temperature).

MULTICOLLINEARITY - When two or more indepe ndent variables in a regression model are highly correlated with each other producing high standard errors for the regression parameter. The mathematics of a regression model fail if th ere is perfect collinea rity, an exact linear relationship between two or more independent variables. If the correlation between independent variables is higher than either has with the de pendent variable, the problems of multicollinearity are highly likely.

NAIC - North American Industry Classification.

NATURAL CHANGE - The change in base usag e over time. Natural change represents the effects of energy-related decisions that would have been made in the absence of the utility programs by both program partic ipants and non-participants.

NET LOAD IMPACT - The total change in load that is attributable to th e utility DSM program. This change in load may include, implicitly or ex plicitly, the effects of free-drivers, free-riders,Evaluators’ Protocols Glossary of Terms

CPUC 234 TecMarket Works Team state or federal energy efficiency standards, ch anges in the level of energy service and natural change effects.

NET-TO-GROSS RATIO (NTGR) - A factor representing net program load impacts divided by gross program load impacts that is applied to gross program load impacts to convert them into net program load impacts. This factor is also sometimes used to convert gross measure costs to

net measure costs.

NEW CONSTRUCTION - Residentia l and nonresidential buildings that have been newly built or have added major additions subject to Calif ornia Code of Regulati on Title 24, the California building standards code.

NON-PARTICIPANT - Any customer who was eligib le but did not partic ipate in the utility program under consideration in a given program year. Each evaluation plan should provide a definition of a non-participant as it applies to a specific study.

NONRESIDENTIAL – Used to descri be facilities used for business, commercial, agricultural,
institutional and industrial purposes.

NONRESIDENTIAL BUILDING - Any building which is heated or cooled in its interior and is of an occupancy type other than Type H, I or J, as defined in the Uniform Building Code, 1973 edition, as adopted by the Internationa l Conference of Bu ilding Officials.

NORMALIZATION - Adjustment of the results of a model due to ch anges in baseline assumptions (non-independent variables) during the test or post-retrofit period.

NTGR – See NET-TO-GROSS RATIO.

NULL HYPOTHESIS – See SIGNIFICANCE TEST.

OCCUPANCY SENSOR - A control device that se nses the presence of a person in a given space, commonly used to control lighting systems in buildings.

ORIENTATION - The position of a building re lative to the points of a compass.

P-VALUE – See PROBABILITY-VALUE.

PARTICIPANT - An individual, household, busine ss or other utility customer that received a
service or financial assistance offe red through a particular utility pr ogram, set of utility programs or particular aspect of a utility program in a given program year. The term “service” is used in this definition to suggest that the service can be a wide variety of services, including financial rebates, technical assistance, product installations, tr aining, energy efficiency information or other services, items or conditions. Each eval uation plan should presen t the definition of a “participant” as it applies to a specific study.Evaluators’ Protocols Glossary of Terms

CPUC 235 TecMarket Works Team PARTICIPANT TEST - A cost-effectiveness test intended to measure the cost-effectiveness of energy efficiency programs from the perspective of electric and/or gas cust omers (individuals or organizations) participating in them.

PARTIES OR INTERESTED PARTIES - Persons and organizations with an interest in energy efficiency that comment on or participate in the CPUC’s efforts to develop and implement ratepayer-funded energy efficiency programs.

PEAK DEMAND - The maximum level of metered demand during a specified period, such as a billing month or during a specified peak demand period.

PEAK DEMAND PERIOD - Noon to 7 p.m. Monday through Friday, June, July, August and September.

PEAK LOAD - The highest electrical demand within a particular period of time. Daily electric peaks on weekdays occur in late afternoon a nd early evening. Annual peaks occur on hot summer days.

PERFORMANCE DEGRADATION - Any over tim e savings degradation (or increases compared to standard efficiency operation) that includes both (1) technical operational characteristics of the measures, including operating conditions and product design, and (2) human interaction components and behavioral measures.

PERSISTENCE STUDY - A study to assess ch anges in net program impacts over time (including retention and degradation).

PGC - See PUBLIC GOODS CHARGE.

PORTFOLIO - All IOU and non-IOU energy effici ency programs funded through the PGC that are implemented during a program year or cycle.

POST-BUY ANALYSIS - A comparison of the actual advertising schedule run to the original expectations of the schedule as purchased, c onsidering adherence to buy specifications, actual audience achieved as measured by audience ratings services when available, and conformity to standard industry practices. The term is used primarily in rela tion to broadcast media (and more frequently performed for TV schedules than for radio), but a similar type of stewardship should be performed for purchases of print and outdoor media as well

POST-RETROFIT PERIOD - The time following a retrofit during which savings are to be
determined.

POWER ANALYSIS - A power analysis, executed when a study is being planned, is used to anticipate the likelihood th at the study will yield a significant effect and is based on the same factors as the significance test it self. Specifically, the larger th e effect size used in the power analysis, the larger the sample size; the larg er (more liberal) the criterion required for significance (alpha), the higher the expectation that the study will yield a statistically significantEvaluators’ Protocols Glossary of Terms

CPUC 236 TecMarket Works Team effect. The probability-value (p-value) provided by the significance test and used to reject the null hypothesis, is a function of three factors: size of the obs erved effect (e.g., gross energy savings), sample size and the crite rion required for significance (al pha, the level of confidence). These three factors, together with power, form a closed system – once any three are established, the fourth is completely determined. The goal of power analysis is to fi nd an appropriate balance among these factors by taking into account the s ubstantive goals of the study and the resources
available to the researcher.

PRACTICE - Generally refers to a change in a cu stomer’s behavior or pr ocedures that reduces energy use (e.g., thermostat settings and maintenance procedures).

PRACTICE RETENTION STUDY - An assessment of the length of time a customer continues the energy conservation behavioral cha nges after adoption of these changes.

PRECISION - The indication of the closeness of agreement among repeated measurements of the same physical quantity. In econometrics, the accuracy of an estimator as measured by the
inverse of its variance.

PROBABILITY-VALUE (P-VALUE) - The probab ility of obtaining a finding at least as "impressive" as that obtained, assuming the null hypothesis is true, so that the finding was the result of chance alone. The p-value is provided by the significance test and used to reject the null hypothesis, and is a function of three factors: size of the obs erved effect (e.g., gross energy savings), sample size and the crite rion required for significance (al pha, the level of confidence). These three factors, together with power, form a closed system – once any three are established, the fourth is completely determined.

PROCESS EVALUATION - A systematic assessment of an energy efficiency program for the purposes of documenting program operations at th e time of the examination, and identifying and recommend improvements that can be made to the program to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant
satisfaction.

PROCESS OVERHAUL - Modifications to industrial or agricultural processes to improve their energy use characteristics.

PROGRAM - An activity, strategy or course of action undertaken by an implementer or Administrator using PGC funds. Each program is defined by a unique combination of program strategy, market segment, marketing approach and energy efficiency measure(s) included.

PROGRAM (IMPLEMENTATION) CYCLE - The pe riod of time during which programs are funded, planned and implemented. Can be an annu al cycle, a bi-annual cy cle or other period of
time.

PROGRAM DESIGN - The method or approach for making, doing or accomplishing an objective by means of a program.Evaluators’ Protocols Glossary of Terms

CPUC 237 TecMarket Works Team PROGRAM DEVELOPMENT - The process by wh ich ideas for new or revised energy efficiency programs are converted into a design to achieve a specific objective.

PROGRAM PENETRATION - The level of prog ram acceptance among qualified customers.

PROGRAM MANAGEMENT - The responsibility and ability to oversee and guide the performance of a program to achieve its objective.

PROGRAM STRATEGIES - Refers to the type of method deployed by the program in order to obtain program participation. Some examples of program strategies includ e: rebates, codes, performance contracting and audits.

PROGRAM THEORY - A presentation of the goals of a program, incorporated with a detailed presentation of the activities that the progr am will use to accomplish those goals and the identification of the causal relationships between the activities and the program’s effects.

PROGRAM YEAR (PY) - The calendar year appr oved for program implementation. Note that program years can be shorter than 12 months if programs are approved after the beginning of a calendar year (after January 1 of a given year).

PROGRAMMABLE CONTROLLER - A device that controls the opera tion of electrical equipment (such as air conditioning units and lig hts) according to a pre-set time schedule.

PROJECT - An activity or cour se of action undertaken by an implementer involving one or multiple energy efficiency measures, usually at a single site.

PROJECT DEVELOPMENT - The process by which an implementer identifies a strategy or creates a design to provide energy efficiency pr oducts, services and pr actices directly to

customers.

PUBLIC GOODS CHARGE (PGC) (E lectric) - Per Assembly Bill (AB) 1890, a universal charge applied to each electric utility Customer’s bill to support th e provision of public goods. Public goods covered by California’s electric PGC include public purpose energy efficiency programs, low-income services, renewables, and ener gy-related research and development.

RATIO ESTIMATOR (SAMPLI NG METHOD) - A sampling me thod to obtain increased precision by taking advantage of th e correlation between an auxiliary variable and th e variable of
interest to reduce the co efficient of variation.

REBATES - A type of incentive provided to encourage the adoption of energy-efficient practices, typically paid after th e measure has been installed. Th ere are typically two types of rebates: a Prescriptive Rebate, which is a prescribed financial incentive/unit for a prescribed list of products, and a Customized Rebate, in which th e financial incentive is determined using an analysis of the customer’s equipment and an ag reement on the specific products to be installed. Upstream rebates are financial incentives provid ed for manufacturing, sale s, stocking or otherEvaluators’ Protocols Glossary of Terms

CPUC 238 TecMarket Works Team per unit energy-efficient product movement activitie s designed to increase us e of particular type of products.

REBOUND EFFECT – SEE TAKEBACK EFFECT

REGRESSION MODEL - A mathematical model based on statistical analysis where the dependent variable is regressed on the independ ent variables which are said to determine its value. In so doing, the relations hip between the variables is estim ated statistically from the data
used.

RELIABILITY - When used in en ergy evaluation refers to the likelihood that the observations can be replicated.

REMODELING – Modifications to or the act of modifying the characteristics of an existing
residential or nonresidential building or energy-using equi pment installed within it.

RENEWABLE ENERGY or RENEWABLE RESOURCES or RENEWABLE ENERGY RESOURCES- Renewable energy re sources are naturally replenis hable, but flow-limited. They are virtually inexhaustible in dur ation but limited in the amount of energy that is available per unit of time. Some (such as geothermal and biom ass) may be stock-limited in that stocks are depleted by use, but on a time scale of decades or perhaps centuries, they can probably be replenished. Renewable energy re sources include: biomass, hydr o, geothermal, solar and wind.
Renewable resources also include some experimental or less-deve loped sources such as the use of ocean thermal, wave and tidal action techno logies. Utility renewabl e resource applications include bulk electricity gene ration, on-site electricity gene ration, distributed electricity generation, non-grid-connected generation and demand-reduction (energy efficiency) technologies.

RENOVATION - Modification to the characteristic(s ) of an existing residential or nonresidential building, including but not limited to windows, insu lation and other modifications to the building
shell.

REPLACEMENT - Refers to the changing of equi pment either due to failure, move to more efficient equipment or other reasons near the end of product life or earlier. Often used to refer to a move to a more energy-efficient product that replaces an inefficient product.

RESEARCH AND DEVELOPMENT (R&D) - Research is the discovery of fundamental, new knowledge. Development is the application of ne w knowledge to develop a potential new service or product. Basic power sector R&D is mo st commonly funded and conducted through the Department of Energy (DOE), its associated gover nment laboratories, university laboratories, the Electric Power Research Institute (EPRI) and private sector companies.

RESIDENTIAL BUILDING - Means any hotel, mo tel, apartment house, lodging house, single dwelling or other residential building that is heated or mechanically cooled.Evaluators’ Protocols Glossary of Terms

CPUC 239 TecMarket Works Team RESIDENTIAL CUSTOMER – Utility customers with accounts for existing single-family residences, multi-family dwellings (whether ma ster-metered or individually metered) and buildings that are essentially re sidential but used for commerci al purposes, including but not
limited to time shares and vacation homes.

RETAIL MARKET - A market in wh ich electricity and other energy se rvices are sold directly to
the end-use customer.

RETENTION (MEASURE) - The degree to which meas ures are retained in use after they are
installed.

RETROFIT - Energy efficiency activities undertaken in existi ng residential or nonresidential buildings where existing inefficient equipm ent is replaced by efficient equipment.

RETROFIT ISOLATION - The savings measurem ent approach defined in IPMVP Options A and B, and ASHRAE Guideline 14 that determines energy or demand savings through the use of meters to isolate the energy flows for the system(s) under consideration.

RIGOR - The level of expected reliability. The higher the level of rigor, the more confident we
are the results of the evaluation are both accurate a nd precise, i.e., reliable. That is, reliability and rigor are treated as synonymous. Reliability is discu ssed in the Sampling and Uncertainty
Protocol and in the Evaluation Framework where it is noted that sa mpling precision does not equate to accuracy. Both are important componen ts of reliability, as used in this Protocol.

SAE - See STATISTICALLY ADJUS TED ENGINEERING MODELS.

SAMPLE DESIGN - The approach used to select the sample units.

SAMPLING ERROR - An error which arises because the data are collected from a part, rather than the whole of the population. It is usually measurable from the sample data in the case of probability sampling.

SAVINGS MEASUREMENT APPROACH - The es timation of energy and demand savings associated with an energy efficiency measure fo r a piece of equipment, a subsystem or a system. The estimated savings are based on some kind of m easured data from before and after the retrofit and may be calculated using a vari ety of engineering techniques.

SERIAL CORRELATION - See AUTOCORRELATION.

SERVICE AREA - The geographical territory served by a utility.

SETBACK THERMOSTAT - S ee THERMOSTAT, SETBACK.

SHADING - The protection from heat gains due to direct solar ra diation. Shading is provided by permanently attached exterior devices, glazing ma terials, and adherent ma terials applied to the glazing or an adjacent building for nonresidenti al buildings, hotels, motels and high riseEvaluators’ Protocols Glossary of Terms

CPUC 240 TecMarket Works Team apartments, and by devices affixed to the struct ure for residential buildings. (See California Code of Regulations, Title 24, Section 2-5302.)

SHADING COEFFICIENT (SC) - The ratio of so lar heat gain through fenestration, with or without integral shading devices , to that occurring through unsh aded 1/8 in. thick clear double strength glass. See also SO LAR HEAT GAIN COEFFICIENT.

SHGC - See SOLAR HEAT GAIN COEFFICIENT.

SIGNIFICANCE TEST – Traditiona lly, data collected in a research study is submitted to a significance test to assess the viab ility of the null hypothe sis. The null hypothesis is a term that
statisticians often use to indicate the statistical hypothesis tested. The purpo se of most statistical tests is to determine if the obtained results provi de a reason to reject th e hypothesis that they are merely a product of chance factors. For example, in an experiment in which two groups of randomly selected subjects have received differe nt treatments and have yielded different means, it is always necessary to as k if the difference between the obtained means is among the differences that would be expected to occu r by chance whenever two groups are randomly selected. In this example, the hypothesis tested is that the two samples are from populations with the same mean. Another way to say this is to a ssert that the in vestigator tests the null hypothesis that the difference between the means of the popul ations, from which the samples were drawn, is
zero. If the difference between th e means of the samples is among those that would occur rarely by chance when the null hypothesis is true, the nu ll hypothesis is rejected and the investigator describes the results as statistically significant.

SIMPLE RANDOM SAMPLING - A me thod of selecting n sample units out of the N population such that every one of the distinct N items has an equa l chance of being selected.

SIMPLIFIED ENGINEERING MODEL - Engineering equations used to calculate energy usage and/or savings. These models are usually based on a quantitative description of physical processes that describe the tran sformation of delivered energy into useful work such as heat, lighting or motor drive. In prac tice, these models may be reduced to simple equations that calculate energy usage or savings as a function of measurable attri butes of customers, facilities or equipment (e.g., lighting use = watts X hours of use). These models do not incorporate billing data and do not produce estimates of energy savings to which tests of statistical validity can be applied.

SNAPBACK EFFECT – SEE TAKEBACK EFFECT

SOLAR HEAT GAIN - Heat added to a space du e to transmitted and absorbed solar energy.

SOLAR HEAT GAIN COEFFICIENT (SHGG) - The ratio of the solar heat gain entering the space through the fenestration area to the incident solar radiation.

SOLAR HEAT GAIN FACTOR - An estimate used in calculating cooling load s of the heat gain due to transmitted and absorbed solar energy through 1/8”-thick, clear glass at a specific latitude,
time and orientation.Evaluators’ Protocols Glossary of Terms

CPUC 241 TecMarket Works Team SOLAR HEATING AND HOT WATER SYSTEMS - Solar heating or hot water systems provide two basic functions: (1) capturing the sun’s radiant energy, converting it into heat energy and storing this heat in insulated storage tank(s ); and (2) delivering the stored energy as needed
to either the domestic hot wate r or heating system. These com ponents are called the collection and delivery subsystems.

SPILLOVER - Reductions in energy consumption and/or demand in a utility’s service area caused by the presence of the DSM program, beyond program related gross or net savings of participants. These effects could result from: (a) additional energy effi ciency actions that program participants take outside the program as a result of ha ving participated; (b) changes in the array of energy-using equipment that manuf acturers, dealers and contractors offer all customers as a result of program availability ; and (c) changes in the energy use of nonparticipants as a result of utility programs, whet her direct (e.g., utility program advertising) or indirect (e.g., stocking practices such as (b) above or change s in consumer buying habits). Spillover impacts are to be evaluated via the Imp act Evaluation Protocols (participant spillover) or by the Market Effects Protocol (non-participant spill over), but spillover im pacts are not to be counted toward goal achievements at this time.

SPURIOUSNESS OR SPURIOUS CORRELATION - The apparent association between two variables that is actually attributable to a thir d variable outside the current analysis, probably a common precedent variable.

STAKEHOLDERS - In program evalua tion, stakeholders refer to the myriad of parties that are impacted by a program. Stakeholders include: re gulatory staff, program designers, implementers and evaluators, energy producers, special interest groups, potential participants and customers.

STANDARD DEVIATION - The square root of the variance.

STATEWIDE MARKETING AND OUTREACH PR OGRAMS - Programs that convey consistent statewide messages to individual consumers through a mass-market advertising campaign.

STATEWIDE PROGRAM - A program available in the service territories of all four large California IOUs, with identical implementation char acteristics in all areas, including incentives and application procedures.

STATISTICAL ANALYSIS - Extrapolation of samp le data up to the population, calculation of
error bounds.

STATISTICAL COMPARISONS - A comparison group of customers serving as a proxy of what program participants would have looked like if the program had not been offered.

STATISTICAL POWER - The probability that statistic al significance will be attained, given that there really is a treatment effect. From Lipsey, Mark W. Design Sensitivity: Statistical Power for Experimental Research . Newbury Park, CA: SAGE Publications, 1990.Evaluators’ Protocols Glossary of Terms

CPUC 242 TecMarket Works Team STATISTICALLY ADJUSTED ENGINEERING (S AE) MODELS - A category of billing analysis models that incorporat e the engineering estimate of savi ngs as a dependent variable. The regression coefficient in these models is th e percentage of the e ngineering estimate of savings observed in changes in energy usage. Fo r example, if the coefficient on the SAE term is 0.8, this means that the customers are on average realizing 80 percent of the savings from their engineering estimates.

STRATIFIED RANDOM SAMPLING – A sampling method in which the population is divided into X units of subpopulations that are non-overlapping and toge ther comprise the entire population, called strata. A simple random sample is taken of each strata to create a sample based upon stratified random sampling.

STRATIFIED RATIO ESTIMATION - A sampling me thod that combines a stratified sample design with a ratio estimator to reduce the coefficient of variation by using the correlation of a known measure for the unit (e.g., exp ected energy savings) to stratify the popul ation and allocate sample from strata for optimal sampling.

SUPPLY-SIDE - Activities conducted on the utility’s side of the customer meter. Activities designed to supply electric power to customer s, rather than meeting load though energy efficiency measures or on-site generati on on the customer side of the meter.

SURVIVAL ANALYSIS - Survival analysis is a class of statistical methods for studying the timing of events or time-to-event models. Orig inally these models were developed for medical research where the time to death was analyzed, hence the name survival analysis. These statistical methods are designed to work with ti me-dependent covariates and censoring. Time dependent covariates are indepe ndent variables whose impacts on the dependent variable vary by not only its occurrence but also its timing. Censored data refers to not knowing when something occurred because it is before your data collection (left- censored) or has yet to occur at the time of data collection (right-censored).

SYSTEM - A combination of equipment and/or controls, accessories, in terconnecting means and terminal elements by which energy is transformed so as to perform a specific function, such as HVAC, service water heating or illumination.

TAKEBACK EFFECT – A change in energy using be havior that yields an increased level of
service and that occurs as a result of taking an energy efficiency action.

TECHNICAL DEGRADATION FACTOR - A multipli er used to account for time- and userelated change in the energy sa vings of a high efficiency meas ure or practice relative to a standard efficiency measure or practice due to technical operational characteristics of the measures, including operating cond itions and product design.

TECHNICAL POTENTIAL - The comple te penetration of all measures analyzed in applications where they were deemed technically feas ible from an engineering perspective.Evaluators’ Protocols Glossary of Terms

CPUC 243 TecMarket Works Team TEMPERATURE - Degree of hotness or coldness m easured on one of several arbitrary scales based on some observable phenomenon (such as the expansion).

THERM - One hundred thousand (100,000) Brit ish thermal units (1 therm = 100,000 Btu).

THERMOSTAT - An automatic control device de signed to be responsive to temperature and typically used to maintain set temp eratures by cycling the HVAC system.

THERMOSTAT, SETBACK - A device containing a clock mechanism, which can automatically change the inside temperature maintained by th e HVAC system according to a pre-set schedule. The heating or cooling requirements can be redu ced when a building is unoccupied or when occupants are asleep. (See Ca lifornia Code of Regulations, Title 24, Section 2- 5352(h).)

TIME-OF-USE (TOU) METER - A measuring devi ce that records the times during which a customer uses various amounts of electricity. This type of meter is used for customers who pay
time-of-use rates.

TIME-OF-USE (TOU) RATES - Electricity prices that vary depending on the time periods in which the energy is consumed. In a time-of- use rate structure, higher prices are charged during utility peak-load times. Such rates can provide an incentive for consumers to curb power use during peak times.

TOTAL FLOOR AREA is the floor area (in square feet) of enclos ed space on all floors of a building, as measured at the floor level of the ex terior surfaces of exteri or walls enclosing the

space.

TOTAL RESOURCE COST TEST – SOCIETAL VERSION - A cost-effectiveness test intended
to measure the overall cost-effectiveness of energy efficiency programs from a societal perspective.

TOU – See TIME-OF-USE METER and TIME-OF-USE RATES.

TRIANGULATION - Comparing the results from two or more di fferent data gathering or measurement techniques on the same problem to derive a “best” estimate from the analysis of the comparison.

UA - A measure of the amount of heat that w ould be transferred through a given surface or enclosure (such as a building envelope) with a 1° F temperature difference between the two sides. The UA is calculated by multiplying the U- value by the area of the surface (or surfaces).

UNCERTAINTY - The range or interval of doubt surrounding a measured or calculated value within which the true value is expected to fall within some degree of confidence.

UNCERTAINTY ANALYSIS - (a) A procedure or method by which the uncertainty of a measured or calculated value is determined; (b ) the process of determining the degree of confidence in the true value when using a meas urement procedure(s) and/or calculation(s).Evaluators’ Protocols Glossary of Terms

CPUC 244 TecMarket Works Team UNCONDITIONED SPACE - A space that is neither directly nor indirectly conditioned space, which can be isolated from conditioned space by partitions and/or closeable doors. (See California Code of Regulati ons, Title 24, Section 2-5302.)

UPGRADE (Electric utility) - Re placement or addition of elec trical equipment resulting in increased generation or transmission capability.

UPSTREAM PROGRAMS - Programs that provide in formation and/or fina ncial assistance to entities in the delivery ch ain of high-efficiency products at th e retail, wholesale or manufacturing
level.

UTILITY METER - The meter used to calculate a monthly energy and/or demand charge at a specific utility/customer connection; more than one may be installed per customer and per site due to different supply voltages, capacity requirements, physical separation distances, installation periods or for specific customer requirements or utility programs.

U-VALUE or U-FACTOR - A measur e of how well heat is transf erred by the entire window - the frame, sash and glass - either into or out of the building. U-value is the opposite of R-value. The lower the U-factor number, the better the window will keep heat inside a home on a cold day.

VALIDITY - The extent to which any measuring in strument measures what it is intended to

measure.

VENTILATION - The process of supplying or removing air by natu ral or mechanical means to or from any space. Such air may or may not have been conditioned or treated.

WATT - A unit of measure of elec tric power at a point in time, as capacity or demand. One watt of power maintained ove r time is equal to one joule/second. Some Christmas tree lights use one
watt. The watt is named after Scottish inventor James Watt and is capitalized when shortened to W and used with other abbreviations, as in kWh.

WATT-HOUR - One watt of power expended for one hour. One thousandth of a kilowatt-hour.

WEATHERSTRIPPING - Specially designed strips, seals and gask ets installed around doors and windows to limit air leakage.

WET-BULB TEMPERATURE - The te mperature at which water, by evaporating into air, can bring the air to saturation at the same temper ature. Wet-bulb temperature is measured by a wetbulb psychrometer.

WHOLE-BUILDING CALIBRATED SIMULATION APPROACH - Th e savings measurement approach defined in IPMVP Option D and ASHRAE Guideline 14, which involves the use of an approved computer simulation program to develop a physical model of the building in order to determine energy and demand savings. The simulati on program is used to model the energy usedEvaluators’ Protocols Glossary of Terms

CPUC 245 TecMarket Works Team by the facility before and after the retrofit. Th e pre- or post-retrofit models are developed by calibration with measured energy use and demand data and weather data.

WHOLE-BUILDING METERED APPROA CH - The savings measurement approach defined in IPMVP Option C and ASHRAE Guideline 14 that determines energy and demand savings through the use of whole-facil ity energy (end-use) data, whic h may be measured by utility meters or data loggers. This appr oach may involve the use of monthl y utility billing data or data gathered more frequently from a main meter.

ZONE - A space or group of spaces within a buil ding with any combination of heating, cooling or lighting requirements sufficien tly similar so that desired conditions can be maintained throughout by a single controlling device.Evaluators’ Protocols Performance Basis Metrics

CPUC 246 TecMarket Works TeamEvaluators’ Protocols Performance Basis Metrics

CPUC 247 TecMarket Works Team 

## APPENDIX C. Performance Basis Metrics

This section identifies the performance basis metrics that the CPUC-ED will use to assess the performance of the Administrator’s energy effi ciency programs. These metrics are to be provided by the Administrato rs to the CPUC-ED.

When an evaluation plan indicates that one or more of the following metrics will be collected, assessed or reported within an evaluation effo rt, the evaluation contractor will report the performance basis metrics for each program being evaluated and for the aggregation of programs when the evaluation includes more than one progra m. The performance basis metrics are to be reported in an appendix to the evaluation plan entitled Performance Basis Metric Reporting . The
evaluation contractor is to work with the Join t Staff during the evalua tion planning efforts to identify the performance basis metrics that are to be reported within the evaluation effort.

The following is a list of the performance ba sis metrics that are reported by the program
administrators to be considered for inclusion in th e evaluation reports. The decision of which metrics to include in the evaluation reports will be made by the Joint Staff and provided to the
evaluation contractor. The evaluatio n contractor is to coordinate with Joint Staff to assure that the appropriate performance basis metric s are included in the evaluation reports.

1. Measure installation counts reported by the program. 2. Program costs reported by the program. 3. Measure-specific unit Energy Sa vings reported by the program. 4. Measure-specific installations by progra m delivery strategy re ported by the program. 5. Program administrator estimat es of Gross Energy Savings. 6. Program administrator estimated net-to-gro ss ratios by measure and delivery strategy. 7. Program administrator estimates of net energy savings. 8. Load factors or daily load shapes used to transform annual savings estimates into peak 9. savings estimates.
10. Incremental measure costs.Evaluators’ Protocols Power Analysis

CPUC 248 TecMarket Works TeamEvaluators’ Protocols Power Analysis

CPUC 249 TecMarket Works Team 

## Appendix D. A Primer for Using Power Analysis to Determine Sample Sizes

Power is the probability that you will detect an “e ffect” that is there in the true population that you are studying. Put another way, the power of a statistical test of a null hypothesis is the probability that it will lead to a rejection of the null hypothesi s when it is false, i.e., the probability that it will result in the conclusion that the phenomenon exists. The “effect” could be a difference between two means, a correlation betw een two variables (r), a regression coefficient (b), a chi-squared, etc. Power analysis is a st atistical technique that can be used (among other things) to determine sample size re quirements to ensure that stat istical significance can be found. This appendix provides an overview of using pow er analysis for determining required sample sizes. It provides references and an exampl e of using power analys is for this purpose.

Power analysis is a required component in severa l of the Energy Efficiency Evaluation Protocols

to assist in determining required sample sizes for th e allowable methods that use any type of regression analysis. The regression-base d methods within the Impact Protocol 142 and the
Effective Useful Life Evaluati on Protocol (Retention and Degrad ation) (e.g., survival analysis) must use power analysis to plan their sample size (unless census samples are being used). (Regression-based methods must also meet th e requirements of the Sampling and Uncertainty Protocol.)

In all of the Protocols, where pow er analysis is required it is one of up to three inputs to be used for the determination of sample size for a non-cens us regression study. Each Protocol states that power analysis, results from prior studies on si milar programs, and professional judgment are to be used to determine the required sample size. Sample size planning is an important component in the evaluation planning activity. The proposed sample size(s) must be within the evaluation plan submitted and approved by Joint Staff pr ior to undertaking samp le data collection.

There are many possible references for power analys is and over the last d ecade it has become a standard component of graduate statistics courses. The seminal work was conducted by Jacob
Cohen and the classic text cited is his 1988 Statistical Power Analysis for the Behavioral

Sciences . Power analysis can be used for many things but is only being requi red in the Protocol for determining required sample sizes.

There are several software pack ages and calculation We b sites that conduct the power analysis calculation. The National Institute of Health provid ed funding to BioStat, Inc. to create standalone software to conduct power an alysis calculations. The current version of this software is
called Power and Precision TM and is offered for sale by Biostat TM ( www.PowerAnalysis.com ). The major statistical software packages that evaluators are likely to use for conducting regression-based analyses have incorporated components that conduct power analysis. For

142 These include the Gross Energy Impact Protocol, Gross Demand Impact Protocol, Participant Net Impact

Protocol, and the Indirect Impact Prot ocol. All of these have at least one minimum allowable method that is regression-based. Regression-based methods discussed in t hese Protocols include, but are not limited to, multiple regression (econometric analysis), Analysis of Covarianc e (ANCOVA), and discrete choice (logistic regression).Evaluators’ Protocols Power Analysis

CPUC 250 TecMarket Works Team example, it is included in SPSS ® and in the 9.1 versions of SAS ® along with the SAS/STAT ®
package (Power and Precision module).

A brief overview of the parameters to be input for conducting power anal ysis for the purpose of determining required sample size for the primary regression model types pr imarily used within energy efficiency evaluation is provided below. This is followed by an example where power analysis is conducted to determin e the required sample size for a su rvival analysis (the preferred methodology for effective useful life analysis). This example illustrate s how the sample size requirement varies according to different input parameters.

A small list of references is prov ided at the end of this Appendix.



### Basics of Power Analys is and the Protocols

There is some variation in the parameters a nd the set-up required to use power analysis to determine sample requirements for different types of analyses. There are some that are common to all power analysis. There ar e four common parameters that cr eate a closed system for power analysis. These are:

• Alpha

• Power

• Effect size

• Sample size

Alpha is the criterion required to establish st atistical significance. For consistency across studies, these Protocols have cal led for 90% confidence level (pre cision) and then varied the error tolerance based upon the rigor level assigned. A 90% precisi on equates to an alpha of 0.10. This represents the probability or proportion of st udies that would result in a Type I error, where the researcher rejects the null hypothesis when it is in fact true. For consistency with the precision requirements elsewhere in the Protocol s, the alpha should be set at 0.10 when using power analysis to determin e the required sample size.

Power is the probability that one find a statistically significant effect (when in reality there is one), assuming the effect size, sample size, and al pha criteria. It is co mmon to set power from 0.7 to 0.9. The EUL Analysis Protocol sets the minimum power to 0.7 for the Basic rigor level and 0.8 for the Enhanced rigor.

The effect size is the expected magnitude of the effect. However, effect size will be expressed differently depending on the unit of measurement of the variables involved and on the type of analysis being performed.

In determining sample sizes in the research planning process, va lues for these parameters can be varied in an attempt to balance a level of stat istical power, the alpha, and the effect size, all determined with an eye on the budget constraints. In the end, the results of the power analysis will be combined with professional judgment and pas t studies to arrive at the required sample
sizes.Evaluators’ Protocols Power Analysis

CPUC 251 TecMarket Works Team For multiple regression, analysis of covariance (ANCOVA), and logistic regression, there are three parameters that one can vary when determining the required sample size:

• Alpha

• Power

• Effect size

For survival analysis, there is a fourth parameter that can be varied, the duration of study. Survival analysis depends upon failures to estimate the function of when failure will occur taking into account that for many of the sites failures w ill not have yet been observed (i.e., the data is right-censored, the point of failure is not determined for many in the sample). The later the study (i.e., the greater the duration), th e greater the power since a greater duration increases probability
that more failures will be observed. For the same alpha, effect size, and power, a study that plans
to collect retention data close to the ex-ante EUL (the median measure life) will require fewer sample points than a study conducted earlier.

We conclude this brief introducti on with a list of power facts.

• The more stringent the significance level, the greater the necessary sample size. More subjects are needed for a 1% leve l test than for a 5% level test.

• Two-tailed tests require larger sample si zes than one-tailed te sts. Assessing two directions at the same time requires a gr eater investment. (At the same time, good science requires that a one-ta il test is only used when ther e is strong proof that it is appropriate to do so and not being used for the purpose of making it simpler to pass a statistical significance test.)

• The smaller the critical effect size, the larg er the sample size. Subtle effects require greater efforts.

• The larger the power required, the larger the necessary sample size. Greater protection from failure requires greater effort.

• The smaller the sample size, the smaller the pow er, i.e., the greater the chance of failure.

• If one proposed to conduct an analysis with a very small sample size, one must be willing to accept a high risk of finding no statistically significant results, or be operating in an
area in which the critical e ffect size is quite large.



### Example of Varying Parameters and Estimating Required Sample Size for Survival Analysis through Power Analysis

The basic level of rigor in the EUL Protocols requires that a 0.70 level of power be planned at
the 90% level of confidence. While the enhanced level of rigor requires that a 0.80 level of power be planned also at the 90 % level of confidence. An ex ercise was conducted using Power
and Precision TM software to provide an example of th e use of power analysis to set requiredEvaluators’ Protocols Power Analysis

CPUC 252 TecMarket Works Team sample size and to demonstrate the impact of th e different power level requirements on sample size requirements. 143

Two hypothetical situations were created around an energy efficiency measure with an ex-ante median EUL of 8 years.

• In the first situation, a researcher is interest ed in setting up a study to detect an effect size (a delta) of two years in both directio ns. In other words, our ex-post estimate around an 8 year median EUL finding would be 6 years to 10 years.

• In the second situation, a researcher is inte rested in setting up a study to detect an effect size (a delta) of only one year in bot h directions. In other words, our ex-post estimate around an 8 year median EUL finding would be 7 years to 9 years.

In both cases it was assumed that the effect was se lected as the smallest effect that would be important to detect, in the se nse that any smaller effect w ould not be of any substantive significance. It is also assumed that the effect si ze is reasonable, in the sense that an effect of this magnitude could be anticipat ed in this field of research.

The conditions of the study were as follows:

• A two-tailed test was used since it is possi ble that the ex-post EUL could be higher or
lower than the ex-ante EUL.

• The computation assumes an attrition rate of zero. This means that all sites will be followed until the measure is no longer operational or is not there (the terminal event) or until the study ends.

• This study assumes a condition in which subjects are entered during a given program period and then followed until either (a) the te rminal event occurs, or (b) the study ends causing us not to know how long the equipment w ill last in all those si tes that still have operational equipment (i.e., the site is random ly right-censored). The study design calls for all subjects to be accrued before the st udy begins, with the retention study to occur at 5 years after the program year under study (a follow-up period of 5 years). In other words, all subjects in the sample will be followed for a maximum of 5 years.

• The alpha level was set at 0.10. (T his equates to 90% precision.)

• This study systematically varied two leve ls of power (0.70 and 0.80) to examine the impacts of varying the required power on the subsequent required sample size.

• Finally, this study systematically varied two levels of effect size to examine the impact of alternative effect size requireme nts. Both assumed that th e ex-ante EUL was 8 years. For the ex-post EUL we first assumed 10 year s, which means that the delta between exante and ex-post is two years. We then assumed 9 years, which means that the delta between ex-ante and ex-post is only one year, a much smaller effect.

Table 25 below shows the effect on sample sizes of varying both the effect and the power. The differences looking from one column to the othe r column demonstrate th e differential impact of

143 Produced by Biostat TM . Information available a t: www.PowerAnalysis.comEvaluators’ Protocols Power Analysis

CPUC 253 TecMarket Works Team requiring a power of 0.7 versus 0.8. Looking from row to row demonstrates the impact on sample size requirements of desiring to obtain a one-year effect vers us a two-year effect (for the 8 year ex-ante survival analysis).

Table 25. Sample Sizes as a Function of Alpha and Power

Power

Effect 0.70 0.80

1 Year 1,050 1,400

2 Years 320 420

As one can see, at alphas of 0.70 and 0.80, th e sample sizes increase by approximately 33 percent for a one-year effect si ze and 31.3 percent for a two-year effect size. Increasing the power requirement from 0.70 to 0.80 increases th e required sample size by approximately onethird. However, as one moves from the effect of one year to two years, the required sample sizes increase by approximately 230 percent, from 320 to 1,050 for a power of 0.7 and from 420 to 1,400 for a power of 0.8. Clearly, the impact of a sm aller effect is greate r than the impact of increasing the power.

While we have attempted to keep the example si mple so that the effect of moving from the standard to the enhanced level of rigor can be cl early seen, we note that there are four parameters that one can adjust for determining the required sample size. These are the:

• Duration of study (the post 5-year study assumption in our example.) 144

• Alpha (The precision level which we set at 90% confidence, as is done throughout the evaluation Protocols, which provides an alpha of 0.1.)

• Power

• Effect size

Consider the case in which the effect is one ye ar at a power of 0.80, requiring a sample size of 1,400. If one chose an alpha of 0.20 (as was done in the pre-1998 Protocols for the EUL analysis) and extended the follow-up period from 5 years to 7 years, then the sample size is
reduced to 770.

In determining sample sizes in the research plan ning process, values for these parameters can be varied in an attempt to balance a level of st atistical power, the alpha, the duration of the study, and the effect size, all determined with an eye on the budget constraints. The values will probably be unique to each measur e selected for study. In the end, the results of the power analysis will be combined with professional judgment and past st udies to arrive at the required sample sizes.

144 The closer the study is to the ex-ante EUL the lower t he sample size requirement since finding enough failures to

complete the analysis is a primary component of sample size requirement and the ability to obtain a survival analysis result.Evaluators’ Protocols Power Analysis

CPUC 254 TecMarket Works Team 

### References

Borenstein, Michael, and Jacob Cohen. (1988) Statistical Power Analysis: A Computer Program . Hillsdale, N.J.: Lawrence Erlbaum Associates, Publishers.

Borenstein, Michael, Hannah Ro thstein, and Jacob Cohen. (1988) Power and Precision TM

. Englewood, N.J.: Biostat, Inc.

Cohen, Jacob. (1988) Statistical Power Analysis for the Behavioral Sciences . Hillsdale,N.J.: Lawrence Erlbaum Associates, Publishers.

Kraemer, Helena Chmura and Sue Thiemann. (1987) How Many Subjects? Statistical Power Analysis in Research . Newbury Park, CA: SAGE Publications.

Lipsey, Mark W. (1990) Design Sensitivity: Statistical Po wer for Experimental Research

. Newbury Park, CA: SAGE Publications.

Statistical Package for the Social Sciences, SPSS ® (See SPSS.com)

Statistical Analysis Software (SAS ® )Evaluators’ Protocols Summary Tables

CPUC 255 TecMarket Works Team 

## Appendix E. Summary Tables for All Protocols

The following tables are provided as a quick re ference to the summary tables found in the
Protocols.Evaluators’ Protocols Summary Tables

CPUC 256 TecMarket Works Team 

### Summary of Protocol-Driven Im pact Evaluation Activities

Required Protocols for Gross Energy Evaluation

Rigor Level Minimum Allowable Methods for Gross Energy Evaluation

Basic 1. Simple Engineering Model (SEM) with M&V equal to IPMVP Option A and meeting all requirements in the M&V Protocol for this method. Sampling according to the Sampling and Uncertainty Protocol.

2. Normalized Annual Consumption (N AC) using pre- and post-program participation consumption from utility bills from the appropriate meters related to the measures undertaken, normalized for weather, using identified weather data to normalize for heating and/or cooling as is appropriate to measures included. Tw elve (12) months pre-retrofit and twelve (12) months post-retrofit consumption data is required. Sampling must be according to the Sampling and Uncertainty Protocol.

Enhanced 1. A fully specified regression analys is of consumption information from utility bills with inclusion/adjustment for changes and background variables over the time period of analysis that could potentially be correlated with the gross energy savings being measured. Twelve (12) months post-retrofit consumption data are required. Twelve (12) months pre-retrofit consumption data are required, unless program design does not allow pre-retrofit billing data, such as in new constr uction. In these cases, well-matched control group s and post-retrofit consumption analysis is allowable. 145 Sampling must be according to the Sampling and Uncertainty Protocol utilizing power analysis as an input to determining required sample size(s).

2. Building energy simulation models t hat are calibrated as described in IPMVP Option D requirements in the M&V Protocols. If appropriate, may alternatively use a process-engineering model (e.g., AirMaster+) with calibration as described in the M&V Protocols. Sampling according to the Sampling and Uncertainty Protocol.

3. Retrofit Isolation engineering models as described in IPMVP Option B requirements in the M&V Protocols. Sampling according to the Sampling and Uncertainty Protocol.

4. Experimental design established within the program implementation process, designed to obtain reliable net energy savings based upondifferences between energy consumption between treatment and nontreatment groups from consumption data. 146 Sampling must be according to the Sampling and Uncertainty Protocol.

145 Post-retrofit only billing collapses the analysis from cross-sectional time-series to cross-sectional. Given this, even

more care and examination is expected with regard to cont rolling for cross-sectional issu es that could potentially bias the savings estimate.

146 The overall goal of the Direct Impact Pr otocols is to obtain reliable net energy and demand savings estimates. If

the methodology directly estimates net savings at the same or better rigor than the required level of rigor, then a gross savings and participant net impact analys is is not required to be shown separately.Evaluators’ Protocols Summary Tables

CPUC 257 TecMarket Works Team Required Protocols for Gross Demand Evaluation

Rigor Level Minimum Allowable Methods for Gross Demand Evaluation

Basic Reliance upon secondary data for estimating demand impacts as a function of energy savings. End-use savings load shapes or end-use load shapes from one of the following will be used to estimate demand impacts:

1. End-use savings load shapes, end-use load shapes or allocation factors
from simulations conducted for DEER

2. Allocation factors from CEC foreca sting models or utility forecasting models with approval through t he evaluation plan review process

3. Allocation based on end-use savings load shapes or end-use load shapes from other studies for related programs/similar markets with approval through the evalua tion plan review process

Enhanced Primary demand impact data must be collected during the peak hour during the peak month for each utility system peak. Estimation of demand impact estimates based on these data is requi red. If the methodology and data used can readily provide 8,760-hour outpu t, these should also be provided. 147 Sampling requirements can be met at the program level but reporting must be by climate zone (according to CEC’ s climate zone classification).

1. If interval or time-of-use consumpt ion data are available for participants through utility bills, these data c an be used for regression analysis, accounting for weather, day type and other pertinent change variables, to determine demand impact estimates. Pre- and post-retrofit billing periods must contain peak periods. Requires using power analysis, evaluations of similar programs, and professional judgment to determine sample size requirements for planni ng the evaluation. Needs to meet the requirements of the Sampli ng and Uncertainty Protocol.

2. Spot or continuous metering/measurement of peak pre and post-retrofit during the peak hour of the peak mo nth for the utility system peak to be used with full measurement Option B or calibrated engineering model Option D meeting all requirements as provided in the M&V Protocol. Pre-retrofit data must be adjusted fo r weather and other pertinent change variables. Must meet the Sampling and Uncertainty Protocol with a program target of 10% precision at a 90% confidence level.

3. Experimental design established within the program implementation process, designed to obtain reliable net demand savings based upon differences between energy consumption during peak demand periods between treatment and non-treatment groups from consumption data or spot or continuous metering. 148 Sampling must be according to the Sampling and Uncertainty Protocol.

147 This includes the use of 15-minute interval data or Building Energy Simulation models whose output is 8,760

hourly data.

148 The overall goal of the Impact Protoc ols is to obtain reliable net energy and demand savings estimates. If the

methodology directly estimates net savings at the same or better rigor than the required level of rigor, then a gross savings and participant net impact analysis is not required to be shown separately.Evaluators’ Protocols Summary Tables

CPUC 258 TecMarket Works Team Required Protocols for Participant Net Impact Evaluation

Rigor Level Minimum Allowable Methods for Participant Net Impact Evaluation

Basic 1. Participant self-report.

Standard 1. Participant and non-participant analys is of utility consumption data that
addresses the issue of self-selection.

2. Enhanced self-report method using other data sources relevant to the decision to install/adopt. These could include, for example, record/business policy and paper review, examination of other similar decisions, interviews with multiple ac tors at end-user, interviews with mid-stream and upstream market actors, Title 24 review of typically built buildings by builders and/ or stocking practices.

3. Econometric or discrete choice 149 with participant and non-participant comparison addressing the issue of self-selection.

Enhanced 1. “Triangulation” using more than one of the methods in the Standard Rigor Level. This must include anal ysis and justification for the method for deriving the triangulation estimate from the estimates obtained.

Required Protocols for Indirect Impact Evaluation

Rigor Level Minimum Allowable Methods for Indirect Impact Evaluation

Basic An evaluation to estimate the progra m’s net changes on the behavior of the participants is required; the impact of the program on participant behavior.

Standard A two-stage analysis is required that will produce energy and demand savings. The first stage is to conduct an evaluation to estimate the program’s net changes on the behavior of the participa nts/targeted-customers. The second is to link the behaviors identified to estimates of energy and demand savings based upon prior studies (as approved through the evaluation planning or evaluation review process).

Enhanced A three-stage analysis is required that will produce energy and demand savings. The first stage is to conduct an evaluation to estimate the program’s net impact on the behavior changes of the participants. The second stage is to link the behavioral changes to estimates of energy and demand savings based upon prior studies (as approved through the evaluation planning or evaluation review process). The third stage is to conduct field observation/testing to verify that the occurrence of the level of net behavioral changes.

Summary of Protocol-Driven Impact Evaluation Activities 1 The Joint Staff identifies which programs and program components will receive an impact evaluation and identify the type of impact evaluation(s) to be conduct ed and at what rigor level.

2 The Joint Staff determines any special needs on a case -by-case basis that will be required from

p articular p ro g ram or p ro g ram com p onent evaluations. CPUC-ED issues re q uest for p ro p osals for

149 The instrumental-decomposition (ID) me thod described and referenced in the Evaluation Framework (page 145) is an allowable method that falls into this category. A propensity score methodology is also an allowable method in this category as described in: Itzhak Yanovitzky, Elaine Zanu tto and Robert Hornik,, “Estimating causal effects of public health education campaigns using propensity score methodology.” Evaluation and Program Planning 28 (2005): 209–220.Evaluators’ Protocols Summary Tables

CPUC 259 TecMarket Works Team impact evaluations, selects evaluation contract ors and establishes scope(s) of work.

3 Program theory and logic models (PT/LM), if available, must be reviewed/assessed as needed to properly identify impacts and evaluation elem ents required to assess net program impacts. Research design and sampling plan developed to m eet Protocol requirements at a program or program component basis as designated by the Joint Staff rigor level assignments. This includes meeting requirements from the Sampling and Unce rtainty Protocol, M&V Protocol and Reporting Protocol, as are applicable given Impact Evaluati on Protocol requirements. Research design and sampling must be designed to meet any of the Joint Staff requirements for additional analyses including, but not limited to, the estimation of net impacts by delivery mechanism, the estimation of transmission and/or distribution benefits, or other ar eas designated of specific concern by the Joint Staff. Develop Evaluation Plan, submit it to the CPUC-ED and revise as necessary to have an approved Evaluation Plan that meets th e Impact Evaluation Protocols.

4 All impact evaluation teams must be staffed so as to meet the skills required for the research design, sampling, appropriate and selected impact evaluation method, uncertainty analysis, and reporting being planned and conducted.

5 Develop precise definitions of participants, non-participants and comparison groups. Obtain
concurrence with the CPUC-ED on these definit ions which are to be used in developing the research design and sampling plans.

6 All impact evaluations must meet the requirem ents of the Sampling and Uncertainty Protocol.

6.e There are 2 primary sampling considerations for regression-based consumption analysis.

(3) Unless a census is utilized, conduct a power analysis to estimate the required sample size. One may also consider prior evaluations for similar programs and professional judgment (must use all of t hese for the Enhanced level of rigor); and

(4) Must use a minimum of 12 months pre a nd post-retrofit consum ption data, except when program approach does not allow pre-retrofit data (e.g., new construction).

6.f All engineering-based methods must:

(3) Estimate the uncertainty in all deemed and measured input parameters and consider propagation of error when determining measured quantities and sample sizes to meet the required error tolerance levels; and

(4) Use a combination of deemed and measured data sources with sufficient sample sizes designed to meet a 30% error toleranc e level in the reported value at a 90% confidence level to meet the Basic rigor level and a 10% error tolerance level at a 90% confidence level for the Enhanced rigor level.

6.g Participant and non-participant comparisons and econometric/discrete-choice methods for Participant Net Impact evaluation will use power analysis combined with examinations of prior evaluation studies for similar programs to derive required sample sizes.

6.h Self-report and Enhanced self-report methods for Participant Net Impact evaluations must at a program level have a minimum sample size of 300 participant decision-makers for at least 300 participant sites (where decision-makers may cover more than one site) or a census attempt, whichever is smaller, (while investigation will be at a measure or end-use level).Evaluators’ Protocols Summary Tables

CPUC 260 TecMarket Works Team 7 All impact evaluations must be planned, conduct ed, analyzed and reported to minimize potential bias in the estimates, justify the methods selected for doing this and report all analysis of potential bias issues as described in the Sampling and Uncertainty Protocol, Impact Evaluation Protocol and M&V Protocol. Primary considerations that must be addressed (based upon method employed)
are as follows:

7.e Regression-based consumpt ion analysis must incorporate:

(6) Addressing the influence of weather when weather sensitive measures have been included in the program evaluation;

(7) Assessing potential bias given inclusion/ exclusion issues due to the 12 month preand post-retrofit consumption minimum requirement;

(8) For the Enhanced rigor level, assess , plan, measure and incorporate background and change variables that might be expect ed to be correlated with gross and net energy and/or demand savings;

(9) Comparison groups must be carefully select ed with justification of the criteria for selection of the comparison group and di scussion of any potential bias and how the selected comparison group provides the best available minimization of any potential bias; and

(10) Interval or TOU consumption data for de mand impact analysis must contain the peak period for the utility system peak. If demand billing data is used for demand impact analysis, the research design must address the issues of building demand versus time period for peak and issues with demand ratchets and how the evaluation can reliably provide demand savings estimates. Demand savings must be reported by CTZ.

7.f Engineering-based meth ods must incorporate:

(4) Addressing the influence of weather when weather sensitive measures have been included in the program evaluation;

(5) Meeting all the requirements in the M&V Protocol including issues of baseline determination; and

(6) For the Enhanced rigor level of demand impact analysis using spot or continuous metering/measurement pre- and post-retrofit for the peak hour of the peak month for the utility system peak. Demand savings must be reported by CTZ.

7.g Experimental design must use spot or c ontinuous metering/measurement pre and postretrofit for the peak hour of the peak mont h for the utility system peak for determining demand impacts. Demand savings must be reported by CTZ.

7.h Indirect impact analys is must incorporate:

(6) Description of expected impacts (direct behavioral and indirect energy and demand impacts) and how they will be measured;

(7) Discussion of identification and measurement of baseline;

(8) Extent of exposure/treatment and its measurement;Evaluators’ Protocols Summary Tables

CPUC 261 TecMarket Works Team (9) Comparison groups must be carefully select ed with justification of the criteria for selection of the comparison group and disc ussion of any potential issues of bias and how the selected comparison group provides the best available minimization of potential bias; and

(10) Assessing, planning for and analyzing to control for self-selection bias.

8 Regression analysis of consumption data must address outliers, missing data, weather adjustment, selection bias, background variables, data screens, au tocorrelation, truncation, error in measuring variables, model specification and omitted vari able error, heteroscedasticity, collinearity and influential data points. These areas must be addr essed and reported in accordance with the Sampling and Uncertainty Protocol.

9 Engineering analysis and M&V based methods are r equired to address sources of uncertainty in parameters, construction of baseline, guarding against measurement error, site selection and nonresponse bias, engineering model bias, modeler bias, deemed parameter bias, meter bias, sensor placement bias and non-random selection of equipment or circuits to monitor. These areas must be addressed and reported in accordance with the Sampling and Uncertainty Protocol.

10 Develop draft evaluation report to include meeti ng all requirements in the Reporting Protocol and incorporating the program’s performance metrics.

11 Develop final evaluation report in accordance with guidance provided by the Joint Staff. Submit final evaluation report to the CPUC-ED.

12 Once accepted by the CPUC-E D, develop abstracts and post them and report on CALMAC Web site following the CALMAC posting instructions.

Note: The steps included in this evaluation summary table must comply with all the requirements within the Impact Evaluation Protocol.Evaluators’ Protocols Summary Tables

CPUC 262 TecMarket Works Team 

### Summary of Protocol-Driven M&V Activities

Summary of M&V Protocol for Basic Level of Rigor

Provision Requirement Verification Physical inspection of installation to verify correct measure installation and installation quality IPMVP Option Option A 150
Source of Stipulated Data DEER assumptions, program work papers, engineering references, manufacturers catalog data, on-site survey data Baseline Definition Consistent with program baseline definition. May include federal or Title 20 appliance standards effective at date of equipment manufacture, Title 24 building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the program
Monitoring Strategy and
Duration Spot or short-term measurements depending on measure type Weather Adjustments Weather dependent measur es: normalize to long-term average weather data as directed by the Im pact Evaluation Protocol Calibration Criteria Not applicable

Additional Provisions None

Summary of M&V Protocol for Enhanced Level of Rigor
Provision Requirement

Verification Physical inspection of installation to verify correct measure installation and installation quality. Review of commissi oning reports or functional performance testing to verify correct operation

IPMVP Option Option B or Option D

Source of Stipulated Data DEER assumptions, program work papers, engineering references, manufacturers catalog data, on-site survey data

Baseline Definition Consistent with program baseline definition. May include federal or Title 20 appliance standards effective at dat e of equipment manufacture, Title 24 building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the

program

Monitoring Duration Sufficient to capture all operational modes and seasons

Weather Adjustments Weather dependent measures: normalize to long-term average weather data as directed by the Impact Evaluation Protocol

Calibration Criteria Option D building energy simulation mo dels calibrated to monthly billing or interval demand data. Optional calibration to end-use metered data

Additional Provisions Hourly building energy simulation pr ogram compliant with ASHRAE Standard
140-2001

150 Exceptions to this provision are programs offering comprehensive measure packages with significant measure

interactions; commissioning, and retrocommissioning program s; and new construction programs. Evaluation of measure savings within these programs conducted using engineering methods must follow the Enhanced rigor M&V Protocol and use building energy simulation modeling under IPMVP Option D.Evaluators’ Protocols Summary Tables

CPUC 263 TecMarket Works Team Summary of Protocol-Driven M&V Activities 1 Receive input from impact evaluation plan. Re ceive M&V site selection and expected rigor level from the impact evaluation plan. 2 Develop overall M&V plan. The M&V option for each site shall be established according to the rigor assignment and allowable options under the Impact Evaluation Protocol. Project baseline definition with justification shall be reported. Overall M&V planning shall consider the needs of process evaluation studies for measure insta llation verification and measure performance information. The overall M&V plan shall be su bmitted for approval to the evaluation project manager as designated by the CPUC-ED. 3 Assess data sources. For each sampled site, t he data resources for the engineering analysis must be identified and reviewed. Data sources may include program descriptions, program databases, DEER estimates and underlying documentation, program work papers and on-site surveys. Uncertainties associated with engineering parameter s must be estimated. Baseline uncertainties, where not explicitly documented elsewhere, ma y be informed by professional judgment. 4 Conduct uncertainty analysis. The uncertainty in the estimated savings must be estimated using a propagation of error analysis. The parameters having the greatest influence on the uncertainty must be identified from the propagation of error analysis. 5 Develop site-specific M&V plan according to the outline in the M&V Protocols. The M&V plan must
address data collection conducted to reduce uncert ainty in the engineering estimates of savings. Sampling of measures within a particular site shall be done in accordance with the Sampling and Uncertainty Protocol. The site-specific M&V plan shall be submitted for review and approval to the evaluation project manager designated by the CPUC -ED prior to commencing field data collection. 6 Conduct pre- and/or post-installation monitoring as indicated by M&V plan. Data collection must be conducted in accordance with the site-specific M &V plan. Changes to the M&V plan resulting from unanticipated field conditions shall be document ed and submitted to the evaluation project manager designated by the CPUC-ED. 7 Conduct data analysis and estimate site-specific savings. Conduct analysis of field data and estimate site savings in accordance with site -specific M&V plan. Energy savings estimates for weather-dependent measures shall be normalized to long-term average weather conditions as directed by the Impact Evaluation Protocol. 8 Prepare site-specific M&V report. Prepare a site -specific M&V report for each site used in the analysis that includes the site-specific M&V plan, data collection, data analysis, calculation of measured engineering parameters and overall savi ngs estimates. Calculate the uncertainties associated with energy savings estimates and meas urement-derived engineer ing parameters. The site-specific uncertainty analysis shall include an estimate of the sampling error associated with individual measure sampling within the site, measur ement error associated with field data collection and uncertainties associated with any non-measur ed (deemed) parameters. Potential sources of bias associated with the measurements and engi neering analysis shall be identified and steps to minimize the bias shall be reported in accor dance with the Sampling and Uncertainty Protocol. 9 Prepare draft overall M&V report. A draft over all M&V project report shall be submitted to the CPUC-ED that meets all the requirements of t he Reporting Protocol, demonstrates compliance with the overall M&V plan developed in step 2 and summarizes the results from each site. Site-specific M&V reports shall be included as an Appendix. Raw field data and data analysis results shall be supplied electronically in accordance with the Reporting Protocol. 10 Prepare final overall M&V report. Prepare final overall M&V report in accordance with review comments provided by the Joint Staff. 11 Submit final M&V report. Submit final M&V r eport and associated datas ets to the CPUC-ED. 12 Post final M&V report on the CALMAC Web site . Once accepted by the CPUC-ED, develop abstracts and post them and final M&V report on the CALMAC Web site following the CALMAC posting instructions.Evaluators’ Protocols Summary Tables

CPUC 264 TecMarket Works Team 

### Emerging Technology

Summary of Protocol-Driven Emerging Technology Evaluation Activities

1 Joint staff selects an evaluation contra ctor to implement the Emerging Technology Program evaluation.

2 The ETP managers, in collaboration with t he evaluation contractor and the CPUC-ED, develop logic models and program theories to inform the evaluation plan.

3 The contractor works with the CPUC-ED on the development of the draft evaluation plan (with possible input from the program implement er) consistent with the ETP Protocol. As necessary, the plan must comply with the ot her Protocols (Impact Evaluation Protocol, Process Evaluation Protocol, Market Effects Protocols, the Sampling and Uncertainty Protocol and the Reporting Protocol) in the development of the evaluation plan and in the implementation and reporting efforts.

4 The CPUC-ED works with the evaluation contractor to finalize and approve an evaluation plan from which the contra ctor can begin the evaluation effort.

5 The contractor carries out all eight of t he required Protocol requirements in order to measures key short, intermediate, and long–ra nge performance indicators identified in the logic model. 6 The contractor reports the results of the final evaluation to the CPUC-ED and Joint Staff consistent with the provisions in the Reporting Protocol.

7 Once the report is accepted by the CP UC-ED, the contactor develops abstracts and posts the report on CALMAC web site follo wing the CALMAC posting instructions.Evaluators’ Protocols Summary Tables

CPUC 265 TecMarket Works Team 

### Codes and Standards

Summary of Protocol-Driven Codes and Standards Evaluation Activities

1 Joint staff selects an evaluation contra ctor to implement the Codes and Standards Program evaluation.

2 The evaluation contractor reviews the pr ogram change theories and the program logic models, identifies the technologies or behaviors that can be evaluated via the Protocol, constructs a draft evaluation plan and submit s the plan for approval to the CPUC-ED.
The contractor works with the CPUC-ED on the development of the draft evaluation plan and rigor levels. The plan must use the Impact Evaluation Protocol, the Sampling and Uncertainty Protocol and the Reporting Prot ocol in the development of the evaluation plan and in the implementation and reporting efforts.

3 The CPUC-ED works with the evaluation contractor to finalize and approve an evaluation plan from which the contra ctor can begin the evaluation effort.

4 The contractor conducts an assessment of the gross market-level energy impacts for each code and standard covered technology or behavior being evaluated consistent with the rigor level assignments.

5 The contractor determines the influence of the program on the adoption of each code and standard covered in the study and alloca tes adoption attribution. The assessment uses an interview approach for this assessm ent. This assessment is accomplished as early in the code change cycle as possible but preferably in the technology selection and demonstration phase of the cycle.

6 The contractor estimates naturally occurr ing code and standard covered technology or behavior adoption rates based on literature reviews and interviews with experts.

7 The contractor adjusts the gross market le vel energy savings estimates to account for the net adjustment factors for naturally occurring technology adoption, naturally occurring code change, and non-compliance. This approach nets out the influence of non-program-induced impacts from the gross mark et-level impacts for each technology.

8 The contractor estimates the timeline asso ciated with adoption of a code and standard without the program, using a Delphi approach with an expert panel.

9 The program administrators remove savings estimates from their programs for codecovered measures.

10 The evaluation contractor assesses the cons truction and sales efforts for each utility company service territory and allocates savings by IOU based on the construction and
sales estimates.

11 The contractor reports the results of t he evaluation to the CPUC-ED and Joint Staff consistent with the provisions in the Reporting Protocol.

12 Once the report is accepted by the CP UC-ED, the contactor develops abstracts and

p osts the re p ort on the CALMAC web site followin g the CALMAC p ostin g instructions.Evaluators’ Protocols Summary Tables

CPUC 266 TecMarket Works Team posts the report on the CALMAC web site following the CALMAC posting instructions.

13 As needed, the CPUC-ED or the Joint Staff can request the evaluation contractor to update and report the actual energy savings ov er time consistent with the Protocol. Updates can be conducted with a different ev aluation contractor than those doing the original assessment.Evaluators’ Protocols Summary Tables

CPUC 267 TecMarket Works Team 

### Effective Useful Life

Required Protocols for Measure Retention Study

Rigor Level Retention Evaluation Allowable Methods

Basic 1. In-place and operable status assessment based upon on-site inspections. Sampling must meet the Basic Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirements for the subsequent EUL study. See below specification.)

2. Non-site methods (such as tel ephone surveys/interviews, analysis of consumption data, or use of other dat a, e.g. from EMS systems) may be proposed but must be explicitly appr oved by Joint Staff through the evaluation planning process. Sampli ng must meet the Basic Rigor Level requirements discussed in this Protoc ol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirements for the subsequent EUL study. See below specification.)

Enhanced 1. In-place and operable status assessment based upon on-site inspections. Sampling must meet the Enhanced Rigor Level requirements discussed in this Protoc ol and must meet the requirements of the Sampling and Uncertainty Protocol. (The sampling requirements of this Protocol may need to meet the sampling requirement for the subsequent EUL study. See below specification.)Evaluators’ Protocols Summary Tables

CPUC 268 TecMarket Works Team Required Protocols for Degradation Study

Rigor Level Allowable Methods for Degradation Studies

Basic 1. Literature review required for technical degradation studies across a range of engineering-based literature, to include but not limited to manufacturer’s studies, ASHRAE studies, and laboratory studies. Review of technology assessments. Assessments using simple engineering models for technology components and which examine key input variables and uncertainty factor s affecting technical degradation.

2. Telephone surveys/interviews with a research design that meets accepted social science behavioral re search expectations for behavioral degradation.

Enhanced 1. For technical degradation: field measurement testing.

2. For behavioral degradation: fiel d observations and measurement.

Required Protocols for EUL Analysis Studies

Rigor Level Allowable Methods for EUL Analysis Studies

Basic 1. Classic survival analysis (defined bel ow) or other analysis methods that specifically control for right-censored data (those cases of failure that might take place some time after data are collected) must be attempted. For methods not accounting for right-censored data, the functional form of the model used to estimate EUL (“model functional form”) must be justified and theoretica lly supported. Sampling must meet the Basic Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. Sample size requirements will be determined through the use of power analysis, results from prior studies on si milar programs, and professional judgment. Power analysis used to determine the required sample size must be calculated by setting power to at least at 0.7 to determine the sample size required at a 90% confidence level (alpha set at 0.10). Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regression-based approaches and a 90% confidence level with 30% precision is to be used for non-regression components.

Enhanced 1. Classic survival analysis (defined bel ow) or other analysis methods that specifically control for right-censored data (those cases of failure that might take place some time after data are collected) must be attempted. The functional form of the model used to estimate EUL (“model functional form”) must be justified an d theoretically supported. Sampling must meet the Enhanced Rigor Level requirements discussed in this Protocol and must meet the requirements of the Sampling and Uncertainty Protocol. Sample size requirements will be determined through the use of power analysis, results from prior studies on similar programs, and professional judgment. Power analysis used will set power to at least to 0.8 to determine the sample size required at a 90% confidence level (alpha set at 0.10). Where other analyses or combined functional forms are used, power analysis should be set at these parameters to determine required sample sizes for regression-based

a pp roaches and a 90% confidence level with 10% p recision is to be usedEvaluators’ Protocols Summary Tables

CPUC 269 TecMarket Works Team for non-regression components.

Summary of Protocol-Driven Impact Evaluation Activities
1 Joint Staff will review retention, EUL, and degradation planning information, perhaps through an initial study of (1) prior retention, EUL, and degradation studies and methods, (2) required retention, EUL, and degradation sample sizes, an d (3) assessment of data collection methods for the prioritized measure and delivery strategy/ application needs. Along with any risk analysis information, Joint Staff will identify which measur es by delivery strategy /application will receive which type of retention, EUL, and degradation evaluation, when, and at what rigor level.

Joint Staff will determine any special needs on a case-by-case basis that will be required for particular retention, EUL, and degradation evaluat ions. Joint Staff will develop preliminary RFPs for groups of studies based upon timing of the neede d data collection or analysis, similar sectors or issues to be addressed, and requiring similar skill sets. CPUC-ED will issue RFPs for retention, EUL, and degradation evaluations, select evaluation contractors, and establ ish scope(s) of work.

2 Evaluators will develop a research design and samp ling plan to meet Protocol requirements as designated by the Joint Staff rigor level assignments. This includes meeting requirements from the Sampling and Uncertainty Protocol, as are applic able given Effective Useful Life Evaluation Protocol requirements. Research design and samp ling must be designed to meet any of the Joint Staff requirements for additional analyses to include but not limited to areas designated of specific concern by the Joint Staff. Evaluators will devel op and submit an Evaluation Plan to Joint Staff, and the plan will be revised as necessary to have an approved Evaluation Plan that meets the
Effective Useful Life Evaluation Protocol.

3 All retention, EUL, and degradation study evaluat ion teams (including panel data collection teams) will make sure their teams are ap propriately staffed, in order to meet the skills required for the research design, sampling, and selected retent ion, EUL, and degradation evaluation method, uncertainty analysis, and reporting being planned and conducted.

4 All retention, EUL, and degradation study evaluat ions will be planned, conducted, and analyzed to minimize potential bias in the estimates (showing the methods for doing th is), and evaluators will report all analyses of potential bias issues as de scribed in the Sampling and Uncertainty Protocol.

5 All retention, EUL, and degradation evaluations will be conducted according to the Evaluation Plan and appropriate Protocols.

6 Evaluators will develop the draft ev aluation report in accordance to guidance provided by the Joint Staff and reporting requirements in this Protocol.

7 Final evaluation report will be developed in accord ance to guidance provided by the Joint Staff, and
then submitted to Joint Staff.

8 Once accepted by Joint Staff, abstracts w ill be developed, and a report will be posted on the CALMAC web site following the CALMAC posting instructions.Evaluators’ Protocols Summary Tables

CPUC 270 TecMarket Works Team 

### Summary of Protocol-Driven Market Effects Evaluation Activities

Required Protocols for Market Effects Evaluation Scoping Studies Level of Rigor Scoping Study Requirements

Basic Define the market by its location, the utilities involved, the equipment, behaviors, sector and the program years of interest. Develop market theory. Identify available secondary data and potential sources for primary data. Outline data collection and analysis approaches

Enhanced Define the market by its location, the utilities involved, the equipment, behaviors, sector and the program years of interest. Develop market theory and logic model. Detail indicato rs. Identify available secondary data and primary data that can be used to track changes in indicators. Outline data collection approach. Recommend hy potheses to test in the market effects study. Recommend the analysis approach most likely to be
effective.

Required Protocol for Market Theory and Logic Models

Level of Rigor Market Theory and Logic Model Requirements

Basic Identification of assumptions about ant icipated changes in the market and associated research questions. Market theory should include market operations and conditions, external influences, and as sumptions about changes in the market (which could include market operational theory, market structure and function studies, and product and communication flows). Develop program theory and logic models across programs in that market. Analyze across both of these to examine program interventions, external influences and associated research questions. Theories and logic models should be generated through interviews with program staff and a sample of market actors.

Enhanced Articulate market theory and, if reas onable, develop graphical model of market theory. Market theory should include market operations and conditions, and changes occurring in the market (could in clude market operational theory, market structure and function studies, and product and communication flows). Develop multiple program theory and logic model s for those programs intervening in the market. Integrate the market theory and program theory/logic models to examine external and programmatic influences, as sumptions about changes in the market and associated research questions. Theories and logic models should be generated through interviews or workshops with program staff from each of the programs and a sample of a wide variety of market actors. Use a literature review and other studies of these markets and iteration with program staff to ensure thoroughness in measuring the critical parameters for both market development
from external influences and market effects.Evaluators’ Protocols Summary Tables

CPUC 271 TecMarket Works Team Required Protocol for Market Effects Evaluation Indicator Studies Level of Rigor Indicator Study Requirements

Basic Select appropriate market actor group for each indicator, survey representative samples of market actors able to report on each indicator from market experience. A baseline study must be conducted as early as possible. On-going tracking provides the basis for comparisons.

Enhanced Select appropriate market actor group for each indicator. Conduct longitudinal study of representativ e samples of market actors able to report on each indicator from market experience. Samples weighted to represent known parameters in the population of interest. A baseline study must be conducted as early as possible, on-going tracking provides the basis for comparisons.

Required Protocol for Preponderance of Evidence Approach to Causal Attribution
Estimation

Level of Rigor Preponderance of Evidence Approach Requirements

Basic A representative sample of market actors surveyed or interviewed to provide self-reports on perceived changes in the market, attribution and the sustainability of those changes.

Enhanced Quasi-experimental or experimental design with comparison groups using a representative sample of mark et actors surveyed or interviewed to provide self-reports on perceived changes in the market, attribution and the sustainability of those changes.

Summary of Protocol-Driven Market Effects Evaluation Activities

1 Joint staff identifies the markets or market sect ors (and the associated se t of programs) that will
receive a market effects evaluation and identif ies the potential approach and rigor level for the scoping study.

2 Joint staff identifies market- or market sector-specific study needs that will be assessed (including program-specific or program group spec ific study needs) from the evaluation. CPUCED issues request for proposals for market effects scoping study, selects the scoping study contractor and establishes a scope(s) of work.

3 Evaluation contractor develops scoping study. A scoping study will more finely define the market boundaries for the study, including its location, t he utilities involved, the equipment or behaviors to be assessed and the program-influenced years of interest. The scoping study will develop a market theory and a logic model; identify the market change indicators to track; and the available primary and secondary dat a sources. The study will al so identify the hypotheses to test and the data collection approach, and provide a recommended analysis approach and model specification (if appropriate).

4 A market change theory and logic model (MCT/LM) should be developed to identify assumed
direction of effects and indicators for measur ing effects. The market theory should include market operations and conditions, and changes occu rring in the market (could include a market operations theory, market st ructure and function scenarios, and product and communication flows) The theory and logic model should be generated through interviews or workshops with

p ro g ram staff from each of the p ro g rams that are ex p ected to influence the market bein gEvaluators’ Protocols Summary Tables

CPUC 272 TecMarket Works Team assessed and a sample of a wide variety of mark et actors and should incorporate a literature
review.

5 Joint staff reviews the scoping study and deter mines how to proceed with the Market Effects Evaluation. CPUC-ED issues r equest for proposals for evaluat ion contractors, selects the contractor, establishes a final scope(s) of work and negotiates the contract.

6 All market effects evaluation t eams must be staffed to meet the skills required for the research design, sampling, appropriate and selected ev aluation method, uncertainty analysis and reporting requirements.

7 A research design and sampling plan should be developed to meet Protocol requirements at the market level to meet the Joint Staff assigned study rigor level. This includes meeting requirements from the Sampling and Uncertaint y Protocol and the Reporting Protocol, as applicable. The evaluation contractor will develop an Evaluation Plan, submit it to the CPUC-ED and revise as necessary.

8 Indicators studies conducted as part of t he Market Effects Evaluation should be based on the results of the scoping study, addr ess the appropriate market actor group(s) for each indicator.

9 All Market Effects Evaluations must meet t he requirements of the Sampling and Uncertainty Protocol. The 90/10 level of precision is a mini mum precision target for the most important data collection efforts on its most important variables. Which data collection efforts and variables are considered to be the most important will be deter mined in close collaboration with the CPUC-ED

10 The gross market effects and the estimate of energy savings associated with the market effects should be estimated. Estimation of gross market effects can be as simple as comparing
indicators between time one and time two and t hen multiplying the energy value derived in an M&V supported impact assessment or from DE ER, or using a CPUC-ED-approved net energy
effects model.

11 Attribution or causality should be address ed to estimate net effects using either a preponderance of evidence approach or a net effects modeling approach.

c. For a preponderance of evidence approach a determination of attribution should use quasi-experimental or experimental design with comparison groups using a representative sample of market actors. This may include interviews to provide selfreports on perceived changes in the marke t, attribution and the sustainability of those changes as well as direct observation or ot her data to support changes resulting from the program.

d. For a net effects modeling approach to estima te causality, the model specifications must reflect the complexity of the market. It is likely that such an approach will require multiple equations to model the various activi ties that occur in a market and the various points of intervention that energy efficiency programs exert on a market.

12 Sustainability should be addressed using a preponderance of evidence approach.

13 Develop draft evaluation report to include meet ing all requirements in the Reporting Protocol and incorporating the program’s performance metrics.

14 Develop final evaluation report in accordanc e to guidance provided by Joint Staff.Evaluators’ Protocols Summary Tables

CPUC 273 TecMarket Works Team 15 Submit final evaluation report to the CPUC-ED.

16 Once the report is accepted by the CPUC-ED, develop abstracts and post them and the report on CALMAC Web site following the CALMAC posting instructionsEvaluators’ Protocols Summary Tables

CPUC 274 TecMarket Works Team 

### Sampling and Uncertainty

Required Protocols for Gross Impacts

Rigor
Level Gross Impact Options

Simplified Engineering Models : The relative precision is 90/30 151 . The sampling unit is the premise. The sample size selected mu st be justified in the evaluation plan and approved as part of the ev aluation planning process.

Basic Normalized Annual Consumption (NAC) Models : There are no targets for relative precision. This is due to the fact that NAC models are typically estimated for all participants with an adequate amount of pre- and post-billing data. Thus, there is no sampling error. However, if sampling is conducted, either a power analysis 152 or justification based upon prior evaluations of similar programs must be used to determine sample sizes. The sample size selected must be justified in the evaluati on plan and approved as part of the evaluation planning process.

Regression : There are no relative precision tar gets for regression models that estimate gross energy or demand impacts. Evaluators are expected to conduct, at a minimum, a statistical power analysis as a way of in itially estimating the required sample size. 153 Other
information can be taken into account such as professional judgment and prior evaluations of similar programs. The sample size select ed must be justified in the evaluation plan and approved as part of the ev aluation planning process. Enhanced

Engineering Models : The target relative precision for gross energy and demand impacts is 90/10. The sampling unit is the premise. The sample size selected must be justified in the evaluation plan and approved as part of the evaluation planning process.

151 Also of interest, in addition to the relative precision, are the actual kW h, kW, and therm bounds of the interval.

152 Statistical power is the probab ility that statistical significance will be atta ined, given that there really is a treatment

effect. Power analysis is a statistical technique that ca n be used (among other things) to determine sample size requirements to ensure statistical sign ificance can be found. Power analysis is only being required in the Protocol for determining required sample sizes. There are se veral software packages and calculation Web sites that conduct the power analysis calculatio n. One of many possible references includes: Cohen, Jacob (1989) Statistical Power Analysis for the Behavioral Sciences , Lawrence Erlbaum Associates, Inc.

153 Ibid.Evaluators’ Protocols Summary Tables

CPUC 275 TecMarket Works Team Required Protocols for Net Impacts

Rigor Level Net Impacts Options

Basic For the self-report approach (Option Basic.1) , given the greater issues with construct validity and variety of layered measurements involved in estimating participant NTGRs, no relative precision target has been established. 154 To ensure consistency and comparability a minimum sample size of 300 sites (or decision-makers in cases where decision-makers cover multiple sites) or a census 155 , whichever is smaller, is required.

Standard If the method used for estimating net energy and demand impacts is regression-based, there are no relative precision targets. If the method used for estimating NTGRs is regression-based (discrete choice), there are no re lative precision targets. In either case, evaluators are expected to conduct, at a minimu m, a statistical power analysis as a way of initially estimating the required sample size. 156 Other information can be taken into account such as professional judgment and pr ior evaluations of similar programs.

For the self-report approach (Option Standard.2 ), there are no precision targets since the estimated NTGR will typically be estimated us ing information collected from multiple decision-makers involving a mix of quantitat ive and qualitative information around which a
standard error cannot be constructed. Thus to ensure consistency and comparability, for such studies, a minimum sample size of 300 sites (or decision-makers in cases where decision-makers cover multiple sites) or a census, whichever is smaller, is required.

Enhanced The requirements described for Enhanced apply depending on the methods chosen.

154 This is considered the best feasible approach at the time of the creatio n of this Protocol. Like the other

approaches to estimating the net-to-gross ratio (NTGR), t here is no precision target when using the self-report method. However, unlike the estimati on of the required sample sizes when using the regression and discrete choice approaches, the self-report approach poses a unique set of challenges to estimating required sample sizes. These challenges stem from the fact that the self-report methods for esti mating free-ridership involve greater issues with construct validity, and often include a variety of layered measurements involv ing the collection of both qualitative and quantitative data fr om various actors involved in the decision to install the efficient equipment. Such a situation makes it difficult to arrive at a prior estima te of the expected variance needed to estimate the sample
size. Alternative proposals and the sup port and justifications that address all of the issues discussed here on the

aggregation of variance for the proposed self-report method may be submitted to Joint Staff as an additional option (but not instead of the Protocol requ irements) in impact evaluation RFPs and in Evaluation Plans. Joint Staff may elect to approve an Evaluation Plan wi th a well-justified alternative.

155 A census is rarely achieved. Rather, one attempts to conduct a census, recognizing that there will nearly always be some sites, participants or non-participants who drop out fo r a variety of reasons such as refusals or insufficient
data.

156 Statistical power is the probab ility that statistical significance will be atta ined, given that there really is a treatment

effect. Power analysis is a statistical technique that ca n be used (among other things) to determine sample size requirements to ensure statistical sign ificance can be found. Power analysis is only being required in the Protocol for determining required sample sizes. There are se veral software packages and calculation Web sites that conduct the power analysis calculation.Evaluators’ Protocols Summary Tables

CPUC 276 TecMarket Works Team Required Protocols for Measure-level Measurement and Verification

Rigor
Level M&V Options

Basic Simplified Engineering Models : The target relative precision for gross energy and demand impacts is 90/30. The sample unit may be the individual measure, a particular circuit or point of control as designated by the M&V plan.

Enhanced Direct Measurement and Energy Simulation Models : The target relative precision for gross energy and demand impacts is 90/10. The sample unit may be the individual measure, a particular circuit or point of control as designated by the M&V plan.

Required Protocols for Sampling of Measures Within a Site

The target relative precision is 90/20 for each measure selected for investigation. The sampling unit (measure, circuit, control point) shall be designated by the M&V plan. The initial assumption regarding the coefficient of variation for determining sample size is 0.5.

Required Protocols for Verification

Rigor Level Verification Options

Basic The target relative precision is 90/10. The key parameter upon which the variability for the sample size calculation is based is binar y (i.e., Is it meeting the basic verification criteria specified in the M&V Protocol?).

Enhanced The target relative precision is 90/10. The key parameter upon which the variability for the sample size calculation is based is binar y (i.e., Is it meeting the enhanced verification criteria specified in the M&V Protocol?).