Cape Light Compact 

Annual Report on 

Energy Efficiency Activities 

in 2007 

Submitted to the 

Massachusetts Department of Public Utilities 

and the Massachusetts Division of Energy Resources 

August 27, 2009 
This page intentionally blank. 
Table of Contents 

I. Executive Summary ..................................................................................................... 1

A. Introduction ........................................................................................................ 1 

B. Report Organization ........................................................................................... 2 

C. Summary of Results ........................................................................................... 4 

D. Summary of Results by Sector........................................................................... 4 

II. Overview of Evaluation Methodology...................................................................... 12

III. Impacts by BCR Activity......................................................................................... 16

A. Residential........................................................................................................ 16 

B. Low-Income ..................................................................................................... 20 

C. Commercial & Industrial.................................................................................. 23 

Appendices....................................................................................................................... 27

Appendix 1. Glossary of Terms and Abbreviations.............................................. 28 

Appendix 2. 2007 Evaluation Impact Parameters................................................. 32 

Appendix 3. Post Program Savings Attributed to Selected 2007 Market 

Transformation Initiatives...................................................................................... 35 

Appendix 4. Calculation of Shareholder Incentive............................................... 36 

Appendix 5. Summary of 2007 Energy Efficiency Evaluation Reports ............... 37 
1

I. Executive Summary 

A. Introduction 

Since July 2001 the Cape Light Compact has delivered energy efficiency programs to all 

member towns on Cape Cod and Martha’s Vineyard. This Annual Report provides 

detailed information on the Compact’s energy efficiency activities and savings during the 

course of calendar year 2007. 

The Compact’s 2007 EEP was approved when the 2007-2012 Energy Efficiency Plan 

was approved on December 24, 2007. Using the 2007 funding as approved by the 

Department, the Compact implemented the following set of efficiency programs in 2007: 

• The Residential ENERGY STAR® New Construction Program, which provides 

home buyers, home builders, and construction trade allies with technical assistance 

and financial incentives to increase the efficiency of homes that are newly built or 

undergo major renovations. Results of this program are shown in the Residential 

Lost Opportunity row of Table 2 and in Section III. Results of the Low-Income 

New Construction Program, which provides low-income housing development 

agencies, weatherization assistance program (“WAP”) providers, and residential 

construction trade allies with incentives to increase the home energy rating of new 

low-income housing, are also included. 

• The Residential Conservation Services RCS/MassSAVE Program, which provides 

all interested residential customers with energy savings education, the opportunity 

for a home energy audit and financial incentives for numerous electric and nonelectric efficiency measures, including financial support to switch electric space 

heating systems to more efficient systems that use alternative fuels. Results of this 

program are shown in the Residential Retrofit 1-4 row of Table 2 and in Section III. 

• The Residential ENERGY STAR Products and Services Program, which seeks to 

increase the availability and use of ENERGY STAR qualified lighting and 

appliances, including: clothes washers, room air conditioners, dehumidifiers and 

refrigerators. This program is used to implement the Northeast Energy Efficiency 

Partnership (“NEEP”) initiatives and other regional market transformation efforts. 

Results of this program are divided appropriately between the Residential Lighting 

and Residential Appliances rows of Table 2 and in Section III. 

• The Low-Income Single Family Program, which provides low-income customers in 

single-family dwellings with assistance in purchasing and installing efficient 

lighting, appliances, and weatherization measures. These services are similar to, 

but more extensive in ability to leverage program benefits and offer higher 

incentives to eligible customers, than in the RCS/MassSAVE program. Results of 

this program are shown in the Low Income Retrofit 1-4 row of Table 3 and in 

Section III. 

• The Low-Income Multi-Family Program, which provides owners and managers of 

low-income multi-family dwellings with assistance in purchasing and installing 
2

efficient lighting, appliances and space heating measures. Results of this program 

are shown in the Low Income Retrofit Multifamily row of Table 3 and in Section 

III. 

• The Commercial and Industrial New Construction Program, which provides 

technical assistance and financial incentives to increase the efficiency in the 

construction, renovation, and/or remodeling of all commercial, industrial, 

government and multi-family housing facilities. Results of this program are 

included in the C&I Lost Opportunity row of Table 4 and in Section III. 

• The Medium and Large Commercial and Industrial Retrofit Program, which 

provides technical and financial assistance to medium and large commercial and 

industrial (“C&I”) customers seeking to do discretionary replacements of existing 

operating equipment and processes in their facilities with high-efficiency 

alternatives. Results of this program are included in the C&I Large Retrofit row of 

Table 4 and in Section III. 

• The Small Commercial and Industrial Retrofit Program, which provides technical 

assistance, financial incentives and direct installation to small C&I customers to 

replace existing operating equipment and systems with high-efficiency equipment. 

Results of this program are included in the C&I Small Retrofit row of Table 4 and 

in Section III. 

• The Government Agencies Program, which provides technical assistance and 

financial incentives1

 to all government facilities, including municipal, state and 

federal facilities. For the purposes of reporting the results of this program in this 

Annual Report, in Table 4 and in Section III, the results of efficiency activities with 

small government customers are included in the C&I Small Retrofit row, while the 

results of efficiency activities with large government customers are included in the 

C&I Large Retrofit row. The results of government new construction activities are 

included in the C&I Lost Opportunity row. 

• The Commercial and Industrial Products and Services Program, which seeks to 

increase the availability and use of more efficient motors, lighting designs, and 

HVAC systems. This program is used to implement NEEP and other regional 

market transformation initiatives. The results of this program are included in the 

C&I Lost Opportunity row of Table 4 and in Section III. 

B. Report Organization 

This Executive Summary provides an overview of the Compact’s energy efficiency 

programs’ (referred to as BCR Activities) benefits and costs. For each sector there are 

 

1

 Unlike the Compact’s other C&I Programs, where a customer co-pay is required, the Government 

program covers the entire cost of energy efficiency services resulting from an audit up to a cap of $75,000 

per project. 
3

tables summarizing the lifetime energy savings, lifetime capacity savings, the nonelectric benefits (NEBs), and the dollar values of the total benefits2

 and the total costs. 

The savings data are presented in terms of both “preliminary” and “evaluated” data. 

• The preliminary data refers to savings estimates that are based on the evaluation 

impact factors that were used in the 2006 Annual Report and the Approved Energy 

Efficiency Plan: 2007 – 2012 (referred to as the 2007 EEP).3 Using this data allows 

for the most direct comparison with the estimated savings from the 2007 EEP. 

• The evaluated data typically isolates for changes to savings results based on 

evaluation impact factors from all of the program evaluations that have been 

prepared since the EEP was filed. However, in 2007, the Compact undertook a 

major initiative to adopt assumptions that were consistent with the other 

Massachusetts Program Administrators. Many measure lives, free ridership rates, 

spillover rates, in-service rates, energy savings, loadshapes, demand savings, 

coincidence factors and non-electric benefits were updated. As a result, the 

comparison of the preliminary and evaluated data does not isolate for changes to 

savings results based on evaluation impact factors in this year. The impact factors 

are a subset of the assumptions used in planning and reporting. The evaluated data 

do present our best estimate of the efficiency savings, based on the evaluation 

information available at this time. Appendix 2 presents the impact factors that were 

used to prepare the evaluated results. 

Section II of this Annual Report provides a discussion of the methodology that is used for 

program monitoring and evaluation. It presents a brief summary of the types of 

evaluations that are used, and a description of the methodology for estimated net energy 

savings. It also includes a list of the evaluation studies that were used to prepare the 2007 

evaluated efficiency savings results. These evaluation studies are also used to inform 

program design and delivery. 

Section III of this Annual Report provides more detailed results of the program activities. 

The tables in this section include information regarding the number of program 

participants, the annual efficiency savings and non-electric benefits, the benefit-cost ratio 

of the program, and the savings impacts by type of end-use (lighting, HVAC, motors, 

refrigeration, hot water, and end-user behavior). This section also summarizes recent 

evaluation report findings where relevant. Finally, the appendices provide more detail 

regarding the monitoring and evaluation results and the program savings. Of particular 

interest in this Annual Report, Appendix 3 provides greater detail of program budgets (by 

category) and savings (by type). 

 

2

 The Compact is submitting, consistent with other Program Administrators practice and statewide 

guidance from the Department of Energy Resources, its benefit-cost ratios for its 2007 energy efficiency 

programs with additional capacity benefits in the form of a demand reduction induced price effect 

(“DRIPE”). 

3

 D.P.U. 07-47, Petition of Cape Light Compact Seeking Certification of Energy Efficiency Plan: 2007-

2012 (Stamp-approved December 24, 2007).
4

C. Summary of Results 

Table 1 provides a summary of the program expenses and savings. It also presents the 

percent change between the final evaluated results and (a) the preliminary evaluated 

results, and (b) the estimates of expenses and savings targets in the Compact’s EEP. The 

values in the “Amount” column are the 2007 results, based on all evaluations available at 

this time. 

(Eval-Pre)/Eval (Eval-Plan)/Eval

Prorgram Implementation Expenses $4.909 $ - Millions 0% 2%

Total Expenses $5.048 $ - Millions 0% 0%

Annual Energy Savings 17.149 GWh -25% 33%

Annual Summer Demand Savings 2.409 MW -34% 32%

Annual Winter Demand Savings 4.142 MW -29% 40%

Lifetime Energy Savings 158.599 GWh -54% 33%

Lifetime Demand Savings 29.510 MW-Years -40% 36%

Total Resource Cost Test 4.428 Benefit / Cost -40% 35%

Performance Incentive - After Taxes - $ - Millions - 0%

SAVINGS AND EXPENSES FOR 2007

TABLE 1

Measurement Amount Percent Change Comparison Units

Program implementation expenses include all of the costs incurred by the Compact, except for monitoring 

and verification costs. Total expenses include program implementation costs, plus monitoring and 

verification costs, plus customer contributions. 

The Compact’s 2007 program implementation expenses were roughly 2% higher than the 

2007 budgets in the EEP. However, the total expenses were the same as the 2007 

budgets in the EEP, indicating that higher implementation costs were offset by slightly 

lower evaluation expenditures. This was due to economies of scale that were achieved by 

participation in joint studies with other Program Administrators. 

The annual energy savings achieved in 2007 were 33% higher than those estimated in the 

2007 EEP. The demand savings achieved in 2007 were also significantly higher than 

those estimated in the 2007 EEP. In general, the Compact experienced greater 

participation in the Residential Lighting program and greater uptake on lighting measures 

in the Low Income Retrofit Multifamily and C&I Lost Opportunity programs than 

planned. The benefit-cost ratio of the 2007 programs in total was 4.43. This indicates 

that the Compact’s programs in total are highly cost-effective, where every $1.00 spent 

reduces the net cost of electricity by $4.43. 

D. Summary of Results by Sector 

1. Residential Programs 

Table 2 presents the lifetime energy savings, lifetime capacity savings, and lifetime nonelectric benefits for each of the residential programs. It also presents the total cumulative 

benefits and costs, in 2007 present value dollars. These total benefits and costs are used 

to determine whether each program is cost-effective, based on the total resource cost 

(TRC) test. 
5

Benefit-Cost Ratio

Activity Preliminary Report Preliminary Report Preliminary Report $-Benefits $-Costs

A02a Residential Lost Opportunity 1,255 1,140 761 756 $375,239 $474,428 $704,150 $306,845

A02b Residential HVAC - - - - $0 $0 $0 $0

A03a Residential Retrofit 1-4 54,568 10,560 605 4,522 $753,119 $1,991,937 $3,653,397 $1,200,152

A03b Residential Retrofit Multifamily NA NA NA NA NA NA NA NA

A03c Residential Load Response NA NA NA NA NA NA NA NA

A04a Residential Lighting 39,217 64,573 2,350 4,189 $321,378 $526,704 $6,612,875 $466,145

A04b Residential Appliances 2,233 3,548 640 1,429 $818,544 $1,067,473 $1,620,579 $327,929

Total 79,821 97,273 4,356 10,896 $2,268,280 $4,060,542 $12,591,000 $2,301,072

TABLE 2

SUMMARY OF RESIDENTIAL BCR ACTIVITIES

Lifetime MWH Lifetime kW TRC Values Lifetime $ NEB

Figures 1 through 4 present the same information as Table 2. They indicate that most of 

the residential energy and capacity savings are obtained from the Residential Retrofit 1-4 

and Residential Lighting programs; that most of the non-electric benefits come from the 

Residential Retrofit 1-4 and Residential Appliances programs; and that all residential 

programs are cost effective. 

FIGURE 1

RESIDENTIAL LIFETIME MWH

-

20,000

40,000

60,000

80,000

100,000

120,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

A04b Residential Appliances

A04a Residential Lighting

A03c Residential Load Response

A03b Residential Retrofit

Multifamily

A03a Residential Retrofit 1-4

A02b Residential HVAC

A02a Residential Lost Opportunity

 
6

FIGURE 2

RESIDENTIAL LIFETIME kW

-

2,000

4,000

6,000

8,000

10,000

12,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME kW

A04b Residential Appliances

A04a Residential Lighting

A03c Residential Load Response

A03b Residential Retrofit

Multifamily

A03a Residential Retrofit 1-4

A02b Residential HVAC

A02a Residential Lost Opportunity

FIGURE 3

RESIDENTIAL LIFETIME $ NEB

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

$4,500,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME $ NE

B

A04b Residential Appliances

A04a Residential Lighting

A03c Residential Load Response

A03b Residential Retrofit Multifamily

A03a Residential Retrofit 1-4

A02b Residential HVAC

A02a Residential Lost Opportunity
7

FIGURE 4

RESIDENTIAL TRC VALUES

$0

$2,000,000

$4,000,000

$6,000,000

$8,000,000

$10,000,000

$12,000,000

$14,000,000

$-Benefits $-Costs

BENEFITS VERSUS COSTS

$ TRC VALUE

A04b Residential Appliances

A04a Residential Lighting

A03c Residential Load Response

A03b Residential Retrofit Multifamily

A03a Residential Retrofit 1-4

A02b Residential HVAC

A02a Residential Lost Opportunity

2. Low-Income Programs 

Table 3 presents the lifetime energy savings, lifetime capacity savings, and lifetime nonelectric benefits for each of the low-income programs. It also presents the total 

cumulative benefits and costs, in 2007 present value dollars. These total benefits and 

costs are used to determine whether each program is cost-effective, based on the total 

resource cost test. 

Benefit-Cost Ratio

Activity Preliminary Report Preliminary Report Preliminary Report $-Benefits $-Costs

B02a Low-Income Lost Opportunity - - - - $0 $0 $0 $0

B03a Low-Income Retrofit 1-4 4,953 4,220 250 282 $779,124 $2,091,049 $2,451,669 $525,065

B03b Low-Income Retrofit Multifamily 3,102 3,486 66 101 $882,299 $1,284,389 $1,561,098 $102,129

Total 8,054 7,705 316 383 $1,661,423 $3,375,439 $4,012,768 $627,195

TABLE 3

SUMMARY OF LOW-INCOME BCR ACTIVITIES

Lifetime MWH Lifetime kW TRC Values Lifetime $ NEB

Figures 5 through 8 present the same information graphically as listed in Table 3. They 

indicate that Low Income Retrofit 1-4 is contributing a greater amount of energy and 

capacity savings and non-electric benefits as compared to the Low Income Retrofit 

Multifamily program and that all programs are cost effective. 
8

FIGURE 5

LOW-INCOME LIFETIME MWH

-

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

B03b Low-Income Retrofit

Multifamily

B03a Low-Income Retrofit 1-4

B02a Low-Income Lost Opportunity

FIGURE 6

LOW-INCOME LIFETIME kW

-

50

100

150

200

250

300

350

400

450

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME kW

B03b Low-Income Retrofit

Multifamily

B03a Low-Income Retrofit 1-4

B02a Low-Income Lost Opportunity
9

FIGURE 7

LOW-INCOME LIFETIME $ NEB

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME $ NE

B

B03b Low-Income Retrofit

Multifamily

B03a Low-Income Retrofit 1-4

B02a Low-Income Lost Opportunity

FIGURE 8

LOW-INCOME TRC VALUES

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

$4,500,000

$-Benefits $-Costs

BENEFITS VERSUS COSTS

$ TRC VALUE

B03b Low-Income Retrofit Multifamily

B03a Low-Income Retrofit 1-4

B02a Low-Income Lost Opportunity

3. Commercial & Industrial Programs 

Table 4 presents the lifetime energy savings, lifetime capacity savings, and lifetime nonelectric benefits for each of the Commercial & Industrial programs. It also presents the 

total cumulative benefits and costs, in 2007 present value dollars. These total benefits 

and costs are used to determine whether each program is cost-effective, based on the total 

resource cost (TRC) test. 
10

Benefit-Cost Ratio

Activity Preliminary Report Preliminary Report Preliminary Report $-Benefits $-Costs

C02a C&I Lost Opportunity 35,916 18,085 8,683 4,769 $8 $18,462 $2,356,411 $455,662

C03a Large C&I Retrofit 47,828 12,999 16,815 3,713 $406 $10,755 $1,751,863 $685,189

C03b Small C&I Retrofit 55,700 39,989 11,183 9,748 $12,861 $134,322 $5,218,426 $1,787,534

Total 139,444 71,073 36,681 18,231 $13,275 $163,539 $9,326,699 $2,928,385

TABLE 4

SUMMARY OF C&I BCR ACTIVITIES

Lifetime MWH Lifetime kW TRC Values Lifetime $ NEB

Figures 9 through 12 present the same information as Table 4. They indicate that most of 

the Compact’s C&I energy and capacity savings and non-electric benefits are obtained 

from the Small C&I Retrofit program and that all programs are cost effective. 

FIGURE 9

C&I LIFETIME MWH

-

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

C03b Small C&I Retrofit

C03a Large C&I Retrofit

C02a C&I Lost Opportunity
11

FIGURE 10

C&I LIFETIME kW

-

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME k

W

C03b Small C&I Retrofit

C03a Large C&I Retrofit

C02a C&I Lost Opportunity

FIGURE 11

 C&I LIFETIME $ NEB

$0

$20,000

$40,000

$60,000

$80,000

$100,000

$120,000

$140,000

$160,000

$180,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME $ NE

B

C03b Small C&I Retrofit

C03a Large C&I Retrofit

C02a C&I Lost Opportunity
12

FIGURE 12

C&I TRC VALUES

$0

$1,000,000

$2,000,000

$3,000,000

$4,000,000

$5,000,000

$6,000,000

$7,000,000

$8,000,000

$9,000,000

$10,000,000

$-Benefits $-Costs

BENFITS VERSUS COSTS

$ TRC VALUE

C03b Small C&I Retrofit

C03a Large C&I Retrofit

C02a C&I Lost Opportunity

II. Overview of Evaluation Methodology 

Preliminary versus Evaluated Results 

As noted above, the savings data in this report are presented in terms of both 

“preliminary” and “evaluated” data. 

• The preliminary data refers to savings estimates that are based on the evaluation 

impact factors4 that were used in the 2006 Annual Report and the Approved Energy 

Efficiency Plan: 2007 – 2012 (referred to as the 2007 EEP).5

 Using this data allows 

for the most direct comparison with the estimated savings from the 2007 EEP. 

• The evaluated data typically isolates for changes to savings results based on 

evaluation impact factors from all of the program evaluations that have been 

prepared since the EEP was filed. However, in 2007, the Compact undertook a 

major initiative to adopt assumptions that were consistent with the other 

Massachusetts Program Administrators. Many measure lives, free ridership rates, 

spillover rates, in-service rates, energy savings, loadshapes, demand savings, 

coincidence factors and non-electric benefits were updated. As a result, the 

comparison of the preliminary and evaluated data does not isolate for changes to 

savings results based on evaluation impact factors in this year. The evaluated data 

 

4

 Evaluation impact factors include measure lives, free-ridership rates, spillover rates, in-service rates, and 

realization rates. 

5

 D.P.U. 07-47, Petition of Cape Light Compact Seeking Certification of Energy Efficiency Plan: 2007-

2012 (Stamp-approved December 24, 2007).
13

does present our best estimate of the efficiency savings, based on all the evaluation 

information available at this time. Appendix 2 presents the impact factors that 

were used to prepare the evaluated results. 

Evaluation Studies Used in Preparing 2007 Evaluated Results 

Since its inception in July 2001, the Compact has participated in many state-wide and 

regional monitoring and evaluation studies, along with other energy efficiency Program 

Administrators. The Compact has also conducted several evaluation studies specific to 

its own programs. It is common for energy efficiency program evaluators to update 

parameters on a multi-year cycle, unless significant program changes warrant more 

frequent study. 

The evaluation studies completed in 2007 that were used to update impact factors or to 

inform the process of program delivery are listed below. In 2007 the studies included a 

mix of process and impact evaluation and other research. The executive summaries of 

these reports are included in Appendix 5. 

• Evaluation of the Massachusetts New Homes with ENERGY STAR, Findings and 

Analysis, April 24 ,2008 by Nexus Market Research, Inc. and Dorothy Conant, 

Consultant. 

• The Massachusetts New Homes with ENERGY STAR PROGRAM, 2007 

Progress Report, Final Report, May 30, 2008 

• MassSAVE Final Summary QA/QC and Impact Study Report, April 8, 2008 by 

RLW Analytics, Inc. 

• Residential Lighting Measure Life Study, June 4, 2008, by Nexus Market 

Research, Inc. and RLW Analytics, Inc. 

• Market Progress and Evaluation Report (MPER) for the 2007 Massachusetts 

ENERGY STAR Lighting Program, June 16, 2008, by Nexus Market Research, 

Inc. and RLW Analytics, Inc. 

• Measure Life Report, Residential and Commercial/Industrial Lighting and HVAC 

Measures, prepared for the New England State Program Working Group (SPWG), 

June 2007, by GDS Associates, Inc.6

• Coincidence Factor Study, Residential Room Air Conditioners, Prepared for 

Northeast Energy Efficiency Partnership New England Evaluation and State 

Program Working Group, June 23, 2008 

• Coincidence Factor Study, Residential and Commercial Industrial Lighting 

Measures, Prepared for New England State Program Working Group, Spring 

2007, Prepared by RLW Analytics 

 

6

 This study was completed and filed in 2007, but results were used to update some measures for the 2007 

Annual Report. 
14

• Multiple Small Business Services Programs Impact Evaluation 2007, Final Report 

Update, by Summit Blue Consulting, September 2, 2008. 

• Freeridership and Spillover Study 2007 by PA Consulting: Cape Light Compact 

Results, June 23, 2008 

• Freeridership and Spillover Study 2007 by PA Consulting: National Grid Results, 

June 23, 2008 

Types of Evaluations 

The evaluation of 2007 energy efficiency program impacts reflects the Compact’s efforts 

to apply appropriate methodologies and adjust them for individual program 

characteristics. The diverse nature of the programs, including the magnitude of 

preliminary kW and kWh impacts, the number of customers served, and the end uses 

affected, calls for the adoption of different evaluation approaches. Evaluations of some 

programs use several methodologies to develop overall impact results and provide 

meaningful feedback on program delivery and direction. Some of these methodologies 

are briefly described below. 

Survey-Based Impact Parameter Studies. Survey-based impact parameter studies focus 

on the analysis of information collected through customer surveys. They are generally 

used to measure free-ridership and spillover. These studies provide timely feedback to 

program managers as well as input to the impact evaluations. 

• In 2007, the Cape Light Compact commissioned a survey-based study to assess 

free-ridership and spillover impacts from its Small Commercial and Industrial 

Retrofit, Medium and large Commercial and Industrial Retrofit and Commercial 

and Industrial New Construction Programs. This study was conducted in 

coordination with other utilities and common survey instruments and research 

methods were used. The results of the study are included in Appendix 5. 

Billing Analyses. Billing analyses involve the analysis of billing data, combined in some 

cases with survey data, to determine impacts for programs where a large number of 

participants install similar measures. Since billing data are available for all customers, 

billing analysis techniques may include representative samples of both participants and 

non-participants in an evaluation. 

• In 2007, the Cape Light Compact commissioned a billing analysis to assess the 

realization rate for energy savings from lighting measures in its Small 

Commercial and Industrial Retrofit program. This study was conducted in 

coordination with other utilities and common research methods were used. 

Site Specific Measurement Analysis. Impact evaluations for many of the end uses and 

programs covered in this report rely on engineering estimates that are based on sitespecific metering and on-site telephone assessments of measure performance and 

persistence. 

Process and Market Progress Evaluation Studies. Process evaluations review energy 

efficiency program design and implementation, and recommend modifications to 
15

program delivery. The scope of these evaluations includes all aspects of the program 

including administrative efficiency, the quality of service provided, and the databases 

used for program tracking and reporting. Process evaluations assess the early stages of 

energy efficiency programs. They specifically provide an assessment of (a) whether 

actual operations resemble the intended program design and operation plan, and 

(b) whether real-world experience shows that the original program design and 

implementation plan are appropriate given the existing field conditions. 

• In 2007, the Cape Light Compact jointly funded a market progress study of 

Residential Lighting. This study updated various market transformation tracking 

factors. 

Economic Modeling and Analysis Studies. The benefits and cost-effectiveness of energy 

efficiency programs are based on modeling and analysis that values energy efficiency in 

relation to the avoided costs of energy supply projected over the life of the programs and 

measures installed. Avoided costs are typically projected based on forecasting models. 

The cost-effectiveness results presented in this report – both preliminary and evaluated – 

are all based on the avoided cost estimates that were used in preparing the 2007 EEP. 

This approach allows for a more direct comparison of the economic results between the 

2007 EEP and the 2007 Annual Report. The avoided cost estimates used for both of 

these studies are taken from the following report: Synapse Energy Economics, Inc., 

Avoided Energy Supply Costs in New England, 2007 Final Report, prepared for the 

Avoided Energy Supply Component (AESC) Study Group, Revised - January 3, 2008. 

Generic Impact Equations 

The general form of the impact equation for most of the measures installed is: 

Net Impacts = Gross Impacts * Realization Rate*(1-Free-Ridership + Spillover) * 

Persistence Factor. 

Realization Rates are study-specific parameters, which typically compare the energy or 

demand performance of installed equipment to initial estimates of performance. They are 

typically based on engineering or billing analysis. 

Free-ridership includes both partial and pure free-ridership, where such information is 

available, as required by D.T.E 98-100. 

In energy efficiency programs, spillover may occur among both participants and 

nonparticipants. Both participant and nonparticipant spillover were used in the 

calculation of savings for commercial and industrial programs, consistent with D.T.E. 98-

100. The nonparticipant spillover impact used in this report is based on the combined 

results of National Grid and Compact surveys. 

Persistence indicates the continued presence of savings over time as indicated by followup surveys that confirm the measure remains installed, and verify it is operating as 

intended. As defined by the 2005 Measure Life Study, “Savings persistence is the 

percent change in expected savings due to changed operating hours, changed process 
16

operation, and/or degradation in equipment efficiency relative to the baseline efficiency 

option”. 

Measure lives are applied to net annual kW and kWh to calculate lifetime kW and kWh. 

As defined by the 2005 Measure Life Study7

, measure life is “The median number of 

years that a measure is installed and operational. This definition implicitly includes 

equipment life and measure persistence, but not savings persistence....In addition, this 

definition conforms in letter or in spirit with the definition of measure life used by most 

national utilities.” 

Performance Metrics 

As a not-for-profit inter-governmental organization, the Compact does not require 

shareholder performance incentives, and thus does not need to monitor or track any form 

of performance metrics. 

III. Impacts by BCR Activity 

A. Residential 

1. By BCR Activity 

Table 5 presents a summary of the number of customers served, the annual savings, the 

lifetime savings, and the costs incurred for the residential programs. It also presents the 

benefit-cost ratio, based on the total resource cost test. The costs and benefits used to 

derive this ratio are the same as those presented in Table 2. 

The Residential Retrofit 1-4 and Residential Lighting Programs provide the greatest 

annual energy and capacity savings. The Residential Lighting program is particularly 

cost-effective. 

Benefit-Cost Ratio BenefitCost

Activity kWh kWh per

Cust kW $-NEB MWH kW $-NEB Activity perCust TRC

A02a Residential Lost Opportunity 146 90,499 620 33.32 $21,383 1,140 756 $474,428 $306,845 $2,102 2.29 

A02b Residential HVAC - - NA - $0 - - $0 $0 NA NA

A03a Residential Retrofit 1-4 1,633 1,087,932 666 246.07 $98,381 10,560 4,522 $1,991,937 $1,200,152 $735 3.04 

A03b Residential Retrofit Multifamily NA NA NA NA NA NA NA NA NA NA NA

A03c Residential Load Response NA NA NA NA NA NA NA NA NA NA NA

A04a Residential Lighting 4,551 9,972,772 2,191 645.57 $83,477 64,573 4,189 $526,704 $466,145 $102 14.19 

A04b Residential Appliances 2,115 273,731 129 131.25 $76,257 3,548 1,429 $1,067,473 $327,929 $155 4.94 

Total 8,445 11,424,934 1,353 1,056.21 $279,497 79,821 10,896 $4,060,542 $2,301,072 $272 5.47 

TABLE 5

IMPACT BY RESIDENTIAL BCR ACTIVITIES

Annual Lifetime Cost

Participant

 

7 Measure Life Study Report prepared for the Massachusetts Joint Utilities by Energy Resource Solutions 

(ERS), October 10, 2005. 
17

2. By End Use 

Table 6 presents a summary of the lifetime energy savings, capacity savings, and nonelectric benefits, by the different end-uses addressed in the residential programs. 

Lighting and HVAC provide the majority of energy savings from the residential 

programs. Many of the residential non-electric benefits are from hot water savings, as a 

result of the saved water from ENERGY STAR clotheswashers. 

End Use

Preliminary Report Preliminary Report Preliminary Report

Lighting 46,383 69,355 2,779 4,502 $361,920 $562,732

HVAC 49,555 7,949 1,008 5,026 $1,652,177 $3,540,605

Refrigeration 537 403 72 35 $0 $0

Hot Water 798 2,114 496 1,333 $552,574 $701,868

Motors 0 0 0 0 $0 $0

End User Behavior NA NA NA NA NA NA

Total 97,273 79,821 4,356 10,896 $2,566,671 $4,805,206

TABLE 6

IMPACT BY RESIDENTIAL END-USES

Lifetime MWH Lifetime kW Lifetime $ NEB

Figures 13 through 15 present the same information as Table 6. 

FIGURE 13

RESIDENTIAL LIFETIME MWH - END-USE

0

20,000

40,000

60,000

80,000

100,000

120,000

Preliminary Report

Lifetime MWH

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting
18

FIGURE 14

RESIDENTIAL LIFETIME kW - END-USE

0

2,000

4,000

6,000

8,000

10,000

12,000

Preliminary Report

Lifetime kW

PRELIMINARY VERSUS EVALUATED

LIFETIME k

W

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting

FIGURE 15

RESIDENTIAL LIFETIME $ NEB - END- USE

$0

$1,000,000

$2,000,000

$3,000,000

$4,000,000

$5,000,000

$6,000,000

Preliminary Report

Lifetime $ NEB

BENEFITS VERSUS COSTS

LIFETIME $ NE

B

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting
19

3. Program Evaluation 

The Residential ENERGY STAR® New Construction Program 

In 2008, the Joint Management Committee (JMC) completed an evaluation report8 on 

evaluation work conducted in 2007. The evaluation included: 

• A process evaluation focusing on the transition to a new implementation 

contractor and program changes introduced in 2007; 

• Interviews with implementation contractor staff, HERS raters, Joint Management 

Committee members, ENERGY STAR builders, and program managers; and, 

• An estimation of the incremental costs required to reach ENERGY STAR 

certification. 

The Residential Lighting Program

In 2008, a process and impact evaluation was conducted on the ENERGY STAR 

Lighting program.9 This study updated numerous market transformation tracking factors 

such as consumer awareness, satisfaction, use and purchases of compact fluorescent 

lights and energy-saving fixtures and CFL sales per household in Massachusetts and the 

United States. In addition, the study: 

• Surveyed marketing and stocking practices as well as salesperson training 

practices 

• Assessed manufacturers’ marketing, product development and sales 

• Documented net-to-gross analysis, market penetration of ENERGY STAR 

lighting products, and 

• Reviewed the data collection process used to track activity in cooperative 

promotions. 

Also in 2008, a measure life study10 provided updated estimates of measure lives for 

compact fluorescent bulbs and exterior fixtures. 

The Residential ENERGY STAR Products and Services Program

Coincidence factor studies for room air conditioners11 and lighting12 updated coincidence 

factors to the 2008 ISO-NE definition. 

 

8 Evaluation of the Massachusetts New Homes with ENERGY STAR, Findings and Analysis, April 24, 

2008 by Nexus Market Research, Inc. and Dorothy Conant, Consultant. 

9 Market Progress and Evaluation Report (MPER) for the 2007 Massachusetts ENERGY STAR Lighting 

Program, June 16, 2008, by Nexus Market Research, Inc. and RLW Analytics, Inc.

10 Residential Lighting Measure Life Study, June 4, 2008, by Nexus Market Research, Inc. and RLW 

Analytics, Inc.
20

Residential Conservation Services/MassSAVE

In 2008, the Compact co-sponsored a study13 in which a series of evaluation activities for 

the MassSAVE Program were conducted, including evaluation of the performance of onsite QA/QC work, measure level impact analysis and a natural gas billing analysis. 

Findings and recommendations from the study include: 

• Sponsors should consider using a single tool consistently among all vendors to 

calculate savings; 

• Sponsors should consider developing a single database accessible to all vendors to 

track activity consistently across the state and remove error between DOER and 

vendor-related program tracking; 

• An ongoing QA process should be initiated to ensure that the quality of 

installation is sustained; and, 

• Vendors should install additional weatherization measures, CFLs and/or domestic 

hot water measures to take advantage of all opportunities in a home. 

B. Low-Income 

1. By BCR Activity 

Table 7 presents a summary of the number of customers served, the annual savings, the 

lifetime savings, and the costs incurred for the low-income programs. It also presents the 

benefit-cost ratio, based on the total resource cost test. The costs and benefits used to 

derive this ratio are the same as those presented in Table 3. 

The Low Income Retrofit 1-4 Program contributes greater annual energy and capacity 

savings due to the fact that there are a greater number of participants in this program. 

Benefit-Cost Ratio BenefitCost

Activity kWh kWh per

Cust kW $-NEB MWH kW $-NEB Activity perCust TRC

B02a Low-Income Lost Opportunity - - NA - $0 - - $0 $0 NA NA

B03a Low-Income Retrofit 1-4 442 336,520 761 31.13 $139,547 4,220 282 $2,091,049 $525,065 $1,188 4.67 

B03b Low-Income Retrofit Multifamily 209 236,515 1,132 6.59 $103,462 3,486 101 $1,358,119 $102,129 $489 16.01 

TOTAL 651 573,035 880 37.72 $243,009 7,705 383 $3,449,168 $627,195 $963 6.52 

TABLE 7

IMPACT BY LOW-INCOME BCR ACTIVITIES

Annual Lifetime Cost

Participant

 

11 Coincidence Factor Study, Residential Room Air Conditioners, Prepared for: Northeast Energy 

Efficiency Partnership New England Evaluation and State Program Working Group, June 23, 2008.

12 Coincidence Factor Study, Residential and Commercial Industrial Lighting Measures, Prepared for New 

England State Program Working Group, Spring 2007, Prepared by RLW Analytics. 

13 MassSAVE Final Summary QA/QC and Impact Study Report, April 8, 2008 by RLW Analytics, Inc.
21

2. By End Use 

Table 8 presents a summary of the lifetime energy savings, capacity savings, and nonelectric benefits, by the different end-uses addressed in the low-income programs. As for 

residential, most of the energy and demand savings are from the Lighting and HVAC end 

uses. Most of the low-income non-electric benefits come from the HVAC measures. 

This is because the home energy audits result in benefits associated with (a) improved 

property values, (b) reduced fire, illness and moving costs, and (c) fossil-fuel savings. 

All of the low-income programs also have non-electric benefits as a result of reduced 

usage of the low-income discount rate. The low income programs also have non-electric 

benefits that are experienced by non-low-income residential customers, such as lighting 

O&M savings and reduced water usage. 

Preliminary Report Preliminary Report Preliminary Report

Lighting 4,894 4,493 276 292 $112,560 $92,567

HVAC 2,962 3,013 12 54 $1,526,253 $3,311,739

Refrigeration 195 195 27 33 $18,863 $22,472

Hot Water 4 5 1 3 $3,747 $22,390

Motors 0 0 0 0 $0 $0

End User Behavior NA NA NA NA NA NA

Total 8,054 7,705 316 383 $1,661,423 $3,449,168

TABLE 8

IMPACT BY LOW-INCOME END-USES

Lifetime MWH Lifetime kW Lifetime $ NEB End Use

Figures 16 through 18 present the same information as Table 8. 

FIGURE 16

LOW-INCOME LIFETIME MWH - END-USE

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting
22

FIGURE 17

 LOW-INCOME LIFETIME kW - END-USE

0

50

100

150

200

250

300

350

400

450

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME kW

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting

FIGURE 18

LOW-INCOME LIFETIME $ NEB - END-USE

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME $ NE

B

End User Behavior

Motors

Hot Water

Refrigeration

HVAC

Lighting
23

3. Program Evaluation 

The Compact conducted no new evaluation activities since the 2004 process evaluation of 

the low income program and the addition, in 2006, of low-income non-electric benefits 

(NEBs) to the estimates of low-income multifamily program impacts. 

C. Commercial & Industrial 

1. By BCR Activity 

Table 9 presents a summary of the number of customers served, the annual savings, the 

lifetime savings, and the costs incurred for the commercial & industrial programs. It also 

presents the benefit-cost ratio, based on the total resource cost test. The costs and 

benefits used to derive this ratio are the same as those presented in Table 4. 

The Small C&I Retrofit Program contributes the most annual and lifetime energy and 

capacity savings and non-electric benefits due to high participation in this program. 

Benefit-Cost Ratio Benefit-Cost

Activity kWh kWh per

Customer kW $-NEB MWH kW $-NEB Activity perCustomer TRC

C02a C&I Lost Opportunity 29 1,205,495 41,569 317.90 $1,231 18,085 4,769 $18,462 $455,662 $15,712 5.17 

C03a Large C&I Retrofit 15 890,949 59,397 244.99 $827 12,999 3,713 $10,755 $685,189 $45,679 2.56 

C03b Small C&I Retrofit 290 3,054,330 10,532 752.23 $10,427 39,989 9,748 $134,322 $1,787,534 $6,164 2.92 

TOTAL 334 5,150,775 15,421 1,315.12 $12,485 71,073 18,231 $163,539 $2,928,385 $8,768 3.18 

TABLE 9

IMPACT BY C&I BCR ACTIVITIES

Annual Lifetime Cost

Participant

2. By End Use 

Table 10 presents a summary of the lifetime energy savings, capacity savings, and nonelectric benefits, by the different end-uses addressed in the commercial & industrial 

programs. 

Most of the energy and capacity savings are obtained primarily from lighting measures. 

The non-electric benefits in the C&I sector are primarily from reduced O&M costs as a 

result of efficient light bulbs with longer operating lives. 

Preliminary Report Preliminary Report Preliminary Report

Lighting 82,624 42,916 20,439 11,124 $8,160 $160,325

HVAC 10,778 3,401 5,233 1,927 $954 $3,215

Motors / Drives 25,496 13,046 8,826 3,576 $501 $0

Refrigeration 20,547 11,710 2,183 1,603 $3,660 $0

Hot Water 0 0 0 0 $0 $0

Compressed Air NA NA NA NA NA NA

Process NA NA NA NA NA NA

End User Behavior NA NA NA NA NA NA

Total 139,444 71,073 36,681 18,231 $13,275 $163,539

TABLE 10

IMPACT BY C&I END-USES

Lifetime MWH Lifetime kW Lifetime $ NEB End Use

Figures 19 through 21 present the same information as Table 10. 
24

FIGURE 19

C&I LIFETIME MWH - END-USE

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME MWH

End User Behavior

Process

Compressed Air

Hot Water

Refrigeration

Motors / Drives

HVAC

Lighting

FIGURE 20

C&I LIFETIME kW - END USE

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME k

W

End User Behavior

Process

Compressed Air

Hot Water

Refrigeration

Motors / Drives

HVAC

Lighting
25

FIGURE 21

C&I LIFETIME $ NEB - END-USE

$0

$20,000

$40,000

$60,000

$80,000

$100,000

$120,000

$140,000

$160,000

$180,000

Preliminary Report

PRELIMINARY VERSUS EVALUATED

LIFETIME $ NE

B

End User Behavior

Process

Compressed Air

Hot Water

Refrigeration

Motors / Drives

HVAC

Lighting

3. Program Evaluation 

In 2007 and 2008, the Cape Light Compact co-sponsored several evaluations pertaining 

to commercial and industrial programs. 

1. The Cape Light Compact commissioned a billing analysis14 to assess the 

realization rate for energy savings from lighting measures in its Small 

Commercial and Industrial Retrofit program. While this study was conducted in 

coordination with other utilities and common research methods were used, 

territory-specific realization rates were estimated. The realization rate of 1.04 

indicates that the engineering estimates of savings from this program provided a 

good, slightly conservative, estimate of changed consumption. 

2. The Cape Light Compact commissioned an update of free-ridership and spillover 

estimates15,16 for key commercial and industrial retrofit and new construction 

measures. This study was conducted in coordination with other utilities and 

common research methods were used. Territory-specific results were developed. 

Where appropriate, for some measures and for nonparticipant spillover, where 

sample sizes from the Cape were extremely small and it was not possible to obtain 

 

14 Multiple Small Business Services Programs Impact Evaluation 2007, Final Report Update, by Summit 

Blue Consulting, September 2, 2008. 

15 Freeridership and Spillover Study 2007 by PA Consulting: Cape Light Compact Results, June 23, 2008.

16 Freeridership and Spillover Study 2007 by PA Consulting: National Grid Results, June 23, 2008.
26

responses to produce meaningful results, the Cape Light Compact applied results 

from the National Grid territory. 

The key findings were as follows: 

• Free-ridership levels for major measures (e.g. lighting) decreased in large 

commercial and industrial retrofit and held steady at under 10% in small 

commercial and industrial retrofit program; 

• “Like” participant spillover remained small, decreasing very slightly in 

these programs; and, 

• Non-participant spillover similarly, remained small, decreasing slightly 

since 2005. 

3. In addition, the Cape Light Compact co-sponsored regional research17 to update 

coincidence factors for commercial and industrial lighting measures to the ISONE definitions of peak coincidence. 

 

17 Coincidence Factor Study, Residential and Commercial Industrial Lighting Measures, Prepared for New 

England State Program Working Group, Spring 2007, Prepared by RLW Analytics.
27

Appendices 
28

Appendix 1. Glossary of Terms and Abbreviations18

Annual kWh Reduction Expected net annual energy savings after all impact factors 

have been taken into consideration. 

AMP Appliance Management Program 

BBRS Board of Building Regulations and Standards 

CAP Community Action Program 

CEE Consortium for Energy Efficiency 

CFL Compact Fluorescent Lamps 

Coincident Peak Demand Demand for electricity at the time of the Company’s peak 

demand. 

Delta Watts The difference in the wattage between pre-existing or 

baseline lighting equipment and energy efficient lighting 

equipment. 

Demand The amount of electric energy used by a customer or a piece 

of equipment at a specific time, expressed in kilowatts. 

Demand Adjustment Factor This factor is a combination of one or more evaluation 

impact parameters applied to gross demand savings in the 

calculation of net demand savings. 

Diversity That characteristic of a variety of electric loads whereby 

individual maximum demands usually occur at different 

times. 

Diversity Factor Percent of savings available at the time of the Company’s 

peak demand. 

DOE Department of Energy 

DOER Massachusetts Division of Energy Resources 

D&R D&R International, the contractor to DOE and EPA that 

monitors sales of ENERGY STAR® appliances. 

DSM Demand Side Management 

DTE Massachusetts Department of Telecommunications and 

Energy 

EFLH Equivalent Full Load Hours 

 

18 Much of this glossary was taken from Massachusetts Electric and Nantucket Electric, 2003 Energy 

Efficiency Annual Report, submitted to the Massachusetts Department of Telecommunications and Energy, 

September 2004. In addition to this glossary, a glossary completed in March 2009 for the Regional EM&V 

Forum with additional terms and acronyms is now available at: http://www.neep.org/EMVinfo.html 
29

Energy Adjustment Factor A factor made up of one or more evaluation impact 

parameters applied to gross kWh savings in the calculation 

of net kWh savings. 

EPA Environmental Protection Agency 

EPACT Energy Policy Act 

ENERGY STAR® Brand name for the voluntary energy efficiency labeling 

initiative sponsored by the U.S. Environmental Protection 

Agency and Department of Energy. 

Free Riders Customers who participate in an energy efficiency program 

but would have installed the same measure(s) on their own if 

the program had not been available. 

Free-Ridership Rate The percent of savings attributable to Free Riders. 

Gross kW Expected demand reduction based on a comparison of 

standard or replaced equipment, and equipment installed 

through an energy efficiency program. 

Gross kWh Expected kWh reduction based on a comparison of standard 

or replaced equipment, and equipment installed through an 

energy efficiency program. 

GWh Gigawatt-hour – a measure of electricity usage over time 

equal to 1,000 megawatt-hours or 1,000,000 kilowatt-hours. 

Hours of Use The estimated number of hours per year that a measure 

operates. 

Hours of Use Realization 

Rate 

Ratio of actual metered hours of use data to estimated hours 

of use data. 

HP Horsepower 

HVAC Heating Ventilation and Air Conditioning 

Impact Factor Generic term for persistence, realization rates, in-service 

rates, non-coincident connected demand factors, etc., 

developed during the evaluation of energy efficiency 

programs and used to calculate net savings. 

JMC The Joint Management Committee of utility and non-utility 

parties that manages the ENERGY STAR® Homes Program. 

kWh Kilowatt-hour – The basic unit of electric energy usage over 

time. One kWh is equal to one kW of power supplied to a 

circuit for a period of one hour. 

kW Kilowatt – A measure of electric demand – 1000 watts 

kW – Years See: Lifetime kW 

Lifetime The expected length of time, in years, that an installed 

measure will be in service and producing savings. 

Lifetime kW The expected demand savings over the lifetime of an 
30

installed measure, calculated by multiplying the annual peak 

kW reduction associated with a measure by the expected 

lifetime of that measure. It is expressed in units of kWyears. 

Lifetime MWh The expected energy savings over the lifetime of an installed 

measure, calculated by multiplying the annual MWh 

reduction associated with a measure by the expected lifetime 

of that measure. 

LIHEAP Low Income Heating Assistance Program 

Maximum Annual kW 

Savings 

Peak annual demand savings of a measure. At the program 

level, this equals the sum of the annual peak demand savings 

across all measures. 

Measure Specific technology or practice that produces energy and/or 

demand savings for which the company provides financial 

incentives. 

MPER Multi-Year Program Evaluation and Market Progress 

Reporting, or Market Progress and Evaluation Report, 

developed for various residential programs. 

MW Megawatt – a measure of electric demand equal to 1,000 

kilowatts. 

MWh Megawatt-hour – a measure of energy use over time equal to 

1,000 kilowatt-hours. 

NATE North American Technician Excellence Program 

NEEP Northeast Energy Efficiency Partnerships 

O&M Operation and Maintenance 

Off-Peak energy kWh The kWh reduction that occurs during the Company’s offpeak hours for energy. (Monday-Friday 9 p.m. to 8 a.m. and 

all day of weekends and holidays) 

On-Peak Energy kWh The kWh reduction that occurs during the Company’s onpeak hours for energy. (Monday-Friday 8 a.m. to 9 p.m., 

except holidays) 

Persistence Rate Percentage of first year energy or demand savings expected 

to persist over the life of the installed energy efficiency 

equipment; developed by conducting surveys of installed 

equipment several years after installation to determine 

presence and operational capability of the equipment. 

RCS Residential Conservation Services. Formerly Energy 

Conservation Services or ECS 

Seasonal (Winter/Summer) 

kW 

The net demand reduction during either the Winter or 

Summer seasons. 

Spillover Additional energy efficient equipment installed by customers 
31

that were influenced by the Company’s sponsored program, 

but without direct financial or technical assistance from the 

program. Spillover is separated into Participant and Nonparticipant factors. Non-participating customers may be 

influenced by product availability, publicity, education, and 

other factors that are affected by the program. 

Spillover Rate Estimate of energy savings attributable to spillover effects 

expressed as a percent of savings installed by participants 

through an energy efficiency program. 

VSD Variable Speed Drive 

WAP Weatherization Assistance Program 

Watt The basic electrical unit of power. 
32

Appendix 2. 2007 Evaluation Impact Parameters 

The table below presents the impact factors that were used to calculate the evaluated 

savings for the commercial and industrial programs in 2007. Impact parameters for the 

C&I New Construction, C&I Large Retrofit, C&I Small Retrofit and C&I Products & 

Services programs were updated based on the evaluation studies conducted by the 

Compact in 2007 and 2008. Impact factors for Government programs were not evaluated 

in 2007 or 2008. Impact factors shown below represent the common assumptions 

developed by Massachusetts program administrators, based on a review of best available 

information on measures in statewide programs. Impact factors are a subset of all of the 

assumptions used in planning and reporting. The impact factors in bold were updated 

based on evaluation studies. 

The impact factors in all of the tables in Appendix 2 are not the only assumptions that 

were updated in 2007, due to the fact that in 2007, the Compact undertook a major 

initiative to adopt assumptions that were consistent with the other Massachusetts Program 

Administrators. Many measure lives, free ridership rates, spillover rates, in-service rates, 

energy savings, loadshapes, demand savings, coincidence factors and non-electric 

benefits were updated. These tables do not provide a comprehensive overview of all of 

the assumptions that changed in 2007. Rather, they provide an overview of the impact 

factors that changed in 2007 due to evaluation studies. 

Table A2.1 Commercial & Industrial Program Evaluation Impact Factors 

BCR Activity Program End Use Measure 

Life

Free 

Ridership 

Rate

Spillover 

[Participant] 

Rate

Spillover 

[NonParticipant] 

Rate

InService 

Rate

kWh 

Realization 

Rate

C02a C&I Lost Opportunity C02a C&I New Construction ALght 15 13.00% 0.00% 2.60% 100% 100%

C02a C&I Lost Opportunity C02a C&I New Construction CMoDr 20 100.00% 0.00% 2.60% 100% 100%

C02a C&I Lost Opportunity C02b C&I Govt New Construction ALght 15 0.00% 0.00% 2.60% 100% 100%

C03a Large C&I Retrofit C03a C&I Large Retrofit ALght 13 35.20% 0.70% 2.60% 100% 100%

C03a Large C&I Retrofit C03a C&I Large Retrofit CMoDr 15 19.30% 0.00% 2.60% 100% 100%

C03a Large C&I Retrofit C03a C&I Large Retrofit DRefr 13 7.80% 0.40% 2.60% 100% 100%

C03a Large C&I Retrofit C03a C&I Large Retrofit BHVAC 23 12.50% 5.20% 2.60% 100% 100%

C03a Large C&I Retrofit C03c C&I Govt Large ALght 13 0.60% 3.40% 2.60% 89% 100%

C03a Large C&I Retrofit C03c C&I Govt Large CMoDr 15 0.00% 0.00% 2.60% 100% 100%

C03a Large C&I Retrofit C03c C&I Govt Large BHVAC 10 0.00% 0.00% 2.60% 100% 100%

C03b Small C&I Retrofit C03b C&I Small Retrofit ALght 13 7.70% 0.30% 2.60% 86% 104%

C03b Small C&I Retrofit C03b C&I Small Retrofit BHVAC 9 2.10% 25.40% 2.60% 100% 100%

C03b Small C&I Retrofit C03b C&I Small Retrofit DRefr 13 7.80% 0.40% 2.60% 100% 100%

C03b Small C&I Retrofit C03d C&I Govt Small ALght 13 0.60% 3.40% 2.60% 89% 100%

C03b Small C&I Retrofit C03d C&I Govt Small BHVAC 10 0.00% 0.00% 2.60% 100% 100%

C03b Small C&I Retrofit C03d C&I Govt Small CMoDr 15 0.00% 0.00% 2.60% 100% 100%

C03b Small C&I Retrofit C03d C&I Govt Small DRefr 11 0.00% 0.00% 2.60% 100% 100%

C02a C&I Lost Opportunity C04c C&I Products & Services ALght 15 27.90% 13.40% 2.60% 100% 100%

C02a C&I Lost Opportunity C04c C&I Products & Services CMoDr 15 28.80% 9.20% 2.60% 100% 100%

C02a C&I Lost Opportunity C04c C&I Products & Services BHVAC 15 14.80% 5.90% 2.60% 100% 100%

Note: Shaded cells indicate impact factors that are neither 100% for the In-Service Rate or kWh 

Realization Rate nor 0% for the Free Ridership, Participant Spillover, or Non-Participant Spillover Rates. 
33

The table below presents the impact factors that were used to calculate the evaluated 

savings for residential programs offered by the Cape Light Compact in 2007. Impact 

factors shown below represent common assumptions developed by Massachusetts 

program administrators, based on a review of best available information on measures in 

statewide programs. The impact factors in bold were updated based on evaluation 

studies. 

Table A2.2 Residential Program Evaluation Impact Factors 

BCR Activity Measure Measure 

Life

Free 

Ridership 

Rate

Spillover 

[Participant] 

Rate

Spillover 

[NonParticipant] 

Rate

InService 

Rate

A02a Residential Lost Opportunity CFL 7 2.00% 0.00% 0.00% 90%

A02a Residential Lost Opportunity FIXTUREIN 20 8.00% 4.00% 0.00% 95%

A02a Residential Lost Opportunity DISHWASHER 12 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity REFRIG 13 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HEATSYSTEM 18 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity AIRSEAL - electric 15 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERS - ES 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSC - ES 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSD - ES 15 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSS - ES 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERS - CP 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSC - CP 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSD - CP 15 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity HERSS - CP 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity LIHERS - ES 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity LIHERSC - ES 25 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity LIHERSD - ES 15 0.00% 0.00% 0.00% 100%

A02a Residential Lost Opportunity LIHERSS - ES 25 0.00% 0.00% 0.00% 100%

BCR Activity Measure Measure 

Life

Free 

Ridership 

Rate

Spillover 

[Participant] 

Rate

Spillover 

[NonParticipant] 

Rate

InService 

Rate

A03a Residential Retrofit 1-4 BOILRWATER 20 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 CFL 7 2.00% 0.00% 0.00% 90%

A03a Residential Retrofit 1-4 HOTWATER 7 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 HVAC 17 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 INDIRECTDH 20 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 SWITCH 20 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 TORCHIERE 8 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 T-STAT 10 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 AIRSEAL - electric 15 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 AIRSEAL - oil 15 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 AIRSEAL - gas 15 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 AIRSEAL - other 15 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 INSULATION - electric 25 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 INSULATION - other 25 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 INSULATION - gas 25 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 INSULATION - oil 25 2.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 FURNACE 20 0.00% 0.00% 0.00% 100%

A03a Residential Retrofit 1-4 REFRIGESV 13 35.00% 36.00% 0.00% 100%

A03a Residential Retrofit 1-4 REFRIGRV 1 35.00% 36.00% 0.00% 100%

BCR Activity Measure Measure 

Life

Free 

Ridership 

Rate

Spillover 

[Participant] 

Rate

Spillover 

[NonParticipant] 

Rate

InService 

Rate

A04a Residential Lighting CFL 5 0.00% 0.00% 0.00% 165%

A04a Residential Lighting CFLNCP 7 0.00% 0.00% 0.00% 165%

A04a Residential Lighting FIXTUREIN 20 8.00% 4.00% 0.00% 95%

A04a Residential Lighting FIXTUREOUT 6 12.00% 7.00% 0.00% 87%

A04a Residential Lighting TORCHIERE 8 6.00% 3.00% 0.00% 83%

A04b Residential Appliances CLOTHESWAS 11 10.00% 0.00% 0.00% 27%

A04b Residential Appliances ECMHEAT 20 0.00% 0.00% 0.00% 100%

A04b Residential Appliances DEHUMESV 12 0.00% 0.00% 0.00% 100%

A04b Residential Appliances DEHUMRV 4 0.00% 0.00% 0.00% 100%

A04b Residential Appliances ROOMACESV 11 0.00% 0.00% 0.00% 100%

A04b Residential Appliances ROOMACRV 4 0.00% 0.00% 0.00% 100%

Note: Shaded cells indicate impact factors that are neither 100% for the In-Service Rate nor 0% for the 

Free Ridership, Participant Spillover, or Non-Participant Spillover Rates. 
34

The table below presents the impact factors that were used to calculate the evaluated 

savings for low income programs offered by the Cape Light Compact in 2007. Impact 

factors shown below represent the common assumptions developed by Massachusetts 

program administrators, based on a review of best available information on measures in 

statewide programs. The Compact’s low income program impact factors were not 

updated in 2007 due to evaluation studies. 

Table A2.3 Low Income Program Evaluation Impact Factors 

BCR Activity Measure Measure 

Life

Free 

Ridership 

Rate

Spillover 

[Participant] 

Rate

Spillover 

[NonParticipant] 

Rate

InService 

Rate

B03a Low-Income Retrofit 1-4 CFL 16 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 HEATSYSTEM 15 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 HOTWATER 7 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 AIRSEAL - electric 15 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 AIRSEAL - oil 15 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 AIRSEAL - gas 15 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 INSULATION - electric 25 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 INSULATION - gas 25 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 INSULATION - oil 25 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 REFRIGESV 13 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 REFRIGRV 1 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 DEHUMESV 12 0.00% 0.00% 0.00% 100%

B03a Low-Income Retrofit 1-4 DEHUMRV 4 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily CFL 16 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily CLOTHESWAS 11 10.00% 0.00% 0.00% 27%

B03b Low-Income Retrofit Multifamily HEATSYSTEM 20 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily HVAC 15 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily T-STAT 5 2.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily AIRSEAL - electric 15 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily AIRSEAL - gas 15 0.00% 0.00% 0.00% 100%

B03b Low-Income Retrofit Multifamily INSULATION - gas 21 0.00% 0.00% 0.00% 100%

Note: Shaded cells indicate impact factors that are neither 100% for the In-Service Rate nor 0% for the 

Free Ridership, Participant Spillover, or Non-Participant Spillover Rates. 
35

Appendix 3. Post Program Savings Attributed to Selected 2007 

Market Transformation Initiatives 

The Compact has not developed estimates of post program savings associated with 

market transformation initiatives. It is our understanding that this issue has not been 

considered a high priority for DOER or other Program Administrators. 
36

Appendix 4. Calculation of Shareholder Incentive 

The Cape Light Compact does not require shareholder incentives to implement its energy 

efficiency programs. Therefore, this section is not relevant to the Compact. 
37

Appendix 5. Summary of 2007 Energy Efficiency Evaluation 

Reports 

The following studies were used in preparing the evaluated results presented in this 

Annual Report. The executive summaries of these reports are attached below. The full 

copies of these reports are available from the Compact upon request. 

• Evaluation of the Massachusetts New Homes with ENERGY STAR, Findings and 

Analysis, April 24 ,2008 by Nexus Market Research, Inc. and Dorothy Conant, 

Consultant. 

• The Massachusetts New Homes with ENERGY STAR PROGRAM, 2007 

Progress Report, Final Report, May 30, 2008. 

• MassSave Final Summary QA/QC and Impact Study Report, April 8, 2008 by 

RLW Analytics, Inc. 

• Market Progress and Evaluation Report (MPER) for the 2007 Massachusetts 

ENERGY STAR Lighting Program, June 16, 2008, by Nexus Market Research, 

Inc. and RLW Analytics, Inc. 

• Residential Lighting Measure Life Study, June 4, 2008, by Nexus Market 

Research, Inc. and RLW Analytics, Inc. 

• Coincidence Factor Study, Residential Room Air Conditioners, Prepared for: 

Northeast Energy Efficiency Partnership New England Evaluation and State 

Program Working Group, June 23, 2008. 

• Coincidence Factor Study, Residential and Commercial Industrial Lighting 

Measures, Prepared for New England State Program Working Group, Spring 

2007, Prepared by RLW Analytics. 

• Multiple Small Business Services Programs Impact Evaluation 2007, Final Report 

Update, by Summit Blue Consulting, September 2, 2008. 

• Freeridership and Spillover Study 2007 by PA Consulting: Cape Light Compact 

Results, June 23, 2008. 

• Freeridership and Spillover Study 2007 by PA Consulting: National Grid Results, 

June 23, 2008. 
22 Haskell Street, Cambridge, MA 02140 

Phone: (617) 497-7544 Fax: (617) 497-7543 

www.nexusmarketresearch.com

Evaluation of the Massachusetts New Homes 

with ENERGY STARÆ 

Findings and Analysis 

Executive Summary 

April 24, 2008 

Submitted to: 

Joint Management Committee 

Submitted by: 

Nexus Market Research, Inc. 

Dorothy Conant 
Nexus Market Research

Table of Contents 

Executive Summary ....................................................................................................................1 

Program Overview ..............................................................................................................................1 

Conclusions ........................................................................................................................................2 

Recommendations...............................................................................................................................3 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 1

Nexus Market Research

Executive Summary 

This document is a market progress evaluation report (MPER) on the Massachusetts New Homes 

with ENERGY STARÆ Program run by the Joint Management Committee (JMC) in 

Massachusetts. This report contains the findings and analysis of the evaluation work conducted 

in 2007, with the individual evaluation reports on which the findings and analysis are based as 

appendices. The evaluation work conducted for the 2007 program includes: 

• A process evaluation focusing on issues related to the transition to a new 

implementation contractor and program changes introduced in 2007. Interviews 

were conducted with JMC members, non-utility parties (NUPs), implementation 

contractor personnel and HERS raters. (Process Evaluation, Appendix A) 

• In-depth interviews with forty ENERGY STAR builders (Builder Interviews, 

Appendix B) 

• Interviews with the managers of six ENERGY STAR Homes programs using 

independent HERS raters to certify homes and/or recruit builders with builders 

paying for HERS raterís services, and four companies providing HERS rating 

services in Massachusetts (Review of Market-Based HERS Model, Appendix C) 

• Estimation of the incremental costs required to reach ENERGY STAR certification 

and more stringent HERS levels for single and multifamily homes with different 

heating systems (Incremental Cost Analysis, Appendix D) 

 

In-depth builder interviews and home buyer surveys were previously completed for the 2002, 

2003, 2004, and 2006 programs while an incremental cost analysis was previously conducted in 

2002. Where appropriate, the findings from the previous year studies are compared. 

Program Overview 

The Massachusetts New Homes with ENERGY STAR Program certified more than 1,200 

housing units in 2007, bringing the total number certified since program inception to more than 

13,300. The estimated percentage of new homes completed in Massachusetts that are ENERGY 

STAR certified climbed from three percent in 1999 to 16% in 2006. In 2007 the penetration of 

ENERGY STAR homes fell to nine percent. The penetration of single family ENERGY STAR 

homes remained consistent at seven percent, but the penetration of multifamily ENERGY STAR 

units dropped sharply because multifamily units in high-rise buildings not built under residential 

building code are no longer eligible for ENERGY STAR certification. 

The total number of housing units certified in 2007 is slightly lower than one-half the number 

certified in 2006. In addition to the exclusion of units in multifamily buildings not built using 

residential building code, total building permits issued in Massachusetts in 2007 were 24% lower 

than in 2006. Excluding multifamily units in high-rise buildings not built using residential 

building code, more than 1,700 housing units were certified in 2006. The number of comparable 

housing units certified in 2007 is 26% lower, which is consistent with the 24% percentage drop 

in permits issued. 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 2

Nexus Market Research

The year 2007 was a transition period which presented several challenges for the program. First, 

it had to deal with changing to a new implementation contractor, ICF. The program also began 

moving toward a more market-driven model. Meanwhile, stricter EPA requirements for 

ENERGY STAR certification, involving Thermal Bypass Checklist (TBC) requirements and 

duct leakage standards, had to be met by all homes, regardless of when they signed up. Thus, all 

builders needed to be informed of the change in contractors, be assigned new account managers 

and HERS raters, and be supported in dealing with stricter requirements for certification. 

Conclusions 

The four reports summarized in this MPER and attached as appendices support the following key 

conclusions: 

• Overall success. The program has overall successfully dealt with the transitions it 

faced in 2007. It is now transitioned to ICF, and the sponsors and NUPs are satisfied 

with ICFís performance implementing the program in 2007. Moreover, the program 

sustained a high level of builder satisfaction despite the many changes builders had to 

deal with in 2007. 

• Overall marketing. Program marketing is likely to be one of the most challenging 

issues the program will have to deal with in 2008. Given budget constraints, the 

programís marketing director is working to identify and introduce marketing 

approaches that will produce the maximum bang for the buck. 

• Marketing support. The 2008 agreement form lists the marketing support available 

through the program and asks builders to indicate what support they are interested in 

getting. This is an easy and cost effective way of letting all participating builders 

know what is available and identifying builders who want marketing support. The 

program is addressing the need to update and expand the amount of information 

available to builders and consumers on their web site. 

• Training. The program is addressing a critical need to provide more training to 

builders and HVAC and insulation contractors to ensure projects pass the TBC and 

meet program duct and air sealing standards. 

• Incremental costs. Incremental costs for reaching the minimum ENERGY STAR 

level from baseline building practices are slightly lower than five years ago; however, 

incremental costs for reaching higher efficiency, lower HERS levels, rise sharply, 

especially going from HERS 85 to HERS 70. Since over half of the housing units 

certified in 2007 achieved HERS indices of 70 or lower, the incremental cost of 

getting to a HERS index of 65 or lower may be manageable for many builders. 

Tiered incentives, such as those being offered in 2008, encourage builders to strive 

for lower HERS indices. 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 3

Nexus Market Research

• Paying for HERS rater services. Most builders do not know the cost of the HERS 

rater services they receive from the program and all interviewed HERS raters say they 

plan to charge builders for extra hours of support in 2008. A big questions going 

forward is whether or not builders will be willing to pay for additional hours of 

support and, if they are not willing to pay, what impact this will have on the number 

of homes failing to pass the final inspection. 

• Moving beyond ENERGY STAR. A majority of sponsors, NUPs, ICF staff, HERS 

raters and builders support raising the Massachusetts Energy Code to ENERGY 

STAR standards. With the average HERS index dropping each year, some 

interviewees strongly support moving the program to very high performance levels 

over time (e.g., zero net-energy homes). Also, the sponsors and NUPs agree that the 

program should look into quantifying and claiming non-energy benefits. 

• Moving to a market-driven model. Multiple program designs are compatible with a 

successful market-based model using independent HERS raters. Moreover, existing 

companies offering HERS rating services in Massachusetts have the resources 

available to meet the Massachusetts programís needs for rating services and are now 

providing those services to builders. 

Recommendations 

Based on these conclusions and findings presented in this report, which are addressed in more 

detail in the appendices, the NMR evaluation team offers several sets of recommendations to 

ensure the program continues to successfully deal with changes in the marketplace and manage 

its transition to a market-driven model. Before listing detailed recommendations, it is important 

to note the need for an organized system of tracking the actions taken throughout the year to 

address the large number of recommendations that are generated by an annual MPER. For 

example, the JMC may wish to include quarterly discussions on the specific actions taken and 

progress made in addressing the MPERís recommendations, including a discussion of the 

reasons some recommendations cannot or should not be addressed at this time. 

The first set of recommendations addresses the programís need to find as many cost effective 

ways as possible to market itself, support builders marketing their ENERGY STAR homes, and 

educate consumers on the benefits of buying and living in an ENERGY STAR home. Builders 

say the public does not value the ENERGY STAR label on homesóthis needs to change for the 

program to grow the market share of ENERGY STAR homes and move to a market-driven 

model. Having said this, the NMR team recognizes that some sponsors reach their participation 

goals and have all available funds committed before the end of the year and, therefore, are not 

necessarily looking to increase marketing in their service territories. 

• Maximize free press opportunities. Alert local newspapers to ENERGY STAR 

model home openings, builders who are installing renewable technologies or building 

ENERGY STAR-certified green homes and any other events that could merit 

coverage. 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 4

Nexus Market Research

• Track marketing activity and, to the extent feasible, its effect on interest and 

participation in the program. Sponsors want to see that marketing efforts are 

producing tangible results before increasing marketing budgets. 

• Formulate a positive, productive response for any oversubscription that may 

occur. Explain in marketing materials that there are a limited number of incentives 

available. Explain that if all available incentives are accounted for, builders already 

participating in the program can hire their HERS rater to certify additional homes, 

and that the program will continue to provide information on HERS rater certification 

and technical support resources available to new builders and homeowners interested 

in building an ENERGY STAR home. Explore being able to claim savings from 

these homes as spillover in regulatory filings. 

As already noted, the program has taken an important step in encouraging and supporting 

builders in the marketing of their ENERGY STAR homes by listing the marketing support 

available in the 2008 agreement and asking builders to indicate what support they are interested 

in getting. Additional recommendations in this area include: 

• Offer co-op advertising to builders. Three-fourths of builders interviewed in 2007 

say they would be ìsomewhatî or ìveryî likely to take advantage of co-op 

advertising. 

• Consider compensating builders willing to have an ENERGY STAR model home 

open to the public for several months. The model home would have informational 

materials available for visitors and displays explaining what goes into building an 

ENERGY STAR home and potential energy savings. Over one-third of interviewed 

builders say they would be very interested in doing this. 

• Consider promoting participating builders as an approach that can: 1) address 

builders' concerns that the program does not partner with them enough, 2) leverage 

builders to do more of what the program wants them to do, such as market the 

ENERGY STAR status of their homes more aggressively and participate in more 

trainings, 3) reward leading builders for their efforts while sending business their 

way, and 4) set leading builders up as ambassadors for the program. Consider setting 

up a tiered promotion schedule: the more builders do (the more homes they have 

certified, the lower the HERS indices they achieve, the more trainings they or their 

subcontractors attend, etc.), the more they will be promoted in web site listings, 

named in advertising, featured in newspaper articles, and awards, etc. 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 5

Nexus Market Research

In the area of consumer marketing, the first recommendation bears repeating from previous 

yearsí MPERs. 

• In consumer marketing, stress that the ONLY way to be sure a home is energy 

efficient is to have it verified by a third party using blower door and duct blaster 

tests, and this is what the program does. Ninety percent of builders say it is 

important to market the program to consumers, and 78% of builders say they would 

be more likely to use ENERGY STAR in their marketing, or to increase the emphasis 

on ENERGY STAR in their marketing, if the program marketed directly to 

consumers. 

• Incorporate ENERGY STAR Homes in sponsor marketing for ENERGY STAR 

lighting and appliances. The message could be a one liner saying, ìAnd, if you are 

buying a new home, look for an ENERGY STAR-labeled home.î Builders say home 

buyers know about ENERGY STAR appliances, but not ENERGY STAR homes. 

• Target marketing to market players working with new-home buyers. For 

example, the program could develop, and distribute ENERGY STAR home brochures 

to mortgage lenders and ask them to hand a brochure to every customer applying for a 

mortgage for a new home in Massachusetts. 

• Consider marketing messages/approaches that leverage other efforts in the state 

that encourage energy-efficient building, such as the net zero energy buildings task 

force recently announced by Governor Patrick.1

• Seek out and take advantage of opportunities to partner with green building 

programs to reach consumers interested in energy efficiency. Homebuyers are 

more likely to have heard of green building than ENERGY STAR-certified homesó

builders say ìgreenî is the buzz word in the new construction market. 

• Find a way to get green building programs to include ENERGY STAR in their 

marketing messages. The message should be that homes need to be ENERGY 

STAR certified to meet their programís energy-efficiency requirements, and that the 

New Homes with ENERGY STAR Program offers incentives and technical support 

to builders of ENERGY STAR homes. 

The next set of recommendations deals with the programís long-term transition to a marketdriven model. Given recent program changes in certification requirements and the 

implementation contractor, the NMR team agrees with interviewees who believe it would be 

good for the program to have a year or two of continuity and stability before introducing any 

major changes affecting builders. Beyond that, the NMR team offers the following 

recommendations for moving on the road to a market-driven model, many of which are also 

suggested by interviewees: 

 

1

 Massachusetts Governor Deval Patrick announced the establishment of a task force on net zero energy buildings on 

March 12, 2008 at the Northeast Sustainable Energy Association's (NESEA) Building Energy Conference and Trade 

Show at the Seaport World Trade Center in Boston. 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 6

Nexus Market Research

• Delay having builders pay for HERS rater services at least until 2010, but begin 

preparing them for having to pay for at least a portion of those services. Many 

builders are totally unaware of the cost of hiring a HERS rater and many will not be 

happy about having to pay for a HERS raterónothing that adds to their costs, 

especially in a slow market, is likely to be well received. It will be important to 

soften the blow by making it clear that until now the Massachusetts program has 

been paying for all HERS rater services and, assuming the Massachusetts 

program increases incentives to help defray the cost of hiring a HERS rater, 

builders should be told how much the incentives were increased to help them 

cover the cost of hiring a HERS rater. 

• All reviewed market-driven model programs offer some level of builder and/or HERS 

rater training. Training is important to making these models work. Assess buildersí, 

subcontractorsí and HERS ratersí needs for additional training and offer free or 

low-cost training opportunities. Builder and subcontractor training are 

especially important as the program looks to move toward higher performance 

tiers. 

• Increase communication and coordination with the CoolSmart Program to better 

address quality central air conditioning installations. 

• Continue to explore options for including high-rise multifamily projects in the 

program, given their importance in the new home market in Massachusetts.

• If the Massachusetts program continues to offer on-site technical support to builders 

who are having trouble meeting program requirements it will be important to have 

clear protocols in place for deciding when the HERS rater should go to the 

program to provide support to a builder, and when it would be appropriate for 

the HERS rater to offer additional support for an additional cost. 

• Tied to the previous recommendation, the program should tell all HERS raters 

marketing their services to builders to clearly distinguish between services that 

address meeting program requirements for certification and value-added 

services available at an additional cost. 

The final set of recommendations deals with maintaining the operational effectiveness of the 

program, especially in view of changes in 2008. 

• Keep everyone informed. It is important that builders get consistent information 

and direction to avoid confusion and complaints. At the program level, annual group 

meetings with builders and HERS raters to review program changes and address 

known issues are a start; perhaps biannual meetings with HERS raters in the first two 

years of the market-based delivery model would ensure even better communication. 

At the builder/project level, the HERS rater is likely to be the builderís main contact 

and it is important to take steps to ensure the technical support and advice being given 
Evaluation of the Massachusetts New Homes with ENERGY STAR Program Page 7

Nexus Market Research

to builders to ensure projects meet the programís technical standards are consistent 

and appropriate. 

• Schedule at least one JMC meeting a year devoted to talking with account 

managers and HERS raters about what is happening in the field. The process 

evaluation interviews revealed that sponsors were unaware of how many hours of 

technical support many builders need and that these builders were not being charged 

for extra support. 

• Address the irregularity that some HERS raters consistently failed homes in 

2007 if there was no manual J load calculation and others did not. This issue has 

already been addressed in meetings with HERS raters serving the program in 2008. 

Any remaining irregularities, if they exist will likely be identified through the quality 

control inspections planned for 2008. 

• Closely track bulb installations in 2008 to assess the impact on the number of bulbs 

installed of having buildersí electricians, rather than HERS raters, order and install 

bulbs. Be prepared to step in, if necessary, to facilitate the process and make sure 

bulbs are available at the desired time and installed in all appropriate locations. 

• Implement a simpler process for reviewing and approving external 

communication pieces. One option is to have one JMC member take responsibility 

for consolidating all JMC member comments and edits and providing one final set of 

comments/edits to ICF. JMC members could take turns doing this so it does not 

become a burden for any one sponsor. 

• Enlist the help of builders and other interested parties in efforts to encourage 

towns and communities to raise their Energy Code to ENERGY STAR 

standards. Many builders support a move to ENERGY STAR standards and they 

can provide first hand testimony based on real experience that achieving ENERGY 

STAR standards is neither onerous nor cost prohibitive. Also, explore options for 

claiming savings if the program directly facilitates code improvements to the higher, 

ENERGY STAR standards. 
The Massachusetts New Homes with ENERGY STAR®

Program

2007 PROGRESS REPORT

Final Report

June 13, 2008

Submitted to:

The Massachusetts New Homes with ENERGY STAR Program

Joint Management Committee

Submitted by:

Dorothy Conant, Consultant 
Table of Contents

1. INTRODUCTION.................................................................................................. 1

2. METRICS ............................................................................................................. 1

3. OVER THE YEARS.............................................................................................. 1

3.1 Massachusetts Housing Permits......................................................................................... 2

3.2 Annual Signed Housing Units ............................................................................................. 4

3.2.1 Recruited Housing Units Compared to Statewide Permits Issued .........................4

3.2.2 Status of Signed Housing Units.............................................................................. 5

3.3 Certified Housing Units as Percentage of Statewide Completed Housing Units................6

3.4 HERS Ratings ..................................................................................................................... 8

3.5 Spending per Participating Housing Unit ..........................................................................11

4. 2007 PROGRAM ACTIVITY............................................................................... 14

4.1 Projects and Housing Units Recruited ..............................................................................14

4.2 2007 Recruited Projects by Size—Number of Units .........................................................17

4.3 Builder Mix—New and Repeat Participants......................................................................18

4.4 2007 Completions and Signings ......................................................................................20

4.4.1 Distribution of Completions across Sponsor Territories .......................................23

4.4.2 Distribution of Signings across Sponsor Territories .............................................26

5. PENETRATION OF ENERGY STAR MEASURES............................................ 29

6. DUCT LEAKAGE............................................................................................... 31
List of Figures

Figure 3.1 Massachusetts Housing Permits Issued 1980 – 2007 ...............................................................2

Figure 3.2 Year-to-Date Total Permits Issued 2002 – 2008 ........................................................................ 3

Figure 3.3 Year-to-Date Single family Permits Issued 2002 – 2008............................................................3

Figure 3.4 Year-to-Date Multifamily Permits Issued 2002 – 2008 ...............................................................3

Figure 3.5 Annual Signed Housing Units ..................................................................................................... 4

Figure 3.6 Year-End 2006 Status of Signed Housing Units......................................................................... 5

Figure 3.7 ENERGY STAR Completions as Percent of State-Wide Completions.......................................7

Figure 3.8 ENERGY STAR, EMU and Code Plus Completions as Percent of State Completions .............7

Figure 3.9 1999 – 2006 HERS Ratings........................................................................................................ 8

Figure 3.10 2007 HERS Indices—All Homes ............................................................................................10

Figure 3.11 2007 HERS Indices—Single Family Homes...........................................................................10

Figure 3.12 2007 HERS Indices—Multifamily Units ..................................................................................10

Figure 3.13 Annual Program Spending......................................................................................................12

Figure 3.14 Annual Spending per Housing Unit ........................................................................................13

Figure 4.1 Historical Signed Projects by Housing Type.............................................................................14

Figure 4.2 Historical Signed Housing Units by Housing Type ...................................................................15

Figure 4.3 Percent of 2003 - 2007 Market Rate versus Low Income Projects Signed..............................16

Figure 4.4 Percent of 2003 - 2007 Market Rate versus Low Income Housing Units Signed ....................16

Figure 4.5 2003—2007 Signed Projects by Number of Housing Units per Project ...................................17

Figure 4.6 2007 Signed Projects by Number of Housing Units per Project...............................................18

Figure 4.7 Historical Percentages of Signed Projects from New Builders.................................................19

Figure 4.8 All 2007 Completions by Housing Type....................................................................................20

Figure 4.9 2007 Market Rate Completions by Housing Type ....................................................................21

Figure 4.10 2007 Low Income Completions by Housing Type ..................................................................22

Figure 4.11 Electric Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus Housing Units....23

Figure 4.12 Gas Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus Housing Units ..........24

Figure 4.13 Electric Sponsor Signed ENERGY STAR and Code Plus Projects and Housing Units .........26

Figure 4.14 Electric Sponsor Signed ENERGY STAR and Code Plus Market Rate and Low Income 

Housing Units..............................................................................................................................................26

Figure 4.15 Gas Sponsor Signed ENERGY STAR and Code Plus Projects and Housing Units ..............27

Figure 4.16 Gas Sponsor Signed ENERGY STAR and Code Plus Market Rate and Low Income Housing 

Units ............................................................................................................................................................27

Figure 5.1 Percent of Certified Housing Units Installing ENERGY STAR Lighting and Appliance 

Measures 2002 – 2007 ...............................................................................................................................29

Figure 5.2 Percent of ENERGY STAR-Certified Housing Units Installing ENERGY STAR Heating 

Systems, Windows and Central Air Conditioning .......................................................................................30

Figure 5.3 Comparison of 2007 ENERGY STAR versus Code Plus Housing Units Installing ENERGY 

STAR Heating Systems, Windows and Air Conditioning ............................................................................31

List of Tables

Table 3.1 Annual Program Spending, Signings and Completions.............................................................11

Table 4.1 Electric Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus Housing Units .......25

Table 4.2 Gas Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus Housing Units .............25

Table 4.3 Electric Sponsor Signed Total ENERGY STAR and Code Plus Projects and Housing Units ...28

Table 4.4 Gas Sponsor Signed Total ENERGY STAR and Code Plus Projects and Housing Units ........28
1. Introduction

The annual Massachusetts New Homes with ENERGY STAR®

2. Metrics

Program 2007 Progress Report is 

a summary of 2007 program activity. Program performance information includes historical as 

well as current information to show the growth of the program over time. 

There were no metrics in 2007. 

3. Over the Years 

The figures on the following pages show historical data on housing permits issued, homes

recruited and certified, and the program’s achievements since 1999. They show the number of 

housing units recruited each year, the average HERS ratings of homes completed in each year, 

the average cost per signed housing unit and per completed housing unit each year, and 

completed housing units each year as a percent of estimated total annual housing units completed 

in Massachusetts. All 2007 data on signings and completions include homes certified by ICF, 

Conservation Services Group (CSG) and Unitil, and homes recruited by ICF. As these figures 

will show, the number of housing permits issued, the number of housing units signed and the 

number of ENERGY STAR housing units completed in 2007 are all lower than in 2006. 

However, single family ENERGY STAR homes maintained their share of new single family 

homes completed in Massachusetts. As of the end of 2007, the program has ENERGY STAR 

certified over 13,000 housing units. 
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 2

3.1 Massachusetts Housing Permits

The numbers of both single family and multifamily permits issued in Massachusetts fell for the 

second year in a row. Single family permits issued in 2007, at 8,928 permits, are at their lowest 

level during the 1980 to 2007 period. (Figure 3.1) Annual multifamily permits issued, which 

grew consistently from 2002 through 2005, also dropped in 2006 and 2007, but remain above 

1990 through 2002 levels. In the last two years, the number of total permits issued has fallen by 

36%, the number of single family permits issued by 37% and the number of multifamily permits 

issued by 35%.

Figure 3.1 Massachusetts Housing Permits Issued 1980 – 2007

24,171

15,358

19,580

45,215

8,928

28,110

6,430

17,105

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

50,000

198019821984198619881990199219941996199820002002200420062008

Massachusetts Housing Permits Issued

Total Single Family Multifamily

Multifamily

Single Family

Total Massachusetts Permits

Figure 3.2, Figure 3.3, and Figure 3.4 on the following page show year-to-date total, single 

family and multifamily permits issued from 2002 through April 2008.1

 1 Total permits for each year are the final revised annual totals which may be higher or lower than the published 

December year-to-date totals.

Total permits issued in 

2007 are 22% lower, single family permits 18% lower and multifamily permits 29% lower than 

in 2006. Total permits issued January through April 2008 are 38% lower, single family permits 

39% lower and multifamily permits 38% lower than in the first four months of 2007. 
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 3

Figure 3.2 Year-to-Date Total Permits Issued 2002 – 2008

17,465

20,257

22,477

24,171

19,580

14,874

3,128

0

5,000

10,000

15,000

20,000

25,000

Jan Feb Mar April May June July Aug Sept Oct Nov Dec

2002 2003 2004 2005 2006 2007 2008

Total Permits

Year-to-Date

Figure 3.3 Year-to-Date Single family Permits Issued 2002 – 2008

13,636

13,037

14,842

14,223

10,916

8,703

1,559

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

Jan Feb Mar April May June July Aug Sept Oct Nov Dec

2002 2003 2004 2005 2006 2007 2008

Single Family Permits

Year-to-Date

Figure 3.4 Year-to-Date Multifamily Permits Issued 2002 – 2008

3,829

7,220

7,635

9,948

8,664

6,171

1,569 0

2,000

4,000

6,000

8,000

10,000

12,000

Jan Feb Mar April May June July Aug Sept Oct Nov Dec

2002 2003 2004 2005 2006 2007 2008

Multifamily Permits

Year-to-Date
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 4

3.2 Annual Signed Housing Units

3.2.1 Recruited Housing Units Compared to Statewide Permits Issued

Figure 3.5 shows new housing units recruited to participate in the Massachusetts program as a 

percentage of all housing permits issued in the state fell only a fraction of a percent in 2007—

from 13.2% in 2006 to 13.0% in 2007. The number of signed housing units and the number of 

permits issued each fell by virtually the same percentage from 2006 to 2007—the number of 

signings fell 23% and the number of permits fell 22%—resulting in almost no change in signings 

as a percentage of permits issued. 

Prior to 2006, signings include only housing units signed and committed to being built to 

ENERGY STAR standards. 2006 signings include both ENERGY STAR and Energy Measure 

Upgrade (EMU) housing units and 2007 signings include both ENERGY STAR and Code Plus 

housing units. Also, from 2003 through 2006 signings were tracked by housing type. The 2007 

signing information is based on signed agreements ICF received in 2007 and the number of 

housing units in those agreements allocated to 2007 and 2008; no breakdown of signings by 

housing type is available. 

Figure 3.5 Annual Signed Housing Units

(Percent of MA Statewide Permits)

5%

12%

16%

14% 15%

20%

8%

12%

13.0%

13.2% 

10%

28%

30%

16.3% 1,409 14%

8%

10.7% 1,171

0%

5%

10%

15%

20%

25%

30%

35%

1999 2000 2001 2002 2003 2004 2005 2006 2007

All Signings Multifamily Single Family

Multifamily

Single Family

ENERGY STAR & Code Plus Signings

 Percent of Permits

Total Signings
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 5

3.2.2 Status of Signed Housing Units

Since its inception, the Massachusetts program has certified over 13,000 ENERGY STAR 

housing units. Previous progress reports showed the status at the end of the current reporting 

year of all homes recruited in each year from 1999 on. With the change to a new implementation 

contractor in 2007, using a different data tracking approach, detailed information on projects by 

the year they originally signed up to participate in the program is no longer available. This does 

not mean one tracking approach is better than another, it simply means information is tracked 

differently, with some information being tracked in more detail and some in less detail. 

Consistent with the purpose of the annual progress reports to provide historical data on program 

activity, Figure 3.6 shows the reported status at the end of 2006 of all 20,118 housing units 

signed up to participate in the program from 1999 through 2006. 

Figure 3.6 Year-End 2006 Status of Signed Housing Units

(1999-2006 Signings)

18%

59%

23%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Year-End 2006 Status of 20,118 Housing Units Signed 1999-2006 

Dropped Out

Active

Completed
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 6

3.3 Certified Housing Units as Percentage of Statewide Completed 

Housing Units

Figure 3.7 on the following page shows annual ENERGY STAR housing units certified each 

year through the program as a percentage of estimated total annual completed housing units in 

Massachusetts. The housing units certified in any year include housing units recruited in 

previous years. From 1999 through 2006 the number of housing units certified through the 

program increased each year, both in number and in percent of total completed housing units, 

peaking in 2006 at 2,610 housing units and 15.5% of the market. In 2007, both the number of 

total housing units certified and their share of total housing units completed in Massachusetts 

dropped sharply.

Figure 3.8 on the following page is the same as Figure 3.7 except that the 2006 and 2007 

numbers include housing units completed through the program under the EMU and Code Plus 

participation paths. Including housing units participating though non-ENERGY STAR paths 

increases the number of housing units completed through the program from 2,610 to 3,318 in 

2006 and from 1,286 to 1,616 in 2007, and increases the percentage of completed housing units 

participating in the program from 15.5% to 19.7% in 2006 and from 9.0% to 11.2% in 2007. 

In 2003, the program began tracking recruited and completed homes under the Census Bureau 

single family and multifamily housing category definitions, which is how housing permit data 

are reported. Under the Census Bureau definitions, single family includes fully detached

housing units, semi detached (semi attached, side-by-side) housing units, row houses, and 

townhouses. In the case of attached units, each must be separated from the adjacent unit by a 

ground-to-roof wall and must not share heating/air-conditioning systems or inter-structural 

public utilities such as water supply, power supply, or sewage disposal lines. Because housing 

units certified as ENERGY STAR since 2003 are tracked using the Census Bureau definitions, it 

is possible to separately calculate the percentages of multifamily and single family housing units 

completed in the state that are ENERGY STAR certified. Including EMU and Code Plus 

housing units in 2006 and 2007 raises the percentage of completed multifamily housing units that 

participated in the program from 35.2% to 47.6% in 2006 and from 14.1% to 19.6% in 2007, and 

the percentage of completed single family homes that participated in the program from 6.8% to 

7.3% in 2006 and from 7.0% to 8.1% in 2007.
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 7

Figure 3.7 ENERGY STAR Completions as Percent of State-Wide Completions

3% 4%

6%

9% 10%

12%

19%

9.0% 1,286

9%

15.5%

2,610

14.1% 533

35.2% 

1,820

18%

25%

7.0% 733 6%

6.8%

6% 7% 790

0%

5%

10%

15%

20%

25%

30%

35%

40%

1999 2000 2001 2002 2003 2004 2005 2006 2007

All ENERGY STAR Completions

Multifamily ENERGY STAR

Single Family ENERGY STAR 

Multifamily

Single Family

Program ENERGY STAR Completions as 

Percent of Statewide Completions

Total Completions

Figure 3.8 ENERGY STAR, EMU and Code Plus Completions as Percent of State 

Completions

3% 4%

6%

9% 9% 10%

12%

19.7%

18% 19%

6% 6% 7%

11.2%

1,616

25%

47.6%

2,462

19.6%

767

7.3%

8.1%

849

0%

10%

20%

30%

40%

50%

60%

1999 2000 2001 2002 2003 2004 2005 2006 2007

All Completions All Multifamily All Single Family

Multifamily

Single Family

Program ENERGY STAR, EMU & Code Plus 

Completions as Percent of Statewide Completions

856

3,318

Total Completions
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 8

3.4 HERS Ratings 

The HERS index approach to rating homes was introduced in 2006. Most homes certified in 

2006 (92%) were rated using the traditional HERS score. Homes completed in 2007 were rated 

using the HERS index approach. 

Figure 3.9 shows 1999 through 2006 average and maximum HERS scores for participating 

certified homes. As shown, the average HERS score for all certified housing units increased 

each year: average single family HERS scores varied little, while average multifamily HERS 

scores consistently increased. The average HERS score of housing units certified in 1999 was 

86.7 and by 2006 climbed to 89.3. This 2.6 point increase in the average HERS score equates to 

an increase of 13% in energy savings.

2 Figure 3.9 also shows that each year some certified 

homes achieved HERS scores exceeding the 86.0 score required for ENERGY STAR

certification by more than five points, scoring 91.0 or better. In three years some housing units

achieved HERS scores of 94.0 or higher, which represents energy savings of 40% over a home 

with a minimum ENERGY STAR qualifying HERS score of 86.0. 

Figure 3.9 1999 – 2006 HERS Ratings

(Completed Housing Units Each Year Rated Using the Traditional HERS Score) 88.2 87.9 94.1 92.1 88.1 88.2 88.2 94.0 92.7 88.1 89.5 89.0 93.6 93.0 88.489.6 89.3 92.6

94.1

88.1

82.0

84.0

86.0

88.0

90.0

92.0

94.0

96.0

Average

Single

Family

Average

Multifamily

Average All

Homes

Max Single

Family

Max

Multifamily

Traditional HERS Score

2003 2004 2005 2006

 

2 Based on five percent increase in savings per point increase in HERS score.
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 9

The HERS Index rating system introduced in 2006 is described as follows on 

the energystar.gov website:3

Figure 3.10

 

“The HERS Index is a scoring system established by the Residential Energy 

Services Network (RESNET) in which a home built to the specifications of the 

HERS Reference Home (based on the 2006 International Energy Conservation 

Code) scores a HERS Index of 100, while a net zero energy home scores a HERS 

Index of 0. The lower a home’s HERS Index, the more energy efficient it is in 

comparison to the HERS Reference Home. Each 1-point decrease in the HERS 

Index corresponds to a 1% reduction in energy consumption compared to the 

HERS Reference Home. Thus a home with a HERS Index of 85 is 15% more 

energy efficient than the HERS Reference Home and a home with a HERS Index 

of 80 is 20% more energy efficient.” 

Homes in Massachusetts must achieve a HERS Index of 85 or lower to be ENERGY STAR 

certified. Assuming a one percent increase in energy savings per point decrease in the HERS 

Index, a HERS Index of 50 reflects 50% energy savings compared to the HERS reference home 

(2006 International Energy Conservation Code) and energy savings 35 percentage points higher

than needed to be ENERGY STAR certified (85.0 HERS Index). 

, Figure 3.11 and Figure 3.12 on the next page show the individual HERS indices 

achieved by homes certified in 2007 for all certified homes, certified single family homes and 

certified multifamily units, respectively. As shown, the average and median HERS indices are 

the same for all homes, single family homes and multifamily units—average HERS index is 68 

and median HERS index is 69. The average 68 HERS Index reflects 32% energy savings

compared to the 2006 International Energy Conservation Code reference home, and energy 

savings 17 percentage points higher than required for ENERGY STAR certification in 

Massachusetts. The HERS indices achieved by single family homes cover a much larger range: 

HERS indices achieved by multifamily units range from 85 to 53, while HERS indices achieved 

by single family homes range from 85 to 14. Thirty-one, or four percent, of the single family 

certified homes achieved HERS indices of 50 or less, which is considered an indication that a 

home would qualify for the $2,000 federal tax credit. 

The 2008 program encourages builders to build to higher efficiency levels by paying a higher 

incentive for homes that achieve a HERS index of 65 or lower. Over one-third (35%) of the 

housing units certified in 2007 achieved HERS indices of 65 or lower—33% of certified single 

family homes and 39% of certified multifamily units.

 

3 http://www.energystar.gov/index.cfm?c=bldrs_lenders_raters.nh_HERS
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 10

Figure 3.10 2007 HERS Indices—All Homes

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

1,227 ENERGY STAR-Certified Homes

HERS Index

Minimum (best) 14

Median

69

Average

68

2007 HERS Indices Tax Credit 

50

2008

Tier II 

Incentive 

Level

65

Figure 3.11 2007 HERS Indices—Single Family Homes

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

700 Single Family ENERGY STAR-Certified Homes

HERS Index

Minimum (best) 14

Median

69

Average

68

2007 Single Family HERS Indices

Tax Credit 

50

2008

Tier II 

Incentive

Level

65

Figure 3.12 2007 HERS Indices—Multifamily Units

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

527 Multifamily ENERGY STAR-Certified Units

HERS Index

Minimum (best) 53

Median

69

Average

68

2007 Multifamily HERS Indices

2008

Tier II

Incentive

Level

65
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 11

3.5 Spending per Participating Housing Unit

Table 3.1 shows that the number of housing units signed in one year peaked in 2005 at 4,761. In 

2006 and 2007, homes were signed under non-ENERGY STAR as well as ENERGY STAR 

participation paths—ENERGY STAR and EMU paths in 2006 and ENERGY STAR and Code 

Plus paths in 2007. The total number of housing units signed dropped sharply in 2006 and again 

in 2007 reflecting the impacts of no longer recruiting multifamily units in buildings over three 

stories and the slow down in the new construction market. The annual number of housing units 

completed through the program rose steadily through 2006, then plunged in 2007. The drop in 

2007 completions again reflects the impacts of the depressed market for new housing and the 

program no longer being able to certify multifamily units in buildings over three stories. 

Table 3.1 Annual Program Spending, Signings and Completions 

Year Spending

$Thousands

Housing Units 

Signed 

Housing Units 

Completed

2000 $3,160 2,085 565

2001 $3,434 2,715 965

2002 $4,078 2,423 1,435

2003 $4,160 2,063 1,630

2004 $5,193 3,320 1,854

2005 $5,284 4,761 2,358

2006* $5,390 2,580 3,318

2007* $3,610** 1,994 1,316

* 2006 and 2007 include ENERGY STAR, EMU and Code Plus housing units.

**Preliminary Estimate
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 12

Figure 3.13 shows annual spending by electric program sponsors increased each year from 2000 

to 2006, then declined sharply in 2007. Preliminary estimates of 2007 spending provided by 

electric program sponsors show the drop in 2007 spending is predominantly due to a decrease in 

low income spending: low income spending fell by $1.2 million and non-low income spending 

by $600 thousand.

4

Figure 3.13 Annual Program Spending

 

$0.8 $0.6 $0.7

$1.5

$2.7 $3.5 $3.4

$3.7

$0.6

$1.4

$0.7

$1.8

$3.0

$3.9

$2.5

$3.6

$0.0

$1.0

$2.0

$3.0

$4.0

$5.0

$6.0

2000 2001 2002 2003 2004 2005 2006 2007

Millions

Low Income Non Low Income

$3.4

$4.1 $4.2

$3.2

$5.2 $5.3

Total Spending

$5.4

$3.6

 4 The cost data are from annual reports filed with the Division of Energy Resources (DOER) by the electric utilities 

and the Cape Light Compact. The cost data include customer incentives plus in-house and contracted out expenses 

for planning and administration, marketing, and implementation. The cost data do not include evaluation expenses,

market research expenses, performance incentives, other costs or participant costs.
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 13

Figure 3.14 shows the annual spending per signed housing unit and per completed housing unit. 

The dramatic decrease in spending per completed housing unit from the early years of the 

program is largely a reflection of the lag between the time housing units are signed up and the 

time they are certified. 2006 and 2007 include housing units signed and completed under both 

ENERGY STAR and non-ENERGY STAR paths. The decrease in 2007 spending per signed 

housing unit is the result of the number of units signed decreasing by a lower percentage than 

spending decreased—spending dropped 33% and the number of housing units signed dropped 

23%. Conversely, the increase in 2007 spending per completed housing unit is the result of the 

number of completed housing units decreasing by a higher percentage than spending 

decreased—spending dropped 33% and the number of completed housing units dropped 60%. 

Figure 3.14 Annual Spending per Housing Unit 

$2,801

$2,241

$1,515 $1,265

$1,683

$2,016

$1,564

$1,110

$2,552 $2,842

$5,592

$3,558

$1,624

$2,743

$2,089

$1,810

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

2000 2001 2002 2003 2004 2005 2006 2007

Year Housing Unit Signed or Completed

(2006 and 2007 Include ENERGY STAR, EMU and Code Plus Housing Units)

Spending Per Completed Housing Unit

Spending Per Signed Housing 

U it
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 14

4. 2007 Program Activity

The program strives to recruit a mix of project and housing types and to bring new builders into 

the program, as well as maintain the involvement of currently participating builders, to sustain 

the program’s momentum. 

4.1 Projects and Housing Units Recruited

Historically, the program has strived to recruit a mix of market rate and low income housing 

types including single family detached, single family attached and multifamily housing. 

Information on 2007 signings includes a breakdown by market rate versus low income, but does 

not include a breakdown by housing type. Figure 4.1 shows historical data on the percentage of 

2003 through 2006 projects recruited by major housing type: single family detached, single 

family attached and multifamily. Figure 4.2 shows historical data on the percentage of 2003 

through 2006 housing units recruited by major housing category: single family detached, single 

family attached and multifamily. As shown, single family detached projects have consistently 

accounted for a sizable majority of the projects recruited each year, but for less than one-third of 

the housing units recruited because most single family detached projects involve only one or two 

homes. 

Figure 4.1 Historical Signed Projects by Housing Type

(Census Bureau Definitions)

80%

70%

59%

72% 69%

8%

7%

15%

19%

12%

12%

24% 26%

9%

19%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2003 2004 2005 2006 2003-2006

Percent of Projects Signed

Single Family Detached Single Family Attached Multifamily
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 15

Figure 4.2 Historical Signed Housing Units by Housing Type

(Census Bureau Definitions)

32% 29%

16%

25% 24%

19%

7%

21%

35%

20%

49%

64% 64%

41%

57%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2003 2004 2005 2006 2003-2006

Percent of Units Signed

Single Family Detached Single Family Attached Multifamily

Going forward, given that the program can not currently recruit multifamily units in buildings 

over three stories except under specific conditions, the percentage of multifamily units recruited 

each year is likely to be lower than in previous years unless the program finds a way to bring 

high-rise multifamily buildings back into the program. 

Figure 4.3 and Figure 4.4 on the following page show, respectively, the annual percentages of 

market rate and low income projects and housing units recruited from 2003 through 2007. Low 

income projects are defined as any project that includes at least one low income unit. As shown, 

the percentage of projects recruited in 2007 that include low income units is higher than in 

previous years, but low income units recruited in 2007, as a percentage of all recruited units, is 

consistent with previous years.
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 16

Figure 4.3 Percent of 2003 - 2007 Market Rate versus Low Income Projects 

Signed 

94% 91% 89% 96%

81%

90%

6% 9% 11% 19%

10% 4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2003 2004 2005 2006 2007 2003-

2007

Percent of Projects Signed

Market Rate Low Income

Figure 4.4 Percent of 2003 - 2007 Market Rate versus Low Income Housing Units 

Signed 

79% 83% 79%

94%

84% 83%

21% 17% 21%

6%

16% 17%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2003 2004 2005 2006 2007 2003-

2007

Percent of Units Signed

Market Rate Low Income
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 17

4.2 2007 Recruited Projects by Size—Number of Units

Figure 4.5 shows the annual percentages of projects and housing units signed from 2003 through 

2007 falling into various size categories based on the number of housing units in the project. Not 

surprisingly, single-homes account for more than half of all projects signed in each year but 2007 

(48% in 2007), but only a small percentage of the housing units, ranging between four percent of 

housing units in 2005 and 2007 and seven percent in 2003. The majority of housing units 

recruited in every year are in projects with over 25 housing units, ranging between 65% of 

housing units recruited in 2004 and 80% in 2003.

Figure 4.5 2003—2007 Signed Projects by Number of Housing Units per Project

70%

55% 56% 64%

48%

6%

8%

14% 7%

12%

18%

8%

22% 25%

69% 67%

7% 6%

25%

11% 20%

19% 23%

16%

22%

65%

80% 75%

13% 12% 15% 8% 11%

0%

20%

40%

60%

80%

100%

2003 2004 2005 2006 2007 2003 2004 2005 2006 2007

Single Homes Small 2-5 Unit Projects

Mediium 6-25 Unit Projects Large Over 25 Unit Projects

Percent of Projects Percent of Housing Units

Figure 4.6 is another way of showing the very high percentage of projects that have very few 

housing units. Going forward it will be interesting to see if the percentage of one and two unit 

projects decreases as the program moves further toward a market driven model using 

independent HERS raters to recruit projects. HERS raters will likely target multiple unit projects 

because they will be more profitable, requiring less time per housing unit to service than singlehome projects. On the other hand, if the number of one and two unit projects is predominantly 

driven by small builders who want to participate in the program and homeowners who want their 

custom home built to ENERGY STAR standards the percentage of one and two unit projects 

may continue to be high.
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 18

Figure 4.6 2007 Signed Projects by Number of Housing Units per Project

118

135

204

276

0

50

100

150

200

250

300

167 Agreements Received in 2007

Number of Housing Units

One or Two Units 3 to 10 Units 11 to 25 Units 26 to 100 Units >100 Units

60% of Agreements Received in 2007 

Have Only One or Two Housing Units

14% of 

Agreements 

Have 3 to 10 

Units

15% of 

Agreements 

Have 11 to 25 

Units

8% of 

Agreements 

have 26 to 100 

Units

2% of Agreements

(4 projects) Have 

Over 100 Units

4.3 Builder Mix—New and Repeat Participants

Historically, program sponsors have been interested in the number of projects and units coming 

from builders new to the program. 2007 signing data by new versus repeat builder are not 

available. Figure 4.7 shows the percentages of annual project and housing unit signings 

associated with builders new to the program. As shown, the percentage of projects coming from 

builders new to the program steadily decreased from 76% in 2002 to 56% in 2006. This decline 

is reasonable because each year the population of builders who have participated in the program 

grows. Over the 2002 through 2006 period, 62% of all projects signed were with builders who 

had not previously participated in the program. The percentage of signed housing units coming 

from new builders has varied from year to year, ranging from a low of 47% in 2002 to a high of 

68% in 2006. Over the 2002 through 2006 period, 58% of all housing units signed were with 

builders who had not previously participated in the program. The program has clearly been 

successful in bringing new builders into the program. With ICF’s ongoing efforts to attract new 

builders, along with the many independent HERS raters now working in the field to recruit 

builders, it seems reasonable to assume that a large percentage of the builders signing 

agreements each year will continue to be new to the program. 
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 19

Figure 4.7 Historical Percentages of Signed Projects from New Builders

76%

47%

68% 66%

60% 57% 59%

53% 56%

68%

62%

58%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percent of Signed Projects & Units From New Builders

Percent of Projects Percent of Units

2002 2003 2004 2005 2006 2002-2006

New Builders
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 20

4.4 2007 Completions and Signings

Figure 4.8 shows the percentages of 2007 completions by housing category for combined 

ENERGY STAR, EMU and Code Plus completions; only ENERGY STAR completions; and 

only EMU and Code Plus completions. 

Figure 4.8 All 2007 Completions by Housing Type

2007 Total Completions

ENERGY STAR, EMU & Code Plus

(n=1,616)

Single Family 

Attached

Mutifamily 29%

47%

Single Family 

Detached

24%

2007 Completions

ENERGY STAR

(n=1,286)

Single Family 

Attached

32%

Mutifamily

43%

Single Family 

Detached

25%

 

2007 Completions

EMU & Code Plus

(n=330)

Single Family 

Attached

18%

Mutifamily

65%

Single Family 

Detached

17%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 21

Figure 4.9 shows the percentages of 2007 market rate completions by housing category for 

combined ENERGY STAR, EMU and Code Plus completions; only ENERGY STAR 

completions; and only EMU and Code Plus completions. 

Figure 4.9 2007 Market Rate Completions by Housing Type

2007 Market Rate Completions

ENERGY STAR, EMU & Code Plus

(n=1,250)

Single Family 

Attached

30%

Mutifamily

43%

Single Family 

Detached

27%

2007 Market Rate Completions

ENERGY STAR

(n=997)

Single Family 

Attached

33%

Mutifamily

40%

Single Family 

Detached

28%

 

2007 Market Rate Completions

ENERGY STAR, EMU & Code Plus

(n=1,250)

Single Family 

Attached

30%

Mutifamily

43%

Single Family 

Detached

27%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 22

Figure 4.10 shows the percentages of 2007 low income completions by housing category for 

combined ENERGY STAR, EMU and Code Plus completions; only ENERGY STAR 

completions; and only EMU and Code Plus completions. 

Figure 4.10 2007 Low Income Completions by Housing Type

2007 Low Income Completions

ENERGY STAR, EMU & Code Plus

(n=366)

Single Family 

Attached

23%

Mutifamily

63%

Single Family 

Detached

14%

2007 Low Income Completions

ENERGY STAR

(n=289)

Single Family 

Attached

28%

Mutifamily

55%

Single Family 

Detached

17%

 

2007 Low Income Completions

EMU & Code Plus

(n=77)

Single Family 

Attached

6%

Single Family 

Detached

1%

Mutifamily

92%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 23

4.4.1 Distribution of Completions across Sponsor Territories

Electric Sponsor territories

Figure 4.11 shows the percentage of total ENERGY STAR, EMU and Code Plus housing units 

completed in 2007 in each of the electric sponsors’ service areas and in municipal electric 

service areas; the percentage of total ENERGY STAR-certified housing units completed in 2007 

in each of the electric sponsors’ service areas and in municipal electric service areas; and the 

percentage of total completed EMU and Code Plus housing units completed in 2007 in each of 

the electric sponsors’ service areas and in municipal electric service areas. 

Figure 4.11 Electric Sponsor 2007 Completed ENERGY STAR, EMU and Code 

Plus Housing Units

All 2007 Units Completed

ENERGY STAR, EMU & Code Plus

(n=1,616)

CLC 8%

Muni

1%

UNITIL 2%

WMECO

8%

National 

Grid

39%

NSTAR

42%

2007 ENERGY STAR Units Completed

(n=1,286)

CLC 10%

Muni

1%

UNITIL 3%

WMECO, 9%

National 

Grid

41%

NSTAR

36%

 

2007 EMU & Code Plus Units Completed

(n=330)

CLC

0.3%

Muni

0.6%

WMECO

5%

National 

Grid

29%

NSTAR

65%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 24

Gas Sponsor Territories

Figure 4.12 shows the percentage of total ENERGY STAR, EMU and Code Plus housing units 

completed in 2007 in each of the gas sponsors’ service areas; the percentage of total ENERGY 

STAR-certified housing units completed in 2007 in each of the gas sponsors’ service; and the 

percentage of total completed EMU and Code Plus housing units completed in 2007 in each of 

the gas sponsors’ service areas. Table 4.1 and Table 4.2 on the following page show the data 

used to generate Figure 4.11 and Figure 4.12. 

Figure 4.12 Gas Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus 

Housing Units

All 2007 Units Completed

ENERGY STAR, EMU & Code Plus

(n=1,616)

BRKSH 4%

Bay State

21%

NE 2%

No Gas 

Sponsor

16%

KeySpan

39%

NSTAR

18%

2007 ENERGY STAR Units Completed

(n=1,286)

NSTAR

21%

KeySpan

33%

No Gas 

Sponsor

15%

NE

2%

Bay State

25%

BRKSH 5%

 

2007 EMU & Code Plus Units Completed

(n=330)

BRKSH 3%

Bay State 5%

NE 6%

No Gas 

Sponsor 

22%

KeySpan

61%

NSTAR 4%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 25

Table 4.1 Electric Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus 

Housing Units

Electric Sponsors

EMU & Code Plus Completions ENERGY STAR Completions Total 

Completions: 

ENERGY STAR 

EMU Code Plus

Total 

Units

Market 

Rate 

Units

Low 

Income 

Units

Total 

Units

Market 

Rate 

Units

Low 

Income 

Units

NGRID 97 84 13 533 419 114 630

NSTAR 215 153 62 460 327 133 675

Western Mass Electric 

(WMECo) 15 13 2 115 87 28 130

Cape Light Compact (CLC) 1 1 0 127 114 13 128

Unitil 0 0 0 36 35 1 36

Municipals (Muni) 2 2 0 15 15 0 17

Totals: 330 253 77 1,286 997 289 1,616

Table 4.2 Gas Sponsor 2007 Completed ENERGY STAR, EMU and Code Plus 

Housing Units

Gas Sponsors

EMU & Code Plus Completions ENERGY STAR Completions Total 

Completions: 

ENERGY STAR

EMU

Code Plus

Total 

Units

Market 

Rate 

Units

Low 

Income 

Units

Total 

Units

Market 

Rate 

Units

Low 

Income 

Units

Bay State Gas 16 8 8 319 259 60 335

KeySpan 200 196 4 427 294 133 627

Berkshire Gas (BRKSH) 10 10 0 61 49 12 71

NSTAR 14 11 3 271 192 79 285

New England 19 19 0 21 21 40

No Gas Sponsor 71 9 62 187 182 5 258

Totals: 330 253 77 1,286 997 289 1,616
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 26

4.4.2 Distribution of Signings across Sponsor Territories

Electric Sponsor Territories

Figure 4.13 shows the percentage of total ENERGY STAR and Code Plus projects and housing 

units signed in 2007 in each of the electric sponsors’ service areas and in municipal electric 

service areas. Figure 4.14 shows the percentage of total market rate and total low income 

housing units signed in 2007 in each of the electric sponsors’ service areas and in municipal 

electric service areas.

Figure 4.13 Electric Sponsor Signed ENERGY STAR and Code Plus Projects and 

Housing Units

Total 2007 Signed Projects

ENERGY STAR & Code Plus

(n=167)

NSTAR

24%

UNITIL 1%

WMECO

22%

National 

Grid

42%

CLC 8%

Muni 3%

 

Total 2007 Signed Units

ENERGY STAR & Code Plus

(n=1,994)

Muni 3%

CLC 5%

National 

Grid

37%

WMECO 8%

UNITIL

1%

NSTAR

46%

Figure 4.14 Electric Sponsor Signed ENERGY STAR and Code Plus Market Rate 

and Low Income Housing Units

Total 2007 Signed Market Rate Units

ENERGY STAR & Code Plus

(n=1,681)

Muni 3%

CLC 6%

National 

Grid

37%

WMECO

8%

UNITIL 1% NSTAR

46%

 

Total 2007 Signed Low Income Units

ENERGY STAR & Code Plus

(n=313)

WMECO

10%

CLC

1%

National 

Grid

42%

NSTAR

47%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 27

Gas Sponsor Territories

Figure 4.15 shows the number and percentage of total ENERGY STAR and Code Plus projects 

and housing units signed in 2007 in each of the gas sponsors’ service areas and in areas where 

natural gas is not available or gas providers do not sponsor the program. Figure 4.16 shows the 

percentage of total market rate and low income housing units signed in 2007 in each of the gas

sponsors’ service areas and in areas where natural gas is not available or gas providers do not 

sponsor the program. Table 4.3 and Table 4.4 on the following page show the data used to 

generate Figure 4.13 though Figure 4.16.

Figure 4.15 Gas Sponsor Signed ENERGY STAR and Code Plus Projects and 

Housing Units

Total 2007 Signed Projects

ENERGY STAR & Code Plus

 (n=167)

NSTAR

12%

KeySpan

30%

No Gas 

Sponsor 

25%

NE 2%

Bay State

22%

BRKSH 8%

 

Total 2007 Signed Units

ENERGY STAR & Code Plus 

(n=1,994)

NSTAR

15%

KeySpan

42%

No Gas Sponsor 4%

NE 3%

Bay State

33%

BRKSH 4%

Figure 4.16 Gas Sponsor Signed ENERGY STAR and Code Plus Market Rate and 

Low Income Housing Units

Total 2007 Signed Market Rate Units

ENERGY STAR & Code Plus

(n=1,681)

NSTAR

14%

KeySpan 

42%

No Gas Sponsor 5%

NE 3%

Bay 

State

32%

BRKSH 4%

 

Total 2007 Signed Low Income Units

ENERGY STAR & Code Plus

(n=313)

NSTAR

19%

KeySpan 

38%

No Gas Sponsor 4%

Bay State

33%

BRKSH 7%
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 28

Table 4.3 Electric Sponsor Signed Total ENERGY STAR and Code Plus Projects 

and Housing Units

All 2007 ENERGY STAR and Code Plus Signed Projects and Units

Electric Sponsors

And

Municipals

Total 

Projects

Market 

Rate

Projects

Low 

Income 

Projects

Total 

Units

Market 

Rate

Units

Low 

Income 

Units

NGRID 70 54 16 746 615 131

NSTAR Electric 40 31 9 921 774 147

Western Mass Electric (WMECo) 37 31 6 158 127 31

Cape Light Compact (CLC) 13 12 1 108 104 4

Unitil 2 2 0 11 11 0

Municipals (Muni) 5 5 0 50 50 0

Totals: 167 135 32 1,994 1,681 313

Table 4.4 Gas Sponsor Signed Total ENERGY STAR and Code Plus Projects and 

Housing Units

All 2007 ENERGY STAR and Code Plus Signed Projects and Units

Gas Sponsors Total 

Projects

Market 

Rate

Projects

Low 

Income 

Projects

Total 

Units

Market 

Rate

Units

Low 

Income 

Units

Bay State Gas 37 29 8 649 545 104

KeySpan 50 35 15 829 711 118

Berkshire Gas (BRKSH) 14 11 3 86 65 21

NSTAR 20 18 2 291 233 58

New England Gas (NE) 4 4 0 51 51 0

No Gas Sponsor 42 38 4 88 76 12

Totals: 167 135 32 1,994 1,681 313
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 29

5. Penetration of ENERGY STAR Measures

Figure 5.1 shows the percentage of housing units certified in each year from 2002 through 2006

that installed ENERGY STAR lighting measures through the program and the percentage of 

housing units certified in each year from 2002 through 2007 that installed ENERGY STAR 

dishwashers and refrigerators. ICF was not able to provide the number of homes installing 

lighting measures in 2007, but says that virtually all the homes they certified installed at least 

some compact fluorescent bulbs (CFLs) because HERS raters were incentivized to have builders 

install CFLs. The total number of CFL bulbs installed in ENERGY STAR, EMU and Code Plus 

housing units in 2007 is 31,819: this is 14,407 bulbs, or 31%, lower than in 2006. All ENERGY 

STAR lighting fixtures installed through the program in 2007 were installed either in homes 

certified by CSG early in 2007 or in housing units certified by Unitil. The total number of CFL 

fixtures installed in 2007 is 1,422: this is 3,109 fixtures, or 69%, lower than in 2006. 

Percentages of ENERGY STAR housing units installing ENERGY STAR dishwashers and 

refrigerators dropped considerably in 2007. The number of ENERGY STAR housing units 

installing ENERGY STAR clothes washers is also tracked, but the information is not shown in

Figure 5.1 because the percentages are so low, ranging between one and five percent: the 

percentage of ENERGY STAR homes installing ENERGY STAR clothes washers in 2007 is two 

percent. 

Figure 5.1 Percent of Certified Housing Units Installing ENERGY STAR Lighting 

and Appliance Measures 2002 – 2007

59%

60%

59%

57%

58%

47%

70%

68%

49%

50%

72%

62%

58%

34%

80%

76%

65%

43%

47%

6%

70%

45%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fixtures Bulbs Dishwashers Refrigerators

Percent of ENERGY STAR-Certified Homes

2002 2003 2004 2005 2006 2007
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 30

Figure 5.2 shows the percentages of ENERGY STAR-certified housing units that installed 

ENERGY STAR heating systems, windows and central air conditioning. As shown, the 

percentage of homes installing ENERGY STAR heating systems increased steadily from 2003 

through 2006 to 87% and then fell to 72% in 2007. The percentage of homes installing 

ENERGY STAR windows has varied over the last few years. In 2005, almost three-fourths 

(73%) of all certified homes installed ENERGY STAR windows; in 2006 the percentage of 

homes installing ENERGY STAR windows fell sharply to 56% and in 2007 rebounded to a high 

of 83% of certified homes. The percentage of housing units installing ENERGY STAR central 

air conditioning remains low: the 2006 to 2007 decline in the percentage of homes installing 

ENERGY STAR central air conditioning is likely a reflection of the change in the ENERGY 

STAR criteria for central air conditioning from SEER 13 to SEER 14. Also, the penetration of 

ENERGY STAR central air conditioning will, reasonably, always be lower than the penetration 

of ENERGY STAR windows and heating systems because all homes have heating systems and 

windows, but only some homes have central air conditioning. Putting the 7% penetration of 

ENERGY STAR central air conditioning into better perspective, just over half (55%) of the 

housing units certified in 2007 have central air conditioning and 12% of these homes installed 

ENERGY STAR equipment. 

Figure 5.2 Percent of ENERGY STAR-Certified Housing Units Installing ENERGY STAR 

Heating Systems, Windows and Central Air Conditioning

 2002 – 200 7 

66%

69%

53%

77%

73%

87%

56%

15%

72%

83%

37%

77%

52%

3%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Heating Systems Windows Central A/C

Percent of ENERGY STAR-Certified Homes

2002 2003 2004 2005 2006 2007
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 31

ICF was able to provide data on the number of 2007 EMU and Code Plus housing units that 

installed ENERGY STAR heating systems, windows and central air conditioning. Figure 5.3

shows that virtually the same percentages of ENERGY STAR and EMU/Code Plus homes 

installed ENERGY STAR heating systems, but that EMU/Code Plus housing units were much 

less likely than ENERGY STAR housing units to have ENERGY STAR windows or central air 

conditioning. 

Figure 5.3 Comparison of 2007 ENERGY STAR versus Code Plus Housing Units 

Installing ENERGY STAR Heating Systems, Windows and Air Conditioning 

7%

83%

72% 73%

1%

52%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ENERGY STAR

Heating Systems

ENERGY STAR

Windows

ENERGY STAR

Central A/C

Percent of Housing Units

ENERGY STAR Housing Units Code Plus Housing Units

6. Duct Leakage

Revised EPA standards for ENERGY STAR certification require ducts be sealed and tested to 

have leakage at or below 6 cfm25 per 100 square feet of conditioned floor area. However, duct 

leakage testing is not required if all ducts and air handling equipment are in conditioned space 

and envelope leakage is 3 ACH50 or less. The majority of homes participating in the 

Massachusetts program have ducts. 

In 2006, the program certified 938 housing units that had ducts and met the new duct leakage 

requirements: the average duct leakage in these homes was 2.3 cfm25 per 100 square feet of 

conditioned floor area. 

In 2007, almost all certified housing units (96%) had ducts. The average duct leakage in 2007 

for ENERGY STAR-certified housing units with ducts in unconditioned space was 4.3 cfm25 

per 100 square feet of conditioned floor area: 5.1 cfm25 per 100 square feet of conditioned floor 

area in multifamily units and 3.9 cfm25 per 100 square feet of conditioned floor area in single 
2007 Massachusetts New Homes with ENERGY STAR® Program Progress Report Page 32

family homes. Of the certified housing units with ducts, 42% had all ducts installed in 

conditioned space and met the envelope leakage requirement to waive duct testing: 51% of 

multifamily units with ducts and 36% of single family homes with ducts. The program 

encourages builders to install ducts in conditioned space and, based on findings from the 2005 

Baseline study, it has been successful. In the 2005 Baseline Study, only 11% of the single family 

homes with ducts had all ducts installed in conditioned space. 
MassSAVE 

 Final Summary 

QA/QC and Impact Study Report 

 

 

April 8, 2008 

 

Prepared for: 

Bay State Gas 

Berkshire Gas 

Cape Light Compact 

KeySpan Energy Delivery 

National Grid 

NSTAR Electric and Gas Corporation 

Unitil/Fitchburg Gas & Electric 

New England Gas Company 

Northeast Utilities/Western Massachusetts Electric 

Prepared by: 

RLW Analytics 

179 Main Street 

Middletown, CT 06457 

(860) 346-5001 
TABLE OF CONTENTS

1 INTRODUCTION .......................................................................... 1

1.1 Objectives ....................................................................................... 3

2 METHODOLOGY .......................................................................... 3

2.1 Summary of QA/QC and Measure Level Impact Methodology ............... 4

2.2 Summary of Billing Analysis Methodology........................................... 6

3 RESULTS .................................................................................. 8

3.1 Objective 1: Perform QA/QC visits on installed measures. ................... 8

3.2 Objective 2: Provide feedback and impacts for targeted measures. .... 10

3.3 Objective 3: Provide a check of gas program impacts at a high level. . 13

4 CONCLUSIONS AND RECOMMENDATIONS .......................................... 14

TABLE OF TABLES

Table 1: Summary of Vendor Provided Data (PY2006) .............................................. 2 

Table 2: Summary of Vendor Data by Fuel with per Home Savings (PY2006) ...... 3 

Table 3: Final On-Site Sample Design ........................................................................... 5 

Table 4: Measure Ratings and Status Summary .......................................................... 9 

Table 5: Comparison of 2004 and 2007 Satisfaction Levels .................................... 10 

Table 6: Summary of Savings (therms) for Fossil Fuel by Measure ....................... 11 

Table 7: Summary of Savings (kWh) for Electric Measures in the Program ......... 11 

Table 8: Final Results (Customer Level) ..................................................................... 13 

Table 9: Final Results (Program Level) ....................................................................... 13 

Table 10: Summary of Savings Impacts ..................................................................... 14 

Table 11: Recommendations for Savings Estimate Improvements ........................ 15 

TABLE OF FIGURES

Figure 1: Overview of 2007 MassSAVE Evaluation Work Approach ......................... 4 

Figure 2: Sampled Lighting Logger Profiles for the Average Weekday and 

Weekend .......................................................................................................................... 12 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 1 

________________________________________________________________________ 

1 Introduction 

This report is intended to provide a summary of results as gathered from a series 

of evaluation activities performed for the MassSAVE Program in 2007 on 

participants from the 2006 program year. These activities comprised of the 

performance of on-site QA/QC work, measure level impact work and a natural 

gas billing analysis. 

The MassSAVE program provides toll free phone energy saving tips and 

screening, a comprehensive home energy assessment and follow-up services to 

all Massachusetts residential electric and gas customers. The State Legislature 

of Massachusetts mandates the MassSAVE program for all regulated electric and 

gas distribution companies, and the MA Division of Energy Resources (“MA 

DOER”) regulates the program. MassSAVE is the new name for the 

Massachusetts Residential Conservation Services Program, which has been 

available since 1980. The overarching goal of the program is to deliver non-lowincome residential customers with services that are intended to simplify customer 

participation and provide a “one-stop shopping” home energy efficiency and 

renewable energy service. The Program refers low-income customers to the 

appropriate provider of low-income programs. There were five primary program 

vendors that provided program services on behalf of the sponsoring entities in 

2006. 

The MassSAVE Program has been designed to provide several benefits to 

participants including reduced energy costs, improved home performance, a 

healthier home environment, and improved home comfort all year long. Services 

provided include the following: 

 Educational materials/services to help consumers save energy, 

 In-home services available at no cost for customers who plan to invest in 

energy saving improvements, 

 Step-by-step guidance to lead customers through the installation of 

energy-saving measures and receiving incentives, 

 Incentives toward the installation of energy-saving measures, 

 Quality installations performed by fully licensed and insured contractors 

who warrant all workmanship and materials for one year, and 

 Quality inspections available to ensure that the job was done right. 

To illustrate the diverse offering of measures and the multi fuel nature of the 

MassSAVE Program, Table 1 presents the 2006 program year participant 

application and annual savings data as gathered from program vendors by fuel 

type in kBtus. It should be noted that these values are slightly different than 

DOER activity for 2006; which may be the result of the timing of the DOER data 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 2 

________________________________________________________________________ 

extract from the vendors or changes in the crediting of savings among programs 

after the data is sent along to DOER. HDMC also have savings in this table with 

completion dates in their data that appeared to fall outside of 2006, although 

they may have been credited to the 2006 program for fiscal reasons. The gas 

and oil savings are generally associated with insulation and air sealing measures 

while electric savings are generally associated with lighting and refrigeration. 

The specific measures provided by the vendors are dependent on their 

contracting utility. As such, not all vendors provide all measures. 

Annual Non-Electric Fuel Savings (in kBtu) 

Measure/Vendor CET CSG HDMC Rise Unitil Total 

Air Sealing/Weatherization 91,633 3,282,198 25,326,356 857,535 67,699 29,625,421 

Attic Insulation 209,786 9,496,745 19,890,865 2,127,130 74,497 31,799,023 

Basement Insulation 19,990 852,943 719,194 184,914 6,110 1,783,151 

DHW 0 77,189 4,730,428 41,752 61,222 4,910,591 

Duct Insulation 56,640 23,108 1,645,777 4,946 31,108 1,761,578 

Duct Sealing 0 24,493 0 21,940 0 46,433 

Heating System 0 5,812,480 0 56,240 0 5,868,720 

Indirect DHW 0 2,664,000 0 64,154 0 2,728,154 

Lighting 0 0 0 0 0 0 

Pipe Insulation 267,044 0 1,495,694 4,639 0 1,767,377 

Refrigerator 0 0 0 0 0 0 

Rim Joist Insulation 0 279,737 0 0 69,159 348,896 

Thermostat 156,668 1,827,432 3,121,640 65,987 116,254 5,287,981 

Wall Insulation 563,245 12,895,430 18,272,786 503,378 39,579 32,274,418 

Windows 0 649,440 0 527,647 0 1,177,087 

Non-Electric Total 1,365,006 37,885,195 75,202,740 4,460,260 465,629 119,378,830 

No. of Applications 48 1,790 5,293 366 20 7,517 

Annual Electric Savings (in kBtu) 

Air Sealing/Weatherization 0 88,645 14,335 118,312 5,130 226,421 

Attic Insulation 0 250,783 0 234,389 15,170 500,341 

Basement Insulation 0 42,533 0 31,329 0 73,862 

DHW 0 28,034 31,182 37,435 1,553 98,205 

Duct Insulation 0 0 0 0 0 0 

Duct Sealing 0 16,552 0 0 0 16,552 

Heating System 0 0 0 0 0 0 

Indirect DHW 0 40,000 0 0 0 40,000 

Lighting 0 26,732,130 5,717,303 1,787,736 46,048 34,283,217 

Pipe Insulation 0 0 0 1,744 0 1,744 

Refrigerator 0 2,311,293 893,523 708,103 0 3,912,919 

Rim Joist Insulation 0 3,060 0 0 0 3,060 

Thermostat 0 393,637 0 70,526 9,350 473,513 

Wall Insulation 0 24,389 0 6,435 0 30,824 

Windows 0 113,421 0 70,014 0 183,435 

Electric Total 0 30,044,477 6,656,343 3,066,023 77,251 39,844,094 

Total Savings 1,365,006 67,929,672 81,859,083 7,526,283 542,880 159,222,924 

No. of Applications 0 6,741 2,713 1,291 58 10,804 

Table 1: Summary of Vendor Provided Data (PY2006) 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 3 

________________________________________________________________________ 

The following table provides a breakdown of vendor provided program data by 

fuel type, in therms1; which is the unit of measurement used to report impacts 

later in this report. Other fuel savings is not provided in this table but is an 

additional 3,128,692 kBTU; or 31,287 therms in 106 homes. It is interesting to 

note the increased average savings per participating home for oil (180 therms 

per home) as opposed to gas (142 therms per home). 

 Oil Gas Electric 

Tracking Savings (kBTU) 49,077,055 67,173,083 39,844,094 

# Homes Treated 2,727 4,739 10,804 

Avg. Savings per Home 179.9 therms 141.7 therms 1,081 kWh 

Table 2: Summary of Vendor Data by Fuel with per Home Savings (PY2006) 

1.1 Objectives 

The overall evaluation work performed in 2007 was designed to accomplish three 

primary objectives. Each of these is provided below. 

Objective 1: Perform QA/QC visits and provide feedback on energy efficiency 

measures as installed by vendors. (Appendix A) 

Objective 2: Provide specific feedback and impacts for targeted oil and gas 

measures provided through the program as well as all electric measures. 

(Appendix B) 

Objective 3: Provide a reality check of gas program impacts at a high level. This 

was done through a billing analysis of natural gas customers. (Appendix C) 

The first two study activities have been reported out previous to this summary 

report (the QAQC results in October, 2007 and the measure level impacts in 

November, 2007). This report is intended to present and synthesize the results 

of all three activities in a single document along with pertinent final conclusions 

and recommendations.

2 Methodology 

Figure 1 presents a brief overview of the 2007 MassSAVE evaluation activities. 

As discussed above, three activities were performed in this study designed to 

achieve the three objectives. These activities are summarized below and include 

the performance of QA/QC work, measure level impact work and a natural gas 

billing analysis. A detailed discussion of the various methodologies employed can 

be found in the appendices to this report. 

 

1

 To convert the values from KBTU to therms, we divided KBTU by 100. A therm is equal to 100,000 BTUs. 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 4 

________________________________________________________________________ Work performed April through December, 2007

 Overview of 2007 MassSAVE Evaluation Work

Natural Gas Billing Analysis QA/QC and Measure Impact 

Work

Data acquisition of 2007 

measure level activity by 

participant from each program 

vendor

Sampling of 90 homes to 

represent vendors and largest 

measures contributing to 

energy savings

Acquisition of gas billing 

data for participants and 

non-participants from 

natural gas utility

Visits to 90 homes with 

appropriate blower door 

testing, infrared imaging and 

audit data collection

Performance of timeseries comparison/cross 

sectional design billing 

analysis

Performance of DOE-2 

simulation modeling and other 

engineering analyses to 

estimate measure level oil, 

gas and electric energy 

savings

Performance of timeseries comparison/cross 

sectional design billing 

analysis

Documentation and delivery of 

QA/QC and measure level 

results to study sponsors.

Documentation and 

delivery of billing analysis 

results to study sponsors.

Synthesis of all evaluation efforts into a single summary 

report. 

Figure 1: Overview of 2007 MassSAVE Evaluation Work Approach 

2.1 Summary of QA/QC and Measure Level Impact Methodology 

To achieve the QA/QC measure level impact objectives, a total of 90 homes were 

selected for visits. These homes were selected from vendor data based upon the 

presence of measures installed that had been designated as a significant energy 

savings measure2 from 2006 DOER data. Generally, we sought to perform 

 

2

 Air Sealing, Attic Insulation, Wall Insulation, Thermostats, and Heating Systems. Refrigerators were also 

later targeted for purposes of determining electric savings. Lighting measures were not targeted due to 

their ubiquity but were analyzed wherever encountered. 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 5 

________________________________________________________________________ 

allocation of the 90 available site visits proportional the distribution of the 

program savings among the vendors. However, we had a minimum threshold of 

6 homes randomly selected for each vendor to ensure a reasonable assessment 

of quality of measure installation practices. The final sample design is provided 

in Table 3 below where the ‘pri’ columns show the primary measure for which a 

home was selected for visit and the total columns show the total number of 

measures observed. 

Major Measure/Vendor 

CET CSG HDMC RISE AES/Unitil Total 

Pri. Total Pri. Total Pri. Total Pri. Total Pri. Total Pri. Total

Attic Insulation 2 3 5 16 7 12 4 9 2 2 20 42 

Air Sealing/Weatherization 2 4 5 15 7 16 4 8 2 3 20 46 

Heating System 0 0 5 9 0 0 0 0 0 0 5 9 

Thermostats 3 4 5 6 7 7 1 1 2 2 18 20 

Wall Insulation 3 4 5 9 7 9 2 3 0 0 17 25 

Supplemental Refrigerators 0 0 0 0 6 7 4 6 0 0 10 13 

Total 10 15 25 55 34 51 15 27 6 7 90 155

Table 3: Final On-Site Sample Design 

At each home, all measures installed were observed for QA/QC purposes but only 

primary gas or oil measures and all electric measures were targeted for the 

measure level impact analysis. At each site, data was collected and diagnostic 

testing was performed as needed to inform simulation or re-engineering analysis 

in the interest of providing the energy impacts of the targeted installed measures 

and to provide feedback on the quality of measure installation. This included 

blower door tests, lighting loggers, infra-red scans of wall insulation applications 

(when possible) and full home audits. Once all data was collected, savings work 

was performed for each electric measure in each home and the major gas and oil 

measures along with QA/QC assessments for all measures encountered. As part 

of the on-site, the auditor also asked the homeowner questions regarding their 

perception and experience with the program. 

Measure level impacts for the targeted oil and gas measures and majority of 

electric measures were calculated using DOE-2 models. DOE-2 is a detailed 

hourly building energy analysis tool that we used to predict the energy use of 

each measure in each home before and after the measure was applied. RLW 

used six typical home models in estimating the savings based upon the condition 

observed on-site (including blower door results) and baseline (pre-retrofit) 

conditions. These six models represented the various home types encountered 

in the field with respect to encountered heating and cooling system 

configurations. The models were calibrated with actual gas and electric billing 

data. More detailed information on the savings approach for all measures and 

fuels is contained in Appendix B of this document. 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 6 

________________________________________________________________________ 

2.2 Summary of Billing Analysis Methodology 

The billing analysis uses a participant group and the control group and employs a 

“time-series comparison/cross sectional experimental design”. The time 

series/cross sectional design helps to reduce concerns about self-selection bias 

and free-ridership and helps the evaluation achieve internal and external validity. 

Internal validity means the evaluation is conducted in a manner that allows the 

results to isolate the impact of the activity being studied. When other factors are 

not recognized, the changes attributed to the program may be the result of other 

phenomena. For example, if the experiment does not recognize the dynamic 

nature of a participant’s operational or end-use characteristics, their change in 

usage could be explained by changes in other participant characteristics. 

 

In addition, the research design can help achieve external validity by ensuring 

that the results are representative of a larger population of interest, allowing for 

the findings to be generalized. For example, for the MassSAVE analysis, the 

information determined by a sample of participants, and the corresponding 

control group, permits the evaluation to represent the total program impacts. 

The MassSAVE natural gas energy impacts were determined through a 

multivariate regression (MVR) analysis. A more detailed discussion of the 

methodology is provided in Appendix C. The MVR uses the temperature 

normalized annual consumption (NAC) for the participants and representative 

control group and tracking system data. The regression protocol used was a 

comprehensive and systematic approach that has been applied with great 

success to the analysis of market based programs. The regression protocol 

consists of six steps that result in the selection of an optimal model that 

accurately quantifies the program impact. The six steps of the regression 

protocol follow. 

Step 1: The Simple Model: During this step an initial regression model was 

developed using ordinary least squares ("OLS"). This simple model determined 

the effect of one important variable (i.e., the participation indicator variable 

status, or the participant’s engineering estimate of savings) on savings while 

controlling for all other variables. 

Step 2: Regression Diagnostics As a result of the residual standard deviation 

being related to the size of the customer's gas usage or demand, one regression 

assumption most often violated is that the standard deviation of the error terms, 

(or "residuals") is not constant across the range of predicted values. When the 

standard deviation residuals are related to the predicted values, the model is said 

to be "heteroscedastic." Heteroscedasticity can often be detected in crosssectional models used to analyze DSM program impact. During this step, 

verification that the regression assumptions are valid was performed and any 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 7 

________________________________________________________________________ 

added multivariate regression analyses to address this issue were performed 

under a weighted least squares ("WLS") approach. 

Step 3: Weighted Least Squares: As discussed above, one of the fundamental 

regression assumptions is that the standard deviation of the error terms (or 

residuals) has a constant variance across the range of predicted values. When the 

residuals are related to the predicted values, the model is said to be 

heteroscedastic. When heteroscedasticity is present, an ordinary least squares 

(OLS) approach to establishing the relationship between the dependent and 

independent variables may be inappropriate. Weighted least squares (WLS) is one 

approach to correct for heteroscedasticity in regression analysis. 

According to econometric theory, WLS provides the most reliable estimate of 

savings and an accurate measure of the resulting reliability. The theory of WLS 

depends on a correct specification of the heteroscedasticity. The theory assumes 

that a positive-valued variable can be specified; say z, such that the residual 

standard deviation is proportional to z. Usually, z is taken to be some measure of 

size (for example, the pre-retrofit NAC consumption). These results of the 

MassSAVE models suggested that the error terms may be heteroscedastic. 

Accordingly, weighted least squares (“WLS”) regression was used to develop the 

final models. 

Step 4: The Unabridged Model: During this step an initial regression analysis 

(using OLS, or if more appropriate, a WLS approach) is performed. A 

multivariate regression full analysis model, the unabridged model, is developed. 

This model consists of any available variable that may be significant in the 

determination of the program impacts. After the development of the unabridged 

model, a residual analysis is performed. This analysis is used to diagnose, 

analyze, and correct if necessary, any outliers. Under WLS, this step is used to 

determine the best gamma for use in creating the optimal weights. 

Step 5: The Refined Model: The fifth step develops the refined model, based on 

the unabridged model, and if using WLS, the optimal value of gamma. A step 

wise regression approach is used to eliminate any insignificant variables of the 

unabridged model. After this step, the refined model will feature only those 

variables that have mathematical significance in the determination of the energy 

or demand savings. 

Step 6: Calculation of Energy Savings: The final step in the analysis estimates 

the energy savings by using the resultant models. In this step the savings are 

calculated using both the unabridged and the refined models to examine the 

impact on savings of removing the statistically insignificant terms. 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________ Page 8 

________________________________________________________________________ 

3 Results 

The following sections present results for each aforementioned study objective. 

3.1 Objective 1: Perform QA/QC visits on installed measures. 

As discussed earlier, a total of 90 quality control visits were performed at homes 

of 2006 participants in the MassSAVE Program. These inspections were 

conducted during the period of June through August, 2007. The table below 

summarizes the QA/QC visits. It shows the number of measures assessed, the 

status of the measure as observed and determined by RLW auditors, the typical 

installations found onsite, and a summary of issues encountered. 

In general terms, we observed the majority of measures to be operating as 

intended and well installed. There were two instances where major measures 

were reported in the tracking system but were not found onsite during the audit. 

In one of these instances, wall insulation was reported to have been installed, 

but the customer informed the auditor that attic insulation had been installed 

instead of wall insulation. In another instance the tracking system reported that 

basement insulation was installed at a site, but upon inspection of the basement, 

none was found. 

Programmable thermostats were found still installed in all cases with the 

exception of one home in which the homeowner had removed it. However, 

there were several instances in which the thermostat was observed to be in 

manual mode and several more instances where the customer had set their 

programmable thermostat to a schedule that mimicked the schedule they had 

used with their old manual thermostat. 

Other measures such as lighting and air sealing are reported below as not 

installed when the homeowner didn’t remember the measure being installed and 

the auditor could not find evidence of the work. It should be noted that the 

customers often had trouble remembering what bulbs or air sealing measures 

had been installed through the program. Evidence of air sealing can be difficult 

to find if it is installed in an attic, or sealing small cracks in the basement. In 

addition, the installed lighting measures could have been removed. As a result 

of these possibilities, the vendor may have installed these measures even though 

they were not found to be installed during the QA/QC audit. 
Massachusetts Final Summary QA/QC and Impact Study Report ___________________________________ Page 9 

 

 ______________________________________________________________________________________ 

Program 

Measure 

Qty 

Assessed 

Onsite 

Rating/Status 

Good Fair Poor 

Not 

Installed Removed Assessment and Issues Encountered 

Air Sealing/ 

WXN 46 29 7 3 6 1 

Majority of installations done very well. Multiple 

instances where customer was unable to recall 

weatherization measure installation. Multiple 

instances where the home showed potential for 

additional tightening (9 homes). Two instances 

where the weather-stripping fell off. 

Attic 

Insulation 40 36 4 0 0 0 

Nearly all installations of high quality. Isolated 

instances where application was not uniform. 

Two instances where the stairs and/or doors 

leading to the treated attic area were not 

insulated. 

Basement/ 

Floor 

Insulation 

10 7 2 0 1 0 

Vast majority of installations done well. In one 

home the insulation was not installed. In 

another home a small section had fallen out of 

place, and in another home some compression 

was observed. 

Domestic 

Hot Water 20 19 0 0 1 3 

Majority of installations installed and operating 

as intended. Three instances where customer 

removed the measure (2 aerators and one low 

flow showerhead) and one instance where 

customer was unable to recall the installation. 

Duct 

Insulation 4 4 0 0 0 0 All installed. 

Heating 

System 9 9 0 0 0 0 All verified to be ENERGY STAR compliant and fully operational. 

Lighting 47 45 0 0 8 9 

Many still installed and operating. At eight 

homes the customers were unable to recall the 

installation of seventeen bulbs and they were 

not found by the auditor. At nine homes 

customers removed a total of thirteen bulbs. 

Refrigerator 13 13 0 0 0 0 All verified to be ENERGY STAR compliant and 

fully operational. 

Thermostat 20 19 0 0 0 1 

Vast majority well installed. One instance 

where customer removed the measure and 

three homes in which the thermostat was on a 

manual setting. 

Wall 

Insulation 25 18 5 1 1 0 

Generally installed well based upon evidence 

available. A few instances where gaps in 

treatment were found. Two instances where 

drill holes not well covered or patched and one 

instance where exterior tiles had been broken 

and not repaired. The one instance where 

insulation was not installed appeared to be an 

error in the tracking data where attic insulation 

was installed in place of wall insulation. 

Windows 3 3 0 0 0 0 All well installed. 

Total 237 202 18 4 17 14 - 

Table 4: Measure Ratings and Status Summary
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 10 

________________________________________________________________________ 

Table 5 presents the satisfaction ratings of the various measure types installed 

by homeowners visited in the QA/QC sample. The rating scale ranges from 1 to 

6 with 1 representing “Very Dissatisfied” and 6 representing “Very Satisfied”. 

Also provided are the satisfaction results from the 2004 QA/QC work, in which 

the same satisfaction question and rating scale was asked of 2003 program 

participants. There is only one measure type that experienced a substantial 

reduction in satisfaction rating from the 2004 work, which was basement 

insulation. This was due primarily to a respondent that was upset that another 

inaccessible portion of the basement could not be treated. One other measure 

had a slight reduction in average satisfaction ratings while the remainder of the 

measures had improvements in ratings; including improvements of ten percent 

or more for three measures (attic insulation, air sealing, and lighting). In 2004, 

it was suggested that satisfaction ratings be trended over time as a metric that is 

regularly tracked as a measurement of program success. 

 Satisfaction Rating 

1 (very dissatisfied) to 6 (very satisfied) 

Measure n 2004 Average n 2007 Average 

Refrigerator 14 5.9 13 5.8 

Attic Insulation 26 5.0 40 5.6 

Air Sealing 26 4.8 35 5.4 

Basement Insulation 13 5.2 9 4.2 

Wall Insulation 22 4.8 25 5.1 

Lighting 37 4.8 47 5.5 

Domestic Hot Water 16 4.9 23 5.1 

Heating Systems - - 9 5.6 

Thermostats 6 5.5 20 5.6 

Table 5: Comparison of 2004 and 2007 Satisfaction Levels 

3.2 Objective 2: Provide feedback and impacts for targeted measures. 

As part of the 90 on-sites performed and discussed above, RLW estimated 

savings for targeted oil and gas measures and all electric measures. Table 6 

provides a summary of all program fossil fuel savings by primary measure, with 

all other program measures not analyzed included as a bundle. Appendix B 

provides additional breakdowns of results, including oil versus gas and by 

vendor. The last row above the total shows the RLW calculated interactive 

heating penalty in therms for all lighting and refrigerator applications also 

analyzed as part of the on-sites. Based upon those measures analyzed in this 

study, we estimate the total program fossil fuel impact of the program to be 

905,781 therms at +/- 18% at 90% confidence. This would reflect a 76% 

realization rate for the programs gas and oil savings. Combining these overall 

results with the total number of fossil fuel homes in the population provides a 

per home savings estimate of nearly 160 therms according the vendor savings 

estimates and 120 therms according to the RLW savings work. Detail on the 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 11 

________________________________________________________________________ 

reasons behind the various measure level results can be found in Appendix B of 

this report. 

Measure Vendor RLW 

Realization 

Rate 

Relative 

Precision 

Air Sealing/Weatherization 296,254 106,277 36% +/- 30% 

Attic Insulation 317,990 234,852 74% +/- 25% 

Wall Insulation 322,744 383,109 119% +/- 24% 

Heating System 58,687 122,531 209% +/- 24% 

Thermostats 52,880 22,996 43% +/- 33% 

All Other Fossil Fuel Measures 145,233 145,233* 100%* N/A 

Interactive Penalty from lighting and refrigerators 0 -109,217 N/A N/A 

Totals 1,193,788 905,781 76% +/- 18% 

 *Savings assumed to have a 100% realization rate as they were not analyzed as part of this study. 

Table 6: Summary of Savings (therms) for Fossil Fuel by Measure 

Total program electric energy (kWh) for all measures, including those installed in 

fossil fuel heated homes, are shown in Table 7, below. Most vendors did not 

count electric savings for measures primarily installed to produce savings in fossil 

fuel heated homes, which pushed the realization rate for ‘All Other Measures’ 

above 100% as RLW did include electric savings due to central air conditioning 

systems found in many of the homes. These results produce a per home electric 

savings estimate of nearly 1,080 kWh according the vendor savings estimates 

and 502 kWh according to the RLW savings work. 

Summary of Electric 

Impacts (kWh) Vendor RLW 

Realization 

Rate 

Relative 

Precision 

Lighting 10,047,836 3,814,420 38% +/- 33% 

Refrigerators 1,146,811 1,071,685 93% +/- 13% 

All Other Measures 482,989 540,117 112% +/- 29% 

Totals 11,677,636 5,426,222 47% +/- 20% 

 Table 7: Summary of Savings (kWh) for Electric Measures in the Program 

Lighting represents the majority of the electric savings in the program, and thus 

we calculated demand and estimated coincident demand reductions. The total 

connected demand reduction for lighting in the sample is 15.8 kW, based on 324 

CFLs observed and monitored by RLW at 49 sites. The installation of bulbs 

averaged 6.6 CFLs per home based on RLW data, where the vendor data 

indicated 7.2 CFLs per home. 

The figure below shows the weighted (by connected lighting Watts reduced) 

averages of monitored on-times for all 291 loggers that were utilized for this 

project. It is important to recognize that this project was conducted during the 

Summer and early Fall, so that there were no Winter measurements of lighting 

usage. Also, it is important to note that these load shapes do not represent 

typical lighting load shapes for a typical home with all lights considered. Instead, 

they are true load profiles for the sample of lights observed and metered on-site. 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 12 

________________________________________________________________________ 

Average Lighting Usage Profiles 

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Hour Ending

Lighting Fraction On

Weekdays

Weekends

Figure 2: Sampled Lighting Logger Profiles for the Average Weekday and 

Weekend

Coincident peak summer demand reduction for the sample, based on the 

average of the four hours from 1 to 5 PM during a typical day in July was found 

by RLW to be 0.949 kW. This is based upon a percent on-time during this 

window of 6.0%. Total program impacts may be inferred by multiplying these 

estimates by the ratio of total program population (homes, or applications) to the 

sample population. This ratio is 327.4 (16,042/49 from Appendix B), so the 

unadjusted program-wide peak demand reductions are about 311 kW for the 

summer ISO window. Note that this procedure for calculating program-wide 

savings fails to adjust the estimates for any sampling bias that may be present in 

the RLW sample. Without vendor estimates of these same demand reductions, it 

is impossible to estimate the sampling bias. 

Since the logger data from this study were gathered during the summer months, 

adjustments must be made to estimate winter peak reductions. From a study 

performed for NECPUC in 20073

, the weighted coincident peak summer 

coincident factor was estimated to be 8.2% and the winter was estimated to be 

29.8%. Using these factors, we can calculate a summer to winter factor ratio of 

3.63. Applying this to our calculated summer coincident factor of 6.0% provides 

us with an estimated winter coincident factor of 21.8% and a calculated 

Coincident peak winter demand reduction of 3.4 kW for the sample (21.8% 

*15.8 kW). 

 

3 “Coincidence Factor Study Residential and Commercial & Industrial Lighting Measures”, 

Prepared for the New England State Program Working Group (SPWG), RLW Analytics Inc. Spring 

2007.
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 13 

________________________________________________________________________ 

3.3 Objective 3: Provide a check of gas program impacts at a high 

level. 

The check of gas program impacts was performed through a multivariate 

regression (MVR) analysis of natural gas participant and non-participant billing 

data. This table shows the customer level results by strata. As this table shows, 

the stratification allows the results to be weighted to accurately represent the 

population. 

 

The average net gas savings from the billing analysis is estimated to be 62 

Therms, or 5.3% of the participant’s pre-program NAC. The confidence interval 

around this estimate is from 53.9 Therms to 70.0 Therms. This is a ±12.0% 

relative confidence interval. An actual savings of 62 Therms is 54% of the 

tracking system savings estimate (i.e., the realization rate). It should be noted 

that this customer level result as well as the program level result below, is a net 

savings estimate. 

Interestingly, the lowest strata (participants with tracking less than 175 Therms) 

had a realization rate of 84%. The savings estimates increase with the stratum4

. 

However, the realization rate decreases. This indicates that the accuracy of the 

tracking estimates of savings decrease as the estimates increase. The largest 

strata tended to have more attic and wall insulation measures installed than the 

lower strata 

Pop.

Strata N Low Limit High Limit Pop.

Analysis 

Data Set Parts

Control 

Group

Estimated 

Savings (ccf)

Realization 

Rate

1 4,145 - 175 60 65 1,137 1,080 50 84%

2 472 175 640 350 351 1,219 1,158 138 39%

3 120 640 10,000 1,102 921 1,742 1,812 165 15%

Total 115 4,737 115 1,161 1,107 62 54%

Stratum Limits (Therm 

Savings) Average Savings (Therm) PreNAC (Therm)

Table 8: Final Results (Customer Level)

Table 9 presents the program level results from the billing analysis. The net 

program gas savings is estimated to be 293,416 Therms. The relative 

confidence interval around this estimate is 255,374 Thermsto 331,457 Therms; 

which calculates to a ±13% interval. 

Strata N

Total Savings 

(Therm)

Conifidence 

Interval (+/- therm) RCI

1 4,145 208,594 21,238 ±10.2%

2 472 65,013 8,177 ±12.6%

3 120 19,809 8,626 ±43.5%

Total 293,416 4,737 38,042 ±13.0%

Table 9: Final Results (Program Level) 

 

4

 Note that the high limit for stratum 3 is a stratum limit and not an actual savings value from the data. 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 14 

________________________________________________________________________ 

4 Conclusions and Recommendations 

Based upon the performance of the on-sites, QA/QC results, measure level 

performance analysis and billing analysis of gas savings we believe the 

MassSAVE Program can be characterized as providing a high quality value added 

service to customers that is producing substantial levels of electric, gas and oil 

energy savings. The following table provides the savings estimates as calculated 

from the measure level on-site work and the billing analysis and compares them 

to the overall program and per home estimates calculated from the vendor data. 

Vendor reporting to the DOER for all of the sponsors should be revised to reflect 

the findings in this study. Since all sponsors do not use vendor data for 

institutional reporting, they need to exercise care when systematically 

incorporating results from these studies into savings calculations used for 

institutional reporting. 

The measure level on-site work estimated gross savings for all fossil fuels and 

provides a program level savings estimate of 905,781 therms and a per home 

estimate of 119.6 therms with a precision of ±18%. The measure level on-site 

work also provides an estimated gross savings for electricity of 5,426,222 kWh at 

the program level and a per home estimate of 502 kWh with a precision of 

±20%. The billing analysis measured net savings for natural gas only and 

provides a program level savings estimate of 293,416 therms and a per home 

estimate of 62 therms with a precision of ±13%. 

Estimate Source Program Level Per Home Precision 

All Fossil Fuels (therms) 

Vendor 1,193,788 157.7 N/A 

RLW On-Site Measure 905,781 119.6 ±18% 

Natural Gas (therms) 

Vendor 545,296 115 N/A 

Billing Analysis 293,416 62 ±13% 

Electricity (kWh) 

Vendor 11,677,636 1,080 N/A 

RLW On-Site Measure 5,426,222 502 ±20% 

Table 10: Summary of Savings Impacts 

During the process of determining program impacts from the on-site work, there 

were opportunities noted in which vendor savings estimates could be adjusted or 

refined to reflect more accurate savings in their tracking data. These are 

presented in detail on a case by case basis in Appendix B and further 

summarized below. We do believe that the sponsors should consider the idea of 

utilizing a single tool consistently among all vendors to calculate savings. Such a 

tool should include all fuel savings and would ensure a regular mechanism is 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 15 

________________________________________________________________________ 

used across vendors. To the extent that savings are going to continue to be 

aggregated by DOER and the program cost effectiveness depends on program 

wide savings assumptions, using a single savings calculation methodology across 

vendors makes a great deal of sense. 

Should the development of a single tool among vendors be deemed too resource 

intensive and logistically prohibitive given the structure and contracts in place 

among the many sponsors and vendors, an alternative is to coalesce 

assumptions to the extent possible for use among all vendors and savings work. 

We provide vendor specific recommendations below as well as measure level 

unitized savings work in Appendix B report to facilitate changes in savings work 

at whatever level of adjustment the program sponsors are able to prioritize 

pending the determination of available resources and time. 

Measure Vendor Suggestion 

Air Sealing 

HDMC Reconsider use of assumed savings per home in lieu of 

calculating savings uniquely for each home treated*. 

CET Consider reviewing the calculation of savings as the impact per 

square foot appears to be low compared to other vendors. 

Attic 

Insulation 

RISE Consider reviewing the calculation of oil savings, as the per 

application (home) savings estimates appear to be low. 

All 

Consider the performance of a targeted review of how vendors 

are estimating attic insulation savings, especially when existing 

insulation is in place*. 

Heating 

system CSG Reconsider use of assumed savings in lieu of customized estimates of savings. 

Thermostats All 

Consider implementing a selection process where 

programmable thermostats are installed in homes where they 

are desired and consider excluding applications where setback 

is already being practiced with the existing thermostat*. 

Lighting All 

To the extent that sockets in areas of higher use are available 

at the time of CFL installation, those areas should be prioritized 

for treatment. Also consider tracking the interactive effects for 

lighting*. 

* Quality control activities that that could be expected to reduce some of the specific savings estimation problems. 

Table 11: Recommendations for Savings Estimate Improvements

Based upon both the on-site measure level impact work and the QA/QC work, we 

provide additional conclusions below, with suggestions made as appropriate. 

 Conclusion 1: Early in the evaluation process, RLW gathered 2006 

program data from both DOER and the individual vendors. We anticipated 

the savings activity from both sources to align closely with one another. 

However, the vendor provided activity was noted to be 2.9% lower than 

the DOER activity with varying vendor level differences; including two that 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 16 

________________________________________________________________________ 

were more than 10% off. While there are several reasons why this may 

occur, such as changes in the crediting of savings among programs after 

the data is sent along to DOER, it is desirable that DOER and vendor 

records of program activity be aligned and consistent. 

o Suggestion 1a: If this is the result of program savings being credited 

differently after data is submitted to DOER, DOER might consider 

providing more time for vendors to settle their savings allocation 

process before providing their records of program activity. 

o Suggestion 1b: The sponsors might consider the development of a 

single database that is accessible to all vendors and other interested 

parties that tracks all program activity in a consistent manner. Such a 

system would allow DOER and other parties to query the data for 

activity and would remove any error between the DOER gathered 

savings and program activity and the actual vendor data in the system. 

 

 Conclusion 2: An integral part of our on-site activities was in verifying 

vendor provided measure level data. In working with the vendor data for 

the MassSAVE Program, several instances occurred in which tracking data 

appeared to be incorrect due to input errors, mis-assigned fuel types or 

mis-categorized measure type.

o Suggestion 2: To the extent that savings measure type 

categorization and fuel savings is important to properly characterizing 

savings at the vendor level and at the aggregate level (DOER), we 

recommend a program improvement goal of establishing checks in the 

vendor tracking systems to ensure the proper association of savings to 

measure types and fuels. 

 Conclusion 3: In the analysis of measure savings from the on-sites, it 

was noted that several vendors do not track electric savings for measures 

installed primarily to generate savings in fossil fueled homes. 

o Suggestion 3: Electric savings should be calculated and tracked for 

those major measures that are anticipated to provide electric cooling 

kWh savings in addition to the primary fossil fuel savings. 

 Conclusion 4: In general, the installations observed on-site were 

determined to be good, with occasional and isolated observations of major 

measure installations that either could not be verified or could be 

improved and some instances of tracking system errors. These findings 

suggest that major measure installations are being installed within 

reasonable quality tolerance bounds, providing a solid foundation for 

continuing measure installation improvements. 

o Suggestion 4a: At a high level, we recommend that an ongoing QA 

Process be initiated as an integral part of the MassSAVE Program. The 

purpose of this activity would not be to “police” the program 

installations, but rather to ensure that the current quality of installation 

is sustained, that ongoing improvement of major measure installations 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 17 

________________________________________________________________________ 

are sought and to make certain that some of the smaller measures 

such as lighting are getting installed and are generating energy 

savings for the program. This process should be incorporated in a way 

that seeks continual process improvements through its outcomes. A 

specific structure for such a QA process is provided in Appendix A of 

this report. 

o Suggestion 4b: While a high level QA process would likely benefit the 

savings being generated in the program, some specific measure level 

QA work available for consideration can also be expected to result in 

higher energy savings. Specifically, the sponsors might consider 

implementing a selection process where programmable thermostats 

are installed in homes where they are desired and have the capability 

to generate the most savings through setbacks that are not already 

behaviorally performed with the current thermostat. In addition, when 

thermostats are installed, the contractor should make sure the 

homeowner knows how to use the thermostat properly. Further, to 

the extent that sockets in areas of higher use are available at the time 

of CFL installation those areas should be prioritized for treatment.

 Conclusion 5: Participants are reporting satisfaction with many elements 

of the program and its delivery. 

o Suggestion 5: As the program continues to evolve and the QA/QC 

process matures, continue to track measure satisfaction and consider it 

an important metric that should be monitored regularly. 

 Conclusion 6: Some vendors track installations by job as opposed to 

number of units installed. 

o Suggestion 6: To facilitate any subsequent QA/QC work, ex-post 

savings work or measure level aggregation work we recommend the 

tracking of specific quantities of units installed among these measure 

types (lighting, weatherization, DHW) and locations installed. 

 Conclusion 7: Several homes visited had single doors treated with 

weather-stripping, small quantities of CFLs installed and/or single 

installations of DHW measures such as faucet aerators or low flow 

showerheads. It was noted at these homes that other opportunities for 

these measures existed but were not treated.

o Suggestion 7: Consider or encourage vendors to install additional 

sets of weatherization measures (weather-stripping), CFLs and/or 

DHW measures to take advantage of all opportunities at a home.

 Conclusion 8: Nearly all attic insulation was well installed at the sites 

visited and few instances of wall insulation gaps were evidenced. 

However, in a few isolated instances, attic insulation was observed to 

have uneven depth and at a couple of homes wall insulation was noted 
Massachusetts Final Summary QA/QC and Impact Study Report _________________ Page 18 

________________________________________________________________________ 

either to have not been done by the homeowner or to have some gaps in 

coverage. 

o Suggestion 8a: We believe the isolated instances of gaps in wall 

insulation and uneven attic insulation do not warrant a substantial 

recommendation. A brief reminder for vendors and contractors may 

be appropriate at this time regarding the need for uniform depth of 

attic insulation to achieve proper material density (and R-Value) in 

treated homes.

o Suggestion 8b: The timing of on-sites in the summer made 

interpretation of infra-red imaging for the wall insulation difficult due 

to low temperature differentials between the indoor and outdoor 

temperatures at many homes. We suggest that during the colder 

months a targeted series of homes with wall insulation treatment 

undergo infra-red imaging in the interest of providing more conclusive 

evidence of their proper installation. 
22 Haskell Street, Cambridge, MA 02140

Phone: (617) 497-7544 Fax: (617) 497-7543

www.nexusmarketresearch.com

Market Progress and

Evaluation Report (MPER)

For the 2007 Massachusetts

ENERGY STAR® Lighting Program

Final

July 1, 2008

Volume 1

Findings and Analysis

Submitted to:

Cape Light Compact

Massachusetts Electric Company

Nantucket Electric Company

NSTAR Electric

Western Massachusetts Electric Company

Unitil

Submitted by:

Nexus Market Research, Inc.

RLW Analytics, Inc.

Dorothy Conant
Massachusetts ENERGY STAR Lighting MPER 2007

Nexus Market Research

Contents

Executive Summary........................................................................................................................ 2

Findings....................................................................................................................................... 2

Program achievements............................................................................................................ 2

The current state of the market ............................................................................................... 3

Torchieres ............................................................................................................................... 7

Involvement of manufacturing and retail partners.................................................................. 7

NCP Program Procedures....................................................................................................... 8

Technical Considerations........................................................................................................ 9

Conclusions............................................................................................................................... 11

Recommendations..................................................................................................................... 12

Appendices (In Separate Volume)

Appendix A Product Quality Assessment and Technological Developments

Appendix B: Consumer Survey

Appendix B-1: Consumer Survey Report 

Appendix B-2: Phone Survey Questionnaire 

Appendix C: Retailer Survey

Appendix C-1: Retailer Survey Memo

Appendix C-2: Retailer Survey Data Collection Tools

Appendix D: Process Documentation

Appendix D-1: Process Documentation Memo

Appendix D-2: Process Discussion Guides

Appendix E: Database Analysis

Appendix F: Lighting Price Analysis Memo 

Appendix G: 2006 Net-to-Gross Memo 

Appendix H: Socket Count

Appendix H-1: Socket Count Memo

Appendix H-2: Socket Count Data Collection Tool
Massachusetts ENERGY STAR Lighting MPER 2007 Page 2

Nexus Market Research

Executive Summary

This report summarizes the results and implications of evaluation activities completed as part of 

the Market Progress and Evaluation Report (MPER) for the 2007 Massachusetts ENERGY 

STAR Lighting Program by Nexus Market Research, Inc. (NMR), RLW Analytics, Inc. 

(RLW), and Dorothy Conant. It reviews key findings, provides an analysis of the degree to 

which the compact fluorescent lamp (CFL), fixture, and torchiere markets have been 

transformed, and offers several recommendations for consideration by the Sponsors: Cape Light 

Compact, Unitil, National Grid, NSTAR, and Western Massachusetts Electric. A full report of 

findings and analysis for the 2007 MPER is provided to the Sponsors under separate cover.

Findings 

A wide variety of research activities were conducted in support of the program evaluation. The 

results can be categorized as documenting the achievements and costs of the program, describing 

the current state of markets for energy-efficient lighting products, outlining the involvement of 

manufacturing and retail partners in the program, and identifying technical considerations. Key 

points in each of these categories are listed below.

Program achievements

• Since 2002, the Sponsors’ Lighting Program has included three basic components: 

− The ENERGY STAR Lights catalog (and website)

− Instant rebate coupons

− Negotiated Cooperative Promotions (NCP)

In 2007, 89% of the products distributed through the Massachusetts ENERGY STAR 

Lighting program came from the NCP component.

• Program-related shipments of CFLs have increased dramatically since the Sponsors first 

began offering a joint efficient lighting program—from 158,000 in 1998 to 2.6 million in 

2007.1

• The multiple program components—the catalog, retail coupons, and the Negotiated 

Cooperative Promotions—appear to meet somewhat different needs of the market and 

provide complementary opportunities for consumers and for retailers. 

 

• Program-related shipments of ENERGY STAR-qualified fixtures and torchieres have 

decreased, after peaking in 2004.

− Relevant shipments of ENERGY STAR-qualified fixtures grew from 46,000 in 1998 

to 119,000 in 2004, falling to 39,000 in 2007.

− Relevant shipments of ENERGY STAR-qualified torchieres increased from 774 in 

1998 to 38,000 in 2004, falling to 3,300 in 2007.

• The program appears to be highly cost effective. 

− Despite the fact that total program spending decreased to $7.3 million in 2007 after 

peaking at $12.8 million in 2004, program expense per product remains stable and 

low at $2.82. It is worth noting that these figures are based only on products directly 

 1 As documented through supporting sales data for coupon, catalog, and NCP markdown programs and through 

documented shipping confirmation for the NCP buydown program.
Massachusetts ENERGY STAR Lighting MPER 2007 Page 3

Nexus Market Research

incentivized through the program, and there is strong evidence that the program has 

yielded significant spillover savings. The majority of the budget is directed to 

customer incentives, including NCP buydown and markdown costs. Collectively, the 

percentage of Sponsors’ budgets allocated per the regulatory reporting categories to 

planning and administration, marketing, and technical support dropped from 61% in 

2003 to 39% in 2007, with a commensurate percentage increase going to customer 

incentives during the same time period. 

− Lifetime savings resulting from the 2007 program year are estimated to be 1,665,000 

MWh, as adjusted for hours of use, in-service rates, free ridership, and spillover. The 

cost per average MWh saved is estimated to be about $5. CFLs, which account for 

98% of program sales in 2007 account for about 95% of these savings.

The current state of the market

Here, we summarize findings with regard to customer awareness, product quality, environmental 

considerations, customer satisfaction, and pricing, as well as trial and usage and sales volumes.

• Customer awareness for CFLs is nearly universal, but awareness of energy-saving 

fixtures has remained stable since 2005. 

− Ninety-four percent (94%)2

• Product quality appears to be relatively high. 

of respondents were familiar with CFLs in 2007—

compared with 77% in 2005, and 28% were familiar with energy-saving fixtures in 

2007—compared with 29% in 2005. 

− Almost half of retailers report no product returns through the program; participating 

retailers with returns estimate an average rate of 2.5%, mostly because products were 

broken or defective at the time of purchase or because the customer did not like the 

look of the product after it was installed. EFI3

− While the product quality of standard CFLs has greatly improved, program 

administrators and manufacturers agree that the quality of some types of specialty 

CFLs, such as three-way, dimmable, and recessed CFLs could be improved. Two 

factors to note in this regard include: 

reports that in the 2007 catalog 

program, the return rate is about 3 to 4%. These return rates include products 

returned because they are damaged (as through shipping), defective, or because 

customers have changed their minds about their choices.

 All reflector bulbs were required to undergo re-qualification for the 

ENERGY STAR program in 2005. This requirement appears to have 

weeded out many of the poorly performing products. 

 The dimming capability of CFLs continues to be problematic at the low 

end of the dimming range, when color shifts become apparent and shut-off 

 2 Except where otherwise noted, the results presented for 2005 through 2007 are weighted by number of households 

in Massachusetts or the number of products. The demographic characteristics of telephone survey respondents do 

not always mirror exactly those of the general population. The weighting system helps correct the differences so that 

product counts better reflect the purchasing habits of the Massachusetts population. Results from 2002 through 2004 

are not weighted, which somewhat limits comparability of results.

3 The Energy Federation Inc. (EFI) is a contractor hired by the Sponsors to provide program fulfillment services for 

the catalog, process rebate coupons for retailer reimburse, process invoices for the NCPs for manufacturer 

reimbursement, and maintain program databases. 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 4

Nexus Market Research

points are not consistent or as low as incandescents. Some manufacturers 

we interviewed are addressing these issues. 

− Results from a separate measure life study conducted by NMR for the Sponsors and 

other residential lighting programs in New England suggest that CFL product quality 

improved over the study period from 2002 through 2005, but dropped in 2006. More 

specifically, except for 2006, each successive generation of CFLs studied survived 

longer than the last. The drop in 2006 could be a random event or may represent a 

shift in quality, but small sample sizes and a limited number of failures in recent years 

curtailed our ability to conduct meaningful statistical analyses to more fully explore 

survival rates over time.

4

− Independent testing of products has been supported by the Sponsors through the 

Program for the Evaluation and Assessment of Residential Lighting (PEARL) 

Board.5 Our analysis in the 2005 MPER and 2006 MPER found that only a small 

proportion of CFLs sold in Massachusetts were delisted as ENERGY STAR-qualified 

products as a result of PEARL testing cycles.6

− There is concern among some PEARL Board members that historical issues with the 

ENERGY STAR program’s timeliness and/or transparency of the de-listing process 

might continue when it takes over the testing process. Both the PEARL Board and 

the Consortium for Energy Efficiency (CEE) have recommended that the Department 

of Energy (DOE) allow comprehensive audits of the test results in order to ensure that 

DOE follows the appropriate protocols regarding product nomination and 

procurement, testing, data reporting, marginal failures, disqualification appeals, and 

delisting. 

 The PEARL Board has decided to 

conduct another round (Cycle 9) of testing in 2008, which will most likely be the final 

product testing cycle for the program. The ENERGY STAR program will assume 

oversight of a third-party CFL testing and verification program in the new CFL 

program requirements, Version 4.0, which becomes effective in December 2008. 

• Environmental considerations, based on the presence of mercury in CFLs, have become 

more important to CFL stakeholders, as evidenced by more media attention, consumer 

concerns, regulatory mandates, product offerings from manufacturers, manufacturer 

marketing strategies, disposal options, and ENERGY STAR specifications. 

− In Massachusetts, the Mercury Management Act (signed July 28, 2006) phases out 

the use of mercury in products (not including CFLs) and regulates the disposal of 

products containing mercury (including CFLs). If recycling efforts do not meet 

targeted levels, the law requires lamp manufacturers to provide up to $1 million per 

 4 NMR, Inc. and RLW Analytics, Inc. Residential Lighting Measure Life Study, Final June 4, 2008. Submitted to 

New England Residential Lighting Program Sponsors.

5 The PEARL program is chaired and coordinated by the Natural Resources Defense Council (NRDC), with a 

testing lab at the Lighting Research Center (LRC) at Rensselaer Polytechnic Institute in Troy, NY. PEARL was 

established to replicate the tests that manufacturers are required to perform before submitting products for ENERGY 

STAR qualification in order to verify compliance with ENERGY STAR specifications. The sponsors include the 

Massachusetts energy efficiency program administrators through the Northeast Energy Efficiency Partnerships, 

other regional program administrators, as well as the EPA and DOE. 

6 Sponsors monitor testing and screen incentivized products throughout the year to ensure that they are ENERGY 

STAR-qualified. Sponsors also included a provision in the 2006 RFP to partners that “The Sponsors reserve the 

right to exclude specific ENERGY STAR qualified products from this promotion for any reason including results of 

independent, third-party testing.” 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 5

Nexus Market Research

year to the Massachusetts Department of Environmental Protection (MA DEP) for 

grants to municipalities and/or regional authorities that are collecting and recycling 

mercury-containing lamps.7

− The Version 4.0 specification for ENERGY STAR-qualified CFLs, effective 

December 2008, sets mercury dosage and labeling requirements for qualified 

products. For lamps under 25 watts the maximum mercury dosage is 5 mg per lamp, 

and for 25 to 40 watt lamps the maximum dosage is 6 mg per lamp. Manufacturers 

must also label product packaging with the ‘Hg’ symbol within a circle, include text 

that the ‘Lamp Contains Mercury,’ and list the website ‘www.epa.gov/bulbrecycling’ 

or ‘www.lamprecycle.org’ on the packaging.8

− The ENERGY STAR program has also produced a consumer information fact sheet 

on mercury in CFLs and a disposal protocol for spent bulbs.

 

9

− Many CFL manufacturers are introducing products with dosing as low as 2 to 3 mg of 

mercury and best practice offerings from manufacturers currently include products in 

the 1 to 2 mg range. These low-mercury products are not universally available across 

all CFL product lines and they are more expensive to produce. There are concerns 

about product quality with decreases in mercury levels. 

− Manufacturers are also responding to international interest in mercury. The European 

Union’s Restriction on Hazardous Waste Substances Directive (RoHS) adopted a 

mercury limit of 5 mg per bulb.10

− An industry-wide solution to financing CFL recycling programs remains unresolved. 

A few energy-efficiency programs or states, such as Wisconsin Focus on Energy and 

Efficiency Maine, have established coordinated regional recycling programs, and 

some manufacturers offer their partners the option of including plastic bags in product 

packaging for containment of used CFLs or offer collection bins or buckets that can 

be placed in retail locations for spent CFL collection and recycling.

 RoHS also includes restrictions on lead used in 

glass and solder. Japan, South Korea, and China also are developing similar mercury 

restrictions.

− The Sponsors see their role in dealing with mercury and recycling as primarily 

educational; they have been approached by recycling vendors but are hesitant to take 

responsibility for recycling efforts. Most believe the manufacturers and retailers (and 

the government) should be responsible for coming up with recycling programs and 

providing disposal sites. 

− Significantly more consumer respondents in 2007 compared to previous years are 

aware of some type of hazard associated with CFLs and disposal requirements; 

however, the vast majority of all current and previous users—77%—are unable to 

identify any hazards associated with CFLs. The few respondents who were aware of 

 7 Fact Sheet, Summary: Massachusetts Mercury Management Act. Massachusetts Department of Environmental 

Protection. http://www.mass.gov/dep/toxics/laws/hglawfax.pdf (Accessed 4-18-07). 8 www.lamprecycle.org is sponsored by the National Electrical Manufacturers Association (NEMA), an organization 

representing many lighting manufacturers. The website serves as a clearinghouse for information on recycling 

fluorescent lamps.

9 http://www.energystar.gov/ia/partners/promotions/change_light/downloads/Fact_Sheet_Mercury.pdf (Accessed 4-

29-08). 10 Directive 2002/95/EC of the European Parliament and of the Council, January 27, 2003 as reported in the Official 

Journal of the European Union, 13.2.2003.

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2003:037:0019:0023:EN:PDF (Accessed 4-29-08).
Massachusetts ENERGY STAR Lighting MPER 2007 Page 6

Nexus Market Research

hazards note that the products require special disposal (8%) or that the bulbs contain 

mercury (8%). Forty-three percent of those who have used CFLs have also had to 

dispose of at least one bulb. Seventy-two percent of these respondents simply threw 

the products away in the trash and 26% recycled the CFL or dropped it off at a 

recycling center. We calculate that about six million CFLs have been removed from 

service in Massachusetts since 1998, which could mean that 72% of those, or more 

than four million, have been thrown in the trash. 

• Customer satisfaction with energy-efficient lighting products is high. A large majority 

of consumers is satisfied with their energy-saving products. Seventy-six percent of those 

who currently use CFLs rate their satisfaction as a 4 or a 5 on a five-point scale, as do 

78% of those who currently use energy-saving fixtures. These results are similar to 

satisfaction levels in previous years.

• Trial and usage of CFLs has increased compared to 2005, but trial and usage of energysaving fixtures has remained stable. 

− Seventy-nine percent of all households in Massachusetts have tried CFLs and 76% 

currently use CFLs. In 2005, 64% of all households in Massachusetts had tried CFLs 

and 56% of households were using CFLs at the time of the survey.

− In 2007, 14% of all Massachusetts households have tried energy-saving fixtures and 

12% currently have at least one installed. In 2005, 13% of all Massachusetts 

households had tried energy-saving fixtures and 11% had at least one energy-saving 

fixture currently installed. 

− About one-quarter of households in Massachusetts are still not using CFLs. Most of 

those households (16% of all households) are aware of the technology, but have not 

tried CFLs; 6% of households are not aware of CFLs; and 3% are previous users. A 

lack of familiarity (73% of all households) is the primary reason why respondents 

have not tried energy-saving fixtures

− As of early 2008, about 21.4% of all bulbs in use in Massachusetts homes were 

CFLs—amounting to about 22.6 million CFLs. As of early 2006, about 11.3% of all 

bulbs in use in Massachusetts homes were CFLs—amounting to about 12 million 

CFLs.

− There are about 5.2 million additional CFLs in storage in Massachusetts homes, or 

about 2.7 bulbs per household (past and current CFL users). In early 2006, there were 

about 3.8 million CFLs in storage locations. The average number of stored bulbs per 

household is similar to that found in 2005 and 2006, but the number of households 

storing bulbs has increased. The fact that CFLs are held in storage suggests that 

many people plan to install CFLs to replace incandescents or other CFLs as they burn 

out, and signals that CFLs are achieving the status of standard options to meet 

lighting needs. The volume of CFLs in storage may be the result of the promotion of 

multi-packs through the NCP program and also suggests the potential value of an 

effort to educate consumers about replacing incandescents with CFLs right away 

rather than waiting for the incandescents to burn out.

• Sales volumes continue to increase, but at a lower rate than in the past few years. 

− Market-level sales of medium screw-based CFLs in Massachusetts have increased 

from 430,000 units in 2000 to about 13.3 million in 2007, and now account for 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 7

Nexus Market Research

about 19% of sales of medium-screw based bulbs in the Commonwealth.11 This 

estimate is validated by an analysis we conducted comparing CFL sales and usage 

patterns from 1998 through 2007 using various sources and assumptions, 

including estimates from the Measure Life Study the sponsors are participating 

in.12

− The increase in market-level CFL sales from 2005 to 2006 was about 4.1 million 

units, or about 65%, while the increase from 2006 to 2007 was only about 2.1 

million units, or about 21%. 

− Meanwhile, U.S. sales of CFLs increased from 82 million units in 2004 to 89 

million in 2005—up by 9%, to 163 million in 2006—up by 81%, and in 2007 they 

increased to 349 million units—or about 114% over 2006. More CFLs per 

household are still sold in Massachusetts than in the U.S. as a whole, indicating 

earlier market adoption, but the rest of the U.S. appears to be catching up. This 

issue will be explored more fully in the Net-to-Gross memo, after more sales data 

from individual states become available, likely in the fall of 2008.

Torchieres

The following are some key findings related to torchieres:

• Program-supported sales of torchieres increased from 774 in 1998 to 38,499 in 2004, but 

has dropped dramatically to 3,320 in 2007. This drop appears to be related to reduced 

Sponsor efforts in response to perceived changes in market demand.

• Energy-efficient torchiere installations account for only 0.1% of the estimated 1,665,000 

MWh in lifetime savings resulting from the 2007 program year (adjusted for hours of use, 

in-service rates, free ridership, and spillover). 

• Sponsor incentive amounts for the instant coupons were $15 to $20 in 2007, the same as 

in 2006. Incentive amounts for torchieres were $17 under buydown agreements and $20 

for markdowns, an increase of $5 per unit compared to 2006. 

• Seven percent (7%) of consumers surveyed report purchasing a torchiere-style lamp in 

2007. Of those purchasing torchieres, 29% say the torchieres they purchased use CFLs. 

The responses from 2007 show a decrease in the use of halogen bulbs and an increase in 

the use of CFLs compared to previous years, but these shifts are not statistically 

significant. 

Involvement of manufacturing and retail partners

• The residential lighting market is changing fast, with rapidly increasing sales of CFLs all 

over the US. Even so, the Massachusetts program, along with other energy-efficient 

lighting programs offered elsewhere, continue to be important to manufacturers and 

retailers and help to stimulate their production and marketing of energy-efficient lighting 

products. 

− Fifty-eight percent (58%) of participating retailers say they would continue to offer 

CFLs and energy-saving fixtures if the program were not in place. 

 11 Not all CFLs sold in the state are currently being used in residential homes; some CFLs are used in non-residential 

applications and some are in storage. Identifying the portion of annual sales that are not currently being used in 

residential applications is beyond the scope of this study.

12 NMR, Inc. and RLW Analytics, Inc. Residential Lighting Measure Life Study, Final June 4, 2008. Submitted to 

New England Residential Lighting Program Sponsors.
Massachusetts ENERGY STAR Lighting MPER 2007 Page 8

Nexus Market Research

• The program continues to attract participating manufacturers and retailers and thus 

increases the availability of energy-efficient lighting products in Massachusetts. The 

number of retail locations selling ENERGY STAR-qualified lighting products through 

the program has increased from 248 in 2002 to 833 in 2007, fewer than the 954 stores in 

2006. Furthermore, the Massachusetts program supports sales of about 734 CFL 

models,13

• The range of store types participating in the Massachusetts program includes bargain, 

hardware, home improvement, department, price club, grocery, drug/convenience, 

specialty/electrical, and home furnishing stores. In 2007, home improvement stores and 

hardware stores were the sources of the greatest number of NCP sales, while two years 

ago, in 2005, bargain stores dominated sales. Program sales at grocery stores more than

doubled in 2007 compared to 2006 and slightly exceeding sales in 2005. Sales at drug 

and convenience stores dropped dramatically in 2007 compared to the previous two 

years. 

including about 126 models in the NCP program. Seventy-four percent (74%) 

of the products sold through the NCP program were standard CFLs in the 13 to 23 watt 

range. 

NCP Program Procedures

• The NCP program, as an industry-sponsored initiative,14

• The structure of the 2007 program follows that of the 2006 program. To encourage 

industry partners to provide the program with “documented, credible, store-level sales 

reports,”

is a dynamic program. Sponsor 

budget allocations and manufacturer/retailer product mix and counts are established at the 

beginning of the program year, but allocations and spending shift according to feedback 

from manufacturers and retailers and Sponsor budgets. 

15

• While there were more buydown agreements compared to markdown agreements

Sponsors offered higher incentives for products sold through the markdown 

model (which bases all of the incentive payments on sales data) compared to those sold 

through buydowns (which pays the majority of the incentive to the industry partner upon 

receipt of confirmed shipment reports and the remaining amount based on confirmed 

sales data). Both markdowns and buydowns provide reduced product pricing for the 

consumer. 

16

 13 This figure is a simple sum of unique models from each of the program delivery modes and may include duplicate 

model types across programs.

14 The Sponsors invite lighting manufacturers to partner with retailers for promotions of qualified lighting products. 

The manufacturer-retailer partner teams are encouraged to submit proposals for a variety of promotion types, 

including product buydowns and markdowns that reduce the price consumers pay for the products, coupons, and 

advertising. 

15 Request for Proposals. Northeast ENERGY STAR Lighting Initiative, Negotiated Cooperative Promotion, 2007 

ENERGY STAR Lighting Campaign, April 1, 2007 – December 31, 2007. January 16, 2007. 16 The Sponsors negotiate NCP agreements, also known as memorandum of understanding agreements (MOUs) with 

these industry partners that detail the promotion terms.

in 

2007, the invoiced value of all 2007 markdown agreements was higher, at about $2.7 

million—more than twice the value of buydown agreements ($1.2 million). Additionally, 

some of the manufacturers/retailers that traditionally have participated in the program 

through buydown agreements shifted to the markdown model in 2007. 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 9

Nexus Market Research

• The NCP program continues to rely on the industry expertise of its contractors, EFI and 

Lockheed Martin to work with manufacturers and retailers. These contractors ensure that 

product shipments and sales occur and that supporting documentation is maintained.

Technical Considerations

The lighting market is dynamic with respect to technological progress. It is thus important to 

track this as well as sales of CFLs. This section addresses two new technologies: the GU24 

standard for fixtures, and solid-state lighting (SSL), represented largely by light-emitting diodes 

(LEDs), the most commonly mentioned emerging SSL technology. The adoption of the Federal 

Energy Independence and Security Act of 2007 (EISA 2007), which sets efficiency standards for 

medium, screw-base lamps beginning in 2012, also may impact the development of more 

efficient lighting technologies, including improvements to incandescents, halogens, CFLs, and 

SSL.

GU-24 Standard for Fixtures

• GU-24 fixture technology establishes a new line voltage socket and pin base for 

replaceable ballasts that is standardized for interchangeability across manufacturers. The 

idea of the standard is to simplify matching of fixtures, ballasts, and pin-based CFLs for 

manufacturers, retailers, and consumers. GU-24 technology can be retrofitted to existing 

incandescent fixtures. It also allows consumers greater choice of features (light output, 

light color, dimmability) from a single fixture depending on the GU-24 ballast/lamp 

combination used, and theoretically extends fixture life because the ballasts are 

replaceable. GU-24 lamps are available in one-and two-piece lamp and ballast 

combinations, but aside from the flexibility that two-piece units offer for replacing just 

one part of the lamp/ballast configuration, the integrated lamp/ballast combination may 

be more consumer-friendly.

• The portion of a finalized Version 4.2 of the ENERGY STAR specification for residential 

light fixtures that addresses GU-24 fixtures will take effect on August 1, 2008. The 

revised version of the specification focuses on the performance requirements for products 

and fixtures that use GU-24 bases. Specifications for fixtures that use other lamp bases 

(not GU-24 or LED) are not affected by the revised Version 4.2 specification. All fixture 

models using GU-24 bases that had previously been qualified under the Version 4.0 

specification must be re-qualified under Version 4.2.

• Fixture manufacturers continue to indicate strong support for GU24 technology; many 

are transitioning to the GU24 standard and phasing out CFL fixtures with other lamp 

bases. 

• This development may have a downside, however. The GU24 standard introduces a new 

set of pin-based fixtures and CFLs to the market, overlaying the somewhat complicated 

array of pin-based fixtures and CFLs that currently exists. Insofar as the GU24 standard 

is successful, it may undercut the sales potential of ENERGY STAR-qualified fixtures 

that are currently on the market and make it more difficult for consumers to find 

replacement bulbs for the fixtures they already have.

• Considering both the benefits and the issues involved, program administrators from 

Massachusetts are mixed on the benefits of relying on GU24 technology, fearing that it 

will not inherently promote the use of ENERGY STAR-qualified products. Moreover, 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 10

Nexus Market Research

there will need to be more pin-based bulbs available for this technological development 

to succeed. 

Solid-State Lighting (SSL)

• Effective immediately, the EPA issued a technical amendment to the Residential Lighting 

Fixture (RLF) criteria, Version 4.2 on June 2, 2008. This amendment incorporates test 

procedures and metrics to allow LED-based fixture performance to be evaluated against 

the requirements of the existing RLF specification.

• Solid-state lighting (SSL), in the form of white light-emitting diodes (LEDs), has a 

number of attributes that make it an attractive option for the future, including the fact that 

they are mercury-free and contain no other known toxins. The new ENERGY STAR 

specification for solid-state luminaires (SSL), Version 1.0 has been finalized and the 

criteria will become effective September 30, 2008. The criteria use a two-tiered approach 

to qualifying products: 

o Category A—Focus on near-term applications, mostly niche lighting (task, 

outdoor, etc.)

o Category B—Likely 3-5 years out, will replace Category A with efficacy targets 

for a wider range of products, possibly including general lighting applications.

• The two-tiered specification approach recognizes that the technology is rapidly changing 

and that there are some applications of the technology that are currently commercially 

viable as energy-efficient lighting alternatives. Flexibility with the development of the 

Category B applications allows for the technology to continue its rapid improvement. 

• LEDs currently are not a near-term or even medium-term replacement for CFLs, although 

there are some promising reflector lamp applications that might be competitive in the 

medium-term. Experts disagree as to whether screw-in LEDs will ever be developed for 

widespread use. Recently EFI started to offer a small selection of recessed or downlight 

LED fixtures that are available through the Sponsors’ Internet catalog. 

• There is a consensus among manufacturers and DOE that not rushing LEDs to market 

will allow the industry and DOE to learn about the technology, including its benefits and 

limitations, and will help to avert inaccurate product claims or inferior products that do 

not meet consumer expectations. 

Energy Independence and Security Act of 2007

• EISA 2007 is a federal act, signed into law on December 19, 2007, that sets maximum 

wattage levels for medium, screw-base lamps by lumen output. The standards are to

become effective under a phased approach beginning in 2012 when general service bulbs 

will be required to use about 20-30% less energy than current incandescent bulbs. 

Products that meet EISA 2007 standards may include more efficient incandescents and 

halogens, which will not necessarily offer the energy savings that are possible with CFLs 

and will likely be possible through SSL products by that time. 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 11

Nexus Market Research

Conclusions 

As was the case when observed in last year’s (2006) MPER, the market for CFLs continues to 

evolve rapidly, at international, national and state levels. In 2005 the rate of change in 

Massachusetts was faster than in the U.S. as a whole—and had been, as far as records indicate, at 

least since 1998. Starting in 2006, however, that trend reversed, a reversal that accelerated in 

2007; the rate of change in the national market appears to have been faster than the rate of 

change in Massachusetts. Having started earlier and more intensively, the Massachusetts CFL 

market remains ahead of the national market, and is building off a larger base on a per-household 

basis—but it shows signs of leveling off even as the national market is accelerating. This 

suggests that the incremental market effects in Massachusetts this year compared to last year are 

decreasing and may indicate that Massachusetts market is maturing. Moreover, pending national 

legislation could provide a new floor or baseline for lighting efficiency by 2012, consolidating 

gains and ensuring sustainability. Given these trends, the Sponsors should consider ways to 

modify the nature of their program support during the current transition period.

With 13.3 million CFLs sold in 2007, the number sold per household in Massachusetts was 5.45 

CFLs, significantly higher than the 3.12 per household average sold in the U.S. as whole—an 

average that includes Massachusetts and other states with active programs. While the U.S. 

household average likely overstates baseline sales and so should be used only qualitatively, not 

to derive market based estimates or net-to-gross estimates. Hence, CFL sales per capita in 

Massachusetts appear to be 65% greater than the national average. However, as recently as 2006 

they were about 193% greater, when sales per household were 4.26 CFLs in Massachusetts, 

compared to 1.46 nationally. As tracked by the U.S. Department of Commerce and adjusted to 

reflect residential sales, there were about 82 million CFLs sold in 2004, increasing to 90 million 

in 2005, 163 million in 2006, and 349 million CFLs in 2007. 

Behind this floodtide of sales, there appears to be a confluence of trends, including the 

following: 

• Higher energy prices—including not only electricity prices but, more visibly, gasoline prices 

at the pump 

• Lower CFL prices—the average price difference between a 25-watt CFL and a 100-watt 

incandescent in Massachusetts in 2007 was $2.80, compared to a difference of $4.39 in 2005 

• Greater industry commitment—for example, Wal-Mart committed to selling 100 million 

CFLs in 2007, and met the target by September17

• Continuing support from energy-efficiency programs; Massachusetts sponsors are among the 

early champions, especially with the markdown/buydown model, which seems to drive 

higher sales than alternative program models such as coupons

• Increased environmental awareness (the “Al Gore Effect”)—for example, 61% of Americans 

surveyed in March of 2008 said that the effects of global warming have already begun to 

happen, compared to 48% who said the same thing in 199718

 17 Wal-Mart. 2007. “Wal-Mart Surpasses Goal to Sell 100 Million Compact Fluorescent Light Bulbs Three Months 

Early.” http://walmartstores.com/FactsNews/NewsRoom/6756.aspx. Bentonville, Ark.: Wal-Mart Stores, Inc. 18 Gallup Poll, accessed in http://www.pollingreport.com/enviro.htm 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 12

Nexus Market Research

In keeping with the high number of CFLs sold per household, 94% of Massachusetts consumers 

are aware of CFLs, 79% of all households have tried them, and 76% currently use them. As of 

early 2008, there are about 22.6 million CFLs installed in the Commonwealth—or 21.4% of all 

installed medium screw-based lamps—and there are another 5.2 million in storage, nearly all of 

which consumers intend to use. The fact that consumers are keeping CFLs for later use indicates 

that they have become a standard option—something they can reach for in the closet rather than 

making a special trip to the store or placing a catalog or online order.

The complement of the 21.4% of sockets being occupied by CFLs is the 62.5% that are not, 

indicating significant remaining potential in the Massachusetts residential market. However, 

with the market evolving so rapidly at the national level, the question is how much more 

additional savings could be captured with the program than without it. The challenge for the 

Sponsors is to target their efforts where market momentum is lacking. 

Despite the uncertainty, it is also important to keep in mind the change in lighting efficiency 

standards that will become effective in 2012 through EISA 2007. EISA 2007 will use a phased 

approach to introduce more efficient lighting standards, beginning in 2012 Some standards and 

codes take a long time to work through the population of measures. Clothes washers, for 

example, last 15 years or so, and half of the less efficient models remain in place for that many 

years after a change in standards. Incandescent bulbs, in contrast, last only for about 1000 hours 

on average, and bulbs placed in frequently used applications may need to be replaced several 

times a year. When the standard becomes effective, one could expect most inefficient bulbs to 

be changed out in a fairly short time. 

In contrast, the market for CFL fixtures is not transformed; it appears to remain dependent on the 

Sponsors’ support for survival. Furthermore, Sponsors face the challenge of determining 

whether to include or change the focus of their ENERGY STAR-qualified fixture efforts to the 

GU24 standard. 

Meanwhile, technological developments with LEDs also are occurring rapidly. LEDs are 

expected to have efficiency levels similar to CFLs, but most LEDs still are not competitive in 

terms of performance or cost. Currently, LEDs are able to replace only limited numbers of 

fixtures and bulbs. 

Recommendations

The findings and conclusions drawn from the research summarized above and through inferences 

that are made throughout the evaluation process lead to several recommendations for 

consideration by the Sponsors.

Program Structure

• Continue support for each of the current program’s components in the short term—that is, 

the next year or so. 

− NCPs are an extremely cost-effective way to increase sales of energy-efficient 

lighting products that are ready for the mass market.

− The two other program components—coupons and the lighting catalog—also fill 

important functions. Coupons provide an important option to allow program 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 13

Nexus Market Research

participation by retailers who do not have electronic sales tracking systems. The 

catalog provides a venue for the introduction of new products to consumers and for 

the sale of products (such as pin-based replacement bulbs and LEDs) that are difficult 

for them to find elsewhere. 

− To expand the range of product offerings—especially fixtures—that retailers may be 

hesitant to carry because of low sales volumes, it may be worthwhile to expand the 

NCP process to include lighting products sold by third parties over the Internet or 

through catalogs by parties other than the Sponsors.

• Continue to improve tracking of NCP program activity.

− Additional documentation on sales and customers may be needed to support 

evaluation needs and documentation in support of the Forward Capacity Market. 

Sponsors need to consider how this may be done within the structure of the NCP 

program.

• Annual program planning may be too slow to respond to the rapid changes occurring in 

the residential lighting market; the Sponsors may, therefore, consider requiring industry 

partners to provide more frequent and comprehensive data exchanges to better assess the 

market and the role of the program in this market.

Product Incentives

• The incentive structure serves at least two needs: to continue to fill the remaining 

potential for CFL use among all Massachusetts consumers and to encourage the 

development, production, sales, and use of better and more specialized CFLs. In the near 

future, the incentive structure may also be used to encourage the development, 

production, sales, and use of other energy-efficient lighting options, such as SSL. 

• We recommend comparing CFL prices for the same packages at the same chains in 

Massachusetts and one or more non-program states, to help determine how much of 

Sponsors’ incentives are being passed on to consumers. 

• Given the increasing levels of sales outside the program, the Sponsors should consider 

further reductions that would lead toward eliminating the incentives for “plain vanilla” 

CFLs in the 13 to 25 watt range. 

• Continuation of higher incentives for specialty CFLs that have passed quality tests is 

warranted because these products are more likely to satisfy consumer expectations. Also 

increased support for newer specialty CFL applications on the market such as floods or 

candelabra based bulbs (once ENERGY STAR labeled products become available) to 

further support their adoption into socket types with lower saturations of CFLs.

• Consider reducing the incentives paid for CFLs sold at “big box” stores relative to the 

incentive levels offered at other types of stores.

• Continue offering lower incentives paid for CFLs in multi-packs relative to incentives 

paid for products sold individually.

• Consider higher incentives for CFLs with color rendition properties that better meet 

customer requirements. Regardless of whether or not such differentiated incentives are 

adopted, the program should seek to educate consumers about choosing CFLs with color 

properties appropriate to needs or expectations.

• Consider higher incentives for CFLs with lower mercury content. (See “Mercury in 

CFLs” below.)
Massachusetts ENERGY STAR Lighting MPER 2007 Page 14

Nexus Market Research

Changing Markets

• Stepped-up advertising may be warranted to persuade those with just a few CFLs to 

purchase more and install them into other sockets. There remains a sizable segment of 

the population that has still not tried a CFL (21% in 2007), but reaching the group of nonusers may prove to be a considerable challenge, given the high level of awareness of 

CFLs among the general population and the rate at which CFLs have diffused to the rest 

of the population, leaving a hard-to-reach group of “laggards” or “late adopters” as the 

remaining non-users. The Sponsors may also want to consider a boost in advertising to 

leverage the consumer interest in environmental concerns by tying CFLs to the climate 

change issue or to leverage the dollar savings that will be achieved by switching. 

− It may be possible to get to those who have not tried CFLs through mainstream 

media outlets, such as television, radio, and print advertising—as expensive as 

that is—or other means such as the Internet, more point-of-purchase materials at 

retailers, or direct mail. The Sponsors’ increased support for advertising could be 

leveraged through cooperative advertising agreements with retailers and 

manufacturers, perhaps tied to NCP agreements. The increased advertising could 

also be partially supported with the funds resulting from reduced incentives. 

− Strategies for reaching consumers who have not tried CFLs or are not familiar 

with CFLs could include continued additional targeted marketing at grocery 

stores. The telephone survey revealed that these consumers are more likely to 

shop for incandescent light bulbs while grocery shopping than are CFL users; 

continued greater exposure of CFLs in the venues where current non-users are 

most likely to shop for light bulbs should help to raise awareness of the 

technology. 

• Given the rapid development of LED technology, there are likely to be more LED 

applications, and the Sponsors should plan on increasing the promotion of those 

applications as they become available.

− The Sponsors should begin promoting LEDs after the ENERGY STAR 

specification comes out, and invite proposals from manufacturers. 

• Consider how EISA 2007 will impact the lighting market, as more efficient incandescent 

and halogen products are introduced. These products may meet EISA 2007 efficiency 

standards and will be mercury free, but the energy savings from CFLs and LEDs (Tier 2 

products will likely be available by then) may be superior. To avoid lost savings in the 

regulatory switch, Sponsors may want to think now about positioning the type of 

products that will give them the most energy savings. 

− The incandescent wattage scale for defining lighting needs will need to change 

under EISA 2007. One way to encourage consumer choices of the most efficient 

products is to encourage an efficacy standard or lumens-per-watt labeling system. 

A categorical lumen-per-watt scale can help ensure that efficiency standards are 

maintained. 

ENERGY STAR-Qualified Fixtures

• Given findings from this and previous MPERs, we recommend that Sponsors continue to 

provide only that support which is necessary to ensure that manufacturers have a market 

for the current generation of ENERGY STAR-qualified fixtures until the next generation 

becomes more widely available. The ENERGY STAR-qualified fixture market appears 
Massachusetts ENERGY STAR Lighting MPER 2007 Page 15

Nexus Market Research

to remain dependent on Sponsors’ support. A more significant commitment to the 

current generation of ENERGY STAR-qualified fixtures might be read as an implied 

promise to consumers that would be difficult to fulfill: that replacement lamps and 

ballasts for any fixture supported by the program in the past will continue to be available 

for years, if not decades. (Meeting this implicit commitment may be possible through the 

ENERGY STAR Lights catalog, so long as the number of models involved is limited. 

That alone may justify the continued use of the catalog.) 

Mercury in CFLs

• There is a need to establish a long-term solution for CFL disposal in Massachusetts. 

Currently, 43% of CFL users have had to dispose of at least one bulb and most of those 

(72%) have simply thrown the CFL in the trash. An industry-wide solution to financing 

CFL recycling programs remains unresolved. Under the Massachusetts Mercury 

Management Act, 19

• Sponsors have encouraged manufacturers and retailers to help build a mercury recycling 

infrastructure for CFLs via retail and other avenues for the 2008 program. In 2007, 

Sponsors provided consumers with some information on recycling at presentations and 

events and two partnered with the Cambridge post office on a pilot recycling effort. 

Recycling centers exist in various parts of the state, including each town in Barnstable 

County, but access to such centers is not available or convenient for all consumers across 

the state. The Sponsors should continue to encourage mercury disposal solutions with 

industry partners

a law enacted July 28, 2006, program sponsors, retailers, and 

manufacturers are not required to collect spent bulbs. As of May 1, 2008, the 

Massachusetts law prohibits the disposal of products containing mercury in trash and 

wastewater. The Act requires that by January 1, 2007, manufacturers of lamps that 

contain mercury must implement a plan for educating users about recycling “end of life” 

lamps. The law establishes recycling targets for mercury-containing lamps, phased in 

over the next few years.

• The Sponsors see their role in dealing with mercury and recycling as primarily 

educational; they have been approached by recycling vendors but are hesitant to take 

responsibility for recycling efforts. Most believe the manufacturers and retailers (and the 

government) should be responsible for coming up with recycling programs and providing 

disposal sites. Given the potential downsides of not addressing mercury—environmental 

damage and missed energy-savings if consumers are reluctant to use CFLs because of the 

mercury content—Sponsors should consider additional education efforts and a continued 

effort to work with industry, regional, and national partners to establish permanent CFL 

recycling solutions.

 19 Fact Sheet, Summary: Massachusetts Mercury Management Act. Massachusetts Department of Environmental 

Protection. http://www.mass.gov/dep/toxics/laws/hglawfax.pdf (Accessed 4-18-07).
Massachusetts ENERGY STAR Lighting MPER 2007 Page 16

Nexus Market Research

Product Quality

• Sponsors have been strong supporters of product quality assurance efforts for CFLs, and 

these efforts have been valuable in improving manufacturer accountability for product 

quality in the ENERGY STAR program. The Sponsors may wish to actively engage with 

the ENERGY STAR program, which will be taking over product quality testing and 

verification functions from PEARL, to voice their support and to provide relevant 

technical information.

• As SSLs, currently in the form of LEDs, become available, Sponsors may also wish to 

actively engage in the on-going planning and discussions for establishing quality 

standards. Strong product quality will help to assure consumer acceptance of the 

technology and the associated energy-savings that will come from using these products to 

displace inefficient lighting technologies.

Considerations for the Forward Capacity Market

• The Sponsors should consider tracking some products sold through the NCP retail 

channel and determining their disposition, hours of use, wattage displaced, etc. This 

could contribute to a refinement of Sponsors’ assumptions about in-service rates and 

sales and bolster the case for including CFL savings in the FCM.20

• In their bid into the FCM, the Sponsors may also wish to consider additional properties of 

the effect of CFLs on demand for electricity, such as heat factor to account for cooling 

savings (in instances in which the retrofit occurs in an electrically cooled space) and 

increased heating fuel needed because of reduced heat (in cases of electric heating fuels.

These factors—in addition to wattage reduction, hours of use, measure life, and inservice rate—are included in Efficiency Vermont’s Technical Resource Manual.

• At the end of the transition period for FCM, the Sponsors will need to be able to provide 

program sales data by load zone. For states with only a single load zone this is not a 

significant concern, however, since Massachusetts has three load zones, a plan for 

allocating lighting program sales should be fully explored, vetted and put into place prior 

to the onset of the full FCM. We recommend that as part of the coming evaluation cycle, 

resources be set aside to ensure that the processes for tracking and confirming retail sales 

and a system to allocate them by load zone is assessed and functioning in a manner 

consistent with the ISO FCM requirements for submitting this demand reduction 

resource.

It is important to point out that these are recommendations for maintaining and improving a 

program that is an unqualified success. The Sponsors should be commended for their design, 

implementation, and continuing support of a program that is having a profound effect on the 

market and that is saving so much energy cost effectively. 

 20 We offered some suggestions for establishing a system to track CFLs in Appendix H of the 2006 Lighting MPER 

report.
22 Haskell Street, Cambridge, MA 02140 

Phone: (617) 497-7544 Fax: (617) 497-7543 

www.nexusmarketresearch.com 

RESIDENTIAL LIGHTING 

MEASURE LIFE STUDY 

FINAL 

June 4 2008 

Submitted to: 

New England Residential Lighting Program Sponsors 

Submitted by: 

Nexus Market Research, Inc. 

RLW Analytics, Inc. 
Nexus Market Research 

Table of Contents 

1 Executive Summary ................................................................................................................ 1

2 Introduction ............................................................................ Error! Bookmark not defined.

2.1 Structure of this Report ................................................... Error! Bookmark not defined.

2.2 List of Terms Used in this Report ................................... Error! Bookmark not defined.

2.3 Current Lighting Programs in New England ................... Error! Bookmark not defined.

3 Sample Design, Methodology, and Measure Life ................. Error! Bookmark not defined.

3.1 Sample Design ................................................................. Error! Bookmark not defined.

3.2 On-Site Methodology ...................................................... Error! Bookmark not defined.

3.3 Potential Bias................................................................... Error! Bookmark not defined.

3.4 Decision Table for Including Products in Analysis ........ Error! Bookmark not defined.

3.5 Examining Survival and Failure Rates of Included Products ....... Error! Bookmark not 

defined.

3.6 Measure Life Analyses .................................................... Error! Bookmark not defined.

3.7 Parametric Regression Analysis ...................................... Error! Bookmark not defined.

4 Installation Rates for Measure Life Products ......................... Error! Bookmark not defined.

5 Current Disposition and Use of Measure Life Products ........ Error! Bookmark not defined.

5.1 Location of Use ............................................................... Error! Bookmark not defined.

5.2 Products That Have Not Been Installed and Are Not Traceable ... Error! Bookmark not 

defined.

5.3 Product Disposition ......................................................... Error! Bookmark not defined.

5.4 Replacement of Removed Measure Life Products .......... Error! Bookmark not defined.

6 Counting Products in the Home ............................................. Error! Bookmark not defined.

6.1 Socket Count of All Lighting Products ........................... Error! Bookmark not defined.

6.2 All Lighting Products in Storage ..................................... Error! Bookmark not defined.

7 Markdown Products and Program Spillover .......................... Error! Bookmark not defined.

7.1 Possible Markdown Products in Respondent Homes ...... Error! Bookmark not defined.

7.2 Coupon and Direct Install Spillover ................................ Error! Bookmark not defined.

8 Recommendations .................................................................. Error! Bookmark not defined.

Appendix A: Sample Design and Bias........................................... Error! Bookmark not defined.

A.1 Sample Development ...................................................... Error! Bookmark not defined.

A.2 Sample Design ................................................................. Error! Bookmark not defined.

A.3 On-Site Methodology ...................................................... Error! Bookmark not defined.

A.4 Bias Resulting from Sample Design and Methodology .. Error! Bookmark not defined.

A.4.1 Potential Sources of Bias from Sampling Procedures ........... Error! Bookmark not 

defined.

A.4.2 Potential Sources of Bias from Respondent Recall . Error! Bookmark not defined.

A.5 Decision Table for Including Products in Analysis ........ Error! Bookmark not defined.

Appendix B: Preliminary Examination of Measure Life Data ...... Error! Bookmark not defined.

B.1 Examining Survival and Failure Rates of Included Products ....... Error! Bookmark not 

defined.

B.2 Product Quality over Time .............................................. Error! Bookmark not defined.
Nexus Market Research 

Appendix C: Measure Life Analyses ............................................. Error! Bookmark not defined.

C.1 Life Tables ....................................................................... Error! Bookmark not defined.

C.2 Logit Regression Models ................................................ Error! Bookmark not defined.

C.3 Parametric Regression Analysis ...................................... Error! Bookmark not defined.

Appendix D: Not Found/Not Recalled Products ........................... Error! Bookmark not defined.

D.1 Background Information ................................................. Error! Bookmark not defined.

D.2 Ability to Recall over Time ............................................. Error! Bookmark not defined.

D.3 Location of Original Installation ..................................... Error! Bookmark not defined.

D.4 Relationship to Participants who Purchase the Most Products ..... Error! Bookmark not 

defined.

D.5 Various Analysis of Model Numbers .............................. Error! Bookmark not defined.

D.5.1 Miscategorization as Spillover ................................. Error! Bookmark not defined.

D.5.3 Rare Products or Incorrect Model Numbers ............ Error! Bookmark not defined.

D.5.3 Overlap with Most Commonly Obtained Products .. Error! Bookmark not defined.
Nexus Market Research 

List of Tables 

Table 1–1: Recommended Estimates of Measure Life – Decimals ................................................ 1

Table 1–2: Recommended Estimates of Measure Life – Integers .................................................. 1

Table 1–3: Status of Measure Life Products for Use in Analysis ................................................... 3

Table 2–1: Annual Product Distribution in Massachusetts by Type and Program Componenta

................................................................................................ Error! Bookmark not defined.

Table 3–1: Products Available for Study Sample .......................... Error! Bookmark not defined.

Table 3–2: Distribution of Sampled CFLs, External Fixtures, and Internal Fixtures by State and 

Program .................................................................................. Error! Bookmark not defined.

Table 3–3: Categorization of Measure Life Products Based on Rules of Decision Table .... Error! 

Bookmark not defined.

Table 3–4: Characterization of Products by Auditor’s Ability to Locate Them . Error! Bookmark 

not defined.

Table 3–5: Coupon CFL Failure and Survival by Year of Purchase ........... Error! Bookmark not 

defined.

Table 3–6: Direct Install CFL Failure and Survival by Year of Purchase ... Error! Bookmark not 

defined.

Table 3–7: Exterior Fixture Failure and Survival by Year of Purchase ...... Error! Bookmark not 

defined.

Table 3–8: Interior Fixtures Failure and Survival by Year of Purchase ...... Error! Bookmark not 

defined.

Table 3–9: Recommended Estimates of Measure Life – Decimals Reported .... Error! Bookmark 

not defined.

Table 3–10: Recommended Estimates of Measure Life – Integers Reported .... Error! Bookmark 

not defined.

Table 4–1: Product Installation Rates by Year .............................. Error! Bookmark not defined.

Table 5–1: Location of Installed Lighting Products ...................... Error! Bookmark not defined.

Table 5–2: Percentage of Products Respondents Did Not Recall by Quality of Product 

Information ............................................................................ Error! Bookmark not defined.

Table 5–3: Locating Interior Fixtures by Type of Fixture ............. Error! Bookmark not defined.

Table 5–4: Disposal Method of lighting products that broke or burned out Error! Bookmark not 

defined.

Table 5–5: Disposal method of lighting products removed while still working . Error! Bookmark 

not defined.

Table 5–6: Replacement of Removed Lighting Products .............. Error! Bookmark not defined.

Table 6–1: Distribution of Lighting Products by State .................. Error! Bookmark not defined.

Table 6–2: Lighting Products by Location in Household .............. Error! Bookmark not defined.

Table 6–3: Lighting Products by Fixture Type .............................. Error! Bookmark not defined.

Table 6–4: Lighting Products by Control ...................................... Error! Bookmark not defined.

Table 6–5: Wattage of All Installed Lighting Products ................. Error! Bookmark not defined.

Table 6–6: Light Bulbs in Storage ................................................. Error! Bookmark not defined.

Table 6–7: Why CFL Was Not Installed ....................................... Error! Bookmark not defined.

Table 6–8: What Bulb Will CFL Replace...................................... Error! Bookmark not defined.

Table 7–1: Overlap of Measure Life Product Models with Markdown Models. Error! Bookmark 

not defined.
Nexus Market Research 

Table 7–2: Source of CFLs Observed in Respondents’ Homes .... Error! Bookmark not defined.

Table 7–3: Calculating Possible CFL Spillover Purchases ............ Error! Bookmark not defined.

Table 7–4: Motivation for Buying CFL Products .......................... Error! Bookmark not defined.

Table 8–1: Recommended Estimates for Measure Life ................. Error! Bookmark not defined.

Table A–1: Products Listed in Databases Received from Sponsors ............ Error! Bookmark not 

defined.

Table A–2: Products Available for Study Sample ......................... Error! Bookmark not defined.

Table A–3: Desired and Actual Completions by Year and Type of Program .... Error! Bookmark 

not defined.

Table A–4: Distribution of Sampled CFLs, External Fixtures, and Internal Fixtures by State and 

Program .................................................................................. Error! Bookmark not defined.

Table A–5: Decision Table for Useful Life ................................... Error! Bookmark not defined.

Table A–6: Categorization of Measure Life Products Based on Rules of Decision Table .... Error! 

Bookmark not defined.

Table A–7: Characterization of Products by Auditor’s Ability to Locate Them Error! Bookmark 

not defined.

Table B–1: Coupon CFL Failure and Survival by Year of Purchase .......... Error! Bookmark not 

defined.

Table B–2: Direct Install CFL Failure and Survival by Year of Purchase .. Error! Bookmark not 

defined.

Table B–3: Exterior Fixture Failure and Survival by Year of Purchase ...... Error! Bookmark not 

defined.

Table B–4: Interior Fixtures Failure and Survival by Year of Purchase ..... Error! Bookmark not 

defined.

Table C–1: Estimated Measure Life, Logit Regression Analysis .. Error! Bookmark not defined.

Table C–2: Estimated Measure Life, Parametric Regression Analyses ...... Error! Bookmark not 

defined.

Table C–3: Estimated Measure Life of Products also Offered in Markdown Programs, Parametric 

Regression Analyses .............................................................. Error! Bookmark not defined.

Table C–4: Recommended Estimates of Measure Life – Decimals Reported .... Error! Bookmark 

not defined.

Table C–5: Recommended Estimates of Measure Life – Integers ReportedError! Bookmark not 

defined.

Table D–1: Number and Percentage of Not Found/Not Recalled Products Error! Bookmark not 

defined.

Table D–2: Percentage of Products Respondents Did Not Recall by Quality of Product 

Information ............................................................................ Error! Bookmark not defined.

Table D–3: Status of Lighting Products with Model Numbers Sometimes Found/Recalled Error! 

Bookmark not defined.

Table D–4: Status of All Lighting Products, Except those Not Found/Not Recalled............ Error! 

Bookmark not defined.
Nexus Market Research 

List of Figures 

Figure 3-1: Survival and Failure Rates of Coupon CFLs (Raw Data) ......... Error! Bookmark not 

defined.

Figure 3-2: Survival and Failure Rates of Direct Install CFLs (Raw Data) Error! Bookmark not 

defined.

Figure 3-3: Survival and Failure Rates of Exterior Fixtures (Raw Data) .... Error! Bookmark not 

defined.

Figure 3-4: Survival and Failure Rates of Interior Fixtures (Raw Data) ..... Error! Bookmark not 

defined.

Figure A-1: Percentage of Products Found by Auditor or Recalled by Respondent ............. Error! 

Bookmark not defined.

Figure B-1: Survival and Failure Rates of Coupon CFLs (Raw Data) ........ Error! Bookmark not 

defined.

Figure B-2: Survival and Failure Rates of Direct Install CFLs (Raw Data) Error! Bookmark not 

defined.

Figure B-3: Survival and Failure Rates of Exterior Fixtures (Raw Data) ... Error! Bookmark not 

defined.

Figure B-4: Survival and Failure Rates of Interior Fixtures (Raw Data) .... Error! Bookmark not 

defined.

Figure B-5: Survival Rates of CFLs by Year Obtained (Raw Data) ........... Error! Bookmark not 

defined.

Figure B-6: Survival Rates of Exterior Fixtures by Year Obtained (Raw Data) Error! Bookmark 

not defined.

Figure B-7: Survival Rates of Interior Fixtures by Year Obtained (Raw Data) . Error! Bookmark 

not defined.

Figure C-1: Cumulative Survival Rates from Life Tables ............. Error! Bookmark not defined.

Figure C-2: Estimated Survival Rates from Logit Regression ...... Error! Bookmark not defined.

Figure C-3: Estimated Measure Life of CFLs, Parametric Regression Analyses .................. Error! 

Bookmark not defined.

Figure C-4: Measure Life of Exterior Fixtures, Parametric Regression Analyses ................ Error! 

Bookmark not defined.

Figure C-5: Measure Life of Interior Fixtures, Parametric Regression Analyses ................. Error! 

Bookmark not defined.

Figure D-1: Percentage of Products Found by Auditor or Recalled by Respondent ............. Error! 

Bookmark not defined.
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 1

Nexus Market Research 

1 Executive Summary 

The purpose of this study is to estimate measure life for lighting products distributed through 

energy efficiency programs in New England. As explained in more detail in the full study (see 

Section Error! Reference source not found. and Appendix C), we recommend three different 

program-specific estimates of measure life for CFLs (coupon, direct install, and markdown1) and 

two for exterior fixtures (markdown and all other programs). These estimates and their 

respective confidence intervals are shown in Table 1–1 (to two decimal places) and in Table 1–2 

(as integers). We do not suggest an estimate of measure life for interior fixtures as we believe 

the data were collected too early in their life cycle to provide a reliable estimate. 

Table 1–1: Recommended Estimates of Measure Life – Decimals 

Product Measure Life 80% Confidence Interval 

Low High 

Coupon CFLs 5.48 5.06 5.91 

Direct Install CFLs 6.67 5.97 7.36 

Markdown CFLs (all states) 6.82 6.15 7.44 

Coupon and Direct Install Exterior Fixtures 5.47 5.00 5.93 

Markdown Exterior Fixtures 5.88 5.24 6.52 

All Interior Fixtures Continue using current estimates of measure life 

Table 1–2: Recommended Estimates of Measure Life – Integers 

Product Measure Life 80% Confidence Interval

Low High 

Coupon CFLs 5 5 6 

Direct Install CFLs 7 6 7 

Markdown CFLs (all states) 7 6 7 

Coupon and Direct Install Exterior Fixtures 5 5 6 

Markdown Exterior Fixtures 6 5 7 

All Interior Fixtures Continue using current estimates of measure life 

Our definition of “measure life” is consistent with that used in the Measure Life Report prepared 

by GDS Associates for the New England State Program Working Group (SPWG).2

 “For 

programs delivered by program administrators in New England, Measure Life includes 

equipment life and measure persistence (not savings persistence). 

 Equipment Life means the number of years that a measure is installed and will operate 

until failure, and 

 Measure Persistence takes into account business turnover, early retirement of installed 

equipment, and other reasons measures might be removed or discontinued.” 

 

1

 Due to the diversity of program types throughout the region, we use the term “markdown” to refer to both 

markdown programs (offered in all the states) and buydown programs (offered in some of the states). In 

Massachusetts, the Negotiated Cooperative Promotions (NCPs) include both markdown and buydown programs. 

2

 GDS Associates, Inc. (2007) Measure Life Report: Residential and Commercial/Industrial Lighting and HVAC 

Measures. Prepared for The New England State Program Working Group for use as an Energy Efficiency 

Measures/Programs Reference Document for the ISO Forward Capacity Market (FCM). 
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 2

Nexus Market Research 

Specifically, our measure life estimates do not distinguish between equipment life and measure 

persistence; our estimates—one for each measure category—include both those products that 

were installed and operated until failure (i.e., equipment life) as well as those that were retired 

early and permanently removed from service for any reason, be it early failure, breakage, or the 

respondent not liking the product (i.e., measure persistence). The remainder of this executive 

summary provides background information about the study and highlights some of the key 

results and recommendations. 

Sample Development and Design: The sample design for this study is based on the number of 

energy efficient lighting products distributed through energy-efficiency programs conducted in 

Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont (See Section 

Error! Reference source not found. and Appendix A). For a program to be included, we 

needed to be able to determine the following information for a product or the person obtaining 

that product: 

1. Knowledge that the respondent had obtained at least one energy efficient lighting product 

through a Sponsor’s program from 2002 to 2006 

2. Detailed information on the model number, manufacturer and wattage for the product in 

order to identify it in the home; for direct install programs, we also looked for the 

location of installation 

3. Customer contact information 

After reviewing the databases of households that had participated in various retail and direct 

install programs, NMR and RLW determined that only the coupon, single-family ENERGY 

STAR Homes and MassSAVE files contained sufficient product and resident contact information 

to use for the study. We drew a random sample of participants based on the type and number of 

products they had obtained through the programs. We collectively refer to these sample products 

as the “measure life products.” Auditors visited a total of 285 homes to conduct an inventory of 

lighting products and a respondent survey designed to learn more about the measure life products 

as well as other lighting products found in the home. 

Bias Resulting from Sample Design and Methodology: The sample design and methodology 

used in this study introduce several potential sources of bias (See Appendix A): 

1. The lack of adequate product and customer contact information limited the sample to the 

coupon and a few direct-install programs while excluding products from all other 

Sponsor-administered lighting programs. 

2. In order to complete the study in a timely and cost effective manner, the later on-site 

surveys targeted homes with large numbers of fixtures. This decision resulted in the 

unintended inclusion of a disproportionate number of electricians, contractors, and 

landlords, as they had purchased large numbers of fixtures to install in locations other 

than their own homes. Because they installed these products at different addresses, we 

were unable to verify the disposition of many of these products. Furthermore, 

respondents with numerous products were less likely to recall the disposition of at least 

some of them (See Appendix D). 
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 3

Nexus Market Research 

3. Given the amount of time that passed between the household obtaining the lighting 

products and being contacted for this study, inaccurate customer recall of products that 

the auditor did not personally observe accounts for the majority of products excluded 

from the analysis and presents a major source of potential bias 

4. Because we contacted respondents at the phone number given at the time of participation, 

the resulting sample included only those who had not moved or changed their phone 

number in at least one and up to six years. This likely means that low-income 

households, renters, and younger adults are not well represented in the sample. 

While we recognize that the potential for bias exists, we cannot say whether such bias would 

produce higher or lower estimates than the ones we present here. Moreover, we find no evidence 

of bias across states or Sponsors. 

Characterizing Products as Survived, Failed, or Excluded: In order to estimate measure life, 

we had to classify individual products as having “survived” or “failed” for a specific period of 

time. In cooperation with the Sponsors, we developed a “Decision Table” to guide the 

classification of products into one of three categories: 1) survived, 2) failed or 3) excluded (See 

Section Error! Reference source not found. and Appendix A). For a product to be classified as 

“survived” the auditor typically had to confirm its continued installation and operation visually. 

An exception to this was the inclusion of products reported installed in rentals, second homes, 

and businesses if the respondent was in the position of knowing the current status of the product. 

“Failed” products are those that burned out, broke, or were permanently removed from service, 

including those that broke or failed and were returned to the store. We excluded products: that 

could not be found (accounting for the majority of excluded products, see Appendix D); that 

were reported installed but the respondent was not in the position to know if the product 

remained in place (e.g., by a contractor); that were installed outside of New England; that were 

being stored for future use; and that had been returned to the store before the product failed (e.g., 

a CFL may not have fit a fixture or the customer decided they did not like a fixture) or given 

away. Table 1–3 summarizes these classifications, but see Section Error! Reference source not 

found. and Appendix A for more detail. 

Table 1–3: Status of Measure Life Products for Use in Analysis 

Product Status CFLs Fixtures 

Coupon Direct Install Exterior Interior 

Survived 48% 56% 37% 55% 

Failed 20% 14% 17% 6% 

Excluded from Analysis 32% 31% 46% 39% 

Total Number of Products 695 441 215 397 

Measure Life Analysis: We relied on three types of “survival analyses” to estimate the measure 

life of the products distributed through the coupon and direct install programs under 

consideration (See Section Error! Reference source not found., Section Error! Reference 

source not found. and Appendix C): 

Method 1: Life Tables 

Method 2: Logit Regression 
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 4

Nexus Market Research 

 Method 3: Parametric Regression Models of Survival Analysis3

We chose estimates resulting from parametric regression analysis. According to our results, the 

measure life of CFLs (coupon, direct install, and markdown) falls between five and one-half and 

seven years, while that for exterior fixtures (coupon, direct install, and markdown) is between 

five and one-half years and six years (Table 1–1). However, we do not believe that the data or 

results are adequate for predicting the measure life of interior fixtures because this study was 

conducted too early in their lifecycle. The measure life data also provide some indication of 

increased survival rates over time for CFLs, perhaps as a result of improved product quality, 

although the small sample size and limited number of failures in recent years curtail our ability to 

conduct meaningful statistical analyses to verify improved quality. 

The reader will note that we provide an estimate of measure life for markdown CFLs and 

exterior fixtures. We did not include the lighting markdown and buydown programs 

(collectively referred to as “markdown” programs in this document since not all Sponsors used 

the buydown approach) in the sample of measure life products due to a lack of participant 

contact information. Even so, markdown programs account for the vast majority of lighting 

products distributed through the Sponsors’ programs. For this reason, we conducted analyses on 

the subset of products with model numbers obtained through the coupon or direct install 

programs that were also distributed through markdown programs in order to provide an estimate 

of measure life for the markdown products (Section Error! Reference source not found.). We 

supply these estimates with three important caveats: 1) the population who purchases markdown 

products may differ from those who take part in coupon or direct install programs, 2) not all 

markdown model numbers were represented in the sample of measure life products, and 3) the 

distribution and usage of products actually purchased in the markdown programs may vary from 

what we observed from these products obtained through coupon and direct install products. We 

believe it would be wise to conduct a follow-up study of the measure life of markdown products 

in the near future. 

Measure Life Product Use and Disposition: Most of the measure life CFLs and many interior 

fixtures were found installed in the living room (22% to 23%), bedroom (16% to 24%), kitchen 

and dining room (11% to 19%), and the basement (9% to 11%) of respondents’ homes (See 

Section Error! Reference source not found.). Interior fixtures were most commonly installed 

in foyers and hallways (32%). Only four percent of coupon CFLs and one percent of all other 

products were found in storage, likely reflecting the fact that many of these products had been in 

the respondents’ home between 18 months and six years by the time we visited (See Section 

Error! Reference source not found.). Most of the products had likely been installed—or 

misplaced—by the time we conducted the on-sites. It is also the case that direct install products 

are typically installed by the auditor during the visit to the customer’s home. Alternatively, it is 

possible that some of the stored CFLs are measure life products on which we had incomplete or 

perhaps incorrect information from the database, or that respondents confused them with 

products they had obtained outside the coupon or direct install programs. Respondents usually 

replace burned out or broken CFLs with new CFLs (59%), but broken energy-efficient fixtures 

are more commonly replaced with regular fixtures and incandescent bulbs (59%) (See Section 

Error! Reference source not found.). Once CFLs or fixtures burn out, most participants throw 

 

3

 We examined five types of Parametric Regression models in this analysis. 
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 5

Nexus Market Research 

them away in the trash (84%). Few respondents report recycling the CFLs (14%). Previous 

research we have conducted indicates that few people are aware of the mercury in CFLs, 

although recent media attention and Sponsor education campaigns have raised awareness.4

 As a 

result, most consumers throw the CFLs away as they would other bulbs. However, it is also the 

case that to recycle CFLs in many of the states participating in this study, users must save broken 

or burned out bulbs and take them to hazardous waste drop-off sites (often associated with towns 

or municipalities), usually on specific dates. 

Identification of Markdown and Spillover Products: A secondary objective of this study is to 

estimate the number of products currently in respondents’ homes that may have potentially been 

purchased through lighting markdown programs run by the Sponsors. We matched a list of 

model numbers from Energy Federation Incorporated (EFI) of all markdown products offered in 

New England (with a separate list for Vermont) with all the measure life products currently 

installed in the 285 households (See Section Error! Reference source not found.). We also 

asked respondents how much they paid for the CFLs and fixtures and where they were 

purchased. Only two fixtures appear to be markdown products, but a total of 21% of all CFLs in 

respondents’ homes (942 in total, or 3.3 per home) are likely markdown purchases. Note that in 

2006 markdown products accounted for about 85% of CFLs distributed through Massachusetts 

programs, but because the participants in the measure life programs have obtained products 

through coupon or direct install programs, they may have had less need to buy markdown 

products than households that are not obtaining CFLs for the first time. 

Finally, we estimated spillover for the coupon and direct install programs (See Section Error! 

Reference source not found.). We limited the estimates to non-markdown CFLs that were 

purchased after the respondents’ participation in the coupon or direct install programs. 

Respondents had to be aware of the program and to state that their participation in the coupon or 

direct install program strongly influenced their decision to purchase the non-program products. 

In total, there are 892 likely spillover CFLs found in the homes of coupon participants (4.9 per 

coupon household) compared to 695 coupon CFLs—amounting to spillover of 128%—and 355 

likely spillover CFLs found in the homes of direct install participants (3.4 per direct install 

household) compared to 441 direct install CFLs—amounting to spillover of 81%. 

However, it should be noted that our estimate of spillover does not take into account program 

influences of which the respondent is not aware, such as the fact that the success of such 

programs has increased the availability and lowered the price of CFLs. Furthermore, it does not 

include any program-induced purchases of CFLs that are no longer in their homes (e.g., they may 

have burned out, been given away, etc.). Taking these other factors into account would tend to 

increase the spillover rate. In contrast, it is also likely that the markdown program is responsible 

for some of the spillover that our methodology attributes to direct install or coupon programs. 

 

4

 NMR (2008) Telephone Survey Results for Market Progress and Evaluation Report (MPER) 2007 Massachusetts 

ENERGY STAR Lighting Program. Submitted to National Grid, Cape Light Compact, NSTAR Electric Company, 

Western Massachusetts Electric Company, and Unitil. Draft April 4, 2008. 
FINAL Report on Lighting Measure Life Study June 4, 2008 Page 6

Nexus Market Research 

Recommendations: The findings from this analysis lead to the following recommendations for 

the Sponsors: 

 Adopt the measure life estimates presented in Table 1–1 

 To the extent possible, collect the following information in direct install programs: 

product type, manufacturer, model number, fixture type, wattage, room/location of 

installation, date of installation, and any other product as well as customer information 

including name, address, and phone number. 

 Conduct a measure life study of interior fixtures in the future, as our study occurred too 

early in their life cycle to provide reliable estimates 

 Conduct a process evaluation to examine problems with tracking databases; consider a 

study that tracks new coupon purchases over time in order to ascertain what happens to 

products after they leave the stores. 

 Conduct a long-term measure life study relying on a panel-based approach and using a 

sample drawn in part from the current Markdown Impact Study being conducted for 

Sponsors in Connecticut, Massachusetts, Rhode Island, and Vermont and marking the 

appropriate products with a sticker or permanent marker for future identification. 

 Continue current support for quality assurance efforts. 

Additional Topics: This report also addresses the following topics: 

 Installation rates of measure life CFL products included in this study (See Section Error! 

Reference source not found.) 

 Analysis of all lighting products currently installed in the participants’ homes (See 

Section Error! Reference source not found.) 

 Analysis of all lighting products currently placed in storage (See Section Error! 

Reference source not found.) 
Final Report

Coincidence Factor Study

Residential Room Air Conditioners

 

 

 

 

 

June 27, 2007

June 23, 2008

Prepared by:

RLW Analytics

179 Main Street

Middletown, CT 06457

(860) 346-5001

Prepared for;

Northeast Energy Efficiency 

Partnerships’ New England Evaluation 

and State Program Working Group 

Sponsored by:

Cape Light Compact

Efficiency Vermont

National Grid USA

NSTAR Electric

Western Massachusetts Electric

Maine PUC

Fitchburg Electric

New Hampshire Electric Cooperative

Public Service of New Hampshire

Unitil

 
Acknowledgement:

The authors wish to thank all of the people at the State Program Working Group (SPWG), 

Northeast Energy Efficiency Partnership (NEEP) who took the time to support and help with this 

study. Regrettably, we cannot thank everyone individually, but we do want to acknowledge the 

contributions made by Gail Azulay, Tom Belair, Lisa Glover, Cheryl Jenkins, Angela Li, Ralph 

Prahl and Elizabeth Titus. The data, insight, and support provided by these individuals helped to 

establish the foundation for this report. RLW assumes sole responsibility for any errors or 

omissions in this report.
Executive Summary

The New England State Program Working Group (SPWG)1 contracted with RLW to calculate on-peak 

and seasonal peak coincidence factors for residential room Air Conditioner (RAC) measures that could be 

consistently applied to energy efficiency programs that may bid into the ISO-NE Forward Capacity 

Market (FCM) in any of the New England states. The study covered four of the six New England states

including Massachusetts, New Hampshire, Rhode Island and Vermont. Maine also sponsored the study 

although there were no participating units in the state and no on-site metering or survey activity was 

conducted in the state. Connecticut did not participate in the study because they no longer offer incentives 

for room AC units.2

The study utilized interval metered power data from 93 on-site visits that were nested within a sample of 

approximately 610 phone surveys. The sample was designed to allocate on-site visits and phone surveys 

equally by the six ISO-NE load zones with participating room AC units from program years 2005 and 

2006. Figure i- 1 shows the actual distribution of data collection activities by load zones, the on-site 

numbers reflect sites with both a phone survey and site visit. NEMA NHRI SEMA VT WCMA

Total

On-site

Survey 0

100

200

300

400

500

600

Distribution of On-sites and Phone Surveys Completed 

On-site 15 16 19 10 15 18 93

Survey 89 83 81 93 89 82 517

NEMA NH RI SEMA VT WCMA Total

Figure i- 1: Distribution of On-site and Phone Surveys

 1 Represented by the state regulatory agencies (CT DPUC, Maine PUC, MA DOER, NH PUC, RI PUC, and VT 

PSB) and associated energy efficiency program administrators (Cape Light Compact, Maine PUC, Efficiency 

Vermont, National Grid (MA, NH & RI), Northeast Utilities (CT&MA), NSTAR, PSNH, United Illuminating, and 

Unitil (MA&NH)).

2 There are no participant sites in Maine, however results will be provided by adjusting the study results to Maine 

weather.
The analysis of the primary data utilized a two step approach the first step was to create a regression 

model of the operation of the Room AC units using the real year weather data and actual metered data. 

The second step was to use the resulting model to predict the operation of the room AC units across the 

ISO-NE FCM performance hours for 2007 and typical year after adjusting for any bias in the on-site 

sample. The nested on-site sample technique was used to control for potential bias in the on-site sample, 

specifically selection bias due to the increased probability that people who are generally home during the 

day would be over represented in the on-site sample.

3

• Type of Area Served (i.e. Bedroom vs. Non-bedroom),

A multi-variant regression model was constructed 

using the metered interval power data (for 114 room AC units) and the survey response data along with

hourly weather data from the appropriate weather station. 

There were six survey variables that were found to have statistically significant impact on the regression 

model as follows:

• Home During Day,

• Cooling Capacity per Area Served (BTU/ft

2

),

• Outside Temperature when Cooling Begins,

• Schedule or Continuous Operation, and

• Cooling Setting.

Each of these variables were tested to determine if there was a statistically different distribution of the 

variables within a load zone when compared with the mean values for the whole dataset using a T-test 

methodology. The largest single change from on-site data occurred in the occupancy variable, which had 

on-site customers reporting that 73% were generally home during the day as opposed to 53% in the larger 

survey sample. Both the space type and occupancy variables did not show significant variation between 

the overall survey results and the load zone level survey results. Table i - 1 provides a summary of the 

changes to the four remaining variables, which show that unique results were calculated for the NEMA, 

RI, SEMA and VT load zones. The results for the NH and WCMA load zones were identical to those 

provided by the model inputs using the average survey response data for all zones. 

 3 Phone survey results from surveys conducted during evening and weekend hours were used to establish occupancy 

rates for the population. 
Load Zone BTU/sqft Outside Temp Cont_Sched Cooling Setpoint

All Zones 32.9 82.4 0.28 70.5

NEMA 35.2 83.8 0.28 70.5

NH 32.9 82.4 0.28 70.5

RI 32.9 83.9 0.35 71.5

SEMA 32.9 81.0 0.28 70.5

VT 29.7 82.4 0.28 70.5

WCMA 32.9 82.4 0.28 70.5

Table i - 1: Summary of Zonal Changes to Survey Variables4

The Coincidence Factors (CFs) and Full Load Equivalent Hours (FLEHs) were developed for the 2007 

summer season using the operating profiles that had been adjusted for all applicable bias using the phone 

survey response data as described above. The calculation of the On-Peak CF was relatively 

straightforward since the performance hours are time dependent and can be calculated without having 

extreme ambient weather conditions. The calculation of the FLEHs was also straightforward and was

calculated from the bias adjusted operating profiles directly. The weather normalized CFs and FLEHs

were computed by using the bias adjusted regression model and using Typical Meteorological Year 

(TMY 2) weather data to calculate the results. Since the results are driven by differences in load zone 

variables and weather file data the results are reported out at the weather file level using survey inputs for 

the applicable load zones. The results were calculated by holding the survey variables static and then 

running the nine different weather files so that the hourly weather variables could be used to provide 

hourly results. Table i - 2 provides a summary of the results for all weather files using the average 

survey inputs for all Load zones for the On-Peak performance hours 1:00 PM to 5:00 PM June through 

August using 2007 and TMY2 weather data. 

Weather Files On-Peak CF Seasonal CF On-Peak CF Seasonal CF 2007 FLEH TMY2 FLEH

Albany, NY 0.154 0.276 0.142 NA 224 184

Boston, MA 0.134 0.304 0.125 NA 228 175

Burlington, VT 0.139 0.276 0.119 NA 166 141

Caribou, ME 0.080 0.131 0.080 NA 60 42

Concord NH 0.143 0.290 0.134 NA 171 149

Hartford, CT 0.170 0.303 0.171 NA 272 253

Portland, ME 0.111 0.270 0.111 NA 119 102

Providence , RI 0.159 0.296 0.144 NA 245 204

Worcester, MA 0.131 0.261 0.113 NA 172 134

Average for All Load Zones

2007 Weather TMY2 Weather

Average for All Load Zones Average for All Load Zones

Table i - 2: Summary of CF and FLEH by Weather File using Average Load Zone Data

 4 The zone specific responses that are different from the average for all zones are shown in bold font. 
Although there were slight differences in CF and FLEHs due to zonal differences in the model inputs the 

difference in the final results were not much more than ± 0.001 for CF and ± 3 hours for FLEH. 

Therefore although the zonal differences in survey responses for some of the model variables were 

statistically significant when these different model input were run the results did not provide numerically 

significant differences in the results.5

Table i - 2

 As a result we recommend that the calculation of DRV for each 

load zone use the CFs provided in .

The project results are reported out by weather file because the CF and FLEHs were calculated using the 

regression model and hourly weather data. The optimum method for determining RAC savings for a 

sponsor that operates in multiple load zones and/or has customers that should be modeled using multiple 

weather files would be to assign load zones and weather file designations to each rebate based upon the 

location of the customer by town and or zip code. Once this has been accomplished then capacity or 

demand reduction weighted allocations can be developed for each load zone where multiple weather files 

are applicable. If all of the demand reduction within a load zone is associated with one weather file then 

the sponsor can simply select the appropriate CF for the weather file as given in Table i - 2. 

The Seasonal Peak performance hours were calculated by determining the hours when the real-time 

system load meets or exceeds 90% of the 50/50 CELT forecast for the summer 2007 period of 27,360 

MW.

6 There were a total of 24 hours during the summer of 2007 when the real–time system load was 

24,624 MW or greater, eight hours during June and 16 hours during August, and the 2007 Seasonal Peak 

CFs were calculated during those hours. It was not possible to calculate the TMY 2 Seasonal peak CF 

values because of the method used to create TMY 2 weather data, which uses “typical” months to create 

an annual file.7

 5 This was due to a combination of factors primarily the relatively small differences in the variables, and changes in 

multiple variables canceling each other out.

6 Data taken from ISO-NE 2007 Capacity, Energy, Load and Transmission (CELT) report dated April 20, 2007. 7 For example the June data for Albany could be from 1976, while the Boston data could be from 1980 and Hartford 

from 1978. In order to develop an accurate typical regional weather model it will be necessary to select typical 

months from the same year for all of the regional files.

 The ISO-NE report entitled “Summer 2007 Weather Normal Peak Load” noted that the 

weather normalized peak load for 2007 was 27,460 MW, 0.4% (100 MW) higher than the April 2007 

forecast of 27,360 MW for the summer of 2007. According to the report “The summer of 2007 can be 

characterized as normal with respect to overall temperature and humidity.” Therefore we would defer to 

ISO-NE characterization of the summer of 2007 as normal with respect to temperature and humidity and 

recommend that both the 2007 On-Peak CFs and 2007 Seasonal Peak CFs be used for future year DRV 

calculation by the project sponsors. 
The relative precision of the estimated impacts provided from the bias adjusted model could not be 

calculated directly because the model used the average inputs from the survey data and thus provided only 

one set of numbers depending upon the load zone and weather file selected. A first order approximation 

of the relative precision is provided by the following equations;

Y = f(x) + E => E = Y-f(x)

Yadj = f(xo) + E

Yadj = Y + [f(xo) – f(x)] Where,

Y = the actual CF for the hour from the metered

f(x) = the predicted value from unadjusted model

f(xo) = the predicted value after adjusting the model for bias

E = expected error in the adjusted model

Yadj = the predicted output from the adjusted model

Table i - 3 provides the estimated relative precision of the monthly and summer On-Peak CF values using 

the methodology explained above. The relative precision ranged from ±14.4% for June to ±10.4% for the 

summer season. Note that the mean value for June was 0.218, which was higher than expected because 

most of the June metered data was collected during a heat wave at the end of the month.8

Standard Relative 

Month sample (n) Mean Deviation Cv Precision

June 82 0.218 0.222 1.02 ±14.4%

July 108 0.156 0.155 0.99 ±12.2%

August 108 0.174 0.164 0.94 ±11.7%

Summer 114 0.175 0.152 0.87 ±10.4%

 

Table i - 3: Estimated Relative Precision of On-Peak CF 

 8 The adjusted model results reflect the mean coincident value during the entire month of June and are therefore 

significantly lower.

Based on ISO-NE characterization of the summer of 2007 as normal with respect to 

temperature and humidity, we recommend that both the 2007 On-Peak CFs and 2007 Seasonal 

Peak CFs be used for future year estimates of Demand Reduction Values. 
Coincidence Factor Study

Residential and Commercial & Industrial Lighting Measures

Spring 2007

Prepared for:

New England State 

Program Working Group (SPWG)

Prepared by:

179 Main Street

Middletown, CT 06457

(860) 346-5001

Prepared for;

New England State Program Working 

Group (SPWG)

For use as an 

Energy Efficiency Measures/Programs 

Reference Document for the

ISO Forward Capacity Market (FCM)

 
Acknowledgement:

The authors wish to thank all of the people at the State Program Working Group (SPWG), 

Northeast Energy Efficiency Partnership (NEEP) who took the time to support and help with this 

study. Regrettably, we cannot thank everyone individually, but we do want to acknowledge the 

contributions made by Julie Michals, Jeff Schlegel Tom Belair, Carol White, Chris Neme, Paul 

Horowitz, Ralph Prahl and Elizabeth Titus. The data, insight, and support provided by these 

individuals helped to establish the foundation for this report. RLW assumes sole responsibility 

for any errors or omissions in this report.
Table of Contents

EXECUTIVE SUMMARY ..................................................................................................1

Residential Lighting Coincidence Factor Results ....................................................................... 2

Commercial & Industrial Lighting Coincidence Factor Results................................................. 5

Commercial & Industrial Occupancy Sensor Coincidence Factor Results................................. 9
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 1

_____________________________________________________________________________

March 25, 2007 Final REPORT

Executive Summary

The New England State Program Working Group (SPWG)1

• Summer On-Peak: average weekday from 1-5 PM throughout June, July and August.

contracted with RLW to calculate 

coincidence factors for residential and commercial & industrial (C&I) lighting measures that 

could be consistently applied to energy efficiency programs that may bid into the ISO-NE 

Forward Capacity Market (FCM) in any of the New England states. As directed by the SPWG, 

the focus of this effort was on lighting measures.

Resulting coincidence factors presented in this report were developed to work as common values 

accepted by all New England states for the FCM that can be applied or used as appropriate; they 

are based on measures installed by energy efficiency programs in the New England states that 

have supported this research effort.

This section of the report describes the analytical results and conclusions for the calculation of the 

Coincidence Factors (CFs) for the Residential and Commercial & Industrial Lighting measures.

Energy Efficiency demand reductions can be classified in the Forward Capacity Market (FCM) as 

one of three different types of assets, based upon the performance hours that will be used for 

evaluation. The most straightforward type of asset is On-Peak, because the performance hours are 

fixed and defined as follows: 

• Winter On-Peak: average weekday from 5-7 PM throughout December and January.

ISO-NE hourly load data and forecast data were obtained for the past several years from recent 

energy efficiency program evaluations throughout New England, as described in Appendix A. 

They were analyzed to determine Seasonal Peak performance hours and Critical Peak 

performance hours, which are defined as follows:

• Seasonal Peak Hours occur when Real Time load is equal to or greater than 90% of the 

50/50 seasonal peak load forecast during Summer (June – August) or Winter (December 

and January) months.

 1 Represented by the state regulatory agencies (CT DPUC, Maine PUC, MA DOER, NH PUC, RI PUC, 

and VT PSB) and associated energy efficiency program administrators (Cape Light Compact, Efficiency 

Maine, Efficiency Vermont, National Grid (MA, NH & RI), Northeast Utilities (CT&MA), NSTAR, 

PSNH, United Illuminating, and Unitil (MA&NH)).
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 2

_____________________________________________________________________________

March 25, 2007 Final REPORT

• Critical Peak Performance Hours occur when the Day Ahead Load forecast is equal to 

or greater than 95% of the 50/50 seasonal peak load forecast during Summer (June –

August) or Winter (December and January) months and also includes shortage hours.

 Shortage hours occur during Operating Procedure2

Coincidence Factors (CFs) are defined in this study as the fractions of the connected (or rated) 

load (based on actual lighting Watts, motor nameplate horsepower and efficiency, AC rated 

capacity and efficiency, etc.) reductions that actually occur during each of the seasonal demand

windows. They are the ratio of the actual demand reductions during the coincident windows to 

the maximum connected load reductions. Under this definition other issues such as diversity and 

load factor are automatically accounted for, and only the coincidence factor will be necessary to 

determine coincident demand reductions from readily observable equipment nameplate (rated) 

information. In other words, coincident demand reduction will simply be the product of the 

coincidence factor and the connected equipment load kW reduction.

4 (OP4) level 6 or higher 

events, at level 6 the 30-minute operating reserve begins to be depleted. 

Residential Lighting Coincidence Factor Results

Table i - 1 and Table i - 2 provide the unweighted and weighted, Summer On-Peak and Winter 

On-Peak CFs as well as the associated relative precisions for all residential lighting. The CFs 

were developed using only metered data that were acquired during the winter (December and 

January) or summer (June, July and August) peak months; the number of loggers used in the 

analysis is provided in the tables. The weighted CFs were developed by weighting the logger 

files based upon the connected load that the logger represents. In most cases the weighted results

are slightly higher than the unweighted results. The CFs for the summer range from a low of 

0.06 for June to a high of 0.094 for August, with the average summer CF between 0.076 

unweighted and 0.082 weighted. If the average is carried to only two decimal places then the 

result is a summer average CF of 0.08 for both methodologies. The relative precision for the 

average summer on-peak period is ±6.1% at the 80% confidence interval. 

 2 Operating Procedures are from the ISO-NE to address potential capacity shortages.
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 3

_____________________________________________________________________________

March 25, 2007 Final REPORT

Sample Size Unweighted Weighted Unweighted

Data Period n CF CF Rel Precision

June 210 0.060 0.069 ±11.6%

July 102 0.081 0.086 ±12.5%

August 189 0.094 0.092 ±8.7%

Average Summer 501 0.076 0.082 ±6.1%

Summer On-Peak Hours 1PM - 5PM

Table i - 1: Summer On-Peak CFs and Relative Precisions Residential Lighting

Sample Size Unweighted Weighted Unweighted

Data Period n CF CF Rel Precision

December 282 0.263 0.281 ±6.5%

January 264 0.301 0.320 ±6.5%

Average Winter 546 0.286 0.298 ±4.5%

Winter On-Peak Hours 5PM - 7PM

Table i - 2: Winter On-Peak CFs and Relative Precisions Residential Lighting

The winter CFs as expected are higher than the summer CFs, ranging from 0.263 for December to 

0.320 for January, with the average winter CF for all lighting at 0.286 unweighted and 0.298 

weighted. The relative precision is better during the winter peak periods primarily because the 

CFs are higher and there is less variation in the data, i.e. the Coefficient of Variation (Cv) is 

lower. The relative precision of the average winter unweighted CF is ±4.5% at the 80% 

confidence interval and the December and January relative precisions are both better than ±10%

at the 80% confidence interval. 

The Seasonal Summer and Winter Peak performance hours were calculated using historical load 

data and the 50/50 Seasonal Peak Forecasts from the most recent Capacity Energy Loads and 

Transmission (CELT) report. The seasonal peak performance hours were weighted based upon 

the frequency distribution of the hours observed where the load met or exceeded 90% of the 

50/50 seasonal peak forecast, and these values were then used to calculate a weighted CF for each 

of the measure types. Table i - 3 and Table i - 4 provide the Summer Seasonal Peak and Winter 

Seasonal Peak CFs for all residential lighting. The CFs during the summer months range from a 

low of about 0.08 for June to a high of 0.10 for August, with an Average Summer CF of about

0.09. The relative precision during each of the summer months is within the range of ±10% at the 

80% confidence interval. The Winter Seasonal Peak CFs, as expected, are higher than the 

Summer Seasonal Peak CFs, ranging from 0.25 in December to 0.28 in January, with an Average 

Winter Seasonal Peak CF for all lighting at 0.26. 
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 4

_____________________________________________________________________________

March 25, 2007 Final REPORT

Sample Size Unweighted Calculated Calculated

Data Period n CF CV Rel Precision

June 210 0.075 2.275 ±6.3%

July 102 0.091 1.884 ±5.3%

August 189 0.104 1.747 ±5.2%

Average Summer 501 0.088 1.967 ±3.6%

Summer Seasonal Peak Hours (90% of 50/50 Peak)

Table i - 3: Summer Seasonal Peak CFs and Relative Precisions Residential Lighting

Sample Size Unweighted Calculated Calculated

Data Period n CF CV Rel Precision

December 282 0.249 1.23 ±4.5%

January 264 0.279 1.19 ±4.5%

Average Winter 546 0.264 1.21 ±3.2%

Winter Seasonal Peak Hours (90% of 50/50 peak)

Table i - 4: Winter Seasonal Peak CFs and Relative Precisions Residential Lighting

Table i - 5 and Table i - 6 presents a comparison of the CFs calculated for the On-Peak 

Performance hours and the Seasonal Peak Performance hours for both the summer and winter 

periods. The results show that the Summer Seasonal Peak CF increases over the Summer OnPeak for each month during the summer period and the Average Summer CF increases by 16% 

from 0.076 to 0.088. The increase is due to a wider range of hours being included in the weighted 

average calculation, among them more evening hours, when the CFs are higher. The reverse is 

true for the Winter Seasonal Peak CFs, which is lower than the Winter On-Peak CFs with the 

Average Winter CF decreasing by 8% from about 0.29 to 0.26. The decrease is due to a wider 

range of hours being included in the weighted average calculation, among them more morning 

and afternoon hours, when the CFs are lower. 

 

 On-Peak Seasonal % Change

Unweighted Unweighted Seasonal/

Data Period CF CF On-Peak

June 0.060 0.075 126%

July 0.081 0.091 112%

August 0.094 0.104 111%

Average Summer 0.076 0.088 116%

Table i - 5: Comparison of Summer On-Peak and Seasonal Peak CFs Residential Lighting 
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 5

_____________________________________________________________________________

March 25, 2007 Final REPORT

 On-Peak Seasonal % Change

Unweighted Unweighted Seasonal/

Data Period CF CF On-Peak

December 0.263 0.249 95%

January 0.301 0.279 93%

Average Winter 0.286 0.264 92%

Table i - 6: Comparison of Winter On-Peak and Seasonal Peak CFs Residential Lighting 

Commercial & Industrial Lighting Coincidence Factor Results

A similar Coincidence Factor analysis was also conducted for Commercial and Industrial 

Lighting and Occupancy Sensor measures. The logger data were analyzed by sector so that 

results could be applied to multiple programs with different participation rates among the 

different sectors. Table i - 7and Table i - 8 provide the On-Peak CFs for the ten C&I sectors 

along with the associated relative precisions and total estimated CFs based on a logger weighted 

strategy and weighting each sector equally. The Summer On-Peak CFs indicates that the Grocery 

sector has the highest CF of about 0.95, while the Other sector has the lowest CF of about 0.54. 

All of the sectors have relative precisions that are within ± 5% at the 80% confidence interval. 

The Grocery sector also had the highest Winter On-Peak CF of about 0.78, while the School 

sector had the lowest CF of about 0.34. Once again the relative precisions were all quite good 

with each sector exceeding ± 10% at the 80% confidence interval. As expected the Winter OnPeak CFs were lower than the Summer On-Peak CFs for all of the C&I lighting sectors, because 

the performance hours occur later in the day as C&I facilities are shutting down and lighting is 

being switched off. 
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 6

_____________________________________________________________________________

March 25, 2007 Final REPORT

Sample Size Calculated Logger Calculated Calculated

Sector Type n CF Weight CV Rel Precision

Grocery 37 0.948 0.026 0.179 ±1.9%

Manufacturing 169 0.729 0.119 0.488 ±2.4%

Medical (Hospital) 58 0.769 0.041 0.425 ±3.6%

Office 259 0.750 0.183 0.438 ±1.7%

Other 192 0.543 0.136 0.675 ±3.1%

Restaurant 43 0.811 0.030 0.347 ±3.4%

Retail 166 0.824 0.117 0.342 ±1.7%

University/College 70 0.680 0.049 0.483 ±3.7%

Warehouse 59 0.781 0.042 0.359 ±3.0%

School 362 0.633 0.256 0.503 ±1.7%

0.704 1.000

0.747

Summer On-Peak Hours 1PM - 5PM

Total Weighted by Logger

Total Equal Weight by Sector

Table i - 7: Summer On-Peak CFs and Relative Precision C&I Lighting

Sample Size Calculated Logger Calculated Calculated

Sector Type n CF Weight CV Rel Precision

Grocery 37 0.776 0.026 0.474 ±7.1%

Manufacturing 169 0.399 0.119 0.983 ±6.9%

Medical (Hospital) 58 0.603 0.041 0.593 ±7.1%

Office 259 0.537 0.183 0.725 ±4.1%

Other 192 0.426 0.136 0.804 ±5.3%

Restaurant 43 0.663 0.030 0.557 ±7.7%

Retail 166 0.655 0.117 0.592 ±4.2%

University/College 70 0.523 0.049 0.679 ±7.4%

Warehouse 59 0.496 0.042 0.787 ±9.3%

School 362 0.343 0.256 1.010 ±4.8%

0.480 1.000

0.542

Winter On-Peak Hours 5PM - 7PM

Total Weighted by Logger

Total Equal Weight by Sector

Table i - 8: Winter On-Peak CFs and Relative Precision C&I Lighting

Table i - 9and Table i - 10 provide the Summer and Winter Seasonal-Peak CFs for the ten C&I 

sectors along with the associated relative precisions and total estimated CFs based on a logger 

weighted strategy and weighting each sector equally (which is the simple average of the CFs across 

all sectors. The Seasonal Peak Performance Hours were determined by analysis of historic ISO-NE 

Load Data and Forecast Data to determine the frequency distribution for each hour where the 

demand was greater than or equal to 90% of the seasonal forecast. A simple probabilistic weighting 

scheme was applied based upon the number of observations during each hour as described in 
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 7

_____________________________________________________________________________

March 25, 2007 Final REPORT

section 3 of this report. The Summer Seasonal-Peak CFs indicates that the Grocery sector has the 

highest CF of about 0.90, while the Other sector has the lowest CF of about 0.48. All of the sectors 

have relative precisions that are within ± 5% at the 80% confidence interval during the Summer 

Seasonal Peak hours. The Grocery sector also had the highest Winter On-Peak CF of about 0.78, 

while the School sector had the lowest CF of about 0.34. Once again the relative precisions were all 

quite good, with each sector exceeding ± 10% at the 80% confidence interval. As expected, the 

Winter On-Peak CFs were lower than the Summer On-Peak CFs for all of the C&I lighting sectors, 

because the performance hours occur later in the day as C&I facilities are shutting down and 

lighting is being switched off. 

Sample Size Calculated Logger Calculated Calculated

Sector Type n CF Weight CV Rel Precision

Grocery 37 0.904 0.026 0.23 ±1.5%

Manufacturing 169 0.671 0.119 0.52 ±1.7%

Medical (Hospital) 58 0.740 0.041 0.45 ±2.5%

Office 259 0.702 0.183 0.48 ±1.2%

Other 192 0.476 0.136 0.75 ±3.0%

Restaurant 43 0.775 0.030 0.40 ±2.5%

Retail 166 0.795 0.117 0.38 ±1.2%

University/College 70 0.650 0.049 0.51 ±2.5%

Warehouse 59 0.727 0.042 0.41 ±2.2%

School 362 0.599 0.256 0.48 ±1.1%

0.660 1.000

Total Equal Weight by Sector 0.704

Summer Seasonal Peak Hours (90% of 50/50 Peak)

Total Weighted by Logger

Table i - 9: Summer Seasonal Peak CFs and Relative Precision C&I Lighting
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 8

_____________________________________________________________________________

March 25, 2007 Final REPORT

Sample Size Calculated Logger Calculated Calculated

Sector Type n CF Weight CV Rel Precision

Grocery 37 0.770 0.026 0.44 ±4.6%

Manufacturing 169 0.432 0.119 0.91 ±4.2%

Medical (Hospital) 58 0.618 0.041 0.58 ±4.5%

Office 259 0.539 0.183 0.71 ±2.6%

Other 192 0.428 0.136 0.80 ±4.4%

Restaurant 43 0.644 0.030 0.59 ±5.3%

Retail 166 0.647 0.117 0.59 ±2.7%

University/College 70 0.528 0.049 0.60 ±4.2%

Warehouse 59 0.535 0.042 0.70 ±5.6%

School 362 0.388 0.256 0.85 ±2.7%

0.497 1.000

0.553

Winter Seasonal Peak Hours (90% of 50/50 Peak)

Total Weighted by Logger

Total Equal Weight by Sector

Table i - 10: Winter Seasonal Peak CFs and Relative Precision C&I Lighting

Table i - 11 provides a comparison of the Summer On-Peak and Seasonal Peak CFs for each of the 

C&I sectors, which shows that for every sector the Summer Seasonal CFs are lower than the

Summer On-Peak CFs. This means that if the C&I lighting were classified as Summer Seasonal 

Peak assets the demand reductions would be lower. 

% Change

On-Peak Seasonal Seasonal /

Sector Type CF CF On-Peak

Grocery 0.948 0.904 95%

Manufacturing 0.729 0.671 92%

Medical (Hospital) 0.769 0.740 96%

Office 0.750 0.702 94%

Other 0.543 0.476 88%

Restaurant 0.811 0.775 96%

Retail 0.824 0.795 96%

University/College 0.680 0.650 96%

Warehouse 0.781 0.727 93%

School 0.633 0.599 95%

Total Weighted by Logger 0.704 0.660 94%

Total Equal Weight by Sector 0.747 0.704 94%

Summer

Table i - 11: Comparison of Summer On-Peak and Seasonal Peak CFs C&I Lighting

Table i - 12 provides a similar comparison of the Winter On-Peak and Seasonal Peak CFs for each 

of the C&I Lighting sectors. In this case the results are mixed, with 7 of the 10 sectors showing an 
NECPUC New England State Working Group

2007 Coincidence Factor Study_________________________________________ Page 9

_____________________________________________________________________________

March 25, 2007 Final REPORT

increase in the Winter Seasonal Peak CFs compared to the Winter On-Peak CF. This seems to 

indicate that in general for the winter, C&I lighting would have more demand reduction if classified 

as a Seasonal Peak asset. 

% Change

On-Peak Seasonal Seasonal /

Sector Type CF CF On-Peak

Grocery 0.776 0.770 99%

Manufacturing 0.399 0.432 108%

Medical (Hospital) 0.603 0.618 103%

Office 0.537 0.539 101%

Other 0.426 0.428 100%

Restaurant 0.663 0.644 97%

Retail 0.655 0.647 99%

University/College 0.523 0.528 101%

Warehouse 0.496 0.535 108%

School 0.343 0.388 113%

Total Weighted by Logger 0.480 0.497 104%

Total Equal Weight by Sector 0.542 0.553 102%

Winter

Table i - 12: Comparison of Winter On-Peak and Seasonal Peak CFs C&I Lighting

Commercial & Industrial Occupancy Sensor Coincidence Factor Results

Table i - 13 and Table i - 14 present the Summer On-Peak and Winter On-Peak CFs for occupancy 

sensors for seven of the ten C&I sectors as well as the total CFs for all seven sectors on a logger 

weighted basis and by weighting each sector equally. During the Summer On-Peak Period the 

occupancy sensors installed in the University/College sector had the highest CF of about 0.30, while 

the Other sector had the lowest CF of about 0.02. The Summer On-Peak CF for the remaining 

sectors ranged from about 0.21 for Manufacturing to 0.27 for the Office Sector. During the Winter 

On-Peak the Office sector had the highest CF of about 0.31 and the Other sector had the lowest CF 

of 0.09. The CFs for the remaining sectors ranged from a low of about 0.17 for the Warehouse 

sector to a high of about 0.23 for the University/College sector. The relative precision for all of the

CFs were estimated by calculating the relative precision of the occupancy sensors profiles, since 

only aggregate savings profiles were developed for the analysis. In this case we would recommend 

using the logger weighted Total CFs since the relative precision for individual sector results are not 

that good particularly during the Winter period. 
NECPUC New England State Working Group

2007 Coincidence Factor Study________________________________________ Page 10

_____________________________________________________________________________

March 25, 2007 Final REPORT

Sample Size Calculated Logger Estimated Estimated

Data Period n CF Weight CV Rel Precision

Manufacturing 12 0.210 0.035 0.688 ±12.7%

Medical 59 0.234 0.170 0.602 ±5.0%

Office 69 0.270 0.199 0.559 ±4.3%

Other 56 0.017 0.161 0.793 ±6.8%

University/College 16 0.304 0.046 0.678 ±10.9%

Warehouse 77 0.266 0.222 0.646 ±4.7%

School 58 0.239 0.167 0.828 ±7.0%

0.217 1.000

0.154

Total Weighted by Logger

Total Equal Weight by Sector

Summer On-Peak Hours 1PM - 5PM

Table i - 13: Summer On-Peak CFs and Relative Precision C&I Occupancy Sensors

Sample Size Calculated Logger Estimated Estimated

Sector Type n CF Weight CV Rel Precision

Manufacturing 12 0.190 0.035 1.301 ±34.1%

Medical 59 0.213 0.170 0.840 ±9.9%

Office 69 0.309 0.199 1.087 ±11.9%

Other 56 0.089 0.161 1.053 ±12.8%

University/College 16 0.233 0.046 0.827 ±18.8%

Warehouse 77 0.175 0.222 1.082 ±11.2%

School 58 0.173 0.167 1.527 ±18.2%

0.197 1.000

0.138

Total Weighted by Logger

Total Equal Weight by Sector

Winter On-Peak Hours 5PM - 7PM

Table i - 14: Winter On-Peak CFs and Relative Precision C&I Occupancy Sensors

Table i - 15 and Table i - 16 provide the Summer Seasonal Peak and Winter Seasonal Peak CFs 

for the occupancy sensors for seven of the ten C&I sectors. Once again, during the Summer

Seasonal Peak hours the University/College sector occupancy sensors had the highest CF of about

0.28 and the Other sector had the lowest CF of about 0.02. The CFs for the remaining sectors 

ranged from about 0.20 to 0.27. The Winter Seasonal Peak CFs were similar to the Winter OnPeak results with the Office sector having the highest CF of about 0.30 and the Other sector 

having the lowest CF of about 0.07. Once again the relative precision of the CFs was estimated 

by using the occupancy sensor profiles and the results from the Seasonal periods are better than 

for the On-Peak periods because the results were taken across more hours. The Summer Seasonal 

Peak estimated relative precisions for the sectors are all within ±10% at the 80% confidence 
NECPUC New England State Working Group

2007 Coincidence Factor Study________________________________________ Page 11

_____________________________________________________________________________

March 25, 2007 Final REPORT

interval, and Winter estimated relative precisions are also within that range for most of the 

sectors.

Sample Size Calculated Logger Estimated Estimated

Data Period n CF Weight CV Rel Precision

Manufacturing 12 0.198 0.035 0.712 ±8.9%

Medical 59 0.239 0.170 0.649 ±3.6%

Office 69 0.274 0.199 0.606 ±3.2%

Other 56 0.024 0.161 0.808 ±4.6%

University/College 16 0.283 0.046 0.720 ±7.6%

Warehouse 77 0.246 0.222 0.700 ±3.3%

School 58 0.209 0.167 0.739 ±4.2%

0.208 1.000

0.147

Total Weighted by Logger

Total Equal Weight by Sector

Summer Seasonal Peak Hours (90% of 50/50 Peak)

Table i - 15: Summer Seasonal-Peak CFs and Relative Precision C&I Occupancy Sensors 

Sample Size Calculated Logger Estimated Estimated

Data Period n CF Weight CV Rel Precision

Manufacturing 12 0.172 0.035 1.063 ±17.3%

Medical 59 0.221 0.170 0.827 ±6.3%

Office 69 0.296 0.199 0.966 ±6.9%

Other 56 0.066 0.161 0.990 ±7.7%

University/College 16 0.231 0.046 0.819 ±11.9%

Warehouse 77 0.183 0.222 0.986 ±6.6%

School 58 0.159 0.167 1.140 ±8.7%

0.191 1.000

0.133

Total Weighted by Logger

Total Equal Weight by Sector

Winter Seasonal Peak Hours (90% of 50/50 Peak)

Table i - 16: Winter Seasonal-Peak CFs and Relative Precision C&I Occupancy Sensors

Table i - 17 and Table i - 18 provide a comparison of the Summer and Winter On-Peak and 

Seasonal Peak CFs for occupancy sensors for seven C&I sectors as well as the totals for all seven 

sectors calculated on a logger weighted and sector weighted basis. The results for the Summer 

period show that the Summer Seasonal CFs are lower than the On-Peak CFs for four of the seven 

sectors and for the total CF using both calculation methods. The results for the Winter period are 

similar, with five of the sectors having lower Seasonal Peak CFs and lower Total CFs using both 

calculation methods. Classifying the occupancy sensors as Seasonal Peak assets would result in a 

slight reduction in demand savings during both periods.
NECPUC New England State Working Group

2007 Coincidence Factor Study________________________________________ Page 12

_____________________________________________________________________________

March 25, 2007 Final REPORT

% Change

On-Peak Seasonal Seasonal /

Sector Type CF CF On-Peak

Manufacturing 0.210 0.198 94%

Medical 0.234 0.239 102%

Office 0.270 0.274 101%

Other 0.017 0.024 144%

University/College 0.304 0.283 93%

Warehouse 0.266 0.246 92%

School 0.239 0.209 87%

Total Weighted by Logger 0.217 0.208 96%

Total Equal Weight by Sector 0.154 0.147 96%

Summer

Table i - 17: Comparison of Summer On-Peak and Seasonal Peak CFs Occupancy Sensors

% Change

On-Peak Seasonal Seasonal /

Sector Type CF CF On-Peak

Manufacturing 0.190 0.172 90%

Medical 0.213 0.221 104%

Office 0.309 0.296 96%

Other 0.089 0.066 75%

University/College 0.233 0.231 99%

Warehouse 0.175 0.183 105%

School 0.173 0.159 92%

Total Weighted by Logger 0.197 0.191 97%

Total Equal Weight by Sector 0.138 0.133 96%

Winter

Table i - 18: Comparison of Winter On-Peak and Seasonal Peak CFs Occupancy Sensors
FINAL REPORT - UPDATE 

MULTIPLE SMALL BUSINESS 

SERVICES PROGRAMS

IMPACT EVALUATION 

2007

Submitted To:

Cape Light Compact

National Grid

NSTAR

Unitil

Western Massachusetts Electric Company
Note: This September 2, 2008 update to the final report includes complete data for WMECO. 

Submitted by: 

Summit Blue Consulting, LLC 

1722 14th Street, Ste. 230 

Boulder, CO 80302 

720.564.1130 

Prepared by: 

Michael Ozog 

970.407.1048 

mozog@summitblue.com

Mary Klos 

608.807.0083 

mklos@summitblue.com
Summit Blue Consulting, LLC 1 

TABLE OF CONTENTS 

1 Executive Summary............................................................................................ 2

2 Introduction ....................................................................................................... 4

3 Study Methodology............................................................................................. 5

3.1 Program Tracking Data.................................................................................... 5

3.2 Merging with Billing Data ................................................................................. 6

3.3 Savings Ratios................................................................................................. 8

3.4 Billing Analysis ................................................................................................ 9

3.5 Applying These Results................................................................................... 10

4 Study Results..................................................................................................... 11

4.1 Cape Light Compact ....................................................................................... 13

4.2 National Grid.................................................................................................. 16

4.3 NSTAR........................................................................................................... 19

4.4 Unitil ............................................................................................................. 22

4.5 Western Massachusetts Electric Company ........................................................ 25

5 Reliability of Estimates...................................................................................... 28

5.1 Precision........................................................................................................ 28

5.2 Bias Issues..................................................................................................... 29

Appendix A – Data Cleaning Steps................................................................................ 1

Appendix B—Complete Regression Results .................................................................. 4
Summit Blue Consulting, LLC 2 

1 EXECUTIVE SUMMARY

The five sponsoring organizations of this study, the Cape Light Compact, National Grid, NSTAR, Unitil 

(Fitchburg Gas and Electric Light Company d/b/a Unitil), and Western Massachusetts Electric Company 

(WMECO), have offered Small Business Services (SBS) programs throughout Massachusetts to small 

business energy consumers for several years. A large share of the electric energy savings from the SBS 

program comes from prescriptive lighting measures. 

Impact evaluations have been conducted in previous years to obtain estimates of realization rates for the 

gross energy savings resulting from the prescriptive lighting measures that are part of the SBS program. 

These realization rates reflect a comparison of estimated savings from the program tracking systems to 

actual customer billing data to verify the gross energy savings that were achieved. The purpose of this 

study is to estimate similar realization rates for 2007 program participants for each individual 

participating sponsor. 

The statistical model that was developed to estimate the savings from the lighting measures installed 

through SBS was framed within the Statistically Adjusted Engineering (SAE) approach. Under this 

approach, the engineering estimate of savings is included as an explanatory variable in a regression 

equation with the billed electricity consumption as the dependent variable. The estimated coefficient on 

the engineering estimate of savings may be interpreted as the realization rate. That is, the coefficient 

indicates the percentage of the engineering estimate of energy savings that is realized on average 

according to the analysis of billing records. 

Table 1-1 presents the results of this statistical modeling effort for each sponsor. 

Table 1-1. Summary of Lighting Savings Realization Rates by Sponsor 

Realization Rate T-value1 

Statistically 

Significant at the 

90% Confidence 

Level? 

Cape Light Compact 1.04 8.58 Yes 

National Grid 1.00 22.38 Yes 

NSTAR 0.89 23.37 Yes 

Unitil 1.02 12.39 Yes 

Western Massachusetts Electric 1.03 5.7 Yes 

The realization rates varied from a low of 0.89 to a high of 1.04 with most of them near the value of one. 

This indicates there is a good correspondence between initial estimates of gross energy savings and 

 

1

 The T-value is equal to the estimated realization rate divided by its standard error. It can be used directly to test 

the hypothesis that the realization rate is equal to zero. A T-value of 1.645 indicates there is 90% confidence that 

the realization rate is statistically significant and is not zero. Higher T-values indicate higher confidence levels. 
Summit Blue Consulting, LLC 3 

verified gross energy savings for prescriptive lighting measures in the SBS program. All of the 

realization rates are statistically significant at the 90% confidence level. 

Table 1-2 shows the precision rates that were achieved for each sponsor. The estimates for National Grid 

and NSTAR have a precision rate of plus or minus 7%. Sponsors with smaller numbers of customers had 

precision levels in the 10% to 30% range at the 90% confidence level. This wider precision range reflects 

the lower certainty of the coefficient estimates given greater variability and fewer observations. 

Table 1-2. Confidence Intervals and Precision Levels for Realization Rates2

Sponsor 

Expected 

Value of 

Realization 

Rate 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Cape Light Compact 1.04 ± 19% 0.84 1.25 

National Grid 1.00 ± 7% 0.92 1.07 

NSTAR 0.89 ± 7% 0.82 0.95 

Unitil 1.02 ± 13% 0.88 1.16 

WMECO 1.03 ± 29% 0.73 1.33 

The realization rates from this study should be applied to gross energy savings estimates from engineering 

calculations or deemed savings to create verified gross energy savings estimates. Additional estimates of 

free-ridership and/or spillover effects would need to come from other studies and be applied to the 

verified gross energy savings to estimate net energy savings. The development of net energy savings 

estimates is beyond the scope of this study. 

 

2

 All results shown in this table are calculated using realization rates and T-values with six decimal points. After the 

calculations, the results are rounded to two decimal points for reporting purposes. This rounding method was used 

for all similar tables in this report. 
Summit Blue Consulting, LLC 4 

2 INTRODUCTION

The five sponsoring organizations of this study, the Cape Light Compact, National Grid, NSTAR, Unitil 

(Fitchburg Gas and Electric Light Company d/b/a Unitil), and Western Massachusetts Electric Company 

(WMECO), have offered Small Business Services (SBS) programs throughout Massachusetts to small 

business energy consumers for several years. A large share of the electric energy savings from the SBS 

program comes from prescriptive lighting measures. 

Impact evaluations have been conducted in previous years to obtain estimates of realization rates for the 

gross energy savings resulting from the prescriptive lighting measures that are part of the SBS program. 

These realization rates reflect a comparison of estimated savings from the program tracking systems to 

actual customer billing data to verify the gross energy savings that were achieved. The purpose of this 

study is to estimate similar realization rates for 2007 program participants for each individual 

participating sponsor. 
Summit Blue Consulting, LLC 5 

3 STUDY METHODOLOGY

This chapter presents the approach used in the billing data analysis to estimate realization rates for the 

SBS programs. It describes the data that were collected, steps taken to prepare the data and the analytical 

methods that were used to perform the billing analysis. 

3.1 Program Tracking Data 

Program tracking data were used to identify program participants who installed lighting measures through 

the small business energy efficiency programs offered by the sponsors. These tracking data provided sitespecific engineering estimates for the lighting measure savings for each participant. 

Program tracking data covered different years for each sponsor. Table 3-1 shows the initial installation 

year for lighting measures for each participant that had tracking data and matching billing data evaluated 

in this study. 

Table 3-1. Years Covered in Program Tracking Data for Evaluation 

2006 Participants 

Evaluated 

2007 Participants 

Evaluated Total 

Cape Light Compact 100 159 259 

National Grid 97 1,200 1,297 

NSTAR 106 680 786 

Unitil 0 22 22 

Western Massachusetts Electric 68 30 98 

TOTAL 371 2,091 2,462 

The program tracking data that were used in the billing data analysis included the following: 

• Account or Location ID 

• Customer name and town 

• Description of measures installed (both lighting and other measures) 

• KWh savings from installed measures 

• Installation date 

In constructing the participation variable used in the billing analysis, a zero was used for all months prior 

to the installation date. After that date, the participation variable was set to the engineering estimate of 

kWh savings. Since single participants may have installed multiple measures that each had different 

installation dates, the engineering (tracking system) estimates of savings for these additional measures 

were added to the participation variable after the subsequent installation dates. 
Summit Blue Consulting, LLC 6 

3.2 Merging with Billing Data 

The next step in this process was to merge the program tracking data with the billing systems customer 

information and billing databases from the sponsors. The critical step in this process was identifying the 

correct billing accounts to be matched to the tracking system customer data. In some cases, the customer 

may have more than one billing account that reflects the savings identified in the tracking system. These 

multiple accounts, or meters, for the same customer must be identified and added together so the total 

savings from the measures will be reflected in the consumption data. 

A location key was created to accomplish this purpose. Each unique combination of sponsor – customer 

name – address was assigned a unique location key in the billing data. In this way, if a customer had two 

accounts at the same address those accounts would be combined. Tracking data was then matched by 

account number to the correct location key in the billing data.3

Table 3-2 summarizes the disposition of all data received from the five sponsors for this study. Billing 

records were received for 3,186 accounts. When service addresses were examined, these accounts could 

be combined into 3,090 unique customer locations. 

Table 3-2. Summary of Data Disposition for All Sponsors 

 Locations Accounts 

Billing Records Received 3,090 3,186 

Savings Records Received 2,976 

Billing Records with no Savings Data 485 493 

Savings Records with no Billing Data 283 

First Round Match of Billing and Savings Data 2,633 2,693 

No Lighting Measures 117 

Mismatched Periods for Billing and Savings Data 28 

No Billing Data for Period before start of Savings 26 

TOTAL MATCHED BILLING AND SAVINGS DATA 2,462 

Savings records were received for 2,976 accounts. When these savings records were matched to the 

billing data, matches were found for 2,693 of the accounts. These accounts covered 2,633 unique 

customer locations. This matching process left 493 accounts in the billing data that had no matching 

savings records, and 283 accounts in the savings data that had no matching billing records. 

This level of unmatched data is to be expected in this type of longitudinal study. A customer with a 

particular account number may have participated in the savings program at some point over the period of 

study, and then they may have moved to a new location or closed their business. If they moved, the new 

 

3

 The LocID (Location ID), which identifies the location/premise, was used for matching NATIONAL GRID 

tracking system information to their billing data. The account number was used for this purpose for all other 

sponsors. 
Summit Blue Consulting, LLC 7 

occupant would have a new account number that would no longer match the account number in the 

tracking system. The new occupant may have completely different energy usage patterns, so it is not 

possible to use the billing data for the new occupant to estimate savings. It is often very difficult to pickup old billing data for accounts that are no longer active. This explains why some savings records do not 

have matching billing data. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All matched data was analyzed 

by location. Since this study is focusing on savings for lighting measures, 117 locations that did not 

receive any lighting measures were excluded. An additional 28 locations were excluded because all of 

the available billing data was for the period before the savings measures were installed. Essentially, there 

was no post-participation data. And, finally, 26 locations were dropped because they did not have any 

pre-participation billing data. This left a total of 2,462 locations with matched billing and savings data. 

This matched billing and savings data were the core data used for the billing analysis for each sponsor. In 

order not to bias the analysis, from this point on customers were not automatically eliminated from the 

analysis. Instead, manual review on a case-by-case basis was done within each sponsor’s model before 

any decisions were made to exclude additional data.

At this point it was possible to look at some of the attributes of the matched dataset. Table 3-3 

summarizes the completeness of the monthly billing data and customer’s installation of other measures in 

addition to lighting measures. 

Table 3-3. Attributes of Matched Billing and Savings Data 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 2,462 100% 

 

Incomplete Billing Data for Jan 2006 through Dec 2007 894 36% 

Complete Billing Data for Jan 2006 through Dec 2007 1,568 64% 

 

Only Lighting Measure Savings 2,318 94% 

Lighting and Other Measure Savings 144 6% 

A majority of the customer locations, 64%, had a complete set of monthly billing data for the study 

period. That means they had twenty-four monthly observations with matched billing and savings data. 

As explained in section 3.4, the methodology for this study utilizes monthly observations rather than 

annual observations. This preserves the ability to include and make use of customer locations that do not 

have a complete set of twenty-four months of data, which reduces the potential bias that may occur if they 

all had to be deleted. Some incomplete sets may be missing only one month of data, and others may be 

missing over twenty months. During the modeling step for each sponsor, each case is reviewed manually 

and only those that are serious outliers are removed from the analysis. 

While the emphasis of this billing analysis is on the verification of gross energy savings from lighting 

measures, some customer locations may install both lighting and other types of energy saving measures. 

Since the billing data reflects all of their energy use and subsequent energy savings, in order to get a good 

fit on the modeling of energy savings all measures installed by the customer location must be identified 

and included in the model. The large majority of customer locations, 94%, installed only lighting 
Summit Blue Consulting, LLC 8 

measures so the impact of the correct specification of other measure savings is not expected to have any 

significant impact on the estimate of the realization rate for lighting measures. 

Technical detail on the specific steps that were taken in the data cleaning process, along with a description 

of the format of the final merged dataset, can be found in Appendix A. 

3.3 Savings Ratios 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. If a customer has multiple accounts/meters at the 

same location, they have been combined for this ratio. 

Table 3-4 presents the summary of the savings ratios calculated for each customer location. Over 80% of 

the customer locations show savings ratios less than 40%, with the remainder spread out over higher 

savings ratios. This distribution is typical for this type of data. 

Table 3-4. Summary of Savings Ratios 

Savings Ratio Locations Percent 

0% to 9% 902 37% 

10% to 19% 583 24% 

20% to 29% 324 13% 

30% to 39% 220 9% 

40% to 49% 127 5% 

50% to 59% 75 3% 

60% to 69% 60 2% 

70% to 79% 36 1% 

80% to 89% 19 1% 

90% to 99% 25 1% 

100% or more 91 4% 

TOTAL 2,462 100% 

As appealing as it may be, it is not possible to create an exact cut-off between savings ratios that are 

‘reasonable’ and those that are not since so many factors can influence the savings ratios for a particular 

customer location. Those factors include the number and type of lighting measures that were installed, 

the percentage of the total lighting load that received measures, the percentage of the overall energy use 

that is used for lighting, the presence of other savings measures besides lighting, and potential changes in 

the metering arrangement or customer occupancy and use patterns during the entire period of study. 

Sponsors with larger numbers of customer locations showing savings ratios greater than 100% 

investigated some of the individual cases and found the typical explanations of customer changes, 

misassigned account numbers, etc. An investigation into each case is very time-consuming since each 

case usually has its own unique story. Since the total number of cases with a savings ratio greater than 

100% is very small, it was determined that they would not have a significant impact on the estimation of 
Summit Blue Consulting, LLC 9 

the overall realization rates. In keeping with the philosophy of minimizing bias by not using arbitrary cutoffs to throw out potentially usable cases, all cases were kept in the initial modeling efforts for each 

sponsor. Manual inspection of individual cases was then used to identify outliers that should be excluded 

from each model. 

3.4 Billing Analysis 

The statistical model that was developed to estimate the savings from the lighting measures installed 

through SBS was framed within the Statistically Adjusted Engineering (SAE) approach. Under this 

approach, the engineering (the program tracking system’s) estimate of savings is included as an 

explanatory variable in a regression equation with the billed electricity consumption as the dependent 

variable. The estimated coefficient on the engineering estimate of savings is interpreted as the realization 

rate. That is, the coefficient indicates the percentage of the engineering estimate of energy savings that is 

realized on average according to the analysis of billing records. 

One assumption made in using billing analysis is that the data in the program tracking systems, which is 

the source of the engineering data used in the study, accurately reflects the individual small business 

project installations. Any errors in the tracking system data are inherently included in this analysis and 

are embedded in the realization rates. 

For this analysis, data are available both across customers (i.e., cross-sectional) and over time (i.e., timeseries). With this type of data, known as “panel” data, it becomes possible to control at once for 

differences across facilities as well as differences across periods in time through the use of a “fixedeffects” panel model. The fixed-effect refers to the assumption that differences across customers can be 

explained in large part by customer-specific intercept terms, as discussed below. 

Because the consumption data in the panel model includes months before and after the installation of 

measures through the program, the period of program participation (or the participation window) may be 

defined specifically for each customer. This feature of the panel model allows for the pre-installation 

months of consumption to act as controls for post-participation months. In addition, this model, unlike 

annual pre/post-participation models such as annual change models, does not require a full year of postparticipation data. 

The fixed effects model can be viewed as a type of differencing model in which all characteristics of the 

customer, which (1) are independent of time and (2) determine the level of energy consumption, are 

captured within the customer-specific constant terms. In other words, differences in customer 

characteristics that cause variation in the level of energy consumption, such as building size and structure, 

are captured by constant terms representing each unique customer facility. 

Algebraically, the fixed-effect panel data model is described as: 

 yit = αi + βxit +RR·Eit+ εit

where: 

yit = Energy consumption for site i during month t

αi = Constant term for site i

β = Vector of coefficients 
Summit Blue Consulting, LLC 10 

x = Vector of variables that represent factors causing changes in monthly 

consumption (i.e., the time-effects variables such as weather) 

RR = The estimated coefficient that represents the realization rate 

Eit = The engineering estimate of savings for site i during month t

ε = Error term 

In practice, rather than estimating a unique intercept term for each customer, an equivalent approach is 

employed which expresses both the dependent and independent variables in terms of deviations from the 

time-series means for each customer. The resulting estimated coefficients from this "deviation from the 

mean" approach are equal to the coefficients found by having facility-specific intercept terms. 

That is, it can be shown that: 

ii i α = − y x β

This implies that the customer-specific intercept term captures the difference between the average energy 

use for that customer and the predicted average energy use (from the model) during the time period used 

in the model. Therefore, the fixed-effects model explains the month to month deviation in energy use 

rather than the level of energy use. 

It is possible to apply the fixed effects model to the pooled data from all of the sponsors to create a single 

estimate of the realization rate, or it can be applied to data for each individual sponsor to estimate a 

unique realization rate for each sponsor’s program. Both applications of the model were tested to 

determine the best fit to the data. 

3.5 Applying These Results 

The methodology of this study was designed to develop realization rates for gross energy savings. The 

gross energy savings reported in the various tracking systems are compared to billing usage to estimate 

the extent to which the predicted level of gross energy savings actually occurred. The customer’s electric 

bill will reflect the reduction in energy usage that occurred, but it does not provide any insight into the 

level of free-ridership associated with that reduction in energy use and cannot reflect what the customer 

would have done in the absence of the program. The realization rate only reflects what actually happened 

and can be measured. 

The realization rates from this study should be applied to gross energy savings estimates from engineering 

calculations or deemed savings to create verified gross energy savings estimates. Additional estimates of 

free-ridership and/or spillover effects would need to come from other studies and be applied to the 

verified gross energy savings to estimate net energy savings. The development of net energy savings 

estimates is beyond the scope of this study. 
Summit Blue Consulting, LLC 11 

4 STUDY RESULTS

This section of the report presents the results from the study. First, there is a discussion of modeling 

specifications that worked well for all sponsors, and a presentation of the overall realization rate results. 

After this discussion there are sections for each individual sponsor to show the results of the data cleaning 

effort and the SAE model developed specifically for them. 

The general model specification focused on the relationship between total energy consumption and 

expected savings, with a few additional explanatory variables. The explanatory variables focused on 

modeling the effects of weather, seasonal variation and economic growth on total energy consumption. 

Different variable specifications were tried to find the best representation of these effects. 

Several weather variables were collected and tested. These included temperature, humidity, heating 

degree days and cooling degree days. Temperature proved to work the best in explaining energy use 

variations due to weather. 

Both monthly and seasonal dummy variables were created to represent seasonal variation. In general, the 

seasonal indicators performed better than the monthly indicators, probably because the study period time 

series only covered two years of data. The seasonal indicators were particularly helpful for modeling the 

fall and winter seasons, although this varied by sponsor. 

Economic growth was best represented by an annual dummy variable that allowed for an increase in 

energy use from one year to the next. Again, the simple specification of this variable probably worked 

best due to the relatively short time period covered in the data for this study. 

The first modeling attempt specified a single fixed effects model for the entire group of sponsors. This 

pooled sponsor model did not perform well. Many of the resultant coefficients were not statistically 

significant. This result is not surprising since there is no reason to expect all realization rates to be 

identical for the independent programs. 

There are many factors which could create differences in the realization rates for each utility. For 

example, each sponsor independently tracks their own SBS program and develops their own energy 

savings estimates. This could create differences. They may have different implementation vendors. 

There may be differences in the firmographics of the customers participating in each program. Weather 

response for each area may vary. 

Given that the pooled data model did not perform well and there was no overriding reason to expect the 

same realization rate for everyone, it was decided that it would be best to develop individual models for 

each sponsor. Those individual models are discussed in the remainder of this section. 
Summit Blue Consulting, LLC 12 

Table 4-1 summarizes the realization rates that were estimated from each of the individual models. 
Summit Blue Consulting, LLC 13 

Table 4-1. Summary of Lighting Savings Realization Rates by Sponsor 

Realization Rate T-value 

Statistically 

Significant at the 

90% Confidence 

Level? 

Cape Light Compact 1.04 8.58 Yes 

National Grid 1.00 22.38 Yes 

NSTAR 0.89 23.37 Yes 

Unitil 1.02 12.39 Yes 

Western Massachusetts Electric 1.03 5.7 Yes 

 
Summit Blue Consulting, LLC 14 

4.1 Cape Light Compact 

Table 4-2 summarizes the disposition of all data received from Cape Light Compact for this study. 

Billing records were received for 390 accounts. When service addresses were examined, these accounts 

could be combined into 343 unique customer locations. 

Table 4-2. Summary of Data Disposition for Cape Light Compact 

 Locations Accounts 

Billing Records Received 343 390 

Savings Records Received 338 

Billing Records with no Savings Data 71 71 

Savings Records with no Billing Data 19 

First Round Match of Billing and Savings Data 272 319 

No Lighting Measures 5 

Mismatched Periods for Billing and Savings Data 8 

No Billing Data for Period before start of Savings 0 

TOTAL MATCHED BILLING AND SAVINGS DATA 259 

Savings records were received for 338 accounts. When these savings records were matched to the billing 

data, matches were found for 319 of the accounts. These accounts covered 272 unique customer 

locations. This matching process left 71 accounts in the billing data that had no matching savings 

records, and 19 accounts in the savings data that had no matching billing records. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All matched data was analyzed 

by location. Since this study is focusing on savings for lighting measures, 5 locations that did not receive 

any lighting measures were excluded. An additional 8 locations were excluded because all of the 

available billing data was for the period before the savings measures were installed. Essentially, there 

was no post-participation data. This left a total of 259 locations with matched billing and savings data. 

Table 4-3 summarizes the completeness of the monthly billing data and customer’s installation of other 

measures in addition to lighting measures. 

Table 4-3. Attributes of Matched Billing and Savings Data for CLC 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 259 100% 

 

Incomplete Billing Data for Jan 2006 through Dec 2007 33 13% 

Complete Billing Data for Jan 2006 through Dec 2007 226 87% 
Summit Blue Consulting, LLC 15 

 

Only Lighting Measure Savings 229 88% 

Lighting and Other Measure Savings 30 12% 

A majority of the customer locations, 87%, had a complete set of monthly billing data for the study 

period. That means they had twenty-four monthly observations with matched billing and savings data. 

A similar majority of customer locations, 88%, installed only lighting measures so the impact of the 

correct specification of other measure savings should not have a significant impact on the estimate of the 

realization rate for lighting measures for CLC. 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. If a customer has multiple accounts/meters at the 

same location, they have been combined for this ratio. 

Table 4-4 presents the summary of the savings ratios calculated for each customer location. Eighty 

percent of the customer locations show savings ratios less than 40%, with the remainder spread out over 

higher savings ratios. This distribution is typical for this type of data. 

Table 4-4. Summary of Savings Ratios for Cape Light Compact 

Savings Ratio Locations Percent 

0% to 9% 88 34% 

10% to 19% 50 19% 

20% to 29% 39 15% 

30% to 39% 31 12% 

40% to 49% 19 7% 

50% to 59% 9 4% 

60% to 69% 6 2% 

70% to 79% 2 1% 

80% to 89% 0 0% 

90% to 99% 5 2% 

100% or more 10 4% 

TOTAL 259 100% 

The initial modeling process began using all of this data. During the modeling process, a few customers 

were identified for exclusion because their consumption was very low or very spotty. 

A problem with autocorrelation of the data was also identified. Autocorrelation means that there is a 

correlation in the error terms. The error terms are the difference between the actual observed energy use 

and the energy use predictions from the model for each month. In the perfect model, these error terms are 

completely random and have no correlation to each other. In the real world, however, autocorrelation 

often occurs when evaluating time-series data because external conditions and/or customer behavior 
Summit Blue Consulting, LLC 16 

during one time period frequently has some carryover from the preceding time period. Things that affect 

usage this month may be similar to what affected usage last month. It is also possible that one month’s 

usage may be directly related to the prior month’s usage. For example, over production in one month 

may be followed by reduced production in the following month. There are many possible causes of 

autocorrelation in time-series data.4

Autocorrelation is more likely to cause model misspecification in small samples than in large samples. It 

was determined that the model for Cape Light Compact would be improved by correcting for 

autocorrelation. 

After making the correction, the final model result was a realization rate of 1.04 with a t-value of 8.58. 

This result is statistically significant at the 90% confidence level. These results are summarized in Table 

4-5. 

Table 4-5. Lighting Savings Realization Rate for Cape Light Compact 

Realization 

Rate T-value 

Statistically 

Significant at 

the 90% 

Confidence 

Level? 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

1.04 8.58 Yes ± 19% 0.84 1.25 

The exact specification and results of the final fixed effects model for Cape Light Compact can be found 

in Appendix B. 

 

4

 For a discussion of possible causes of autocorrelation, see pp. 139-140 in A Guide to Econometrics, Fifth Edition, 

Peter Kennedy, The MIT Press, 2003. 
Summit Blue Consulting, LLC 17 

4.2 National Grid 

Table 4-6 summarizes the disposition of all data received from National Grid for this study. Billing 

records were received for 1,846 accounts. When service addresses were examined, these accounts could 

be combined into 1,799 unique customer locations. 

Table 4-6. Summary of Data Disposition for National Grid 

 Locations Accounts 

Billing Records Received 1799 1846 

Savings Records Received 1438 

Billing Records with no Savings Data 403 411 

Savings Records with no Billing Data 3 

First Round Match of Billing and Savings Data 1424 1435 

No Lighting Measures 110 

Mismatched Periods for Billing and Savings Data 17 

No Billing Data for Period before start of Savings 1 

TOTAL MATCHED BILLING AND SAVINGS DATA 1297 

Savings records were received for 1,438 accounts. When these savings records were matched to the 

billing data, matches were found for 1,435 of the accounts. These accounts covered 1,424 unique 

customer locations. This matching process left 411 accounts in the billing data that had no matching 

savings records, and 3 accounts in the savings data that had no matching billing records. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All matched data was analyzed 

by location. Since this study is focusing on savings for lighting measures, 110 locations that did not 

receive any lighting measures were excluded. An additional 17 locations were excluded because all of 

the available billing data was for the period before the savings measures were installed. Essentially, there 

was no post-participation data. And, finally, 1 location was dropped because they did not have any preparticipation billing data. This left a total of 1,297 locations with matched billing and savings data. 

At this point it was possible to look at some of the attributes of the matched dataset for National Grid. 

Table 4-7 summarizes the completeness of the monthly billing data and customer’s installation of other 

measures in addition to lighting measures. 

Table 4-7. Attributes of Matched Billing and Savings Data for National Grid 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 1,297 100% 

 

Incomplete Billing Data for Jan 2006 through Dec 2007 418 32% 

Complete Billing Data for Jan 2006 through Dec 2007 879 68% 
Summit Blue Consulting, LLC 18 

 

Only Lighting Measure Savings 1,252 96% 

Lighting and Other Measure Savings 45 4% 

A majority of the customer locations, 68%, had a complete set of monthly billing data for the study 

period. That means they had twenty-four monthly observations with matched billing and savings data. 

A large majority of customer locations, 96%, installed only lighting measures so the impact of the correct 

specification of other measure savings should not have a significant impact on the estimate of the 

realization rate for lighting measures for National Grid. 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. If a customer has multiple accounts/meters at the 

same location, they have been combined for this ratio. 

Table 4-8 presents the summary of the savings ratios calculated for each customer location. Eighty-five 

percent of the customer locations show savings ratios less than 40%, with the remainder spread out over 

higher savings ratios. This distribution is typical for this type of data. 

Table 4-8. Summary of Savings Ratios for National Grid 

Savings Ratio Locations Percent 

0% to 9% 514 40% 

10% to 19% 325 25% 

20% to 29% 162 12% 

30% to 39% 103 8% 

40% to 49% 60 5% 

50% to 59% 34 3% 

60% to 69% 30 2% 

70% to 79% 16 1% 

80% to 89% 11 1% 

90% to 99% 9 1% 

100% or more 33 2% 

TOTAL 1,297 100% 

The initial modeling process began using all of this data. During the modeling process, two customers 

were identified for exclusion because they did not fit the model well.5

 After excluding these customers, 

 

5

 These two customers were identified as outliers because they dramatically changed the results when they were 

included in the model, suggesting that they were both outliers (different from other people), and had undue influence 
Summit Blue Consulting, LLC 19 

the final model result was a realization rate of 1.00 with a t-value of 22.3. This result is statistically 

significant at the 90% confidence level. These results are summarized in Table 4-9. 

Table 4-9. Lighting Savings Realization Rate for National Grid 

Realization 

Rate T-value 

Statistically 

Significant at 

the 90% 

Confidence 

Level? 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

1.00 22.38 Yes ± 7% 0.92 1.07 

The exact specification and results of the final fixed effects model for National Grid can be found in 

Appendix B. 

 

on the model. Every individual customer was tested for their influence on the model, and no other customers 

displayed this type of impact on the model results. 
Summit Blue Consulting, LLC 20 

4.3 NSTAR 

Table 4-10 summarizes the disposition of all data received from NSTAR for this study. Billing records 

were received for 801 accounts. When service addresses were examined, these accounts could be 

combined into 800 unique customer locations. 

Table 4-10. Summary of Data Disposition for NSTAR 

 Locations Accounts 

Billing Records Received 800 801 

Savings Records Received 1050 

Billing Records with no Savings Data 0 0 

Savings Records with no Billing Data 249 

First Round Match of Billing and Savings Data 800 801 

No Lighting Measures 0 

Mismatched Periods for Billing and Savings Data 4 

No Billing Data for Period before start of Savings 10 

TOTAL MATCHED BILLING AND SAVINGS DATA 786 

Savings records were received for 1,050 accounts. When these savings records were matched to the 

billing data, matches were found for 801 of the accounts. These accounts covered 800 unique customer 

locations. All billing data that was received was matched to a savings record, but 249 accounts in the 

savings data had no matching billing records. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All matched data was analyzed 

by location. All of the NSTAR locations had received lighting measures. Four locations were excluded 

because all of the available billing data was for the period before the savings measures were installed. 

Essentially, there was no post-participation data. Ten locations were dropped because they did not have 

any pre-participation billing data. This left a total of 786 locations with matched billing and savings data. 

At this point it was possible to look at some of the attributes of the matched dataset 

for NSTAR. 

Table 4-11 summarizes the completeness of the monthly billing data and customer’s installation of other 

measures in addition to lighting measures. 

Table 4-11. Attributes of Matched Billing and Savings Data for NSTAR 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 786 100% 

 
Summit Blue Consulting, LLC 21 

Incomplete Billing Data for Jan 2006 through Dec 2007 436 55% 

Complete Billing Data for Jan 2006 through Dec 2007 350 45% 

 

Only Lighting Measure Savings 752 96% 

Lighting and Other Measure Savings 34 4% 

Forty-five percent of customers had a complete set of monthly billing data for the study period. That 

means they had twenty-four monthly observations with matched billing and savings data. 

The large majority of customer locations, 96%, installed only lighting measures so the impact of the 

correct specification of other measure savings should not have a significant impact on the estimate of the 

realization rate for lighting measures for NSTAR. 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. If a customer has multiple accounts/meters at the 

same location, they have been combined for this ratio. 

Table 4-12 presents the summary of the savings ratios calculated for each customer location. Eighty-one 

percent of the customer locations show savings ratios less than 40%, with the remainder spread out over 

higher savings ratios. This distribution is typical for this type of data. 

Table 4-12. Summary of Savings Ratios for NSTAR 

Savings Ratio Locations Percent 

0% to 9% 275 35% 

10% to 19% 183 23% 

20% to 29% 107 14% 

30% to 39% 71 9% 

40% to 49% 40 5% 

50% to 59% 27 4% 

60% to 69% 16 2% 

70% to 79% 11 1% 

80% to 89% 7 1% 

90% to 99% 9 1% 

100% or more 40 5% 

TOTAL 786 100% 

The initial modeling process began using all of this data. During the modeling process, approximately 

400 monthly observations were eliminated as outliers. This represents about 2% of the monthly data. 

Results from the final model are summarized in Table 4-13. 

Table 4-13. Lighting Savings Realization Rate for NSTAR 
Summit Blue Consulting, LLC 22 

Realization 

Rate T-value 

Statistically 

Significant at 

the 90% 

Confidence 

Level? 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

0.89 23.37 Yes ± 7% 0.82 0.95 

The exact specification and results of the fixed effects model for NSTAR can be found in Appendix B. 
Summit Blue Consulting, LLC 23 

4.4 Unitil 

Table 4-14 summarizes the disposition of all data received from Unitil for this study. Billing records 

were received for 23 accounts. When service addresses were examined, there were no indications of 

multiple accounts at the same customer location. 

Table 4-14. Summary of Data Disposition for Cape Light Compact 

 Locations Accounts 

Billing Records Received 23 23 

Savings Records Received 28 

Billing Records with no Savings Data 1 1 

Savings Records with no Billing Data 6 

First Round Match of Billing and Savings Data 22 22 

No Lighting Measures 0 

Mismatched Periods for Billing and Savings Data 0 

No Billing Data for Period before start of Savings 0 

TOTAL MATCHED BILLING AND SAVINGS DATA 22 

Savings records were received for 28 accounts. When these savings records were matched to the billing 

data, matches were found for 22 of the accounts. This matching process left 1 accounts in the billing data 

that had no matching savings records, and 6 accounts in the savings data that had no matching billing 

records. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All 22 cases for Unitil 

contained the required information. 

At this point it was possible to look at some of the attributes of the matched dataset for Unitil. Table 4-15 

summarizes the completeness of the monthly billing data and customer’s installation of other measures in 

addition to lighting measures. 

Table 4-15. Attributes of Matched Billing and Savings Data for Unitil 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 22 100% 

 

Incomplete Billing Data for Jan 2006 through Dec 2007 0 0% 

Complete Billing Data for Jan 2006 through Dec 2007 22 100% 

 

Only Lighting Measure Savings 22 100% 

Lighting and Other Measure Savings 0 0% 
Summit Blue Consulting, LLC 24 

All customer locations had a complete set of monthly billing data for the study period. That means they 

had twenty-four monthly observations with matched billing and savings data. 

In addition, all of the customer locations installed only lighting measures so the impact of the correct 

specification of other measure savings will not have any impact on the estimate of the realization rate for 

lighting measures for Unitil. 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. 

Table 4-16 presents the summary of the savings ratios calculated for each customer location. Seventyseven percent of the customer locations show savings ratios less than 40%, with the remainder spread out 

over higher savings ratios. This distribution is typical for this type of data. 

Table 4-16. Summary of Savings Ratios for Unitil 

Savings Ratio Locations Percent 

0% to 9% 5 23% 

10% to 19% 4 18% 

20% to 29% 4 18% 

30% to 39% 4 18% 

40% to 49% 2 9% 

50% to 59% 0 0% 

60% to 69% 2 9% 

70% to 79% 1 5% 

80% to 89% 0 0% 

90% to 99% 0 0% 

100% or more 0 0% 

TOTAL 22 100% 

The initial modeling process began using all of this data. During the modeling process, two customers 

were identified as outliers. However, due to the small total number of customers in Unitil dataset, extra 

work was done to keep as much data as possible from those two customers in the model. Individual 

monthly observations for those two customers were evaluated and only twenty-two monthly observations 

were identified as outliers and excluded from the model. This reduced the total number of monthly 

observations from 504 to 482. 

A problem with autocorrelation of the data was also identified. Autocorrelation means that there is a 

correlation in the error terms. The error terms are the difference between the actual observed energy use 

and the energy use predictions from the model for each month. In the perfect model, these error terms are 

completely random and have no correlation to each other. In the real world, however, autocorrelation 

often occurs when evaluating time-series data because external conditions and/or customer behavior 

during one time period frequently has some carryover from the preceding time period. Things that affect 

usage this month may be similar to what affected usage last month. It is also possible that one month’s 
Summit Blue Consulting, LLC 25 

usage may be directly related to the prior month’s usage. For example, over production in one month 

may be followed by reduced production in the following month. There are many possible causes of 

autocorrelation in time-series data.6

Autocorrelation is more likely to cause model misspecification in small samples than in large samples. It 

was determined that the model for Unitil would be improved by correcting for autocorrelation. 

After making the correction, the final model result was a realization rate of 1.02 with a t-value of 12.39. 

This result is statistically significant at the 90% confidence level. These results are summarized in Table 

4-17. 

Table 4-17. Lighting Savings Realization Rate for Unitil 

Realization 

Rate T-value 

Statistically 

Significant at 

the 90% 

Confidence 

Level? 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

1.02 12.39 Yes ± 13% 0.88 1.16 

The exact specification and results of the final fixed effects model for Unitil can be found in Appendix B. 

 

6

 For a discussion of possible causes of autocorrelation, see pp. 139-140 in A Guide to Econometrics, Fifth Edition, 

Peter Kennedy, The MIT Press, 2003. 
Summit Blue Consulting, LLC 26 

4.5 Western Massachusetts Electric Company 

Table 4-18 summarizes the disposition of all data received from Western Massachusetts Electric 

Company for this study. Billing records were received for 126 accounts. When service addresses were 

examined, these accounts could be combined into 125 unique customer locations. 

Table 4-18. Summary of Data Disposition for Western Massachusetts Electric Co. 

 Locations Accounts 

Billing Records Received 125 126 

Savings Records Received 122 

Billing Records with no Savings Data 10 10 

Savings Records with no Billing Data 6 

First Round Match of Billing and Savings Data 115 116 

No Lighting Measures 2 

Mismatched Periods for Billing and Savings Data 0 

No Billing Data for Period before start of Savings 15 

TOTAL MATCHED BILLING AND SAVINGS DATA 98 

Savings records were received for 122 accounts. When these savings records were matched to the billing 

data, matches were found for 116 of the accounts. These accounts covered 115 unique customer 

locations. This matching process left 10 accounts in the billing data that had no matching savings 

records, and 6 accounts in the savings data that had no matching billing records. 

After the two sets of data were initially matched, additional data checking was done to make sure the 

matched data contained the information required for the billing analysis. All matched data was analyzed 

by location. Since this study is focusing on savings for lighting measures, 2 locations that did not receive 

any lighting measures were excluded. All locations had billing data available for the period after the 

savings measures were installed. However, there were 15 locations that were excluded because there was 

no billing data for the period before the installation of the savings measures. This left a total of 98 

locations with usable matched billing and savings data. 

Table 4-19 presents some attributes of the matched data. The table first summarizes the completeness of 

the monthly billing data, and then presents information on customers’ installation of other measures in 

addition to lighting measures. 
Summit Blue Consulting, LLC 27 

Table 4-19. Attributes of Matched Billing and Savings Data for WMECO 

Locations Percent 

TOTAL MATCHED BILLING AND SAVINGS DATA 98 100% 

 

Incomplete Billing Data for Jan 2006 through Dec 2007 7 7% 

Complete Billing Data for Jan 2006 through Dec 2007 91 93% 

 

Only Lighting Measure Savings 63 64% 

Lighting and Other Measure Savings 35 36% 

A majority of the customer locations, 93%, had a complete set of monthly billing data for the study 

period. That means they had twenty-four monthly observations with matched billing and savings data. 

A majority of customer locations, 64%, installed only lighting measures while 36% of the locations 

installed lighting and other measures. This is a large share of locations with savings from other measures. 

It is important to include savings from these other measures in the WMECO model to properly isolate the 

realization rate for lighting measures. 

Another attribute of the matched tracking and billing data that was analyzed was the savings ratio. The 

savings ratio is the annual savings estimate from the tracking system as a fraction of the total kWh use 

from the billing records for that customer location. If a customer has multiple accounts/meters at the 

same location, they have been combined for this ratio. 

Table 4-20 presents the summary of the savings ratios calculated for each customer location. Twentynine percent of the customer locations show savings ratios greater than 50% for WMECO. This is 

considerably higher than the typical ten percent of cases in this range for the other sponsors. It is an 

indication of possible mis-match between savings and billing data. Since it is unlikely that savings are 

really greater than 50% of total use in this many cases, the twenty-eight cases in this group were 

considered bad data and excluded from the modeling process. 
Summit Blue Consulting, LLC 28 

Table 4-20. Summary of Savings Ratios for Western Massachusetts Electric Co. 

Savings Ratio Locations Percent 

0% to 9% 20 21% 

10% to 19% 21 22% 

20% to 29% 12 12% 

30% to 39% 11 11% 

40% to 49% 6 6% 

50% to 59% 5 5% 

60% to 69% 6 6% 

70% to 79% 6 6% 

80% to 89% 1 1% 

90% to 99% 2 2% 

100% or more 8 8% 

TOTAL 98 100% 

The initial modeling process began using the 70 cases with good data. During the modeling process, 

monthly observations were eliminated for four customers identified as outliers. Results from the final 

model are summarized in Table 4-21. 

Table 4-21. Lighting Savings Realization Rate for Western Massachusetts Elec. Co. 

Realization 

Rate T-value 

Statistically 

Significant at 

the 90% 

Confidence 

Level? 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

1.03 5.7 Yes ± 29% 0.73 1.33 

The exact specification and results of the final fixed effects model for Western Massachusetts Electric 

Company can be found in Appendix B. 
Summit Blue Consulting, LLC 29 

5 RELIABILITY OF ESTIMATES

This section will start with a comparison of the statistical reliability of the estimates of realization rates 

for each sponsor. This will be followed by an identification and discussion of methods used to control 

relevant types of bias associated with the data collection and analysis efforts that were part of this study. 

5.1 Precision 

The statistical reliability of the estimates of the realization rates come directly from the fixed effects 

regression models. Since the realization rates are estimated as coefficients in the models, the standard 

errors that are calculated for each of the coefficients provide a measure of the reliability of the results. 

The standard errors can be evaluated at any specified confidence level to determine the range of precision 

for each estimate at that confidence level. 

It was hoped that this study would provide estimates of realization rates with plus or minus 10% precision 

at the 90% confidence level. Given that this study is based on the population of participants, and not a 

sample, the resulting precision rates reflect the best information available. Table 5-1 shows the precision 

rates that were achieved for each sponsor. The estimates for National Grid and NSTAR have a precision 

rate of plus or minus 7% which is better than the 10% goal. Sponsors with smaller numbers of customers 

had precision levels in the 10% to 30% range at the 90% confidence level. This wider precision range 

reflects the lower certainty of the coefficient estimates given greater variability and fewer observations. 

Table 5-1. Confidence Intervals and Precision Levels for Realization Rates 

Sponsor 

Expected 

Value of 

Realization 

Rate 

Precision at 

the 90% 

Confidence 

Level 

Lower Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Upper Bound 

of Realization 

Rate at 90% 

Confidence 

Level 

Cape Light Compact 1.04 ± 19% 0.84 1.25 

National Grid 1.00 ± 7% 0.92 1.07 

NSTAR 0.89 ± 7% 0.82 0.95 

Unitil 1.02 ± 13% 0.88 1.16 

WMECO 1.03 ± 29% 0.73 1.33 

It is not possible to improve the precision range for Cape Light Compact, Unitil and Western 

Massachusetts Electric Company given that each model was developed using all available data for the 

population. There is no way to add more observations. The precision range simply is what it is. 
Summit Blue Consulting, LLC 30 

Figure 5-1 compares the precision range for each realization rate estimate at the 90% confidence level. 

There is a 90% chance that the real realization rate is somewhere within the illustrated bar, and the 

expected value of the realization rate is the midpoint, or diamond, on the bar. This figure graphically 

illustrates the similarity of realization rates for all of the sponsors. The one exception is NSTAR which 

has a lower probable realization rate. 
Summit Blue Consulting, LLC 31 

Figure 5-1. 90% Confidence Intervals for Realization Rates 

0.70

0.80

0.90

1.00

1.10

1.20

1.30

1.40

Cape Light

Compact

National Grid NSTAR Unitil WMECO

Realization Rate

5.2 Bias Issues 

The potential for bias is always a concern in any statistical evaluation. Every effort is made to minimize 

potential bias in the study design and analysis methodology. Even so, bias is difficult to eliminate 

entirely. Potential sources of bias in this study will be identified and discussed here. 

Selection bias. Selection bias refers to problems that may arise because the participants selected for 

evaluation may be different than the general population of all participants. The methodology for this 

study was specifically designed to include all program participants rather than just a sample of 

participants. This eliminates the possibility of sampling error or non-response error. 

A different type of selection bias does occur, though, in that the necessary data needed for analysis is not 

always available for all customers who participated in the program. This may happen because customers 

move or go out of business and they do not have complete billing data for the study period. This type of 

selection bias was reduced by using a fixed effects regression model based on monthly observations. By 

combining all customers into a single model, it was not necessary to exclude customers just because they 

didn’t have a complete set of data. Whatever data that was available could be used, as long as there was 

data available for before and after the program participation date. The use of monthly observations 

instead of annual data also allowed for the inclusion of more customers. While previous sections of this 

report thoroughly reported the customer exclusions that had to be made because of insufficient data, every 
Summit Blue Consulting, LLC 32 

effort was made to retain information for as many customers as possible. This was done to minimize 

selection bias. 

Measurement bias. Measurement bias occurs when measurement techniques do not accurately represent 

what has actually occurred. The measurements used in this study were primarily billing data for energy 

use, tracking systems for program participation data, and weather data. 

Billing data is generally very accurate at the meter level since it is the basis for customer payments and 

distribution company revenues. Measurement bias can occur when meter-level data has to be combined 

to match the facility that installed the energy saving measures. The potential for this type of measurement 

bias was minimized by paying careful attention to the facility location information and creating 

algorithms to combine meter information at the facility location level. 

The measurement bias from the tracking systems is unknown. Since tracking systems are unique to each 

sponsor, the measurement bias could easily vary by sponsor. An important aspect of the design of this 

study to minimize measurement bias was the inclusion of savings for other measures besides lighting if 

the customer installed several types of measures. In this way, changes to the bill related to savings from 

non-lighting measures could be quantified separately and kept from influencing the realization rate for 

lighting savings. Likewise, if a customer installed multiple lighting measures at different times, 

measurement bias was reduced by adding each additional lighting increment in the appropriate month. 

Careful use of all the information available in the tracking systems helped reduce measurement bias. 

The weather data came from the National Oceanic and Atmospheric Administration (NOAA) database. 

This is a reliable source of weather measurements with a high internal standard for minimizing 

measurement bias. The most likely source of bias related to weather data could come from the selection 

of weather-monitoring sites to use for the study. If the chosen site does not have weather that matches the 

locations of the program participants, there could be some measurement bias. By agreement of the 

sponsors, data from the Worcester, Massachusetts monitoring site was used for this study. It was felt that 

this would be representative weather data for all participants. 

Even when care is taken to reduce measurement bias, it is still a real issue that can’t be completely 

addressed in a billing analysis. Problems like 1) mis-recording of how many lights have been changed 

out on a project; 2) mis-stating the wattage deltas; 3) mis-estimating hours of use; 4) errors in copying 

information from paper to the tracking systems, etc. create measurement bias which is difficult to correct. 
Summit Blue Consulting, LLC A-1 

APPENDIX A – DATA CLEANING STEPS
Summit Blue Consulting, LLC A-1 

The data received from the five participating sponsors was first formatted to make it ready for the SAE 

model. Four main steps were carried out to format the data: 

Step 1: Bring in tracking data 

• Import the tracking data for all the sponsors. Run a program to clean up the data and move it into 

standard field names of the same length and type so that the files can be merged together. 

• Add a sponsor name column to the records so the lines can be separated by sponsor. 

• Create a measure description field which contains as much information as possible about the 

measures, to help with categorizing them. 

• Use the Location ID instead of the Account Number for National Grid. 

• If the Install Start Date is unknown, make it the same as the Install End Date. 

• Create a list of all measures by sponsor. Manually assign a measure category to each. The 

categories are INT-Interior Lighting, EXT-Exterior Lighting, CNT-Lighting Controls, LED-LED 

Exit Lights, and OTH-Other non-lighting measures. Note: If the measure description for HID 

lights did not identify whether it was an interior or exterior installation, all large HID lights were 

coded as EXT. 

• Assign the Measure Categories to the measures. Sum savings for each Account Number and each 

Measure Category. 

• If there is an InstallStartDate and an InstallStopDate, assume the savings start is halfway in 

between. 

• For each customer measure summary, create a record for every day from the measure start to the 

current day. Each day is annual savings divided by 365.25. 

• Summarize by year and month, creating a file which contains the savings for that customer and 

category for each calendar month. Savings can vary by month because a customer does not 

necessarily install all measures at the same time. 

• Merge in the number of calendar days for each month. 

Step 2: Bring in weather data 

• Get daily weather data from NOAA for January 2006 to the present for Worcester, 

Massachusetts. 

• Convert date and time values to a standard format. 

• Get rid of duplicate hourly data. 

• Summarize by day, giving maximum, minimum and average temperature and humidity values. 

• Calculate Heating Degree Days and Cooling Degree Days. These are defined as: A "degree day" 

is a unit of measure for recording how hot or how cold it has been over a 24-hour period. The 

number of degree days applied to any particular day of the week is determined by calculating the 

mean temperature for the day and then comparing the mean temperature to a base value of 65 

degrees F. (The "mean" temperature is calculated by adding together the high for the day and the 

low for the day, and then dividing the result by 2.)If the mean temperature for the day is, say, 5 

degrees higher than 65, then there have been 5 cooling degree days. On the other hand, if the 
Summit Blue Consulting, LLC A-2 

weather has been cool, and the mean temperature is, say, 55 degrees, then there have 10 heating 

degree days (65 minus 55 equals 10.7

• Summarize data by calendar month. 

Step 3: Bring in the billing data. 

• Bring in data for each sponsor. 

• Filter out records with zero billing kWh or days billed = 0 

• Create a total address field for each record. It is a combination of fields that will equate to a 

street address and city. This will be used to determine multiple accounts/meters at one location. 

• For Cape Light Compact data, eliminate true duplicates. (If a record was for the same account and 

the same bill read date and it had the same kWh reading, it was considered a duplicate record and 

excluded. If it matched on account and bill read date but had a different kWh reading, it was 

considered an additional meter reading for that customer and added in.) Add up all kWh for 

records with the same account and date. 

• If the billing start date is missing, look at the previous record’s end date and use that to estimate 

the start date. 

• Create a unique LocationKey for each location based on sponsor, customer name and total 

address. 

• Add LocationKey to all billing records. 

• Split the billing cycles into days and average use per day, and re-combine to create average use 

per day for calendar months for each customer. 

• Summarize by LocationKey and date because there can be multiple accounts/meters at a location. 

Create a field to keep number of meters per location. 

Step 4: Merge all data together 

The final format of the data is as shown in Table A-1. 

 

7

 http://www.srh.noaa.gov/ffc/html/degdays.shtml 
Summit Blue Consulting, LLC A-3 

Table A-1. Final Field Names and Descriptions 

Sponsor CLC (Cape Light Compact) 

NGRIDEI (National Grid EI Program) 

NGRIDSBS (National Grid SBS Program) 

NSTAR 

UNITIL 

WMECO 

Location Key This identifies a unique customer location. The Location Key is 

created by SBC to identify all billings accounts/meters that should be 

combined for the same customer location. They are unique across 

sponsors. 

Year 4 digit year 

Month 2 digit month 

aveDailykWh Average daily kWh for all accounts at the given location during the 

calendar month. Multiply times the number of days in the calendar 

month to get monthly kWh use. 

AveMeters The number of accounts/meters that were combined at this location. 

CalendarDays The number of days in this calendar month. 

INTAveSavingsbyDay Average daily kWh savings for all INTERIOR LIGHTING measures 

at this location. It is zero before installation and pro-rated during the 

month of installation. After installation, it is the annual savings/365. 

Multiply times the number of days in the calendar month to get 

monthly kWh savings. 

EXTAveSavingsbyDay Same as above, but for EXTERIOR LIGHTING measures 

CNTAveSavingsbyDay Same as above, but for LIGHTING CONTROLS measures 

LEDAveSavingsbyDay Same as above, but for LED EXIT LIGHT measures 

OTHAveSavingsbyDay Same as above, but for OTHER NON-LIGHTING measures 

AveMonthTemp Average temperature over the calendar month 

AveMonthHumid Average humidity over the calendar month 

AveMonthTHI Average Temperature-Humidity Index over the calendar mo 

AveMonthCDD Average daily Cooling Degree Days over the calendar month. 

Multiply times the number of days in the calendar month to get 

monthly CDD. 

AveMonthHDD Average daily Heating Degree Days over the calendar month. 

Multiply times the number of days in the calendar month to get 

monthly HDD. 
Summit Blue Consulting, LLC A-4 

APPENDIX B—COMPLETE REGRESSION RESULTS
Summit Blue Consulting, LLC A-5 

Cape Light Compact 

Fixed Effects 

Valid cases: 6626 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 1121894812.046 Degrees of freedom: 6621 

R-squared: 0.008 Rbar-squared: 0.007 

Residual SS: 1113205978.299 Std error of est: 410.040 

F(5,6621): 10.336 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -0.556819 0.102258 -5.445243 0.000 -0.071749 -0.050515 

Y2006 -43.977805 11.024341 -3.989155 0.000 -0.052690 -0.027371 

TEMP -0.222113 0.498526 -0.445540 0.656 -0.008498 0.021098 

WINTER -11.459092 16.223236 -0.706338 0.480 -0.013121 -0.030007 

FALL 48.945383 13.281218 3.685308 0.000 0.051131 0.048186 

Full R-Squared 0.99142136 

Correcting for Auto - Fixed Effects 

Valid cases: 6320 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 944872919.361 Degrees of freedom: 6315 

R-squared: 0.014 Rbar-squared: 0.014 

Residual SS: 931411635.767 Std error of est: 384.047 

F(5,6315): 18.254 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -1.044631 0.121806 -8.576193 0.000 -0.116372 -0.083389 

Y2006 -83.943844 14.135899 -5.938345 0.000 -0.081191 -0.037345 

TEMP -0.266063 0.538866 -0.493746 0.622 -0.008305 -0.003017 

WINTER 5.189615 16.503706 0.314451 0.753 0.005494 -0.011542 

FALL 45.261941 14.112699 3.207178 0.001 0.045031 0.034562 

Rho 0.39420908 

Full R-Squared 0.99282231 

Where: 

ENGEST is the engineering estimate of savings for lighting installations 

Y2006 is a 1-0 indicator variable for the year (2006 = 1, 2007 = 0) 

TEMP is the average outdoor temperature 

WINTER is a 1-0 indicator for the winter season (Nov, Dec, Jan, Feb) 

FALL is a 1-0 indicator for the fall season (Aug, Sep, Oct) 
Summit Blue Consulting, LLC A-6 

National Grid 

Fixed Effects 

Valid cases: 30687 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 1173571906.175 Degrees of freedom: 30681 

R-squared: 0.024 Rbar-squared: 0.024 

Residual SS: 1145032063.390 Std error of est: 193.185 

F(6,30681): 127.454 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -0.996846 0.044549 -22.376604 0.000 -0.137559 -0.121435 

Y2006 -10.319893 2.492057 -4.141115 0.000 -0.025828 0.037706 

TEMP 1.672938 0.109735 15.245208 0.000 0.133899 0.078244 

WINTER 26.236214 3.570675 7.347691 0.000 0.062854 -0.035702 

FALL 0.976119 2.872861 0.339772 0.734 0.002171 0.024890 

OTH -0.405922 0.120313 -3.373897 0.001 -0.019333 -0.015588 

Full R-Squared 0.97682783 

Where: 

ENGEST is the engineering estimate of savings for lighting installations 

Y2006 is a 1-0 indicator variable for the year (2006 = 1, 2007 = 0) 

TEMP is the average outdoor temperature 

WINTER is a 1-0 indicator for the winter season (Nov, Dec, Jan, Feb) 

FALL is a 1-0 indicator for the fall season (Aug, Sep, Oct) 

OTH is the engineering estimate of savings for other non-lighting installations 
Summit Blue Consulting, LLC A-7 

NSTAR 

Fixed Effects 

Valid cases: 17242 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 229178552.244 Degrees of freedom: 17237 

R-squared: 0.047 Rbar-squared: 0.046 

Residual SS: 218493996.234 Std error of est: 112.587 

F(5,17237): 168.581 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -0.885723 0.037906 -23.366015 0.000 -0.186509 -0.165201 

Y2006 -10.676367 1.888320 -5.653896 0.000 -0.045519 0.029048 

TEMP 0.531244 0.113074 4.698214 0.000 0.072038 0.104894 

SPRING -26.088793 2.966681 -8.793934 0.000 -0.097974 -0.081442 

WINTER -7.718591 4.090038 -1.887168 0.059 -0.031022 -0.052191 

Full R-Squared 0.94147675 

Where: 

ENGEST is the engineering estimate of savings for lighting installations 

Y2006 is a 1-0 indicator variable for the year (2006 = 1, 2007 = 0) 

TEMP is the average outdoor temperature 

SPRING is a 1-0 indicator for the spring season (Mar, Apr, May) 

WINTER is a 1-0 indicator for the winter season (Nov, Dec, Jan, Feb) 
Summit Blue Consulting, LLC A-8 

Unitil 

Fixed Effects 

Valid cases: 504 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 1730032.034 Degrees of freedom: 499 

R-squared: 0.196 Rbar-squared: 0.189 

Residual SS: 1391714.651 Std error of est: 52.811 

F(5,499): 24.261 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -0.930846 0.097731 -9.524597 0.000 -0.439871 -0.320634 

Y2006 -23.587213 5.462856 -4.317744 0.000 -0.198191 0.032768 

TEMP 1.587119 0.234567 6.766155 0.000 0.425325 0.142999 

FALL -7.618144 6.145638 -1.239602 0.216 -0.056276 -0.008843 

WINTER 30.358766 7.550044 4.021005 0.000 0.243257 -0.001259 

Full R-Squared 0.90739409 

Correcting for Auto - Fixed Effects 

Valid cases: 482 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 1018574.737 Degrees of freedom: 477 

R-squared: 0.289 Rbar-squared: 0.283 

Residual SS: 724134.666 Std error of est: 38.963 

F(5,477): 38.791 Probability of F: 0.000 

 Standard Prob Standardized Cor with 

Variable Estimate Error t-value >|t| Estimate Dep Var 

------------------------------------------------------------------------------- 

ENGEST -1.019792 0.082285 -12.393360 0.000 -0.543793 -0.477561 

Y2006 -17.849496 6.390393 -2.793177 0.005 -0.123638 0.149736 

TEMP 1.223129 0.206217 5.931270 0.000 0.294394 0.077451 

FALL -1.677211 5.138308 -0.326413 0.744 -0.014126 -0.053817 

WINTER 26.607001 6.020478 4.419416 0.000 0.227542 0.106901 

Rho 0.49804523 

Full R-Squared 0.97783136 

Where: 

ENGEST is the engineering estimate of savings for lighting installations 

Y2006 is a 1-0 indicator variable for the year (2006 = 1, 2007 = 0) 

TEMP is the average outdoor temperature 

WINTER is a 1-0 indicator for the winter season (Nov, Dec, Jan, Feb) 

FALL is a 1-0 indicator for the fall season (Aug, Sep, Oct) 
Summit Blue Consulting, LLC A-9 

WMECO 

Fixed Effects 

Valid cases: 66 Dependent variable: KWH 

Missing cases: 0 Deletion method: None 

Total SS: 59866765.910 Degrees of freedom: 1483 

Mean SS: 40368.69 Std error of est: 200.919 

F(66,1483): 206.86 Probability of F: 0.000 

 Standard Prob 

Variable Estimate Error t-value >|t| 

--------------------------------------------------------- 

ENGEST -1.03228 0.1811 -5.7 0.000 

Y2006 -35.4394 11.0316 -3.21 0.001 

TEMP 0.548403 0.3304 1.66 0.097 

OTHER 0.565913 0.1256 4.5 0.000 

Full R-Squared 0.9523 

 

Where: 

ENGEST is the engineering estimate of savings for lighting installations 

Y2006 is a 1-0 indicator variable for the year (2006 = 1, 2007 = 0) 

TEMP is the average outdoor temperature 

OTHER is the engineering estimate of savings for other non-lighting installations 
Cape Light Compact 

2007 Commercial and Industrial Programs 

Free-ridership and Spillover Study 

Final Executive Summary 

June 23, 2008 
Cape Light Compact 6/23/08 

Cape Light Compact 

2007 Commercial and Industrial Programs 

Free-ridership and Spillover Study 

Final Executive Summary 

June 23, 2008 

© PA Knowledge Limited 2008 

PA Consulting Group 

6410 Enterprise Lane 

Suite 300 

Madison, WI 53719 

Tel: +1 608 443 2700

Fax: +1 608 661 5181

www.paconsulting.com

Prepared by: Laura Schauer 

Pam Rathbun 

Jeremy Kraft 

Eric Rambo 

Version: 1.0 
i

Cape Light Compact.6/23/08 

TABLE OF CONTENTS 

1. Executive Summary: Cape Light Compact 1-1

1.1 Study Objectives 1-1

1.2 Study Methodology 1-1

1.3 Total Participant Free-ridership Estimates 1-2

1.4 Participant “Like” Spillover Estimates 1-5

1.5 Nonparticipant Spillover Estimates 1-7
1-1

Cape Light Compact.6/23/08 

1. EXECUTIVE SUMMARY: CAPE LIGHT COMPACT 

This Executive Summary summarizes the findings of the program year 2007 Commercial and 

Industrial Free-ridership and Spillover Study for Cape Light Compact. The emphasis of this 

study was to assess program free-ridership, participant spillover, and nonparticipant spillover 

for the following programs offered by Cape Light Compact: 

• New Construction 

• Medium and Large Commercial and Industrial Retrofit 

• Small Commercial and Industrial Retrofit 

The 2007 Free-ridership and Spillover Studies ran concurrently for National Grid, Cape Light 

Compact, United Illuminating, and Unitil. 

1.1 STUDY OBJECTIVES 

The primary objective of the year 2007 Commercial and Industrial Free-ridership and 

Spillover Study was to assist Cape Light Compact in quantifying the net impacts of their 

commercial and industrial energy efficiency programs by estimating the extent of: 

• Program free-ridership 

• Participant “like” spillover 

This executive summary provides the free-ridership and participant spillover estimates for 

2007. The study also includes a nonparticipant spillover analysis. Cape Light Compact 

vendors were not included in this particular portion of the study due to lack of information; 

however, the results, determined in aggregate across all participating sponsors, are 

presented at the end of this section. First, a summary of the methodology is provided. 

1.2 STUDY METHODOLOGY 

The methodology used for this year’s study follows the standardized methods developed in 

2003 for a group of Massachusetts energy efficiency program administrators1

. 

To accomplish the above objectives, telephone surveys were conducted with samples of 

2007 program participants in each of the programs, with design professionals and equipment 

vendors involved in these 2007 installations. The program participant sample consisted of 

unique utility accounts, not unique customers names. The same customer name, or business 

entity, can have multiple accounts in multiple locations, but program technical support and 

incentives are provided on behalf of an individual account. Thus, for the purposes of this 

study, a customer or participant is defined as a unique account. 

The majority of these telephone interviews were completed with program participants 

 

1

 Pamela Rathbun, Carol Sabo, and Bryan Zent, Standardization Methods for Free-ridership and 

Spillover Evaluation—Task 5 Final Report (Revised), prepared for National Grid, NSTAR Electric, 

Northeast Utilities, Unitil, and Cape Light Compact, June 16, 2003. 
1. Executive Summary: Cape Light Compact... 

1-2

Cape Light Compact.6/23/08 

between March 14 and May 2, 2008. All sampled participating customers were mailed a letter 

on Cape Light Compact letterhead in advance of the telephone call. This letter explained the 

purpose of the call, informed customers that someone would be calling them in the next 

couple of weeks to ask them some questions about their experience with the programs, and 

thanked them for their cooperation in advance. This advance letter and repeated call attempts 

resulted in a high response rate of 81 percent, which increases the level of confidence in the 

survey results. The duration of interviews with program participants averaged nine minutes. 

In addition to the customer surveys, surveys were conducted with design professionals and 

vendors identified by customers as being the most knowledgeable about the decisions to 

install the equipment through Cape Light Compact’s programs. These surveys were used for 

estimating free-ridership for those installations where the design professional/equipment 

vendor was more influential in the decision than the customer. 

The number of survey completions for some measure categories is low because the number 

of installations within these measure categories for program year 2007 was small. Thus, 

although a high percentage of the 2007 program participants completed surveys, some 

caution should be used when interpreting the results. 

1.3 TOTAL PARTICIPANT FREE-RIDERSHIP ESTIMATES 

A program’s free-ridership rate is the percentage of program participants deemed to be free 

riders. A free rider refers to a customer who received an incentive through an energy 

efficiency program who would have installed the same or a smaller quantity of the same high 

efficiency measure on their own within one year if the program had not been offered. For free 

riders, the program is assumed to have had no influence or only a slight influence on their 

equipment purchase decision. Consequently, none or only some of the energy savings of 

equipment purchased by this group of customers should be credited to the energy efficiency 

program. Free riders account for costs, but not benefits, to the program, driving benefit-cost 

ratios down. 

For programs that offer monetary incentives for multiple measure categories (e.g., motors, 

lighting, HVAC), it is important to estimate free-ridership by specific measure category. 

Category-specific estimates produce feedback on the program at the level at which it actually 

operates and allow for cost-effectiveness testing by measure category. 

In addition, for commercial and industrial incentive programs, free-ridership has often been 

found to be highly variable among measure categories, making it essential to produce 

measure category-specific estimates. The ability to provide reliable estimates by measure 

category is dependent on the number of installations within that measure category—the fewer 

installations, the less reliable the estimation. 

It is also important to measure the extent of free-ridership for each customer. Pure free riders 

(100%) would have installed exactly the same quantity and type of equipment within one year 

in the absence of the program. Partial free riders (1–99%) are those customers who would 

have installed some equipment within one year on their own, but a smaller quantity and/or a 

lesser efficiency. Thus, the program had some impact on their decision. Non-free riders (0%) 

are those who would not have installed any high efficiency qualifying equipment within one 

year in the absence of the program services. The total free-ridership estimates in this report 

include pure, partial, and non-free riders. 
1. Executive Summary: Cape Light Compact... 

1-3

Cape Light Compact.6/23/08 

This year’s approach to estimating free riders follows the approach outlined in the 

Standardization Methods... report, which consisted of a sequential question technique to 

identify free riders. This sequential approach asks program participants about the actions they 

would have taken if the program had not been offered. This approach is considered an 

accurate method of estimating the actual level of free-ridership among program participants 

because it addresses the program’s impact upon project timing, measure quantity, and 

efficiency levels while explicitly recognizing that the cost of energy-efficient equipment can be 

a barrier to installation in the absence of utility-sponsored energy efficiency programs. This 

method is also recommended because it walks survey respondents through their decision 

process with the objective of helping them recall the program’s impact upon all aspects of 

project decision-making. 

One issue with the method is how to handle responses of “don’t know.” The “don’t know” 

responses to the initial free-ridership question are assigned a free-ridership value of zero 

percent. For these cases, we then check their responses to the consistency questions and 

their response to open-ended question and adjust the free-ridership rate as appropriate. Note 

that program total free-ridership (pure and partial) rates illustrated in the following tables are 

weighted by measure category kWh savings as well as the disproportionate probability of 

being sampled. 

New Construction Participant Free-ridership Rates. Table 1-1 summarizes the total freeridership results overall and by measure category for 2007 New Construction installations. 

The total free-ridership for the 2007 program year was 14.6 percent. 

Table 1-1 

Cape Light Compact New Construction Program Total Participant Free-ridership Rates 

All 2007 Installations 

Measure Description Total Participant Free-ridership Rate

# Accounts 

(Survey/Pop) 2007 

2007 90% 

Error 

Margin 

2005 2004

Variable Speed Drives 1/1 100.0% ±0.0% — — 

Lighting 3/5 13.0% ±20.2% 72.6% — 

Overall New Construction 3/5 14.6% ±21.2% 61.4% —

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories.

No surveys were conducted with New Construction participants in 2004. No surveys were done with VSD 

measures in 2005. 
1. Executive Summary: Cape Light Compact... 

1-4

Cape Light Compact.6/23/08 

Medium and Large C&I Retrofit Program Participant Free-ridership Rates. Table 1-2 

summarizes Products and Services free-ridership results for 2007 HVAC installations. 

Refrigeration and HVAC installations could not be assessed; only one account was identified 

for each measure, and those accounts were not reachable for this study. The total freeridership for the 2007 program year was 23.1 percent. 

Table 1-2 

Cape Light Compact Medium and Large C&I Retrofit Program Total Participant 

Free-ridership Rates 

All 2007 Installations 

Measure Description Total Participant Free-ridership Rate

# Accounts 

(Survey/Pop) 2007 

2007 

90% 

Error 

Margin 

2005 2004

Variable speed Drives 2/2 19.3% ±0.0% 81.1% 22.4% 

HVAC 0/1 — — — — 

Refrigeration 0/1 — — — — 

Lighting 2/2 35.2% ±0.0% — — 

Overall Products & 

Services 4/6 23.1% ±20.0% 81.1% 19.5% 

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories.

*Variable Speed Drives and Lighting measures were not included in the 2005 or 2004 studies under the 

Products and Services program. 

Small C&I Retrofit Participant Free-ridership Rates. Table 1-3 summarizes the total freeridership results overall and by measure category for 2007 Small C&I Retrofit Program 

installations. The total free-ridership for the 2007 program year was 7.4 percent, slightly lower 

than the 2005 rate for the Small C&I Retrofit program. 
1. Executive Summary: Cape Light Compact... 

1-5

Cape Light Compact.6/23/08 

Table 1-3 

Cape Light Compact Small C&I Retrofit Program 

Total Participant Free-ridership Rates 

All 2007 Installations 

Measure Description Total Participant Free-ridership Rate

# Accounts 

(Survey/Pop) 2007 2007 90% Error Margin 2005 2004*

HVAC 22/35 2.1% ±3.1% 55.7% 8.1% 

Refrigeration 45/58 7.8% ±3.1% 9.0% — 

Lighting 65/166 7.7% ±4.2% 7.1% 5.8% 

Overall Retrofit 108/215 7.4% ±2.9% 8.0% 6.1% 

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories.

*Rates were pulled from the 2004 Small C&I Retrofit category. Refrigeration was not part of the Retrofit 

sample in 2004. 

1.4 PARTICIPANT “LIKE” SPILLOVER ESTIMATES 

Spillover refers to additional energy-efficient equipment installed by a customer due to 

program influences but without any financial or technical assistance from the program. 

Participant “like” spillover refers to the situation where a customer installed equipment 

through the program in the past year and then installed additional equipment of the same type 

due to program influences. In contrast to free-ridership, spillover adds benefits to the program 

at no additional cost, increasing the program benefits and benefit-cost ratio. 

Survey free-ridership questions were followed by questions designed to measure "like" 

spillover. These questions asked about recent purchases (since program participation in 

2007) of any additional energy-efficient equipment of the same type as installed through the 

program that were made without any technical or financial assistance from the utility. A “like” 

spillover estimate was computed based on how much more of the same energy-efficient 

equipment the participant installed outside the program and did so because of their positive 

experience with the program.

One of the issues with attempting to quantify spillover savings is how to value the savings of 

measures installed outside the program since we are relying on customer self-reports of the 

quantity and efficiency of any measures installed. We used a conservative approach and 

reported only those measures installed outside the program that were of exactly the same 

type and efficiency as the ones installed through the program. Our conservative approach 

allowed customers to be more certain about whether the equipment they installed outside the 

program was the same type as the program equipment. This, in turn, made it possible for us 

to use the estimated program savings for that measure to calculate the customer’s “like” 

spillover savings. 

When reviewing the measure category “’like’ spillover,” it is important to consider the number 

of survey completions that the estimate is based upon. The number of survey completions for 
1. Executive Summary: Cape Light Compact... 

1-6

Cape Light Compact.6/23/08 

some measure categories is low because very few customers in the sample installed the 

measure. 

New Construction Participant “Like” Spillover Rates. Table 1-4 presents the like spillover 

rate for year 2007 New Construction participants overall and by measure. The New 

Construction program has no spillover saving attributions for the purchase of like equipment 

outside of the program. 

Table 1-4 

Cape Light Compact New Construction Program Participant 

“Like” Spillover Rates 

All Year 2007 Installations 

Total Participant “Like” Spillover Rate

Measure 

Description # Accounts 

(Survey/Pop) 2007 

2007 90% 

Error 

Margin 

2005 2004

Variable Speed 

Drives 1/1 0.0% — 0.0% — 

Lighting 3/5 0.0% — 0.0% — 

Overall 3/5 0.0% — 0.0% — 

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories. 

No surveys were conducted with New Construction participants in 2004. No surveys were done with VSD 

measures in 2005. 

Medium and Large C&I Retrofit Program Participant “Like” Spillover Rates. Table 1-5 

presents the like spillover rate for year 2007 Products and Services installations. The rate is 

lower than in 2005; however, the population and sample size were extremely small. 

Table 1-5 

Cape Light Compact Medium and Large C&I Retrofit Program Participant “Like” 

Spillover Rates 

All Year 2007 Installations 

Measure Description Total Participant “Like” Spillover Rate

# Accounts 

(Survey/Pop) 2007 

2007 

90% 

Error 

Margin 

2005 2004

Variable speed Drives 2/2 0.0% ±0.0% — — 

HVAC 0/1 — — 6.3% 0.0% 

Refrigeration 0/1 — — — — 

Lighting 2/2 0.7% ±0.0% — — 

Overall Products & 

Services 4/6 0.2% ±2.0% 6.3% 0.0% 
1. Executive Summary: Cape Light Compact... 

1-7

Cape Light Compact.6/23/08 

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories.

*Variable Speed Drives and Lighting measures were not included in the 2005 or 2004 studies under the 

Products and Services program. 

Retrofit Participant “Like” Spillover Rates. Table 1-6 summarizes the like spillover rate for 

year 2007 Retrofit installations overall and by measure. The spillover rate in 2007 is slightly 

higher than 2005. 

Table 1-6 

Cape Light Compact Small C&I Retrofit Program 

Participant “Like” Spillover Rates 

All Year 2007 Installations 

Measure Description Total Participant “Like” Spillover Rate

# Accounts 

(Survey/Pop) 2007 2007 90% Error Margin 2005 2004+

HVAC 22/35 25.4% ±9.3% 0.0% 0.4% 

Refrigeration 45/58 0.4% ±0.7% 9.0% — 

Lighting 65/166 0.3% ±0.9% 0.6% 3.5% 

Overall Retrofit 108/215 1.7% ±1.4% 0.5% 2.7% 

Overall survey and population participant counts do not equal the sum of measure category survey and 

population participant counts; the same participant may be represented in multiple measure categories.

*Rates were pulled from the 2004 Small C&I Retrofit category. Refrigeration was not part of the Retrofit 

sample in 2004. 

+Rates were pulled from the 2004 Small C&I Retrofit category. Refrigeration was not part of the Retrofit 

sample in 2004. 

1.5 NONPARTICIPANT SPILLOVER ESTIMATES 

Nonparticipant spillover refers to energy efficient measures installed by program 

nonparticipants due to the program's influence. The program can have an influence on design 

professionals and vendors as well as an influence on product availability, product acceptance, 

customer expectations, and other market effects, all of which may induce nonparticipants to 

buy high efficiency products. Total nonparticipant spillover would also include responses from 

nonparticipating designers and vendors. 

The methodology for the 2007 study estimated only a portion of nonparticipant like-measure 

spillover based on responses from design professionals and vendors participating in National 

Grid, United Illuminating, and Unitil’s Medium and Large Commercial programs2. Cape Light 

Compact vendors were not included in this study due to insufficient data; however, two of the 

three vendors that were indicated within Cape Light Compacts’ vendor data overlapped with 

National Grid’s vendor sample and were surveyed. 

 

2

Nonparticipant spillover for small business programs was not estimated because of the small number of vendors involved in 

delivering the program. 
1. Executive Summary: Cape Light Compact... 

1-8

Cape Light Compact.6/23/08 

The data for the analysis could have been collected from nonparticipants directly or from the 

design professionals and vendors who recommended, sold, and/or installed qualifying high 

efficiency equipment. We chose to survey the design professionals and vendors primarily 

because they could typically provide much more accurate information about the efficiency 

level of installed equipment than could the nonparticipants. Experience has shown that 

customers cannot provide enough data about the new equipment they have installed to allow 

for accurate estimates of the energy savings achieved from the equipment. While they usually 

can report what type of equipment was installed, they typically cannot provide sufficient 

information about the quantity, size, efficiency, and/or operation of that equipment to allow us 

to determine whether the equipment is "program-eligible." On the other hand, design 

professionals and equipment vendors who have worked with the program are typically more 

knowledgeable about equipment and are familiar with what is and is not "program-eligible." 

Another argument in favor of using design professionals and equipment vendors to estimate 

nonparticipant spillover was that we could use data in the program tracking system database 

to attach kWh savings estimates to nonparticipant spillover. In the program tracking system 

database, measure-specific program kWh savings are associated with each design 

professional and vendor who participated in the program in 2007. 

To determine nonparticipant spillover, design professionals and equipment vendors were 

asked (by measure category they installed in the program) what percent of their sales were 

program-eligible and what percent of these sales did not receive an incentive through the 

programs. They were then asked about the program’s impact on their decision to 

recommend/install this efficient equipment outside the program. Using the survey responses 

and measure savings data from the program tracking system, the participating vendor 

nonparticipant like spillover savings could be estimated for each design professional/vendor 

and the results extrapolated to the total program savings. 

This method of estimating nonparticipant spillover is a conservative estimate for two reasons. 

First, not all design professionals and equipment vendors who are familiar with the programs 

specified and/or installed equipment through the program in 2007. Thus, we miss any 

nonparticipant spillover that was associated with these other design professionals/vendors 

(although it is less likely these design professionals/vendors had nonparticipant spillover if 

they were not involved with the program in 2007). 

Second, this method only allows us to extrapolate nonparticipant spillover for those same 

measure categories that a particular design professional/vendor was associated with for the 

2007 programs. Thus, if a vendor installed program-eligible equipment in other measure 

categories in the year 2007 outside the program, but none through the program, we did not 

capture nonparticipant spillover savings with that particular type of equipment. In essence, we 

measured only "like" nonparticipant spillover; that is, spillover for measures like those 

installed through the program in 2007. 

The nonparticipant spillover results for the Medium and Large Commercial and Industrial 

programs are based on surveys with 106 design professionals and vendors out of a 

population of 237 National Grid, United Illuminating, and Unitil vendors. Because of the 

significant overlap in sponsors’ territories, as well as vendors across sponsors, we report the 

results in aggregate rather than by sponsor. The analysis indicates that the combined 

nonparticipant spillover from the medium and large commercial and industrial programs 

amounted to 2,603,307 kWh in the 2007 program year, which is approximately 2.6 percent of 

the total savings produced by these programs combined (Table 1-7). 
1. Executive Summary: Cape Light Compact. . . 

Table 1-7 

Nonparticipant “Like” Spillover Results for Program Year 2007 

National Grid, Unitil, and United Illuminating Vendors

A B C D E F G H 

Survey 

Categories 

Vendor 

Population 

kWh 

Savings3

Number of Firms 

Surveyed with 

kWh Savings/ 

Number of Firms 

in Program with 

kWh Savings 

Surveyed 

kWh 

Savings4

Surveyed 

Savings 

Coverage 

Rate 

(D/B) 

Nonparticipant 

Spillover from 

Surveyed 

Firms (kWh)5

Estimated 

Spillover 

Percent 

(F/D) 

90% CI 

Motors 102,873 6/16 38,077 37.0% 0 0.0% 0HVAC 10,877,314 27/60 2,175,565 20.0% 79,149 3.6% 0.7% to 3VSD6 2,393,842 11/28 802,202 33.5% 115,569 14.4% 4.8% to 25Lighting 56,560,136 60/151 20,074,391 35.3% 603,572 3.0% 1.0% to 4Compressed Air 4,671,464 10/22 1,743,112 37.3% 60,498 3.5% 1.3% to 6Refrigeration 4,758,046 2/6 1,197,312 25.2% 0 0.0% 0Other7 19,474,884 10/33 4,998,940 25.7% 0 0.0% 0Total 98,838,559 106/254 31,029,599 31.4% 858,788 2.6% 1.0% to 3

3 The vendor population kWh savings represents the total savings for all measures for Medium and Large C&I programs for actual vendors. Spillover is m

associated with the program. 

4 The total surveyed kWh savings represents the total savings for all surveyed design professionals and surveyed vendors in the program tracking system 

suggested they were actual vendors, not participants.

5 Net of “like” spillover for the customers associated with the surveyed design professionals/vendors, as identified from the participating customer survey. 6 One VSD response suggested spillover but could not respond to the percentage question (VNP3). We imputed the percentage with the values from oth

respond to this question. Only one case was considered in the imputation, with a value of 50 percent.

7 “Other” is a residual category consisting of measures remaining from “Custom” after equipment was reassigned to existing categories such as “Motors,” 

as process equipment, process cooling equipment, and comprehensive chillers.
National Grid 

2007 Commercial and Industrial Programs 

Free-ridership and Spillover Study 

Final Executive Summary 

June 23, 2008 
National Grid 

2007 Commercial and Industrial Programs 

Free-ridership and Spillover Study 

Final Executive Summary 

June 23, 2008 

© PA Knowledge Limited 2008 

PA Consulting Group 

6410 Enterprise Lane 

Suite 300 

Madison 

Wisconsin 53719 

Tel: +1 608 443 2700

Fax: +1 608 661 5181

www.paconsulting.com

Prepared by: Laura Schauer 

Pam Rathbun 

Jeremy Kraft 

Eric Rambo 

Version: 1.0 
i 

National Grid 6/23/08 

TABLE OF CONTENTS 

1. Executive Summary: National Grid 1-1

1.1 Study Objectives 1-1

1.2 Study Methodology 1-1

1.3 Total Participant Free-ridership Estimates 1-2

1.4 Participant “Like” Spillover Estimates 1-7

1.5 Nonparticipant Spillover Estimates 1-11
1-1 

National Grid 6/23/08 

1. EXECUTIVE SUMMARY: NATIONAL GRID 

This Executive Summary summarizes the findings of the program year 2007 Commercial and 

Industrial Programs Free-ridership and Spillover Study for National Grid customers. The 

purpose of this study was to assess program free-ridership, participant spillover, and 

nonparticipant spillover for the following programs offered by National Grid: 

• Energy Initiative 

• Design 2000plus 

• Small Business Services 

The 2007 Free-ridership and Spillover Studies ran concurrently for National Grid, Cape Light 

Compact, United Illuminating, and Unitil.

1.1 STUDY OBJECTIVES 

The primary objective of the program year 2007 Commercial and Industrial Programs Freeridership and Spillover Study was to assist National Grid in quantifying the net impacts of their 

commercial and industrial energy efficiency programs by estimating the extent of: 

• Program free-ridership 

• Participant “like” spillover 

• Nonparticipant “like” spillover 

This executive summary provides the free-ridership, participant spillover and nonparticipant 

spillover estimates for 2007. First, a summary of the study methodology is provided. 

1.2 STUDY METHODOLOGY 

The methodology used for this year’s study follows the standardized methods developed in 

2003 for a group of Massachusetts energy efficiency program administrators1. 

To accomplish the above objective, telephone surveys were conducted with samples of 2007 

program participants in each of the programs and with design professionals and equipment 

vendors involved in these 2007 installations. The program participant sample consisted of 

unique electric utility accounts, not unique customer names. The same customer name, or 

business identity, can have multiple accounts in multiple locations, but program technical 

support and incentives are provided on behalf of an individual account. Thus, for the 

purposes of this study, a customer or participant is defined as a unique account. 

The majority of these telephone interviews were completed with program participants 

between March 5 and May 9, 2008. All sampled participating customers were mailed a letter 

on National Grid letterhead in advance of the telephone call. This letter explained the purpose 

 

1

 Pamela Rathbun, Carol Sabo, and Bryan Zent, Standardization Methods for Free-ridership and 

Spillover Evaluation—Task 5 Final Report (Revised), prepared for National Grid, NSTAR Electric, 

Northeast Utilities, Unitil, and Cape Light Compact, June 16, 2003. 
1. Executive Summary: National Grid... 

1-2

National Grid 6/23/08 

of the call, informed customers that someone would be calling them in the next couple of 

weeks to ask them some questions about their experience with the programs, and thanked 

them for their cooperation in advance. This advance letter and repeated call attempts resulted 

in a high response rate of 76 percent, which increases the level of confidence in the survey 

results. The duration of interviews with program participants averaged ten minutes. 

In addition to the customer surveys, surveys were conducted with: 

• Design professionals and vendors identified by customers as being the most 

knowledgeable about the decisions to install the equipment through National Grid’s 

Design 2000plus or Energy Initiative programs. These surveys were used for 

estimating free-ridership for those installations where the design 

professional/equipment vendor was more influential in the decision than the customer. 

• Design professionals and equipment vendors who had recommended, sold and/or 

installed equipment through National Grid’s Design 2000plus or Energy Initiative 

programs, as well as Unitil and United Illuminating’s Medium and Large Commercial 

design professionals and vendors. These surveys were used for estimating the extent 

of nonparticipant “like” spillover for the Sponsor’s programs. 

The number of survey completions for some measure categories is low because the number 

of installations within these measure categories for program year 2007 was small. Thus, 

although a high percentage of the 2007 program participants completed surveys, some 

caution should be used when interpreting the results. 

1.3 TOTAL PARTICIPANT FREE-RIDERSHIP ESTIMATES 

A program’s free-ridership rate is the percentage of program participants deemed to be free 

riders. A free rider refers to a customer who received an incentive through an energy 

efficiency program who would have installed the same or a smaller quantity of the same high 

efficiency measure on their own within one year if the program had not been offered. For free 

riders, the program is assumed to have had no influence or only a slight influence on their 

equipment purchase decision. Consequently, none or only some of the energy savings of 

equipment purchased by this group of customers should be credited to the energy efficiency 

program. Free riders account for costs, but not benefits, to the program, driving benefit-cost 

ratios down. 

For programs that offer monetary incentives for multiple measure categories (e.g., motors, 

lighting, HVAC), it is important to estimate free-ridership by specific measure category. 

Category-specific estimates produce feedback on the program at the level at which it actually 

operates and allow for cost-effectiveness testing by measure category. 

In addition, for commercial and industrial incentive programs, free-ridership has often been 

found to be highly variable among measure categories, making it essential to produce 

measure category-specific estimates. The ability to provide reliable estimates by measure 

category is dependent on the number of installations within that measure category—the fewer 

installations, the less reliable the estimation. 

It is also important to measure the extent of free-ridership for each customer. Pure free riders 

(100%) would have installed exactly the same quantity and type of equipment within one year 

in the absence of the program. Partial free riders (1-99%) are those customers who would 
1. Executive Summary: National Grid... 

1-3

National Grid 6/23/08 

have installed some equipment within one year on their own, but a smaller quantity and/or a 

lesser efficiency. Thus, the program had some impact on their decision. Non-free riders (0%) 

are those who would not have installed any high efficiency qualifying equipment within one 

year in the absence of the program services. The total free-ridership estimates in this report 

include pure, partial, and non-free riders. 

This year’s approach to estimating free riders follows the approach outlined in the 

Standardization Methods... report, which consists of a sequential question technique to 

identify free riders. This sequential approach asks program participants about the actions they 

would have taken if the program had not been offered. This approach is considered an 

accurate method of estimating the actual level of free-ridership among program participants 

because it addresses the program’s impact upon project timing, measure quantity, and 

efficiency levels while explicitly recognizing that the cost of energy-efficient equipment can be 

a barrier to installation in the absence of utility-sponsored energy efficiency programs. This 

method is also recommended because it walks survey respondents through their decision 

process with the objective of helping them recall the program’s impact upon all aspects of 

project decision-making. 

One issue with the method is how to handle responses of “don’t know.” The “don’t know” 

responses to the initial free-ridership question are assigned a free-ridership value of zero 

percent. For these cases, we then check their responses to the consistency questions and 

their response to open-ended question and adjust the free-ridership rate as appropriate. Note 

that program total free-ridership (pure and partial) rates illustrated in the following tables are 

weighted by measure category kWh savings as well as the disproportionate probability of 

being sampled. When reviewing the measure category free-ridership rates it is important to 

consider the number of survey completions that the estimate is based upon. 
1. Executive Summary: National Grid... 

1-4

National Grid 6/23/08 

Energy Initiative Participant Free-ridership Rates. Table 1-1 summarizes the total freeridership results overall and by measure category for 2007 Energy Initiative installations. The 

overall Energy Initiative program free-ridership for the 2007 program year was 10.5 percent, 

which is higher than the 8.9 percent rate found for 2005 installations, and higher than the 6.7 

percent rate found in 2004. The HVAC installations’ free-ridership rates dropped significantly 

in 2007, which reflects changes in program requirements related to HVAC efficiency levels. 

The rate was highest for VSD measures (33.1 percent), which changed significantly from 

previous years. A close review of the data shows that the relatively small population, and high 

savers with high free-ridership rates, drove the rate up for this measure category. 

Table 1-1

National Grid Energy Initiative Program Total Participant Free-ridership Rates 

All 2007 Installations 

Total Participant Free-ridership Rate

Measure Description # Accounts 

(Survey/Pop) 2007 

2007 90% 

Error 

Margin 

2005 2004 2002 

Custom: Process, 

HVAC, Drivepower, 

Lighting, O&M 

50/118 7.4% ±4.6% 8.4% 5.5% 10.9% 

Motors 6/6 21.0% ±0.0% 32.4% 15.2% 9.1% 

HVAC 23/29 12.5% ±5.5% 40.9% 0.3% 43.8% 

VSD 12/18 33.1% ±12.9% 2.0% 0.1% 0.5% 

Lighting: T8, Other 

Fluorescent, CFL, 

Controls, HID, LED Exit 

Signs 

93/397 10.2% ±4.5% 5.9% 7.5% 14.5% 

Compressed Air 15/19 5.7% ±4.5% 36.8% 26.4% 17.1% 

Overall Energy Initiative 

Program 178/558 10.5% ±3.1% 8.9% 6.7% 15.3% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 
1. Executive Summary: National Grid... 

1-5

National Grid 6/23/08 

Design 2000plus Participant Free-ridership Rates. Table 1-2 summarizes Design 

2000plus free-ridership results overall and by measure category for 2007 installations. The 

overall Design 2000plus program free-ridership rate for 2007 was 19.0 percent, which is lower 

than the 21.7 percent rate found for 2005 installations and slightly higher than the 18.1 

percent rate found for 2004 installations. As with the Energy Initiative program, HVAC 

measures’ free-ridership rates dropped significantly between 2005 and 2007. VSD and 

compressed air measures had the highest free-ridership rates. 

Table 1-2 

National Grid Design 2000plus Program Total Participant Free-ridership Rates 

All 2007 Installations 

Total Participant Free-ridership Rate

Measure Description # Accounts 

(Survey/Pop) 2007

2007 

90% 

Error 

Margin 

2005 2004 2002 

Custom: Process, 

HVAC, Drivepower, 

Lighting 

42/84 14.6% ±6.3% 10.1% 2.4% 16.6% 

Motor: New 37/88 28.8% ±9.3% 15.1% 15.5% 40.1% 

Motor: Failed/Stock 22/32 11.2% ±6.2% 23.5% 21.8% 23.2% 

HVAC (Unitary): 

Packaged A/C and 

Water Source Heat 

Pump 

55/124 14.8% ±5.9% 56.4% 5.3% 40.0% 

HVAC (Non-unitary): 

Programmable 

Thermostat, Energy 

Management System, 

Chiller, Control 

36/69 8.3% ±5.2% 49.1% 64.0% 39.5% 

VSD 10/16 58.7% ±15.7% 8.6% 43.5% 19.3% 

Lighting: T8, Other 

Fluorescent, CFL, 

Controls, HID, LED 

Exit Signs 

51/123 27.9% ±7.9% 28.2% 50.6% 36.3% 

Compressed Air 55/113 33.8% ±7.5% 32.7% 17.6% 20.6% 

Overall Design 

2000plus Program 241/516 19.0% ±3.0% 21.7% 18.1% 27.2% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 

Custom measures include six Comprehensive Program participants. The Comprehensive Program free-ridership 

rate was zero percent for 2007 installations. 
1. Executive Summary: National Grid... 

1-6

National Grid 6/23/08 

Small Business Services Participant Free-ridership Rates. Table 1-3 summarizes the 

results overall and by measure category for 2007 Small Business Services installations. The 

total free-ridership rate for 2007 was 5.5 percent, which is slightly higher than past years. 

Table 1-3 

National Grid Small Business Services Program Total Participant Free-ridership Rates 

All 2007 Installations 

Total Participant Free-ridership Rate

Measure Description # Accounts 

(Survey/Pop) 2007

2007 90% 

Error 

Margin 

2005 2004 2002 

Lighting: Fluorescent 

with ELIG/3’4’8’ Lamp 

& EEMAG, Standard 

Ballast, Exit Sign, 

Compact Fluorescent, 

HID 

243/1,329 5.8% ±2.2% 2.3% 1.0% 1.0% 

Non-lighting: Water 

Heater Wrap, 

Programmable 

Thermostat, 

Economizer 

64/155 3.6% ±2.9% 2.0% 1.3% 1.0% 

Overall Small 

Business Services 

Program 

284/1,441 5.5% ±2.0% 2.2% 1.0% 1.0% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 
1. Executive Summary: National Grid... 

1-7

National Grid 6/23/08 

State-level Participant Free-ridership Rates. Table 1-4 shows Massachusetts has a total 

2007 free-ridership rate of 12.7 percent, New Hampshire 21.1 percent, and Rhode Island 10.0 

percent. 

Table 1-4 

National Grid State-level Total Participant Free-ridership Rates 

All 2007 Installations 

Total Participant Free-ridership Rates 

Massachusetts New Hampshire Rhode Island 

Program

# 

Accounts 

Surveyed

2007 

Total 

Freeridership

90% 

Error 

Margin

# 

Accounts 

Surveyed

2007 

Total 

Freeridership

90% 

Error 

Margin 

# 

Accounts 

Surveyed

2007 

Total 

Freeridership

90% 

Error 

Margin

Energy 

Initiative 128 11.3% ±3.8% 8 24.2% ±11.1% 42 7.9% ±5.9% 

Design 

2000plus 154 18.6% ±3.8% 8 51.1% ±15.2% 79 18.1% ±5.1% 

Small 

Business 

Services 

127 6.9% ±3.4% 45 0.5% ±0.8% 112 4.3% ±2.8% 

ALL 392 12.7% ±2.4% 60 21.1% ±3.4% 228 10.0% ±2.8% 

Overall survey and population participant counts do not equal the sum of program survey and population

participant counts; the same participant may be represented in multiple programs. 

1.4 PARTICIPANT “LIKE” SPILLOVER ESTIMATES 

Spillover refers to additional energy-efficient equipment installed by a customer due to 

program influences but without any financial or technical assistance from the program. 

Participant “like” spillover refers to the situation where a customer installed equipment 

through the program in the past year and then installed additional equipment of the same type 

due to program influences. In contrast to free-ridership, spillover adds benefits to the program 

at no additional cost, increasing the program benefits and benefit-cost ratio. 

Survey free-ridership questions were followed by questions designed to measure "like" 

spillover. These questions asked about recent purchases (since program participation in 

2007) of any additional energy-efficient equipment of the same type as installed through the 

program that were made without any technical or financial assistance from the utility. A “like” 

spillover estimate was computed based on how much more of the same energy-efficient 

equipment the participant installed outside the program and did so because of their positive 

experience with the program.

One of the issues with attempting to quantify spillover savings is how to value the savings of 

measures installed outside the program since we are relying on customer self-reports of the 

quantity and efficiency of any measures installed. We used a conservative approach and 

reported only those measures installed outside the program that were of exactly the same 

type and efficiency as the ones installed through the program. Our conservative approach 

allowed customers to be more certain about whether the equipment they installed outside the 
1. Executive Summary: National Grid... 

1-8

National Grid 6/23/08 

program was the same type as the program equipment. This, in turn, makes it possible for us 

to use the estimated program savings for that measure to calculate the customer’s “like” 

spillover savings. 

Note that the “like” spillover rates illustrated in the following tables are weighted by measure 

category kWh savings and the disproportionate probability of being sampled. When reviewing 

the measure category “’like’ spillover,” it is important to consider the number of survey 

completions that the estimate is based upon. The number of survey completions for some 

measure categories is low because very few customers in the sample installed the measure. 

Thus, although a high percentage of the 2007 program participants completed surveys, some 

caution should be used when interpreting the results. 

Energy Initiative Participant “Like” Spillover Rates. Table 1-5 presents the “like” spillover 

rates for year 2007 Energy Initiative participants, overall and by measure category. The 

estimate of “like” spillover savings attributable to the overall Energy Initiative program for the 

purchase of like equipment outside of the program is 3.3%, which is slightly higher than 

previous years’ spillover rates. 

Table 1-5 

National Grid Energy Initiative Program Participant “Like” Spillover Rates 

All Year 2007 Installations 

Total Participant “Like” Spillover Rate 

Measure Description # Accounts 

(Survey/Pop) 2007 

2007 

90% 

Error 

Margin 

2005 2004 2002

Custom: Process, 

HVAC, Drivepower, 

Lighting, O&M 

50/118 8.5% ±4.9% 1.9% 0.7% 1.4% 

Motors 6/6 13.9% ±0.0% 1.7% 0.0% 2.5% 

HVAC 23/29 5.2% ±3.5% 0.3% 27.0% 7.9% 

VSD 12/18 0.0% ±0.0% 0.2% 16.2% 12.0% 

Lighting: T8, Other 

Fluorescent, CFL, 

Controls, HID, LED 

Exit Signs 

93/397 1.8% ±2.0% 1.4% 0.4% 2.1% 

Compressed Air 15/19 0.0% ±0.0% 0.0% 0.0% 0.0% 

Overall Energy 

Initiative Program 178/558 3.3% ±1.8% 1.4% 2.7% 2.7% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 
1. Executive Summary: National Grid... 

1-9

National Grid 6/23/08 

Design 2000plus Participant “Like” Spillover Rates. Table 1-6 presents the “like” spillover 

rates for year 2007 Design 2000plus installations overall and by measure category. The

overall Design 2000plus program spillover rate for the 2007 program year was 5.9 percent, 

which is lower than 8.8 percent rate found for 2005 installations but higher than 2004 and 

2002 rates. 

Table 1-6 

National Grid Design 2000plus Program Participant “Like” Spillover Rates 

All Year 2007 Installations 

Total Participant “Like” Spillover Rate

Measure Description # Accounts 

(Survey/Pop) 2007 

2007 

90% 

Error 

Margin 

2005 2004 2002 

Custom: Process, HVAC, 

Drivepower, Lighting 42/84 3.7% ±3.4% 11.3% 1.8% 1.0% 

Motor: New 37/88 9.2% ±5.9% 5.3% 2.4% 8.4% 

Motor: Failed/Stock 22/32 4.5% ±4.0% 10.4% 3.4% 21.4% 

HVAC (Unitary): 

Packaged A/C and Water 

Source Heat Pump 

55/124 5.9% ±3.9% 6.6% 2.5% 2.3% 

HVAC (Non-unitary): 

Programmable 

Thermostat, Energy 

Management System, 

Chiller, Control 

36/69 15.2% ±6.8% 0.2% 1.4% 4.1% 

VSD 10/16 0.0% ±0.0% 0.0% NA 3.9% 

Lighting: T8, Other 

Fluorescent, CFL, 

Controls, HID, LED Exit 

Signs 

51/123 13.4% ±6.0% 8.4% 0.3% 1.1% 

Compressed Air 55/113 0.0% ±0.0% 0.2% 1.0% 0.2% 

Overall Design 

2000plus Program 241/516 5.9% ±1.8% 8.8% 1.4% 2.0% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 

Custom measures include six Comprehensive Program participants. The Comprehensive Program spillover rate 

was zero percent for 2007 installations. 
1. Executive Summary: National Grid... 

1-10

National Grid 6/23/08 

Small Business Services Participant “Like” Spillover Rates. Table 1-7 summarizes the 

“like” spillover rates for year 2007 Small Business installations overall and by measure 

category. The overall Small Business Services program spillover rate was 2.0 percent, which 

is comparable with previous years’ rates. 

Table 1-7 

National Grid Small Business Services Program Participant “Like” Spillover Rates 

All Year 2007 Installations 

Total Participant “Like” Spillover Rate 

Measure Description # Accounts 

(Survey/Pop) 2007 

2007 

90% 

Error 

Margin 

2005 2004 2002 

Lighting: Fluorescent 

with ELIG/3’4’8’ Lamp & 

EEMAG, Standard 

Ballast, Exit Sign, 

Compact Fluorescent, 

HID 

243/1,329 2.2% ±1.4% 2.0% 0.5% 2.0% 

Non-lighting: Water 

Heater Wrap, 

Programmable 

Thermostat, Economizer

64/155 0.7% ±1.3% 1.0% 0.3% 0.6% 

Overall Small 

Business Services 

Program 

284/1,441 2.0% ±1.2% 1.9% 0.4% 1.9% 

Overall survey and population participant counts do not equal the sum of measure category survey and population 

participant counts; the same participant may be represented in multiple measure categories. 
1. Executive Summary: National Grid... 

1-11

National Grid 6/23/08 

State-level Participant “Like” Spillover Rates. Table 1-8 shows the “like” spillover rate for 

Massachusetts is 3.5 percent, New Hampshire is 2.1 percent, and Rhode Island is 4.6 

percent. The surveyed number represents the number of accounts surveyed within each 

state. 

Table 1-8 

National Grid State-level Program Participant “Like” Spillover Rates by Account 

All Year 2007 Installations 

Total Participant “Like” Spillover Rates 

Massachusetts New Hampshire Rhode Island 

Program # 

Accounts 

Surveyed

2007 

Spillover

90% 

Error 

Margin

# 

Accounts 

Surveyed

2007 

Spillover

90% 

Error 

Margin

# 

Accounts 

Surveyed

2007 

Spillover

90% 

Error 

Margin

Energy 

Initiative 128 4.3% ±2.4% 8 0.0% — 42 0.4% ±1.4% 

Design 

2000plus 154 2.2% ±1.5% 8 0.0% — 78 14.2% ±4.6% 

Small 

Business 

Services 

127 2.5% ±2.1% 45 3.9% ±2.1% 112 1.1% ±1.5% 

ALL 392 3.5% ±1.3% 60 2.1% ±1.4% 228 4.6% ±2.0% 

Overall survey and population participant counts do not equal the sum of program survey and population

participant counts; the same participant may be represented in multiple programs. 

1.5 NONPARTICIPANT SPILLOVER ESTIMATES 

Nonparticipant spillover refers to energy efficient measures installed by program 

nonparticipants due to the program's influence. The program can have an influence on design 

professionals and vendors as well as an influence on product availability, product acceptance, 

customer expectations, and other market effects, all of which may induce nonparticipants to 

buy high efficiency products. Total nonparticipant spillover would also include responses from 

nonparticipating designers and vendors. 

The methodology for the 2007 study estimated only a portion of nonparticipant like-measure 

spillover based on responses from design professionals and vendors participating in National 

Grid, United Illuminating, and Unitil’s Medium and Large Commercial programs2. Cape Light 

Compact vendors were not included in this study due to insufficient data; however, two of the 

three vendors that were indicated within Cape Light Compacts’ vendor data overlapped with 

National Grid’s vendor sample and were surveyed. 

The data for the analysis could have been collected from nonparticipants directly or from the 

design professionals and vendors who recommended, sold, and/or installed qualifying high 

 

2

Nonparticipant spillover for small business programs was not estimated because of the small number of vendors involved in 

delivering the program. 
1. Executive Summary: National Grid... 

1-12

National Grid 6/23/08 

efficiency equipment. We chose to survey the design professionals and vendors primarily 

because they could typically provide much more accurate information about the efficiency 

level of installed equipment than could the nonparticipants. Experience has shown that 

customers cannot provide enough data about the new equipment they have installed to allow 

for accurate estimates of the energy savings achieved from the equipment. While they usually 

can report what type of equipment was installed, they typically cannot provide sufficient 

information about the quantity, size, efficiency, and/or operation of that equipment to allow us 

to determine whether the equipment is "program-eligible." On the other hand, design 

professionals and equipment vendors who have worked with the program are typically more 

knowledgeable about equipment and are familiar with what is and is not "program-eligible." 

Another argument in favor of using design professionals and equipment vendors to estimate 

nonparticipant spillover was that we could use data in the program tracking system database 

to attach kWh savings estimates to nonparticipant spillover. In the program tracking system 

database, measure-specific program kWh savings are associated with each design 

professional and vendor who participated in the program in 2007. 

To determine nonparticipant spillover, design professionals and equipment vendors were 

asked (by measure category they installed in the program) what percent of their sales were 

program-eligible and what percent of these sales did not receive an incentive through the 

programs. They were then asked about the program’s impact on their decision to 

recommend/install this efficient equipment outside the program. Using the survey responses 

and measure savings data from the program tracking system, the participating vendor 

nonparticipant like spillover savings could be estimated for each design professional/vendor 

and the results extrapolated to the total program savings. 

This method of estimating nonparticipant spillover is a conservative estimate for two reasons. 

First, not all design professionals and equipment vendors who are familiar with the programs 

specified and/or installed equipment through the program in 2007. Thus, we miss any 

nonparticipant spillover that was associated with these other design professionals/vendors 

(although it is less likely these design professionals/vendors had nonparticipant spillover if 

they were not involved with the program in 2007). 

Second, this method only allows us to extrapolate nonparticipant spillover for those same 

measure categories that a particular design professional/vendor was associated with for the 

2007 programs. Thus, if a vendor installed program-eligible equipment in other measure 

categories in the year 2007 outside the program, but none through the program, we did not 

capture nonparticipant spillover savings with that particular type of equipment. In essence, we 

measured only "like" nonparticipant spillover; that is, spillover for measures like those 

installed through the program in 2007. 

The nonparticipant spillover results for the Medium and Large Commercial and Industrial 

programs are based on surveys with 106 design professionals and vendors out of a 

population of 237 National Grid, United Illuminating, and Unitil vendors. Because of the 

significant overlap in sponsors’ territories, as well as vendors across sponsors, we report the 

results in aggregate rather than by sponsor. The analysis indicates that the combined 

nonparticipant spillover from the medium and large commercial and industrial programs 

amounted to 2,603,307 kWh in the 2007 program year, which is approximately 2.6 percent% 

of the total savings produced by these programs combined (Table 1-9). 
1. Executive Summary: National Grid... 

Table 1-9 

Nonparticipant “Like” Spillover Results for Program Year 2007 

National Grid, Unitil, and United Illuminating Vendors

A B C D E F G H 

Survey 

Categories 

Vendor 

Population 

kWh 

Savings3

Number of Firms 

Surveyed with 

kWh Savings/ 

Number of Firms 

in Program with 

kWh Savings 

Surveyed 

kWh 

Savings4

Surveyed 

Savings 

Coverage 

Rate 

(D/B) 

Nonparticipant 

Spillover from 

Surveyed 

Firms (kWh)5

Estimated 

Spillover 

Percent 

(F/D) 

90% CI 

Motors 102,873 6/16 38,077 37.0% 0 0.0% 0HVAC 10,877,314 27/60 2,175,565 20.0% 79,149 3.6% 0.7% to 3VSD6 2,393,842 11/28 802,202 33.5% 115,569 14.4% 4.8% to 25Lighting 56,560,136 60/151 20,074,391 35.3% 603,572 3.0% 1.0% to 4Compressed Air 4,671,464 10/22 1,743,112 37.3% 60,498 3.5% 1.3% to 6Refrigeration 4,758,046 2/6 1,197,312 25.2% 0 0.0% 0Other7 19,474,884 10/33 4,998,940 25.7% 0 0.0% 0Total 98,838,559 106/254 31,029,599 31.4% 858,788 2.6% 1.0% to 3

3 The vendor population kWh savings represents the total savings for all measures for Medium and Large C&I programs for actual vendors. Spillover is m

associated with the program. 

4 The total surveyed kWh savings represents the total savings for all surveyed design professionals and surveyed vendors in the program tracking system 

suggested they were actual vendors, not participants.

5 Net of “like” spillover for the customers associated with the surveyed design professionals/vendors, as identified from the participating customer survey.6 One VSD response suggested spillover but could not respond to the percentage question (VNP3). We imputed the percentage with the values from oth

respond to this question. Only one case was considered in the imputation, with a value of 50 percent.

7 “Other” is a residual category consisting of measures remaining from “Custom” after equipment was reassigned to existing categories such as “Motors,” 

as process equipment, process cooling equipment, and comprehensive chillers.