U.S. ENVIRONMENTAL PROTECTION AGENCY 

OFFICE OF INSPECTOR GENERAL 

Catalyst for Improving the Environment 

 Evaluation Report 

Improvements Needed to Validate 

Reported ENERGY STAR Benefits 

 Report No. 09-P-0061 

 December 17, 2008 
Report Contributors: Laurie Adams 

 Natasha Besch-Turner 

 Daniel Carroll 

 Jerri Dorsey

 Jeffrey Harris 

Abbreviations 

CCAP Climate Change Action Plan 

CFLs Compact Fluorescent Lights 

DOE Department of Energy 

DSM demand-side management 

EPA U.S. Environmental Protection Agency 

kWh kilowatt hours 

MMTCE million metric tons of carbon equivalent 

OIG Office of Inspector General 

PBC public benefits charge 

Cover art:	 The illustrations depict the four primary components of EPA’s 

ENERGY STAR program, from left: products, commercial, 

industrial, and residential (from EPA). 
U.S. Environmental Protection Agency 09-P-0061 

December 17, 2008 Office of Inspector General 

At a Glance
 

Catalyst for Improving the Environment 

Why We Did This Review 

We initiated this review to 

examine the validity and 

accuracy of the reported 

energy savings for the U.S. 

Environmental Protection 

Agency’s (EPA’s) ENERGY

STAR program. This was part 

of our agenda to assess new 

approaches to environmental 

protection. We specifically

sought to determine whether 

the savings reported were

valid and fully supportable. 

Background 

ENERGY STAR is a 

voluntary program designed to 

help businesses and 

individuals enhance their 

energy efficiency. In 2006, 

the ENERGY STAR 

program reported avoiding a 

total of 37.6 million metric 

tons of carbon equivalent. It 

further reported that ENERGY 

STAR helped prevent 

greenhouse gas emissions 

equivalent to those from 

25 million vehicles while 

savings Americans $14 billion 

on their energy bills. 

For further information, 

contact our Office of 

Congressional, Public Affairs, 

and Management at 

(202) 566-2391. 

To view the full report, 

click on the following link: 

www.epa.gov/oig/reports/2009/ 

20081217-09-P-0061.pdf 

Improvements Needed to Validate Reported 

ENERGY STAR Benefits 

What We Found 

Reported ENERGY STAR benefits represented one-half of EPA’s total 

greenhouse gas emissions avoided in 2006. ENERGY STAR benefits are a major 

component of efforts reducing such emissions. The accuracy of the program’s 

reported energy savings is important in monitoring the United States’ efforts to 

reduce greenhouse gas emissions. 

We found the ENERGY STAR program’s reported savings claims were inaccurate 

and the reported annual savings unreliable. We identified several deficiencies 

with the shipment data and the process used in calculating benefits. Deficiencies 

included the lack of a quality review of the data collected; reliance on estimates, 

forecasting, and unverified third party reporting; and the potential inclusion of 

exported items. Also, EPA included savings for one Department of Energy (DOE) 

product that DOE also claimed. 

Additionally, sales of formerly qualified products are used to determine ENERGY 

STAR’s market transformation benefits, but we found that this benefit was 

computed inconsistently. Also, the methodology used to compute the ENERGY 

STAR commercial sector benefits uses unverified assumptions. 

What We Recommend 

We recommended that EPA: 

•	 Establish and implement improved quality controls. 

•	 Develop and consistently apply a data-driven methodology to compute 

market transformation effects. 

•	 Validate the model for calculating the benefits of the ENERGY STAR 

commercial sector to ensure it accurately reflects the sector’s impacts. 

EPA disagreed with many of our conclusions, but stated it had implemented some 

of the recommendations. However, some of EPA’s planned actions do not meet 

the intent of our recommendations, and we consider these recommendations open 

and unresolved. 
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

OFFICE OF 

INSPECTOR GENERAL 

December 17, 2008 

MEMORANDUM 

SUBJECT:	 Improvements Needed to Validate Reported ENERGY STAR Benefits 

 Report No. 09-P-0061 

FROM:	 Wade T. Najjum 

Assistant Inspector General for Program Evaluation 

TO: Robert J. Meyers 

Principal Deputy Assistant Administrator 

Office of Air and Radiation 

This is our report on the ENERGY STAR benefits evaluation conducted by the Office of 

Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report 

contains findings that describe the problems the OIG has identified and corrective actions the 

OIG recommends. This report represents the opinion of the OIG and does not necessarily 

represent the final EPA position. Final determinations on matters in this report will be made by 

EPA managers in accordance with established audit resolution procedures. 

The estimated cost of this report – calculated by multiplying the project’s staff days by the 

applicable daily full cost billing rates in effect at the time – is $538,867. 

Action Required 

In accordance with EPA Manual 2750, you are required to provide a written response to this 

report within 90 calendar days. You should include a corrective actions plan for agreed upon 

actions, including milestone dates. We have no objections to the further release of this report to 

the public. This report will be available at http://www.epa.gov/oig. 

If you or your staff has any questions regarding this report, please contact me at 202-566-0827; 

or Jeffrey Harris, Director of Cross Media, at 202-566-0831 or harris.jeffrey@epa.gov. 
Improvements Needed to Validate 09-P-0061 

Reported ENERGY STAR Benefits 

Table of Contents 

Chapters 

1 Introduction ........................................................................................................ 1 

Purpose ...................................................................................................... 1 

Background ................................................................................................. 1 

 Noteworthy Achievements........................................................................... 3 

 Scope and Methodology.............................................................................. 4 

 Prior Review ............................................................................................... 5 

2 Inaccurate and Unsupported Data Used to Calculate 

2006 ENERGY STAR Product Annual Benefits ............................................... 6 

Process Used to Calculate Product Sector Savings.................................... 6 

Shipment Totals Used in Savings Calculations Are Unreliable ................... 8 

Other Factors Contribute to Uncertainties........................................... 10 

DOE Product Included in EPA ENERGY STAR Reported Savings ............ 12 

Conclusion................................................................................................... 13 

 Recommendation ........................................................................................ 13 

Agency Comments and OIG Evaluation...................................................... 14 

3 ENERGY STAR Market Transformation Benefits 

 Calculated Inconsistently.................................................................................. 15 

Market Transformation Caused by Innovations in Efficiency....................... 15 

Market Transformation Benefits Calculated with Assumed Data ................ 16 

Market Transformation Obscures Program Accomplishments .................... 17 

Conclusion................................................................................................... 18 

Recommendation ........................................................................................ 18 

Agency Comments and OIG Evaluation...................................................... 18 

4 ENERGY STAR Commercial Benefits Model Utilizes 

 Unverified Assumptions ................................................................................... 20 

Formula Used to Calculate Commercial Sector Savings............................. 20 

Methodology Not Tailored for ENERGY STAR ....... .................................. 20 

Formula Methodologies and Assumptions Need to Be Verified .................. 23 

Conclusion................................................................................................... 23 

 Recommendation ........................................................................................ 24 

Agency Comments and OIG Evaluation...................................................... 24 

Status of Recommendations and Potential Monetary Benefits.............................. 25 

- continued -
Improvements Needed to Validate 09-P-0061 

Reported ENERGY STAR Benefits 

Appendices 

A Logic Model for ENERGY STAR Products ...................................................... 26 

B Logic Model for ENERGY STAR Commercial & Industrial Programs ........... 27 

C Details on Scope and Methodology.................................................................. 28 

D Agency Comments and OIG Evaluation........................................................... 30 

E Distribution ......................................................................................................... 43 
 09-P-0061
 

Chapter 1

Introduction 

Purpose 

We initiated this review to examine the validity and accuracy of the reported 

energy savings for the U.S. Environmental Protection Agency’s (EPA’s) 

ENERGY STAR program. This was part of our agenda to assess new approaches 

to environmental protection. We specifically sought to determine whether the 

savings reported in 2006 were valid and fully supportable. 

Background 

In 1990, Section 103(g) of the Clean Air Act directed the EPA Administrator to 

“conduct a basic engineering research and technology program to develop, 

evaluate, and demonstrate non regulatory strategies and technologies for air 

pollution prevention.” In 1992, EPA’s Office of Air and Radiation established the 

ENERGY STAR Product Labeling Program (the ENERGY STAR program) as an 

innovative approach to environmental protection. Congress formally authorized 

the ENERGY STAR program in the Energy Policy Act of 2005.1 

ENERGY STAR is a voluntary program designed to help businesses and 

individuals protect the environment through superior energy efficiency. The 

ENERGY STAR program was designed to overcome selected market barriers. 

The program was first introduced to recognize and promote energy-efficient 

computers. It has since grown to cover many additional consumer products and 

services. In 1996, EPA partnered with the Department of Energy (DOE) to 

promote the ENERGY STAR label and broaden the range of products covered.2 

The ENERGY STAR program is one part of a larger U.S. government-wide 

agenda to address climate change. In 2002, the President announced a goal of 

reducing America’s greenhouse gas intensity 18 percent by 2012. EPA’s 

programs are expected to contribute over 70 percent of the emissions reductions 

needed to meet the President’s greenhouse gas intensity goal. ENERGY STAR is 

the most significant of the EPA's greenhouse gas avoidance programs. In 2006, 

the program accounted for over 50 percent of EPA's contribution. The success of 

ENERGY STAR is therefore central to whether the United States will meet the 

President’s goal for reducing greenhouse gas intensity. 

1

 Public Law 109-58-Aug. 8, 2005; Subtitle C-Energy Efficient Products, Sec. 131. ENERGY STAR program. 2

 A Memorandum of Cooperation was signed jointly on May 29, 1996. 

1 
 09-P-0061
 

The ENERGY STAR program reported avoiding a total of 37.6 million metric 

tons of carbon equivalent (MMTCE) in 2006. According to the EPA, the program 

helped prevent greenhouse gas emissions equivalent to those from 25 million 

vehicles while saving Americans $14 billion on their energy bills. ENERGY 

STAR has four program sectors: products, commercial, industrial, and residential. 

The 2006 reported carbon emissions avoided by program sector are illustrated in 

Figure 1-1; the paragraphs that follow describe the four sectors. 

Figure 1-1: Carbon Emissions Avoided by Sector for 2006 

Industrial 17.3% 

Commercial 

40.4 %

Products 41.2% 

Residential 

1.1 % 

Source: OIG Analysis of ENERGY STAR 2006 Annual Report 

Products 

According to ENERGY STAR staff the product sector is designed to promote 

energy-efficient products for purchase by consumers. This sector is the original 

component of EPA’s ENERGY STAR program and is still the most recognizable 

part of the program.3

 EPA reports that approximately 1,700 manufacturers are 

using the ENERGY STAR label on over 40,000 product models across more than 

50 product categories. Annually, consumers are reportedly purchasing 

approximately 300 million ENERGY STAR-qualified products and have invested 

in over 2 billion products since 1992. ENERGY STAR reported within its 2006 

annual report that the product sector was responsible for 15.5 MMTCE emissions 

avoided. 

3

 A logic model illustrating how the ENERGY STAR program is designed to promote energy efficiency among 

consumer products is depicted in Appendix A. 

2 
 09-P-0061
 

Commercial and Industrial Sectors 

The commercial and industrial sectors are designed to promote superior corporate 

energy management approaches and provide partners guidance on assessing 

current energy use and developing plans that will lead to energy reductions and 

overall energy efficiency. Annually, commercial and industrial buildings use 

approximately $200 billion worth of electricity and natural gas, which results in 

nearly half of the U.S. greenhouse gas emissions. ENERGY STAR reported 

within the 2006 annual report that the commercial and industrial sectors avoided 

combined total emissions of 21.7 MMTCE.4 EPA reported that increasing energy 

prices and greater concern for the environment and global warming have recently 

increased interest and participation in ENERGY STAR’s commercial and 

industrial program efforts. 

Residential Sector 

The residential sector is designed to help make residential homes more energy 

efficient. This sector is composed of two primary components: home 

improvements and new homes. ENERGY STAR reported that households can 

reduce their energy use up to 30 percent and save $600 annually on utility bills by 

using qualified products and services. In the 2006 annual report, EPA reported 

that ENERGY STAR’s residential sector avoided total emissions of 0.4 MMTCE. 

Noteworthy Achievements 

According to ENERGY STAR, the program has grown steadily in terms of the 

energy efficient solutions it offers, the variety of partners, and the benefits it 

delivers. EPA has been successful in marketing the ENERGY STAR brand and 

label. In 2007, EPA reported that 74 percent of households nationwide recognize 

the ENERGY STAR label and 62 percent associated the label with “energy 

efficiency or energy savings.” 

Since 2000, the ENERGY STAR program has consistently outperformed its 

annual goals, as illustrated in Figure 1-2. 

4

 A logic model illustrating how the ENERGY STAR program is designed to promote energy efficiency among 

commercial buildings is depicted in Appendix B. 

3 
 09-P-0061
 

Figure 1-2: ENERGY STAR Goals Are Surpassed 

0 

5 

10 

15 

20 

25 

30 

35 

40 

MMTCE Avoided 

2000 2001 2002 2003 2004 

Year 

2005 2006 

Goal Reported 

Source: ENERGY STAR and Other Climate Protection Partnership Annual Reports, 

2000 through 2006 

The ENERGY STAR program’s overall reported achievements in 2006 exceeded 

the program’s goals as illustrated in Table 1-1. 

Table 1-1: ENERGY STAR 2006 Goals and Achievements by Program 

ENERGY STAR Program 

2006 Goal 

(MMTCE Avoided) 

2006 Achievements 

(MMTCE Avoided) 

Product Labeling 14.5 15.5 

Residential 0.5 0.4 

Commercial Buildings 11.5 15.2 

Industrial 3.7 6.5 

Total 30.2 37.6 

Source: Based on data presented in the ENERGY STAR and Other Climate Protection 

Partnerships 2006 Annual Report 

Scope and Methodology 

We conducted this performance evaluation in accordance with generally accepted 

government auditing standards. Those standards require that we plan and perform 

the evaluation to obtain sufficient, appropriate evidence to provide a reasonable 

basis for our findings and conclusions based on our objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based upon our objectives. We performed our field work from 

January through September 2008. 

Our review included an examination of applicable laws and regulations as well as 

Agency guidance. We reviewed those internal controls that were relevant to our 

objectives. We reviewed ENERGY STAR annual reports, and Agency guidance 

4 
 09-P-0061
 

documents. We reviewed planning documents, including logic models. We met 

with Agency staff and contractors. We reviewed the methodologies governing the 

savings benefit calculations for the commercial and products program sectors.5 

Additionally, for the products sector, we reviewed documentation in support of 

the 2006 reported savings benefits. 

Appendix C provides further details on our scope and methodology. 

Prior Review 

An EPA Office of Inspector General (OIG) report, ENERGY STAR Program Can 

Strengthen Controls Protecting the Integrity of the Label,6 reported that 

improvements should be made to better assure the integrity of the ENERGY 

STAR label for home and office products. Criteria for revising specifications 

were unclear, and EPA did not have reasonable assurance that the selfcertification process was effective. EPA relied on some alternative verification 

mechanisms, but lacked any quality assurance or review of these reported results. 

EPA did not agree with all of the conclusions, but agreed to implement the OIG 

recommendations to strengthen management controls to protect the integrity of 

the ENERGY STAR label. 

5

 The residential and industrial program sectors were excluded from our review. The commercial and products
 

sector combined encompassed 81.6 percent of the 2006 reported carbon emissions avoided. 

6

 EPA OIG Report No. 2007-P-00028, issued August 1, 2007, at www.epa.gov/oig/reports/2007/20070801-2007-P00028.pdf. 

5 
 09-P-0061
 

Chapter 2

Inaccurate and Unsupported Data Used to Calculate 

2006 ENERGY STAR Product Annual Benefits 

The ENERGY STAR program reported savings for 2006 that were inaccurate or 

unsupported. A key component of the program’s annual and lifetime savings 

calculation process for consumer products is the total ENERGY STAR shipments 

for that year. We identified several concerns with shipment data, including: lack 

of quality review of the data submittals; reliance on estimates, forecasting, and 

third party reporting; and the potential inclusion of international shipments with 

domestic ENERGY STAR product shipments. When annual shipment data are not 

supportable or reliable, the validity of the savings calculation process is in 

question. 

Process Used to Calculate Product Sector Savings 

The product sector’s annual savings are computed using the Climate Change 

Action Plan (CCAP) model. This model is used to compute the energy savings for 

each qualified EPA ENERGY STAR product. A key number in the computation 

is the annual ENERGY STAR-qualified shipment total by product category.7 

For the majority of the EPA product categories (29 of 37), the manufacturers 

annually submit shipment data in one of two ways. They either submit data 

directly via unit shipment data submittal forms or indirectly through third party 

associations. One EPA contractor has sole responsibility for the receipt of these 

data and the production of the annual summary report. This report includes the 

overall numbers of qualified ENERGY STAR units shipped by product category, 

which are used in the annual savings calculation process. Figure 2-1 depicts the 

process. Figure 2-2 illustrates how the annual shipments per qualified ENERGY 

STAR product category are multiplied by the unit energy savings8

 to derive the 

annual energy savings attributable to the program. 

7

 The ENERGY STAR program uses the term shipments interchangeably with sales. When a qualified ENERGY 

STAR manufacturer ships its ENERGY STAR-qualified product, the shipment is considered sold for annual savings 

calculation purposes. For the reporting period reviewed, EPA did not require manufacturers to submit annual 

shipment data for office products or for programmable thermostats. Office products consist of: computers, copiers, 

fax machines, mailing machines, multifunction devices, printers, and scanners. 

8

 Unit energy savings for each product type is the difference between the business-as-usual unit energy consumption 

and the ENERGY STAR unit energy consumption in a given year. Unit energy savings change over time for most 

product types due to specification revisions, usage pattern changes, and changes to the business-as-usual efficiency. 

6 
 09-P-0061
 

Figure 2-1: ENERGY STAR Products Annual Savings Calculation Process 

Partners Report 

Shipment Data to EPA* 

EPA and two 

contractors review 

and update key 

parameters such 

as product lifetime, 

Business-as-usual 

projections 

Contractor updates 

CCAP model* EPA reviews model 

Energy savings from 

existing program 

stock + (2006 

Product Shipments 

due to program x Unit 

Energy Consumption 

– 2006 Adjustments) 

(CCAP model results 

as supplied by 

contractor are 

modified by 

adjustment factors) 

EPA prepares 

adjustments to reflect 

understanding of 

third party promotion 

1. Data incorporated 

into Annual Report 

2. Summary Memos 

written 

Final 2006 Results 

75.2 billion kWh 

58.5 trillion British 

thermal units natural 

gas and other fuels 

Legend 

Pink - Contractor/Partner 

Actions 

Blue – EPA Actions 

Orange – Methodology 

Formula 

Yellow – Final 2006 Results 

Green – Post-Analysis 

Actions 

*Manufacturer submits data directly to EPA contractor or via third party associations 

Source: EPA ENERGY STAR Immediate Office 

Figure 2-2: ENERGY STAR Products Annual Savings Calculation 

Current Year 

ENERGY STAR 

Qualified Shipments 

X 

Current Year 

Unit Energy Savings 

Current Year 

Annual 

Energy 

Savings 

= + = 

Total 

Annual 

Energy 

Savings 

Savings from 

Products Sold in 

Previous Years 

Source: OIG based on ENERGY STAR program materials 

The total annual saving calculation also includes a portion of savings computed 

for products shipped in previous years but are assumed to still be in use. EPA 

continues to claim savings for shipped qualified products over the estimated 

lifetime of the product. The lengths for which the lifetime annual savings benefits 

are calculated vary from 4 to 20 years after initial product shipment. 

Additionally, the total annual savings calculation includes an estimate of benefits 

from sales of products that previously met ENERGY STAR specifications but no 

longer qualify (further details are in Chapter 3). 

7 
 09-P-0061
 

Shipment Totals Used in Savings Calculations Are Unreliable 

2006 Annual Shipment Data are Irreconcilable 

We found that the annual shipment data used for the 2006 savings calculations 

were unsupportable. Based on our review of the documents provided, we were 

unable to reconcile the original shipment data with final reported totals for 18 of 

the 29 product categories. These reported totals are used to calculate ENERGY 

STAR program savings. Shipment totals for 13 product categories were 

understated and 5 were overstated. The results of our analysis are in Table 2-1. 

Table 2-1: OIG Reconciliation of Reported Product Shipment Totals for 

18 of 29 ENERGY STAR Categories 

ENERGY STAR 

Qualified Product 

2006 Unit Shipments 

Used in Benefits 

Calculation9 

2006 Unit Shipment 

Totals Reported by 

Manufacturers and 

Associations10 Difference Impact 

Audio/DVD 3,572,316 3,881,346 -309,030 Understated 

Boilers 382,770 405,573 -22,803 Understated 

Commercial Hot 

Food Holding 

Cabinets 

17,059 17,403 -344 Understated 

Commercial 

Refrigerators & 

Freezers 

83,493 106,658 -23,165 Understated 

Dehumidifiers 1,203,216 776,431 +426,785 Overstated 

End Use Products 

with Qualified 

External Power 

Supplies 

18,446,331 19,623,346 -1,177,015 Understated 

Exit Signs 1,425,554 3,450,117 -2,024,563 Understated 

External Power 

Supplies 

128,849,173 128,939,237 -90,064 Understated 

Furnaces 1,172,843 1,170,124 +2,719 Overstated 

Monitors 13,859,666 34,581,203 -20,721,537 Understated 

Residential Light 

Fixtures 

11,361,701 11,363,633 -1,932 Understated 

Roofing - 

Commercial Gallons 

21,352,516 15,966,045 +5,386,471 Overstated 

Roofing - 

Commercial & 

Residential Sq. Ft. 

1,916,655,032 2,227,937,102 -311,282,070 Understated 

Room Air Cleaners 287,354 289,815 -2,461 Understated 

Telephony 6,761,627 7,006,979 -245,352 Understated 

Transformers 

(Commercial & 

Industrial) 

173,390 166,669 +6,721 Overstated 

TV/VCR 17,685,967 13,166,140 +4,519,827 Overstated 

Ventilating Fans 853,406 857,232 -3,826 Understated 

Source: OIG analysis of shipment data submittals 

9

 Per ENERGY STAR Unit Shipment Data Report Calendar Year 2006, July 1, 2007. These figures are reported by
 

the contractor and used in the annual savings calculation process. 10 Total compiled by OIG based on supporting documents (manufacturer data shipment submittal forms and
 

third party association summary spreadsheets) provided by contractor. 

8 
 09-P-0061
 

According to EPA’s contractor, the differences may have been caused by a 

number of reasons, including: 

•	 Subsequent to shipment data submittals, manufacturers may have called to 

update or correct previously provided written submissions. 

•	 The contractor may have called the manufacturer to follow up concerning 

the data submittal and new information received. 

•	 Some private labelers11 and original manufacturers submit the shipment 

data for the same products, which causes double reporting of shipments. 

•	 Some manufacturers who reported shipments were not qualified 

manufacturers for that product category. 

•	 Manufacturers made errors in reporting. 

However, the contractor could not provide evidence in support of these possible 

explanations. 

If the shipment data used in the annual savings calculation are incorrect or 

unsupportable, the Agency may be overstating or understating the savings 

benefits for the program. For example, in our reconciliation of the 2006 shipment 

data, the monitor category was understated by nearly 21 million unit shipments or 

150 percent. Monitors are one of the six ENERGY STAR products that account 

for 70 percent of the ENERGY STAR product sector carbon reductions to date. 

TV/VCR, another one of the six large contributors to carbon reductions, was 

overstated by approximately 4.5 million unit shipments or 26 percent. 

ENERGY STAR overall program benefit are the sum of the savings calculated for 

each individual product. The importance of accurate data is discussed within a 

recently published article12 that summarizes the methodology used to compute the 

annual ENERGY STAR product program sector benefits. The article stated: 

We implement the bottom-up model with awareness that uncertainty 

for each product type contributes to uncertainty in total ENERGY 

STAR impacts. This means that many small inaccuracies are 

additive overall and any one inaccuracy for a product type with 

large energy savings can significantly affect the overall results. 

We could not reconcile shipment data in 2006 for 62 percent of the product 

categories reviewed. The significance of this deficiency extends beyond a single 

year’s benefits calculation. The deficiency impacts the savings calculations for 

future years as well because savings are computed and claimed annually over the 

11 Private labeling is defined as licensing a product to another company to sell under its own name, rather than under 

the name of the manufacturer. 

12 Marla C. Sanchez, Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008. Savings estimates for the 

United States Environmental Protection Agency's ENERGY STAR voluntary product labeling program. Energy 

Policy. vol. 36, no. 6, pp. 2098-2108. 

9 
 09-P-0061
 

lifetime of a product. As previously discussed, the lifetime of calculated annual 

benefits can reach up to 20 years after initial product shipment. 

Lack of Quality Review of Shipment Data 

The reported shipment data submitted annually by manufacturers were not being 

adequately reviewed. According to ENERGY STAR’s guidance document,13 the 

ENERGY STAR Product Labeling branch collects national shipment data from 

ENERGY STAR partner manufacturers and reconciles inconsistencies through 

the contractor. As described by the contractor, the contractor performs a limited 

review that involves reviewing the information to see if the data appears 

reasonable. EPA officials have not requested the supporting documentation;14 

therefore we found no evidence that EPA conducts any quality reviews of 

shipment data submittals. If a reconciliation of shipment data had occurred, the 

discrepancies we identified in our analysis could have been noted and addressed 

in a timely manner. 

Number of Manufacturers Submitting Data May be Wrong 

We were unable to reconcile the number of manufacturers who submitted their 

annual data to the number reported. In some cases, we found fewer 

manufacturers submitting data than the contractor reported. In other cases, more 

manufacturers reported than were qualified. Additionally, we noted some 

manufacturers reporting annual ENERGY STAR product shipments that were not 

listed as a manufacturer of ENERGY STAR-qualified products. For these, there 

was no evidence of any follow-up to address why this occurred. 

Other Factors Contribute to Uncertainties 

Third Party Association Reporting Inconsistent and Unsupported 

Reporting from third party associations was inconsistent and primarily included 

only summary totals by manufacturer. For most of the associations reporting 

shipment data, there was no way to verify the numbers based on any support or 

actual shipment data documentation, but we identified some problems. For 

example, there were cases where the manufacturer reported directly to both the 

contractor and the association, which resulted in shipment number differences. 

There was no evidence that the contractor questioned the differences or conducted 

any follow-up. The contractor acknowledged that the third party association 

reporting is inconsistent, lacks uniformity, and data has been submitted past 

agreed due dates. According to the contractor, in instances when there is a 

13OAR (Office of Air and Radiation) Climate Protection Partnerships Division’s ENERGY STAR Performance 

Management Plan.

14 The contractor had not been maintaining the supporting documentation for previous shipment totals, and the 

contractor is not required to do so by the contract. However, the contractor recently began retaining supporting 

documentation for reported overall numbers. 

10 
 09-P-0061
 

reporting difference, it has been its general practice to use the shipment totals 

reported by the third party association rather than the manufacturer. We could not 

verify from documentation provided whether these numbers were double counted. 

Use of Estimates and Forecasting Increases Uncertainty 

Selected manufacturer shipment data submittal forms provided shipment 

estimates rather than actual shipment totals. We found no documentation of any 

follow-up with these manufacturers as to why estimates were used, nor did the 

contractor mention this as a step taken. 

EPA did not require manufacturers to submit annual shipment data for office 

products. For the 2006 annual reported savings, forecasting reports were used to 

estimate shipment totals for some office products. We reviewed the two primary 

forecasting reports used. We noted that one report used for this forecasting was 

dated 2001. The other report did not clearly indicate how domestic shipments 

were separated from international shipments. Concerns associated with using a 

dated forecasting report may be corrected due to recent office product 

specification revisions.15 Also, beginning in 2008, manufacturers were required 

to submit shipment data annually for office products. 

Annual programmable thermostat shipments were estimated by the contractor 

based on past conversations with one U.S.-based manufacturer. Currently there 

are over 50 qualified manufacturers in this category, including many companies 

outside of the United States. Further, the ENERGY STAR Partner Commitment 

Agreement for Programmable Thermostats16 specifically requires that 

manufacturers submit the total number of qualified products shipped annually. 

We could not verify or reconcile the accuracy of support for benefits from this 

item since no actual shipment data were submitted, despite EPA’s requirements. 

ENERGY STAR Totals May Include International Shipments 

According to ENERGY STAR staff, shipment totals used to calculate energy 

savings are supposed to be limited to U.S. shipments. However, ENERGY 

STAR’s management plan states that as part of the partnership agreement, 

manufacturers must supply EPA with the “total number of ENERGY STAR 

qualified products shipped in the United States / Canada on annual basis.” We 

noted discrepancies in manufacturer shipment submittal forms. For example, on 

one submittal the manufacturer was identified as Company X, Canada, but had 

been changed to Company X, USA, within the final contractor’s reports. Also, on 

selected forms, the manufacturer said it could provide total international 

shipments with only an estimate of U.S. shipments. EPA’s contractor said that it 

15 In 2007, new specifications were made to the following office products: computers, copiers, facsimile, mailing 

machines, multifunction devices, printers, and scanners. 

16 To participate in the ENERGY STAR program and become eligible to become a partner, each manufacturer must 

agree to and adhere to this commitment.
 

11 
 09-P-0061
 

was unaware of domestic and international shipments being commingled, but 

noted it has not done any detailed work in this area. We believe that being 

unaware of whether international shipments are being included in calculations is 

not a reason to conclude they are not. 

Late Shipment Data Not Included in Benefits Calculations 

We found that approximately 33 million units reported as shipped were not 

included in the shipment totals used to calculate saving benefits for 2006. Per the 

Performance Management Plan, manufacturers are to provide shipment data by 

March 1 of the subsequent year (i.e., March 2007 for 2006 shipments). EPA’s 

contractor then compiles this information and prepares a July report of ENERGY 

STAR shipments to be used in the savings benefit calculations. The contractor 

prepares an updated report in October that includes shipment data submitted after 

the deadline. However, due to late submittals, we noted 33 million units captured 

in the October 2007 report that had not been captured in the July 2007 report. 

Therefore, savings benefits from these products were not captured in the 2006 

reporting period. 

According to the EPA Office of Air and Radiation’s Director of the Climate 

Protection Partnership Division, these shipments were also not included in the 

2007 calculations because the amount was considered immaterial. According to 

EPA’s contractor, the late shipment numbers were folded into the 2007 

calculations. However, the contractor was unable to provide evidence that this 

had occurred. Therefore, we concluded that the late shipments are not being 

captured by the current methodology. The lack of a methodology to capture 

potential savings benefits for late submittals reduces the reliability of ENERGY 

STAR reporting. 

DOE Product Included in EPA ENERGY STAR Reported Savings 

EPA reports a portion of savings benefits from the shipments of compact 

fluorescent lights (CFLs) even though CFLs are managed by DOE. EPA’s 2006 

ENERGY STAR Annual Report claimed 75.2 billion kilowatt hours (kWh) in 

energy savings. However, 5.7 billion kWh of those savings, or 8 percent, came 

from the DOE-managed CFLs. EPA did not disclose in its Annual Report that 

savings benefits reported include benefits from DOE-managed products. 

Also, we found differences in the total benefits claimed for CFLs by EPA and 

DOE. DOE reported 6.2 billion kWh in energy savings versus EPA’s 7.6 billion 

kWh for 2006. Rather than obtain total savings benefits directly from DOE, EPA 

calculated the CFLs energy savings benefits in the same manner as other EPA 

product savings (this process is described earlier in this chapter). This difference 

in energy savings between DOE and EPA for the same product questions the 

reliability of the computing and reporting of energy savings. 

12 
 09-P-0061
 

EPA ENERGY STAR staff said they claim one-third of CFL energy savings 

because of EPA’s marketing efforts. They also said they were involved in the 

specification development for CFLs. For these reasons, they claim a portion of 

the CFLs savings benefits. However, DOE ENERGY STAR staff were not aware 

that EPA claimed this percentage of CFL savings, and had claimed 100 percent of 

their calculated CFL savings in 2006. This resulted in double-counting of CFL 

benefits by EPA and DOE. A DOE ENERGY STAR representative said that 

while EPA does contribute to the success of CFLs by EPA’s marketing efforts, 

EPA should coordinate with DOE regarding the calculation and reporting. 

Conclusion 

We found that the reported savings claims were inaccurate and the reported 

annual savings unreliable. We identified several concerns with shipment data, 

including: lack of quality review of the data submittals; reliance on estimates, 

forecasting, and third party reporting; and potential inclusion of international 

shipments with domestic ENERGY STAR product shipments. After attempting 

to reconcile the support for these numbers by product category, we found the 

overall claims to be inaccurate due to irreconcilable manufacturer shipment data 

and use of estimates. We conclude that the savings reported are not accurate or 

verifiable. The accuracy of the program’s reported energy savings is important in 

the United States’ efforts to reduce greenhouse gas emissions. 

Recommendation 

2-1 To improve the validity of reported annual savings for the ENERGY 

STAR program, we recommend that the Principal Deputy Assistant 

Administrator for the Office of Air and Radiation establish and perform 

quality controls to ensure that: 

•	 Data in benefits calculations, whether from partners or third 

parties, are timely, complete, valid, and documented. 

•	 The contractor and third party associations receiving the 

manufacturer data submittal forms reconcile submittals in a 

manner that ensures the total annual shipments reported by product 

category are accurate and reflect actual numbers (not estimates) 

and are for domestic shipments only. 

•	 Agency officials improve contractor oversight by obtaining actual 

support for annual savings in a manner that demonstrates that the 

numbers are valid and can be reconciled. 

•	 Data in benefits calculations attributable to DOE products should 

be clearly identified and developed in consultation with DOE to 

avoid redundancy. 

13 
 09-P-0061
 

Agency Comments and OIG Evaluation 

EPA did not agree to the portions of Recommendation 2-1 that related to 

improvements needed to ensure data used in benefits calculations are timely, 

complete, valid, and documented. The Agency requested the OIG to delete the 

related portions of the recommendation. EPA said it addresses the quality of 

annual shipment numbers used for ENERGY STAR benefits calculations through 

a systematic review and improvement of the data submitted based on additional 

follow-up with reporting manufacturers. However, based on our findings in this 

area, we do not agree that the Agency currently has a systematic review process in 

place. The OIG evaluated the existing process and the findings show that the 

process used to collect and verify the data was deficient and needs improvement. 

OIG identified deficiencies with the process of collecting, reviewing, and 

reporting product data shipment information both from third parties or actual 

manufacturers. EPA also stated that estimates were not used as part of estimating 

annual program benefits. However, at the time of our review, estimates of 

shipment totals were used for both office products and programmable thermostats 

in calculating annual benefits. 

EPA provided corrective actions to address the portions of the recommendation 

that relate to improving contractor oversight and reporting savings from DOE 

products. Specifically, EPA agreed to remove the benefits of CFL products from 

the program’s 2007 benefits estimates and will only reintroduce these benefits 

after appropriate coordination with DOE. The OIG concurs with the Agency’s 

plan to address the reporting of CFLs. However, the Agency’s corrective actions 

for contractor oversight do not fully meet the intent of the recommendation. The 

OIG identified deficiencies with the process of collecting, reviewing, and 

reporting product data shipment information. The Agency’s corrective action 

plan will need to address these shortcomings. 

The recommendation remains open and unresolved. The Agency’s comments and 

our evaluation of those comments are in Appendix D. 

14 
 09-P-0061
 

Chapter 3

ENERGY STAR Market Transformation Benefits 

Calculated Inconsistently 

The reporting of ENERGY STAR market transformation benefits is not 

transparent. ENERGY STAR does not disclose that it includes savings from 

products that are not ENERGY STAR qualified. Further, the process does not 

compute the effect for products with stable or rising sales after a specification 

revision. EPA’s market transformation benefits are generated when innovations 

are introduced into the marketplace and are accepted by a large portion of the 

market. However, EPA calculates the market transformation effect only when 

ENERGY STAR product shipments fall after a specification revision. 

Market Transformation Caused by Innovations in Efficiency 

The ENERGY STAR programs attempt to transform the markets (i.e,. consumer 

electronics, office equipment, lighting, etc.) from less to more energy-efficient 

products. Market transformation is the process whereby these efficiencies are 

introduced into the marketplace and are increasingly accepted by a large portion 

of the market. According to EPA, the ENERGY STAR program produces such 

an effect on the market. ENERGY STAR staff asserted that the program has two 

primary mechanisms to spur market transformation for labeled products: 

•	 Introduce standards for new product categories. 

•	 Revise specifications for existing products to further increase their energy 

efficiency. 

According to EPA, the program has revised specifications for over 50 ENERGY 

STAR product categories. After a specification is revised, shipments of 

ENERGY STAR units may decrease until manufacturers institute product design 

changes to meet the new requirements. Nonetheless, products that once met 

program standards but are then no longer ENERGY STAR-qualified continue to 

be sold.17 Shipments of these products are included in reported ENERGY STAR 

benefits; this assumes that the products continue to be manufactured as energy 

efficient despite not meeting the new ENERGY STAR revised standard. 

17 The methodology does not address the loss of potential energy savings that occurs because the consumer 

purchased a formerly qualified product over a newly qualified higher performing alternative. 

15 
 09-P-0061
 

Market Transformation Benefits Calculated with Assumed Data 

EPA’s ENERGY STAR market transformation benefits process calculates and 

reports savings based on a combination of assumed and actual shipment numbers. 

The ENERGY STAR program does not have data to support the assumptions the 

market transformation effect uses. Further, the assumed sales that are attributed 

to the market transformation effect represent non-qualified products. 

A peer reviewed article18 describing ENERGY STAR’s market transformation 

effect included the following limitations statement: 

“General limitations to a bottom-up approach occur in two main 

areas: (1) the model requires numerous detailed inputs to generate 

the end result and (2) uncertainty in those inputs are additive 

through the process. These limitations mean that collecting and 

documenting high-quality inputs is essential...targeting data 

collection and verification activities at those areas is key to 

successful results.” 

In this market transformation effect the main input – previously qualified 

ENERGY STAR products – is based on assumptions rather than collected or 

verified data. Program requirements do not require manufacturers to report sales 

of formerly qualified ENERGY STAR products. EPA uses a model based on 

assumptions to calculate the market transformation effect in the absence of actual 

shipment data. By design, the model ensures that shipment amounts of products 

are never lower than the amount in the year prior to the specification change. 

The specification revision for DVD players illustrates EPA’s method of 

accounting for the market transformation effect. New specifications for DVD 

players took effect January 1, 2003. Shipments of ENERGY STAR-qualified 

DVD players rose that year, so no adjustment for the market transformation effect 

was made. However, in 2004 and beyond, manufacturer-reported shipments of 

ENERGY STAR-qualified DVD players fell below the 2003 level. To make up 

for this drop in sales, which was attributed to the specification revision, the model 

adjusted the current-year shipment figure based on the market transformation 

effect. The model increased the shipment totals for 2004 by adding in an 

estimated number of non-qualified ENERGY STAR DVD players shipped. This 

assumption is based on the theory that even though this estimated number 

represents non-qualified ENERGY STAR products, these products are more 

energy efficient than they would have been without Energy Star. Therefore, 

ENERGY STAR’s impact was to transform the market to produce more efficient 

products. The model continued to rely on this estimated level of product 

18 Marla C. Sanchez., Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008. Savings estimates for the 

United States Environmental Protection Agency's ENERGY STAR voluntary product labeling program. Energy 

Policy. vol. 36, no. 6. 

16 
 09-P-0061
 

shipments for 2005 and 2006 despite further decline in the number of units 

shipped. This is shown in Figure 3-1. 

Figure 3-1: DVD Players Market Transformation Effect 

0 

1000 

2000 

3000 

4000 

5000 

6000 

7000 

8000 

9000 

10000 

DVD units 

2001 2002 2003 2004 2005 2006 

Market transformation assumed shipments 

Actual Energy Star shipments 

Source: OIG analysis of CCAP 

 Market Transformation Obscures Program Accomplishments 

Based on our review of ENERGY STAR products that had specification 

revisions, the market transformation effect was calculated for nine product 

categories in 2006: 

• DVD players • Home audio equipment • VCR & VCR/DVDs 

• Monitors • TVs • Air source heat pumps 

• Telephony • TVs/VCRs • Central air conditioners 

The savings derived from the market transformation effect may have a significant 

impact on ENERGY STAR reported savings, because energy savings will 

continue to accumulate for the lifetime of a product. To illustrate the impact of 

the market transformation effect, we recomputed ENERGY STAR benefits by 

eliminating assumed shipment data and comparing these to the original benefit 

calculation. Based on our recalculation, ENERGY STAR energy savings reported 

in 2006 were overstated, as shown in Table 3.1.19 

19 This calculation represents one year; it does not include the potential overstatement from prior or future year 

benefits calculated for the same products. 

17 
 09-P-0061 

 Table 3-1: Market Transformation Effect 

ENERGY STAR OIG 

Summary20 Re-calculation Overstatement 

Energy Saved 

(Billion kWh) 

70.5 66.3 4.2 

Carbon Avoided 14.3 13.5 0.8 

(MMTCE) 

Source: OIG analysis of EPA data 

Shipment amounts were not based on actual or reported shipment data. EPA 

counted benefits from non-qualified ENERGY STAR products into its annual 

ENERGY STAR benefits amounts for 2006. Also, EPA did not compute a 

market transformation effect for products with stable or rising sales after a 

specification revision. The model is therefore inconsistently applied, since there 

should be a benefit derived from products with increasing as well as decreasing 

shipments. 

Conclusion 

ENERGY STAR does not disclose that it includes savings from products that are 

not ENERGY STAR-qualified. The computation of the market transformation 

effect does not use shipment or sales data, and only applies to products with both 

specification revisions and decreasing sales. Further, reported ENERGY STAR 

energy savings for 2006 omit mention of the market transformation effect, though 

it is included in individual product energy savings calculations. ENERGY STAR 

needs to improve controls over estimating and reporting benefits from the market 

transformation effect. Partnership agreements should include requirements for 

reporting shipment data of formerly qualified products, and the “effect” should be 

applied consistently. Also, market transformation benefits should be separately 

disclosed in ENERGY STAR’s annual reporting. 

Recommendation 

3-1 We recommend that the Principal Deputy Assistant Administrator for the 

Office of Air and Radiation develop and consistently apply a data-driven 

methodology to compute the market transformation effect of all product 

categories and report the benefits separately from ENERGY STARqualified products. 

Agency Comments and OIG Evaluation 

EPA said that the OIG did not provide a basis for recommending a change in the 

market transformation methodology and suggested removing the 

recommendation. The methodology is based on assumed shipment amounts and 

20 The 70.5 billion kWh of energy saved comes from EPA-managed ENERGY STAR products. 

18 
 09-P-0061
 

reports savings from non-qualified products as ENERGY STAR qualified product 

savings without proper disclosure. Therefore, the OIG does not agree with EPA’s 

suggestion to remove the recommendation. The recommendation is open and 

unresolved. The Agency’s comments and our evaluation of those comments are 

in Appendix D. 

19 
 09-P-0061
 

Chapter 4

ENERGY STAR Commercial Benefits Model Utilizes 

Unverified Assumptions 

The methodology used to compute the ENERGY STAR commercial sector 

savings was based on unverified assumptions. EPA calculates program savings as 

the difference between all commercial sector gains in energy efficiency and 

claims by utility and State programs for that year. However, we do not believe 

the formula is sufficiently tailored to the specific needs of the ENERGY STAR 

program. The formula needs to be verified to ensure that the end results – 

benefits attributable to ENERGY STAR’s commercial activities – are valid and 

accurately represent ENERGY STAR’s contributions to energy savings. 

Formula Used to Calculate Commercial Sector Savings 

EPA’s ENERGY STAR program uses a formula created, operated, and 

maintained by its contractor to compute the annual commercial sector savings. In 

this formula, the contractor (a) calculates all commercial sector gains in energy 

efficiency, (b) subtracts amounts from utility and State programs, and 

(c) concludes that the remaining energy savings are attributed to the ENERGY 

STAR program. ENERGY STAR staff said they review the formula with the 

contractor but they do not document the review results. The formula uses data 

from other agencies, along with related energy studies; it does not use any data 

from the ENERGY STAR program. 

Methodology Not Tailored for ENERGY STAR 

The methodology used for ENERGY STAR is not tailored specifically to 

calculate impacts attributable to that program and contains unverified 

assumptions. The methodology, which calculates the commercial benefits 

amount, is described in a 2007 paper by EPA’s contractor.21 This paper states: 

“In this study, energy efficiency programs refer collectively to all such 

governmental efforts, irrespective of organizational origin or mode.” This 

methodology looks at all energy efficiency programs in total; it is not tailored to 

pull out the impacts of the ENERGY STAR program. There is only one mention 

of ENERGY STAR in the paper, because it is used to define the start of the 

treatment period. 

21 Horowitz, Marvin J. (2007). “Changes in Electricity Demand in the United States from the 1970s to 2003.” The 

Energy Journal, Vol. 28, No. 3, pp. 93-119. 

20 
 09-P-0061
 

The paper’s goal was to answer the following three questions: 

•	 How has electricity demand changed in the United States over the past 

three decades? 

•	 If electricity demand changed, do these changes differ between States 

depending on their degree of commitment to encouraging energy 

efficiency? 

•	 Does program commitment have observable, long-term impacts on the 

behaviors that affect electricity demand? 

These questions are not related directly to ENERGY STAR’s impact on 

electricity demand. Rather, they seek to determine energy efficiency programs’ 

effects on energy demand. 

The methodology was peer reviewed, but its application as the basis for 

measuring ENERGY STAR’s impact on the commercial sector was not part of the 

published methodology. EPA needs to verify that the benefits attributed to 

ENERGY STAR are supported. Specifically, EPA should verify that the 

methodology: (a) avoids double counting savings from the product sector, 

(b) accounts for impacts of other federal energy efficiency programs, and 

(c) accounts for impacts of non-federal energy efficiency programs. 

Method to Avoid Double Counting Products Savings Unclear 

The formula used to compute ENERGY STAR commercial sector energy savings 

does not use any data, results, or statistics from the EPA’s ENERGY STAR 

program. Rather, the formula uses databases maintained by other government 

agencies, such as the Federal Reserve Board, National Climate Data Center, and 

Energy Information Administration. EPA ENERGY STAR staff said that the 

formula uses the best information available. To account for benefits from the 

ENERGY STAR products sector, the formula uses a Federal Reserve Bank’s 

market group index for information processing and related equipment. 

Approximately half of all ENERGY STAR product savings in 2006 were for 

commercial products, totaling 37.1 billion kWh. When the Agency reports total 

annual savings it includes savings from products, residential, industrial and 

commercial. If the commercial savings do not account for the savings from 

commercial products reported by the Agency, there may be the potential for 

overstatement of savings. The contractor and EPA officials asserted there was no 

double counting, but they could provide no evidence to support that position. 

Method for Accounting for Impacts of DOE Program Unclear 

The ENERGY STAR program’s methods for accounting for the impacts of a 

DOE program are unclear. The Energy Policy Act of 1992 set standards for 

certain commercial equipment and set schedules requiring DOE to make, review, 

21 
 09-P-0061
 

and update standards. The DOE’s Building Technologies Program works to 

improve the efficiency of buildings and the equipment, components, and systems 

within them. These DOE programs should contribute to some of the energy 

savings within the commercial sector. However, EPA’s formula does not 

evaluate or compensate for the energy efficiency contributions from DOE 

programs. Rather, the formula only deducts savings from utility, State, and local 

programs. ENERGY STAR staff said that the formula accounts for the DOE 

programs through the use of the Federal Reserve Board’s market index group. 

The accuracy of using this index compared to actual reported data from both EPA 

and DOE should be validated to determine if the formula is reporting ENERGY 

STAR commercial benefits accurately. 

Method for Accounting for Impacts of Non-federal Energy Efficiency 

Programs Unclear 

The method for accounting for the impacts of non-federal energy efficiency 

programs is unclear. The formula subtracts energy savings that are attributable to 

utility companies, States, and local programs – such as demand-side management 

(DSM) and public benefits charges (PBC) from total commercial savings – to 

arrive at net ENERGY STAR savings for the commercial sector. The contractor 

calculates savings from these other programs22 by using information on DSM and 

PBC programs from the study A Nationwide Assessment of Utility Sector 

Efficiency Spending, Savings, and Integration with Utility System Resource 

Acquisition. The contractor performs the following alterations to the numbers 

taken from this study: 

1. Project or estimate DSM and PBC savings from 2004 to 2006. (The 

study’s results were from 2004; the contractor updated this data to 2006 by 

using a growth rate that it determines.) 

2. Allocate the share of energy savings that occur in the commercial 

sector. (The study’s reported amounts were a combination of residential, 

commercial, and industrial savings.) 

3. Apply a net-realization rate that lowers the computed DSM and PBC 

savings because, as explained by the contractor, the full amount of energy 

saved reported by electric utilities for their DSM programs cannot be 

taken at face value. 

Based on the first two steps, the computed savings from the DSM and PBC 

programs were 38,502 gigawatt hours, but EPA’s contractor took 42.5 percent23 

of this amount, lowering the savings figure to 16,363 gigawatt hours. This was 

22 Other programs consist of State, utility and local energy efficiency programs. 23 This percentage is supported by a memorandum prepared by the contractor that spells out how the program
 

allocation factor for the formula was calculated. Specifically, the 42.5 percent is referred to as a net-to-gross factor, 

or a net realization rate. 

22 
 09-P-0061
 

then attributed to the other energy efficiency programs. The application of the 

net-realization rate significantly impacts the ENERGY STAR-derived energy 

benefit. The lower the rate the greater the portion of savings is attributable to 

ENERGY STAR. ENERGY STAR staff said that all formula inputs are from the 

best available and/or peer reviewed data. 

Formula Methodologies and Assumptions Need to Be Verified 

The spreadsheet used to calculate the energy savings for the commercial sector 

was not verified to ensure that its results are an accurate computation of energy 

savings attributable specifically to the ENERGY STAR commercial sector. 

ENERGY STAR plans to expand the use of this formula to compute energy 

savings for the industrial sector, so it is additionally important to verify and 

validate the formula’s results. 

The EPA Quality Manual for Environmental Programs CIO 2105-P-01-0 

(formerly 5360 A1) defines a peer review as: 

A documented critical review of work by qualified individuals (or 

organizations) who are independent of those who performed the 

work, but are collectively equivalent in technical expertise. A peer 

review is conducted to ensure that activities are technically 

adequate, competently performed, properly documented, and satisfy 

established technical and quality requirements. The peer review is 

an in-depth assessment of the assumptions, calculations, 

extrapolations, alternate interpretations, methodology, acceptance 

criteria, and conclusions pertaining to specific work and of the 

documentation that supports them. 

A peer review could accomplish two objectives. First, it could validate that the 

formula reasonably computes all energy savings. Second, it could provide 

assurance that savings are attributable to EPA’s ENERGY STAR commercial 

sector’s activities. The validation should be conducted in accordance with EPA 

Quality Manual for Environmental Programs CIO 2105-P-01-0. 

Conclusion 

The methodology used to compute the ENERGY STAR commercial savings is 

based on unverified assumptions that impact the accuracy of the reported energy 

savings. The methodology is based on a study of the impact of all energy 

efficiency programs on electric demand, not only ENERGY STAR; it assumes 

that all net savings are attributable to the ENERGY STAR commercial program. 

EPA needs an independent assessment or validation of how ENERGY STAR 

savings are computed to establish a reasonable assurance that the energy saved is 

accurate. 

23 
 09-P-0061
 

Recommendation 

4-1 We recommend that the Principal Deputy Assistant Administrator for the 

Office of Air and Radiation validate the formula (methodology) used for 

calculating the benefits of the ENERGY STAR commercial program in 

accordance with EPA Quality Manual for Environmental Programs CIO 

2105-P-01-0, to ensure that it accurately reflects the impacts of EPA 

actions. 

Agency Comments and OIG Evaluation 

EPA maintains that the estimates generated by the methodology are rigorous, that 

where marginal uncertainties are present they have been handled conservatively, 

and that the results are sound. EPA said that it will secure additional outside 

expert review of the entire methodology being used to estimate the benefits of the 

ENERGY STAR program in the commercial sector. The EPA action meets the 

intent of the recommendation; however the recommendation remains open until 

the completion of the expert review. The review should assure that assumptions, 

data sources, and methods used to estimate the ENERGY STAR commercial 

benefits savings are reasonable and supported. The Agency’s comments and our 

evaluation of those comments are in Appendix D. 

24 
 09-P-0061 

Status of Recommendations and 

Potential Monetary Benefits 

POTENTIAL MONETARY 

RECOMMENDATIONS BENEFITS (in $000s) 

Rec. 

No. 

Page 

No. Subject Status1 Action Official 

Planned 

Completion 

Date 

Claimed 

Amount 

Agreed To 

Amount 

2-1 

3-1 

4-1 

13 

18 

24 

To improve the validity of reported annual savings 

for the ENERGY STAR program, establish and 

perform quality controls to ensure that: 

• Data in benefits calculations, whether from 

partners or third parties, are timely, complete, 

valid, and documented. 

• The contractor and third party associations 

receiving the manufacturer data submittal 

forms reconcile submittals in a manner that 

ensures the total annual shipments reported 

by product category are accurate and reflect 

actual numbers (not estimates) and are for 

domestic shipments only. 

• Agency officials improve contractor oversight 

by obtaining actual support for annual 

savings in a manner that demonstrates that 

the numbers are valid and can be reconciled. 

• Data in benefits calculations attributable to 

DOE products should be clearly identified 

and developed in consultation with DOE to 

avoid redundancy. 

Develop and consistently apply a data-driven 

methodology to compute the market transformation 

effect of all product categories and report the 

benefits separately from ENERGY STAR-qualified 

products. 

Validate the formula (methodology) used for 

calculating the benefits of the ENERGY STAR 

commercial program in accordance with EPA 

Quality Manual for Environmental Programs CIO 

2105-P-01-0, to ensure that it accurately reflects 

the impacts of EPA actions. 

O 

O 

O 

Principal Deputy 

Assistant Administrator, 

Office of Air and Radiation 

Principal Deputy 

Assistant Administrator, 

Office of Air and Radiation 

Principal Deputy 

Assistant Administrator, 

Office of Air and Radiation 

O = recommendation is open with agreed-to corrective actions pending 

C = recommendation is closed with all agreed-to actions completed 

U = recommendation is undecided with resolution efforts in progress 

25 

1 
Logic Model for ENERGY STAR Products

Resources Activities 

Program Administration -- Up to date product lists -- Up to date partner information -- Data collection on  product shipments Build Supply of Efficient products -- Add new products (using  established guidelines) 

-Dollars -Staff 

Revise Product Specs -- as appropriate (using  established guidelines) Build Demand -- Outreach -- robust web-based info -- consumer hotline -- direct consumer efforts -- key product strategies -- partner by partner  assistance -- coordinated national  campaigns -- train distribution channels -- product procurement Ensure Quality -- require proper use of  logo in ads -- spot test products -- improve retail experience Evaluation -- consumer awareness,  understanding, etc
 -- progress assessments 

Outputs Assessment/ Intermediate Reassessment Outcomes Outcomes 

Products
 

-- # product categories -- # product models -- # product sales (annual) -- market penetration
 Partners
 

-- # manufacturers -- # retailers -- # program sponsors -- # states and others Partner activity -- activity/investment by 

 retailers, program sponsors,  manufacturers Outreach
 -- # web visitors -- key visitor interests
 -- # media impressions -- # media impressions by

Energy Savings 

-- in kWh -- by product area -- program level Reductions in GHG 

emissions -- in mmtce -- by product area -- program level Investment in technology -- in $$ -- program level Net Savings on Energy bills 

-- in $$ -- program level Program Cost-effectiveness Behavioral Change Improved Energy Efficiency Policies for Product Efficiency 

Annual review of key outputs against program near, mid, and long term goals for program and key product areas and revision of strategies, as appropriate 

 key strategy and product Compliance/Quality
 

-- # advertising clips -- # companies advertising -- # violations -- # companies with 

violations 

-- Indication of manufacturer  QA/QC issues
 -- lighting fixture violations -- Indication of retailer issues Brand Information -- level of consumer 

 awareness, understanding, etc Benefits Projections -- estimates by product of 

 future sales/benefits 

 based on continued efforts
 

Source: ENERGY STAR program 

26 
Logic Model for ENERGY STAR Commercial & Industrial Programs 

Resources Activities 

Program Administration – Up to date partner

 information -- Maintain portfolio manager 

 and other key tools 

Build Demand -- Outreach -- robust web-based info -- customer hotline -- national campaign on  building improvements -- encourage state/local lead  by example 

-Dollars -Staff 

-- web-based and other  trainings for end-users -- recognition for excellence  -- broad corporate leadership  -- achieving key milestones  -- exemplary buildings Build Supply of Services -- partner with service  providers -- train service providers on  benchmarking and other  ENERGY STAR resources -- partner with utilities; train;  and help develop programs  offering benchmarking. Ensure Quality -- require proper use of  logo in ads -- audit labeled buildings 

 Assessment/ Intermediate 

Outcomes Outcomes 

 Outputs 

 Reassessment

Partners Committed to Improvements -- # private sector organ -- # public sector -- # key sub sectors -- # states Partner activity -- activity/investment by  end-user partners, program sponsors, third-party providers, etc -- number of ENERGY STAR  leaders Standard Measurement System -- # building types -- % of market covered -- number of benchmarked  buildings, % of key subsectors  benchmarked, square footage  benchmarked -- number of buildings labeled,  % of key subsectors labeled,  square footage labeled Building Service Market -- # SSP partners -- # third party providers of  benchmarking services -- # trainings -- number of benchmarks  provided Outreach -- # web visitors -- key visitor interests -- # media impressions -- number campaign participants Compliance/Quality -- # advertising clips -- # companies advertising -- # violations -- # companies with 

violations 

Annual review of key outputs against program near, mid, and long term goals for program and sales of key products and revision of strategies, as appropriate 

Energy Savings 

-- in kWh -- program level Reductions in GHG 

emissions 

-- in mmtce -- by program level Investment in technology 

-- in $$ -- program level Net Savings on Energy bills 

-- in $$ -- program level Program Costeffectiveness Behavioral Change Improved Energy Efficiency Policies for Building Efficiency 

Source: ENERGY STAR program 

27 
09-P-0061 

Appendix C 

Details on Scope and Methodology 

To determine whether the savings reported for products and commercial ENERGY STAR 

program sectors were valid and fully supportable, we analyzed supporting documentation for 

2006 reported annual savings. We reviewed the ENERGY STAR and Other Climate Protection 

Partnerships 2006 Annual Report (September 2007) and the U.S. Climate Action Report – 2006 

for savings and benefits claims made by the program. We also reviewed relevant reports and 

spreadsheets by Lawrence Berkeley National Laboratory and various contractors. 

We conducted a review of the savings benefits for the products and commercial sectors, which 

combined encompassed 81.6 percent of the 200624 reported carbon emissions avoided; we did 

not conduct a detailed review of the residential and industrial sectors. To analyze the methods 

used to calculate the annual savings by sector, we met with EPA ENERGY STAR officials and 

the primary contractors involved. For the product sector, we obtained available support data and 

documented the roles of both EPA and the contractor in the computation and methodology 

process. Specifically, we requested the supporting documentation for the 2006 shipment totals 

used in the savings benefit calculation for 2006 (as described in Chapter 2). We were provided 

manufacturer shipment data submittal forms and shipment reports from third party associations. 

We performed a detailed analysis and reconciliation of the manufacturer shipment data 

submittals for all 2006 reported amounts by qualified ENERGY STAR product category. We 

documented the forecasting methods for the other EPA product categories for which submission 

of direct data shipment data was not required for 2006 reporting. We judgmentally selected a 

sample of four qualified products based on a number of factors including: ENERGY STAR 

shipment totals for 2006 and specification revision history. We traced and reconciled these four 

qualified products through the Lawrence Berkeley National Laboratory CCAP model. These 

products were computers, DVDs, external power supplies, and monitors. 

For the commercial sector, we reviewed the methodologies used to calculate ENERGY STAR 

reported benefits. We met with the contractor to obtain an understanding on how commercial 

benefits are calculated. We also analyzed the contractor’s spreadsheet, containing the inputs and 

assumptions used in calculating the 2006 commercial ENERGY STAR reported benefits. 

To determine what management controls EPA had in place to ensure the validity and accuracy of 

reported annual savings, we analyzed available data and met with EPA and contracting officials. 

We reviewed the OAR (Office of Air and Radiation) Climate Protection Partnerships Division’s 

ENERGY STAR Performance Management Plan. Our review included a detailed follow-up with 

the ENERGY STAR Director on plan specifics. We met with the contractor involved in 

implementing the plan to determine the role quality assurance/quality control plays in its work in 

the computation process. We also analyzed some of the controls/procedures within the plan to 

verify whether they were applicable and had been adequately performed as described within 

Chapter 2. All internal control weaknesses noted during our evaluation are discussed in 

Chapters 2, 3, and 4 of this report. 

24 At the time of our evaluation, 2006 was the most current data available. 

28 
09-P-0061 

To determine the basis for goal setting, we meet with ENERGY STAR staff and reviewed the 

program’s logic models. We also reviewed recent U.S. Climate Action Reports, Guidelines for 

Designing EPA Partnership Programs, Government Performance Results Act of 1993, and the 

Energy Policy Act of 2005. 

29 
09-P-0061 

Appendix D 

Agency Comments and OIG Evaluation 

MEMORANDUM 

SUBJECT: Response to Draft Evaluation Report: Improvements Needed to Validate Reported 

ENERGY STAR Benefits: Project No. 2007-0081 

FROM: Robert J. Meyers 

Principal Deputy Assistant Administrator 

TO: Jeffrey Harris 

Director for Special Studies 

Office of the Inspector General 

We appreciate the opportunity to review the OIG’s draft report on the benefits of the 

ENERGY STAR program and their recommendations for improving the benefit estimation 

methods OAR employs. OAR has or will address two of the three recommendations and asks 

that the third recommendation be removed. 

OAR maintains a strong commitment to using and continually improving the methods it 

employs to provide robust estimates of the benefits of the ENERGY STAR programs. In this 

context, OAR will proceed to address the OIG recommendations, where they have not already 

been addressed. 

OIG Response: OIG understands the ENERGY STAR program reports estimates. The 

statement that, “More than 37 million metric tons of greenhouse gas emissions were avoided, 

equivalent to the greenhouse gas emissions from 25 million vehicles,” does not disclose that 

reported program benefits are robust estimates. The other reportable program benefits are 

reported in a similar matter. 

OAR is very concerned about the tone and exaggeration of issues in the OIG report as 

well as the general implication that OAR’s estimates of ENERGY STAR program benefits are 

not valid. OAR stands by the validity of the reported estimates of the ENERGY STAR program 

benefits. 

•	 OAR methods and procedures conform to Agency policy and guidance. 

•	 OAR employs peer-reviewed methods and appropriate QA/QC procedures. 

•	 OAR approaches are consistent with well-documented best practices for developing 

estimates of the benefits of energy efficiency programs. 

•	 OIG did not identify any issues that have a significant impact on the reported results for 

the ENERGY STAR program. 

30 
09-P-0061 

OIG Response: OIG reported that the data used to calculate benefits was inaccurate 

and unsupported; therefore we could not attest to the overall validity of the actual 

reported numbers. For the product and commercial program sectors we documented 

how benefits were calculated, evaluated the supporting data, and outcome of EPA’s 

calculations with the claimed impacts. In our opinion there were numerous 

deficiencies in the processes and data supporting the benefit calculations. 

It is important to understand that program benefit estimates are estimates. Such estimates are 

routinely based on available data and reasonable assumptions. OAR employs conservative 

estimates for assumptions where there is a lack of available data and some uncertainty. 

Further, OAR does not believe that the OIG identified any issues that have a significant 

impact on the reported results for the ENERGY STAR program. The OIG review of the 

ENERGY STAR benefits calculations was narrowly focused on several issues and processes. In 

addition, there are important inaccuracies in the OIG report and misunderstandings of the 

methods that OAR employs (see supporting information). 

OIG Response: It is the OIG’s opinion that the Agency approach for estimating program 

benefits is not conservative. For example, we do not believe that the Agency’s use of the 

market transformation effect is a conservative approach. The OIG takes no position on the 

validity of the market transformation effect, but we disagree with the methodology used by the 

Agency to compute this effect. An additional issue we noted was the lack of disclosure in the 

annual report. Benefits derived from the market transformation are not identified in the annual 

report and are included along with benefits from ENERGY STAR qualified products. 

Furthermore, the OIG’s objective was to determine the validity of the ENERGY STAR benefit 

calculations. We conducted a comprehensive review of the methodology, data, and analysis 

used by EPA to calculate over 80 percent of the ENERGY STAR reported carbon emissions 

avoided in 2006. We found deficiencies in the design, data, and execution of the benefit 

calculations. We did not attempt to quantify the net numerical impact of these deficiencies. 

OIG believes that the net impact of reported deficiencies may be significant. For example, 

when calculating individual consumer product benefits if the initial data shipment numbers are 

unsupported or inaccurate, not only is the benefit calculation for that current year invalid but the 

subsequent calculation of benefits are impacted over the lifetime of that product category. As 

noted in the Agency’s peer reviewed article: 

We implement the bottom-up model with awareness that uncertainty for each product 

type contributes to uncertainty in total ENERGY STAR impacts. This means that many 

small inaccuracies are additive overall and any one inaccuracy for a product type with 

large energy savings can significantly affect the overall results. 

We do not agree that the OIG report contains inaccuracies. The Agency’s response did not 

provide evidence in support of this statement. 

31 
09-P-0061 

The result of the OIG review is that 

•	 The OIG identified several ways for OAR to improve its documentation of data and 

QA/QC procedures, steps that OAR has already or will undertake this coming year. 

•	 The OIG questions an approach that OAR uses to represent the market transformation 

benefits of the ENERGY STAR product labeling program; however, because this 

approach is a well accepted practice, has been peer-reviewed, addresses the uncertainty in 

a conservative manner, and represents about 2 percent of overall program benefits, OAR 

does not believe a change is necessary. 

•	 The OIG identified two addition errors that would serve to increase the ENERGY STAR 

program benefits for 2006 by about 1 percent. These errors have been corrected. 

OIG Response: OIG does not agree with the above assessment of the results of the OIG’s review. 

The supporting information OAR provided in this response will be addressed by each subject as 

deemed necessary. 

In addition, the OIG made three recommendations to improve the methods and processes OAR 

uses for estimating the annual benefits for the ENERGY STAR program. These 

recommendations and OAR’s responses are provided below. 

Recommendation 2-1 To improve the validity of reported annual savings for the ENERGY 

STAR program, we recommend that the Principal Deputy Assistant Administrator for the Office 

of Air and Radiation establish and perform quality controls to ensure that: 

•	 Data in benefits calculations, whether from partners or third parties, are timely, complete, 

valid and documented. 

•	 The contractor and third party associations receiving the manufacturer data submittal forms 

reconcile submittals in a manner that ensures the total annual shipments reported by product 

category are accurate and reflect actual numbers (not estimates) and are for domestic 

shipments only. 

•	 Agency officials improve contractor oversight by obtaining actual support for annual 

savings in a manner that demonstrates that the numbers are valid and can be reconciled. 

•	 Data in benefits calculations attributable to DOE products should be clearly identified and 

developed in consultation with DOE to avoid redundancy. 

OAR Response: OAR addresses the quality of annual shipment numbers used for 

ENERGY STAR benefits calculations through a systematic review and improvement of the 

data submitted based on additional follow up with reporting manufacturers. There are clear 

deadlines for the reporting of data and ramifications for partners not reporting data by these 

deadlines. Estimates and international shipments, while collected in a few limited cases, 

are not used as part of estimating annual program benefits. Documentation of these 

requirements, data submittal timelines and penalties for not meeting them is available at 

www.energystar.gov/usd. 

32 
09-P-0061 

OIG Response: OIG evaluated the existing process. The findings show that the process used to 

collect and verify the data was deficient and needs improvement. Corrective action may include 

improved implementation and oversight of the existing process as well as new procedures. For 

example, we found large discrepancies in the shipment data used in the benefits calculation for 

exit signs and monitors. The table provided by the Agency not only acknowledges errors 

occurred, but validates the lack of a quality control system. The large “spreadsheet errors,” 

uncorrected until disclosed by OIG, demonstrate deficiencies in the existing Quality 

Assurance/Quality Control process. Additionally, at the time of our review, estimates of 

shipment totals were used for both office products and for programmable thermostats in 

calculating annual benefits. 

Status: Suggest removing the first two bullets. The remaining two recommendations have 

been addressed: 1) The EPA contractor has been directed to fully document the data 

collection and review process for product shipment data for 2007 and subsequent years. 

This has been completed for 2007 2) For the future, EPA has removed the benefits 

estimates attributable to EPA efforts with compact fluorescent lighting (CFL) products as of 

the 2007 program benefits estimates and will only reintroduce these benefits after 

appropriate coordination with DOE. 

OIG Response: OIG does not agree with the removing of the first two bullets. Further, the 

Agency’s corrective action plan will need to address the shortcomings related to improving 

contractor oversight. The Agency should identify improvements in management controls over 

the existing process. OIG identified deficiencies with the process of collecting, reviewing, and 

reporting product data shipment information both from third parties or actual manufacturers. 

The status of the recommendations reported by the Agency does not meet the intent of the 

recommendation. This recommendation is open and unresolved. 

Recommendation 3-1 We recommend that the Principal Deputy Assistant Administrator for the 

Office of Air and Radiation develop and consistently apply a data-driven methodology to 

compute the market transformation effect of all product categories and report the benefits 

separately from ENERGY STAR-qualified products. 

OAR Response: Calculating market transformation benefits is a widely accepted concept 

within the energy community and is a meaningful indicator of the ongoing effect of 

ENERGY STAR in the market. The current methodology for calculating market 

transformation benefits has been peer reviewed (see below) and is applied consistently 

across the product categories, where applicable. Further, the market transformation effect 

accounts for a small fraction of overall ENERGY STAR program savings (e.g. less than 2 

percent), meaning that uncertainty in the data and assumptions behind the calculations 

(uncertainty that has already been addressed through conservative assumptions) have a 

relatively low impact on overall modeled results. 

Method has been peer reviewed in the journal Energy Policy: 

33 
09-P-0061 

Sanchez, Marla C., Richard E. Brown, Carrie Webber, and Gregory K. Homan. 

2008. Savings estimates for the United States Environmental Protection Agency's 

ENERGY STAR voluntary product labeling program. Energy Policy. vol. 36, no. 

6, pp. 2098-2108 

OIG Response: OIG did not question the concept, only the Agency’s methodology and data 

sources used to compute benefits for ENERGY STAR. The methodology and data used to 

calculate ENERGY STAR market transformation benefits has not been validated. 

Additionally, the peer reviewed article includes a limitations section which provides the 

following: 

These limitations mean that collecting and documenting high-quality inputs is 

essential, which can be a labor-intensive and expensive process. As a result, 

identifying areas of critical uncertainty and sensitivity and then targeting 

data collection and verification activities at those areas is key to successful 

results. [Emphasis added] 

The OIG evaluated reported benefits incurred in 2006. For 2006, the market transformation 

effect was 6 percent of the reported benefits for the product category. The significance of 

savings from the market transformation calculation can vary depending on the number of new 

specifications in a year, as well as the reported shipment totals. We do not consider using 

unsupported shipment totals to compute benefit savings in the market transformation 

calculation a conservative assumption. Furthermore, the OIG analysis does not take into 

account the reported savings that can be realized in future years since these units will be 

counted for their expected product life. 

Status: Since, the OIG has not provided a basis for recommending a change in 

methodology, particularly in light of the complexity and cost associated with doing so, 

OAR suggests removing this recommendation. 

OIG Response: It is the OIG’s opinioin that the computation of the market transformation 

effect requires identifying areas of critical uncertainty and sensitivity and then targeting data 

collection and verification activities at those areas for successful results. That has not been 

done. While the benefits calculated from market transformation make up 6% of the product 

savings, given the downturn in the economy and changes to major consumer products (like 

DVDs) the impact of a market transformation calculation is likely to grow larger. This 

recommendation is open and unresolved. 

Recommendation 4-1: We recommend that the Principal Deputy Assistant Administrator for 

the Office of Air and Radiation validate the formula (methodology) used for calculating the 

benefits of the ENERGY STAR commercial program in accordance with EPA Quality Manual 

for Environmental Programs CIO-2105-P-01-0, to ensure that it accurately reflects the impacts of 

EPA actions. 

34 
09-P-0061 

OAR Response: OAR's approach to estimating the benefits of the ENERGY STAR 

program in the commercial sector has evolved as the program has expanded and adopted 

up-to-date econometric analysis and information. Each refinement has been reviewed by 

outside experts. OAR believes that the estimates generated by the methodology are 

rigorous, that where marginal uncertainties are present they have been handled 

conservatively, and that the results are sound. OAR’s program benefits estimates are 

performed in accordance with the EPA Quality Manual as the Office of Atmospheric 

Programs has established a Quality Management Plan and these benefits estimates have 

been developed in accordance with that plan. 

OIG Response: The peer reviewed methodology does not directly support Office of Air and 

Radiation’s claimed benefits for the ENERGY STAR commercial sector. EPA uses parts of that 

methodology to derive the estimated benefits for commercial sector. The actual computation 

methodology used needs to be validated. 

Further, OAR recommends that the title for Chapter 4 be modified to be “Further 

Validate Commercial Sector Methods 

OIG Response: The OIG believes the current title accurately reflects the content of the chapter. 

Status: OAR will secure additional outside expert review of the entire methodology 

being used to estimate the benefits of the ENERGY STAR program in the commercial 

sector. We can provide the OIG for a schedule for this review within two months. 

OIG Response: The Office of Air and Radiation action meets the intent of the recommendation. 

The expert review should assure that assumptions, data sources, and methods used to estimate the 

ENERGY STAR commercial benefits savings are reasonable and supported. 

Supporting Information 

OAR has identified a number of errors in the OIG report and several important 

misunderstandings in the OIG report as detailed below. 

2006 Annual Shipment Data 

OAR has reviewed the table the OIG presented in its draft report showing that it could not 

reproduce the 2006 product shipment totals in a July 2007 ICF report for 18 of the 29 product 

categories that it examined. OAR looked at eleven of the eighteen product categories that the 

OIG examined, the ones which would potentially have the most significant impact on the 2006 

stated program results due to the size of the differences in the OIG table. OAR has the following 

observations based on its review (see attached summary table): 

•	 OAR has confirmed the validity of the shipment numbers used in the 2006 benefits 

calculation for nine of the eleven product categories OAR examined. 

35 
09-P-0061 

•	 The differences that the OIG found are largely explained by the OIG not completing the 

QA/QC process that is in place to review and improve the raw submittal data. 

•	 Two computational errors were uncovered as a result of the audit. These two errors serve to 

increase the ENERGY STAR program benefits and together represent a small fraction (about 

1%) of the programs overall reported benefits. These errors have been corrected. 

•	 The OIG findings do not support the broad claim that the ENERGY STAR program benefits 

are invalid. 

OIG Response: The Agency provided no additional documentation or support for its 

analysis, so we cannot comment on its accuracy. However, given the quality of the 

documentation supporting ENERGY STAR benefit calculations, OIG understands Office of 

Air and Radiation’s difficulty. OIG analysis and numbers were based on the documentation 

provided by the EPA contractor tasked with reporting these numbers. In fact, we met with the 

contractor concerning this issue and it was unable to provide us with any further 

documentation that would change any of our reported numbers or discrepancies. Our detailed 

analysis identified large inaccuracies in the reported shipment totals used and in what the 

actual shipment data supported for products such as exit signs and monitors. The Agency’s 

table provided in this response acknowledges that there were in fact “spreadsheet errors” for 

both products and the numbers should have been the much larger number as identified by the 

OIG. These two “spreadsheet errors” demonstrate deficiencies with the existing Quality 

Assurance/Quality Control process since large errors were not identified. The errors were 

reported by the OIG. The Agency work was done after the fact, in an attempt to verify the 

original numbers. It should also be noted that the underestimates were already used in the 

2006 benefits calculations process and were already reported within the Annual Report. 

Further, OAR disagrees with the characterization of the OIG in a number of other areas in the 

report and provides the following clarifications. 

Quality Review of Data. OAR’s contractor responsible for collecting and compiling shipment 

data performs a systematic review of all shipment reports that looks for and addresses a number 

of potential issues. The contactor has been directed to document the results of this quality 

review. The issues the contractor addresses include: 

•	 Multiple submissions received from a single partner 

•	 Inclusion of a partner's data in an aggregate submission from an association, when the 

partner had already reported to contractor directly 

•	 An apparent math error on the data collection form 

•	 Submission of shipment data for a product that cannot earn the ENERGY STAR, e.g., 

qualified shipments reported for ENERGY STAR camcorders 

•	 Submission of data on a non-standard form, e.g., a NYSERDA form 

•	 Indication that reported, qualified shipments went to a private labeler, who was also 

required to report 

•	 Alteration of the standard form, e.g., partner modification of units of measure 

36 
 

09-P-0061 

•	 Failure of the partner to break data into requested subcategories 

•	 ENERGY STAR qualified shipments exceeding total shipments 

•	 Indication that the submission is only an estimate 

•	 Indication of negative shipments for a partner 

•	 A partner's claim of qualified shipments for a product they do not manufacturer, have 

not yet qualified with ENERGY STAR, or for which they’ve not partnered with 

ENERGY STAR. 

In addition to working with partners and third party associations to resolve data anomalies, 

the contractor performs the following steps to improve the quality of the report: 

•	 Reviews past year’s process recommendations and reviews with EPA if any changes 

are needed to solicit the most accurate information from partners in the clearest way. 

•	 Shares information including contact lists with third party associations well in 

advance, to ensure that all eligible partners contribute to the collection effort. 

•	 Contractor product leads review submitted data for their respective categories to flag 

any apparent discrepancies or unexpected changes from previous years. 

•	 Task Order Manager checks data entry, calculations, and consistency among summary 

and breakout tables. 

The EPA contractor has been directed to fully document the data collection and review process 

for product shipment data for 2007 and subsequent years. This is complete for 2007. 

OIG Response: If the Quality Assurance/Quality Control system the Agency summarizes in 

this section was implemented and enforced as the Agency implies then the errors we identified 

would have been caught by either the contractor or EPA. The Quality Assurance/Quality 

Control process existing at the time of the OIG’s review was deficient as demonstrated by the 

errors identified in the report. Many of these steps were either not in place or had not been 

implemented during our review. We were advised by the EPA contractor that follow-up 

actions were not always documented, which may have contributed to the discrepancies in 

shipment totals used in the benefit calculations. The EPA contractor told us that the third party 

reporting was inconsistent, lacked uniformity, and information was often submitted late. 

•	 Data Submitted Through Third Parties. The ENERGY STAR partnership agreement 

explicitly allows for the submittal of shipment data through a third party aggregator in the 

event a partner has competitiveness concerns associated with releasing data. OAR’s 

contractor works closely with the relevant trade associations to ensure that any potential data 

issues are resolved as the data is compiled. This is an appropriate and necessary practice to 

employ as part of collecting information for a partnership program. 

37 
09-P-0061 

OIG Response: OIG took no position on the use of third parties. OIG identified 

deficiencies with the reporting of the data received by the contractor. The contractor 

recognized there were problems with the third party association data submittals. We were 

told that it was the contractor’s practice to accept the third party reported numbers even if a 

manufacturer had self-reported a different set of numbers. 

•	 International Shipments. The OIG did not provide any credible basis for OAR to question 

that shipment totals used in the benefits calculations include anything but U.S. shipments. In 

fact, the two instances cited give no indication other than that the submitter clearly 

understood the request to be for U.S. data alone. OAR does not use international shipment 

information to estimate program benefits. 

OIG Response: OIG provided examples of potential control issues with accounting for 

international shipments. 

Compact Fluorescent Light (CFL) Bulbs 

OAR estimated benefits from EPA efforts with compact fluorescent light bulbs. EPA has made 

important contributions to the area of ENERGY STAR CFLs for more than eight years. This 

includes contributions to key aspects of the ENERGY STAR specification for CFLs, quality 

testing for CFLs, and ramping up consumer awareness and interest in purchasing ENERGY 

STAR qualifying CFLs. EPA has removed these benefits from its ENERGY STAR program 

benefits as of 2007. 

OIG Response: OIG reported that EPA and DOE were both claiming the CFL benefits 

which results in double counting of the greenhouse gas savings. OIG did not question 

EPA’s contribution to CFLs. We agree that removal of the CFL benefits until they are 

discussed with the appropriate DOE officials is the appropriate action. 

Market Transformation Effect 

OAR employs a widely accepted concept within the energy community for estimating the market 

transformation benefits of the ENERGY STAR program. This captures the set of products in the 

marketplace, that are more efficient than they otherwise would have been, that were once eligible 

for the ENERGY STAR, but are no longer due to increased stringency of the ENERGY STAR 

specification. The current methodology for calculating these market transformation benefits has 

been peer reviewed in the journal Energy Policy: 

Sanchez, Marla C., Richard E. Brown, Carrie Webber, and Gregory K. Homan. 2008. 

Savings estimates for the United States Environmental Protection Agency's ENERGY 

STAR voluntary product labeling program. Energy Policy. vol. 36, no. 6, pp. 2098-2108. 

38 
09-P-0061 

The method is applied consistently across the product categories for which an ENERGY STAR 

specification has been revised and a substantial number of products are still in the market that 

met the earlier specification. Further, the market transformation effect accounts for a small 

fraction of overall ENERGY STAR program savings (about 2%). This means that uncertainty in 

the data and assumptions behind the calculations (uncertainty that has already been addressed 

through conservative assumptions) has a relatively low impact on program estimates. 

Accordingly, the OIG has not provided a basis for recommending a change in methodology, 

particularly in light of the complexity and cost associated with doing so. 

OIG Response: The method is not applied consistently across the product categories. The 

computation for market transformation is only used when shipment of units falls after a 

specification change. The market transformation concept as stated would also apply to units 

that increased in sales after a specification, which has occurred, but there was no computation 

of the savings derived from these units. The ENERGY STAR contractor’s formula does not 

account for market transformation savings for units that increase after a specification change. 

Furthermore, the OIG believes the savings reported from the market transformation units have 

a potential to significantly impact reported savings. We do not consider knowingly reporting 

unsupported and possibly incorrect data as a conservative practice even if it were properly 

disclosed, which is not done in ENERGY STAR reports. The number of formerly qualified 

units used in the computation is derived from reported shipments of ENERGY STAR qualified 

units. There is no actual relationship, it is an assumption. The result is a flawed process that 

will produce an inaccurate result, particularly given current economic conditions. 

 July 2007 

Report 

Total 

OIG 

Total 

Summer 

2008 

Review* 

Notes 

Boilers 382,770 405,573 382,770 Reported values add up to 382,770. 

No underestimation. 

Commercial 

Refrigerators/ 

Freezers 

83,493 106,658 83,493 

OIG appears to have 

• excluded one manufacturer’s data, because 

their brand name, rather than their 

manufacturer name, was listed on their 

reporting form which ICF addressed through 

their QA/QC process 

• excluded another manufacturer’s data because 

the company was not on the list of partners 

required to submit data, though this was 

addressed by ICF through their QA/QC 

process 

• Included one manufacturer’s reported data 

that was excluded as a result of the ICF 

QA/QC process. 

No underestimation. 

Dehumidifiers 1,203,216 776,431 1,203,216 Reported values add up to 1,203,216. 

No overestimation. 

39 
09-P-0061 

Exit Signs 1,425,554 3,450,117 3,460,176 

Spreadsheet error. Compiled number should 

have been 3,460,176. This underestimate has 

been corrected in subsequent analysis 

Furnaces 1,172,843 1,170,124 1,172,843 

OIG appears to have excluded data that was 

reported by one furnace manufacturer on their 

air-source heat pump data form which was 

addressed through the QA/QC process. 

No overestimation. 

Monitors 13,859,666 34,581,203 32,584,882 

Spreadsheet error. Compiled number should 

have been 32,584,882. This underestimate has 

been corrected in subsequent analysis 

Roof Products 

(Commercial 

Gallons) 

21,352,516 15,966,045 21,358,657 

Shipments reported by one company were 

inadvertently not included. ICF number should 

have been 21,358,657. 

No significant effect. 

Room Air 

Cleaners 287,354 289,815 287,354 

OIG appears to have included one 

manufacturer’s reported data that was excluded 

as a result of the QA/QC process. 

No underestimation. 

Transformers 173,390 166,669 173,390 

OIG appears to have 

• excluded one manufacturer’s data because 

their brand name(s) rather than their 

manufacturer name was listed on their 

reporting forms which ICF addressed through 

their QA/QC process 

• chosen to use data from one of the two forms 

submitted by one manufacturer whereas ICF 

confirmed with the manufacturer that the 

other form was the correct one. 

No overestimation. 

TV/VCR 17,685,967 13,166,140 17,689,230 

Shipments reported by one company for one 

product type (TV/VCR combination units) were 

inadvertently not included. ICF total should 

have been 17,689,230. 

No significant effect. 

Ventilating 

Fans 853,406 857,232 853,406 

OIG appears to have 

• included one manufacturer’s reported data 

that was excluded as a result of the QA/QC 

process 

• included incorrect data for one company that 

had been addressed through ICF’s QA/QC 

process. 

No underestimation. 

40 
 

09-P-0061 

* Re-examination of what shipment totals were supportable based on manufacturer information 

provided as of May 2007. 

OIG Response: The OIG analysis was based on the documentation used to produce EPA’s 

report of ENERGY STAR benefits in 2006. We do note that the Agency’s analysis does not 

address all the irreconcilable products as reported on by the OIG within Chapter 2, Table 2-1. 

Based on the Agency's chart provided, the Agency only reviewed 11 of the 18 products and thus 

the Agency's analysis does not address the differences noted for the following product categories: 

•	 Audio/DVD 

•	 Commercial Hot Food Holding Cabinets 

•	 End Use Products 

•	 External Power Supplies 

•	 Residential Light Fixtures 

•	 Roofing Commercial & Residential Sq. Ft. 

•	 Telephony 

Commercial Program Methodology (Addressed in Chapter 4 of OIG Report) 

OAR uses best available information to estimate program benefits in the commercial market and 

updates the methodology and inputs as the program evolves and new information is made 

available. These methods can be summarized as follows: 

•	 Program estimates are based on well-documented peer-reviewed methods. 

•	 The data inputs are either peer-reviewed or publicly available data themselves or based on 

reasonable and well-documented assumptions. 

•	 In areas of marginal uncertainty, OAR applies program evaluation best practice methods to 

avoid overstatement of program benefits. 

•	 The current program benefits estimation method includes techniques to account for the 

energy savings from commercial products associated with information processing and related 

equipment used by businesses. The method was peer-reviewed, found to be appropriate and 

sounds, and published in a 2007 Energy Journal article. 

•	 The current program benefits estimation method also includes techniques to account for 

benefits from energy efficiency programs across the country. The approach of subtracting 

state-utility savings from the national total was peer-reviewed and published in a 2001 

Energy Journal article. 

OAR disagrees with the OIG characterization of the methods used to avoid double-counting with 

federal programs like ENERGY STAR qualified products and DOE’s Building Technologies 

Program (BTP) as “unclear” and “questionable.” Again, these methods have been peerreviewed by experts in the field and found to be sound and appropriate. 

41 
09-P-0061 

OAR continues to improve the models it uses to estimate ENERGY STAR program benefits. 

OAR will undertake another peer-review of the overall approach and its key elements. 

OIG Response: The peer-reviewed methodology referenced by the Office of Air and 

Radiation does not directly support the claimed benefits for the ENERGY STAR commercial 

sector. The contractor who prepares EPA’s estimate uses parts of the methodology in its 

computation of the benefits. The contractor’s computation of estimated benefits for 

commercial sector is what needs to be validated. 

42 
09-P-0061 

Appendix E 

Distribution 

Office of the Administrator 

Principal Deputy Assistant Administrator, Office of Air and Radiation 

Agency Follow-up Official (the CFO) 

Agency Follow-up Coordinator 

Office of General Counsel 

Associate Administrator for Congressional and Intergovernmental Relations 

Associate Administrator for Public Affairs 

Audit Follow-up Coordinator, Office of Air and Radiation 

Deputy Inspector General 

43