From: Frey, Nathan J. <Nathan_J._Frey@omb.eop.gov> 

Sent: Friday, July 17, 2015 6:13 PM 

To: Culligan, Kevin; Harvey, Reid; Vasu, Amy 

Cc: Szabo, Aaron; Grossman, Andrea 

Subject: Summary of Interagency Comments Under EO12866 and 13563 -- Evaluation 

Measurement and Verification (EM&V) Guidance for CPP 

Attached please find an initial summary of interagency comments under EO 12866 and 13563 on 

the draft EPA Evaluation Measurement and Verification (EM&V) Guidance for Demand-Side 

Energy Efficiency (EE). 

 

Thank you and please let me know if you have any questions. 

Nathan 

<< SummaryofinteragencycommentsunderEO12866_13563_Evaluation Measurement and 

Verification Guidance.docx >> 
Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

1

Evaluation Measurement and Verification (EM&V) Guidance 

for Demand‐Side Energy Efficiency (EE) 

1) Some	commenters	do	not	believe	this	guidance	is	sufficiently	forward‐leaning	into	

advanced	EM&V	approaches.	This	is	a	rule	that	will	not	take	effect	for	another	seven	

years,	and	we	believe	there	is	ample	opportunity	to	incentivize	better	EM&V	than	

what	is	stated	in	this	guidance.	In	particular,	we	think	the	use	of	deemed	savings	

approaches,	particularly	those	that	are	not	grounded	in	empirics	and	those	that	are	

not	frequently	updated,	should	be	deprecated	or	at	least	discouraged.	Specific	

recommendations	are	

a.							To	disallow	any	deemed	savings	approach	that	has	not	been	empirically	

verified	by	a	rigorous	study	involving	RCTs	or	a	quasi‐experimental	design.	

b.						To	continually	do	empirical	checks	of	all	deemed	savings	quantities	based	on	

observed	outcomes	(perhaps	based	on	a	regular	sampling	of	installations)	and	to	

update	the	deemed	savings	values	based	on	the	result	of	those	empirics	(as	opposed	

to	every	three	years	as	is	suggested	in	the	proposal)	

2) In	general,	the	some	commenters	recommend	discouraging	any	EM&V	approach	

that	relies	purely	on	modeling.	This	is	particularly	important	given	a	recent	study	

that	shows	a	commonly	used	model	overestimating	the	savings	from	home	retrofits	

by	around	40%.	We	recommend	requiring	that	all	models	must	be	rigorously	

verified	through	actual	measurement	in	real‐world	applications	before	being	

allowed	to	be	used	for	EM&V	and	that	this	verification	be	continued	on	a	regular	

basis.	

3) It	would	be	really	helpful	to	have	EPA	explain	how	it	expects	to	use	this	guidance.	It	

is	described	differently	throughout,	including	as	“illustration	of	how	these	

approaches	should be	applied,”	“intended	to	help	states...implement	the	

requirements,”	“practical	clarification	of	the	preamble	and	emissions	guidelines,”	

and	“provided	to	help	EE	providers	develop	E&M&V	plans	that	document how	the	

requirements	will	be	implemented.”		

o Ultimately,	if	a	state	submits	an	EM&V	plan	that	doesn’t	meet	the	level	of	

rigor	reflected	in	the	guidance,	but	arguably	meets	the	broad	requirements	in	

the	rule,	will	EPA	approve	it?	Would	it	have	a	basis	for	rejecting	it?	

o If	a	state	submits	an	approvable	plan	but	in	practice	doesn’t	live	up	to	the	

guidance,	what	happens?	

4) We’d	like	to	strongly	discourage	the	use	of	top‐down	macroeconomic	studies	as	a	

primary	approach	to	estimating	savings	values	(Ken	accidentally	omitted	this	from	

his	comments).	Josh	and	I	had	suggested	complementing rigorous	bottom‐up	

approaches	with	periodic	top‐down	“cross‐checks.”	But	the	language	on	p.13	
Summary	of	Interagency	Working	Comments	on	Draft	Language	under

EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Review.

2

(labeled	p.8)	seems	to	leave	this	option	open	(although	“not	recommended”).	We’d	

like	some	stronger	language	here,	as	this	kind	of	approach	is	way	too	gameable.		

5) Reviewers	suggests	that	the	EPA	much	more	strongly	discourage	older	methods	for	

EM&V	of	EE	programs.	Specifically,	there	are	real	questions	in	the	current	academic	

literature	about	relying	on	“deemed	savings	values”	for	EM&V.	For	example,	

Greenstone	et	al.	(2015)	indicates	that	the	engineering	models	often	used	for	

estimating	the	energy	savings	of	the	Weatherization	Assistance	Program	over‐

estimate	the	energy	savings	by	roughly	40	percent.	If	the	methods	are	not	rigorous,	

this	could	significantly	reduce	the	effectiveness	of	the	CPP.	It	may	be	plausible	to	

expect	that	many	of	these	questions	to	be	resolved	in	the	next	several	years,	so	the	

commenters	suggests	wording	that	guides	states	to	use	the	latest	methods	and	

findings.	For	example,	wording	that	strongly	suggests	that	“deemed	savings	values”	

are	only	used	when	they	have	been	verified	with	estimates	from	quasi‐experimental	

designs	or	RCTs	would	be	very	valuable.	

6) Commenters	suggests	a	discussion	of	“rebound	effect”	should	be	mentioned	in	the	

guidance	and	that	appropriate	efforts	are	taken	to	incorporate	a	rebound	effect	in	

the	analysis.