From: Frey, Nathan J. <Nathan_J._Frey@omb.eop.gov>

Sent: Friday, July 17, 2015 6:13 PM

To: Culligan, Kevin; Harvey, Reid; Vasu, Amy

Cc: Szabo, Aaron; Grossman, Andrea
Subject: Summary of Interagency Comments Under EO12866 and 13563 -- Evaluation Measurement and Verification (EM&V) Guidance for CPP

Attached please find an initial summary of interagency comments under EO 12866 and 13563 on the draft EPA Evaluation Measurement and Verification (EM&V) Guidance for Demand-Side Energy Efficiency (EE).

Thank you and please let me know if you have any questions.

Nathan

<< SummaryofinteragencycommentsunderEO12866_13563_Evaluation Measurement and
Verification Guidance.docx >>

### Summary	of	Interagency	Working	Com ments	on	Draft	Language	under EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Re view.

1 

## Evaluation Measurement and Verification (EM&V) Guidance

for Demand

‐ Side Energy Efficiency (EE)

1) Some	commenters	do	not	believe	th is	guidance	is	sufficiently	fo rward‐leaning	into advanced	EM&V	approaches.	This	is	a	rule	that	will	not	take	eff ect	for	another	seven years,	and	we	believe	there	is	ample	opportunity	to	incentivize better	EM&V	than

what	is	stated	in	this guidance.	In	particular,	we	think	the	us e	of	deemed	savings approaches,	particularly	those	that	are	not	grounded	in	empiric s	and	those	that	are not	frequently	updated,	should	b e	deprecated	or	at	least	discou raged.	Specific

recommendations	are a.							To	disallow	any	deemed	savings	approach	that	has	not	b een	empirically verified	by	a	rigorous	study	inv olving	RCTs	or	a	quasi‐experime ntal	design. b.						To	continually	do	empirical	checks	of	all	deemed	saving s	quantities	based	on observed	outcomes	(perhaps	based on	a	regular	sampling	of	insta llations)	and	to update	the	deemed	savings	values based	on	the	result	of	those	e mpirics	(as	opposed to	every	three	years	as	is suggested	in	the	proposal)

2) In	general,	the	some	commenters	recommend	discouraging	any	EM&V approach that	relies	purely	on	modeling. This	is	particularly	important given	a	recent	study that	shows	a	commonly	used	model overestimating	the	savings	fro m	home	retrofits by	around	40%.	We	recommend	requi ring	that	all	models	must	be	r igorously verified	through	actual	measurem ent	in	real‐world	applications before	being

allowed	to	be	used	for	EM&V	and that	this	verification	be	conti nued	on	a	regular

basis.

3) It	would	be	really	helpful	to	ha ve	EPA	explain	how	it	expects	t o	use	this	guidance.	It is	described	differently	through out,	including	as	“illustration of	how	these approaches should be	applied,”	“intended	to	help	states...implement	the requirements,”	“practical	clarification	of	the	preamble	and	emi ssions	guidelines,” and	“provided	to	help	EE	provide rs	develop	E&M&V	plans	that document how the requirements	will	be	implemented.”

o Ultimately,	if	a	state	submits	a n	EM&V	plan	that	doesn’t	meet	t he	level	of rigor	reflected	in	the	guidance,	but	arguably	meets	the	broad	r equirements	in the	rule,	will	EPA	approve	it?	Would	it	have	a	basis	for	reject ing	it?

o If	a	state	submits	an	approvable	plan	but	in	practice	doesn’t	l ive	up	to	the guidance,	what	happens?

4) We’d	like	to	strongly	discourage the	use	of	top‐down	macroecono mic	studies	as	a primary	approach	to	estimating	savings	values	(Ken	accidentally omitted	this	from his	comments).	Josh	and	I	had	suggested complementing rigorous	bottom‐up approaches	with	periodic	top‐dow n	“cross‐checks.”	But	the	langu age	on	p.13

### Summary	of	Interagency	Working	Com ments	on	Draft	Language	under EO12866/13563	Interagency	Review.		Subject	to	Further	Policy	Re view.

2 (labeled	p.8)	seems	to	leave	thi s	option	open	(although	“not	re commended”).	We’d like	some	stronger	language	here ,	as	this	kind	of	approach	is	w ay	too	gameable.

5) Reviewers	suggests	that	the	EPA	mu ch	more	strongly	discourage	o lder	methods	for EM&V	of	EE	programs.	Specifically ,	there	are	real	questions	in the	current	academic literature	about	relying	on	“deem ed	savings	values”	for	EM&V.	F or	example, Greenstone	et	al.	(2015)	indicate s	that	the	engineering	models often	used	for estimating	the	energy	savings	of the	Weatherization	Assistance Program	over‐ estimate	the	energy	savings	by	roughly	40	percent.	If	the	metho ds	are	not	rigorous, this	could	significantly	reduce	the	effectiveness	of	the	CPP.	I t	may	be	plausible	to expect	that	many	of	these	questi ons	to	be	resolved	in	the	next several	years,	so	the commenters	suggests	wording	that	guides	states	to	use	the	lates t	methods	and findings.	For	example,	wording	that	strongly	suggests	that	“dee med	savings	values” are	only	used	when	they	have	been	verified	with	estimates	from quasi‐experimental designs	or	RCTs	would	be	very	valuable.

6) Commenters	suggests	a	discussion	o f	“rebound	effect”	should	be mentioned	in	the guidance	and	that	appropriate	efforts	are	taken	to	incorporate a	rebound	effect	in the	analysis.