Peer Review Documents: Meta

‐ Analysis of Willingness to Pay for Water Quality Improvements Page i DCN SE06136

Peer Review Documents: Meta

‐ Analysis of Willingness to Pay for Water Quality

Improvements

May 2015

ADDENDUM: REPORT TRANSMISSION MEMO

From: U.S. EPA, Office of Water

To: Steam Electric ELG Rule Record

Date: September 30, 2015

Subject: Peer Review Documents: Meta-Analysis of Willingness to Pa y for Water Quality Improvements

In general, the peer reviewers commended EPA fo r this work, and agreed that this work is important and difficult. To the extent practicable , the attached peer review comments have been incorporated into the final analysis for the St eam Electric Effluent Limitations Guidelines and Standards (ELGs). This memo explains a few additional points and supplements the discussion in Chapter 4 and Appendix H of the Benef it and Cost Analysis (BCA) document (DCN SE05977).

EPA asked the peer reviewers about whether th e Agency should impose the adding up condition ( i.e. , choose MRM2 over MRM1). The peer reviewers were not very concerned that the metaregression satisfy adding up. Nonetheless, EP A has a strong preference for imposing adding up, because it implies that different benefit-cost anal yses can be compared or combined and that these comparisons and combinations are meaningf ul. Thus, EPA selected MRM2 as its primary model for estimating water quality benefits of the ELGs. In the BCA document, MRM2 is referred to as Model 1. EPA developed a second m odel, referred to as MRM2-S in the memo “Revised Memorandum: Accounting for Scope in the MRM2 Meta-Regression Model” dated August 19, 2015 (DCN SE05985). MRM2-S is referre d to as Model 2 in the BCA document.

The peer reviewers expressed some concern that EPA was using the meta-analysis to predict outside of the range of the meta -data with regard to the size of the water quality improvement ( ln_qualitych in the MRM1 model). EPA understands the general concern expressed by the peer reviewers. EPA chose not to rely on MRM1 in the BCA, and in doing so, has addressed the peer reviewers’ concern. MRM2 (Model 1 of the BC A) and MRM2-S (Model 2 of the BCA) do not

include ln_qualitych as an explanatory variable. In these models, the willingness to pay (WTP) is divided by the water quality change to create the dependent variable for the model. The easiest way to think about MRM2 is as an estimating equation for a water quality demand curve. Picture a downward sloping curve on a graph with margin al WTP (price) on the y-axis, and the waterPeer Review Documents: Meta

‐ Analysis of Willingness to Pay for Water Quality Improvements Page ii quality level, Q, on the x-axis. We approximate these pairs by taking the average WTP over the range of water quality change in each study, i.e. , WTP/ Δ Q, as the marginal WTP at the mid-point of that change. (Recall that, by th e intermediate value theorem, there must be some Q within the range of the water quality change where the marginal WTP equals the average WTP.) Other variables on the right hand side of the estimati ng equation can be thought of as demand shifters. To apply the model for benefit transfer, we in tegrate under the marginal WTP curve between the initial and final water quality levels. That's how this model solves the adding-up problem (of MRM1): WTP calculated as the in tegral under a well-defined dema nd curve will have to satisfy the adding-up condition by construction.

Model 1 maintains the assumption that the underly ing data generating pro cess—the statements of WTP by the survey respondents in the original studies—complies with the adding-up condition.
That is, the functional form of Model 1 (in the BCA) was chosen in part to assure that the adding-up property holds for the estim ated benefit transfer function.

However, we can achieve a better fit to the da ta by including a measure of the water quality change as an additional explanat ory variable. Adding such a variab le has the effect of making the water quality demand curve undefined, since a well-defined demand curve (marginal WTP function) depends on the quantity of the good not the change in quantity. Therefore, a benefit

transfer function estimated in this way would not satisfy the adding-up condition. We can retain the adding-up property in such a be nefit transfer function if the qua lity change variable is held constant at a pre-determined fixed value— i.e. , it is treated as a dema nd curve shifter that does

not shift—so that WTP can be calculated as th e integral under a fixed demand curve. So as a form of sensitivity analysis we estimated this model, which we call Model 2 (in the BCA), and we examined the influence of choosing different le vels for the fixed value of the quality change

variable on the benefit transfer results.

Specifically, Model 2 is iden tical to Model 1 except ln( Δ WQ) is included as an additional explanatory variable. The estimation results for Model 2 are shown in Table 4. The coefficient

on ln( Δ WQ) is highly statisticallysignificant, and the R2 for Mode l 2 is significantly greater than that for Model 1. This suggests that either the underlying data generating process does not satisfy the adding-up condition (because of irrational be haviors by the survey respondents or one or more forms of hypothetical bias in the stated pr eference elicitation proces s itself), or there are one or more mis-measured or missing explanatory variables correlated with ln( Δ WQ) that if precisely measured or added to the model w ould cause it to satisfy the adding-up condition.

Closer examination of this issue will be an important area for follo w-up research. In the
meantime, we retain Model 1 as our preferred estimating equa tion for the benefit transfer

function because it avoids the n eed to arbitrarily specify the fi xed level of the quality change "demand shifter" when applying the model, and we used Model 2 to illustrate the sensitivity of

the benefit transfer results to this dimension of model uncertainty.Peer Review Documents: Meta

‐ Analysis of Willingness to Pay for Water Quality Improvements Page iii It is true that the Δ Q's examined in the underlying stated preference studies were always larger

than most of the Δ Q's we are examining in this policy sett ing. However, most stated preference surveys will ask about water qual ity changes that are big enough to register a WTP response that

rises above the "noise" inherent in the stated preference data generating process—that is, we typically wouldn't expect very accurate answ ers to SP questions focused on very small increments, so survey designers typically don't as k about very small increments. If we were to

take the admonition to "n ever extrapolate out of sample" along this dimension to heart, then we may find that NO existing SP studies can be used for benefit transfers. This seems like an unreasonably stringent standard.

It is also important to note th at MRM2 does not have the same kind of extrapolation problem. As explained above, MRM2 estimates a demand cu rve for water quality based on observations (approximations) of marginal WTP regressed on th e associated water quality levels. The water quality levels covered in the underl ying meta-data span a range that c overs most (if not all) of the water quality levels examined in the Steam Electric rule. That is, we use MRM2 to estimate WTP for changes between Q 0 and Q 1 for each census block group by integrating under the demand curve between those two levels. And the Q 0 's and Q 1 's for most or all of the block groups fall well within the range of Q's represented in the meta-data.

In general, the peer reviewers approved of th e inclusion of the GIS analysis and variables extending the underlying meta-data reported in the studies. However, the very brief explanation

of the ln_ar_agr variable left the reviewers questioni ng its inclusion. EPA has included this variable, because it is strongly significant, a nd has included the followi ng justification for its inclusion. Areas dominated by agricultural land uses have particular characteristics which suggest that WTP for water quality improvements could be lower than other types of areas, ceteris paribus. First, unlike non- agricultural rural areas, heavily ag ricultural areas have generally been altered from their natural ecosystem cond itions, and do not tend to be highly prized for water-based recreation (or character ized as pristine natural areas) – this would be expected to decrease WTP for improvements in agricultura l areas compared to many other rural areas. Second, unlike more heavily populated suburban or urban areas, agricultur al lands do not have the population base to support well-developed a nd used recreational areas. Improvements to water bodies in suburban areas, for example, are often highly valued because these areas support

extensive recreational a nd other uses. Third, a greater propor tion of the populati on in agricultural areas has employment linked to farm activities that may be associated directly or indirectly with water pollution. Those whose employment depends on farming activities may be hesitant to support programs to improve water quality (in stat ed preference surveys or otherwise), for fear that these policies may lead to greater restric tions on farms and farming activities. For all of these reasons, one would expect areas dominated by agriculture to have lower WTP to improve water quality, again holding all else constant. Th is intuition is str ongly supported by model results. Also, removing this variable causes subs tantial changes elsewhere in the model, a sign that removing this variable could cause non-tr ivial omitted variables bias. In such cases,Peer Review Documents: Meta

‐ Analysis of Willingness to Pay for Water Quality Improvements Page iv econometric guidance suggests that the variable s hould be retained rather than omitted; if the

inclusion of a variable within a model is questionable, the defau lt solution should be to include the variable (including an irreleva nt variable only reduces efficien cy; omitting a relevant variable creates bias).

Finally, EPA views its revisions to the meta-analy sis employed here as a work in progress. As part of the revisions, EPA has identified some issu es that require further analysis, and intends to

continue this work after the Steam Electric ELGs are promulgated. EPA also intends to have the Science Advisory Board (SAB) review the results of that progress when it has reached a point of completion where SAB review is appropriate. In particular, EPA has identified the following issues to address: conducting additional robustn ess tests and cross validation, investigating model over-fitting, investigating whether a distance decay effect can be gleaned from the metadata (to substitute for the 100-mile radius assump tion used in the benefits transfer application here), and to consider employing Bayesian estimation techniques.

## Peer Review Documents

Meta-Analysis of Willingness to Pay for Water Quality Improvements May 2015

Table of Contents

Kathleen Bell Review.............................................................................................................. 2

Kevin Boyle Review................................................................................................................ 9

Jacqueline Geoghegan Review.............................................................................................. 20

Timothy Haab Review........................................................................................................... 28

Laura Taylor Review............................................................................................................... 32PEER R EVIEW : META

‐ ANALYSIS OF WILLINGNESS

‐ TO

‐ PAY FOR WATER QUALITY IMPROVEMENTS

Thank you for the opportunity to conduct this review. My responses to the questions reflect my

expe rtise in environmental economics and spatial analysis. -Kathleen Bell

1

. The report presents two meta-regression models with distinct functional form s. EPA anticipates using one or possibly both of the models to estimate WTP for water quality changes that can be smaller than any of the changes valued in the primary studies. The models may also be applied to waters with baseline conditions outside of the rang e observed in the primary studies.

o Meta-regression model one (MRM1) uses WTP as the dependent variable and includes the natural log of the water quality change as an explanatory variable. As explained in Section 6.2 of the report, the MRM1 WTP function does not satisfy Diamond’s (1996) adding-up conditi on. The MRM1 function implies that marginal WTP goes to infinity as the size of the water quality improvement goes to zero. In the application of the model, this resulted in fairly large WTP values being assigned to very small changes in water quality. The linear interpolation appr oach described in Section 6.2 was developed to address these issues. Do you think it adequately addre ss these concerns about extrapolation for MRM1? Would you suggest any other alternatives?

o Meta-regression model two (MRM2) uses an approx imation of marginal WTP as the dependent variable and includes the associated water quality l evel as an explanatory variable. We formulated MRM2 as a way to address the adding up condition and the problem with large predicted values for small changes, albeit with some costs in explanatory power. How important do you think it is to satisfy the adding up condition in applied benefit transfers? Are the tradeoffs presented worth moving to a
model other than MRM2?

Please review the usefulness of the two models in the context of a benefit tr ansfer, and explain your preference between the two functional forms.

Response: The US EPA report presents two distinct meta

‐ regression models, referred to as meta

‐ regression one (MRM1) and meta

‐ regression two (MRM2). The report explains MRM1 in greater detail,

with extensive discussion of methods and assumptions. The discussion of the second meta

‐ regression model, MRM2, is less comprehensive and perhaps reflects its origins, a model developed in reaction to
the results and application of another model. Both models make use of the same dataset; these data

summarize studies focused on the willingness to pay for water quality improvements in the US. The major difference between the two meta

‐ regression models is the functional form of the ultimate willingness to pay function. Reviewing the report and summaries of these two models, I developed a
sincere appreciation of the challenges EPA faces when developing rigorous and robust policy analysis

tools for application in a variety of policy settings.

Meta

‐ regression model one (MRM1)

. Meta

‐ regression model one is an updated version of two prior

meta

‐ regression modeling efforts supported by US EPA (2004, 2009). This most current version incorporates updates that give greater consideration to the spatial aspects of valuing changes to water

resources, including the extent of water resources, water quality changes, substitutes, and population sampled. In general, the development of the model appeared to follow established guidelines oneconomics meta

‐ analyses. It is encouraging to know that the initial meta

‐ regression model was published (Johnston et al. 2005) and was therefore peer

‐ reviewed. The updates to the meta

‐ regression

model seem responsive to past concerns raised when applying the model during rulemaking. A clear advantage of this model is its grounding in curren t meta

‐ regression practices (a weak structural utility

theoretic meta

‐ analysis) and its linkages to the two prior versions which have been employed for rulemaking. That said, I believe meta

‐ regression models are most useful when applied to water quality changes that resemble those of the underlying studies. I have many questions about the utility of such

an approach for changes that are distinct from the range of the underlying studies.

Further, the incorporation of the spa tial variables seems like a big change for meta

‐ regression modeling. While these additions respond to increased awareness of spatial heterogeneity in benefits

and costs, greater sensit ivity analysis of such additions may be warranted to understand fully the repercussions of these additions and the sensitivity of results to different spatial measures and to
different approaches for considering the spatial aspects of changes in water quality and the valuation

of these changes. Similar to the issu es that arise around water quality changes beyond the range of the studies, there are likely analogous issues around the spatial measures. For example, it is not clear the

extent to which participants in the original studies were aware of the spatial factors now controlled for

ex

‐ post in this meta

‐ anal ysis.

The extrapolation approach described in Section 6.2. offers one approach to “avoid” the large WTP

values for very small changes in water quality. While the description is clear and this adjustment

would achieve the desired goal, I am not necessarily convinced by the current text that this is a prefer red strategy for extrapolation. The report mentions numerous options for responding to such

concerns. It would have been interesting to compare the results from multiple approaches. I remain puzzled by whether or not MRM1 should be applied to rules or policies involving such small changes

and have questions about whether, how, and wh en to extrapolate.

Considerable time and effort have gone into the development and application of multiple versions of

this meta

‐ regression model. Model development has taken place through multiple iterations, and

MRM1 has proved useful in rulemaking applications. Continuing to build on this foundation and to improve and test the approach under differen t assumptions seems quite productive.

Meta

‐ regression model two (MRM2)

. Meta

‐ regression model two, developed in response to the

concerns about the appropriateness of MRM1 to handle small changes in water quality and its lack of

satisfaction of the adding up condition, employs the same data but adopts a different functional form

to approximate marginal willingness to pay. I was surprised by the limit ed justification and explanation

of MRM2. Has prior work adopted this specification? Suggested this specification? I realize this modeling analysis is preliminary (and some of the standard errors and fit issues raise flags). But the

brief overview complicated my assessment of the approach, especially af ter having read a thorough description of MRM1.

As noted in the report, it would be extremely beneficial to see additional comparisons of MRM1 and

MRM2 under common assumptions. In closing, I remain less concerned about adding up conditions and

more concerned about the ability of the underlying data from these studies to get at what the agencyneeds.

The explanatory text of the first review question includes the following text ‐ EPA anticipates using one

or possibly both of the models to estimate WTP for water quality changes that can be smaller than any of the changes valued in the primary studies. The models may also be applied to waters with baseline

conditions outside of the range observed in the primary studies. Were the 2004, 2009, and current analyses done with small water quality changes in mind as suggested by the wording of the review question? While I fully recognize the utility and necessity of using meta

‐ regression models for rulemaking and policy analysis, I concluded my reading of the review pac kage intrigued by the challenges and weaknesses of applying the current models. I was also excited by the promise of future

efforts that might continue to test alternative meta

‐ regression models and incentivize primary research

of the type and sequencing of water quality chan ges more regularly induced by US EPA regulations.

2. The proposed benefit trans fer approach for both mode ls defines the commodity change to be valued as the change in average water quality within 100 mile s of the home. This implies that the quantity and quality of surface waters are perfec t substitutes, i.e., a 1 point impr ovement for 10 shoreline miles would be equivalent to a 10 point improvement for 1 shorelin e mile. Is this appropriate for a benefit transfer of water quality? If not, what would you suggest as an alternative, and how would you implement it?

Response: My initial response is that quantity and quality are not perfect substitutes. The length

‐ weighting has

both advantages and disadvantages. I have additional concerns about the basis of the 100 miles assumption and would like to see greater justification of this assumption.

I would also encourage gr eater restraint in discussing theoretical expectations related to spatial

outcomes. The report strongly asserts monotonic decreases in willingness to pay with distance. It is my
view that we have much to learn about “distance

‐ decay” and the definition and use of spatial measures such as the shoreline length weights used in the change in average wate r quality calculation.

Would be interesting to see how the results would change using different measures of commodity change or if the analysis included different measures, such as the size of the affected area.

The description of some of the spatial variables and the assumptions and steps used in the analysis

could be improved to help readers better understand how the commodity change to be valued is actually operationalized. The benefit transfer examples included in the report were very helpful and helped clarif y some issues. Clearly, more applications and examples help reveal critical details about

the usefulness and appropriateness of the benefit

‐ transfer approach.3. As explained in Section 6.1 of the report, EPA considers the blockgroup approach an improvement over previous meta-regression models applied at the state level. However, using Census blockgroups as the unit of analysis requires an exp licit choice to be made about the di stance within which water quality changes are valued by households (as indicated in question 2 above). We w ould expect that, for the typical household, the value placed on surface wate r quality will decline monot onically with distance from the home, so the threshold di stance used for the purposes of bene fit transfer should be chosen to approximate as closely as possible the integral unde r the willingness to pay cur ve as it declines over space. EPA selected a distance for the benefits transfer example in Section 6.1, and describes the rationale behind that distance. Is this a good assump tion for the proposed benefi ts transfer application? Aside from explicitly controlling for the distance of the waters from the home in the meta-regression model (which may not be feasible), is there a more appropriate way to control fo r the spatial extent of benefits?

Response:

I would like to see additional discussion of the rationale for the distance assumption in Section 6.1 (100 Miles). At present, the support is a single study’s finding about the majority of recreation trips occurring

within a 2 ho ur drive from home. Given the importance of this threshold distance assumption,

additional work should be done to research and document the appropriateness of this assumption for

benefit transfer analysis. This may involve more thorough consideration of use and non

‐ use values, and

greater reflection on how such an assumption might vary acr oss different water resources.

I am puzzled by the discussion of several spatial issues. Notably, why does EPA expect the value to
decline monotonically with distance from the home? There remain many questions about the

interactions of distance, use/non

‐ use, and value relationships. This assumption/assertion of monotonic

distance decay should be better justified. Our theoretical and empirical models have not advanced to provide us with such confidence.

Also, it is important to note that state

‐ scale analyses also make a distance assumption when bounding

values within state boundaries.4. Section 4 of the report describes the estimati on approach for MRM1, which is based on a random effects model to account for within-study correla tions of observed WTP estimates. Should other error structures or weighting schemes also be considered for either MRM1 or MRM2? For example, should the observations be weighted by the inverse of the number of observations drawn from their parent studies to give each study equal weight in the regression? Could y ou suggest additional variables that would distinguish variation across studies from varia tion between valuation scenarios within a study?

Response:

As suggested in the report, it would be informative to see more thorough, direct comparisons of MRM1

and MRM2, including results where common assumptions about error structures and weighting

schemes are made.

There may be benefits from including additional variables that would distinguish variation across studies from variation between scenarios within a study. I believe there would be valu e from understanding the extent to which spatial issues were stressed by the researchers and their methods (e.g., did the sampling specifically address relative proximity to the affected water resources (many

random samples will not necessarily give researchers great var iation in terms of relative distance and

other key variables)?; did respondents understand the spatial extent?). I am somewhat concerned

about controlling for issues ex

‐ post without knowing if the original study really emphasized items such

as the spatial extent of the affected water resources.

It seems as if there are real chall enges at coming up with refined definitions of affected water

resources and potential substitutes of those resources that can be consistently applied across many studies. MRM1 offers a valuable starting point from which to build on and change how such variables

are represented.5. Section 3.4 of the report describes several geo-s patial explanatory variables included in the meta regression model. These are water quality change (lnquality_ch), baseline water quality (lnbase), the relative size of the area (i.e., watershed) affected by water quality changes to the surveyed population area (i.e., state) to indicate extent of market effe cts (ln_ar_ratio1), proportion of the agricultural land in the affected area (ln_ar_agr), presence of subs titute water bodies (sub_proportion), and whether multiple waterbody types are affected (mult_bod). Several of these were constructed with supplementary information from outside of the or iginal studies based on descriptions of the study area. Do you think each of these variables impro ves the meta-analysis and subseque nt benefit transfer application?

Response:

Water quality change ( lnquality_ch) and baseline water quality ( lnbase ) are measured in terms of the

100

‐ point Water Quality Index. The report clearly documented the assumptions that were made for

each study. While some question the use of such indices and ladders, these tools permit

standardization across disparate studies. These variables are essential to the benefit transfer application and improve the meta

‐ analysis (though the parameter on the baseline variable was
sometimes not significant).

Relative size of the area (i.e., watershed) affected by water quality changes to the surveyed population

area (i.e., state) to indicate exte nt of market effects ( ln_ar_ratio1). I understand the motivation behind including extent of market effects and believe such effects improve a benefit

‐ transfer analysis. I

commend the researchers for doing the work to operationalize this measure. Yet, I am not entirely

convinced that this ratio, as currently measured

, gets at extent of market effects well. The description

and justification of this variable were confusing.

Proportion of the agricultural land in the affected area ( ln_ar_agr). I did not understand the explanation of this variable as a measure of development characteristics, and I was also not clear on the economic intuition behind the variable. While the presence of agricul tural land may imply limited development in some regions of the US, it would not make a good measure of this outcome in other regions, especially those that are highly forested and not particularly agricultural. Recommend using a

measure that more directly gets at the desired measur e (e.g., development or population or housing density or impervious surface). The NLCD development categories can be problematic; they are measuring cover and not use. They have tried to improve representation of low

‐ density development

but it can still be challenging.

Presenc e of substitute water bodies (sub_proportion) is a logical, appropriate addition to the analysis.

The continuous measure has the potential to outperform commonly used binary measures. Got a bit

confused about the spatial scale at which this proportion was calculated. In practice the variable

seemed to improve the analysis.

Whether multiple waterbody types are affected ( mult_bod)

. This binary variable distinguishes affected

areas including multiple waterbody ty pes and appears to improve the analysis. That said, it does seem

as if this effect could also be treated differently.EXTENSIONS BEYOND THE PRESENT APPLICATION (OPTIONAL)

9. Are there other geo-spatial facto rs that might influence WTP for wa ter quality? In particular, should direct measures of the distance to and size (area or shoreline length) of the affected water bodies be included as explanatory variables? Should EPA consider alternative c onstructions of a variable to measure the availability of subs titute waters (currently represen ted by the sub_prorportion variable, which uses waterbody type and size in comparing s ubstitutes), possibly also taking relative distance and water quality into account?

Response:

Strongly encourage additional consideration and research of additional spatial factors or distinct spatial

measures of the spatial factors/variables currently in MRM1 and MRM2 as you continue to develop

these meta

‐ regression modeling tools. However, I also suggest more extensive justification and testing
of such measures and encourage greater restraint in incorporation of such effects.

Including relative distance sounds appealing. Yet, in this context, summary measures of relative

distance may not work particularly well.Comments On:

Meta

- analysis of the Willingness

- to

- Pay for Water Quality Improvements

By

Kev in J. Boyle

May 24, 2015

Ove rall this is a well

- done analysis. I note the careful:

• Sel ection of studies, which is consistent with best practices in the literature;

• Use of GIS data to enhance the empirical analysis ;

• Coding of water quality, which is the foundation of the analysis and exceeds the standard practice in the literature; and

• The extensive robustness checks that have been undertaken to establish the credibility

of the estimated meta

- equation.

This is one of the best meta

- analyses of nonmarket values that has ever been conducted and is

the best meta

- analysis to support policy decisions of which I am aware.

My o nly concern is what appears to be a desire to make the meta

- analysis estimate values for conditions that the original studies were not designed to address, i.e., a lack of values for small changes in water quality and all original studies provide partial equilibrium value estimates.

Rather than making what may be ad hoc adjustments to an excellent meta

- analysis, these limitations highlight the need for EPA to initiate research to fill these major gaps in the empirical literature. To be clear, these limitations should not be used to condemn the meta

data and the meta

- regressions, but are simply omissions in the literature that cannot be explicitly addressed in the current meta

- analysis

.

I addr ess the charge questions provided by EPA and then provide some general comments. The general comments expand on my responses to the charge questions, suggest changes to improve the report, and provide items for thought. I noted a number of typos, none severe, throughout the report and a careful editing is needed.

Priori ty Q uestions:

1. I fi nd th at both MRM1 and MRM2 are potentially useful models. Each specification has strengths and each has weaknesses. I suggest below that a four

- quadrant table be developed that documents the strengths and weaknesses of each specification. Recommendations for use can be based on the content of this table. In addition, futureanalysts can review this in formation to make choices for use based on selecting strengths and avoiding weaknesses specific to the policy applications under consideration.

I ha ve a tendency to favor MRM1 because it avoids the ad hoc manipulations of the dependent variable and the potential biases this dependent variable enters into the estimation. Given this equation, it might be reasonable for any multiple increment change of K to value k ( QQ 0 → QQ KK ) and not the individual increments ( kk ∈ KK )

. Attributing benefits for individual increments could be accomplished by using the existing data to assign proportions with simulations to develop error bounds.

Es timating benefits for changes outside the range of primary studies demonstrates a critical need for the EPA to support studies to fi ll these empirical gaps in the literature. Construing models estimated with existing data to fit what these values might look like is not a good approach. However, it is not uncommon to make predictions outside of the range of observed data, but these pr edictions need to be used a face value with appropriate consideration for the embedded prediction errors. Given the timing of decision making and available budgets, this will not happen immediately, but it is an initiative that needs to be given a very high priority.

2. I e xplain in my comments below that 100 miles is a conservative assumption for both use and nonuse values and suggest some additional citations that support the conservativeness of this boundary. It may be that in cases where benefits and co sts are close to investigate how a change in this distance would affect benefit estimates to consider the robustness of

benefit

- cost calculations.

It i s difficult to know what the substitutes are. They could be another outdoor recreational activity or an y other consideration that people would spend their disposable income on.

Meta

- analyses rarely consider substitutes in the estimation of the meta

- equation and the approach used here is the best I have seen to date in any empirical study.

3. A co nsideration that is missing here in the report is that recreation demand models often use a distance greater than 100 miles for water

- based recreation (e.g., three hours driving time) and nonuse value studies have shown a distance decay of greater than 100 miles.

Thu s, while stated

- preference studies have shown that values decline with distance, the empirical results report an average for all study participants over all distances. Thus, if one is to consider a 100 mile boundary versus a 150 mile boundary (~3 hours driving), the 100 mile boundary study would be expected to have a higher mean value estimate than the 150 mile boundary assuming all other factors are held equal. Thus, if existing studies consider boundaries of greater than 100 miles the use of the meta

- eq uation to conduct benefit

transfers may underestimate benefits to this affected group and omit benefits of those located at a greater distance. Thus, the approach is conservative to benefit estimation.The exception to the logit I just outlined might be a small, local water improvement project. I do not think this is a limitation in the current context as EPA policies occur at a much larger spatial scale.

4. I w ould suggest weighting of MRM1 by the inverse of study observations as was done for

MRM2. The n umber of observations is potentially an arbitrary decision of the empirical investigator. However, you have a small number of observations for all studies where as

other meta

- analyses have had data where single studies each contribute large percentages

of the total observations. Original studies can influence meta

- equation estimates through the number of observations and being systematically different from the mass of observations provided by other studies. I think it will contribute to the robustness an alyses to consider estimates with and without weighting and horizontal, leave

- one

- study

- out robustness checks. Random effects estimation is fairly standard for meta

- analyses and I do not think another approach will substantially change your results.

5. Th e use of spatial variables is innovative and enhances the analyses

. I note my concerns

with ln_ar

- ratio1 below; this variable just does not make sense to me. Perhaps additional explanation could convince me otherwise. I have discussed sub_proportion above . While ln_ar_agr addresses and important source of nonpoint source pollution loadings, I am not sure what this variable is measuring because WTP is in preference space. Does this mean that people in rural, agricultural areas value water quality improvem ents less than people in

more urbanized areas?

Ex tensions beyond the Present Application:

6. Study selection is consistent with best practices in the meta

- analysis literature.

7. Th is study, while there may be data interpolation errors, represents the very best practice in incorporating the change values in the meta

- equation. It would be interesting, as part of the robustness analysis, to run the model without the interpolated water quality and consider how the results change. Would this limit the observed baseline water quality and the changes values? If yes, how so?

8. I a m not a fan of the term “publication bias”. I prefer publication “effect”. Perhaps some studies do not get published that do not meet some criteria, but the peer

- review process is

an im portant criterion for study quality. Is this a bias or is this sorting of quality. The likely limitation, for example, might be ce studies that do not find a significant coefficient on a water quality attribute. I would not be overly concerned about this issue given the careful study selection and the extensive robustness checks that are employed.

9. Ye s, it is worthwhile to consider other spatial variables and yes, include them as regressors. I think people consider standing and flowing waters differentl y and this is not included in the current analysis. Did the application include a particularly iconic water body? I do notsee how you can address distance explicitly since you do not know the location of each person. Perhaps you could do something with distance from city centers to each water body (center of lakes and center of river segments as proxies for distance for most people that would be in a general population survey.

10. Sta te of the art robustness approaches are used and the analyses go beyond what is proposed by Boyle et al. (2013) in terms of practical application. I think that this study should be submitted for journal publication as an example of applying state

- of

- the

- art
robustness checks.

Gen eral C omments:

• Section 2 – It would be good to explicitly state that you were only searching for stated

preference studies and explain why this choice was made. Also in this context explain that
you were seeking studies that estimated total value and explain the reasoning for this

choice.

• P. 3 – it is surprising that “value” was not one of the keywords?

• P. 3, bottom – While the steps you follow below are important to a credible meta

- analysis, they do not “ensure validity”.

• P. 4 – You might want to explain how you applied conditions such as “general (ly) accepted”
and “sufficient information” in addressing the six conditions.

• Sec tion 3 – It is standard to have two people code the data and then to have a third person resolve in any differences in coding to ensure data accuracy. Your statement about review by multiple individuals is not clear.

• P. 7 – Why was 2007 selected as the base year for dollars?

• Tab le 2 – “ thesis ” implies MS thesis a nd PhD dissertation?

• Tab le 2 – “ lump_sum ” seems odd coding to include payment periods of 5 or less years?

• Tab le 2, nonusers – recreational nonusers is odd terminology and potentially confusing.
Why not just say nonusers who do not participate in recreation activities.

• Tab le 2, ln_ar_ratio – Are both the numerator and denominator measure in terms of

counties?• Tab le 2 – Why not include a variable indicating if a decrement or increment in water quality
was valued? Is symmetry expected? Not typically done in meta

- analyses, but worth

consideration.

• Se ction 3.1 – I have found that the publication year is often know, but the year the data are
collections is not reported. How did you handle this?

• P. 12

– While not necessary for the current analysis, it would be interesting to investigate how uncertainties in coding changes in water quality affect the outcome of the m eta

analysis. I say that this is unnecessary here because best practices have been used, but as a
research question to inform and improve future meta

- analysis this could be a useful
investigation.

• Figu res 1 and 2 – There appears to be clusters in the data as noted in the text; 4 categories for the baselines and three categories for changes. Does this limit the general applicability

of the meta

- equation to support benefit transfers outside of the interactions of these data
groupings? That is, does the 100

- point water quality index provide a false sense of accuracy
in measuring changes in water quality?

• P. 13

– I h ave a concern with the statement: “ EPA expects, all else being equal, that WTP for water quality improvements would be directly related to the size and number of affected

waterbodies .” This need not hold. For example, a program for an individual water body may have a very specific action plan and outcomes whereas a program for an area of
waterbodies might, by definition, be more general and les s specific. This could lead to
respondent uncertainty about outcomes and the single water body having a higher value.
This a challenge for any meta

- analysis, not the just the current study. Such variations in the design of sated

- preference surveys are v ery difficult to model in a meta

- equation and as
such, caution is warranted in the hypotheses proposed.

• Tab le 3, Lg_reg – This simply represents everything that is different about this one study; an individual study fixed effect.

• P. 14, bottom – I am confused by the statement: “ Sa_area is not correlated with the size of the affected resource because various factors affect researcher’s sample selection including
budget and resource constraints and study objectives. As a result, the metadata does not

pro vide clear information regarding the extent of the market ...” There is no reason they
should be correlated so I do not understand the problem you are attempting to articulate

here.• P. 14, top – I am struggling somewhat with your use of sampled area and af fected resource
area. It would seem that affected resource area should affect welfare estimates; the
standard more is better argument. On the other hand, the sampled area should affect
aggregate welfare estimation. Given this, I am not sure the ratio re levant ?

• P. 15, bottom – Consider that substitutes may not be other water bodies. For a user it may be an alternative outdoor activity that is not associated with water, and for nonusers it may
be the current alternative for the discretionary income that they would allocate to nonuse
preferences. Addressing substitutes is good, but you should note that whatever is done is imperfect.

• Equa tion (1) – Why not model Q ojs as D js *Q ojs ? Ostensibly, people are asked to value the
change in water quality. Baselin e conditions should only come into play if the change
valued depends on baseline conditions, i.e., is not constant. This leads to the second question of whether Q ojs because you have already specified a nonlinear relationship
between willingness to pay and quality? This could explain the lack of significance you
discuss on the bottom of page 20.

• P. 19

– I disagree with the st atement: “... MRM1 is best characterized as being a weak
structural utility theoretic meta

- analysis ...” It is a reduced form, which i s the standard
approach in the literature, and you should not claim more than this.

• P. 20, 1 st full para. – Statement about the natural log of zero being zero is not correct, ln(0)<0 and ln(1)=0. Yes, the unlogged version of equation (1) (equation (3)) does force
WTP to be zero when the water quality change is zero. So, it does not force WTP through
the origin when the change in water quality is zero (footnote 14)

. Given the importance that this range of the function seems to imply for policy (e.g., the changes designated in Table 12) I would like to see a table with the refined water quality changes in the data that
provides more information than Figure 2. It would be good to also present a cros s tab of
the data in Figures 1 and 2 as an appendix table.

• P. 20

– It would be good to provide the major specifications investigated and a little intuition for the specifications considered instead of saying many were considered. What

, specifically

, were the model

- selection criteria? It would help the credibility of the analysis.

• Tab le 4 – the estimated equation is not truly a trans

- log function as coefficients are not reported for squared and interaction terms.

• P. 26, 7 th line – change to “... that are ‘potentially’ influential ...”• Table 5 – H ighlighting does not show well when the report is printed. Might want to consider a more explicit form of identification for influential studies.

• Figu re 6 – in addition to the general discussion of robustness results, I think it is worth paying specific attention to the lump_sum and ce results as these are investigator choices from the original studies that you must code for benefit

- transfer predict ions. Why are the
coefficients for these variables generally positive, but not always?

• Figu re 7 – Similar comments for variables that are potentially influential. The effects shown

for ln_ar_ratio and ln base are not surprising to me given the concerns/c onfusion I expressed above. I am not sure the former is measuring an appropriate concept and am not sure the latter should be included in the equation. Could the +/
- effects for the user
variables be due to some studies valuing increments and others decrements that are not
controlled for in the estimated equation?

• Tab le 10 – Discussion might cite Kaul et al. (2013) as it shows that your percentage transfer

error is consistent with benefit

- transfer validity studies in the literature ( http://su8bj7jh4j.scholar.serialssolutions.com/?sid=google&auinit=S&aulast=Kaul&atitle=
What+can+we+learn+from+benefit+transfer+errors%3F+Evidence+from+20+years+of+rese arch+on+convergent+validity&id=doi:10.1016/j.jeem.2013.03.001&title=Journal+of+enviro
nmental+economics+and+management&volume=66&issue=1&date=2013&spage=90&issn=

0095

- 0696 ). A better approach for computing benefit

- transfer errors is outlined by

Chatto padyay ( 2003 ): [((study case value - policy case value)/(study case value + policy case value))/2]* 100.

• P. 41

, top – I suggest also citing studies that show the distance decay for nonuse values to support your assertion that the estimate is conservative. Also, if you consider recreation demand RUMs they often use a three

- hour travel distance, which also support your

conservative assertion.

• P. 41, 2 nd para. – I wonder if for some projects a 100

- mile buffer is too s mall . I see the logic for the example computation here, but think that analysts should be encouraged to consider the size of the buffer in conjunction with the characteristics of the program being

valued.

• Figu re 9 – Needs improvement and legend to identify the meaning of the colors/symbols

u sed.

• Tab le 11 – I have several thoughts on the variable coding.o Use of a choice experiment or contingent valuation needs to be thought through
carefully in original studies and this translates to benefit transfers. There are two key

considerations

1) Do es the policy change to be evaluated call for implicit prices to value policies based

on various attribute levels? If Yes, then a ce

.

2) Do people think of the resource in terms of separable attributes or do they think of
an integrated ecosystem? If the latter, then cv

.
While a benefit transfer does not allow the flexibility of ce attributes, this can provide
some guidance about what method you might choose for the control. Further, I think
too little thought has been given to these two points and many of the applications for
EPA may be better valued through cv than a ce

.

o Thesis – I think some of the best studies I have observed are theses because the
graduate student focuses solely on this one study while more experienced researchers juggle multiple studies at the same time. Further, theses have intense peer review from

advisors and committee members. I do not have a concern with the coding, per se, but
the justification of “preferred” may be questionable.

o If w e believe valuation methods are improving over time and that these improvements
have the potential to remove overestimat ion of values , then coding time to the most
recent year may be appropriate.

• Tab le 12 – I suggest more explanation of the content of this table in the text to bring the reader along with you for these important policy descriptors in the meta

- analysis

cal culations. One directional sentence on page 41 is not sufficient.

• P. 46, top – the log

- log functional form is not used “... in virtually all ... meta

- regressions ...”
I believe that linear and log of dependent variable are also very common.

• P. 46, 1 st lim itation – T his is an artifact of the lack of data in this region of the functional relationship and the functional form chosen. It implies to me that this is not the “best” use

of the meta

- equation to predict values estimates because it is outside of the range of the
data and the wrong functional specification is chosen

. Rather than stating that “... future water regulations may result in smaller improvements ... (and) ... it is important that the
Agency be able to use MMR1 to accurately estimate WTP for small water quality
improvements ...”, this is a clear indication that the agency needs to pursue original research to value small increments in water quality to fill this gap in the empirical literature.• P. 46, bottom – Another alternative to the linear interpol ation is to simulated what values
might be in the lower range and re

- estimate the meta

- equation as having these data could
change the shape of the whole function, which is an issue that that has not been addressed.

• P. 47, 2 nd limitation – It is important to remember that adding up is based on an assumption about preferences imposed by the assumed economic model and is not required by the
general axioms of revealed preferences. Further, the meta

- data were not estimated in an
experimental setting where adding up was imposed by the study designs nor empirically
investigated as test of the underlying preferences.

• P. 48, top – I recognize your challenge with addressing multiple programs that each makes
small, marginal improvements in water quality. This is a long

- standing concern, even with
original studies, given the partial equilibrium nature of the empirical estimates. It seems to
me that it is important to ask whether you are going to treat the changes as a sequence of
small changes or as one big collecti ve change. If it is a sequence, then it is important to get
the sequence correct and then apply value estimates appropriately to value each change. If
it is one large change then it is possible to dispense with the sequencing issue. Here I see

two funda mental concerns.

o Valu e estimates for small changes are likely biased due to the absence of original
benefit estimates for small changes and consequently a meta

- equation that might be incorrectly specified. By the latter, I mean the functional relationship for small value
changes is an artifact of the values observed for larger changes.

o Yo u are taking partial equilibrium estimates to estimate the meta

- equation and forcing economic logic/conditioning on the econometrics summary of these estimate s that was
not imposed in the original estimation. Now, a meta

- equation can be used to test
adding up, but this is a very strict test because it forces adding up to hold across studies,
but adding up is innately a within

- study phenomena if it exists.

Rat he r than forcing the meta

- equation , and by consequence the existing value data, to address the adding

- up condition, which the original studies were not designed to address
and may be inconsistent with underlying preferences, I think the alternative approac h
considering marginal values is reasonable

.

• Equation (8) – This equation provides some important considerations.

o First , if we assume the WTP function is concave, consistent with the specification of
MRM1, then marginal willingness to pay is consiste ntly underestimated by this linearinterpolation process given the available data. The underestimation increases as the
change in water quality increases.

o Th e concern that data for small changes is limited and the estimated function is driven
by value estimates for larger changes remains; the estimated MRM2 function may also
be misspecified.

o The specification of the dependent variable, which includes components of Q js suggests that water quality is no longer a truly exogenous regressor.

Th ese concerns a pply to equation (9) as well. However, I believe that it would be
worthwhile carrying through the nonlinear estimation to confirm your supposition that the
end results would not change much.

• P. 49, 1 st para. o f Section 7.1 – Here I think that excluding ln Q 0s is a concern as the marginal value does depend on the change from the base water quality condition.

• P. 50 , 1 st para. o f Section 7. 2 – It is worth noting in the text that most met

- analyses do use
cluster and that boot strapping is not as common, and that studies in the literature have
fewer observations that are available to use here to place these analysis procedures in
context. As written right now a uniformed reader may have the tendency to dismiss the
cluster analysis.

• P. 51, top – discussion of the outlier observation is unnecessarily confusing.

• P. 51, 2 nd full para. – “The coefficient on Q is negative ... indicating a downward sloping

demand curve.” WTP is not conditioned on the baseline quality so and you are making
comparisons across studies and not within a single study. While this negative coefficient is reassuring information you should be careful how you frame this discussion.

• P. 52, top – It is also worth reiterating here that income may not income as observed in the study data. In addition, noting that full income may not be the income that people used in
answering the valuati on questions, e.g., they may have considered available disposable income after all current bills were paid. All the same, considering other specification to see
if income becomes significant is worthwhile given the important role of this variable, but

the above caveats indicate why it may remain insignificant for very plausible reasons given

the available measure of income.• P. 56, 1sr para. – That MRM2 does not fit the data as well as MRM 1 Is not surprising given the error introduced in the computation of marginal WTP and potential additional
misspecification of explanatory variables.

• Figu re 13 – Label panels for MRM1 and MRM2.

• Tab les 18 and 19 – while I appreciate all of the information you present in the tables, it
would be nice to see the tables here, and elsewhere in the report, presented in single tables

to facilitate comparisons of models and estimation procedure results. I found is difficult to
make comparisons across pages and an underlying theme of this report is to consider which

model is best from a conceptual and empirical perspective. Such merging of tables would
facilitate the comparisons of estimation results. It would also be helpful to have a four

quadrant table that lists the strengths and weaknesses of the MRM1 and MRM2 equations.

I know the MRM2 model was a late addition so I assume you have not had enough time at this point to flush out such an important comparison.Peer Review for the U.S. Environmental Protection Agency of: “Meta-analysis of the Willingness-to-Pay for Water Quality Improvements”

Jacqueline Geoghegan, Ph.D.

Professor and Chair of Economics Clark University, Worcester MA

May 28, 2015

**Introduction**

As I discussed with Professor Kiel at the onset of this peer review, I am not an expert in stated preference, meta-analysis nor benefit transfer. Rather, what I bring to this peer review is twenty years of experience in spatial environmental economic modeling. 1 Consequentl y, the majority of my comments will focus on the spatial data and modeling aspects of the reviewed document, although I will make other overall comments when warranted.

Most generally, it became immediately clear to even a non-expert like myself, that “MRM1” is a well-established, peer-reviewed model that has undergone extensive and varied robustness tests, while “MRM2” is at a very preliminary stage of development. Indeed, as EPA writes on page 56 “MRM2 was a late addition to this report, so EPA views the results presented in this section as still preliminary since they have been only partially stress tested.” Therefore, a complete peer review on “The Battle of the MRMs” at this juncture would be not overly productive. However, in the conference call of 05/05/2015, (which I was not able to participate in, as it was scheduled for a time that I was not available, but I listened to the recording of the call), EPA staff provided the peer review panel with background comments on the context of the models as well as indications that EPA has already established a planned line of research for moving forward with these two models. Furthermore, as EPA writes on page 56 “EPA intends to apply more sophisticated estimation approaches to MRM2 in follow up work.” Finally, on page 66, section 11 “Next Steps”, one of the two examples included is “incorporation of distance decay”, a specific spatial data refinement, that I hope my comments below can assist EPA as the Agency moves forward with this work.

Given this background, while I will answer the “Priority Questions” and “Extension” questions to the best of my ability in this peer review, many of my comments will be in the spirit of suggestions to EPA for future research (specifically in regard to the spatial aspects, such as comments on distance decay) in meta-analysis and benefit transfer applications, either in regard to water quality issues as in this specific case, but also more broadly to the overall mission of the Agency to protect human health and the environment. In addition, there are some overlaps in the spatial aspects contained in the Priority Questions to the peer review panel of the report (i.e.

1 Geo ghegan, Wainger and Bockstael (1997) was the first paper in environmental economics to use GIS data and spatial analysis. As of May 2015, it has been cited 636 times (google scholar). In addition, any benefit estimation work that I do has always been in a revealed preference framework (i.e. hedonic models) with related work on air quality and open space amenities, .so no specific research experience with water quality either.Que stions #2, #3 and #5 all refer to spatial issues), resulting in some repetition in my responses to these questions, especially with regard to spatial scale issues and the spatial unit of analysis. I also note that it is likely that many of my suggestions might require more extensive GIS expertise and resources than are currently available to EPA, but I believe that the investment by EPA (and more broadly throughout the public sector) in this burgeoning area of spatial data creation and analysis is highly critical for future data collection, management, and analysis.

Overall, upon my first read of this EPA document, as well as other additional background material on meta-analysis and benefit transfer 2

, I was quite surprised by the limited amount of research that has been performed in these applications on environmental quality that has taken either a spatially-explicit or more broadly, spatially-informed approach to these methods. Space must play a critical role in environmental MRMs for benefit transfer and policy analysis, with regard to the geospatial scale of the impacted resource, the market extent of the potential residents who value such changes and the extent and availability of substitutes for the impacted resource. The MRMs presented in this report, which begin specifically to include these critical spatial characteristics, I believe, are a clear improvement over previous MRM models used for

benefit transfer.

**Responses to Priority Questions**

1. As this questi on is specifically on the functional forms used for the MRMs, I will leave this question to other panel members with expertise in this area to address this question in detail. The numerical example given in section 9 of the report is illustrative, but given the nascent stage of the development of MRM2, I am not sure if the current example is a useful comparison of the two models. As EPA moves forward with both of these models developed further, different simulations, such as the one given in the report as an example, by using each model with the same benefit transfer scenario, could help determine if the empirical outcomes dominate any potential theoretical effects relating to the issue of the adding-up condition.

However, more broadly, as an applied environmental economist, I do have more general

concerns on the use of any model to predict impacts so far outside the range of observed metadata. I realize the constraints that meta-analysis researchers and EPA face: that there are no primary analyses that use baseline or changes in the range that EPA policy considers in this specific application. I would encourage EPA then to consider funding research in the future that would be helpful for the development of data (and methods, see my comments below) that would be in this range for policy analysis and implementation.

2. I view Question #2 as having two parts. The first part, explicitly stated in the question, concerns the modeling of the trade-offs in preferences between a measure of water quality and the (linear) area affected. The second part, not explicitly stated in this question, but is one of the main points of Question #3: “is 100 miles the ‘right’ spatial distance to use in this model?” For the first part,

2 Nelson and Kennedy (2009) and Johnston and Rosenberger (2010).conc erning the trade-offs, it is an empirical question about how individuals value these two aspects. Are there any stated, revealed or experimental data to support this modeling choice of these two aspects as perfect substitutes for each other? If not, can different weights be used to experimentally test how much the different specifications actually matters for predicted

outcomes?

The water quality index could also be a function of surface area or some other measure of “size” or water body type. For example, my intuition is that a “shoreline mile” for a river is affected more by a certain change in water quality than a “shoreline mile” of a large lake (i.e. where the benefits would be more diffuse over the larger area). On the other hand, should each “side” of a river “count”, should this be a function of river width? On page 14, EPA writes: “By delineating resources using GIS, EPA is able to incorporate detailed spatial and hydrological features of the waterbodies.” What exactly is meant by “detailed”? Related, has EPA done any sensitivity analysis with these “details”, such as different variable specifications based on the type of surface water along these ideas? On page 15, EPA writes: “EPA considered, but did not select, a variety of alternative specifications of the geographic scale and extent of market variables (e.g., shoreline length).” What does “considered” mean? EPA thought about different specifications and did not try, or implemented different specifications and that the specifications did not improve the model? 3

More broa dly, while much documentation in this report is given on the different standard (for meta-analysis) robustness checks for MRM1, what other sensitivity analysis were done on the new spatial variables included in MRM1? As this approach of using external spatial data (i.e. not included in the original studies) and GIS-created variables using these data appears to be so new in this meta-analysis literature, there might not exist standard or agreed upon methods on this specifically for meta-analysis, as there appears for “traditional” meta-analysis research. 4

The refore, if the spatial approaches and methodologies suggested in MRM1 are to be used more widely by other researchers and/or in other applications, reported sensitivity analysis would be helpful in order to develop such protocols for future applications. This is another area of research that EPA should consider for future work: the further development of methods for determining how best to include these important spatial features of the landscape derived from readily available, external GIS data into meta-analysis and benefit transfer models.

The not-specifically-asked second part of this question is about the choice of the 100 miles cutoff, I will discuss in my response to Question #3 next.

3 M y comments here also are reflective of Extension Question #9 that specifically asks for suggestions on this variable and specification.

4 Su ch as is found in Navrud and Ready (2007) and Johnston and Rosenberger (2010).3. This question beg ins: “As explained in Section 6.1 of the report, EPA considers the blockgroup approach an improvement over previous meta-regression models applied at the state level.” However, I don’t see any argument making that case in this section. 5 (Per haps this is due to some mislabeling in the document, as this is the section of the document with an “August17August17 ( sic ), 2012” and “316(b) Stated Preference Survey Peer Review Document” in the footer?) But more specifically, I believe that in MRM1, Census data are used only to construct an internally consistent income measure for each observation in the meta-data, which is a reasonable thing to do. In any event, with regards specifically to benefit transfer (which is what section 6.1 contains), using blockgroups (BG) as unit of analysis for the creation of specific values for variables seems reasonable, as this is a generally recognizable and available spatial (GIS) geography for the basis of analysis. In general, as an applied micro-spatial environmental economist, all else equal, I prefer to use the smallest available spatial unit of analysis for research. However, perhaps as EPA moves forward with this work, some additional types of sensitivity analysis on the spatial level of aggregation (i.e. using larger geographies) to determine how influential this choice is on the empirical outcomes.

In Section 6.1, EPA writes “It is assumed that households in each BG value changes in the affected waterbodies within a specified distance, here set to 100 miles. A radius of 100 miles is consistent with Viscusi et al. (2008) based on the assumption that the majority of recreational trips will occur with a 2 hour drive from home.” (p. 41). In reading Viscusi et al. (2008), the authors state that they used the 100 mile distance in their survey, as US EPA NCEE had calculated that about 80% of recreational trips contained in the 1996 National Survey on

Recreation on the Environment occurred within the 100 mile radius. These data are almost 20 years old. It appears that this survey was done again in 2010, so perhaps EPA could consider analyzing these newer data to determine if these distances are stable over time. That said, recreational use is only one of the uses and non-uses that EPA is interested in valuing, so it is unclear to me that this is enough of a reason for the choice of 100 miles.

More generally, beyond being an empirical question of what the “right” distance should be, it is

also unclear if it should be a constant fixed distance for all locations in the United States. The extent of the market is an empirical question so perhaps smaller buffers for smaller watersheds and/or different population densities etc. For example, the “right” distance could be a function of population density and related congestion effects. 6 That is, there are (at least) two potential types of congestion effects: one is of the resource itself (in higher population density areas, a resident would have to travel further to reach an uncongested site if that is preferred) the other is the transportation costs of access, if road congestion is greater in higher population density regions, this would increase the time costs to reach a destination. Therefore, another potential robustness check would be for EPA to experiment with different sized buffers, and not

5 On page 5 6 EPA discusses the use of an “exogenously specified threshold distance cutoff when using the metaregression models described in this report for benefit transfers.”

6 I focus here on population density as an example, as population counts are readily available and population densities easily created in a GIS.nec essarily constant across all observations, or interaction terms, although I realize that there are potential issues with degrees of freedom and over-fitting of models here. Again, my point in response to Question #2 on sensitivity analysis of MRM1 applies here.

In this panel review charge question, EPA writes “We would expect that, for the typical household, the value placed on surface water quality will declined monotonically with distance from home...” Is there any justification for EPA’s claim of “monotonically”? Why not quadratic? Or any other functional form? Again, this relates to the specific values under consideration: distance decay might be appropriate for recreational or other use values, but not necessarily for all values. For example, as Johnston and Rosenberger (2010) write “it is possible for certain public goods to have a constant average impact on residents’ utility within a given geographical market that does not decline – or changes by only a trivial amount – over distance.” (p. 492). Therefore, while a distance decay function should be experimented with, there are other options that are possible that could be tried in the MRMs to test for scalar effects. For example, as the spatial extent of a BG could vary with population density (as BG are delineated by population and not geographical area), could average BG spatial size be included on the RHS of the MRM? If that matters in the MRM, then perhaps that would be evidence in the benefit transfer for having the distance decay or buffers be also a function of population density? (For the reasons given in the previous paragraph.)

4. While this question is outside my area of expertise, the GLS approach used in MRM1, given the constraints of the underlying data, appears appropriate. However, I was confused by the specifics of this question. In footnote #12 on page 19, EPA writes “Weighting observations based on the number of WTP observations reported in each study produced nearly identical results.” So hasn’t EPA already performed some of this kind of sensitivity analysis?

5. This question lists a number of the spatial variables developed for MRM1 and resulting benefittransfer. The peer reviewers are asked “Do you think each of these variables improves the metaanalysis and subsequent benefit transfer analysis?” Well, generally, yes, given my assorted comments above on my view of the value of adding spatially-explicit relevant variables. Do I

think that “each” variable individually “improves” the analysis? I do not think it is currently possible to answer that particular question, although I have given a number of suggestions above and to follow here for other specification and the need for more sensitivity analysis. Any variable should be included in a model for theoretical reasons, but in many cases, it is an empirical question of the “right” form of a variable is to use.

Of particular concern is the current “proportion of the agricultural land in the affected area”. First, I didn’t really understand the description and rational for the benefit transfer example (given on page 44 in Table 11). But more importantly, EPA includes this variable, but there appears to be no theoretical justification specifically articulated by EPA in this report for the inclusion of this variable. EPA writes on page 16 “The agricultural fraction is calculated as the area-weighted average of counties within the affected resource area. EPA also developed and tested an analogous variable based on the fraction of land that is in developed land use categoriesbut it not ( sic ) improve model fit.” Again, there needs to be theoretical justification for the inclusion/exclusion of variables, including these land cover variables, and not just simply including “available” data to create variables that “improve model fit”. Why should these variables affect WTP? It would be helpful for EPA to articulate this here. In addition, similar to my previous comments above, what kind of sensitivity analysis with these variables were performed to determine suitability for inclusion in the MRM or how variation in them affects specific benefit transfer applications?

**Responses to Extension Questions**

Ex tension Questions #6, #8 and #10 are specific questions concerning meta-analysis and Extension Question #7 focuses on water quality measures, which is also out of my area of expertise, so I will leave these to the other peer review panel members.

My contributions to these Extension Questions focus exclusively on Question #9 which asks about “other geo-spatial features that might influence WTP for water quality?” I raised a number of questions and potential suggestions regarding these issues in my responses above to the Priority Questions. Additional comments and suggestions are included next.

On page 15, EPA writes: “... EPA used intersecting counties to define all affected resources, including discrete rivers, as geographic areas while avoiding the arbitrary selection of a boundary distance.” It is unclear to me why county boundaries are particularly less “arbitrary” than a fixed pre-determined and/or variable/experimental distance, as county boundaries can be arbitrary as well. Counties can vary greatly in size and in some areas of the country. In addition, while counties exist throughout the United States, they can play almost no jurisdictional or social/political role in some states (e.g. in Massachusetts, the key identification is the “town”). The affected resources could be measured by boundary distance or some function of population density as well (as discussed in my responses to Priority Questions above).

On page 15, EPA writes: “... EPA considers the state to be a reasonable basis for the calculating ( sic ) the substitute proportion.” While I agree with EPA’s discussion of the important of spatial/cultural homogeneity, it is unclear to me if state boundaries capture this to the best degree possible. Again, similar to counties, states can vary greatly in both geographical and population size across the United States. For example, Rhode Island is tiny, so should the available substitutes be all of New England instead of only that state? Or for Rhode Island, should the available substitutes include the states of southern New England (i.e. associate Rhode Island with Massachusetts and Connecticut)? On the other end of the spectrum, California is a large land area with a huge population: Northern California is very different than Southern California. Perhaps EPA could consider comparing their state approach to determining geographically-available substitutes to a definition based on OMB’s Metropolitan Statistical Areas (MSA) as these are created to contain geographical areas that have a “high degree of social and economic integration” ( www.census.gov ), which can cross states lines and could be a better

fit for these areas than state boundaries.On pag e 16, EPA writes: “For non-river inland waterbodies (e.g., lakes and ponds), the substitute proportion is calculated based on area relative to areas of lakes with the states(s).” Is

this done just for lakes, or are “ponds” included too? For example, has EPA considered subdividing and/or weighting these data by the USGS Hydrologic Unit System (another potential sensitivity analysis)? While EPA writes on page 14 “For rivers and lakes, the Agency opted to match to the NHDPlus dataset over other publically available georeferenced hydrological datasets (e.g., E2RF1) because...” but given the importance of these data for this application, EPA might consider adding more detail on the different datasets available and the pros and cons of the use of each data source. Or perhaps make clear these are the data that are generally used by EPA for these kind of purposes.

On page 23, EPA discusses different potential sample selection biases of the studies included in the metadata, and included on page 24 is a map of the United States where the original studies occurred, but there is no mention of the potential of over-sampling of states in which specific researchers live. EPA asserts on page 23 “a relatively good geographic and waterbody type coverage of the studies”, but considering the dominance of studies in North Carolina, Virginia, Maryland, and West Virginia and the paucity of observations from the western half of the country, perhaps EPA could do a fuller justification for this claim? Are there any MRM and/or political implications for extrapolating these data to other states? Have MRM researchers developed robustness tests to consider this aspect? More broadly, even within each study (observation for the MRMs), for sample areas by both jurisdictional and watershed boundaries, how was the survey population samples in the original studies? That is, where the survey respondents selected based on any specific spatial criteria? Or does EPA simply have to assume a uniform spatial distribution of survey respondents across the sample area? If so, has EPA considered how the spatial sample compares to the spatial distribution of the population?

**Conclusions**

As I wrote in my introductory comments, while I am not an expert in meta-analysis and benefit transfer, I am convinced that the incorporation of spatial/geographical characteristics into these MRMs and resulting benefit transfer models is a vast improvement in both theory and application as well as subsequent use of this approach for policy and decision making. There are three broad areas of research that EPA should focus on as they proceed: the first two in the shorter term and the last is a longer term issue. The first is the further development and testing of MRM2 and how it compares to MRM1 for a more even comparison of the two model specifications to determine if there is indeed a trade-off between theory and empirics (already self-acknowledged by EPA in this peer review document as well as earlier phone call discussed above). The second is for the further methodological development of protocols for the inclusion of these new additional externally-generated GIS variables into meta-analysis used for benefit transfer. The third, longer term issue, is for the support for the new original research necessary to create observed data in the relevant range that EPA needs for policy analysis.**Re ferences** 

Ge oghegan, Jacqueline, Lisa Wainger and Nancy Bockstael. 1997. “Spatial Landscape Indices in a Hedonic Framework: An Ecological Economics Analysis Using GIS,” Ecological

Economics 23(3): 251-264.

Johnston, Robert J. and Randall S. Rosenberger. 2010. “Methods, Trends and Controversies in Contemporary Benefit Transfer,” Journal of Economics Surveys 24(3): 479-510.

Navrud, Stale, Richard Ready (Eds). 2007. Environmental Value Transfer: Issues and Methods

. Springer: The Netherlands.

Nelson, Jon P. and Peter E. Kennedy. 2009. “The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: As Assessment,” Environmental and Resource

Economics 42: 345-377.

Viscusi, W. Kip, Joel C. Huber and Jason Bell. 2008. “The Economic Value of Water Quality,”

Environmental and Resource Economics , 41(2): 169-187.Meta-Analysis of Willingness to Pay for Water Quality

Improvements Responses to Charge Questions -Timothy Haab

Priority Questions:

1) Priority question 1) consists of two parts. In part A we are asked whether the use of linear
interpolation in MRM1 adequately addresses concerns about extrapolation beyond the realm of
observable water quality changes. In part B we are asked to assess whether using an approximation of marginal willingness to pay in MRM2 adequately addresses concerns about MRM1 not meeting Diamond’s (1996) adding up condition and the problem of MRM1 predicting
values for small water quality changes that are perceived to be ‘too large.’

Before directly addressing either of these issues, I would like to first give a brief comment on Diamond’s adding -up condition in the hopes of ruling it out as a deciding factor between the
two models. In short, the adding-up condition is an extreme form of the NOAA Panel’s 1993
recommendation that contingent valuation results in natural resource damage assessment should show adequate responsiveness to the scope of the resource being damaged. In the current context, this should be interpreted as there be a positive relationship between water
quality changes and willingness to pay. This is clearly the case in MRM1 as evidenced by the positive and significant coefficient on lnquality_ch in Table 4 on page 22. Unfortunately, the

NOAA Panel did not define what is meant by ‘adequacy to scope.’ Some have interpreted Diamond’s adding -up test to be the gold standard for scope tests. EPA seems concerned that MRM1 does not meet the more stringent conditions of Diamond’s adding -up test. This is not

correct.

In a response to Hausman’s 2012 piece in in the Journal of Economic Perspectives , Haab et al (2013) address the issue of the adding up test and more generally defining adequacy to scope.
They point out:

 “A simple theoretical model of WTP...can be used to show that WTP is nondecreasing in quality or quantity.” (Whitehead et al, 1998)

 “The size of scope effects may be limited by diminishing marginal utility or substitution among nonmarket goods...” (Rollins and Lyke, 1998)

 “The size of the scope e ffect in [a formal model of scope with bounded utility functions] depends on the substitutability between market and nonmarket goods.” (Amiran and Hagen, 2010)

In related discussions, Haab et al note that since 1996, very few researchers have chosen to tes t CV with the adding up condition. It is not the standard test of scope in the literature and as such it should not be used as the sole means of accepting or rejecting model specifications.

As such I find very little convincing evidence in the report to distinguish between MRM1 and
MRM2 based on adequacy to scope responses or the adding up condition. Both models exhibit responsiveness to scope consistent with what has been found in the literature since 1996.

This leaves the question of whether MRM1 or M RM2 provide a ‘better’ means of interpolating changes in water quality beyond the observed studies (in this case, changes that are smallerthan those that are published). Before addressing this question directly, a caveat: the fact that

there are very fe w published or ‘grey’ studies that value small changes in water quality raises
the issue of publication bias in the chosen studies. Beyond the issue of interpolation, this raises concerns in determining adequacy of scope. If only studies that pass a scope test (or value an adequately large change in water quality) are published, then of course the meta-analysis will find a significant response to scope (or fail to capture small changes in water quality adequately). While not limited to the current study, publication bias raises questions for policy application of meta-analysis, particularly for valuing changes beyond the observable realm of
existing studies.

Prediction beyond the realm of observable data has plagued the stated preference literature since its early days. As such, there is very little guidance in how to handle out of range prediction. In any parametric setting, the out of range prediction is going to be highly sensitive to the chosen functional form. Bounded functional forms (like the log-log and semi-log) are notorious for sensitivity in the tails of the error distribution. Fixes to functional forms to simply make the results look better (pass the laugh test?), result in arbitrary results subject to easy

criticism.

Based on this, I find little evidence that MRM2 improves the meta-analysis. There is little theoretical justification for the additional structure imposed by MRM2 (i.e. I don’t buy the adding-up argument) and adjusting functional forms to get results that look better is arbitrary at
best. As such, I recommend the use of MRM1 and recommend ensuring conservative estimates of WTP for changes beyond the observable range using linear interpolation as seen in figure 10. The linear interpolation gives a lower bound estimate for any concave function form. While not the subject of the current study, and unlikely to happen in a reasonable policy, for changes above the observed range (in the upper tail), I recommend capping WTP at the top end of the observable range. Doing so will ensure a conservative and defensible WTP measure.

2) Priority question 2) focuses on the specification of the commodity change and asks specifically if specifying water quality and water quantity as ‘perfect substitutes’ makes sense? My short response is given the data collected, I can think of no better way to handle this. Unless separate
data is available for each study on quantity and quality, and they could be incorporated into the
meta-equation separately, an arbitrary assumption is needed. Linearity seems as good as any
other arbitrary assumption.

3) Similar to response 2, unless distance to water bodies is available for all data points in the study,
and can be incorporated into the estimated meta-equation, an arbitrary assumption is necessary. Distance decay is well-documented in the literature (for both use and non-use values). How quickly that decay occurs is an open question. While risking highlighting the
arbitrary nature of the 100 mile threshold, it might be worth reporting calculations based on a couple of other distances (75 and 125 miles?) to show sensitivity to the assumption. While the 100 mile radius is consistent with Viscusi et al (2008), additional studies find that distance-decay occurs at faster or slower rates depending on whether the affected populations are users or non-users and the scope of the change. For example, Hanley, Schlapfer and Spurgeon (2003) state: “In accord with a priori expectations, we find that a more rapid distance decay exists for
use values than for non-use values. We also find no distance significant effect for a general classof environmental good, where a significant effect exists for a specific local example of this class.”

So while 100 miles may be defensible, it is worth noting that other distances may produc e
different results.

4) I have always been a fan of including methodological and study specific variables in metaregression equations. Because it is well known that willingness to pay is sensitive to assumed functional forms, distributional assumptions, author fixed effects, I would like to see some additional controls included. In many cases, such controls will serve as a fixed effect and will swamp the random effect. As to the question of weighting, because each observation of the
dependent variable is an average, it is tempting to treat the dependent variable as an average as
you would with county-level averages or census-track averages. However, doing so would seem to indicate that you are estimating a regression where the dependent variable is actually an individual and not an average. Because these samples are not drawn from a common population, there would need to be many more controls in place to correct for the inconsistent
sampling across studies.

5) In short

— yes. I like the addition of additional variables from outside the original studies.

6) Were ‘contingent behavior’ studies considered? Also, while there is significant overlap in the
steps and many of the journals were likely covered in other steps, the list of journals looks incomplete with journals like the American Journal of Agricultural Economics, the Journal of Environmental Management, Journal of Agricultural and Resource Economics...). Likewise, the
author list in footnote 3 is only a fraction of the authors active in this field. Again, most of the authors are likely picked up in other steps, but the list looks strikingly short.

7) I found the WQI and WQL translations to be satisfactory.

8) Yes, particularly in the context of the scope/adding up discussions. It is unlikely that a stated
preference study that fails to find scope effects would be published. Unfortunately, addressing
this issue is nearly impossible without additional information on unpublished studies. This might be particularly troublesome for MRM2 as the dependent variable is specified as WTP/Q. As such, the error term is likely truncated from below as studies with small WTP/Q are likely
omitted from the sample. Taking the log of the dependent variable is an arbitrary fix that rules out the possibility of WTP/Q = 0. Th e fact that you don’t observe it is not an indication that WTP/Q is never equal to zero, but rather an indication that such studies will rarely appear in the
literature. Failure to find adequate scope effects in a meta-analysis would be shocking as a

result.

9) I like the idea of including distance as an explanatory variable as long as it is available. My guess is that many studies do not report such information.

10) I have no comment here.References:

Amiran, Edoh Y., and Daniel A. Hagen. 2010. “ The scope trials: Variation in sensitivity to scope and WTP with directionally bounded utility functions, ” Journal of Environmental Economics and Management

59:293

– 301.

Diamond, P.A. 1996. “Testing the Internal Consistency of Contingent Valuation Surveys.” Journal of Environmental Economics and Management , 30: 337-347.

Haab, T.C., Matthew G. Interis, Daniel R. Petrolia, and John C. Whitehead. 2013. “From Hopeless to Curious? Thoughts on Hausman's “Dubious to Hopeless” Critique of Contingent Valuation” Appl. Econ. Perspect. Pol. 35 (4): 593-612

Hanley, N., F. Schläpfer, J. Spurgeon . 2003. “ Aggregating the benefits of environmental improvements: distance-decay functions for use and non- use values,” Journal of Environmental Management , 68:297-

304.

Hausman, Jerry. 2012. "Contingent Valuation: From Dubious to Hopeless." Journal of Economic Perspectives , 26(4): 43-56.

Rollins, Kimberly and Audrey Lyke . 1998. “ The case for diminishing marginal existence values, ” Journal of Environmental Economics and Managemen t 36(3): 324- 44

. Viscusi, W.K., J. Huber, and J. Bell. 2008. “The Economic Value of Water Quality.” Environmental and

Resource Economics , 41(2):169-187.

Whitehead, J

. C., T. C. Haab and J. Huang. 1998. “ Part-whole bias in contingent valuation: Will scope effects be detected with inexpensive survey methods? ” Southern Economic Journal , 65(1):160-168.COMMENTS ON THE PEER REVIEW PACKAGE:
META -ANALYSIS OF WILLINGNESS- TO -PAY FOR WATER QUALITY IMPROVEMENTS

First, I’d like to say that I’m impressed with the work thus far. The EPA has done a thoughtful and careful analysis and is to be commended for the work done thus far, and for requesting input at this stage in the work’s development. -Laura Taylor

Priority Questions

1. One of the greater difficulties I see is that the EPA is interested in valuing water quality changes that are outside (below) the range of the observed meta data on water quality changes. As such, any estimates of the WTP for Δ WQ<2.5 rely wholly on functional form. It is clear EPA recognizes this, but it is worth repeating that this is really problematic, and is evidenced by the issues with MRM1 as Δ WQ becomes very small.

While I don’t disagree with the proposed linear interpolation as a way to “push down” the
values for small WQ changes using MRM1, there is a certain logical in consistency that is unappealing. To me, the discussion reads as follows : The EPA prefers the translog function because it forces WTP(D=0) to be zero. Y et , the EPA finds the estimation results from this functional form to be unpalatable at ranges less than 2.5 and thus alters the function between 2.5 and the origin... so what then is the benefit of the translog? One doesn’t have to choose a functional form that goes through the origin to make an assumption and draw a straight line in the WTP function from D=2.5 to D=0.

One approach to consider is a linear or cubic spline function. Since valuation is mostly based on the WQL (or an index/description that is easily mapped to the WQL), you have very natural knots in the data. Through a linear (or cubic) spline, you can get away from the assumed smoothness of the log-log function. Perhaps knots at each 10 units (0, 10, 20, 30, etc.)?

Further, given the fact that value estimates for Δ WQ<2.5 will be determined by choice of functional form, it seems to me that having explored many functional forms is an important exercise and should be incorporated into the benefits transfer framework. Perhaps the EPA could consider an “internal meta analysis” (of their meta data) in which WTP( Δ WQ<2.5) is generated for a large array of functional forms and the median (or

mean ) estimate is used for benefits transfer, along with 5 th /95 th percentiles (or whatever range is deemed appropriate).

In sum, I think EPA should vigorously pursue the logic in MRM2 (MWTP) as well as in MRM1 – meaning pursue a large variety of meta regression model functional forms, including using dependent variables that are WTP and MWTP and not wedding themselves to translog. Note, I don’t believe MRM2 should be pursued because of adding-up. I am not convinced that satisfying the adding-up condition is a first-order priority (although it may be a practical one from the perspective of satisfying a widevariety of stakeholders, and thus can be included as one of many pieces of evidence).

2. I don’t believe that it is a good assumption that quantity and quality of surface waters are perfect substitutes. Unfortunately, I am not aware of evidence that suggests what the relevant tradeoff is between quantity and quality of water resources. I do have a suggestion that may be more palatable in these dimensions, although it doesn’t directly address the heart of the issue (which is the absence of a real understanding of how people tradeoff the quantity of WQ changes with the scale of the WQ changes).

The goal is to value a specific policy outcome in which each unit of water (Li) that has a baseline condition (WQ i B ) and a post-policy condition (WQ i A ) contributes to the benefits measure. The EPA is currently suggesting that it compute the WTP for the average Δ WQ
within 100 miles of a household:

∆

, ! = ∆ ! × ! !

! ! , ! × ! !

! ! ,

where the average baseline condition (B WQ i ) is highlighted as well in the WTP function (see comment 5a below).

An alternative, is to compute the WTP for each homogeneous segment (i.e., same type of water body, same baseline and same change) and then compute average WTP over these segments, which uses the nonlinearity of the WTP function differently:

= ( ( ∆ ! , ! ) × ! ) !

! ! .

To introduce the area over the change more directly, one could amend the meta function to be (see comment 5a below for the discussion of the Δ WQ and BWQ interaction term) :

WTP = β 1 * Δ WQ + β 2 *B WQ + β 3 *Area

+ β 4 * Δ WQ *BWQ + β 5 * Δ WQ *Area + β 6 * Δ WQ *BWQ*Area + ...

where Area is the total area affected by the policy (scaled/transformed in some way to adjust units which are bound to be very large).

Then average WTP for a particular policy impact within 100 miles of a home is: :

= ( ( ∆ ! , ! , ! ) × ! ) !

! ! .

3. Ideally, there would be some evidence on the spatial extent of the market in the literature, and ideally, this would be present for different types of water bodies. For instance, one might expect there to be a different extent of the market for large economically importantbodies of water (e.g., Lake Lanier in Georgia or Chesapeake Bay) versus a network of small, but wadeable streams in a watershed.

Recognizing that there is very little information available, I wonder if the EPA could attempt to recover some evidence from recent studies. For example, could the EPA contact authors of studies that seem like good candidates for re-calculating WTP for different groups of respondents according to distance to the resource. For studies that used well-defined resources (i.e., a specific river or lake) and a large enough sample (i.e., a random sample of residents in the state), the EPA might be able to get the authors to provide some rough estimates of the distance-decay relationship. Without knowing the details of each study, it seems Londono Cadavid and Ando (2013), Welle and Hodgson (2011) and Whitehead (2006) might be candidates. Or perhaps even Stumborg et al (2001, although this is quite old for keeping data/models). Could the EPA fund additional work on the data collected through these studies that attempts (using original methods) to breakout values by proximity of the respondent to the water body inquestion? I don’t think this work would be large in scope or used as additional datapoints in the actual metadata, but rather just provide rough estimates that are used to inform judgments about the distance-decay relationship.

Without going back into the original data underlying the metadata, I can’t see a way to better define the distance-decay relationship (and I can’t see a way to build it into the MRM given the information gleaned from studies).

There may be a way to add a bit more heterogeneity in the benefits estimation within the confines of the MRM/data you have available. Perhaps you could use the variable

nonusers to generate heterogeneous values that are applied to a set of rings around the census block. For instance, you could keep the assumption that use values are primarily held for waters within 100 miles. In this case, one could use predicted WTP from the
MRM when nonusers =0 as the average value for waters within 100 miles, and then for waters greater than 100 miles, but within the same state, use the predicted WTP as determined by the MRM when nonusers =1.

4. One error structure that is easily considered is to group WTP estimates by author, rather than by study. This takes some judgment, but it captures the correlation among WTP estimates that may be present due to the same researcher making similar judgments in the research process (including which models ultimately are “preferred, ” and thus reported. For instance, looking at the list of studies, one might consider grouping the following: Banzhaf et al 2006/2011; Bockstael et al 1988/1989; studies with Rosenberger (if more than one); studies with Whitehead; and so forth.

One certainly can weight by the inverse of number of observations from each study, and the impact on the results would be good to see at least in a robustness section.

One weighting scheme that is sometimes suggested is to weight by a measure of the precision of the estimate (e.g., weight by t- statistic) . I’m not a big fan of this method as it focuses on precision, but is silent on bias (and I find bias to be the more important of thetwo issues). Further, poor econometric analyses can lead to overstated t- statist ics, and this weighting scheme ignores this fact.

5. Yes, I think the variables in question improve the usefulness of the meta analysis for benefits transfer. I do, however, think they could be explored further to better capture the intended relationships and have some comments/suggestions:

a. First, and importantly, it seems to me that the change in water quality should be interacted with the baseline water quality. It seems to me that the appropriate specification is:

WTP = β 1 *D + β 2 *Q + β 3 *D*Q + ...

This specification allows WTP to be conditional on the baseline level, capturing the fact that the WTP for a 5-point change off a baseline of 75 may not be valued the same as a 5-point change off a baseline of 45.

This is also more consistent with MRM2, that specifies MWTP to be a function of the baseline WQ.

b. I find the definition of shoreline length for rivers odd (EPA doubled the reach length to reflect both shorelines of a river). Would a lay-person interpret an improvement along 10 miles of a river as “20-miles of shoreline improvement”? I find this unlikely. The analogy that comes to mind for me is a car: because I can access my automobile from two sides, does it mean I have two cars? I don’t
believe this definition matters for the variable sub_proportion since both the numerator and the denominator are multiplied by two in the case of rivers, but it would be good to confirm the use of this assumption.

c. Capturing the extent of the market is hard to do, and I’m not sure I can suggest a better approach than what is currently being used. It seems to me there are two broad types of sampling frames for survey recipients: targeted (e.g., residents in a set of counties, region, or a city) or broad (state-wide) and the resource being valued is either contained within the sample or the sample is contained within the
resource. Your variable ar_ratio does capture this, but perhaps more flexibility could be introduced to allow the impact of ar_ratio to be different depending on whether the resource is contained within the sampling frame or vice versa. For example, let CW = 1 if the resource is contained within the survey sample frame (Hayes et al in Figure 3 and Matthews et al in Figure 4 would both have CW=1, while the others would have CW=0). Then the regression model would be:

WTP = β 1 *CW + β 2 *ar_ratio + β 3 *CW*ar_ratio

d. A small point: I believe ar_ratio would be much more intuitive and easier to explain if it were the inverse of the current definition (i.e., = % of affected area / % sampled area). We would expect the coefficient to be positive and thereasoning why we expect a positive sign avoids using negatives combined with inverse relationships (which is confusing), as the current definition requires. This is just a minor expositional point.

e. In general, it seems more interaction terms could capture spatial influences on WTP. For example, mult_bod could be interacted with sub_proportion to capture the fact that the influence of substitute lakes (identifiable, specific features) may influence WTP differently than the presence of substitutes rivers and streams throughout a watershed (or whatever spatial definition). This relates to my
comment in 9c below.

f. Note, I would expect your variable “ sub_proportion ” to follow the same logic as the needed assumptions in benefits transfer. Currently, sub_proportion uses the state as the backdrop for substitute water bodies. However, in the BT, you use
100-mile buffers.

Extensions beyond the Present Application (Optional)

6. The selection criteria seem fine, although it was unclear to me if you went back and applied all the criteria to the original 2004 and 2009 compilations. Did the criteria change over time? If so, go back and make sure the studies from 2004/09 compilation meet the current criteria if you haven’t already done so (it sounds like you did).

Also, it was not clear to me if you included multiple observations from a study that arise from specification choices in the modeling (rather than just those that address a different component of the study data or values for alternative resource changes). If not, why not? If not, and there were multiple estimates that arose based on modeling/specification choices, how did you choose which one to include? Perhaps this is not an issue and studies generally report WTPs based on one estimate model/specification?

Lastly, you could consider doing a social science citations index (SSCI or currently the “web of science” database) on the existing papers in your dataset. In other words, search a database that records the papers that have cited your current set of WQ valuation studies. I suspect you have probably caught most, if not all, papers – this is just a suggestion for your consideration.

7. I thought this section was thorough and reasonable in its approach (I did not review each
row in Table B-20 to assess EPA’s coding of each individual study).

8. No comment.

9. I have several comments here.a. I agree that sub_proportion could be better defined. I appreciated the EPA’s justification for state, but it seems to miss some potentially important factors that
limit the extent of the market.

I would suggest considering “day-trip” distance (100 miles approximately) instead of the state. Something larger than 100 miles could be used, but the state seems a bad approximation in some instances (e.g., California – or even N.C., for which a mountain lake is a poor substitute for a coastal inland lake).

b. I was unclear on how substitute lakes were defined. It appears that rivers are matched based on stream order – are lakes similarly matched for comparability? A large reservoir with public boat ramps is not the same as a closed- access reservoir, or a small lake with zero boat access. Should lakes have clearly defined ‘comparable’ substitutes in the same way rivers do? (which brings up the subtle point of using aggregate size as the measure of the amount of substitutes – do 10 small lakes really substitute for 1 big lake? It seems this approach works better with diffuse resources, such as wadeable streams rather than large features).

c. EPA asks about using WQ as another factor that defines “substitute”. This is a good question. My inclination would be to not use WQ as way to limit the definition of a substitute. Substitutes can be “better” or “worse” – thus having better or worse water quality – and be weak or strong substitutes. I still believe that EPA could refine the substitute sets for lakes a bit more (capturing very large differences in lakes as suggested in the point above), but your current means of defining similar rivers and estuaries/coasts seems appropriate to me.

d. See also related comments in 5.

10. No comment.

Other Comments (not addressing specific questions asked by the EPA)

1. I believe it would be very helpful to the reader in future versions to use TWTP and MWTP to distinguish when you are speaking of total and marginal WTP, respectively.

2. I think your income variable needs more attention. It is an important variable for benefits transfer and I have questions about the measure itself and your specification choice. How many missing observations are there? Are the sample incomes similar to the census variables you use? What’s the correlation? I wonder if you are introducing significant measurement error. If classical, you would be biasing the coefficient toward zero. However, if survey respondents have, on average, higher incomes and education than the general population in the survey area, then the coefficient estimate would be biased up. You can get some sense of this issue by comparing your census definition with the sample incomes (means and/or medians) when available. If differences are substantial, you could consider using the original income measures (means and/or medians) and use abias function to adjust the census-based income measure that is used to impute missing observations. For instance, if you find that sample income is on average 1.3xCensus Income, they you could impute income for missing observations as 1.3xCensus Income.

3. In MRM1, I was surprised by the sign of swim

_ use and boat

_ use (and they are significant) – I would have expected these to be positive. Is there a logical explanation (or am I wrong to expect these to be positive)? Also, I find the coefficient on voluntary to be surprising – again I would have expected this to be positive (if it is not incentive compatible, the assumption is usually that hypothetical bias is the dominant result and values would be biased up). Is there something else that this might be correlated with that is not coded (i.e., is there an omitted variables problem relating to study characteristics)?

4. Why not also weight MRM1 (by inverse of # observations drawn from each study)? In future work, the EPA should apply consistent methods to each MRM (such as similar weig hting) to the extent possible. The document indicates that future work will do this, I just wanted to underscore that I agree it should be done. The preferred MRMs upon which benefits transfers are conducted need not match, but consistent approaches should be reported so that the reader and the analyst can see the influence of these judgments.

5. I wonder about the way in which you include study year. Do we really expect that each year leads to a certain percentage change in WTP, after controlling for inflation, income, methods, etc. In fact, once controlling for all factors that are expected to influence WTP, then year of the study should have no influence, correct? If it does have an influence, it must be because of an important omitted variable correlated with year... what might that be? Results indicate that later years are associated with lower WTP. Over time, has publication pressures and/or standard practices led authors to estimate/report “more conservative” values from stated preferences studies (again, after controlling for economically important influences that vary with time)? Also, I’m curious what happens if you allow more flexibility (i.e., include dummy variables for decade, for instance) – is the relationship really monotonic? Maybe so, but it would be nice to check.

6. You have a variable that indicates if a WTP estimate comes from a thesis. Are thesis, by definition, “ not reviewed ”? If so, then the effect of a thesis on WTP is = 0.6509 – 0.7419, correct? Is the combined value significantly different than zero?

7. When reading Section 3.3., the following question came to mind: did all studies elicit WTP for an improvement in WQ? Or did some studies elicit WTP to avoid a decline in WQ? If the latter is true, then this needs to be coded into the metadata, since property rights are different and the WTP to get an increase in WQ is not expected to be the same as WTP to avoid a decline. With choice experiments, this can be tricky to assess if the levels of some attributes include both increases and decreases as compared to the baseline (or status quo) option.

8. Do you have a sense whether or not the range of ar_ratio in the metadata is likely to encompass policy applications? I’m wondering if the issues associated with predictingout of sample might arise here too (or for ar_agr , or sub_proportion ).

9. In the BT application (page 43), your choices over covariate values led to several questions. First, how is the variable “outlier” related to the variable CE? Do choice experiments have “outliers” that they trim (there are studies which use subsets of choices, but are these present in your data)? I ask, because I wonder about your choice of setting
Outlier = 1.

Second, on page 43, you indicate that set “ nonparam ” to zero because “older studies” just report mean WTP across responses. Should these studies be contributing to estimation of all coefficients (not just nonparam ) if their methods are that suspect? Could you consider dropping them?

The variables swim_use

, gamefish , and boat_use are specified for each BG based on the length-weighted average of baseline water quality & WQ changes for the affected resources. Here is where my comment on the definition of river “affected areas” (= 2 x length of river) is important. You may ultimately decide that your definition makes the most sense, but it is important to make sure that it is truly the better measure as compared to using just the length of the river (see comment 5b).

Lastly, I’m confused about the variable ln_ar_ratio on page 44. It seems to me that you’d want the area of each BG divided by the area of resource within the buffer that is improved. You use counties to create the “area of resource within the buffer”, but that doesn’t make sense to me. It seems to me that you would, at a minimum, also intersect county boundaries with the 100-mile radius and only compute area that falls within both affected counties and the 100-mile buffer. (Maybe this is what you did?) Below is a simple schematic of what I think you did:

The blue circle is a blockgroup, each black square is a county, blue lines are affected rivers, and the red circle is a 100-mile buffer around the blockgroup.I believe you used the area of all five counties (black boxes) as your denominator. However, it seems to me a better approximation is the intersection of the counties with the buffer. In the example above, there are four little “triangles” that lie within the circle but that are not within any of the affected counties. These areas would not be included in the denominator either. Only the area of the 100-mile buffer that intersects the affected counties ought to be included.

10. As an aside, your prediction of WTP (from ln(WTP)) relies on normality of the error term. As discussed in Wooldridge (Introductory Econometrics, 2012, page 213), there are predictions of WTP that are not reliant on the assumption of normal errors that are easily computed.