Peer Review Documents: Meta‐Analysis of Willingness to Pay for Water Quality Improvements Page i

DCN SE06136

Peer Review Documents: Meta‐Analysis of Willingness to Pay for Water Quality

Improvements

May 2015

ADDENDUM: REPORT TRANSMISSION MEMO

From: U.S. EPA, Office of Water 

To: Steam Electric ELG Rule Record 

Date: September 30, 2015 

Subject: Peer Review Documents: Meta-Analysis of Willingness to Pay for Water Quality 

Improvements 

In general, the peer reviewers commended EPA for this work, and agreed that this work is 

important and difficult. To the extent practicable, the attached peer review comments have been 

incorporated into the final analysis for the Steam Electric Effluent Limitations Guidelines and 

Standards (ELGs). This memo explains a few additional points and supplements the discussion 

in Chapter 4 and Appendix H of the Benefit and Cost Analysis (BCA) document (DCN 

SE05977). 

EPA asked the peer reviewers about whether the Agency should impose the adding up condition 

(i.e., choose MRM2 over MRM1). The peer reviewers were not very concerned that the metaregression satisfy adding up. Nonetheless, EPA has a strong preference for imposing adding up, 

because it implies that different benefit-cost analyses can be compared or combined and that 

these comparisons and combinations are meaningful. Thus, EPA selected MRM2 as its primary 

model for estimating water quality benefits of the ELGs. In the BCA document, MRM2 is 

referred to as Model 1. EPA developed a second model, referred to as MRM2-S in the memo 

“Revised Memorandum: Accounting for Scope in the MRM2 Meta-Regression Model” dated 

August 19, 2015 (DCN SE05985). MRM2-S is referred to as Model 2 in the BCA document. 

The peer reviewers expressed some concern that EPA was using the meta-analysis to predict 

outside of the range of the meta-data with regard to the size of the water quality improvement 

(ln_qualitych in the MRM1 model). EPA understands the general concern expressed by the peer 

reviewers. EPA chose not to rely on MRM1 in the BCA, and in doing so, has addressed the peer 

reviewers’ concern. MRM2 (Model 1 of the BCA) and MRM2-S (Model 2 of the BCA) do not 

include ln_qualitych as an explanatory variable. In these models, the willingness to pay (WTP) is 

divided by the water quality change to create the dependent variable for the model. The easiest 

way to think about MRM2 is as an estimating equation for a water quality demand curve. Picture 

a downward sloping curve on a graph with marginal WTP (price) on the y-axis, and the water 
Peer Review Documents: Meta‐Analysis of Willingness to Pay for Water Quality Improvements Page ii

quality level, Q, on the x-axis. We approximate these pairs by taking the average WTP over the 

range of water quality change in each study, i.e., WTP/ΔQ, as the marginal WTP at the mid-point 

of that change. (Recall that, by the intermediate value theorem, there must be some Q within the 

range of the water quality change where the marginal WTP equals the average WTP.) Other 

variables on the right hand side of the estimating equation can be thought of as demand shifters. 

To apply the model for benefit transfer, we integrate under the marginal WTP curve between the 

initial and final water quality levels. That's how this model solves the adding-up problem (of 

MRM1): WTP calculated as the integral under a well-defined demand curve will have to satisfy 

the adding-up condition by construction. 

Model 1 maintains the assumption that the underlying data generating process—the statements of 

WTP by the survey respondents in the original studies—complies with the adding-up condition. 

That is, the functional form of Model 1 (in the BCA) was chosen in part to assure that the 

adding-up property holds for the estimated benefit transfer function. 

However, we can achieve a better fit to the data by including a measure of the water quality 

change as an additional explanatory variable. Adding such a variable has the effect of making the 

water quality demand curve undefined, since a well-defined demand curve (marginal WTP 

function) depends on the quantity of the good not the change in quantity. Therefore, a benefit 

transfer function estimated in this way would not satisfy the adding-up condition. We can retain 

the adding-up property in such a benefit transfer function if the quality change variable is held 

constant at a pre-determined fixed value—i.e., it is treated as a demand curve shifter that does 

not shift—so that WTP can be calculated as the integral under a fixed demand curve. So as a 

form of sensitivity analysis we estimated this model, which we call Model 2 (in the BCA), and 

we examined the influence of choosing different levels for the fixed value of the quality change 

variable on the benefit transfer results. 

Specifically, Model 2 is identical to Model 1 except ln(ΔWQ) is included as an additional 

explanatory variable. The estimation results for Model 2 are shown in Table 4. The coefficient 

on ln(ΔWQ) is highly statistically significant, and the R2 for Model 2 is significantly greater than 

that for Model 1. This suggests that either the underlying data generating process does not satisfy 

the adding-up condition (because of irrational behaviors by the survey respondents or one or 

more forms of hypothetical bias in the stated preference elicitation process itself), or there are 

one or more mis-measured or missing explanatory variables correlated with ln(ΔWQ) that if 

precisely measured or added to the model would cause it to satisfy the adding-up condition. 

Closer examination of this issue will be an important area for follow-up research. In the 

meantime, we retain Model 1 as our preferred estimating equation for the benefit transfer 

function because it avoids the need to arbitrarily specify the fixed level of the quality change 

"demand shifter" when applying the model, and we used Model 2 to illustrate the sensitivity of 

the benefit transfer results to this dimension of model uncertainty. 
Peer Review Documents: Meta‐Analysis of Willingness to Pay for Water Quality Improvements Page iii

It is true that the ΔQ's examined in the underlying stated preference studies were always larger 

than most of the ΔQ's we are examining in this policy setting. However, most stated preference 

surveys will ask about water quality changes that are big enough to register a WTP response that 

rises above the "noise" inherent in the stated preference data generating process—that is, we 

typically wouldn't expect very accurate answers to SP questions focused on very small 

increments, so survey designers typically don't ask about very small increments. If we were to 

take the admonition to "never extrapolate out of sample" along this dimension to heart, then we 

may find that NO existing SP studies can be used for benefit transfers. This seems like an 

unreasonably stringent standard. 

It is also important to note that MRM2 does not have the same kind of extrapolation problem. As 

explained above, MRM2 estimates a demand curve for water quality based on observations 

(approximations) of marginal WTP regressed on the associated water quality levels. The water 

quality levels covered in the underlying meta-data span a range that covers most (if not all) of the 

water quality levels examined in the Steam Electric rule. That is, we use MRM2 to estimate 

WTP for changes between Q0 and Q1 for each census block group by integrating under the 

demand curve between those two levels. And the Q0's and Q1's for most or all of the block groups 

fall well within the range of Q's represented in the meta-data. 

In general, the peer reviewers approved of the inclusion of the GIS analysis and variables 

extending the underlying meta-data reported in the studies. However, the very brief explanation 

of the ln_ar_agr variable left the reviewers questioning its inclusion. EPA has included this 

variable, because it is strongly significant, and has included the following justification for its 

inclusion. Areas dominated by agricultural land uses have particular characteristics which 

suggest that WTP for water quality improvements could be lower than other types of areas, 

ceteris paribus. First, unlike non-agricultural rural areas, heavily agricultural areas have generally 

been altered from their natural ecosystem conditions, and do not tend to be highly prized for 

water-based recreation (or characterized as pristine natural areas) – this would be expected to 

decrease WTP for improvements in agricultural areas compared to many other rural areas. 

Second, unlike more heavily populated suburban or urban areas, agricultural lands do not have 

the population base to support well-developed and used recreational areas. Improvements to 

water bodies in suburban areas, for example, are often highly valued because these areas support 

extensive recreational and other uses. Third, a greater proportion of the population in agricultural 

areas has employment linked to farm activities that may be associated directly or indirectly with 

water pollution. Those whose employment depends on farming activities may be hesitant to 

support programs to improve water quality (in stated preference surveys or otherwise), for fear 

that these policies may lead to greater restrictions on farms and farming activities. For all of 

these reasons, one would expect areas dominated by agriculture to have lower WTP to improve 

water quality, again holding all else constant. This intuition is strongly supported by model 

results. Also, removing this variable causes substantial changes elsewhere in the model, a sign 

that removing this variable could cause non-trivial omitted variables bias. In such cases, 
Peer Review Documents: Meta‐Analysis of Willingness to Pay for Water Quality Improvements Page iv

econometric guidance suggests that the variable should be retained rather than omitted; if the 

inclusion of a variable within a model is questionable, the default solution should be to include 

the variable (including an irrelevant variable only reduces efficiency; omitting a relevant variable 

creates bias). 

Finally, EPA views its revisions to the meta-analysis employed here as a work in progress. As 

part of the revisions, EPA has identified some issues that require further analysis, and intends to 

continue this work after the Steam Electric ELGs are promulgated. EPA also intends to have the 

Science Advisory Board (SAB) review the results of that progress when it has reached a point of 

completion where SAB review is appropriate. In particular, EPA has identified the following 

issues to address: conducting additional robustness tests and cross validation, investigating 

model over-fitting, investigating whether a distance decay effect can be gleaned from the metadata (to substitute for the 100-mile radius assumption used in the benefits transfer application 

here), and to consider employing Bayesian estimation techniques.
Peer Review Documents

Meta-Analysis of Willingness to Pay for Water Quality Improvements

May 2015

Table of Contents

Kathleen Bell Review..............................................................................................................2

Kevin Boyle Review................................................................................................................9

Jacqueline Geoghegan Review..............................................................................................20

Timothy Haab Review...........................................................................................................28

Laura Taylor Review...............................................................................................................32

1
baseline conditions outside of the range observed in the primary studies. 

o Meta-regression model one (MRM1) uses WTP as the dependent variable and includes the natural log

of the water quality change as an explanatory variable. As explained in Section 6.2 of the report, the 

MRM1 WTP function does not satisfy Diamond’s (1996) adding-up condition. The MRM1 function 

implies that marginal WTP goes to infinity as the size of the water quality improvement goes to zero. In 

the application of the model, this resulted in fairly large WTP values being assigned to very small 

changes in water quality. The linear interpolation approach described in Section 6.2 was developed to 

address these issues. Do you think it adequately address these concerns about extrapolation for MRM1? 

Would you suggest any other alternatives? 

o Meta-regression model two (MRM2) uses an approximation of marginal WTP as the dependent

variable and includes the associated water quality level as an explanatory variable. We formulated 

MRM2 as a way to address the adding up condition and the problem with large predicted values for 

small changes, albeit with some costs in explanatory power. How important do you think it is to satisfy 

the adding up condition in applied benefit transfers? Are the tradeoffs presented worth moving to a 

model other than MRM2? 

Please review the usefulness of the two models in the context of a benefit transfer, and explain your 

preference between the two functional forms. 

Response: The US EPA report presents two distinct meta‐regression models, referred to as meta‐

regression one (MRM1) and meta‐regression two (MRM2). The report explains MRM1 in greater detail,

with extensive discussion of methods and assumptions. The discussion of the second meta‐regression

model, MRM2, is less comprehensive and perhaps reflects its origins, a model developed in reaction to

the results and application of another model. Both models make use of the same dataset; these data

summarize studies focused on the willingness to pay for water quality improvements in the US. The

major difference between the two meta‐regression models is the functional form of the ultimate

willingness to pay function. Reviewing the report and summaries of these two models, I developed a

sincere appreciation of the challenges EPA faces when developing rigorous and robust policy analysis

tools for application in a variety of policy settings.

Meta‐regression model one (MRM1). Meta‐regression model one is an updated version of two prior

meta‐regression modeling efforts supported by US EPA (2004, 2009). This most current version

incorporates updates that give greater consideration to the spatial aspects of valuing changes to water

resources, including the extent of water resources, water quality changes, substitutes, and population

sampled. In general, the development of the model appeared to follow established guidelines on

2
Further, the incorporation of the spatial variables seems like a big change for meta‐regression

modeling. While these additions respond to increased awareness of spatial heterogeneity in benefits

and costs, greater sensitivity analysis of such additions may be warranted to understand fully the

repercussions of these additions and the sensitivity of results to different spatial measures and to

different approaches for considering the spatial aspects of changes in water quality and the valuation

of these changes. Similar to the issues that arise around water quality changes beyond the range of the

studies, there are likely analogous issues around the spatial measures. For example, it is not clear the

extent to which participants in the original studies were aware of the spatial factors now controlled for

ex‐post in this meta‐analysis.   

The extrapolation approach described in Section 6.2. offers one approach to “avoid” the large WTP

values    for very small changes in water quality. While the description is clear and this adjustment

would achieve the desired goal, I am not necessarily convinced by the current text that this is a

preferred strategy for extrapolation. The report mentions numerous options for responding to such

concerns. It would have been interesting to compare the results from multiple approaches. I remain

puzzled by whether or not MRM1 should be applied to rules or policies involving such small changes

and have questions about whether, how, and when to extrapolate.   

Considerable time and effort have gone into the development and application of multiple versions of

this meta‐regression model. Model development has taken place through multiple iterations, and

MRM1 has proved useful in rulemaking applications. Continuing to build on this foundation and to

improve and test the approach under different assumptions seems quite productive.

Meta‐regression model two (MRM2). Meta‐regression model two, developed in response to the

concerns about the appropriateness of MRM1 to handle small changes in water quality and its lack of

satisfaction of the adding up condition, employs the same data but adopts a different functional form

to approximate marginal willingness to pay. I was surprised by the limited justification and explanation

of MRM2. Has prior work adopted this specification? Suggested this specification? I realize this

modeling analysis is preliminary (and some of the standard errors and fit issues raise flags). But the

brief overview complicated my assessment of the approach, especially after having read a thorough

description of MRM1.   

As noted in the report, it would be extremely beneficial to see additional comparisons of MRM1 and

MRM2 under common assumptions. In closing, I remain less concerned about adding up conditions and

more concerned about the ability of the underlying data from these studies to get at what the agency

3
rulemaking and policy analysis, I concluded my reading of the review package intrigued by the

challenges and weaknesses of applying the current models. I was also excited by the promise of future

efforts that might continue to test alternative meta‐regression models and incentivize primary research

of the type and sequencing of water quality changes more regularly induced by US EPA regulations.   

2. The proposed benefit transfer approach for both models defines the commodity change to be valued as

the change in average water quality within 100 miles of the home. This implies that the quantity and 

quality of surface waters are perfect substitutes, i.e., a 1 point improvement for 10 shoreline miles would 

be equivalent to a 10 point improvement for 1 shoreline mile. Is this appropriate for a benefit transfer of 

water quality? If not, what would you suggest as an alternative, and how would you implement it? 

Response:

My initial response is that quantity and quality are not perfect substitutes. The length‐weighting has

both advantages and disadvantages. I have additional concerns about the basis of the 100 miles

assumption and would like to see greater justification of this assumption.   

I would also encourage greater restraint in discussing theoretical expectations related to spatial

outcomes. The report strongly asserts monotonic decreases in willingness to pay with distance. It is my

view that we have much to learn about “distance‐decay” and the definition and use of spatial measures

such as the shoreline length weights used in the change in average water quality calculation.   

Would be interesting to see how the results would change using different measures of commodity

change or if the analysis included different measures, such as the size of the affected area.   

The description of some of the spatial variables and the assumptions and steps used in the analysis

could be improved to help readers better understand how the commodity change to be valued is

actually operationalized. The benefit transfer examples included in the report were very helpful and

helped clarify some issues. Clearly, more applications and examples help reveal critical details about

the usefulness and appropriateness of the benefit‐transfer approach.

4
gpfppffpp

Aside from explicitly controlling for the distance of the waters from the home in the meta-regression 

model (which may not be feasible), is there a more appropriate way to control for the spatial extent of 

benefits? 

Response:

I would like to see additional discussion of the rationale for the distance assumption in Section 6.1 (100

Miles). At present, the support is a single study’s finding about the majority of recreation trips occurring

within a 2 hour drive from home. Given the importance of this threshold distance assumption,

additional work should be done to research and document the appropriateness of this assumption for

benefit transfer analysis. This may involve more thorough consideration of use and non‐use values, and

greater reflection on how such an assumption might vary across different water resources.

I am puzzled by the discussion of several spatial issues. Notably, why does EPA expect the value to

decline monotonically with distance from the home? There remain many questions about the

interactions of distance, use/non‐use, and value relationships. This assumption/assertion of monotonic

distance decay should be better justified. Our theoretical and empirical models have not advanced to

provide us with such confidence.   

Also, it is important to note that state‐scale analyses also make a distance assumption when bounding

values within state boundaries.

5
As suggested in the report, it would be informative to see more thorough, direct comparisons of MRM1

and MRM2, including results where common assumptions about error structures and weighting

schemes are made.

There may be benefits from including additional variables that would distinguish variation across

studies from variation between scenarios within a study. I believe there would be value from

understanding the extent to which spatial issues were stressed by the researchers and their methods

(e.g., did the sampling specifically address relative proximity to the affected water resources (many

random samples will not necessarily give researchers great variation in terms of relative distance and

other key variables)?; did respondents understand the spatial extent?). I am somewhat concerned

about controlling for issues ex‐post without knowing if the original study really emphasized items such

as the spatial extent of the affected water resources.   

It seems as if there are real challenges at coming up with refined definitions of affected water

resources and potential substitutes of those resources that can be consistently applied across many

studies. MRM1 offers a valuable starting point from which to build on and change how such variables

are represented.

6
Response:

Water quality change (lnquality_ch) and baseline water quality (lnbase) are measured in terms of the

100‐point Water Quality Index. The report clearly documented the assumptions that were made for

each study. While some question the use of such indices and ladders, these tools permit

standardization across disparate studies. These variables are essential to the benefit transfer

application and improve the meta‐analysis (though the parameter on the baseline variable was

sometimes not significant).

Relative size of the area (i.e., watershed) affected by water quality changes to the surveyed population

area (i.e., state) to indicate extent of market effects (ln_ar_ratio1). I understand the motivation behind

including extent of market effects and believe such effects improve a benefit‐transfer analysis. I

commend the researchers for doing the work to operationalize this measure. Yet, I am not entirely

convinced that this ratio, as currently measured, gets at extent of market effects well. The description

and justification of this variable were confusing.

Proportion of the agricultural land in the affected area (ln_ar_agr). I did not understand the

explanation of this variable as a measure of development characteristics, and I was also not clear on

the economic intuition behind the variable. While the presence of agricultural land may imply limited

development in some regions of the US, it would not make a good measure of this outcome in other

regions, especially those that are highly forested and not particularly agricultural. Recommend using a

measure that more directly gets at the desired measure (e.g., development or population or housing

density or impervious surface). The NLCD development categories can be problematic; they are

measuring cover and not use. They have tried to improve representation of low‐density development

but it can still be challenging.   

Presence of substitute water bodies (sub_proportion) is a logical, appropriate addition to the analysis.

The continuous measure has the potential to outperform commonly used binary measures. Got a bit

confused about the spatial scale at which this proportion was calculated. In practice the variable

seemed to improve the analysis.

Whether multiple waterbody types are affected (mult_bod). This binary variable distinguishes affected

areas including multiple waterbody types and appears to improve the analysis. That said, it does seem

as if this effect could also be treated differently.   

7
p

Strongly encourage additional consideration and research of additional spatial factors or distinct spatial

measures of the spatial factors/variables currently in MRM1 and MRM2 as you continue to develop

these meta‐regression modeling tools. However, I also suggest more extensive justification and testing

of such measures and encourage greater restraint in incorporation of such effects.   

Including relative distance sounds appealing. Yet, in this context, summary measures of relative

distance may not work particularly well.

8
Comments On:

Meta-analysis of the Willingness-to-Pay for Water Quality Improvements

By 

Kevin J. Boyle

May 24, 2015

Overall this is a well-done analysis. I note the careful:

• Selection of studies, which is consistent with best practices in the literature;

• Use of GIS data to enhance the empirical analysis;

• Coding of water quality, which is the foundation of the analysis and exceeds the

standard practice in the literature; and

• The extensive robustness checks that have been undertaken to establish the credibility

of the estimated meta-equation.

This is one of the best meta-analyses of nonmarket values that has ever been conducted and is 

the best meta-analysis to support policy decisions of which I am aware.

My only concern is what appears to be a desire to make the meta-analysis estimate values for 

conditions that the original studies were not designed to address, i.e., a lack of values for small 

changes in water quality and all original studies provide partial equilibrium value estimates. 

Rather than making what may be ad hoc adjustments to an excellent meta-analysis, these 

limitations highlight the need for EPA to initiate research to fill these major gaps in the 

empirical literature. To be clear, these limitations should not be used to condemn the metadata and the meta-regressions, but are simply omissions in the literature that cannot be 

explicitly addressed in the current meta-analysis. 

I address the charge questions provided by EPA and then provide some general comments. The 

general comments expand on my responses to the charge questions, suggest changes to 

improve the report, and provide items for thought. I noted a number of typos, none severe, 

throughout the report and a careful editing is needed.

Priority Questions:

1. I find that both MRM1 and MRM2 are potentially useful models. Each specification has

strengths and each has weaknesses. I suggest below that a four-quadrant table be

developed that documents the strengths and weaknesses of each specification.

Recommendations for use can be based on the content of this table. In addition, future

9
analysts can review this information to make choices for use based on selecting strengths 

and avoiding weaknesses specific to the policy applications under consideration.

I have a tendency to favor MRM1 because it avoids the ad hoc manipulations of the 

dependent variable and the potential biases this dependent variable enters into the 

estimation. Given this equation, it might be reasonable for any multiple increment change 

of K to value k (QQ0 → QQKK) and not the individual increments (kk ∈ KK). Attributing benefits 

for individual increments could be accomplished by using the existing data to assign 

proportions with simulations to develop error bounds.

Estimating benefits for changes outside the range of primary studies demonstrates a critical 

need for the EPA to support studies to fill these empirical gaps in the literature. Construing 

models estimated with existing data to fit what these values might look like is not a good 

approach. However, it is not uncommon to make predictions outside of the range of 

observed data, but these predictions need to be used a face value with appropriate 

consideration for the embedded prediction errors. Given the timing of decision making and 

available budgets, this will not happen immediately, but it is an initiative that needs to be 

given a very high priority.

2. I explain in my comments below that 100 miles is a conservative assumption for both use

and nonuse values and suggest some additional citations that support the conservativeness

of this boundary. It may be that in cases where benefits and costs are close to investigate

how a change in this distance would affect benefit estimates to consider the robustness of

benefit-cost calculations.

It is difficult to know what the substitutes are. They could be another outdoor recreational

activity or any other consideration that people would spend their disposable income on.

Meta-analyses rarely consider substitutes in the estimation of the meta-equation and the

approach used here is the best I have seen to date in any empirical study.

3. A consideration that is missing here in the report is that recreation demand models often

use a distance greater than 100 miles for water-based recreation (e.g., three hours driving

time) and nonuse value studies have shown a distance decay of greater than 100 miles.

Thus, while stated-preference studies have shown that values decline with distance, the

empirical results report an average for all study participants over all distances. Thus, if one

is to consider a 100 mile boundary versus a 150 mile boundary (~3 hours driving), the 100

mile boundary study would be expected to have a higher mean value estimate than the 150

mile boundary assuming all other factors are held equal. Thus, if existing studies consider

boundaries of greater than 100 miles the use of the meta-equation to conduct benefittransfers may underestimate benefits to this affected group and omit benefits of those

located at a greater distance. Thus, the approach is conservative to benefit estimation.

10
The exception to the logit I just outlined might be a small, local water improvement project. 

I do not think this is a limitation in the current context as EPA policies occur at a much larger 

spatial scale.

4. I would suggest weighting of MRM1 by the inverse of study observations as was done for

MRM2. The number of observations is potentially an arbitrary decision of the empirical

investigator. However, you have a small number of observations for all studies where as

other meta-analyses have had data where single studies each contribute large percentages

of the total observations. Original studies can influence meta-equation estimates through

the number of observations and being systematically different from the mass of

observations provided by other studies. I think it will contribute to the robustness analyses

to consider estimates with and without weighting and horizontal, leave-one-study-out

robustness checks. Random effects estimation is fairly standard for meta-analyses and I do

not think another approach will substantially change your results.

5. The use of spatial variables is innovative and enhances the analyses. I note my concerns

with ln_ar-ratio1 below; this variable just does not make sense to me. Perhaps additional

explanation could convince me otherwise. I have discussed sub_proportion above. While

ln_ar_agr addresses and important source of nonpoint source pollution loadings, I am not

sure what this variable is measuring because WTP is in preference space. Does this mean

that people in rural, agricultural areas value water quality improvements less than people in

more urbanized areas?

Extensions beyond the Present Application:

6. Study selection is consistent with best practices in the meta-analysis literature.

7. This study, while there may be data interpolation errors, represents the very best practice

in incorporating the change values in the meta-equation. It would be interesting, as part of

the robustness analysis, to run the model without the interpolated water quality and

consider how the results change. Would this limit the observed baseline water quality and

the changes values? If yes, how so?

8. I am not a fan of the term “publication bias”. I prefer publication “effect”. Perhaps some

studies do not get published that do not meet some criteria, but the peer-review process is

an important criterion for study quality. Is this a bias or is this sorting of quality. The likely

limitation, for example, might be ce studies that do not find a significant coefficient on a

water quality attribute. I would not be overly concerned about this issue given the careful

study selection and the extensive robustness checks that are employed.

9. Yes, it is worthwhile to consider other spatial variables and yes, include them as regressors.

I think people consider standing and flowing waters differently and this is not included in

the current analysis. Did the application include a particularly iconic water body? I do not

11
see how you can address distance explicitly since you do not know the location of each 

person. Perhaps you could do something with distance from city centers to each water 

body (center of lakes and center of river segments as proxies for distance for most people 

that would be in a general population survey.

10. State of the art robustness approaches are used and the analyses go beyond what is

proposed by Boyle et al. (2013) in terms of practical application. I think that this study

should be submitted for journal publication as an example of applying state-of-the-art

robustness checks.

General Comments:

• Section 2 – It would be good to explicitly state that you were only searching for statedpreference studies and explain why this choice was made. Also in this context explain that

you were seeking studies that estimated total value and explain the reasoning for this

choice.

• P. 3 – it is surprising that “value” was not one of the keywords?

• P. 3, bottom – While the steps you follow below are important to a credible meta-analysis,

they do not “ensure validity”.

• P. 4 – You might want to explain how you applied conditions such as “general(ly) accepted”

and “sufficient information” in addressing the six conditions.

• Section 3 – It is standard to have two people code the data and then to have a third person

resolve in any differences in coding to ensure data accuracy. Your statement about review

by multiple individuals is not clear.

• P. 7 – Why was 2007 selected as the base year for dollars?

• Table 2 – “thesis” implies MS thesis and PhD dissertation?

• Table 2 – “lump_sum” seems odd coding to include payment periods of 5 or less years?

• Table 2, nonusers – recreational nonusers is odd terminology and potentially confusing.

Why not just say nonusers who do not participate in recreation activities.

• Table 2, ln_ar_ratio – Are both the numerator and denominator measure in terms of

counties?

12
• Table 2 – Why not include a variable indicating if a decrement or increment in water quality

was valued? Is symmetry expected? Not typically done in meta-analyses, but worth

consideration.

• Section 3.1 – I have found that the publication year is often know, but the year the data are

collections is not reported. How did you handle this?

• P. 12 – While not necessary for the current analysis, it would be interesting to investigate

how uncertainties in coding changes in water quality affect the outcome of the metaanalysis. I say that this is unnecessary here because best practices have been used, but as a

research question to inform and improve future meta-analysis this could be a useful

investigation.

• Figures 1 and 2 – There appears to be clusters in the data as noted in the text; 4 categories

for the baselines and three categories for changes. Does this limit the general applicability

of the meta-equation to support benefit transfers outside of the interactions of these data

groupings? That is, does the 100-point water quality index provide a false sense of accuracy

in measuring changes in water quality?

• P. 13 – I have a concern with the statement: “EPA expects, all else being equal, that WTP for

water quality improvements would be directly related to the size and number of affected

waterbodies.” This need not hold. For example, a program for an individual water body

may have a very specific action plan and outcomes whereas a program for an area of

waterbodies might, by definition, be more general and less specific. This could lead to

respondent uncertainty about outcomes and the single water body having a higher value.

This a challenge for any meta-analysis, not the just the current study. Such variations in the

design of sated-preference surveys are very difficult to model in a meta-equation and as

such, caution is warranted in the hypotheses proposed.

• Table 3, Lg_reg – This simply represents everything that is different about this one study; an

individual study fixed effect.

• P. 14, bottom – I am confused by the statement: “Sa_area is not correlated with the size of

the affected resource because various factors affect researcher’s sample selection including

budget and resource constraints and study objectives. As a result, the metadata does not

provide clear information regarding the extent of the market ...” There is no reason they

should be correlated so I do not understand the problem you are attempting to articulate

here.

13
• P. 14, top – I am struggling somewhat with your use of sampled area and affected resource

area. It would seem that affected resource area should affect welfare estimates; the

standard more is better argument. On the other hand, the sampled area should affect

aggregate welfare estimation. Given this, I am not sure the ratio relevant?

• P. 15, bottom – Consider that substitutes may not be other water bodies. For a user it may

be an alternative outdoor activity that is not associated with water, and for nonusers it may

be the current alternative for the discretionary income that they would allocate to nonuse

preferences. Addressing substitutes is good, but you should note that whatever is done is

imperfect.

• Equation (1) – Why not model Qojs as Djs*Qojs? Ostensibly, people are asked to value the

change in water quality. Baseline conditions should only come into play if the change

valued depends on baseline conditions, i.e., is not constant. This leads to the second

question of whether Qojs because you have already specified a nonlinear relationship

between willingness to pay and quality? This could explain the lack of significance you

discuss on the bottom of page 20.

• P. 19 – I disagree with the statement: “... MRM1 is best characterized as being a weak

structural utility theoretic meta-analysis ...” It is a reduced form, which is the standard

approach in the literature, and you should not claim more than this.

• P. 20, 1st full para. – Statement about the natural log of zero being zero is not correct,

ln(0)<0 and ln(1)=0. Yes, the unlogged version of equation (1) (equation (3)) does force

WTP to be zero when the water quality change is zero. So, it does not force WTP through

the origin when the change in water quality is zero (footnote 14). Given the importance

that this range of the function seems to imply for policy (e.g., the changes designated in

Table 12) I would like to see a table with the refined water quality changes in the data that

provides more information than Figure 2. It would be good to also present a cross tab of

the data in Figures 1 and 2 as an appendix table.

• P. 20 – It would be good to provide the major specifications investigated and a little

intuition for the specifications considered instead of saying many were considered. What,

specifically, were the model-selection criteria? It would help the credibility of the analysis.

• Table 4 – the estimated equation is not truly a trans-log function as coefficients are not

reported for squared and interaction terms.

• P. 26, 7th line – change to “... that are ‘potentially’ influential ...”

14
• Table 5 – Highlighting does not show well when the report is printed. Might want to

consider a more explicit form of identification for influential studies.

• Figure 6 – in addition to the general discussion of robustness results, I think it is worth

paying specific attention to the lump_sum and ce results as these are investigator choices

from the original studies that you must code for benefit-transfer predictions. Why are the

coefficients for these variables generally positive, but not always?

• Figure 7 – Similar comments for variables that are potentially influential. The effects shown

for ln_ar_ratio and lnbase are not surprising to me given the concerns/confusion I

expressed above. I am not sure the former is measuring an appropriate concept and am not

sure the latter should be included in the equation. Could the +/- effects for the user

variables be due to some studies valuing increments and others decrements that are not

controlled for in the estimated equation?

• Table 10 – Discussion might cite Kaul et al. (2013) as it shows that your percentage transfer

error is consistent with benefit-transfer validity studies in the literature

(http://su8bj7jh4j.scholar.serialssolutions.com/?sid=google&auinit=S&aulast=Kaul&atitle=

What+can+we+learn+from+benefit+transfer+errors%3F+Evidence+from+20+years+of+rese

arch+on+convergent+validity&id=doi:10.1016/j.jeem.2013.03.001&title=Journal+of+enviro

nmental+economics+and+management&volume=66&issue=1&date=2013&spage=90&issn=

0095-0696). A better approach for computing benefit-transfer errors is outlined by

Chattopadyay (2003): [((study case value - policy case value)/(study case value + policy case

value))/2]* 100.

• P. 41, top – I suggest also citing studies that show the distance decay for nonuse values to

support your assertion that the estimate is conservative. Also, if you consider recreation

demand RUMs they often use a three-hour travel distance, which also support your

conservative assertion.

• P. 41, 2nd para. – I wonder if for some projects a 100-mile buffer is too small. I see the logic

for the example computation here, but think that analysts should be encouraged to

consider the size of the buffer in conjunction with the characteristics of the program being

valued.

• Figure 9 – Needs improvement and legend to identify the meaning of the colors/symbols

used.

• Table 11 – I have several thoughts on the variable coding.

15
o Use of a choice experiment or contingent valuation needs to be thought through

carefully in original studies and this translates to benefit transfers. There are two key

considerations

1) Does the policy change to be evaluated call for implicit prices to value policies based

on various attribute levels? If Yes, then a ce.

2) Do people think of the resource in terms of separable attributes or do they think of

an integrated ecosystem? If the latter, then cv.

While a benefit transfer does not allow the flexibility of ce attributes, this can provide 

some guidance about what method you might choose for the control. Further, I think 

too little thought has been given to these two points and many of the applications for 

EPA may be better valued through cv than a ce. 

o Thesis – I think some of the best studies I have observed are theses because the

graduate student focuses solely on this one study while more experienced researchers

juggle multiple studies at the same time. Further, theses have intense peer review from

advisors and committee members. I do not have a concern with the coding, per se, but

the justification of “preferred” may be questionable.

o If we believe valuation methods are improving over time and that these improvements

have the potential to remove overestimation of values, then coding time to the most

recent year may be appropriate.

• Table 12 – I suggest more explanation of the content of this table in the text to bring the

reader along with you for these important policy descriptors in the meta-analysis

calculations. One directional sentence on page 41 is not sufficient.

• P. 46, top – the log-log functional form is not used “... in virtually all ... meta-regressions ...”

I believe that linear and log of dependent variable are also very common.

• P. 46, 1st limitation – This is an artifact of the lack of data in this region of the functional

relationship and the functional form chosen. It implies to me that this is not the “best” use

of the meta-equation to predict values estimates because it is outside of the range of the

data and the wrong functional specification is chosen. Rather than stating that “... future

water regulations may result in smaller improvements ... (and) ... it is important that the

Agency be able to use MMR1 to accurately estimate WTP for small water quality

improvements ...”, this is a clear indication that the agency needs to pursue original

research to value small increments in water quality to fill this gap in the empirical literature.

16
• P. 46, bottom – Another alternative to the linear interpolation is to simulated what values

might be in the lower range and re-estimate the meta-equation as having these data could

change the shape of the whole function, which is an issue that that has not been addressed.

• P. 47, 2nd limitation – It is important to remember that adding up is based on an assumption

about preferences imposed by the assumed economic model and is not required by the

general axioms of revealed preferences. Further, the meta-data were not estimated in an

experimental setting where adding up was imposed by the study designs nor empirically

investigated as test of the underlying preferences.

• P. 48, top – I recognize your challenge with addressing multiple programs that each makes

small, marginal improvements in water quality. This is a long-standing concern, even with

original studies, given the partial equilibrium nature of the empirical estimates. It seems to

me that it is important to ask whether you are going to treat the changes as a sequence of

small changes or as one big collective change. If it is a sequence, then it is important to get

the sequence correct and then apply value estimates appropriately to value each change. If

it is one large change then it is possible to dispense with the sequencing issue. Here I see

two fundamental concerns.

o Value estimates for small changes are likely biased due to the absence of original

benefit estimates for small changes and consequently a meta-equation that might be

incorrectly specified. By the latter, I mean the functional relationship for small value

changes is an artifact of the values observed for larger changes.

o You are taking partial equilibrium estimates to estimate the meta-equation and forcing

economic logic/conditioning on the econometrics summary of these estimates that was

not imposed in the original estimation. Now, a meta-equation can be used to test

adding up, but this is a very strict test because it forces adding up to hold across studies,

but adding up is innately a within-study phenomena if it exists.

Rather than forcing the meta-equation, and by consequence the existing value data, to 

address the adding-up condition, which the original studies were not designed to address 

and may be inconsistent with underlying preferences, I think the alternative approach 

considering marginal values is reasonable. 

• Equation (8) – This equation provides some important considerations.

o First, if we assume the WTP function is concave, consistent with the specification of

MRM1, then marginal willingness to pay is consistently underestimated by this linear

17
interpolation process given the available data. The underestimation increases as the 

change in water quality increases.

o The concern that data for small changes is limited and the estimated function is driven

by value estimates for larger changes remains; the estimated MRM2 function may also

be misspecified.

o The specification of the dependent variable, which includes components of Qjs suggests

that water quality is no longer a truly exogenous regressor.

These concerns apply to equation (9) as well. However, I believe that it would be 

worthwhile carrying through the nonlinear estimation to confirm your supposition that the 

end results would not change much.

• P. 49, 1st para. of Section 7.1 – Here I think that excluding lnQ0s is a concern as the marginal

value does depend on the change from the base water quality condition.

• P. 50, 1st para. of Section 7.2 – It is worth noting in the text that most met-analyses do use

cluster and that boot strapping is not as common, and that studies in the literature have

fewer observations that are available to use here to place these analysis procedures in

context. As written right now a uniformed reader may have the tendency to dismiss the

cluster analysis.

• P. 51, top – discussion of the outlier observation is unnecessarily confusing.

• P. 51, 2nd full para. – “The coefficient on Q is negative ... indicating a downward sloping

demand curve.” WTP is not conditioned on the baseline quality so and you are making

comparisons across studies and not within a single study. While this negative coefficient is

reassuring information you should be careful how you frame this discussion.

• P. 52, top – It is also worth reiterating here that income may not income as observed in the

study data. In addition, noting that full income may not be the income that people used in

answering the valuation questions, e.g., they may have considered available disposable

income after all current bills were paid. All the same, considering other specification to see

if income becomes significant is worthwhile given the important role of this variable, but

the above caveats indicate why it may remain insignificant for very plausible reasons given

the available measure of income.

18
• P. 56, 1sr para. – That MRM2 does not fit the data as well as MRM1 Is not surprising given

the error introduced in the computation of marginal WTP and potential additional

misspecification of explanatory variables.

• Figure 13 – Label panels for MRM1 and MRM2.

• Tables 18 and 19 – while I appreciate all of the information you present in the tables, it

would be nice to see the tables here, and elsewhere in the report, presented in single tables

to facilitate comparisons of models and estimation procedure results. I found is difficult to

make comparisons across pages and an underlying theme of this report is to consider which

model is best from a conceptual and empirical perspective. Such merging of tables would

facilitate the comparisons of estimation results. It would also be helpful to have a fourquadrant table that lists the strengths and weaknesses of the MRM1 and MRM2 equations.

I know the MRM2 model was a late addition so I assume you have not had enough time at

this point to flush out such an important comparison.

19
Peer Review for the U.S. Environmental Protection Agency of: 

“Meta-analysis of the Willingness-to-Pay for Water Quality Improvements” 

Jacqueline Geoghegan, Ph.D. 

Professor and Chair of Economics 

Clark University, Worcester MA 

May 28, 2015 

Introduction 

As I discussed with Professor Kiel at the onset of this peer review, I am not an expert in 

stated preference, meta-analysis nor benefit transfer. Rather, what I bring to this peer review is 

twenty years of experience in spatial environmental economic modeling.1

 Consequently, the 

majority of my comments will focus on the spatial data and modeling aspects of the reviewed 

document, although I will make other overall comments when warranted. 

Most generally, it became immediately clear to even a non-expert like myself, that 

“MRM1” is a well-established, peer-reviewed model that has undergone extensive and varied 

robustness tests, while “MRM2” is at a very preliminary stage of development. Indeed, as EPA 

writes on page 56 “MRM2 was a late addition to this report, so EPA views the results presented 

in this section as still preliminary since they have been only partially stress tested.” Therefore, a 

complete peer review on “The Battle of the MRMs” at this juncture would be not overly 

productive. However, in the conference call of 05/05/2015, (which I was not able to participate 

in, as it was scheduled for a time that I was not available, but I listened to the recording of the 

call), EPA staff provided the peer review panel with background comments on the context of the 

models as well as indications that EPA has already established a planned line of research for 

moving forward with these two models. Furthermore, as EPA writes on page 56 “EPA intends to 

apply more sophisticated estimation approaches to MRM2 in follow up work.” Finally, on page 

66, section 11 “Next Steps”, one of the two examples included is “incorporation of distance 

decay”, a specific spatial data refinement, that I hope my comments below can assist EPA as the 

Agency moves forward with this work. 

Given this background, while I will answer the “Priority Questions” and “Extension” 

questions to the best of my ability in this peer review, many of my comments will be in the spirit 

of suggestions to EPA for future research (specifically in regard to the spatial aspects, such as 

comments on distance decay) in meta-analysis and benefit transfer applications, either in regard 

to water quality issues as in this specific case, but also more broadly to the overall mission of the 

Agency to protect human health and the environment. In addition, there are some overlaps in the 

spatial aspects contained in the Priority Questions to the peer review panel of the report (i.e. 

 

1

 Geoghegan, Wainger and Bockstael (1997) was the first paper in environmental economics to use GIS data and 

spatial analysis. As of May 2015, it has been cited 636 times (google scholar). In addition, any benefit estimation 

work that I do has always been in a revealed preference framework (i.e. hedonic models) with related work on air 

quality and open space amenities, .so no specific research experience with water quality either. 

20
Questions #2, #3 and #5 all refer to spatial issues), resulting in some repetition in my responses 

to these questions, especially with regard to spatial scale issues and the spatial unit of analysis. I 

also note that it is likely that many of my suggestions might require more extensive GIS 

expertise and resources than are currently available to EPA, but I believe that the investment by 

EPA (and more broadly throughout the public sector) in this burgeoning area of spatial data 

creation and analysis is highly critical for future data collection, management, and analysis. 

Overall, upon my first read of this EPA document, as well as other additional background 

material on meta-analysis and benefit transfer2

, I was quite surprised by the limited amount of 

research that has been performed in these applications on environmental quality that has taken 

either a spatially-explicit or more broadly, spatially-informed approach to these methods. Space 

must play a critical role in environmental MRMs for benefit transfer and policy analysis, with 

regard to the geospatial scale of the impacted resource, the market extent of the potential 

residents who value such changes and the extent and availability of substitutes for the impacted 

resource. The MRMs presented in this report, which begin specifically to include these critical 

spatial characteristics, I believe, are a clear improvement over previous MRM models used for 

benefit transfer. 

Responses to Priority Questions 

1. As this question is specifically on the functional forms used for the MRMs, I will leave this 

question to other panel members with expertise in this area to address this question in detail. The 

numerical example given in section 9 of the report is illustrative, but given the nascent stage of 

the development of MRM2, I am not sure if the current example is a useful comparison of the 

two models. As EPA moves forward with both of these models developed further, different 

simulations, such as the one given in the report as an example, by using each model with the 

same benefit transfer scenario, could help determine if the empirical outcomes dominate any 

potential theoretical effects relating to the issue of the adding-up condition. 

However, more broadly, as an applied environmental economist, I do have more general 

concerns on the use of any model to predict impacts so far outside the range of observed metadata. I realize the constraints that meta-analysis researchers and EPA face: that there are no 

primary analyses that use baseline or changes in the range that EPA policy considers in this 

specific application. I would encourage EPA then to consider funding research in the future that 

would be helpful for the development of data (and methods, see my comments below) that would 

be in this range for policy analysis and implementation. 

2. I view Question #2 as having two parts. The first part, explicitly stated in the question, concerns 

the modeling of the trade-offs in preferences between a measure of water quality and the (linear) 

area affected. The second part, not explicitly stated in this question, but is one of the main points 

of Question #3: “is 100 miles the ‘right’ spatial distance to use in this model?” For the first part, 

 

2

 Nelson and Kennedy (2009) and Johnston and Rosenberger (2010). 

21
concerning the trade-offs, it is an empirical question about how individuals value these two 

aspects. Are there any stated, revealed or experimental data to support this modeling choice of 

these two aspects as perfect substitutes for each other? If not, can different weights be used to 

experimentally test how much the different specifications actually matters for predicted 

outcomes? 

The water quality index could also be a function of surface area or some other measure of “size” 

or water body type. For example, my intuition is that a “shoreline mile” for a river is affected 

more by a certain change in water quality than a “shoreline mile” of a large lake (i.e. where the 

benefits would be more diffuse over the larger area). On the other hand, should each “side” of a 

river “count”, should this be a function of river width? On page 14, EPA writes: “By delineating 

resources using GIS, EPA is able to incorporate detailed spatial and hydrological features of the 

waterbodies.” What exactly is meant by “detailed”? Related, has EPA done any sensitivity 

analysis with these “details”, such as different variable specifications based on the type of 

surface water along these ideas? On page 15, EPA writes: “EPA considered, but did not select, a 

variety of alternative specifications of the geographic scale and extent of market variables (e.g., 

shoreline length).” What does “considered” mean? EPA thought about different specifications 

and did not try, or implemented different specifications and that the specifications did not 

improve the model?3

 

More broadly, while much documentation in this report is given on the different standard (for 

meta-analysis) robustness checks for MRM1, what other sensitivity analysis were done on the 

new spatial variables included in MRM1? As this approach of using external spatial data (i.e. 

not included in the original studies) and GIS-created variables using these data appears to be so 

new in this meta-analysis literature, there might not exist standard or agreed upon methods on 

this specifically for meta-analysis, as there appears for “traditional” meta-analysis research.4 

Therefore, if the spatial approaches and methodologies suggested in MRM1 are to be used more 

widely by other researchers and/or in other applications, reported sensitivity analysis would be 

helpful in order to develop such protocols for future applications. This is another area of 

research that EPA should consider for future work: the further development of methods for 

determining how best to include these important spatial features of the landscape derived from 

readily available, external GIS data into meta-analysis and benefit transfer models. 

The not-specifically-asked second part of this question is about the choice of the 100 miles cutoff, I will discuss in my response to Question #3 next. 

 

3

 My comments here also are reflective of Extension Question #9 that specifically asks for suggestions on this 

variable and specification. 

4

 Such as is found in Navrud and Ready (2007) and Johnston and Rosenberger (2010). 

22
3. This question begins: “As explained in Section 6.1 of the report, EPA considers the blockgroup 

approach an improvement over previous meta-regression models applied at the state level.” 

However, I don’t see any argument making that case in this section.5

 (Perhaps this is due to 

some mislabeling in the document, as this is the section of the document with an 

“August17August17 (sic), 2012” and “316(b) Stated Preference Survey Peer Review Document” 

in the footer?) But more specifically, I believe that in MRM1, Census data are used only to 

construct an internally consistent income measure for each observation in the meta-data, which is 

a reasonable thing to do. In any event, with regards specifically to benefit transfer (which is 

what section 6.1 contains), using blockgroups (BG) as unit of analysis for the creation of specific 

values for variables seems reasonable, as this is a generally recognizable and available spatial 

(GIS) geography for the basis of analysis. In general, as an applied micro-spatial environmental 

economist, all else equal, I prefer to use the smallest available spatial unit of analysis for 

research. However, perhaps as EPA moves forward with this work, some additional types of 

sensitivity analysis on the spatial level of aggregation (i.e. using larger geographies) to determine 

how influential this choice is on the empirical outcomes. 

In Section 6.1, EPA writes “It is assumed that households in each BG value changes in the 

affected waterbodies within a specified distance, here set to 100 miles. A radius of 100 miles is 

consistent with Viscusi et al. (2008) based on the assumption that the majority of recreational 

trips will occur with a 2 hour drive from home.” (p. 41). In reading Viscusi et al. (2008), the 

authors state that they used the 100 mile distance in their survey, as US EPA NCEE had 

calculated that about 80% of recreational trips contained in the 1996 National Survey on 

Recreation on the Environment occurred within the 100 mile radius. These data are almost 20 

years old. It appears that this survey was done again in 2010, so perhaps EPA could consider 

analyzing these newer data to determine if these distances are stable over time. That said, 

recreational use is only one of the uses and non-uses that EPA is interested in valuing, so it is 

unclear to me that this is enough of a reason for the choice of 100 miles. 

More generally, beyond being an empirical question of what the “right” distance should be, it is 

also unclear if it should be a constant fixed distance for all locations in the United States. The 

extent of the market is an empirical question so perhaps smaller buffers for smaller watersheds 

and/or different population densities etc. For example, the “right” distance could be a function 

of population density and related congestion effects.6

 That is, there are (at least) two potential 

types of congestion effects: one is of the resource itself (in higher population density areas, a 

resident would have to travel further to reach an uncongested site if that is preferred) the other is 

the transportation costs of access, if road congestion is greater in higher population density 

regions, this would increase the time costs to reach a destination. Therefore, another potential 

robustness check would be for EPA to experiment with different sized buffers, and not 

 

5

 On page 56 EPA discusses the use of an “exogenously specified threshold distance cutoff when using the metaregression models described in this report for benefit transfers.” 

6

 I focus here on population density as an example, as population counts are readily available and population 

densities easily created in a GIS. 

23
necessarily constant across all observations, or interaction terms, although I realize that there are 

potential issues with degrees of freedom and over-fitting of models here. Again, my point in 

response to Question #2 on sensitivity analysis of MRM1 applies here. 

In this panel review charge question, EPA writes “We would expect that, for the typical 

household, the value placed on surface water quality will declined monotonically with distance 

from home...” Is there any justification for EPA’s claim of “monotonically”? Why not 

quadratic? Or any other functional form? Again, this relates to the specific values under 

consideration: distance decay might be appropriate for recreational or other use values, but not 

necessarily for all values. For example, as Johnston and Rosenberger (2010) write “it is possible 

for certain public goods to have a constant average impact on residents’ utility within a given 

geographical market that does not decline – or changes by only a trivial amount – over distance.” 

(p. 492). Therefore, while a distance decay function should be experimented with, there are 

other options that are possible that could be tried in the MRMs to test for scalar effects. For 

example, as the spatial extent of a BG could vary with population density (as BG are delineated 

by population and not geographical area), could average BG spatial size be included on the RHS 

of the MRM? If that matters in the MRM, then perhaps that would be evidence in the benefit 

transfer for having the distance decay or buffers be also a function of population density? (For 

the reasons given in the previous paragraph.) 

4. While this question is outside my area of expertise, the GLS approach used in MRM1, given the 

constraints of the underlying data, appears appropriate. However, I was confused by the 

specifics of this question. In footnote #12 on page 19, EPA writes “Weighting observations 

based on the number of WTP observations reported in each study produced nearly identical 

results.” So hasn’t EPA already performed some of this kind of sensitivity analysis? 

5. This question lists a number of the spatial variables developed for MRM1 and resulting benefit 

transfer. The peer reviewers are asked “Do you think each of these variables improves the metaanalysis and subsequent benefit transfer analysis?” Well, generally, yes, given my assorted 

comments above on my view of the value of adding spatially-explicit relevant variables. Do I 

think that “each” variable individually “improves” the analysis? I do not think it is currently 

possible to answer that particular question, although I have given a number of suggestions above 

and to follow here for other specification and the need for more sensitivity analysis. Any 

variable should be included in a model for theoretical reasons, but in many cases, it is an 

empirical question of the “right” form of a variable is to use. 

Of particular concern is the current “proportion of the agricultural land in the affected area”. 

First, I didn’t really understand the description and rational for the benefit transfer example 

(given on page 44 in Table 11). But more importantly, EPA includes this variable, but there 

appears to be no theoretical justification specifically articulated by EPA in this report for the 

inclusion of this variable. EPA writes on page 16 “The agricultural fraction is calculated as the 

area-weighted average of counties within the affected resource area. EPA also developed and 

tested an analogous variable based on the fraction of land that is in developed land use categories 

24
but it not (sic) improve model fit.” Again, there needs to be theoretical justification for the 

inclusion/exclusion of variables, including these land cover variables, and not just simply 

including “available” data to create variables that “improve model fit”. Why should these 

variables affect WTP? It would be helpful for EPA to articulate this here. In addition, similar to 

my previous comments above, what kind of sensitivity analysis with these variables were 

performed to determine suitability for inclusion in the MRM or how variation in them affects 

specific benefit transfer applications? 

Responses to Extension Questions

 Extension Questions #6, #8 and #10 are specific questions concerning meta-analysis and 

Extension Question #7 focuses on water quality measures, which is also out of my area of 

expertise, so I will leave these to the other peer review panel members. 

My contributions to these Extension Questions focus exclusively on Question #9 which 

asks about “other geo-spatial features that might influence WTP for water quality?” I raised a 

number of questions and potential suggestions regarding these issues in my responses above to 

the Priority Questions. Additional comments and suggestions are included next. 

On page 15, EPA writes: “... EPA used intersecting counties to define all affected 

resources, including discrete rivers, as geographic areas while avoiding the arbitrary selection of 

a boundary distance.” It is unclear to me why county boundaries are particularly less “arbitrary” 

than a fixed pre-determined and/or variable/experimental distance, as county boundaries can be 

arbitrary as well. Counties can vary greatly in size and in some areas of the country. In addition, 

while counties exist throughout the United States, they can play almost no jurisdictional or 

social/political role in some states (e.g. in Massachusetts, the key identification is the “town”). 

The affected resources could be measured by boundary distance or some function of population 

density as well (as discussed in my responses to Priority Questions above). 

On page 15, EPA writes: “... EPA considers the state to be a reasonable basis for the 

calculating (sic) the substitute proportion.” While I agree with EPA’s discussion of the 

important of spatial/cultural homogeneity, it is unclear to me if state boundaries capture this to 

the best degree possible. Again, similar to counties, states can vary greatly in both geographical 

and population size across the United States. For example, Rhode Island is tiny, so should the 

available substitutes be all of New England instead of only that state? Or for Rhode Island, 

should the available substitutes include the states of southern New England (i.e. associate Rhode 

Island with Massachusetts and Connecticut)? On the other end of the spectrum, California is a 

large land area with a huge population: Northern California is very different than Southern 

California. Perhaps EPA could consider comparing their state approach to determining 

geographically-available substitutes to a definition based on OMB’s Metropolitan Statistical 

Areas (MSA) as these are created to contain geographical areas that have a “high degree of social 

and economic integration” (www.census.gov), which can cross states lines and could be a better 

fit for these areas than state boundaries. 

25
On page 16, EPA writes: “For non-river inland waterbodies (e.g., lakes and ponds), the 

substitute proportion is calculated based on area relative to areas of lakes with the states(s).” Is 

this done just for lakes, or are “ponds” included too? For example, has EPA considered 

subdividing and/or weighting these data by the USGS Hydrologic Unit System (another potential 

sensitivity analysis)? While EPA writes on page 14 “For rivers and lakes, the Agency opted to 

match to the NHDPlus dataset over other publically available georeferenced hydrological 

datasets (e.g., E2RF1) because...” but given the importance of these data for this application, 

EPA might consider adding more detail on the different datasets available and the pros and cons 

of the use of each data source. Or perhaps make clear these are the data that are generally used 

by EPA for these kind of purposes. 

On page 23, EPA discusses different potential sample selection biases of the studies 

included in the metadata, and included on page 24 is a map of the United States where the 

original studies occurred, but there is no mention of the potential of over-sampling of states in 

which specific researchers live. EPA asserts on page 23 “a relatively good geographic and 

waterbody type coverage of the studies”, but considering the dominance of studies in North 

Carolina, Virginia, Maryland, and West Virginia and the paucity of observations from the 

western half of the country, perhaps EPA could do a fuller justification for this claim? Are there 

any MRM and/or political implications for extrapolating these data to other states? Have MRM 

researchers developed robustness tests to consider this aspect? More broadly, even within each 

study (observation for the MRMs), for sample areas by both jurisdictional and watershed 

boundaries, how was the survey population samples in the original studies? That is, where the 

survey respondents selected based on any specific spatial criteria? Or does EPA simply have to 

assume a uniform spatial distribution of survey respondents across the sample area? If so, has 

EPA considered how the spatial sample compares to the spatial distribution of the population? 

Conclusions 

As I wrote in my introductory comments, while I am not an expert in meta-analysis and 

benefit transfer, I am convinced that the incorporation of spatial/geographical characteristics into 

these MRMs and resulting benefit transfer models is a vast improvement in both theory and 

application as well as subsequent use of this approach for policy and decision making. There are 

three broad areas of research that EPA should focus on as they proceed: the first two in the 

shorter term and the last is a longer term issue. The first is the further development and testing 

of MRM2 and how it compares to MRM1 for a more even comparison of the two model 

specifications to determine if there is indeed a trade-off between theory and empirics (already 

self-acknowledged by EPA in this peer review document as well as earlier phone call discussed 

above). The second is for the further methodological development of protocols for the inclusion 

of these new additional externally-generated GIS variables into meta-analysis used for benefit 

transfer. The third, longer term issue, is for the support for the new original research necessary 

to create observed data in the relevant range that EPA needs for policy analysis. 

26
References

Geoghegan, Jacqueline, Lisa Wainger and Nancy Bockstael. 1997. “Spatial Landscape Indices 

in a Hedonic Framework: An Ecological Economics Analysis Using GIS,” Ecological 

Economics 23(3): 251-264. 

Johnston, Robert J. and Randall S. Rosenberger. 2010. “Methods, Trends and Controversies in 

Contemporary Benefit Transfer,” Journal of Economics Surveys 24(3): 479-510. 

Navrud, Stale, Richard Ready (Eds). 2007. Environmental Value Transfer: Issues and Methods. 

Springer: The Netherlands. 

Nelson, Jon P. and Peter E. Kennedy. 2009. “The Use (and Abuse) of Meta-Analysis in 

Environmental and Natural Resource Economics: As Assessment,” Environmental and Resource 

Economics 42: 345-377. 

Viscusi, W. Kip, Joel C. Huber and Jason Bell. 2008. “The Economic Value of Water Quality,” 

Environmental and Resource Economics, 41(2): 169-187. 

27
Meta-Analysis of Willingness to Pay for Water Quality 

Improvements Responses to Charge Questions -Timothy Haab

Priority Questions: 

1) Priority question 1) consists of two parts. In part A we are asked whether the use of linear

interpolation in MRM1 adequately addresses concerns about extrapolation beyond the realm of

observable water quality changes. In part B we are asked to assess whether using an

approximation of marginal willingness to pay in MRM2 adequately addresses concerns about

MRM1 not meeting Diamond’s (1996) adding up condition and the problem of MRM1 predicting

values for small water quality changes that are perceived to be ‘too large.’

Before directly addressing either of these issues, I would like to first give a brief comment on 

Diamond’s adding-up condition in the hopes of ruling it out as a deciding factor between the 

two models. In short, the adding-up condition is an extreme form of the NOAA Panel’s 1993 

recommendation that contingent valuation results in natural resource damage assessment 

should show adequate responsiveness to the scope of the resource being damaged. In the 

current context, this should be interpreted as there be a positive relationship between water 

quality changes and willingness to pay. This is clearly the case in MRM1 as evidenced by the 

positive and significant coefficient on lnquality_ch in Table 4 on page 22. Unfortunately, the 

NOAA Panel did not define what is meant by ‘adequacy to scope.’ Some have interpreted 

Diamond’s adding-up test to be the gold standard for scope tests. EPA seems concerned that 

MRM1 does not meet the more stringent conditions of Diamond’s adding-up test. This is not 

correct. 

In a response to Hausman’s 2012 piece in in the Journal of Economic Perspectives, Haab et al 

(2013) address the issue of the adding up test and more generally defining adequacy to scope. 

They point out: 

 “A simple theoretical model of WTP...can be used to show that WTP is nondecreasing in

quality or quantity.” (Whitehead et al, 1998)

 “The size of scope effects may be limited by diminishing marginal utility or substitution

among nonmarket goods...” (Rollins and Lyke, 1998)

 “The size of the scope effect in [a formal model of scope with bounded utility functions]

depends on the substitutability between market and nonmarket goods.” (Amiran and

Hagen, 2010)

In related discussions, Haab et al note that since 1996, very few researchers have chosen to test 

CV with the adding up condition. It is not the standard test of scope in the literature and as such 

it should not be used as the sole means of accepting or rejecting model specifications. 

As such I find very little convincing evidence in the report to distinguish between MRM1 and 

MRM2 based on adequacy to scope responses or the adding up condition. Both models exhibit 

responsiveness to scope consistent with what has been found in the literature since 1996. 

This leaves the question of whether MRM1 or MRM2 provide a ‘better’ means of interpolating 

changes in water quality beyond the observed studies (in this case, changes that are smaller 

28
than those that are published). Before addressing this question directly, a caveat: the fact that 

there are very few published or ‘grey’ studies that value small changes in water quality raises 

the issue of publication bias in the chosen studies. Beyond the issue of interpolation, this raises 

concerns in determining adequacy of scope. If only studies that pass a scope test (or value an 

adequately large change in water quality) are published, then of course the meta-analysis will 

find a significant response to scope (or fail to capture small changes in water quality 

adequately). While not limited to the current study, publication bias raises questions for policy 

application of meta-analysis, particularly for valuing changes beyond the observable realm of 

existing studies. 

Prediction beyond the realm of observable data has plagued the stated preference literature 

since its early days. As such, there is very little guidance in how to handle out of range 

prediction. In any parametric setting, the out of range prediction is going to be highly sensitive 

to the chosen functional form. Bounded functional forms (like the log-log and semi-log) are 

notorious for sensitivity in the tails of the error distribution. Fixes to functional forms to simply 

make the results look better (pass the laugh test?), result in arbitrary results subject to easy 

criticism. 

Based on this, I find little evidence that MRM2 improves the meta-analysis. There is little 

theoretical justification for the additional structure imposed by MRM2 (i.e. I don’t buy the 

adding-up argument) and adjusting functional forms to get results that look better is arbitrary at 

best. As such, I recommend the use of MRM1 and recommend ensuring conservative estimates 

of WTP for changes beyond the observable range using linear interpolation as seen in figure 10. 

The linear interpolation gives a lower bound estimate for any concave function form. While not 

the subject of the current study, and unlikely to happen in a reasonable policy, for changes 

above the observed range (in the upper tail), I recommend capping WTP at the top end of the 

observable range. Doing so will ensure a conservative and defensible WTP measure. 

2) Priority question 2) focuses on the specification of the commodity change and asks specifically if 

specifying water quality and water quantity as ‘perfect substitutes’ makes sense? My short 

response is given the data collected, I can think of no better way to handle this. Unless separate 

data is available for each study on quantity and quality, and they could be incorporated into the 

meta-equation separately, an arbitrary assumption is needed. Linearity seems as good as any 

other arbitrary assumption. 

3) Similar to response 2, unless distance to water bodies is available for all data points in the study, 

and can be incorporated into the estimated meta-equation, an arbitrary assumption is 

necessary. Distance decay is well-documented in the literature (for both use and non-use 

values). How quickly that decay occurs is an open question. While risking highlighting the 

arbitrary nature of the 100 mile threshold, it might be worth reporting calculations based on a 

couple of other distances (75 and 125 miles?) to show sensitivity to the assumption. While the 

100 mile radius is consistent with Viscusi et al (2008), additional studies find that distance-decay 

occurs at faster or slower rates depending on whether the affected populations are users or 

non-users and the scope of the change. For example, Hanley, Schlapfer and Spurgeon (2003) 

state: “In accord with a priori expectations, we find that a more rapid distance decay exists for 

use values than for non-use values. We also find no distance significant effect for a general class 

29
of environmental good, where a significant effect exists for a specific local example of this class.” 

So while 100 miles may be defensible, it is worth noting that other distances may produce 

different results. 

4) I have always been a fan of including methodological and study specific variables in metaregression equations. Because it is well known that willingness to pay is sensitive to assumed 

functional forms, distributional assumptions, author fixed effects, I would like to see some 

additional controls included. In many cases, such controls will serve as a fixed effect and will 

swamp the random effect. As to the question of weighting, because each observation of the 

dependent variable is an average, it is tempting to treat the dependent variable as an average as 

you would with county-level averages or census-track averages. However, doing so would seem 

to indicate that you are estimating a regression where the dependent variable is actually an 

individual and not an average. Because these samples are not drawn from a common 

population, there would need to be many more controls in place to correct for the inconsistent 

sampling across studies. 

5) In short—yes. I like the addition of additional variables from outside the original studies. 

6) Were ‘contingent behavior’ studies considered? Also, while there is significant overlap in the 

steps and many of the journals were likely covered in other steps, the list of journals looks 

incomplete with journals like the American Journal of Agricultural Economics, the Journal of 

Environmental Management, Journal of Agricultural and Resource Economics...). Likewise, the 

author list in footnote 3 is only a fraction of the authors active in this field. Again, most of the 

authors are likely picked up in other steps, but the list looks strikingly short. 

7) I found the WQI and WQL translations to be satisfactory. 

8) Yes, particularly in the context of the scope/adding up discussions. It is unlikely that a stated 

preference study that fails to find scope effects would be published. Unfortunately, addressing 

this issue is nearly impossible without additional information on unpublished studies. This 

might be particularly troublesome for MRM2 as the dependent variable is specified as WTP/Q. 

As such, the error term is likely truncated from below as studies with small WTP/Q are likely 

omitted from the sample. Taking the log of the dependent variable is an arbitrary fix that rules 

out the possibility of WTP/Q = 0. The fact that you don’t observe it is not an indication that 

WTP/Q is never equal to zero, but rather an indication that such studies will rarely appear in the 

literature. Failure to find adequate scope effects in a meta-analysis would be shocking as a 

result. 

9) I like the idea of including distance as an explanatory variable as long as it is available. My guess 

is that many studies do not report such information. 

10) I have no comment here. 

30
References: 

Amiran, Edoh Y., and Daniel A. Hagen. 2010. “The scope trials: Variation in sensitivity to scope and WTP 

with directionally bounded utility functions,” Journal of Environmental Economics and Management

59:293–301. 

Diamond, P.A. 1996. “Testing the Internal Consistency of Contingent Valuation Surveys.” Journal of 

Environmental Economics and Management, 30: 337-347. 

Haab, T.C., Matthew G. Interis, Daniel R. Petrolia, and John C. Whitehead. 2013. “From Hopeless to 

Curious? Thoughts on Hausman's “Dubious to Hopeless” Critique of Contingent Valuation” Appl. Econ. 

Perspect. Pol. 35 (4): 593-612

Hanley, N., F. Schläpfer, J. Spurgeon. 2003. “Aggregating the benefits of environmental improvements: 

distance-decay functions for use and non-use values,” Journal of Environmental Management, 68:297-

304. 

Hausman, Jerry. 2012. "Contingent Valuation: From Dubious to Hopeless." Journal of Economic 

Perspectives, 26(4): 43-56. 

Rollins, Kimberly and Audrey Lyke. 1998. “The case for diminishing marginal existence values,” Journal of 

Environmental Economics and Management 36(3): 324-44. 

Viscusi, W.K., J. Huber, and J. Bell. 2008. “The Economic Value of Water Quality.” Environmental and 

Resource Economics, 41(2):169-187.

Whitehead, J. C., T. C. Haab and J. Huang. 1998. “Part-whole bias in contingent valuation: Will scope 

effects be detected with inexpensive survey methods?” Southern Economic Journal, 65(1):160-168. 

 

31
COMMENTS ON THE PEER REVIEW PACKAGE: 

META-ANALYSIS OF WILLINGNESS-TO-PAY FOR 

WATER QUALITY IMPROVEMENTS 

First, I’d like to say that I’m impressed with the work thus far. The EPA has done a thoughtful 

and careful analysis and is to be commended for the work done thus far, and for requesting input 

at this stage in the work’s development. -Laura Taylor 

Priority Questions 

1. One of the greater difficulties I see is that the EPA is interested in valuing water quality

changes that are outside (below) the range of the observed metadata on water quality

changes. As such, any estimates of the WTP for ΔWQ<2.5 rely wholly on functional

form. It is clear EPA recognizes this, but it is worth repeating that this is really

problematic, and is evidenced by the issues with MRM1 as ΔWQ becomes very small.

While I don’t disagree with the proposed linear interpolation as a way to “push down” the

values for small WQ changes using MRM1, there is a certain logical inconsistency that is

unappealing. To me, the discussion reads as follows: The EPA prefers the translog

function because it forces WTP(D=0) to be zero. Yet, the EPA finds the estimation

results from this functional form to be unpalatable at ranges less than 2.5 and thus alters

the function between 2.5 and the origin... so what then is the benefit of the translog?

One doesn’t have to choose a functional form that goes through the origin to make an

assumption and draw a straight line in the WTP function from D=2.5 to D=0.

One approach to consider is a linear or cubic spline function. Since valuation is mostly

based on the WQL (or an index/description that is easily mapped to the WQL), you have

very natural knots in the data. Through a linear (or cubic) spline, you can get away from

the assumed smoothness of the log-log function. Perhaps knots at each 10 units (0, 10,

20, 30, etc.)?

Further, given the fact that value estimates for ΔWQ<2.5 will be determined by choice of

functional form, it seems to me that having explored many functional forms is an

important exercise and should be incorporated into the benefits transfer framework.

Perhaps the EPA could consider an “internal meta analysis” (of their meta data) in which

WTP(ΔWQ<2.5) is generated for a large array of functional forms and the median (or

mean) estimate is used for benefits transfer, along with 5th/95th percentiles (or whatever

range is deemed appropriate).

In sum, I think EPA should vigorously pursue the logic in MRM2 (MWTP) as well as in

MRM1 – meaning pursue a large variety of meta regression model functional forms,

including using dependent variables that are WTP and MWTP and not wedding

themselves to translog. Note, I don’t believe MRM2 should be pursued because of

adding-up. I am not convinced that satisfying the adding-up condition is a first-order

priority (although it may be a practical one from the perspective of satisfying a wide

32

bodies of water (e.g., Lake Lanier in Georgia or Chesapeake Bay) versus a network of 

small, but wadeable streams in a watershed. 

Recognizing that there is very little information available, I wonder if the EPA could 

attempt to recover some evidence from recent studies. For example, could the EPA 

contact authors of studies that seem like good candidates for re-calculating WTP for 

different groups of respondents according to distance to the resource. For studies that 

used well-defined resources (i.e., a specific river or lake) and a large enough sample (i.e., 

a random sample of residents in the state), the EPA might be able to get the authors to 

provide some rough estimates of the distance-decay relationship. Without knowing the 

details of each study, it seems Londono Cadavid and Ando (2013), Welle and Hodgson 

(2011) and Whitehead (2006) might be candidates. Or perhaps even Stumborg et al 

(2001, although this is quite old for keeping data/models). Could the EPA fund 

additional work on the data collected through these studies that attempts (using original 

methods) to breakout values by proximity of the respondent to the water body in 

question? I don’t think this work would be large in scope or used as additional datapoints in the actual metadata, but rather just provide rough estimates that are used to 

inform judgments about the distance-decay relationship. 

Without going back into the original data underlying the metadata, I can’t see a way to 

better define the distance-decay relationship (and I can’t see a way to build it into the 

MRM given the information gleaned from studies). 

There may be a way to add a bit more heterogeneity in the benefits estimation within the 

confines of the MRM/data you have available. Perhaps you could use the variable 

nonusers to generate heterogeneous values that are applied to a set of rings around the 

census block. For instance, you could keep the assumption that use values are primarily 

held for waters within 100 miles. In this case, one could use predicted WTP from the 

MRM when nonusers=0 as the average value for waters within 100 miles, and then for 

waters greater than 100 miles, but within the same state, use the predicted WTP as 

determined by the MRM when nonusers=1. 

4. One error structure that is easily considered is to group WTP estimates by author, rather

than by study. This takes some judgment, but it captures the correlation among WTP

estimates that may be present due to the same researcher making similar judgments in the

research process (including which models ultimately are “preferred,” and thus reported.

For instance, looking at the list of studies, one might consider grouping the following:

Banzhaf et al 2006/2011; Bockstael et al 1988/1989; studies with Rosenberger (if more

than one); studies with Whitehead; and so forth.

One certainly can weight by the inverse of number of observations from each study, and

the impact on the results would be good to see at least in a robustness section.

One weighting scheme that is sometimes suggested is to weight by a measure of the

precision of the estimate (e.g., weight by t-statistic). I’m not a big fan of this method as it

focuses on precision, but is silent on bias (and I find bias to be the more important of the

34
two issues). Further, poor econometric analyses can lead to overstated t-statistics, and 

this weighting scheme ignores this fact. 

5. Yes, I think the variables in question improve the usefulness of the meta analysis for

benefits transfer. I do, however, think they could be explored further to better capture the

intended relationships and have some comments/suggestions:

a. First, and importantly, it seems to me that the change in water quality should be

interacted with the baseline water quality. It seems to me that the appropriate

specification is:

WTP = β1*D + β2*Q + β3*D*Q + ...

This specification allows WTP to be conditional on the baseline level, capturing

the fact that the WTP for a 5-point change off a baseline of 75 may not be valued

the same as a 5-point change off a baseline of 45.

This is also more consistent with MRM2, that specifies MWTP to be a function of

the baseline WQ.

b. I find the definition of shoreline length for rivers odd (EPA doubled the reach

length to reflect both shorelines of a river). Would a lay-person interpret an

improvement along 10 miles of a river as “20-miles of shoreline improvement”? I

find this unlikely. The analogy that comes to mind for me is a car: because I can

access my automobile from two sides, does it mean I have two cars? I don’t

believe this definition matters for the variable sub_proportion since both the

numerator and the denominator are multiplied by two in the case of rivers, but it

would be good to confirm the use of this assumption.

c. Capturing the extent of the market is hard to do, and I’m not sure I can suggest a

better approach than what is currently being used. It seems to me there are two

broad types of sampling frames for survey recipients: targeted (e.g., residents in a

set of counties, region, or a city) or broad (state-wide) and the resource being

valued is either contained within the sample or the sample is contained within the

resource. Your variable ar_ratio does capture this, but perhaps more flexibility

could be introduced to allow the impact of ar_ratio to be different depending on

whether the resource is contained within the sampling frame or vice versa. For

example, let CW = 1 if the resource is contained within the survey sample frame

(Hayes et al in Figure 3 and Matthews et al in Figure 4 would both have CW=1,

while the others would have CW=0). Then the regression model would be:

WTP = β1*CW + β2*ar_ratio + β3*CW*ar_ratio

d. A small point: I believe ar_ratio would be much more intuitive and easier to

explain if it were the inverse of the current definition (i.e., = % of affected area /

% sampled area). We would expect the coefficient to be positive and the

35
reasoning why we expect a positive sign avoids using negatives combined with 

inverse relationships (which is confusing), as the current definition requires. This 

is just a minor expositional point. 

e. In general, it seems more interaction terms could capture spatial influences on

WTP. For example, mult_bod could be interacted with sub_proportion to capture

the fact that the influence of substitute lakes (identifiable, specific features) may

influence WTP differently than the presence of substitutes rivers and streams

throughout a watershed (or whatever spatial definition). This relates to my

comment in 9c below.

f. Note, I would expect your variable “sub_proportion” to follow the same logic as

the needed assumptions in benefits transfer. Currently, sub_proportion uses the

state as the backdrop for substitute water bodies. However, in the BT, you use

100-mile buffers.

Extensions beyond the Present Application (Optional) 

6. The selection criteria seem fine, although it was unclear to me if you went back and

applied all the criteria to the original 2004 and 2009 compilations. Did the criteria

change over time? If so, go back and make sure the studies from 2004/09 compilation

meet the current criteria if you haven’t already done so (it sounds like you did).

Also, it was not clear to me if you included multiple observations from a study that arise

from specification choices in the modeling (rather than just those that address a different

component of the study data or values for alternative resource changes). If not, why not?

If not, and there were multiple estimates that arose based on modeling/specification

choices, how did you choose which one to include? Perhaps this is not an issue and

studies generally report WTPs based on one estimate model/specification?

Lastly, you could consider doing a social science citations index (SSCI or currently the

“web of science” database) on the existing papers in your dataset. In other words, search

a database that records the papers that have cited your current set of WQ valuation

studies. I suspect you have probably caught most, if not all, papers – this is just a

suggestion for your consideration.

7. I thought this section was thorough and reasonable in its approach (I did not review each

row in Table B-20 to assess EPA’s coding of each individual study).

8. No comment.

9. I have several comments here.

36
a. I agree that sub_proportion could be better defined. I appreciated the EPA’s

justification for state, but it seems to miss some potentially important factors that

limit the extent of the market.

I would suggest considering “day-trip” distance (100 miles approximately)

instead of the state. Something larger than 100 miles could be used, but the state

seems a bad approximation in some instances (e.g., California – or even N.C., for

which a mountain lake is a poor substitute for a coastal inland lake).

b. I was unclear on how substitute lakes were defined. It appears that rivers are

matched based on stream order – are lakes similarly matched for comparability?

A large reservoir with public boat ramps is not the same as a closed-access

reservoir, or a small lake with zero boat access. Should lakes have clearly defined

‘comparable’ substitutes in the same way rivers do? (which brings up the subtle

point of using aggregate size as the measure of the amount of substitutes – do 10

small lakes really substitute for 1 big lake? It seems this approach works better

with diffuse resources, such as wadeable streams rather than large features).

c. EPA asks about using WQ as another factor that defines “substitute”. This is a

good question. My inclination would be to not use WQ as way to limit the

definition of a substitute. Substitutes can be “better” or “worse” – thus having

better or worse water quality – and be weak or strong substitutes. I still believe

that EPA could refine the substitute sets for lakes a bit more (capturing very large

differences in lakes as suggested in the point above), but your current means of

defining similar rivers and estuaries/coasts seems appropriate to me.

d. See also related comments in 5.

10. No comment.

Other Comments (not addressing specific questions asked by the EPA) 

1. I believe it would be very helpful to the reader in future versions to use TWTP and

MWTP to distinguish when you are speaking of total and marginal WTP, respectively.

2. I think your income variable needs more attention. It is an important variable for benefits

transfer and I have questions about the measure itself and your specification choice. How

many missing observations are there? Are the sample incomes similar to the census

variables you use? What’s the correlation? I wonder if you are introducing significant

measurement error. If classical, you would be biasing the coefficient toward zero.

However, if survey respondents have, on average, higher incomes and education than the

general population in the survey area, then the coefficient estimate would be biased up.

You can get some sense of this issue by comparing your census definition with the

sample incomes (means and/or medians) when available. If differences are substantial,

you could consider using the original income measures (means and/or medians) and use a

37
bias function to adjust the census-based income measure that is used to impute missing 

observations. For instance, if you find that sample income is on average 1.3xCensus 

Income, they you could impute income for missing observations as 1.3xCensus Income. 

3. In MRM1, I was surprised by the sign of swim_use and boat_use (and they are

significant) – I would have expected these to be positive. Is there a logical explanation

(or am I wrong to expect these to be positive)? Also, I find the coefficient on voluntary

to be surprising – again I would have expected this to be positive (if it is not incentive

compatible, the assumption is usually that hypothetical bias is the dominant result and

values would be biased up). Is there something else that this might be correlated with

that is not coded (i.e., is there an omitted variables problem relating to study

characteristics)?

4. Why not also weight MRM1 (by inverse of # observations drawn from each study)? In

future work, the EPA should apply consistent methods to each MRM (such as similar

weighting) to the extent possible. The document indicates that future work will do this, I

just wanted to underscore that I agree it should be done. The preferred MRMs upon

which benefits transfers are conducted need not match, but consistent approaches should

be reported so that the reader and the analyst can see the influence of these judgments.

5. I wonder about the way in which you include study year. Do we really expect that each

year leads to a certain percentage change in WTP, after controlling for inflation, income,

methods, etc. In fact, once controlling for all factors that are expected to influence WTP,

then year of the study should have no influence, correct? If it does have an influence, it

must be because of an important omitted variable correlated with year... what might that

be? Results indicate that later years are associated with lower WTP. Over time, has

publication pressures and/or standard practices led authors to estimate/report “more

conservative” values from stated preferences studies (again, after controlling for

economically important influences that vary with time)? Also, I’m curious what happens

if you allow more flexibility (i.e., include dummy variables for decade, for instance) – is

the relationship really monotonic? Maybe so, but it would be nice to check.

6. You have a variable that indicates if a WTP estimate comes from a thesis. Are thesis, by

definition, “not reviewed”? If so, then the effect of a thesis on WTP is = 0.6509 –

0.7419, correct? Is the combined value significantly different than zero?

7. When reading Section 3.3., the following question came to mind: did all studies elicit

WTP for an improvement in WQ? Or did some studies elicit WTP to avoid a decline in

WQ? If the latter is true, then this needs to be coded into the metadata, since property

rights are different and the WTP to get an increase in WQ is not expected to be the same

as WTP to avoid a decline. With choice experiments, this can be tricky to assess if the

levels of some attributes include both increases and decreases as compared to the baseline

(or status quo) option.

8. Do you have a sense whether or not the range of ar_ratio in the metadata is likely to

encompass policy applications? I’m wondering if the issues associated with predicting

38
out of sample might arise here too (or for ar_agr, or sub_proportion). 

9. In the BT application (page 43), your choices over covariate values led to several

questions. First, how is the variable “outlier” related to the variable CE? Do choice

experiments have “outliers” that they trim (there are studies which use subsets of choices,

but are these present in your data)? I ask, because I wonder about your choice of setting

Outlier = 1.

Second, on page 43, you indicate that set “nonparam” to zero because “older studies” just

report mean WTP across responses. Should these studies be contributing to estimation of

all coefficients (not just nonparam) if their methods are that suspect? Could you consider

dropping them?

The variables swim_use, gamefish, and boat_use are specified for each BG based on the

length-weighted average of baseline water quality & WQ changes for the affected

resources. Here is where my comment on the definition of river “affected areas” (= 2 x

length of river) is important. You may ultimately decide that your definition makes the

most sense, but it is important to make sure that it is truly the better measure as compared

to using just the length of the river (see comment 5b).

Lastly, I’m confused about the variable ln_ar_ratio on page 44. It seems to me that

you’d want the area of each BG divided by the area of resource within the buffer that is

improved. You use counties to create the “area of resource within the buffer”, but that

doesn’t make sense to me. It seems to me that you would, at a minimum, also intersect

county boundaries with the 100-mile radius and only compute area that falls within both

affected counties and the 100-mile buffer. (Maybe this is what you did?) Below is a

simple schematic of what I think you did:

The blue circle is a blockgroup, each black square is a county, blue lines are affected 

rivers, and the red circle is a 100-mile buffer around the blockgroup. 

39
I believe you used the area of all five counties (black boxes) as your denominator. 

However, it seems to me a better approximation is the intersection of the counties with 

the buffer. In the example above, there are four little “triangles” that lie within the circle 

but that are not within any of the affected counties. These areas would not be included in 

the denominator either. Only the area of the 100-mile buffer that intersects the affected 

counties ought to be included. 

10. As an aside, your prediction of WTP (from ln(WTP)) relies on normality of the error

term. As discussed in Wooldridge (Introductory Econometrics, 2012, page 213), there

are predictions of WTP that are not reliant on the assumption of normal errors that are

easily computed.

40