Peer Review Package for Meta-analysis of the Willingness-toPay for Water Quality Improvements
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 1

Table of Contents

1 Project Summary.............................................................................................................................. 2

2 Literature Review to Identify Additional Studies.......................................................................... 3

3 Variable Development and Coding................................................................................................. 7

3.1 Study Methodology and Year ..................................................................................................9

3.2 Region and Surveyed Populations .........................................................................................10

3.3 Reconciliation of Water Quality Baseline and Change..........................................................11

3.4 Delineation of the Affected Resource and Substitutes...........................................................13

4 MRM1 Estimation and Results..................................................................................................... 19

5 Robustness Tests............................................................................................................................. 22

5.1 Horizontal Robustness ...........................................................................................................25

5.2 Vertical Robustness................................................................................................................29

5.3 Summary and Findings from Horizontal and Vertical Robustness Tests ..............................38

5.4 Cross-Validation Test ............................................................................................................39

6 Benefit Transfer Application......................................................................................................... 40

6.1 Benefit Transfer Example ......................................................................................................41

6.2 Benefit Transfer Application Limits......................................................................................45

7 An Alternative Approach: MRM2................................................................................................ 48

7.1 Estimation and Model Selection Approach............................................................................48

7.2 Results....................................................................................................................................50

8 Model Limitations and Uncertainty.............................................................................................. 56

9 Example WTP Predictions for MRM1 and MRM2 .................................................................... 57

10 Overfitting and Step-wise Regression: MRM2 ............................................................................ 59

11 Next Steps........................................................................................................................................ 66

12 References ....................................................................................................................................... 66

Appendix A: Summary Statistics from the Metadata for Key Geospatial Variables......................... 74

Appendix B: Summary of Study-Specific WQL Translations.............................................................. 75

Appendix C: MRM2 with Observation 17 Excluded............................................................................. 85

Appendix D: MRM2 Bootstrapped Histograms of Step-wise Regression Coeffcent Estimates ........ 89
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 2

1 Project Summary

The U.S. Environmental Protection Agency (EPA) is re-examining and updating a meta-regression model 

(MRM) that relates household willingness-to-pay (WTP) to water quality improvements and various other 

independent variables. EPA began work on the original MRM in 2004, and the MRM was last updated in 

2009 as part of EPA’s benefits analysis for the Construction and Development Effluent Guidelines (U.S. 

EPA 2009).

1 The 2009 revisions included adding 11 studies (34 WTP observations) that were not 

included in the original metadata. The meta-data was published in 2009 as part of the rulemaking 

documentation for the effluent limitation guidelines and standards for Construction and Development 

(C&D) (U.S. EPA 2009). The 2009 revisions did not address the modeling approach (e.g., specification of 

independent variables and use of various functional forms). Household WTP for water quality 

improvements estimated based on the 2009 MRM have been considered in several Clean Water Act 

(CWA) rulemakings, including:

 Final Effluent limitation guidelines and standards for C&D (U.S. EPA 2009);

 Final water quality standards for Florida lakes and flowing waters and proposed water quality 

standards for Florida’s estuaries, coastal water, and South Florida inland flowing waters (U.S. 

EPA 2010; U.S. EPA 2012); and

 Proposed effluent limitation guidelines for Steam Electric power generating sources (U.S. EPA 

2013).

As the analysis has been used in several applications, it has become important to consider whether the 

MRM is broadly applicable or whether changes would enhance the breadth of appropriate application 

situations. For instance, the coefficients for single-river and multi-river variables in the 2009 MRM are an 

issue of particular concern to EPA. This is because the current model coefficients can appear 

counterintuitive by suggesting that total WTP is lower for multi-river improvements than single river 

improvements.2 Theory suggests that WTP estimates should be sensitive to geospatial factors including 

scale (the size of affected resources or areas), market extent (the size of the market area over which WTP 

is estimated) and the availability of substitutes. Neither the 2004 (2009) MRMs nor other published 

MRMs enable simultaneous adjustments for these factors, leading to potentially biased benefit transfer 

results. The current revisions include incorporation of additional studies in the metadata and the 

development of new variables for core economic factors including geospatial scale of the affected 

resources, size of the surveyed area, and substitute availability. By directly incorporating these geospatial 

factors into the MRM, EPA expects to enhance its content validity and improve the accuracy of benefit 

transfers.

This report provides detail on each step involved in the MRM update. Section 2 describes EPA’s 

literature review to identify additional studies included in the meta-data. Section 3 details development 

and coding of variables included in the revised WTP function (e.g., geographic area covered by the study, 

size of the affected aquatic resource, presence of substitute sites). The report presents two MRMs, 

referred to as MRM1 and MRM2, presented in Section 4 and Section 7, respectively. A key issue that 

pervades the potential use of MRM1 and MRM2 for benefit transfer is their suitability for predicting 

 

1 The 2004 MRM was also published as Johnston et al. (2005). 

2 EPA’s review of the affected resources in the single and multiple rivers studies suggests that variables for single and 

multiple rivers do not fully capture variation in spatial scale and resource significance across groups because the metadata 

for single rivers is dominated by large, regionally important rivers. This factor likely contributes to ambiguity in the 

interpretation of single and multiple river coefficients.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 3

WTP for separate regulations that are analyzed and implemented sequentially, in particular, regulations 

that generate small water quality improvements below the range of those observed in the metadata. This 

issue is highlighted through benefit transfer examples and discussion of uncertainties and limitations in 

Section 6. Section 5 presents detailed robustness testing results for MRM1 and Section 9 presents 

example WTP predictions using both MRM1 and MRM2. Section 10 discusses a step-wise regression 

approach and potential for over-fitting. Section 11 describes potential next steps. 

2 Literature Review to Identify Additional Studies

EPA used the 2009 meta-data as the starting point for the current revisions and conducted a literature 

review for additional studies to include in the metadata. The Agency followed Stanley et al. (2013)’s 

guidelines for meta-regression analyses in economics in documenting the literature search including (a) 

the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date 

completed. To identify new studies, EPA relied on the following:

 Searches of general literature databases and search engines (EBSCO, Google Scholar, Google). 

Search keywords were selected to be sufficiently broad so not to miss relevant studies:

o First terms: (1) water quality, (2) water clarity, (3) nutrient removal/improvement, (4) 

water quality index/ladder, (5) clean water, (6) water pollution reduction, (7) water 

habitat improvement, and (8) stream flow.

o Second terms: (1) willingness to pay, (2) stated preference, (3) contingent valuation, (4) 

choice experiments, (5) contingent activity, and (6) conjoint analysis

 Searches of online reference and abstract databases (Environmental Valuation Resource 

Inventory (EVRI), Benefits Use Valuation Database (BUVD), AgEcon Search, RePEc/IDEAs, 

and the Oregon State University College of Forestry Recreation Use Values Database);

 Visits to webpages of authors and university programs known to publish stated preference studies 

and/or water quality valuation research;3

 Searches of web sites for Resources for the Future and EPA’s National Center for Environmental 

Economics, both known to conduct environmental and resource economics valuation research;

 Searches of key resource economics journals for 2005 to 2013 (Land Economics, Environmental 

and Resource Economics, Marine Resource Economics, Journal of Environmental Economics and 

Management, Water Resources Research, and Ecological Economics). EPA focused these 

journal-specific searches on the recent period of 2005 to 2013 given the likelihood that older 

studies were captured under the steps above or captured during literature reviews conducted for 

prior versions of the MRM; and

 Reviewed bibliographies of other valuation meta-analyses (Van Houtven et al. 2007; Ge et al. 

2013; Brander and Brouwer 2011).

Studies identified during the search were screened according to the following criteria prior to inclusion in 

the metadata to ensure validity, consistency, and applicability. The identification and review of studies 

was completed and verified by multiple individuals following recommendations of Stanley et al. (2013):

 

3 This included G. Poe (Cornell), J. Bergstrom (University of Georgia), T. Haab (Ohio State), K. Viscusi (Vanderbilt), R. 

Carson (U.C. San Diego), W. H. Desvousges, J. Whitehead (Appalachian State), K. Boyle (Virginia Tech), R. Rosenberger 

(Oregon State), J. Loomis (Colorado State), J. Corrigan (Kenyon College), and R.H. von Haefen (North Carolina State0. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 4

 Commodity consistency – Study must value water quality changes affecting ecosystem services 

provided by waterbodies, including recreational activities (such as fishing, boating, and 

swimming), aquatic life support and other nonuse values.

4 EPA did not include studies that 

estimate WTP for improvements in surface waters used primarily for drinking water.

 Welfare consistency – The study must use general accepted stated preference approaches and 

report theoretically comparable Hicksian welfare measures (Boyle et al. 2013).

 Amenity detail – As described by Johnston et al. (2005), “the study must provide sufficient 

information regarding resource, context, and study attributes to warrant inclusion in the metadata” (p.223). 

 Study location – The study must be conducted in the U.S.

 Research methods – The study must apply research methods that are supported by the literature 

and that provide WTP estimates consistent with neoclassical welfare theory.

 Duplicative studies – Some studies may be released in multiple forms, such as working papers, 

conference papers, journal articles, and book chapters. Studies are screened to remove duplicative 

analysis and analysis that were subsequently revised and published. Multiple values can be 

included for a study if they address a different component of the study data or estimate values for 

alternative resource changes.

Table 1 summarizes studies in the revised metadata after identifying and screening additional studies. In 

total, the revised metadata includes 51 stated preference studies that estimated total WTP (use plus 

nonuse) per household for water quality changes in U.S. waterbodies. The studies address various 

waterbody types including, rivers, lakes, salt ponds/marshes, and estuaries. The fifteen studies added to 

the metadata during the current revisions are shaded in Table 1. Two of the new papers (Corrigan et al. 

2009; Whitehead 2006) replaced unpublished versions of the same studies (Azevedo et al. 2001; 

Whitehead et al. 2002). The revised metadata excludes two studies by Viscusi et al. (2008) and Carson

and Mitchell (1993). Carson and Mitchell (1993) estimate WTP for water quality improvements nationwide. It is excluded because its national scale means that the size of the affected resource area is 

substantially different from other studies in the meta-data, which at most, assess improvements within a 

region. On the other hand, Viscusi et al. (2008) asked respondents to value water quality improvements 

within 100 miles of their home. To be included in the metadata, a WTP observation must correspond to a 

specific affected resource or set of resources that can be delineated in GIS. Viscusi et al. (2008) does not 

fit into EPA’s data coding framework because the affected resource varies across the sample, with each 

respondent asked to think about their own area. The WTP estimates reported by Viscusi et al. (2008) 

reflect an average across many affected areas, rather than WTP for a specific affected resource. Many 

studies contribute multiple observations due to in-study variations in factors such as number or type 

affected waterbodies, magnitude of water quality improvement, sampled market area, and elicitation 

methods. The inclusion of multiple observations per study is standard in resource valuation metadata 

(Nelson and Kennedy 2009). In total, the 51 studies provide 140 WTP observations. 

 

4

For example, a study that estimates WTP for recreational fishing improvements due to fish stocking would not be included 

in the meta-data. However, a study that estimates WTP for recreational fishing improvements due to nutrient reductions 

would be included in the meta-data, presuming it satisfies all other criteria.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 5

Table 1. Primary Studies in the Metadata.

A,B,C,D

Author(s) and 

Publication Year

Obs. in 

Metadata State(s) Water Body Type(s)

Willingness to Pay per Household

(2007$)

Mean Min. Max.

Aiken (1985) 1 CO river and lake 193.18 193.18 193.18

Anderson and 

Edwards (1986)

1 RI salt pond/marshes 180.71 180.71 180.71

Banzhaf et al. (2006) 2 NY Lake 57.47 54.09 60.85

Banzhaf et al. (2011) 1 VA, WV, TN, 

NC, GA

river/stream 31.30 31.30 31.30

Bockstael et al (1988) 1 DC, MD, VA estuary 149.03 149.03 149.03

Bockstael et al. (1989) 2 MD estuary 158.30 75.67 240.93

Borisova et al. (2008) 3 WV, VA river/stream 44.94 18.05 65.82

Cameron and Huppert 

(1989)

1 CA estuary 49.53 49.53 49.53

Carson et al. (1994) 2 CA estuary 59.40 41.21 77.59

Clonts and Malone 

(1990)

3 AL river/stream 103.20 78.31 127.48

Collins and 

Rosenberger (2007)

1 WV river/stream 18.19 18.19 18.19

Collins et al. (2009) 7 WV river/stream 120.52 2.84 217.57

Corrigan et al. (2009) 1 IA Lake 123.30 123.30 123.30

Croke et al. (1987) 9 IL river/stream 77.47 61.31 93.68

De Zoysa (1995) 1 OH river/stream 70.18 70.18 70.18

Desvousges et al. 

(1987)

12 PA river/stream 59.19 19.84 137.26

Downstream 

Strategies (2008)

2 PA river/stream 12.74 10.70 14.77

Farber and Griner 

(2000)

6 PA river/stream 76.16 16.58 148.59

Hayes et al. (1992) 2 RI estuary 397.44 390.68 404.19

Herriges and Shogren

(1996)

2 IA Lake 134.55 61.71 207.40

Hite (2002) 2 MS river/stream 60.08 58.24 61.93

Huang et al. (1997) 2 NC estuary 258.65 255.01 262.29

Irvin et al. (2007) 4 OH all freshwater 21.67 19.65 23.23

Johnston et al. (1999) 1 RI river/stream 180.95 180.95 180.95

Kaoru (1993) 1 MA salt pond/marshes 218.61 218.61 218.61

Lant and Roberts 

(1990)

3 IA, IL river/stream 143.93 124.04 154.31

Lant and Tobin (1989) 9 IA, IL river/stream 55.63 40.58 67.64

Lichtkoppler and 

Blaine (1999)

1 OH river and lake 41.93 41.93 41.93

Lindsey (1994) 8 MD estuary 66.80 33.40 102.20

Lipton (2004) 1 MD estuary 63.98 63.98 63.98

Londoño Cadavid and 

Ando (2013)

2 IL river/stream 38.68 35.93 41.44

Loomis (1996) 1 WA river/stream 93.07 93.07 93.07

Lyke (1993) 2 WI river and lake 78.75 59.75 97.74

Matthews et al. (1999) 2 MN river/stream 21.73 18.14 25.32

Opaluch et al. (1998) 1 NY estuary 138.47 138.47 138.47

Roberts and Leitch 

(1997)

1 MN, SD Lake 8.35 8.35 8.35

Rowe et al. (1985) 1 CO river/stream 134.59 134.59 134.59

Sanders et al. (1990) 4 CO river/stream 160.69 81.01 210.04
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 6

Table 1. Primary Studies in the Metadata.

A,B,C,D

Author(s) and 

Publication Year

Obs. in 

Metadata State(s) Water Body Type(s)

Willingness to Pay per Household

(2007$)

Mean Min. Max.

Schulze et al. (1995) 2 MT river/stream 20.84 17.34 24.33

Shrestha and 

Alavalapati (2004)

2 FL river and lake 156.46 137.97 174.95

Stumborg et al. (2001) 2 WI Lake 84.29 66.73 101.86

Sutherland and Walsh 

(1985)

1 MT river and lake 146.03 146.03 146.03

Takatsuka (2004) 4 TN river/stream 286.88 181.90 391.85

Wattage (1993) 3 IA river/stream 53.89 40.24 74.59

Welle (1986) 6 MN Lake 167.28 109.60 238.42

Welle and Hodgson 

(2011)

3 MN Lake 145.10 10.59 285.06

Wey (1990) 2 RI salt pond/marshes 147.26 63.95 230.58

Whitehead and 

Groothuis (1992)

3 NC river/stream 41.01 31.90 53.16

Whitehead (2006) 3 NC river/stream 187.18 27.52 365.54

Whitehead et al. 

(1995)

2 NC estuary 95.44 78.29 112.59

Whittington et al. 

(1994)

1 TX estuary 194.72 194.72 194.72

Notes:

(A):Shading indicates studies added since the 2009 meta-analysis. 

(B): Journal publications by Corrigan et al. (2009) and Whitehead (2006) replaced staff papers authored by 

Azevedo et al. (2001) and Whitehead (2002), respectively.

(C):A study by Olsen et al. (1991) from the 2009 meta-analysis was excluded from the current metadata following 

robustness testing and additional review of study details.

(D):The revised metadata excludes studies by Viscusi et al. (2008) and Carson and Mitchell (1993) that surveyed 

households across the U.S.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 7

3 Variable Development and Coding

EPA entered values for all existing variables in the metadata for each new study identified during the 

literature review. Next, EPA developed and coded new variables (for all studies) that address gaps in the 

2009 metadata. The variables in the revised MRM fall into four general categories:

(1) Study methodology and year variables characterize such features as the year in which a study was 

conducted, payment vehicle and elicitation formats, WTP estimation method, and publication 

type. These variables are included to explain differences in WTP across studies but are not 

expected to vary across benefit transfer for different policy applications. 

(2) Region and surveyed populations variables characterize such features as the US region in which 

the study was conducted, the average income of respondent households and the representation of 

users and nonusers within the survey sample.

(3) Sampled market and affected resource variables characterize features such as the geospatial scale 

(or size) of affected water bodies, the size of the market area over which populations were 

sampled, as well as land cover and the quantity of substitute water bodies.

(4) Water quality (baseline and change) variables characterize baseline conditions and the extent of 

the water quality change.

Table 2 presents and defines key variables from the metadata by category. Shading indicates that the 

variable was developed during these revisions. The development of new variables focused primarily on 

developing variables that capture geospatial factors including extent of sampled market, surveyed 

populations and affected resources.5 The 2009 MRM largely relied on binary variables to distinguish 

broad categories of affected resources (e.g., single vs multiple rivers, regional freshwater, etc.). Extensive 

GIS mapping is required in order to quantitatively incorporate new spatial factors using continuous 

variables because many primary studies give broad outlines but omit the detailed geospatial information 

regarding affected resources and sampled populations. This type of supplementation has been used by 

others to include various types of information into valuation MRMs other than that necessary for the 

current metadata (e.g., Ghermandi et al. 2010; Ghermandi and Nunes 2013). By linking to publicly 

available GIS data layers, EPA provides a consistent basis for the construction and coding of the new, 

spatial variables. The following subsections describe both development and coding of water quality and 

spatial variables for surveyed populations, extent of sampled market, study site, and affected resources. In

accordance with Stanley et al. (2013) guidelines, all variable coding was reviewed by multiple individuals 

and documented clearly in the metadata and data dictionary.

Annual WTP values are typically desired for policy analysis to support a comparison of annualized costs 

and benefits. The metadata includes total WTP as reported by the original study, adjusted to 2007$. For 

over 80 percent of observations (114 studies), total WTP reflects annual payments in perpetuity (113 

studies) and one observation reflects annual WTP for a 10-year payment period. For the remaining 26 

observations, WTP reflects a short-term payment period (11 observations use a one-time lump sum 

payment, 1 observation uses a 3-year payment period, and 14 observations use a 5-year payment period). 

EPA used the lump_sum binary variable to account for short-term payment periods (lump_sum=1). EPA 

also tested models with WTP values for short-term payment periods converted to perpetual streams but 

found that results were sensitive to discount rate assumptions and that these models did not provide any 

practical advantages for benefit transfer over models using the lump_sum variable.

 

5 EPA also coded one new methodological variable, ce, to identify studies that used a choice experiment approach.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 8

Table 2. Definition and Summary Statistics for Model Variables.

A

Variable Definition Units Mean St. Dev.

Study Methodology and Year

ce Binary variable indicating that the study is a choice 

experiment

Binary 

(Range: 0 or 1)

0.1071 0.3104

thesis Binary variable indicating that the study is a thesis. Binary 

(Range: 0 or 1)

0.1143 0.3193

lnyear Natural log of the year in which the study was conducted

(i.e., data was collected), converted to an index by 

subtracting 1980.

Natural log of 

years (year 

ranges from 

1981 to 2011).

2.2127 0.9282

volunt Binary variable indicating that WTP was estimated using a 

payment vehicle described as voluntary as opposed to, for 

example, property taxes. 

Binary 

(Range: 0 or 1)

0.0857 0.2809

outliers_trim Binary variable indicating that outlier bids were excluded 

when estimating WTP. 

Binary 

(Range: 0 or 1)

0.1929 0.3960

nonparam Binary variable indicating that WTP was estimated using 

non-parametric methods.

Binary 

(Range: 0 or 1)

0.4286 0.4966

non_reviewed Binary variable indicating that the study was not published in 

a peer-reviewed journal. 

Binary 

(Range: 0 or 1)

0.2357 0.4260

lump_sum Binary variable indicating that the study provided WTP as a 

one-time, lump sum or provided annual WTP values for a 

payment period of five years or less.. This variable enables 

the benefit transfer analyst to estimate annual WTP values 

by setting lump_sum=0. 

Binary 

(Range: 0 or 1)

0.1857 0.3903

wtp_median Binary variable indicating that the WTP measure from the 

study is the median.

Binary 

(Range: 0 or 1)

0.0714 0.2585

Region and Surveyed Populations

northeast Binary variable indicating that the survey included 

respondents from states within the Northeast U.S., defined 

as ME, NH, VT, MA, RI, CT, and NY. This is equivalent to 

EPA Region 1 plus NY or the Northeast USDA region. 

Binary 

(Range: 0 or 1)

0.0714 0.2585

central Binary variable indicating that the survey included 

respondents from states within the Central U.S., defined as 

OH, MI, IN, IL, WI, MN, IA, MO, ND, SD, NE, KS, MT, WY, 

UT, and CO. This is equivalent to EPA regions 5, 7, and 8 or 

the Midwest and Mountain Plains USDA regions. 

Binary 

(Range: 0 or 1)

0.3643 0.4830

southB,C Binary variable indicating that the survey included 

respondents from states within the Southern U.S, defined as 

NC, SC, GA, FL, KY, TN, MS, AL, AR, LA, OK, TX, and NM.

This is equivalent EPA regions 4 and 6, or the Southeast 

and Southwest USDA regions.

Binary 

(Range: 0 or 1)

0.1571 0.3652

nonusers Binary variable indicating that the survey was implemented 

over a population of recreational nonusers (default category 

for this variable is a survey of any population that includes

both users and nonusers). 

Binary 

(Range: 0 or 1)

0.0857 0.2809

lnincome Natural log of the median income (in 2007$) for the sample 

area of each study based on historical U.S. Census data. It 

was designed to provide a consistent income variable given 

differences in reporting of respondent income across studies 

in the metadata (i.e., mean vs. median). Also, some studies 

do not report respondent income. This variable was 

estimated for all studies in the metadata regardless of 

whether the study reported summary statistics for 

respondent income.

Natural log of 

income (2007$)

(sample area 

income ranges 

from $34,332 to 

$78,444)

10.7453 0.1731

Sampled Market and Affected Resource

mult_bod Binary variable indicating that the survey addressed multiple 

waterbody types. The eight waterbody type categories are 

(1) river/stream, (2) lake, (3) all freshwater, (4) estuary, (5) 

wetlands, (6) river and lake, (7) salt pond/marshes, and (8) 

multiple (estuary and fresh water). Takes on a value of 1 if 

the study is coded as category (3), (6), or (8). 

Binary 

(Range: 0 or 1)

0.0786 0.2700

river Binary variable that takes on a value of 1 if the study affects 

a river, such that river length>0, and zero otherwise.

Binary 

(Range: 0 or 1)

0.6857 0.4659

swim_use Binary variable indicating that the affected use(s) stated in 

the study include swimming.

Binary 

(Range: 0 or 1)

0.2643 0.4425

gamefish Binary variable indicating that the affected use stated in the Binary 0.0571 0.2329
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 9

Table 2. Definition and Summary Statistics for Model Variables.

A

Variable Definition Units Mean St. Dev.

study is gamefishing. (Range: 0 or 1)

boat_use Binary variable indicating that the affected use(s) stated in 

the study includes boating.

Binary 

(Range: 0 or 1)

0.1143 0.3193

ln_ar_agr Natural log of the proportion of the affected resource area 

which is agricultural based on NLCD, reflecting the nature of 

development in the area surrounding the resource. The 

affected resource area is defined as all counties that 

intersect the affected resource(s). EPA also tested a 

variable for the fraction of lands that is developed land 

classes, but it did not improve model fit. 

Natural log of 

proportion 

(Range: 0 to 1)

-1.4329 0.9031

ln_ar_ratio A ratio of the sampled area, in km2, relative to the affected 

resource area (defined as the total area of counties that 

intersect the affected resource(s), in km2). The total area of 

counties that intersect the affected resource(s) is given by 

the variable ar_total_area. From here, ln_ar_ratio = 

log(sa_area / ar_total_area), where sa_area is the size of 

the sampled area in km2.

Natural log of 

ratio (km2/km2)

-1.1278 2.6067

sub_proportion The proportion of water bodies of the same hydrological type 

affected by the water quality change, within affected state(s). 

For rivers, this is measured as the length of the affected 

river reaches as a proportion of all reaches of the same 

order. For lakes and ponds, this is defined as the area of 

the affected water body as a proportion of all water bodies of 

the same National Hydrography Dataset classification. For 

bays and estuaries, this is defined as the shoreline length of 

the water body as a proportion of all analogous (e.g., 

coastal) shoreline lengths. To account for observations 

where multiple waterbody types are affected, the variable 

sub_proportion is defined as maximum of separate 

substitute proportions for rivers, lakes, and estuaries/bays.

The affected resource appears in both the numerator and 

denominator when calculating sub_proportion.

Proportion 

(Range: 0 to 1) 

(km/km)

0.1880 0.2911

Water Quality Baseline and Change

lnquality_ch Natural log of the change in mean water quality (quality_ch), 

specified on the WQI (McClelland 1974; Mitchell and Carson 

1989). 

WQI units 2.9070 0.6039

lnbase Natural log of baseline water quality, specified on the WQI 

(McClelland 1974; Mitchell and Carson 1989).

WQI units 3.5889 0.6697

Notes:

(A): Shading indicates that the variable was developed and coded during the current set of revisions.

(B): EPA merged studies from the Southeast and Southwest region into a single group because the metadata includes only one 

observation from the Southwest region, Whittington et al. (1994) which studied estuaries in Texas

(C): The USDA regions omitted from the regional binary variables are the Mid-Atlantic region (NJ, DE, MD, DC, PA, WV, and VA) 

and the Western region (WA, OR, ID, CA, NV, AZ, and AK).

3.1 Study Methodology and Year

As often found in meta-analyses within the valuation and benefit transfer literature (Navrud and Ready

2007), a variety of study and methodology effects can be shown to influence WTP for water quality 

improvements. The 2009 metadata included numerous variables characterizing factors including, but not 

limited to publication type, elicitation method, response rate, treatment of outliers, payment period, and 

study year. EPA coded values for these variables for the new primary studies following the variable 

definitions from the 2009 MRM. The majority of the methodological variables in the metadata are binary, 

with 1 indicating that a given methodology was used in the primary study. Some others, such as study 

year, are continuous. Most of the methodological variables available in the metadata did not improve 

model fit and were not included in the revised MRM. Given that choice experiments are being used with 

increased frequency in the recent literature, EPA coded a new binary variable, ce, to flag observations that 

are based on a choice experiment framework. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 10

3.2 Region and Surveyed Populations

The 2009 metadata included variables describing the location of the study by state and region of the 

country although these are used in a fairly simplistic way in the meta-analysis. EPA expanded the 

metadata by delineating sampled area for each of the 140 WTP observations as GIS polygons. The

sampled area is defined as the geographic area over which the primary survey was fielded (i.e., the 

population its meant to be representative of). The Agency based its GIS delineation on the description of 

the study area from the study documentation and matched to polygon boundaries from publicly available 

GIS datasets. The sampled area can vary across observations from the same primary study if the WTP 

values are based on separate survey samples (this is true for 5 of 51).

The sampled areas are typically defined by either jurisdictional boundaries or watershed boundaries:

 For sampled areas defined by jurisdictional boundaries (e.g., states, counties, or cities), EPA 

defined boundaries using the Census Topologically Integrated Geographic Encoding and 

Referencing (TIGER) state and county shapefiles for 1990, 2000, and 2010. By using the 

shapefile vintage that most closely corresponds to the year in which the study was fielded, EPA is 

able to accurately identify boundaries at the time of the study.

 For sampled areas defined by watersheds, EPA approximated sampled area boundaries by 

matching to the U.S. Geologic Survey Hydrologic Unit Code (HUC) Watershed Boundary 

Dataset (WBD) to approximate the boundaries of the study area.6 The Agency first matched the 

watershed polygon in the HUC dataset based on the watershed name presented in the study. If the 

name could not be matched, EPA consulted the survey materials or maps included in the study to 

either locate the corresponding HUC watershed or use the materials to trace a watershed polygon 

manually. If maps are not provided, then EPA used any additional locational information 

presented in the study to identify a HUC watershed to serve as a proxy for the study watershed. 

This approach generally involved more interpretation than if the study defined the sampled area 

based on jurisdictional boundaries (as in previous bullet).7

Income is inconsistently reported across primary studies, with some reporting median income, others 

reporting mean income, and others not reporting any income statistics for the sample. After delineating 

the sample area for all studies, EPA calculated income statistics for the population within the sampled 

area using georeferenced historic U.S. Census income data to overcome reporting deficiencies and 

improve consistency across observations. Historical median income data is available for 1980, 1990, 

2000, and 2010. Each sampled area was matched to the data vintage that is nearest to the year in which 

the study was fielded. EPA derived sampled area income as a population-weighted average of median 

income across counties that intersect the sampled are polygons. The population weights were adjusted 

based on the fraction of county area that is within the sample area. Resulting income estimates were 

adjusted to 2007$ using the Consumer Price Index (CPI).

 

6 The HUC WBD is a package of watershed shapefiles for the contiguous U.S. subdivided to seven levels of resolution. Each 

resolution level is assigned a specific number of HUC digits, with the number of digits increasing with higher resolutions. 

The lowest resolution is the HUC 2-digit, or HUC-2, watershed which divide the country into 21 distinct drainage regions. 

The highest classification is the HUC12 dataset, consisting of approximately 98,000 watersheds. Additional information 

about the HUC watershed and delineation process can be found at http://water.usgs.gov/GIS/huc.html.

7 EPA encourages authors of future studies to describe the sampled area clearly, to avoid the need for judgment or 

interpretation when including studies in future meta-analyses.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 11

The 2009 metadata included regional dummies for the study location based on the boundaries of U.S. 

Department of Agriculture (USDA) regions.8 The USDA regions were used as the starting point for 

regional assignments. The boundaries of the seven USDA regions are closely aligned with the ten EPA 

regions, EPA generated two new binary variables that combine some regions for which effects were 

found to be similar. Northeast identifies studies that sampled respondents in the Northeast U.S. (as far 

south as New York, central identifies studies that sampled in the Midwest or Mountain Plains regions of 

the U.S, and south identifies those that sampled in the Southeast and Southwest U.S.(as far west as New 

Mexico). The state boundaries for each variable are listed in Table 2. 

EPA also calculated the continuous variables sa_area, defined as the size of the area sample in square 

kilometers. Sa_area feeds into the index variable, ln_ar_ratio, described in detail in the following 

subsection. 

3.3 Reconciliation of Water Quality Baseline and Change

An important component metadata development is the reconciliation of variables across observations 

(Johnston et al. 2005; Smith and Pattanayak 2002; Smith et al. 2002; Van Houtven et al. 2007). Although 

the calculation and reconciliation of most independent variables requires little explanation, there are some 

variables for which additional detail is warranted. These include variables characterizing surface water 

quality and its measurement. To reconcile measures of water quality across studies, EPA adapted the prior 

approach of Johnston et al. (2005), mapping water quality changes to the 100-point Water Quality Index 

(WQI).9

A large number of the original studies in the metadata (10 studies and 31 percent of observations) include 

the 10-point water quality ladder (WQL) measures as a native component of the original primary studies

either using its numerical values (e.g., 2.5, 5.0, etc.) or descriptive levels (e.g., boatable, gamefishing).

10

 

In other cases, EPA used descriptive information from studies to map baseline and post-improvement to 

points on the WQL. To calculate 100-point WQI values EPA multiplied the reported WQL values by ten. 

In most cases, descriptions correspond closely to levels on WQL, rendering mapping straightforward. 

EPA used the following guidelines for mapping:

 WQL assignments are made based on consideration of information provided in the primary study 

documentation and, if available, the survey materials. The assignments are not supplemented by 

external water quality databases or reports. They should reflect the descriptions and metrics 

presented to respondents within the valuation scenarios.

 Identify recreational uses (e.g., boating, fishing, and swimming) provided in the description of 

baseline and improved (or declined) conditions. These uses may be stated directly or embedded 

in the definition of water quality metrics (e.g., low, moderate, high) and can include additional 

descriptive information such as effects on sensitive aquatic species or indication of presence of or 

amount of specific pollutants.

 

8 Using ecoregion boundaries to define regional dummies is not feasible because it would result in a very small number of 

observations per region. Each USDA region includes several ecoregions and thus are better suited for grouping studies. 

9 Van Houtven et al. (2007) also mapped water quality changes to the WQL.

10 The WQL (Vaughan 1986) is expressed on a scale of 0 to 10 and can be mapped to the WQI by multiplying by 10. Refer to 

Chapter 10 and Appendix G of the benefits analysis for the C&D Effluent Guidelines (U.S. EPA 2009) for additional detail 

on the WQI and WQL.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 12

 Only consider those metrics that can be reasonably mapped to the WQL. Ancillary improvements 

from the improvement plans, such as shoreline trash pickup or terrestrial bird species, are not 

considered.

 Exclude observations that describe an improvement in recreational use that is not tied to a water 

quality improvement (e.g., an increase in fish abundance not based on water quality 

improvement).

 Start by assigning values corresponding to use thresholds defined on the WQL (e.g., 2.5, 4.5). 

Assign intermediate values when the changes occur within a specific use category or changes 

extend beyond the minimum for provision of use. EPA notes that the majority of assignments 

directly match values corresponding to WQL use thresholds. 

 To the extent possible, assigned WQL values should reflect the “affected” portion of resource(s) 

described in the survey, that is, the portion of the resource that is subject to water quality changes 

under the valuation scenario.

 In some cases, uses may be supported intermittently due to algae blooms, for example. If 

provided, use information regarding the frequency of service provision to calculate intermediate 

WQL values by weighting use threshold values.

 For some types of environmental contamination, such as high acidity, the survey may state that 

the affected resource supports swimming but has degraded fisheries. It supports a higher use on 

the WQL (swimming), but not a lower use (gamefishing). . In these cases, we specify baseline 

WQL and WQL changes based on fishing conditions, the use that is actually improved under the 

valuation scenario (e.g., a baseline of 4.5 for rough fishing and a post-improvement value of 5.0

for game fishing).

Appendix B presents water quality description from the original studies and the assigned baseline and 

improved water quality conditions for each study. The metadata include two sets of baseline and postimprovement WQL values, one reflecting a “low range” and another reflecting a “high range.” In cases 

where coding was straightforward, these ranges are identical. In other cases, the ranges may differ, 

reflecting uncertainty regarding uses supported under baseline or post-compliance conditions. For 

example, a study might include a low range of 2.5 to 4.5 and high range of 2.5 to 5.0 if it is unclear 

whether rough fishing or game fishing is supported under improved conditions. The water quality change 

variables in the MRMs are based on the high range consistent with the 2009 MRM because of slightly 

better model fit. 

EPA did not identify any systematic variation in results associated with studies for which the WQI was a 

native component, versus those for which quality changes were mapped to the WQI. A binary variable 

indicating that the observation used the WQL (CHNG_WQL) was not individually significant and did not 

improve model fit. Figure 1 and Figure 2 illustrate baseline WQI values and WQI changes from the 

metadata. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 13

Figure 1. Histogram of Baseline WQI Values from the Metadata (Percent of Studies).

Figure 2. Histogram of WQI Changes from the Metadata (Percent of Studies).

3.4 Delineation of the Affected Resource and Substitutes

WTP for WQI changes is likely better explained in concert with geospatial factors, given the geographic 

heterogeneity in the metadata. In the context of water quality valuation, the geospatial scale of a water 

quality change reflects the area over which the environmental change occurs. EPA expects, all else being

equal, that WTP for water quality improvements would be directly related to the size and number of 

affected waterbodies. EPA defines an affected resource as a waterbody that experience a water quality 

change under the valuation scenarios presented in the primary study. The number of affected waterbodies 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 14

assessed in the primary studies varies widely across the metadata from single waterbodies to all 

waterbodies throughout a state or small region. The existing metadata identifies the geographic scale of 

the affected resources using four categories: (1) single waterbody, (2) multiple waterbodies, (3) a small 

region, or (4) a large region. Table 3 summarizes the number and percent of observations in each of the 

four geographic scale categories. As part of the revisions, EPA delineated the affected waterbodies by 

matching waterbody descriptions and maps from the primary studies to publicly available georeferenced 

datasets. By delineating resources using GIS, EPA is able to incorporate detailed spatial and hydrological 

features of the waterbodies. As with sampled area, some primary studies provide separate WTP values for 

multiple affected resources or sets of affected resources. These are always treated as separate observations 

within the metadata. 

Table 3. Summary of Scale of Resource Changes in the Meta-data.

Category Variable Description Observation 

Count

% of 

Observations

Single 

Waterbody Single Single, discrete waterbody such as a river or lake. 49 35.0%

Multiple 

Waterbodies Multi Multiple discrete waterbodies (e.g., 2 rivers) 13 9.3%

Small Region Sm_reg Small region, includes watersheds and 

state-wide analyses 77 55.0%

Large Region Lg_reg Large region, includes one estuary 

analysis that includes multiple states. 1 0.7%

Total - - 140 100.0%

The datasets used to map affected resources vary by affected waterbody type. For rivers and lakes, the 

Agency opted to match to the NHDPlus dataset over other publicly available georeferenced hydrologic 

datasets (e.g., E2RF1) because it is a comprehensive, national dataset of hydrologic features such as 

rivers, lakes, ponds and catchments, and contains tabular data that can be linked to these hydrologic 

features. For estuaries and coastlines, EPA used the National Oceanic and Atmospheric Administration

(NOAA) Global Self-Consistent Hierarchical, High-Resolution Geography Database (GSHHD). EPA 

attempted to match the name of the affected resource to the datasets, and if a match could not be found, 

used narrative descriptions and maps from the primary study to manually select the affected waterbodies 

in ArcGIS. Once delineated, EPA calculated affected shoreline length for all waterbody types as proxy for 

the human/water interface, although some uses actually occur in open water. For rivers, shoreline length 

is double the reach length to reflect both shorelines. EPA also calculated the area of affected lakes.

Observed mean WTP in a primary study is also related to the extent of market analyzed (Loomis 2000; 

Loomis and Rosenberger 2006). It is established in economic literature that mean WTP often declines as 

distance increases to an affected resource (i.e., distance decay) (Bateman et al. 2006; Jørgensen et al. 

2013; Schaafsma et al. 2012). Because the scale of the sampled market area varies greatly across primary

studies and past work has shown that such differences can have important implications for welfare 

estimates, the accuracy of benefit transfers depends on the ability to adjust for the distance between 

affected populations and the resource (Bateman et al. 2006; Johnston and Duke 2009; Johnston and 

Ramachandran 2013; Loomis and Rosenberger 2006). The size of the sampled area (sa_area) in the 

metadata varies greatly across the metadata mean of 49,065 mi2 with 5th and 95th percentiles of 13.3 and 

104,094 mi2, respectively. Sa_area is not correlated with the size of the affected resource because various 

factors affect researcher’s sample selection including budget and resource constraints and study 

objectives. As a result, the metadata does not provide clear information regarding the extent of the market 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 15

for the affected resources or distance effects. Figure 3 and Figure 4 show sampled areas and affected 

resources for four examples from the metadata. The examples were selected to show a range of resource 

types, sizes, and sampled areas. 

Preliminary modeling indicated that model performance was enhanced, in terms of model fit, variable 

significance, and consistency with theoretical expectations, when resource size was specified as a

function of sampled market area (sa_area). The ln_ar_ratio variable reflects the size of the sampled 

market relative to the size of the affected resource, defined as the (natural log of the) size of the sampled 

market area (sa_area) divided by the geographic area encompassing the affected waters in square 

kilometers (ar_total_area). Ar_total_area is calculated based on the area of counties that insect the 

affected resource. For example, if a study valued all freshwater in Iowa, then ar_total_area would equal

the area of Iowa. On other hand, ar_total_area would equal the area of a single Iowa county if the 

affected resource falls entirely within the county boundary. The area ratio, ln_ar_ratio, is expected to 

have negative sign reflecting two effects. First, holding all else constant, stated preference surveys over 

larger market areas imply greater distances between individual households and affected waterbodies, and 

thus lower WTP per household. Secondly, improvements to more waters within the sampled area should 

be associated with greater WTP, ceteris paribus.

About 55 percent of observations in the metadata estimated WTP for water quality changes in a single 

watershed, group of watersheds, or a state. The other 45 percent focused on a single waterbody or small 

set of waterbodies (e.g., one lake or 3 lakes). Specifying the affected resource area requires some 

assumptions if the affected resource is defined as a river or lake (e.g., EPA assumed that counties 

intersecting the affected water bodies represent the affected resource area). As noted above, EPA used 

intersecting counties to define all affected resources, including discrete rivers, as geographic areas while 

avoiding the arbitrary selection of a boundary distance.

Relative to the alternatives, ln_ar_ratio provides a more intuitive means to capture the scale of the 

affected resource occurring throughout an area, such as watershed or region which is likely to be the case 

in the context of national rulemakings. Model fit decreases when the affected area ratio is included in 

non-logged form and the variable (ar_ratio) is no longer individually significant. Logging the ratio is also 

intuitive because WTP is zero when the resource size is zero. EPA considered, but did not select, a variety 

of alternative specifications of the geographic scale and extent of market variables (e.g., shoreline length).

The availability and quality of substitutes in the surrounding geographic area are also expected to 

influence welfare estimates (Loomis and Rosenberger 2006; Schaafsma et al. 2012). All else equal, WTP 

should be negatively related to substitute availability. EPA developed a continuous variable, 

sub_proportion, defined as the ratio of affected waterbodies to available use and nonuse substitutes within 

the state. EPA expects that the relationship between distance and substitutes will differ for resource users

and nonusers. For users, there is “a positive cost of access which depends on distance” (Hanley et al. 

2003, p.300), therefore, sites that are farther away will generally be less attractive substitutes all other 

things being equal. On the other hand, “there seems no theoretical reason to expect such a [distance 

decay] trend to be seen within the responses of those who are present non-users” (Bateman et al. 2006, p. 

453). There is some evidence that a sense of spatial and cultural identity or “ownership” may be important 

for nonuse values for some environmental resources (Bateman et al. 2006; Hanley et al. 2003). Hanley et 

al. (2003) provides the example that “I may have stronger non-use values for Scottish wildlife sites if I am 

Scottish than for English wildlife sites” (p.300). In the U.S., this may be manifested as affiliation with 

one’s state of residence. Given these factors, EPA considers the state to be a reasonable basis for the 

calculating the substitute proportion.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 16

The substitute proportion variable (sub_proportion) represents a potential advance over less sophisticated 

binary variables traditionally used to represent the size or scope of affected resources. For studies 

restricted to rivers, the calculation of substitutes is restricted to rivers of the same stream order(s) as the 

affected resource to ensure comparability. For non-river inland waterbodies (e.g., lakes and ponds), 

substitute proportion is calculated based on area relative to area of lakes within the state(s).11 For estuaries 

and coasts, EPA calculated the denominator as all GSHHD coastline miles in the state(s). EPA also tested 

models with separate substitute variables for each waterbody type, but model performance was not 

improved. 

By delineating an affected area, in addition to the waterbodies themselves, EPA is also able to analyze the 

development characteristics of areas bordering the waterbodies. EPA used the National Land Cover 

Dataset (NLCD) to develop a land cover profile of the affected resource area. The NLCD is a highresolution (30 meter) spatial dataset of land cover across the contiguous U.S. and available for the years 

1992, 2001 and 2006. The NLCD has 16 or 21 categories of land cover depending on the year. For the 

purpose of consistency and simplicity, EPA defined the agricultural land as the sum of “cultivated 

cropland” and “pasture/hay”. The Agency used the change in agricultural land between NLCD vintages to 

interpolate values for the actual year of the study. The agricultural fraction is calculated as the areaweighted average of counties within the affected resource area. EPA also developed and tested an 

analogous variable based on the fraction of land that is in developed land use categories but it not improve 

model fit. 

 

11 NHDPlus does not provide widths of river and streams. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 17

Figure 3. Example Sampled Areas and Affected Resources from Aiken (1985) and 

Hayes et al. (1992). Sampled areas are highlighted in orange and affected resources are 

highlighted in blue. Aiken (1985) surveyed residents in and around Fort Collins, Colorado 

regarding state-wide improvements to rivers and lakes. Hayes et al. (1992) surveyed Rhode 

Island residents regarding improvements to upper Narragansett Bay. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 18

Figure 4. Example Sampled Areas and Affected Resources from Matthews et al. (1999) and Lant 

and Tobin (1989). Sampled areas are highlighted in orange and affected resources are highlighted in 

blue. Matthews et al (1999) surveyed residents of nearby counties regarding improvements to the 

Minnesota River. Lant and Tobin (1989) surveyed residents of Ames City, Iowa (location shown in insert)

regarding the improvements to the South Skunk River Basin. Lant and Tobin (1989) also surveyed 

residents of other cities regarding other affected resources. They were included as separate observations 

and are not pictured here. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 19

4 MRM1 Estimation and Results

Following established practices, EPA estimated MRM1 as a standard multi-level model (Bateman 

and Jones 2003; Johnston et al. 2005) of the type recommended within the literature (Nelson and 

Kennedy 2009; Johnston and Rosenberger 2010). As described in greater detail below, the model is 

specified as a trans-log model. The model allows for cross-sectional panel correlation among 

observations from the same study. If left unaddressed, such correlation could lead to heteroskedastic 

errors and inefficient, inconsistent parameter estimates (Rosenberger and Loomis 2000). For each 

empirical study in the metadata, a central tendency measure (mean or median) of WTP for water 

quality improvements for the representative individual is given by WTPjs, the natural log of which is 

the measured effect size:

ln WTPjs = ln(Djs) βD + ln(Q0js)βQ0 + ln(Yjs) βY + X1jsβX1 + ln(X2js)βX2 + εjs [1]

Here, WTPjs is average (or median) WTP for observation j from study s, Djs =Q1js – Q0jsis the water 

quality change calculated as the difference between starting and ending water quality values, Q0js is 

the baseline water quality level, and Yjs is the household income for the sampled population. X1js and 

X2js are vectors of variables measuring environmental conditions and change, research methods, 

temporal factors, and other factors expected to influence value surfaces across observations and/or 

sites. The vectors βD, βQ0, βY, βX1, and βX2 represent conforming vectors of parameters to be 

estimated. 

To allow for potential effects of study-specific unobservable factors, EPA partitioned εjs into two 

components such that

εjs = us + ejs [2]

Here, us represents a systematic, normally distributed, study-level random effect with E(us) = 0 and 

Var(us) = σu

2

, and ejs is a standard iid estimation level error, distributed with a zero mean and 

constant variance σe

2

(Shrestha and Loomis 2001). Clustering by study to account for within-study 

correlation is standard practice within MRMs in the benefit transfer literature (Nelson and Kennedy

2009). Other aspects of the econometric model follow standard conventions; EPA estimated MRM1 

as a random effects, generalized least squares model with robust standard errors (Nelson and 

Kennedy 2009).12 Note that this specification implicitly assumes that the WTP estimates reported in 

the primary studies represent the true underlying WTP in the study population value rather than an 

estimate (i.e., there is no sampling uncertainty).

Independent variables included in the meta-analysis are derived from a list of attributes with potential to 

influence WTP, based on theory and prior findings in the literature. Hence, MRM1 is best characterized 

as being a weak structural utility theoretic meta-analysis (Bergstrom and Taylor 2006). When specifying 

MRM1, EPA emphasized core scope and scale variables relevant to benefit transfer. Care was taken to 

avoid over-parameterizing the model with methodological variables. Unlike the weak structural model, a

strong structural utility theoretic model would involve: (i) specifying a structural form of the theoretic 

indirect utility function; and (ii) fitting a theory-consistent mathematical form to the strictly defined set of 

the economic variables (Bergstrom and Taylor 2006). A disadvantage of the strong structural utility 

 

12 Although EPA estimated both unweighted and weighted trans-log models only the unweighted model is presented in this 

section. Weighting observations based on the number of WTP observations reported in each study produced nearly identical 

results. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 20

theoretic approach is that “restrictions imposed on the utility function specification may preclude some 

explanatory variables of interest” (Bergstrom and Taylor, p.358) from the model. This type of modeling 

approach has not been adopted in the meta-analysis literature because (a) results tend to be highly 

sensitive to assumed utility functions, and (b) generally sacrifice empirical fit for a strong assumption 

regarding the structure of utility.

The trans-log form was used for the final estimated models based on input from preliminary models that 

include a variety of alternative functional specifications. This also follows one of the primary model 

specifications used in the original water quality valuation meta-analysis of Johnston et al. (2005). The 

trans-log specification incorporates the natural log of the dependent variable (the natural log of WTP per 

household) on the left hand side and the natural log of household income (lnincome), log-water quality 

change (lnquality_ch), relative geospatial scale (ln_ar_ratio), and log-proportion of agricultural land 

within the affected area (ln_ar_agr) on the right hand side. All binary independent variables enter linearly

(see variable definitions in Table 1). General advantages of the trans-log specification for meta-analysis 

are discussed by Johnston et al. (2005) and Londoño and Johnston (2012), and include an ability to 

capture curvature in the valuation function, multiplicative rather than additive effect of independent 

variables on WTP, and the implied and intuitive constraint that WTP is zero in the limit when key 

continuous variables (such as the size of water quality change or relative geospatial scale) are also zero. 

For logged continuous variables, coefficient estimates may be interpreted directly as elasticities, or the 

percent change in the dependent variable caused by a 1 percent change in the independent variable, all 

else constant. For the linear variables, given a single unit change in the k-th characteristic, the mean 

percentage change in WTP is given by (e

k

-1)100, where βk is the estimated coefficient on the variable. 

The model presented in Table 4 was selected after the estimation of numerous preliminary models with 

different specifications and groups of independent variables. It was selected based on both statistical fit 

and correspondence with theoretical expectations. The trans-log specification was very close to the betsfitting Box-Cox model. As noted above, EPA estimated the models using generalized least squares with 

random effects and robust standard errors to account for study-level dependencies. Measures of fit in the 

log-space for the illustrated model are good (χ

2

= 683.44, df=24, p<0.0001; R2= 0.65). These compare 

favorably to prior meta-analyses in the published literature. Study-level errors (σu) are found to be zero.

The model performs well, with intuitive results for virtually all statistically significant variables. The 

model identifies numerous statistically significant coefficients for variables characterizing (1) study 

methodology, (2) region and surveyed populations, (3) extent of the market, study site, and affected 

resources, and 4) water quality. In total, 20 out of 24 non-intercept coefficient estimates are statistically 

significant at p<0.10, and 16 of 24 are significant at p<0.01. The ln_ar_ratio and sub_proportion

geospatial variables are both significant at p<0.01 with signs on coefficients that are consistent with 

expectations. The change in water quality (lnqualicy_ch) is also highly significant with a positive sign, 

indicating that, as expected, higher water quality changes are associated with higher WTP. The variable 

for baseline WQI (lnbase) is not individually significant. EPA tested many variables specifications with 

the natural log of WTP (lnWTP) as the dependent variable, and a large, statistically significant effect was 

never observed for baseline WQI.13 This effect could be caused by multimodal distribution and might be 

better characterized as discrete rather than continuous.

Given the trans-log functional form, these results for lnquality_ch and ln_ar_ratio imply diminishing 

marginal returns to scope and scale (e.g., WTP per unit of water quality improvement declines as the 

 

13 During tests, EPA only found baseline WQI to be individually significant when specified as non-logged with non-logged 

WTP as the dependent variable. However, EPA selected the model with lnWTP as the dependent variable because of the 

property that forces WTP through the origin when there is no water quality improvement. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 21

scope of water quality change increases). WTP is lower among nonusers (nonusers), another expected 

result, because users may hold both use and nonuse values, while nonusers hold only nonuse values, by 

definition. Methodological variables also show expected patterns. For example, households are willing to 

pay a greater nominal amount in a one-shot payment (lump_sum) than they would be willing to pay in 

repeated annual payments.14 WTP also varies according to the type of uses potentially affected by the 

proposed changes (e.g., swim_use, boat_use). Although these variables may be correlated with the water 

quality condition, it is not always the case: For example, waters may be swimmable but the study focuses 

on improvements in game fishing (e.g., Banzhaf et al., 2006). EPA found that model fit was better with 

swim_use, boat_use, and gamefish included (R2of 0.65 with them included compared to 0.60 without) 

and the lnincome variable was no longer individually significant when they were excluded. The 

significance of the lnquality_ch variable is also higher when the three use variables are included (p>|z| of 

0.005 compared to p>|z| of 0.062 without them). Thus, dropping these variables could result in omitted 

variable bias. 

As expected, sub_proportion has a positive coefficient, indicating that WTP increases as the affected 

resource constitutes a larger fraction of available substitutes. The coefficient for ln_ar_ratio is negative 

because the sampled area (sa_area) is in the numerator indicating that average WTP decreases as the 

sampled market area increases relative to the affected resource. This is consistent with the expectation 

that a larger sample area will include a greater proportion of respondents who are resource nonusers, who 

are less familiar with the resource, and/or who live at a greater distance from the affected waters.

 

14 As discussed in Section 3, EPA also tested models with WTP values for short-term payment periods converted to perpetual 

streams but found that results were sensitive to discount rate assumptions and that these models did not provide any practical 

advantages for benefit transfer over models including the lump_sum variable.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 22

Table 4. MRM1 Estimation Results, Random Effects with Robust Standard Errrors.

Variable Parameter 

Estimate

Robust 

Standard 

Error

Z P>|z|

95% Confidence Interval

5% 95%

ce 0.5286 0.1994 2.65 0.008 0.1379 0.9194

thesis 0.6509 0.1991 3.27 0.001 0.2607 1.0411

lnyear -0.4688 0.0749 -6.26 0.000 -0.6156 -0.3219

volunt -1.3225 0.2221 -5.95 0.000 -1.7579 -0.8871

outliers_trim -0.3843 0.1215 -3.16 0.002 -0.6225 -0.1462

nonparam -0.4709 0.1210 -3.89 0.000 -0.7082 -0.2337

non_reviewed -0.7419 0.1662 -4.46 0.000 -1.0677 -0.4161

lump_sum 0.8250 0.1397 5.91 0.000 0.5512 1.0988

wtp_median -0.2543 0.2169 -1.17 0.241 -0.6794 0.1707

northeast 0.5198 0.2351 2.21 0.027 0.0591 0.9805

central 0.6526 0.1122 5.82 0.000 0.4327 0.8725

south 1.4611 0.1395 10.48 0.000 1.1878 1.7344

nonusers -0.4352 0.1186 -3.67 0.000 -0.6677 -0.2027

lnincome 0.7451 0.3749 1.99 0.047 0.0103 1.4798

mult_bod -0.5380 0.1384 -3.89 0.000 -0.8092 -0.2668

river -0.1745 0.1298 -1.34 0.179 -0.4290 0.0799

swim_use -0.4361 0.2179 -2.00 0.045 -0.8631 -0.0091

gamefish 0.2680 0.1749 1.53 0.125 -0.0747 0.6108

boat_use -0.2918 0.1626 -1.79 0.073 -0.6104 0.0269

ln_ar_agr -0.3919 0.0911 -4.30 0.000 -0.5705 -0.2133

ln_ar_ratio -0.0729 0.0253 -2.88 0.004 -0.1225 -0.0232

sub_proportion 0.6680 0.1811 3.69 0.000 0.3131 1.0229

lnquality_ch 0.2993 0.1064 2.81 0.005 0.0908 0.5078

lnbase -0.0364 0.1233 -0.29 0.768 -0.2780 0.2052

intercept -4.0431 4.3030 -0.94 0.347 -12.4768 4.3906

R

2

0.6455 (overall), 0.2414 (within), 0.8361 (between)

Number of Obs. 140, 51 clusters (range 1 – 12 observations per cluster)

χ

2

683.44 (p<0.0001)

Study Level (σu) 0.000

Residual (σe) 0.541

Rho (share of overall 

variance σe that is 

account for by σu)

0.000

rWTP

2

(squared 

correlation coefficient 

between the predicted 

and observed WTP)

0.5051

5 Robustness Tests

Because the meta-analysis of WTP for water quality improvements is intended to be used for benefit 

transfer, following Boyle et al. (2013), EPA examined various factors that indicate whether the individual 

parameter estimates are robust or fragile. One goal of testing a meta-regression for robustness is to 

compel benefit transfer practitioners to “... carefully consider how each study included in a meta-analysis 

matches policy-case conditions when the estimated equations are used to predict policy-case value 

estimates” (Boyle et al., 2013, p. 380). EPA notes that these robustness tests do not and cannot provide 

information on the best or most appropriate specification – nor does a lack of robustness associated with 

individual variables or observations imply that these variables or observations should be excluded. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 23

However, these tests can provide insight into the confidence that may be held in certain hypothesis tests 

drawn from model results.

The tests presented here focus on horizontal robustness (inclusion/exclusion of observations or studies) 

and vertical robustness (inclusion/exclusion of variables). All horizontal and vertical tests were run on 

MRM1 presented in Section 4. Section 5.1 describes horizontal robustness tests using measures of 

influence on overall regression fit and on parameter estimates and significance. Section 5.2 describes 

vertical robustness tests using extreme bounds analysis (EBA). Section Error! Reference source not 

found. concludes with a summary of robustness tests and findings.

EPA did not conduct robustness tests to address study or sample selection bias, as suggested by Boyle et 

al. (2013). EPA included in the metadata all relevant studies that met criteria for validity, consistency, and 

applicability described in Section 2. The objective of examining sample selection is to determine whether 

the studies included in the metadata represent a random sample from the possible universe of studies and 

study sites, or whether they follow some systematic pattern of selection. If there is a systematic pattern of 

selection, then the WTP estimates predicted by the MRM may be biased. However, in this case, it is not 

feasible to specify a meaningful sample selection equation because various factors are known to effect 

researcher’s site selection for water quality valuation studies. These factors include, but are not limited to, 

the national or regional importance of the water bodies, the extent of current or future water quality 

impairments, and concern of the local public for environmental issues. Given the number and nature of 

factors that affect selection, specifying a valid and unbiased selection equation is not straightforward, and 

a mis-specified equation could provide misleading results. EPA also notes a relatively good geographic 

and waterbody type coverage of the studies included in metadata indicating that a selection bias (if any) is 

unlikely to affect geographic location or body type. Figure 5Error! Reference source not found. below 

depicts a geographic distribution of the studies. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 24

Figure 5. Map Depicting Geographic Distribution of Studies by Waterbody Type. The numbers indicate the number of studies for that state. 

Their sum does not equal the total number of studies because some studies include multiple states.

Type 

all freshwater 

nver!Stream 

river/stream and lake 

riveriS~eam and ostiJary 

esluary 

sail pOI'Idfmarshes 

m!Atipl& warerbody 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 25

5.1 Horizontal Robustness

5.1.1 Introduction 

Boyle et al. (2013) describe “horizontal robustness tests” as a series of analyses which examine the 

influence of individual observations and studies on both the global fit of meta-regressions and the 

precision and sign of individual estimates. While iteratively removing one observation at a time helps to 

identify influential observations, prior research has shown that study-specific groups of observations can 

exert influence when individual observations do not. Because the present water quality meta-analysis 

explicitly acknowledges within-study dependencies of multiple observations per study, EPA proceeded

directly to leave-one-study- out robustness checks. Leave-one-study-out horizontal robustness attempts to 

determine the joint influence of observations from a single study. Further, observations from a study can 

[collectively, if there are more than one] lead parameter estimates to deviate from estimates based on the 

full meta-dataset. If studies or observations have substantial influence on coefficient signs and/or 

significances, it suggests that they are sufficiently different from the remaining body of literature. When 

regression coefficients estimated from reduced datasets deviate substantially from the main regression 

estimates, such differences can indicate the omitted observations are outliers from a dataset with 

otherwise narrow focus, or it can indicate that the meta-analysis has captured a broad range of 

heterogeneous estimates. Note that having captured a range of heterogeneous estimates is not necessarily 

bad, since having a wide variety of data enables the analyst to better comment on broad trends in the 

literature (rather than having narrow-sighted focus on a particular case). Hence, these results should not 

be used in the absence of other information to dictate the removal of individual studies from the metadata. 

Such decisions require a theoretically informed consideration of all factors surrounding metadata content 

and consistency.

EPA implemented a variety of horizontal robustness test procedures which Boyle et al. (2013) proposed. 

To implement horizontal robustness tests EPA used the PROC MIXED procedure in SAS (SAS/STAT 

9.2, 2010), which is appropriate for the models with dependencies across studies. PROC MIXED 

performs a leave-one-study-out (iterative) assessment of the following three robustness measures: overall 

model fit, the degree to which individual parameter estimates change, and the degree to which the 

statistical significance of individual parameter estimates changes. The PROC MIXED procedure for 

horizontal robustness generated errors when run with study-level random effects. This is not surprising 

given that study-level random effects were found to be zero (see Table 4). EPA dropped study-level 

random effects from the PROC MIXED procedure in favor of repeated measures estimation with study 

identification number as the subject. 

5.1.2 Horizontal Robustness Testing Methods

5.1.2.1 Overall Influence

Removing observations can change the objective function being minimized during estimation. Examining 

changes in this function before and after removing a group of observations provides a global, summary 

measure expressing the joint influence of the removed observations on all of the estimated parameters.

 PROC MIXED employs the likelihood displacement measure, which is equal the amount by 

which the log likelihood of the full data changes if one were to use an estimate based on fewer 

data points times two.

 Likelihood displacement needs to be compared to a χ2distribution with q degrees of freedom, 

where q is the number of fixed effects and covariance parameters that are subject to updating 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 26

when the group of studies was removed.. Since the 25 coefficients are updated each time a study 

is removed, EPA compares the likelihood displacement statistic to χ2= 37.653 (α= 0.05).

No studies in the meta-data exert a significant degree of influence on the global regression based on 

comparison to the χ2criteria of 37.653. Three studies, Collins et al. (2009), Desvousges et al. (1997), and 

Lant and Tobin (1989), have restricted log likelihood distances greater than 10 but do not exceed the χ

2

criteria. EPA used 10 only as a heuristic cut-off to highlight the studies that are influential relative to 

other studies in the metadata.15

5.1.2.2 Change in Individual Parameter Estimates

Although no likelihood displacement statistics suggested that removing a study would induce substantial 

overall changes to the meta-regression, EPA conducted additional tests to determine if studies influence 

results in more specific aspects relevant to meta-analysis interpretation and applications. Beginning with 

changes in individual parameter estimates when individual studies are excluded, EPA followed 

Schabenberger’s (2004) suggested steps:

 Remove a group of observations and examine the change in the overall parameter vector. Because 

the number of random effects and variables can be large, examining this change is a good 

summary of the influence that a group of observations has on the component variables in the 

meta-regression.

 SAS implements two summary statistics that represent study’s influence on coefficient estimates: 

the quadratic form of Cook’s D and MDFFITS, the multivariate statistic DFFITS. For both of 

these statistics, large values indicate that the change in the parameter estimate is large relative to 

the variability of the estimate. 

 Because both statistics describe the (normalized) change in coefficient estimates when removing 

an observation (or group of them), a value of 1 for these statistics indicate no change. Following 

Kennedy (2008), EPA flagged studies with D > 1 as influential. Flagged values are highlighted in 

blue in Table 5..

 Table 5 indicates that four studies have either a D or MDFFITS greater than one: Croke et al. 

(1987), Desvousges et al. (1987), Downstream Strategies (2008), and Lant and Tobin (1989). The

D and MDFFITS values for these studies range from 2.08 to 4.63, indicating that these studies are 

influential, with a large change in the parameter estimate relative to the variability of the estimate.

5.1.2.3 Change in the precision of parameter estimates

Studies that have little influence on the magnitude of parameter estimates (vis a vis small Cook’s D or 

MDFFITS) may still influence the precision of estimates, and thus also affect hypothesis tests and 

confidence intervals (Schabenberger 2004). By comparing variance of estimates based on the full and 

reduced datasets, PROC MIXED computes trace and determinants of the variance-covariance matrix in 

each round of estimation. When study i is removed, the covariance ratio (COVRATIO) reports the 

increase in variance (i.e., the decrease in estimate precision). 

The point at which covariance ratios become worrisome is subjective, but the five- to 98 fold increases 

seen for seven studies stand out in comparison to other studies in the sample (Table 5: Collins et al. 

(2009); Croke et al. (1987); Desvousges et al. (1987); Lant and Roberts (1990); Lyke (1993); Takatsuka 

 

15 These studies may represent relatively rare contexts, such as reducing acid mine drainage (Collins et al., 2009) and large 

water bodies (Desvouges et al. (1997) that address the Monongahela River in Pennsylvania). They also contribute a fairly 

high number of observations; 7 observations for Collins et al. (2009) , 12 observations for Desvousges et al. (1987), and 9 

observations for Lant and Tobin (1989).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 27

(2004), and Welle (1986)). COVRATIO values for these studies are highlighted in blue in Table 5. This 

degree of influence could be explained by features of these studies. For example, compared to other 

studies, these influential studies often focus on somewhat different resource characteristics and/or 

geographic scope of improvement (i.e., 100 percent of the affected waters in a given state). EPA

investigated Croke et al. (1987) further because its COVRATIO of 97.96 is far and away the highest. This 

study focuses on urban rivers in Chicago; studies of urban rivers are rare in the metadata. By comparing 

variable standard errors with and without Croke et al. (1987) included in estimation, EPA found that the 

variance for the boat_use variable increased by 117 percent with Croke et al. (1987), by far the greatest 

percentage increase for any variable.16 This is not surprising given that of the 16 total observations with

boat_use=1, six (38 percent) are from Croke et al. (1987).

 

16 Ten other variables had a percentage increase in variance of 20 percent or greater when Croke et al. (1987) was excluded

from estimation. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 28

Table 5. Horizontal Robustness Results by Study.

A

Study # Obs Restricted LL 

Distance Cook’s D MDFFITS COVRATIO

Aiken (1985) 1 0.482 0.0593 0.1026 0.390

Anderson and Edwards (1986) 1 0.933 0.0758 0.1831 0.025

Banzhaf et al. (2006) 2 0.724 0.0499 0.0531 0.350

Banzhaf et al. (2011) 1 0.014 0.0027 0.0019 2.394

Bockstael et al. (1984) 1 0.155 0.0300 0.0396 0.350

Bockstael et al. (1989) 2 0.157 0.0201 0.0154 1.481

Borisova et al. (2008) 3 2.473 0.1531 0.2074 0.129

Cameron and Huppert (1989) 1 0.129 0.0332 0.0115 3.539

Carson et al. (1994) 2 0.041 0.0085 0.0042 4.606

Clonts and Malone (1990) 3 2.476 0.1612 0.4330 0.037

Collins and Rosenberger (2007) 1 1.105 0.0699 0.1712 0.128

Collins et al. (2009) 7 12.607 0.4825 0.8614 5.394

Corrigan et al. (2008) 1 0.083 0.0243 0.0178 2.529

Croke et al. (1987) 9 5.535 2.0765 0.2251 97.961

De Zoysa (1995) 1 0.546 0.0512 0.0398 3.148

Desvousges et al. (1987) 12 17.873 4.6323 0.2923 6.230

Downstream Strategies (2008) 2 5.172 0.2803 3.9763 0.000

Farber and Griner (2000) 6 0.313 0.0549 0.2211 1.344

Hayes et al. (1992) 2 0.051 0.0080 0.0064 4.863

Herriges and Shogren (1996) 2 1.516 0.1683 0.1029 2.422

Hite (2002) 2 0.480 0.0809 0.0298 3.858

Huang et al. (1997) 2 0.022 0.0013 0.0011 3.169

Irvin et al. (2007) 4 1.030 0.3513 0.0367 4.103

Johnston et al. (1999) 1 0.389 0.0358 0.0463 0.943

Kaoru (1993) 1 0.502 0.0292 0.0315 0.220

Lant and Roberts (1990) 3 0.139 0.0271 0.0169 9.369

Lant and Tobin (1989) 9 11.205 0.6308 3.6023 0.000

Lichtkoppler and Blaine (1999) 1 0.177 0.0346 0.0401 0.652

Lindsey (1994) 8 5.272 0.4811 0.5875 0.005

Lipton (2004) 1 0.046 0.0146 0.0091 3.968

Londoño et al. (2013) 2 2.956 0.1807 0.8961 0.001

Loomis (1996) 1 0.078 0.0175 0.0116 1.261

Lyke (1993) 2 0.033 0.0122 0.0099 7.427

Matthews et al. (1999) 2 0.811 0.0956 0.0606 2.667

Opaluch et al. (1998) 1 0.577 0.0614 0.0395 1.056

Roberts and Leitch (1997) 1 0.174 0.0371 0.0339 0.290

Rowe et al. (1985) 1 0.944 0.1153 0.3665 0.006

Sanders et al. (1990) 4 0.932 0.1216 0.0900 0.741

Schulze et al. (1995) 2 1.054 0.1271 0.0918 4.274

Shrestha and Alavalapati (2004) 2 0.248 0.0679 0.0299 2.013
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 29

Table 5. Horizontal Robustness Results by Study.

A

Study # Obs Restricted LL 

Distance Cook’s D MDFFITS COVRATIO

Stumborg et al. (2001) 2 0.043 0.0073 0.0062 2.956

Sutherland and Walsh (19850 1 0.022 0.0047 0.0029 2.292

Takatsuka (2004) 4 0.756 0.2083 0.0731 6.965

Wattage (1993) 3 0.135 0.0249 0.0274 3.897

Welle (1986) 6 0.507 0.1320 0.0407 9.419

Welle and Hodgson (2011) 3 1.341 0.1755 0.1380 3.643

Wey (1990) 2 1.915 0.0946 0.1412 0.646

Whitehead and Groothuis (1992) 3 4.562 0.2037 0.6047 0.000

Whitehead (2002) 3 0.819 0.0574 0.0481 1.521

Whitehead et al. (1995) 2 0.048 0.0107 0.0032 4.543

Whittington et al. (1994) 1 0.007 0.0009 0.0007 2.451

Notes:

(A): Highlighted in blue are Cook’s D and MDFFITS values greater than 1, COVRATIO values greater than 5, 

and restricted log likelihood values greater than 10. 

5.2 Vertical Robustness

5.2.1 Introduction 

Extreme bounds analysis (EBA) is the process of conducting, “a sensitivity test that examines how 

robustly the dependent variable of a regression model is related to a variety of possible determinants” 

(Hlavac, 2014a). Hlavac describes EBA as a process of,

“...finding out which variables from the set X are robustly associated with the dependent variable 

y. To do so, we run a large number of regression models. Each has y as the dependent variable 

and includes a set of standard explanatory variables F that are included in each regression model. 

In addition, each model includes a different subset D of the variables in X. Following the 

convention in the literature, we will refer to F as the free variables and to X as the doubtful 

variables. Some subset of the doubtful variables X might be so-called focus variables that are of 

particular interest to the researcher. The doubtful variables whose regression coefficients retain 

their statistical significance in a large enough proportion of estimated models are declared to be 

robust, whereas those that do not are labelled fragile. More formally, to find out whether a focus 

variable v in X is robustly correlated with the dependent variable y, we estimate a set of 

regression models of the following form:

y = αj + βj v +γj F + δjDj + e 

where j indexes [each] regression model, F is the full set of free variables that will be included in 

every regression model, Djis a vector of k variables taken from the set X of doubtful variables, 

and e is the error term.” (Hlavac, 2014a, p. 3)

According to Hlavac (2014a), Dj has “conventionally been limited to no more than three doubtful 

variables per model (Levine and Renelt 1992; Achen 2005).” Levine and Renelt (1992) commented on 

use of such a limited number of doubtful variables (and the choice of which to test), suggesting that for 

reasonableness and simplicity in analysis, they “... only search over seven indicators that [arguably] 

represent a reasonable conditioning set [on the significance of variables of interest].” The authors also, 

“... further restrict the pool of variables from which we choose D-variables by excluding variables that, a 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 30

priori, might measure the same phenomenon [as the variable of interest v].” In practice, EBA could treat 

as “doubtful” the full set of a researcher’s explanatory variables, if the researcher held no basic a priori 

expectations or evidence about which types of variables are most robustly associated with one another 

(Boyle, et al., 2013; Hlavac, 2014b).

Like horizontal robustness, vertical robustness tests cannot be used alone to determine whether individual 

variables should be included or excluded from a meta-regression model. Such assessments must consider 

a variety of issues including theoretical expectations, the risk of omitted variable bias, the potential 

impact of multicollinearity, and other factors. Also, these tests do not provide insight into the ability of 

the overall model to predict WTP. In some cases, models including non-robust (or fragile) variables can 

predict better than those excluding these variables. Hence, results of these tests must be considered 

carefully, within the context of all other information available about a particular meta-regression. 

However, the results can inform the confidence that researchers give to hypothesis tests.

5.2.2 Approach and application to WQ WTP Analysis

EPA implemented EBA using ‘ExtremeBounds,’ a new user-submitted packaged program for the R 

Statistical Computing Environment (Hlavac, 2014b). This package is more advantageous than other 

approaches that EPA is aware of, such as using the .ado program ‘eba’ in Stata (which has limited 

functionality), or hand-coding an original EBA script. 

The broadest EBA of the WQ WTP meta-regression would treat all variables as “doubtful,” and would 

estimate regressions with all possible combination of explanatory variables in Table 4, ranging from one 

explanatory variable to all 24. Such computations are potentially valuable in that no a priori assumptions 

need to be made about relative stability of some variables over others, but the millions of regressions 

required at this scale of analysis is beyond the computational power and resources available. As an initial 

test of EBA, EPA specified the ‘ExtremeBounds’ model with the following settings:

 All variables in MRM1 were included during robustness testing. The MRM needs to account for 

basic indicators of study type, method, and quality because it aggregates data from studies using a 

wide variety of survey methods and statistical estimators. Failure to include controls for these 

factors will result in omitted variable bias on other variables. EPA thus required a set of eight 

“free” variables to appear in every regression: thesis, volunt, non_reviewed, nonparam, 

lump_sum, wtp_median, ce, and outliers_trim. Because these variables are no longer tested in the 

regression-varying combinations of variables, it reduces the number of regressions to 63,018.

 EPA treated all remaining variables as “focus” variables, and thus consider their relationship with 

lnWTP as “doubtful”. One or more of the “focus” variables appears in each of the estimated 

regressions. EPA added combinations of variables in each specification, in combinations of up to 

10 extra variables at a time17. Added variables are picked from the set of: lnyear, nonusers, 

lnincome, northeast, central, south, river, swim_use, gamefish, boat_use, mult_bod, ln_ar_agr, 

sub_proportion, ln_ar_ratio, lnbase, and lnquality_ch.

 To eliminate the influence of coefficient estimates from model specifications with high 

multicollinearity, EPA specified a maximum acceptable variance inflation factor (VIF) of 10 

(Kennedy 2008). 

 To reduce the time required to complete the analysis, EPA estimated a random sample of 10,000 

of the 63,018 possible regressions by the above model settings (16 percent). This procedure, 

 

17 Note that this produces regressions that are smaller in the number of variables than the estimated meta-regression in Table 4, 

but EPA used the Stata-recommended 50 degrees of freedom in all tests.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 31

“yields unbiased estimates of the quantities of interest that can give the researcher a good sense of 

the full EBA results within a reasonable time frame” (Sala-i-Martin et al. 2004). 

5.2.3 EBA Vertical Robustness Results

Table 6 lists the number of regressions on which the EBA analysis is conducted. The right-hand column 

reports the total number of regressions which include each of our focus variables. Because no regression 

estimates were eliminated from EBA due to high VIF, EBA is based on this number of coefficients. 

Histograms in Error! Reference source not found. and Error! Reference source not found. display the 

distribution of coefficient estimates for free and focus variables in our series of regressions, respectively. 

Where a large portion of a variable’s coefficient distribution falls to the right of the vertical red bar at 

zero, most of the estimated coefficients for a variable are positive. Estimated coefficients for several key 

variables, including lnquality_ch, south, and thesis are positive in all regressions, and the estimated 

coefficients for lump_sum, lnincome, northeast, sub_proportion, ce, central, and gamefish are 

predominantly, although not always, in the positive. 

On the other hand, in cases where a large portion of the distribution falls to the left of (below) zero, most 

of the estimated coefficients are negative. Estimated coefficients for volunt, non_reviewed, lnyear, 

nonusers, river, and ln_ar_agr were always negative in our regressions, and the estimated coefficient for 

nonparam, wtp_median, outliers_trim, swim_use, boat_use, mult_bod, and lnbasewas usually negative. 

The result for ln_ar_ratio is more mixed, with this variable’s distribution covering substantial negative 

and positive regions. Also, ln_ar_ratio appears bi-modal: in one cluster of estimation results, the 

coefficient is approximately zero, and in another cluster, it is negative. Potential causes of this effect are 

discussed below.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 32

Table 6. Number of Estimates Used to Estimate EBA of Free and Focus 

Variables.

A

Variable Free or Focus? Number of estimates

intercept Free 10,000

thesis Free 10,000

volunt Free 10,000

non_reviewed Free 10,000

nonparam Free 10,000

lump_sum Free 10,000

wtp_median Free 10,000

ce Free 10,000

outliers_trim Free 10,000

lnyear Focus 4,938

nonusers Focus 4,849

lnincome Focus 4,915

northeast Focus 4,943

central Focus 4,899

south Focus 4,912

river Focus 4,915

swim_use Focus 4,913

gamefish Focus 4,878

boat_use Focus 4,881

mult_bod Focus 4,952

ln_ar_agr Focus 4,954

sub_proportion Focus 4,855

ln_ar_ratio Focus 4,982

Lnbase Focus 4,799

lnquality_ch Focus 4,905

Notes: 

(A): 63,018 regressions were possible, of which 15.9 percent were estimated (n=10,000). 

Each estimate comes from a separate regression.

Table 7 reports the information on the distribution of coefficient estimates in tabular format. For each 

variable, it also shows the mean coefficient estimate and the mean of the standard errors on estimated 

regression coefficients; whether or not the variable was predominantly positive or negative, and the 

percent of cases it held that sign; and the percent of cases where the estimated coefficient was both of the 

dominant sign and statistically significantly different from zero (at α = 0.05). 

Table 8 and Table 9 extend the raw results, demonstrating EPA’s conclusions about vertical robustness 

based on Leamer and Sala-i-Martin robustness tests. The main goal of the extreme bounds analysis is to 

provide insight on the degree of robustness in the parameter estimates of policy variables, under the 

inclusion/exclusion of valuation method variables. 

In general, EPA found that key variables (e.g., water quality changes) are robust under reasonable testing 

criteria, speaking to the overall quality of the estimated benefit transfer function. Other variables denoting 

resource and sampled population characteristics are more fragile (e.g., sub_proportion, lnincome). In 

some cases, this fragility may be attributed to relationships between two variables. For example, the bimodal distribution of ln_ar_ratio suggests the relationship between the size index of a water body and 

willingness to pay for water quality improvements may be stratified across different resource or study 

characteristics. For example, ln_ar_ratio is weakly correlated with both sub_proportion (rho = -0.08) and 

ln_ar_agr (rho = -0.39). Thus, in the robustness check they tend to show lower stability.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 33

In the case of other fragile variables, the results of robustness tests can inform values used in benefit 

transfer applications. For example, fragile free variables (e.g., non_reviewed) or resource characteristic 

variables may be assigned a value of zero. 

Several more detailed comments can be made regarding interpretation of the test results:

1. Leamer’s criteria suggest most coefficient estimates are fragile. However, Leamer’s criteria are 

often criticized as a priori conservative: they have been described as both “demanding” (Hlavac, 

2014a) and “extremely strict” (Boyle et al. 2013, p. 381). Sala-i-Martin commented that because 

the Leamer’s EBA test is “...too strong for any variable to pass it ...giving the label of nonrobust to all variables is all but guaranteed” (Sala-i-Martin 1997, p. 179). Thus, given other 

economists’ experience implementing these tests, EPA is neither surprised nor alarmed at results 

based on Leamers’ criteria. 

2. Sala-i-Martin’s procedure bases robustness tests on a more “reasonable” approach considering the 

weight of evidence (rather than deferring to the single most extreme result) of the series of 

regressions (Hlavac 2014a) . Results of this test suggest that eight of the variables are robust 

(thesis, volunt, lnyear, nonusers, south, river, ln_ar_agr, and lnquality_ch). Here, robust variables 

are those for which 95 percent or more of the CDF lies either above or below zero (following 

Sala-i-Martin’s criterion). Under somewhat less-stringent cutoff of 90 percent, three additional 

variables approach robustness (non_reviewed, nonparam, and outliers_trim). These results are 

encouraging since the water quality change variable (lnquality_ch) is the primary focus for 

benefit transfer.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 34

Figure 6. Histograms Summarizing Coefficient Estimates Resulting from Extreme Bounds Analysis of Free Variables.

Notes: Vertical red lines are at zero, and blue kernel density curves are non-parametric approximations of a regression coefficient’s distribution. 

Source: Uses Hlavac’s (2014b) R package ‘ExtremeBounds.’

outliers_trim
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 35

Figure 7. Histograms Summarizing Coefficient Estimates Resulting from Extreme Bounds Analysis of Focus Variables.

Notes: Vertical red lines are at zero, and blue kernel density curves are non-parametric approximations of a regression coefficient’s distribution. 

Source: Uses Hlavac’s (2014b) R package ‘ExtremeBounds.’

sub_proportion ln_ar_ratio

lnincome

river
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 36

Table 7. Means and Standard Errors of Estimated Coefficients over 10,000 Estimated 

Regressions.

Variable Mean Coefficient 

Estimate (range)

Mean 

SE

Predominant 

Sign

% Est.’s Are 

Predominan

t Sign

% Est.’s Are 

Predominant 

Sign and 

SignificantA

intercept 0.931 (-14.733 to 9.701) 2.688 Positive 64.8 50.9

thesis 0.982 (0.518 to 1.39) 0.322 Positive 100.0 97.8

volunt -0.898 (-1.506 to -0.443) 0.236 Negative 100.0 100.0

non_reviewed -0.57 (-1.122 to -0.083) 0.286 Negative 100.0 46.1

nonparam -0.295 (-0.698 to 0.067) 0.163 Negative 98.2 44.1

lump_sum 0.075 (-0.331 to 0.762) 0.209 Positive 62.0 10.5

wtp_median -0.262 (-0.574 to 0.111) 0.247 Negative 97.5 5.6

ce 0.127 (-0.401 to 0.764) 0.220 Positive 66.7 13.5

outliers_trim -0.361 (-0.733 to 0.019) 0.192 Negative 99.9 42.8

lnyear -0.318 (-0.56 to -0.103) 0.111 Negative 100.0 81.7

nonusers -0.346 (-0.594 to -0.157) 0.161 Negative 100.0 49.8

lnincome 0.667 (-0.339 to 1.779) 0.454 Positive 97.5 31.4

northeast 0.498 (-0.168 to 1.377) 0.345 Positive 96.2 30.5

central 0.128 (-0.224 to 0.708) 0.160 Positive 65.1 28.4

south 0.996 (0.645 to 1.494) 0.179 Positive 100.0 100.0

river -0.374 (-0.798 to 0.035) 0.169 Negative 100.0 58.5

swim_use -0.151 (-0.576 to 0.215) 0.235 Negative 86.5 1.1

gamefish 0.258 (-0.107 to 0.708) 0.210 Positive 96.4 25.5

boat_use -0.152 (-0.427 to 0.155) 0.261 Negative 92.9 0.1

mult_bod -0.152 (-0.749 to 0.251) 0.273 Negative 79.4 8.2

ln_ar_agr -0.302 (-0.519 to -0.169) 0.096 Negative 100.0 99.6

sub_proportion 0.347 (-0.16 to 0.915) 0.241 Positive 97.8 25.5

ln_ar_ratio -0.016 (-0.092 to 0.041) 0.032 Negative 61.3 11.1

lnbase -0.079 (-0.358 to 0.247) 0.124 Negative 76.9 5.3

lnquality_ch 0.295 (0.129 to 0.425) 0.126 Positive 100.0 82.5

Notes:

(A): At 95 percent confidence.
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 37

Table 8. Leamer’s Extreme Bounds Analysis Results.

Variable Lower EB Upper EB Robust or Fragile

intercept -22.225 20.449 Fragile

thesis -0.150 2.110 Fragile

volunt -1.982 -0.008 Robust

non_reviewed -1.616 0.545 Fragile

nonparam -1.115 0.346 Fragile

lump_sum -0.831 1.052 Fragile

wtp_median -1.161 0.619 Fragile

ce -0.900 1.296 Fragile

outliers_trim -1.116 0.364 Fragile

lnyear -0.762 0.144 Fragile

nonusers -0.864 0.201 Fragile

lnincome -1.340 2.471 Fragile

northeast -0.956 1.955 Fragile

central -0.595 0.950 Fragile

south 0.217 1.821 Robust

river -1.115 0.400 Fragile

swim_use -0.996 0.674 Fragile

gamefish -0.556 1.111 Fragile

boat_use -1.082 0.642 Fragile

mult_bod -1.354 0.907 Fragile

ln_ar_agr -0.723 0.020 Fragile

sub_proportion -0.633 1.299 Fragile

ln_ar_ratio -0.148 0.114 Fragile

lnbase -0.663 0.485 Fragile

lnquality_ch -0.081 0.690 Fragile
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 38

Table 9. Sala-i-Martin’s Extreme Bounds Analysis Results.

Variable CDFA(β<0) CDFA(β>0) Robust or FragileB

intercept 32.757 67.243 Fragile

thesis 0.474 99.526 Robust

volunt 99.854 0.146 Robust

non_reviewed 93.780 6.220 Fragile

nonparam 90.652 9.348 Fragile

lump_sum 39.714 60.286 Fragile

WTP_median 82.545 17.455 Fragile

ce 35.628 64.372 Fragile

outliers_trim 94.091 5.909 Fragile

lnyear 98.420 1.580 Robust

nonusers 96.536 3.464 Robust

lnincome 13.634 86.366 Fragile

northeast 14.288 85.712 Fragile

central 33.229 66.771 Fragile

south 0.002 99.998 Robust

river 95.928 4.072 Robust

swim_use 71.409 28.591 Fragile

gamefish 16.720 83.280 Fragile

boat_use 70.520 29.480 Fragile

mult_bod 67.907 32.093 Fragile

ln_ar_agr 99.673 0.327 Robust

sub_proportion 12.132 87.868 Fragile

ln_ar_ratio 62.134 37.866 Fragile

lnbase 67.292 32.708 Fragile

lnquality_ch 1.428 98.572 Robust

Notes: 

(A): Generic model of CDF; no assumption is made about the distribution of beta coefficients across 

models. Results are similar to, but not the same as, Sala-i-Martin EBA conducted assuming coefficient 

estimates are distributed normally across models.

(B): Based on Sala-i-Martin’s (1997) criteria, in which a variable is deemed robust if more than 95 

percent of the cumulative distribution function of the beta estimate falls above or below zero.

5.3 Summary and Findings from Horizontal and Vertical Robustness 

Tests

This section reports results of horizontal and vertical robustness tests to determine the relative 

influence of individual studies on the regression estimates and significances and the stability of 

parameter estimates with respect to inclusion of other covariates. While EPA has found some 

evidence of both horizontal and vertical fragility in each of these dimensions, on net the weight of 

evidence suggests the meta-regression is relatively robust. It would be rare to conduct these tests 

on any meta-regression and find a perfectly robust regression. 

Nonetheless, understanding areas of fragility is important for practitioners who may apply this 

meta-regression in the future. For example, 

 Most studies appear to exert a modest amount of influence on global fit and (at least at a 

summary level) individual coefficient estimates. This finding suggests that, in the broad 

picture, model results are not overly sensitive to the inclusion or exclusion of any single 

study in the meta-data. 
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 39

 However, nine of 50 studies appear to exert some influence on global fit or specific 

variables. For example, studies by Croke et al. (1987), Lant and Roberts (1990), and 

Welle (1986) generate a nine-fold increase in variance when omitted, as measured by 

COVRATIO. As previously discussed, these studies’ influence is likely driven by the 

studies’ focus on specific water bodies with unique properties not observed throughout 

the general sample of meta-data.

 Relationships between lnWTP and key policy-related variables are generally robust to the 

inclusion of various other explanatory variables, suggesting a high degree of confidence 

in the sign and significance of the relationship between water quality improvements

(lnquality_ch) and willingness-to-pay for these improvements. 

On the other hand, robustness testing did identify a number of fragile variables. These include: 

ln_ar_ratio and lump_sum. Analysts should place relatively more trust in the robust variables 

(e.g., lnquality_ch) than in fragile ones, at least from a vertical robustness point of view (Boyle et 

al. 2013). As a result of these robustness tests, EPA has determined it is appropriate to retain all 

the studies and explanatory variables.

5.4 Cross-Validation Test

EPA also tested the out-of-sample predictive accuracy of the MRM1 using a leave-oneobservation-out, cross-validation approach. Under this approach, the model is re-estimated 140 

times (once for each observation in the metadata), with each observation j sequentially omitted. 

EPA generated a WTP prediction for each observation j, using parameter estimates from the –j

model and observation j variable values from the metadata. 

EPA calculated the accuracy of WTP prediction by comparing it to the observed primary WTP 

estimate for observation j from the metadata. Table 10 and Figure 8 summarize the crossvalidation results in terms of dollars and percentage differences. EPA found that out-of-sample 

predictive performance of MRM1 was generally similar to that of MRM2, the alternative model 

described in subsequent sections, with similar errors on average. 

Table 10. MRM1 Cross-Validation Results.

Metric Mean Percentile Min Max 5% 50% 95%

Transfer Error (predicted WTP 

minus observed WTP, 2007$) -7.0 -119.1 -1.5 66.6 -220.3 258.6

Absolute Value of Transfer Error 

(predicted WTP minus observed 

WTP, 2007$)

41.2 1.5 21.9 148.4 0.1 258.6

Percentage Transfer Error (%) 29.2% -61.5% -2.6% 145.1% -69.9% 1,826.4%

Absolute Value of Percentage 

Transfer Error (%) 62.1% 2.1% 34.2% 145.1% 0.2% 1,826.4%
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 40

Figure 8. Results of Leave-One-Observation-Out Cross-Validation Test for MRM1. (a) 

Scatter plot of predicted vs observed ln(WTP). (b) Scatter plot of predicted vs observed WTP. (c) 

Histogram of “transfer error”, defined as the WTP prediction minus the observed WTP. (d) 

Histogram of the absolute value of percentage transfer error. 

(a) (b)

(c) (d)

6 Benefit Transfer Application

This section illustrates application of MRM1 to estimation of benefits for water quality 

improvements under a hypothetical policy scenario, laying out the potential steps and datasets 

used for the transfer. The purpose of the hypothetical policy scenario is to abstract from the 

particulars of a specific policy under consideration. For clarity, the example focuses on water 

quality improvements to a limited number of river reaches in a relatively small geographic area. 

EPA expects that these steps would be similar when implementing a benefit transfer for other 

waterbody types, multiple water types, or a high number of waterbodies spread throughout a large 

geographic area as may be required to support a benefits analysis for a national rulemaking.

0

.05

.1

.15

.2

Fraction of Observations

-300 -200 -100 0 100 200 300

Transfer Error ($/hh/yr)
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 41

6.1 Benefit Transfer Example

The benefit transfer approach uses Census block group (BG) as the geographic unit of analysis.18

It is assumed that households in each BG value changes in the affected waterbodies within a 

specified distance, here set to 100 miles. A radius of 100 miles is consistent with Viscusi et al. 

(2008) based on the assumption that the majority of recreational trips will occur within a 2 hour 

drive from home. The waterbodies not affected by a regulation or water quality change within this 

buffer zone are assumed to represent substitute sites. EPA notes that this assumption is 

conservative in that it presumes that all households that are not within 100 miles of an affected 

resource have zero WTP.

By focusing on a buffer around the BG as a unit of analysis, rather than buffers are affected 

waterbodies, each household is only included in the assessment once, eliminating the potential for 

double-counting of households even though the buffer around a BG will likely overlap with other 

BG areas.

19 Total national WTP is calculated as the sum of estimated BG-level WTP across all 

BGs that have at least one affected waterbody within 100 miles. There will be zero WTP for BGs 

with no affected waters within 100 miles by the assumptions of this analysis. Limiting 

populations to a specific study area (e.g., watershed) may result in underestimation of benefitting 

population and may bias the substitution effect. Figure 9 presents a hypothetical example 

focusing on two BGs near 5 affected waterbodies, with affected resources and BGs are 

highlighted in red and the centroid of census both census BGs encircled by a 100-mile radius 

buffer.

20

Table 11 summarizes EPA’s approach for assigning values for all variables. Following benefit 

transfer using the prior water quality MRM (U.S. EPA 2009; U.S. EPA 2012; U.S.EPA 2013),

methodological variable values are specified based on judgments regarding preferred practice for 

primary studies. Their values are held fixed across all analyzed BGs. Table 12 presents 

hypothetical baseline WQI and WQI changes for the affected waterbodies.

Table 13 summarizes BG-specific variable assignments and resulting household WTP estimates 

using MRM1. Annual household WTP is multiplied by the total number of households in the BG 

to calculate total WTP for the BG. Both per household WTP and total WTP are greater for BG B 

than BG A. BG B’s affected rivers experience a greater water quality improvement, 2.44 on the 

WQI compared to 1.69 for BG A, and BG B has a larger household population. The difference in 

water quality is only partially offset by the greater affected area in BG A.

 

18 A Census block group is a group of Census blocks (the smallest geographic unit for the Census) in a contiguous 

area that never crosses a State or county boundary. A block group typically contains a population between 600 and 

3,000 persons. There are 217,740 block groups in the 2010 Census. See 

http://www.census.gov/geo/maps-data/data/tallies/tractblock.html .

19 Population double-counting issues can arise when using “distance to water body” to assess simultaneous 

improvements to many water bodies.

20 When applied to an actual scenario each affected waterbody would corresponds to a reach in the National 

Hydrography Dataset identified by COMID number. For estuaries, coasts, and the Great Lakes, shorelines should 

be matched to the Global Self-Consistent, Hierarchical, High-Resolution Geography Database (GSHHD). It is 

important to use these two datasets (NHD and GSHHD) to prevent any misspecification due to differences in data 

resolution relative to the metadata.
August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document| pg 42

Figure 9. Illustration of Intersection of BGs and COMIDs.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 43

Table 11. Approach for Specifying Variable Values for Analyzed BGs.

Variable Assigned Value Description

Study Methodology and Year

ce 0.5185 Set to reflect the mean for the last ten years of the metadata (2002 

on) because choice experiments have become more common in 

recent years.

thesis 0 Set to zero because studies published in peer-reviewed journals are 

preferred

lnyear 2.5257 Set to reflect the mean of the year index value for studies in the 

metadata (12.5). lnyear is the natural log of year index.

volunt 0 Set to zero because hypothetical voluntary payment mechanisms are 

not incentive compatible (Mitchell and Carson 1989)

outliers_trim 1 Set to one because survey data that exclude outlier bids are

preferable.

nonparam 0 Set to zero because most studies used in the meta-analysis used 

regression analysis to calculate WTP values. Some older studies in 

the metadata with nonparam=1 just reported the mean WTP value 

across responses. This is not the preferred approach in the current 

literature. 

non_reviewed 0 Set to zero because studies published in peer-reviewed journals are 

preferred.

lump_sum 0 Set to zero to obtain a prediction that is an annual WTP.

wtp_median 0 Set to zero because only average or mean WTP values that when 

multiplied by the number of affected households will always yield total 

benefits.

Region and Surveyed Population

northeast 0 Three binary variables that indicate the region in which the BG is 

located (northeast, central, south). The regional boundaries are 

defined by states. For this example, assume that affected BGs are 

located in the Mid-Atlantic U.S. (default) and assign zeros for

northeast, central, and south. Regional specification will vary across 

BGs when implementing a national benefit transfer.

central 0

south 0

nonusers 0 Zero because BGs are expected to contain both users and nonusers 

of the affected resources within the buffer.

lnincome BG-specific Set equal to the natural log of median income for each BG based on 

Census data.

Sampled Market and Affected Resource

mult_bod 0 Binary variable indicating that multiple waterbody types are affected 

(e.g., river and lakes). For this example, assign a value of zero 

because calculations are based exclusively on rivers.

river 1 Binary variable indicating that rivers are affected. For this example, 

assign a value of 1 because calculations are based exclusively on 

stream miles.

swim_use BG-specific Binary variables that identify studies which swimming, gamefish, and 

boating uses are specifically identified. Specify values for all three 

variables for each BG based on the length-weighted average of 

baseline water quality and water quality changes for affected 

resources within the 100 mile buffer based on the water quality 

ladder. 

gamefish BG-specific

boat_use BG-specific

ln_ar_agr BG-specific Use Census county boundary layers to identify counties that intersect 

affected resources within 100 mile buffer of each BG. For intersecting 

counties, calculate the fraction of total land area that is agricultural 

using the National Land Cover Dataset (NLCD). The ln_ar_agr

variable was coded in the metadata to reflect the area surrounding 

the affected resources. Setting this variable to reflect the entire buffer 

would bias WTP predictions if the average rate over the entire buffer 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 44

Table 11. Approach for Specifying Variable Values for Analyzed BGs.

Variable Assigned Value Description

is different from the rate of agricultural development nearby the 

affected resources. Differences in rates are more likely in cases 

where there are few affected resources within the buffer or the 

affected resources are located in a limited portion of the buffer.

ln_ar_ratio BG-specific Set the sampled area (sa_area) equal to the area of each BG as if a 

primary valuation study was being conducted for the BG that asks 

households within the BG to value improvements to affected 

resources within the buffer. Calculate the denominator 

(ar_total_area) by using GIS to intersect affected reaches with the 

Census county boundary layer and calculate their total area with 

duplicate counties dropped. Calculating the denominator for each BG 

tailors the WTP prediction to the size of affected resources within the 

buffer.

sub_proportion BG-specific Calculate the ratio of affected reaches miles to the total number of 

reach miles within the buffer that are the same order(s) as the 

affected reaches within the buffer. Its value can range from 0 to 1.

Water Quality Baseline and Change

lnquality_ch BG-specific Calculate the length-weighted average WQI change for affected 

COMIDs within the 100 mi buffer of each BG based on water quality 

modeling. lnquality_ch is the natural log of the weighted average WQI 

change.

lnbase BG-specific Calculate the length-weighted average baseline WQI for affected 

COMIDs within the 100 mi buffer of each BG based on water quality 

modeling. lnbase is the natural log of the weighted average baseline 

WQI.

Table 12. Hypothetical Water Quality Improvements Used for Benefit Transfer Illustration.

Census Block 

Group ID

Affected 

COMID

COMID 

Length (mi)

COMID-Specific WQI 

Values

BG Length-Weighted 

Average

Baseline Change Baseline Change

A 1 2 50 3 47.81 1.69

3 7 46 1

4 3 45 2

5 4 52 2

B 8 2 40 4 47.00 2.44

4 3 45 2

5 4 52 2
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 45

Table 13. BG-Specific Variable Assignment for Benefit Transfer Example and Estimated WTP.

Variable Assigned Value/ Notes BG A BG B

Study Methodology and Year

ce See Table 11 0.5185 0.5185

thesis 0.0000 0.0000

lnyear 2.5257 2.5257

volunt 0.0000 0.0000

outliers_trim 1.0000 1.0000

nonparam 0.0000 0.0000

non_reviewed 0.0000 0.0000

lump_sum 0.0000 0.0000

wtp_median 0.0000 0.0000

Region and Surveyed Population

northeast Assume both BGs are in the Mid-Atlantic, 

therefore assign zeros for all regional variables

0.0000 0.0000

central 0.0000 0.0000

south 0.0000 0.0000

nonusers See Table 11 0.0000 0.0000

lnincome Assume $50,000 for BG A and $55,000 for BG B 10.8198 10.9151

Sampled market and Affected Resources

mult_bod Assign 0 because only one waterbody type is 

affected (rivers) 0.0000 0.0000

river Assign 1 because analyzing rivers 1.0000 1.0000

swim_use

Assign to reflect water quality values based WQL 

uses corresponding to WQI values in Table 11

0.0000 0.0000

gamefish 0.0000 0.0000

boat_use 1.0000 1.0000

ln_ar_agr Assume agricultural fraction of 0.30 for BG A and 

0.20 for BG B -1.2040 -1.6094

ln_ar_ratio

Calculate assuming an affected area of 270 .3 

mi2and sampled area of 2.7 mi2for BG A, and an 

affected area of 115.8 mi2and sampled area of 

3.1 mi2for BG B. The sampled area varies 

because the areas of BGs are not uniform.

-4.6052 -3.6243

sub_proportion Assume substitute proportion of 0.02 for BG A 

and 0.01 for BG B 0.0200 0.0100

Baseline WQI and WQI Change

lnquality_ch Correspond to length-weighted averages 

presented in Table 11 0.5247 0.8920

lnbase Correspond to length-weighted averages 

presented in Table 11 3.8501 3.8672

WTP Results

Estimated Household 

WTP per Year

Consistent with U.S. EPA (2013), EPA 

adds half of the model residual variance 

to the natural log of WTP prior to 

calculating the exponential.21

$25.58 $33.21

Households in BG Assume 200 households in BG A and 300 

households in BG B. 200 300

Total BG WTP (2007$) - $5,116.23 $9,963.22

6.2 Benefit Transfer Application Limits

The specific functional form chosen for MRM1 (the log-log form) is based on statistical fit within a weak 

structural utility theoretic framework (Bergstrom and Taylor 2006), following approaches in virtually all 

 

21 This adjustment was not included in the cross-validation predictions from Section 5.4, but this should not have a significant 

effect on the interpretation of cross-validation results. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 46

published valuation meta-regression models in the literature. For example, the final functional form 

contains many factors that are expected (based on theory) to influence WTP –including income, the water 

quality change, and the baseline water quality, among possibly many other factors. It also implies that 

WTP necessarily will equal 0 when the water quality change equals 0—another theoretically expected 

pattern. Beyond this, no additional functional constraints or strong assumptions are imposed on the 

model—the specification is based on that which appears to best fit the data.

Equation [1] from Section 4 implies the following WTP function:

WTPjs = αjsYjs

βYQjs

βQDjs

βD

[3]

where αjsis a reduced form parameter containing the error term, εjs. This functional form has two practical 

limitations on the circumstances under which MRM1 can be directly used for a benefits transfer. 

First, the WTP function in equation [3] tends to predict larger than expected values for very small water 

quality changes (smaller than the smallest change reported in the meta-data). Commonly used utility 

functions generally imply that WTP grows (shrinks) nearly linearly with small increases or decreases in 

the quantity or quality of a resource. However, the WTP function in equation [3] implies that the marginal 

WTP for water quality improvements increases rapidly as D approaches zero, i.e., the limit of ∂WTP/∂D

as D goes to zero is ∞.

Studies in the metadata tend to value substantial WQI improvements, in many cases equivalent to a step 

or more on the WQL (i.e., 20 points or more on the WQI) (see Figure 2 in Section 3.3). The mean WQI 

change in the metadata is 21.3 and the minimum is 2.5. However, future water regulations may result in 

smaller improvements in the WQI for some local areas. Therefore, it is important that the Agency be able 

to use MRM1 to accurately estimate WTP for small water quality improvements as well.

There are a variety of options to address this concern. One possibility is to use the MRM1 itself to predict 

WTP in the orange of water quality changes observed in the underlying metadata (i.e., D>=2.5), and to 

use a linear interpolation of model results to predict WTP for water quality changes below the observed 

range (i.e., D<2.5). Figure 10 illustrates the interpolation approach. The red line is total annual household 

WTP for various WQI improvements for BG A from Section 6.1 as predicted by MRM1. The curve is 

steep near the origin, resulting in high marginal WTP for small WQI improvements. The dotted line is 

interpolated WTP values for water quality changes less than 2.5. Applying this approach to the benefit 

transfer example in 6.1 results in household WTP of $9.70 and total WTP of $1,939 for BG A, and 

household WTP of $19.17 and total WTP of $5,752 for BG B. These interpolated estimates for BG A and 

BG B are 62 percent and 42 percent less than estimates predicted directly by MRM1 (Table 13).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 47

Figure 10. Illustration of WTP for WQI Improvements for BG A (in red). The 

dotted line shows interpolated WTP for WQI improvements less than 2.5.

The second limitation of MRM1is that the WTP function in equation [3] does not satisfy the “adding-up” 

condition, which Diamond (1996) describes as a necessary condition for internal consistency in benefit 

estimation, with particular emphasis on stated preference studies. The adding-up condition states that for 

all Q0 < Q1 < Q2:

WTP(Y,Q0,Q2

) = WTP(Y,Q0,Q1) + WTP(Y − WTP(Y,Q0,Q1),Q1,Q2) [4]

In words, the WTP for a change from state 0 to 2 conditional on baseline income must equal the WTP for 

a change from 0 to 1 conditional on baseline income, plus the WTP from state 1 to 2 conditional on the 

income remaining after paying for the change from 0 to 1. As noted by Carson (2012, p. 36), “such a test 

is logically correct given its assumptions,” but often fails to hold for both market and non-market values 

in empirical settings. 

The WTP function given in equation [4] does not in general satisfy the adding-up condition because

Y

βY(Q2 − Q0)βD ≠ YβY(Q1 − Q0)βD + (Y − αYβYQ0

βQ(Q1 − Q0)

βD)

βY

(Q2 − Q1)

βD [5]

The problem can be seen more readily by abstracting from the income effect (which is often small enough 

to be neglected in practical applications):

(Q2 − Q0)

βD ≠ (Q1 − Q0)βD + (Q2 − Q1)βD [6]

Hence, benefit transfers resulting from MRM1 generally will not satisfy the adding-up condition. 

In cases where a single policy action is expected to lead to significant water quality improvements that 

attain a water quality objective MRM1 would provide defensible household WTP estimates. For example, 

MRM1 could be used for a cost-benefit analysis of a TMDL that is expected to result in an impaired 

waterbody attaining all of the applicable designated uses. This is because the water quality goals would be 

achieved and no further policy actions would be expected. However, the Agency often proposes national 

regulations that address discharges from a single industrial category, such as discharges from steam 

electric power plants, or from municipal sources, like combined sewer overflows. These sector-by-sector 

national regulations are a necessary incremental approach to implementing the Clean Water Act, yet they 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 48

often result in small improvements to many waters across the country. It is a long standing practice that 

the Agency takes partial credit for partial improvements on the WQL resulting from individual 

regulations. As one regulation may result in a small water quality change, the cumulative effect of 

multiple regulations could well be significant. If EPA is going to be able to account for the economic 

benefits associated with these many small incremental changes it must have a model that can reliably 

estimate numerous small changes to water quality in an additive way.

7 An Alternative Approach: MRM2

To address limitations of MRM1 in terms of satisfying adding-up and predicting very large WTP for very 

small water quality changes, the WTP function based on MRM1 can be specified directly in terms of 

marginal willingness to pay. Then the estimated marginal willingness to pay function can be integrated to 

to get a WTP function that automatically satisfies the adding-up condition, i.e.,

WTPj

(Q0,Q2) = ∫

WTPj

Q

dQ Q2

Q0

= ∫

WTPj

Q

dQ Q1

Q0

+ ∫

WTPj

Q

dQ Q2

Q1

[7]

To implement this approach, EPA used the following estimating equation, referred to as MRM2 in this 

report:

ln (

WTPjs

Djs ) = QjsβQ + ln(Yjs) βY + X1jsβX1 + ln(X2js)βX2 + εjs [8]

where, consistent with equation [1] from Section 4, Yjsis the household income for the sampled 

population and X1js and X2js are vectors of variables measuring environmental conditions and change, 

research methods, temporal factors, and other factors expected to influence value surfaces across 

observations and/or sites. To implement the model, EPA approximated marginal willingness to pay, 

WTPjs/Djs, with the stated willingness to pay amounts, WTPjs, divided by the water quality change, Djs

(where Djs =Q1j s- Q0js), evaluated at the mid-point of the range over which water quality was changed, 

Qjs= (1/2)(Q0js + Q1js), for each observation j and study s. The vectors βD, βQ0, βY, βX1, and βX2 represent 

conforming vectors of parameters to be estimated.

This approximation could be avoided by integrating the marginal WTP function in equation [8] over the 

range of the water quality index change to get an exact expression for the incremental WTP for each 

observation:

ln(WTPjs) = ln(Yjs) βY + X1jsβX1 + ln(X2js) βX2 + ln [

1

βQ

(e

βQQ1js − eβQQ0js )] + εjs [9]

This expression clarifies how stated WTP is assumed to depend, according to equation [8], on the initial 

and final water quality levels. However, equation [9] is nonlinear in the parameters so would require a 

custom nonlinear least squares or maximum likelihood routine for estimation. EPA would not expect 

large differences between these alternative estimating equations, but for completeness the Agency may 

pursue this elaboration in follow up work. For all results presented in the present report, EPA used the 

approximation in [8] as the estimating equation and the approximation described in the preceding 

paragraph to allow estimation of MRM2 using ordinary or weighted least squares.

7.1 Estimation and Model Selection Approach

EPA began by estimating a preliminary model that included the same set of regressors that appear in 

MRM1 except lnQ0s and lnDj, which were excluded for the reasons explained above. To estimate the 

parameters of the candidate model specifications EPA used ordinary least squares (OLS) and weighted 

least squares (WLS). When using WLS, each observation was weighted by the inverse of the number of 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 49

observations drawn from its parent study. This approach gives each study rather than each observation 

equal weight in the regression (e.g., Mrozek and Taylor 2002, Van Houtven et al. 2007).22 This is distinct 

from MRM1, which gives each observation equal weight. Both weighted and unweighted MRMs are 

common in the published literature, with no consensus on which approach is more appropriate (Johnston 

et al. 2005).

In all cases, EPA reports cluster-robust standard errors (Cameron and Miller 2013) and clusteredbootstrap standard errors (Ren et al. 2010). The Agency calculated the R

2

for the WLS model using the 

sum of weighted squared residuals based on the weighted data, i.e.,

RWLS

2 = 1 −

∑ wj(yj − xjβ̂WLS)

2

j

∑ (√wjyj− (

1

N

∑j √wjyj))

2

j

[10]

RWLS

2

indicates the proportion of the variation of the weighted dependent variable that is explained by the 

weighted independent variables. This quantity is maximized by WLS estimation and so is comparable 

across alternative WLS specifications, but it is not directly comparable to the R

2

of the unweighted OLS 

model (Greene 1997 p 257). For a more comparable statistic, EPA also calculated a pseudo R

2

for the 

WLS model, R̃WLS

2

, which is based on the unweighted data and so is directly comparable to the R

2

from 

an OLS model (Willett and Singer 1988):

RWLS

2 = 1 −

∑ (yj − xjβ̂WLS)

2

j

∑ (yj−

1

N

∑j yj)

2

j

[11]

If the observation weights were all equal—i.e., if all studies supplied the same number of observations—

then the weighted least squares estimator would collapse to ordinary least squares and both RWLS

2

and 

R̃WLS

2 would be equal to ROLS2

. In the present dataset the weights are not all equal, so R̃WLS

2 will necessarily 

be less than ROLS

2

but RWLS

2

could be larger or smaller than ROLS

2

.

EPA also presents measures of out-of-sample predictive accuracy for all models based on their respective 

leave-one-out cross-validation performance. For OLS models the cross-validation measure of fit is:

RCV

2 = 1 −

∑ [yj − ŷj(β̂−j)]

N 2

j=1

∑ (yj −y)

N 2

j=1

[12]

For WLS models the unweighted yj’s would be replaced by their weighted counterparts, √wjyj, in 

equation [12]. RCV

2

provides the basis for model selection in the step-wise regression approach described 

above for both OLS and WLS models.

 

22 Both OLS and WLS should be consistent, but which will be more efficient? This will depend on how each primary 

investigator chose which estimate(s) to report from among the possibly many more that were generated in the course of each 

study, and on the relative magnitudes of the within- and across-study error variances. For example, if all of the studies were 

based on comparably large datasets (so that their sampling error variances are roughly equal) and if the reported estimates 

are encompassing subset means of all underlying (including possibly many unreported) estimates generated by each study, 

then the means of each study-level group of estimates would be comparably precise and so WLS would be more efficient 

than OLS. Alternatively, if the reported estimates are random draws from the underlying estimates, and if the within study 

error variances are sufficiently large relative to the across study error variance, then each reported estimate would be 

comparably precise and so OLS may be more efficient than WLS. It should be possible to use a random effects model to 

estimate the within- and across-study error variances and thereby develop an estimator that may be more efficient than either 

OLS or WLS, but EPA has left that task for follow-up work. In this section, the Agency focuses exclusively on the OLS and 

WLS estimators.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 50

Because MRM1 and MRM2 use different dependent variables, their respective R

2

values are not directly 

comparable. This is distinct from the non-comparability between R

2

values of OLS and WLS models. To 

facilitate comparisons of fit between MRM1 and MRM2 estimated by either OLS or WLS, EPA

calculated the squared correlation coefficients between the observed and predicted WTP values, rWTP

2

, for 

all models.23 To generate predictions of WTP using MRM2, the Agency plugged the estimated model 

parameters into the benefit transfer function given by equation [9].

7.2 Results

Results for the full specification of MRM2 estimated by OLS are presented in Table 14 and Figure 11. 

Table 14 shows that a majority of the variables included in the model are statistically significant at the 5% 

level according to the cluster-robust standard errors. However, the bootstrapped confidence intervals are 

roughly twice as wide as those implied by the cluster-robust standard errors, so by the bootstrap measures 

far fewer variables are statistically significant. For data sets of small to modest size such as this one, 

bootstrapped statistics may be viewed as more reliable because they are nonparametric and do not rely on 

asymptotic assumptions (Efron 1979, Efron 2003, Horowitz 2003). However, it is not clear to us why the 

bootstrapped confidence intervals are so much larger than the cluster-robust standard errors in this case, 

so EPA reports both measures for comparison and we flag this discrepancy for closer examination in 

follow-up work.

Measures of model fit are shown at the bottom of Table 14. The RCV

2

value is significantly smaller than 

the R

2

value for the model. This suggests that the out-of-sample predictive performance of the model is 

substantially worse than its within-sample fit, which is to be expected. The predictive performance of the 

model can be improved by excluding (or refraining from including) those explanatory variables that 

reduce the precision more than they increase the accuracy of the predictions.24

Table 14 shows scatterplots of the predicted versus observed observations, as the approximated marginal 

WTPs in panel (a) and as stated WTPs in panel (b). The latter were calculated using the estimated model 

parameters in the benefit transfer function given in equation [9], and the observed explanatory variable 

values. The squared correlation coefficient between the predicted and observed WTP values was rWTP

2 = 

0.528. 

There is an obvious outlier in panel (b), which is observation 17. This is one of two observations from the 

study by Hayes et al. (1992). The other observation (number 16) differs from 17 in only two dimensions: 

‘swim_use’ = 1 for observation 16 rather than 0, and Q = 60 for observation 16 rather than 70. So in the 

marginal WTP-Q space, shown in panel (a) of Figure 11, observation 17 does not appear as an obvious 

outlier. However, ΔQ(16) = 20 while ΔQ(17) = 40, so the predicted WTP for 17 is much larger than that 

for 16, as seen in panel (b) of Figure 11. To examine the influence of observation 17 on the estimation 

results EPA re-estimated the model excluding that observation. The full model results with observation 17 

excluded are presented in Appendix C. The fit of the model in marginal WTP-Q space increases only 

marginally without observation 17 (from r2= 0.612 to 0.618), but the fit in WTP-Q space increases 

significantly (from rWTP

2 = 0.528 to 0.574). The Agency might conjecture that this ‘stray’ prediction 

would be tamed if EPA were to fit the parameters using equation [9] as the estimating equation directly, 

using non-linear least squares or maximum likelihood, so the Agency highlights that task for follow-up 

 

23 EPA notes, however, that this comparison is unfair to the WLS models, which minimize the residual sum of squared errors 

between the within-study average observations and predictions rather than the individual observations and predictions. To 

balance the comparisons, EPA also should examine the rWTP

2 values based on the study means for all models.

24 This is the goal of the step-wise regressions, the results of which are discussed below.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 51

work. In the meantime, since EPA has so far relied exclusively on estimating equation [7], all results in 

the remainder of this section use all of the data including observation 17.

Because the dependent variable of MRM2 is (an approximation of) marginal WTP, if the coefficient on Q

were equal to zero this would imply a flat demand curve, i.e., marginal WTP for water quality 

improvements would not depend on the level of water quality. The coefficient on Q is negative in both the 

OLS and WLS models, indicating a downward sloping demand curve. The Q coefficient is statistically 

significant according to both the asymptotic cluster-robust standard errors and the bootstrap confidence 

intervals in the OLS model, but only according to the former statistic in the WLS model. To illustrate the 

shape of the demand curve, panels (c) in Figure 11 and Figure 12 show the marginal WTP curve over the 

entire domain of the water quality index for the full specification of MRM2 estimated by OLS and WLS. 

To create these curves, the geo-spatial variables were set to their mean values in the dataset and the 

remaining variables in X1 and X2 were specified as indicated in Table 11. The dotted red lines are the 

corresponding 2.5th, 50th, and 97.5th percentiles of bootstrapped marginal WTP predictions at the fixed 

values of X1 and X2.

As seen in panel (c) of Figure 11, the OLS estimates using the full dataset suggest that, on average, 

marginal WTP declines from about $3 per index point at Q = 0 to around $0.5 per index point at Q = 100. 

For comparison, in an earlier meta-analysis study based on the 10-point water quality ladder, Van 

Houtven et al. (2007) estimated an average WTP by users of $56-58 using a linear function and $32-37 

using a log-linear function per point on the 10-point scale.

Results for the preliminary specification of MRM2 estimated by WLS are presented in Table 15 and 

Figure 12. The results are broadly similar to the OLS results with a few notable differences. The insample fit of the model to the observed marginal WTPs is lower, as it must be since here EPA has

minimized the squared weighted residuals, but the auxiliary fit of the predicted stated WTPs is nearly as 

good: the rWTP

2

values for the OLS and WLS models were 0.528 and 0.518, respectively. The marginal 

WTP curve, shown in panel (c) of Figure 12, is also somewhat lower than that estimated by OLS at low 

values of Q, starting at just over $2 rather than roughly $3.

One of the notable differences between MRM1 and MRM2 is that in MRM2 the log of median income is 

generally not statistically significant (it was also near the margin of being insignificant in MRM1). This is 

the case in the full specification estimated by OLS and WLS, as shown in Table 14 and Table 15.

25 The 

estimated coefficient is positive, suggesting that water quality is a normal good, but it is estimated very 

imprecisely. It may be instructive to conduct a power analysis for this regression coefficient (Dupont et al.

1998), so EPA flagged this issue for follow-up work. 

To compare benefit transfer predictions by MRM1 and MRM2, the Agency used the estimated parameters 

of MRM2 in the benefit transfer function given in equation [9] to predict stated WTPs for the two 

hypothetical examples described in Table 13 (with X1 and X2 specified at the same fixed values used to 

draw the marginal WTP curves, as indicated above). These results are shown in Table 16Error! 

Reference source not found.. Example A involves a change from 47.8 to 49.5 on the WQI, and 

Example B involves a change from 47.0 to 49.4. Using MRM2 to estimate the WTP for these changes at 

the specified values of X1 and X2 indicated in the previous paragraph, for Example A, the mean is $2.4 per 

household per year using OLS and $2.2 using WLS, and for Example B, the mean is $3.4 per household 

per year using OLS and $3.1 using WLS. 

 

25 This variable also only rarely appears in the final models selected by step-wise regression, as shown later, in Figures 6 and 8.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 52

Table 14. MRM2 Preliminary Specification Estimated by OLS. In the far right columns, a single * 

indicates b/standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not 

include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept 0.150 4.580 -18.000 15.300

ce 0.530 0.306 -0.642 1.610

thesis 0.631 0.245 -0.197 1.800 *

lnyear -0.356 0.101 -0.928 -0.074 * **

volunt -1.470 0.223 -2.150 -0.681 * **

outliers_trim -0.494 0.174 -0.956 0.202 *

non_param -0.577 0.118 -0.917 0.059 *

non_reviewed -0.578 0.198 -1.380 -0.031 * **

lump_sum 0.879 0.189 0.315 1.520 * **

wtp_median -0.348 0.340 -1.340 0.646

northeast 1.060 0.319 0.134 2.240 * **

central 0.545 0.160 -0.241 1.180 *

south 1.350 0.170 0.654 2.020 * **

nonusers -0.454 0.150 -0.802 -0.156 * **

lnincome 0.212 0.418 -1.100 1.840

mult_bod -0.680 0.204 -1.480 -0.004 * **

river -0.092 0.150 -0.662 0.500

swim_use -0.384 0.183 -0.861 0.105 *

gamefish 0.526 0.306 -0.367 1.350

boat_use -0.715 0.225 -1.330 0.014 *

ln_ar_agr -0.344 0.099 -0.628 0.114 *

ln_ar_ratio -0.075 0.025 -0.148 0.055 *

sub_proportion 1.140 0.236 0.029 1.900 * **

Q -0.017 0.005 -0.034 -0.004 * **

R2 0.612

RCV2 0.421

rWTP2 0.528
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 53

Figure 11. MRM2 Preliminary Specification Estimated by OLS. (a) Scatter plot of predicted vs 

observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal willingness 

to pay curve. Demand curve shifters are set as described in the main text. The black line is the point 

estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th 

percentiles.

(a) (b)

(c)
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 54

Table 15. MRM2 Preliminary Specification Estimated by WLS. In the far right columns, a single * 

indicates b/standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not 

include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept -1.040 5.300 -19.900 16.400

ce 0.377 0.322 -0.725 1.550

thesis 0.866 0.270 -0.240 1.880 *

lnyear -0.412 0.130 -0.912 -0.033 * **

volunt -1.390 0.239 -2.150 -0.638 * **

outliers_trim -0.367 0.201 -0.976 0.298

non_param -0.408 0.121 -0.845 0.105 *

non_reviewed -0.709 0.199 -1.390 -0.077 * **

lump_sum 0.843 0.188 0.189 1.540 * **

wtp_median -0.161 0.331 -1.370 0.926

northeast 1.180 0.337 0.177 2.340 * **

central 0.561 0.214 -0.240 1.260 *

south 1.400 0.235 0.640 2.130 * **

nonusers -0.586 0.146 -0.910 -0.178 * **

lnincome 0.333 0.473 -1.180 2.040

mult_bod -0.827 0.211 -1.710 -0.019 * **

river -0.079 0.174 -0.719 0.477

swim_use -0.234 0.177 -0.747 0.301

gamefish 0.233 0.387 -0.678 1.460

boat_use -0.725 0.237 -1.480 0.006 *

ln_ar_agr -0.271 0.110 -0.589 0.132 *

ln_ar_ratio -0.034 0.035 -0.149 0.072

sub_proportion 1.100 0.272 -0.097 1.950 *

Q -0.015 0.006 -0.035 0.001 *

R2 0.745

RCV2 0.499

RWLS2 0.548

rWTP2 0.518
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 55

Figure 12. MRM2 Preliminary Specification Estimated by WLS. (a) Scatter plot of predicted vs 

observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal willingness 

to pay curve. Demand curve shifters are set as described in the main text. The black line is the point 

estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th 

percentiles.

(a) (b)

(c)
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 56

Table 16. WTP [$/yr/household] for Examples A and B from Table 13 Calculated using the Full 

Specification of MRM2 Estimated by OLS and WLS.

A Confidence intervals are based on 1000 bootstrap 

samples conditional on full model specification.

Parameter Q0 Q1

OLS WLS

Mean 2.5th 50th 97.5th Mean 2.5th 50th 97.5th

Example A 47.81 49.50 3.42 0.89 3.40 10.61 2.54 0.82 2.69 8.35

Example B 47.00 49.44 5.36 1.53 5.29 17.62 4.06 1.33 4.32 12.91

(A) Consistent with WTP predictions in Table 13, EPA adds half of the model residual variance to the natural log of the 

dependent variable prior to calculating the exponential. This adjustment was not included in the cross-validation 

predictions in Figure 11and Figure 12, but this should not have a significant effect on the interpretation of crossvalidation results.

8 Model Limitations and Uncertainty 

Because both MRM1 and MRM2 are drawn from the same underlying metadata, they are subject to the 

same limitations related to data availability. However, the two model structures differ, leading to 

different advantages and disadvantages. In general, MRM2 may be viewed as making somewhat stronger 

assumptions (that may or may not coincide with patterns in the underlying metadata), in return for a set of 

theoretically-desired properties in resulting benefit estimates. Two important limitations of MRM1 were 

discussed above in Section 6.2. The primary limitation of MRM2 is that it does not fit the data as well as 

MRM1. For example, for the squared correlations between the observed and predicted WTP values for 

MRM1 and MRM2 (when both are estimated by OLS) are 0.633 and 0.528, respectively. Additional 

work may be warranted to more fully compare and contrast the models when estimated using similar 

assumptions—e.g., weighted versus unweighted, random effects versus OLS—but we conjecture that the 

superior statistical fit of MRM1 will be robust to these additional analyses. 

MRM2 was a late addition to this report, so EPA views the results presented in this section as still 

preliminary since they have been only partially stress tested. EPA intends to apply more sophisticated 

estimation approaches to MRM2 in follow up work. In particular, a random effects model could be used 

to estimate MRM2, with or without weighting by the inverse number of observations per study. EPA will 

also consider estimating MRM2 via nonlinear least squares using equation [9] as the estimating equation 

directly. Instead of the step-wise regression approach used here, other types of shrinkage estimators could 

be used such as ridge, lasso, or stage-wise regression (Tibshirani 1996; Efron et al. 2004). Additional 

sensitivity analyses also could be conducted to examine the influence of each study and possibly other 

aspects of the model specification that might substantially influence the results. One sensitivity analysis 

that EPA has conducted was to estimate models with squared and cubed Q terms to allow a more flexible 

shape for the marginal WTP curve. The model fits were not significantly improved according to any of 

measures of fit EPA considered, and the marginal WTP curves were broadly similar but they exhibited 

more idiosyncratic curvature and had wider confidence intervals.

Aside from the statistical issues that were the main focus of this section, it is also important to bear in 

mind that the distance between the surveyed households and the affected water bodies is not well 

measured by any of the explanatory variables in the model. The closest proxy is the log of the ratio of the 

sampled area to the area of the counties containing the affected water bodies (ln ar ratio). However, this 

is at best a crude proxy for distance and in any case is not intended to be used to delineate the area over 

which households are assumed to ascribe value to water quality improvements. As discussed Section 6.1, 

EPA intends to use an exogenously specified threshold distance cutoff when using the meta-regression 

models described in this report for benefit transfers.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 57

Finally, the relatively large (positive) magnitude of the parameter estimate for the southern U.S. regional 

dummy variable (south) leads EPA to question the appropriate interpretation of this effect. While it is 

theoretically possible that WTP for water quality changes is substantially higher in the south, the 

magnitude of the effect suggested by the model seems unlikely from an intuitive perspective. As 

suggested above, it is possible that spurious, unexplained factors influence the magnitude of this 

parameter in the present model. However, assessments of preliminary model runs suggest that this effect 

is relatively robust given the present data and selection of variables available. Nonetheless, EPA 

recommends that the magnitude of the predicted shift in WTP associated with the south region should be 

viewed with caution.

The use (and interpretation) of all reduced-form MRMs for benefit transfer is also subject to the 

constraints and concerns expressed elsewhere in the literature (e.g., Desvousges et al., 1998; Poe et al., 

2001; Vandenberg et al. 2001; Smith and Pattanayak 2002)

9 Example WTP Predictions for MRM1 and MRM2

Table 17 presents predicted annual household WTP for various WQI changes and baselines using MRM1 

and MRM2. Variable values are set to match the means from the meta-data, with the exception of 

baseline WQI and WQI change, which vary across predictions. MRM1 predictions include a linear 

adjustment for WQI improvements less than 2.5, as described in Section 6.2. For MRM2, WTP 

predictions are the product estimated WTP per unit WQI change and the WQI change. As evident in 

Figure 14, WTP predictions from the two models diverge dramatically for larger WQI changes, with 

MRM2 predicting higher WTP for large changes (e.g., WQI change = 40).

Table 17 also shows that MRM1 generally will not satisfy the adding-up condition while MRM2 

generally will. For example, consider water quality improvements from 20 to 30 to 40 on the WQI. 

MRM2 gives WTP(2030) = $79.74, WTP(3040) = $67.41, and WTP(2040) = $146.62. The first 

two of these add up to the third (nearly exactly, presumably to within rounding error), as they should. On 

the other hand, MRM1 gives WTP(2030) = $74.35, WTP(3040) = $73.26, and WTP(2040) = 

$91.49. The first two of these add up to an amount significantly greater than the third, thereby clearly 

violating the adding-up condition.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 58

Table 17. Predicted WTP from MRM1 and MRM2 for Various WQI Baselines and WQI 

Changes (2007$)A,B,C

Baseline 

WQI

WQI Change

0.01 0.1 0.5 1 5 10 20 40

MRM1 (dependent variable is lnWTP) with linear adjustment for WQI improvements < 2.5

10 $0.20 $2.01 $10.07 $20.14 $61.96 $76.25 $93.83 $115.46

20 $0.20 $1.96 $9.82 $19.64 $60.42 $74.35 $91.49 $112.58

30 $0.19 $1.94 $9.68 $19.35 $59.54 $73.26 $90.15 $110.94

40 $0.19 $1.92 $9.58 $19.15 $58.92 $72.50 $89.21 $109.78

50 $0.19 $1.90 $9.50 $19.00 $58.44 $71.91 $88.49 $108.90

60 $0.19 $1.89 $9.44 $18.87 $58.05 $71.44 $87.91 $108.18

70 $0.19 $1.88 $9.38 $18.77 $57.73 $71.04 $87.42 n/a

80 $0.19 $1.87 $9.34 $18.67 $57.45 $70.70 $86.99 n/a

90 $0.19 $1.86 $9.30 $18.60 $57.21 $70.39 n/a n/a

MRM2 (dependent variable is ln(WTP per 1-point WQI change)

10 $0.10 $1.03 $5.11 $10.17 $49.18 $94.32 $173.45 $293.25

20 $0.09 $0.87 $4.32 $8.60 $41.58 $79.74 $146.62 $247.90

30 $0.07 $0.73 $3.65 $7.27 $35.15 $67.41 $123.95 $209.56

40 $0.06 $0.62 $3.09 $6.15 $29.71 $56.98 $104.78 $177.15

50 $0.05 $0.52 $2.61 $5.20 $25.12 $48.17 $88.58 $149.76

60 $0.04 $0.44 $2.21 $4.39 $21.23 $40.72 $74.88 $126.60

70 $0.04 $0.37 $1.86 $3.71 $17.95 $34.42 $63.30 n/a

80 $0.03 $0.32 $1.58 $3.14 $15.17 $29.10 $53.51 n/a

90 $0.03 $0.27 $1.33 $2.65 $12.83 $24.60 n/a n/a

(A): Variable values are set to match the means from the meta-data.

(B): “n/a” indicates that WTP was not estimated because (Baseline WQI + WQI Change) >100.

(C) Consistent with WTP predictions in Table 13, EPA adds half of the model residual variance to the natural 

log of the dependent variable prior to calculating the exponential.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 59

Figure 13. Predicted WTP from MRM1 and MRM2 for Various WQI Baselines and 

WQI Changes (2007$). “Base” refers to the Baseline WQI value. MRM1 predictions 

include a linear adjustment for WQI improvements less than 2.5, as described in Section 

6.2. (a) MRM1 with linear adjustment for WQI improvements < 2.5. (b) MRM2. 

(a)

(b)

10 Overfitting and Step-wise Regression: MRM2

The full specification of MRM2 contains 25 degrees of freedom, and the dataset includes 140 

observations from 51 studies. The studies are assumed to be independent, but EPA expects that when 

multiple observations are drawn from a single study they will be correlated, so there are somewhere 

between 51 and 140 effective observations, or between only 2 and 6 effective observations per degree of 

freedom. With so few observations per parameter, EPA has some fear that the preliminary model based 

on the full specification may suffer from overfitting and therefore have substantially lower out-of-sample 

predictive power than indicated by its nominal (e.g., Breiman 1992).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 60

To address this concern, EPA used forward and backward step-wise regression and leave-one-out cross 

validation (CV) to select variables for inclusion in a more parsimonious model. This approach selects the 

model that minimizes the quadratic loss from predictive errors, i.e.,

β̂ ∗ = arg min {

1

N

∑ [yj − ŷj(β̂

−j)]

N 2

j=1

} [13]

where ŷj(β̂

−j) is the predicted value of based on parameters fit to all data excluding observation j

(Hastie et al. 2009 p 241-249). The intuition for this approach follows from the tradeoff between bias and 

variance (Breiman 1992). Beginning with a null model—e.g., including only a constant term in a standard 

linear regression—out-of-sample predictive performance typically will initially increase as regressors 

drawn from a set of candidate explanatory variables are added to the model. But as more regressors are 

added to the model, the variance of the model predictions will eventually become so large that the mean 

squared error of the out-of-sample predictions will increase even as the mean squared error of the withinsample predictions and the bias of the out-of-sample predictions continue to decrease. Minimizing out-ofsample predictive errors is a natural objective for meta-regression models intended to be used for benefit 

transfers, which by definition involve out-of-sample predictions.

To correct for the over-optimism of the nominal standard errors associated with the model selected by 

step-wise regression, EPA estimated standard errors by bootstrapping the data 500 times and re-running 

the entire model selection algorithm on each bootstrap sample. As explained by Hastie et al. (2009), “In 

general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of 

modeling steps.” The sampling distributions estimated in this way account for the uncertainty associated 

with selecting the final model specification plus the sampling uncertainty conditional on the selected 

specification. Nominal standard error estimates reflect only the latter component of the un- certainty. By 

bootstrapping the entire model selection algorithm, EPA is able to avoid the over-optimistic (biased low) 

estimates of standard errors produced by the nominal calculations that ignore the degrees of freedom used 

up by the data mining process (Ye 1998).

Results for the MRM2 final specification selected by step-wise regression are shown in Table 18 and 

Table 19. The most notable aspects of these results are: 1) only a handful of variables are dropped from 

the full specification, indicating a modest amount of over-fit, 2) the nominal standard errors are much 

more optimistic than the confidence intervals created by bootstrapping the entire model selection and 

estimation routine, 3) the bootstrapped confidence intervals and the frequency distributions of the 

coefficient estimates are very similar to the results of the robustness analysis for MRM1, and 4) the 

estimated demand curves for the final specifications are very similar to those for the full specifications, 

though the 95% confidence intervals are substantially wider.

Panels (d) in Figure 14 and Figure 15 show the training (within-sample) and test (out-of-sample) error 

curves for the MRM2 step-wise OLS and WLS regression models, respectively. Both the forward and 

backward step-wise paths are plotted on the figure, so each line appears to be doubled over. The training 

error curves decrease monotonically as more variables are included in the model, as they must, but the 

test error curves reach a minimum before all variables are included. The final selected models correspond 

to the minimum points on the test error curves.

Appendix D presents histograms of the coefficient estimates of the final specifications based on 500 

bootstrap iterations of the full step-wise model selection and estimation routine. These are the basis of the 

bootstrap confidence intervals shown in Table 18 and Table 19. Only two variables appear in the final 

specifications in 97.5% or more of the bootstrap iterations: ‘South’ in the OLS model and ‘voluntary’ in 

the WLS model. These results are qualitatively similar to the robustness test applied to MRM1, which are 

described in Section 5. There it was shown that the coefficients on volunt and south were robust according 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 61

to Leamer’s extreme bounds criteria, and thesis, volunt, lnyear, nonuser, south, river, ln ar agr, and ln 

quality ch (not included in MRM2) were robust according to Sala-i-Marin’s extreme bounds criteria. Here 

EPA used a 95% confidence interval, which is intermediate between the criteria of Leamer and Sala-imartin. Leamer’s criteria is equivalent to a 100% confidence interval, and Sala-i-Martin’s criteria is based 

on a 90% confidence interval, so the results reported in Table 18 and Table 19 for MRM2 are broadly 

consistent with the robustness analysis applied to MRM1. Accounting for this additional component of 

uncertainty greatly increases the confidence intervals on the individual coefficient estimates as well as the 

estimated marginal WTP curve conditional on a fixed X1 and X2, as shown in panel (c) of Figure 14 and 

Figure 15. The high (essentially) flat upper 95% confidence interval bound shown there arises from the 

2.5% or more cases where Q was not included in the final model specification.

That only a few variables are robust in this sense does not change the fact that the final specifications in 

Table 18 and Table 19 are the ones with maximum estimated out-of-sample predictive performance 

(among those considered along the forward and backward step-wise regression paths). The main take 

home message of the bootstrap confidence intervals shown in Table 18 and Table 19 and the robustness 

analysis results for MRM1 is that the nominal standard errors and confidence intervals typically reported 

with an author’s preferred model specification, which is often selected through a series of supervised 

estimations and statistical tests, are likely to be highly optimistic. That is, if the study could be repeated 

very many times, EPA should not expect to see a variable survive the specification selection gauntlet 

nearly as often as suggested by its nominal significance level reported in the original study.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 62

Table 18. MRM2 Final Specification Selected by Step-Wise OLS. In the far right columns, a single * 

indicates b/standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not 

include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept 2.250 0.358 -21.200 18.100 *

ce 0.423 0.334 -0.887 1.610

thesis 0.619 0.213 0.000 1.750 *

lnyear -0.339 0.093 -0.960 0.000 *

volunt -1.550 0.210 -2.340 -0.497 * **

outliers_trim -0.501 0.173 -1.110 0.000 *

non_param -0.543 0.112 -1.010 0.000 *

non_reviewed -0.593 0.193 -1.390 0.000 *

lump_sum 0.802 0.174 0.000 1.560 *

wtp_median 0.000 0.000 -1.380 0.727

northeast 1.150 0.298 0.000 2.250 *

central 0.538 0.184 0.000 1.260 *

south 1.350 0.154 0.000 2.060 *

nonusers -0.417 0.139 -0.799 0.000 *

lnincome 0.000 0.000 -1.370 2.180

mult_bod -0.659 0.186 -1.540 0.000 *

river 0.000 0.000 -0.818 0.564

swim_use -0.408 0.181 -0.916 0.228 *

gamefish 0.538 0.293 -0.769 1.430

boat_use -0.701 0.216 -1.530 0.900 *

ln_ar_agr -0.365 0.109 -0.690 0.000 *

ln_ar_ratio -0.070 0.027 -0.217 0.048 *

sub_proportion 1.200 0.193 -0.742 1.901 *

Q -0.016 0.004 -0.332 0.000 *

R2 0.604

RCV2 0.452

pseudo RWLS2 0.548

rWTP2 0.503
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 63

Table 19. MRM2 Final Specification Selected by Step-Wise WLS. In the far right columns, a single * 

indicates b/standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not 

include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept 2.450 0.670 -20.200 17.500 *

ce 0.000 0.000 -0.727 1.580

thesis 1.030 0.221 0.000 1.860 *

lnyear -0.303 0.140 -0.891 0.000 *

volunt -1.260 0.187 -2.180 -0.422 * **

outliers_trim 0.000 0.000 -1.080 0.223

non_param -0.411 0.119 -0.882 0.000 *

non_reviewed -0.696 0.192 -1.410 0.000 *

lump_sum 0.646 0.166 0.000 1.450 *

wtp_median 0.000 0.000 -1.270 1.000

northeast 1.350 0.271 0.000 2.320 *

central 0.434 0.279 0.000 1.230

south 1.200 0.237 0.000 2.050 *

nonusers -0.561 0.160 -0.903 0.000 *

lnincome 0.000 0.000 -1.440 2.040

mult_bod -0.763 0.225 -1.560 0.000 *

river 0.000 0.000 -0.971 0.422

swim_use 0.000 0.000 -0.800 0.335

gamefish 0.000 0.000 -0.930 1.500

boat_use -0.909 0.249 -1.480 0.797 *

ln_ar_agr -0.214 0.106 -0.654 0.237 *

ln_ar_ratio 0.000 0.000 -0.195 0.071

sub_proportion 1.170 0.259 -0.484 1.900 *

Q -0.017 0.006 -0.052 0.000 *

R2 0.712

RCV2 0.619

pseudo RWLS2 0.415

rWTP2 0.467
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 64

Figure 14. MRM2 Parsimonious Model Selected by Step-Wise OLS Regression. (a) Scatter plot of 

predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) 

Marginal WTP curve. Demand curve shifters are set as described in the main text. The black line is the 

point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th 

percentiles. (d) Training (in-sample) and test (out-of-sample) prediction errors for step-wise model 

selection algorithm.

(a) (b)

(c) (d)
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 65

Figure 15. MRM2 Parsimonious Model Selected by Step-Wise WLS Regression. (a) Scatter plot of 

predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) 

Marginal WTP curve. Demand curve shifters are set as described in the main text. The black line is the 

point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th 

percentiles. (d) Training (in-sample) and test (out-of-sample) prediction errors for step-wise model 

selection algorithm.

(a) (b)

(c) (d)
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 66

11 Next Steps

EPA plans to conduct additional analyses and tests to further refine the meta-analysis and improve 

transfer estimates. These analyses may include:

 Incorporation of distance decay – The distance between the surveyed households and the affected 

waterbodies is not well measured by any of the explanatory variables in MRM1 or MRM2. The 

closest proxy is the ln_ar_ratio variable which is the log of the ratio of sampled area to affected 

resource area. However, this variable cannot be used to directly capture the distance of each 

household to the affected water bodies in a benefit transfer to a policy case. As a result, an 

exogenous distance cutoff (see the example in Section 6.1) or distance decay function will still be 

needed for practical benefit transfer applications. Substantial GIS analysis of study-specific 

spatial features would be required to directly incorporate distance decay by calculating distances 

between sampled populations and affected waterbodies. Options might include calculating 

distance as the centroid of the respondents for each study or calculating distance for each Census 

BG in the sampling frame and calculating a population-weighted average across blocks. The 

distance decay analysis may also include estimating distance and WQI values for substitute 

bodies within a specified range (e.g., within 100 miles of the BG).

 Bayesian Models – It may be useful to explore approaches for using Bayesian modeling results in 

benefit transfers. Meta-analysis using Bayesian modeling techniques may offer additional 

insights through: (1) incorporation of qualitative information about some of the parameters (i.e., 

informative priors), (2) estimation of meta-regression for subsets of studies defined by the type of 

water resource (e.g., rivers, lakes, freshwater, estuaries), and (3) assessment of the impact of 

sampling uncertainty in the study-specific estimates on the precision of the parameter estimates. 

EPA did preliminary testing of Bayesian versions of MRM1 considering these three factors and 

found a lot of consistency between the Bayesian meta-regression estimates and classical metaregression estimates. Bayesian modeling offers additional advantages in accounting for the 

extrapolation uncertainty that comes from the model choice (including imposition of the theorybased restrictions, as was done in MRM2). In that, Bayesian model averaging techniques can be 

used to adequately represent the uncertainty associated with the out-of-range extrapolations, 

which are heavily influenced by the assumptions about the model structure.

12 References

Achen, C.H. 2005. “Let’s Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong.” 

Conflict Management and Peace Science, 22(4): 327-339.

Aiken, R.A. 1985. Public Benefits of Environmental Protection in Colorado. Masters thesis, Colorado 

State University.

Anderson, G.D. and S.F. Edwards. 1986. “Protecting Rhode Island’s Coastal Salt Ponds: An Economic 

Assessment of Downzoning.” Coastal Zone Management, 14( 1⁄2): 67-91.

Azevedo, C., J.A. Herriges, and C.L. Kling. 2001. “Valuing Preservation and Improvements of Water 

Quality in Clear Lake.” Staff Report 01-SR 94, Center for Agricultural and Rural Development (CARD), 

Iowa State University.

Banzhaf, H.S., D. Burtraw, S. Chung, D.A. Evans, A. Krupnik, and J. Siikamaki. 2011. "Valuation of 

Ecosystem Services in the Southern Appalachian Mountains." Paper Presented at the Annual Meeting of 

the Association of Environmental and Resource Economists (AERE).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 67

Banzhaf, Spencer H, Dallas Burtraw, David Evans, and Alan Krupnick. 2006. "Valuation of Natural 

Resource Improvements in the Adirondacks." Land Economics, 82(3): 445-464.

Bateman, I.J., B.H. Day, S. Georgiou and I. Lake. 2006. “The Aggregation of Environmental Benefit 

Values: Welfare Measures, Distance Decay and Total WTP.” Ecological Economics, 60(2): 450-460. 

Bateman, I.J. and A.P. Jones. 2003. “Contrasting conventional with multi-level modeling approaches to 

meta-analysis: Expectation consistency in U.K. woodland recreation values.” Land Economics, 79(2): 

235-258.

Bergstrom, J.C. and L.O. Taylor. 2006. “Using Meta-Analysis for Benefits Transfer: Theory and 

Practice.” Ecological Economics, 60(2): 351-360.

Bockstael, N.E., McConnell, K.E., and Strand, I.E. 1988. “Benefits from improvements in Chesapeake 

Bay Water Quality.” Department of Agricultural & Resource Economics, University of Maryland, 1988.

Bockstael, N.E., K.E. McConnell and I.E. Strand. 1989. “Measuring the Benefits of Improvements in 

Water Quality: The Chesapeake Bay.” Marine Resource Economics 6: 1-18.

Borisova, Tatiana, Alan Collins, Gerard D’Souza, Matthew Benson, Mary Leigh Wolfe, and Brian 

Benham, 2008. “A Benefit-Cost Analysis of Total Maximum Daily Load Implementation.” Journal of the 

American Water Resources Association, 44(4):1009-1023.

Boyle, Kevin J., Christopher C. Parmeter, Bent Boehlert, and Robert Paterson. 2013. “Due Diligence in 

Meta-Analyses to Support Benefit Transfers.” Environmental and Resource Economics, 55(3): 357-386.

Brander, L., and R. Brouwer. 2011. “A multi-level meta-analysis regression model of contingent values 

for water ecosystem services.” Paper presented at the European Association of Environmental and 

Resource Economists 18th Annual Conference, 29 June - 2 July 2011, Rome.

Breiman L. 1992. “The little bootstrap and other methods for dimensionality selection in regression: Xfixed prediction error.” Journal of the American Statistical Association, 87(419):7 38-754.

Cameron, T.A. and D.D. Huppert. 1989. “OLS versus ML Estimation of Non-market Resource Values 

with Payment Card Interval Data.” Journal of Environmental Economics and Management 17, 230-246.

Cameron AC, Miller DL. 2013. A practitioner’s guide to cluster-robust inference. 

http://cameron.econ.ucdavis.edu/research/Cameron_Miller_Cluster_Robust_October152013.pdf

Carson, R.T. 2012. “Contingent Valuation: A Practical Alternative when Prices Aren’t Available.” 

Journal of Economic Perspectives, 4: 27-42.

Carson, R.T., and R.C. Michell. 1993. “The Value of Clean Water: The Public’s Willingness to Pay for 

Boatable, Fishable, and Swimmable Water Quality.” Water Resources Research, 29(70), 2445-2454.

Carson, R.T., W.M. Hanemann, R.J. Kopp, J.A. Krosnick, R.C. Mitchell, S. Presser, P.A. Ruud, and V.K. 

Smith. 1994. Prospective Interim Lost Use Value due to DDT and PCB Contamination in the Southern

California Bight. Volume 2. Report to the National Oceanic and Atmospheric Administration, Produced 

by Natural Resources Damage Assessment Inc., LA Jolla, CA.

Clonts, H.A., and J.W. Malone. 1990. “Preservation Attitudes and Consumer Surplus in Free Flowing 

Rivers.” In J. Vining (ed.), Social Science and Natural Resource Recreation Management, Westview 

Press: Boulder, CO, pp. 301-317.

Collins, Alan R. and Rosenberger, R.S. 2007. “Protest Adjustments in the Valuation of Watershed 

Restoration Using Payment Card Data.” Agricultural and Resource Economics Review, 36(2): 321-335.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 68

Collins, A.R., R.S. Rosenberger and J.J. Fletcher. 2009. “Valuing the restoration of acidic streams in the 

Appalachian Region: A stated choice method.” In H.W. Thurstone, M.T. Heberling and A. Schrecongost 

(eds.), Environmental Economics for Watershed Restoration. Boca Raton, FL: CRC/Taylor Francis. 

pp.29-52.

Corrigan, J.R., C.L. Kling, and J. Zhao. 2009. "Willingness to Pay and the Cost of Commitment: An 

Empirical Specification and Test." Environmental and Resource Economics, 40: 285-298.

Croke, Kevin, R.G. Fabian, and G. Brenniman. 1986-87. “Estimating the Value of Improved Water 

Quality in an Urban River System.” Journal of Environmental Systems 16(1): 13-24.

De Zoysa, A.D.N. 1995. A Benefit Evaluation of Programs to Enhance Groundwater Quality, Surface 

Water Quality and Wetland Habitat in Northwest Ohio. Dissertation, Ohio State University.

Desvouges, W.H., K. Mathews, and K. Train. 2012. “Adequate responsiveness to scope in contingent 

valuation.” Ecological Economics, 84: 121-128.

Desvousges, W.H., F.R. Johnson, and H.S. Banzhaf. 1998., Environmental Policy Analysis with Limited 

Information: Principles and Applications of the Transfer Method. Edward Elgar, Northampton, MA.

Desvousges, W.H., V.K. Smith and A. Fisher. 1987. “Option Price Estimates for Water Quality 

Improvements: A Contingent Valuation Study for the Monongahela River.” Journal of Environmental 

Economics and Management, 14: 248-267.

Diamond, P.A. 1996. “Testing the Internal Consistency of Contingent Valuation Surveys.” Journal of 

Environmental Economics and Management, 30: 337–347.

Downstream Strategies LLC. 2008. An Economic Benefit Analysis for Abandoned Mine Drainage 

Remediation in the West Branch Susquehanna River Watershed, Pennsylvania. Prepared for Trout 

Unlimited.

Dupont W.D, and W.D. Plummer. 1998. “Power and sample size calculations for studies involving linear 

regression.” Controlled Clinical Trials, 19: 589-601.

Efron B. 1979. “Bootstrap methods: another look at the jackknife.” The Annals of Statistics, 7(1): 1-26.

Efron B. 2003. ‘Second thoughts on the bootstrap.” Statistical Science, 18(2): 135-140.

Efron B., T. Hastie, I. Johnstone, and R. Tibshirani. 2004. “Least angle regression.” The Annals of 

Statistics, 32(2): 407-499.

Farber, S., and Griner, B. 2000. “Using Conjoint Analysis to Value Ecosystem. Change,” Environmental 

Science and Technology, 34(8): 1407-1412.

Ge, J., C. Kling, and J. Herriges. 2013. “How much is Clean Water Worth? Valuing Water Quality 

Improvement Using a Meta Analysis.” Iowa State University, WP #13016, March 2013.

Ghermandi, A. and P.A.L.D. Nunes. 2013. “A Global Map of Coastal Recreation Values: Results from a 

Spatially Explicit Meta-analysis.” Ecological Economics, 86(1): 1-15.

Ghermandi, A., J.C.J.M. van den Bergh, L.M. Brander, H.L.F. de Groot, and P.A.L.D. Nunes. 2010. 

“Values of Natural and Human-made Wetlands: A Meta-analysis.” Water Resources Research, 46: 

W12516.

Greene WH. 1997. Econometric Analysis. 3rd edition. Upper Saddle River, NJ: Prentice-Hall, Inc.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 69

Hanley, N., F. Schlapfer, and J. Spurgeon. 2003. Aggregating the benefits of environmental 

improvements: distance-decay function for use and non-use values. Journal of Environmental 

Management, 68: 297-304.

Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning: Data Mining, 

Inference, and Prediction. 2

nd edition. Springer-Verlag.

Hayes, K.M., T.J. Tyrell and G. Anderson. 1992. “Estimating the Benefits of Water Quality 

Improvements in the Upper Narragansett Bay.” Marine Resource Economics 7: 75-85.

Herriges, J.A. and J.F. Shogren. 1996. “Starting point bias in dichotomous choice valuation with followup 

questioning.” Journal of Environmental Economics and Management, 30(1):112-131.

Hite, D. 2002. Willingness to Pay for Water Quality Improvements: The Case of Precision Application 

Technology. Department of Agricultural Economics and Rural Sociology, Auburn University, Auburn, 

AL; 8 August 2002.

Hlavac, M. 2014a. “ExtremeBounds: Extreme bounds analysis in R (pp. 1 - 21)”. Cambridge, MA: 

Harvard University.

Hlavac, M. 2014b. Package ‘ExtremeBounds’ (Version 0.1.3 (2014-03-21, 07:00:10)) [Package].

Hoehn, J.P., and A. Randall. 1989. Too Many Proposals Pass the Benefit Cost Test. The American 

Economic Review, 79(3): 544-551.

Horowitz, J.L. 2003. “The bootstrap in econometrics.” Statistical Science, 18(2): 211-218.

Huang, J.C., T.C. Haab and J.C. Whitehead. 1997. “Willingness to Pay for Quality mprovements: Should 

Revealed and Stated Preference Data Be Combined?” Journal of Environmental Economics and

Management, 34(3): 240-255.

Irvin, S., T. Haab, and F.J. Hitzhusen. 2007. "Estimating willingness to pay for additional protection of 

Ohio surface waters: contingent valuation of water quality". In F.J. Hitzhusen (ed.), Economic Valuation 

of River Systems, Cheltenham: Edward Elgar, pp. 35-51.

Johnston, R.J., E.Y. Besedin and R. Stapler. 2015. “Enhanced Geospatial Meta-Analysis and 

Environmental Benefit Transfer: An Application to Water Quality Improvements.” W-3133: Benefits and 

Costs of Natural Resources Policies Affecting Ecosystem Services on Public and Private Lands. 

Pensacola, FL, February 25-27.

Johnston, R.J., E.Y. Besedin, R. Iovanna, C. Miller, R. Wardwell, and M. Ranson. 2005. “Systematic 

Variation in Willingness to Pay for Aquatic Resource Improvements and Implications for Benefit 

Transfer: A Meta-Analysis.” Canadian Journal of Agricultural Economic,s 53(2005): 221-248.

Johnston, R.J. and J.M. Duke. 2009. “Willingness to Pay for Land Preservation Across States and 

Jurisdictional Scale: Implications for Benefit Transfer.” Land Economics, 85(2): 217–237.

Johnston, R.J. and M. Ramachandran. 2014. “Modeling Spatial Patchiness and Hot Spots in Stated 

Preference Willingness to Pay.” Environmental and Resource Economics, 59(3): 363-387.

Johnston, R.J. and R.S. Rosenberger. 2010. “Methods, trends and controversies in contemporary benefit 

transfer.” Journal of Economic Surveys, 24(3): 479–510.

Johnston, S.K. Swallow, and T.F. Weaver. 1999. "Estimating Willingness to Pay and Resource Tradeoffs 

with Different Payment Mechanisms: An Evaluation of a Funding Guarantee for Watershed 

Management." Journal of Environmental Economics and Management, 38: 97-120.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 70

Jørgensen, S.L., S.B. Olsen, J. Ladenburg, L. Martinsen, S.R. Svenningsen and B. Hasler. 2013. 

“Spatially Induced Disparities in Users’ and Non-users’ WTP for Water Quality Improvements – Testing 

the Effect of Multiple Substitutes and Distance Decay.” Ecological Economics, 92(1): 58-66.

Kaoru, Y. 1993. “Differentiating Use and Nonuse Values for Coastal Pond Water Quality 

Improvements.” Environmental and Resource Economics, 3: 487-494. 

Kennedy, P. E. 2008. A Guide to Econometrics (6 ed.). Malden, MA: Blackwell Publishing.

Lant, C.L., and R. S. Roberts. 1990. “Greenbelts in the Cornbelt: Riparian Wetlands, Intrinsic Values, and 

Market Failure.” Environment and Planning 22: 1375-1388.

Lant, C.L., and G.A. Tobin. 1989. “The Economic Value of Riparian Corridors in Cornbelt Floodplains: 

A Research Framework.” Professional Geographer (41): 337-349.

Levine, R., and D. Renelt. 1992. “A sensitivity analysis of cross-country growth regressions.” The 

American Economic Review, 82(4): 942 - 963. 

Lichtkoppler, F.R. and T.W. Blaine. 1999. “Environmental Awareness and Attitudes of Ashtabula County 

Voters Concerning the Ashtabula River Area of Concern: 1996-1997.” Journal of Great LakesResources 

25: 500-514.

Lindsey, G. 1994. “Market Models, Protest Bids, and Outliers in Contingent Valuation.” Journal of 

WaterResources Planning and Management 12: 121-129.

Lipton, D. 2004. “The Value of Improved Water Quality to Chesapeake Bay Boaters.” Marine Resource 

Economics 19: 265-270

Londoño Cadavid, C. and A. W. Ando . 2013. "Valuing preferences over stormwater management 

outcomes including improved hydrologic function." Water Resources Research 49: 4114-4125.

Londoño, L.M. and R.J. Johnston. 2012. “Enhancing the Reliability of Benefit Transfer over 

Heterogeneous Sites: A Meta-Analysis of International Coral Reef Values.” Ecological Economics, 78(1): 

80-89.

Loomis, J.B. 1996. “How Large is the Extent of the Market for Public Goods: Evidence from a Nation 

Wide Contingent Valuation Survey.” Applied Economics, 28(7): 779-782.

Loomis, J.B. 2000. “Vertically Summing Public Good Demand Curves: An Empirical Comparison of 

Economic versus Political Jurisdictions.” Land Economics, 76: 312-321.

Loomis, J.B. and R.S. Rosenberger. 2006. “Reducing barriers in future benefit transfers: needed 

improvements in primary study design and reporting.” Ecological Economics, 60: 343-350.

Lyke, A.J. 1993. Discrete Choice Models to Value Changes in Environmental Quality: A Great Lakes 

Case Study. Dissertation submitted to the Graduate School of the University of Wisconsin, Madison.

Matthews, L.G., F.R. Homans, and K.W. Easter. 1999. “Reducing Phosphorous Pollution in the 

Minnesota River: How Much is it Worth?” Staff Paper. Department of Applied Economics, University of 

Minnesota.

McClelland, N.I. 1974. Water Quality Index Application in the Kansas River Basin. EPA-907/9-74-001, 

US EPA Region VII, Kansas City, MO.

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation 

Method. Resources for the Future, Washington, DC.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 71

Mrozek, J.R., and L.O. Taylor. 2002. “What determines the value of life? A meta-analysis.” Journal of 

Policy Analysis and Management, 21(2): 253-270. 

Navrud, S. and R. Ready. 2007. “Review of Methods for Value Transfer.” In: Navrud, S. and R. Ready 

(eds.), Environmental Value Transfer: Issues and Methods. Dordrecht, The Netherlands: Springer. pp.1-

10.

Nelson, J.P. and P.E. Kennedy. 2009. “The Use (and Abuse) of Meta-Analysis in Environmental and 

Resource Economics: An Assessment.” Environmental and Resource Economics, 42(3): 345-377.

Olsen, D., Richards, Jack, and Scott, R. D. 1991. “Existence and Sport Values for Doubling the Size of 

Columbia River Basin Salmon and Steelhead Runs.” Rivers. 2(1): 44-56.

Opaluch, J.J., T. Grigalunas, M.J. Mazzotta, J. Diamantides, and R. Johnston. 1998. Resource and 

Recreational Economic Values for the Peconic Estuary. Report prepared for Peconic Estuary Program, 

Suffolk County Department of Health Services, Riverhead, NY, by Economic Analysis, Inc., Peace Dale, 

Rhode Island.

Poe, G. L., K. J. Boyle, and J. C. Bergstrom, 2001. “ A preliminary meta analysis of contingent values for 

ground water quality revisited”, in The Economic Value of Water Quality, edited by J. C. Bergstrom, K. J. 

Boyle, and G. L. Poe, pp. 137–162, Edward Elgar, Northampton, Mass.

Ren, S., H. Lai, W. Tong, M. Aminzadeh, X. Hou, and S. Lai. 2010. “Nonparametric bootstrapping for 

hierarchical data.” Journal of Applied Statistics, 37(9): 1487-1498.

Roberts, L.A., and J.A. Leitch. 1997. “Economic Valuation of Some Wetland Outputs of Mud Lake.”

Agricultural Economics Report No. 381, Department of Agricultural Economics, North Dakota 

Agricultural Experiment Station, North Dakota State University.

Rosenberger, R. S. and J. B. Loomis. 2000. “Using meta-analysis for benefit transfer: In-sample 

convergent validity tests of an outdoor recreation database.” Water Resources Research, 36: 1097-1107.

Rowe, R.D., W.D. Schulze, B. Hurd and D. Orr. 1985. Economic Assessment of Damage Related to the 

Eagle Mine Facility. Energy and Resource Consultants, Inc., Boulder, CO.

Sala-i-Martin, X. X. 1997. “I just ran two million regressions.” The American Economic Review, 87(2), 

178-183. 

Sanders, L. B., R.G. Walsh, and J.B. Loomis. 1990. “Toward Empirical Estimation of the Total Value of 

Protecting Rivers.” Water Resources Research. 26(7): 1345-1357.

SAS/STAT 9.2. 2010. The MIXED Procedure. SAS/STAT(R) 9.2 User's Guide, Second Edition 

Retrieved April 29, 2014, from 

http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#mixed_toc.htm

Schaafsma, M., R. Brouwer and J. Rose. 2012. “Directional Heterogeneity in WTP models for 

Environmental Valuation.” Ecological Economics, 79(1): 21-31.

Schabenberger, O. 2004. Mixed model influence diagnostics. Proceedings of the Twenty-Ninth Annual 

SAS® Users Group International ConferenceProceedings of the Twenty-Ninth Annual SAS® Users 

Group International Conference (pp. 1-17). Cary, NC: SAS Institute Inc. Retrieved from 

http://www2.sas.com/proceedings/sugi29/189-29.pdf. 

Schulze, W.D., R.D. Rowe, W.S. Breffle, R.R. Boyce, and G.H. McClelland. 1995. Contingent Valuation 

of Natural Resource Damages Due to Injuries to the Upper Clark Fork River Basin. State of Montana, 

Natural Resource Damage Litigation Program. Prepared by: RCG/Hagler Bailly, Boulder, CO.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 72

Shrestha, R. K., J.R.R. Alavalapati. 2004. “Valuing Environmental Benefits of Silvopasture Practice: A 

Case Study of the Lake Okeechobee Watershed in Florida.” Ecological Economics 49: 349-359.

Shrestha, R.K. and J.B. Loomis. 2001. “Testing a meta-analysis model for benefit transfer in international 

outdoor recreation.” Ecological Economics, 39(1): 67-83.

Smith, V.K. and S.K. Pattanayak. 2002. “Is Meta-Analysis a Noah’s Ark for Non-Market Valuation?”

Environmental and Resource Economics, 22(1-2): 271-296.

Smith, V.K., G. Van Houtven and S.K. Pattanayak. 2002. “Benefit Transfer via Preference Calibration: 

‘Prudential Algebra’ for Policy.” Land Economics, 78(1): 132-152.

Stanley, T.D., H. Doucouliagos, M. Giles, J. H. Heckemeyer, R. J. Johnston, P. Laroche, J. P. Nelson, M. 

Paldam, J. Poot, and G. Pugh. 2013. “Meta-Analysis of Economic Research Reporting Guidelines.”

Journal of Economic Surveys, 27(2): 390–394.

Stumborg, B.E., K.A. Baerenklau and R.C. Bishop. 2001. “Nonpoint Source Pollution and Present 

Values: A Contingent Valuation of Lake Mendota.” Review of Agricultural Economics. 23(1): 120-132.

Sutherland, R. J., and R.G. Walsh. 1985. “Effect of Distance on the Preservation Value of Water 

Quality.” Land Economics 61(3): 282-29.

Takatsuka, Yuki. 2004. Comparison of the Contingent Valuation Method and the Stated Choice Model 

for Measuring Benefits of Ecosystem Management: A Case Study of the Clinch River Valley, Tennessee. 

PhD dissertation, University of Tennessee.

Tibshirani, R. 1996. “Regression shrinkage and selection via the lasso.” Journal of the Royal Statistical 

Society. Series B (Methodological), 58(1): 267-288.

United States Environmental Protection Agency (U.S. EPA). 2004. Meta-Analysis of Non-Use Benefits 

for the Section 316(b) Rulemaking for Cooling Water Intake Structures at Existing Facilities. Peer review 

Report. Prepared by U.S. EPA, Office of Water, Economic Analysis Division, Washington, D.C., 

February 13, 2004.

United States Environmental Protection Agency (U.S. EPA). 2009. Environmental Impact and Benefits 

Assessment for Final Effluent Guidelines and Standards for the Construction and Development Category. 

November, 2009.

United States Environmental Protection Agency (U.S. EPA). 2010. Water Quality Standards for the State

of Florida’s Lakes and Flowing Waters. Final Rule. Federal Register, 75(233): 75762-75807.

United States Environmental Protection Agency (U.S. EPA). 2012. Water Quality Standards for the State

of Florida’s Estuaries, Coastal Waters, and South Florida Inland Flowing Waters. Proposed Rule. Federal 

Register, 77(243): 74924-74985.

United States Environmental Protection Agency (U.S. EPA). 2013. Benefit and Cost Analysis for 

the Proposed Effluent Limitations Guidelines and Standards for the Steam Electric Power 

Generating Point Source Category. May 2013.

Van Houtven, G., J. Powers and S.K. Pattanayak. 2007. “Valuing Water Quality Improvements 

in the United States Using Meta-analysis: Is the Glass Half-full or Half-empty for National 

Policy Analysis?” Resource and Energy Economics, 29(3): 206–228.

VandenBerg, T. P., Poe, G. L. and Powell, J. R. 2001. “Assessing the accuracy of benefits transfers: 

evidence from a multi-site contingent valuation study of groundwater quality.” In J. C. Bergstrom, K. J. 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 73

Boyle & G. L. Poe (Eds.), The Economic Value of Water Quality, pp. 100-120, Cheltenham, UK: Edward 

Elgar. 

Vaughan, W.J. 1986. The RFF Water Quality Ladder. Appendix B in R.C. Mitchell and R.T. Carson. The 

Use of Contingent Valuation Data for Benefit/Cost Analysis in Water Pollution Control, Final Report. 

Washington: Resources for the Future.

Viscusi, W.K., J. Huber, and J. Bell. 2008. “The Economic Value of Water Quality.” Environmental and 

Resource Economics, 41(2): 169-187.

Wattage, P. M. 1993. Measuring the benefits of water resource protection from agricultural 

contamination: Results from a contingent valuation study. PhD dissertation, Forestry, Iowa State 

University.

Welle, P.G. 1986. Potential Economic Impacts of Acid Deposition: A Contingent Valuation Study of 

Minnesota. Dissertation, University of Wisconsin-Madison.

Welle, P.G., and J.B. Hodgson. 2011. Property Owner’s Willingness to Pay for Water Quality 

Improvements: Contingent Valuation Estimates in Two Central Minnesota Watersheds. Journal of 

Applied Business and Economics 12(1): 81-94.

Wey, K.A. 1990. Social Welfare Analysis of Congestion and Water Quality of Great Salt Pond, Block 

Island, Rhode Island. Dissertation, University of Rhode Island.

Whitehead, J.C., and P.A. Groothuis. 1992. “Economic Benefits of Improved Water Quality: a case study 

of North Carolina’s Tar-Pamlico River.” Rivers, 3: 170-178.,

Whitehead, J.C. 2006. “Improving Willingness to Pay Estimates for Quality Improvements through Joint 

Estimation with Quality Perceptions.” Southern Economic Journal, 73(1): 100-111.

Whitehead, J.C., G.C. Blomquist, T.J. Hoban and W.B. Clifford. 1995. “Assessing the Validity and 

Reliability of Contingent Values: A Comparison of On-Site Users, Off-Site Users, and Nonusers.” 

Journal of Environmental Economics and Management, 29: 238-251.

Whitehead, J.C., T. Hoban and W. Clifford. 2002. “Landowners’ Willingness to Pay for Water Quality 

Improvements: Jointly Estimating Contingent Valuation and Behavior with Limited Information.” White 

paper developed in part by U.S. EPA, NCDENR and the College of Agriculture and Life Sciences at 

NSCU.

Whittington, D., G. Cassidy, D. Amaral, E. McClelland, H. Wang and C. Poulos. 1994. “The Economic 

Value of Improving the Environmental Quality of Galveston Bay.” Department of Environmental 

Sciences and Engineering, University of North Carolina at Chapel Hill. GBNEP-38, 6/94.

Willett, J.B., and J.D. Singer. 1988. “Another cautionary note about R2: its use in weighted least-squares 

regression analysis.” The American Statistician, 42(3): 236-238.

Ye, J. 1998. “On measuring and correcting the effects of data mining and model selection.” Journal of the 

American Statistical Association, 93(441): 120-131.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 74

Appendix A: Summary Statistics from the Metadata for Key Geospatial 

Variables

Table A-1. Summary statistics from the metadata for selected variables describing sampled 

market area, affected resources, and water quality

Variable Units Mean Median Min Max St. Dev. Percentiles

5

th 95th

ar_agr Proportion 

(0 to 1) 0.33 0.26 0.01 0.92 0.26 0.04 0.88

ar_total_area km2

39,685 11,797 1,356 451,134 59,718 2,015 152,213

ar_ratio1 Ratio 

(km2/km2)

7.91 0.74 <0.01 773.74 65.49 <0.01 15.53

sub_proportion Proportion 

(0 to 1) 0.19 0.03 <0.01 1.00 0.29 <0.01 0.93

sa_area km2

127,077 5,329 4.6 9,833,518 832,426 34.4 269,601

WQIbase WQI 

(0 to 100) 42.3 45.0 5.0 80.0 18.1 5.0 72.5

quality_ch WQI 

(0 to 100) 21.3 20.0 2.5 57.5 10.9 5.0 45.0
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 75

Appendix B: Summary of Study-Specific WQL Translations

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Aiken (1985) 1 no CO State-wide rivers 

and lakes

Current conditions with existing programs 

were described (80% of CO rivers and 

lakes support game fish such as trout, and 

20% are too polluted for trout survival). 

Baseline set to 4.5 on WQL to reflect rough 

fishing. 

Scenario would improve water quality 

to allow swimming, boating, and 

fishing. Post-improvement WQL 

expressed as a range from 5.0 to 7.0 

on WQL (change of 0.5 to 2.5) to 

reflect uncertainty in average statewide improvement. 

Anderson and 

Edwards (1986)

1 yes RI Salt 

pond/marshes 

near the RI coast

Study used the WQL

Banzhaf et al. 

(2006)

2 no NY Hundreds of 

lakes in 

Adirondack Park 

upstate NY

Lakes described as safe for swimming at 

baseline, but some fish populations have 

declined or been eliminated in lakes of 

concern. Assigned baseline using a range 

of 4.5 to 5.0 on WQL to reflect rough 

fishing to game fishing. 

Improvements affect fish abundance 

and the overall ecosystem. Improved 

conditions expressed as a range from 

5.0 to 6.0 on WQL (change of 0.5 to 

1.0) to reflect uncertainty regarding 

quality of game fishing and additional 

ecosystem improvements. 

Banzhaf et al. 

(2011)

1 no VA, 

WV, 

TN, NC, 

GA

Many streams 

impacted by 

acidification in 

the Southern 

Appalachian 

Region

The affected streams have been impacted 

by air pollution and are not in good 

condition, with various fish species 

affected. Assigned baseline using a range 

of 4.5 to 5.0 on WQL to reflect rough 

fishing to game fishing.

Improved conditions expressed as a 

range from 5.0 to 6.0 on WQL 

(change of 0.5 to 1.0) to reflect 

uncertainty regarding quality of game

fishing and ecosystem improvements.

Bockstael et al 

(1988)

1 no DC, 

MD, VA

Chesapeake Bay Study assessed impacts to recreational 

uses, swimming in particular, due to 

reductions in nutrient loadings. Baseline 

coded as a range from 4.5 to 5.0 on WQL

for rough fishing to game fishing to reflect 

uncertainty regarding quality of fishing 

supported at baseline. 

Assigned post-improvement WQL of 

7.0 to reflect quality suitable for 

swimming (change of 2.5 and 2.0). 
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 76

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Bockstael et al. 

(1989)

2 no MD Chesapeake Bay 

estuarine system

The valuation question was asked of those 

respondents who believed that water 

quality in the Bay was unacceptable for 

swimming regarding an improvement to 

allow swimming. Baseline set to 5 on WQL 

to reflect game fishing, but not swimming.

Assigned post-improvement WQL of 7 

to reflect quality suitable for swimming 

(change of 2.0). 

Borisova et al. 

(2008)

3 no WV, VA Opequan and 

Abrams Creeks

The creeks are currently polluted with dirt 

and sediment along with sewage and 

bacteria. Because of these pollutants, no 

swimming or wading is recommended in 

the VA portion of these creeks In the 

survey questions, improvements were 

described in terms of watershed clean-up 

to improve the safety of swimming and 

wading. . Assigned baseline of 5.0 to 

reflect baseline game fish population in 

portions of the rivers but no swimming.

Assigned post-improvement WQL of 7 

to reflect water quality improvements 

to support swimming and wading as 

described in the survey (change of 

2.0).

Cameron and 

Huppert (1989)C

1 no CA San Francisco 

Bay and ocean 

area

Survey asked recreational users to value 

improvement focusing on doubling catch 

rates for salmon and striped bass using 

methods including habitat restoration. 

Assigned range of 5.0 to 7.0 on WQL for 

baseline to reflect uncertainty regarding 

baseline service provision.

Assigned post-improvement WQL 

range from 6.75 to 8.5 to reflect 

improvements beyond the fishable 

range (change of 1.75 and 1.5). 

Carson et al. 

(1994)

2 no CA Area on the 

Southern 

California Bight

There are reproduction problems among 

four fish species in the affected area 

caused by DDT and PCBs. Assigned 

range of 4.5 to 5.0 on WQL to reflect rough 

fishing to game fishing given presence of 

fishing at baseline (just depressed), 

reflecting uncertainty regarding baseline 

services.

Assigned post-improvement WQL 

range from 6 to 7 (change of 1.5 to 

2.0) to reflect uncertainty regarding 

the degree of improvement within the 

fishable range. 

Clonts and 

Malone (1990)

3 no AL Fifteen freeflowing riversEstimated WTP to prevent degradation to 

fifteen free-flowing rivers in Alabama.

Assigned baseline of 5 to 7 on WQL to 

reflect the uncertainty regarding average 

recreational uses supported by these rivers 

at baseline. 

Assigned WQL of 4.5 for rough fishing 

to reflect the degraded state in the 

absence of protection (absolute 

change of 0.5 to 2.5).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 77

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Collins and 

Rosenberger 

(2007)

1 no WV Cheat River 

Watershed

At baseline, water quality problems include 

acid mine drainage, lack of aquatic life, 

trash, sewage, and sedimentation. The 

watershed has fish in only the upper 

portions of the Cheat River. Northern 

section of the river is popular for rafting.

Assigned baseline of 4.5 to reflect rough 

fishing conditions (but not game fishing) 

within the study area.

Full clean-up would include treatment 

of acid mine drainage from all stream 

tributaries of the Cheat River. 

Assigned post-improvement WQL of 6 

to 6.5 (change of 1.5 to 2.0) to reflect 

improvements beyond the minimum

water quality to support game fish. 

Collins et al. 

(2009)

7 no WV Deckers Creek Describes services using three attributes 

(aquatic, scenic, and swimming). A WTP 

observation associated exclusively with the 

scenic attribute was dropped because it 

could not be tied to the WQL. Assigned 

baselines of 4.5 to 7.0 for rough fishing to 

swimming depending on whether the 

aquatic or swimming attributes (or both) 

were evaluated. 

Assignments based on support of 

fishing, game fishing, and swimming, 

with intermediate values assigned in 

some cases to reflect improvements 

beyond the minimum to support

relevant use. Post-improvement WQL 

range from 5.0 to 8.0 (changes of 0.5 

to 1.5) depending on whether the 

aquatic or swimming attributes (or 

both) were evaluated.

Corrigan et al. 

(2009)

1 no IA Clear Lake Estimated WTP for water quality 

improvements in a lake that supports game

fishing and swimming at baseline, but is 

occasionally affected by algae blooms. 

Assigned baseline WQL of 6.5 to 7.0 to 

reflect uncertainty regarding the frequency 

of impacts. 

Assigned post-improvement WQL of 

7.5 to 8.5 (changes 1.0 to 1.5) to 

reflect improvements beyond the 

minimum for swimming use. 

Croke et al. 

(1987)

9 yes IL Rivers in the 

Chicago urban 

river system

Study used the WQL

De Zoysa (1995) 1 no OH Maumee River 

Basin

Study focuses on degradation of fish 

habitat. At baseline, clear water fish are 

being replaced by fish more tolerant of low 

oxygen levels and greater turbidity.

Assigned baseline WQL of 4.5 to reflect 

rough fishing. 

Assigned post-improvement WQL of 

5.0 (change of 0.5) to reflect support 

of game fishing. 

Desvousges et 

al. (1987)

12 yes PA Monongahela 

River

Study used the WQL
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 78

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Downstream 

Strategies (2008)

2 no PA West Branch 

Susquehanna 

River Watershed

Water does not support trout populations or 

warm water fishing (like bass) because of 

acid mine drainage. Assigned baseline 

WQL of 4.5 to reflect rough fishing because 

fish have not been totally extirpated 

throughout the watershed.

If acid mine drainage were cleaned 

up, river would once again support 

trout populations and, in some areas, 

warm water fish. It would also 

provide cleaner water for community 

and industrial water supplies and 

improved recreational experiences 

for swimming, canoeing, rafting, etc. 

Assigned post-improvement WQL of 

7.0 to reflect various recreational 

uses including game fishing and 

swimming. 

Farber and 

Griner (2000)

6 no PA Loyalhanna 

Creek and 

Conemaugh 

River 

Study expresses level of pollution as 

“unpolluted”, ”moderately polluted”, or 

“severely polluted” depending on support

for fish, wildlife, and other organisms. 

Assigned baseline WQL values of 2.5 to 

“severely polluted”4.5 to “moderately

polluted” as 4.5, and 5.0 to “unpolluted” on 

descriptions. WQL assignments focus on 

support for fishing because human health is 

not typically affected. 

Post-improvement WQL values 

range from 4.5 to 5 (changes ranges 

from 0.5 to 2.5) depending on 

whether WTP observation 

corresponds to an increase to 

“moderately polluted” or “unpolluted”.

Hayes et al. 

(1992)

2 no RI Upper 

Narragansett Bay

The affected resource is unsuitable for 

swimming at baseline because of 

agricultural, domestic, and industrial 

pollutants. Assign baseline WQL values 

ranging from 4.5 to 5 for rough fishing to 

game fishing due to uncertainty regarding 

the type of fishing supported.

Estimated WTP is for improvements 

that would benefit swimming and 

shellfishing. Assigned postcompliance WQL range from 7.0 to 

9.0 across scenarios (change of 2.5 

to 4) depending on scenario to reflect 

improvements beyond minimum for 

fishing in some cases to reflect high 

sensitivity of shellfishing. 

Herriges et al. 

(1996)

2 no IA Storm Lake Study estimates WTP to prevent 

degradation in recreational uses from a 

baseline that supports some fishing and 

boating. Assigned baseline WQL values of 

4.5 for both observations to reflect rough 

fishing. 

Assigned degraded WQL values of 

ranging from 1.25 to 2.5 to reflect 

unusable to boatable waters in the 

absence of intervention (absolute 

change of 3.25 and 2).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 79

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Hite (2002) 2 no MS Lower Mississippi 

River Basin

Study estimates WTP for improvements in 

water quality in the Lower Mississippi River 

which includes many 303(d) listed waters. 

Assigned baseline WQL values ranging 

from 2.5 to 4.0 to reflect boating and quality 

close to supporting rough fishing. 

Assigned WQL values to reflect 

modest improvements (10-20% 

improvements) ranging from 4.25 to 

4.5 to reflect support, or nearing 

support, for rough fishing (changes of 

0.25 to 0.5).

Huang et al. 

(1997)

2 no NC Albemarle and 

Pamlico Sounds

Baseline water quality was described as 

degraded from 1981 levels in terms of 

reduced fish catches (60%) and reduced 

number of open shellfish beds (25%).

Assigned baseline WQL values of 5.0 to 

reflect game fishing.

Assigned post-improvement WQL 

values of 8 to 9 (changes of 3 to 4) to 

reflect improvements beyond 

swimmable, reflecting high sensitivity 

of shellfish beds to water quality.

Irvin et al. (2007) 4 no OH State-wide 

surface waters

Water quality changes described in terms of 

percent of set-aside value under the 

Antidegradation Rule to achieve varies 

quality categories (limited quality, general 

high quality, superior high quality, state 

resource waters, and outstanding national 

resource waters). “Outstanding national 

resource waters” have special 

characteristics such as high water quality, 

or as having exceptional recreational, 

cultural, aesthetic or scientific value. 

Assigned baseline WQl values of 4.5 for 

“limited quality”, reflecting rough fishing.

Post-improvement WQL levels 

correspond to four water quality 

categories above “limited quality” 

(general high quality, superior high 

quality, state resource waters, and 

outstanding national resource 

waters). Set “outstanding national 

resource waters” to 9.5, reflecting the 

highest water quality use 

classification on the WQL, because 

these are surface waters of the 

highest national ecological or 

recreational significance with water 

quality that shall not be degraded by 

any substance. Approximate WQL 

levels for intermediate categories 

based on the percent set aside 

assuming a step-wise path between 

WQl values of 4.5 (“limited quality”) 

and 9.5 (“outstanding national 

resource waters”). Results in postimprovements values of 4.86, 5.86, 

7.42, and 9.5 (changes of 0.36, 1.36, 

2.92, and 5.00), respectively.

Johnston et al. 

(1999)

1 yes RI Wood-Pawcatuck 

Watershed

Study used the WQL
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 80

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Kaoru (1993) 1 no MA 3 saltwater ponds 

on Martha’s 

Vineyard 

Study is focused on impacts to shellfishing 

in saltwater ponds in Martha’s Vineyard.

Assigned baseline WQL values ranging 

from 4.5 to 5 for rough fishing to game 

fishing to reflect support of some 

recreational uses.

Assigned post-improvement WQL 

values of 8 to 9 (changes of 3.5 to 4) 

to reflect improvements beyond 

swimmable, reflecting high sensitivity 

of shellfish beds to water quality.

Lant and Roberts 

(1990)

3 yes IA, IL Edwards River, 

South Skunk 

River, and 

Wapsipinicon 

River Basins

Study used the WQL

Lant and Tobin 

(1989)

9 yes IA, IL Multiple major IA 

and IL river 

basins

Study used the WQL

Lichtkoppler and 

Blaine (1999)

1 no OH Ashatabula River 

and Harbor

The study area was designated as an "area 

of concern" due to dredging restrictions 

because of PCB contamination, fish 

consumption advisory, and degraded 

wildlife habitat. Assigned baseline WQL 

values ranging from 1.25 to 2.5 to reflect 

not suitable to boating. 

Assigned post-improvement WQL 

values of 4.5 to 5 (changes of 3.25 

and 2.5) to reflect rough and game 

fishing, respectively. 

Lindsey (1994) 8 no MD Chesapeake Bay Study focuses on improvements in 

recreational use to support swimming.

Assigned baseline WQL values of 4.5 to 5 

to reflect rough fishing to game fishing.

Assigned post-improvement WQL of 

7.0 to reflect swimming (changes of 

2.5 to 2.0)

Lipton (2004) 1 no MD Chesapeake Bay Respondents are asked to rate water 

quality on a five tiered scale that is tied to 

whether they feel their boating is restricted 

due to water quality conditions. Assigned 

baseline WQLvalues ranging from 0.5 to 

1.25 for polluted conditions not suitable for 

boating. 

Assigned post-improvement WQL 

values of 1.25 to 2.5 to reflect 

average-values below boating and 

for boating, respectively (changes of 

0.75 to 1.25).

Londoño Cadavid 

and Ando (2013)

2 yes IL Streams near 

cities of 

Champaign and 

Urbana

Study used the WQL
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 81

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Loomis (1996) 1 no WA Elwha River The study focuses on declines in historic 

fish runs and potential restoration. Assigned 

baseline WQL value of 5.0 because Elwha 

River supports game fishing. 

Improvements would occur within the 

game fishing range. Assigned values 

of 6.0 to 7.0 (changes of 1.0 to 2.0) 

on WQL to reflect the midpoint and 

maximum of the game fishing range.

Lyke (1993) 2 no WI WI Great Lakes 

and tributaries 

that full under 

Great Lakes 

fishing 

regulations

The study estimates values based on a 

sample of nonusers with conditions 

described in terms of catch rates and 

contaminant levels. Assigned baseline WQL 

value of 5.0 to reflect game fishing because 

trout and salmon fishing are present at 

baseline.

Post-improvement WQL value 

assignments range from 5.5 to 9.0 

(changes of 0.5 to 4.0) across 

scenarios depending on effect on 

catch rates and contaminant levels. 

Matthews et al. 

(1999)

2 no MN Minnesota River Recreational uses are limited under the 

polluted baseline condition largely due to 

nutrient loadings. Assigned baseline WQL 

value of 2.5 to reflect boating and canoeing. 

Assigned post-improvement WQL 

levels ranging from 4.5 to 5.0 

(change of 2.0 to 2.5) to reflect 

increased rough and game fishing. 

Opaluch et al. 

(1988)

1 no NY Peconic Estuary Study uses multiple attributes including 

undeveloped land, wetland acreage, 

farmland acreage, eelgrass acreage, and 

acres of safe shellfish beds. Pollution has 

resulted in the closure of 4,000 acres of 

shellfish beds. Assigned baseline WQL 

value of 7.0 to reflect support for multiple 

recreational uses, including fishing and 

swimming. 

Assigned post-improvement WQL 

level of 9.0 to reflect reduction in 

shellfish closures (change of 2.0). 

Assignment reflects high sensitivity 

of shellfish beds to water quality.

Roberts and 

Leitch (1997)

1 no MN, SD Mud Lake Survey indicates that the lake supports a 

wide range of uses. Swimming is excluded 

from the assignment because the lake is a 

wetland not suited for swimming. Assigned 

baseline WQL value of 5.0 to reflect boating 

and game fishing. 

Survey focuses on WTP for 

preventing degradation. Assigned 

WQL level of 4.5 for rough fishing 

(absolute change of 0.5) in the 

absence of intervention to reflect 

reduction in fishing services. 

Rowe et al. 

(1985)

1 yes CO Stretch of Eagle 

River

Study used the WQL
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 82

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Sanders et al. 

(1990)

4 no CO Up to 11 wild 

and/or scenic 

rivers

Wild or scenic rivers, by definition, are likely 

to be of excellent (swimmable) water 

quality, therefore assigned baseline WQL 

value of 7.0. 

Survey focuses on WTP for 

preventing degradation to wild and 

scenic rivers. Assigned WQL values 

range from 4.5 to 5.0 (absolute 

change of 2.5 and 2.0) in the 

absence of intervention to reflect a 

reduction in uses to below the 

swimmable range. 

Schulze et al. 

(1995)

2 no MT Clark Fork River 

Basin

The survey discusses 8 hazardous sites in 

Montana that have been put on EPA's 

National Priorities List for cleanup under the 

Comprehensive Environmental Response, 

Compensation, and Liability Act (CERCLA).

At baseline, nearby surface waters are 

affected by industrial and mining wastes. 

Assigned baseline WQL value of 1.25 to 2.5 

to reflect high levels of pollution and 

potential support for boating. 

Assigned post-improvement WQL 

levels of 4.5 to 5 (changes ranging 

from 2.0 to 3.75) to reflect effects of 

restoration on rough and game 

fishing services. 

Shrestha and 

Alavalapati 

(2004)

2 no FL Lake 

Okeechobee 

Basin

The study focuses on use of silvopasture 

(which combines trees, forages, and shrubs 

with livestock operations) for limiting 

phosphorus runoff. Assigned baseline WQL 

values ranging from 2.5 to 3.5 to reflect 

limited recreational uses, below the fishable 

range on average. 

Assigned post-improvement WQL 

levels of 3.5 to 4.5 (changes of 1.0) 

to reflect additional support for fish 

habitat and rough fishing. 

Stumborg et al. 

(2001)

2 no WI Lake Mendota The Lake Mendota watershed had been on

the Wisconsin Department of Natural 

Resources (WDNR) Priority Watershed 

Program. The baseline probability of an 

algae bloom on any given day during 

summer time was 50% (1 of every 2 days).

Assigned baseline WQL value of 2.5 to 

reflect limited recreational use given 

frequency of algae blooms.

Assigned post-improvements WQL 

levels of 4.5 to 5 (change of 2.0 to 

2.5) to reflect improvements in fish 

populations and rough and game 

fishing given 50% reductions in 

phosphorus loadings.
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 83

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Sutherland and 

Walsh (1985)

1 no MT Flathead River 

and Lake Area

The study area has high quality water at 

baseline suitable for fishing, boating, 

shoreline camping, sunbathing, and 

swimming. Therefore, assigned baseline 

WQL value of 7.0. 

Assigned values of 4.5 to 5.0 

(absolute change of 2.5 and 2.0) to 

reflect reduction in recreational 

services in the absence of protection. 

Takatsuka (2004) 4 no TN Clinch River 

Valley

Survey expressed conditions in terms of

three metrics (aquatic life, sport fish, and 

general water quality) rated in terms of 

poor, fair, good. Assigned baseline WQL 

value of 5 because the river valley supports 

bass. 

Assigned post-improvement WQI 

levels of 6.0 to 7.0 across 

observations to reflect increase in 

gamefish abundance and support for 

swimming (changes of 1.0 to 2.0).

Wattage (1993) 3 yes IA Bear Creek Study used the WQL.

Welle (1986) 6 no MN MN lakes that 

could be 

damaged by acid 

rain

At baseline, the study lakes have high 

water quality. As stated in the study, “Lakes 

and streams are generally clean and 

unpolluted. Remote lakes and forest are 

relatively pristine.” Assigned baseline WQL 

values of 7.5 to 8.5 to reflect water that are 

“generally clean and unpolluted” or better 

and supports multiple recreational uses. 

Assigned WQL levels of 4.5 to 5.0 

(absolute changes of 2.5 to 4) in the 

absence of intervention to reflect 

impacts on recreational uses and 

aesthetics including angling. 

Welle and 

Hodgson (2011)

3 no MN Sauk River Chain 

of Lakes 

Watershed and 

Lake MargaretGull Lake 

Watershed

Description of baseline and changes 

focuses on prevalence of algae blooms that 

diminish finishing and swimming. Calculate 

WQL values for baseline and postimprovement by weighting WQL levels 

between 2.5 and 7.0 by the frequency of 

algae blooms. Results in baseline WQL 

value of 3.85. 

The improvement program would 

include reductions in algae blooms 

from 60-80 % of during summer 

months to 30 to 50% . Calculate 

post-improvement WQL values by 

weighting WQL levels between 2.5 

and 7.0 by the frequency of algae 

blooms. Results in post-improvement 

level of 5.2 (change of 1.35).

Wey (1990) 2 yes RI Great Salt Pont 

on Block Island

Study used the WQL.

Whitehead and 

Groothuis (1992)

3 no NC Tar-Pamlico 

River

The river has experienced declining fish 

catches, diseases in fish, algae blooms, 

and losses in shellfish beds and underwater 

grasses as a result of nonpoint source 

pollution. Assigned baseline WQL values of 

4.5 to 5 to reflect rough fishing to game 

fishing at baseline in spite of declines. 

Valuation scenario would result in a 

doubling of recreational catch rate 

and general improvements for 

boating and swimming. Assigned 

post-improvement level of 7.0 to 

reflect swimming (change of 2.5 and 

2.0).
September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 84

Table B-20. Coding of WQL Values for Primary Studies in the MetadataA,B

Author(s) and 

Publication Year

Obs. in 

Metadata

Study 

Used 

the 

WQL?

(yes/no)

State(s) Waterbody Baseline Improvement

Whitehead 

(2006)

3 no NC Neuse River Study asks respondents about perceived 

water quality in terms of poor, fair, good, 

and excellent when thinking about uses 

such as boating, fishing, and swimming. 

These categories are assigned WQL levels 

of 2.5, 4.5, 5, and 7, respectively. Baseline 

WQL values vary across observations from 

2.5 to 5.0.

All observations evaluate 

improvement to “excellent”. Postimprovement WQL values. Assigned 

post-improvement WQL value of 7.0 

consistent with swimming. Changes 

range from 2.0 to 4.5 across 

observations. 

Whitehead et al. 

(1995)

2 no NC AlbemarlePamlico Estuary 

System

The survey described individual uses of the 

system such as fishing, boating, and 

swimming. Assigned baseline WQL values 

range from 5.0 to 7.0 to reflect uncertainty 

regarding the respondent perceived 

baseline. 

The study elicited respondent 

willingness-to-pay “to make sure 

water pollution does not get worse 

and wildlife habitat remains the same 

in the A-P system.” Assigned WQL 

value of 4.5 for rough fishing to 

reflect degradation in the absence of 

restoration (absolute change of 0.5 to 

2.5).

Whittington et al. 

(1994)

1 no TX Galveston Bay The bay supports fish, shellfish, and coastal 

birds, but half the bay is closed for 

shellfishing due to public health concerns. 

Assigned baseline WQL value of 5.0 to 

reflect game fishing. 

Assigned post-improvement WQL 

value of 7.0 to reflect approximate 

water quality to support 50% more 

oyster beds for public use, as well as 

improvements in fishing and 

swimming.

(A): All studies were initially coded using the 10-point WQL. WQL values were then multiplied by 10 to calculate baseline and post WQI values. EPA coded WQL 

values as four variables, baseline_low and post_low, and baseline_high and post_high. These values allow EPA to capture a range of starting and ending values 

given uncertainties during analyst translation and coding, Low and high WQL changes are calculated as the difference between their respective start and end 

points. Following the 2009 MRM, lnquality_ch is the natural log of ((post_high- baseline_high)*10) and lnbase is the natural log of (baseline_low*10). MRM1 is 

insensitive to the use of baseline_low versus basline_high. Model fit, estimated coefficients, and variable significance are all very similar.

(B): Shading indicates studies added since the 2009 meta-analysis. 

(C): As described in Section 2, to be included in the metadata, studies must value water quality changes affecting ecosystem services provided by waterbodies, 

including recreational activities (such as fishing, boating, and swimming), aquatic life support and other nonuse values. EPA conducted additional review of the 

Cameron and Huppert (1989) survey materials during preparation of this appendix. The study asked recreational users to value improvements focusing on a 

doubling of catch rates for salmon and striped bass using methods including fishing stocking and habitat restoration. The survey describes demand for water 

generally but does not specifically mention water quality. Because it does not mention water quality specifically, EPA has determined that the Cameron and 

Huppert (1989) study and its one observation should be dropped from the metadata prior to finalization of the MRM. In order to assess sensitivity to this change, 

EPA estimated MRM1 with the Cameron and Huppert (1989) observation dropped from the metadata and found that results were nearly identical to the results 

presented in Section 4.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 85

Appendix C: MRM2 with Observation 17 Excluded

Table C-1. MRM2 preliminary specification estimated by OLS excluding observation 17. In the far 

right columns, a single * indicates b/standard error > 1.96 and ** indicates that the 95% bootstrap 

confidence intervals do not include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept 0.231 4.640 -18.100 15.500

ce 0.546 0.315 -0.714 1.640

thesis 0.591 0.248 -0.317 1.710 *

lnyear -0.386 0.100 -0.966 -0.115 * **

volunt -1.520 0.215 -2.240 -0.761 * **

outliers_trim -0.504 0.173 -0.975 0.187 *

non_param -0.609 0.114 -0.971 0.008 *

non_reviewed -0.593 0.200 -1.430 -0.052 * **

lump_sum 0.917 0.189 0.358 1.570 * **

wtp_median -0.340 0.340 -1.320 0.663

northeast 1.180 0.319 0.261 2.300 * **

central 0.576 0.157 -0.202 1.220 *

south 1.380 0.167 0.696 2.090 * **

nonusers -0.458 0.152 -0.813 -0.156 * **

lnincome 0.206 0.424 -1.100 1.860

mult_bod -0.695 0.201 -1.500 -0.015 * **

river -0.086 0.143 -0.613 0.495

swim_use -0.433 0.184 -0.929 0.035 *

gamefish 0.521 0.299 -0.360 1.320

boat_use -0.709 0.221 -1.280 0.036 *

ln_ar_agr -0.357 0.100 -0.642 0.094 *

ln_ar_ratio -0.076 0.026 -0.152 0.055 *

sub_proportion 1.190 0.235 0.052 2.000 * **

Q -0.016 0.005 -0.033 -0.002 * **

R2 0.618

RCV2 0.426

rWTP2 0.574
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 86

Figure C-1. MRM2 preliminary specification estimated by OLS excluding observation 17. (a) Scatter 

plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) 

Marginal willingness to pay curve. Demand curve shifters are set as described in the main text. The black 

line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 

97.5th percentiles.

(a) (b)

(c)
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 87

Table C-2. MRM2 preliminary specification estimated by WLS excluding observation 17. In the far 

right columns, a single * indicates b/standard error> 1.96 and ** indicates that the 95% bootstrap 

confidence intervals do not include zero.

Parameter b

Standard 

Error

Bootstrap Confidence 

Interval

b/Standard 

Error > 1.96 

(*=yes)

CI include 

zero 

2.5 (**=yes) th 97.5th

intercept -1.120 5.240 -19.700 15.500

ce 0.379 0.327 -0.714 1.600

thesis 0.827 0.273 -0.294 1.790 *

lnyear -0.438 0.127 -0.934 -0.080 * **

volunt -1.450 0.229 -2.290 -0.760 * **

outliers_trim -0.376 0.202 -1.010 0.284

non_param -0.424 0.118 -0.846 0.076 *

non_reviewed -0.741 0.203 -1.480 -0.112 * **

lump_sum 0.882 0.186 0.230 1.520 * **

wtp_median -0.158 0.325 -1.280 0.798

northeast 1.240 0.313 0.291 2.300 * **

central 0.589 0.212 -0.291 1.190 *

south 1.420 0.231 0.639 2.090 * **

nonusers -0.590 0.146 -0.929 -0.208 * **

lnincome 0.339 0.467 -1.130 2.010

mult_bod -0.856 0.210 -1.710 -0.082 * **

river -0.061 0.171 -0.668 0.502

swim_use -0.312 0.176 -0.890 0.174

gamefish 0.214 0.377 -0.674 1.400

boat_use -0.702 0.231 -1.420 0.054 *

ln_ar_agr -0.282 0.111 -0.602 0.115 *

ln_ar_ratio -0.035 0.035 -0.151 0.078

sub_proportion 1.140 0.268 0.029 2.000 * **

Q -0.014 0.006 -0.035 0.005 *

R2 0.761

RCV2 0.524

pseudoRWLS2 0.554

rWTP2 0.543
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 88

Figure C-2. MRM2 preliminary specification estimated by WLS excluding observation 17. (a) 

Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated 

WTPs. (c) Marginal willingness to pay curve. Demand curve shifters are set as described in the main text. 

The black line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 

50th, and 97.5th percentiles.

(a) (b)

(c)
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 89

Appendix D: MRM2 Bootstrapped Histograms of Step-wise 

Regression Coeffcent Estimates

Figure D-1. MRM2 Bootstrapped Histograms of Step-wise OLS Regression Coefficient Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 90

Figure D-1. (contd) MRM2 Bootstrapped Histograms of Step-wise OLS Regression Coefficient 

Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 91

Figure D-1. (contd) MRM2 Bootstrapped Histograms of Step-wise OLS Regression Coefficient 

Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 92

Figure D-1. (contd) MRM2 Bootstrapped Histograms of Step-wise OLS Regression Coefficient 

Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 93

Figure D-2. MRM2 Bootstrapped Histograms of Step-wise WLS Regression Coefficient Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 94

Figure D-2. (contd) MRM2 Bootstrapped Histograms of Step-wise WLS Regression Coefficient 

Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 95

Figure D-2. (contd) MRM2 Bootstrapped Histograms of Step-wise WLS Regression Coefficient 

Estimates.
September 29, 2015WQ Meta-Analysis Peer Review Document | pg 96

Figure D-2. (contd) MRM2 Bootstrapped Histograms of Step-wise WLS Regression Coefficient 

Estimates.