# Peer Review Package for Meta

- analysis of the Willingness

- to

Pay for Water Quality ImprovementsSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 1



# Table of Contents

1 Project Summary

................................ ................................ ................................ .............................. 2

2 Literature Review to Identify Additional Studies

................................ ................................ .......... 3
3 Variable Development and Coding

................................ ................................ ................................ . 7
3.1 Study Methodology and Year

................................ ................................ ................................ .. 9
3.2 Region and Surveyed Populations

................................ ................................ ......................... 10

3.3 Reconciliation of Water Quality Baseline and Change

................................ .......................... 11

3.4 Delineation of the Affected Resource and Substitutes

................................ ........................... 13

4 MRM1 Estimation and Results

................................ ................................ ................................ ..... 19

5 Robustness Tests

................................ ................................ ................................ ............................. 22

5.1 Horizontal Robustness

................................ ................................ ................................ ........... 25

5. 2 Vertical Robustness

................................ ................................ ................................ ................ 29

5.3 Summary and Findings from Horizontal and Vertical Robustness Tests

.............................. 38

5.4 Cross

- Validation Test

................................ ................................ ................................ ............ 39

6 Benefit Transfer Application

................................ ................................ ................................ ......... 40

6.1 Benefit Transfer Example

................................ ................................ ................................ ...... 41

6.2 Benefit Transfer Application Limits

................................ ................................ ...................... 45

7 An Alternative Approach: MRM2

................................ ................................ ................................ 48

7.1 Estimation and Model Selection Approach

................................ ................................ ............ 48

7.2 Results

................................ ................................ ................................ ................................ .... 50

8 Model Limitations and Uncertainty

................................ ................................ .............................. 56
9 Example WTP Predictions for MRM1 and MRM2

................................ ................................ .... 57
10 Overfitting and Step

- wise Regression: MRM2

................................ ................................ ............ 59

11 Next Steps

................................ ................................ ................................ ................................ ........ 66

12 Referen ces

................................ ................................ ................................ ................................ ....... 66 Appendix A: Summary Statistics from the Metadata for Key Geospatial Variables

......................... 74 Appendix B: Summary of Study

- Specific WQL Translations

................................ .............................. 75 Appendix C: MRM2 with Observation 17 Excluded

................................ ................................ ............. 85 Appendix D: MRM2 Bootstrapped Histograms of Step

- wise Regression Coeffcent Estimates

........ 89September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 2



# 1 Project Summary

The U.S. Environmenta l Protection Agency (EPA) is re

- examining and updating a meta

- regression model (MRM) that relates household willingness

- to

- pay (WTP) to water q uality improvements and various other independent variables. EPA began work on the original MRM in 2004 , and the MRM was last updated in

200 9 as part of EPA’s benefits analysis for the Construction and Development Effluent Guidelines (U.S. EPA 2009)

. 1 

### The 200 9 revisions included adding 11 studies (34 WTP observations) that were not included in the original metadata

. T he meta

- data was published in 200 9 as part of the rulemaking

documentation for the e ffluent limitation guidelines and standards for Construction and Development (C&D) (U.S. EPA 2009). The 2009 revisions did not address the modeling approach (e.g., specifi cation of independent variables and use of various functional forms). Household WTP for water quality improvements estimated based on the 200 9 MRM have been considered in several Clean Water Act (CWA) rulemakings, including:

 Final Effluent limitation guid elines and standards for C&D (U.S. EPA 2009) ;

 Final water quality standards for Florida lakes and flowing waters and p roposed w ater quality standards for Florida’s estuaries, coastal water, and South Florida inland flowing waters (U.S.

EPA 2010 ; U.S. EPA 2 012 ) ; and

 Proposed e ffluent limitation guidelines for Steam Electric power generating sources (U.S. EPA 2013)

. As the analysis has been used in several applications, it has become important to consider whether the MRM is broadly applicable or whether chang es would enhance the breadth of appropriate application

situations. For instance

, the coefficients for single

- river and multi

- river variables in the 2009 MRM are an issue of particular concern to EPA. This is because the current model coefficients can appear counterintuitive by suggesting that total WTP is lower for multi

- river improvements than single river improvements. 2 

### T heory sugges ts that WTP estimates sh ould be sensitive to geospatial factors including scale (the size of affected resources or areas), market extent (the size of the market area over which WTP is estimated) and the availability of substitutes. Neither the 2004 (2009) MRMs nor other publishe d MRMs enable simultaneous adjustments for these factors, leading to potentially biased benefit transfer

results

. T he current revisions include incorporation of additional studies in the metadata and the development of new variables for core economic facto rs including geo spatial scale of the affected resources, size of the surveyed area, and substitute availability. By directly incorporating these geospatial factors into the MRM, EPA expects to enhance its content validity and improve the accuracy of benefi t
transfers.

This report provides detail on each step involved in the MRM update. Section 2 describes EPA’s literature review to identify additional studies includ ed in the meta

- data. Section 3 details development and coding of variables included in the revised WTP function (e.g., geographic area covered by the study,

size of the affected aquatic resource , presence of substitute sites ). The report presents two MRMs, referred to as MRM1 and MRM2, presented in Section 4 and Section 7 , respectively. A key issue that pervades the potential use of MRM1 and MRM2 for benefit transfer is their suitability for predicting

1 The 2004 MRM was also published as Johnston et al. (2005).

2 EPA’s review of the affected resources in the single and multiple rivers studies suggests that variables for single and multiple rivers do not fully capture variation in spatial scale and resource significance across groups because the metadata for single rivers is dominated by large, regionally important rivers. This factor likely contributes to ambiguity in the interpretation of single and multiple river coefficients.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 3



### WTP for separate regulations that are analyzed and impleme nted sequentially, in particular, regulations that generate small water quality improvements below the range of those observed in the metadata. This issue is highlighted through benefit transfer examples and discussion of uncertainties and limitations in

S ection 6 . Section 5 presents detailed robustness testing results for MRM1 and Section 9 presents example WTP predictions using both MRM1 and MRM2. Section 10 discusses a step

- wise regression

appro ach and potential for over

- fitting. Sec tion 11 describes potential next steps.



# 2 Literature Review to Identify Additional Studies

EPA used the 2009 meta

- data as the starting point for the current revisio ns and conducted a literature

review for additional studies to include in the metadata

. The Agency followed Stanley et al. (2013)’s guidelines for meta

- regression analyses in economics in documenting the literature search including (a)

the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date completed. To identify new studies, EPA relied on the following:

 Searches of general literature databases and search engines (EBSCO, Google Scholar, Google). Search keywords were selected to be sufficiently broad so not to miss relevant studies:

o First terms: (1) water quality, (2) water clarity, (3) nutrient removal/improvement, (4) water quality index/ladder, (5) clean water, (6) water pollution reduction, (7) water
habitat improvement, and (8) stream flow.

o Second terms: (1) willingness to pay, (2) stated preference, (3) contingent valuation, (4) choice experiments, (5) contingent activity, and (6) conjoint analysis

 Searches of online reference and abstract databases (Environ mental Valuation Resource Inventory (EVRI), Benefits Use Valuation Database (BUVD), AgEcon Search, RePEc/IDEAs,

and the Oregon State University College of Forestry Recreation Use Values Database );

 Visits to webpages of authors and university programs known to publish stated preference studies

and/or w ater quality valuation research; 3



###  Searches of web sites for Resources for the Future and EPA’s National Cen ter for Environmental Economics, both known to conduct environmental and resource economics valuation research ;

 Searches of key resource economics journals for 2005 to 2013 (Land Economics, Environmental and Resource Economics, Marine Resource Economics, Journal of Environmental Economics and Management, Water Resources Rese arch, and Ecological Economics ). EPA focused these journal

- specific searches on the recent period of 2005 to 2013 given the likelihood that older studies were captured under the steps above or captured during literature reviews conducted for prior versions of the MRM ; and

 Reviewed biblio graphies of other valuation meta

- analyses (Van Houtven et al. 2007; Ge et al. 2013; Brander and Brouwer 2011).

Studies id entified during the search were screened according to the following criteria prior to inclusion in the metadata to ensure validity, consistency, and applicability. The identification and review of studies

was completed and verified by multiple individuals following recommendations of Sta nley et al. (2013):

3 This included G. Poe (Cornell), J. Bergstrom (University of Georgia), T. Haab (Ohio State), K. Viscusi (V anderbilt), R. Carson (U.C. San Diego), W. H. Desvousges, J. Whitehead (Appalachian State), K. Boyle (Virginia Tech), R. Rosenberger (Oregon State), J. Loomis (Colorado State), J. Corrigan (Kenyon College), and R.H. von Haefen (North Carolina State0.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 4



###  Commodity consistency

– Study must value water quality changes affecting ecosys tem services provided by waterbodies, including recreational activities (such as fishing, boating, and swimming), aquatic life support and other nonuse value s

. 4 

### EPA did not include studies that

estimate WTP for improvements in surface waters used primarily for drinking water.

 Welfare consistency

– The study must us e general accepted stated preference approaches and report theoretically comparable Hicksian welfare measures (Boyle et al. 2013)

.  Amenity detail

– As described by Johnston et al. (2005), “the study must provide sufficient information regarding resource, context, and study attributes to warrant inclusion in the meta

data” (p.223).

 Study location

– The study must be conducted in the U.S.

 Research meth ods

– The s tudy must apply research methods that are supported by the literature and that provide WTP estimates consistent with neoclassical welfare theory.

 Duplicative studies

– Some studies may be released in multiple forms, such as working papers,

confe rence papers, journa l articles, and book chapters. Studies are screened to remove duplica tive analysis and analysis that were subsequently revised and published. Multiple valu es can be included for a study if they address a different component of the study data or estimate values for alternative resource changes.

Table 1 summarizes studies in the revised metadata after identifying and screening additional studies. In total, the revised metadata includes 51 stated preference studies that estimated total WTP (use plus nonuse) per household for water quality changes in U.S. waterbodies. The studies address various waterbody types including, rivers, lakes, salt ponds/marshes, and estuaries. The fifteen studies added to the metadata during the current revisions a re shaded in Table 1. Two of the new papers (Corrigan et al. 2009; Whitehead 2006) replaced unpublished versions of the same studies (Azevedo et al. 2001;

Whitehead et al. 2002). The revised metadata excludes two studies by Viscusi et al. (2008) and Carson and Mitchell (1993). Carson and Mitchell (1993) estimate WTP for water quality improvements nation

wide. It is excluded because its national scale means that the size of the affected resource area is substantially different from other s tudies in the meta

- data, which at most, assess impr ovements within a region. On the other hand, Viscusi et al. (2008) asked respondents to value water quality improvements

within 100 mile s of their home. To be included in the metadata, a WTP observation m ust correspond to a specific affected resource or set of resources that can be delineated in GIS. Viscusi et al. (2008) does not
fit into EPA’ s data coding framework because the affected reso urce varies across the sample, with each respondent asked to thin k about their own area. The WTP estimates reported by Viscusi et al. (2008) reflect an average across many affected areas, rather than WTP for a specific affected resource. Many studies contribute multiple observations due to in

- study variations in factors such as number or type affected waterbodies, magnitude of water quality improvement, sampled market area, and elicitation

methods. The inclusion of multiple observations per study is standard in resource valuation metadata (Nelson and Kennedy 2009). In to tal, the 51 studies provide 140 WTP observations

.

4 F or example, a study that estimates WTP for recreational fishing improvements due to fish stocking would not be included

in the meta

- data. However, a study that estimates WTP for recreational fishing improvements due to nutrient reductions

would be included in the meta

- data, presuming it satisfies all other criteria.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 5



### Table 1 . Primary Studies in the Metadata

. A,B,C,D

Author(s) and Publication Year Obs. in Metadata State(s) Water Body Type(s) Willingness to Pay per Household ( 2007 $ )

Mean Min. Max. Aiken (1985) 1 CO river and lake 193.18 193.18 193.18

Anderson and Edwards (1986) 1 RI salt pond/marshes 180.71 180.71 180.71

Banzhaf et al. (2006) 2 NY L ake 57.47 54.09 60.85 Banzhaf et al. (2011) 1 VA, WV, TN, NC, GA river/stream 31.30 31.30 31.30

Bockstael et al (1988) 1 DC, MD, VA estuary 149.03 149.03 149.03 Bockstael et al. (1989) 2 MD estuary 158.30 75.67 240.93 Borisova et al. (2008) 3 WV, VA river/stream 44.94 18.05 65.82 Cameron and Huppert (1989) 1 CA estuary 49.53 49.53 49.53

Carson et al. (1994) 2 CA estuary 59.40 41.21 77.59

Clonts and Malone (1990) 3 AL river/stream 103.20 78.31 127.48

Collins and Rosenberger (2007) 1 WV river/stream 18.19 18.19 18.19

Collins et al. (2009) 7 WV river/stream 120.52 2.84 217.57 Corrigan et al. (2009) 1 IA L ake 123.30 123.30 123.30 Croke et al. (1987) 9 IL river/stream 77.47 61.31 93.68 De Zoysa (1995) 1 OH river/stream 70.18 70.18 70.18 Desvousges et al. (1987) 12 PA river/stream 59.19 19.84 137.26

Downstream Strategies (2008) 2 PA river/stream 12.74 10.70 14.77

Farber and Griner (2000) 6 PA river/stream 76.16 16.58 148.59

Hayes et al. (1992) 2 RI estuary 397.44 390.68 404.19 Herriges and Shogren (1996) 2 IA L ake 134.55 61.71 207.40

Hite (2002) 2 MS river/stream 60.08 58.24 61.93 Huang et al. (1997) 2 NC estuary 258.65 255.01 262.29 Irvin et al. (2007) 4 OH all freshwater 21.67 19.65 23.23 Johnston et al. (1999) 1 RI river/stream 180.95 180.95 180.95 Kaoru (1993) 1 MA salt pond/marshes 218.61 218.61 218.61

Lant and Roberts (1990) 3 IA, IL river/stream 143.93 124.04 154.31

Lant and Tobin (1989) 9 IA, IL river/stream 55.63 40.58 67.64 Lichtkoppler and Blaine (1999) 1 OH river and lake 41.93 41.93 41.93

Lindsey (1994) 8 MD estuary 66.80 33.40 102.20 Lipton (2004) 1 MD estuary 63.98 63.98 63.98

Londoño Cadavid and Ando (2013) 2 IL river/stream 38.68 35.93 41.44

Loomis (1996) 1 WA river/stream 93.07 93.07 93.07 Lyke (1993) 2 WI river and lake 78.75 59.75 97.74 Matthews et al. (1999) 2 MN river/stream 21.73 18.14 25.32 Opaluch et al. (19 9 8) 1 NY estuary 138.47 138.47 138.47

Roberts and Leitch (1997) 1 MN, SD L ake 8.35 8.35 8.35

Rowe et al. (1985) 1 CO river/stream 134.59 134.59 134.59 Sanders et al. (1990) 4 CO river/stream 160.69 81.01 210.04September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 6



### Table 1 . Primary Studies in the Metadata

. A,B,C,D

Author(s) and Publication Year Obs. in Metadata State(s) Water Body Type(s) Willingness to Pay per Household ( 2007 $ )

Mean Min. Max. Schulze et al. (1995) 2 MT river/stream 20.84 17.34 24.33

Shrestha and Alavalapati (2004) 2 FL river and lake 156.46 137.97 174.95

Stumborg et al. (2001) 2 WI L ake 84.29 66.73 101.86

Sutherland and Walsh (1985) 1 MT river and lake 146.03 146.03 146.03

Takatsuka (2004) 4 TN river/stream 286.88 181.90 391.85 Wattage (1993) 3 IA river/stream 53.89 40.24 74.59 Welle (1986) 6 MN L ake 167.28 109.60 238.42 Welle and Hodgson (2011) 3 MN L ake 145.10 10.59 285.06

Wey (1990) 2 RI salt pond/marshes 147.26 63.95 230.58

Whitehead and Groothuis (1992) 3 NC river/stream 41.01 31.90 53.16

Whitehead (200 6 ) 3 NC river/stream 187.18 27.52 365.54

Whitehead et al. (1995) 2 NC estuary 95.44 78.29 112.59

Whittington et al. (1994) 1 TX estuary 194.72 194.72 194.72

Notes : (A): Shading indicates studies added since the 2009 meta

- analysis. (B): J ournal publications by Corrigan et al. (2009) and Whitehead (2006 ) replaced staff papers authored by Azevedo et al. (2001) and Whitehead (2002), respectively. (C): A study by Olsen et al. (1991) from the 2009 meta

- analysis was excluded from the current metadata following robustness testing and additional review of study details. (D): The revised metadata excludes studies by Viscusi et al. (2008) and Carson and Mitchell (1993) that surveyed households across the U.S.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 7



# 3 Variable Development and Coding

EPA entered values for all existing variables in the metadata for each new study identified during the

literature review

. Next, EPA developed and coded new variables (for all studies) that address gaps in the

200 9 metadata. The variables in the revised MRM fall into four general categories:

(1) Study methodology and year v ariables characterize such features as the year in which a study was conducted, payment vehicle and elicitation formats, WTP estimatio n method, and publication type. These variables are included to explain differences in WTP across studies but are not expected to vary across benefit transfer for different policy applications.

(2) Region and surveyed populat ions variables characterize such features as the US region in which the study was conducted, the average income of respondent households and the representation of users and nonusers within the survey sample.

(3) Sampled market and affected resource variables c haracterize features such as the geospatial scale (or size) of affected water bodies, the size of the market area over which populations were sampled, as well as land cover and the quantity of substitute water bodies.

(4) Water quality (baseline and change) v ariables characterize baseline conditions and the extent of the water quality change.

Table 2 presents and defines key variables from the metadata by category. Shading indicates that the variable was developed during these revisions. The development of new variables focused primarily on developing variables that capture geospatial factors including extent of sampled market, surveyed populations and affected resources. 5 

### The 2009 MRM largely relied on binary variable s to distinguish broad categories of affect ed resources (e.g., single vs multiple rivers, regional freshwater, etc .) . Extensive GIS mapping is required in order to quantitatively incorporate new spatial factors using continuous variables because many primary studies give broad outlines but omit the detailed geospatial information regarding affected resources and sampled populations. This type of supplementation has been used by

others to include various types of information into valuation MRMs other than that necessary for the

current metadata (e.g.

, Ghermandi et al. 2010; Ghermandi and Nunes 2013 ). By linking to publicly available GIS data layers, EPA provides a consistent basis for the construction and coding of the new, spatial variables. The following subsections describe both development and cod ing of water quality and spatial variables for surveyed populations, extent of sampled market, study site, and affected resources. In accordance with Stanley et al. (2013) guidelines, all variable coding was reviewed by multiple ind ividuals

and documented clearly in the metadata and data dictionary

. Annual WTP values are typically desired for policy analysis to support a comparison of annualized costs and benefits. The metadata includes total WTP as reported by the original study, adjusted to 2007$. For over 80 percent of observations (114 studies) , total WTP reflects annual payments in perpetuity (113 studies) and one observation reflects annual WTP for a 10

- year payment period

. For the remaining 26 observations, WTP reflec ts a short

- term payment period ( 11 observations use a one

- time lump sum
payment, 1 observati on uses a 3

- year payment period, and 14 observations use a 5

- year payment period )

. EPA used the lump_sum binary variable to account for short

- term payment periods ( lump_sum =1 )

. EPA
also tested models with WTP values for short

- term payment periods converted to perpetual streams but found that results were sensitive to discount rate assumptions and that these models did not provide any practical advantages for benefit transfer over models using the lump_sum variable.

5 EPA also coded one new methodological variable, ce , to identify studies that used a choice experiment approach.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 8



### Table 2

. Definition and Summary Statistics for Model Variables

. A

Variable Definition Units Mean St. Dev. Study Methodology and Year

c e Binary variable indicating that the study is a choice experiment Binary (Range: 0 or 1) 0.10 71 0.3 104

t hesis Binary variable indicating that the study is a thesis. Binary (Range: 0 or 1) 0.11 43 0.3 193

l nyear Natural log of the year in which the study was conducted (i.e., data was collected) , converted to an index by subtracting 1980. Natural log of
years (year ranges from 1981 to 2011). 2.212 7 0. 9282

v olunt Binary variable indicating that WTP was estimated using a payment vehicle described as voluntary as opposed to, for example, property taxes. Binary (Range: 0 or 1) 0.0 857 0.2 809

outlier s

_ trim Binary variable indicating that outlier bids were excluded when estimating WTP. Binary (Range: 0 or 1) 0.1 929 0.39 60

n onparam Binary variable indicating that WTP was estimated using

non

- parametric methods. Binary (Range: 0 or 1) 0.4 286 0.49 66

non_reviewed Binary variable indicating that the study was not published in
a peer

- reviewed journal. Binary (Range: 0 or 1) 0.23 57 0.42 60

lump_sum Binary variable indicating that the study provided WTP as a

one

- time, lump sum or provided annual WTP values for a payment period of five years or less.

. This variable enables the benefit transfer analyst to estimate annual WTP values by setting lump_sum =0. Binary (Range: 0 or 1) 0.18 57 0.3 903

wtp_median B inary variable indicating that the WTP measure from the study is the median. Binary (Range: 0 or 1) 0.0 714 0.25 85

Region and Surveyed Populations

n ortheast Binary variable indicating that the survey included respondents from states within the Northeast U.S., defined

as ME, NH, VT, MA, RI, CT, and NY. This is equivalent to EPA Region 1 plus NY or the Northeast USDA region

. Binary (Range: 0 or 1) 0.0714 0.2585

c entral Binary variable indicating that the survey included respondents from states within the Central U.S., defined as OH, MI, IN, IL, WI, MN, IA, MO, ND, SD, NE, KS, MT, WY, UT, and CO . This is equivalent to EPA regions 5, 7, and 8 or the Midwest and Mountain Plains USDA regions. Binary (Range: 0 or 1) 0.3643 0.4830

s outh B,C Binary variable indicating that the survey included respondents from states within the Southern U.S, defined as NC, SC, GA, FL, KY, TN, MS, AL, AR, LA, OK, TX, and NM. This is equivalent EPA regions 4 and 6, or the Southeast and Southwest USDA regions

. Binary (Range: 0 or 1) 0.1571 0.3652

n onusers Binary variable indicating that the survey was implemented over a population of recreational nonusers (default category for this variable is a survey of any population that includes both users and nonusers ). Binary (Range: 0 or 1) 0.0 857 0.2 809

lnincome Natural log of the median income (in 2007$) for the sample area of each study based on historical U.S. Census data. It was designed to provide a consistent income variable given differences in reporting of respondent income across studies in the metadata (i.e., mean vs. median). Also, some studies do not report respondent income. This variable was estimated for all st udies in the metadata regardless of whether the study reported summary statistics for respondent income. Natural log of income ( 2007$ ) (sample area income ranges from $34,332 to $78,444) 10.745 3 0.1 731

Sampled Market and Affected Resource
mult_bod Binary variable indicating that the survey addressed multiple waterbody types . The eight waterbody type categories are (1) river/stream, (2) lake, (3) all freshwater, (4) estuary, (5) wetlands, (6) river and lake, (7) salt pond/marshes, and (8) multiple (e stuary and fresh water). Takes on a value of 1 if the study is coded as category (3), (6), or (8). Binary (Range: 0 or 1) 0.07 86 0.2 700

river Binary variable that takes on a value of 1 if the study affects a river, such that river length>0, and zero oth erwise. Binary (Range: 0 or 1) 0.6 857 0.46 59

swim_use Binary variable indicating that the affected use (s) stated in the study include swimming

. Binary (Range: 0 or 1) 0.2 643 0.4 425

gamefish Binary variable indi cating that the affected use stated in the Binary 0.05 71 0.23 29September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 9



### Table 2

. Definition and Summary Statistics for Model Variables

. A

Variable Definition Units Mean St. Dev. study is gamefishing. (Range: 0 or 1) boat_use Binary variable i ndicating that the affected use(s) stated in the study include s boating

. Binary (Range: 0 or 1) 0.11 43 0.31 93

ln_ar_agr Natural log of the proportion of the affected resource area which is agricultural based on NLCD , reflecting the nature of development in the area surrounding the resource. The

a ffected resource area i s defined as all counties that intersect the affected resource (s). EPA also tested a variable for the fraction of lands that is developed land classes, but it did not improve model fit. Natural log of proportion (Range: 0 to 1) - 1.43 29 0. 9031

ln_ar_ratio A ratio of the sampled area, in km 2 , relative to the affected resource area (defined as the total area of counties that intersect the affected resource(s), in km 2 ). The total area of counties that intersect the affected resource(s) is given by the variable ar_total_area. From here, ln_ar_ratio = log( sa_area / ar_tota l_area ), where sa_area is the size of the sampled area in km 2

. Natural log of ratio (km 2 /km 2 ) - 1. 1278 2.6 067

sub_ p roportion The proportion of water bodies of the same hydrological type affected by the water quality change, within affected state(s). For rivers, this is measured as the length of the affected river reaches as a proportion of all reaches of the same order. For la kes and ponds, this is defined as the area of the affected water body as a proportion of all water bodies of the same National Hydrography Dataset classification. For bays and estuaries, this is defined as the shoreline length of the water body as a propo rtion of all analogous (e.g., coastal) shoreline lengths. To account for observations where multiple waterbody types are affected, t he variable sub_ proportion is defined as maximum of separate substitute proportions for rivers, lakes, and estuaries/bays.
T he affected resource appears in both the numerator and denominator when calculating sub_proportion

. Proportion (Range: 0 to 1) (km/km) 0.188 0 0.2 911

Water Quality Baseline and Change
lnquality_ch Natural log of the change in mean water quality ( quality_ch ), specified on the WQI (McClelland 1974; Mitchell and Carson 1989). WQI units 2.90 70 0. 6039

lnbase Natural log of baseline water quality, specified on the WQI (McClelland 1974; Mitchell and Carson 1989). WQI units 3.5 889 0.66 97

Notes: (A): Shading indicates that the variable was developed and coded during the current set of revisions. (B): EPA merged studies from the Southeast and Southwest region into a single group because the metadata includes only one observation from the Southwest regi on, Whittington et al. (1994) which studied estuaries in Texas (C): The USDA regions omitted from the regional binary variables are the Mid

- Atlantic region (NJ, DE, MD, DC, PA, WV, and VA) and the Western region (WA, OR, ID, CA, NV, AZ, and AK).



## 3.1 Study Methodology and Year

As often found in meta

- analyses within the valuation and benefit transfer literature (Navrud and Ready 2007), a variety of study and methodology effects can be shown to influence WTP for water quality improvements. The 2009 metadata in cluded numerous variables characterizing factors including, but not limited to publication type, elicitation method, response rate, treatment of outliers, payment period, and study year. EPA coded values for these variables for the new primary studies fol lowing the variable definitions from the 2009 MRM. The majority of the methodological variables in the metadata are binary, with 1 indicating that a given methodology was used in the primary study. Some others, such as study year, are continuous. Most of t he methodological variable s available in the metadata did not improve
model fit and were not included in the revised MRM. Given that choice experiments are being used with increased frequency in the recent literature, EPA coded a new binary variable

, ce , t o flag observations that are based on a choice experiment framework.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 10



## 3.2 Region and Surveyed Populations

The 200 9 metadata included variables describing the location of the study by state and region of the
country although these are used in a fairly simplisti c way in the meta

- analysis. EPA expanded the metadata by delineating sampled area for each of the 140 W TP observations as GIS polygons. The sampled area is defined as the geographic area over which the primary survey was fielded (i.e., the population its m eant to be representative of)

. The Agency based its GIS delineation on the description of the study area from the study documentation and matched to polygon boundaries from publicly available

GIS datasets. The sampled area can vary across observations from the same primary study if the WTP values are based on separate survey samples (this is true for 5 of 51 )

. The sampled areas are typically defined by either jurisdictional boundaries or watershed boundaries :

 For sampled areas defined by jurisdiction al boundaries (e.g., s tates, counties, or cities ) , EPA
defined boundaries using the Census Topologically Integrated Geographic Encoding and Referencing (TIGER) state and county shape files for 1990, 2000, and 2010. By using the shapefile vintage that most closely corresponds to the year in which the study was fielded, EPA is able to accurately identify boundaries at the time of the study.

 For sampled areas defined by watersheds

, EPA approximated sampled area bounda ries by matching to the U.S. Geologic Survey Hydrologic Unit Code (HUC) Watershed Boundary Dataset (WBD) to approximate the boundaries of the study area. 6 

### The Agency first matched the watershed polygon in the HUC dataset based on the watershed name present ed in the study. If the name could not be matched, EPA consulted the survey materials or maps included in the study to either locate the corresponding HUC watershed or use the materials to trace a watershed polygon manually. If maps are not provided, then EPA used any additional locational information presented in the study to identify a HUC watershed to serve as a proxy for the study watershed. This approach generally involved more interpretation than if the study defined the sampled area based on jurisdic tional boundaries ( as in previous bullet). 7



### Income is inconsistently reported across primary studies, with some reporting median income, others reporting mean income, and other s not reporting any income statistics for the sample. After delineating the samp le area for all studies , EPA calculated income statistics for the population within the sampled area using georeferenced historic U.S. Census income data to overcome reporting deficiencies and improve consistency across observations. Historical median inco me data is available for 1980, 1990, 2000, and 2010

. Each sampled area was matched to the data vintage that is nearest to the year in which the study was fielded. EPA derived sampled area income as a population

- weighted average of median

income across cou nties that intersect the sampled are polygons. The population weights were adjusted based on the fraction of county area that is within the sample area. Resulting income estimates were adjusted to 2007$ using the Consumer Price Index (CPI).

6 The HUC WBD is a package of watershed shapefiles for the contiguous U.S. sub divided to seven levels of resolution. Each resolution level is assigned a specific number of HUC digits, with the number of digits increasing with higher resolutions.

The lowest resolution is the HUC 2

- digit, or HUC

- 2, watershed which divide the country i nto 21 distinct drainage regions. The highest classification is the HUC12 dataset, consisting of approximately 98,000 watersheds. Additional information about the HUC watershed and delineation process can be found at http://water.usgs.gov/GIS/huc.html

.

7 EPA encourages authors of future studies to describe the sampled area clearly, to avoid the need for judgment or interpretation when including studies in future meta

- analyses.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 11



### The 200 9 metadata included regional dummies for the study location based on the boundaries of U.S. Department of Agriculture (USDA) regions. 8 

### The USDA regions were used as the starting point for regional assignments. The boundaries of the seven USDA regions are cl osely aligned with the ten EPA regions, EPA generated two new binary variables that combine some regions for which effects were

found to be similar. Northeast identifies studies that sampled respondents in the Northeast U.S. (as far south as New York, c ent ral identifies studies that sampled in the Midwest or Mountain Plains regions of the U.S, and south identifies those that sampled in the Southeast and Southwest U.S.(as far west as New Mexico). The state boundaries for each variable are listed in Table 2

.
EPA also calculated the continuous variables sa_area , defined as the size of the area sample in square

kilometers. Sa_area feeds into the index varia ble, ln_ar_ratio , described in detail in the following

subsection.



## 3.3 Reconciliation of Water Quality Baseline and Change

An important component metadata development is the reconciliation of variables across observations (Johnston et al. 2005; Smith and Patt anayak 2002; Smith et al. 2002; Van Houtven et al. 2007). Although the calculation and reconciliation of most independent variables requires little explanation, there are some

variables for which additional detail is warranted. These include variables cha racterizing surface water quality and its measurement. To reconcile measures of water quality across studies , EPA adapted the prior approach of Johnston et al. (2005), mapping water quality changes to the 100

- point Water Quality Index (WQI). 9



### A large number of the original studies in the metadata ( 10 studies and 3 1 percent of observations) include

the 10

- point water quality ladder (WQL) measures as a native component of the original primary studies either using its numerical values (e.g., 2.5, 5.0, etc

. ) or descriptive levels (e.g., boatable, gamefishing)

. 10


### In other cases, EPA used descriptive information from studies to map baseline and post

- improvement to points on the WQL. To calculate 100

- point WQI values EPA multipl i ed the reported WQL values by t en. In most cases, descriptions correspond closely to levels on WQL

, rendering mapping straightforward. EPA used the following guidelines for mapping:

 WQL a ssignments are made based on consideration of information provided in the primary study

documentati on and, if available, the survey materials. The assignments are not supplemented by external water quality databases or reports. They should reflect the descriptions and metrics presented to respondents within the valuation scenarios.

 Identify recreational uses (e.g., boating, fishing, and swimming) provided in the description of

baseline and improved (or declined) conditions. These uses may be stated directly or embedded in the definition of water quality metrics (e.g., low, moderate, high) and can inclu de additional descriptive information such as effects on sensitive aquatic species or indication of presence of or amount of specific pollutants.

8 Using ecoregion boundaries to define regional dummies is not feasible because it would result in a very small number of observations per region. Each USDA region includes several ecoregions and thus are better suited for grouping studies.

9 Van Houtven et al. (2007) also mappe d water quality changes to the WQL.

10 The WQL (Vaughan 1986) is expressed on a scale of 0 to 10 and can be mapped to the WQI by multiplying by 10. Refer to Chapter 10 and Appendix G of the benefits analysis for the C&D Effluent Guidelines (U.S. EPA 2009) for additional detail on the WQI and WQL.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 12



###  Only consider those metrics that can be reasonably mapped to the WQL. Ancillary improvements

from the improvement plans, such as shoreline trash pickup or terrestrial bird species, are not

considered

.  Exclude observations that describe an improvement in recreational use that is not tied to a water quality improvement (e.g., an increase in fish abundance not based on water quality improvement).

 Start by assigning values corresponding to use thresholds defined on the WQL (e.g., 2.5, 4.5)

. Assign intermediate values when the changes occur within a specific use category or changes extend beyond the minimum fo r provision of use. EPA notes that the majority of assignments directly match values corresponding to WQL use thresholds.

 To the extent possible, assigned WQL values should reflect the “affected” portion of resource(s) described in the survey, that is, t he portion of the resource that is subject to water quality changes

under the valuation scenario.

 In some cases, uses may be supported intermittently due to algae blooms, for example. If provided, use information regarding the frequency of service provisio n to calculate intermediate WQL values by weighting use threshold values.

 For s ome types of environmental contamination, such as high acidity, the survey may state that the affected resource supports swimming but has degraded fisheries. It supports a high e r use on the WQL (swimming), but not a lower use (gamefishing). . In these cases, we specify baseline WQL and WQL changes based on fishing conditions, the use that is actually improved under the

valuation scenario (e.g., a baseline of 4.5 for rough fishing and a post

- improvement value of 5.0 for game fishing ).

Appendix B presents water quality description from the original studies and the assigned baseline and improved water quality conditions for each study. The metadata include two sets of baseline and p ost

improvement WQL values, one reflecting a “low range” and another reflecting a “high range.” In cases where coding was straightforward, these ranges are identical. In other cases, the ranges may differ, reflecting uncertainty regarding uses supported un der baseline or post

- compliance conditions. For example, a study might include a low range of 2.5 to 4.5 and high range of 2.5 to 5.0 if it is unclear whether rough fishing or game fishing is supported under improved conditions. The water quality change

va riables in the MRMs are based on the high ran ge consistent with the 2009 MRM because of slightly

better model fit.

EPA did not identify any systematic variation in results associated with studies for which the WQI was a native component, versus those for which quality changes were mapped to the WQI. A binary variable indicating that the observation used the WQL ( CHNG_WQL ) was not individually significant and did not improve model fit. Figure 1 and Figure 2 illustrate baseline WQI values and WQI changes from the

metadata.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 13 

### Figure 1

. Histogram of Baseline WQI Values from the Metadata (Percent of Studies)

.

Figure 2 . Histogram of WQI Changes from the Metadata (Percent of Studies)

.



## 3.4 Delineation of the Affected Resource and Substitutes

WTP for WQI changes is likely better explained in concert with geospatial factors

, given the geographic heterogeneity in the metadata

. In the context of water quality valuation, the geospatial scale of a water quality change reflects the area over which the environmental change occurs. EPA expects, all else being equal, that WTP for water quality improvements would be directly related to the size and number of

affected waterbodies. EPA defines an affected resource as a waterbody that experience a water quality change under the valuation scenarios presented in the primary study. The number of affec ted waterbodiesSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 14



### assessed in the primary studies varies widely across the metadata from single waterbodies to all waterbodies throughout a state or small region. The existing metadata identifies the geographic scale of the affected resources using four cate gories : (1) single waterbody, (2) multiple waterbodies, (3) a small region, or (4) a large region. Table 3 summarizes the number and percent of observ ations in each of the

four geographic scale categories. As part of the revisions, EPA delineated the affected waterbodies by matching waterbody descriptions and maps from the primary studies to publicly available georeferenced

datasets. By delineating reso urces using GIS, EPA is able to incorporate detailed spatial and hydrological

features of the waterbodies. As with sampled area, some primary studies provide separate WTP values for multiple affected resources or sets of affected resources. These are always treated as separate observations

within the metadata.

Table 3

. Summary of Scale of Resource Changes i n the Meta

- data

. Category Variable Description Observation
Count % of

Observations Single Waterbody Single Single, discrete waterbody such as a river
or lake. 49 35.0 %

Multiple
Waterbodies Multi Multiple discrete waterbodies (e.g., 2 rivers) 13 9. 3 %

Small Region Sm_reg S mall region, includes watersheds and

state

- wide analyses 77 55.0 %

Large Region Lg_reg Large region , includes one estuary analysis that includes multiple states. 1 0.7%

Total

- - 140 100.0%



### The datasets used to map affected resources vary by affected waterbody type. For rivers and lakes, t he Agency opted to match to the NHDPlus dataset over other publicly available georeferenced hydrologic datasets (e.g., E2RF1) because it is a comprehensive, national dat aset of hydrologic features such as rivers, lakes, ponds and catchments, and contains tabular data that can be linked to these hydrologic

features. For estuaries and coastlines, EPA used the National Oceanic and Atmospheric Administration (NOAA) Global Sel f

- Consistent Hierarchical, High

- Resolution Geography Database (GSHHD). EPA attempted to match the name of the affected resource to the datasets, and if a match could not be found,

u sed narrative descriptions and maps from the primary study to manually sele ct the affected waterbodies

in ArcGIS. Once delineated, EPA calculated affected shoreline length for all waterbody types as proxy for the human/water interface, al though some uses actually occur in open water . For rivers, shoreline length

is double the rea ch length to reflect both shorelines. EPA also calculate d the area of affected lakes.

Observed mean WTP in a primary study is also related to the extent of market analyzed (Loomis 2000; Loomis and Rosenberger 2006). It is established in economic literature that mean WTP often declines as distance increases to an affected resource (i.e., distance decay) (Bateman et al. 2006; Jørgensen et al. 2013; Schaafsma et al. 2012)

. Because t he scale of the sampled market area varies g reatly across primary

studies and past work has shown that such differences can have important implications for welfare estimates, the accuracy of benefit transfers depends on the ability to adjust for the distance between affected populations and the resource (Bateman et al. 2006; Johnsto n and Duke 2009 ; Johnston and Ramachandran 2013; Loomis and Rosenberger 2006 ). The size of the sampled area ( sa_area ) in the

metadata var ies greatly across the metadata mean of 49, 065 mi 2 

### with 5 th 

### and 95 th 

### percentiles of 13.3 and 104,094 mi 2 

### , respectively. Sa_area is not correlated with the size of the affected resource because various factors affect researcher’s sample selection including budget and resource constraints and study objectives. As a result, the metadata does not provide clear i nformation regarding the extent of the marketSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 15



### for the affected resources or distance effects. Figure 3 and Figure 4 show sampled areas and affected resources for four examples from the metadata . The examples were selected to show a range of resource types, sizes, and sampled areas.

Preliminary mode ling indicated that model performance wa s enhanced, in terms of model fit, variable significance, and consistency with theoretical expectations, when resource size was specified as a function of sampled market area ( sa_area )

. The l n_ar_ratio variable reflects the size of the sampled market relative to the size of the affected resource, defined as the (natural log of the) size of the sampled market area ( sa_area ) divided by the geographic area encompassing the affected waters in square kilometers ( ar_to tal_area ). Ar_total_area is calculated based on the area of counties that insect the affected resource. For example, if a study valued all freshwater in Iowa, then ar_total_area would equal
the area of Iowa . On other hand, ar_total_area would equal the are a of a single Iowa county if the

affected resource fall s enti rely within the county boundary. The area ratio, ln_ar_ratio

, is expected to have negative sign reflecting two effects. First, holding all else constant, stated preference surveys over larger ma rket areas imply greater distances between individual households and affected waterbodies, and thus lower WTP per household . Secondly, improvements to more waters within the sampled area should be associated with greater WTP, ceteris paribus

. About 55 per cent of observations in the metadata estimated WTP for water quality changes in a single watershed, group of watersheds, or a state. The other 45 percent focused on a single waterbody or small

set of waterbodies (e.g., one lake or 3 lakes )

. Specifying the affected resource area requires some assumptions if the affected resource is defined as a river or lake (e.g., EPA assumed that counties

inter secting the affected water bodies represent the affected resource area). As noted above, EPA used intersecting counties to define all affected resources , includ ing discrete rivers, as geographic areas while avoiding the arbitrary selection of a boundary distance.

Relative to the alternatives

, l n_ar_ratio provides a more intuiti ve means to capture the scale of the affected resource occurring throughout an area, such as watershed or region which is likely to be the case in the context of national rulemakings. Model fit decreases when the affected area ratio is included in

non

- logg ed form and the variable ( ar_ratio ) is no longer individually significant. Logging the ratio is also

intuitive because WTP is zero when the resource size is zero. EPA considered, but did not select, a variety

of alternative specifications of the geographic scale and extent of market variables (e.g., shoreline length )

. The availability and quality of substitutes in the surrounding geographic area are also expected to
influence welfare estimates (Loomis and Rosenberger 2006; Schaafsma et al. 2012). All else e qual, WTP should be negatively related to substitute availability. EPA developed a continuous variable, sub_ proportion

, defined as the ratio of affected waterbodies to available use and nonuse substitutes within

the state . EPA expect s that the relationship between distance and substitutes will differ for resource user s and nonusers. For users, there is “a positive cost of access which depends on distance” (Hanley et al.

2003 , p.300 ) , therefore, sites that are farther away will generally be less attractive substitutes all other things being equal . On the other hand, “there seems no theoretical reason to expect such a [distance decay] trend to be seen within the responses of those who are present non

- users” (Bateman et al. 2006, p. 453). There is some evidenc e that a sense of spatial and cultural identity or “ownership” may be important

for nonuse values for some environmental resources (Bateman et al. 2006; Hanley et al. 2003)

. Hanley et al. (2003) provides the example that “ I may have stronger non

- use values for Scottish wildlife sites if I am Scottish than for English wildlife sites” (p.300)

. In the U.S., this may be manifested as affiliation with one’s state of residence. Given these factors, EPA considers the state to be a reasonable basis for the

calcu lat ing the substitute proportion.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 16



### The substitute proportion variable ( sub_proportion ) represents a potential advance over less sophisticated binary variables traditionally used to represent the size or scope of affected resources. For studies

restricted to r ivers, the calculation of substitutes is restricted to rivers of the same stream order(s) as the affected resource to ensure comparability. For non

- river inland waterbodies (e.g., lakes and ponds),

substitute proportion is calculated based on area relative to area of lakes within the state(s). 11 

### For estuaries and coasts, EPA calculated the denominator as all GSHHD coastline miles in the state(s). EPA also tested models with separate substitute variables for each waterbody type, but model performance was not improved.

By delineating an affected area, in addition to the waterbodies themselves, EPA is also able to analyze the development characteristics of areas bordering the waterbodies. EPA used the National Land Cover Dataset (NLCD) to develop a land cover profile of the affected resource area. The NLCD is a high

resolution (30 meter) spatial dataset of land cover across the contiguous U.S. and available for the years 1992, 2001 and 2006 . The NLCD has 16 or 21 categories of land cover depending on the year . For the purpose of consistency and simplicity, EPA defined the agricultural land as the sum of “cultivated

c ropland” and “pa s ture/hay”. The A gency used the change in agricultural land between NLCD vintages to interpolate values for the actual year of the study. The agricultural fraction is calculated as the are a

weighted average of counties within the affected resource area. EPA also developed and tested an analogous variable based on t he fraction of land that is in developed land use categories but it not improve

model fit.

11 NHDPlus does not provide widths of river and streams.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 17 

### Figure 3 . Example Sampled Areas and Affected Resources from Aiken (1985) and Hayes et al. (1992). Sampled areas are highlighted in orange and affected resources are highlighted in blue. Aiken (1985) surveyed residents in and around Fort Collins, Colorado regarding state

- wide improvements to rivers and lakes. Hayes et al. (1992) surveyed Rhode
Island r esidents regarding improvements to upper Narragansett Bay.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 18



### Figure 4

. Example Sampled Areas and Affected Resources f rom Matthews et al

. (199 9 ) and Lant
and Tobin (19 89 ). Sampled areas are highlighted in orange a nd affected resources are highlighted in
blue. Matthews et al (1999) surveyed residents of nearby counties regarding improvements to the Minnesota River. Lant and Tobin (1989) surveyed residents of Ames City, Iowa (location shown in insert) regarding the improvements to the South Skunk River Basin. Lant and Tobin (1989) also surveyed residents of other cities regarding other affected resources. They were included as separate observations and are not pictured here.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 19



# 4 MRM 1 Estimation and Results

Following established practices

, EPA estimated MRM 1 as a standard multi

- level model (Bateman and Jones 2003; Johnston et al. 2005) of the type recommended within the literature (Nelson and Kennedy 2009; Johnston and Rosenberger 2010). As described in greater detail below, the model is specified as a trans

- log model. The model allows for cross

- sectional panel correlation among observations from the same study. If left unaddressed, such correlation could lead to heteroskedastic errors and inefficient, inconsistent p arameter estimates (Rosenberger and Loomis 2000). For each empirical study in the metadata, a central tendency measure ( mean or median ) of WTP for water quality improvements for the representative individual is given by WTP js 

### , the natural log of which is

the measured effect size:

ln WTP

js 

### = ln ( D

js 

### ) β D 

### + ln ( Q 0 js 

### ) β Q 0 

### + ln ( Y js 

### ) β Y 

### + X 1 js 

### β

X 1 

### + ln ( X 2 js 

### ) β

X 2 

### + ε js 

### [ 1 ]

Here, WTP

js 

### is average (or median) WTP for observation j from study s , D j s 

### = Q 1 j s 

### – Q 0 j s 

### is the water quality change calculated as the difference between starting and ending water quality values, Q 0 js 

### is the baseline water quality level, and Y js 

### is the household income for the sampled population. X 1 js 

### and

X 2 js 

### are vectors of variables measur ing environmental conditions and change, research methods, temporal factors, and other factors expected to influence value surfaces across observations and/or
sites. The vectors β D 

### , β Q0 

### , β Y 

### , β X1 

### , and β

X 2 

### represent conforming vectors of parameters to be
estimated.

To allow for potential effects of study

- specific unobservable factors, EPA partition ed ε js 

### into two components such that

ε js 

### = u s 

### + e

js 

### [ 2 ]

Here, u s 

### represents a systematic, normally distributed, study

- level random effect with E( u s 

### ) = 0 and Var( u s 

### ) = σ

u 2 

### , and e js 

### is a standard iid estimation level error, distributed with a zero mean and

constant variance σ e 2 

### (Shrestha and Loomis 2001). Clustering by study to account for within

- study correlation is standard practice within MRMs in the benefit transfer literature (Nelson and Kennedy 2009). Other aspects of the econometric model follow standard conventions; EPA estimated MRM1

as a random effects

, generalized least squares model with robust standard err ors (Nelson and Kennedy 2009). 12 

### Note that this specification implicitly assume s that the WTP estimates reported in the primary studies represent the true underlying WTP in the study population value rather than an estimate (i.e., there is no sampling uncertainty).

Independent variables included in the meta

- analysis are derived from a list of attributes with potential to influence WTP, based on theory and prior findings in the literature. Hence, MRM1 is best characterized as being a weak structural utility theoretic meta

- analysis (Bergstrom and Taylor 2006). When specifying MRM1, EPA emphas ized core scope and scale variables relevant to benefit transfer

. Care was taken to
avoid over

- parameterizing the model with methodological variables. Unlike t he weak structural model , a strong structural utility theoretic model would involve : (i) specifying a structural form of the theoretic indirect utility function ; and (ii) fitting a theory

- consistent mathematical form to the strictly defin ed set of the economic variables (Bergstrom and Taylor 2006). A disadvantage of the strong structural uti lity

12 Although EPA estimated both unweighted and weighted trans

- log models only the unweighted model is presented in this section. Weighting observations based on the number of WTP observations reported in each study produced nearly identical

results.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 20



### theoretic approach is that “restrictions imposed on the utility function specification may preclude some explanatory variables of interest” (Bergstrom and Taylor, p.358) from the model. This type of modeling approach ha s not been adopted in the meta

- a nalysis literature because (a) results tend to be highly sensitive to assumed utility functions, and (b) generally sacrifice empirical fit for a strong assumption regarding the structure of utility.

The trans

- log form was used for the final estimated models based on input from preliminary models that include a variety of alternative functional specifications. This also follows one of the primary model specifications used in the original water quality valuation meta

- analysis of Johnston et al. (2005 ). The

trans

- log specification incorporates the natural log of the dependent variable (the natural log of WTP per household) o n the left h and side and the natural log of household income ( lnincome ), log

- water quality change ( lnquality_ch ), relative geospatial scale ( ln_ar_ratio ), and log

- proportion of agricultural land within the affected area ( ln_ar_agr ) on the right hand side. All binary in dependent variables enter linearly (see variable definitions in Table 1 ). General advantages of the trans

- log specification for meta

- analysis are discussed by Johnston et al. (2005) and Londoño and Johnston ( 2012 ), and include an ability to capture curvature in the valuation function, multiplicative rather than additive effect of independent variables on WTP, and the implied and intuitive constraint that WTP is zero in the limit when key continuous variables (such as the size of water quality change or relative geospatial scale ) are also zero. For logged continuous variables, coefficient estimates may be interpreted directly as elasticities, or the percent change in the dependent variable caused by a 1 percent chan ge in the independent variable, all

else constant. For the linear variables, given a single unit change in the k

- th characteristic, the mean percentage change in WTP is given by ( e  k



### - 1)  100, where β k 

### is the estimated coefficient on the variable

.
The mode l presented in Table 4 was selected after the estimation of numerous preliminary models with different specifications and groups of independent variables. It was selected based on both statistical fit and correspo ndence with theoretical expectations. The trans

- log specification was very close to the bets

fitting Box

- C ox model. As noted above, EPA estimated the models using generalized least squares with

random effects and robust standard errors to account for stud y

- level dependencies. Measures of fit in the log

- space for the illustrated model are good ( χ 2 

### = 683.44, df=24, p<0.0001; R 2 

### = 0. 65 ). These compare favorably to prior meta

- analyses in the published literature. Study

- level errors (σ

u 

### ) are found to be zero

.

T he model performs well, with intuitive results for virtually all statistically significant variables. The model identifies numerous statistically significant coefficients for variables characterizing (1) study methodology, (2) region and surveyed populatio ns, (3) e xtent of the market, study site

, and affected

resources , and 4) water quality. In total, 20 out of 2 4 non

- intercept coefficient estimates are statis tically significant at p<0.10, and 16 of 24 are significant at p<0.01. The ln_ar_ratio and sub_ proportion geospatial variables are both significant at p<0.01 with signs on coefficients that are consistent with expectations. The change in water quality ( lnqualicy_ch ) is also highly significant with a positive sign, indicating that, as expected, highe r water quality changes are associated with higher WTP. The variable for baseline WQI ( lnbase ) is not individually significant. EPA tested many variables specifications with the natural log of WTP ( lnWTP ) as the dependent variable, and a large, statistical ly significant effect was never observed for baseline WQI. 13 

### This effect could be caused by multimodal distribution and might be

better characterized as discrete rather than continuous.

Given the tra ns

- log functional form, these results for lnquality_ch and ln

_ ar_ratio imply diminishing marginal returns to scope and scale (e.g., WTP per unit of water quality improvement declines as the

13 During tests, EPA only found baseline WQI to be individually significant when specified as non

- logged with non

- logged WTP as the dependent variable. However, E PA selected the model with lnWTP as the dependent variable because of the property that forces WTP through the origin when there is no water quality improvement.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 21



### scope of water quality change increases). WTP is lower among nonusers ( nonusers ), another expected

result , because users ma y hold both use and nonuse values, while nonusers hold only nonuse values, by

definition . Methodological variables also show expected patterns. For example, households are willing to pay a greater nominal amount in a one

- shot payment ( lump_sum ) than they w ould be willing to pay in repeated annual payments. 14 

### WTP also varies according to the type of uses potentially affected by the proposed changes (e.g., swim_use, boat_use ). Although these variables may be correlated with the water quality condition

, it is not always the case : For example , waters may be swimmable but the study focuses on improvements in game fishing ( e.g., Banzhaf et al.

, 2006 ). EPA found that model fit was better with
swim_use

, boat_use , and gamefish included (R 2 

### of 0.65 with them in cluded compared to 0.60 without)

and the lnincom e variable was no longer individually significant when they were excluded. The significance of the lnquality _ch variable is also higher when the three use variables are included (p>|z| of 0.005 compared to p> |z| of 0.062 without them ). Th us, dropping the se variables could result in omitted

variable bias.

As expected, sub_proportion has a positive coefficient, indicating that WTP increases as the affected

resource constitute s a larger fraction of available substitutes. The coefficient for ln_ar_ratio is negative because the sampled area ( sa_area ) is in the numerator indicating that average WTP decreases as the sampled market area increases relative to the affected resource. Th is is consistent with the expectation that a larger sample area will include a greater proportion of respondents who are resource nonusers, who are less familiar with the resource, and/or who live at a greater distance from the affected waters.

14 As discussed in Section 3, EPA also tested models with WTP values for short

- term payment p eriods converted to perpetual streams but found that results were sensitive to discount rate assumptions and that these models did not provide any practica l advantages for benefit transfer over models including the lump_sum variable.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 22



### Table 4

. MRM 1 Estimation Results, R andom E ffects with R obust S tandard Er rrors

.

Variable Parameter Estimate Robust

Standard
Error Z P>|z| 95% Confidence Interval

5% 95%

c e 0.5286 0.1994 2.65 0.008 0.1379 0.9194

t hesis 0.6509 0.1991 3.27 0.001 0.2607 1.0411

l nyear

- 0.4688 0.0749

- 6.26 0.000

- 0.6156

- 0.3219

v olunt

- 1.3225 0.2221

- 5.95 0.000

- 1.7579

- 0.8871 outliers_trim

- 0.3843 0.1215

- 3.16 0.002

- 0.6225

- 0.1462

n onparam

- 0.4709 0.1210

- 3.89 0.000

- 0.7082

- 0.2337 non_reviewed

- 0.7419 0.1662

- 4.46 0.000

- 1.0677

- 0.4161 lump_sum 0.8250 0.1397 5.91 0.000 0.5512 1.0988 wtp _median

- 0.2543 0.2169

- 1.17 0.241

- 0.6794 0.1707

n ortheast 0.5198 0.2351 2.21 0.027 0.0591 0.9805

c entral 0.6526 0.1122 5.82 0.000 0.4327 0.8725

s outh 1.4611 0.1395 10.48 0.000 1.1878 1.7344

n onusers

- 0.4352 0.1186

- 3.67 0.000

- 0.6677

- 0.2027

l nincome 0.7451 0.3749 1.99 0.047 0.0103 1.4798 mult_bod

- 0.5380 0.1384

- 3.89 0.000

- 0.8092

- 0.2668
river

- 0.1745 0.1298

- 1.34 0.179

- 0.4290 0.0799 swim_use

- 0.4361 0.2179

- 2.00 0.045

- 0.8631

- 0.0091

g amefish 0.2680 0.1749 1.53 0.125

- 0.0747 0.6108 boat_use

- 0.2918 0.1626

- 1.79 0.073

- 0.6104 0.0269 ln_ar_agr

- 0.3919 0.0911

- 4.30 0.000

- 0.5705

- 0.2133 ln_ar_ratio

- 0.0729 0.0253

- 2.88 0.004

- 0.1225

- 0.0232 sub_ proportion 0.6680 0.1811 3.69 0.000 0.3131 1.0229 lnquality_ch 0.2993 0.1064 2.81 0.005 0.0908 0.5078

l nbase

- 0.0364 0.1233

- 0.29 0.768

- 0.2780 0.2052

i ntercept

- 4.0431 4.3030

- 0.94 0.347

- 12.4768 4.3906

R 2 0.6455 (overall), 0.2414 (within), 0.8361 (between) Number of Obs. 140, 51 clusters (range 1

– 12 observations per cluster)

χ 2 683.44 (p<0.0001) Study Level (σ u ) 0.000 Residual (σ e ) 0.541
Rho (share of overall variance σ e that is account for by σ u ) 0.000



### r WTP 2 (squared
correlation coefficient between the predicted and observed WTP) 0. 5051



# 5 Robustness Tests

Because the meta

- analysis of WTP for water quality improvements is intended to be used for benefit

tr ansfer, following Boyle et al. ( 2013 )

, EPA examine d various factors that indicate whether the in dividual
parameter estimates are robust or fragile. One goal of testing a meta

- regression for robustness is to compel benefit transfer practitioners to “... carefully consider how each study included in a meta

- analysis matches policy

- case conditions when the estimated equations are used to predict policy

- case value estimates” (Boyle et al., 2013, p. 380). EPA note s that these robustness tests do not and cannot provide information on the best or most appropriate specification

– nor does a lack of robustness a ssociated with individual variables or observations imply that these variables or observations should be excluded.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 23



### H owever, these tests can provide insight into the confidence that may be held in certain hypothesis tests

drawn from model results.

The test s presented here focus on horizontal robustness (inclusion/exclusion of observations or studies) and vertical robustness (inclusion/exclusion of variables)

. All horizontal and vertical tests were run on MRM1 presented in Section 4

. Section 5.1 describes horizontal robustness tests using measures of influence on overall regres sion fit and on parameter estimates and significance. Section 5.2 describes vertical robustness tests using extreme bounds analysis (EBA). Section Error! Reference source not
found. concludes with a summary of robustness tests and findings.

EPA did not conduct robustness tests to address study or sample s election bias , as suggested by Boyl e et
al. ( 2013 )

. EP A included in the metadata all relevant studies that met criteria for validity, consistency, and applicability described in Section 2

. The objective of examining s ample selection is to determine whether

the studies included in the metadata represent a random sample from the possible universe of studies and study sites , or whether they follow some systematic pattern of selection. If there is a systematic pattern of

s election, then the WTP estimates predicted by the MRM may be biased. However, in this case, it is not feasible to specify a meaningful sample selection equation because various factors are known to effect researcher’s site selection for water quality valua tion studies. These factors include, but are not limited to, the national or regional importance of the water bodies, the extent of current or future water quality impairments, and concern of the local public for environmental issues. Given the number and nature of factors that affect selection, specifying a valid and unbiased selection equation is not straightforward, and

a mis

- specified equation could provide misleading results. EPA also note s a relatively good geographic

a nd water body type coverage of th e studies included in meta data indicating that a selection bias (if any) is unlikely to affect geographic location or body type. Figure 5 Error! Reference source not found. below depicts a geographic distribution of the studies.September 29, 2015 WQ Meta-Analysis Peer Review Document | pg 24



### Figure 5

. Map Depicting Geographic Distribution of Studies by Waterbody Type. The numbers indicate the number of studies for that state. Their sum does not equal the total number of studies because some studies include multiple states. 

## Type

all freshwater

nver!Stream

river/stream and lake**riveriS~eam** 

### and ostiJary

esluary

sail pOI'Idfmarshes

m!Atipl& warerbodySeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 25



## 5.1 Horizontal Robustness

5.1.1 Introduction



### Boyle et al. (2013) describe “horizontal robustness tes ts” as a series of analyses which examine the influence of individual observations and studies on both the global fit of meta

- regressions and the precision and sign of individual estimates. While iteratively removing one observation at a time helps to

iden tify influential observations, prior research has shown that study

- specific groups of observations can exert influence when individual observations do not. Because the present water quality meta

- analysis explicitly acknowledges within

- study dependencies of multiple observations per study, EPA proceed ed directly to leave

- one

- study

- out robustness checks. Leave

- one

- study

- out horizontal robustness attempts to determine the joint influence of observations from a single study. Further, observations from a study can [collectively, if there are more than one] lead parameter estimates to deviate from estimates based on the

full meta

- dataset. If studies or observations have substantial influence on coefficient signs and/or significances, it suggests that they are suf ficiently different from the remaining body of literature. When regression coefficients estimated from reduced datasets deviate substantially from the main regression estimates, such differences can indicate the omitted observations are outliers from a dat aset with otherwise narrow focus, or it can indicate that the meta

- analysis has captured a broad range of heterogeneous estimates. Note that having captured a range of heterogeneous estimates is not necessarily bad, since having a wide variety of data enab les the analyst to better comment on broad trends in the literature (rather than having narrow

- sighted focus on a particular case). Hence, these results should not

be used in the absence of other information to dictate the removal of individual studies fr om the metadata. Such decisions require a theoretically informed consideration of all factors surrounding metadata content and consistency.

EPA implement ed a vari ety of horizontal robustness test procedures which Boyle et al. (2013) proposed

. To implement horizontal robustness tests EPA used the PROC MIXED procedure in SAS ( SAS/STAT 9.2, 2010 ) , which is appropriate for the models with dependencies across studies. PROC MIXED performs a leave

- one

- study

- out (iterative) assessment of the following three robustness measures: overall model fit, the degree to which individual parameter estimates change, and the degree to which the statistical significance of individual parameter estimates changes. The PROC MIXED procedure for horizontal robustness generated errors when run with study

- level random effects. This is not surprising given that study

- level random effec ts were found to be zero (see Table 4 ). EPA dropped study

- level random effects from the PROC MIXED procedure in favor of repeated measures estimation w ith study identification number as the subject.

5.1.2 Horizontal Robustness Testing Methods

5 .1.2.1 Overall Influence



### Removing observations can change the objective function being minimized during estimation. Examining changes in this function before and after removing a group of observations provides a global, summary measure expressing the joint influence of the removed observations on all of the estimated parameters.

 PROC MIXED employs the likelihood displacement measure, which is equal the amount by

which th e log likelihood of the full data changes if one were to use an estimate based on fewer data points times two

.  Likelihood displacement needs to be compared to a χ 2 

### distribution with q degrees of freedom,

where q is the number of fixed effects and covariance parameters that are subject to updatingSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 26



### when the group of studies was removed.. Since the 25 coefficients are updated each time a study

is removed

, EPA compare s the likelihood displacement statistic to χ 2 

### = 37.653 (α= 0.05).

No studies in the me ta

- data exert a significant degree of influence on the global regression based on comparison to the χ 2 

### criteria of 37.653

. Three stu dies, Collins et al. (2009), Desvousges et al. (1997), and Lant and Tobin (1989), have restricted log likelihood distances greater than 10 but do not exceed the χ 2



### criteria . EPA used 10 only as a heuristic cut

- off to highlight the studies that are influential relative to

other studies in the metadata. 15



### 5 .1.2.2 Change in Individual Parameter Estimates



### Although no likelihood di splacement statistics suggested that removing a study would induce substantial overall changes to the meta

- regression, EPA conducted additional tests to determine if studies influence results in more specific aspects relevant to meta

- analysis interpretatio n and applications. Beginning with changes in individual parameter estimates when individual studies are excluded, EPA followed Schabenberger’s (2004) suggested steps:

 Remove a group of observations and examine the change in the overall parameter vector. Because the number of random effects and variables can be large, examining this change is a good summary of the influence that a group of observations has on the component variables in the

meta

- regression.
 SAS implements two summary statistics that represe nt study’s influence on coefficient estimates:

the quadratic form of Cook’s D and MDFFITS, the multivariate statistic DFFITS

. For both of

these statistics

, large values indicate that the change in the parameter estimate is large relative to

the variabilit y of the estimate.
 Because both statistics describe the (normalized) change in coefficient estimates when removing an observation (or group of them), a value of 1 for these statistics indicate no change. Following Kennedy (2008), EPA flag ged studies with D > 1 as influential. Flagged values are highlighted in

blue in Table 5

. .  Table 5 indicates that four studies have either a D or M DFFITS greater than one: Croke et al. (1987), Desvousges et al. (1987), Downstream Strategies (2008), and Lant and Tobin (1989)

. The

D and M DFF ITS values for these studies range from 2.08 to 4.63 , indicating that these studies are influential, with a large change in the parameter estimate relative to the variability of the estimate.

5 .1.2.3 Change in the precision of parameter estimates



### Studies that have little influence on the magnitude of parameter estimates (vis a vis small Cook’s D or

M DFFITS) may still influence the precision of estimates, and thus also affect hypothesis tests and

confidence intervals ( Scha benberger 2004 ) . By comparing variance of estimates based on the full and

reduced datasets, PROC MIXED computes trace and determinants of the variance

- covariance matrix in each round of estimation. When study i is removed, the covariance ratio (COVRATIO ) reports the increase in variance (i.e., the decrease in estimate precision).

The point at which covariance ratios become worrisome is subjective, but the five

- to 9 8 fold increases

seen for seven studies stand out in comparison to other studies in the s ample ( Table 5 : Collins et al. (2009); Croke et al. (1987); Desvousges et al. (1987); Lant and Roberts (1990); Lyke (1993); Takatsuka

15 These studies may r epresent relatively rare contexts, such as reducing acid mine drainage ( Collins et al., 2009) and large water bodies (Desvouges et al. (1997) that address the Monongahela River in Pennsylvania). They also contribute a fairly high number of observations; 7 observations for Collins et al. (2009) , 12 observations for Desvousges et al. (1987), and 9 observations for Lant and Tobin (1989).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 27



### (2004), and Well e (1986))

. COVRATIO values for these studies are highlighted in blue in Table 5 . This degree of influence could be explained by features of these studies. For example, compared to other studies, these influential studies often focus on somewhat different resource characteristics and/or geographic scope of improvement (i.e., 100 perce nt of the affected waters in a given state). EPA investigated Croke et al. (1987) further because its COVRATIO of 97.96 is far and away the highest . This study focuses on urban rivers in Chicago; studies of urban rivers are rare in the metadata. By compari ng variable standard errors with and without Croke et al. (1987) included in estimation, EPA found that the variance for the boat_use variable increased by 117 percent with Croke et al. (1987), by far the greatest percentage increase for any variable. 16 

### Thi s is not surprising given that of the 16 total observations with boat_use=1, six (38 percent) are from Croke et al. (1987).

16 Ten other variables had a percentage increase in variance of 20 percent or greater when Cro k e et al. (1987) was excluded

from estimation.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 28



### Table 5 . Horizontal R obustness R esults by S tudy

. A

Study # Obs Restricted LL
Distance Cook’s D MDFFITS COVRATIO

Aiken (1985) 1 0.482 0.0593 0.1026 0.390
Anderson and Edwards (1986) 1 0.933 0.0758 0.1831 0.025
Banzhaf et al. (2006) 2 0.724 0.0499 0.0531 0.350
Banzhaf et al. (2011) 1 0.014 0.0027 0.0019 2.394
Bockstael et al. (1984) 1 0.155 0.0300 0.0396 0.350

Bockstael et al. (1989) 2 0.157 0.0201 0.0154 1.481
Borisova et al. (2008) 3 2.473 0.1531 0.2074 0.129
Cameron and Huppert (1989) 1 0.129 0.0332 0.0115 3.539
Carson et al. (1994) 2 0.041 0.0085 0.0042 4.606
Clonts and Malone (1990) 3 2.476 0.1612 0.4330 0.037
Collins and Rosenberger (2007) 1 1.105 0.0699 0.1712 0.128
Collins et al. (2009) 7 12.607 0.4825 0.8614 5.394
Corrigan et al. (2008) 1 0.083 0.0243 0.0178 2.529
Croke et al. (1987) 9 5.535 2.0765 0.2251 97.961
De Zoysa (1995) 1 0.546 0.0512 0.0398 3.148
Desvousges et al. (1987) 12 17.873 4.6323 0.2923 6.230
Downstream Strategies (2008) 2 5.172 0.2803 3.9763 0.000
Farber and Griner (2000) 6 0.313 0.0549 0.2211 1.344
Hayes et al. (1992) 2 0.051 0.0080 0.0064 4.863
Herriges and Shogren (1996) 2 1.516 0.1683 0.1029 2.422

Hite (2002) 2 0.480 0.0809 0.0298 3.858
Huang et al. (1997) 2 0.022 0.0013 0.0011 3.169
Irvin et al. (2007) 4 1.030 0.3513 0.0367 4.103
Johnston et al. (1999) 1 0.389 0.0358 0.0463 0.943
Kaoru (1993) 1 0.502 0.0292 0.0315 0.220

Lant and Roberts (1990) 3 0.139 0.0271 0.0169 9.369
Lant and Tobin (1989) 9 11.205 0.6308 3.6023 0.000
Lichtkoppler and Blaine (1999) 1 0.177 0.0346 0.0401 0.652
Lindsey (1994) 8 5.272 0.4811 0.5875 0.005
Lipton (2004) 1 0.046 0.0146 0.0091 3.968
Londoño et al. (2013) 2 2.956 0.1807 0.8961 0.001

Loomis (1996) 1 0.078 0.0175 0.0116 1.261
Lyke (1993) 2 0.033 0.0122 0.0099 7.427
Matthews et al. (1999) 2 0.811 0.0956 0.0606 2.667
Opaluch et al. (1998) 1 0.577 0.0614 0.0395 1.056
Roberts and Leitch (1997) 1 0.174 0.0371 0.0339 0.290
Rowe et al. (1985) 1 0.944 0.1153 0.3665 0.006

Sanders et al. (1990) 4 0.932 0.1216 0.0900 0.741
Schulze et al. (1995) 2 1.054 0.1271 0.0918 4.274
Shrestha and Alavalapati (2004) 2 0.248 0.0679 0.0299 2.013September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 29



### Table 5 . Horizontal R obustness R esults by S tudy

. A

Study # Obs Restricted LL
Distance Cook’s D MDFFITS COVRATIO

Stumborg et al. (2001) 2 0.043 0.0073 0.0062 2.956
Sutherland and Walsh (19850 1 0.022 0.0047 0.0029 2.292
Takatsuka (2004) 4 0.756 0.2083 0.0731 6.965
Wattage (1993) 3 0.135 0.0249 0.0274 3.897
Welle (1986) 6 0.507 0.1320 0.0407 9.419
Welle and Hodgson (2011) 3 1.341 0.1755 0.1380 3.643
Wey (1990) 2 1.915 0.0946 0.1412 0.646
Whitehead and Groothuis (1992) 3 4.562 0.2037 0.6047 0.000
Whitehead (2002) 3 0.819 0.0574 0.0481 1.521

Whitehead et al. (1995) 2 0.048 0.0107 0.0032 4.543
Whittington et al. (1994) 1 0.007 0.0009 0.0007 2.451

Notes: (A): Highlighted in blue are Cook’s D and MDFFITS values greater than 1, COVRATIO values greater than 5, and restricted log likelihood values greater than 10.



## 5.2 Vertical Robustness

5.2.1 Introduction



### Extreme bounds analysis (EBA) is the process of conducting, “a sensitivity test that examines how robustly the dependent variable of a regression model is related to a variety of possible determinants” ( Hlavac, 2014a ) . Hlavac describes EBA as a process of,

“...finding out which variables from the set X are robustly associated with the dependent variable

y . To do so, we run a large number of regression models. Each has y as the dependent variable and includes a set of standard explanatory variables F that are included in each regression model.
In additio n, each model includes a different subset D of the variables in X . Following the convention in the literature, we will refer to F as the free variables and to X as the doubtful
variables. Some subset of the doubtful variables X might be so

- called focus var iables that are of particular interest to the researcher. The doubtful variables whose regression coefficients retain their statistical significance in a large enough proportion of estimated models are declared to be robust, whereas those that do not are l abelled fragile. More formally, to find out whether a focus
variable v in X is robustly correlated with the dependent variable y , we estimate a set of regression models of the following form:

y = α j 

### + β j 

### v +γ j 

### F + δ j 

### D j 

### + e

where j indexes [each] regress ion model, F is the full set of free variables that will be included in every regression model, D j 

### is a vector of k variables taken from the set X of doubtful variables,
and e is the error term.” ( Hlavac, 2014a, p. 3 ) According to Hlavac (2014 a ), D j 

### has “conventionally been limited to no more than three doubtful variables per model (Levine and Renelt 1992; Achen 2005).” Levine and Renelt ( 1992 ) commented on

use of such a limited nu mber of doubtful variables (and the choice of which to test), suggesting that for reasonableness and simplicity in analysis, they “... only search over seven indicators that [arguably] represent a reasonable conditioning set [on the significance of variables of interest].” The authors also, “... further restrict the pool of variables from which we choose D

- variables by excluding variables that, aSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 30



### priori, might measure the same phenomenon [as the variable of interest v].” In practice, EBA could treat
as “doubtfu l” the full set of a researcher’s explanatory variables, if the researcher held no basic a priori expectations or evidence about which types of variables are most robustly associated with one another ( Boyle, et al., 2013 ; Hlavac, 2014b )

. Like horizontal robustness, ver tical robustness tests cannot be used alone to determine whether individual

variables should be included or excluded from a meta

- regression model. Such assessments must consider a variety of issues including theoretical expectations, the risk of omitted v ariable bias, the potential impact of multicollinearity, and other factors. Also, these tests do not provide insight into the ability of the overall model to predict WTP. In some cases, models including non

- robust (or fragile) variables can predict bette r than those excluding these variables. Hence, results of these tests must be considered carefully, within the context of all other information available about a particular meta

- regression. However, the results can inform the confidence that researchers g ive to hypothesis tests.

5.2.2 Approach and application to WQ WTP Analysis



### EPA implemented EBA using ‘ExtremeBounds,’ a new user

- submitted packaged program for the R Statistical Computing Environment ( Hlavac, 2014b ) . This package is more advantageous than other approaches that EPA is awar e of, such as using the .ado program ‘eba’ in Stata (which has limited functionality), or hand

- coding an original EBA script.

The broadest EBA of the WQ WTP meta

- regression would treat all variables as “doubtful,” and would estimate regressions with all possible combination of explanatory variables in Table 4

, ranging from one explanatory variable to all 24. Such computations are potentially valuable in that no a priori assumptions need to be made about relative stability of some variables over others, but th e millions of regressions required at this scale of analysis is beyond the computational power and resources available. As an initial test of EBA, EPA specified the ‘ExtremeBounds’ model with the following settings:

 All variables in MRM1 were included duri ng robustness testing

. The MRM needs to account for basic indicators of study type, method, and quality because it aggregates data from studies using a wide variety of survey methods and statistical estimators. Failure to include controls for these

factors will result in omitted variable bias on other variables. EPA thus required a set of eight “free” variables to appear in every regression: thesis, volunt, non_reviewed, nonparam, lump_sum, wtp_median, ce , and outliers_trim . Because these variables are no l onger tested in the regression

- varying combinations of variables, it re duces the number of regressions to 63,018.

 EPA treated all remaining variables as “focus” variables, and thus consider their relationship with
lnWTP as “doubtful”. One or more of the “f ocus” variables appears in each of the estimated regressions. EPA add ed combinations of variables in each specification, in combinations of up to

10 extra variables at a time 17 

### . Added variables are picked from the set of : lnyear, nonusers, lnincome, northea st, central, south, river, swim_use, gamefish, boat_use, mult_bod, ln_ar_agr, sub_proportion , ln_ ar_ratio

, lnbase, and lnquality_ch

.  To eliminate the influence of coefficient estimates from model specifications with high multicollinearity, EPA specified a maximum acceptable variance inflation factor (VIF) of 10 ( Kennedy 2008 )

.  To reduce the time r equired to complete the analy sis, EPA estimated a random sample of 10,000

of the 63,018 possible regressions by the above model settings ( 16 percent). This procedure,

17 Note that this produces regressions that are smaller in the number of variables than the estimated meta

- regression in Table 4

, but EPA used the Stata

- recommended 50 degrees of freedom in all tests

.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 31



### “yields unbiased estimates of the quantities of interest that can give the researcher a g ood sense of

the full EBA results within a reasonable ti me frame” (Sala

- i

- Martin et al. 2004).

5.2.3 EBA Vertical Robustness Results



### Table 6 lists the number of regressions on which the EBA analysis is conducted. The right

- hand column reports the total number of regressions which include each of our focus variables . Because no reg ression estimates were eliminated from EBA due to high VIF, EBA is based on this number of coefficients.

Histograms in Error! Reference source not found. and Error! Reference sou rce not found. display the

distribution of coeffici ent estimates for free and focus variables in our series of regressions, respectively.

Where a large portion of a variable’ s coefficient distribution falls to the right of the vertical red bar at

zero, most of the estimated coefficients for a variable are positive. Estimated coefficients for several key variables, including lnquality_ch

, south , and thesis are positive in all r egressions , and the estimated

coefficients for lump_sum

, lnincome

, northeast

, sub_proportion

, ce , central , and gamefish are predominantly , although not always, in the positive

.
On th e other hand, in cases where a large portion of the distribution falls to the left of (below) zero, most of the estimated coefficients are negative. Estimated coefficients for volunt

, non_reviewed

, lnyear

,

nonusers , river , and ln_ar_agr were always negative in our regressions, and the estimated coefficient for

nonparam , wtp_med ian

, outlier s_trim

, swim_use

, boat_use

, mult_bod

, and lnbase was usually negative

. The result for ln_ ar_ratio is more mixed, with th is variable ’s distribution covering substantial negative

and positive regions

. Also

, ln_ ar_ratio appears bi

- modal: in one cluster of estimation results , the

coefficient is approximately zero, and in another cluster, it is negative

. P otent ial causes of this effect are discussed below.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 32 

### Table 6 . Number of E stimates U sed to E stimate EBA of F ree and F ocus V ariables

. A

Variable Free or Focus? Number of estimates

i ntercept F ree 10,000

t hesis F ree 10,000

v olunt F ree 10,000 non_reviewed F ree 10,000

n onparam F ree 10,000 lump_sum F ree 10,000 wtp _median F ree 10,000

c e F ree 10,000 outliers_trim F ree 10,000

l nyear F ocus 4,9 38

n onusers F ocus 4,849
lnincome F ocus 4,915

n ortheast F ocus 4,943

c entral F ocus 4,899

s outh F ocus 4,912
river F ocus 4,915 swim_use F ocus 4,913

g amefish F ocus 4,878 boat_use F ocus 4,881 mult_bod F ocus 4,952 ln_ar_agr F ocus 4,954 sub_ proportion F ocus 4,855 ln_ ar_ratio F ocus 4,982

L nbase Focus 4,799 lnquality_ch F ocus 4,905
Notes: (A): 63

, 018 regressions were possible, of which 15.9 percent were estimated (n=10,000). Each estimate comes from a separate regression.



### Table 7 reports the information on the distribution of coefficient estimates in tabular format. For each variable, it also shows the mean coefficient estimate and the mean of the standard errors on estimated regression coefficients ; whether or not the variable was predomi nantly positive or negative

, and the percent of cases it held that sign; and the percent of cases where the estimated coefficient was both of the dominant sign and statistically significantly different from zero (at α = 0.05).

Table 8 and Table 9 extend the raw results, d emonstrating EPA’s conclusions about vertical robustness

based on Leamer and Sala

- i

- Martin robustness tests. The main goal of the extreme bounds analysis is to provide insight on the degree of robustness in the parameter estimates of policy variables, unde r the

inclusion/exclusion of valuation method variables.

In general, EPA found that key variables (e.g., water quality changes) are robust under reasonable testing criteria, speaking to the overall quality of the estimated benefit transfer function. Other variables denoting resource and sampled population characteristics are more fragile (e.g., sub_proportion

, lnincome ). In

some case s, this fragility may be attributed to relationships between two variables. For example, the bi

modal distribution of ln_ar_r atio suggests the relationship between the size index of a water body and willingness to pay for water quality improvements may be stratified across different resource or study characteristics. For example, ln_ ar_ratio is weakly correlated with both sub_pr oportion (rho =

- 0. 08 ) and ln_ar_agr (rho =

- 0. 39 ). Thus, in the robustness check they tend to show lower stability.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 33



### In the case of other fragile variables , the results of robustness tests can inform value s used in benefit transfer applications. For example, fragile free variables ( e.g., non_reviewed ) or resource characteristic variables may be assigned a value of zero.

Several more detailed comments can be made regarding interpretation of the test results:

1. Lea mer’s criteria suggest most coefficient estimates are fragile. However, Leamer’s criteria are

o ften criticized as a priori conservative: they have been described as both “demanding” ( Hlavac,

2014a ) and “ extremely strict ” ( Boyle et al. 2013, p. 381 )

. Sala

- i

- Martin commented that because the Leamer’s EBA test is “...too strong for any variable to pass it ...giving the label of non

robust to all variables is all but guaranteed” ( Sala

- i

- Martin 1997, p. 179 )

. Thus, g iven other economists’ experience implementing these tests , E PA is neither surprised nor alarmed at results
based on Leamers’ criteria

.
2. Sala

- i

- Martin’s procedure bases robustness tests on a more “reasonable” approach considering the weight of evidence (rather than deferring to the single most extreme result) of the series of regressions ( Hlavac 2014a ) . Results of this test suggest that eight of the variables are robust ( thesis

, volunt

, lnyear

, nonusers , south

, river

, ln_ar_agr , and lnquality_ch ). Here, robust variables are those for which 95 percent or more of the CDF lies either above or below zero (following

Sala

- i

- Martin’s criterion). Under somewh at less

- stringent cutoff of 90 percent, three additional variables approach robustness ( non_reviewed

, nonparam , and outliers_trim ). These results are encouraging since the water quality change variable ( lnquality_ch ) is the primary focus for

benefit transf

er

.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 34



### Figure 6 . Histograms S ummarizing C oefficient E stimates R esulting from E xtreme B ounds A nalysis of F ree V ariables. Notes: Vertical red lines are at zero, and blue kernel density curves are non

- parametric approximations of a regression coefficient’s distribution.
Source: Uses Hlavac’s (2014 b ) R package ‘ExtremeBounds.’ outliers_trimSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 35



### Figure 7 . Histograms S ummarizing C oefficient E stimates R esulting from E xtreme B ounds A nalysis of F ocus V ariables. Notes: Vertical red lines are at zero, and blue kernel density curves are non

- parametric approximations of a regression coefficient’s distribution.
Source: Uses Hlavac’s (2014 b ) R package ‘ExtremeBounds.’ sub_proportion ln_ar_ratio lnincome

riverAugust 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 36



### Table 7 . Means and S tandard E rrors of E stimated C oefficients over 10,000 E stimated
R egressions.

Variable Mean Coefficient Estimate (range) Mean SE Predominant

Sign % Est.’s Are

Predominan t Sign % Est.’s Are

Predominant Sign and Significant A

i ntercept 0.931 (

- 14.733 to 9.701) 2.688 Positive 64.8 50.9

thesis 0.982 (0.518 to 1.39) 0.322 Positive 100.0 97.8
volunt

- 0.898 (

- 1.506 to

- 0.443) 0.236 Negative 100.0 100.0 non_reviewed

- 0.57 (

- 1.122 to

- 0.083) 0.286 Negative 100.0 46.1

nonparam

- 0.295 (

- 0.698 to 0.067) 0.163 Negative 98.2 44.1 lump_sum 0.075 (

- 0.331 to 0.762) 0.209 Positive 62.0 10.5 wtp _median

- 0.262 (

- 0.574 to 0.111) 0.247 Negative 97.5 5.6

ce 0.127 (

- 0.401 to 0.764) 0.220 Positive 66.7 13.5 outliers_trim

- 0.361 (

- 0.733 to 0.019) 0.192 Negative 99.9 42.8 lnyear

- 0.318 (

- 0.56 to

- 0.103) 0.111 Negative 100.0 81.7

nonusers

- 0.346 (

- 0.594 to

- 0.157) 0.161 Negative 100.0 49.8
lnincome 0.667 (

- 0.339 to 1.779) 0.454 Positive 97.5 31.4
northeast 0.498 (

- 0.168 to 1.377) 0.345 Positive 96.2 30.5

central 0.128 (

- 0.224 to 0.708) 0.160 Positive 65.1 28.4
south 0.996 (0.645 to 1.494) 0.179 Positive 100.0 100.0
river

- 0.374 (

- 0.798 to 0.035) 0.169 Negative 100.0 58.5 swim_use

- 0.151 (

- 0.576 to 0.215) 0.235 Negative 86.5 1.1 gamefish 0.258 (

- 0.107 to 0.708) 0.210 Positive 96.4 25.5 boat_use

- 0.152 (

- 0.427 to 0.155) 0.261 Negative 92.9 0.1 mult_bod

- 0.152 (

- 0.749 to 0.251) 0.273 Negative 79.4 8.2 ln_ar_agr

- 0.302 (

- 0.519 to

- 0.169) 0.096 Negative 100.0 99.6 sub_proportion 0.347 (

- 0.16 to 0.915) 0.241 Positive 97.8 25.5 ln_ar_ratio

- 0.016 (

- 0.092 to 0.041) 0.032 Negative 61.3 11.1

lnbase

- 0.079 (

- 0.358 to 0.247) 0.124 Negative 76.9 5.3 lnquality_ch 0.295 (0.129 to 0.425) 0.126 Positive 100.0 82.5
Notes: (A): At 95 percent confidence.August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 37 

### Table 8 . Leamer’s Extreme Bounds Analysis Results.

Variable Lower EB Upper EB Robust or Fragile

i ntercept

- 22.225 20.449 Fragile

t hesis

- 0.150 2.110 Fragile

v olunt

- 1.982

- 0.008 Robust non_reviewed

- 1.616 0.545 Fragile

n onparam

- 1.115 0.346 Fragile lump_sum

- 0.831 1.052 Fragile wtp _median

- 1.161 0.619 Fragile

c e

- 0.900 1.296 Fragile outliers_trim

- 1.116 0.364 Fragile

l nyear

- 0.762 0.144 Fragile

n onusers

- 0.864 0.201 Fragile

lnincome

- 1.340 2.471 Fragile

n ortheast

- 0.956 1.955 Fragile

c entral

- 0.595 0.950 Fragile

s outh 0.217 1.821 Robust

river

- 1.115 0.400 Fragile swim_use

- 0.996 0.674 Fragile

g amefish

- 0.556 1.111 Fragile boat_use

- 1.082 0.642 Fragile mult_bod

- 1.354 0.907 Fragile ln_ar_agr

- 0.723 0.020 Fragile sub_proportion

- 0.633 1.299 Fragile ln_ar_ratio

- 0.148 0.114 Fragile

lnbase

- 0.663 0.485 Fragile lnquality_ch

- 0.081 0.690 FragileAugust 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 38 

### Table 9 . Sala

- i

- Martin’s Extreme Bounds Analysis Results.

Variable CDF A (β<0) CDF A (β>0) Robust or Fragile B

i ntercept 32.757 67.243 Fragile

t hesis 0.474 99.526 Robust

v olunt 99.854 0.146 Robust non_reviewed 93.780 6.220 Fragile

n onparam 90.652 9.348 Fragile lump_sum 39.714 60.286 Fragile WTP_median 82.545 17.455 Fragile

c e 35.628 64.372 Fragile outliers_trim 94.091 5.909 Fragile

l nyear 98.420 1.580 Robust

n onusers 96.536 3.464 Robust

lnincome 13.634 86.366 Fragile

n ortheast 14.288 85.712 Fragile

c entral 33.229 66.771 Fragile

s outh 0.002 99.998 Robust

river 95.928 4.072 Robust swim_use 71.409 28.591 Fragile

g amefish 16.720 83.280 Fragile boat_use 70.520 29.480 Fragile mult_bod 67.907 32.093 Fragile ln_ar_agr 99.673 0.327 Robust sub_proportion 12.132 87.868 Fragile ln_ar_ratio 62.134 37.866 Fragile

lnbase 67.292 32.708 Fragile lnquality_ch 1.428 98.572 Robust

Notes: (A): Generic model of CDF; no assumption is made about the distribution of beta coefficients across models. Results are similar to, but not the same as, Sala

- i

- Martin EBA conducted assuming coefficient estimates are distributed normally across models. (B): Based on Sala

- i

- Martin’s (1997) criteria, in which a variable is deemed robust if more than 95 percent of the cumulative distribution function of the beta estimate falls above or below zero.



## 5.3 Summary and Findings from Horizontal and Vertical Robustness
Tests

This section report s results of horizontal and vertical robustness tests to determine the relative influence of individual studies on the regression estimates and significances and the stability of parameter estimates with respect to inclusion of other covariates. While EPA has found some

evidence of both horizontal and vertical f ragility in each of these dimensions, on net the weight of evidence suggests the meta

- regression is relativ ely robust. It would be rare to conduct these tests
on any meta

- regression and find a perfectly robust regression.

Nonetheless, understanding areas of fragility is important for practitioners who may apply this

meta

- regression in the future. For example,

 Most studies appear to exert a modest amount of influence on global fit and (at least at a summary level) individual coefficient estimates . This fi nding suggests that, in the broad picture, model results are not overly sensitive to the inclusion or exclusion of any single study in the meta

- data.August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 39



###  Howe ver, nine of 5 0 studies appear to exert some influence on global fit or specific

variables . For examp le, studies by Croke et al. (1987), Lant and Roberts (1990)

, and

Welle (1986) generate a nine

- fold increase in variance when omitted, as measured by

COVRATIO. As previously discussed, these studies’ influence is likely driven by the
studies’ focus on specific water bodies with unique properties not observed throughout the general sample of meta

- data.

 Relationships between lnWTP and key policy

- related variables are generally robust to the inclusion of various other explanatory variables , suggesting a high degree of confidence in the sign and significance of the relationship between water quality improvements ( lnquality_ch ) and willingness

- to

- pay for these improvements.

On the other hand, robustness testing did identify a number of fragile variables. These include: ln_ ar_ratio and lump_sum . Analysts should place relatively more trust in the robust variables (e.g., lnquality_ch ) than in fragile ones, at least from a vertical robustness point of view (Boyle et

al. 2013). As a result of these robustness tests, EPA has determined it is appropriate to retain all the studies and explanatory variables.



## 5.4 Cross

- Validation Test

EPA also tested the out

- of

- sample predictive accuracy of the MRM1 using a leave

- one

observation

- out, cross

- validation approach

. Under this approach, t he model is re

- estimated 140 times (once for each observation in the metadata), with each observation j sequentially omitted. EPA generated a WTP prediction for each observation j , using parameter estimates from the

– j
model and ob servation j variable values from the metadata.

EPA calculated the accuracy of WTP prediction by comparing it to the observed primary WTP

estimate for observation j from the metadata. Table 10 and Figure 8 summarize the cross

validation results in terms of dollars and percentage differences. EPA found that out

- of

- sample predictive performance of MRM1 was generally similar to that of MRM2, the alternative model described in subsequent sections, with similar errors on average.

Table 10

. MRM1 Cross

- Validation Results.

Metric Mean Percentile Min Max 5% 50% 95% Transfer Error (predicted WTP minus observed WTP, 2007$) - 7.0

- 119.1

- 1.5 66.6

- 220.3 258.6

Absolute Value of Transfer Error (predicted WTP minus observed WTP, 2007$) 41.2 1.5 21.9 148.4 0.1 258.6

Percentage Transfer Error (%) 29.2%

- 61.5%

- 2.6% 14 5 .1%

- 69.9% 1,826.4% Absolute Value of Percentage Transfer Error (%) 62.1% 2.1% 34.2% 145.1% 0.2% 1,826.4%August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 40



### Figure 8

. Results of Leave

- One

- Observation

- Out Cross

- Validation Test for MRM1

. (a) Scatter plot of predicted vs observed ln( WTP ) . (b) Scatter plot of predicted vs observed WTP . (c) Histogram of “transfer error”, defined as the WTP prediction minus the observed WTP. (d) Histogram of the absolute value of percentage transfer error.

(a) (b)

( c ) ( d )



# 6 Benefit Transfer Application

This section illustrates application of MRM 1 to estimation of benefits for water quality improvements under a hypothetical policy scenario, laying out the potential steps and datasets

used for the transfer. The purpose of the hypothetical policy scenario is to abstract from the particulars of a speci fic policy under consideration. For clarity, the example focuses on water quality improvements to a limited number of river reaches in a relatively small geographic area. EPA expects that these steps would be similar when implementing a benefit transfer fo r other waterbody types, multiple water types, or a high number of waterbodies spread throughout a large geographic area as may be required to support a benefits analysis for a national rulemaking.August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 41



## 6.1 Benefit Transfer Example

The benefit transfer approach use s C ensus block group (BG) as the geographic unit of analysis. 18



### It is assumed that households in each BG value changes in the affected waterbodies within a specified distance, here set to 100 miles. A radius of 100 miles is consistent with Viscusi et al. (2 008) based on the assumption that the majority of recreational trips will occur within a 2 hour
drive from home. The waterbodies not affected by a regulation or water quality change within this buffer zone are assumed to represent substitute sites. EPA not es that this assumption is conservative in that it presumes that all households that are not within 100 miles of an affected

resource have zero WTP.

By focusing on a buffer around the BG as a unit of analysis , rather than buffers are affected waterbodies, each household is only included in the assessment once, eliminating the potential for

double

- counting of households even though the buffer around a BG will likely overlap with other

BG areas

. 19 

### Total national WTP is calculated as the sum of estimated BG

- level WTP across all BGs that have at least one affected waterbody within 100 miles. There will be zero WTP for BGs with no affected waters within 100 miles by the assumptions of this analysis. Limiting populations to a specific study a rea (e.g., watershed) may result in underestimation of benefitting population and may bias the substitution effect. Figure 9 presents a hypothetical e xample focusing on two BGs near 5 affected waterbodies, with affected resources and BGs are highlighted in red and the centroid of census both census BGs encircled by a 100

- mile radius

buffer

. 20



### Table 11 summarizes EPA’s approach for assigning values for all variables. Following benefit transfer using the prior water quality MRM (U.S. EPA 2009; U.S. EPA 2012 ; U.S.EPA 2013 )

, methodological variable values are specified based on judgments regarding preferred practice for primary studies. Their values are held fixed across all analyzed BGs. Table 12 presents hypothetical baseline WQI and WQI changes for the affected waterbodies.

Table 13 summarizes BG

- specific variable a ssignments and resulting household WTP estimates using MRM1 . Annual household WTP is multiplied by the total number of households in the BG

to calculate total WTP for the BG . Both per household WTP and total WTP are greater for BG B
than BG A. BG B’s affe cted rivers experience a greater water quality improvement, 2.44 on the WQI compared to 1.69 for BG A, and BG B has a larger household population. The difference in water quality is only partially offset by the greater affected area in BG A.

18 A Census block group is a group of Census blocks (the smallest geographic unit for the Census) in a contiguous

area that never crosses a State or county boundary. A block group typically contains a population between 600 and 3,000 persons. There are 217,740 block groups in the 2010 Census. See http://www.census.gov/geo/maps

- data/data/tallies/tractblock.html

.

19 Population double

- counting issues can arise when using “distance to water body” to assess simultaneous improvements to many water bodies.

20 When applied to an actual scenario each affected waterbody would corresponds to a reach in the National Hydrography Dataset identified by COMID number. For estuaries, coasts, and the Great Lakes, shorelines should
be matched to the Global Self

- Consistent, Hierarchical, High

- Resolut ion Geography Database (GSHHD). It is important to use these two datasets (NHD and GSHHD) to prevent any misspecification due to differences in data

resolution relative to the metadata.August 17August 17, 2012 316(b) Stated Preference Survey Peer Review Document | pg 42 

### Figure 9

. Illustration of Intersection of BGs and COMIDs

.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 43



### Table 11



### . Approach for Specifying Variable Values for Analyzed BGs

. Variable Assigned Value Description Study Methodology and Year

c e 0.5185 Set to reflect the mean for the last ten years of the metadata (2002 on) because choice experiments have become more common in recent years

. t hesis 0 Set to zero because studies published in peer

- reviewed journals are preferred lnyear 2.5257 Set to reflect the mean of the year index value for studies in the metadata (12.5). ln

year is the natural log of year index.

v olunt 0 Set to zero because hypothetical voluntary payment mechanisms are not incentive compatible (Mitchell and Carson 1989)

ou tliers_trim 1 Set to one because survey data that exclude outlier bids are preferable

. nonparam 0 Set to zero because most studies used in the meta

- analysis used regression analysis to calculate WTP values . Some older studies in
the metadata with nonparam =1 just reported the mean WTP value across responses. This is not the preferred approach in the current
literature. non_reviewed 0 Set to zero because studies published in peer

- reviewed journals are preferred

. lump_sum 0 Set to zero to obtain a predicti on that is an annual WTP. wtp_median 0 Set to zero because only average or mean WTP values that when multiplied by the number of affected households will always yield total
benefits. Region and Surveyed Population

northeast 0 Three binary variables that indicate the region in which the BG is located ( northeast, central, south ). The regional boundaries are
defined by state s . For this example, assume that affected BGs are located in the Mid

- Atlantic U.S. (default) and assign zeros for northeast, central, and south. Regional specification will vary across BGs when implementing a national benefit transfer. central 0
south 0

nonusers 0 Zero because BGs a re expected to contain both users and nonusers
of the affected resources within the buffer.

l nincome BG

- specific Set equal to the natural log of me dian income for each BG based on
Census data

. Sampled Market and Affected Resource
mult_bod 0 Binary variable indicating that multiple waterbody types are affected (e.g., river and lakes). For this example, assign a value of zero because calculations are based exclusively on rivers

. river 1 Binary variable indicating that rivers are affected. For this example, assign a value of 1 because calculations are based exclusively on

stream miles

. swim_use BG

- specific Binary variables that identify studies which swimming, gamefish, and boating uses are specifically identified. Specify values for all three variables for each BG based on the length

- weighted average of baseline water quality and water quality changes for a ffected resources within the 100 mile buffer based on the water quality
ladder. gamefish BG

- specific boat_use BG

- specific

ln_ar_agr BG

- specific Use C ensus county boundary layers to identify counties that intersect
affected resources within 100 mile buffer of each BG . For intersecting counties, calculate the fraction of total land area that is agricultural using the National Land Cover Dataset (NLCD). T he ln_ar_agr variable was coded in the metadata to reflect the area surrounding the affected resources. Setting this variable to reflect the entire buffer
would bias WTP predictions if the average rate over the entire bufferSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 44



### Table 11



### . Approach for Specifying Variable Values for Analyzed BGs

. Variable Assigned Value Description

is different from the rate of a gricultural development nearby the
affected resources . Differences in rates are more likely in cases
where there are few affected resources within the buffer or the affected resources are located in a limited portion of the buffer. ln_ar_ratio BG

- specific Set the sampled area ( sa_area ) equal to the area of each BG as if a primary valuation study was being conducted for the BG that asks households within the BG to value improvements to affected
resources within the buffer. Calculate the denominator ( ar_total_area ) by using GIS to intersect affected reaches with the Census county boundary layer and calculate their total area with duplicate counties dropped. Calculating the denominator for each BG tailors the WTP prediction to the size of a ffected resources within the
buffer. sub_ proportion BG

- specific Calculate the ratio of affected reaches miles to the total number of
reach miles within the buffer that are the same order (s) as the
affected reaches within the buffer. Its value can range from 0 to 1. Water Quality Baseline and Change lnquality_ch BG

- specific Calculate the length

- weighted average WQI change for affected COMIDs within the 100 mi buffer of each BG based on water quality modeling. lnquality_ch is the natural log of the weighted average WQI change

. lnbase BG

- specific Calculate the length

- weighted average baseline WQI for affected COMIDs within the 100 mi buffer of each BG based on water quality modeling. lnbase is the natural log of the weight ed average baseline WQI

.



### Table 12

. Hypothetical Water Quality Improvements Used for Benefit Transfer Illustration

.

Census Block Group ID Affected

COMID COMID

Length (mi) COMID

- Specific WQI
Values BG Length

- Weighted Average
Baseline Change Baseline Change

A 1 2 50 3 47.81 1.69

3 7 46 1

4 3 45 2

5 4 52 2

B 8 2 40 4 47.00 2.44

4 3 45 2

5 4 52 2September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 45



### Table 13

. BG

- Specific Variable Assignment for Benefit Transfer Example and Estimated WTP.

Variable Assigned Value/ Notes BG A BG B Study Methodology and Year

ce See Table 11 0.5185 0.5185
thesis 0.0000 0.0000 lnyear 2.5257 2.5257
volunt 0.0000 0.0000 outliers_trim 1.0000 1.0000

nonparam 0.0000 0.0000 non_reviewed 0.0000 0.0000 lump_sum 0.0000 0.0000 wtp_median 0.0000 0.0000 Region and Surveyed Population

northeast Assume both BGs are in the Mid

- Atlantic, therefore assign zeros for all regional variables 0.0000 0.0000
central 0.0000 0.0000
south 0.0000 0.0000

nonusers See Table 11 0.0000 0.0000
lnincome Assume $50,000 for BG A and $55,000 for BG B 10.8198 10.9151 Sampled market and Affected Resources

mult_bod Assign 0 because only one waterbody type is affected (rivers) 0.0000 0.0000

river Assign 1 because analyzing rivers 1.0000 1.0000 swim_use Assign to reflect water quality values based WQL uses corresponding to WQI values in Table 11 0.0000 0.0000 gamefish 0.0000 0.0000

boat_use 1.0000 1.0000

ln_ar_agr Assume agricultural fraction of 0.30 for BG A and
0.20 for BG B - 1.2040

- 1.6094

ln_ar_ratio Calculate assuming an affected area of 270 .3
mi 2 and sampled area of 2.7 mi 2 f or BG A

, and an affected area of 115.8 mi 2 and sampled area of
3.1 mi 2 for BG B. The sampled area varies
becau se the areas of BGs are not uniform. - 4.6052

- 3.6243

sub_proportion Assume substitute proportion of 0.02 for BG A
and 0.01 for BG B 0.0200 0.0100

Baseline WQI and WQI Change

lnquality_ch Correspond to length

- weighted averages presented in Table 11 0.52 47 0.89 20

lnbase Correspond to length

- weighted averages presented in Table 11 3.8 501 3.8 672
WTP Results

Estimated Household WTP per Year Consistent with U.S. EPA (2013), EPA
adds half of the model residual variance to the natural log of WTP prior to calculating the exponential. 21 $25.58 $33.21

Households in BG Assume 200 households in BG A and 300
households in BG B. 200 300

Total BG WTP (2007$)

- $5,116.23 $9,963.22



## 6.2 Benefit Transfer Application Limits

The specific functional form chosen for MRM1 (the log

- log form) is based on statistical fit within a weak structural utility theoretic framework (Bergstrom and Taylor 2006), following approaches in virtually all

21 This adjustment was not included in the cross

- validation prediction s from Section 5.4 , but this should not have a significant effect on the interpretation of cross

- validation results.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 46



### published valuation meta

- regression models i n the literature. For example, t he final functional form contains many factors that are expected (based on theory) to influence WTP

– including income, the water quality change, and the baseline water quality, among possibly many other factors. It also imp lies that WTP necessarily will equal 0 when the water quality change equals 0

— another theoretically expected pattern. Beyond this, no additional functional constraints or strong assumptions are imposed on the

model

— the specification is based on that which appears to best fit the data.

Equation [1] from Section 4 implies the following WTP function:

WTP js 

### = α js 

### Y js β Y 

### Q js β Q 

### D js β D 

### [ 3 ]

where α

j s 

### is a reduced form parameter containing the error term, ε j s 

### . This functional form has two practical limitations on the circumstances under which MRM1 can be directly used for a benefits transfer.

First , the WTP func tion in equation [3] tends to predict larger than expected values for very small water quality changes ( smaller than the smallest change reported in t he meta

- data)

. Commonly us ed utility

functions generally imply that WTP grows (shrinks) nearly linearly wi th small increases or decreases in the quantity or quality of a resource. However, the WTP function in equation [ 3 ] implies that the marginal

WTP for water quality improvements increases rapidly as D approaches zero, i.e., the limit of ∂ WTP / ∂ D

as D goes to zero is ∞.

Studies in the metadata tend to value substantial WQI improvements, in man y cases equivalent to a step or more on the WQL (i.e., 20 points or more on the WQI) (see Figure 2 in Section 3.3 )

. The mean WQI change in the metadata is 21.3 and the minimum is 2.5. However, f uture water regulations may result in

small er improvements in the WQI for some local areas

. Therefore, it is important that the Agency be able to use MRM1 to accurately estimate WTP for small water quality improvements as well.

There are a variety of options to address this concern. One possibility is to use the MRM1 itself to predict WTP in the orange of water quality changes observed in the underlying metadata (i.e., D >=2.5) , and to

use a linear interpolation of model results to predict WTP for water quality changes below the observed range (i.e., D <2.5). Figure 10 illustrates the interpolation approach. The red line is total annual household WTP for various WQI improvements for BG A from Section 6.1 as predicted by MRM1. The curve is steep near the origin, resulting in high marginal WTP for small WQI improvements. The dotted line is interpolated WTP values for water quality changes less than 2.5. Applying this a pproach to the benefit transfer example in 6.1 results in household WTP of $9.70 and total WTP of $1,939 for BG A, and

household WTP of $19.17 and total WTP of $5,752 for BG B. These interpolated estimates for BG A and BG B are 62 percent and 42 percent less than estimates predicted directly by MRM 1 ( Table 13 ).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 47 

### Figure 10

. Illustration of WTP for WQI Improvements for BG A (in red). The dotted line shows interpolated WTP for WQI improvements less than 2.5.



### The second limitation of MRM1is tha t the WTP function in equation [3] does not satisfy the “adding

- up” condition, which Diamond (1996) describes as a necessary condition for internal consistency in benefit estimation, with particular emphasis on stated preference studies. The adding

- up condition states that for

all Q 0 

### < Q 1 

### < Q 2 

### :

WTP ( Y

, Q 0 

### , Q 2 

### ) = WTP ( Y

, Q 0 

### , Q 1 

### ) + WTP ( Y

− WTP ( Y

, Q 0 

### , Q 1 

### )

, Q 1 

### , Q 2 

### ) [ 4 ]

In words, the WTP for a change from state 0 to 2 conditional on baseline income must equal the WTP for a change from 0 to 1 conditional on baseli ne income, plus the WTP from state 1 to 2 conditional on the income remaining after paying for the change from 0 to 1. As noted by Carson (2012, p. 36), “such a test is logically correct given its assumptions,” but often fails to hold for both market and non

- market values

in em pirical settings.

The WTP function given in equation [4] does not in general satisfy the adding

- up condition because

Y β Y 

### ( Q 2 

### − Q 0 

### ) β D 

### ≠ Y β Y 

### ( Q 1 

### − Q 0 

### ) β D 

### + ( Y

− α Y β Y 

### Q 0 β Q 

### ( Q 1 

### − Q 0 

### ) β D 

### ) β Y 

### ( Q 2 

### − Q 1 

### ) β D 

### [ 5 ]

The problem can b e seen more readily by abstracting from the income effect (which is often small enough to be neglected in practical applications):

( Q 2 

### − Q 0 

### ) β D 

### ≠ ( Q 1 

### − Q 0 

### ) β D 

### + ( Q 2 

### − Q 1 

### ) β D 

### [ 6 ]

Hence, benefit transfers resulting from MRM 1 generally will not satisfy the adding

- up condition

.
In cases where a single policy action is expected to lead to significant water quality improvements that attain a water quality objective MRM1 would provide defensible household WTP estimates. For exam ple,

MRM1 could be used for a cost

- benefit analysis of a TMDL that is expected to result in an impaired waterbody attaining all of the applicable designated uses. This is because the water quality goals would be achieved and no further policy actions would be ex pected. However, the A gency often proposes national regulations that address discharges from a single industrial category, such as discharges from steam electric power plants, or from municipal sources, like combined sewer overflows. These sector

- by

- sector national regulations are a necessary incremental approach to implementing the Clean Water Act, yet theySeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 48



### often result in small improvements t o many waters across the country . It is a long standing practice that the Agency takes partial credit for par tial improvements on the WQL resulting from individual regulations. As one regulation may result in a small water quality change, the cumulative effect of multiple regulations could well be significant. If EPA is going to be able to account for the economi c benefits associated with these many small incremental changes it must have a model that can reliably estimate numerous small changes to water quality in an additive way.



# 7 An Alternative Approach : MRM2

To address limitations of MRM1 in terms of satisfying adding

- up and predicting very large WTP for very small water quality changes, the WTP function based on MRM1 can be specified directly in terms of marginal willingness to pay. Then the estimated marginal willingness to pay function can be integrated to

to get a WTP function that automatically satisfies the adding

- up condition, i.e.,

WTP j 

### ( Q 0 

### , Q 2 

### ) = ∫ WTP j Q 

### dQ Q 2 Q 0 

### = ∫ WTP j Q 

### dQ Q 1 Q 0 

### + ∫ WTP j Q 

### dQ Q 2 Q 1 

### [ 7 ]

To implement this approach

, EPA used the following estimating equation, refer red to a s MRM2 in this
report :

ln ( WTP js

D js 

### )

= Q js 

### β Q 

### + ln ( Y js 

### ) β Y 

### + X 1 js 

### β

X 1 

### + ln ( X 2 js 

### ) β

X 2 

### + ε js 

### [ 8 ]

where, consistent with equation [1] from Section 4

, Y j s 

### is the household in come for the sampled population and X 1 js 

### and X

2 js 

### are vectors of variables measuring environmental conditions and change, research methods, temporal factors, and other factors expected to influence value surfaces across observations and/or sites. To implement the model, EPA approximated marginal willingn ess to pay,
WTP js 

### / D j s 

### , with the stated willingness to pay amounts, WTP js 

### , divided by the water quality change, D js 

### (where D

j s 

### = Q 1 j s 

### - Q 0 js 

### ), evaluated at the mid

- point of the range over which water quality was changed, Q js 

### = (1/2)( Q 0js 

### + Q 1js 

### ), for each observation j and study s . The vectors β D 

### , β Q0 

### , β Y 

### , β X1 

### , and β

X 2 

### represent conforming vectors of parameters to be estimated.

This approximation could be avoided by integrating the marginal WTP function in equation [ 8 ] over the range of the wat er quality index change to get an exact expression for the incremental WTP for each

observation:

ln ( WTP

js 

### )
= ln ( Y

js 

### ) β Y 

### + X 1 js 

### β

X 1 

### + ln ( X 2 js 

### ) β

X 2 

### + ln [ 1

β Q 

### ( e β Q Q 1 js



### − e β Q Q 0 js 

### ) ] + ε js 

### [ 9 ]

This expression clarifies how stated WTP is assumed to depend, according to equation [ 8 ], on the initial and final water quality levels. However, equation [ 9 ] is nonlinear in the parameters so would require a custom nonlinear least squares or maximum likel ihood routine for estimation. EPA would not expect large differences between these alternative estimating equations, but for completeness the Agency may pursue this elaboration in follow up work. For all results presented in the present report, EPA used th e approximation in [ 8 ] as the estimating equation and the approximation described in the preceding paragraph to allow estimation of MRM2 using ordinary or weighted least squares.



## 7.1 Estimation and Model Selection Approach

EPA began by estimating a preliminary model that included the same set of regressors that appear in MRM1 except ln Q 0 s 

### and ln D j 

### , which were excluded for the reasons explained above. To estimate the parameters of the candidate model specifications EPA used ordinary least squares (OLS) and weigh ted least squares (WLS). When using WLS, each observation was weighted by the inverse of the number ofSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 49



### observations drawn from its parent study. This approach gives each study rather than each observation equal weight in the regression (e.g., Mrozek and Ta ylor 2002, Van Houtven et al . 2007). 22 

### This is distinct from MRM1, which gives each observation equal weight. Both weighted and unweighted MRMs are common in the published literature, with no consensus on which approach is more appropriate (Johnston

et a l

. 2005).

In all cases, EPA report s cluster

- robust standard errors (Cameron and Miller 2013) and clustered

bootstrap standard errors (Ren et al

. 2010). The Agency calculated the R 2 

### for the WLS model using the sum of weighted squared residuals based on the weighted data, i.e.,

R WLS 2 

### = 1

− ∑ w j ( y j − x j β ̂ WLS ) 2 j

∑ ( √ w j y j − ( 1

N ∑ √ w j y j j ) ) 2 j 

### [ 10 ]

R WLS 2 

### indicates the proportion of the variation of t he weighted dependent variable that is explained by the weighted independent variables. This quantity is maximized by WLS estimation and so is comparable across alternative WLS specifications, but it is not directly comparable to the R 2 

### of the unweighted O LS model (Greene 1997 p 257). For a more comparable statistic, EPA also calculated a pseudo R 2 

### for the
WLS model, R

̃ WLS 2 

### , which is based on the unweighted data and so is directly comparable to the R 2 

### from an OLS model (Willett and Singer 1988):

R WLS 2 

### = 1

− ∑ ( y j − x j β ̂ WLS ) 2 j
∑ ( y j − 1

N ∑ y j j ) 2 j 

### [ 11 ]

If the observation weights were all equal

— i.e., if all studies supplied the same number of observations

— then the weighted least squares estimator would collapse to ordinary least squares and both R WLS 2 

### and

R ̃ WLS 2 

### would be equal to R OLS 2 

### . In the present dataset the weights are not all equal, so R ̃ WLS 2 

### will necessarily

be less than R O L S

2 

### but R WLS

2 

### could be larger or smaller than R OLS 2



### .

EPA also present s measures of out

- of

- sample predictive accuracy for all models based on their respective

l eave

- one

- out cross

- validation performance. For OLS models the cross

- validation measure of fit is:

R CV 2 

### = 1

− ∑ [ y j − y ̂ j ( β ̂

− j ) ] 2 N j = 1 ∑ ( y j − y ) 2 N j = 1 

### [ 12 ]

For WLS models the unweighted y j 

### ’s would be replaced by their weighted counterparts, √ w j 

### y j 

### , in
equation [12]

. R CV

2 

### provides the basis for model selection in the step

- wise regression approach described
above for both OLS and WLS models.

22 Both OLS and WLS should be consistent, but which will be more efficient? This will depend on how each primary investigator chose which estimate(s) to report from among the possibly many more that were generated in the course of each study, and on the relative magnitudes of the within

- and across

- st udy error variances. For example, if all of the studies were based on comparably large datasets (so that their sampling error variances are roughly equal) and if the reported estimates are encompassing subset means of all underlying (including possibly man y unreported) estimates generated by each study, then the means of each study

- level group of estimates would be comparably precise and so WLS would be more efficient than OLS. Alternatively, if the reported estimates are random draws from the underlying es timates, and if the within study error variances are sufficiently large relative to the across study error variance, then each reported estimate would be comparably precise and so OLS may be more efficient than WLS. It should be possible to use a random ef fects model to
estimate the within

- and across

- study error variances and thereby develop an estimator that may be more efficient than either OLS or WLS, but EPA has left that task for follow

- up work. In this section, the Agency focuses exclusively on the O LS and

WLS estimators.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 50



### Because MRM1 and MRM2 use different dependent variables, their respective R 2 

### values are not directly comparable. This is distinct from the non

- comparability between R 2 

### values of OLS and WLS models. To facilitate comparisons of fit between MRM1 and MRM2 estimated by either OLS or WLS, EPA calculated the squared correlation coefficients between the observed and predicted WTP values, r WTP 2 

### , for

all models. 23 

### To generate predictions of WTP using MRM2, the Agency plugged the estimated model parameters into the benefit transfer function given by equation [ 9 ]

.



## 7.2 Results

Results for the full specification of MRM2 estimated by OLS are presented in Table 14 and Figure 11

. Table 14 shows that a majority of the variables included in the model are statistically significant at the 5% level according to the cluster

- robust standard errors. However, the bootstrapped confidenc e intervals are roughly twice as wide as those implied by the cluster

- robust standard errors, so by the bootstrap measures far fewer variables are statistically significant. For data sets of small to modest size such as this one, bootstrapped statistics ma y be viewed as more reliable because they are nonparametric and do not rely on asymptotic assumptions (Efron 1979, Efron 2003, Horowitz 2003). However, it is not clear to us why the bootstrapped confidence intervals are so much larger than the cluster

- robu st standard errors in this case,

so EPA reports both measures for comparison and we flag this discrepancy for closer examination in

follow

- up work.

Measures of model fit are shown at the bottom of Table 14 . The R CV 2 

### value is significantly smaller than

the R 2 

### value for the model. This suggests that the out

- of

- sample predictive performance of the model is substantially worse than its within

- sample fit, which is to be expected. The predictive performance of the model can be improved by excluding (or refraining from including) those explanatory variables that reduce the precision more than they increase the accuracy of the predictions. 24



### Table 14 shows scatterplots of the predicted versus observed observations, as the approximated marginal WTPs in panel (a) and as stated WTPs in panel (b). The latter we re calculated using the estimated model parameters in the benefit transfer function given in equation [ 9 ] , and the observed explanatory variable values. The squared correlation coefficient between the predicted and observed WTP values was r WTP 2 

### = 0.528.

There is an obvious outlier in panel (b), which is observation 17. This is one of two observations from the study by Hayes et al . (1992) . The other observation (number 16) differs from 17 in only two dimensions: ‘swim_use’ = 1 for observation 1 6 rather than 0, and Q = 60 for observation 16 rather than 70. So in the marginal WTP

- Q space, shown in panel (a) of Figure 11 , observation 17 does no t appear as an obvious outlier. However, ΔQ(16) = 20 while ΔQ(17) = 40, so the predicted WTP for 17 is much larger than that for 16, as seen in panel (b) of Figure 11 . To examine the influence of observation 17 on the estimation

results EPA re

- estimated the model excluding that observation. The full model results with observation 17 excluded are presented in Appendix C. The fit of the model in marginal WTP

- Q space increases only marginally without observation 17 (from r 2 

### = 0.612 to 0.618), but the fit in WTP

- Q space increases significantly (from r WTP 2 

### = 0.528 to 0.574). The Agency might conjecture that this ‘stray’ prediction

would be tamed if EPA we re to fit the parameters using equation [ 9 ] as the estimating equation directly, using non

- linear least squares or maximum likelihood, so the Agency highlights that task for follow

- up

23 EPA notes, however, that this comparison is unfair to the WLS models, which minimize the residual sum of squared errors

between the within

- study average observations and predictions rather than the individual observations and pred ictions. To balance the comparisons, EPA also should examine the r WTP 2 values based on the study means for all models.

24 This is the goal of the step

- wise regressions, the results of which are discussed below.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 51



### work. In the meantime, since EPA has so far relied excl usively on estima ting equation [7] , all results in the remainder of this section use all of the data including observation 17.

Because the dependent variable of MRM2 is ( an approximation of ) marginal WTP , if the coefficient on Q were equal to zero this would imply a flat demand curve, i.e., marginal WTP for water quality improvements would not depend on the level of water quality. The coefficient on Q is negative in both the OLS and WLS models, indicating a downward sloping deman d curve. The Q coefficient is statistically significant according to both the asymptotic cluster

- robust standard errors and the bootstrap confidence intervals in the OLS model, but only according to the former statistic in the WLS model. To illustrate the shape of the demand curve, panels (c) in Figure 11 and Figure 12 show the marginal WTP curve over the entire domain of the water quality index for the full specification of MRM2 estimated by OLS and WLS

. To create these curves, the geo

- spatial variables were set to their mean values in the dataset a nd the remaining variables in X 1 

### and X

2 

### were specified as indicated in Table 11 . The dotted red lines are the corresponding 2.5th, 50th, and 97.5th percentiles of bootstrapped marginal WTP predictions at the fixed

values of X

1 

### and X

2


### . As seen in panel (c) of Figure 11 , the OLS estimates using the full dataset suggest that, on average, marginal WTP declines from about $3 per index point at Q = 0 to around $0.5 per index point at Q = 100. For comparison, in an earlier meta

- analysis study based on the 10

- point water quality ladder, Van

Houtven et al. ( 2007) estimated an average WTP by users of $56

- 58 using a linear function and $32

- 37 using a log

- linear function per point on the 10

- point scale.

Results for the preliminary specification of MRM2 estimated by WLS are presented in Table 15 and Figure 12 . The results are broadly similar to the OLS results with a few notable differences. The in

sample fit of the model to the observed marginal WTPs is lower, as it must be since here EPA ha s minimized the squared weighted residuals, but the auxiliary fit of the predicted stated WTPs is nearly as good: the r

WTP 2 

### values for th e OLS and WLS models were 0.528 and 0.518, respectively. The marginal WTP curve, shown in panel (c) of Figure 12 , is also somewhat lower than that est imated by OLS at low

values of Q , starting at just over $2 rather than roughly $3.

One of the notable differences between MRM1 and MRM2 is that in MRM2 the log of median income is generally not statistically significant (it was also near the margin of bein g insignificant in MRM1). This is the case in the full specification estimated by OLS and WLS, as shown in Table 14 and Table 15

. 25 

### The estimated coefficient is positive, suggesting that water quality is a normal good, but it is estimated very imprecisely. It may be inst ructive to conduct a power analysis for this regression coefficient (Dupont et al

. 1998), so EPA flag ged this issue for follow

- up work.

To compare benefit transfer predictions by MRM1 and MRM2, the Agency used the estimated parameters

of MRM2 in the benef

it trans fer function given in equation [ 9 ] to predict stated WTPs for the two hypothetical examples described in Table 13 (with X 1 

### and X

2 

### specified at the same fixed values used to draw the marginal WTP curves, as indicated above). These results are shown in Table 16 Error!
Reference source not found.

. Example A involves a change from 47.8 to 49.5 on the WQI , and Example B involves a change from 47.0 to 49.4. Using MRM2 to estimate the WTP for these changes at the specified valu es of X 1 

### and X

2 

### indicated in the pr evious paragraph, for Example A, the mean is $ 2.4 per household per year using OLS and $2.2 using WLS , and for Example B, the mean is $ 3.4 per household per year using OLS and $3.1 using WLS.

25 This variable also only rarely appears in the final models selected by step

- wise regression, as shown later, in Figures 6 and 8.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 52



### Table 14

. MRM2 P reliminary S pecification E stimated by OLS

. In the far right columns, a single *
indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not
include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept 0.15 0 4.58 0

- 18 .000 15.3 00

ce 0.53 0 0.306

- 0.642 1.61 0
thesis 0.631 0.245

- 0.197 1.8 00 * lnyear

- 0.356 0.101

- 0.928

- 0.07 4 * **

volunt

- 1.47 0 0.223

- 2.15 0

- 0.681 * ** outliers_trim

- 0.494 0.174

- 0.956 0.202 *

non_param

- 0.577 0.118

- 0.917 0.059 * non_reviewed

- 0.578 0.198

- 1.38 0

- 0.03 1 * ** lump_ sum 0.879 0.189 0.315 1.52 0 * ** wtp_median

- 0.348 0.34 0

- 1.34 0 0.646

n ortheast 1.06 0 0.319 0.134 2.24 0 * **

c entral 0.545 0.16 0

- 0.241 1.18 0 *

s outh 1.35 0 0.17 0 0.654 2.02 0 * **

nonusers

- 0.454 0.15 0

- 0.802

- 0.156 * **

lnincome 0.212 0.418

- 1.1 00 1.84 0 mult_bod

- 0.68 0 0.204

- 1.48 0

- 0.004 * **

river

- 0.09 2 0.15 0

- 0.662 0.5 00 swim_use

- 0.384 0.183

- 0.861 0.105 * gamefish 0.526 0.306

- 0.367 1.35 0 boat_use

- 0.715 0.225

- 1.33 0 0.014 * ln_ar_ agr

- 0.344 0.099

- 0.628 0.114 * ln_ar_ ratio

- 0.07 5 0.025

- 0.148 0.055 * sub_proportion 1.14 0 0.236 0.029 1.9 00 * **
Q

- 0.01 7 0.00 5

- 0.03 4

- 0.004 * **

R2 0.612
RCV2 0.421

rWTP2 0.528September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 53



### Figure 11

. MRM2 P reliminary S pecification E stimated by OLS. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal willingness to pay curve. Demand curve shifters are set as described in the main t ext. The black line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th percentiles.

(a) (b)

(c)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 54



### Table 15

. MRM2 P reliminary S pecification E stimated by WLS. In the far right columns, a single *
indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not
include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept

- 1.04 0 5.3 00

- 19.9 00 16.4 00

ce 0.377 0.322

- 0.725 1.55 0
thesis 0.866 0.27 0

- 0.24 0 1.88 0 * lnyear

- 0.412 0.13 0

- 0.912

- 0.03 3 * **

volunt

- 1.39 0 0.239

- 2.15 0

- 0.638 * ** outliers_trim

- 0.367 0.201

- 0.976 0.298

non_param

- 0.408 0.121

- 0.845 0.105 * non_reviewed

- 0.709 0.199

- 1.39 0

- 0.07 7 * ** lump_ sum 0.843 0.188 0.189 1.54 0 * ** wtp_median

- 0.161 0.331

- 1.37 0 0.926

n ortheast 1.18 0 0.337 0.177 2.34 0 * **

c entral 0.561 0.214

- 0.24 0 1.26 0 *

s outh 1.4 00 0.235 0.64 0 2.13 0 * **

nonusers

- 0.586 0.146

- 0.91 0

- 0.178 * **

lnincome 0.333 0.473

- 1.18 0 2.04 0 mult_bod

- 0.827 0.211

- 1.71 0

- 0.01 9 * **

river

- 0.07 9 0.174

- 0.719 0.477 swim_use

- 0.234 0.177

- 0.747 0.301 gamefish 0.233 0.387

- 0.678 1.46 0 boat_use

- 0.725 0.237

- 1.48 0 0.006 * ln_ar_ agr

- 0.271 0.11 0

- 0.589 0.132 * ln_ar_ ratio

- 0.034 0.03 5

- 0.149 0.072 sub_proportion 1.1 00 0.272

- 0.09 7 1.95 0 *
Q

- 0.01 5 0.006

- 0.035 0.00 1 *

R2 0.745
RCV2 0.499
RWLS2 0.548

rWTP2 0.518September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 55



### Figure 12 . MRM2 P relimin ary S pecification Estimated by W LS. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal willingness to pay curve. Demand curve shifters are set as described in the main text. The black line is the point
estimate based on th e primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th percentiles.

(a) (b)

(c)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 56



### Table 16

. WTP [$/yr/household] for Examples A and B from Table 13 C alculated using the F ull
S pecification of MRM2 Estimated by OLS and WLS

. A 

### Confidence intervals are based on 1000 bootstrap samples conditional on full model specification.

Parameter Q 0 Q 1 OLS WLS

Mean 2.5 th 50 th 97 .5 th Mean 2.5 th 50 th 97 .5 th
Example A 47.8 1 49. 50 3.42 0.89 3.40 10.61 2.54 0.82 2.69 8.35 Example B 47.0 0 49.4 4 5.36 1.53 5.29 17.62 4.06 1.33 4.32 12.91 (A) Consistent with WTP prediction s in Table 13

, EPA adds half of the model residual variance to the natural log of the dependent variable prior to calculating the exponential. This adjustment was not included i n the cross

- validation predictions in Figure 11 and Figure 12 , but this should not have a significant effect on the interpretation of cross

validation results.



# 8 Model Limitations and Uncertainty

Because both MRM1 and MRM2 are drawn from the same under lying metadata, they are subject to the same limitations related to data availability. However, the two model structures differ, leading to different advantages and disadvantages. In general, MRM2 may be viewed as making somewhat stronger assumptions (th at may or may not coincide with patterns in the underlying metadata), in return for a set of theoretically

- desired properties in resulting benefit estimates. Two important limitations of MRM1 were

discussed above in Section 6.2 . The primary limitation of MRM2 is that it does not fit the data as well as MRM1. For example, for the squared correlations between the observed and predicted WTP values for MRM1 and MRM2 (w hen both are estimated by OLS) are 0.633 and 0.528, respectively. Additional work may be warranted to more fully compare and contrast the models when estimated using similar assumptions

— e.g., weighted versus unweighted, random effects versus OLS

— but we co njecture that the superior statistical fit of MRM1 will be robust to these additional analyses.

MRM2 was a late addition to this report, so EPA views the results presented in this section as still preliminary since they have been only partially stress tested. EPA intends to apply more sophisticated estimation approaches to MRM2 in follow up work. In particular, a random effects model could be used

to e stimate MRM2, with or without weighting by the inverse number of observations per study. EPA will

also consider estimating MRM2 via nonlinear least squares using equation [9 ] as the estimating equation directly. Instead of the step

- wise regression approach used here, other types of shrinkage estimators could be used such as ridge, lasso, or stage

- w ise regression (Tibshirani 1996; Efron et al

. 2004). Additional sensitivity analyses also could be conducted to examine the influence of each study and possibly o ther aspects of the model specification that might substantially influence the results. One sensitivity analysis

that EPA ha s conducted was to estimate models with squared and cubed Q terms to allow a more flexible shape for the marginal WTP curve. The mod el fits were not significantly improved according to any of

measures of fit EPA considered, and the marginal WTP curves were broadly similar but they exhibited more idiosyncratic curvature and had wider confidence intervals.

Aside from the statistical issu es that were the main focus of this section, it is also important to bear in mind that the distance between the surveyed households and the affected water bodies is not well measured by any of the explanatory variables in the model. The closest proxy is th e log of the ratio of the sampled area to the area of the counties containing the affected water bodies ( ln ar ratio ). However, this

is at best a crude proxy for distance and in any case is not intended to be used to delineate the area over

which household s are assumed to ascribe value to water quality improvements. As discussed Section 6.1

, EPA intends to use an exogenously specified threshold distance cutoff when u sing the meta

- regression models described in this report for benefit transfers.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 57



### Finally, the relatively large (positive) magnitude of the parameter estimate for the southern U.S. regional dummy variable ( south ) leads EPA to question the appropriate interpr etation of this effect. While it is theoretically possible that WTP for water quality changes is substantially higher in the south, the magnitude of the effect suggested by the model seems unlikely from an intuitive perspective. As suggested above, it is p ossible that spurious, unexplained factors influence the magnitude of this parameter in the present model. However, assessments of preliminary model runs suggest that this effect is relatively robust given the present data and selection of variables availa ble. Nonetheless, EPA recommends that the magnitude of the predicted shift in WTP associated with the south region should be

viewed with caution.

The use (and interpretation) of all reduced

- form MRMs for benefit transfer is also subject to the

constraints and concerns expressed elsewhere in the literature (e.g., Desvousges et al., 1998; Poe et al., 2001; Vandenberg et al. 2001; Smith and Pattanayak 2002)



# 9 Example WTP Predictions for MRM1 and MRM2

Table 17 presents predicted annual household WTP for various WQI changes and baselines using MRM1

and MRM2. Variable values are set to match the means from the meta

- data , with the exception of baseline WQI and WQI change, which vary across predictions. MRM1 predictions include a linear adjustment for WQI improvements less than 2.5, as described in Section 6.2. For MRM2, WTP predictions are the product estimated WTP per u nit WQI change and the WQI change. As evident in Figure 14, WTP predictions from the two models diverge dramatically for larger WQI changes, with MRM2 predicting higher WTP for large changes (e.g., WQI change = 40)

. Table 17 also shows that MRM1 generally will not satisfy the adding

- up condition while MRM2 generally will. For example, consider water quality improvements from 20 to 30 to 40 on the WQI. MRM2 gives WTP(20  30) = $ 79.74 , WTP(30  40) = $ 67.41 , and WTP(20  40) = $ 146.62 . The first

two of these ad d up to the third (nearly exactly, presumably to within rounding error), as they should. On the other hand, MRM1 gives WTP(20  30) = $74.35, WTP(30  40) = $73.26, and WTP(20  40) = $91.49. The first two of these add up to an amount significantly greater tha n the third, thereby clearly violating the adding

- up condition.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 58



### Table 17



### . Predicted WTP from MRM1 and MRM2 for Various WQI Baselines and WQI Changes (2007$) A

, B,C

Baseline WQI WQI Change

0.01 0.1 0.5 1 5 10 20 40 MRM1 (dependent variable is lnWTP) with linear adjustment for WQI improvements < 2.5

10 $0.20 $2.01 $10.07 $20.14 $61.96 $76.25 $93.83 $115.46

20 $0.20 $1.96 $9.82 $19.64 $60.42 $74.35 $91.49 $112.58

30 $0.19 $1.94 $9.68 $19.35 $59.54 $73.26 $90.15 $110.94

40 $0.19 $1.92 $9.58 $19.15 $58.92 $72.50 $89.21 $109.78

50 $0.19 $1.90 $9.50 $19.00 $58.44 $71.91 $88.49 $108.90

60 $0.19 $1.89 $9.44 $18.87 $58.05 $71.44 $87.91 $108.18

70 $0.19 $1.88 $9.38 $18.77 $57.73 $71.04 $87.42 n/a

80 $0.19 $1.87 $9.34 $18.67 $57.45 $70.70 $86.99 n/a

90 $0.19 $1.86 $9.30 $18.60 $57.21 $70.39 n/a n/a MRM2 (dependent variable is ln(WTP per 1

- point WQI change)

10 $0.10 $1.03 $5.11 $10.17 $49.18 $94.32 $173.45 $293.25

20 $0.09 $0.87 $4.32 $8.60 $41.58 $79.74 $146.62 $247.90

30 $0.07 $0.73 $3.65 $7.27 $35.15 $67.41 $123.95 $209.56

40 $0.06 $0.62 $3.09 $6.15 $29.71 $56.98 $104.78 $177.15

50 $0.05 $0.52 $2.61 $5.20 $25.12 $48.17 $88.58 $149.76

60 $0.04 $0.44 $2.21 $4.39 $21.23 $40.72 $74.88 $126.60

70 $0.04 $0.37 $1.86 $3.71 $17.95 $34.42 $63.30 n/a

80 $0.03 $0.32 $1.58 $3.14 $15.17 $29.10 $53.51 n/a

90 $0.03 $0.27 $1.33 $2.65 $12.83 $24.60 n/a n/a (A): Variable values are set to match the means from the meta

- data

. (B): “n/a” indicates that WTP was not estimated because (Baseline WQI + WQI Change) >100. (C) Consistent with WTP prediction s in Table 13

, EPA adds half of the model residual variance to the natural log of the dependent variable prior to calculating the exponential.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 59 

### Figure 13

. Predicted WTP from MRM1 and MRM2 for Various WQI Baselines and WQI Changes (2007$). “Base” refers to the Baseline WQI value. MRM1 predictions include a linear adjustment for WQI improvements less than 2.5, as described in Section
6.2. (a) MRM1 with linear adjustment for WQI impro vements < 2.5. (b) MRM2. (a)

(b )



# 10 Overfitting and Step

- wise Regression : MRM2

Th e full specification of MRM2 contains 25 degrees of freedom, and the dataset includes 140 observations from 51 studies. The studies are assumed to be independent, but EPA expect s that when multiple observations are drawn from a single study they will be correlated, so there are somewhere between 51 and 140 effective observations, or between only 2 and 6 effective observations per degree of

freedom. With so few observations per parameter, EPA has some fear that the preliminary model based on the full specification may suffer from overfitting and therefore have substantially lower out

- of

- sample predictive power than indicated by its nominal (e.g., Breiman 1992).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 60



### To address th is concern, EPA used forward and backward step

- wise regression and leave

- one

- out cross validation (CV) to select variables for inclusion in a more parsimonious model. This approach selects the model that minimizes the quadratic loss from predictive errors, i.e.,

β ̂ ∗ 

### = arg min { 1

N 

### ∑ [ y

j 

### − y ̂ j 

### ( β ̂

− j 

### ) ] 2 N j = 1 

### } [ 13 ]

where y ̂ j 

### ( β ̂

− j 

### ) is the predicted value of based on parameters fit to all data excluding observation j (Hastie et al. 2009 p 241

- 249). The intuition for this approach follows from the tradeoff between bias and variance (Breiman 1992). Beginning with a null model

— e.g., including only a constant term in a standard linear regression

— out

- of

- sample predictive performance typically will initially increase as regressors

drawn from a s et of candidate explanatory variables are added to the model. But as more regressors are added to the model, the variance of the model predictions will eventually become so large that the mean squared error of the out

- of

- sample predictions will increase ev en as the mean squared error of the within

sample predictions and the bias of the out

- of

- sample predictions continue to decrease. Minimizing out

- of

sample predictive errors is a natural objective for meta

- regression models intended to be used for benefit

t ransfers, which by definition involve out

- of

- sample predictions.

To correct for the over

- optimism of the nominal standard errors associated with the model selected by
step

- wise regression, EPA estimated standard errors by bootstrapping the data 500 times a nd re

- running the entire model selection algorithm on each bootstrap sample. As explained by Hastie et al. (2009), “In general, with a multistep modeling procedure, cross

- validation must be applied to the entire sequence of modeling steps.” The sampling di stributions estimated in this way account for the uncertainty associated with selecting the final model specification plus the sampling uncertainty conditional on the selected specification. Nominal standard error estimates reflect only the latter componen t of the un

- certainty. By bootstrapping the entire model selection algorithm, EPA is able to avoid the over

- optimistic (biased low) estimates of standard errors produced by the nominal calculations that ignore the degrees of freedom used up by the data mi ning process (Ye 1998).

Results for the MRM2 final specification selected by step

- wise regression are shown in Table 18 and

Table 19 . The most notable aspects of these results are: 1) only a handful of vari ables are dropped from the full specification, indicating a modest amount of over

- fit, 2) the nominal standard errors are much more optimistic than the confidence intervals created by bootstrapping the entire model selection and estimation routine, 3) the bootstrapped confidence intervals and the frequency distributions of the coefficient estimates are very similar to the results of the robustness analysis for MRM1, and 4) the estimated demand curves for the final specifications are very similar to those fo r the full specifications, though the 95% confidence intervals are substantially wider.

Panels (d) in Figure 14 and Figure 15 show the training (within

- sample) and test (out

- of

- sample) error curves for the MRM2 step

- wise OLS and WLS regression models, respectively. Both the forward and backward step

- wise paths are plotted on the figure, so each line appears to be doubled over. The training error curves decrease monotonically as more variables are included in the model, as they must, but the

test error curves reach a minimum before all variables are in cluded. The final selected models correspond to the minimum points on the test error curves.

Appendix D presents histograms of the coefficient estimates of the final specifications based on 500 bootstrap iterations of the full step

- wise model selection and estimation routine. These are the basis of the bootstrap confidence intervals shown in Table 18 and Table 19 . Only two variables appear in the final specifications in 97.5% or more of the bootstrap iterations: ‘South’ in the OLS model and ‘voluntary’ in the WLS model. These results are qualitatively similar to the robustness test applied to MRM1, which are

described in Section 5 . There it was shown that the coefficients on volunt and south were robust accordi ngSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 61



### to Leamer’ s extreme bounds criteria, and thesis

, volunt

, lnyear

, nonuser , south

, river

, ln ar agr, and ln quality c h (not included in MRM2) were robust according to Sala

- i

- Marin’s extreme bounds criteria. Here

EPA used a 95% confidence interval, which i s intermediate between the criteria of Leamer and Sala

- i

martin. Leamer’s criteria is equivalent to a 100% confidence interval, and Sala

- i

- Martin’s criteria is based on a 90% confidence interval, so the results reported in Table 18 and Table 19 for MRM2 are broadly consistent with the robustness analysis applied to MRM1. Accounting for this additional component of uncertainty greatly increases the confidence intervals on the individ ual coefficient estimates as well as the estimated marginal WTP curve con ditional on a fixed X 1 

### and X

2 

### , as shown in panel (c) of Figure 14 and Figure 15 . The high (essentially) flat upper 95% confidence interval bound shown there arises from the 2.5% or more cases where Q was not included in the final model specification.

That only a few variables are robust in this sense does not change the fact that the final specifications in

Table 18 and Table 19 are the ones with maximum estimated out

- of

- sample predictive performance (among those considered along the forward and backward step

- wise regre ssion paths). The main take home message of the bootstrap confidence intervals shown in Table 18 and Table 19 and the robustness analysis results for MRM1 is that the nominal standard errors and confidence intervals typically reported with an author’s preferred model specification, which is often sel ected through a series of supervised estimations and statistical tests, are likely to be highly optimistic. That is, if the study could be repeated very many times, EPA should not expect to see a variable survive the specification selection gauntlet nearly as often as suggested by its nominal significance level reported in the original study.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 62



### Table 18

. MRM2 F inal S pecification S elected by S tep

- W ise OLS

. In the far right columns, a single *
indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not
include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept 2.250 0.358

- 21.2 00 18.1 00 *

ce 0.423 0.334

- 0.887 1.61 0
thesis 0.619 0.213 0.000 1.75 0 * lnyear

- 0.339 0.093

- 0.96 0 0.000 *

volunt

- 1.550 0.210

- 2.34 0

- 0.497 * ** outliers_trim

- 0.501 0.173

- 1.11 0 0.000 *

non_param

- 0.543 0.112

- 1.01 0 0.000 * non_reviewed

- 0.593 0.193

- 1.39 0 0.000 * lump_ sum 0.802 0.174 0.000 1.56 0 * wtp_median 0.000 0.000

- 1.38 0 0.727

n ortheast 1.150 0.298 0.000 2.25 0 *

c entral 0.538 0.184 0.000 1.26 0 *

s outh 1.350 0.154 0.000 2.06 0 *

nonusers

- 0.417 0.139

- 0.799 0.000 *

lnincome 0.000 0.000

- 1.37 0 2.18 0 mult_bod

- 0.659 0.186

- 1.54 0 0.000 *

river 0.000 0.000

- 0.818 0.564 swim_use

- 0.408 0.181

- 0.916 0.228 * gamefish 0.538 0.293

- 0.769 1.43 0 boat_use

- 0.701 0.216

- 1.53 0 0.9 00 * ln_ar_ agr

- 0.365 0.109

- 0.690 0.000 * ln_ar_ ratio

- 0.070 0.027

- 0.217 0.048 * sub_proportion 1.200 0.193

- 0.742 1.901 *
Q

- 0.016 0.004

- 0.332 0.000 *

R2 0.604
RCV2 0.452 pseudo RWLS2 0.548

rWTP2 0.503September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 63



### Table 19

. MRM2 F inal S pecification S elected by S tep

- W ise WLS

. In the far right columns, a single *
indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap confidence intervals do not
include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept 2.45 0 0.67 0

- 20.2 00 17.5 00 *

ce 0.000 0.000

- 0.727 1.580
thesis 1.030 0.221 0.000 1.860 * lnyear

- 0.303 0.140

- 0.891 0.000 *

volunt

- 1.26 0 0.187

- 2.18 0

- 0.422 * ** outliers_trim 0.000 0.000

- 1.080 0.223

non_param

- 0.411 0.119

- 0.882 0.000 * non_reviewed

- 0.696 0.192

- 1.410 0.000 * lump_ sum 0.646 0.166 0.000 1.450 * wtp_median 0.000 0.000

- 1.270 1.000

n ortheast 1.350 0.271 0.000 2.320 *

c entral 0.434 0.279 0.000 1.230

s outh 1.200 0.237 0.000 2.050 *

nonusers

- 0.561 0.160

- 0.903 0.000 *

lnincome 0.000 0.000

- 1.440 2.040 mult_bod

- 0.763 0.225

- 1.560 0.000 *

river 0.000 0.000

- 0.971 0.422 swim_use 0.000 0.000

- 0.800 0.335 gamefish 0.000 0.000

- 0.930 1.500 boat_use

- 0.909 0.249

- 1.480 0.797 * ln_ar_ agr

- 0.214 0.106

- 0.654 0.237 * ln_ar_ ratio 0.000 0.000

- 0.195 0.071 sub_proportion 1.170 0.259

- 0.484 1.900 *
Q

- 0.017 0.006

- 0.052 0.000 *

R2 0.712
RCV2 0.619 pseudo RWLS2 0.415

rWTP2 0.467September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 64



### Figure 14

. MRM2 P arsimonious M odel S elected by S tep

- W ise OLS R egression

. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal WTP curve. Demand curve shifters are set as described in the main text. The black line is the point estimate based on the primary d ataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th percentiles. (d) Training (in

- sample) and test (out

- of

- sample) prediction errors for step

- wise model selection algorithm.

(a) (b)

(c) (d)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 65



### Figure 15

. MRM2 P arsimonious M odel S elected by S tep

- W ise WLS R egression

. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal WTP curve. Demand curve shifters are set as described in the main text. The black line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th percentiles. (d) Training (in

- sample) and test (out

- of

- sample) prediction errors for step

- wise model selection algorithm.

(a) (b)

(c) (d)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 66



# 11 Next Steps

EPA plans to conduct additional analyses and tests to further refine the meta

- analysis and improve

transfer estimates. Th ese analyses may include:

 Incorporation of distance decay

– The distance between the surveyed households and the affected waterbodies is not well measured by any of the explanatory variables in MRM1 or MRM2 . The closest proxy is the ln_ar_ratio variable which is the log of the ratio of sampled area to affected

r esource area. However, this variable cannot be used to directly capture the distance of each household to the affected water bodies in a benefit transfer to a policy case. As a result , an exogenous distance cutoff (see the example in Section 6.1 ) or distance decay function will still be

need ed for practical benefit transfer applications. Substantial GIS analysis of study

- specific spatial features would be required to direct ly incorporate distance decay by calculating distances between sampled populations and affected waterbodies

. Option s might include calculating distance as the centroid of the respondents for each study or calculating distance for each Census

BG in the sampling frame and calculating a population

- weighted average across blocks. The distance decay analysis may also include estimating distance and WQI values for substitute

bodies within a specified range (e.g., within 100 miles of the BG ).
 Bayesian Mod els

– It may be useful to explore approaches for using Bayesian modeling results in

benefit transfers. Meta

- analysis using Bayesian modeling techniques may offer additional insights through: (1) incorporation of qualitative information about some of the p arameters (i.e., informative priors), (2) estimation of meta

- regression for subsets of studies defined by the type of water resource (e.g., rivers, lakes, freshwater, estuaries), and (3) assessment of the impact of sampling uncertainty in the study

- specifi c estimates on the precision of the parameter estimates.

EPA did preliminary test ing of Bayesian versions of MRM1 considering these three factors and found a lot of consistency between the Bayesian meta

- regression estimates and classical meta

regression es timates. Bayesian modeling offers additional advantages in accounting for the extrapolation uncertainty that comes from the model choice (including imposition of the theory

based restrictions, as was done in MRM2). In that, Bayesian model averaging techniques can be used to adequately represent the uncertainty associated with the out

- of

- range extrapolations, which are heavily influenced by the assumptions about the model structure.



# 12 References

Achen, C.H. 2005. “Let’s Put Garbage

- Can Regressions and Garbage

- Can Probits Where They Belong.” Conflict Management and Peace Science , 22(4): 327

- 339.

Aiken, R.A. 1985. Public Benefits of Environmental Protection in Colorado. Masters thesis, Colorado

Stat e University.

Anderson, G.D. and S.F. Edwards. 1986. “ Protecting Rhod e Island’s Coastal Salt Ponds: An Economic Assessment of Downzoning.” Coastal Zone Management

, 14( 1⁄2): 67

- 91.

Azevedo, C., J.A. Herriges, and C.L. Kling. 2001. “ Valuing Preservation and I mprovements of Water Quality in Clear Lake. ” Staff Report 01

- SR 94, Center for Agricultural and Rural Development (CARD), Iowa State University.

Banzhaf, H.S., D. Burtraw, S. Chung, D.A. Evans, A. Krupnik, and J. Siikamaki. 2011. "Valuation of Ecosystem Se rvices in the Southern Appalachian Mountains." Paper Presented at the Annual Meeting of the Association of Environmental and Resource Economists (AERE).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 67



### Banzhaf, Spencer H, Dallas Burtraw, David Evans, and Alan Krupnick. 2006. "Valuation of Natural

Resourc e Improvements in the Adirondacks." Land Economics , 82(3): 445

- 464.

Bateman, I.J., B.H. Day, S. Georgiou and I. Lake. 2006. “ The Aggregation of Environmental Benefit Values: Welfare Measures, Distance Decay and Total WTP. ” Ecological Economics

, 60(2): 4 50

- 460.

Bateman, I.J. and A.P. Jones. 2003. “Contrasting conventional with multi

- level modeling approaches to

meta

- analysis: Expectation consistency in U.K. woodland recreation values.” Land Economics , 79(2):

235

- 258.

Bergstrom, J.C. and L.O. Taylor. 2006 . “Using Meta

- Analysis for Benefits Transfer: Theory and
Practice.” Ecological Economics , 60(2): 351

- 360.

Bockstael, N.E., McConnell, K.E., and Strand, I.E. 1988. “ Benefits from improvements in Chesapeake Bay Water Quality. ” Department of Agricultural & R esource Economics, University of Maryland, 1988.

Bockstael, N.E., K.E. McConnell and I.E. Strand. 1989. “Measuring the Benefits of Improvements in Water Quality: The Chesapeake Bay.” Marine Resource Economics 6: 1

- 18.

Borisova, Tatiana, Alan Collins, Gerar d D’Souza, Matthew Benson, Mary Leigh Wolfe, and Brian Benham, 2008. “A Benefit

- Cost Analysis of Total Maximum Daily Load Implementation.” Journal of the

American Water Resources Association , 44(4):1009

- 1023.

Boyle, Kevin J., Christopher C. Parmeter, Bent Boehlert, and Robert Paterson. 2013. “ Due Diligence in

Meta

- Analyses to Support Benefit Transfers. ” Environmental and Resource Economics, 55(3): 357

- 386.

Brander, L., and R. Brouwer. 2011. “ A multi

- level meta

- analysis regression model of contingent values for water ecosystem services .” Paper presented at the European Association of Environmental and Resource Economists 18th Annual Conference, 29 June

- 2 July 2011, Rome.

Breiman L. 1992. “The little bootstrap and other methods for dimensionality selection i n regression: X

fixed prediction error.” Journal of the American Statistical Association

, 87(419):7 38

- 754.

Cameron, T.A. and D.D. Huppert. 1989. “OLS versus ML Estimation of Non

- market Resource Values with Payment Card Interval Data.” Journal of Environmental Economics and Management 17, 230

- 246.

Cameron AC, Miller DL. 2013. A practitioner’s guide to cluster

- robust inference. http://cameron.econ.ucdavis.edu/research/Cameron_Miller_Cluster_Robust_October152013.pdf

Carson, R.T. 2012. “Con tingent Valuation: A Practical Alternative when Prices Aren’t Available.” Journal of Economic Perspectives , 4: 27

- 42.

Carson, R.T., and R.C. Michell. 1993. “The Value of Clean Water: The Public’s Willingness to Pay for Boatable, Fishable, and Swimmable Wat er Quality.” Water Resources Research , 29(70), 2445

- 2454.

Carson, R.T., W.M. Hanemann, R.J. Kopp, J.A. Krosnick, R.C. Mitchell, S. Presser, P.A. Ruud, and V.K. Smith. 1994. Prospective Interim Lost Use Value due to DDT and PCB Contamination in the Southern California Bight. Volume 2. Report to the National Oceanic and Atmospheric Administration, Produced by Natural Resources Damage Assessment Inc., LA Jolla, CA.

Clonts, H.A., and J.W. Malone. 1990. “ Preservation Attitudes and Consumer Surplus in Free Flowin g Rivers. ” In J. Vining (ed.), Social Science and Natural Resource Recreation Management , Westview Press: Boulder, CO, pp. 301

- 317.

Collins, Alan R. and Rosenberger, R.S. 2007. “Protest Adjustments in the Valuation of Watershed Restoration Using Payment Ca rd Data.” Agricultural and Resource Economics Review , 36(2): 321

- 335.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 68



### Collins, A.R., R.S. Rosenberger and J.J. Fletcher. 2009. “Valuing the restoration of acidic streams in the Appalachian Region: A stated choice method.” In H.W. Thurstone, M.T. Heberling and A. Schrecongost (eds.), Environmental Economics for Watershed Restoration . Boca Raton, FL: CRC/Taylor Francis. pp.29

- 52.

Corrigan, J.R., C.L. Kling, and J. Zhao. 2009. "Willingness to Pay and the Cost of Commitment: An Empirical Specification and Test." Environmental and Resource Economics , 40: 285

- 298.

Croke, Kevin, R.G. Fabian, and G. Brenniman. 1986

- 87. “Estimating t he Value of Improved Water Quality in an Urban River System.” Journal of Environmental Systems 16(1): 13

- 24.

De Zoysa, A.D.N. 1995. A Benefit Evaluation of Programs to Enhance Groundwater Quality, Surface Water Quality and Wetland Habitat in Northwest Ohio . Dissertation, Ohio State University.

Desvouges, W.H., K. Mathews, and K. Train. 2012. “Adequate responsiveness to scope in contingent
valuation.” Ecological Economics , 84: 121

- 128.

Desvousges, W.H., F.R. Johnson, and H.S. Banzhaf. 1998

. , Environmental Policy Analysis with Limited Information: Principles and Applications of the Transfer Method . Edward Elgar, Northampton, MA.

Desvousges, W.H., V.K. Smith and A. Fisher. 1987. “Option Price Estimates for Water Quality Improvements: A Contingent Valuation St udy for the Monongahela River.” Journal of Environmental Economics and Management

, 14: 248

- 267.

Diamond, P.A. 1996. “Testing the Internal Consistency of Contingent Valuation Surveys.” J ournal of Environmental Economics and Management

, 30: 337

– 347.

Downstre am Strategies LLC. 2008. An Economic Benefit Analysis for Abandoned Mine Drainage Remediation in the West Branch Susquehanna River Watershed, Pennsylvania. Prepared for Trout

Unlimited.

Dupont W.D, and W.D. Plummer. 1998. “Power and sample size calculation s for studies involving linear regression.” Controlled Clinical Trials

, 19: 589

- 601.

Efron B. 1979. “Bootstrap methods: another look at the jackknife.” The Annals of Statistic s, 7(1): 1

- 26.

Efron B. 2003. ‘Second thoughts on the bootstrap.” Statistical Sci ence , 18(2): 135

- 140.

Efron B., T. Hastie, I. Johnstone, and R. Tibshirani. 2004. “Least angle regression.” The Annals of

Statistics , 32(2): 407

- 499.

Farber, S., and Griner, B. 2000. “Using Conjoint Analysis to Value Ecosystem. Change,” Environmental Science and Technology , 34(8): 1407

- 1412.

Ge, J

., C. Kling, and J. Herriges. 2013. “How much is Clean Water Worth? Valuing Water Quality Improvement Using a Meta Analysis.” Iowa State University, WP #13016, March 2013.

Ghermandi, A. and P.A.L.D. Nunes. 201 3. “ A Global Map of Coastal Recreation Values: Results from a Spatially Explicit Meta

- analysis. ” Ecological Economics , 86(1): 1

- 15.

Ghermandi, A., J.C.J.M. van den Bergh, L.M. Brander, H.L.F. de Groot, and P.A.L.D. Nunes. 2010.

“ Values of Natural and Human

- made Wetlands: A Meta

- analysis. ” Water Resources Research , 46:

W12516.

Greene WH. 1997. Econometric Analysis . 3 rd 

### edition. Upper Saddle River, NJ: Prentice

- Hall, Inc.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 69



### Hanley, N., F. Schlapfer, and J. Spurgeon. 2003. Aggregating the benefits of environment al improvements: distance

- decay function for use and non

- use values. Journal of Environmental Management

, 68: 297

- 304.

Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2 nd 

### editi on. Springer

- Verlag.

Hayes, K.M., T.J. Tyrell and G. Anderson. 1992. “Estimating the Benefits of Water Quality Improvements in the Upper Narragansett Bay.” Marine Resource Economics 7: 75

- 85.

Herriges, J.A. and J.F. Shogren. 1996. “Starting point bias in d ichotomous choice valuation with followup questioning.” Journal of Environmental Economics and Management

, 30(1):112

- 131.

Hite, D. 2002. Willingness to Pay for Water Quality Improvements: The Case of Precision Application Technology. Department of Agricult ural Economics and Rural Sociology, Auburn University, Auburn, AL; 8 August 2002.

Hlavac, M. 2014a. “ExtremeBounds: Extreme bounds analysis in R (pp. 1

- 21) ” . Cambridge, MA: Harvard University.

Hlavac, M. 2014b. Package ‘ExtremeBounds’ (Version 0.1.3 (2014

- 03

- 21, 07:00:10)) [Package].

Hoehn, J.P., and A. Randall. 1989. Too Many Proposals Pass the Benefit Cost Test. The American Economic Review, 79(3): 544

- 551.

Horowitz, J.L. 2003. “The bootstrap in econometrics.” Statistical Science

, 18(2): 211

- 218.

H uang, J.C., T.C. Haab and J.C. Whitehead. 1997. “Willingness to Pay for Quality mprovements: Should
Revealed and Stated Preference Data Be Combined?” Journal of Environmental Economics and Management

, 34(3): 240

- 255.

Irvin, S., T. Haab, and F.J. Hitzhusen. 2007. "Estimating willingness to pay for additional protection of Ohio surface waters: contingent valuation of water quality". In F.J. Hitzhusen (ed.), Economic Valuation of River Systems , Cheltenham: Edward Elgar, pp. 35

- 51.

Johnston, R.J., E .Y. Besedin and R. Stapler. 2015. “ Enhanced Geospatial Meta

- Analysis and Environmental Benefit Transfer: An Application to Water Quality Improvements .” W

- 3133 : Benefits and Costs of Natural Resources Policies Affecting Ecosystem Services on Public and Private Lands

. Pensacola, FL, February 25

- 27.

Johnston, R.J., E.Y. Besedin, R. Iovanna, C. Miller, R. Wardwell, and M. Ranson. 2005. “ Systematic Variation in Willingness to Pay for Aquatic Resource Improvements and Implications for Benefit

Transfer: A Meta

- Analysis. ” C anadian Journal of Agricultural Economic,s 53(2005): 221

- 248.

Johnston, R.J. and J.M. Duke. 2009. “Willingness to Pay for Land Preservation Across States and Jurisdictional Scale: Implications for Benefit Transfer.” Land Economics, 85(2): 217

– 237.

Johnston , R.J. and M. Ramachandran. 2014. “Modeling Spatial Patchiness and Hot Spots in Stated Preference Willingness to Pay.” Environmental and Resource Economics , 59(3): 363

- 387.

Johnston, R.J. and R.S. Rosenberger. 2010. “Methods, trends and controversies in co ntemporary benefit
transfer.” Journal of Economic Surveys , 24(3): 479

– 510.

Johnston, S.K. Swallow, and T.F. Weaver. 1999. "Estimating Willingness to Pay and Resource Tradeoffs with Different Payment Mechanisms: An Evaluation of a Funding Guarantee for Watershed Management." Journal of Environmental Economics and Management , 38: 97

- 120.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 70



### Jørgensen, S.L., S.B. Olsen, J. L adenburg, L. Martinsen, S.R. Svenningsen and B. Hasler. 2013.

“ Spatially Induced Disparities in Users’ and Non

- users’ WTP for Water Quality Improvements

– Testing the Effect of Multiple Substitutes and Distance Decay. ” Ecological Economics , 92(1): 58

- 66.

K aoru, Y. 1993. “Differentiating Use and Nonuse Values for Coastal Pond Water Quality Improvements.” Environmental and Resource Economics

, 3: 487

- 494.

Kennedy, P. E. 2008. A Guide to Econometrics (6 ed.). Malden, MA: Blackwell Publishing.

Lant, C.L., and R . S. Roberts. 1990. “Greenbelts in the Cornbelt: Riparian Wetlands, Intrinsic Values, and
Market Failure.” Environment and Planning 22: 1375

- 1388.

Lant, C.L., and G.A. Tobin. 1989. “The Economic Value of Riparian Corridors in Cornbelt Floodplains:

A Resear ch Framework.” Professional Geographer (41): 337

- 349.

Levine, R., and D. Renelt. 1992. “A sensitivity analysis of cross

- country growth regressions.” The

American Economic Review, 82(4): 942

- 963.

Lichtkoppler, F.R. and T.W. Blaine. 1999. “Environmental A wareness and Attitudes of Ashtabula County Voters Concerning the Ashtabula River Area of Concern: 1996

- 1997

. ” Journal of Great LakesResources

25: 500

- 514.

Lindsey, G. 1994. “Market Models, Protest Bids, and Outliers in Contingent Valuation.” Journal of

Wat erResources Planning and Management 12: 121

- 129.

Lipton, D. 2004. “The Value of Improved Water Quality to Chesapeake Bay Boaters.” Marine Resource

Economics 19: 265

- 270

Londoño Cadavid, C. and A. W. Ando . 2013. "Valuing preferences over stormwater managem ent outcomes including improved hydrologic function." Water Resources Research 49: 4114

- 4125.

Londoño, L.M. and R.J. Johnston. 2012. “Enhancing the Reliability of Benefit Transfer over Heterogeneous Sites: A Meta

- Analysis of International Coral Reef Values .” Ecological Economics , 78(1):

80

- 89.

Loomis, J.B. 1996. “How Large is the Extent of the Market for Public Goods: Evidence from a Nation Wide Contingent Valuation Survey.” Applied Economics

, 28(7): 779

- 782.

Loomis, J.B. 2000. “Vertically Summing Public Good Demand Curves: An Empirical Comparison of
Economic versus Political Jurisdictions.” Land Economics, 76: 312

- 321.

Loomis, J.B. and R.S. Rosenberger. 2006. “Reducing barriers in future benefit transfers: needed improvements in primary study design and r eporting.” Ecological Economics, 60: 343

- 350.

Lyke, A.J. 1993. Discrete Choice Models to Value Changes in Environmental Quality: A Great Lakes Case Study. Dissertation submitted to the Graduate School of the University of Wisconsin, Madison.

Matthews, L.G. , F.R. Homans, and K.W. Easter. 1999. “ Reducing Phosphorous Pollution in the

Minnesota River: How Much is it Worth? ” Staff Paper. Department of Applied Economics, University of

Minnesota.

McClelland, N.I. 1974. Water Quality Index Application in the Kansas River Basin. EPA

- 907/9

- 74

- 001, US EPA Region VII, Kansas City, MO.

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation Method. Resources for the Future, Washington, DC.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 71



### Mrozek, J.R., and L.O. Taylor. 2002. “ What determines the value of life? A meta

- analysis.” Journal of Policy Analysis and Management

, 21(2): 253

- 270.

Navrud, S. and R. Ready. 2007. “Review of Methods for Value Transfer.” In: Navrud, S. and R. Ready (eds.), Environmental Value Transfer: Issues and Methods . Dordrecht, The Netherlands: Springer. pp.1

- 10.

Nelson, J.P. and P.E. Kennedy. 2009. “ The Use (and Abuse) of Meta

- Analysis in Environmental and

Resource Economics: An Assessment. ” Environmental and Resource Economics , 42(3): 345

- 377.

Olsen, D. , Richards, Jack, and Scott, R. D. 1991. “ Existence and Sport Values for Doubling the Size of

Columbia River Basin Salmon and Steelhead Runs. ” Rivers . 2(1): 44

- 56.

Opaluch, J.J., T. Grigalunas, M.J. Mazzotta, J. Diamantides, and R. Johnston. 1998. Resource and Recreational Economic Values for the Peconic Estuary. Report prepared for Peconic Estuary Program, Suffolk County Department of Health Services, Riverhead, NY, by Economic Analysis, Inc., Peace Dale,

Rhode Island.

Poe, G. L., K. J. Boyle, and J. C. Be rgstrom, 2001. “ A preliminary meta analysis of contingent values for ground water quality revisited ” , in The Economic Value of Water Quality , edited by J. C. B ergstrom, K. J. Boyle, and G. L

. Poe, pp. 137

– 162, Edward Elgar, Northampton, Mass.

Ren, S., H. Lai, W. Tong, M. Aminzadeh, X. Hou, and S. Lai. 2010. “Nonparametric bootstrapping for
hierarchical data.” Journal of Applied Statistics

, 37(9): 1487

- 1498.

Roberts, L.A., and J.A. Leitch. 1997. “ Economic Valuation of Some Wetland Outputs of Mud Lake. ” Agri cultural Economics Report No. 381, Department of Agricultural Economics, North Dakota Agricultural Experiment Station, North Dakota State University.

Rosenberger, R. S. and J. B. Loomis. 2000. “Using meta

- analysis for benefit transfer: In

- sample
convergent validity tests of an outdoor recreation database.” Water Resources Research , 36: 1097

- 1107.

Rowe, R.D., W.D. Schulze, B. Hurd and D. Orr. 1985. Economic Assessment of Damage Related to the Eagle Mine Facility. Energy and Resource Consultants, Inc., Boulde r, CO.

Sala

- i

- Martin, X. X. 1997. “I just ran two million regressions.” The American Economic Review , 87(2),

178

- 183.

Sanders, L. B., R.G. Walsh, and J.B. Loomis. 1990. “Toward Empirical Estimation of the Total Value of Protecting Rivers.” Water Resources Research . 26(7): 1345

- 1357.

SAS/STAT 9.2. 2010. The MIXED Procedure. SAS/STAT(R) 9.2 User's Guide, Second Edition Retrieved April 29, 2014, from http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#mixed_toc.htm

Schaafsma, M., R. Brouwer and J. Rose. 2012. “Directional Heterogeneity in WTP models for

Environmental Valuation .” Ecological Economics , 79(1): 21

- 31.

Schabenberger, O. 2004. Mixed model influence diagnostics. Proceedings of the Twenty

- Ninth Annual SAS® Users Group International ConferenceProceedings of the Twenty

- Ninth Annual SAS® Users Group International Conference (pp. 1

- 17). Cary, NC: SAS Institute Inc. Retrieved from http://www2.sas.com/proceedings/sugi29/189

- 29.pdf

. Schulze, W.D., R.D. Rowe, W.S. Breffle, R.R. Boyce, and G.H. McClelland. 1995. Contingent Valuation of Natural Resource Damages Due to Injuries to the Upper Clark Fork River Basin. State of Montana, Natural Resource Damage Litigation Program. Prepared by: RCG/Hagler Bailly, Boulder, CO.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 72



### Shrestha, R. K., J.R.R. Alavalapati. 200 4. “Valuing Environmental Benefits of Silvopasture Practice: A Case Study of the Lake Okeechobee Watershed in Florida.” Ecological Economics 49: 349

- 359.

Shrestha, R.K. and J.B. Loomis. 2001. “Testing a meta

- analysis model for benefit transfer in internati onal
outdoor recreation.” Ecological Economics , 39(1): 67

- 83.

Smith, V .K. and S.K. Pattanayak. 2002. “ Is Meta

- Analysis a Noah’ s Ark for Non

- Market Valuation?”

Environmental and Resource Economics

, 22(1

- 2): 271

- 296.

Smith, V.K., G. Van Houtven and S.K. Patt anayak. 200 2. “ Benefit Transfer via Preference Calibration: ‘Prudential Algebra’ for Policy.” Land Economics

, 78(1): 132

- 152.

Stanley, T.D., H. Doucouliagos, M. Giles, J. H. Heckemeyer, R. J. Johnston, P. Laroche, J. P. Nelson, M. Paldam, J. Poot, and G. P ugh. 2013. “ Meta

- Analysis of Economic Research Reporting Guidelines. ” Journal of Economic Surveys, 27(2): 390

– 394.

Stumborg, B.E., K.A. Baerenklau and R.C. Bishop. 2001. “Nonpoint Source Pollution and Present Values: A Contingent Valuation of Lake Mendota. ” Review of Agricultural Economics . 23(1): 120

- 132.

Sutherland, R. J., and R.G. Walsh. 1985. “Effect of Distance on the Preservation Value of Water Quality.” Land Economics 61(3): 282

- 29.

Takatsuka, Yuki. 2004. Comparison of the Contingent Valuation Method and the Stated Choice Model for Measuring Benefits of Ecosystem Management: A Case Study of the Clinch River Valley, Tennessee. PhD dissertation, University of Tennessee.

Tibshirani, R. 1996. “Regression shrinkage and selection via the lasso.” Journal of the Royal Statistical Society. Series B (Methodological)

, 58(1): 267

- 288.

United States Environmental Protection Agency (U.S. EPA). 2004. Meta

- Analysis of Non

- Use Benefits for the Section 316(b) Rulemaking for Cooling Water Intake Structures at Existing F acilities. Peer review Report. Prepared by U.S. EPA, Office of Water, Economic Analysis Division, Washington, D.C., February 13, 2004.

United States Environmental Protection Agency (U.S. EPA). 2009. Environmental Impact and Benefits

Assessment for Final Ef

fluent Guidelines and Standards for the Construction and Development Category. November, 2009.

United States Environmental Pr otection Agency (U.S. EPA). 201 0

. Water Quality Standards for the State of Florida’s Lakes and Flowing Waters . Final Rule. Federal Register, 7 5 ( 233 ): 75762

- 75807

. United States Environmental Protection Agency (U.S. EPA). 2012. Water Quality Standards for the State of Florida’s Estuaries, Coastal Waters, and South Florida Inland Flowing Waters

. Proposed Rule. Federal Register, 77(243): 74924

- 74985.



## United States Environmental Protection Agency (U.S. EPA). 2013. Benefit and Cost Analysis for the Proposed Effluent Limitations Guidelines and Standards for the Steam Electric Power Generating Point Source Category. May 2013.

Van Hou tven, G., J. Powers and S.K. Pattanaya k. 2007. “ Valuing Water Quality Improvements in the United States Using Meta

- analysis: Is the Glass Half

- full or Half

- empt y for National Policy Analysis?” Resource and Energy Economics

, 29(3): 206

– 228.

VandenBerg, T. P ., Poe, G. L. and Powell, J. R. 2001

. “ Assessing the accuracy of benefits transfers:

evidence from a multi

- site contingent valuation study of groundwater quality. ” In J. C. Bergstrom, K. J.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 73



### Boyle & G. L. Poe (Eds.), The E conomic V alue of W ate r Q uality

, pp. 100

- 120

, Cheltenham, UK: Edward Elgar.

Vaughan, W.J. 1986. The RFF Water Quality Ladder. Appendix B in R.C. Mitchell and R.T. Carson. The Use of Contingent Valuation Data for Benefit/Cost Analysis in Water Pollution Control, Final Report. Washington: Re sources for the Future.

Viscusi, W.K., J. Huber, and J. Bell. 2008. “The Economic Value of Water Quality.” Environmental and

Resource Economics , 41(2): 169

- 187.

Wattage, P. M. 1993. Measuring the benefits of water resource protection from agricultural

cont amination: Results from a contingent valuation study. PhD dissertation, Forestry, Iowa State University.

Welle, P.G. 1986. Potential Economic Impacts of Acid Deposition: A Contingent Valuation Study of Minnesota. Dissertation, University of Wisconsin

- Madis on.
Welle, P.G., and J.B. Hodgson. 2011. Property Owner’s Willingness to Pay for Water Quality Improvements: Contingent Valuation Estimates in Two Central Minnesota Watersheds. Journal of Applied Business and Economics 12(1): 81

- 94.

Wey, K.A. 1990. Social Welfare Analysis of Congestion and Water Quality of Great Salt Pond, Block Island, Rhode Island. Dissertation, University of Rhode Island.

Whitehead, J.C., and P.A. Groothuis. 1992. “Economic Benefits of Improved Water Quality: a case study

of North Caroli na’s Tar

- Pamlico River.” Rivers, 3: 170

- 178.,

Whitehead, J.C. 2006. “ Improving Willingness to Pay Estimates for Quality Improvements through Joint Estimation with Quality Perceptions. ” Southern Economic Journal , 73(1): 100

- 111.

Whitehead, J.C., G.C. Blomqu ist, T.J. Hoban and W.B. Clifford. 1995. “Assessing the Validity and Reliability of Contingent Values: A Comparison of On

- Site Users, Off

- Site Users, and Nonusers.” Journal of Environmental Economics and Management, 29: 238

- 251.

Whitehead, J.C., T. Hoban a nd W. Clifford. 2002. “ Landowners’ Willingness to Pay for Water Quality Improvements: Jointly Estimating Contingent Valuation and Behavior with Limited Information. ” White paper developed in part by U.S. EPA, NCDENR and the College of Agriculture and Life Sciences at

NSCU.

Whittington, D., G. Cassidy, D. Amaral, E. McClelland, H. Wang and C. Poulos. 1994. “ The Economic Value of Improving the Environmental Quality of Galveston Bay. ” Department of Environmental Sciences and Engineering, University of North Ca rolina at Chapel Hill. GBNEP

- 38, 6/94.

Willett, J.B., and J.D. Singer. 1988. “Another cautionary note about R 2 

### : its use in weighted least

- squares regression analysis.” The American Statistician

, 42(3): 236

- 238.

Ye, J. 1998. “On measuring and correcting the effects of data mining and model selection.” Journal of the
American Statistical Association

, 93(441): 120

- 131.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 74



# Appendix A : Summary Statistics from the Metadata for Key Geospatial
Variables

Table A

- 1

. Summary statistics from the metadata for selected variables describing sampled market area, affected resources, and water quality

Variable Units Mean Median Min Max St. Dev. Percentiles

5 th 95 th

ar_agr Proportion

(0 to 1) 0.33 0.26 0.01 0.92 0.26 0.04 0.88

ar_ total_area km 2 39,685 11,797 1,356 451,134 59,718 2,015 152,213 ar_ratio1 Ratio

(km 2 /km 2 ) 7.91 0.74 <0.01 773.74 65.49 <0.01 15.53

sub_proportion Proportion

(0 to 1) 0.19 0.03 <0.01 1.00 0.29 <0.01 0.93

sa_area km 2 127,077 5,329 4.6 9,833,518 832,426 34.4 269,601
WQIbase WQI

(0 to 100) 42.3 45.0 5.0 80.0 18.1 5.0 72.5

quality_ch WQI

(0 to 100) 21.3 20.0 2.5 57.5 10.9 5.0 45.0September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 75



# Appendix B: Summary of Study

- Specific WQL Translations

Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Aiken (1985) 1 no CO S tate

- wide rivers and lakes Current conditions with existing programs

were described (80% of CO rivers and lakes support game fish such as trout, and 20% are too polluted for trout survival). Baseline set to 4.5 on WQL to reflect rough fishi ng. Scenario would improve water quality to allow swimming, boating, and fishing. Post

- improvement WQL expressed as a range from 5 .0 to 7 .0 on WQL (change of 0.5 to 2.5) to reflect uncertainty in average state

wide improvement.
Anderson and Edwards (1986) 1 yes RI S alt pond/marshes

near the RI coast Study used the WQL

Banzhaf et al. (2006) 2 no NY H undreds of lakes in Adirondack Park upstate NY Lakes d escribed as safe for swimming at

baseline, but some fish populations have declined or been eliminated in lakes of concern. Assign ed baseline using a range of 4.5 to 5 .0 on WQL to reflect rough fishing to game fishing

. Improvements affect fish abundance and t he overall ecosystem. I mprove d conditions expressed as a range from
5 .0 to 6 .0 on WQL (c hange of 0.5 to 1.0) to reflect uncertainty regarding quality of game fishing and additional ecosystem improvements.
Banzhaf et al. (2011) 1 no VA, WV, TN, NC, GA Many streams

impacted by acidification in the Southern Appalachian Region The affected streams have been impacted

by air pollution and are not in good condition, with various fish species affected. Assign ed baseline using a range of 4.5 to 5 .0 on WQL to reflect rough fishing to game fishing

. Improved conditions expressed as a range from 5 .0 to 6 .0 on WQL (change of 0.5 to 1.0) t o reflect uncertainty regarding quality of game fishing and ecosystem improvements.

Bockstael et al (1988) 1 no DC, MD, VA Chesapeake Bay Study assessed impacts to recreational uses, swimming in particular, due to reductions in nutrient loadings. Baseline coded as a range from 4.5 to 5 .0 on WQL for rough fishing to game fishing to reflect uncertainty regarding quality of fishing supported at baseline. Assign ed pos t

- improvement WQL of
7 .0 to reflect quality suitable for swimming (change of 2.5 and 2.0).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 76



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Bockstael et al. (1989) 2 no MD Chesapeake Bay estuarine system The valuation question was asked of those

respondents who believed that water quality in the Bay was unacceptable for swimming regarding an improvement to allow swimming. Baseline set to 5 on WQL to reflect game fishing, but not swimming. Assign ed post

- improvement WQL of 7 to reflect quality suitable for swimming (change of 2. 0).

Borisova et al. (2008) 3 no WV, VA Opequan and Abrams Creeks The creeks are currently polluted with dirt

and sediment along with sewage and bacteria. Because of these pollutants, no swimming or wading is recommended in the VA portion of these creeks In the survey questions, improvements were described in terms of watershed clean

- up to improve the safety of swimming and wading.

. Assigned baseline of 5.0 to reflect baseline game fish population in portions of the rivers but no swimming. Assign ed post

- improvement WQL of 7 to reflect water quality improvements to support swimming and wading as described in the survey (change of 2.0).

Cameron and Huppert (1989) C 1 no CA San Francisco Bay and ocean

area Survey asked recreational users to value

improvement focusing on doubling catch rates for salmon and striped bass using methods including habitat restoration. Assigned range of 5.0 to 7.0 on WQL for baseline to reflect uncertainty regarding baseline service provision. Assigned post

- improvement W QL range from 6.75 to 8.5 to reflect improvements beyond the fishable range (change of 1.75 and 1.5).

Carson et al. (1994) 2 no CA Area on the Southern California Bight There are r eproduction problems among

four fish species in the affec ted area caused by DDT and PCBs. Assigned range of 4.5 to 5.0 on WQL to reflect rough fishing to game fishing given presence of fishing at baseline (just depressed), reflecting uncertainty regarding baseline services. Assigned post

- improvement WQL range from 6 to 7 (chang e of 1.5 to 2.0) to reflect uncertainty regarding the degree of improvement within the fishable range.

Clonts and Malone (1990) 3 no AL Fifteen free

flowing rivers Estimated WTP to prevent degradation to

fifteen free

- flowing rivers in Alabama. Assigned baseline of 5 to 7 on WQL to reflect the uncertainty regarding average recreational uses supported by these rivers at baseline. Assigned WQL of 4.5 for rough fishing to reflect the degraded state in the absence of protection (absolute change of 0.5 to 2.5).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 77



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Collins and Rosenberger (2007) 1 no WV Cheat River Watershed At baseline, water quality problems include

acid mine drainage, lack of aquatic life, trash, sewage, and sedimentation. The watershed has fish in only the upper portions of the Cheat River. Northern section of the river is popular for rafting. Assigned baseline of 4.5 to reflect rough fishing conditions (but not game fishing) within the study area. Full clean

- up would include treatment of acid mine drainage from all stream tribut aries of the Cheat River. Assigned post

- improvement WQL of 6 to 6.5 (change of 1.5 to 2.0) to reflect improvements beyond the minimum water quality to support game fish.

Collins et al. (2009) 7 no WV Deckers Creek Describes services using three attributes ( aquatic, scenic, and swimming). A WTP observation associated exclusively with the scenic attribute was dropped because it could not be tied to the WQL. Assigned baselines of 4.5 to 7.0 for rough fishing to swimming depending on whether the
aqua tic or swimming attributes (or both) were evaluated. Assignments based on support of fis hing, game fishing, and swimming

, with intermediate values assign ed in some cases to reflect improvements beyond the minimum to support relevant use. Post

- improvement WQL range from 5.0 to 8.0 (changes of 0.5 to 1.5) depending on whether the aquatic or swimming attributes (or both) were evaluated. Corrigan et al. (2009) 1 no IA Clear Lake Estimated WTP for water quality improvements in a lake that support s game fishing and swimming at baseline, but is occasio nally affected by algae blooms

. Assigned baseline WQL of 6.5 to 7.0 to reflect uncertainty regarding the frequency of impacts

. Assigned post

- improvement WQL of 7.5 to 8.5 (changes 1.0 to 1.5) to reflect imp rovements beyond the minimum for swimming use.

Croke et al. (1987) 9 yes IL Rivers in the Chicago urban river system Study used the WQL

De Zoysa (1995) 1 no OH Maumee River Basin Study focuses on degradation of fish

habitat. At baseline

, clear water fish are being replaced by fish more tolerant of low oxygen levels and greater turbidity. Assigned baseline WQL of 4.5 to reflect rough fishing. Assigned post

- improvement WQL of 5.0 (change of 0.5) to reflect support of game fishing.

Desvou sges et al. (1987) 12 yes PA Monongahela River Study used the WQLSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 78



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Downstream Strategies (2008) 2 no PA West Branch Susquehanna
River Watershed Water does not support trout populations or

warm water fishing (like bass) because of acid mine drainage. Assigned baseline WQL of 4.5 to reflect rough fishing because fish have not been totally extirpated throughout the watershed. If acid mine drainage were cleaned up, river would once a gain support trout populations and, in some areas, warm water fish. It would also provide cleaner water for community and industrial water supplies and improved recreational experiences for swimming, canoeing, rafting, et c. Assigned post

- improvement WQL of 7.0 to reflect various recreational uses including game fishing and swimming.
Farber and Griner (2000) 6 no PA Loyalhanna Creek and C onemaugh River Study e xpresses level of pollution as

“unpolluted”, ”moderately polluted”, or “severely polluted” depending on support for fish, wildlife, and other organisms. Assigned baseline WQL values of 2.5 to

“ severely polluted ”4.5 to “ moderately polluted” as 4.5, and 5.0 to “ unpolluted ” on descriptions. WQL assignments f ocus on support for fishing because human health is not typically affected. Post

- improvement WQL values range from 4.5 to 5 (changes ranges from 0.5 to 2.5) depending on whether WTP observation correspond s to an increase to “moderately polluted” or “unpolluted”.

Hayes et al. (1992) 2 no RI Upper Narragansett Bay The affected resource is unsuitable for

swimming at baseline because of agricultural, domestic, and industrial pollutants. Assign baseline WQL values ranging from 4.5 to 5 for rough fishing to game fishing due to uncertainty regarding the type of fishing supported. Estimated WTP is for improvements that would benefit swimming and shellfishing. Assigned p ost

compliance WQL rang e from 7.0 to
9. 0 across scenarios (change of 2.5 to 4) depending on scenario to reflect improvements beyond minimum for fishing in some cases to reflect high sensitivity of shellfishing. Herriges et al. (1996) 2 no IA Storm Lake Study estimates WTP to prevent degradation in recreational uses from a baseline that supports some fishing and boating. Assigned baseline WQL values of 4.5 for both observations to reflect rough fishing. Assigned degraded WQL values of ranging from 1.25 to 2.5 to reflec t unusable to boatable waters in the absence of intervention (absolute change of 3.25 and 2).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 79



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Hite (2002) 2 no MS Lower Mississippi River Basin Study estimates WTP for improvements in

water quality in the Lower Mississippi River which includes many 303(d) listed waters. Assigned baseline WQL values ranging from 2.5 to 4.0 to reflect boating and quality close to supporting rough fishing. Assigned WQL values to reflect modest improvements (10

- 20% imp rovements) ranging from 4.25 to 4.5 to reflect support, or nearing support, for rough fishing (changes of 0.25 to 0.5). Huang et al. (1997) 2 no NC Albemarle and Pamlico Sounds Baseline water quality was described as

degraded from 1981 levels in terms of reduced fish catches (60%) and reduced number of open shellfish beds (25%). Assign ed baseline WQL values of 5.0 to reflect game fishing. Assigned post

- improvement WQL values of 8 to 9 (changes of 3 to 4) to reflect improvements beyond swimmable, reflecting high sensitivity of shellfish beds to water quality.

Irvin et al. (2007) 4 no OH State

- wide surface waters Water quality changes described in terms of

percent of set

- aside value under the Antidegradation Rule to achieve varies quality categories ( limited quality, general high quality, superior high quality, state resource waters, and outstanding national resource waters). “O utstanding national
resource waters ” ha ve special characteristics such as high water quality, or as having exceptional recreat ional, cultural, aesthetic or scientific value. Assigned baseline WQl values of 4.5 for “limited quality”, reflecting rough fishing. Post

- improvement WQL levels correspond to four water quality categories above “limited quality” ( general high quality, supe rior high quality, state resource waters, and outstanding national resource
waters ). Set “outstanding national resource waters” to 9.5, reflecting the highest water quality use classification on the WQL, because these are surface waters of the highest nati onal ecological or recreational significance with water quality that shall not be degraded by any substance. Approximate WQL levels for intermediate categories based on the percent set aside assuming a step

- wise path between WQl values of 4.5 (“limited qua lity”) and 9.5 (“outstanding national resource waters”). Results in post

improvements values of 4.86, 5.86, 7.42, and 9.5 (changes of 0.36, 1.36, 2.92, and 5.00), respectively.
Johnston et al. (1999) 1 yes RI Wood

- Pawcatuck
W atershed Study used the WQLSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 80



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Kaoru (1993) 1 no MA 3 saltwater ponds on Martha’s Vineyard Study is focused on impacts to shellfishing

in saltwater ponds in Martha’s Vineyard.
A ssigned baseline WQL values ranging from 4.5 to 5 for rough fishing to game fishing to reflect support of som e recreational uses

. Assigned post

- improvement WQL values of 8 to 9 (changes of 3.5 to 4) to reflect improvements beyond swimmable, reflecting high sensitivity of shellfish beds to water quality.

Lant and Roberts (1990) 3 yes IA, IL Edwards River, South Skunk River, and Wapsipinicon River Basins Study used the WQL

Lant and Tobin (1989) 9 yes IA, IL Multiple major IA and IL river basins Study used the WQL

Lichtkoppler and Blaine (1999) 1 no OH Ashatabula River and Harbor The study area was designated as an "area

of c oncern" due to dredging restrictions because of PCB contamination, fish consumption advisory, and degraded wildlife habitat

. Assign ed baseline WQL values ranging from 1.25 to 2.5 to reflect not suitable to boating. Assigned post

- improvement WQL values of 4.5 to 5 (changes of 3.25 and 2.5) to reflect rough and game fishing, respectively.

Lindsey (1994) 8 no MD Chesapeake Bay Study focuses on i mprovements in recreational use to support swimming. Assigned baseline WQL values of 4.5 to 5 to reflect rough fishing to game fishing. Assigned post

- improvement WQL of 7.0 to reflect swimming (changes of 2.5 to 2.0)

Lipton (2004) 1 no MD Chesapeake Bay Respondents are asked to rate water quality on a five tiered scale that is tied to whether they feel their boating is restricted due to water quality conditions . Assigned baseline WQLvalues ranging from 0.5 to 1.25 for polluted conditions not suitable for boating. Assigned post

- improvement WQL values of 1.2 5 to 2.5 to reflect
average

- values below boating and for boating, respectively (changes of 0.75 to 1.25).

Londoño Cadavid and Ando (2013) 2 yes IL Streams near cities of Champaign and Urbana Study used the WQLSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 81



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Loomis (1996) 1 no WA Elwha River The study focuses on declines in historic fish runs and potential restoration. Assigned baseline WQL value of 5.0 because Elwha River supports game fishing. Improvements would occur within the game fishing range. Assigned values of 6.0 to 7.0 (changes of 1.0 to 2.0) on WQL to reflect the midpoint and maximum of the game fishing range. Lyke (1993) 2 no WI WI Great Lakes
and tributaries that full under Great Lakes fishing regulations The study estimates values based on a

sample of nonusers with conditions described in terms of catch rates and contaminant levels. Assigned baseline WQL value of 5.0 to reflect game fishing because trout and salmon fishing are present at baseline. Post

- improvement WQL value assignments range from 5.5 to 9.0 (changes of 0.5 to 4 .0) across scenarios depending on effect on catch rates and contaminant levels.

Matthews et al. (1999) 2 no MN Minnesota River Recreational uses are limited under the polluted baseline condition largely due to nutrient loadings. Assigned baseline WQL value of 2.5 to reflect boating and canoeing. Assigned post

- improvement WQL levels ranging from 4.5 to 5.0 (change of 2.0 to 2.5) t o reflect increased rough and game fishing. Opaluch et al. (1988) 1 no NY Peconic Estuary Study uses multiple attributes including undeveloped land, wetland acreage, farmland acreage, eelgrass acreage, and acres of safe shellfish beds. Pollution has resulted in the closure of 4,000 acres of shellfish beds. Assigned baseline WQL value of 7.0 to reflect support for multiple recreational uses, including fishing and swimming. Assigned post

- improvement WQL level of 9.0 to reflect reduction in shellfish closures (change of 2.0). Assignment reflects high sensitivity of shellfish beds to water qualit y.

Roberts and Leitch (1997) 1 no MN, SD Mud Lake Survey indicates that the lake supports a wide range of uses. Swimming is excluded from the assignment because the lake is a wetland not suited for swimming. Assigned baseline WQL value of 5.0 to reflect boating and game fishing. Survey focuses on WTP for preventing degradation. Assigned WQL level of 4.5 for rough fishing (absolute change of 0.5) in the absence of intervention to reflect reduction in fishing services.
Rowe et al. (1985) 1 yes CO Stretch of Eagle
River Study used the WQLSeptember 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 82



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Sanders et al. (1990) 4 no CO Up to 11 wild and/or scenic
rivers Wild or scenic rivers, by definition, are likely

to be of excellent (swimmable) water quality, therefore assigned baseline WQL value of 7.0. Survey focuses on WTP for preventing degradation to wild and scenic rivers. Assigned WQL values range from 4.5 to 5.0 (absolute change of 2.5 and 2.0) in the absence of intervention to reflect a
reduc tion in uses to below the swimmable range. Schulze et al. (1995) 2 no MT Clark Fork River Basin The survey discusses 8 hazardous sites in

Montana that have been put on EPA's National Priorities List for cleanup under the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA)

. At baseline, nearby surface waters are affected by industrial and mining wastes. Assigned baseline WQL value of 1.25 to 2.5 to reflect high levels of pollution and potential support for boating. Assigned post

- improvement WQL levels of 4.5 to 5 (changes ranging from 2.0 to 3.75) to reflect effects of restoration on rough and game fishing services.

Shrestha and Alavalapati (2004) 2 no FL Lake Okeechobee Basin The study focuses on use of silvopasture

(which combines trees, forages, and shrubs with livestock operations) for limiting phosphorus runoff. Assigned baseline WQL values ranging from 2.5 to 3.5 to reflect limited recreational uses, below the fishable

ran ge on average. Assigned post

- improvement WQL levels of 3.5 to 4.5 (changes of 1.0) to reflect additional support for fish habitat and rough fishing.

Stumborg et al. (2001) 2 no WI Lake Mendota The Lake Mendota watershed had been on the Wisconsin Department of Natural Resources (WDNR) Priority Watershed Program . The baseline probability of an algae bloom on any given day during

summer time was 50% (1 of every 2 days). Assigned baseline WQL value of 2.5 to reflect limited recreational use given frequency of algae blooms. Assigned post

- improvements WQL levels of 4.5 to 5 (change of 2.0 to 2.5) to reflect improvements in fish populations and rough and game fishing given 50% reductions in phosphorus loadings.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 83



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Sutherland and Walsh (1985) 1 no MT Flathead River and Lake Area The study area has high quality water at

baseline suitable for fishing, boating, shoreline camping, sunbathing, and swimming . Therefore, assigned baseline WQL value of 7.0. Assigned values of 4.5 to 5.0 (absolute chang e of 2.5 and 2.0) to reflect reduction in recreational services in the absence of protection.

Takatsuka (2004) 4 no TN Clinch River Valley Survey expressed conditions in terms of

three metrics (aquatic life, sport fish, and general water quality) rated in terms of poor, fair, good. Assign ed baseline WQL value of 5 because the river valley supports bass. Assigned post

- improvement WQI levels of 6.0 to 7 .0 across observations to reflect increase in gamefish abundance and support for swimming (changes of 1. 0 to 2.0).

Wattage (1993) 3 yes IA Bear Creek Study used the WQL. Welle (1986) 6 no MN MN lakes that could be damaged by acid rain At baseline, the study lakes have high

water quality. As stated in the study, “ Lakes and streams are generally clean and unpolluted. Remote lakes and forest are relatively pristine. ” Assigned baseline WQL values of 7.5 to 8.5 to reflect water that are “generally clean and unpolluted” or better and supports multiple recreational uses. Assigned WQL levels of 4.5 to 5.0 (abso lute changes of 2.5 to 4) in the absence of intervention to reflect impacts on recreational uses and aesthetics including angling.

Welle and Hodgson (2011) 3 no MN Sauk River Chain of Lakes Watersh ed and Lake Margaret

Gull Lake Watershed Description of baseline and changes

focus es on prevalence of algae blooms that diminish finishing and swimming. Calculate WQL values for baseline and post

improvement by weighting WQL levels between 2.5 and 7.0 by the frequency of algae blooms. Results in baseline WQL

v alue of 3.85. The improvement program would include reductions in algae blooms from 60

- 80 % o f during summer months to 30 to 50%

. Calculate post

- improvement WQL values by weighting WQL levels between 2.5 and 7.0 by the frequency of algae blooms. Results in post

- improvement level of 5.2 (change of 1.35). Wey (1990) 2 yes RI Great Salt Pont on Block Island Study used the WQL.

Whitehead and Groothuis (1992) 3 no NC Tar

- Pamlico River The river has experienced declining fish

catches, diseases in fish, algae blooms, and losses in shellfish bed s and underwater grasses as a result of nonpoint source pollution. Assigned baseline WQL values of 4.5 to 5 to reflect rough fishing to game fishing at basel ine in spite of declines. Valuation scenario would result in a doubling of recreational catch rate and general improvements for boating and swimming. Assigned post

- improvement level of 7.0 to reflect swimming (change of 2.5 and 2.0).September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 84



### Table B

- 20 . Coding of WQL Values for Primary Studies in the Metadata A,B

Author(s) and Publication Year Obs. in

Metadata Study

Used the WQL? (yes/no) State(s) Waterbody Baseline Improvement

Whitehead (2006) 3 no NC Neuse River Study asks respondents about perceived water quality in terms of poor, fair, good, and excellent when thinking about uses such as boating, fishing, and swimming. These c ategories are assigned WQL levels of 2.5, 4.5, 5, and 7, respectively

. Baseline WQL values vary across observations from
2.5 to 5.0. All observations evaluate improvement to “excellent”. Post

improvement WQL values. Assigned post

- improvement WQL value of 7.0 consistent with swimming. Changes range from 2.0 to 4.5 across
obs ervations.

Whitehead et al. (1995) 2 no NC Albemarle

Pamlico Estuary S ystem The survey described individual uses of the

system such as fishing, boa ting, and swimming. Assigned baseline WQL values range from 5.0 to 7.0 to reflect uncertainty regarding the respondent perceived baseline. The study elicited respondent willingness

- to

- pay “to make sure water pollution does not get worse and wildlife habit at remains the same in the A

- P system.” Assigned WQL value of 4.5 for rough fishing to reflect degradation in the absence of restoration (absolute change of 0.5 to 2.5). Whittington et al. (1994) 1 no TX Galveston Bay The bay supports fish, shellfish, and coastal birds, but half the bay is closed for shellfishing due to public health concerns. Assigned baseline WQL value of 5.0 to reflect game fishing. Assigned post

- improvement WQL value of 7.0 to reflect approximate water quality to support 50% more oyster beds for public use, as well as improvements in fishing and swimming. (A): All studies were initially coded using the 10

- point WQL. WQL values were then multiplied by 10 to calculate baseline and post WQI values. E PA coded WQL values as four variables, baseline_low and pos t _low , and baseline_high and post_high . These values allow EPA to capture a range of starting and ending values given uncertainties during analyst translation and coding, Low and high WQL changes are calculated as the difference between their respective start and end points. Following the 2009 MRM, lnquality_ch is the natural log of (( post_high

- baseline_high)*10) and l nbase is the natural log of ( baseline_low *10). MRM1 is insensitive to the use of baseline_low versus basline_high . Model fit, estimated coefficients, and variable significance ar e all very similar. (B): Shading indicates studies added since the 2009 meta

- analysis. (C): As described in Section 2, to be included in the metadata, studies must value water quality changes affecting ecosystem services provided by waterbodies, including recreational activities (such as fishing, boating, and swimming), aquatic life support and other nonuse values. EPA conducted additional review of the Cameron and Huppert (1989) survey materials during preparation of this appendix . The study asked recreational users to value improvements focusing on a doubling of catch rates for salmon and str iped bass using methods including fishing stocking and habitat restoration. The survey describes demand for water generally but does not specifically mention water quality. Because it does not mention water quality specifically, EPA has de termined that the Cameron and Huppert (1989) study and its one observation should be dropped from the metadata prior to finalization of the MRM. In order t o assess sensitivity to this change, EPA estimated MRM1 with the Cameron and Huppert (1989) observation dropped fro m the metadata and found that results were nearly identical to the results presented in Section 4.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 85



# Appendix C: MRM2 with Observation 17 Excluded

Table C

- 1

. MRM2 preliminary specification estimated by OLS excluding observation 17

. In the far right columns, a single * indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap
confidence intervals do not include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept 0.231 4.64 0

- 18.1 00 15.5 00

ce 0.546 0.315

- 0.714 1.64 0
thesis 0.591 0.248

- 0.317 1.71 0 * lnyear

- 0.386 0.1 00

- 0.966

- 0.115 * **

volunt

- 1.52 0 0.215

- 2.24 0

- 0.761 * ** outliers_trim

- 0.504 0.173

- 0.975 0.187 *

non_param

- 0.609 0.114

- 0.971 0.00 8 * non_reviewed

- 0.593 0.2 00

- 1.43 0

- 0.05 2 * ** lump_ sum 0.917 0.189 0.358 1.57 0 * ** wtp_median

- 0.34 0 0.34 0

- 1.32 0 0.663

n ortheast 1.18 0 0.319 0.261 2.3 00 * **

c entral 0.576 0.157

- 0.202 1.22 0 *

s outh 1.38 0 0.167 0.696 2.09 0 * **

nonusers

- 0.458 0.152

- 0.813

- 0.156 * **

lnincome 0.206 0.424

- 1.1 00 1.86 0 mult_bod

- 0.695 0.201

- 1.5 00

- 0.015 * **

river

- 0.08 6 0.143

- 0.613 0.495 swim_use

- 0.433 0.184

- 0.929 0.035 * gamefish 0.521 0.299

- 0.36 0 1.32 0 boat_use

- 0.709 0.221

- 1.28 0 0.036 * ln_ar_ agr

- 0.357 0. 100

- 0.642 0.09 4 * ln_ar_ ratio

- 0.07 6 0.026

- 0.152 0.05 5 * sub_proportion 1.19 0 0.235 0.052 2 .000 * **
Q

- 0.016 0.00 5

- 0.03 3

- 0.002 * **

R2 0.618
RCV2 0.426

rWTP2 0.574September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 86



### Figure C

- 1

. MRM2 preliminary specification estimated by OLS excluding observation 17

. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Marginal willingness to pay curve. Demand curve shifters are set as described in the main text. The black line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, an d 97.5th percentiles.

(a) (b)

(c)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 87



### Table C

- 2

. MRM2 preliminary specification estimated by WLS excluding observation 17

. In the far right columns, a single * indicates b/ standard error > 1.96 and ** indicates that the 95% bootstrap
confidence intervals do not include zero.

Parameter b Standard
Error Bootstrap Confidence

Interval b/Standard

Error > 1.96 (*=yes) CI include

zero (**=yes) 2.5 th 97 .5 th
intercept

- 1.12 0 5.24 0

- 19.7 00 15.5 00

ce 0.379 0.327

- 0.714 1.6 00
thesis 0.827 0.273

- 0.294 1.79 0 * lnyear

- 0.438 0.127

- 0.934

- 0.080 * **

volunt

- 1.45 0 0.229

- 2.29 0

- 0.76 0 * ** outliers_trim

- 0.376 0.202

- 1.01 0 0.284

non_param

- 0.424 0.118

- 0.846 0.07 6 * non_reviewed

- 0.741 0.203

- 1.48 0

- 0.112 * ** lump_sum 0.882 0.186 0.23 0 1.52 0 * ** wtp_median

- 0.158 0.325

- 1.28 0 0.798
northeast 1.24 0 0.313 0.291 2.3 00 * **

central 0.589 0.212

- 0.291 1.19 0 *

south 1.42 0 0.231 0.639 2.09 0 * **

nonusers

- 0.59 0 0.146

- 0.929

- 0.208 * **

lnincome 0.339 0.467

- 1.13 0 2.01 0 mult_bod

- 0.856 0.21 0

- 1.71 0

- 0.082 * **

river

- 0.06 1 0.171

- 0.668 0.502 swim_use

- 0.312 0.176

- 0.89 0 0.174 gamefish 0.214 0.377

- 0.674 1.4 00 boat_use

- 0.702 0.231

- 1.42 0 0.054 * ln_ar_agr

- 0.282 0.111

- 0.602 0.115 * ln_ar_ratio

- 0.035 0.035

- 0.151 0.078 sub_proportion 1.14 0 0.268 0.029 2 .000 * **
Q

- 0.01 4 0.006

- 0.035 0.00 5 *

R2 0.761
RCV2 0.524 pseudoRWLS2 0.554

rWTP2 0.543September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 88



### Figure C

- 2

. MRM2 preliminary specification estimated by WLS excluding observation 17

. (a) Scatter plot of predicted vs observed marginal WTPs. (b) Scatter plot of predicted vs observed stated WTPs. (c) Margina l willingness to pay curve. Demand curve shifters are set as described in the main text. The black line is the point estimate based on the primary dataset. Red lines are the bootstrapped 2.5th, 50th, and 97.5th percentiles.

(a) (b)

(c)September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 89



# Appendix D: MRM2 Bootstrapped Histograms of Step

- wise Regression Coeffcent Estimates

Figure D

- 1 . MRM2 B ootstrapped H istograms of S tep

- wise OLS R egression C oefficient E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 90



### Figure D

- 1

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise OLS R egression C oefficient
E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 91



### Figure D

- 1

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise OLS R egression C oefficient
E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 92



### Figure D

- 1

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise OLS R egression C oefficient
E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 93



### Figure D

- 2 . MRM2 B ootstrapped H istograms of S tep

- wise W LS R egression C oefficient E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 94



### Figure D

- 2

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise W LS R egression C oefficient
E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 95



### Figure D

- 2

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise W LS R egression C oefficient
E stimates.September 29, 2015 WQ Meta

- Analysis Peer Review Document | pg 96



### Figure D

- 2

. (contd) MRM2 B ootstrapped H istograms of S tep

- wise W LS R egression C oefficient
E stimates.