Peer Review Report

316(b)Stated

Preference Survey

Report Document

(Final Submission)

Prepared by

Applied Planning Corporation

EPA Contract: EP-H-000334
Table of Contents

PURPOSE OF PEER REVIEW.................................................................................................. 1

SUMMARY OF EPA VALUATION STUDY............................................................................ 2

Benefit Estimation....................................................................................................................... 2

The Choice Experiment............................................................................................................... 3

PEER REVIEW............................................................................................................................ 6

SUMMARY................................................................................................................................. 6

REVIEWERS’ GENERAL COMMENTS ................................................................................ 8

REVIEWERS’ RESPONSES TO CHARGE QUESTIONS .................................................... 9

Is the choice experiment survey format an appropriate means to estimate total

willingness to pay for the characterized policies, including use and nonuse values? ..... 9

Were the procedures EPA used to develop and test the proposed survey instrument

consistent with current best practices for stated preference surveys, as reflected in the

peer-reviewed literature? ................................................................................................... 12

Is the overall quality and design of the survey booklet adequate for its intended

purpose? Is the information provided by the survey booklet sufficient and

appropriate (e.g., is there appropriate emphasis on the budget constraint facing

respondents, are appropriate steps taken to minimize hypothetical and strategic bias)?

Does the information provide a sufficiently unbiased perspective to allow valid welfare

estimation............................................................................................................................. 14

Does the experimental design allow for the estimation of robust values across the

range of regulatory policy options and the scope of the resource changes?.................. 16

Are the sample sizes and response rates for the regional and national surveys adequate

to produce statistically robust estimates? ......................................................................... 17

EPA estimated unweighted linear, weighted linear and quadratic RUM models for the

regional and national data. For the calculation of implicit prices and monetization

EPA used the results of the weighted linear models across all the survey regions and

national analysis because of empirical and theoretical concerns. Is this approach

appropriate for estimating welfare changes associated with expected changes in survey

attributes under the 316b regulation? Are there other functional forms that EPA

should investigate that might be theoretically or empirically superior? ....................... 19
Is the nonresponse study design adequate to evaluate the potential for non-response

bias? Are the steps EPA used to correct for non-response bias adequate to address the

potential biases identified? Is EPA’s development of weighting values and their use

consistent with current best practices in the literature? ................................................. 21

Are the scope tests performed correctly and are the results interpreted appropriately?

Are there additional tests that could be conducted to further assess the validity of

responses? ............................................................................................................................ 23

What are your expectations regarding the validity and reliability of the benefit

estimates that are provided by this survey, compared to the current stated preference

literature? Would you expect the results provided by the survey to be superior to

those that EPA prepared as part of the national benefits assessment for the proposed

316(b) existing facilities rule? Would you recommend that the information provided

by this survey be used in EPA's cost benefit analysis of the 316(b) rule?...................... 25

Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed

the potential implications of uncertainties in the analysis, and are the effects on the

conclusions drawn presented in a transparent manner? ................................................ 27

SPECIFIC COMMENTS ON REPORT................................................................................. 29

ROADMAP FOR THE EPA...................................................................................................... 34

Estimation Robustness............................................................................................................. 34

Sample Selection and Non-Response Study............................................................................ 35

Validity Checks......................................................................................................................... 35

Discussion of Over/Under Estimation of WTP....................................................................... 36

Reporting and Documentation ................................................................................................ 36

CONCLUSION ........................................................................................................................... 36

APPENDIX I ............................................................................................................................... 38

Kevin Boyle.............................................................................................................................. 39

Diane Dupont........................................................................................................................... 45

Jim Kahn.................................................................................................................................. 55

Dan Phaneuf ............................................................................................................................ 65

John Whitehead ...................................................................................................................... 72

APPENDIX II.............................................................................................................................. 80

Responses to EPA Clarification Questions - Boyle.......................................................... 81
Responses to EPA Clarification Questions – Dupont...................................................... 83

Responses to EPA Clarification Questions – Kahn.......................................................... 85

Responses to EPA Clarification Questions – Phaneuf .................................................... 87

Responses to EPA Clarification Questions –Whitehead ................................................ 89

APPENDIX III............................................................................................................................ 91

Clarification Questions........................................................................................................... 92

Kevin Boyle’s response to clarification question.................................................................. 92

Diane Dupont’s response to clarification questions............................................................. 93

Jim Kahn response to clarification questions....................................................................... 93

Dan Phaneuf’s response to clarification questions .............................................................. 93

John Whitehead’s response to clarification questions......................................................... 94
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 1

I. PURPOSE OF PEER REVIEW

Section 316(b) of the Clean Water Act

The U.S. Environmental Protection Agency is developing rules under Section 316(b) of the

Clean Water Act (33 U.S.C. 1251 et seq). Section 316(b) establishes national performance

requirements that are designed to minimize adverse environmental impacts caused by cooling

water intake structures (CWISs). These structures negatively impact aquatic biota through

impingement mortality (where fish or other aquatic life are trapped and killed on equipment at

the entrance to the CWIS) and entrainment mortality (where fish and other aquatic organisms are

taken into the cooling system, passed through the heat exchanger, killed, then discharged back

into the source body). Additional adverse impacts on biota are associated with the operation of

CWISs such as thermal discharges, chemical effluents, flow modifications and nonlethal effects

of impingement caused by the plants.

Due to these lethal and nonlethal effects section 316(b) requires that CWISs minimize their

adverse environmental impacts through reduction of volume, frequency and/or seasonality of

water withdrawals in order to enhance ecosystem functions of affected water bodies. The EPA

has developed section 316(b) rule in several phases since 2000 to address new and existing

manufacturers, power plants and offshore oil and gas facilities using CWISs that withdraw over

2 million gallons of water daily from U.S. waters.

Under Executive Orders 12866 and 13563 the EPA is required to estimate the potential benefits

and costs to society of rules it proposes. In order to monetize the ecological gains from the

Section 316(b) rule, the EPA requested and received approval from the Office of Management

and Budget to conduct a stated-preference survey that included choice questions to estimate the

environmental benefits of changes in operations of CWIs. The stated preference (SP) study was

designed to estimate marginal willingness to pay (WTP) for selected environmental attributes

that would be affected by implementation of the rule. The use of choice questions provides an

estimated valuation model that can be used for a variety of implementation procedures associated

with the 316(b) rules.

Purpose and Process of the Peer Review

The purpose of this report is to present a Peer Review of the survey design and implementation,

data analyses and reporting. Peer review is a process for identifying and enhancing the scientific

content of a study, which will enable EPA to understand the confidence and limitations they may

place on study results to support policy analyses. The EPA report and associated materials were

reviewed by:

 Dr. Kevin J. Boyle, Virginia Tech – Blacksburg, VA,

 Dr. Diane P. Dupont, Brock University - St. Catharines, ON, Canada,

 Dr. James R. Kahn, Washington and Lee University – Lexington, VA,

 Dr. Daniel J. Phaneuf, University of Wisconsin – Madison WI, and

 Dr. John C. Whitehead, Appalachian State University – Boone, NC.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 2

Drs. Boyle and Kahn were part of a 2006 peer review panel that reviewed a stated-preference

survey developed in 2005 to assess the environmental benefits of changes in the operations of

CWIs. EPA used comments from that review to develop a new survey instrument that is being

reviewed here.

The main document the reviewers were asked to focus on is entitled, “316(b) Stated Preference

Survey Peer Review Document”. This document includes EPA’s documentation and analysis of

the stated preference survey design and analysis of the resultant data. The review panel was also

provided a document entitled “Summary of Public Comments on the 316(b) Stated Preference

Survey,” which contains the comments received during the Notice of Data Availability (NODA)

public comment period and EPA responses. EPA also provided the following background

documents and files to be used as a resource if additional information was necessary for

reviewing EPA’s methodology and results:

1. ICR supporting statement

2. Focus Groups Executive Summary Memorandum

3. All versions of the survey instrument for four regional surveys and a national survey

4. An outline of the quality assurance measures use during the implementation of the survey

5. The Stated Preference Survey NODA

6. The NODA supporting document and Memorandum

7. An Excel file including all data from the main and non-response surveys

8. The Economic and Environmental Benefits Assessment Report for the 316(b) proposed

rule

There were three stages of the peer review process prior to the compilation of this peer review

report. In the first stage, each member of the panel individually provided written responses to

charge questions provided by the EPA. In the second stage, the written comments were shared

among the panelists and each reviewer was asked to comment on the comments of other panel

members. The third stage was a conference call with the peer review panel organized by Mr.

Peter Chaikin. After the conference call, this peer review report was written summarizing the

panelists’ comments on the EPA report. This document was reviewed by the panel members

before submission to the EPA. We will refer to this process as the Applied Planning Corporation

peer review.

This peer review is intended to assess the work that EPA has completed to date on the 316(b)

stated-preference research. The peer reviewers were asked to identify areas where modifications

to EPA’s estimation and validation approaches would improve the accuracy and precision of

willingness-to-pay estimates.

II. SUMMARY OF EPA VALUATION STUDY

Benefit Estimation
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 3

EPA’s benefit analysis for the proposed 316(b) rule includes estimates of changes in use values

of commercial and recreational fisheries, and a partial estimate of changes in non-use values

(USEPA 2011a).1 Non-use values are values that people may hold for an environmental

improvement that are not tied to any direct observable use of the resource such as recreational

fishing.

After the proposed rule was released, EPA conducted a stated preference study to estimate total

(use plus non-use) benefits to the public places of ecological improvements arising from

regulating CWISs. The use of a SP survey reflects recent EPA guidelines for benefits analysis

(USEPA 2010, p. 7-41) that recognize the “advantages of [stated preference] methods

include[ing] their ability to estimate non-use values and to incorporate hypothetical scenarios

that closely correspond to a policy case.” It is this stated preference study for which this

document provides a peer review.

The objectives of the SP study were to “explore how public values (including non-use values) for

fish and aquatic organisms are affected by I&E mortality at cooling water intake structures

(CWIS) located at existing 316(b) facilities, as reflected in individuals’ willingness to pay for

programs that would prevent such losses” (USEPA 2011, p. 30).

The Choice Experiment

Stated preference surveys ask individuals to make choices from which researchers elicit

information to estimate individuals’ values for specified changes in an environmental amenity

and are typically framed in terms of estimating willingness-to-pay (WTP). WTP is the

maximum amount of money that an individual or household will pay for a specified

environmental change and respondents express information to estimate their WTP through

choices over policy options. Advantages of choice-based questions include similarity to

referenda or market choices where individuals are familiar with choosing among alternative

policy options or commodities at specified costs (Freeman 2003). Carefully designed SP

questions include features to reduce hypothetical and other possible biases that can result from

asking survey questions versus assessing WTP through market transactions or binding referenda.

The 316(b) survey was designed as a choice experiment following established procedures

(Adamowicz et al. 1998; Louviere et al. 2000; Bennett and Blamey 2001; Bateman et al. 2002).

Choice experiments are a SP technique in which people’s values are estimated based on their

choices over a set of hypothetical alternative states defined by attribute levels which may or may

not contain policy options. Respondents are presented with a set of multi-attribute alternatives

and asked to select their preferred alternative. This format has been applied to assess WTP for

ecological resource improvements similar to those at issue in the 316(b) policy case (e.g.,

Bennett and Blamey 2001; Hanley et al. 2006a, b; Hoehn et al. 2004; Johnston et al. 2002,

2011a, b; Milon and Scrogin 2006; Morrison and Bennett 2004; Morrison et al. 2002; and

Opaluch et al. 1999).

1 The Environmental and Economic Benefits Analysis (EEBA) for the proposed rule is available online at

http://water.epa.gov/lawsregs/lawsguidance/cwa/316b/upload/environbenefits.pdf.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 4

An example of a choice question used in the 316(b) Northeast survey is the following:

Respondents considered Policy Option A, Policy Option B, and No Policy (current situation) and

chose choosing the option that they preferred. The attributes and their levels in each choice

option reflected a feasible outcome under alternative 316(b) regulatory scenarios. Respondents

were asked to answer choice questions that differed in the levels assigned to the attributes.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 5

This SP study was designed as a mail survey sent to households in different regions of the

country and to a nationally representative sample. The target population for the SP survey was all

households in each region or from the continental U.S., and respondents from each household

were 18 years of age or older. Alaska and Hawaii were excluded because they include only four

in-scope non-recirculating facilities, represent a small percentage of overall household

population, and are separated geographically from the states in each survey region. The

population of continental households was stratified into four survey regions (Northeast,

Southeast, Inland, and Pacific) and each region had a version of the survey that presented

attribute levels specific to it. These regions were defined by state boundaries and differed from

the 316(b) benefits regions used in the Environmental and Economic Benefits Analysis (EEBA)

for the proposed rule. The national survey presented attribute values for all U.S. waters. The

surveys were allocated across the four regions based on the number of households in each region

relative to the total number of households in the continental United States. In addition, a

minimum number of completed surveys were required for each region. The survey instrument

was designed to maximize the response rate and procedures were employed to identify potential

non-response bias.

In July 2011, The Office of Management and Budget (OMB) approved implementation of the

Northeast survey version as a pilot study to inform survey design prior to full implementation in

all regions and the national sample. Implementation of the remaining survey versions was

approved in November 2011. EPA mailed the survey to 7,840 households in total, and received a

total of 2,313 completed surveys with an average response rate of 32% across the five sampling

regions.

EPA also conducted a follow-up study of households who did not return a completed mail

survey. The follow-up examined whether non-respondents answered certain questions in a

statistically different way from respondents, as well as checked observable socio-demographic

characteristics for statistical differences. The follow-up study included a set of key attitudinal

questions and socio-demographic variables that are likely to be associated with WTP for

reducing fish mortality from cooling water intake structures (CWISs) and improving fish

populations and conditions in the affected aquatic ecosystems. EPA implemented the follow-up

study using two subsamples: the first subsample received a paper questionnaire via priority mail

and the second subsample was surveyed by telephone. Both non-response subsamples were

asked the same set of attitudinal and demographic questions.

The survey sections included:

 Relative Importance of Issues Associated with Industrial Cooling Water.

Respondents considered the relative importance of key issues associated with the use of

cooling water by industrial facilities.

 Concern for Policy Issues. Respondents were asked questions about the relative

importance of different policy issues.

 Relative Importance of Effects. Respondents were asked questions to promote

understanding of the metrics included in the stated-preference questions.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 6

 Voting for Regulations to Prevent Fish Losses in the Respondent’s Region (or

Nationally). The SP questions in this section were the key part of the survey asking

respondents’ choices when presented with specific fish-related and other ecosystem

resource changes within their region at specified costs to their households.

 Reasons for Voting “No Policy”. This question provided information used by EPA to

identify protest responses; however, not all “no policy” responses are protests.

 Respondent Certainty and Reasons for Voting. This section is designed to identify

respondents who incorrectly interpreted the choice questions or the uncertainty of

outcomes.

 Recreational Experience. This question elicits recreational experience data to

investigate if respondent characteristics influence responses to the SP choice questions.

 Demographics. Responses to these questions are used to estimate the influence of

demographic characteristics on respondents’ SP choices, and ultimately, their WTP to

prevent I&E mortality losses of fish.

EPA’s analysis of the 316(b) SP survey data is grounded in the random utility model of

consumer choice (Hanemann, 1984; McConnell, 1990). The use of the random utility model is

standard in the SP literature for attribute-based valuation exercises such as choice experiments,

and allows the estimation of well-defined welfare measures (i.e., WTP). EPA used regression

models (particularly the mixed logit model) based on random utility specification of choices to

estimate annual household willingness-to-pay for an improvement in the environmental attribute

levels included in the survey (fish saved, commercial fish populations, fish populations (all fish),

and aquatic ecosystem condition). EPA used WTP as the value measure in the estimation of the

regional and national benefits of the regulatory options presented in the proposed rule.

III. PEER REVIEW

SUMMARY

Overall, the panel felt that the design procedures were consistent with best practices. They felt it

was “a strong study of high caliber that will be useful to the EPA now and in the future”.

However, they identified several issues of importance that they feel should be addressed to

enhance the usefulness of the study results and the overall credibility of the study. These

considerations are summarized immediately below.

Survey Design

- Investigate the effects of overlapping attribute definitions by looking for attribute

interaction effects and the effects of dropping attributes from the estimation.

- Due to the survey design, focus solely on the “fish saved” variable in the policy analysis

due to the overlapping definition with attribute definitions.

Sample Selection and Non-Response Study
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 7

- Discuss why the variables used in the sample-selection analyses are relevant and would

impact WTP.

- Provide a breakdown of the responses to specific survey questions such as “how

important is the protection of aquatic ecosystems to you” by the socio-demographic

variables (gender, race, income, etc.) that were found to differ between the main samples

and non-response samples.

- Conduct more statistical tests on the effects of sample selection and on the assumptions

imposed in the sample selection process.

- Address whether the weighted model should be used for benefit estimation and applied to

all people in the sample frames, or if the unweighted model should be used and applied

only to the proportion of the population consistent with the survey response rates.

Reporting/Documentation

- Enhance the discussion of motivation for the overall study, as well as the motivation for

the steps taken in each subsection.

- Provide a stronger justification for the procedures that are used.

- More thoroughly discuss the implications of analysis procedures on estimated WTP.

- Emphasize regional results as opposed to national results.

- Include a new section on the over and underestimation of WTP integrated into

uncertainty analyses to clarify the directional effects of potential biases. Specific

examples include:

o Address potential confusing and overlapping attribute descriptions.

o Map used in the survey may give respondents the impression that they are located

near an affected area and that the effects are widespread geographically; however,

the wording in the survey does not support such a conclusion.

Robustness Checks

- Investigate additional specifications such as conditional logit and multinomial logit

models.

o Simple conditional logit model could be used as a baseline

o Multinomial logit models will help consider the robustness of the mixed logit

assumptions.

- Report how estimates change for the mixed logit model under different assumptions.

- Investigate pooling regions which would allow for more split sample analyses.

- Investigate nonlinearities.

Validity Tests

- The external scope tests conducted are not convincing; conduct external validity tests that

include responsiveness to scope with a split sample survey design

- Report statistical tests for the analyses conducted rather than only reporting summary

statistics for data splits.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 8

- Use analyses and information from the focus groups, if possible, to address some of the

peer-review comments and industry comments on survey design and interpretation of

statistical results.

The panel felt that the survey design, data analyses and reporting supported the use of estimates

of willingness to pay in benefit analyses of the 316(b) rule. The comments are intended to

inform the uncertainty analyses of the study to enhance the clarity and validity of study results.

REVIEWERS’ GENERAL COMMENTS

Some of the panel members elected to make an overall statement prior to expanding on each of

the Charge Questions, while others choose to forgo an opening statement. The statements of

those members who chose to provide overall comments are found in this section below.

Kevin Boyle

“This study uses design practices that are better than is standard in the literature, and the data

analyses follow standard practice. As with any study, in hind sight I have identified components

that could have been designed better. These issues cannot be addressed at this point, but potential

concerns need to be considered in the computation of aggregate benefits. There are elements of

the econometric analysis and documentation that need to be addressed and can still be improved

before the final computation of benefits.”

Jim Kahn

“The first thing I would like to point out is that researchers make a large number of choices when

researching any topic, but this is particularly true with choice-modeling experiments. It is always

possible to second-guess these choices and criticize the researchers for choosing A instead of B,

when, in reality, both A and B have merits and have flaws. There is no such thing as a perfect

choice-modeling experiment. The important questions are whether the survey was conducted

with best practices in a manner in which is unlikely to have significant biases, and whether the

survey was conducted in a fashion in which the confidence intervals of the value estimates is

likely to be tight enough for meaningful policy analysis.”

“I would like to say that I have a great deal of confidence in this survey. First, it was conducted

by two consultants who are truly experts in the field. Second, when they encountered challenges

(proper definition of attributes, nonresponse issues, etc.) they addressed these challenges with

vigor and rigor. Third, and most importantly, the researchers did not try to overreach, but

measured those values that could be most confidently measured. Finally, the documents explain

complicated issues in a manner that can be understood by people that are not experts in the field.

After securing permission from EPA and the authors, I would like to use this study in my

teaching.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 9

John Whitehead

“My specific comments come off rather strong in my review of the document, but I believe that

there is a lot of good work here that is useful to EPA.”

REVIEWERS’ RESPONSES TO CHARGE QUESTIONS

1. Is the choice experiment survey format an appropriate means to estimate total

willingness to pay for the characterized policies, including use and nonuse values?

Overall, the panel agreed that the choice experiment survey format conformed to best practice in

obtaining use and non-use values for the evaluation of rule 316(b), particularly for the “fish

saved” variable. However, they also agreed that more work is needed to make clear that the WTP

estimates of non-use value are valid and reliable. Suggestions for additional analyses are

included throughout responses to subsequent Charge Questions.

Kevin Boyle

“Yes, this approach is in common use in the economics literature for estimating uses and nonuse

values for changes in natural resources. Applications have been published in the peer-reviewed

literature, and there is a substantial literature investigating the validity of choice experiments

(also known as choice modeling, choice questions, attribute based questions, conjoint analysis,

and other labels).”

Diane Dupont

“With specific reference to this choice experiments survey, the format is appropriate insofar as

the goods/services being valued are clearly understood by the respondents. After reading the

survey materials, I believe that the researchers are able to obtain a total WTP for the

characterized polices and that this total WTP will contain both use and non-use values. I should

also note that CVM, the main competitor question format, would not be as amenable to finding

both use and nonuse values. I am less certain about the ability of the survey to separate out use

from non-use components. Focus group comments suggest that respondents may believe in a

different type of implicit relationship between the various attributes than the one presented. This

is important because it may be that, when they see ecosystem values rise in a particular choice

task, they may infer that this will have some additional favorable impact upon another attribute

of interest (commercial fishing, for example). So, they infer a relationship between the attributes

and not treat them as separate. I am concerned that this might result in some kind of double

counting in the Total WTP, in particular, when people assume a very strong complementary

relationship between use and non-use components.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 10
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 11

Jim Kahn

“Choice modeling is not only an appropriate means to estimate total willingness to pay for these

types of policies, but it is the best method for several reasons. First, when choice-modeling has

been used in potential new product applications (called conjoint analysis in this literature),

hypothetical choices have been very close to actual choices. In contrast, contingent valuation has

been notoriously unsuccessful in having hypothetical choices match actual choices, even with

market goods. Contingent valuation represents a good choice for simple problems with limited

budgets. In the case of the 316b regulation, which is complex in all dimensions, choice-modeling

is the only choice. Fortunately, EPA committed sufficient resources to allow a state of the art

study to take place and the greater expense associated with the choice modeling exercise was not

an issue.”

“Choice modeling is the best method for modeling both direct use and non-use values, although

it is often difficult to conduct an experiment which allows the researcher to produce separate

estimates of direct and indirect use values. However, if the policy question requires an estimate

of total value (rather than separate estimates of direct and indirect use values) this is not an issue

and the value estimates can help inform the policy decision process. Nothing has happened to

change my 2006 conclusions that choice-modeling is the appropriate direction to go to

understand the benefits of the proposed rule-making.”

Dan Phaneuf

“Stated preference methods in general are flexible enough to estimate both use and non-use

values. The choice between contingent valuation and choice experiments in this regard is not

about which one can measure which value. Rather, the choice reflects the needs of the

application. When the objective is to understand how multiple, different attributes affect value –

regardless of the type of value – a choice experiment is the appropriate choice. Given the

importance of the multiple ecological outcomes in this application a choice experiment was a

sensible approach.”

John Whitehead

“A stated preference discrete choice experiment will elicit total willingness to pay (WTP).

However, more work is needed to make clear that the WTP estimates reported in the EPA report

are valid and reliable estimates of nonuse value. Since Krutilla (AER, 1967), nonuse values are

often associated with unique natural resources facing irreversible development. Both uniqueness

and irreversibility are subjective and relative concepts and should be considered as sufficient but

not necessary conditions. Where the resources are not unique (i.e., many substitutes) and

development is not reversible (i.e., the species are not threatened or endangered), more effort is

needed to justify the use of nonuse values in benefit-cost analysis.”

“For example, Krutilla described the motives that are likely to lead to nonuse values. The survey

gathers information on motives for nonuse values to preserve fish for recreation (evidence of an

altruistic motive for those who do not fish), preserve fish for aquatic ecosystems, wanting to

know that fish exist in local lakes, rivers and bays, and preserve fish for future generations
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 12

[question 9]. The report states that most of the respondents had some motive for nonuse values

but this only provides weak evidence that nonuse values are valid. If the responses were analyzed

out of the RUM framework so that an index of motives was a right hand side variable, the WTP

model could be used to extract motivated nonuse values from non-motivated values.”

“On page 17, the first paragraph illuminates a potential problem with the analysis. It is clear that

survey respondents were not likely to have any prior information about the issue before they took

the survey. It is well-known from the CVM literature that survey respondents can have

constructed preferences in a no prior information context and that WTP estimates from

constructed preferences are less valid than preferences based on real experience. Without the

prior information question it is impossible to know whether there is a subsample of invalid WTP

estimates.”

“For these reasons, there is some concern that the nonuse component of the WTP estimates may

lack validity relative to the use value component and the WTP estimated with the full sample

may be aggregated over too large a population. The analysis could be conducted with splitsamples of users vs nonusers (there is no question asking about prior information) and those with

significant nonuse value motives vs. those without to demonstrate the validity of the WTP

estimates of those respondents with questionable rationales for valuing these resources. If subsamples are found to have invalid WTP responses then in a sensitivity analysis, WTP should be

aggregated over only the sample proportion with valid values for benefit-cost analysis.”

2. Were the procedures EPA used to develop and test the proposed survey instrument

consistent with current best practices for stated preference surveys, as reflected in the

peer-reviewed literature?

The panelists largely agreed that the procedures used by the EPA to develop and test the

proposed survey instruments were consistent with the industry standard and followed best

practice. They believed that the use of focus groups was a strength. However, they also

commented that the procedures could be flushed out more in the report as some aspects were

unclear. For example, there was minimal discussion in the report regarding how the pilot study in

the Northeast informed the design and implementation of surveys in other regions.

Kevin Boyle

“Using focus groups, cognitive interviews and pilot studies is consistent with the very best

practice for designing survey instruments. The investigators conducted more focus groups and

cognitive interviews than is typical; the work here is consistent with the number conducted for

high-profile litigation applications. The report mentions the pilot study, but provides no

explanation comparable to the discussion of the focus groups and cognitive interviews. It is

unclear if the Northeast survey implementation is the pilot or if another pilot was also conducted.

The authors mention that the Northeast survey was implemented to inform the other survey

applications, but no mention is made of the specific goals of this effort or if the other surveys

were changed from what was learned from the prior implementation of the Northeast survey

instrument. These latter comments are not intended to be a critique of the survey design process;

rather they are a comment of a lack of clarity in the documentation contained in the report.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 13

Diane Dupont

“In general the procedures followed are consistent with the industry standard (focus groups in

different regions, one-on-one cognitive interviews, peer review, testing of specific wordings, use

of graphics, investigation of possible use of heuristics, etc). My concerns are the following:

- As far as I can see, there was no pre-testing on a larger scale in the field (actual mail

survey pre-test); only focus group testing within a controlled setting.

- None of the focus groups took place in the Pacific Region. Given the weaker results from

the models using data from this region, this is a shortcoming.

- It is not clear whether the national versions were pre-tested similar to the regional

versions. This is important to develop the salience of the attributes at a national, as

opposed to more personal, regional scale.

- It was not clear from the executive report discussion about the focus groups whether

other attributes were salient to focus group respondents. I am wondering in particular

about the thermal discharge. On page 2 the survey talks about how pumping warm water

back into the environment affects ecological systems. Then, later on page 5 there is a

discussion about how closed cycle cooling may reduce fish losses and also thermal

discharge. Was this an attribute that was of concern or interest to focus groups? The

thermal discharge component jumped out at me. If it was not an issue in focus groups,

then perhaps it is worthy of a footnote.”

Jim Kahn

“EPA contracted with two of the most experienced and well-respected researchers in this area.

All the relevant literature is cited, and at each point where problems are encountered, the

solutions are consistent with the literature. I found the discussion of the test surveys and focus

groups to be very useful...I found the focus group comments on the utility of the graphics to be

reassuring. I also thought that the survey was visually appealing and was easy to read even with

the view scale set to “actual size”. Color contrast on the survey made it easy to read and less

fatiguing to the eyes.”

Dan Phaneuf

“The survey development process seems to be one of the strengths of this study. The number of

focus groups and amount of pre-fielding testing exceeds what one usually finds in the published

literature. As such I judge the procedures to have conformed to current best practice.”

John Whitehead

“The EPA’s survey design methods were thorough and state of the art.”

3. Is the overall quality and design of the survey booklet adequate for its intended

purpose? Is the information provided by the survey booklet sufficient and appropriate

(e.g., is there appropriate emphasis on the budget constraint facing respondents, are
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 14

appropriate steps taken to minimize hypothetical and strategic bias)? Does the

information provide a sufficiently unbiased perspective to allow valid welfare

estimation?

The largest complaint about the survey booklet was that the wording was confusing in some

places and that attributes were overlapping. (Overlapping attributes could cause problems if the

EPA focuses on an outcomes other than “fish saved”.) Several panelists also mentioned the lack

of cheap talk and budget reminders in the booklet.

Kevin Boyle

“For the considerable effort invested in the design of the survey booklet, I found the wording in

the survey rough and confusing in places. I do not know how this affected the survey outcome in

all instances, but it does leave unresolved questions. Some of these clarity questions might be

answered by knowledge gained from the focus groups and cognitive interviews that is not easily

conveyed in the report. However, the roughness of the descriptions and overlapping categories is

something that I would expect to have been refined and resolve in such an extensive design

process. I also did not see a clear budget reminder in the survey wording, but respondents were

told that the price increase was “unavoidable” (Northeast survey, p. 7, bottom). There was not a

reminder of substitutes and there was no cheap talk script. While I am not convinced of the

appropriateness of cheap talk scripts, they have become common practice in stated-preference

surveys and their presence does serve to reduce value estimates.”

“Consequentiality was addressed in the survey, but not as clearly as it could have been. The

survey does state that respondent answers will help to make policy decisions (Northeast survey,

p. 1, top), but this is not repeated immediately before the choice questions nor is it specifically

stated that survey results “will be used” in making policy decisions. Respondents are asked to

answer as if the questions were real and binding (Northeast survey, p. 10, bottom), but are not

asked to answer as if they are voting in a real referendum at the moment they answer the choice

questions. Thus, there are elements of consequentiality, but they are not as poignant as they

could have been.”

Diane Dupont

“The overall design and quality of the survey booklet is impressive and enhances the saliency of

the exercise. I generally found the wording to be unbiased with one exception in the discussion

on page 7 that describes the attributes where I have some concerns about overlap of attributes. I

also feel there is a lack of clarity at some points in the information screens. The budget constraint

was addressed but not really emphasized. However, I think the choice of higher electricity prices

is a good one and believable. On another note, the policy effects described to people relate to

improvements that will be felt within 3-5 years; however, the survey left unstated when

household cost increases would begin. I did not see any question in the survey designed to elicit

time preference information nor did I see a question designed to elicit people’s beliefs about

when they would start paying these higher costs. Different people may be applying very different

discount rates to the household cost attribute.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 15

Jim Kahn

“I thought the survey booklets were informative without creating biases. The valuation problem

in this case is complex, and technical information must be provided on how environmental

impacts occur, what the environmental impacts are, and how policy actions can reduce the

impacts. The survey does an excellent job of this which is particularly important with respect to

the quality variables. The information in the survey booklet helps the respondent think about

how quality can be influenced by the proposed policy remedies without inserting a particular set

of values into the information. In particular, the treatment of option C, the status quo, was well

constructed in the survey compared to many choice-modeling exercises where the status quo is

ambiguous. Although the preliminary information was a bit on the long side, the researchers

compensated for this by only asking the respondents to evaluate three choice sets, rather than

five or six which is usually the case in choice-modeling. For this reason, it is unlikely that

respondent fatigue set in before the choice sets where completed.”

Dan Phaneuf

“I generally find the design of the survey to be adequate for the task. The systematic way in

which the study is able to directly map between policy objectives, the definition and levels of the

stated preference attributes, and household preferences is a clear strength of the study and

outweighs the concerns mentioned above. I find the description of the environmental problem to

be suitably neutral in tone. I do have several small negative comments regarding the design of

the survey booklet as well as the potential for strategic behavior, which I do not believe affect the

validity of the survey on net. I mention them mainly for completeness.”

“I also have a few observations on efforts to minimize hypothetical bias. On net I am inclined to

think that the importance reminders, general professionalism of the survey, and complexity of

the issue led people to perceive the survey as consequential. Two other points regarding

hypothetical bias relate to the payment vehicle. As best I could tell the survey did not contain

the usual reminder for people to consider their budget constraint, what they would give up, etc.

when answering the payment question. I think it would be useful to highlight some focus group

or cognitive interview evidence that people accepted this payment vehicle as realistic, and were

considering their budget constraints when answering.”

“Finally, I think the survey did an adequate job of including rationality checks. Questions 7 and

8 look like useful diagnostic vehicles, and might perhaps be useful for some estimation tasks.”

John Whitehead

“The overall quality and design of the survey booklet was “adequate” but some features of a

stated preference survey were not included. Respondents were told to “answer all questions as if

this were a real binding vote.” Respondents were told how the cost of the program would affect

their budget (in higher prices, etc.) but not reminded that this money could have been spent on

other things, which is a typical budget reminder. Respondents were not asked directly about the

consequentiality of their votes (i.e., did they think that their votes would be used by policy

makers; Carson and Groves, Env and Res Econ, 2007). Respondents were asked for confidence
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 16

in their answers and whether they would vote the same way in an election, but these responses

are not used in the empirical analysis.”

“A split-sample analysis of confident vs. not confident respondents and same way voters vs.

those who would vote another way would shed some light on this issue. If the split-sample

models are similar then the EPA’s approach is not problematic. If WTP has little validity for

respondents who are not confident about their responses or who would vote a different way, then

the EPA’s aggregation approach is overly optimistic.”

4. Does the experimental design allow for the estimation of robust values across the range

of regulatory policy options and the scope of the resource changes?

The panel agreed that the EPA should only focus on the “fish saved” attribute in the analysis as

this was the most robust outcome measure due to the overlapping definitions of attributes. The

panelists also largely agreed that the EPA could conduct more robustness checks through

additional statistical analyses (e.g., conditional logit), categorical variables and split samples to

increase the quality of the analysis. They also agreed that the analysis should focus on regional

rather than national results.

Kevin Boyle

“Yes, if and only if the policy just focuses on one attribute (fish saved) and the linearity of this

variable in the estimated equations is confirmed by more rigorous investigation. To investigate

the robustness of models to support value estimation for a one percent change I suggest

estimation where each attributes is identified using effects coding to investigate linearity instead

of the quadratic specification. The quadratic models seems odd at best given the small ranges of

levels posed for some attributes (4-6%) and the large jumps (coarseness) for the key fish saved

variable (report, p. 15, Table 3-1).”

Diane Dupont

“I am concerned that the scenarios presented have no instances of increases in some attributes

and decreases in others relative to the status quo. This may give rise to less robust estimates if

respondents find that scenarios do not fit with preconceptions. Furthermore, the design uses only

attribute main effects. According to Carson and Louviere Environmental and Resource

Economics, 2011), in such a case the researcher must assume that the indirect utility function is

strictly additive for all respondents. This worries me given my concern about the (possibly

varying) implicit relationships respondents may believe exist between the use and nonuse

attributes described in the choice tasks. Given the complexity of the problem, I would like to

have seen something other than a main effects design. Finally, within each regional (and

national) survey the same sample question is used for each booklet but then the first choice task

mostly contains different values from the sample values. Do people treat the sample as a base or

reference set of values against which they compare values in the subsequent questions? This may

induce a kind of loss aversion response. More generally, is there some kind of ordering or

comparison of values in earlier and later questions (since the ordering is fixed for each booklet

and not randomized across respondents).”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 17

Jim Kahn

“I am fairly confident in the robustness of the estimates for several reasons. First, the attributes

were constructed in a fashion where they could enter the logit estimation as continuous variables

rather than categorical variables. The continuous form of the attribute levels allows for a test

across policy options that would be much more difficult in the presence of categorically defined

variables. Nevertheless, I also think that the models should be estimated with categorical

variables as another robustness test and to search for nonlinearities. Second, for the most part, the

coefficients are very similar across the different regions and any significant differences can be

accounted for by region-specific characteristics in the type of fish. One critical aspect to assure

the validity of a choice-modeling study is to choose an appropriate set of attribute bundles to

compare in the choice sets. The EPA employed a fractional factorial experimental design,

eliminating those comparisons that might lead to protest responses or puzzlement. The definition

of the 72 alternative sets of comparisons follows the literature, and sample size was appropriate

for the number of choice sets evaluated. The 72 sets were appropriately distributed across the

alternative versions of the surveys in each region.”

Dan Phaneuf

“I think the experimental design is fine conditional on the decision to use three levels for the

important attributes. In this sense I believe it was the correct decision to focus on a main effects

design so as to avoid the need to ask respondents additional questions (a larger sample or more

responses per person would be needed to estimate interaction effects). The one point of criticism

I have relates to the use of only three attributes for fish saved, and perhaps for the other

ecological indicators. I would expect there to be a non-linear relationship in marginal utilities for

these outcomes. While three levels is in principle enough to trace a curve it can be difficult I

practice, as was shown by the effort to include a quadratic term in the models. I think this

feature of the design can be overcome with some additional analysis, as I discuss below.”

John Whitehead

“Yes, this is an important feature of a discrete choice experiment.”

5. Are the sample sizes and response rates for the regional and national surveys adequate

to produce statistically robust estimates?

The Panel agreed Sample sizes were consistent with experimental studies published in the peerreviewed literature, but are less than expected for a policy study of this magnitude. Members of

the panel were concerned about the representativeness of the national sample and recommends

that the EPA focus on the regional results. Among the regional samples, they were also

concerned about the Pacific sample as it has the smallest population and the largest number of

undeliverable surveys. Questions were also raised regarding the number of males who answered

the surveys compared to females.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 18

Kevin Boyle

“The sample sizes are acceptable minimums, but below what would be considered best practice

for a study of this magnitude to support a major policy decision. Sample sizes as employed in

this study are consistent with experimental studies published in the peer-reviewed literature, but

are less than one would expect for a major policy study. The response rates are consistent with

what is observed in the literature for best practice applications of choice modeling. The

nonresponse follow-up survey is consistent with best practice. I wish the authors has discussed

whether they believe the identified effects were relevant for value estimation; no intuitive

discussion was provided or insight.”

Diane Dupont

“The rationale behind the sample sizes is sound and the final response rates are just adequate.

However, I have some related concerns. It looks to me as if there is a problem with the use of

postal delivery addresses to find respondents since the Pacific region has the smallest population

but the biggest number of undeliverable surveys. This calls into question whether the Pacific

sample is less representative than the others. In terms of demographics, males are systematically

over-represented. This makes me think that there was some kind of sample selectivity that took

place since the survey was a mail survey. It is possible that women gave the survey to their male

partners thinking that it was about fish and of more interest to the males. This could be checked

by looking at whether female respondents in the sample were more likely to be living in one

person households. I would also like to see a breakdown of the responses to the question... ‘how

important is the protection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and nonresponse samples). This would assist in better understanding whether the samples provide robust

estimates. I find the Inland sample is problematic in that it contains very different sociodemographics from the non-response sample.”

Jim Kahn

“The implementation of the survey involved no shortcuts deviating from the original plan. The

sample sizes and response rates meet or exceed the guidelines established by the peer reviewed

literature and the federal government. Response rates were adequate and potential biases

associated with non-response were appropriately treated with the weighted model (see response

immediately below).”

Dan Phaneuf

“I found the arguments used to decide on the original sample convincing, and I think the number

of completed surveys received ex post is adequate for the statistical task. I don’t have further

comments or complaints on this aspect of the study.”

John Whitehead
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 19

“The regional sample sizes are adequate to obtain statistically significant coefficient estimates

especially with discrete choice experiment data where repeated measures are used in a mixed

logit. It is difficult not to produce statistically robust estimates with this sort of data and

empirical model.”

“It seems odd that more resources were not devoted to the national survey which has a sample

size too small to represent the national population. It is also too small to obtain WTP estimates

with confidence intervals tight enough to adequately compare to the weighted regional WTP

estimates.”

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the

regional and national data. For the calculation of implicit prices and monetization EPA

used the results of the weighted linear models across all the survey regions and national

analysis because of empirical and theoretical concerns. Is this approach appropriate

for estimating welfare changes associated with expected changes in survey attributes

under the 316b regulation? Are there other functional forms that EPA should

investigate that might be theoretically or empirically superior?

While members of the panel believe that the specification used by the EPA is strong and

convincing, they recommend that additional specifications also be estimated to increase the

robustness and validity of the main model results. With regard to the sample weights, most of

the panelists supported the effort to use the weighted model. However, the EPA could address

whether the weighted model should be used for benefit estimation and applied to all people in the

sample frames, or if the unweighted model should be used and applied only to the proportion of

the population consistent with the survey response rates.

Kevin Boyle

“I recommend discontinuing the use of the term RUM. While the model is based on random

utility, this is a total value study and RUM in the literature refers to a specific type of travel-cost

model that just estimates use values. Thus, I fear the use of the term RUM might cause some to

misconstrue the values that were actually estimated. Furthermore, I do not think the quadratic

specification is reasonable. As noted above, I suggest re-estimation where the levels of each

attribute are dummied out through effect coding to provide a more robust test of the linearity

assumption for the fish saved variable. As for the use of weighted models, I believe that this is

consistent with best practice when survey data do not match known population statistics. On

another note, not knowing how other variables are coded such as the ASC make me

uncomfortable fully supporting the interpretation placed on it.”

Diane Dupont

“I support the effort to use the weighted model results, particularly given the significant

demographic differences between the main sample and the non-response sample, as well as the

region-census level data. I think this is the appropriate approach to use for estimating the welfare
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 20

changes associated with the expected changed in survey attributes under the 316b regulation.

However, a simple CL model would also be informative. Given the concern for estimates across

the different regions, I think the authors ought to include some measure of distance decay in their

models because I am concerned about the likelihood of systematic differences in responses from

respondents living close to the facility points on the maps provided in the booklets. Furthermore,

the report notes that demographic interactions between ecological attributes and gender and

education were included in a preliminary Northeast model. I would like to see a similar model

with interactions using the Inland data set where most of the demographic variables differ from

the non-response sample. As the models currently stand, I am less certain that the estimated

WTP values for the Inland region are representative of average WTP since it is not obvious that

the sample is random. I would like to see a breakdown of some of the debriefing responses on

the importance of the different attributes by gender and education.”

Jim Kahn

“This is an important question, as empirical results are often dictated by the choice of functional

form. Given the non-response bias which, if uncorrected, could lead to biases, the weighted

linear model is the most appropriate functional form for calculating values associated with

marginal changes in the attribute levels. Even though the EPA is not using the values calculated

from the unweighted linear models, it was appropriate to show those values in the tables so the

reader could compare these values to the values based on the weighted regressions. It is certainly

appropriate not to calculate values based on the quadratic regressions, as the coefficients on the

quadratic term do not conform to economic theory. Furthermore, the choice of mixed logit

models to deal with heterogeneity in the sample is a well-justified choice. In many other

dimensions, the mixed model is less restrictive than traditional multinomial logit. Similarly, the

mix of random and fixed coefficients in the estimation represents valid choices given the data set

that was developed.”

Dan Phaneuf

“There are several things I might have liked to have seen as robustness checks, before

concluding that the weighted/linear/full covariance/MXL model should be preferred. First, I like

the mixed logit model and as such I agree with the decision to use it here. However, seeing how

other models perform would give me a better sense of what the mixed logit specification is

buying such as the simple conditional logit model. While it may be only personal opinion, I

don’t think it provides enough robustness checking to only present on class of model

(particularly one that might be prone to some numerical anomalies).”

“Second, I would like to see additional specifications. Looking at models with dummy variables

for different non-price attribute levels (rather than making them continuous) would help give a

sense of the types of non-linearity might be present. While such a specification would not meet

the policy needs directly it would help determine the extent to which the linear model is a good

approximation. It would also be a good way to test for internal scope consistency – i.e. the

magnitudes on the estimates of the dummy coefficients should increase. Also related to nonlinearity I think it would be worthwhile to look at a model that uses logs. This would allow for a

smoothly diminishing marginal utility of the 0 to 100 coded attributes, which I think is
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 21

important.”

“Finally, as I describe in the next question (with caveats), I found the use of sample weights

convincing. My advice would be to focus any additional estimation on the weighted sampled

generally, though for the conditional logits or other simple specifications I would present both.”

John Whitehead

“Using the weighted linear models seems appropriate and is adequately justified. I know of no

other functional forms that might outperform the linear and produce values with diminishing

marginal utility. But I wonder if this goal is worthy? It may be that there is no diminishing

marginal utility within the narrow range of policy impacts proposed by this survey for attributes

other than fish saved. And constant marginal utility is not unlikely for fish saved. Economic

theory does not have much to say about the demand that generates nonuse value.”

7. Is the nonresponse study design adequate to evaluate the potential for non-response

bias? Are the steps EPA used to correct for non-response bias adequate to address the

potential biases identified? Is EPA’s development of weighting values and their use

consistent with current best practices in the literature?

The panelists agreed that the non-response study followed best practice in accordance with the

literature; however, several commented that clarifications could be made in the documentation

regarding this part of the survey. Additionally, answers from the non-response survey could be

further broken down, analyzed and used in statistical analyses to increase the robustness of the

main results.

Kevin Boyle

“The nonresponse study follows best practice. I would like to have seen the report provide more

intuition of whether the significant effects identified really are expected to influence estimated

values. I would also like to have seen an investigation of the subjective decisions employed in

the development of weights for estimation.”

Diane Dupont

“Since the regional results are being compared to one another I think it is unfortunate that a

single weighting method could not be applied to each of the five data sets. However, I think the

EPA did the best that could be done with the information available. I would like to see further

discussion of the potential implications of these two different weighting schemes. I have two

questions about the decision made (on page 47) to exclude respondents from the weighted model

estimation if they do not include answers for all variables being used in the weighting. How

many respondents are removed from each region and are there systematic differences in

respondents that were removed when compared to respondents who were kept in the sample? A

final issue arises with the Pacific sample. There was a significant difference between the

importance assigned to ecosystems between the main sample and the non-response sample. But,
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 22

an adjustment was not undertaken for this. I understand the concern that this is likely to be an

endogenous variable however, this brings up a concern about possible sample selection bias.”

Jim Kahn

“This is an area in which I expressed some concern in the 2006 peer review, and I was glad to

see it so prominently addressed in the research, with a dedicated nonresponse study. The

methodology used to avoid the lazy husband problem completely negates this source of bias, and

it can be determined if cellphone only households, which have certain socio-demographic

characteristics, are underrepresented in the survey. I do not believe that this negatively

influenced the nonresponse study. With regard to philosophical self-selection, the non-response

study indicated that nonresponse was due to people who were less concerned about aquatic

ecosystems. I think it is important to see a breakdown of the responses to the question...‘how

important is the protection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and nonresponse samples). This would assist in better understanding whether the samples provide robust

estimates. Some patterns in nonresponses were noticed in relation to certain socio-demographic

variables, such as age, education, gender and employment were discovered in the analysis. These

formed these patterns and the potential biases that might be generated in the final value estimates

generated the need to conduct weighted logit regressions instead of unweighted estimates. The

methods that EPA used to create the weights corresponded to best practice in the literature, and

should result in estimates that are not biased by patterns of nonresponse.”

Dan Phaneuf

“I found the non-response analysis to be one of the strengths of the study. It is unusual even

among the highest quality studies to devote this much effort to analyzing the selection properties

of the estimation sample. The presence of some statistical differences aside, I found the nonresponse data encouraging in the extent to which it validated the general representativeness of

the primary sample. In short I find the data gathering for the purpose of supporting non-response

analysis exemplary.”

“I am less certain about the methods used to develop the actual weights. First, there is mention

of some observations needing to be dropped (because of missing data) in order to implement the

sub-group weighting routine. It would be useful to know if this involved more than just a few

data lines – i.e. are estimates from non-weighted models that do and do not include the to-bedropped observations the same? Second, the ‘raking method’ used for two of the regions seems

a bit speculative. The only citation to it is mention of a SAS macro that implements the process.

I think more needs to be said about the general performance and acceptability of this methods

vis-à-vis using a not-weighted model.”

John Whitehead

“The response rates are low (albeit, about as high as you can reasonably expect these days) and

raise the issue of nonresponse and sample selection bias. Nonresponse bias exists when the

sample is not representative on known characteristics. The EPA makes an honest effort to gather
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 23

information on respondents with an overall 25% response rate for the nonrespondent survey, but

still manages to obtain information on less than 10% (=619/7118) of nonrespondents. There is

evidence in the survey methods literature that shows that response rates are a function of survey

salience. The respondents, especially to a mail survey, are those who care most about the subject.

Those respondents who care most about the subject are likely to have higher WTP for the policy.

One approach is to estimate a Heckman type sample selection model with the samples of

respondents and nonrespondents with the dependent voting variable converted to a zero-one

variable for a probit. If there is no statistical evidence of sample selection bias then the EPA’s

aggregation approach may be appropriate.”

“In the absence of evidence to the contrary and given the possibility of sample selection bias, the

aggregation rule, assuming that weighted WTP responses adequately reflect nonrespondent WTP

is overly optimistic. It cannot be ruled out that nonrespondent WTP is less than respondent

WTP.”

8. Are the scope tests performed correctly and are the results interpreted appropriately?

Are there additional tests that could be conducted to further assess the validity of

responses?

The panelists did not find the external scope tests used by the EPA to be enlightening. They did

offer additional tests that could be conducted to assess validity such as pooling regional results

and examining split samples. Panelists suggested using focus group comments to address peerreview and industry concerns as well as reporting statistical tests for analyses conducted by EPA.

Kevin Boyle

“Choice experiments, by their design, provide a within subject test of scope. The scope issue in

the literature relates to between subject tests and the tests reported do not address this concern.

The scope test stratifying on choices A and B is neither appropriate nor insightful. The groupings

do not stratify so different people from the same sampling frame are responding to the alternative

percentages. Since the ASCs are not significant it is not surprising that the ASCs for alternatives

A and B are not significant. It is good to investigate if there is a leaning toward alternative A or

B. I think that the investigation of fish saved with the other attributes is appropriate. I would also

suggest estimating the model, always including fish saved and cost, but dropping other attributes

sequentially. Given the overlapping attribute definitions and exclusions of extreme combinations

suggests that responses to the fish saved attribute may be conditioned on the other attributes. I

know this is not the same as dropping attributes from the design, but it is one way to look at this

issue. I do not understand why some of these analyses were just conducted with the Northeast

data, and not all sample groupings.”

“In terms of the geographic scope showing willingness to pay to be higher for regional

improvements than for a national improvement, I find this is not a problem as some regions can

be higher than the national; the region vs. national is an apples and oranges comparison and

different baselines and preferences can cause this outcome as well.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 24

Diane Dupont

“The external scope test of looking at proportions of responses using split samples seems like a

reasonable way to examine the issue. However, no formal testing of these proportions is done

and, in some cases, the differences are very small and the standard deviations are very large. I

would like to see significance levels from non-parametric Chi-squared tests of differences in

proportions (where the proportions are people accepting a particular option). I would also like to

see the test done in a way that is sensitive to the question ordering. The only issue that I have

with respect to the argument that choice experiments provide their own internal scope test (and

so it is not necessary to do) is the following: if a respondent ignores certain attributes in his/her

choices, then this non-attendance may mean that internal scope tests for the ignored attributes are

not satisfied in a conventional sense.”

Jim Kahn

“First let me say that I thought scope tests were correctly performed, but I would like to point out

that scope tests are really a red herring and the scope issues are not as important as some portray

them to be. Accidently generating a scope issue through inadequate understanding of survey

questions and policy effects can be guarded against by appropriate use of pilot surveys,

individual debriefings and focus groups.”

“It is appropriate to test for scope issues, even it is expected that they are not likely. EPA deals

with the issue directly with several tests for the problem. The first test is the natural test that

coefficients on attribute levels have a positive slope, indicating that people are willing to pay

more for higher levels of the attribute. They were not able to demonstrate declining levels of

marginal value with the quadratic estimations, but this is not a real problem. The “external

validity” test of scope that they initiated was cleverly done, splitting the sample into option a

responses and option b responses, and showing that for both option a and option b, the choice set

was more likely to be chosen if it higher levels of the attributes.

The geographic scope test showed willingness to pay to be higher for regional improvements

than for a national improvement, although economic theory would suggest that the national

improvements should have a higher value than the regional improvements. However, this does

not surprise me, as people are more focused on improvements in their own vicinity, and this is

not a global public good in the sense of reducing global warming or species extinction. These

tests were appropriate and well-executed. I do not see the need to conduct additional tests.”

Dan Phaneuf

“As mentioned earlier I would like to see an additional variant on the internal scope test that

makes use of dummy variables for the different non-price attribute levels. This will help make

more transparent issues of scope and the non-linear relationship between the attribute indices and

utility. Specifically on the ‘external’ scope tests I think more can be done to formalize them.

The idea seems like a good one, or at least something intuitive to look at. However, the

summary statistics that are reported do not provide a way of formally or even informally

assessing the statistical presence of scope. I think it would be useful to run a linear probability
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 25

model on the Option A and Option B sub-samples, including dummy variables for each of the

attribute levels outside of price (this could stay continuous), or perhaps keeping the other

attributes continuous and just checking the different levels of fish saved. This would allow a

formal test of whether the coefficients on the three fish saved levels are different and properly

ordered.”

John Whitehead

“The external scope tests are not adequate to demonstrate responsiveness to scope, at least

according to the external scope test standard set by CVM critics. The EPA report performs

internal validity tests using multiple scenarios. This test is relatively weak since respondents

need only be consistent within themselves to produce WTP values that increase with policy

impacts. An external validity test compares responsiveness to scope with split sample survey

design.”

“Within a discrete choice experiment an external validity test can be conducted by using only the

first scenario presented to respondents in the RUMs. In effect, the first scenario is a split sample

test. The EPA considers the split-sample data in Table 8-10 but only applies the eyeball test.

Differences in frequencies across the fish saved scenarios can be tested with a likelihood ratio

test. The external validity test should be conducted with the RUM model (e.g., Lew and Wallmo,

Env and Res Econ, 2011). Note also that in a mail survey the first response may be contaminated

by the following scenarios since respondents could have looked ahead and developed internally

consistent answers during the survey.”

9. What are your expectations regarding the validity and reliability of the benefit

estimates that are provided by this survey, compared to the current stated preference

literature? Would you expect the results provided by the survey to be superior to those

that EPA prepared as part of the national benefits assessment for the proposed 316(b)

existing facilities rule? Would you recommend that the information provided by this

survey be used in EPA's cost benefit analysis of the 316(b) rule?

While the panel did find issues of concern regarding the study, they agreed that “no study is

perfect” and that the current study is superior to many existing studies. Panelists suggested that

the survey content be further developed in the report and that emphasis be placed on regional

results. Once additional robustness checks and sensitivity analyses have been conducted, they

recommend that the survey be used in the EPA’s cost benefit analysis of the 316(b) rule.

Kevin Boyle

“The design practices are better than what is done in most current studies in the stated preference

literature and the data analyses follow standard practice. I am concerned that the survey content

is not as well developed as I would have liked and, as presented, might lead to respondent

confusions and certain items may have led respondents to value more than the scenarios were

intended to value. The current study is superior to the existing benefit estimates, and I would

recommend the current study for national policy analysis. No study is perfect and any benefit

analysis must take into consideration potential imperfections in benefit estimation. In the current
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 26

analysis the robustness analyses of the survey data analysis are appropriate and can be expanded

as noted above. There needs to be more sensitivity analysis in computing the aggregate benefit

estimates.”

Diane Dupont

“Compared to the current stated preference literature I believe that the data collected within this

survey is capable of providing both valid and reliable estimates for benefits. I place more

confidence in results from the regional models/datasets (with the exception of those for the

Pacific Region) than from results associated with the National dataset, in part because of the

saliency of the issues within a regional context and in part because of the larger (relatively

speaking) sample sizes used in the regions. I use the word “capable” in the first sentence because

I think the research effort would gain important insights after incorporating distance decay

effects, ordering and context effects, and further investigation of the role of differences in

demographics, particularly insofar as these differences manifest themselves in expressed

opinions on the debriefing questions. I have confidence that the information obtained from the

survey can be used to assist in EPA’s CBA analysis of the 316(b) rule. I would recommend

undertaking some additional model estimation along the lines noted above in order to verify that

validity of the estimates already provided.”

Jim Kahn

“This question is the most important of the entire peer review of the survey and resulting values.

I am confident that the research was properly done and conforms to the highest scientific

standards. It is not just acceptable, but actually provides a blueprint for others to follow when

conducting their own research. The survey provides important information that was not in the

original benefits assessment. Given the complexity of the environmental variables, the likelihood

that different responses will be forthcoming from different regions and the availability of

resources to conduct a state of the art survey, it is better to use measure the values directly than

to use benefit transfer. Moreover, I like the way the estimates of the value of fish saved can be

used as a lower bound on non-use values. This is a significant addition to the analysis. Since the

survey and analysis were conducted with the appropriate adherence to theoretical and empirical

precedents, I would enthusiastically endorse using these values in a cost benefit analysis of the

316(b) rule.”

Dan Phaneuf

“I find that the design work and data gathering aspects of the study generally meet the necessary

conditions for the resulting data to be valid and reliable. In addition, the raw data summaries (on

p. 21) show that respondents did seem to make tradeoffs between money and the non-price

attributes. However, I find that the current set of reported results alone are not sufficient to judge

if the value predictions contained in the peer review document are themselves of high quality.

Thus I see the study as having the potential to be superior to that which was used as part of the

existing facilities rule, conditional on some robustness checking and additional analysis. As such

I would hold back from making a firm recommendation that this survey be used to inform the

316(b) rule cost benefits analysis, but my expectation is that I would eventually be comfortable
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 27

making such a recommendation. Furthermore, my tendency would be to rely on the regional

models for policy purposes.”

John Whitehead

“These estimates are superior to those used in the 2011 report since they cover the entire nation

and are based on a primary valuation study. However, in spite of relatively low statistical

uncertainty, the WTP estimates are highly uncertain for reasons described above. First, it is not

clear if roughly 25% of the sample has valid nonuse values (see #1). Second it is not clear if

roughly 67% of the population represented by nonrespondents have WTP values equal to those

of the respondents (see #7). If both of these groups of the populations have WTP equal to zero

then the WTP estimates could be as low as 25% of those reported (=100*.75*.33).

Unless evidence can be provided that all of the respondents have valid nonuse values (#2) and

nonrespondents have similar WTP values to respondents (#7), I would not recommend that the

information provided by this survey be used in EPA’s benefit-cost analysis. The aggregate

benefit estimates in Table 10-3 could be used if they are described as the best case analysis

(instead of the base case) and sensitivity analysis be performed over the potential range of

values. Given the current information, a uniform distribution should be used for the range of

values.”

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed

the potential implications of uncertainties in the analysis, and are the effects on the

conclusions drawn presented in a transparent manner?

The main suggestion of the panel was to include an additional section in the report that

summarized all the uncertainties on the over and underestimation of WTP and direction of

potential biases. This also includes addressing potential respondent confusion of attribute

descriptions in the survey. It is important that uncertainties are understood before results can be

used.

Kevin Boyle

“See my comments above.”

Diane Dupont

“I think the report contains a great deal of information in different locations. The footnotes, in

particular, are very helpful to further understanding of decisions made along the way. However, I

think the report should collect all various types of uncertainties and potential implications of

uncertainties in the analysis and put them together in a single table, so that directions of potential

biases could be examined and clarified. Many steps were undertaken and many decisions were

made prior to the generation of the WTP estimates in Table 10.2. The inclusion of a table

showing the types of decisions that had to be made with respect to uncertainties in modeling,

parameter values, model results, estimated WTP values and likely directions of biases would
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 28

help the reader to better understand the implications of the uncertainties.”

Jim Kahn

“Uncertainty is omnipresent in empirical research and must be understood before the results can

be used. It is important to discuss likely sources of uncertainty, the magnitude of the uncertainty

from these sources, and the implications of the uncertainty for basing decisions on the results of

the research. In all areas, I think EPA does a good job. The first source of uncertainty is whether

survey respondents really understand the survey which is a particular concern in the 316b survey,

because of the complexity of the environmental changes that are being examined. The EPA

addresses this uncertainty in two fashions: focus groups and the use of particular questions in the

survey itself.”

“I am not going to summarize the 17 pages of discussion in this memo, but comment on a few of

the things I thought most important. First, going in, I was a bit nervous about the “fish saved”

variable and if people could really wrap their minds around this concept. I thought that it was

particularly important that “fish saved” was well understood, because of the four environmental

attributes, fish saved was the one that was most directly affected by the 316b policies. If the

variable was well-understood, then the willingness to pay estimates for changes in this attribute

could serve as a lower bound on non-use values. After reading the discussion of the

understanding of the variable by the respondents, I feel that they did understand it well enough

that it could serve as a basis of policy analysis. Second, I was concerned about the possible

existence of warm glow effects. Again, the discussion of those participating in the focus groups

convinced me that they are responding because of the anticipated environmental changes, and

not because of an overall desire to do good. This, coupled with their consideration of budget

constraints, leads to a conclusion that a lack of understanding of the surveys or the policies is not

a significant source of uncertainty in the final results. The survey itself contains a question (#8)

which addresses some of the uncertainty issues. The responses to this question are contained in

Table 6.3. In my opinion, the uncertainty is at a level that it does not interfere with the ability to

use the results in policy analysis.”

Dan Phaneuf

“Continuing my point above, I think there are remaining uncertainties as to what other

specifications would predict and how robust the findings might be to modeling decisions. Thus I

think there are some uncertainties in the analysis whose effects on the conclusions are not

transparent. To this I will add that the treatment of the ASC in predicting welfare effects for

non-marginal changes in the attributes needs some additional thought. There is a line in the

document about ignoring it since it was not significantly different from zero. I think there were

cases when the ASC was significant, and in any case I think it needs to be a part of the KrinskyRobb bootstrapping procedure – i.e. the distribution of values should enter into the welfare

calculations in the same way as the other parameters (some of which are also insignificant) do.”

“In other dimensions I think EPA has done an admirable job of documenting uncertainties. The

quantification of uncertainty arising both from random parameters as well as sampling error is a

useful way to illustrate the range of possible predictions.”
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 29

John Whitehead

“Uncertainties in the analysis are not fully described or considered. Key uncertainties include

validity of nonuse values over the entire sample of respondents and potential differences between

respondent and nonrespondent WTP. A number of split sample analyses are necessary to provide

information for potential sensitivity analyses. Without these it may be necessary to be more

conservative with aggregate benefit estimates (see #9). Another approach is to estimate latent

class models which are able to estimate different (probabilistic) groups of responses. With data

such as these, without information on prior information for the respondents, a latent class model

could estimate two or more vectors of coefficients for the choice models with probabilities that

any respondent belongs to each group. It is possible that one or more vectors reveals the

probability that latent groups of respondents have invalid WTP estimates (i.e., statistically

insignificant cost and benefit coefficients). Haab et al. (Marine Res Econ, forthcoming) estimate

conditional logit, mixed logit and latent class models and produce a range of WTP estimates.

They suggest that this range could be used to provide a range of WTP estimates for sensitivity

analyses. A similar approach could be taken here.”

SPECIFIC COMMENTS ON REPORT

Kevin Boyle

“For the considerable effort invested in the design of the survey booklet, I found the wording in

the survey rough and confusing in places. I do not know how this affected the survey outcome in

all instances, but it does leave unresolved questions. Some examples of these concerns are (all

based on Northeast survey):

• P. 1st line – reference to habitat is not clear. Fish habitat or all aquatic habitat, or aquatic

and terrestrial habitat?

• P.1 – you do not show diagrammatically human effects on the natural food web. Not

critical perhaps, but would have been a good inclusion to close the information loop.

• P. 2, diagram – last bullet refers to return water, but this is not sown in the diagram so

seem out of place and the diagram not fully developed to accompany the text.

• P. 2 map – I am concerned that the map does not coincide with the 93% figure most are

on inland waters. I think the coastal sites should have been highlighted. This figure is

repeated again just before the choice questions. I am concerned that responses to the

choice questions could have been driven by the large number and spatially distributed

cooling water sites and this led to overestimation. In this light it is good that the ASCs are

insignificant, but this does not address effects on the attribute coefficient estimates.

• P. 3, 2nd para. – refer to recreational fishing, but do not explicitly exclude later and only

reference commercial fishing in the attribute description. May have caused respondent

confusion.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 30

• P. 3, 3rd para. – not clear if the species listed at the end are forage species of species that

eat the forage species.

• Q. 1 – since this is the first time many respondents may have heard of this issue I am

surprised that you did not allow for a “don’t know” response option.

• P. 4, 1st para. – use of “equivalent” makes this long sentence awkward and potentially

confusing.

• P. 4, last sentence – confusing because you refer to multiple causes of losses in the

preceding paragraph, but I think you are just referring to cooling losses in this sentence.

• P. 5, current situation diagram – line is confusing with 0% losses because a line means

<0.1% losses on the previous page.

• P. 5, 3rd para. – wonder if wording here makes people think that the policy also

conserves water. Here is very good that the ASC’s are not significant, providing evidence

that this is not a problem.

• P. 5, bottom – wording really makes it sound like policy values 5% or 95% changes, and

does not explain that the relative mix of both can lead to many different combine

percentages.

• P. 6, Q. 6 – placement of question is strong given that preceding text discussed fish loses

and there was no transition to habitat here.

 P. 6, bottom – not choosing A or B does not mean a respondent supports the status quo, it

simply means that A and B are not sufficiently better, given the cost, to choose one.

• P. 7, top – You only reference protecting fish and do not introduce the habitat attribute

here.

• P. 7 – commercial fish is consistent with what you intend, but do people know that the

vast majority of commercial fish come from saltwater? If they do not, map issue I raised

earlier continues to be a concern. It also seems odd to have the attribute label be

commercial fish, but the description includes recreational fishing.

• P. 7 – scale here seems to totally overlook over-harvesting of commercial species in the

Northeast. This is an issue that has received a lot of coverage in the press and I wonder if

this influenced how people responded to this attribute in the absence of a clear survey

explanation.

• P. 7 – attributes overlap and this is not consistent with the standard design of choice

experiments. Thus, the standard additive of implicit prices to estimate consumer surplus

as stated in equation 7 on page 65 of the report is applicable for this study. However,

given that policy only looks at on attribute, this is not a concern for how the estimation

results are used to support policy decision.

• P. 7 -- the wording of the fish saved attribute is awkward and potentially confusing

(100=0 lost). I wonder if the significant standard deviations on the fish saved attribute in

the estimated equations is due to respondent differences due to different levels of

confusion or heterogeneity in preferences?

• P. 10, 4th bullet – I wonder if the monthly leveling of costs contributed to scenario being

unrealistic since respondents know that electricity costs in the Northeast are highly

seasonal and may think the costs of this program will be closely linked to electricity

costs.

• P. 10, 7th bullet – introduce “environment” here and just leave it hanging, while you do

follow-up on the “economy”.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 31

• P15-16, Q. 8-9 – responses to these questions can be suspect because reasons could vary

by choice question and you force overall responses.”

“Some of these points I raised might be answered by knowledge gained from the focus groups

and cognitive interviews that is not easily conveyed in the report. However, the roughness of the

descriptions and overlapping categories is something that I would expect to have been refined

and resolve in such an extensive design process.”

Diane Dupont

• “In the notes from the focus groups, the following comment is made...

(On page 9 of the Executive Summary...) “Participants generally appeared to have

some understanding of how the ecological scores interact and this influenced their

responses. ... Some participants’ responses indicated that the choice questions

would have been more difficult if some effects were to go up and others down

within the same policy option. (#5-13).

Given that focus group respondents commented on this possibility suggests that

respondents may believe in a different type of implicit relationship between the various

attributes than the one presented. Why is this important? It may be that, when they see

ecosystem values rise in a particular choice task, they may infer that this will have some

additional favourable impact upon another attribute of interest (commercial fishing, for

example). So, they infer a relationship between the attributes and not treat them as

separate. I am concerned that this might result in some kind of double counting in the

Total WTP, in particular, when people assume a very strong complementary relationship

between use (eg. Commercial and recreation fish populations) and non-use components

(e.g., all fish populations that help to provide feed for commercial populations and

aquatic ecosystems that provide habitat).

• (On page 15 of the Executive Summary...from a respondent...“But see everything else

follows the conditions of aquatic ecosystem. [...] the commercial fish population hinges

upon the condition of aquatic ecosystem, as does the fish population [...]” (#3-15).

I know that the Johnston et al. (2011, 2012) papers from which the EPA survey draws

much inspiration employ a similar choice experiment setup that combines both use and

nonuse components. Both the EPA survey and the Johnston et al survey leave unspecified

the relationship between environment and the components that make it up. I note,

however, that Johnston et al employ a main effects with two-way interactions task choice

design that may help to mitigate this problem. So, my mind is still not made up on how to

interpret people’s responses. What underlying household production function drives

people’s choices? Does it matter if some people see the different attributes as being either

independent or substitutes and others see them as complements?
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 32

• In the discussion on page 7 that describes the attributes, the score for the ecological

condition is defined between 0 and 100 % when compared to the most natural water in

the [..] region. And,... “higher scores mean the area is more natural.” However, on the

actual choice task pages the upper bound of 100% is defined as being in pristine

condition. To my mind “pristine” is not necessarily natural and may be a somewhat

value-laden word. I didn’t see any discussion of this in the focus group report or in the

actual report. Whether this type of wording might lead to a strategic bias is something

that could be considered. I note that some debriefing questions were included but I do not

see them being used to assist in understanding responses.

 I have a couple of concerns regarding information provided. In the InlandVs5b version,

on page 4 the reader is told that about 0.6 billion young adult fish are lost each year in

Inland waters due to cooling water use. Then, it says that more [my emphasis on the

more] than 0.6 billion fish could be saved. But, this is greater than what people have been

told is lost. Then, on the next page, respondents are told that the maximum gain in fish

saved is is 95 % - this seems inconsistent with the previous information. I checked the

NE/SE/National/Pacific versions and there is not the same mismatch between what

scientists estimate are the losses due to cooling and what the maximum gains from the

programs might be. In each of the other cases, the fish saved is clearly smaller than the

original amount being lost.

 The report says...“Because the Priority Mail response was lower than expected, the target

number of telephone completed surveys was increased to obtain the desired number of

responses.” Does this observation call into question the validity of the original mail

approach as leading to some kind of selection bias (relative to telephone responses at any

rate)?

 On page 66 the report states that the ASC is not included in the calculation of

compensating surplus because it was found to be insignificant in the model. There is

some discussion in the literature on the merits of including/excluding ASC values.

However, it is widely accepted that inclusion of the ASC provides more conservative

estimates since the exercise involves measure utility gains/losses of a program away from

one’s status quo situation. Even though the coefficient itself may be insignificantly

different from zero, its level is important for inclusion in calculations. I should also note

that the ASC captures the effects on utility of any other attributes not included in the

choice task that may be important to the respondent. Finally, I note that this is not a

problem for the annual household WTP estimates provided in the report for the

regulatory options since these are calculated using only the marginal WTP for a one unit

change in fish saved.”

John Whitehead

• “Page 8: “Appropriately designed choice-based SP methods may also reduce strategic,
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 33

hypothetical and other possible biases that can result from asking survey questions versus

assessing WTP through market transactions or binding referenda.” During the CVM

debate discrete choice experiments (i.e., “choice-based SP methods”) were introduced.

The mood amongst many environmental economists was that choice experiments were

preferred for reasons not related to the problems of the CVM (which are mentioned in the

sentence above) such as consistency with random utility theory, an interest in new and

advance econometric methods and publishability. As such, it was asserted that choice

experiments were a silver bullet that solved many problems associated with the CVM. I

have not seen much comparative research to back up this assertion. For example, the next

paragraph describes the extensive research conducted on hypothetical bias in the CVM.

No reference is made to studies assessing the potential hypothetical bias that may result

from stated preference choice experiments. Unless there are a number of studies that can

be referenced here to back up the assertion, this sentence should be removed.

• Page 9: I’m not sure what “older forms of SP analysis” refers to. Is this the CVM? It

might be better to go ahead and state that the comparison is being made to the older

forms of the CVM. But why is this comparison appropriate? The EPA should be

comparing discrete choice experiments to newer forms of CVM that are not “typically

contingent upon a single specification of ecological effects.” It is straightforward to

design CVM surveys to ask multiple WTP questions (e.g., Christie and Azevedo, J of Agr

Econ, 2008). If the EPA feels that it is more efficient to estimate the WTP for

characteristics of a policy with discrete choice experiments relative to the multiple

question CVM then some supporting references should be provided.

• Page 13: “an unharvested population is typically believed to be approximately three

times as large as MSY.” Please provide a reference.

• Page 21: Why was a mail survey chosen over an internet survey with all of the advances

in internet sampling that have occurred over the past 10 years or so? With all of the

mailings it seems that even Knowledge Networks would have been less expensive and

more effective. Note that Lew and Wallmo (Env and Res Econ, 2011) used Knowledge

Networks for a federal agency survey.

• Page 21: Was the Pacific response rate 32% or 40% = 311/(1040-255)? Was the Inland

response rate 35% or 33% = 787/(2480-65)? Why was there such a lower rate of

undeliverable surveys for the inland sample (2%) compared to the others (7% - 25%)?

• Page 22: 46% plus seems like a high rate of recreational seafood consumption. Are there

any representative sample statistics with which to compare?

• Pages 27-42: Reading section 7 gets tedious with the cut and pasting. Some of this could

go into an appendix with a summary in the text.

• Page 51: Much of the text above Table 8-5 should be relegated to a footnote. It really

doesn’t matter if this study has similar pseudo-R2s as other studies.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 34

• Page 55: “Unlike open-ended contingent valuation questions, choice experiments provide

a direct mechanism for respondents to react to scope and scale of resource changes ....”

Why is open-ended CVM the relevant comparison? Open-ended questions have not been

the norm since Bishop and Heberlein (AJAE, 1979).

• Page 66: What is the WTP per fish saved? How does it compare to other studies?

• Table 9-2: The regional implicit prices from the weighted linear model are in a different

order (NE, SE, IN, PA) compared to Table 9-1 (NE, SE, PA, IN).

• Page 68: The point estimate of the weighted regional implicit WTP, $1.28, is within the

90% confidence interval of the national fish saved WTP estimate so there is no reason to

discuss any difference in point estimates.

• Page 69: The AIE acronym is used before it is introduced as an acronym in this

paragraph.

• Table 10-1: How is the “reduction in AIE losses” column measured? Is this millions of

fish saved?”

IV. ROADMAP FOR THE EPA

Through the Applied Planning Corporation approach, the peer review panel came up with a road

map for the EPA to strengthen the report. The road map is outlined below.

Estimation Robustness

The panel found that the basic estimation of the report is fine and is largely consistent with the

literature. However, several items can be enhanced:

 The investigation of nonlinearities using a quadratic term is not adequate. Nonlinearities

can be better investigated with attribute levels coded as binary variables and possibly

using effects coding.

 Given the overlapping attribute definitions, it would be beneficial to look for attribute

interaction effects and the effects of dropping attributes from the estimation. The former

is important because the attribute design excluded attribute level combinations that

potentially did not make sense.

 Mixed logit models are based on distribution assumptions made by the investigators as

clearly noted in the report. While the distribution assumptions imposed make sense, it

would be useful to see how different assumptions would affect the estimation results.

 Estimating a simple conditional logit model to be used as a baseline for model

comparisons.

 Estimating multinomial logit models will help to consider the robustness of the mixedlogit assumptions.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 35

 Estimating logarithmic transformation of the models may also be useful.

These analyses can largely be reported in an Appendix to the report and key findings

summarized in the text of the report explaining how different specifications lead to

different/similar aggregate welfare predictions. The main report would contain the preferred

estimation.

Sample Selection and Non-Response Study

The design of the sample selection investigation is appropriate and is consistent with current

survey standards. However, the analyses that follow are not performed to the same standard as

the design.

 Discuss why variables used in the sample-selection analyses are relevant, i.e., would

influence WTP.

 Provide a breakdown of the responses to specific questions such as ‘how important is the

protection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and

the non-response samples). These variables may not have been included in the

nonresponse follow-up surveys, but there are a number that logically are correlated with

WTP, perhaps more so than the variables in the nonresponse follow-up surveys.

 Conduct statistical tests of the effects of sample selection.

 Conduct robustness checks of assumptions imposed in the sample selection process, e.g.,

conducting analyses based on different sets of variables and different recordings of

variables.

The larger question not really addressed in the current report is whether the weighted model

should be used for benefit estimation and applied to all people in the sample frames, or if the

unweighted model should be used and applied only to the proportion of the population consistent

with the survey response rates. Both approaches have been used in the literature. The latter may

provide and absolute lower bound welfare estimate; therefore, if benefits exceed costs then the

EPA is confident that they really do. The former is likely an overestimate because it may include

people who have zero values (WTP=$0) and this is not addressed in the EPA’s statistical

analyses.

Validity Checks

The validity checks conducted are appropriate, with the exception of the scope investigation, but

they do not go far enough.

 As with sample selection, report statistical tests for the analyses that the EPA conducts

rather than only reporting summary statistics for data splits.

 EPA has a number of very good probing questions after the choice questions and the

responses to these questions should be used in the validity analyses. For example,

questions pertaining to whether a respondent fishes could be used to identify use versus

non-use motivations.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 36

The panel recommends that the EPA use these analyses and information from the focus groups,

to the extent possible, to try to address some of the peer-review comments and industry

comments.

Discussion of Over/Under Estimation of WTP

There are numerous elements of the survey design, data analyses and computation of aggregate

benefits that can lead to over or under estimation of benefits and the current draft of the report is

largely silent on these potential effects. Some of these are analysis and aggregation issues that

are mentioned above. Examples of potential survey issues are:

 Introduction of thermal discharge effects, but not explicitly addressed in the choice

question.

 Potential respondent confusion of attribute descriptions.

 Map in survey may give impression to respondents that they are located near affected

areas and the effects are widespread geographically, but wording in survey does not

support such a conclusion. Wording is not as explicit as the map is visually explicit.

The panel suggests adding a new section in the report that summarizes and integrates the EPA’s

uncertainty analyses in order to clarify the directions of potential biases. It would also be useful

to look at split sample analysis with the respondents who indicated they were at the top of the

‘confident in their answers/understood’ scale versus those who were not. It is doubtful that the

EPA will be able to clearly say that WTP is over or under estimated, but a logical argument can

be made that the EPA has provided the best possible estimate using state-of-the-art procedures.

Reporting and Documentation

The report is largely well written. However, weaknesses that need to be improved include

justifying procedures used and finishing discussions to the implications of these procedures for

estimated WTP. The procedures used a generally well described, but the motivation for their use

and the consequences of their use are largely silent in the report.

V. CONCLUSION

This report summarizes the comments of a peer review panel evaluating a stated preference

choice experiment on rules for Section 316(b) of the Clean Water Act (33 U.S.C. 1251 et seq).

Section 316(b) establishes national performance requirements that are designed to minimize

adverse environmental impacts caused by cooling water intake structures (CWISs). The choice

experiment was conducted to further investigate potential policy options stemming from these

rules and the willingness-to-pay of consumers on ecological attributes such as “fish saved”.

This peer review is intended to assess the work that EPA has completed with regard to this rule

as well as the results of EPA’s 316(b) stated preference research. The peer reviewers were asked

to identify areas where modifications to the EPA’s estimation and validation approaches would

improve the accuracy and precision of the resulting willingness-to-pay estimates generated from

the models developed using the survey data.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 37

Overall, the panel felt that the survey design followed best practice. Through the review process

the panel identified several issues of importance that they strongly encourage the EPA to address

prior to releasing the report. They have provided the EPA with a roadmap to address their

concerns regarding documentation, sample selection, robustness, validity and

over/underestimation of WTP. Further specifications analyses recommended in these report will

ensure that the results are robust both conceptually and empirically.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 38

APPENDIX I

REVIEWER COMMENTS
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 39

PEER REVIEW COMMENTS

Kevin Boyle
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 40

316(b) Peer Review

Kevin J. Boyle

Virginia Tech

This study uses design practices that are better than is standard in the literature and the data

analyses follow standard practice. As with any study, in hind sight I have identified components that

could have been designed better. These issues can not addressed at this point, but potential concerns

need to be considered in the computation of aggregate benefits. There are elements of the

econometric analysis and documentation that need to be addressed and can still be improved before

the final computation of benefits.

1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay

for the characterized policies, including use and nonuse values?

Yes, this approach is in common use in the economics literature for estimating uses and nonuse

values for changes in natural resources. Applications have been published in the peer-reviewed

literature, and there is a substantial literature investigating the validity of choice experiments (also

known as choice modeling, choice questions, attribute based questions, conjoint analysis, and other

labels).

2. Were the procedures EPA used to develop and test the proposed survey instrument consistent with

current best practices for stated preference surveys, as reflected in the peer-reviewed literature?

Using focus groups, cognitive interviews and pilot studies is consistent with the very best practice for

designing survey instruments. The investigators conducted more focus groups and cognitive

interviews than is typical; the work here is consistent with the number conducted for high-profile

litigation applications. The report mentions the pilot study, but provides no explanation comparable

to the discussion of the focus groups and cognitive interviews. It is unclear if the Northeast survey

implementation is the pilot or if another pilot was also conducted. The authors mention that the

Northeast survey was implemented to inform the other survey applications, but no mention is made of

the specific goals of this effort or if the other surveys were changed from what was learned from the

prior implementation of the Northeast survey instrument. These latter comments are not intended to

be a critique of the survey design process; rather they are a comment of a lack of clarity in the

documentation contained in the report.

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the

information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate

emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize

hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to

allow valid welfare estimation?

For the considerable effort invested in the design of the survey booklet, I found the wording in the

survey rough and confusing in places. I do not know how this affected the survey outcome in all

instances, but it does leave unresolved questions. Some examples of these concerns are (all based on

Northeast survey):

• P. 1st line – reference to habitat is not clear. Fish habitat or all aquatic habitat, or aquatic and

terrestrial habitat?
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 41

• P.1 – you do not show diagrammatically human effects on the natural food web. Not critical

perhaps, but would have been a good inclusion to close the information loop.

• P. 2, diagram – last bullet refers to return water, but this is not sown in the diagram so seem out of

place and the diagram not fully developed to accompany the text.

• P. 2 map – I am concerned that the map does not coincide with the 93% figure most are on inland

waters. I think the coastal sites should have been highlighted. This figure is repeated again just

before the choice questions. I am concerned that responses to the choice questions could have been

driven by the large number and spatially distributed cooling water sites and this led to

overestimation. In this light it is good that the ASCs are insignificant, but this does not address

effects on the attribute coefficient estimates.

• P. 3, 2nd para. – refer to recreational fishing, but do not explicitly exclude later and only reference

commercial fishing in the attribute description. May have caused respondent confusion.

• P. 3, 3rd para. – not clear if the species listed at the end are forage species of species that eat the

forage species.

• Q. 1 – since this is the first time many respondents may have heard of this issue I am surprised that

you did not allow for a “don’t know” response option.

• P. 4, 1st para. – use of “equivalent” makes this long sentence awkward and potentially confusing.

• P. 4, last sentence – confusing because you refer to multiple causes of losses in the preceding

paragraph, but I think you are just referring to cooling losses in this sentence.

• P. 5, current situation diagram – line is confusing with 0% losses because a line means <0.1%

losses on the previous page.

• P. 5, 3rd para. – wonder if wording here makes people think that the policy also conserves water.

Here is very good that the ASC’s are not significant, providing evidence that this is not a problem.

• P. 5, bottom – wording really makes it sound like policy values 5% or 95% changes, and does not

explain that the relative mix of both can lead to many different combine percentages.

• P. 6, Q. 6 – placement of question is strong given that preceding text discussed fish loses and there

was no transition to habitat here.

 P. 6, bottom – not choosing A or B does not mean a respondent supports the status quo, it simply

means that A and B are not sufficiently better, given the cost, to choose one.

• P. 7, top – You only reference protecting fish and do not introduce the habitat attribute here.

• P. 7 – commercial fish is consistent with what you intend, but do people know that the vast majority

of commercial fish come from saltwater? If they do not, map issue I raised earlier continues to be a

concern. It also seems odd to have the attribute label be commercial fish, but the description includes

recreational fishing.

• P. 7 – scale here seems to totally overlook over-harvesting of commercial species in the Northeast.

This is an issue that has received a lot of coverage in the press and I wonder if this influenced how

people responded to this attribute in the absence of a clear survey explanation.

• P. 7 – attributes overlap and this is not consistent with the standard design of choice experiments.

Thus, the standard additive of implicit prices to estimate consumer surplus as stated in equation 7 on

page 65 of the report is applicable for this study. However, given that policy only looks at on

attribute, this is not a concern for how the estimation results are used to support policy decision.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 42

• P. 7 -- the wording of the fish saved attribute is awkward and potentially confusing (100=0 lost). I

wonder if the significant standard deviations on the fish saved attribute in the estimated equations is

due to respondent differences due to different levels of confusion or heterogeneity in preferences?

• P. 10, 4th bullet – I wonder if the monthly leveling of costs contributed to scenario being unrealistic

since respondents know that electricity costs in the Northeast are highly seasonal and may think the

costs of this program will be closely linked to electricity costs.

• P. 10, 7th bullet – introduce “environment” here and just leave it hanging, while you do follow-up

on the “economy”.

• P15-16, Q. 8-9 – responses to these questions can be suspect because reasons could vary by choice

question and you force overall responses.

Some of these points I raised might be answered by knowledge gained from the focus groups and

cognitive interviews that is not easily conveyed in the report. However, the roughness of the

descriptions and overlapping categories is something that I would expect to have been refined and

resolve in such an extensive design process.

I did not see a clear budget reminder in the survey wording, but respondents were told that the price

increase was “unavoidable” (Northeast survey, p. 7, bottom).

There was not a reminder of substitutes.

There was no cheap talk script. While I am not convinced of the appropriateness of cheap talk

scripts, they have become common practice in stated-preference surveys and their presence does

serve to reduce value estimates.

Consequentiality was addressed in the survey, but not as clearly as it could have been. The survey

does sate that respondent answers will help to make policy decisions (Northeast survey, p. 1, top),

but this is not repeated immediately before the choice questions nor is it specifically stated that

survey results “will be used” in making policy decisions. Respondents are ask to answer as if the

questions were real and binding (Northeast survey, p. 10, bottom), but are not asked to answer as if

they are voting in a real referendum at the moment they answer the choice questions. Thus, there are

elements of consequentiality, but they are not as poignant as they could have been.

4. Does the experimental design allow for the estimation of robust values across the range of

regulatory policy options and the scope of the resource changes?

Yes, if and only if the policy just focuses on one attribute (fish saved) and the linearity of this

variable in the estimated equations is confirmed by more rigorous investigation. To investigate the

robustness of models to support value estimation for a one percent change I suggest estimation

where each attributes is identified using effects coding to investigate linearity instead of the

quadratic specification. The quadratic models seems odd at best given the small ranges of levels

posed for some attributes (4-6%) and the large jumps (coarseness) for the key fish saved variable

(report, p. 15, Table 3-1).

5. Are the sample sizes and response rates for the regional and national surveys adequate to produce

statistically robust estimates?
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 43

The sample sizes are acceptable minimums, but below what would be considered best practice for a

study of this magnitude to support a major policy decision. Sample sizes as employed in this study

are consistent with experimental studies published in the per-reviewed literature, but are les than one

would expect for a major policy study where each sample would typically target at least 1,000 usable

observations (check).

The response rates are consistent with what is observed in the literature for best practice

applications of choice modeling. The nonresponse follow-up survey is consistent with best practice. I

wish the authors has discussed whether they believe the identified effects were relevant for value

estimation; no intuitive discussion was provided or insight.

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and

national data. For the calculation of implicit prices and monetization EPA used the results of the

weighted linear models across all the survey regions and national analysis because of empirical and

theoretical concerns. Is this approach appropriate for estimating welfare changes associated with

expected changes in survey attributes under the 316b regulation? Are there other functional forms

that EPA should investigate that might be theoretically or empirically superior?

I recommend discontinuing the use of the term RUM. While the model is based on random utility, this

is a total value study and RUM in the literature refers to a specific type of travel-cost model that just

estimates use values. Thus, I fear the use of the term RUM might cause some to misconstrue the

values that were actually estimated.

I do not think the quadratic specification is reasonable. As noted above, I suggest re-estiamtion

where the levels of each attribute are dummied out through effect coding to provide a more robust

test of the linearity assumption for the fish saved variable.

I believe the use of weighted models is consistent with best practice when survey data do not match

known population statistics.

The text does not specify how the constant, which the authors refer as ASCs are coded? Was the

ASC=1 if the alternative was A of B and 0 otherwise? This seems to be the interpretation applied to

the estimation results.

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the

steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is

EPA’s development of weighting values and their use consistent with current best practices in the

literature?

The nonresponse study follows best practice. I would like to have seen the report provide more

intuition of whether the significant effects identified really are expected to influence estimated values.

I would also like to have seen an investigation of the subjective decisions employed in the

development of weights for estimation.

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there

additional tests that could be conducted to further assess the validity of responses?

Choice experiments, by their design, provide a within subject test of scope. The scope issue in the

literature relates to between subject tests and the tests reported do not address this concern.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 44

The scope test stratifying on choices A and B is neither appropriate nor insightful. The groupings do

not stratify so different people from the same sampling frame are responding to the alternative

percentages.

Since the ASCs are not significant it is not surprising that the ASCs for alternatives A and B are not

significant. It is good to investigate if there is a leaning toward alternative A or B.

Investigation of fish saved with the other attributes is appropriate.

I would also suggest estimating the model, always including fish saved and cost, but dropping other

attributes sequentially. Given the overlapping attribute definitions and exclusions of extreme

combinations suggests that responses to the fish saved attribute may be conditioned on the other

attributes. I know this is not the same as dropping attributes from the design, but it is one way to look

at this issue.

I do not understand why some of these analyses were just conducted with the Northeast data, and not

all sample groupings.

I was surprised that you did not use results from questions 8 and 9 (p. 15 &16) to evaluate the

credibility of your survey response/equation estimates in the report?

9. What are your expectations regarding the validity and reliability of the benefit estimates that are

provided by this survey, compared to the current stated preference literature? Would you

expect the results provided by the survey to be superior to those that EPA prepared as part of the

national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend

that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b)

rule?

The design practices are better than what is done in most current studies in the stated preference

literature and the data analyses follow standard practice. I am concerned that the survey content is

not as well developed as I would have liked and as presented might lead to respondent confusions

and certain items may have led respondents to value more than the scenarios were intended to value.

The current study is superior to superior to the existing benefit estimates.

Yes, I would recommend the current study for national policy analysis. No study is perfect and any

benefit analysis must take into consideration potential imperfections in benefit estimation. In the

current analysis the robustness analyses of the survey data analysis are appropriate and can be

expanded as noted above. There needs to be more sensitivity analysis in computing the aggregate

benefit estimates.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the

potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn

presented in a transparent manner?

See my comments above.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 45

PEER REVIEW COMMENTS

Diane Dupont
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 46

Review of EPA Stated Preference Survey and Responses to EPA’s 10 Charge Questions

1. Is the choice experiment survey format an appropriate means to estimate total

willingness to pay for the characterized policies, including use and nonuse values?

My response:

Stated preference surveys, of which the choice experiments question format is an example,

are the preferred approach used by economists to obtain non-market values for goods and

services that provide benefits to people. While indirect methods (such as averting expenditures or

travel costs) can be used to infer use values, they are incapable of obtaining nonuse values due to

the absence of a related market transaction. Choice experiments, in particular, are well suited to

obtaining estimates of both use and nonuse values since they describe individual components or

attributes of the good/service of interest to survey respondents. Calculated design of packages

with different amounts (levels) of the attributes are presented to survey respondents and they are

asked to make choices between a status quo (no choice option) and one or more alternative

choices that describe different packages. By varying the levels of the attributes across several

such choice situations and designing a large enough number of different packages spread across

respondents, the researcher can obtain both marginal willingness-to-pay values associated with

one unit changes in a particular attribute, as well as total willingness-to-pay values associated

with specified unit changes in all attributes that comprise a total package or policy change.

With specific reference to this choice experiments survey the format is appropriate insofar as

the goods/services being valued are clearly understood by the respondents. After reading the

survey materials, I believe that the researchers are able to obtain a total WTP for the

characterized polices and that this total WTP will contain both use and non-use values. I should

also note that CVM, the main competitor question format, would not be as amenable to finding

both use and nonuse values.

I am less certain about the ability of the survey to separate out use from non-use components.

While I do not know whether this is an additional consideration for the work, I think this point

does have a bearing upon the total WTP being sought in the event that respondents see use and

nonuse values as being overlapping in some way, that is, the two values are not separable. The

key issue is: as each person views the attributes specified in the choice task and then makes

choices by examining attribute levels, he or she is assuming a particular relationship among these

attributes. As presented in the survey, each of the levels presented for the alternative programs

always rises relative to the status quo level. This has not gone unnoticed by respondents. In the

notes from the focus groups, the following comment is made...

(On page 9 of the Executive Summary...)“Participants generally appeared to have some

understanding of how the ecological scores interact and this influenced their responses. ...

Some participants’ responses indicated that the choice questions would have been more

difficult if some effects were to go up and others down within the same policy option. (#5-

13).

Given that focus group respondents commented on this possibility suggests that respondents

may believe in a different type of implicit relationship between the various attributes than the one
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 47

presented. Why is this important? It may be that, when they see ecosystem values rise in a

particular choice task, they may infer that this will have some additional favourable impact upon

another attribute of interest (commercial fishing, for example). So, they infer a relationship

between the attributes and not treat them as separate. I am concerned that this might result in

some kind of double counting in the Total WTP, in particular, when people assume a very strong

complementary relationship between use (eg. Commercial and recreation fish populations) and

non-use components (e.g., all fish populations that help to provide feed for commercial

populations and aquatic ecosystems that provide habitat).

(On page 15 of the Executive Summary...from a respondent...“But see everything else

follows the conditions of aquatic ecosystem. [...] the commercial fish population hinges upon

the condition of aquatic ecosystem, as does the fish population [...]” (#3-15).

I know that the Johnston et al. (2011, 2012) papers from which the EPA survey draws much

inspiration employ a similar choice experiment setup that combines both use and nonuse

components. Both the EPA survey and the Johnston et al survey leave unspecified the

relationship between environment and the components that make it up. I note, however, that

Johnston et al employ a main effects with two-way interactions task choice design that may help

to mitigate this problem. So, my mind is still not made up on how to interpret people’s responses.

What underlying household production function drives people’s choices? Does it matter if some

people see the different attributes as being either independent or substitutes and others see them

as complements?

2. Were the procedures EPA used to develop and test the proposed survey instrument

consistent with current best practices for stated preference surveys, as reflected in the

peer-reviewed literature?

My Response:

In general the procedures followed are consistent with the industry standard (focus

groups in different regions, one-on-one cognitive interviews, peer review, testing of specific

wordings, use of graphics, investigation of possible use of heuristics, etc). However, I have a

couple of concerns.

 As far as I can see, there was no pre-testing on a larger scale in the field (actual mail

survey pre-test); only focus group testing within a controlled setting.

 None of the focus groups took place in the Pacific Region. Given the weaker results

from the models using data from this region, this is a shortcoming.

 I am assuming that regionally-focused survey versions were tested in the regional

focus groups? However, it is not clear whether the national versions were also pretested. This is important to develop the salience of the attributes at a national, as

opposed to more personal, regional scale. It may help to explain the lower (than

expected) WTP obtained from the National sample.

 It was not clear from the executive report discussion about the focus groups whether

other attributes were salient to focus group respondents. I am wondering in particular
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 48

about the thermal discharge. On page 2 the survey talks about how pumping warm

water back into the environment affects ecological systems. Then, later on page 5

there is a discussion about how closed cycle cooling may reduce fish losses and also

thermal discharge. Was this an attribute that was of concern or interest to focus

groups? The thermal discharge component jumped out at me. If it was not an issue in

focus groups, then perhaps it is worthy of a footnote.

3. Is the overall quality and design of the survey booklet adequate for its intended

purpose? Is the information provided by the survey booklet sufficient and appropriate

(e.g., is there appropriate emphasis on the budget constraint facing respondents, are

appropriate steps taken to minimize hypothetical and strategic bias)? Does the

information provide a sufficiently unbiased perspective to allow valid welfare

estimation?

My Response:

 The overall design and quality of the survey booklet is impressive and enhances the

saliency of the exercise. I generally found the wording to be unbiased with one

exception. In the discussion on page 7 that describes the attributes, the score for the

ecological condition is defined between 0 and 100 % when compared to the most

natural water in the [..] region. And,... “higher scores mean the area is more natural.”

However, on the actual choice task pages the upper bound of 100% is defined as

being in pristine condition. To my mind “pristine” is not necessarily natural and may

be a somewhat value-laden word. I didn’t see any discussion of this in the focus

group report or in the actual report. Whether this type of wording might lead to a

strategic bias is something that could be considered. I note that some debriefing

questions were included but I do not see them being used to assist in understanding

responses.

 The budget constraint was addressed but not really emphasized. However, I think the

choice of higher electricity prices is a good one and believable. I would have

expected a little more cheap talk prior to the choice tasks, e.g, provide respondents

with some additional reasons why some people would not vote in favour of the

alternatives, as well as reasons to vote in favour.

 The policy effects described to people relate to improvements that will be felt within

3-5 years, however, the survey left unstated when household cost increases would

begin. I did not see any question in the survey designed to elicit time preference

information nor did I see a question designed to elicit people’s beliefs about when

they would start paying these higher costs. Different people may be applying very

different discount rates to the household cost attribute.

 I have a couple of concerns regarding information provided. In the InlandVs5b

version, on page 4 the reader is told that about 0.6 billion young adult fish are lost

each year in Inland waters due to cooling water use. Then, it says that more [my

emphasis on the more] than 0.6 billion fish could be saved. But, this is greater than

what people have been told is lost. Then, on the next page, respondents are told that
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 49

the maximum gain in fish saved is is 95 % - this seems inconsistent with the previous

information. I checked the NE/SE/National/Pacific versions and there is not the same

mismatch between what scientists estimate are the losses due to cooling and what the

maximum gains from the programs might be. In each of the other cases, the fish

saved is clearly smaller than the original amount being lost.

Screen Capture from Inland...

Screen Capture From SE

Screen Capture From NE

4. Does the experimental design allow for the estimation of robust values across the range

of regulatory policy options and the scope of the resource changes?

My Response:

 I am concerned that the scenarios presented have no instances of increases in some
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 50

attributes and decreases in others relative to the status quo. See the verbatim comment

noted above from a focus group participant. This may give rise to less robust estimates if

respondents find that scenarios do not fit with preconceptions.

 The design uses only attribute main effects. According to Carson and Louviere

Environmental and Resource Economics, 2011), in such a case the researcher must

assume that the indirect utility function is strictly additive for all respondents. This

worries me given my concern about the (possibly varying) implicit relationships

respondents may believe exist between the use and nonuse attributes described in the

choice tasks.

 Within each regional (and national) survey the same sample question is used for each

booklet but then the first choice task mostly contains different values from the sample

values. Do people treat the sample as a base or reference set of values against which they

compare values in the subsequent questions? This may induce a kind of loss aversion

response similar to what was found in a paper that I did with Ian Bateman, Brett Day and

Stavros Georgiou in RESTAT (2009). I know that respondents are told not to look back

but with a mail survey this is impossible to enforce. More generally, is there some kind of

ordering or comparison of values in earlier and later questions (since the ordering is fixed

for each booklet and not randomized across respondents). This is what Holmes and Boyle

found (Land, 2005).

5. Are the sample sizes and response rates for the regional and national surveys adequate

to produce statistically robust estimates?

My Response:

 The rationale behind the sample sizes is sound and the final response rates are just

adequate. However, I some related concerns.

 It looks to me as if there is a problem with the use of postal delivery addresses to find

respondents since the Pacific region has the smallest population but the biggest number

of undeliverable surveys. Is there a problem with this postal information, e.g., people

have vacated properties due to the financial crisis? This calls into question whether the

Pacific sample is less representative than the others.

 In terms of demographics, males are systematically over-represented. This makes me

think that there was some kind of sample selectivity that took place since the survey was

a mail survey. I looked at the invitation letters going out to households and did not see

anything in the information letter asking the person with the closest birthday to answer

the survey. So, did many women simply give the survey to their male partners thinking

that it was about fish and of more interest to the males? One could check this by looking

at whether female respondents in the sample were more likely to be living in one person

households. (I note in the telephone non-response script that they ask for the respondent

with the closest birthday.) I think this over-representation of men is a real problem.

 I would like to see a breakdown of the responses to the question... “how important is the

protection of aquatic ecosystems to you” by gender, race, income, etc. (the socio-
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 51

demographic variables that were found to differ between the various main samples and

non response samples). This would assist in better understanding whether the samples

provide robust estimates.

 The Inland sample is problematic in that it contains very different socio-demographics

from the non-response sample.

 The report says...“Because the Priority Mail response was lower than expected, the target

number of telephone completed surveys was increased to obtain the desired number of

responses.” Does this observation call into question the validity of the original mail

approach as leading to some kind of selection bias (relative to telephone responses at any

rate)?

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the

regional and national data. For the calculation of implicit prices and monetization EPA

used the results of the weighted linear models across all the survey regions and national

analysis because of empirical and theoretical concerns. Is this approach appropriate

for estimating welfare changes associated with expected changes in survey attributes

under the 316b regulation? Are there other functional forms that EPA should

investigate that might be theoretically or empirically superior?

My Response:

 I support the effort to use the weighted model results, particularly given the significant

demographic differences between the main sample and the non-response sample, as well

as the region-census level data. I think this is the appropriate approach to use for

estimating the welfare changes associated with the expected changed in survey attributes

under the 316b regulation.

 Given the concern for estimates across the different regions, I think the authors ought to

include some measure of distance decay in their models. They can look at the literature

on distance decay. A useful recent paper is “Characterizing Spatial Pattern in Ecosystem

Service Values when Distance Decay Doesn’t Apply: Choice Experiments and Local

Indicators of Spatial Association” by Robert J. Johnston, Mahesh Ramachandran, Eric T.

Schultz, Kathleen Segerson, Elena Y. Besedin. This was presented at the 2011 AAEA

meetings. The reason I argue in favour of this is that I am concerned about the likelihood

of systematic differences in responses from respondents living close to the facility points

on the maps provided in the booklets. I would imagine that the researchers have zip code

information that could be used since it was a mail survey. In particular, it is an

interesting empirical question whether non-use values are more or less subject to distance

decay than use values.

 The report notes that demographic interactions between ecological attributes and gender

and education were included in a preliminary Northeast model. However, because the

interaction terms were not found to improve model fit, demographic variables were not

used in other models. I am not convinced by the explanation in the report. When I look at

the results in Appendix G-1, it seems to me that there is significant information in the
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 52

model with interactions. Both college and gender have significant interaction terms with

fish saved, the attribute of most interest in this analysis. What were the AIC statistics for

these two models? I would like to see a similar model with interactions using the Inland

data set where most of the demographic variables differ from the non-response sample.

As the models currently stand, I am less certain that the estimated WTP values for the

Inland region are representative of average WTP since it is not obvious that the sample is

random. I am thinking particularly of the over-representation of males. I know that

weighting has been done on the parameters. However, the underlying utility function only

includes the main attributes without interactions. I would like to see a breakdown of some

of the debriefing responses on the importance of the different attributes by gender and

education.

7. Is the nonresponse study design adequate to evaluate the potential for non-response

bias? Are the steps EPA used to correct for non-response bias adequate to address the

potential biases identified? Is EPA’s development of weighting values and their use

consistent with current best practices in the literature?

My Response:

 Since the regional results are being compared to one another I think it is unfortunate that

a single weighting method could not be applied to each of the five data sets. However, I

think the EPA did the best that could be done with the information available. I would

like to see further discussion of the potential implications of these two different

weighting schemes.

 I have two questions about the decision made (on page 47) to exclude respondents from

the weighted model estimation if they do no include answers for all variables being used

in the weighting. How many respondents are removed from each region and are there

systematic differences in respondents that were removed when compared to respondents

who were kept in the sample?

 A final issue arises with the Pacific sample. There was a significant difference between

the importance assigned to ecosystems between the main sample and the non-response

sample. But, an adjustment was not undertaken for this. I understand the concern that this

is likely to be an endogenous variable however, this points up a concern that I raise

earlier about possible sample selection bias. I also note for what it is worth that the

household size in the non-response sample is a lot larger than in the main sample. Since

people were asked to consider household level cost increases, this may introduce a

potential bias.

8. Are the scope tests performed correctly and are the results interpreted appropriately?

Are there additional tests that could be conducted to further assess the validity of

responses?

My Response:
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 53

 The external scope test of looking at proportions of responses using split samples seems

like a reasonable way to examine the issue. However, no formal testing of these

proportions is done and, in some cases, the differences are very small and the standard

deviations are very large. I would like to see significance levels from non-parametric Chisquared tests of differences in proportions (where the proportions are people accepting a

particular option). I would also like to see the test done in a way that is sensitive to the

question ordering (that is, did the respondent say yes to A involving 95% fish saved in

each of the three question orderings or separately). There is some evidence in the Pacific

and National samples of a lack of scope response to fish saved (fish saved (fish_sav)

and/or fish saved squared (fish_sq) were not significant in the quadratic models for the

Pacific region and national survey.) This does call into question the salience of the

attribute for these two samples.

 The only issue that I have with respect to the argument that choice experiments provide

their own internal scope test (and so it is not necessary to do) is the following. If a

respondent ignores certain attributes in his/her choices, then this non-attendance may

mean that internal scope tests for the ignored attributes are not satisfied in a conventional

sense.

9.What are your expectations regarding the validity and reliability of the benefit estimates

that are provided by this survey, compared to the current stated preference literature?

Would you expect the results provided by the survey to be superior to those that EPA

prepared as part of the national benefits assessment for the proposed 316(b) existing

facilities rule? Would you recommend that the information provided by this survey be

used in EPA's cost benefit analysis of the 316(b) rule?

My Response:

 Compared to the current stated preference literature I believe that the data collected

within this survey is capable of providing both valid and reliable estimates for benefits. I

place more confidence in results from the regional models/datasets (with the exception of

those for the Pacific Region) than from results associated with the National dataset, in

part because of the saliency of the issues within a regional context and in part because of

the larger (relatively speaking) sample sizes used in the regions. I use the word “capable”

in the first sentence because I think the research effort would gain important insights after

incorporating distance decay effects, ordering and context effects, and further

investigation of role of differences in demographics, particularly insofar as these

differences manifest themselves expressed opinions on the debriefing questions. At a

minimum I think the researchers need to do these additional models in order to forestall

questions about the reliability of the results to alternative specifications.

 As far as I understand it the EPA has already calculated some use benefits in a prior study

(presumably from fishing). Since the values collected in the current study potentially

include use and nonuse values (although these cannot be separately identified), I assume

that the previous use benefits will not be added to the estimates from this study. That

would likely include some form of double counting. However, if the interest is in
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 54

providing use and nonuse benefits together, then the approach in this current study would

provide superior estimates for the benefits.

 I have confidence that the information obtained from the survey can be used to assist in

EPA’s CBA analysis of the 316(b) rule. I would recommend undertaking some additional

model estimation along the lines noted above in order to verify that validity of the

estimates already provided.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed

the potential implications of uncertainties in the analysis, and are the effects on the

conclusions drawn presented in a transparent manner?

My Response:

 I think the report contains a great deal of information in different locations. The

footnotes, in particular, are very helpful to further understanding of decisions made along

the way. However, I think the report should collect all potential implications of

uncertainties in the analysis and put them together in a single location. Many steps were

undertaken and many decisions were made prior to the generation of the WTP estimates

in Table 10.2. The inclusion of a table showing the types of decisions that had to be made

with respect to uncertainties in modeling, parameter values, model results, estimated

WTP values and likely directions of biases would help the reader to better understand the

implications of the uncertainties.

Additional comment

On page 66 the report states that the ASC is not included in the calculation of compensating

surplus because it was found to be insignificant in the model. There is some discussion in the

literature on the merits of including/excluding ASC values. However, it is widely accepted that

inclusion of the ASC provides more conservative estimates since the exercise involves measure

utility gains/losses of a program away from one’s status quo situation. Even though the

coefficient itself may be insignificantly different from zero, its level is important for inclusion in

calculations. I should also note that the ASC captures the effects on utility of any other attributes

not included in the choice task that may be important to the respondent. Finally, I note that this

is not a problem for the annual household WTP estimates provided in the report for the

regulatory options since these are calculated using only the marginal WTP for a one unit change

in fish saved.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 55

PEER REVIEW COMMENTS

Jim Kahn
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 56

Kahn Review

Introduction:

The first thing I would like to point out is that researchers make a large number of choices

when researching any topic, but this is particularly true with choice-modeling experiments. It

is always possible to second-guess these choices and criticize the researchers for choosing A

instead of B, when, in reality, both A and B have merits and have flaws. There is no such

thing as a perfect choice-modeling experiment. The important questions are whether the

survey was conducted with best practices in an manner in which is unlikely to have

significant biases, and whether the survey was conducted in a fashion in which the

confidence intervals of the value estimates is likely to be tight enough for meaningful policy

analysis.

I would like to say that I have a great deal of confidence in this survey. First, it was

conducted by two consultants who are truly experts in the field. Second, when they

encountered challenges (proper definition of attributes, nonresponse issues, etc.) they

addressed these challenges with vigor and rigor. Third, and most importantly, the researchers

did not try to overreach, but measured those values that could be most confidently measured.

Finally, the documents explain complicated issues in a manner that can be understood by

people that are not experts in the field. After securing permission from EPA and the authors,

I would like to use this study in my teaching.

RESPONSES TO SPECIFIC PEER REVIEW CHARGE QUESTIONS:

1. Is the choice experiment survey format an appropriate means to estimate total willingness to

pay for the characterized policies, including use and nonuse values?

Choice modeling is not only an appropriate means to estimate total willingness to pay for these

types of policies, but it is the best method for several reasons. First, when choice-modeling has

been used in potential new product applications (called conjoint analysis in this literature),

hypothetical choices have been very close to actual closes. For example, researchers asked

potential apartment renters to participate in a choice experiment looking at their preferences for

apartments with different characteristics and prices, and found that when they actually went to

rent, their actual choices were similar to their hypothetical choices. This has been true for a

number of market goods. In contrast, contingent valuation has been notoriously unsuccessful in

having hypothetical choices match actual choices, even with market goods. The decision-making

model with choice experiments is one which people are more familiar.

Choice modeling has been successfully used in a number of applications, at every level between

very local and global, and in both developed and developing country contexts. It is the best

method for modeling both direct use and non-use values, although it is often difficult to conduct

an experiment which allows the researcher to produce separate estimates of direct and indirect
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 57

use values. However, if the policy question requires an estimate of total value (rather than

separate estimates of direct and indirect use values) this is not an issue and the value estimates

can help inform the policy decision process.

One very useful aspect of choice-modeling is that it is multi-attribute, so different policy options

can be analyzed, leading to the valuation of different levels of environmental quality. Neither

contingent valuation, nor the revealed preference methods (travel cost or the hedonic methods)

have this desirable property.

Choice-modeling experiments can also be structured to include other decision-making criteria,

such as equity or sustainability, but this was not the charge of this survey. This survey was

designed solely to answer questions related to economic efficiency, measuring the total

willingness to pay for the policy induced improvements in environmental quality.

Nothing has happened to change my 2006 conclusions that choice-modeling is the appropriate

direction to go to understand the benefits of the proposed rule-making. Neither the way the

survey was implemented nor any new developments in the environmental economics literature

have changed my perspective on this issue.

In the original review in 2006, Dr. Krupnick stated that there was no indication in the literature

that choice-modeling was a better choice for policy analysis than contingent valuation. Although

EPA effectively responded to this, I think it important to add some of my own comments.

In the early 1990s as choice-modeling made its way from the marketing/transportation literature

into environmental economics, it was not well-understood, and there was considerable discussion

among environmental economists if it represented an improvement. Choice-modeling was used

for relatively simple problems in the early attempts, in which there was no real advantage over

contingent valuation. However, after roughly two decades of experience, it is almost universally

acknowledged that choice modeling is more appropriate when there are multiple types of

environmental change or environmental resources, and when there are multiple policies or levels

of policies to be analyzed.

Contingent valuation represents a good choice for simple problems with limited budgets. For

example, if a group of students came to me and wanted to determine student willingness to pay

for green energy in the dorms, I would advise them to set this up as a contingent valuation

problem. Similarly, if a poor city in a developing country wanted to look at the willingness to

pay for improved water services, I might suggest contingent valuation. In these cases, the policy

options are limited and the good is very easy to define, so sacrificing the ability of choice

modeling to deal with complexity for the ease of contingent valuation could be a good idea.

In the case of the 316b regulation, which is complex in all dimensions, choice-modeling is the

only choice. Fortunately, EPA committed sufficient resources to allow a state of the art study to

take place and the greater expense associated with the choice modeling exercise was not an issue.

2. Were the procedures EPA used to develop and test the proposed survey instrument consistent

with current best practices for stated preference surveys, as reflected in the peer-reviewed

literature?

EPA contracted with two of the most experienced and well-respected researchers in this area,
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 58

who took great pains to do everything “by the book.” All the relevant literature is cited, and at

each point where problems are encountered, the solutions are consistent with the literature. I

found the discussion of the test surveys and focus groups to be very useful, as a key issue in

evaluating the appropriateness of the survey is whether people really understood the somewhat

complicated attributes, the relationship to each other, and how policy might affect them. I found

the focus group comments on the utility of the graphics to be reassuring.

I also thought that the survey was visually appealing. This may sound like a minor point, but a

survey which is dull-looking will generate respondent fatigue much more rapidly. The survey

was also easy to read. I made sure to examine these surveys with the view scale set to “actual

size” and I could read it without reading glasses. Color contrast on the survey made it easy to

read and less fatiguing to the eyes.

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is

the information provided by the survey booklet sufficient and appropriate (e.g., is there

appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken

to minimize hypothetical and strategic bias)? Does the information provide a sufficiently

unbiased perspective to allow valid welfare estimation?

I thought the survey booklets were informative without creating biases. In general, survey

researchers walk a fine line in terms of providing the appropriate information but not actually

influencing the values that people hold. The valuation problem in this case is complex, and

technical information must be provided on how environmental impacts occur, what the

environmental impacts are, and how policy actions can reduce the impacts. The survey does an

excellent job of this without sinking to the level of propaganda or advocacy, or giving accidental

cues that might cause people to disproportionately accept or reject a policy bundle.

This is particularly important with respect to the quality variables. This is a rather complicated

situation because three of the environmental variables (commercial fish populations, fish

populations in general, and aquatic ecosystem condition) are only partially influenced by 316b

policies, and the fourth environmental variable (fish saved) can also influence the other three.

The information in the survey booklet helps the respondent think about how quality can

influenced by the proposed policy remedies without inserting a particular set of values into the

information.

In particular, the treatment of option C, the status quo, was well constructed in the survey. In

many choice-modeling exercises the status quo is ambiguous, left to the respondent to define and

implemented in the estimations with all zeros in the attribute levels (in the case of categorical

attribute variables). In the 316(b) survey study, respondents know the level of the four quality

variables that are associated with the status quo, and how this compared to potential changes as a

result of policy.

Although the preliminary information was a bit on the long side, the researchers compensated for

this by only asking the respondents to evaluate three choice sets, rather than five or six which is

usually the case in choice-modeling. For this reason, it is unlikely that respondent fatigue set in

before the choice sets where completed.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 59

4. Does the experimental design allow for the estimation of robust values across the range of

regulatory policy options and the scope of the resource changes?

I am fairly confident in the robustness of the estimates for several reasons. First, the attributes

were constructed in a fashion where they could enter the logit estimation as continuous variables

rather than categorical variables. This avoids the problem of all statistical tests being relative to

the left out variable. The continuous nature of the variables allows some of the attribute levels to

be right around the magnitude of changes associated with the proposed policies, and some to

either side of this interval. It is a good range of attribute levels which does not require

extrapolations to be made outside of the range of the data that is collected in the choice

experiment. In short, the continuous form of the attribute levels allows for a test across policy

options that would not be much more difficult in the presence of categorically defined variables.

Second, for the most part, the coefficients are very similar across the different regions and any

significant differences can be accounted for by region-specific characteristics in the type of fish

(highly anadromous in the Pacific compared to the South-East, for example).

One critical aspect to assure the validity of a choice-modeling study is to choose an appropriate

set of attribute bundles to compare in the choice sets. It is impractical to try to include all

possible comparisons in the choice sets, so the researchers must come up with a workable

number of comparisons to capture the variation associated with the complete set of variation,

leading to coefficient estimates that are efficient. EPA employed a fractional factorial

experimental design, eliminating those comparisons that might lead to protest responses or

puzzlement (a comparison where one option was better than the other across all attributes, for

example, the definition of the 72 alternative sets of comparisons follows the literature, and

sample size was appropriate for the number of choice sets evaluated. The 72 sets were

appropriately distributed across the alternative versions of the surveys in each region.

5. Are the sample sizes and response rates for the regional and national surveys adequate to

produce statistically robust estimates?

The implementation of the survey involved no shortcuts deviating from the original plan. The

sample sizes and response rates meet or exceed the guidelines established by the peer reviewed

literature and the federal government. Response rates were adequate and potential biases

associated with non-response were appropriately treated with the weighted model (see response

immediately below.

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the

regional and national data. For the calculation of implicit prices and monetization EPA used

the results of the weighted linear models across all the survey regions and national analysis

because of empirical and theoretical concerns. Is this approach appropriate for estimating

welfare changes associated with expected changes in survey attributes under the 316b

regulation? Are there other functional forms that EPA should investigate that might be

theoretically or empirically superior?

This is an important question, as empirical results are often dictated by the choice of functional

form. In terms of discrete choice models, there are many choices that can be made and it is
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 60

appropriate to conduct estimations with multiple functional forms. Linear models are standard

and quadratic models are often utilized to try to capture diminishing marginal utility associated

with increasing attribute levels. Given the small number of attribute levels, I did not find it

surprising that the peak of the quadratic function came within the range of attribute levels,

implying negative marginal utility associated with moving to the highest attribute levels. For this

reason, it was appropriate to skip the quadratic logit functions when computing values for policy

analysis.

Given the non-response bias which, if uncorrected, could lead to biases, the weighted linear

model is the most appropriate functional form for calculating values associated with marginal

changes in the attribute levels. Even though the EPA is not using the values calculated from the

unweighted linear models, it was appropriate to show those values in the tables so the reader

could compare these values to the values based on the weighted regressions. It is certainly

appropriate not to calculate values based on the quadratic regressions, as the coefficients on the

quadratic term do not conform to economic theory.

I was surprised that this question did not ask about the choice of mixed logit models versus

traditional multinomial logit models and the estimation of the cost variable as a fixed coefficient,

as these seems to me to be more controversial issues. However, the choice of mixed logit models

to deal with heterogeneity in the sample is a well-justified choice. In many other dimensions, the

mixed model is less restrictive than traditional multinomial logit. Similarly, the mix of random

and fixed coefficients in the estimation represents valid choices given the data set that was

developed.

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias?

Are the steps EPA used to correct for non-response bias adequate to address the potential biases

identified? Is EPA’s development of weighting values and their use consistent with current best

practices in the literature?

This is an area in which I expressed some concern in the 2006 peer review, and I was glad to see

it so prominently addressed in the research, with a dedicated nonresponse study.

My first concerns were actually addressed in the EPA response to the 2006 peer review

comments. I was concerned with the “lazy husband” problem and the absence of cellphone-only

households in the study. The methodology used to avoid the lazy husband problem completely

negates this source of bias. Unfortunately, the survey research community has not yet come up

with a way to adequately address the cellphone only household problem, but to the extent that

cellphone only households are likely to have certain socio-demographic characteristics (young,

single, no children, etc.) it can be determined if households that have these characteristics are

underrepresented in the survey. I do not believe that this negatively influenced the nonresponse

study.

As I prepared to complete this new peer review, I was very worried that the survey may

disproportionately have nonresponses from people who thought the government should have a

minimal role. This could cause a bias that would be uncorrectable if people with this philosophy

were not specifically correlated with demographic variables that appeared in the census. My

fears were alleviated by the question related to whether the participants in the nonresponse study
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 61

thought about the role of government involvement in environmental protection. For most of the

regions, slightly less than ninety percent of the participants thought that the government should

be at least somewhat involved in environmental protection, with forty percent stating it should be

highly involved. This was a few percentage points lower in the south. The response to this

question indicates that their initial nonresponse was due to other factors and not due to a

fundamental philosophical conflict with the proposed policy.

Another interesting test of the importance of biases related to philosophical self-selection in the

survey can be found in the questions related to the environment. Both respondents to the original

survey and respondents to the nonresponse study answered the same question related to their

perspective on the importance of aquatic ecosystems. The percentage of responses at each step of

the five step importance scale was remarkably similar, indicating that nonresponse was not due

to people who were less concerned about aquatic ecosystems.

Some patterns in nonresponses were noticed in relation to certain socio-demographic variables,

such as age, education, gender and employment were discovered in the analysis. These formed

these patterns and the potential biases that might be generated in the final value estimates

generated the need to conduct weighted logit regressions instead of unweighted estimates. The

methods that EPA used to create the weights corresponded to best practice in the literature, and

should result in estimates that are not biased by patterns of nonresponse.

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are

there additional tests that could be conducted to further assess the validity of responses?

First let me say that I thought scope tests were correctly performed, but I would like to point out

that scope tests are really a red herring and the scope issues are not as important as some portray

them to be. Even with contingent valuation, scope issues are overstated.

The scope issue began to be discussed as a result of the contingent value work that was done to

measure the benefits of the Clean Air Act and Amendments, with a lot of research focusing on

the willingness to pay for visibility in national parks. Several studies looked at willingness to pay

for a particular western national park and for the system of parks in the west, and found that the

two values were not substantially different. Economists were quick to criticize this, as a more

encompassing good should have a higher value than a less encompassing good. What the critics

failed to realize, however, is that it is highly likely that the respondents would think that a set of

policies that would reduce air pollution in one western park would reduce air pollution in other

western parks, as well as nearby cities. If they are actually looking at the policy in a broader

sense, the result is to be expected. Accidently generating a scope issue through inadequate

understanding of survey questions and policy effects can be guarded against by appropriate use

of pilot surveys, individual debriefings and focus groups.

More excitement was generated about scope in the Exxon Valdez debacle, but this was largely by

studies that were designed to create a scope problem. The bottom line is that I don’t think scope

issues are all that important for contingent valuation, and as EPA points out in the supporting

documents, it is even less of a problem in choice modeling experiments, because scope is

naturally built into the alternative levels of attributes that constitute the choice sets.

It is appropriate to test for scope issues, even it is expected that they are not likely. EPA deals
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 62

with the issue directly with several tests for the problem. The first test is the natural test that

coefficients on attribute levels have a positive slope, indicating that people are willing to pay

more for higher levels of the attribute. They were not able to demonstrate declining levels of

marginal value with the quadratic estimations, but this is not a real problem. The “external

validity” test of scope that they initiated was cleverly done, splitting the sample into option a

responses and option b responses, and showing that for both option a and option b, the choice set

was more likely to be chosen if it higher levels of the attributes.

The geographic scope test showed willingness to pay to be higher for regional improvements

than for a national improvement, although economic theory would suggest that the national

improvements should have a higher value than the regional improvements. However, this does

not surprise me, as people are more focused on improvements in their own vicinity, and this is

not a global public good in the sense of reducing global warming or species extinction.

Respondents will be uncertain about how national improvements are manifest in their own

regions, and this uncertainty and less willingness to pay for improvements which benefit people

in other regions adequately explain the difference between the regional and national values. I

would not have bothered to do this test, for exactly these reasons.

These tests were appropriate and well-executed. I do not see the need to conduct additional tests.

9. What are your expectations regarding the validity and reliability of the benefit estimates that

are provided by this survey, compared to the current stated preference literature? Would you

expect the results provided by the survey to be superior to those that EPA prepared as part of the

national benefits assessment for the proposed 316(b) existing facilities rule? Would you

recommend that the information provided by this survey be used in EPA's cost benefit analysis of

the 316(b) rule?

Of course, this question is the most important of the entire peer review of the survey and

resulting values. First as indicated in my introductory remarks and the answers to the particular

questions, I am confident that the research was properly done and conforms to the highest

scientific standards. It is not just acceptable, but actually provides a blueprint for others to follow

when conducting their own research.

The survey provides important information that was not in the original benefits assessment. The

original benefit assessment used a benefits transfer method for evaluating non-use benefits. I do

not want to imply that benefits transfer methods are a second-class method, but when resources

are available to do a valuation study of the specific policy question, it is much better to do the

valuation study and use the results, then to try to massage the results of studies of other policies

or other regions to get some sort of estimate of the values of the potential 316(b) changes. Given

the complexity of the environmental variables, the likelihood that different responses will be

forthcoming from different regions and the availability of resources to conduct a state of the art

survey, it is better to use measure the values directly than to use benefit transfer. Moreover, I like

the way the estimates of the value of fish saved can be used as a lower bound on non-use values.

This is a significant addition to the analysis. Since the survey and analysis were conducted with

the appropriate adherence to theoretical and empirical precedents, I would enthusiastically

endorse using these values in a cost benefit analysis of the 316(b) rule.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 63

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the

potential implications of uncertainties in the analysis, and are the effects on the conclusions

drawn presented in a transparent manner?

Uncertainty is omnipresent in empirical research and must be understood before the results can

be used. It is important to discuss likely sources of uncertainty, the magnitude of the uncertainty

from these sources, and the implications of the uncertainty for basing decisions on the results of

the research. In all areas, I think EPA does a good job.

The first source of uncertainty is whether survey respondents really understand the survey. If

they are responding to information that they do not understand, or are misinterpreting, then the

results are unlikely to be meaningful. This is a particular concern in the 316b survey, because of

the complexity of the environmental changes that are being examined.

EPA addresses this uncertainty in two fashions. First, this is addressed through the use of focus

groups and second through the use of particular questions in the survey itself.

The focus group discussion in the 316(b) Stated Preference Survey Peer Review Document is not

very intensive, but the supplemental document Executive Summary of Findings from 2010 Focus

Groups Conducted Under EPA ICR #2402.01 is very complete. Discussion includes the

following topics:

1. Presentation of Information to Survey Respondents

2. Interpretation of Information by Respondents

3. Survey Scenario and Respondent Decision-making

4. Budget Issues

5. Respondent Motivations

I am not going to summarize the 17 pages of discussion in this memo, but comment on a few of

the things I thought most important. First, going in, I was a bit nervous about the “fish saved”

variable and if people could really wrap their minds around this concept. I thought that it was

particularly important that “fish saved” was well understood, because of the four environmental

attribute, fish saved was the one that was most directly affected by the 316b policies. If the

variable was well-understood, then the willingness to pay estimates for changes in this attribute

could serve as a lower bound on non-use values. It would be a lower bound because it would not

fully consider non-use values associated with overall aquatic ecosystem health.

I was concerned about the fish saved variable for two reasons. First, I was not sure if people

would understand the aggregation of eggs, larvae and young fish into the fish saved variable.

Second, I was concerned that people might solely value fish saved to prevent death and suffering

of fish, and not consider the broader ecological effects of fish saved. After reading the discussion

of the understanding of the variable by the respondents, I feel that they did understand it well

enough that it could serve as a basis of policy analysis.

Second, I was concerned about the possible existence of warm glow effects. Again, the

discussion of those participating in the focus groups convinced me that they are responding
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 64

because of the anticipated environmental changes, and not because of an overall desire to do

good. This, coupled with their consideration of budget constraints, leads to a conclusion that a

lack of understanding of the surveys or the policies is not a significant source of uncertainty in

the final results.

The survey itself contains a question (#8) which addresses some of the uncertainty issues. The

responses to this question are contained in Table 6.3. The first set of questions has to do with

their understanding of the survey and the confidence in their answers. The second set of

questions have to do with their understanding of uncertainty, with one questions asking whether

the degree to which they agree that the effect of the policies depend on many factors and the

extent to which they agree that the future ecological conditions are never 100% certain.

Question 8 and the discussion of the focus groups allows the reader to evaluate the degree of

uncertainty generated by the information (or lack of) and the survey itself. The estimations

themselves contain information which can be used to develop confidence intervals on the value

estimates which provide insight into the uncertainty generated by the error terms of the

regression.

The discussion of uncertain is transparent and puts the reader in a good position to determine

how confident that they are in the results. In my opinion, the uncertainty is at a level that it does

not interfere with the ability to use the results in policy analysis.

Please provide EPA with any additional comments not covered by the above questions that you

think are important to any assessment of this survey effort.

At this point, I think I have covered the important aspects of the research. I would be happy to

answer any questions or respond in a more detailed fashion.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 65

PEER REVIEW COMMENTS

Dan Phaneuf
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 66

Comments on “316(b) stated preference survey peer review document” – Dan Phaneuf

Response to specific questions

Is the choice experiment survey format an appropriate means to estimate total willingness to pay

for the characterized policies, including use and nonuse values?

Stated preference methods in general are flexible enough to estimate both use and non-use

values. The choice between contingent valuation and choice experiments in this regard is not

about which one can measure which value. Rather, the choice reflects the needs of the

application. When the objective is to understand how multiple, different attributes affect value –

regardless of the type of value – a choice experiment is the appropriate choice. Given the

importance of the multiple ecological outcomes in this application a choice experiment was a

sensible approach.

Were the procedures EPA used to develop and test the proposed survey instrument consistent

with current best practices for stated preference surveys, as reflected in the peer-reviewed

literature?

The survey development process seems to be one of the strengths of this study. The number of

focus groups and amount of pre-fielding testing exceeds what one usually finds in the published

literature. As such I judge the procedures to have conformed to current best practice.

Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the

information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate

emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize

hypothetical and strategic bias)? Does the information provide a sufficiently unbiased

perspective to allow valid welfare estimation?

I have several comments regarding the design of the survey booklet. Most of these are small

negative comments, which I do not believe affect the validity of the survey on net. I mention

them mainly for completeness.

Regarding the design of the survey I have three specific and one general observation. First, the

environmental good is complex, which necessitated the use of substantial detail in describing it.

This is a positive in that stated preference surveys are more likely to produce valid results when

the good in question (and the constructed market) are described with a high level of specificity.

There are downsides to substantial detail, however. I found the first few pages of the survey

required a lot of effort to read and digest. The graphics are generally clear and helpful, but

translating the general information into the 100 point attribute scales required more than casual

study. This complexity might introduce noise into the estimates – i.e. the scale of utility might

be large for some – which could explain some of the insignificant main effects in some of the

models.

Second, the 100 point scale for the attributes works well as a communication vehicle, but it
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 67

makes it more difficult to digest the physical quantities they represent. For example, it is hard to

place 0.1 billion fish saved into the context of the full population – particularly later in the

survey, several pages after figures on proportionate losses were presented. As such peoples’

answers could reflect in part a noisy interpretation of the magnitude and ecological importance of

the physical benefits.

My third observation relates to the second, and is based on what the primary source of value is

for people. I generally accept the notion that there may be non-use value for this or similar

resources. However, if people are not clear on the magnitude and meaning of the policy – i.e.

saving a small proportion of common fish stocks – they may attribute more non-use value than

they otherwise would. More generally, the intuition for why this commodity – non-threatened,

common fish populations – would have substantial non-use value should be made clearer. We

usually think of non-use value as arising from something fairly unique with few substitutes, and

so perhaps more evidence marshaling from the focus group records is necessary to provide a

qualitative sense of the potential for substantial non-use value – relative to the potential for

people to have inferred a greater ecological threat is being avoided.

These details aside, I generally find the design of the survey to be adequate for the task. The

systematic way in which the study is able to directly map between policy objectives, the

definition and levels of the stated preference attributes, and household preferences is a clear

strength of the study and outweighs the concerns mentioned above. I find the description of the

environmental problem to be suitably neutral in tone.

Regarding the potential for strategic behavior I have two observations. First, the use of a

compulsory payment mechanism is incentive compatible, which is a plus. Formally, however,

two additional features of the design would technically be considered incentive incompatible. A

departure from the binary choice referendum format can only be demand revealing when

multiple levels of the commodity can simultaneously exist. Here the commodity is close to a

pure public good since the attributes are defined for the entire region, yet people are able to

select from two different outcomes along with the status quo. One can tell stories in which a

person would, for strategic reasons, select his non-preferred option. The use of multiple choice

tasks per respondent is also, technically, not incentive compatible. I should be clear, however,

that the latter is nearly universally dismissed as a concern in choice modeling and hence is not a

problem here. The former is of marginally more concern. However, in practice I do not believe

this design feature influenced the results in any measurable way.

I also have a few observations on efforts to minimize hypothetical bias. Recent research stresses

the need for respondents to view the survey as consequential, in the sense that its results have the

potential to influence something they care about. I have mixed reactions on this here. On the

one hard there are a few places where the survey reminds people that their answers will help the

government decide which policies to enact or will help the government learn if households are

willing to pay the costs. On the other hand the survey itself does not contain a clear, central

message that one might call a consequentiality treatment (I did not see a cover letter for the

actual survey – there might have been more there). On net I am inclined to think that the

importance reminders, general professionalism of the survey, and complexity of the issue led

people to perceive the survey as consequential.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 68

Two other points regarding hypothetical bias relate to the payment vehicle. As best I could tell

the survey did not contain the usual reminder for people to consider their budget constraint, what

they would give up, etc. when answering the payment question. Likewise I am not a big fan of

the ‘general cost of living’ payment vehicle, though I have used it myself and can appreciate its

merits as applying to everyone. I think it would be useful to highlight some focus group or

cognitive interview evidence that people accepted this payment vehicle as realistic, and were

considering their budget constraints when answering.

Finally, I think the survey did an adequate job of including rationality checks. Questions 7 and 8

look like useful diagnostic vehicles, and might perhaps be useful for some estimation tasks.

Does the experimental design allow for the estimation of robust values across the range of

regulatory policy options and the scope of the resource changes?

I think the experimental design is fine conditional on the decision to use three levels for the

important attributes. In this sense I believe it was the correct decision to focus on a main effects

design so as to avoid the need to ask respondents additional questions (a larger sample or more

responses per person would be needed to estimate interaction effects). The one point of criticism

I have relates to the use of only three attributes for fish saved, and perhaps for the other

ecological indicators. I would expect there to be a non-linear relationship in marginal utilities for

these outcomes. While three levels is in principle enough to trace a curve it can be difficult I

practice, as was shown by the effort to include a quadratic term in the models. I think this

feature of the design can be overcome with some additional analysis, as I discuss below.

Are the sample sizes and response rates for the regional and national surveys adequate to

produce statistically robust estimates?

I found the arguments used to decide on the original sample convincing, and I think the number

of completed surveys received ex post is adequate for the statistical task. I don’t have further

comments or complaints on this aspect of the study.

EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional

and national data. For the calculation of implicit prices and monetization EPA used the results

of the weighted linear models across all the survey regions and national analysis because of

empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes

associated with expected changes in survey attributes under the 316b regulation? Are there

other functional forms that EPA should investigate that might be theoretically or empirically

superior?

There are several things I might have liked to have seen as robustness checks, before concluding

that the weighted/linear/full covariance/MXL model should be preferred. First, I like the mixed

logit model and as such I agree with the decision to use it here. However, seeing how other

models perform would give me a better sense of what the mixed logit specification is buying.

An advantage of the simple conditional logit model is its transparency, and so I would have liked

to see this presented as a straw man to get a sense of how the data look when stripped down.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 69

Similarly it is often numerically difficult to estimate non-zero covariance elements of the mixing

distribution, particularly from a classical perspective. Seeing mixed logit results with a

coefficient independence assumption might, again, help one judge the quality of the preferred

model. In sum, while it may be only personal opinion, I don’t think it provides enough

robustness checking to only present on class of model (particularly one that might be prone to

some numerical anomalies).

Second, I would like to see additional specifications. Looking at models with dummy variables

for different non-price attribute levels (rather than making them continuous) would help give a

sense of the types of non-linearity might be present. While such a specification would not meet

the policy needs directly it would help determine the extent to which the linear model is a good

approximation. It would also be a good way to test for internal scope consistency – i.e. the

magnitudes on the estimates of the dummy coefficients should increase. I did see that such a

model was discussed in an appendix, but its interpretation was difficult owing to the fact that is

was combined with a fairly complicated mixing distribution and numerical problems. Also

related to non-linearity I think it would be worthwhile to look at a model that uses logs. This

would allow for a smoothly diminishing marginal utility of the 0 to 100 coded attributes, which I

think is important – it makes intuitive sense that the value of additional fish saved should fall as

we approach the maximum. Furthermore some of the policy options considered in the regulatory

analysis involve large, non-marginal changes in attributes that span a large part of the range.

This opens up plenty of chances for inaccuracies if the linear model is a poor approximation.

My next point here is speculative. It seems to me there could be significant differences in the

scale of utility among those who reported fully understanding the survey in question 8, relative to

those who did not. It may be possible to add a relative scale parameter to a conditional logit

model, allowing respondents who were less certain – and therefore presumably less informative

– to have a smaller influence on parameter estimates. Again, this is speculative – I would want

to search the literature for similar approaches before recommending it in a policy determining

study.

Finally, as I describe in the next question (with caveats), I found the use of sample weights

convincing. My advice would be to focus any additional estimation on the weighted sampled

generally, though for the conditional logits or other simple specifications I would present both.

Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are

the steps EPA used to correct for non-response bias adequate to address the potential biases

identified? Is EPA’s development of weighting values and their use consistent with current best

practices in the literature?

I found the non-response analysis to be one of the strengths of the study. It is unusual even

among the highest quality studies to devote this much effort to analyzing the selection properties

of the estimation sample. The presence of some statistical differences aside, I found the nonresponse data encouraging in the extent to which it validated the general representativeness of

the primary sample. In short I find the data gathering for the purpose of supporting non-response

analysis exemplary.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 70

I am less certain about the methods used to develop the actual weights, mainly because I do not

have a lot of personal experience with the techniques employed. I’ll make two comments,

conditional on this lack of expertise. First, there is mention of some observations needing to be

dropped (because of missing data) in order to implement the sub-group weighting routine. It

would be useful to know if this involved more than just a few data lines – i.e. are estimates from

non-weighted models that do and do not include the to-be-dropped observations the same?

Second, the ‘raking method’ used for two of the regions seems a bit speculative. The only

citation to it is mention of a SAS macro that implements the process. I think more needs to be

said about the general performance and acceptability of this methods vis-à-vis using a notweighted model.

Are the scope tests performed correctly and are the results interpreted appropriately? Are there

additional tests that could be conducted to further assess the validity of responses?

As mentioned earlier I would like to see an additional variant on the internal scope test that

makes use of dummy variables for the different non-price attribute levels. This will help make

more transparent issues of scope and the non-linear relationship between the attribute indices and

utility.

Specifically on the ‘external’ scope tests I think more can be done to formalize them. The idea

seems like a good one, or at least something intuitive to look at. However, the summary

statistics that are reported do not provide a way of formally or even informally assessing the

statistical presence of scope. I think it would be useful to run a linear probability model on the

Option A and Option B sub-samples, including dummy variables for each of the attribute levels

outside of price (this could stay continuous), or perhaps keeping the other attributes continuous

and just checking the different levels of fish saved. This would allow a formal test of whether

the coefficients on the three fish saved levels are different and properly ordered.

What are your expectations regarding the validity and reliability of the benefit estimates that are

provided by this survey, compared to the current stated preference literature? Would you expect

the results provided by the survey to be superior to those that EPA prepared as part of the

national benefits assessment for the proposed 316(b) existing facilities rule? Would you

recommend that the information provided by this survey be used in EPA's cost benefit analysis of

the 316(b) rule?

I find that the design work and data gathering aspects of the study generally meet the necessary

conditions for the resulting data to be valid and reliable. In addition, the raw data summaries (on

p. 21) show that respondents did seem to make tradeoffs between money and the non-price

attributes. However, I find that the current set of reported results alone are not sufficient to judge

if the value predictions contained in the peer review document are themselves of high quality.

Thus I see the study as having the potential to be superior to that which was used as part of the

existing facilities rule, conditional on some robustness checking and additional analysis. As such

I would hold back from making a firm recommendation that this survey be used to inform the

316(b) rule cost benefits analysis, but my expectation is that I would eventually be comfortable

making such a recommendation.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 71

One caveat to this is that I am not sure what to make of the national model. The issue seems

more regional to me, and it is a bit disconcerting to see the lack of geographical scope. My

tendency would be to rely on the regional models for policy purposes.

Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the

potential implications of uncertainties in the analysis, and are the effects on the conclusions

drawn presented in a transparent manner?

Continuing my point above, I think there are remaining uncertainties as to what other

specifications would predict and how robust the findings might be to modeling decisions. Thus I

think there are some uncertainties in the analysis whose effects on the conclusions are not

transparent. To this I will add that the treatment of the ASC in predicting welfare effects for

non-marginal changes in the attributes needs some additional thought. There is a line in the

document about ignoring it since it was not significantly different from zero. I think there were

cases when the ASC was significant, and in any case I think it needs to be a part of the KrinskyRobb bootstrapping procedure – i.e. the distribution of values should enter into the welfare

calculations in the same way as the other parameters (some of which are also insignificant) do.

In other dimensions I think EPA has done an admirable job of documenting uncertainties. The

quantification of uncertainty arising both from random parameters as well as sampling error is a

useful way to illustrate the range of possible predictions.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 72

PEER REVIEW COMMENTS

John Whitehead
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 73

316(b) Peer Review -- Whitehead

The specific charge questions are as follows:

1. Is the choice experiment survey format an appropriate means to estimate total

willingness to pay for the characterized policies, including use and nonuse values?

A stated preference discrete choice experiment (using terminology in Carson and Louviere, Env

and Res Econ, 2011) will elicit total willingness to pay (WTP). However, more work is needed

to make clear that the WTP estimates reported in the EPA report are valid and reliable estimates

of nonuse value. Since Krutilla (AER, 1967), nonuse values are often associated with unique

natural resources facing irreversible development. Both uniqueness and irreversibility are

subjective and relative concepts and should be considered as sufficient but not necessary

conditions. Where the resources are not unique (i.e., many substitutes) and development is not

reversible (i.e., the species are not threatened or endangered), more effort is needed to justify the

use of nonuse values in benefit-cost analysis.

For example, Krutilla described the motives that are likely to lead to nonuse values. The survey

gathers information on motives for nonuse values to preserve fish for recreation (evidence of an

altruistic motive for those who do not fish), preserve fish for aquatic ecosystems, wanting to

know that fish exist in local lakes, rivers and bays, and preserve fish for future generations

[question 9]. The report states that most of the respondents had some motive for nonuse values

but this only provides weak evidence that nonuse values are valid. If the responses were analyzed

out of the RUM framework so that an index of motives was a right hand side variable, the WTP

model could be used to extract motivated nonuse values from non-motivated values. In a RUM

framework this could be accomplished with interaction of an index of motives with alternative

specific constants (i.e., are motivated respondents less likely to choose something other than the

status quo?) or with split sample analyses. The RUM models could be estimated with split

samples of highly motivated vs. less highly motivated respondents.

On page 17, the first paragraph illuminates a potential problem with the analysis. It is clear that

survey respondents were not likely to have any prior information about the issue before they took

the survey. It is well-known from the CVM literature that survey respondents can have

constructed preferences in a no prior information context and that WTP estimates from

constructed preferences are less valid than preferences based on real experience. For example,

Whitehead et al. (JEEM, 1995) show that WTP estimates from survey respondents who had no

prior information about the topic before the survey did not pass internal validity tests.

Respondents who had prior information but were not on-site users and on-site users did pass

these tests. When these sub-samples are pooled, the invalid responses are masked by the valid

responses. Without the prior information question it is impossible to know whether there is a

subsample of invalid WTP estimates.

For these reasons, there is some concern that the nonuse component of the WTP estimates may

lack validity relative to the use value component and the WTP estimated with the full sample

may be aggregated over too large a population. The analysis could be conducted with splitsamples of users vs nonusers (there is no question asking about prior information) and those with
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 74

significant nonuse value motives vs. those without to demonstrate the validity of the WTP

estimates of those respondents with questionable rationales for valuing these resources. If subsamples are found to have invalid WTP responses then in a sensitivity analysis, WTP should be

aggregated over only the sample proportion with valid values for benefit-cost analysis.

2. Were the procedures EPA used to develop and test the proposed survey instrument

consistent with current best practices for stated preference surveys, as reflected in the peerreviewed literature?

The EPA’s survey design methods were thorough and state of the art.

3. Is the overall quality and design of the survey booklet adequate for its intended

purpose? Is the information provided by the survey booklet sufficient and appropriate

(e.g., is there appropriate emphasis on the budget constraint facing respondents, are

appropriate steps taken to minimize hypothetical and strategic bias)? Does the information

provide a sufficiently unbiased perspective to allow valid welfare estimation?

The overall quality and design of the survey booklet was “adequate” but some features of a

stated preference survey were not included. Respondents were told to “answer all questions as if

this were a real binding vote.” Respondents were told how the cost of the program would affect

their budget (in higher prices, etc.) but not reminded that this money could have been spent on

other things, which is a typical budget reminder.

Respondents were not asked directly about the consequentiality of their votes (i.e., did they think

that their votes would be used by policy makers; Carson and Groves, Env and Res Econ, 2007).

Respondents were asked for confidence in their answers and whether they would vote the same

way in an election, but these responses are not used in the empirical analysis. Footnote 11

describes the “binding vote” question as evidence for consequentiality. Twenty-two percent of

the respondents would not vote the same way and 27% of respondents are not confident about

their answers. Nevertheless, the EPA does not consider if confidence had any effect on the

validity of responses and aggregates sample WTP over the entire population as if these

respondents provide equally valid WTP estimates.

A split-sample analysis of confident vs. not confident respondents and same way voters vs. those

who would vote another way would shed some light on this issue. If the split-sample models are

similar then the EPA’s approach is not problematic. If WTP has little validity for respondents

who are not confident about their responses or who would vote a different way, then the EPA’s

aggregation approach is overly optimistic.

4. Does the experimental design allow for the estimation of robust values across the range

of regulatory policy options and the scope of the resource changes?

Yes, this is an important feature of a discrete choice experiment.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 75

5. Are the sample sizes and response rates for the regional and national surveys adequate to

produce statistically robust estimates?

The regional sample sizes are adequate to obtain statistically significant coefficient estimates

especially with discrete choice experiment data where repeated measures are used in a mixed

logit. It is difficult not to produce statistically robust estimates with this sort of data and

empirical model.

It seems odd that more resources were not devoted to the national survey which has a sample

size too small to represent the national population. It is also too small to obtain WTP estimates

with confidence intervals tight enough to adequately compare to the weighted regional WTP

estimates (see page 68 comment below).

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the

regional and national data. For the calculation of implicit prices and monetization EPA

used the results of the weighted linear models across all the survey regions and national

analysis because of empirical and theoretical concerns. Is this approach appropriate for

estimating welfare changes associated with expected changes in survey attributes under the

316b regulation? Are there other functional forms that EPA should investigate that might

be theoretically or empirically superior?

Using the weighted linear models seems appropriate and is adequately justified. I know of no

other functional forms that might outperform the linear and produce values with diminishing

marginal utility. But I wonder if this goal is worthy? It may be that there is no diminishing

marginal utility within the narrow range of policy impacts proposed by this survey for attributes

other than fish saved. And constant marginal utility is not unlikely for fish saved. Economic

theory does not have much to say about the demand that generates nonuse value.

7. Is the nonresponse study design adequate to evaluate the potential for non-response

bias? Are the steps EPA used to correct for non-response bias adequate to address the

potential biases identified? Is EPA’s development of weighting values and their use

consistent with current best practices in the literature?

The response rates are low (albeit, about as high as you can reasonably expect these days) and

raise the issue of nonresponse and sample selection bias. Nonresponse bias exists when the

sample is not representative on known characteristics. The EPA makes an honest effort to gather

information on respondents with an overall 25% response rate for the nonrespondent survey, but

still manages to obtain information on less than 10% (=619/7118) of nonrespondents.

Sample selection bias exists when the sample is not representative on unknown characteristics.

There is evidence in the survey methods literature that shows that response rates are a function of

survey salience. The respondents, especially to a mail survey, are those who care most about the

subject. Those respondents who care most about the subject are likely to have higher WTP for

the policy.

One approach is to estimate a Heckman type sample selection model with the samples of
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 76

respondents and nonrespondents with the dependent voting variable converted to a zero-one

variable for a probit. If there is no statistical evidence of sample selection bias then the EPA’s

aggregation approach may be appropriate.

In the absence of evidence to the contrary and given the possibility of sample selection bias, the

aggregation rule, assuming that weighted WTP responses adequately reflect nonrespondent WTP

is overly optimistic. It cannot be ruled out that nonrespondent WTP is less than respondent WTP.

An overly pessimistic aggregation rule would add WTP responses over only the 33%

(=2313/7118) of the population that responded. This is the aggregation rule suggested by

Mitchell and Carson in their 1989 CVM book for WTP values with little information on

nonrespondent WTP. Considering this issue, it is possible that aggregate benefit estimates are

biased upwards.

8. Are the scope tests performed correctly and are the results interpreted appropriately?

Are there additional tests that could be conducted to further assess the validity of

responses?

The external scope tests are not adequate to demonstrate responsiveness to scope, at least

according to the external scope test standard set by CVM critics. The EPA report performs

internal validity tests using multiple scenarios. This test is relatively weak since respondents

need only be consistent within themselves to produce WTP values that increase with policy

impacts. An external validity test compares responsiveness to scope with split sample survey

design.

Within a discrete choice experiment an external validity test can be conducted by using only the

first scenario presented to respondents in the RUMs. In effect, the first scenario is a split sample

test. The EPA considers the split-sample data in Table 8-10 but only applies the eyeball test.

Differences in frequencies across the fish saved scenarios can be tested with a likelihood ratio

test. The external validity test should be conducted with the RUM model (e.g., Lew and Wallmo,

Env and Res Econ, 2011). Note also that in a mail survey the first response may be contaminated

by the following scenarios since respondents could have looked ahead and developed internally

consistent answers during the survey.

9. What are your expectations regarding the validity and reliability of the benefit

estimates that are provided by this survey, compared to the current stated preference

literature? Would you expect the results provided by the survey to be superior to those

that EPA prepared as part of the national benefits assessment for the proposed 316(b)

existing facilities rule? Would you recommend that the information provided by this

survey be used in EPA's cost benefit analysis of the 316(b) rule?

These estimates are superior to those used in the 2011 report since they cover the entire nation

and are based on a primary valuation study. However, in spite of relatively low statistical

uncertainty, the WTP estimates are highly uncertain for reasons described above. First, it is not

clear if roughly 25% of the sample has valid nonuse values (see #1). Second it is not clear if
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 77

roughly 67% of the population represented by nonrespondents have WTP values equal to those

of the respondents (see #7). If both of these groups of the populations have WTP equal to zero

then the WTP estimates could be as low as 25% of those reported (=100*.75*.33).

Unless evidence can be provided that all of the respondents have valid nonuse values (#2) and

nonrespondents have similar WTP values to respondents (#7), I would not recommend that the

information provided by this survey be used in EPA’s benefit-cost analysis. The aggregate

benefit estimates in Table 10-3 could be used if they are described as the best case analysis

(instead of the base case) and sensitivity analysis be performed over the potential range of

values. Given the current information, a uniform distribution should be used for the range of

values.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the

potential implications of uncertainties in the analysis, and are the effects on the conclusions

drawn presented in a transparent manner?

Uncertainties in the analysis are not fully described or considered. Key uncertainties include

validity of nonuse values over the entire sample of respondents and potential differences between

respondent and nonrespondent WTP. A number of split sample analyses are necessary to provide

information for potential sensitivity analyses. Without these it may be necessary to be more

conservative with aggregate benefit estimates (see #9).

Another approach is to estimate latent class models which are able to estimate different

(probabilistic) groups of responses. With data such as these, without information on prior

information for the respondents, a latent class model could estimate two or more vectors of

coefficients for the choice models with probabilities that any respondent belongs to each group.

It is possible that one or more vectors reveals the probability that latent groups of respondents

have invalid WTP estimates (i.e., statistically insignificant cost and benefit coefficients). Haab et

al. (Marine Res Econ, forthcoming) estimate conditional logit, mixed logit and latent class

models and produce a range of WTP estimates. They suggest that this range could be used to

provide a range of WTP estimates for sensitivity analyses. A similar approach could be taken

here.

Please provide EPA with any additional comments not covered by the above questions that

you think are important to any assessment of this survey effort.

Page 8: “Appropriately designed choice-based SP methods may also reduce strategic,

hypothetical and other possible biases that can result from asking survey questions versus

assessing WTP through market transactions or binding referenda.” During the CVM debate

discrete choice experiments (i.e., “choice-based SP methods”) were introduced. The mood

amongst many environmental economists was that choice experiments were preferred for reasons

not related to the problems of the CVM (which are mentioned in the sentence above) such as

consistency with random utility theory, an interest in new and advance econometric methods and

publishability. As such, it was asserted that choice experiments were a silver bullet that solved

many problems associated with the CVM. I have not seen much comparative research to back up
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 78

this assertion. For example, the next paragraph describes the extensive research conducted on

hypothetical bias in the CVM. No reference is made to studies assessing the potential

hypothetical bias that may result from stated preference choice experiments. Unless there are a

number of studies that can be referenced here to back up the assertion, this sentence should be

removed.

Page 9: I’m not sure what “older forms of SP analysis” refers to. Is this the CVM? It might be

better to go ahead and state that the comparison is being made to the older forms of the CVM.

But why is this comparison appropriate? The EPA should be comparing discrete choice

experiments to newer forms of CVM that are not “typically contingent upon a single

specification of ecological effects.” It is straightforward to design CVM surveys to ask multiple

WTP questions (e.g., Christie and Azevedo, J of Agr Econ, 2008). If the EPA feels that it is more

efficient to estimate the WTP for characteristics of a policy with discrete choice experiments

relative to the multiple question CVM then some supporting references should be provided.

Page 13: “an unharvested population is typically believed to be approximately three times as

large as MSY.” Please provide a reference.

Page 21: Why was a mail survey chosen over an internet survey with all of the advances in

internet sampling that have occurred over the past 10 years or so? With all of the mailings it

seems that even Knowledge Networks would have been less expensive and more effective. Note

that Lew and Wallmo (Env and Res Econ, 2011) used Knowledge Networks for a federal agency

survey.

Page 21: Was the Pacific response rate 32% or 40% = 311/(1040-255)? Was the Inland response

rate 35% or 33% = 787/(2480-65)? Why was there such a lower rate of undeliverable surveys for

the inland sample (2%) compared to the others (7% - 25%)?

Page 22: 46% plus seems like a high rate of recreational seafood consumption. Are there any

representative sample statistics with which to compare?

Pages 27-42: Reading section 7 gets tedious with the cut and pasting. Some of this could go into

an appendix with a summary in the text.

Page 51: Much of the text above Table 8-5 should be relegated to a footnote. It really doesn’t

matter if this study has similar pseudo-R2s as other studies.

Page 55: “Unlike open-ended contingent valuation questions, choice experiments provide a

direct mechanism for respondents to react to scope and scale of resource changes ....” Why is

open-ended CVM the relevant comparison? Open-ended questions have not been the norm since

Bishop and Heberlein (AJAE, 1979).

Page 66: What is the WTP per fish saved? How does it compare to other studies?

Table 9-2: The regional implicit prices from the weighted linear model are in a different order
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 79

(NE, SE, IN, PA) compared to Table 9-1 (NE, SE, PA, IN).

Page 68: The point estimate of the weighted regional implicit WTP, $1.28, is within the 90%

confidence interval of the national fish saved WTP estimate so there is no reason to discuss any

difference in point estimates.

Page 69: The AIE acronym is used before it is introduced as an acronym in this paragraph.

Table 10-1: How is the “reduction in AIE losses” column measured? Is this millions of fish

saved?

Suggestions:

1. Further test the validity of nonuse values. Conduct split-sample analyses with:

a. highly (nonuse value) motivated vs. less highly motivated respondents

b. users vs. nonusers

c. confident vs. not confident respondents

d. same way voters vs. those who would vote a different way in a real referendum

2. Conduct sensitivity analysis considering the possibility that nonrespondent WTP is lower

than respondent WTP
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 80

APPENDIX II

CLARIFICATION QUESTION RESPONSES
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 81

Responses to EPA Clarification Questions - Boyle

1. Tests of question ordering - Clarification as to specific tests (if any) that are being

requested to address effects of question ordering. Issue raised by Diane Dupont on p.26

of the peer review report.

I concur with Diane’s response in terms of investigating sequencing effects in responses

to the choice questions. I do not think that your design will allow the same

investigations as conducted in Day et al. or Holmes and Boyle. Diane can correct me if I

am wrong about the Day paper as she is a co-author.

From the EPA report I conclude that there were 24 versions with each version having

three Option A/B pairings. This implies to me that the sequence of choice questions

was fixed within each of the 24 versions, but varied across versions. Thus, the choice

question sequence is not completely fixed as Diane assumes, but the fixed order within

each of the 24 versions would need to be accounted for in any analysis. In Holmes and

Boyle, ordering of pairs was random so we did not have this fixed component in our

analysis.

We know that sequencing effects exists in sequential choice questions so your study is

not unique in this concern. Moreover, most studies do not investigate this potential

confounding feature. In addition, once you identify potential sequence effects I do not

believe there is consensus in the literature on the responses that are most appropriate

to use. The Day paper helps us further understand this issue, but does not provide a

conclusive answer.

2. Alternative log model - Clarifications as to the specific functional form being

recommended. Raised by Dan Phaneuf on p.21 of the peer review report.

I believe Dan was referring to logging the quantitative variables as opposed to linear

specifications. However, I think discrete coding, as dummy variables or effects coding,

would be more appropriate. Logging these variables continues to assume continuous

response effects and this assumption may or may not be correct, and can be

investigated with discrete coding.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any)

that are recommended by reviewers. Diane Dupont questioned the representativeness

of the sample at multiple points in her comments, focusing on the Pacific and Inland

regions.

There are two approaches here. One is to perform a Heckman style selection model, but

as I noted in my previous comments the statistical properties have not been worked out

a random parameters model. Woolridge outlines a panel estimator for the selection

model that can be applied with fixed parameter models. If you do not want to go the
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 82

panel estimator approach you could randomly draw one choice observation from each

respondent and estimate the selection model. If you repeat this, say with 500 draws,

you will find that the result should be very robust to the random selection.

The other approach is to consider what variables you are going to use to weight your

data and if these are the variables that truly influence responses to the choice questions.

While your procedures for considering sample selection are consistent with best

practice you follow the same mistake of most of these analyses of using available

variables and not asking if these variables affect responses to the choice questions and

ultimately value estimates. Ask what covariates that you have population data on and

influence choice responses and use these variables in the robustness checks you do for

sample selection.

4. Does the main effects experimental design of the survey interfere or complicate the

interpretation of models that use interactive terms and split samples. If so, can

reviewers offer suggestions on use and interpretation of results?

The main effects design implies that you do not have the most efficient design for

estimating interaction effects, but it does not preclude you from doing this analysis. It

may be the case that you are unable to identify some interaction effects that would

become significant with a different design. If, an interaction effect is highly insignificant

I would not expect that are different design would change such an outcome to a

significant effect.

At face value, the main effects design should not interfere or complicate split sample

analyses, but I think that your sample sizes may result in a problem. First, I think your

sample sizes are minimally sufficient for estimation with all data so I am concerned

about the credibility of any sample splits. The concern for the main effects design is if

the designs are not equally represented in each of your split samples, i.e., respondents

with one set of design points coincidentally happen to be more represented in one

sample and less in another sample. This is something you can and should check in each

split sample before progressing with the statistical analyses.

5. Can reviewers offer suggestions as to how the combine the regional data, given the

different baselines across regions. How would you transform data to accommodate the

fact that the meaning of a percentage point change in each of the environmental

attributes differs across the regions?

One possibility is to include region fixed effect variables and then interact the fixed

effect variables with the attribute variables.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 83

Responses to EPA Clarification Questions – Dupont

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to

address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review

report.

Since the survey was a mail survey and the order of the questions was not varied across

respondents, I was concerned that responses might reflect the particular and unvarying order

used. Specifically, I was concerned that respondents might make comparisons across choice

tasks that would result in a type of systematic bias. There are tests that can be made to determine

whether there are precedent dependent effects (that is, where responses to choice tasks 2,3,4

depend upon attributes seen in prior choice tasks 1,2,3). This essentially involves creating a

series of additional explanatory variables that provide comparative information on previously

seen attributes and looking at whether the coefficients are individually or jointly statistically

significant. Please see Day et al (2012) in the Journal of Environmental Economics and

Management (“Ordering Effects and Choice Set Awareness in Repeat-response Stated

Preference Studies”). Holmes and Boyle (2005) have a similar type of test in their Land

Economics paper “Dynamic Learning and Context-Dependence in Sequential, Attribute-Based,

Stated-Preference Valuation Questions”.

2. Alternative log model - Clarifications as to the specific functional form being

recommended. Raised by Dan Phaneuf on p.21 of the peer review report.

I agreed with Dan on this point. My understanding is that he meant that the attributes that are

valued between 0 and 100% should be transformed into natural log values and the model reestimated with these natural logs. Presumably, household costs would also be transformed.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that

are recommended by reviewers. Diane Dupont questioned the representativeness of the sample

at multiple points in her comments, focusing on the Pacific and Inland regions.

In order to deal with sample issues that may arise between the main sample and the non-response

sample, a Heckman type approach could be used that jointly estimates a selection equation with

the conditional logit and/or mixed logit choice outcome equation to see whether the errors are

correlated. The selection equation could be parameterized with either socio-demographic

responses or responses to common questions in surveys to participants and non-response

individuals who completed the follow-up short survey.

It is much harder to deal with non-representativeness that may arise between census data

characteristics and sample characteristics. The researchers do not have responses from the census

population to questions relating to attribute importance and recreational interests, etc. What the

researchers do have is information from both the main sample and the non-response sample to
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 84

the common set of debriefing questions. As I recall, there are some socio-demographic

differences between these two samples, and, in some cases, the non-responses sample was more

representative of the underlying population in terms of observable characteristics. It would be

helpful to see a breakdown by gender, race, income, etc. (the socio-demographic variables that

were found to differ between the various main samples and non-response samples) of the

responses by main sample and non-response sample to the question... “how important is the

protection of aquatic ecosystems to you?” This would assist in better understanding whether the

main samples provide robust estimates for a more general population.

4. Does the main effects experimental design of the survey interfere or complicate the

interpretation of models that use interactive terms and split samples. If so, can reviewers offer

suggestions on use and interpretation of results?

In general, I do not think that the main effects experimental design interferes with or complicates

the interpretation of models using split samples based upon variables that are not in the design.

However, in this study split samples are used to perform a type of external scope test described

on page 57. We have already commented that this is a weak form of test.

There may be a problem with the use of interactive terms since the main effects design contains

common combinations of attribute levels, so it may be difficult to disentangle the separate effects

of attributes. Systematic use of responses to the debriefing questions may assist in interpreting

results in this case.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different

baselines across regions. How would you transform data to accommodate the fact that the

meaning of a percentage point change in each of the environmental attributes differs across the

regions?

I think it will be necessary to step back from this level of detail (% change point tied to a specific

quantity change in the environmental attribute where the quantity changes differ across regions).

One solution is to categorize the attributes into different groups. The logical groups would be

low, medium and high changes relative to the specific baseline for that region. Then, I am

wondering whether it would be possible to pool the data and estimate an overall model with

some recognition that the scale parameter could differ across the four regions.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 85

Responses to EPA Clarification Questions – Kahn

Let me preference my remarks by saying that I do not consider myself to be an expert in the

statistical issues associated with choice-modeling, although I think I am a competent practitioner.

My comparative advantage in the review process is in the choices that are made in the structure

of the survey such as the definition of environmental variables, choice of payment vehicles,

additional questions and so on. Consequently, I feel that the EPA should place the greatest

weight on the responses of the other reviewers to these questions. At this point, I have only seen

Professor Dupont’s responses, and I concur with what she has said. I offer some additional

remarks in response to the questions.

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to

address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review

report.

I did not raise this issue in my original report, but I appreciate Diane Dupont’s insight on this

issue. I agree with Professor Dupont’s suggestion.

2. Alternative log model - Clarifications as to the specific functional form being

recommended. Raised by Dan Phaneuf on p.21 of the peer review report.

I agree with Professor Dupont in the interpretation of Professor Phaneuf’s remarks. A simple

natural log transformation of the key variables would be simple and informative.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that

are recommended by reviewers. Diane Dupont questioned the representativeness of the sample

at multiple points in her comments, focusing on the Pacific and Inland regions.

Of course, it is very difficult to deal with sample selection issues after the sample has been

conducted. The best way that I can think of is to choose a large number of the “attitude” or

“preference” questions and see how they vary across the different socio-economic

characteristics, being careful to examine cross-cells (e.g. the combination of minority, low

income and male) rather than just looking at the broader socio-economic characteristics

separately.

I agree with Professor Dupont that the Heckman method could be usefully employed. Professor

Dupont talks about the parameterization of the selection equation with socio-economic

characteristics or attitudinal questions. Care must be taken that both equations are properly

instrumented.

Another test could be developed with the aggregation of the samples using categorical quality

variables. If the data from all regions could be aggregated, then this data set could be split into
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 86

separate socio-demographic categories and separate estimates generated for each category. If the

number of categories is held low, the sample size for each category should be sufficient.

In order to do this, the distribution of sample designs across all groups must be the

approximately equal. If for example, one socioeconomic group was given a disproportionate

number of the same sample design, this would degrade the results.

The issue of over or under representing a region in the aggregate sample could be easily

corrected with a weighted regression. Total benefits could be then be easily computed by

multiplying each estimate by the number of people in the category and summing these estimates.

4. Does the main effects experimental design of the survey interfere or complicate the

interpretation of models that use interactive terms and split samples. If so, can reviewers offer

suggestions on use and interpretation of results?

I do not think there is a problem with split samples, so long as sample size is sufficient.

If one is interested in interactive terms, such as the impact of income on willingness to pay, I

agree with Professor Dupont on the potential difficulty associated with the main effects

experimental design.

However, a potential way to get around this issue is to aggregate the data as suggested above (by

using categorical quality variables) and estimated separate regressions for different income levels

(or whatever the variable of interest). This will, at a minimum, provide some information of the

effects of socioeconomic variables on willingness to pay. In order to do this, the distribution of

sample designs across all groups must be the same.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different

baselines across regions. How would you transform data to accommodate the fact that the

meaning of a percentage point change in each of the environmental attributes differs across the

regions?

I agree with Professor Dupont that the way to go is to transform the policy interventions into

light, medium and strong responses. This will interject some noise into the experiments, but if tstatistics remain strong, one could go forward with willingness to pay estimates. As mentioned in

the responses to questions 3 and 4, moving forward in this direction could aid in answering some

of the other questions posed by EPA.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 87

Responses to EPA Clarification Questions – Phaneuf

1. Tests of question ordering: Clarification as to specific tests (if any) that are being requested

to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer

review report.

Intellectually I would be interested in seeing the results of any ordering effects tests that

might be conducted, but in terms of making the study more useful for its intended policy

purposes I am less certain. Understanding the causes, consequences, and fixes for ordering

effects in choice experiments is still something that is being actively researched. Given the

Carson and Groves work we might expect there to be ordering effects, so unless we can think

of a way that the study’s value would be enhanced by explicitly identifying said effects I am

inclined to focus efforts on other exercises.

This said Diane is the person who has worked on ordering issues recently. If she thinks there

are ways to improve the policy relevance of the study by conducting some tests, I would be

willing to follow her lead.

2. Alternative log model: Clarifications as to the specific functional form being recommended.

Raised by Dan Phaneuf on p.21 of the peer review report.

My interest was in seeing two types of models: one that coded the different attribute levels

qualitatively (i.e. as dummy variables) and one that used a natural log transformation, rather

than the linear transformation. The former would allow us to flexibly see the extent of nonlinear effects, and the latter would provide a non-linear but still parsimonious alternative to

the linear model. The log specification might be something like the following:

0 1 2 3 4 5 V fish save com fish fish pop aquatic cost     ln( _ ) ln( _ ) ln( _ ) ln( ) ( ),           

where the cost variable is kept in the level so as to make the marginal utility of income

constant. The issue of looking at conditional (rather than mixed) logit models in order to

better explore functional forms came up in this context. I believe my suggestion was to run

the dummy variable, linear, and log models as simple logits to make a better case for

whichever specification ultimately is deemed best. I had the sense that a continuous model

(as opposed to the dummy variable model) was needed for welfare purposes, to allow

extrapolation away from the values of the designed attributes. Seeing if the (continuous)

linear or (continuous) log specification better matched the pattern from the dummy variable

model would help make the case for one or the other.

3. Statistical tests for the effects of sample selection: Clarification on specific tests (if any) that

are recommended by reviewers. Diane Dupont questioned the representativeness of the

sample at multiple points in her comments, focusing on the Pacific and Inland regions.

I don’t have specific comments on this issue.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 88

4. Does the main effects experimental design of the survey interfere or complicate the

interpretation of models that use interactive terms and split samples? If so, can reviewers

offer suggestions on use and interpretation of results?

If the interaction terms are (attribute)×(person characteristic) there should be no problem –

this would just divide out the population average main effect into components that vary with

respondent characteristics. A split sample should not be a problem with this so long as each

of the sub-samples still contains the same mix of survey versions that allow estimation of the

main effects in the full sample (could become a practical issue given overall sample sizes).

The only point of caution is if the sample split is based on the value of one of the personspecific characteristics. This should not cause econometric problems – just care in

interpretation.

If the interaction terms are (attribute)×(attribute) I think something close to the same logic

applies, but the practical difficulties will be greater. The main effects design does not mean

the model will be incorrect if you try to estimate interactions – only that you may not have

the variability needed to do so with any precision. Splitting the sample will exacerbate –

both overall and in comparison, since one sub-sample might end up with more of the ‘useful’

variability for estimating the interactions.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different

baselines across regions. How would you transform data to accommodate the fact that the

meaning of a percentage point change in each of the environmental attributes differs across

the regions?

This is speculation, but I wonder if the log model written out above could help? The

coefficients in that case are proportionate changes in utility, which could be meaningful in

reference to any of the baselines.

If this does not make sense, perhaps a model that constrains the marginal utility of income to

be constant across the regions but then estimates region-specific marginal utilities for the

attributes (using region interactions) would provide some precision benefits in terms of a

larger sample and some better comparability.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 89

Responses to EPA Clarification Questions –Whitehead

Let me preference my remarks by saying that I do not consider myself to be an expert in the

statistical issues associated with choice-modeling ...

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to

address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review

report.

I do not have anything to add.

2. Alternative log model - Clarifications as to the specific functional form being

recommended. Raised by Dan Phaneuf on p.21 of the peer review report.

I do not have anything to add.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that

are recommended by reviewers. Diane Dupont questioned the representativeness of the sample

at multiple points in her comments, focusing on the Pacific and Inland regions.

Nonresponse bias exists when the sample is not representative on known characteristics. The

EPA does a noble job in testing for nonresponse bias. Sample selection bias exists when the

sample is not representative on unknown characteristics. There is evidence in the survey methods

literature that shows that response rates are a function of survey salience. The respondents,

especially to a mail survey, are those who care most about the subject. Those respondents who

care most about the subject may have higher WTP for the policy. See Whitehead, Groothuis and

Blomquist (Econ Letters, 1993) for these alternative tests in a CVM context.

One approach to test for sample selection bias is to estimate a bivariate probit with sample

selection model. The selection equation could include demographics and attitudinal variables

explaining the response to the first survey. The three voting choices (status quo, option A, option

B) could be converted to a zero-one variable to estimate a probit. The zeros would be the statusquo choice. The ones would be the policy choice if it was chosen (with the other policy

alternative discarded). If the status quo was chosen then one of the policy alternatives could be

randomly discarded. Obviously, the study was not designed for this analysis and the proposed

model is not ideal (I’m not aware of a sample selection model for the multinomial logit and its

variants), but if there is no statistical evidence of sample selection (i.e., the error terms are not

correlated) then there is evidence that EPA’s aggregation approach is appropriate.

4. Does the main effects experimental design of the survey interfere or complicate the
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 90

interpretation of models that use interactive terms and split samples. If so, can reviewers offer

suggestions on use and interpretation of results?

I do not have anything to add.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different

baselines across regions. How would you transform data to accommodate the fact that the

meaning of a percentage point change in each of the environmental attributes differs across the

regions?

I do not have anything to add.
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 91

APPENDIX III

RESPONSES TO CLARIFICATION QUESTIONS
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 92

Clarification Questions

1. Alternative log models:

Some reviewers proposed to transform attribute levels (including household costs) valued

between 0 and 100% into natural log values and re-estimate the model with natural logs.

The two variables that are of particular interest in this study (fish_saved and cost) take

zero values. We could assign some very small arbitrary values to these variables (e.g.,

0.001) and estimate the suggested model. Alternatively, we could use inverse hyperbolic

sine (IHS) function instead to investigate nonlinear effects following a study by Layton

(2001).

Layton, D. F. 2001. Alternative Approaches for Modeling Concave Willingness to Pay

Functions in Conjoint Valuation. Amer. J. of Agric. Econ. 83(5): 1314-1320.

Would the use of the IHS model address your comment/concern?

2. Statistical tests for the effects of sample selection

With regard to sample selection, the peer reviewers have suggested a variety of possible

options. EPA has not been able to locate a published model for Heckman selection with

multinomial responses. Can reviewers provide a reference for a published model? We

would also welcome any additional guidance that the reviewers might provide. In case

the Woolridge (Wooldridge?) paper provides this, could the complete reference be

provided to us?

In the absence of a first-best solution, the bivariate probit approach suggested by

Whitehead seems to be a feasible alternative, and is attractive because the Heckmancorrected bivariate probit is an established approach (i.e., we would be grounding our

approach in an established method). In order to accomplish this, we would transform our

data into bivariate form by dropping or grouping observations, as suggested by the

reviewer. We could then evaluate statistical evidence of sample selection in the

transformed data.

We would like clarification on: (1) whether we are correct in our assumption that there is

no published and accepted Heckman-correction model for multinomial responses that we

could readily apply, and (2) whether the peer reviewers agree that the bivariate approach

suggested by Whitehead is a feasible alternative.

Kevin Boyle’s response to clarification question:

1. I think your proposal is fine and is much better than adding a very small number to

zero.

2. Here are a couple of citations to help with panel data:
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 93

http://www.sciencedirect.com/science/article/pii/030440769401645G

http://www.sciencedirect.com/science/article/pii/S0304407610000825

While I was not clear, this is what I was suggesting with the random draws for one choice, but

did not specify the bivariate probit, which I should have. Woolridge has approved using this

approach for a paper I am currently working on.

Kevin Boyle’s response to John Whitehead comment::

This is the same as my proposal with the exception of randomly drawing one question rather than

just focusing on the first question. The random selection is the approach Woolridge

recommended to me. We did 500 draws and found no evidence of selection.

Doing you suggestion and the random draw would be a good robustness check.

Diane Dupont’s response to clarification questions:

1. I think the use of the inverse hyperbolic sine function is a good one. For values in the

"middle of the distribution", it pretty much can be interpreted like a log function but is

also defined at zero. Here are two further articles that use it and support its use.

John B. Burbidge, Lonnie Magee and A. Leslie Robb, 1988. "Alternative Transformations to

Handle Extreme Values of the Dependent Variable" Journal of the American Statistical

Association Vol. 83, No. 401, pp. 123-127

MacKinnon, James G & Magee, Lonnie, 1990. "Transforming the Dependent Variable in

Regression Models," International Economic Review, vol. 31(2), pages 315-39, May.

2. I do not know of any published articles using Heckman type corrections for the type of

multinomial model being used in the EPA study. I think Whitehead's bivariate approach

is a feasible alternative.

Jim Kahn response to clarification questions:

Sorry I am late in replying. I think Diane's suggestions are very good and do not have anything to

add beyond this.

Dan Phaneuf’s response to clarification questions:
316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 94

1. I think the IHS transformation would be fine - better than a small adjustment, since this

would cause the transformed value to stretch large negative. If there is to be an

adjustment I think it would be better to add one - i.e. log(fish_saved + 1) - since this

would just be a horizontal shift if the axis, preserving the value of zero when the designed

level is zero. If the IHS and +1 shift are similar the latter might be preferred on the basis

of simplicity, though the former is more elegant.

2. It is correct that Heckman style corrections are not worked out for non-linear models like

the logit. The Whitehead approach seems sensible, though I would defer to the others on

this topic.

John Whitehead’s response to clarification questions:

I think Diane's suggestions are very good and do not have anything to add beyond this.

(Dr. Whitehead added a few days later)

I might add that to simplify the task a bivariate probit could first be estimated on the first

question only for each respondent. That would allow EPA to avoid the scary Woolridge panel

stuff in the initial test for sample selection. If there is no sample selection then it might be

reasonable to assume that the benefit estimates are not severely biased. If there is sample

selection then adjusting the estimates might require some scary Woolridge panel selection

model.

JohnWhitehead’s response to Kevin Boyle’s comment:

Ah, now I get what you meant by random selection. I thought it was random selection of the

unchosen alternatives (to get a binary dependent variable) with the panel still included.