# Peer Review Report 316(b)Stated Preference Survey Report Document (Final Submission)

Prepared by

Applied Planning Corporation EPA Contract: EP-H-000334

# Table of Contents

PURPOSE OF PEER REVIEW .................................................................................................. 1

SUMMARY OF EPA VALUATION STUDY............................................................................ 2 Benefit Estimation ....................................................................................................................... 2 The Choice Experiment............................................................................................................... 3

PEER REVIEW ............................................................................................................................ 6

SUMMARY

................................................................................................................................. 6

REVIEWERS’ GENERAL COMMENTS ................................................................................ 8
REVIEWERS’ RESPONSES TO CHARGE QUESTIONS .................................................... 9 Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values? ..... 9
Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature? ................................................................................................... 12
Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation ............................................................................................................................. 14
Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes? .................. 16
Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates? ......................................................................... 17
EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior? ....................... 19Is the nonresponse study design adequate to evaluate the potential for non-response

bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature? ................................................. 21
Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses? ............................................................................................................................ 23
What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule? ...................... 25
Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner? ................................................ 27

SPECIFIC COMMENTS ON REPORT................................................................................. 29

ROADMAP FOR THE EPA ...................................................................................................... 34

Estimation Robustness ............................................................................................................. 34
Sample Selection and Non-Response Study............................................................................ 35
Validity Checks......................................................................................................................... 35 Discussion of Over/Under Estimation of WTP....................................................................... 36
Reporting and Documentation ................................................................................................ 36

CONCLUSION ........................................................................................................................... 36

APPENDIX I ............................................................................................................................... 38 Kevin Boyle .............................................................................................................................. 39
Diane Dupont........................................................................................................................... 45

Jim Kahn.................................................................................................................................. 55

Dan Phaneuf ............................................................................................................................ 65

John Whitehead ...................................................................................................................... 72

APPENDIX II .............................................................................................................................. 80 Responses to EPA Clarification Questions - Boyle .......................................................... 81Responses to EPA Clarification Questions – Dupont...................................................... 83

Responses to EPA Clarification Questions – Kahn .......................................................... 85 Responses to EPA Clarification Questions – Phaneuf .................................................... 87 Responses to EPA Clarification Questions –Whitehead ................................................ 89

APPENDIX III ............................................................................................................................ 91
Clarification Questions ........................................................................................................... 92 Kevin Boyle’s response to clarification question.................................................................. 92 Diane Dupont’s response to clarification questions ............................................................. 93
Jim Kahn response to clarification questions ....................................................................... 93
Dan Phaneuf’s response to clarification questions .............................................................. 93 John Whitehead’s response to clarification questions ......................................................... 94316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 1 I. PURPOSE OF PEER REVIEW
Section 316(b) of the Clean Water Act
The U.S. Environmental Protection Agency is developing rules under Section 316(b) of the Clean Water Act (33 U.S.C. 1251 et seq). Section 316(b) establishes national performance requirements that are designed to minimize adverse environmental impacts caused by cooling water intake structures (CWISs). These structures negatively impact aquatic biota through impingement mortality (where fish or other aquatic life are trapped and killed on equipment at the entrance to the CWIS) and entrainment mortality (where fish and other aquatic organisms are taken into the cooling system, passed through the heat exchanger, killed, then discharged back into the source body). Additional adverse impacts on biota are associated with the operation of CWISs such as thermal discharges, chemical effluents, flow modifications and nonlethal effects of impingement caused by the plants.
Due to these lethal and nonlethal effects section 316(b) requires that CWISs minimize their adverse environmental impacts through reduction of volume, frequency and/or seasonality of water withdrawals in order to enhance ecosystem functions of affected water bodies. The EPA has developed section 316(b) rule in several phases since 2000 to address new and existing manufacturers, power plants and offshore oil and gas facilities using CWISs that withdraw over 2 million gallons of water daily from U.S. waters.
Under Executive Orders 12866 and 13563 the EPA is required to estimate the potential benefits and costs to society of rules it proposes. In order to monetize the ecological gains from the Section 316(b) rule, the EPA requested and received approval from the Office of Management and Budget to conduct a stated-preference survey that included choice questions to estimate the environmental benefits of changes in operations of CWIs. The stated preference (SP) study was designed to estimate marginal willingness to pay (WTP) for selected environmental attributes that would be affected by implementation of the rule. The use of choice questions provides an estimated valuation model that can be used for a variety of implementation procedures associated with the 316(b) rules.
Purpose and Process of the Peer Review
The purpose of this report is to present a Peer Review of the survey design and implementation, data analyses and reporting. Peer review is a process for identifying and enhancing the scientific content of a study, which will enable EPA to understand the confidence and limitations they may place on study results to support policy analyses. The EPA report and associated materials were reviewed by:

 Dr. Kevin J. Boyle, Virginia Tech – Blacksburg, VA,

 Dr. Diane P. Dupont, Brock University - St. Catharines, ON, Canada,

 Dr. James R. Kahn, Washington and Lee University – Lexington, VA,

 Dr. Daniel J. Phaneuf, University of Wisconsin – Madison WI, and

 Dr. John C. Whitehead, Appalachian State University – Boone, NC.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 2 Drs. Boyle and Kahn were part of a 2006 peer review panel that reviewed a stated-preference survey developed in 2005 to assess the environmental benefits of changes in the operations of CWIs. EPA used comments from that review to develop a new survey instrument that is being reviewed here.
The main document the reviewers were asked to focus on is entitled, “316(b) Stated Preference Survey Peer Review Document”. This document includes EPA’s documentation and analysis of the stated preference survey design and analysis of the resultant data. The review panel was also provided a document entitled “Summary of Public Comments on the 316(b) Stated Preference Survey,” which contains the comments received during the Notice of Data Availability (NODA) public comment period and EPA responses. EPA also provided the following background documents and files to be used as a resource if additional information was necessary for reviewing EPA’s methodology and results:

1. ICR supporting statement 2. Focus Groups Executive Summary Memorandum 3. All versions of the survey instrument for four regional surveys and a national survey 4. An outline of the quality assurance measures use during the implementation of the survey 5. The Stated Preference Survey NODA 6. The NODA supporting document and Memorandum 7. An Excel file including all data from the main and non-response surveys 8. The Economic and Environmental Benefits Assessment Report for the 316(b) proposed rule
There were three stages of the peer review process prior to the compilation of this peer review report. In the first stage, each member of the panel individually provided written responses to charge questions provided by the EPA. In the second stage, the written comments were shared among the panelists and each reviewer was asked to comment on the comments of other panel members. The third stage was a conference call with the peer review panel organized by Mr. Peter Chaikin. After the conference call, this peer review report was written summarizing the panelists’ comments on the EPA report. This document was reviewed by the panel members before submission to the EPA. We will refer to this process as the Applied Planning Corporation peer review.
This peer review is intended to assess the work that EPA has completed to date on the 316(b) stated-preference research. The peer reviewers were asked to identify areas where modifications to EPA’s estimation and validation approaches would improve the accuracy and precision of willingness-to-pay estimates.

II. SUMMARY OF EPA VALUATION STUDY
Benefit Estimation316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 3 EPA’s benefit analysis for the proposed 316(b) rule includes estimates of changes in use values of commercial and recreational fisheries, and a partial estimate of changes in non-use values (USEPA 2011a). 1 Non-use values are values that people may hold for an environmental improvement that are not tied to any direct observable use of the resource such as recreational fishing.
After the proposed rule was released, EPA conducted a stated preference study to estimate total (use plus non-use) benefits to the public places of ecological improvements arising from regulating CWISs. The use of a SP survey reflects recent EPA guidelines for benefits analysis (USEPA 2010, p. 7-41) that recognize the “advantages of [stated preference] methods include[ing] their ability to estimate non-use values and to incorporate hypothetical scenarios that closely correspond to a policy case.” It is this stated preference study for which this document provides a peer review.
The objectives of the SP study were to “explore how public values (including non-use values) for fish and aquatic organisms are affected by I&E mortality at cooling water intake structures (CWIS) located at existing 316(b) facilities, as reflected in individuals’ willingness to pay for programs that would prevent such losses” (USEPA 2011, p. 30).
The Choice Experiment
Stated preference surveys ask individuals to make choices from which researchers elicit information to estimate individuals’ values for specified changes in an environmental amenity and are typically framed in terms of estimating willingness-to-pay (WTP). WTP is the maximum amount of money that an individual or household will pay for a specified environmental change and respondents express information to estimate their WTP through choices over policy options. Advantages of choice-based questions include similarity to referenda or market choices where individuals are familiar with choosing among alternative policy options or commodities at specified costs (Freeman 2003). Carefully designed SP questions include features to reduce hypothetical and other possible biases that can result from asking survey questions versus assessing WTP through market transactions or binding referenda.
The 316(b) survey was designed as a choice experiment following established procedures (Adamowicz et al. 1998; Louviere et al. 2000; Bennett and Blamey 2001; Bateman et al. 2002). Choice experiments are a SP technique in which people’s values are estimated based on their choices over a set of hypothetical alternative states defined by attribute levels which may or may not contain policy options. Respondents are presented with a set of multi-attribute alternatives and asked to select their preferred alternative. This format has been applied to assess WTP for ecological resource improvements similar to those at issue in the 316(b) policy case (e.g., Bennett and Blamey 2001; Hanley et al. 2006a, b; Hoehn et al. 2004; Johnston et al. 2002, 2011a, b; Milon and Scrogin 2006; Morrison and Bennett 2004; Morrison et al. 2002; and Opaluch et al. 1999).

1 The Environmental and Economic Benefits Analysis (EEBA) for the proposed rule is available online at http://water.epa.gov/lawsregs/lawsguidance/cwa/316b/upload/environbenefits.pdf

.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 4 An example of a choice question used in the 316(b) Northeast survey is the following:

Respondents considered Policy Option A, Policy Option B, and No Policy (current situation) and chose choosing the option that they preferred. The attributes and their levels in each choice option reflected a feasible outcome under alternative 316(b) regulatory scenarios. Respondents were asked to answer choice questions that differed in the levels assigned to the attributes.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 5 This SP study was designed as a mail survey sent to households in different regions of the country and to a nationally representative sample. The target population for the SP survey was all households in each region or from the continental U.S., and respondents from each household were 18 years of age or older. Alaska and Hawaii were excluded because they include only four in-scope non-recirculating facilities, represent a small percentage of overall household population, and are separated geographically from the states in each survey region. The population of continental households was stratified into four survey regions (Northeast, Southeast, Inland, and Pacific) and each region had a version of the survey that presented attribute levels specific to it. These regions were defined by state boundaries and differed from the 316(b) benefits regions used in the Environmental and Economic Benefits Analysis (EEBA) for the proposed rule. The national survey presented attribute values for all U.S. waters. The surveys were allocated across the four regions based on the number of households in each region relative to the total number of households in the continental United States. In addition, a minimum number of completed surveys were required for each region. The survey instrument was designed to maximize the response rate and procedures were employed to identify potential non-response bias.
In July 2011, The Office of Management and Budget (OMB) approved implementation of the Northeast survey version as a pilot study to inform survey design prior to full implementation in all regions and the national sample. Implementation of the remaining survey versions was approved in November 2011. EPA mailed the survey to 7,840 households in total, and received a total of 2,313 completed surveys with an average response rate of 32% across the five sampling regions.
EPA also conducted a follow-up study of households who did not return a completed mail survey. The follow-up examined whether non-respondents answered certain questions in a statistically different way from respondents, as well as checked observable socio-demographic characteristics for statistical differences. The follow-up study included a set of key attitudinal questions and socio-demographic variables that are likely to be associated with WTP for reducing fish mortality from cooling water intake structures (CWISs) and improving fish populations and conditions in the affected aquatic ecosystems. EPA implemented the follow-up study using two subsamples: the first subsample received a paper questionnaire via priority mail and the second subsample was surveyed by telephone. Both non-response subsamples were asked the same set of attitudinal and demographic questions.
The survey sections included:

 Relative Importance of Issues Associated with Industrial Cooling Water

. Respondents considered the relative importance of key issues associated with the use of cooling water by industrial facilities.

 Concern for Policy Issues. Respondents were asked questions about the relative importance of different policy issues.

 Relative Importance of Effects. Respondents were asked questions to promote understanding of the metrics included in the stated-preference questions.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 6  Voting for Regulations to Prevent Fish Losses in the Respondent’s Region (or Nationally). The SP questions in this section were the key part of the survey asking respondents’ choices when presented with specific fish-related and other ecosystem resource changes within their region at specified costs to their households.

 Reasons for Voting “No Policy”. This question provided information used by EPA to identify protest responses; however, not all “no policy” responses are protests.

 Respondent Certainty and Reasons for Voting. This section is designed to identify respondents who incorrectly interpreted the choice questions or the uncertainty of outcomes.

 Recreational Experience. This question elicits recreational experience data to investigate if respondent characteristics influence responses to the SP choice questions.

 Demographics. Responses to these questions are used to estimate the influence of demographic characteristics on respondents’ SP choices, and ultimately, their WTP to prevent I&E mortality losses of fish

. EPA’s analysis of the 316(b) SP survey data is grounded in the random utility model of consumer choice (Hanemann, 1984; McConnell, 1990). The use of the random utility model is standard in the SP literature for attribute-based valuation exercises such as choice experiments, and allows the estimation of well-defined welfare measures (i.e., WTP). EPA used regression models (particularly the mixed logit model) based on random utility specification of choices to estimate annual household willingness-to-pay for an improvement in the environmental attribute levels included in the survey (fish saved, commercial fish populations, fish populations (all fish), and aquatic ecosystem condition). EPA used WTP as the value measure in the estimation of the regional and national benefits of the regulatory options presented in the proposed rule.

III. PEER REVIEW

SUMMARY
Overall, the panel felt that the design procedures were consistent with best practices. They felt it was “a strong study of high caliber that will be useful to the EPA now and in the future”. However, they identified several issues of importance that they feel should be addressed to enhance the usefulness of the study results and the overall credibility of the study. These considerations are summarized immediately below.

Survey Design

- Investigate the effects of overlapping attribute definitions by looking for attribute interaction effects and the effects of dropping attributes from the estimation.

- Due to the survey design, focus solely on the “fish saved” variable in the policy analysis due to the overlapping definition with attribute definitions.

Sample Selection and Non-Response Study316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 7 - Discuss why the variables used in the sample-selection analyses are relevant and would impact WTP.

- Provide a breakdown of the responses to specific survey questions such as “how important is the protection of aquatic ecosystems to you” by the socio-demographic variables (gender, race, income, etc.) that were found to differ between the main samples and non-response samples.

- Conduct more statistical tests on the effects of sample selection and on the assumptions imposed in the sample selection process.

- Address whether the weighted model should be used for benefit estimation and applied to all people in the sample frames, or if the unweighted model should be used and applied only to the proportion of the population consistent with the survey response rates.

Reporting/Documentation

- Enhance the discussion of motivation for the overall study, as well as the motivation for the steps taken in each subsection.

- Provide a stronger justification for the procedures that are used.

- More thoroughly discuss the implications of analysis procedures on estimated WTP.

- Emphasize regional results as opposed to national results.

- Include a new section on the over and underestimation of WTP integrated into uncertainty analyses to clarify the directional effects of potential biases. Specific examples include:

o Address potential confusing and overlapping attribute descriptions.

o Map used in the survey may give respondents the impression that they are located near an affected area and that the effects are widespread geographically; however, the wording in the survey does not support such a conclusion.

Robustness Checks

- Investigate additional specifications such as conditional logit and multinomial logit models.

o Simple conditional logit model could be used as a baseline

o Multinomial logit models will help consider the robustness of the mixed logit assumptions.

- Report how estimates change for the mixed logit model under different assumptions.

- Investigate pooling regions which would allow for more split sample analyses.

- Investigate nonlinearities.

Validity Tests

- The external scope tests conducted are not convincing; conduct external validity tests that include responsiveness to scope with a split sample survey design

- Report statistical tests for the analyses conducted rather than only reporting summary statistics for data splits.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 8 - Use analyses and information from the focus groups, if possible, to address some of the peer-review comments and industry comments on survey design and interpretation of statistical results.

The panel felt that the survey design, data analyses and reporting supported the use of estimates of willingness to pay in benefit analyses of the 316(b) rule. The comments are intended to inform the uncertainty analyses of the study to enhance the clarity and validity of study results.

REVIEWERS’ GENERAL COMMENTS
Some of the panel members elected to make an overall statement prior to expanding on each of the Charge Questions, while others choose to forgo an opening statement. The statements of those members who chose to provide overall comments are found in this section below.

Kevin Boyle
“This study uses design practices that are better than is standard in the literature, and the data analyses follow standard practice. As with any study, in hind sight I have identified components that could have been designed better. These issues cannot be addressed at this point, but potential concerns need to be considered in the computation of aggregate benefits. There are elements of the econometric analysis and documentation that need to be addressed and can still be improved before the final computation of benefits.”

Jim Kahn
“The first thing I would like to point out is that researchers make a large number of choices when researching any topic, but this is particularly true with choice-modeling experiments. It is always possible to second-guess these choices and criticize the researchers for choosing A instead of B, when, in reality, both A and B have merits and have flaws. There is no such thing as a perfect choice-modeling experiment. The important questions are whether the survey was conducted with best practices in a manner in which is unlikely to have significant biases, and whether the survey was conducted in a fashion in which the confidence intervals of the value estimates is likely to be tight enough for meaningful policy analysis.”
“I would like to say that I have a great deal of confidence in this survey. First, it was conducted by two consultants who are truly experts in the field. Second, when they encountered challenges (proper definition of attributes, nonresponse issues, etc.) they addressed these challenges with vigor and rigor. Third, and most importantly, the researchers did not try to overreach, but measured those values that could be most confidently measured. Finally, the documents explain complicated issues in a manner that can be understood by people that are not experts in the field. After securing permission from EPA and the authors, I would like to use this study in my teaching.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 9 John Whitehead
“My specific comments come off rather strong in my review of the document, but I believe that there is a lot of good work here that is useful to EPA.”

REVIEWERS’ RESPONSES TO CHARGE QUESTIONS

1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values?
Overall, the panel agreed that the choice experiment survey format conformed to best practice in obtaining use and non-use values for the evaluation of rule 316(b), particularly for the “fish saved” variable. However, they also agreed that more work is needed to make clear that the WTP estimates of non-use value are valid and reliable. Suggestions for additional analyses are included throughout responses to subsequent Charge Questions.

Kevin Boyle
“Yes, this approach is in common use in the economics literature for estimating uses and nonuse values for changes in natural resources. Applications have been published in the peer-reviewed literature, and there is a substantial literature investigating the validity of choice experiments (also known as choice modeling, choice questions, attribute based questions, conjoint analysis, and other labels).”

Diane Dupont
“With specific reference to this choice experiments survey, the format is appropriate insofar as the goods/services being valued are clearly understood by the respondents. After reading the survey materials, I believe that the researchers are able to obtain a total WTP for the characterized polices and that this total WTP will contain both use and non-use values. I should also note that CVM, the main competitor question format, would not be as amenable to finding both use and nonuse values. I am less certain about the ability of the survey to separate out use from non-use components. Focus group comments suggest that respondents may believe in a different type of implicit relationship between the various attributes than the one presented. This is important because it may be that, when they see ecosystem values rise in a particular choice task, they may infer that this will have some additional favorable impact upon another attribute of interest (commercial fishing, for example). So, they infer a relationship between the attributes and not treat them as separate. I am concerned that this might result in some kind of double counting in the Total WTP, in particular, when people assume a very strong complementary relationship between use and non-use components.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 10316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 11 Jim Kahn
“Choice modeling is not only an appropriate means to estimate total willingness to pay for these types of policies, but it is the best method for several reasons. First, when choice-modeling has been used in potential new product applications (called conjoint analysis in this literature), hypothetical choices have been very close to actual choices. In contrast, contingent valuation has been notoriously unsuccessful in having hypothetical choices match actual choices, even with market goods. Contingent valuation represents a good choice for simple problems with limited budgets. In the case of the 316b regulation, which is complex in all dimensions, choice-modeling is the only choice. Fortunately, EPA committed sufficient resources to allow a state of the art study to take place and the greater expense associated with the choice modeling exercise was not an issue.”
“Choice modeling is the best method for modeling both direct use and non-use values, although it is often difficult to conduct an experiment which allows the researcher to produce separate estimates of direct and indirect use values. However, if the policy question requires an estimate of total value (rather than separate estimates of direct and indirect use values) this is not an issue and the value estimates can help inform the policy decision process. Nothing has happened to change my 2006 conclusions that choice-modeling is the appropriate direction to go to understand the benefits of the proposed rule-making.”

Dan Phaneuf
“Stated preference methods in general are flexible enough to estimate both use and non-use values. The choice between contingent valuation and choice experiments in this regard is not about which one can measure which value. Rather, the choice reflects the needs of the application. When the objective is to understand how multiple, different attributes affect value – regardless of the type of value – a choice experiment is the appropriate choice. Given the importance of the multiple ecological outcomes in this application a choice experiment was a sensible approach.”

John Whitehead
“A stated preference discrete choice experiment will elicit total willingness to pay (WTP). However, more work is needed to make clear that the WTP estimates reported in the EPA report are valid and reliable estimates of nonuse value. Since Krutilla (AER, 1967), nonuse values are often associated with unique natural resources facing irreversible development. Both uniqueness and irreversibility are subjective and relative concepts and should be considered as sufficient but not necessary conditions. Where the resources are not unique (i.e., many substitutes) and development is not reversible (i.e., the species are not threatened or endangered), more effort is needed to justify the use of nonuse values in benefit-cost analysis.”
“For example, Krutilla described the motives that are likely to lead to nonuse values. The survey gathers information on motives for nonuse values to preserve fish for recreation (evidence of an altruistic motive for those who do not fish), preserve fish for aquatic ecosystems, wanting to know that fish exist in local lakes, rivers and bays, and preserve fish for future generations316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 12 [question 9]. The report states that most of the respondents had some motive for nonuse values but this only provides weak evidence that nonuse values are valid. If the responses were analyzed out of the RUM framework so that an index of motives was a right hand side variable, the WTP model could be used to extract motivated nonuse values from non-motivated values.”
“On page 17, the first paragraph illuminates a potential problem with the analysis. It is clear that survey respondents were not likely to have any prior information about the issue before they took the survey. It is well-known from the CVM literature that survey respondents can have constructed preferences in a no prior information context and that WTP estimates from constructed preferences are less valid than preferences based on real experience. Without the prior information question it is impossible to know whether there is a subsample of invalid WTP estimates.”
“For these reasons, there is some concern that the nonuse component of the WTP estimates may lack validity relative to the use value component and the WTP estimated with the full samplemay be aggregated over too large a population. The analysis could be conducted with splitsamples of users vs nonusers (there is no question asking about prior information) and those with significant nonuse value motives vs. those without to demonstrate the validity of the WTPestimates of those respondents with questionable rationales for valuing these resources. If subsamples are found to have invalid WTP responses then in a sensitivity analysis, WTP should be aggregated over only the sample proportion with valid values for benefit-cost analysis.”

2. Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature?
The panelists largely agreed that the procedures used by the EPA to develop and test the proposed survey instruments were consistent with the industry standard and followed best practice. They believed that the use of focus groups was a strength. However, they also commented that the procedures could be flushed out more in the report as some aspects were unclear. For example, there was minimal discussion in the report regarding how the pilot study in the Northeast informed the design and implementation of surveys in other regions.

Kevin Boyle
“Using focus groups, cognitive interviews and pilot studies is consistent with the very best practice for designing survey instruments. The investigators conducted more focus groups and cognitive interviews than is typical; the work here is consistent with the number conducted for high-profile litigation applications. The report mentions the pilot study, but provides no explanation comparable to the discussion of the focus groups and cognitive interviews. It is unclear if the Northeast survey implementation is the pilot or if another pilot was also conducted. The authors mention that the Northeast survey was implemented to inform the other survey applications, but no mention is made of the specific goals of this effort or if the other surveys were changed from what was learned from the prior implementation of the Northeast survey instrument. These latter comments are not intended to be a critique of the survey design process; rather they are a comment of a lack of clarity in the documentation contained in the report.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 13 Diane Dupont
“In general the procedures followed are consistent with the industry standard (focus groups in different regions, one-on-one cognitive interviews, peer review, testing of specific wordings, use of graphics, investigation of possible use of heuristics, etc). My concerns are the following:

- As far as I can see, there was no pre-testing on a larger scale in the field (actual mail survey pre-test); only focus group testing within a controlled setting.

- None of the focus groups took place in the Pacific Region. Given the weaker results from the models using data from this region, this is a shortcoming.

- It is not clear whether the national versions were pre-tested similar to the regional versions. This is important to develop the salience of the attributes at a national, as opposed to more personal, regional scale.

- It was not clear from the executive report discussion about the focus groups whether other attributes were salient to focus group respondents. I am wondering in particular about the thermal discharge. On page 2 the survey talks about how pumping warm water back into the environment affects ecological systems. Then, later on page 5 there is a discussion about how closed cycle cooling may reduce fish losses and also thermal discharge. Was this an attribute that was of concern or interest to focus groups? The thermal discharge component jumped out at me. If it was not an issue in focus groups, then perhaps it is worthy of a footnote.”

Jim Kahn
“EPA contracted with two of the most experienced and well-respected researchers in this area. All the relevant literature is cited, and at each point where problems are encountered, the solutions are consistent with the literature. I found the discussion of the test surveys and focus groups to be very useful...I found the focus group comments on the utility of the graphics to be reassuring. I also thought that the survey was visually appealing and was easy to read even with the view scale set to “actual size”. Color contrast on the survey made it easy to read and less fatiguing to the eyes.”

Dan Phaneuf
“The survey development process seems to be one of the strengths of this study. The number of focus groups and amount of pre-fielding testing exceeds what one usually finds in the published literature. As such I judge the procedures to have conformed to current best practice.”

John Whitehead
“The EPA’s survey design methods were thorough and state of the art.”

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 14 appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation?
The largest complaint about the survey booklet was that the wording was confusing in some places and that attributes were overlapping. (Overlapping attributes could cause problems if the EPA focuses on an outcomes other than “fish saved”.) Several panelists also mentioned the lack of cheap talk and budget reminders in the booklet.

Kevin Boyle
“For the considerable effort invested in the design of the survey booklet, I found the wording in the survey rough and confusing in places. I do not know how this affected the survey outcome in all instances, but it does leave unresolved questions. Some of these clarity questions might be answered by knowledge gained from the focus groups and cognitive interviews that is not easily conveyed in the report. However, the roughness of the descriptions and overlapping categories is something that I would expect to have been refined and resolve in such an extensive design process. I also did not see a clear budget reminder in the survey wording, but respondents were told that the price increase was “unavoidable” (Northeast survey, p. 7, bottom). There was not a reminder of substitutes and there was no cheap talk script. While I am not convinced of the appropriateness of cheap talk scripts, they have become common practice in stated-preference surveys and their presence does serve to reduce value estimates.”
“Consequentiality was addressed in the survey, but not as clearly as it could have been. The survey does state that respondent answers will help to make policy decisions (Northeast survey, p. 1, top), but this is not repeated immediately before the choice questions nor is it specifically stated that survey results “will be used” in making policy decisions. Respondents are asked to answer as if the questions were real and binding (Northeast survey, p. 10, bottom), but are not asked to answer as if they are voting in a real referendum at the moment they answer the choice questions. Thus, there are elements of consequentiality, but they are not as poignant as they could have been.”

Diane Dupont
“The overall design and quality of the survey booklet is impressive and enhances the saliency of the exercise. I generally found the wording to be unbiased with one exception in the discussion on page 7 that describes the attributes where I have some concerns about overlap of attributes. I also feel there is a lack of clarity at some points in the information screens. The budget constraint was addressed but not really emphasized. However, I think the choice of higher electricity prices is a good one and believable. On another note, the policy effects described to people relate to improvements that will be felt within 3-5 years; however, the survey left unstated when household cost increases would begin. I did not see any question in the survey designed to elicit time preference information nor did I see a question designed to elicit people’s beliefs about when they would start paying these higher costs. Different people may be applying very different discount rates to the household cost attribute.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 15 Jim Kahn
“I thought the survey booklets were informative without creating biases. The valuation problem in this case is complex, and technical information must be provided on how environmental impacts occur, what the environmental impacts are, and how policy actions can reduce the impacts. The survey does an excellent job of this which is particularly important with respect to the quality variables. The information in the survey booklet helps the respondent think about how quality can be influenced by the proposed policy remedies without inserting a particular set of values into the information. In particular, the treatment of option C, the status quo, was well constructed in the survey compared to many choice-modeling exercises where the status quo is ambiguous. Although the preliminary information was a bit on the long side, the researchers compensated for this by only asking the respondents to evaluate three choice sets, rather than five or six which is usually the case in choice-modeling. For this reason, it is unlikely that respondent fatigue set in before the choice sets where completed.”
Dan Phaneuf
“I generally find the design of the survey to be adequate for the task. The systematic way in which the study is able to directly map between policy objectives, the definition and levels of the stated preference attributes, and household preferences is a clear strength of the study and outweighs the concerns mentioned above. I find the description of the environmental problem to be suitably neutral in tone. I do have several small negative comments regarding the design of the survey booklet as well as the potential for strategic behavior, which I do not believe affect the validity of the survey on net. I mention them mainly for completeness.”
“I also have a few observations on efforts to minimize hypothetical bias. On net I am inclined to think that the importance reminders, general professionalism of the survey, and complexity of the issue led people to perceive the survey as consequential. Two other points regarding hypothetical bias relate to the payment vehicle. As best I could tell the survey did not contain the usual reminder for people to consider their budget constraint, what they would give up, etc. when answering the payment question. I think it would be useful to highlight some focus group or cognitive interview evidence that people accepted this payment vehicle as realistic, and were considering their budget constraints when answering.”
“Finally, I think the survey did an adequate job of including rationality checks. Questions 7 and 8 look like useful diagnostic vehicles, and might perhaps be useful for some estimation tasks.”

John Whitehead
“The overall quality and design of the survey booklet was “adequate” but some features of a stated preference survey were not included. Respondents were told to “answer all questions as if this were a real binding vote.” Respondents were told how the cost of the program would affect their budget (in higher prices, etc.) but not reminded that this money could have been spent on other things, which is a typical budget reminder. Respondents were not asked directly about the consequentiality of their votes (i.e., did they think that their votes would be used by policy makers; Carson and Groves, Env and Res Econ, 2007). Respondents were asked for confidence316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 16 in their answers and whether they would vote the same way in an election, but these responses are not used in the empirical analysis.”
“A split-sample analysis of confident vs. not confident respondents and same way voters vs. those who would vote another way would shed some light on this issue. If the split-sample models are similar then the EPA’s approach is not problematic. If WTP has little validity for respondents who are not confident about their responses or who would vote a different way, then the EPA’s aggregation approach is overly optimistic.”

4. Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes?
The panel agreed that the EPA should only focus on the “fish saved” attribute in the analysis as this was the most robust outcome measure due to the overlapping definitions of attributes. The panelists also largely agreed that the EPA could conduct more robustness checks through additional statistical analyses (e.g., conditional logit), categorical variables and split samples to increase the quality of the analysis. They also agreed that the analysis should focus on regional rather than national results.

Kevin Boyle
“Yes, if and only if the policy just focuses on one attribute (fish saved) and the linearity of this variable in the estimated equations is confirmed by more rigorous investigation. To investigate the robustness of models to support value estimation for a one percent change I suggest estimation where each attributes is identified using effects coding to investigate linearity instead of the quadratic specification. The quadratic models seems odd at best given the small ranges of levels posed for some attributes (4-6%) and the large jumps (coarseness) for the key fish saved variable (report, p. 15, Table 3-1).”

Diane Dupont
“I am concerned that the scenarios presented have no instances of increases in some attributes and decreases in others relative to the status quo. This may give rise to less robust estimates if respondents find that scenarios do not fit with preconceptions. Furthermore, the design uses only attribute main effects. According to Carson and Louviere Environmental and Resource Economics, 2011), in such a case the researcher must assume that the indirect utility function is strictly additive for all respondents. This worries me given my concern about the (possibly varying) implicit relationships respondents may believe exist between the use and nonuse attributes described in the choice tasks. Given the complexity of the problem, I would like to have seen something other than a main effects design. Finally, within each regional (and national) survey the same sample question is used for each booklet but then the first choice task mostly contains different values from the sample values. Do people treat the sample as a base or reference set of values against which they compare values in the subsequent questions? This may induce a kind of loss aversion response. More generally, is there some kind of ordering or comparison of values in earlier and later questions (since the ordering is fixed for each booklet and not randomized across respondents).”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 17 Jim Kahn
“I am fairly confident in the robustness of the estimates for several reasons. First, the attributes were constructed in a fashion where they could enter the logit estimation as continuous variables rather than categorical variables. The continuous form of the attribute levels allows for a test across policy options that would be much more difficult in the presence of categorically defined variables. Nevertheless, I also think that the models should be estimated with categorical variables as another robustness test and to search for nonlinearities. Second, for the most part, the coefficients are very similar across the different regions and any significant differences can be accounted for by region-specific characteristics in the type of fish. One critical aspect to assure the validity of a choice-modeling study is to choose an appropriate set of attribute bundles to compare in the choice sets. The EPA employed a fractional factorial experimental design, eliminating those comparisons that might lead to protest responses or puzzlement. The definition of the 72 alternative sets of comparisons follows the literature, and sample size was appropriate for the number of choice sets evaluated. The 72 sets were appropriately distributed across the alternative versions of the surveys in each region.”
Dan Phaneuf
“I think the experimental design is fine conditional on the decision to use three levels for the important attributes. In this sense I believe it was the correct decision to focus on a main effects design so as to avoid the need to ask respondents additional questions (a larger sample or more responses per person would be needed to estimate interaction effects). The one point of criticism I have relates to the use of only three attributes for fish saved, and perhaps for the other ecological indicators. I would expect there to be a non-linear relationship in marginal utilities for these outcomes. While three levels is in principle enough to trace a curve it can be difficult I practice, as was shown by the effort to include a quadratic term in the models. I think this feature of the design can be overcome with some additional analysis, as I discuss below.”

John Whitehead
“Yes, this is an important feature of a discrete choice experiment.”

5. Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates?
The Panel agreed Sample sizes were consistent with experimental studies published in the peerreviewed literature, but are less than expected for a policy study of this magnitude. Members of the panel were concerned about the representativeness of the national sample and recommends that the EPA focus on the regional results. Among the regional samples, they were also concerned about the Pacific sample as it has the smallest population and the largest number of undeliverable surveys. Questions were also raised regarding the number of males who answered the surveys compared to females.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 18 Kevin Boyle
“The sample sizes are acceptable minimums, but below what would be considered best practice for a study of this magnitude to support a major policy decision. Sample sizes as employed in this study are consistent with experimental studies published in the peer-reviewed literature, but are less than one would expect for a major policy study. The response rates are consistent with what is observed in the literature for best practice applications of choice modeling. The nonresponse follow-up survey is consistent with best practice. I wish the authors has discussed whether they believe the identified effects were relevant for value estimation; no intuitive discussion was provided or insight.”

Diane Dupont
“The rationale behind the sample sizes is sound and the final response rates are just adequate. However, I have some related concerns. It looks to me as if there is a problem with the use of postal delivery addresses to find respondents since the Pacific region has the smallest population but the biggest number of undeliverable surveys. This calls into question whether the Pacific sample is less representative than the others. In terms of demographics, males are systematically over-represented. This makes me think that there was some kind of sample selectivity that took place since the survey was a mail survey. It is possible that women gave the survey to their male partners thinking that it was about fish and of more interest to the males. This could be checked by looking at whether female respondents in the sample were more likely to be living in one person households. I would also like to see a breakdown of the responses to the question... ‘howimportant is the protection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and nonresponse samples). This would assist in better understanding whether the samples provide robustestimates. I find the Inland sample is problematic in that it contains very different sociodemographics from the non-response sample.”

Jim Kahn
“The implementation of the survey involved no shortcuts deviating from the original plan. The sample sizes and response rates meet or exceed the guidelines established by the peer reviewed literature and the federal government. Response rates were adequate and potential biases associated with non-response were appropriately treated with the weighted model (see response immediately below).”

Dan Phaneuf
“I found the arguments used to decide on the original sample convincing, and I think the number of completed surveys received ex post is adequate for the statistical task. I don’t have further comments or complaints on this aspect of the study.”

John Whitehead316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 19 “The regional sample sizes are adequate to obtain statistically significant coefficient estimates especially with discrete choice experiment data where repeated measures are used in a mixed logit. It is difficult not to produce statistically robust estimates with this sort of data and empirical model.”
“It seems odd that more resources were not devoted to the national survey which has a sample size too small to represent the national population. It is also too small to obtain WTP estimates with confidence intervals tight enough to adequately compare to the weighted regional WTP estimates.”

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior?
While members of the panel believe that the specification used by the EPA is strong and convincing, they recommend that additional specifications also be estimated to increase the robustness and validity of the main model results. With regard to the sample weights, most of the panelists supported the effort to use the weighted model. However, the EPA could address whether the weighted model should be used for benefit estimation and applied to all people in the sample frames, or if the unweighted model should be used and applied only to the proportion of the population consistent with the survey response rates.

Kevin Boyle
“I recommend discontinuing the use of the term RUM. While the model is based on random utility, this is a total value study and RUM in the literature refers to a specific type of travel-cost model that just estimates use values. Thus, I fear the use of the term RUM might cause some to misconstrue the values that were actually estimated. Furthermore, I do not think the quadratic specification is reasonable. As noted above, I suggest re-estimation where the levels of each attribute are dummied out through effect coding to provide a more robust test of the linearity assumption for the fish saved variable. As for the use of weighted models, I believe that this is consistent with best practice when survey data do not match known population statistics. On another note, not knowing how other variables are coded such as the ASC make me uncomfortable fully supporting the interpretation placed on it.”

Diane Dupont
“I support the effort to use the weighted model results, particularly given the significant demographic differences between the main sample and the non-response sample, as well as the region-census level data. I think this is the appropriate approach to use for estimating the welfare316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 20 changes associated with the expected changed in survey attributes under the 316b regulation. However, a simple CL model would also be informative. Given the concern for estimates across the different regions, I think the authors ought to include some measure of distance decay in their models because I am concerned about the likelihood of systematic differences in responses from respondents living close to the facility points on the maps provided in the booklets. Furthermore, the report notes that demographic interactions between ecological attributes and gender and education were included in a preliminary Northeast model. I would like to see a similar model with interactions using the Inland data set where most of the demographic variables differ from the non-response sample. As the models currently stand, I am less certain that the estimated WTP values for the Inland region are representative of average WTP since it is not obvious that the sample is random. I would like to see a breakdown of some of the debriefing responses on the importance of the different attributes by gender and education.”

Jim Kahn
“This is an important question, as empirical results are often dictated by the choice of functional form. Given the non-response bias which, if uncorrected, could lead to biases, the weighted linear model is the most appropriate functional form for calculating values associated with marginal changes in the attribute levels. Even though the EPA is not using the values calculated from the unweighted linear models, it was appropriate to show those values in the tables so the reader could compare these values to the values based on the weighted regressions. It is certainly appropriate not to calculate values based on the quadratic regressions, as the coefficients on the quadratic term do not conform to economic theory. Furthermore, the choice of mixed logit models to deal with heterogeneity in the sample is a well-justified choice. In many other dimensions, the mixed model is less restrictive than traditional multinomial logit. Similarly, the mix of random and fixed coefficients in the estimation represents valid choices given the data set that was developed.”

Dan Phaneuf
“There are several things I might have liked to have seen as robustness checks, before concluding that the weighted/linear/full covariance/MXL model should be preferred. First, I like the mixed logit model and as such I agree with the decision to use it here. However, seeing how other models perform would give me a better sense of what the mixed logit specification is buying such as the simple conditional logit model. While it may be only personal opinion, I don’t think it provides enough robustness checking to only present on class of model (particularly one that might be prone to some numerical anomalies).”
“Second, I would like to see additional specifications. Looking at models with dummy variables for different non-price attribute levels (rather than making them continuous) would help give a sense of the types of non-linearity might be present. While such a specification would not meet the policy needs directly it would help determine the extent to which the linear model is a good approximation. It would also be a good way to test for internal scope consistency – i.e. themagnitudes on the estimates of the dummy coefficients should increase. Also related to nonlinearity I think it would be worthwhile to look at a model that uses logs. This would allow for a smoothly diminishing marginal utility of the 0 to 100 coded attributes, which I think is316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 21 important.”
“Finally, as I describe in the next question (with caveats), I found the use of sample weights convincing. My advice would be to focus any additional estimation on the weighted sampled generally, though for the conditional logits or other simple specifications I would present both.”

John Whitehead
“Using the weighted linear models seems appropriate and is adequately justified. I know of no other functional forms that might outperform the linear and produce values with diminishing marginal utility. But I wonder if this goal is worthy? It may be that there is no diminishing marginal utility within the narrow range of policy impacts proposed by this survey for attributes other than fish saved. And constant marginal utility is not unlikely for fish saved. Economic theory does not have much to say about the demand that generates nonuse value.”

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature?
The panelists agreed that the non-response study followed best practice in accordance with the literature; however, several commented that clarifications could be made in the documentation regarding this part of the survey. Additionally, answers from the non-response survey could be further broken down, analyzed and used in statistical analyses to increase the robustness of the main results.

Kevin Boyle
“The nonresponse study follows best practice. I would like to have seen the report provide more intuition of whether the significant effects identified really are expected to influence estimated values. I would also like to have seen an investigation of the subjective decisions employed in the development of weights for estimation.”

Diane Dupont
“Since the regional results are being compared to one another I think it is unfortunate that a single weighting method could not be applied to each of the five data sets. However, I think the EPA did the best that could be done with the information available. I would like to see further discussion of the potential implications of these two different weighting schemes. I have two questions about the decision made (on page 47) to exclude respondents from the weighted model estimation if they do not include answers for all variables being used in the weighting. How many respondents are removed from each region and are there systematic differences in respondents that were removed when compared to respondents who were kept in the sample? A final issue arises with the Pacific sample. There was a significant difference between the importance assigned to ecosystems between the main sample and the non-response sample. But,316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 22 an adjustment was not undertaken for this. I understand the concern that this is likely to be an endogenous variable however, this brings up a concern about possible sample selection bias.”

Jim Kahn
“This is an area in which I expressed some concern in the 2006 peer review, and I was glad to see it so prominently addressed in the research, with a dedicated nonresponse study. The methodology used to avoid the lazy husband problem completely negates this source of bias, and it can be determined if cellphone only households, which have certain socio-demographic characteristics, are underrepresented in the survey. I do not believe that this negatively influenced the nonresponse study. With regard to philosophical self-selection, the non-response study indicated that nonresponse was due to people who were less concerned about aquatic ecosystems. I think it is important to see a breakdown of the responses to the question...‘howimportant is the protection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and nonresponse samples). This would assist in better understanding whether the samples provide robust estimates. Some patterns in nonresponses were noticed in relation to certain socio-demographic variables, such as age, education, gender and employment were discovered in the analysis. These formed these patterns and the potential biases that might be generated in the final value estimates generated the need to conduct weighted logit regressions instead of unweighted estimates. The methods that EPA used to create the weights corresponded to best practice in the literature, and should result in estimates that are not biased by patterns of nonresponse.”

Dan Phaneuf
“I found the non-response analysis to be one of the strengths of the study. It is unusual even among the highest quality studies to devote this much effort to analyzing the selection propertiesof the estimation sample. The presence of some statistical differences aside, I found the nonresponse data encouraging in the extent to which it validated the general representativeness of the primary sample. In short I find the data gathering for the purpose of supporting non-response analysis exemplary.”
“I am less certain about the methods used to develop the actual weights. First, there is mention of some observations needing to be dropped (because of missing data) in order to implement the sub-group weighting routine. It would be useful to know if this involved more than just a fewdata lines – i.e. are estimates from non-weighted models that do and do not include the to-bedropped observations the same? Second, the ‘raking method’ used for two of the regions seems a bit speculative. The only citation to it is mention of a SAS macro that implements the process. I think more needs to be said about the general performance and acceptability of this methods vis-à-vis using a not-weighted model.”

John Whitehead
“The response rates are low (albeit, about as high as you can reasonably expect these days) and raise the issue of nonresponse and sample selection bias. Nonresponse bias exists when the sample is not representative on known characteristics. The EPA makes an honest effort to gather316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 23 information on respondents with an overall 25% response rate for the nonrespondent survey, but still manages to obtain information on less than 10% (=619/7118) of nonrespondents. There is evidence in the survey methods literature that shows that response rates are a function of survey salience. The respondents, especially to a mail survey, are those who care most about the subject. Those respondents who care most about the subject are likely to have higher WTP for the policy. One approach is to estimate a Heckman type sample selection model with the samples of respondents and nonrespondents with the dependent voting variable converted to a zero-one variable for a probit. If there is no statistical evidence of sample selection bias then the EPA’s aggregation approach may be appropriate.”
“In the absence of evidence to the contrary and given the possibility of sample selection bias, the aggregation rule, assuming that weighted WTP responses adequately reflect nonrespondent WTP is overly optimistic. It cannot be ruled out that nonrespondent WTP is less than respondent WTP.”

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses?
The panelists did not find the external scope tests used by the EPA to be enlightening. They did offer additional tests that could be conducted to assess validity such as pooling regional resultsand examining split samples. Panelists suggested using focus group comments to address peerreview and industry concerns as well as reporting statistical tests for analyses conducted by EPA.

Kevin Boyle
“Choice experiments, by their design, provide a within subject test of scope. The scope issue in the literature relates to between subject tests and the tests reported do not address this concern.
The scope test stratifying on choices A and B is neither appropriate nor insightful. The groupings do not stratify so different people from the same sampling frame are responding to the alternative percentages. Since the ASCs are not significant it is not surprising that the ASCs for alternatives A and B are not significant. It is good to investigate if there is a leaning toward alternative A or B. I think that the investigation of fish saved with the other attributes is appropriate. I would also suggest estimating the model, always including fish saved and cost, but dropping other attributes sequentially. Given the overlapping attribute definitions and exclusions of extreme combinations suggests that responses to the fish saved attribute may be conditioned on the other attributes. I know this is not the same as dropping attributes from the design, but it is one way to look at this issue. I do not understand why some of these analyses were just conducted with the Northeast data, and not all sample groupings.”
“In terms of the geographic scope showing willingness to pay to be higher for regional improvements than for a national improvement, I find this is not a problem as some regions can be higher than the national; the region vs. national is an apples and oranges comparison and different baselines and preferences can cause this outcome as well.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 24 Diane Dupont
“The external scope test of looking at proportions of responses using split samples seems like a reasonable way to examine the issue. However, no formal testing of these proportions is done and, in some cases, the differences are very small and the standard deviations are very large. I would like to see significance levels from non-parametric Chi-squared tests of differences in proportions (where the proportions are people accepting a particular option). I would also like to see the test done in a way that is sensitive to the question ordering. The only issue that I have with respect to the argument that choice experiments provide their own internal scope test (and so it is not necessary to do) is the following: if a respondent ignores certain attributes in his/her choices, then this non-attendance may mean that internal scope tests for the ignored attributes are not satisfied in a conventional sense.”

Jim Kahn
“First let me say that I thought scope tests were correctly performed, but I would like to point out that scope tests are really a red herring and the scope issues are not as important as some portray them to be. Accidently generating a scope issue through inadequate understanding of survey questions and policy effects can be guarded against by appropriate use of pilot surveys, individual debriefings and focus groups.”
“It is appropriate to test for scope issues, even it is expected that they are not likely. EPA deals with the issue directly with several tests for the problem. The first test is the natural test that coefficients on attribute levels have a positive slope, indicating that people are willing to pay more for higher levels of the attribute. They were not able to demonstrate declining levels of marginal value with the quadratic estimations, but this is not a real problem. The “external validity” test of scope that they initiated was cleverly done, splitting the sample into option a responses and option b responses, and showing that for both option a and option b, the choice set was more likely to be chosen if it higher levels of the attributes.
The geographic scope test showed willingness to pay to be higher for regional improvements than for a national improvement, although economic theory would suggest that the national improvements should have a higher value than the regional improvements. However, this does not surprise me, as people are more focused on improvements in their own vicinity, and this is not a global public good in the sense of reducing global warming or species extinction. These tests were appropriate and well-executed. I do not see the need to conduct additional tests.”

Dan Phaneuf
“As mentioned earlier I would like to see an additional variant on the internal scope test that makes use of dummy variables for the different non-price attribute levels. This will help make more transparent issues of scope and the non-linear relationship between the attribute indices and utility. Specifically on the ‘external’ scope tests I think more can be done to formalize them. The idea seems like a good one, or at least something intuitive to look at. However, the summary statistics that are reported do not provide a way of formally or even informally assessing the statistical presence of scope. I think it would be useful to run a linear probability316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 25 model on the Option A and Option B sub-samples, including dummy variables for each of the attribute levels outside of price (this could stay continuous), or perhaps keeping the other attributes continuous and just checking the different levels of fish saved. This would allow a formal test of whether the coefficients on the three fish saved levels are different and properly ordered.”

John Whitehead
“The external scope tests are not adequate to demonstrate responsiveness to scope, at least according to the external scope test standard set by CVM critics. The EPA report performs internal validity tests using multiple scenarios. This test is relatively weak since respondents need only be consistent within themselves to produce WTP values that increase with policy impacts. An external validity test compares responsiveness to scope with split sample survey design.”
“Within a discrete choice experiment an external validity test can be conducted by using only the first scenario presented to respondents in the RUMs. In effect, the first scenario is a split sample test. The EPA considers the split-sample data in Table 8-10 but only applies the eyeball test. Differences in frequencies across the fish saved scenarios can be tested with a likelihood ratio test. The external validity test should be conducted with the RUM model (e.g., Lew and Wallmo, Env and Res Econ, 2011). Note also that in a mail survey the first response may be contaminated by the following scenarios since respondents could have looked ahead and developed internally consistent answers during the survey.”

9. What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule?
While the panel did find issues of concern regarding the study, they agreed that “no study is perfect” and that the current study is superior to many existing studies. Panelists suggested that the survey content be further developed in the report and that emphasis be placed on regional results. Once additional robustness checks and sensitivity analyses have been conducted, they recommend that the survey be used in the EPA’s cost benefit analysis of the 316(b) rule.

Kevin Boyle
“The design practices are better than what is done in most current studies in the stated preference literature and the data analyses follow standard practice. I am concerned that the survey content is not as well developed as I would have liked and, as presented, might lead to respondent confusions and certain items may have led respondents to value more than the scenarios were intended to value. The current study is superior to the existing benefit estimates, and I would recommend the current study for national policy analysis. No study is perfect and any benefit analysis must take into consideration potential imperfections in benefit estimation. In the current316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 26 analysis the robustness analyses of the survey data analysis are appropriate and can be expanded as noted above. There needs to be more sensitivity analysis in computing the aggregate benefit estimates.”

Diane Dupont
“Compared to the current stated preference literature I believe that the data collected within this survey is capable of providing both valid and reliable estimates for benefits. I place more confidence in results from the regional models/datasets (with the exception of those for the Pacific Region) than from results associated with the National dataset, in part because of the saliency of the issues within a regional context and in part because of the larger (relatively speaking) sample sizes used in the regions. I use the word “capable” in the first sentence because I think the research effort would gain important insights after incorporating distance decay effects, ordering and context effects, and further investigation of the role of differences in demographics, particularly insofar as these differences manifest themselves in expressed opinions on the debriefing questions. I have confidence that the information obtained from the survey can be used to assist in EPA’s CBA analysis of the 316(b) rule. I would recommend undertaking some additional model estimation along the lines noted above in order to verify that validity of the estimates already provided.”

Jim Kahn
“This question is the most important of the entire peer review of the survey and resulting values. I am confident that the research was properly done and conforms to the highest scientific standards. It is not just acceptable, but actually provides a blueprint for others to follow when conducting their own research. The survey provides important information that was not in the original benefits assessment. Given the complexity of the environmental variables, the likelihood that different responses will be forthcoming from different regions and the availability of resources to conduct a state of the art survey, it is better to use measure the values directly than to use benefit transfer. Moreover, I like the way the estimates of the value of fish saved can be used as a lower bound on non-use values. This is a significant addition to the analysis. Since the survey and analysis were conducted with the appropriate adherence to theoretical and empirical precedents, I would enthusiastically endorse using these values in a cost benefit analysis of the 316(b) rule.”

Dan Phaneuf
“I find that the design work and data gathering aspects of the study generally meet the necessary conditions for the resulting data to be valid and reliable. In addition, the raw data summaries (on p. 21) show that respondents did seem to make tradeoffs between money and the non-price attributes. However, I find that the current set of reported results alone are not sufficient to judge if the value predictions contained in the peer review document are themselves of high quality. Thus I see the study as having the potential to be superior to that which was used as part of the existing facilities rule, conditional on some robustness checking and additional analysis. As such I would hold back from making a firm recommendation that this survey be used to inform the 316(b) rule cost benefits analysis, but my expectation is that I would eventually be comfortable316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 27 making such a recommendation. Furthermore, my tendency would be to rely on the regional models for policy purposes.”

John Whitehead
“These estimates are superior to those used in the 2011 report since they cover the entire nation and are based on a primary valuation study. However, in spite of relatively low statistical uncertainty, the WTP estimates are highly uncertain for reasons described above. First, it is not clear if roughly 25% of the sample has valid nonuse values (see #1). Second it is not clear if roughly 67% of the population represented by nonrespondents have WTP values equal to those of the respondents (see #7). If both of these groups of the populations have WTP equal to zero then the WTP estimates could be as low as 25% of those reported (=100*.75*.33).
Unless evidence can be provided that all of the respondents have valid nonuse values (#2) and nonrespondents have similar WTP values to respondents (#7), I would not recommend that the information provided by this survey be used in EPA’s benefit-cost analysis. The aggregate benefit estimates in Table 10-3 could be used if they are described as the best case analysis (instead of the base case) and sensitivity analysis be performed over the potential range of values. Given the current information, a uniform distribution should be used for the range of values.”

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner?
The main suggestion of the panel was to include an additional section in the report that summarized all the uncertainties on the over and underestimation of WTP and direction of potential biases. This also includes addressing potential respondent confusion of attribute descriptions in the survey. It is important that uncertainties are understood before results can be used.

Kevin Boyle
“See my comments above.”

Diane Dupont
“I think the report contains a great deal of information in different locations. The footnotes, in particular, are very helpful to further understanding of decisions made along the way. However, I think the report should collect all various types of uncertainties and potential implications of uncertainties in the analysis and put them together in a single table, so that directions of potential biases could be examined and clarified. Many steps were undertaken and many decisions were made prior to the generation of the WTP estimates in Table 10.2. The inclusion of a table showing the types of decisions that had to be made with respect to uncertainties in modeling, parameter values, model results, estimated WTP values and likely directions of biases would316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 28 help the reader to better understand the implications of the uncertainties.”

Jim Kahn
“Uncertainty is omnipresent in empirical research and must be understood before the results can be used. It is important to discuss likely sources of uncertainty, the magnitude of the uncertainty from these sources, and the implications of the uncertainty for basing decisions on the results of the research. In all areas, I think EPA does a good job. The first source of uncertainty is whether survey respondents really understand the survey which is a particular concern in the 316b survey, because of the complexity of the environmental changes that are being examined. The EPA addresses this uncertainty in two fashions: focus groups and the use of particular questions in the survey itself.”
“I am not going to summarize the 17 pages of discussion in this memo, but comment on a few of the things I thought most important. First, going in, I was a bit nervous about the “fish saved” variable and if people could really wrap their minds around this concept. I thought that it was particularly important that “fish saved” was well understood, because of the four environmental attributes, fish saved was the one that was most directly affected by the 316b policies. If the variable was well-understood, then the willingness to pay estimates for changes in this attribute could serve as a lower bound on non-use values. After reading the discussion of the understanding of the variable by the respondents, I feel that they did understand it well enough that it could serve as a basis of policy analysis. Second, I was concerned about the possible existence of warm glow effects. Again, the discussion of those participating in the focus groups convinced me that they are responding because of the anticipated environmental changes, and not because of an overall desire to do good. This, coupled with their consideration of budget constraints, leads to a conclusion that a lack of understanding of the surveys or the policies is not a significant source of uncertainty in the final results. The survey itself contains a question (#8) which addresses some of the uncertainty issues. The responses to this question are contained in Table 6.3. In my opinion, the uncertainty is at a level that it does not interfere with the ability to use the results in policy analysis.”
Dan Phaneuf
“Continuing my point above, I think there are remaining uncertainties as to what other specifications would predict and how robust the findings might be to modeling decisions. Thus I think there are some uncertainties in the analysis whose effects on the conclusions are not transparent. To this I will add that the treatment of the ASC in predicting welfare effects for non-marginal changes in the attributes needs some additional thought. There is a line in the document about ignoring it since it was not significantly different from zero. I think there werecases when the ASC was significant, and in any case I think it needs to be a part of the KrinskyRobb bootstrapping procedure – i.e. the distribution of values should enter into the welfare calculations in the same way as the other parameters (some of which are also insignificant) do.”
“In other dimensions I think EPA has done an admirable job of documenting uncertainties. The quantification of uncertainty arising both from random parameters as well as sampling error is a useful way to illustrate the range of possible predictions.”316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 29 John Whitehead
“Uncertainties in the analysis are not fully described or considered. Key uncertainties include validity of nonuse values over the entire sample of respondents and potential differences between respondent and nonrespondent WTP. A number of split sample analyses are necessary to provide information for potential sensitivity analyses. Without these it may be necessary to be more conservative with aggregate benefit estimates (see #9). Another approach is to estimate latent class models which are able to estimate different (probabilistic) groups of responses. With data such as these, without information on prior information for the respondents, a latent class model could estimate two or more vectors of coefficients for the choice models with probabilities that any respondent belongs to each group. It is possible that one or more vectors reveals the probability that latent groups of respondents have invalid WTP estimates (i.e., statistically insignificant cost and benefit coefficients). Haab et al. (Marine Res Econ, forthcoming) estimate conditional logit, mixed logit and latent class models and produce a range of WTP estimates. They suggest that this range could be used to provide a range of WTP estimates for sensitivity analyses. A similar approach could be taken here.”

SPECIFIC COMMENTS ON REPORT
Kevin Boyle
“For the considerable effort invested in the design of the survey booklet, I found the wording in the survey rough and confusing in places. I do not know how this affected the survey outcome in all instances, but it does leave unresolved questions. Some examples of these concerns are (all based on Northeast survey):

• P. 1st line – reference to habitat is not clear. Fish habitat or all aquatic habitat, or aquatic and terrestrial habitat?

• P.1 – you do not show diagrammatically human effects on the natural food web. Not critical perhaps, but would have been a good inclusion to close the information loop.

• P. 2, diagram – last bullet refers to return water, but this is not sown in the diagram so seem out of place and the diagram not fully developed to accompany the text.

• P. 2 map – I am concerned that the map does not coincide with the 93% figure most are on inland waters. I think the coastal sites should have been highlighted. This figure is repeated again just before the choice questions. I am concerned that responses to the choice questions could have been driven by the large number and spatially distributed cooling water sites and this led to overestimation. In this light it is good that the ASCs are insignificant, but this does not address effects on the attribute coefficient estimates.

• P. 3, 2nd para. – refer to recreational fishing, but do not explicitly exclude later and only reference commercial fishing in the attribute description. May have caused respondent confusion.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 30 • P. 3, 3rd para. – not clear if the species listed at the end are forage species of species that eat the forage species.

• Q. 1 – since this is the first time many respondents may have heard of this issue I am surprised that you did not allow for a “don’t know” response option.

• P. 4, 1st para. – use of “equivalent” makes this long sentence awkward and potentially confusing.

• P. 4, last sentence – confusing because you refer to multiple causes of losses in the preceding paragraph, but I think you are just referring to cooling losses in this sentence.

• P. 5, current situation diagram – line is confusing with 0% losses because a line means <0.1% losses on the previous page.

• P. 5, 3rd para. – wonder if wording here makes people think that the policy also conserves water. Here is very good that the ASC’s are not significant, providing evidence that this is not a problem.

• P. 5, bottom – wording really makes it sound like policy values 5% or 95% changes, and does not explain that the relative mix of both can lead to many different combine percentages.

• P. 6, Q. 6 – placement of question is strong given that preceding text discussed fish loses and there was no transition to habitat here.

 P. 6, bottom – not choosing A or B does not mean a respondent supports the status quo, it simply means that A and B are not sufficiently better, given the cost, to choose one.

• P. 7, top – You only reference protecting fish and do not introduce the habitat attribute here.

• P. 7 – commercial fish is consistent with what you intend, but do people know that the vast majority of commercial fish come from saltwater? If they do not, map issue I raised earlier continues to be a concern. It also seems odd to have the attribute label be commercial fish, but the description includes recreational fishing.

• P. 7 – scale here seems to totally overlook over-harvesting of commercial species in the Northeast. This is an issue that has received a lot of coverage in the press and I wonder if this influenced how people responded to this attribute in the absence of a clear survey explanation.

• P. 7 – attributes overlap and this is not consistent with the standard design of choice experiments. Thus, the standard additive of implicit prices to estimate consumer surplus as stated in equation 7 on page 65 of the report is applicable for this study. However, given that policy only looks at on attribute, this is not a concern for how the estimation results are used to support policy decision.

• P. 7 -- the wording of the fish saved attribute is awkward and potentially confusing (100=0 lost). I wonder if the significant standard deviations on the fish saved attribute in the estimated equations is due to respondent differences due to different levels of confusion or heterogeneity in preferences?

• P. 10, 4th bullet – I wonder if the monthly leveling of costs contributed to scenario being unrealistic since respondents know that electricity costs in the Northeast are highly seasonal and may think the costs of this program will be closely linked to electricity costs.

• P. 10, 7th bullet – introduce “environment” here and just leave it hanging, while you do follow-up on the “economy”.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 31 • P15-16, Q. 8-9 – responses to these questions can be suspect because reasons could vary by choice question and you force overall responses.”
“Some of these points I raised might be answered by knowledge gained from the focus groups and cognitive interviews that is not easily conveyed in the report. However, the roughness of the descriptions and overlapping categories is something that I would expect to have been refined and resolve in such an extensive design process.”

Diane Dupont

• “In the notes from the focus groups, the following comment is made...
(On page 9 of the Executive Summary...) “Participants generally appeared to have some understanding of how the ecological scores interact and this influenced their responses. ... Some participants’ responses indicated that the choice questions would have been more difficult if some effects were to go up and others down within the same policy option. (#5-13).
Given that focus group respondents commented on this possibility suggests that respondents may believe in a different type of implicit relationship between the various attributes than the one presented. Why is this important? It may be that, when they see ecosystem values rise in a particular choice task, they may infer that this will have some additional favourable impact upon another attribute of interest (commercial fishing, for example). So, they infer a relationship between the attributes and not treat them as separate. I am concerned that this might result in some kind of double counting in the Total WTP, in particular, when people assume a very strong complementary relationship between use (eg. Commercial and recreation fish populations) and non-use components (e.g., all fish populations that help to provide feed for commercial populations and aquatic ecosystems that provide habitat).

• (On page 15 of the Executive Summary...from a respondent ...“But see everything else follows the conditions of aquatic ecosystem. [...] the commercial fish population hinges upon the condition of aquatic ecosystem, as does the fish population [...]” (#3-15).
I know that the Johnston et al. (2011, 2012) papers from which the EPA survey draws much inspiration employ a similar choice experiment setup that combines both use and nonuse components. Both the EPA survey and the Johnston et al survey leave unspecified the relationship between environment and the components that make it up. I note, however, that Johnston et al employ a main effects with two-way interactions task choice design that may help to mitigate this problem. So, my mind is still not made up on how to interpret people’s responses. What underlying household production function drives people’s choices? Does it matter if some people see the different attributes as being either independent or substitutes and others see them as complements?316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 32 • In the discussion on page 7 that describes the attributes, the score for the ecological condition is defined between 0 and 100 % when compared to the most natural water in the [..] region. And,... “higher scores mean the area is more natural.” However, on the actual choice task pages the upper bound of 100% is defined as being in pristine condition. To my mind “pristine” is not necessarily natural and may be a somewhat value-laden word. I didn’t see any discussion of this in the focus group report or in the actual report. Whether this type of wording might lead to a strategic bias is something that could be considered. I note that some debriefing questions were included but I do not see them being used to assist in understanding responses.

 I have a couple of concerns regarding information provided. In the InlandVs5b version, on page 4 the reader is told that about 0.6 billion young adult fish are lost each year in Inland waters due to cooling water use. Then, it says that more [my emphasis on the more] than 0.6 billion fish could be saved. But, this is greater than what people have been told is lost. Then, on the next page, respondents are told that the maximum gain in fish saved is is 95 % - this seems inconsistent with the previous information. I checked the NE/SE/National/Pacific versions and there is not the same mismatch between what scientists estimate are the losses due to cooling and what the maximum gains from the programs might be. In each of the other cases, the fish saved is clearly smaller than the original amount being lost.

 The report says...“Because the Priority Mail response was lower than expected, the target number of telephone completed surveys was increased to obtain the desired number of responses.” Does this observation call into question the validity of the original mail approach as leading to some kind of selection bias (relative to telephone responses at any rate)?

 On page 66 the report states that the ASC is not included in the calculation of compensating surplus because it was found to be insignificant in the model. There is some discussion in the literature on the merits of including/excluding ASC values. However, it is widely accepted that inclusion of the ASC provides more conservative estimates since the exercise involves measure utility gains/losses of a program away from one’s status quo situation. Even though the coefficient itself may be insignificantly different from zero, its level is important for inclusion in calculations. I should also note that the ASC captures the effects on utility of any other attributes not included in the choice task that may be important to the respondent. Finally, I note that this is not a problem for the annual household WTP estimates provided in the report for the regulatory options since these are calculated using only the marginal WTP for a one unit change in fish saved.”

John Whitehead

• “Page 8: “Appropriately designed choice-based SP methods may also reduce strategic,316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 33 hypothetical and other possible biases that can result from asking survey questions versus assessing WTP through market transactions or binding referenda.” During the CVM debate discrete choice experiments (i.e., “choice-based SP methods”) were introduced. The mood amongst many environmental economists was that choice experiments were preferred for reasons not related to the problems of the CVM (which are mentioned in the sentence above) such as consistency with random utility theory, an interest in new and advance econometric methods and publishability. As such, it was asserted that choice experiments were a silver bullet that solved many problems associated with the CVM. I have not seen much comparative research to back up this assertion. For example, the next paragraph describes the extensive research conducted on hypothetical bias in the CVM. No reference is made to studies assessing the potential hypothetical bias that may result from stated preference choice experiments. Unless there are a number of studies that can be referenced here to back up the assertion, this sentence should be removed.

• Page 9: I’m not sure what “older forms of SP analysis” refers to. Is this the CVM? It might be better to go ahead and state that the comparison is being made to the older forms of the CVM. But why is this comparison appropriate? The EPA should be comparing discrete choice experiments to newer forms of CVM that are not “typically contingent upon a single specification of ecological effects.” It is straightforward to design CVM surveys to ask multiple WTP questions (e.g., Christie and Azevedo, J of Agr Econ, 2008). If the EPA feels that it is more efficient to estimate the WTP for characteristics of a policy with discrete choice experiments relative to the multiple question CVM then some supporting references should be provided.

• Page 13: “an unharvested population is typically believed to be approximately three times as large as MSY.” Please provide a reference.

• Page 21: Why was a mail survey chosen over an internet survey with all of the advances in internet sampling that have occurred over the past 10 years or so? With all of the mailings it seems that even Knowledge Networks would have been less expensive and more effective. Note that Lew and Wallmo (Env and Res Econ, 2011) used Knowledge Networks for a federal agency survey.

• Page 21: Was the Pacific response rate 32% or 40% = 311/(1040-255)? Was the Inland response rate 35% or 33% = 787/(2480-65)? Why was there such a lower rate of undeliverable surveys for the inland sample (2%) compared to the others (7% - 25%)?

• Page 22: 46% plus seems like a high rate of recreational seafood consumption. Are there any representative sample statistics with which to compare?

• Pages 27-42: Reading section 7 gets tedious with the cut and pasting. Some of this could go into an appendix with a summary in the text.

• Page 51: Much of the text above Table 8-5 should be relegated to a footnote. It really doesn’t matter if this study has similar pseudo-R2s as other studies.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 34 • Page 55: “Unlike open-ended contingent valuation questions, choice experiments provide a direct mechanism for respondents to react to scope and scale of resource changes ....” Why is open-ended CVM the relevant comparison? Open-ended questions have not been the norm since Bishop and Heberlein (AJAE, 1979).

• Page 66: What is the WTP per fish saved? How does it compare to other studies?

• Table 9-2: The regional implicit prices from the weighted linear model are in a different order (NE, SE, IN, PA) compared to Table 9-1 (NE, SE, PA, IN).

• Page 68: The point estimate of the weighted regional implicit WTP, $1.28, is within the 90% confidence interval of the national fish saved WTP estimate so there is no reason to discuss any difference in point estimates.

• Page 69: The AIE acronym is used before it is introduced as an acronym in this paragraph.

• Table 10-1: How is the “reduction in AIE losses” column measured? Is this millions of fish saved?”

IV. ROADMAP FOR THE EPA
Through the Applied Planning Corporation approach, the peer review panel came up with a road map for the EPA to strengthen the report. The road map is outlined below.

Estimation Robustness The panel found that the basic estimation of the report is fine and is largely consistent with the literature. However, several items can be enhanced:

 The investigation of nonlinearities using a quadratic term is not adequate. Nonlinearities can be better investigated with attribute levels coded as binary variables and possibly using effects coding.

 Given the overlapping attribute definitions, it would be beneficial to look for attribute interaction effects and the effects of dropping attributes from the estimation. The former is important because the attribute design excluded attribute level combinations that potentially did not make sense.

 Mixed logit models are based on distribution assumptions made by the investigators as clearly noted in the report. While the distribution assumptions imposed make sense, it would be useful to see how different assumptions would affect the estimation results.

 Estimating a simple conditional logit model to be used as a baseline for model comparisons.

 Estimating multinomial logit models will help to consider the robustness of the mixedlogit assumptions.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 35  Estimating logarithmic transformation of the models may also be useful.

These analyses can largely be reported in an Appendix to the report and key findings summarized in the text of the report explaining how different specifications lead to different/similar aggregate welfare predictions. The main report would contain the preferred estimation.
Sample Selection and Non-Response Study The design of the sample selection investigation is appropriate and is consistent with current survey standards. However, the analyses that follow are not performed to the same standard as the design.

 Discuss why variables used in the sample-selection analyses are relevant, i.e., would influence WTP.

 Provide a breakdown of the responses to specific questions such as ‘how important is theprotection of aquatic ecosystems to you’ by gender, race, income, etc. (the sociodemographic variables that were found to differ between the various main samples and the non-response samples). These variables may not have been included in the nonresponse follow-up surveys, but there are a number that logically are correlated with WTP, perhaps more so than the variables in the nonresponse follow-up surveys.

 Conduct statistical tests of the effects of sample selection.

 Conduct robustness checks of assumptions imposed in the sample selection process, e.g., conducting analyses based on different sets of variables and different recordings of variables.
The larger question not really addressed in the current report is whether the weighted model should be used for benefit estimation and applied to all people in the sample frames, or if the unweighted model should be used and applied only to the proportion of the population consistent with the survey response rates. Both approaches have been used in the literature. The latter may provide and absolute lower bound welfare estimate; therefore, if benefits exceed costs then the EPA is confident that they really do. The former is likely an overestimate because it may include people who have zero values (WTP=$0) and this is not addressed in the EPA’s statistical analyses.
Validity Checks The validity checks conducted are appropriate, with the exception of the scope investigation, but they do not go far enough.

 As with sample selection, report statistical tests for the analyses that the EPA conducts rather than only reporting summary statistics for data splits.

 EPA has a number of very good probing questions after the choice questions and the responses to these questions should be used in the validity analyses. For example, questions pertaining to whether a respondent fishes could be used to identify use versus non-use motivations.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 36 The panel recommends that the EPA use these analyses and information from the focus groups, to the extent possible, to try to address some of the peer-review comments and industry comments.
Discussion of Over/Under Estimation of WTP There are numerous elements of the survey design, data analyses and computation of aggregate benefits that can lead to over or under estimation of benefits and the current draft of the report is largely silent on these potential effects. Some of these are analysis and aggregation issues that are mentioned above. Examples of potential survey issues are:

 Introduction of thermal discharge effects, but not explicitly addressed in the choice question.

 Potential respondent confusion of attribute descriptions.

 Map in survey may give impression to respondents that they are located near affected areas and the effects are widespread geographically, but wording in survey does not support such a conclusion. Wording is not as explicit as the map is visually explicit.
The panel suggests adding a new section in the report that summarizes and integrates the EPA’s uncertainty analyses in order to clarify the directions of potential biases. It would also be useful to look at split sample analysis with the respondents who indicated they were at the top of the ‘confident in their answers/understood’ scale versus those who were not. It is doubtful that the EPA will be able to clearly say that WTP is over or under estimated, but a logical argument can be made that the EPA has provided the best possible estimate using state-of-the-art procedures.
Reporting and Documentation The report is largely well written. However, weaknesses that need to be improved include justifying procedures used and finishing discussions to the implications of these procedures for estimated WTP. The procedures used a generally well described, but the motivation for their use and the consequences of their use are largely silent in the report.

V. CONCLUSION
This report summarizes the comments of a peer review panel evaluating a stated preference choice experiment on rules for Section 316(b) of the Clean Water Act (33 U.S.C. 1251 et seq). Section 316(b) establishes national performance requirements that are designed to minimize adverse environmental impacts caused by cooling water intake structures (CWISs). The choice experiment was conducted to further investigate potential policy options stemming from these rules and the willingness-to-pay of consumers on ecological attributes such as “fish saved”.
This peer review is intended to assess the work that EPA has completed with regard to this rule as well as the results of EPA’s 316(b) stated preference research. The peer reviewers were asked to identify areas where modifications to the EPA’s estimation and validation approaches would improve the accuracy and precision of the resulting willingness-to-pay estimates generated from the models developed using the survey data.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 37 Overall, the panel felt that the survey design followed best practice. Through the review process the panel identified several issues of importance that they strongly encourage the EPA to address prior to releasing the report. They have provided the EPA with a roadmap to address their concerns regarding documentation, sample selection, robustness, validity and over/underestimation of WTP. Further specifications analyses recommended in these report will ensure that the results are robust both conceptually and empirically.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 38 APPENDIX I

REVIEWER COMMENTS316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 39 PEER REVIEW COMMENTS

Kevin Boyle316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 40 316(b) Peer Review Kevin J. Boyle Virginia Tech
This study uses design practices that are better than is standard in the literature and the data analyses follow standard practice. As with any study, in hind sight I have identified components that could have been designed better. These issues can not addressed at this point, but potential concerns need to be considered in the computation of aggregate benefits. There are elements of the econometric analysis and documentation that need to be addressed and can still be improved before the final computation of benefits.
1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values?
Yes, this approach is in common use in the economics literature for estimating uses and nonuse values for changes in natural resources. Applications have been published in the peer-reviewed literature, and there is a substantial literature investigating the validity of choice experiments (also known as choice modeling, choice questions, attribute based questions, conjoint analysis, and other labels).
2. Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature?
Using focus groups, cognitive interviews and pilot studies is consistent with the very best practice for designing survey instruments. The investigators conducted more focus groups and cognitive interviews than is typical; the work here is consistent with the number conducted for high-profile litigation applications. The report mentions the pilot study, but provides no explanation comparable to the discussion of the focus groups and cognitive interviews. It is unclear if the Northeast survey implementation is the pilot or if another pilot was also conducted. The authors mention that the Northeast survey was implemented to inform the other survey applications, but no mention is made of the specific goals of this effort or if the other surveys were changed from what was learned from the prior implementation of the Northeast survey instrument. These latter comments are not intended to be a critique of the survey design process; rather they are a comment of a lack of clarity in the documentation contained in the report.
3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation?
For the considerable effort invested in the design of the survey booklet, I found the wording in the survey rough and confusing in places. I do not know how this affected the survey outcome in all instances, but it does leave unresolved questions. Some examples of these concerns are (all based on Northeast survey):

• P. 1st line – reference to habitat is not clear. Fish habitat or all aquatic habitat, or aquatic and terrestrial habitat?316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 41 • P.1 – you do not show diagrammatically human effects on the natural food web. Not critical perhaps, but would have been a good inclusion to close the information loop.

• P. 2, diagram – last bullet refers to return water, but this is not sown in the diagram so seem out of place and the diagram not fully developed to accompany the text.

• P. 2 map – I am concerned that the map does not coincide with the 93% figure most are on inland waters. I think the coastal sites should have been highlighted. This figure is repeated again just before the choice questions. I am concerned that responses to the choice questions could have been driven by the large number and spatially distributed cooling water sites and this led to overestimation. In this light it is good that the ASCs are insignificant, but this does not address effects on the attribute coefficient estimates.

• P. 3, 2ndpara. – refer to recreational fishing, but do not explicitly exclude later and only reference commercial fishing in the attribute description. May have caused respondent confusion.

• P. 3, 3rdpara. – not clear if the species listed at the end are forage species of species that eat the forage species.

• Q. 1 – since this is the first time many respondents may have heard of this issue I am surprised that you did not allow for a “don’t know” response option.

• P. 4, 1stpara. – use of “equivalent” makes this long sentence awkward and potentially confusing.

• P. 4, last sentence – confusing because you refer to multiple causes of losses in the preceding paragraph, but I think you are just referring to cooling losses in this sentence.

• P. 5, current situation diagram – line is confusing with 0% losses because a line means <0.1% losses on the previous page.

• P. 5, 3rdpara. – wonder if wording here makes people think that the policy also conserves water. Here is very good that the ASC’s are not significant, providing evidence that this is not a problem.

• P. 5, bottom – wording really makes it sound like policy values 5% or 95% changes, and does not explain that the relative mix of both can lead to many different combine percentages.

• P. 6, Q. 6 – placement of question is strong given that preceding text discussed fish loses and there was no transition to habitat here.

 P. 6, bottom – not choosing A or B does not mean a respondent supports the status quo, it simply means that A and B are not sufficiently better, given the cost, to choose one.

• P. 7, top – You only reference protecting fish and do not introduce the habitat attribute here.

• P. 7 – commercial fish is consistent with what you intend, but do people know that the vast majority of commercial fish come from saltwater? If they do not, map issue I raised earlier continues to be a concern. It also seems odd to have the attribute label be commercial fish, but the description includes recreational fishing.

• P. 7 – scale here seems to totally overlook over-harvesting of commercial species in the Northeast. This is an issue that has received a lot of coverage in the press and I wonder if this influenced how people responded to this attribute in the absence of a clear survey explanation.

• P. 7 – attributes overlap and this is not consistent with the standard design of choice experiments. Thus, the standard additive of implicit prices to estimate consumer surplus as stated in equation 7 on page 65 of the report is applicable for this study. However, given that policy only looks at on attribute, this is not a concern for how the estimation results are used to support policy decision.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 42 • P. 7 -- the wording of the fish saved attribute is awkward and potentially confusing (100=0 lost). I wonder if the significant standard deviations on the fish saved attribute in the estimated equations is due to respondent differences due to different levels of confusion or heterogeneity in preferences?

• P. 10, 4th bullet – I wonder if the monthly leveling of costs contributed to scenario being unrealistic since respondents know that electricity costs in the Northeast are highly seasonal and may think the costs of this program will be closely linked to electricity costs.

• P. 10, 7th bullet – introduce “environment” here and just leave it hanging, while you do follow-up on the “economy”.

• P15-16, Q. 8-9 – responses to these questions can be suspect because reasons could vary by choice question and you force overall responses.
Some of these points I raised might be answered by knowledge gained from the focus groups and cognitive interviews that is not easily conveyed in the report. However, the roughness of the descriptions and overlapping categories is something that I would expect to have been refined and resolve in such an extensive design process.
I did not see a clear budget reminder in the survey wording, but respondents were told that the price increase was “unavoidable” (Northeast survey, p. 7, bottom).
There was not a reminder of substitutes.
There was no cheap talk script. While I am not convinced of the appropriateness of cheap talk scripts, they have become common practice in stated-preference surveys and their presence does serve to reduce value estimates.
Consequentiality was addressed in the survey, but not as clearly as it could have been. The survey does sate that respondent answers will help to make policy decisions (Northeast survey, p. 1, top), but this is not repeated immediately before the choice questions nor is it specifically stated that survey results “will be used” in making policy decisions. Respondents are ask to answer as if the questions were real and binding (Northeast survey, p. 10, bottom), but are not asked to answer as if they are voting in a real referendum at the moment they answer the choice questions. Thus, there are elements of consequentiality, but they are not as poignant as they could have been.
4. Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes?
Yes, if and only if the policy just focuses on one attribute (fish saved) and the linearity of this variable in the estimated equations is confirmed by more rigorous investigation. To investigate the robustness of models to support value estimation for a one percent change I suggest estimation where each attributes is identified using effects coding to investigate linearity instead of the quadratic specification. The quadratic models seems odd at best given the small ranges of levels posed for some attributes (4-6%) and the large jumps (coarseness) for the key fish saved variable (report, p. 15, Table 3-1).
5. Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates?316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 43 The sample sizes are acceptable minimums, but below what would be considered best practice for a study of this magnitude to support a major policy decision. Sample sizes as employed in this study are consistent with experimental studies published in the per-reviewed literature, but are les than one would expect for a major policy study where each sample would typically target at least 1,000 usable observations (check).
The response rates are consistent with what is observed in the literature for best practice applications of choice modeling. The nonresponse follow-up survey is consistent with best practice. I wish the authors has discussed whether they believe the identified effects were relevant for value estimation; no intuitive discussion was provided or insight.
6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior?
I recommend discontinuing the use of the term RUM. While the model is based on random utility, this is a total value study and RUM in the literature refers to a specific type of travel-cost model that just estimates use values. Thus, I fear the use of the term RUM might cause some to misconstrue the values that were actually estimated.
I do not think the quadratic specification is reasonable. As noted above, I suggest re-estiamtion where the levels of each attribute are dummied out through effect coding to provide a more robust test of the linearity assumption for the fish saved variable.
I believe the use of weighted models is consistent with best practice when survey data do not match known population statistics. The text does not specify how the constant, which the authors refer as ASCs are coded? Was the ASC=1 if the alternative was A of B and 0 otherwise? This seems to be the interpretation applied to the estimation results.
7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature?
The nonresponse study follows best practice. I would like to have seen the report provide more intuition of whether the significant effects identified really are expected to influence estimated values. I would also like to have seen an investigation of the subjective decisions employed in the development of weights for estimation.
8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses?
Choice experiments, by their design, provide a within subject test of scope. The scope issue in the literature relates to between subject tests and the tests reported do not address this concern.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 44 The scope test stratifying on choices A and B is neither appropriate nor insightful. The groupings do not stratify so different people from the same sampling frame are responding to the alternative percentages.
Since the ASCs are not significant it is not surprising that the ASCs for alternatives A and B are not significant. It is good to investigate if there is a leaning toward alternative A or B. Investigation of fish saved with the other attributes is appropriate.
I would also suggest estimating the model, always including fish saved and cost, but dropping other attributes sequentially. Given the overlapping attribute definitions and exclusions of extreme combinations suggests that responses to the fish saved attribute may be conditioned on the other attributes. I know this is not the same as dropping attributes from the design, but it is one way to look at this issue.
I do not understand why some of these analyses were just conducted with the Northeast data, and not all sample groupings.
I was surprised that you did not use results from questions 8 and 9 (p. 15 &16) to evaluate the credibility of your survey response/equation estimates in the report?
9. What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule?
The design practices are better than what is done in most current studies in the stated preference literature and the data analyses follow standard practice. I am concerned that the survey content is not as well developed as I would have liked and as presented might lead to respondent confusions and certain items may have led respondents to value more than the scenarios were intended to value. The current study is superior to superior to the existing benefit estimates.
Yes, I would recommend the current study for national policy analysis. No study is perfect and any benefit analysis must take into consideration potential imperfections in benefit estimation. In the current analysis the robustness analyses of the survey data analysis are appropriate and can be expanded as noted above. There needs to be more sensitivity analysis in computing the aggregate benefit estimates.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner?
See my comments above.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 45 PEER REVIEW COMMENTS
Diane Dupont316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 46 Review of EPA Stated Preference Survey and Responses to EPA’s 10 Charge Questions

1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values? My response: Stated preference surveys, of which the choice experiments question format is an example, are the preferred approach used by economists to obtain non-market values for goods and services that provide benefits to people. While indirect methods (such as averting expenditures or travel costs) can be used to infer use values, they are incapable of obtaining nonuse values due to the absence of a related market transaction. Choice experiments, in particular, are well suited to obtaining estimates of both use and nonuse values since they describe individual components or attributes of the good/service of interest to survey respondents. Calculated design of packages with different amounts (levels) of the attributes are presented to survey respondents and they are asked to make choices between a status quo (no choice option) and one or more alternative choices that describe different packages. By varying the levels of the attributes across several such choice situations and designing a large enough number of different packages spread across respondents, the researcher can obtain both marginal willingness-to-pay values associated with one unit changes in a particular attribute, as well as total willingness-to-pay values associated with specified unit changes in all attributes that comprise a total package or policy change. With specific reference to this choice experiments survey the format is appropriate insofar as the goods/services being valued are clearly understood by the respondents. After reading the survey materials, I believe that the researchers are able to obtain a total WTP for the characterized polices and that this total WTP will contain both use and non-use values. I should also note that CVM, the main competitor question format, would not be as amenable to finding both use and nonuse values. I am less certain about the ability of the survey to separate out use from non-use components. While I do not know whether this is an additional consideration for the work, I think this point does have a bearing upon the total WTP being sought in the event that respondents see use and nonuse values as being overlapping in some way, that is, the two values are not separable. The key issue is: as each person views the attributes specified in the choice task and then makes choices by examining attribute levels, he or she is assuming a particular relationship among these attributes. As presented in the survey, each of the levels presented for the alternative programs always rises relative to the status quo level. This has not gone unnoticed by respondents. In the notes from the focus groups, the following comment is made... (On page 9 of the Executive Summary...)“Participants generally appeared to have some understanding of how the ecological scores interact and this influenced their responses. ... Some participants’ responses indicated that the choice questions would have been more difficult if some effects were to go up and others down within the same policy option. (#5- 13). Given that focus group respondents commented on this possibility suggests that respondents may believe in a different type of implicit relationship between the various attributes than the one316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 47 presented. Why is this important? It may be that, when they see ecosystem values rise in a particular choice task, they may infer that this will have some additional favourable impact upon another attribute of interest (commercial fishing, for example). So, they infer a relationship between the attributes and not treat them as separate. I am concerned that this might result in some kind of double counting in the Total WTP, in particular, when people assume a very strong complementary relationship between use (eg. Commercial and recreation fish populations) and non-use components (e.g., all fish populations that help to provide feed for commercial populations and aquatic ecosystems that provide habitat). (On page 15 of the Executive Summary...from a respondent ...“But see everything else follows the conditions of aquatic ecosystem. [...] the commercial fish population hinges upon the condition of aquatic ecosystem, as does the fish population [...]” (#3-15). I know that the Johnston et al. (2011, 2012) papers from which the EPA survey draws much inspiration employ a similar choice experiment setup that combines both use and nonuse components. Both the EPA survey and the Johnston et al survey leave unspecified the relationship between environment and the components that make it up. I note, however, that Johnston et al employ a main effects with two-way interactions task choice design that may help to mitigate this problem. So, my mind is still not made up on how to interpret people’s responses. What underlying household production function drives people’s choices? Does it matter if some people see the different attributes as being either independent or substitutes and others see them as complements?

2. Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature? My Response: In general the procedures followed are consistent with the industry standard (focus groups in different regions, one-on-one cognitive interviews, peer review, testing of specific wordings, use of graphics, investigation of possible use of heuristics, etc). However, I have a couple of concerns.

 As far as I can see, there was no pre-testing on a larger scale in the field (actual mail survey pre-test); only focus group testing within a controlled setting.

 None of the focus groups took place in the Pacific Region. Given the weaker results from the models using data from this region, this is a shortcoming.

 I am assuming that regionally-focused survey versions were tested in the regionalfocus groups? However, it is not clear whether the national versions were also pretested. This is important to develop the salience of the attributes at a national, as opposed to more personal, regional scale. It may help to explain the lower (than expected) WTP obtained from the National sample.

 It was not clear from the executive report discussion about the focus groups whether other attributes were salient to focus group respondents. I am wondering in particular316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 48

about the thermal discharge. On page 2 the survey talks about how pumping warm water back into the environment affects ecological systems. Then, later on page 5 there is a discussion about how closed cycle cooling may reduce fish losses and also thermal discharge. Was this an attribute that was of concern or interest to focus groups? The thermal discharge component jumped out at me. If it was not an issue in focus groups, then perhaps it is worthy of a footnote.

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation? My Response:

 The overall design and quality of the survey booklet is impressive and enhances the saliency of the exercise. I generally found the wording to be unbiased with one exception. In the discussion on page 7 that describes the attributes, the score for the ecological condition is defined between 0 and 100 % when compared to the most natural water in the [..] region. And,... “higher scores mean the area is more natural.” However, on the actual choice task pages the upper bound of 100% is defined as being in pristine condition. To my mind “pristine” is not necessarily natural and may be a somewhat value-laden word. I didn’t see any discussion of this in the focus group report or in the actual report. Whether this type of wording might lead to a strategic bias is something that could be considered. I note that some debriefing questions were included but I do not see them being used to assist in understanding responses.

 The budget constraint was addressed but not really emphasized. However, I think the choice of higher electricity prices is a good one and believable. I would have expected a little more cheap talk prior to the choice tasks, e.g, provide respondents with some additional reasons why some people would not vote in favour of the alternatives, as well as reasons to vote in favour.

 The policy effects described to people relate to improvements that will be felt within 3-5 years, however, the survey left unstated when household cost increases would begin. I did not see any question in the survey designed to elicit time preference information nor did I see a question designed to elicit people’s beliefs about when they would start paying these higher costs. Different people may be applying very different discount rates to the household cost attribute.

 I have a couple of concerns regarding information provided. In the InlandVs5b version, on page 4 the reader is told that about 0.6 billion young adult fish are lost each year in Inland waters due to cooling water use. Then, it says that more [my emphasis on the more] than 0.6 billion fish could be saved. But, this is greater than what people have been told is lost. Then, on the next page, respondents are told that316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 49

the maximum gain in fish saved is is 95 % - this seems inconsistent with the previous information. I checked the NE/SE/National/Pacific versions and there is not the same mismatch between what scientists estimate are the losses due to cooling and what the maximum gains from the programs might be. In each of the other cases, the fish saved is clearly smaller than the original amount being lost. Screen Capture from Inland...

Screen Capture From SE

Screen Capture From NE

4. Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes? My Response:

 I am concerned that the scenarios presented have no instances of increases in some316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 50 attributes and decreases in others relative to the status quo. See the verbatim comment noted above from a focus group participant. This may give rise to less robust estimates if respondents find that scenarios do not fit with preconceptions.

 The design uses only attribute main effects. According to Carson and Louviere Environmental and Resource Economics, 2011), in such a case the researcher must assume that the indirect utility function is strictly additive for all respondents. This worries me given my concern about the (possibly varying) implicit relationships respondents may believe exist between the use and nonuse attributes described in the choice tasks.

 Within each regional (and national) survey the same sample question is used for each booklet but then the first choice task mostly contains different values from the sample values. Do people treat the sample as a base or reference set of values against which they compare values in the subsequent questions? This may induce a kind of loss aversion response similar to what was found in a paper that I did with Ian Bateman, Brett Day and Stavros Georgiou in RESTAT (2009). I know that respondents are told not to look back but with a mail survey this is impossible to enforce. More generally, is there some kind of ordering or comparison of values in earlier and later questions (since the ordering is fixed for each booklet and not randomized across respondents). This is what Holmes and Boyle found (Land, 2005).

5. Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates? My Response:

 The rationale behind the sample sizes is sound and the final response rates are just adequate. However, I some related concerns.

 It looks to me as if there is a problem with the use of postal delivery addresses to find respondents since the Pacific region has the smallest population but the biggest number of undeliverable surveys. Is there a problem with this postal information, e.g., people have vacated properties due to the financial crisis? This calls into question whether the Pacific sample is less representative than the others.

 In terms of demographics, males are systematically over-represented. This makes me think that there was some kind of sample selectivity that took place since the survey was a mail survey. I looked at the invitation letters going out to households and did not see anything in the information letter asking the person with the closest birthday to answer the survey. So, did many women simply give the survey to their male partners thinking that it was about fish and of more interest to the males? One could check this by looking at whether female respondents in the sample were more likely to be living in one person households. (I note in the telephone non-response script that they ask for the respondent with the closest birthday.) I think this over-representation of men is a real problem.

 I would like to see a breakdown of the responses to the question... “how important is the protection of aquatic ecosystems to you” by gender, race, income, etc. (the socio-316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 51 demographic variables that were found to differ between the various main samples and non response samples). This would assist in better understanding whether the samples provide robust estimates.

 The Inland sample is problematic in that it contains very different socio-demographics from the non-response sample.

 The report says...“Because the Priority Mail response was lower than expected, the target number of telephone completed surveys was increased to obtain the desired number of responses.” Does this observation call into question the validity of the original mail approach as leading to some kind of selection bias (relative to telephone responses at any rate)?

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior? My Response:

 I support the effort to use the weighted model results, particularly given the significant demographic differences between the main sample and the non-response sample, as well as the region-census level data. I think this is the appropriate approach to use for estimating the welfare changes associated with the expected changed in survey attributes under the 316b regulation.

 Given the concern for estimates across the different regions, I think the authors ought to include some measure of distance decay in their models. They can look at the literature on distance decay. A useful recent paper is “Characterizing Spatial Pattern in Ecosystem Service Values when Distance Decay Doesn’t Apply: Choice Experiments and Local Indicators of Spatial Association” by Robert J. Johnston, Mahesh Ramachandran, Eric T. Schultz, Kathleen Segerson, Elena Y. Besedin. This was presented at the 2011 AAEA meetings. The reason I argue in favour of this is that I am concerned about the likelihood of systematic differences in responses from respondents living close to the facility points on the maps provided in the booklets. I would imagine that the researchers have zip code information that could be used since it was a mail survey. In particular, it is an interesting empirical question whether non-use values are more or less subject to distance decay than use values.

 The report notes that demographic interactions between ecological attributes and gender and education were included in a preliminary Northeast model. However, because the interaction terms were not found to improve model fit, demographic variables were not used in other models. I am not convinced by the explanation in the report. When I look at the results in Appendix G-1, it seems to me that there is significant information in the316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 52 model with interactions. Both college and gender have significant interaction terms with fish saved, the attribute of most interest in this analysis. What were the AIC statistics for these two models? I would like to see a similar model with interactions using the Inland data set where most of the demographic variables differ from the non-response sample. As the models currently stand, I am less certain that the estimated WTP values for the Inland region are representative of average WTP since it is not obvious that the sample is random. I am thinking particularly of the over-representation of males. I know that weighting has been done on the parameters. However, the underlying utility function only includes the main attributes without interactions. I would like to see a breakdown of some of the debriefing responses on the importance of the different attributes by gender and education.

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature? My Response:

 Since the regional results are being compared to one another I think it is unfortunate that a single weighting method could not be applied to each of the five data sets. However, I think the EPA did the best that could be done with the information available. I would like to see further discussion of the potential implications of these two different weighting schemes.

 I have two questions about the decision made (on page 47) to exclude respondents from the weighted model estimation if they do no include answers for all variables being used in the weighting. How many respondents are removed from each region and are there systematic differences in respondents that were removed when compared to respondents who were kept in the sample?

 A final issue arises with the Pacific sample. There was a significant difference between the importance assigned to ecosystems between the main sample and the non-response sample. But, an adjustment was not undertaken for this. I understand the concern that this is likely to be an endogenous variable however, this points up a concern that I raise earlier about possible sample selection bias. I also note for what it is worth that the household size in the non-response sample is a lot larger than in the main sample. Since people were asked to consider household level cost increases, this may introduce a potential bias.

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses? My Response:316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 53  The external scope test of looking at proportions of responses using split samples seems like a reasonable way to examine the issue. However, no formal testing of these proportions is done and, in some cases, the differences are very small and the standarddeviations are very large. I would like to see significance levels from non-parametric Chisquared tests of differences in proportions (where the proportions are people accepting a particular option). I would also like to see the test done in a way that is sensitive to the question ordering (that is, did the respondent say yes to A involving 95% fish saved in each of the three question orderings or separately). There is some evidence in the Pacific and National samples of a lack of scope response to fish saved (fish saved ( fish_sav) and/or fish saved squared ( fish_sq) were not significant in the quadratic models for the Pacific region and national survey.) This does call into question the salience of the attribute for these two samples.

 The only issue that I have with respect to the argument that choice experiments provide their own internal scope test (and so it is not necessary to do) is the following. If a respondent ignores certain attributes in his/her choices, then this non-attendance may mean that internal scope tests for the ignored attributes are not satisfied in a conventional

sense.

9.What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule? My Response:

 Compared to the current stated preference literature I believe that the data collected within this survey is capable of providing both valid and reliable estimates for benefits. I place more confidence in results from the regional models/datasets (with the exception of those for the Pacific Region) than from results associated with the National dataset, in part because of the saliency of the issues within a regional context and in part because of the larger (relatively speaking) sample sizes used in the regions. I use the word “capable” in the first sentence because I think the research effort would gain important insights after incorporating distance decay effects, ordering and context effects, and further investigation of role of differences in demographics, particularly insofar as these differences manifest themselves expressed opinions on the debriefing questions. At a minimum I think the researchers need to do these additional models in order to forestall questions about the reliability of the results to alternative specifications.

 As far as I understand it the EPA has already calculated some use benefits in a prior study (presumably from fishing). Since the values collected in the current study potentially include use and nonuse values (although these cannot be separately identified), I assume that the previous use benefits will not be added to the estimates from this study. That would likely include some form of double counting. However, if the interest is in316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 54 providing use and nonuse benefits together, then the approach in this current study would provide superior estimates for the benefits.

 I have confidence that the information obtained from the survey can be used to assist in EPA’s CBA analysis of the 316(b) rule. I would recommend undertaking some additional model estimation along the lines noted above in order to verify that validity of the estimates already provided.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner?

My Response:

 I think the report contains a great deal of information in different locations. The footnotes, in particular, are very helpful to further understanding of decisions made along the way. However, I think the report should collect all potential implications of uncertainties in the analysis and put them together in a single location. Many steps were undertaken and many decisions were made prior to the generation of the WTP estimates in Table 10.2. The inclusion of a table showing the types of decisions that had to be made with respect to uncertainties in modeling, parameter values, model results, estimated WTP values and likely directions of biases would help the reader to better understand the implications of the uncertainties.

Additional comment On page 66 the report states that the ASC is not included in the calculation of compensating surplus because it was found to be insignificant in the model. There is some discussion in the literature on the merits of including/excluding ASC values. However, it is widely accepted that inclusion of the ASC provides more conservative estimates since the exercise involves measure utility gains/losses of a program away from one’s status quo situation. Even though the coefficient itself may be insignificantly different from zero, its level is important for inclusion in calculations. I should also note that the ASC captures the effects on utility of any other attributes not included in the choice task that may be important to the respondent. Finally, I note that this is not a problem for the annual household WTP estimates provided in the report for the regulatory options since these are calculated using only the marginal WTP for a one unit change in fish saved.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 55 PEER REVIEW COMMENTS

Jim Kahn316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 56 Kahn Review

Introduction:
The first thing I would like to point out is that researchers make a large number of choices when researching any topic, but this is particularly true with choice-modeling experiments. It is always possible to second-guess these choices and criticize the researchers for choosing A instead of B, when, in reality, both A and B have merits and have flaws. There is no such thing as a perfect choice-modeling experiment. The important questions are whether the survey was conducted with best practices in an manner in which is unlikely to have significant biases, and whether the survey was conducted in a fashion in which the confidence intervals of the value estimates is likely to be tight enough for meaningful policy analysis. I would like to say that I have a great deal of confidence in this survey. First, it was conducted by two consultants who are truly experts in the field. Second, when they encountered challenges (proper definition of attributes, nonresponse issues, etc.) they addressed these challenges with vigor and rigor. Third, and most importantly, the researchers did not try to overreach, but measured those values that could be most confidently measured. Finally, the documents explain complicated issues in a manner that can be understood by people that are not experts in the field. After securing permission from EPA and the authors, I would like to use this study in my teaching.

RESPONSES TO SPECIFIC PEER REVIEW CHARGE QUESTIONS:

1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values? Choice modeling is not only an appropriate means to estimate total willingness to pay for these types of policies, but it is the best method for several reasons. First, when choice-modeling has been used in potential new product applications (called conjoint analysis in this literature), hypothetical choices have been very close to actual closes. For example, researchers asked potential apartment renters to participate in a choice experiment looking at their preferences for apartments with different characteristics and prices, and found that when they actually went to rent, their actual choices were similar to their hypothetical choices. This has been true for a number of market goods. In contrast, contingent valuation has been notoriously unsuccessful in having hypothetical choices match actual choices, even with market goods. The decision-making model with choice experiments is one which people are more familiar. Choice modeling has been successfully used in a number of applications, at every level between very local and global, and in both developed and developing country contexts. It is the best method for modeling both direct use and non-use values, although it is often difficult to conduct an experiment which allows the researcher to produce separate estimates of direct and indirect316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 57 use values. However, if the policy question requires an estimate of total value (rather than separate estimates of direct and indirect use values) this is not an issue and the value estimates can help inform the policy decision process. One very useful aspect of choice-modeling is that it is multi-attribute, so different policy options can be analyzed, leading to the valuation of different levels of environmental quality. Neither contingent valuation, nor the revealed preference methods (travel cost or the hedonic methods) have this desirable property. Choice-modeling experiments can also be structured to include other decision-making criteria, such as equity or sustainability, but this was not the charge of this survey. This survey was designed solely to answer questions related to economic efficiency, measuring the total willingness to pay for the policy induced improvements in environmental quality. Nothing has happened to change my 2006 conclusions that choice-modeling is the appropriate direction to go to understand the benefits of the proposed rule-making. Neither the way the survey was implemented nor any new developments in the environmental economics literature have changed my perspective on this issue. In the original review in 2006, Dr. Krupnick stated that there was no indication in the literature that choice-modeling was a better choice for policy analysis than contingent valuation. Although EPA effectively responded to this, I think it important to add some of my own comments. In the early 1990s as choice-modeling made its way from the marketing/transportation literature into environmental economics, it was not well-understood, and there was considerable discussion among environmental economists if it represented an improvement. Choice-modeling was used for relatively simple problems in the early attempts, in which there was no real advantage over contingent valuation. However, after roughly two decades of experience, it is almost universally acknowledged that choice modeling is more appropriate when there are multiple types of environmental change or environmental resources, and when there are multiple policies or levels of policies to be analyzed. Contingent valuation represents a good choice for simple problems with limited budgets. For example, if a group of students came to me and wanted to determine student willingness to pay for green energy in the dorms, I would advise them to set this up as a contingent valuation problem. Similarly, if a poor city in a developing country wanted to look at the willingness to pay for improved water services, I might suggest contingent valuation. In these cases, the policy options are limited and the good is very easy to define, so sacrificing the ability of choice modeling to deal with complexity for the ease of contingent valuation could be a good idea. In the case of the 316b regulation, which is complex in all dimensions, choice-modeling is the only choice. Fortunately, EPA committed sufficient resources to allow a state of the art study to take place and the greater expense associated with the choice modeling exercise was not an issue.

2. Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature? EPA contracted with two of the most experienced and well-respected researchers in this area,316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 58 who took great pains to do everything “by the book.” All the relevant literature is cited, and at each point where problems are encountered, the solutions are consistent with the literature. I found the discussion of the test surveys and focus groups to be very useful, as a key issue in evaluating the appropriateness of the survey is whether people really understood the somewhat complicated attributes, the relationship to each other, and how policy might affect them. I found the focus group comments on the utility of the graphics to be reassuring. I also thought that the survey was visually appealing. This may sound like a minor point, but a survey which is dull-looking will generate respondent fatigue much more rapidly. The survey was also easy to read. I made sure to examine these surveys with the view scale set to “actual size” and I could read it without reading glasses. Color contrast on the survey made it easy to read and less fatiguing to the eyes.

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation? I thought the survey booklets were informative without creating biases. In general, survey researchers walk a fine line in terms of providing the appropriate information but not actually influencing the values that people hold. The valuation problem in this case is complex, and technical information must be provided on how environmental impacts occur, what the environmental impacts are, and how policy actions can reduce the impacts. The survey does an excellent job of this without sinking to the level of propaganda or advocacy, or giving accidental cues that might cause people to disproportionately accept or reject a policy bundle. This is particularly important with respect to the quality variables. This is a rather complicated situation because three of the environmental variables (commercial fish populations, fish populations in general, and aquatic ecosystem condition) are only partially influenced by 316b policies, and the fourth environmental variable (fish saved) can also influence the other three. The information in the survey booklet helps the respondent think about how quality can influenced by the proposed policy remedies without inserting a particular set of values into the information. In particular, the treatment of option C, the status quo, was well constructed in the survey. In many choice-modeling exercises the status quo is ambiguous, left to the respondent to define and implemented in the estimations with all zeros in the attribute levels (in the case of categorical attribute variables). In the 316(b) survey study, respondents know the level of the four quality variables that are associated with the status quo, and how this compared to potential changes as a result of policy. Although the preliminary information was a bit on the long side, the researchers compensated for this by only asking the respondents to evaluate three choice sets, rather than five or six which is usually the case in choice-modeling. For this reason, it is unlikely that respondent fatigue set in before the choice sets where completed.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 59 4. Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes? I am fairly confident in the robustness of the estimates for several reasons. First, the attributes were constructed in a fashion where they could enter the logit estimation as continuous variables rather than categorical variables. This avoids the problem of all statistical tests being relative to the left out variable. The continuous nature of the variables allows some of the attribute levels to be right around the magnitude of changes associated with the proposed policies, and some to either side of this interval. It is a good range of attribute levels which does not require extrapolations to be made outside of the range of the data that is collected in the choice experiment. In short, the continuous form of the attribute levels allows for a test across policy options that would not be much more difficult in the presence of categorically defined variables. Second, for the most part, the coefficients are very similar across the different regions and any significant differences can be accounted for by region-specific characteristics in the type of fish (highly anadromous in the Pacific compared to the South-East, for example). One critical aspect to assure the validity of a choice-modeling study is to choose an appropriate set of attribute bundles to compare in the choice sets. It is impractical to try to include all possible comparisons in the choice sets, so the researchers must come up with a workable number of comparisons to capture the variation associated with the complete set of variation, leading to coefficient estimates that are efficient. EPA employed a fractional factorial experimental design, eliminating those comparisons that might lead to protest responses or puzzlement (a comparison where one option was better than the other across all attributes, for example, the definition of the 72 alternative sets of comparisons follows the literature, and sample size was appropriate for the number of choice sets evaluated. The 72 sets were appropriately distributed across the alternative versions of the surveys in each region.

5. Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates? The implementation of the survey involved no shortcuts deviating from the original plan. The sample sizes and response rates meet or exceed the guidelines established by the peer reviewed literature and the federal government. Response rates were adequate and potential biases associated with non-response were appropriately treated with the weighted model (see response immediately below.

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior? This is an important question, as empirical results are often dictated by the choice of functional form. In terms of discrete choice models, there are many choices that can be made and it is316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 60 appropriate to conduct estimations with multiple functional forms. Linear models are standard and quadratic models are often utilized to try to capture diminishing marginal utility associated with increasing attribute levels. Given the small number of attribute levels, I did not find it surprising that the peak of the quadratic function came within the range of attribute levels, implying negative marginal utility associated with moving to the highest attribute levels. For this reason, it was appropriate to skip the quadratic logit functions when computing values for policy analysis. Given the non-response bias which, if uncorrected, could lead to biases, the weighted linear model is the most appropriate functional form for calculating values associated with marginal changes in the attribute levels. Even though the EPA is not using the values calculated from the unweighted linear models, it was appropriate to show those values in the tables so the reader could compare these values to the values based on the weighted regressions. It is certainly appropriate not to calculate values based on the quadratic regressions, as the coefficients on the quadratic term do not conform to economic theory. I was surprised that this question did not ask about the choice of mixed logit models versus traditional multinomial logit models and the estimation of the cost variable as a fixed coefficient, as these seems to me to be more controversial issues. However, the choice of mixed logit models to deal with heterogeneity in the sample is a well-justified choice. In many other dimensions, the mixed model is less restrictive than traditional multinomial logit. Similarly, the mix of random and fixed coefficients in the estimation represents valid choices given the data set that was developed.

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature? This is an area in which I expressed some concern in the 2006 peer review, and I was glad to see it so prominently addressed in the research, with a dedicated nonresponse study. My first concerns were actually addressed in the EPA response to the 2006 peer review comments. I was concerned with the “lazy husband” problem and the absence of cellphone-only households in the study. The methodology used to avoid the lazy husband problem completely negates this source of bias. Unfortunately, the survey research community has not yet come up with a way to adequately address the cellphone only household problem, but to the extent that cellphone only households are likely to have certain socio-demographic characteristics (young, single, no children, etc.) it can be determined if households that have these characteristics are underrepresented in the survey. I do not believe that this negatively influenced the nonresponse study. As I prepared to complete this new peer review, I was very worried that the survey may disproportionately have nonresponses from people who thought the government should have a minimal role. This could cause a bias that would be uncorrectable if people with this philosophy were not specifically correlated with demographic variables that appeared in the census. My fears were alleviated by the question related to whether the participants in the nonresponse study316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 61 thought about the role of government involvement in environmental protection. For most of the regions, slightly less than ninety percent of the participants thought that the government should be at least somewhat involved in environmental protection, with forty percent stating it should be highly involved. This was a few percentage points lower in the south. The response to this question indicates that their initial nonresponse was due to other factors and not due to a fundamental philosophical conflict with the proposed policy. Another interesting test of the importance of biases related to philosophical self-selection in the survey can be found in the questions related to the environment. Both respondents to the original survey and respondents to the nonresponse study answered the same question related to their perspective on the importance of aquatic ecosystems. The percentage of responses at each step of the five step importance scale was remarkably similar, indicating that nonresponse was not due to people who were less concerned about aquatic ecosystems. Some patterns in nonresponses were noticed in relation to certain socio-demographic variables, such as age, education, gender and employment were discovered in the analysis. These formed these patterns and the potential biases that might be generated in the final value estimates generated the need to conduct weighted logit regressions instead of unweighted estimates. The methods that EPA used to create the weights corresponded to best practice in the literature, and should result in estimates that are not biased by patterns of nonresponse.

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses? First let me say that I thought scope tests were correctly performed, but I would like to point out that scope tests are really a red herring and the scope issues are not as important as some portray them to be. Even with contingent valuation, scope issues are overstated. The scope issue began to be discussed as a result of the contingent value work that was done to measure the benefits of the Clean Air Act and Amendments, with a lot of research focusing on the willingness to pay for visibility in national parks. Several studies looked at willingness to pay for a particular western national park and for the system of parks in the west, and found that the two values were not substantially different. Economists were quick to criticize this, as a more encompassing good should have a higher value than a less encompassing good. What the critics failed to realize, however, is that it is highly likely that the respondents would think that a set of policies that would reduce air pollution in one western park would reduce air pollution in other western parks, as well as nearby cities. If they are actually looking at the policy in a broader sense, the result is to be expected. Accidently generating a scope issue through inadequate understanding of survey questions and policy effects can be guarded against by appropriate use of pilot surveys, individual debriefings and focus groups. More excitement was generated about scope in the Exxon Valdez debacle, but this was largely by studies that were designed to create a scope problem. The bottom line is that I don’t think scope issues are all that important for contingent valuation, and as EPA points out in the supporting documents, it is even less of a problem in choice modeling experiments, because scope is naturally built into the alternative levels of attributes that constitute the choice sets. It is appropriate to test for scope issues, even it is expected that they are not likely. EPA deals316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 62 with the issue directly with several tests for the problem. The first test is the natural test that coefficients on attribute levels have a positive slope, indicating that people are willing to pay more for higher levels of the attribute. They were not able to demonstrate declining levels of marginal value with the quadratic estimations, but this is not a real problem. The “external validity” test of scope that they initiated was cleverly done, splitting the sample into option a responses and option b responses, and showing that for both option a and option b, the choice set was more likely to be chosen if it higher levels of the attributes. The geographic scope test showed willingness to pay to be higher for regional improvements than for a national improvement, although economic theory would suggest that the national improvements should have a higher value than the regional improvements. However, this does not surprise me, as people are more focused on improvements in their own vicinity, and this is not a global public good in the sense of reducing global warming or species extinction. Respondents will be uncertain about how national improvements are manifest in their own regions, and this uncertainty and less willingness to pay for improvements which benefit people in other regions adequately explain the difference between the regional and national values. I would not have bothered to do this test, for exactly these reasons. These tests were appropriate and well-executed. I do not see the need to conduct additional tests.

9. What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule? Of course, this question is the most important of the entire peer review of the survey and resulting values. First as indicated in my introductory remarks and the answers to the particular questions, I am confident that the research was properly done and conforms to the highest scientific standards. It is not just acceptable, but actually provides a blueprint for others to follow when conducting their own research. The survey provides important information that was not in the original benefits assessment. The original benefit assessment used a benefits transfer method for evaluating non-use benefits. I do not want to imply that benefits transfer methods are a second-class method, but when resources are available to do a valuation study of the specific policy question, it is much better to do the valuation study and use the results, then to try to massage the results of studies of other policies or other regions to get some sort of estimate of the values of the potential 316(b) changes. Given the complexity of the environmental variables, the likelihood that different responses will be forthcoming from different regions and the availability of resources to conduct a state of the art survey, it is better to use measure the values directly than to use benefit transfer. Moreover, I like the way the estimates of the value of fish saved can be used as a lower bound on non-use values. This is a significant addition to the analysis. Since the survey and analysis were conducted with the appropriate adherence to theoretical and empirical precedents, I would enthusiastically endorse using these values in a cost benefit analysis of the 316(b) rule.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 63 10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner? Uncertainty is omnipresent in empirical research and must be understood before the results can be used. It is important to discuss likely sources of uncertainty, the magnitude of the uncertainty from these sources, and the implications of the uncertainty for basing decisions on the results of the research. In all areas, I think EPA does a good job. The first source of uncertainty is whether survey respondents really understand the survey. If they are responding to information that they do not understand, or are misinterpreting, then the results are unlikely to be meaningful. This is a particular concern in the 316b survey, because of the complexity of the environmental changes that are being examined. EPA addresses this uncertainty in two fashions. First, this is addressed through the use of focus groups and second through the use of particular questions in the survey itself. The focus group discussion in the 316(b) Stated Preference Survey Peer Review Document is not very intensive, but the supplemental document Executive Summary of Findings from 2010 Focus Groups Conducted Under EPA ICR #2402.01 is very complete. Discussion includes the following topics:

1. Presentation of Information to Survey Respondents 2. Interpretation of Information by Respondents 3. Survey Scenario and Respondent Decision-making 4. Budget Issues 5. Respondent Motivations

I am not going to summarize the 17 pages of discussion in this memo, but comment on a few of the things I thought most important. First, going in, I was a bit nervous about the “fish saved” variable and if people could really wrap their minds around this concept. I thought that it was particularly important that “fish saved” was well understood, because of the four environmental attribute, fish saved was the one that was most directly affected by the 316b policies. If the variable was well-understood, then the willingness to pay estimates for changes in this attribute could serve as a lower bound on non-use values. It would be a lower bound because it would not fully consider non-use values associated with overall aquatic ecosystem health. I was concerned about the fish saved variable for two reasons. First, I was not sure if people would understand the aggregation of eggs, larvae and young fish into the fish saved variable. Second, I was concerned that people might solely value fish saved to prevent death and suffering of fish, and not consider the broader ecological effects of fish saved. After reading the discussion of the understanding of the variable by the respondents, I feel that they did understand it well enough that it could serve as a basis of policy analysis. Second, I was concerned about the possible existence of warm glow effects. Again, the discussion of those participating in the focus groups convinced me that they are responding316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 64 because of the anticipated environmental changes, and not because of an overall desire to do good. This, coupled with their consideration of budget constraints, leads to a conclusion that a lack of understanding of the surveys or the policies is not a significant source of uncertainty in the final results. The survey itself contains a question (#8) which addresses some of the uncertainty issues. The responses to this question are contained in Table 6.3. The first set of questions has to do with their understanding of the survey and the confidence in their answers. The second set of questions have to do with their understanding of uncertainty, with one questions asking whether the degree to which they agree that the effect of the policies depend on many factors and the extent to which they agree that the future ecological conditions are never 100% certain. Question 8 and the discussion of the focus groups allows the reader to evaluate the degree of uncertainty generated by the information (or lack of) and the survey itself. The estimations themselves contain information which can be used to develop confidence intervals on the value estimates which provide insight into the uncertainty generated by the error terms of the regression. The discussion of uncertain is transparent and puts the reader in a good position to determine how confident that they are in the results. In my opinion, the uncertainty is at a level that it does not interfere with the ability to use the results in policy analysis. Please provide EPA with any additional comments not covered by the above questions that you think are important to any assessment of this survey effort. At this point, I think I have covered the important aspects of the research. I would be happy to answer any questions or respond in a more detailed fashion.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 65 PEER REVIEW COMMENTS

Dan Phaneuf316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 66 Comments on “316(b) stated preference survey peer review document” – Dan Phaneuf

Response to specific questions
Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values?
Stated preference methods in general are flexible enough to estimate both use and non-use values. The choice between contingent valuation and choice experiments in this regard is not about which one can measure which value. Rather, the choice reflects the needs of the application. When the objective is to understand how multiple, different attributes affect value – regardless of the type of value – a choice experiment is the appropriate choice. Given the importance of the multiple ecological outcomes in this application a choice experiment was a sensible approach.
Were the procedures EPA used to develop and test the proposed survey instrument consistent with current best practices for stated preference surveys, as reflected in the peer-reviewed literature?
The survey development process seems to be one of the strengths of this study. The number of focus groups and amount of pre-fielding testing exceeds what one usually finds in the published literature. As such I judge the procedures to have conformed to current best practice.
Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation?
I have several comments regarding the design of the survey booklet. Most of these are small negative comments, which I do not believe affect the validity of the survey on net. I mention them mainly for completeness.
Regarding the design of the survey I have three specific and one general observation. First, the environmental good is complex, which necessitated the use of substantial detail in describing it. This is a positive in that stated preference surveys are more likely to produce valid results when the good in question (and the constructed market) are described with a high level of specificity. There are downsides to substantial detail, however. I found the first few pages of the survey required a lot of effort to read and digest. The graphics are generally clear and helpful, but translating the general information into the 100 point attribute scales required more than casual study. This complexity might introduce noise into the estimates – i.e. the scale of utility might be large for some – which could explain some of the insignificant main effects in some of the models.
Second, the 100 point scale for the attributes works well as a communication vehicle, but it316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 67 makes it more difficult to digest the physical quantities they represent. For example, it is hard to place 0.1 billion fish saved into the context of the full population – particularly later in the survey, several pages after figures on proportionate losses were presented. As such peoples’ answers could reflect in part a noisy interpretation of the magnitude and ecological importance of the physical benefits.
My third observation relates to the second, and is based on what the primary source of value is for people. I generally accept the notion that there may be non-use value for this or similar resources. However, if people are not clear on the magnitude and meaning of the policy – i.e. saving a small proportion of common fish stocks – they may attribute more non-use value than they otherwise would. More generally, the intuition for why this commodity – non-threatened, common fish populations – would have substantial non-use value should be made clearer. We usually think of non-use value as arising from something fairly unique with few substitutes, and so perhaps more evidence marshaling from the focus group records is necessary to provide a qualitative sense of the potential for substantial non-use value – relative to the potential for people to have inferred a greater ecological threat is being avoided.
These details aside, I generally find the design of the survey to be adequate for the task. The systematic way in which the study is able to directly map between policy objectives, the definition and levels of the stated preference attributes, and household preferences is a clear strength of the study and outweighs the concerns mentioned above. I find the description of the environmental problem to be suitably neutral in tone.
Regarding the potential for strategic behavior I have two observations. First, the use of a compulsory payment mechanism is incentive compatible, which is a plus. Formally, however, two additional features of the design would technically be considered incentive incompatible. A departure from the binary choice referendum format can only be demand revealing when multiple levels of the commodity can simultaneously exist. Here the commodity is close to a pure public good since the attributes are defined for the entire region, yet people are able to select from two different outcomes along with the status quo. One can tell stories in which a person would, for strategic reasons, select his non-preferred option. The use of multiple choice tasks per respondent is also, technically, not incentive compatible. I should be clear, however, that the latter is nearly universally dismissed as a concern in choice modeling and hence is not a problem here. The former is of marginally more concern. However, in practice I do not believe this design feature influenced the results in any measurable way.
I also have a few observations on efforts to minimize hypothetical bias. Recent research stresses the need for respondents to view the survey as consequential, in the sense that its results have the potential to influence something they care about. I have mixed reactions on this here. On the one hard there are a few places where the survey reminds people that their answers will help the government decide which policies to enact or will help the government learn if households are willing to pay the costs. On the other hand the survey itself does not contain a clear, central message that one might call a consequentiality treatment (I did not see a cover letter for the actual survey – there might have been more there). On net I am inclined to think that the importance reminders, general professionalism of the survey, and complexity of the issue led people to perceive the survey as consequential.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 68 Two other points regarding hypothetical bias relate to the payment vehicle. As best I could tell the survey did not contain the usual reminder for people to consider their budget constraint, what they would give up, etc. when answering the payment question. Likewise I am not a big fan of the ‘general cost of living’ payment vehicle, though I have used it myself and can appreciate its merits as applying to everyone. I think it would be useful to highlight some focus group or cognitive interview evidence that people accepted this payment vehicle as realistic, and were considering their budget constraints when answering.
Finally, I think the survey did an adequate job of including rationality checks. Questions 7 and 8 look like useful diagnostic vehicles, and might perhaps be useful for some estimation tasks.
Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes?
I think the experimental design is fine conditional on the decision to use three levels for the important attributes. In this sense I believe it was the correct decision to focus on a main effects design so as to avoid the need to ask respondents additional questions (a larger sample or more responses per person would be needed to estimate interaction effects). The one point of criticism I have relates to the use of only three attributes for fish saved, and perhaps for the other ecological indicators. I would expect there to be a non-linear relationship in marginal utilities for these outcomes. While three levels is in principle enough to trace a curve it can be difficult I practice, as was shown by the effort to include a quadratic term in the models. I think this feature of the design can be overcome with some additional analysis, as I discuss below.
Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates?
I found the arguments used to decide on the original sample convincing, and I think the number of completed surveys received ex post is adequate for the statistical task. I don’t have further comments or complaints on this aspect of the study.
EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior?
There are several things I might have liked to have seen as robustness checks, before concluding that the weighted/linear/full covariance/MXL model should be preferred. First, I like the mixed logit model and as such I agree with the decision to use it here. However, seeing how other models perform would give me a better sense of what the mixed logit specification is buying. An advantage of the simple conditional logit model is its transparency, and so I would have liked to see this presented as a straw man to get a sense of how the data look when stripped down.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 69 Similarly it is often numerically difficult to estimate non-zero covariance elements of the mixing distribution, particularly from a classical perspective. Seeing mixed logit results with a coefficient independence assumption might, again, help one judge the quality of the preferred model. In sum, while it may be only personal opinion, I don’t think it provides enough robustness checking to only present on class of model (particularly one that might be prone to some numerical anomalies).
Second, I would like to see additional specifications. Looking at models with dummy variables for different non-price attribute levels (rather than making them continuous) would help give a sense of the types of non-linearity might be present. While such a specification would not meet the policy needs directly it would help determine the extent to which the linear model is a good approximation. It would also be a good way to test for internal scope consistency – i.e. the magnitudes on the estimates of the dummy coefficients should increase. I did see that such a model was discussed in an appendix, but its interpretation was difficult owing to the fact that is was combined with a fairly complicated mixing distribution and numerical problems. Also related to non-linearity I think it would be worthwhile to look at a model that uses logs. This would allow for a smoothly diminishing marginal utility of the 0 to 100 coded attributes, which I think is important – it makes intuitive sense that the value of additional fish saved should fall as we approach the maximum. Furthermore some of the policy options considered in the regulatory analysis involve large, non-marginal changes in attributes that span a large part of the range. This opens up plenty of chances for inaccuracies if the linear model is a poor approximation.
My next point here is speculative. It seems to me there could be significant differences in the scale of utility among those who reported fully understanding the survey in question 8, relative to those who did not. It may be possible to add a relative scale parameter to a conditional logit model, allowing respondents who were less certain – and therefore presumably less informative – to have a smaller influence on parameter estimates. Again, this is speculative – I would want to search the literature for similar approaches before recommending it in a policy determining study.
Finally, as I describe in the next question (with caveats), I found the use of sample weights convincing. My advice would be to focus any additional estimation on the weighted sampled generally, though for the conditional logits or other simple specifications I would present both.
Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature?
I found the non-response analysis to be one of the strengths of the study. It is unusual even among the highest quality studies to devote this much effort to analyzing the selection propertiesof the estimation sample. The presence of some statistical differences aside, I found the nonresponse data encouraging in the extent to which it validated the general representativeness of the primary sample. In short I find the data gathering for the purpose of supporting non-response analysis exemplary.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 70 I am less certain about the methods used to develop the actual weights, mainly because I do not have a lot of personal experience with the techniques employed. I’ll make two comments, conditional on this lack of expertise. First, there is mention of some observations needing to be dropped (because of missing data) in order to implement the sub-group weighting routine. It would be useful to know if this involved more than just a few data lines – i.e. are estimates from non-weighted models that do and do not include the to-be-dropped observations the same? Second, the ‘raking method’ used for two of the regions seems a bit speculative. The only citation to it is mention of a SAS macro that implements the process. I think more needs to besaid about the general performance and acceptability of this methods vis-à-vis using a notweighted model.
Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses?
As mentioned earlier I would like to see an additional variant on the internal scope test that makes use of dummy variables for the different non-price attribute levels. This will help make more transparent issues of scope and the non-linear relationship between the attribute indices and utility.
Specifically on the ‘external’ scope tests I think more can be done to formalize them. The idea seems like a good one, or at least something intuitive to look at. However, the summary statistics that are reported do not provide a way of formally or even informally assessing the statistical presence of scope. I think it would be useful to run a linear probability model on the Option A and Option B sub-samples, including dummy variables for each of the attribute levels outside of price (this could stay continuous), or perhaps keeping the other attributes continuous and just checking the different levels of fish saved. This would allow a formal test of whether the coefficients on the three fish saved levels are different and properly ordered.
What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule?
I find that the design work and data gathering aspects of the study generally meet the necessary conditions for the resulting data to be valid and reliable. In addition, the raw data summaries (on p. 21) show that respondents did seem to make tradeoffs between money and the non-price attributes. However, I find that the current set of reported results alone are not sufficient to judge if the value predictions contained in the peer review document are themselves of high quality. Thus I see the study as having the potential to be superior to that which was used as part of the existing facilities rule, conditional on some robustness checking and additional analysis. As such I would hold back from making a firm recommendation that this survey be used to inform the 316(b) rule cost benefits analysis, but my expectation is that I would eventually be comfortable making such a recommendation.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 71 One caveat to this is that I am not sure what to make of the national model. The issue seems more regional to me, and it is a bit disconcerting to see the lack of geographical scope. My tendency would be to rely on the regional models for policy purposes.
Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner?
Continuing my point above, I think there are remaining uncertainties as to what other specifications would predict and how robust the findings might be to modeling decisions. Thus I think there are some uncertainties in the analysis whose effects on the conclusions are not transparent. To this I will add that the treatment of the ASC in predicting welfare effects for non-marginal changes in the attributes needs some additional thought. There is a line in the document about ignoring it since it was not significantly different from zero. I think there werecases when the ASC was significant, and in any case I think it needs to be a part of the KrinskyRobb bootstrapping procedure – i.e. the distribution of values should enter into the welfare calculations in the same way as the other parameters (some of which are also insignificant) do.
In other dimensions I think EPA has done an admirable job of documenting uncertainties. The quantification of uncertainty arising both from random parameters as well as sampling error is a useful way to illustrate the range of possible predictions.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 72 PEER REVIEW COMMENTS

John Whitehead316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 73 316(b) Peer Review -- Whitehead
The specific charge questions are as follows:
1. Is the choice experiment survey format an appropriate means to estimate total willingness to pay for the characterized policies, including use and nonuse values?
A stated preference discrete choice experiment (using terminology in Carson and Louviere, Env and Res Econ, 2011) will elicit total willingness to pay (WTP). However, more work is needed to make clear that the WTP estimates reported in the EPA report are valid and reliable estimates of nonuse value. Since Krutilla (AER, 1967), nonuse values are often associated with unique natural resources facing irreversible development. Both uniqueness and irreversibility are subjective and relative concepts and should be considered as sufficient but not necessary conditions. Where the resources are not unique (i.e., many substitutes) and development is not reversible (i.e., the species are not threatened or endangered), more effort is needed to justify the use of nonuse values in benefit-cost analysis.
For example, Krutilla described the motives that are likely to lead to nonuse values. The survey gathers information on motives for nonuse values to preserve fish for recreation (evidence of an altruistic motive for those who do not fish), preserve fish for aquatic ecosystems, wanting to know that fish exist in local lakes, rivers and bays, and preserve fish for future generations [question 9]. The report states that most of the respondents had some motive for nonuse values but this only provides weak evidence that nonuse values are valid. If the responses were analyzed out of the RUM framework so that an index of motives was a right hand side variable, the WTP model could be used to extract motivated nonuse values from non-motivated values. In a RUM framework this could be accomplished with interaction of an index of motives with alternative specific constants (i.e., are motivated respondents less likely to choose something other than the status quo?) or with split sample analyses. The RUM models could be estimated with split samples of highly motivated vs. less highly motivated respondents.
On page 17, the first paragraph illuminates a potential problem with the analysis. It is clear that survey respondents were not likely to have any prior information about the issue before they took the survey. It is well-known from the CVM literature that survey respondents can have constructed preferences in a no prior information context and that WTP estimates from constructed preferences are less valid than preferences based on real experience. For example, Whitehead et al. (JEEM, 1995) show that WTP estimates from survey respondents who had no prior information about the topic before the survey did not pass internal validity tests. Respondents who had prior information but were not on-site users and on-site users did pass these tests. When these sub-samples are pooled, the invalid responses are masked by the valid responses. Without the prior information question it is impossible to know whether there is a subsample of invalid WTP estimates.
For these reasons, there is some concern that the nonuse component of the WTP estimates may lack validity relative to the use value component and the WTP estimated with the full samplemay be aggregated over too large a population. The analysis could be conducted with splitsamples of users vs nonusers (there is no question asking about prior information) and those with316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 74 significant nonuse value motives vs. those without to demonstrate the validity of the WTPestimates of those respondents with questionable rationales for valuing these resources. If subsamples are found to have invalid WTP responses then in a sensitivity analysis, WTP should be aggregated over only the sample proportion with valid values for benefit-cost analysis.

2. Were the procedures EPA used to develop and test the proposed survey instrumentconsistent with current best practices for stated preference surveys, as reflected in the peerreviewed literature?
The EPA’s survey design methods were thorough and state of the art.

3. Is the overall quality and design of the survey booklet adequate for its intended purpose? Is the information provided by the survey booklet sufficient and appropriate (e.g., is there appropriate emphasis on the budget constraint facing respondents, are appropriate steps taken to minimize hypothetical and strategic bias)? Does the information provide a sufficiently unbiased perspective to allow valid welfare estimation?
The overall quality and design of the survey booklet was “adequate” but some features of a stated preference survey were not included. Respondents were told to “answer all questions as if this were a real binding vote.” Respondents were told how the cost of the program would affect their budget (in higher prices, etc.) but not reminded that this money could have been spent on other things, which is a typical budget reminder.
Respondents were not asked directly about the consequentiality of their votes (i.e., did they think that their votes would be used by policy makers; Carson and Groves, Env and Res Econ, 2007). Respondents were asked for confidence in their answers and whether they would vote the same way in an election, but these responses are not used in the empirical analysis. Footnote 11 describes the “binding vote” question as evidence for consequentiality. Twenty-two percent of the respondents would not vote the same way and 27% of respondents are not confident about their answers. Nevertheless, the EPA does not consider if confidence had any effect on the validity of responses and aggregates sample WTP over the entire population as if these respondents provide equally valid WTP estimates.
A split-sample analysis of confident vs. not confident respondents and same way voters vs. those who would vote another way would shed some light on this issue. If the split-sample models are similar then the EPA’s approach is not problematic. If WTP has little validity for respondents who are not confident about their responses or who would vote a different way, then the EPA’s aggregation approach is overly optimistic.

4. Does the experimental design allow for the estimation of robust values across the range of regulatory policy options and the scope of the resource changes?
Yes, this is an important feature of a discrete choice experiment.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 75 5. Are the sample sizes and response rates for the regional and national surveys adequate to produce statistically robust estimates?
The regional sample sizes are adequate to obtain statistically significant coefficient estimates especially with discrete choice experiment data where repeated measures are used in a mixed logit. It is difficult not to produce statistically robust estimates with this sort of data and empirical model.
It seems odd that more resources were not devoted to the national survey which has a sample size too small to represent the national population. It is also too small to obtain WTP estimates with confidence intervals tight enough to adequately compare to the weighted regional WTP estimates (see page 68 comment below).

6. EPA estimated unweighted linear, weighted linear and quadratic RUM models for the regional and national data. For the calculation of implicit prices and monetization EPA used the results of the weighted linear models across all the survey regions and national analysis because of empirical and theoretical concerns. Is this approach appropriate for estimating welfare changes associated with expected changes in survey attributes under the 316b regulation? Are there other functional forms that EPA should investigate that might be theoretically or empirically superior?
Using the weighted linear models seems appropriate and is adequately justified. I know of no other functional forms that might outperform the linear and produce values with diminishing marginal utility. But I wonder if this goal is worthy? It may be that there is no diminishing marginal utility within the narrow range of policy impacts proposed by this survey for attributes other than fish saved. And constant marginal utility is not unlikely for fish saved. Economic theory does not have much to say about the demand that generates nonuse value.

7. Is the nonresponse study design adequate to evaluate the potential for non-response bias? Are the steps EPA used to correct for non-response bias adequate to address the potential biases identified? Is EPA’s development of weighting values and their use consistent with current best practices in the literature?
The response rates are low (albeit, about as high as you can reasonably expect these days) and raise the issue of nonresponse and sample selection bias. Nonresponse bias exists when the sample is not representative on known characteristics. The EPA makes an honest effort to gather information on respondents with an overall 25% response rate for the nonrespondent survey, but still manages to obtain information on less than 10% (=619/7118) of nonrespondents.
Sample selection bias exists when the sample is not representative on unknown characteristics. There is evidence in the survey methods literature that shows that response rates are a function of survey salience. The respondents, especially to a mail survey, are those who care most about the subject. Those respondents who care most about the subject are likely to have higher WTP for the policy.
One approach is to estimate a Heckman type sample selection model with the samples of316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 76 respondents and nonrespondents with the dependent voting variable converted to a zero-one variable for a probit. If there is no statistical evidence of sample selection bias then the EPA’s aggregation approach may be appropriate.
In the absence of evidence to the contrary and given the possibility of sample selection bias, the aggregation rule, assuming that weighted WTP responses adequately reflect nonrespondent WTP is overly optimistic. It cannot be ruled out that nonrespondent WTP is less than respondent WTP. An overly pessimistic aggregation rule would add WTP responses over only the 33% (=2313/7118) of the population that responded. This is the aggregation rule suggested by Mitchell and Carson in their 1989 CVM book for WTP values with little information on nonrespondent WTP. Considering this issue, it is possible that aggregate benefit estimates are biased upwards.

8. Are the scope tests performed correctly and are the results interpreted appropriately? Are there additional tests that could be conducted to further assess the validity of responses?
The external scope tests are not adequate to demonstrate responsiveness to scope, at least according to the external scope test standard set by CVM critics. The EPA report performs internal validity tests using multiple scenarios. This test is relatively weak since respondents need only be consistent within themselves to produce WTP values that increase with policy impacts. An external validity test compares responsiveness to scope with split sample survey design.
Within a discrete choice experiment an external validity test can be conducted by using only the first scenario presented to respondents in the RUMs. In effect, the first scenario is a split sample test. The EPA considers the split-sample data in Table 8-10 but only applies the eyeball test. Differences in frequencies across the fish saved scenarios can be tested with a likelihood ratio test. The external validity test should be conducted with the RUM model (e.g., Lew and Wallmo, Env and Res Econ, 2011). Note also that in a mail survey the first response may be contaminated by the following scenarios since respondents could have looked ahead and developed internally consistent answers during the survey.

9. What are your expectations regarding the validity and reliability of the benefit estimates that are provided by this survey, compared to the current stated preference literature? Would you expect the results provided by the survey to be superior to those that EPA prepared as part of the national benefits assessment for the proposed 316(b) existing facilities rule? Would you recommend that the information provided by this survey be used in EPA's cost benefit analysis of the 316(b) rule?
These estimates are superior to those used in the 2011 report since they cover the entire nation and are based on a primary valuation study. However, in spite of relatively low statistical uncertainty, the WTP estimates are highly uncertain for reasons described above. First, it is not clear if roughly 25% of the sample has valid nonuse values (see #1). Second it is not clear if316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 77 roughly 67% of the population represented by nonrespondents have WTP values equal to those of the respondents (see #7). If both of these groups of the populations have WTP equal to zero then the WTP estimates could be as low as 25% of those reported (=100*.75*.33).
Unless evidence can be provided that all of the respondents have valid nonuse values (#2) and nonrespondents have similar WTP values to respondents (#7), I would not recommend that the information provided by this survey be used in EPA’s benefit-cost analysis. The aggregate benefit estimates in Table 10-3 could be used if they are described as the best case analysis (instead of the base case) and sensitivity analysis be performed over the potential range of values. Given the current information, a uniform distribution should be used for the range of values.

10. Has the 316(b) Stated Preference Survey Peer Review Document adequately discussed the potential implications of uncertainties in the analysis, and are the effects on the conclusions drawn presented in a transparent manner?
Uncertainties in the analysis are not fully described or considered. Key uncertainties include validity of nonuse values over the entire sample of respondents and potential differences between respondent and nonrespondent WTP. A number of split sample analyses are necessary to provide information for potential sensitivity analyses. Without these it may be necessary to be more conservative with aggregate benefit estimates (see #9).
Another approach is to estimate latent class models which are able to estimate different (probabilistic) groups of responses. With data such as these, without information on prior information for the respondents, a latent class model could estimate two or more vectors of coefficients for the choice models with probabilities that any respondent belongs to each group. It is possible that one or more vectors reveals the probability that latent groups of respondents have invalid WTP estimates (i.e., statistically insignificant cost and benefit coefficients). Haab et al. (Marine Res Econ, forthcoming) estimate conditional logit, mixed logit and latent class models and produce a range of WTP estimates. They suggest that this range could be used to provide a range of WTP estimates for sensitivity analyses. A similar approach could be taken here.

Please provide EPA with any additional comments not covered by the above questions that you think are important to any assessment of this survey effort.
Page 8: “Appropriately designed choice-based SP methods may also reduce strategic, hypothetical and other possible biases that can result from asking survey questions versus assessing WTP through market transactions or binding referenda.” During the CVM debate discrete choice experiments (i.e., “choice-based SP methods”) were introduced. The mood amongst many environmental economists was that choice experiments were preferred for reasons not related to the problems of the CVM (which are mentioned in the sentence above) such as consistency with random utility theory, an interest in new and advance econometric methods and publishability. As such, it was asserted that choice experiments were a silver bullet that solved many problems associated with the CVM. I have not seen much comparative research to back up316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 78 this assertion. For example, the next paragraph describes the extensive research conducted on hypothetical bias in the CVM. No reference is made to studies assessing the potential hypothetical bias that may result from stated preference choice experiments. Unless there are a number of studies that can be referenced here to back up the assertion, this sentence should be removed.
Page 9: I’m not sure what “older forms of SP analysis” refers to. Is this the CVM? It might be better to go ahead and state that the comparison is being made to the older forms of the CVM. But why is this comparison appropriate? The EPA should be comparing discrete choice experiments to newer forms of CVM that are not “typically contingent upon a single specification of ecological effects.” It is straightforward to design CVM surveys to ask multiple WTP questions (e.g., Christie and Azevedo, J of Agr Econ, 2008). If the EPA feels that it is more efficient to estimate the WTP for characteristics of a policy with discrete choice experiments relative to the multiple question CVM then some supporting references should be provided.
Page 13: “an unharvested population is typically believed to be approximately three times as large as MSY.” Please provide a reference.
Page 21: Why was a mail survey chosen over an internet survey with all of the advances in internet sampling that have occurred over the past 10 years or so? With all of the mailings it seems that even Knowledge Networks would have been less expensive and more effective. Note that Lew and Wallmo (Env and Res Econ, 2011) used Knowledge Networks for a federal agency survey.
Page 21: Was the Pacific response rate 32% or 40% = 311/(1040-255)? Was the Inland response rate 35% or 33% = 787/(2480-65)? Why was there such a lower rate of undeliverable surveys for the inland sample (2%) compared to the others (7% - 25%)?
Page 22: 46% plus seems like a high rate of recreational seafood consumption. Are there any representative sample statistics with which to compare?
Pages 27-42: Reading section 7 gets tedious with the cut and pasting. Some of this could go into an appendix with a summary in the text.
Page 51: Much of the text above Table 8-5 should be relegated to a footnote. It really doesn’t matter if this study has similar pseudo-R2s as other studies.
Page 55: “Unlike open-ended contingent valuation questions, choice experiments provide a direct mechanism for respondents to react to scope and scale of resource changes ....” Why is open-ended CVM the relevant comparison? Open-ended questions have not been the norm since Bishop and Heberlein (AJAE, 1979).
Page 66: What is the WTP per fish saved? How does it compare to other studies?
Table 9-2: The regional implicit prices from the weighted linear model are in a different order316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 79 (NE, SE, IN, PA) compared to Table 9-1 (NE, SE, PA, IN).
Page 68: The point estimate of the weighted regional implicit WTP, $1.28, is within the 90% confidence interval of the national fish saved WTP estimate so there is no reason to discuss any difference in point estimates.
Page 69: The AIE acronym is used before it is introduced as an acronym in this paragraph.

Table 10-1: How is the “reduction in AIE losses” column measured? Is this millions of fish saved?

Suggestions:

1. Further test the validity of nonuse values. Conduct split-sample analyses with:

a. highly (nonuse value) motivated vs. less highly motivated respondents b. users vs. nonusers
c. confident vs. not confident respondents d. same way voters vs. those who would vote a different way in a real referendum 2. Conduct sensitivity analysis considering the possibility that nonrespondent WTP is lower than respondent WTP316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 80 APPENDIX II

CLARIFICATION QUESTION RESPONSES316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 81

Responses to EPA Clarification Questions - Boyle

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review report.
I concur with Diane’s response in terms of investigating sequencing effects in responses to the choice questions. I do not think that your design will allow the same investigations as conducted in Day et al. or Holmes and Boyle. Diane can correct me if I am wrong about the Day paper as she is a co-author.
From the EPA report I conclude that there were 24 versions with each version having three Option A/B pairings. This implies to me that the sequence of choice questions was fixed within each of the 24 versions, but varied across versions. Thus, the choice question sequence is not completely fixed as Diane assumes, but the fixed order within each of the 24 versions would need to be accounted for in any analysis. In Holmes and Boyle, ordering of pairs was random so we did not have this fixed component in our analysis.
We know that sequencing effects exists in sequential choice questions so your study is not unique in this concern. Moreover, most studies do not investigate this potential confounding feature. In addition, once you identify potential sequence effects I do not believe there is consensus in the literature on the responses that are most appropriate to use. The Day paper helps us further understand this issue, but does not provide a conclusive answer.
2. Alternative log model - Clarifications as to the specific functional form being recommended. Raised by Dan Phaneuf on p.21 of the peer review report.
I believe Dan was referring to logging the quantitative variables as opposed to linear specifications. However, I think discrete coding, as dummy variables or effects coding, would be more appropriate. Logging these variables continues to assume continuous response effects and this assumption may or may not be correct, and can be investigated with discrete coding.
3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that are recommended by reviewers. Diane Dupont questioned the representativeness of the sample at multiple points in her comments, focusing on the Pacific and Inland regions.
There are two approaches here. One is to perform a Heckman style selection model, but as I noted in my previous comments the statistical properties have not been worked out a random parameters model. Woolridge outlines a panel estimator for the selection model that can be applied with fixed parameter models. If you do not want to go the316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 82 panel estimator approach you could randomly draw one choice observation from each respondent and estimate the selection model. If you repeat this, say with 500 draws, you will find that the result should be very robust to the random selection.
The other approach is to consider what variables you are going to use to weight your data and if these are the variables that truly influence responses to the choice questions. While your procedures for considering sample selection are consistent with best practice you follow the same mistake of most of these analyses of using available variables and not asking if these variables affect responses to the choice questions and ultimately value estimates. Ask what covariates that you have population data on and influence choice responses and use these variables in the robustness checks you do for sample selection.
4. Does the main effects experimental design of the survey interfere or complicate the interpretation of models that use interactive terms and split samples. If so, can reviewers offer suggestions on use and interpretation of results?
The main effects design implies that you do not have the most efficient design for estimating interaction effects, but it does not preclude you from doing this analysis. It may be the case that you are unable to identify some interaction effects that would become significant with a different design. If, an interaction effect is highly insignificant I would not expect that are different design would change such an outcome to a significant effect.
At face value, the main effects design should not interfere or complicate split sample analyses, but I think that your sample sizes may result in a problem. First, I think your sample sizes are minimally sufficient for estimation with all data so I am concerned about the credibility of any sample splits. The concern for the main effects design is if the designs are not equally represented in each of your split samples, i.e., respondents with one set of design points coincidentally happen to be more represented in one sample and less in another sample. This is something you can and should check in each split sample before progressing with the statistical analyses.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different baselines across regions. How would you transform data to accommodate the fact that the meaning of a percentage point change in each of the environmental attributes differs across the regions?
One possibility is to include region fixed effect variables and then interact the fixed effect variables with the attribute variables.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 83

Responses to EPA Clarification Questions – Dupont

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review report.
Since the survey was a mail survey and the order of the questions was not varied across respondents, I was concerned that responses might reflect the particular and unvarying order used. Specifically, I was concerned that respondents might make comparisons across choice tasks that would result in a type of systematic bias. There are tests that can be made to determine whether there are precedent dependent effects (that is, where responses to choice tasks 2,3,4 depend upon attributes seen in prior choice tasks 1,2,3). This essentially involves creating a series of additional explanatory variables that provide comparative information on previously seen attributes and looking at whether the coefficients are individually or jointly statistically significant. Please see Day et al (2012) in the Journal of Environmental Economics and Management (“Ordering Effects and Choice Set Awareness in Repeat-response Stated Preference Studies”). Holmes and Boyle (2005) have a similar type of test in their Land Economics paper “Dynamic Learning and Context-Dependence in Sequential, Attribute-Based, Stated-Preference Valuation Questions”.

2. Alternative log model - Clarifications as to the specific functional form being recommended. Raised by Dan Phaneuf on p.21 of the peer review report.
I agreed with Dan on this point. My understanding is that he meant that the attributes that arevalued between 0 and 100% should be transformed into natural log values and the model reestimated with these natural logs. Presumably, household costs would also be transformed.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that are recommended by reviewers. Diane Dupont questioned the representativeness of the sample at multiple points in her comments, focusing on the Pacific and Inland regions.

In order to deal with sample issues that may arise between the main sample and the non-response sample, a Heckman type approach could be used that jointly estimates a selection equation with the conditional logit and/or mixed logit choice outcome equation to see whether the errors are correlated. The selection equation could be parameterized with either socio-demographic responses or responses to common questions in surveys to participants and non-response individuals who completed the follow-up short survey. It is much harder to deal with non-representativeness that may arise between census data characteristics and sample characteristics. The researchers do not have responses from the census population to questions relating to attribute importance and recreational interests, etc. What the researchers do have is information from both the main sample and the non-response sample to316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 84 the common set of debriefing questions. As I recall, there are some socio-demographic differences between these two samples, and, in some cases, the non-responses sample was more representative of the underlying population in terms of observable characteristics. It would be helpful to see a breakdown by gender, race, income, etc. (the socio-demographic variables that were found to differ between the various main samples and non-response samples) of the responses by main sample and non-response sample to the question... “how important is the protection of aquatic ecosystems to you?” This would assist in better understanding whether the main samples provide robust estimates for a more general population.

4. Does the main effects experimental design of the survey interfere or complicate the interpretation of models that use interactive terms and split samples. If so, can reviewers offer suggestions on use and interpretation of results?
In general, I do not think that the main effects experimental design interferes with or complicates the interpretation of models using split samples based upon variables that are not in the design. However, in this study split samples are used to perform a type of external scope test described on page 57. We have already commented that this is a weak form of test. There may be a problem with the use of interactive terms since the main effects design contains common combinations of attribute levels, so it may be difficult to disentangle the separate effects of attributes. Systematic use of responses to the debriefing questions may assist in interpreting results in this case.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different baselines across regions. How would you transform data to accommodate the fact that the meaning of a percentage point change in each of the environmental attributes differs across the regions?
I think it will be necessary to step back from this level of detail (% change point tied to a specific quantity change in the environmental attribute where the quantity changes differ across regions). One solution is to categorize the attributes into different groups. The logical groups would be low, medium and high changes relative to the specific baseline for that region. Then, I am wondering whether it would be possible to pool the data and estimate an overall model with some recognition that the scale parameter could differ across the four regions.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 85

Responses to EPA Clarification Questions – Kahn
Let me preference my remarks by saying that I do not consider myself to be an expert in the statistical issues associated with choice-modeling, although I think I am a competent practitioner. My comparative advantage in the review process is in the choices that are made in the structure of the survey such as the definition of environmental variables, choice of payment vehicles, additional questions and so on. Consequently, I feel that the EPA should place the greatest weight on the responses of the other reviewers to these questions. At this point, I have only seen Professor Dupont’s responses, and I concur with what she has said. I offer some additional remarks in response to the questions.

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review report.
I did not raise this issue in my original report, but I appreciate Diane Dupont’s insight on this issue. I agree with Professor Dupont’s suggestion.

2. Alternative log model - Clarifications as to the specific functional form being recommended. Raised by Dan Phaneuf on p.21 of the peer review report.
I agree with Professor Dupont in the interpretation of Professor Phaneuf’s remarks. A simple natural log transformation of the key variables would be simple and informative.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that are recommended by reviewers. Diane Dupont questioned the representativeness of the sample at multiple points in her comments, focusing on the Pacific and Inland regions.
Of course, it is very difficult to deal with sample selection issues after the sample has been conducted. The best way that I can think of is to choose a large number of the “attitude” or “preference” questions and see how they vary across the different socio-economic characteristics, being careful to examine cross-cells (e.g. the combination of minority, low income and male) rather than just looking at the broader socio-economic characteristics separately. I agree with Professor Dupont that the Heckman method could be usefully employed. Professor Dupont talks about the parameterization of the selection equation with socio-economic characteristics or attitudinal questions. Care must be taken that both equations are properly instrumented. Another test could be developed with the aggregation of the samples using categorical quality variables. If the data from all regions could be aggregated, then this data set could be split into316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 86 separate socio-demographic categories and separate estimates generated for each category. If the number of categories is held low, the sample size for each category should be sufficient. In order to do this, the distribution of sample designs across all groups must be the approximately equal. If for example, one socioeconomic group was given a disproportionate number of the same sample design, this would degrade the results. The issue of over or under representing a region in the aggregate sample could be easily corrected with a weighted regression. Total benefits could be then be easily computed by multiplying each estimate by the number of people in the category and summing these estimates.

4. Does the main effects experimental design of the survey interfere or complicate the interpretation of models that use interactive terms and split samples. If so, can reviewers offer suggestions on use and interpretation of results?
I do not think there is a problem with split samples, so long as sample size is sufficient. If one is interested in interactive terms, such as the impact of income on willingness to pay, I agree with Professor Dupont on the potential difficulty associated with the main effects experimental design. However, a potential way to get around this issue is to aggregate the data as suggested above (by using categorical quality variables) and estimated separate regressions for different income levels (or whatever the variable of interest). This will, at a minimum, provide some information of the effects of socioeconomic variables on willingness to pay. In order to do this, the distribution of sample designs across all groups must be the same.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different baselines across regions. How would you transform data to accommodate the fact that the meaning of a percentage point change in each of the environmental attributes differs across the regions?
I agree with Professor Dupont that the way to go is to transform the policy interventions intolight, medium and strong responses. This will interject some noise into the experiments, but if tstatistics remain strong, one could go forward with willingness to pay estimates. As mentioned in the responses to questions 3 and 4, moving forward in this direction could aid in answering some of the other questions posed by EPA.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 87

Responses to EPA Clarification Questions – Phaneuf

1. Tests of question ordering: Clarification as to specific tests (if any) that are being requested to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review report.

Intellectually I would be interested in seeing the results of any ordering effects tests that might be conducted, but in terms of making the study more useful for its intended policy purposes I am less certain. Understanding the causes, consequences, and fixes for ordering effects in choice experiments is still something that is being actively researched. Given the Carson and Groves work we might expect there to be ordering effects, so unless we can think of a way that the study’s value would be enhanced by explicitly identifying said effects I am inclined to focus efforts on other exercises.
This said Diane is the person who has worked on ordering issues recently. If she thinks there are ways to improve the policy relevance of the study by conducting some tests, I would be willing to follow her lead.

2. Alternative log model: Clarifications as to the specific functional form being recommended. Raised by Dan Phaneuf on p.21 of the peer review report.
My interest was in seeing two types of models: one that coded the different attribute levels qualitatively (i.e. as dummy variables) and one that used a natural log transformation, ratherthan the linear transformation. The former would allow us to flexibly see the extent of nonlinear effects, and the latter would provide a non-linear but still parsimonious alternative to the linear model. The log specification might be something like the following:

0 1 2 3 4 5 ln( _ ) ln( _ ) ln( _ ) ln( ) ( ), V fish save com fish fish pop aquatic cost               

where the cost variable is kept in the level so as to make the marginal utility of income constant. The issue of looking at conditional (rather than mixed) logit models in order to better explore functional forms came up in this context. I believe my suggestion was to run the dummy variable, linear, and log models as simple logits to make a better case for whichever specification ultimately is deemed best. I had the sense that a continuous model (as opposed to the dummy variable model) was needed for welfare purposes, to allow extrapolation away from the values of the designed attributes. Seeing if the (continuous) linear or (continuous) log specification better matched the pattern from the dummy variable model would help make the case for one or the other.

3. Statistical tests for the effects of sample selection: Clarification on specific tests (if any) that are recommended by reviewers. Diane Dupont questioned the representativeness of the sample at multiple points in her comments, focusing on the Pacific and Inland regions.

I don’t have specific comments on this issue.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 88 4. Does the main effects experimental design of the survey interfere or complicate the interpretation of models that use interactive terms and split samples? If so, can reviewers offer suggestions on use and interpretation of results?
If the interaction terms are (attribute) ×(person characteristic) there should be no problem – this would just divide out the population average main effect into components that vary with respondent characteristics. A split sample should not be a problem with this so long as each of the sub-samples still contains the same mix of survey versions that allow estimation of the main effects in the full sample (could become a practical issue given overall sample sizes).The only point of caution is if the sample split is based on the value of one of the personspecific characteristics. This should not cause econometric problems – just care in interpretation.
If the interaction terms are (attribute) ×(attribute) I think something close to the same logic applies, but the practical difficulties will be greater. The main effects design does not mean the model will be incorrect if you try to estimate interactions – only that you may not have the variability needed to do so with any precision. Splitting the sample will exacerbate – both overall and in comparison, since one sub-sample might end up with more of the ‘useful’ variability for estimating the interactions.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different baselines across regions. How would you transform data to accommodate the fact that the meaning of a percentage point change in each of the environmental attributes differs across the regions?
This is speculation, but I wonder if the log model written out above could help? The coefficients in that case are proportionate changes in utility, which could be meaningful in reference to any of the baselines.
If this does not make sense, perhaps a model that constrains the marginal utility of income to be constant across the regions but then estimates region-specific marginal utilities for the attributes (using region interactions) would provide some precision benefits in terms of a larger sample and some better comparability.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 89

Responses to EPA Clarification Questions –Whitehead
Let me preference my remarks by saying that I do not consider myself to be an expert in the statistical issues associated with choice-modeling ...

1. Tests of question ordering - Clarification as to specific tests (if any) that are being requested to address effects of question ordering. Issue raised by Diane Dupont on p.26 of the peer review report.
I do not have anything to add.

2. Alternative log model - Clarifications as to the specific functional form being recommended. Raised by Dan Phaneuf on p.21 of the peer review report.
I do not have anything to add.

3. Statistical tests for the effects of sample selection - clarification on specific tests (if any) that are recommended by reviewers. Diane Dupont questioned the representativeness of the sample at multiple points in her comments, focusing on the Pacific and Inland regions.
Nonresponse bias exists when the sample is not representative on known characteristics. The EPA does a noble job in testing for nonresponse bias. Sample selection bias exists when the sample is not representative on unknown characteristics. There is evidence in the survey methods literature that shows that response rates are a function of survey salience. The respondents, especially to a mail survey, are those who care most about the subject. Those respondents who care most about the subject may have higher WTP for the policy. See Whitehead, Groothuis and Blomquist (Econ Letters, 1993) for these alternative tests in a CVM context.
One approach to test for sample selection bias is to estimate a bivariate probit with sample selection model. The selection equation could include demographics and attitudinal variables explaining the response to the first survey. The three voting choices (status quo, option A, optionB) could be converted to a zero-one variable to estimate a probit. The zeros would be the statusquo choice. The ones would be the policy choice if it was chosen (with the other policy alternative discarded). If the status quo was chosen then one of the policy alternatives could be randomly discarded. Obviously, the study was not designed for this analysis and the proposed model is not ideal (I’m not aware of a sample selection model for the multinomial logit and its variants), but if there is no statistical evidence of sample selection (i.e., the error terms are not correlated) then there is evidence that EPA’s aggregation approach is appropriate.

4. Does the main effects experimental design of the survey interfere or complicate the316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 90 interpretation of models that use interactive terms and split samples. If so, can reviewers offer suggestions on use and interpretation of results? I do not have anything to add.

5. Can reviewers offer suggestions as to how the combine the regional data, given the different baselines across regions. How would you transform data to accommodate the fact that the meaning of a percentage point change in each of the environmental attributes differs across the regions?
I do not have anything to add.316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 91 APPENDIX III

RESPONSES TO CLARIFICATION QUESTIONS316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 92 Clarification Questions

1. Alternative log models:
Some reviewers proposed to transform attribute levels (including household costs) valued between 0 and 100% into natural log values and re-estimate the model with natural logs. The two variables that are of particular interest in this study (fish_saved and cost) take zero values. We could assign some very small arbitrary values to these variables (e.g., 0.001) and estimate the suggested model. Alternatively, we could use inverse hyperbolic sine (IHS) function instead to investigate nonlinear effects following a study by Layton (2001).
Layton, D. F. 2001. Alternative Approaches for Modeling Concave Willingness to Pay Functions in Conjoint Valuation. Amer. J. of Agric. Econ. 83(5): 1314-1320.
Would the use of the IHS model address your comment/concern?

2. Statistical tests for the effects of sample selection
With regard to sample selection, the peer reviewers have suggested a variety of possible options. EPA has not been able to locate a published model for Heckman selection with multinomial responses. Can reviewers provide a reference for a published model? We would also welcome any additional guidance that the reviewers might provide. In case the Woolridge (Wooldridge?) paper provides this, could the complete reference be provided to us?
In the absence of a first-best solution, the bivariate probit approach suggested byWhitehead seems to be a feasible alternative, and is attractive because the Heckmancorrected bivariate probit is an established approach (i.e., we would be grounding our approach in an established method). In order to accomplish this, we would transform our data into bivariate form by dropping or grouping observations, as suggested by the reviewer. We could then evaluate statistical evidence of sample selection in the transformed data.
We would like clarification on: (1) whether we are correct in our assumption that there is no published and accepted Heckman-correction model for multinomial responses that we could readily apply, and (2) whether the peer reviewers agree that the bivariate approach suggested by Whitehead is a feasible alternative.

Kevin Boyle’s response to clarification question:

1. I think your proposal is fine and is much better than adding a very small number to

zero.

2. Here are a couple of citations to help with panel data:316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 93 http://www.sciencedirect.com/science/article/pii/030440769401645G
http://www.sciencedirect.com/science/article/pii/S0304407610000825
While I was not clear, this is what I was suggesting with the random draws for one choice, but did not specify the bivariate probit, which I should have. Woolridge has approved using this approach for a paper I am currently working on.
Kevin Boyle’s response to John Whitehead comment::
This is the same as my proposal with the exception of randomly drawing one question rather than just focusing on the first question. The random selection is the approach Woolridge recommended to me. We did 500 draws and found no evidence of selection.
Doing you suggestion and the random draw would be a good robustness check.

Diane Dupont’s response to clarification questions:

1. I think the use of the inverse hyperbolic sine function is a good one. For values in the "middle of the distribution", it pretty much can be interpreted like a log function but is also defined at zero. Here are two further articles that use it and support its use.
John B. Burbidge, Lonnie Magee and A. Leslie Robb, 1988. "Alternative Transformations to Handle Extreme Values of the Dependent Variable" Journal of the American Statistical Association Vol. 83, No. 401, pp. 123-127
MacKinnon, James G & Magee, Lonnie, 1990. "Transforming the Dependent Variable in Regression Models," International Economic Review, vol. 31(2), pages 315-39, May.

2. I do not know of any published articles using Heckman type corrections for the type of multinomial model being used in the EPA study. I think Whitehead's bivariate approach is a feasible alternative.

Jim Kahn response to clarification questions:
Sorry I am late in replying. I think Diane's suggestions are very good and do not have anything to add beyond this.

Dan Phaneuf’s response to clarification questions:316(b) Stated Preference Survey Peer Review

Applied Planning Corporation Page 94 1. I think the IHS transformation would be fine - better than a small adjustment, since this would cause the transformed value to stretch large negative. If there is to be an adjustment I think it would be better to add one - i.e. log(fish_saved + 1) - since this would just be a horizontal shift if the axis, preserving the value of zero when the designed level is zero. If the IHS and +1 shift are similar the latter might be preferred on the basis of simplicity, though the former is more elegant.

2. It is correct that Heckman style corrections are not worked out for non-linear models like the logit. The Whitehead approach seems sensible, though I would defer to the others on this topic.

John Whitehead’s response to clarification questions:
I think Diane's suggestions are very good and do not have anything to add beyond this.
(Dr. Whitehead added a few days later)
I might add that to simplify the task a bivariate probit could first be estimated on the first question only for each respondent. That would allow EPA to avoid the scary Woolridge panel stuff in the initial test for sample selection. If there is no sample selection then it might be reasonable to assume that the benefit estimates are not severely biased. If there is sample selection then adjusting the estimates might require some scary Woolridge panel selection model.
JohnWhitehead’s response to Kevin Boyle’s comment:
Ah, now I get what you meant by random selection. I thought it was random selection of the unchosen alternatives (to get a binary dependent variable) with the panel still included.