COMMENTS SUMMARY REPORT

PEER REVIEW PACKAGE FOR “WILLINGNESS TO PAY SURVEY INSTRUMENT FOR §316(B) PHASE III COOLING WATER INTAKE STRUCTURES”

Prepared for:

U.S. Environmental Protection Agency Office of Water Office of Science and Technology Health and Ecological Criteria Division 1301 Constitution Ave., N.W.

Washington, D.C. 20004

Prepared by:

Versar, Inc. 6850 Versar Center Springfield, Virginia 22151

February 2006Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 

### i TABLE OF CONTENTS

I. INTRODUCTION

............................................................ 1

II. CHARGE TO THE PEER REVIEWERS

........................................... 3

III. GENERAL COMMENTS

...................................................... 5

IV. RESPONSE TO CHARGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

V. SPECIFIC COMMENTS ON REPORT

........................................... 25

VI. SPECIFIC COMMENTS ON INFORMATIONAL SLIDE SHOW

..................... 39

VII. COMMENTS ON FULL TEXT OF STATED PREFERENCE SURVEY

................ 46

VIII. ADDITIONAL REFERENCES

................................................. 49

APPENDIX A: REVIEWER COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 Richard C. Bishop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2 Kevin J. Boyle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-16 Timothy C. Haab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-24 James R. Kahn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-28 Alan J. Krupnick

........................................................... A-38Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 1 I. INTRODUCTION

The U.S. Environmental Protection Agency is de veloping regulations under section 316(b) of the Clean Water Act (33 U.S.C. 1251 et seq ). Section 316(b) requires that the location, design, construction and capacity of cooling water intake structures reflect the best technology available for minimizing adverse environmental impact. The EPA estimates that more than 1,500 industrial facilities use large volumes of water from lakes, rivers, estuaries or oceans, or other waters of the U.S. for cooling, including steam electric power plants, pulp and paper makers, chemi cal manufacturers, petroleum refiners, manufacturers of primary metals like iron, steel and al uminum, and manufacturers of other products.

Cooling water intake structures pull aquatic organism s including fish, shellfish, and their eggs and young, into cooling systems (entrainment). There, the organisms may be killed or injured by heat, physical stress, or by chemicals used to clean the cooling sy stem. Larger organisms may be killed or injured when trapped against screens or other structures at the front of an intake structure (impingement). EPA estimates that 533,378,000 (age-one e quivalent) aquatic organisms are impinged and entrained annually at Phase III facilities. The impacts, if any, of these losses on population levels and ecosystem functions are not known.

Under a court-ordered schedule, EPA is developing section 316(b) regulations in three phases. In December 2001, EPA published fina l regulations addressing all new power plants and facilities that withdraw more than two million gallons of water daily from waters of the U.S. (Phase I). The Phase II rule for implementing section 316(b) was promul gated in February 2004 and applies to most existing power-producing facilities that utilize a cooling wate r intake structure and withdraw 50 million gallons per day from waters of the U.S. The Phase III proposed regulation was published for comment in November 2004 and applies to existing manufacturers a nd power plants using smaller volumes of cooling water (i.e., power plants that withdraw less than 50 million gallons per day). EPA must take final action on these regulations by June 1, 2006. The following website contains information about the three phases of the regulations and the supporting analysis: http://www.epa.gov/waterscience/316b/basic.htm

As required under Executive Order 12866, EPA is conduc ting economic impact and cost-benefit analyses for the section 316(b) regulation for Phase III facilities. A comprehensive estimate of total resource value should include both use and non-use values, such that the resulting total social benefit estimates may be compared to total social cost. Developing comp rehensive quantified benefit estimates for the section 316(b) regulation requires consideration of non-use va lues because nearly all (96 percent) of impingement and entrainment losses at CWIS consist of either forage species, or non-landed recreational and commercial species that do not have direct uses or, as a result, direct use values. Although individuals do not use these resources directly they may nevertheless be affected by changes in resource status or quality, such that they would be willing to pay to maintain these resources. It is generally accepted that non-use values may be substantial in some cases, and that failure to recognize such values may lead to improper inferences regarding policy benefits. Many public comments on the proposed section 316(b) regulation for Phase II facilities and the Phase II Notice of Data Availability suggested that a properly designed and conducted stated preference, or conti ngent valuation (CV), survey would be the most appropriate and acceptable method to estimate the total (including use and non-use) benefits of the rule. Stated preference survey methodology is the genera lly accepted means to estimate total values. Stated preference surveys use carefully designed questions to elicit respondents' willingness to pay (WTP) for particular ecological improvements, based on their responses to either discrete choice or open-endedPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 2 questions regarding hypothetical resource improveme nts or programs. Such improvements may include increased protection of aquatic habitats or species with particular attributes.

To assess public policy significance or importance of the ecological gains from the section 316(b) regulation for Phase III facilities, EPA proposes to conduct a stated preference study to measure non-use benefits of reduced fish losses at CWIS due to the regulation. The study would focus on a broad range of aquatic species, including forage fish and a vari ety of fish species harvested by commercial and recreational fishermen. The findings from this study would be used in developing estimates of the economic benefits of the section 316(b) regulation for Phase III facilities. These findings would also be pertinent to economists and policy makers studying changes in fishpopulations and aquatic habitat improvements.

The survey will ask respondents to choose how th ey would vote, if presented with two different hypothetical regulatory options characterized by [a ] changes in annual impingement and entrainment losses of fish and other organisms, [b] effects on l ong-term fish populations, [c] effects on recreational and commercial catch, and [d] an unavoidable cost of living increase for the respondent's household. Respondents will be allowed to "vote" for one of the presented regulatory options, or to choose not to vote for either option. The survey will also ask r espondents to answer questions about their reasons for voting, their level of concern about various policy i ssues, and their affiliations and recreational activities.

Peer review is a process for enhancing a scientific or technical work document so that the decision or position taken by the Agency, based on the technical doc ument, has a sound, credible basis. The goal of the Agency’s Peer Review Policy is to ensure that scientific and technical work products receive appropriate levels of critical scrutiny from scientific and technical experts as part of the overall decision making process. This document was reviewed by Dr. Richard C. Bishop, Dr. Kevin J. Boyle, Dr. Timothy C. Haab, Dr. James R. Kahn, and Dr. Alan J. Krupnick.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 3 II. CHARGE TO THE PEER REVIEWERS

1. Is the choice experiment survey an appropriate stated preference survey format for the estimation of marginal willingness to pay for the characterized policies? Will the survey likely provide useful insights into preferences, tradeoffs, a nd willingness to pay associated with the described policies?

2. Does the effort used in developing and testing the proposed survey meet current standards for stated preference surveys re flected in the literature?

3. Can the policy options be sufficiently well defined such that the survey is likely to provide useful insights into preferences, tradeoffs and willingness to pay? As structured, can respondents – by taking the survey – report values for the actual environmental improvement, where impacts may be large from the perspective of forage fish onl y or impingement and entrainment, but small from the perspective of only recreational and commercia l fish species? Will the survey likely provide useful insights into preferences, tradeoffs, a nd willingness to pay associated with the range of likely policies?

4. Does the information provided in the survey on the baseline ecological conditions, the expected changes, and the expected costs and their in cidence seem sufficient, appropriate, unbiased and accurate? Does the survey provide appropriateand sufficient introductory and debriefing questions to enable identification of protest responses and symbolic responses, to test retention and understanding of information, etc.? Would you suggest any additional debriefing questions?

5. Does the survey sufficiently and appropriately characterize the policy options under consideration? Does the survey place sufficient and appropriate emphasis on the scope of resource changes? Does the survey appropriate ly communicate the scientific uncertainties as well as uncertainties associated with the policies in question?

6. What are your expectations regarding the poten tial validity and reliability of benefit estimates that might be provided by the proposed survey, comparedto the current state of the art in the stated preference literature?

7. Does the survey take appropriate steps to mi nimize hypothetical bias? Does the survey place sufficient and appropriate emphasis on the budget constraint facing respondents, and on potential substitute uses of their budget? Is the payment ve hicle used in the surv ey credible? Does the survey incorporate appropriate language to amelio rate strategic bias, symbolic responses and “yea saying”? Is there any sign of systematic bias in the survey or presented information? If so, how can it be ameliorated?

8. Will the answers to this survey lend themselv es to straightforward economic interpretation of willingness to pay? Are the proposed econometric modeling approach and validity tests appropriate? Would you suggest an alternative modeling approach or alternative or additional validity tests?

9. The national benefits analysis for Phase III of the 316(b) rulemakings must be completed within the next several months and a final rule issued by June 1, 2006. Are there improvements to the survey and associated model that would be feasibl e to implement within the available time frame? If more time was available, how would your answer change?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 4 III. GENERAL COMMENTS

Richard C. Bishop Before beginning my comments, let me suggest that y ou take them with a grain of salt in the following sense. I will definitely focus on potential problems, fo rgetting to tell you about all the things that are done well. Much hard work is evident in what I saw here and it has produced a lot of progress. Also, survey design is as much an art as a science and we who work in the field frequently disagree about best practice. In the end, those working on this study will have to use their own judgment about which of my comments and suggestions to focus on in final revisions before the pretest. I am kind of like a dentist doing a checkup. Indeed, reading my comments may seem like a trip to the dentist! I like to probe around, searching for the survey equivalent of cavities. I hope that some of my comments will help you do an excellent study. But you will and indeed shoul d decide to ignore others either because you are not sure they are helpful or you can’t take the time or expend the energy to fix what may be low priority problems. With these caveats, here are my general comments:

• The work is being conducted in a total valuation framework, which is appropriate to the case. A narrower framework that ignores nonuse values would ri sk neglecting most of the benefits of the regulations.

• So far as I know, the researchers are correct when th ey suggest that no previous studies have valued changes in fish stocks including forage, recreational and commercial species over a broad geographic area. This will be a path-breaking study in that regard.

• To some extent, I suppose I am passing the buck, but I would recommend that a survey expert be employed as a consultant to help with this study . A heavy hitter like Stanley Presser or Don Dillman or Jon Krosnick would be great if you can get oneof them; another possibility would be Nora Cate Shaeffer of UW-Madison. Nora is less well known in environmental economics, but her credentials in survey methodology are impeccable. I have pe rsonally worked with her on stated-preference studies (e.g., she is a co-author of the paper cite d as Heberlein, 2005, in the document I was sent). Such a person could help evaluate and remedy pr oblems with the instrument including those listed below. In addition, for reasons stated belo w, I am concerned about the plan for addressing nonresponse issues. The sort of person I am recommending might be able to help a lot.

• I am particularly concerned that the slide show is not ready for prime time. In studies like this one, conveying information is crucial and very difficult. A large number of nitty-gritty issues are probed under Specific Comments. Let me raise some broader issues here.

< The document that was sent to me for this review was for the most part very clearly written. Most of the time, I was able to figure out what had been accomplished and what the plans were. Perhaps the biggest shortcoming—if I missed it somewhere, I apologize—was the lack of a description of the procedures that would be usedin the slide show. In what follows, I will assume that respondents will be required to view the slides before they are allowed to open the survey proper; that the pace through the s lide show will be controlled through Knowledge Networks, and not at the discretion of the subject; that the script in Attachment 2 will be applied as voice-over with each slide; and that there will beno way for subjects to review slides after they have gone through the slide show. I will come back to this last issue in a moment.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 5 < The slide show and the survey are currently planne d as two separate steps. This is a major part of the research strategy and merits very careful consideration. There is no easy formula for conveying all the information needed by subjects ina study like this. Admittedly, there are some precedents in the literature for setting up the in formation as a separate step, as noted in EPA’s write-up. A nice clean break between information and survey questions may work better, but after considering this approach I think I still pr efer to mix information and questions together, which is how I am tackling the information probl em these days. In the Internet surveys I am working on, subjects are free to go back and forth between frames. Information is presented in modest doses. A frame or two of information is given and then a question pops up and they can go back to the information if they want to. Indeed, properly framed questions are an incentive to look back if some parts of the information are not yet clear.

< Frankly, the amount of information to be communicates all at once seems huge. Given detail in some of the slides, would it be unreasonable to assume it will take subjects close half an hour (30 seconds per slide) to view the slide show? If so , that is half an hour before they answer one survey question. The total process might take 45 minutes to an hour. That is asking a lot even of Knowledge Networks subjects. If you do a separ ate slide show, it should be cut to 15 minutes or 20 minutes maximum and the pace should be set to be conducive to learning. Has having a separate slide show perhaps made it too tempting to add more and more slides?

< If you stick with the slide show, I would recomme nd that you make it interactive with the survey so that subjects can go back and review informati on if they want. Try putting links in the survey so they can go back to one or more slides and th en return to the place in the survey where they stopped. Also, they need to be able to turn th e slide show on and off. (What if the phone rings or the kids start yelling in the other room?)

• Again, I apologize if I missed it somewhere, but I w as a little surprised to find that the slide show and survey only referred to the Northeast. I would in fer that subjects from different regions will get versions tailored to their region. Maybe that woul d make sense, but I would be tempted to explain it in the slideshow to avoid confusion. Tell them that this is a national policy but that they need to be concerned only about their region since people in other regions will be paying costs there. If a national sample will be considering only changes th at are proposed for the Northeast, then there is more explaining to be done. What about the fish in my region?

• As a general comment on design, except for wande ring out to Wisconsin and Oregon for a couple of groups, all the qualitative work was done in a fairly tight area of the East. Why? I often find it instructive to have a wider geographic range. Also, the focus group memo points out that while people are generally more interested in their home regions, some care about other areas. Focusing on regions could cause some problems in this study. If I live in Maryland, for example, is Maine more important to me than Virginia? If I care somewhat about other regions, shoul dn’t I be informed about them so I can include them in my values? (Perhaps I am not appreciating the difficulty of expanding the information to cover the whole country. Also, limiting people to their home region might be supported as a way of making value estimates conservative.)Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 6 Kevin J. Boyle • There are many global statements about valuation and the design of this specific study that are purely assertions and are not based on any hard empirical research ortheoretical foundations. These statements detract from what is really a basically sound research design. Examples of the former are statements with respect to the incentive compatibility of choice questions and hypothetical bias, and examples of the latter are extrapolations from what was learned from the focus groups.

• The entire discussions of protest responses, hypothe tical bias, and other problematic responses appear to select items from the literature at face value. The key point isthat you did try to address these issues, but you probably have not in the design norw ill not be able to in the estimation do as much to control for these problems as you imply in the review document. The key point really is how much subsets of people inthese categories actually affect your welfare estimates. My reading of the literature is that the only major concern in this regard in hypothetical bias and you are going to do as much as you can to control this issue, but not to the extent you claim.

• You talk about a sample design that includes multiple weighting schemes and there is no discussion about how you are going to resolve these different levels of weights nor how you are going to investigate how weights affect your welfare estimates. One key issue is that you, like many others, assume that making the sample look like the U.S.population will solve all nonrespone biases, but if sample selection arises from something other than Census demographics you have not corrected the problem, you have an endogenous weighting problem.

• There is no discussion of how you are going to a dd up the values from different regions to get a benefit estimate for national policy. What if res pondents value an area larger than their sampling region? What if respondents think that people fro m other regions will pay for their region? See my comments below.

• You assume that the type/mix of fish does not matter to people. This is a strong and perhaps necessary assumption. You must make it explic it because information on specific species while substantially increasing the information load fo r respondents may also change values estimates.

• I think this is a well designed study, but there is definitely a need for improvement. While considerable effort has gone into the scenario design, the design is not as tight as the authors of the review document purport it to be. You have done a very good job, but by over-selling your design you take away from the good features and make the entire design more controversial.

Timothy C. Haab Thank you for the opportunity to review this document and survey. I found the document to be informative, well-written and easy to follow. Th e document does a good job of outlining the procedures to be used and for the most part follows acceptable procedures for the valuation of increased fish stocks from cooling water intake regulation. Particularly, the document provides thorough documentation of the planned sampling scheme and procedures for ensuri ng a valid representative sample from an internet based survey. Many of the fears decision makers face in regard to internet surveys should be allayed. Clearly the authors of the document—in conjuncti on with the experts at Knowledge Network—have spent considerable time designing a sampling scheme a nd estimation strategy for correcting any potential sampling bias created by the internet panel.

Keeping in mind my overall positive impression of the ju stification and survey, I do have some concerns that can be readily addressed. For the most part these are minor concerns that can be addressed quicklyPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 7 and should not cause a significant delay in the administration of the survey. My comments and concerns are detailed below in response to the charge questions provided by Versar. Perhaps my most significant concern deals with the analysis of the survey r esponses. It seems as though the authors have proposed a fairly basic method of analysis of the choice e xperiment response and have ignored some important econometric issues that should be addressed. Further, I have concerns about the aggregation of fish stocks into a single number in the survey responses. I will provide details below.

Again, I want to emphasize that overall I think this survey is well-conceived, meets standard academic practices and when administered will provide useful and accurate information to EPA.

James R. Kahn The U.S. Environmental Protection Agency's plan for implementation of a willingness to pay surveyinstrument to assess the benefits of §316(B) Phase III Cooling Water Intake Structures is a wellconceived plan that should produce results sufficient to guide the decision-making process. After reading the document explaining the survey plan, I find th at the authors are quite knowledgeable of stated preference valuation measures, the controversies asso ciated with their use, and the best means of implementation. I am very pleased that the EPA decide d to use the choice experiment (conjoint analysis) stated preference approach, as I believe this appro ach to be less subject to biases and more conducive to the evaluation of alternative policy options in comparis on to contingent valuation. The justification of the approach, the background information slide show, and the survey itself are all first rate work. If the research plan is implemented as proposed, the EPA will be in a good position to make a decision relative to §316(B) Phase III Cooling Water Intake Structures and to defend the decision that is made. Although my overall assessment of the proposed survey and implem entation plan is quite favorable, I have several suggestions that the EPA may want to consider before implementing the plan.

Conclusions: The proposed survey looks like it will accomplish th e desired purposes, and is extremely well done. However, some small changes can make it better. In addition, there are several sections, particularly the sections on biases and econometric specification that n eed more exposition in order to make the final results of the survey believable. Moreover, the refe rence list is not very impressive and does not provide an adequate survey of existing applied studies us ing this technique. This unimpressive list gives an impression that the researchers have not done all of their homework. For example, I did a Google Scholar search on the keywords "choice experiments environmen tal valuation" and 25 of the first 40 hits were directly relevant. A partial list of additional references is supplied.

In summary, with a few modifications, the surv ey and supporting documentation will provide the appropriate type of information withthe appropriate level of confidence in the empirical results. I am very pleased that EPA has taken this route, instead of rely ing on benefits transfer or a contingent valuation
survey.
Alan J. Krupnick I commend the survey team for their tremendous amount of hard work and boldness in attempting to capture preferences for a pretty ephemeral commodity – fish kills that have no ecological consequences. I also commend EPA for providing what must be a huge survey budget for 5,000 completes. While I actually think this is excessive for what is planned, it is heartening to see the agency make a commitment to such a major study. At the same time, in a study of this complexity there are inevitable tradeoffs that have to made, say between consequentiality and bias, where the literature is insufficient as a guide. There are many such choices here and, given my experience, I think I would have made some different choices, or at least altered the design to develop a strategy of protecting the survey against the possibility thatPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 8 some of the choices were bad ones, primarily through developing additional treatments.

Detail on major problems :

• There is no baseline or the baseline is poorly defined . There are three ecological attributes in the valuation exercise: numbers of fish saved, eff ect on long-term fish populations and effect on recreational and commercial catch. The first has no b aseline, except relative to the total fish currently “lost” in cooling water intakes. While some indi viduals might care about fish death per se, I would imagine this is not a large fraction of the populati on. Most would want to know if this amount of death is large relative to the overall populations of various fish species. They might want to know this to gauge whether fish populations are at risk, which is really dealt with by the second attribute or just about having a number to scale the deaths to. If it is the former, then the number of fish saved attribute is redundant and should be dropped. If it is the latter, then some number is needed. In this case, my advice is to make up a few different b aselines and then use them in several different treatments to test whether the baseline that was ma de up makes any difference to WTP. I suspect it will, but in that way can be corrected or adjustedfor. If something like this isn’t done, I wouldn’t know how to interpret any results from the survey.

The second attribute’s baseline is poorly defined. The scale gives an endpoint of 100%, while some screens fail to show what this 100% refers to. I r ecognize that the text may define this, but I think it needs to be on the screen. The text definition refers to the maximum historic levels of fish. This is poorly defined in my view. The maximum may not be the optimum or even be sustainable. I think that some sentence like: Scientists and ecologists have determined that appropriate fish populations consistent with the health of the ecosystems affect ed. The 100% refers to population levels consistent with that determination.

Finally, the third attribute baseline is also poorly defined. Why is the catch in 1950 significant? We are not told on the screen. Again, if this is interp reted as peak catch then this may not be optimal at all. Indeed, some respondents might interpret an increase in catch as a bad thing, thereby screwing up the scope tests.

I am also troubled by what we term “expansive priors” about the baseline. The documentation says that (at least one) respondents may think that the fi sh populations are declining. Their response to a declining baseline situation will certainly be different than that if they think the baseline is static or increasing.

Overall, much more care is needed to define base lines and test for baseline effects when different (or made up) definitions are used.

• Definitions of the commodities. One of my biggest concerns is that the commodity is defined in terms of fish, not as an indicator of ecosystem health , but as the fish themselves. Respondents may well substitute their own views on ecosystem linkages withfish kills and you won’t know what they are thinking.

And not just fish, but commercial fish (in part) which respondents should have no logical reason to be willing to pay for. Respondents may ask themselves whether prices of fish will come down with the options being considered, thereby canceling out someor all of their increased costs from the option itself. So there are really two issues: whether respondents are willing to ignore ecosystem health concerns and what the role played by commercial fi shing is in their thinking. On the latter I would stipulate that scientists have determined that ther e will be no effect on commercial fish availability orPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 9 prices because such fish come from all over the world, and drop any further mention of it. I would leave in recreational fishing as there may be opti on and use value associated with it and it is placedbased. Because recreational fishing is placed-based, th e text would need to clarify how patchy the effects would be across substitutable recreational fishing sites.

A broader issue is that the three attributes you list are not independent, but ar e related to one another in complex ways. Probably, fewer kills will resu lt in larger long-term fish populations and greater recreation and commercial catch. But, your or thogonal design varies attribute levels for these attributes independently (as it should), which mean s that fish kills could be going down a little while long term populations could be going up by a lot – and that doesn’t really make sense from any sort of simple story. Indeed, increases in catch could have effects on long-term fish populations, irrespective of fish kills from entrainment, so the interdependencies are everywhere. The documentation says that some inadmissible combin ations of levels were dropped from the design space, so maybe this problem has been at least partly addressed. What is not addressed, though, is that if interdependencies exist in people’s minds then the marginal WTP estimates attached to each attribute will not be reliable.

Another issue with the commodity is aggregation over species. The survey lumps all types of fish together and talks about average effects over all speci es. I would want to be sure that this does not create problems for the respondent in interpretation.

Another issue is uncertainty in the commodity. Our research shows that admitting uncertainty results in a downward bias and increased variance. People thi nk that if you are uncertain about effects either the effect can’t be very important or large or th at you don’t know what you’re talking about so why should they take the survey seriously. Both tend to push WTP down.

The authors want to have it both ways. They want to invoke uncertainty, so they can claim they took it into account, and then ignore it in the choice questions because respondents thought that presenting intervals of improvements was too confusing. I don't think they can have it both ways without making changes in the treatment of uncertainty.

As noted elsewhere, my suggestion is to vary in separate treatments how uncertainty is handled, both about the baseline and about the changes.

• Overall experimental design. With such a large sample size being planned I am surprised that there are no plans for treatments, e.g., on information, un certainty, baselines, etc. At the same time, the large sample is being used to get a better unders tanding of how WTP varies regionally. Why is such regional precision required? Further, why is the commodity being limited regionally? In our work, the payment vehicle definition required a limit to the state, but no such limit is needed here as there are no taxes involved. Indeed, one could argue that a regional limitation is unrealistic. Market effects will be the result of cumulative changes across the country (to a large degree), so by not discussing them the survey may seem less credible and WTP may be biased.

• Consequentiality versus neutrality. My view may be controversial, but I think the survey errs too far towards consequentiality (and to far away from neutra lity (the absence of bias)). Every survey has to make choices about whether or not to state the sponsor of the survey, its real purpose, and the implications of voting. By being truthful, one may easily create bias. To me, this survey faces the same problem we faced in whether to tell respondents that they are voting on a question about air pollution controls (with costs passed on to them) to improve conditions in the Adirondacks versusPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 10 trying to find an biased way of getting at preferences for cleanup and in so doing having to be untruthful and perhaps lose some consequentiality . If people know that the USEPA is doing this survey, they will assume that voting for the options is encouraged. Saying that EPA will use these results to assess values begs the public to vote on a warm glow basis rather than their true preferences.

• Payment Vehicle. The payment vehicle begs respondents to wonder why they should have to pay when business is causing the problem. The notion th at increased costs result in higher prices is not one understood by most people, particularly for co sts incurred by powerful and perceived, high profit industries, such as electric power. They may think that industry will make them pay, but they don’t think they should have to pay and may therefore vote against such a plan rather than reveal their true preferences for the fish.

• Extent of the Market. I don’t understand why the survey is ge ographically limited in the way it is. Since the payment vehicle is a change in market prices, these changes can be realized by anyone in an affected market. For some products, this is the entire nation; for others, perhaps effects would be more regional. For power plants, regional effects are conceivable. But these effects for the Northeast, would surely include Pennsylvania (which is excluded now) because this state is part of a regional power grid that includes states labeled as northeastern states. From the benefits side, the extent of the market is even more cloudy. Eff ects of entrainment are likely to affect an entire watershed, so that is the extent of the market . Again, Pennsylvania and Maryland share a common watershed, given the Susquehanna that runs through both states; yet PA is excluded and MD is included in the definition of the Northeast region.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 11 1. Is the choice experiment survey an appropri ate stated preference survey format for the estimation of marginal willingness to pay for the characterized policies? Will the survey likely provide useful insights into preferen ces, tradeoffs, and willingness to pay associated with the described policies? 

### IV. RESPONSE TO CHARGE

Richard C. Bishop Provided that stated preference surveys are well d esigned and executed, they have wide support for applications like this, both in the scientific literature , as cited in the materials provided to this reviewer and by governmental agencies, including not only USEPA, 2000, as cited, but also the Office of Management and Budget, Regulatory Analysis. Circ ular No. A-4. U.S. Office of Management and Budget, 2003. http://www.whitehouse.gov/omb/inforeg/circular_a4.pdf. Having worked in the field for many years, I am convinced that stated-prefere nce methods are capable of providing useful insights into preferences, tradeoffs, and values in cases like this. The choice experiments being designed here are in keeping with one of the major branches of the literature that has many respected proponents.

Kevin J. Boyle The choice experiment is an appropriate methodology and is the best methodology available to estimate values for the Phase III policy.

The survey results will provide useful informa tion, but perhaps not as strong as asserted in the peer-review document (see my comments below).

Timothy C. Haab In short my answer is yes. The choice experimentis the obvious and most appropriate choice for the valuation of multi-attribute policy changes—such as 316( B). I can think of no other method that will allow for the valuation of multiple attributes in such a clean manner. As designed the survey collects: • background information on perceptions towards policies in general • trade-offs among fish stock attributes • debriefing information to establish the validity of responses • demographic information

Combined, all of this information will allow the author s to provide a detailed description and estimate of the values respondents place on fish stock increases dues to Phase III regulations.

The use of the Knowledge Networks internet panel is accepted in the academic community and provides an accurate but less expensive alternative to in-person surveys. The reviewed documents do a nice job of describing the potential pitfalls with intern et surveys and the appropriate solutions.

I have a few concerns about ques tion ordering and wording in the survey that I will outline below.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 12 James R. Kahn Not only is the choice experiment survey an appr opriate stated preference survey format for the estimation of marginal willingness to pay for the charact erized policies, it is the best stated preference approach for this purpose. I believe the survey to be well-constructed and it w ill provide useful insights into preferences, trade-offs and the willingness to pay associated with the described policies. Based on everything I have read in the document, I am confid ent that the U.S. EPA will have the best possible information on which to base these decisions.

Alan J. Krupnick a. The choice experiment (CE) approach is not necessarily the best way of obtaining WTP estimates. It is more complex than the CV alternative. The mo st compelling justification for the former over the latter is if the attributes affected by entrainment need to be unbundled, i.e., if separate shadow prices (marginal WTP) need to be estimated for each of them, presumably so they can be used in benefit transfer exercises to apply in locations where the improvements in these attributes are different. A case can be made here to avoid false precision over these effects that may come from a CE and treat them as bundled in a CV, using several different treatments to capture a relevant range of attribute differences.

b. Whether the survey produces useful values depends on all its features ex ante as well as how well the analysis turns out. On balance, I think there is some reason to be concerned about its features, as detailed below, and one can’t know until the survey results ha ve been analyzed if they will withstand scrutiny.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 13 2. Does the effort used in developing and test ing the proposed survey meet current standards for stated preference surveys reflected in the literature?

Richard C. Bishop Those designing the study display excellent knowledge of the scientific literature on stated-preference methods; experts on the biological side have obviously been consulted; attention has been given to theoretical issues; survey materials were deve loped; focus groups were conducted and the survey materials revised; and a peer review is being conducte d now to gain additional input before the pilot study is implemented. Except perhaps for gaining more i nput from an expert in survey methods (See General Comments above) and some specific questions raised below that may warrant further attention, these procedures meet current standards.

Kevin J. Boyle The design effort does meet current standards, but the inferences from this design process are too strong (see my comments below).

Timothy C. Haab Yes, the use of 12 focus groups and cognitive interv iews for survey development is appropriate and warranted. The 500 individual pilot survey will provi de EPA with valuable insights into how the public is interpreting the survey. I would encourage the survey administrator to follow up with pilot survey subjects to get open ended feedback on the instrument itself. Such feedback will prove valuable in making sure subjects are interpreting questions appropriately.

James R. Kahn Yes, the focus groups and pre-testing more than meet s current standards that have been discussed in the literature.

Alan J. Krupnick The effort with 12 focus groups seems reasonable to me, certainly not excessive for a “commodity” as complex as this one. In our survey on the WTP to improve the Adirondack ecosystem, we used over 30 focus groups. This probably was excessive, in retrospect.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 14 3. Can the policy options be sufficiently well defined such that the survey is likely to provide useful insights into preferences, tradeoffs and willingness to pay?

Richard C. Bishop The breadth of the impacts of I&E regulations in te rms of both geography and affected species makes this a challenging study. It is impossible to develop a scenario that touches on all the aspects that may be of relevance to all subjects. And, like so many othe r environmental economic studies, scientific uncertainty about both the baseline and the effects of the regulations add to the difficulties. We need to recognize the limitations of stated-preference methods in such a c ontext. For example, if it were possible to tailor surveys to the effects of the regulations in each s ubject’s home area, their answers might change. Still, useful—albeit imprecise—insights into preferences, tr adeoffs and WTP can probably be gained with this study. More comments drawing on the results of the focus groups might increase the level of confidence. Did most participants seem to “get it” when they talked about what they were being asked to think through? Could most talk meaningful ways about the tradeoffs involved and why they might care about them? Did some members of the groups say they didn’ t care? (Not everyone needs to care about the fish affected by use of water for cooling and other such problems. I find it reassuring when members of groups I am conducting have learned enough and felt comfortable enough to say that they are not interested.)

Kevin J. Boyle Yes, the policy options can be sufficiently well defi ned. I have made some specific recommendations that should be addressed and the limitations/assump tions should be more explicitly stated.

Timothy C. Haab I have a number of concerns here. First, the slid e show and survey spend considerable time trying to convey that the numbers presented are ‘uncertain.’ I’m not sure this is successful and it is critical that it be. In particular, the choice experiment questions themselves seem to convey ‘certain’ information—“1360 billion fish saved”, “Increase to 70% ”. There is no uncertainty conveyed. In the blow-ups to the slide-show some ambiguity ari ses in the wording: “Policies may increase stocks to between 60 and 75%” How is the respondent to interp ret the actual survey question? Will it definitely increase to 70%? Will it increase to somewhere betw een 60% and 70%? Did your focus groups show any confusion over this? On a similar note, do respondents all have a comm on baseline for comparison? They are told that “100%=maximum historical fish levels”. Is that with or without human intervention? Does everyone have an accurate assessment of what that baseline will be, or are respondents comparing apples to oranges? Again, these are questions that were pr obably addressed in the focus groups but without more information it’s not clear what the respondent is valui ng. Is it an absolute change or a relative change? Do subjects know what catch levels were in 1950? There seems to be a non-sequitur in the slide show. It wasn’t clear to me that the choice experiment was an example until after I read through everything once. Nowhere does it say, “Here is a sample of what one of these questions will look like.”Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 15 Also, I have concerns that the sample questions may bias the answers to the actual choice questions. For the same reason you want to control for potential orde r effects in the repeated paired comparisons, you may need to control for the values reported in the sample problems. The respondent may be tempted to compare the actual choice policies to the sample policies. I’m concerned about aggregating all fish species in to a single aggregate population change. The slideshow give subjects information on the species aff ected and the table gives detailed information on changes in fish species. Are s ubjects supposed to assume proportional increases in stocks for all species? Or is it assumed that just the total matters? The National Marine Fisheries Service has detailed estimates of the value of various species to recreational angler s. Those numbers clearly show that anglers value species differently. Is it the same for non-anglers? I’m not sure you can answer this with the current
survey.
James R. Kahn Yes, the survey does this in quite a nice fashion. Since the policies have varying levels of cost and impacts on fish, they naturally fit into choice set scenarios.

Alan J. Krupnick This question is really about whether the levels of the attributes are appropriate. I have no information to make such a judgment.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 16 4. Does the information provided in the survey on the baseline ecological conditions, the expected changes, and the expected costs a nd their incidence seem sufficient, appropriate, unbiased and accurate? Does the survey provi de appropriate and sufficient introductory and debriefing questions to enable iden tification of protest responses and symbolic responses, to test retention and understanding of information, etc.? Would you suggest any additional debriefing questions?

Richard C. Bishop As I have stated, I think the slide show can be improved, but the basic information on baselines and changes is there and seems to be presented in an unbi ased way. I have suggested some improvements in the debriefing questions in my detailed comments on th e survey (see below), but they involve fine tuning, rather than major changes. The introductory questions designed to assess whether the respondent understood the information (now Questi ons 2 and 3) should be beefed up. I like to use a series of truefalse questions when I am trying to evaluate what people have learned from the information provided. When I introduce the true-false questions, the instructi ons clearly say that the goal is not to test the participant’s knowledge, like in school, but to testhow well we did in communicating information to them. This allows me to go into more depth thanQuestions 2 and 3 in the current version of this instrument. And saying that respondents averaged 80% or 90% in answering the questions (which is my goal in information provision) is a good way to silence those who might otherwise try to argue that people didn’t read or didn’t understand the information. Let me hasten to add that not everyone agrees that this is a good idea. I think they are afraid of turning off respondents, thus contributing to overall and item nonresponse. But I have checked this approach out in focus groups and other qualitative research and so long as people understand the intent, I do not detect much resistance. I say more about the specifics below.

Additionally, Question 2 does not seem like a very good test for “Testing for Appropriate Understanding of the Introductory Materials.” Slide 8 of the intr oductory materials states that there are many causes of fish decline. Slide 20 says (emphasis in origin al) that intake structures “are usually smaller than the effects of commercial fishing.” The voice-over for S lide 40 makes a vague reference to “a wide range of past human activities that have caused fish stocks to decline.” Unless I missed something buried in the slide show, respondents do not have a basis in the intr oductory information to answer Question 2. To the extent that they get the ranking correct, they must be falling back on their general knowledge.

Kevin J. Boyle The ecological condition information is good. Respondents need to be told what to assume about baseline conditions. I presume that you want them to assume th at current conditions will be maintained if they do not choose a policy alternative but this in not cl ear in the information presented to respondents. Limitations, like no information on specific species, needto be explicitly stated and recognized in the study documentation. I think the information is well balanced. I have noted below that I am skeptical of debriefing questions in ge neral and do not have any specific recommendations.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 17 Timothy C. Haab For the first part of the question, see the answer to question above. For the second part , I have concerns about the placement of the introductory questions in the survey. The first two questions are designed to assess general attitudes towards policy and the rankings will be severely compromised by the slide-show. The slide show provides 15 minutes of information on the negative environmental effects of power plant operations and th en subjects are asked to rank environmental objectives among a group of other policies. It seems like this question should come before the slide show.

James R. Kahn In general, I think the survey provided appropria te information on baseline ecological conditions. The introductory and debriefing questions also seemed appropriate to identify protest responses and symbolic responses. However, as I noted in my discussion of 6.6.2, people might believe that reducing impingement of fish would also reduce impingement of birds, reptiles and amphibians, and that some technologies might be associated with less thermal pollution than other technologies. The potential existence of this problem could be tested by adding an additional response to question ten, that would be something like this:

___I thought there were impacts on ecosystems in additi on to those created by changes in fish populations and partially based my answers on these impacts.

Alan J. Krupnick a. The information on baseline ecological conditions is l acking, as the materials readily indicate. I think it will be hard to believe the results for two reas ons that come up here: one, an effort is made to eliminate in people’s minds the links with fish kills to ecological damage and, two, there is no baseline for total fish stocks. Concerning the fo rmer, in my experience, respondents will default to believing there is a link and just telling them ther e is not is not enough. At a minimum, add a debriefing question to see if they bought this restri ction. Concerning the latter reason, having the baseline defined by I&E losses seems wrong to me. The baseline should be fish populations. If these are not known I would recommend making them up, treating this as a parameter and trying alternative information treatments varying this base line to see if responses are sensitive to it.

b. I think the survey goes overboard on consequentiality giving too little attention to its flip side – bias and warm glow. The questions to test for this are very complicated and one can never know if these perceptions are being fully eliminated.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 18 5. Does the survey sufficiently and appropriately characterize the policy options under consideration? Does the survey place sufficient and appropriate emphasis on the scope of resource changes? Does the surv ey appropriately communicate the scientific uncertainties as well as uncertainties associated with the policies in question?

Richard C. Bishop See my responses to Question 3.

Kevin J. Boyle I find this question hard to answer. First, the ultimate ecological effects are not known. Thus I do not know if the selected baseline conditions and attribut e levels will be able to address the actual policy question. Second, while the analysis discussion talk s about linear and quadratic specifications, with a small number of attribute levels the best that you may be able to estimate is a piece-wise approximation to a nonlinear function, which leaves question if the actual policy conditions fall on intermediate levels.

I think the scope of the change is well communicatedwith words, numbers and graphics, except for the region of focus.

Uncertainty is not addressed well (see my comments below).

Timothy C. Haab See response to question 3.

James R. Kahn The survey adequately characterizes the policy options under consideration and places appropriate emphasis on the scope of resource change. I think thatfor the most part, the survey adequately addresses scientific uncertainty, although as noted above, I did not like the choice of words ("common sense") in slide 30. I thought the use of these words cheapen ed the quality of the scientific information.

Alan J. Krupnick Mostly redundant to above questions. The new part is about uncertainty. In my work I find that when a survey admits that effects are uncertain, respondent s tend to reject the survey (and therefore, biasing estimates downwards, unless such people are eliminatedfrom the sample) and think the survey makers don’t know what they are talking about. The paper discounts this effect; but we see this effect very strongly in our work.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

1 Green, D., K. Jacowitz, D. Kahnemann, and D. Mc Fadden (1998) “Referendum contingent valuation, anchoring and willingness to pay for public goods,” Resource and Energy Economics, 20: 85-116.

2/06 19 6. What are your expectations regarding the potential validity and reliability of benefit estimates that might be provided by the prop osed survey, compared to the current state of the art in the stated preference literature?

Richard C. Bishop I have stated some reservations about details both above and in the specific comments below. Much attention to the design of the slide show and the surv ey is still needed, but I think this study is well along toward being state-of-the-art for one that mu st deal with such a broad regulatory initiative.

Kevin J. Boyle I believe that with further revisions this will be a st ate-of-the-art instrument and data collection process. I have concerns that the data collection process may not end up being as sophisticated as proposed. The data analysis section is too vague at this point to address whether this portion of the study will be state-of-the-art.

Timothy C. Haab I feel the proposed survey takes the necessary precautions to provide a reliable, conservative estimate of willingness to pay. While the authors argue agai nst cheap talk in the documentation, the survey implements a watered-down version cheap talk by st rongly encouraging respondents to take the valuation questions seriously. Further, by reminding subjects th at the survey results will be used to inform the policy decision, the responses are taken from the realm of purely inconsequential to possibly consequential. Many would argue that this is enough to meet the c onsequntiality criterion Carson, Groves and Machina and Green, Jacow itz, Kahnemann and McFadden 1 set out for incentive compatibility.

James R. Kahn My expectations are that the validity and reliability of the benefit estimates are very strong in comparison to the current state of the stated preference valuati on literature. I would suspect that after the dust settles from the rule-making process, the authors will be able to publish this study in a top journal, such as the Journal of Environmental Economics and Management

, Ecological Economics , or Environment and Resource Economics.

Alan J. Krupnick I think getting credible marginal WTP values by attributes is unlikely because the attributes aren’t independent (see more on this below).Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 20 7. Does the survey take appropriate steps to minimize hypothetical bias? Does the survey place sufficient and appropriate emphasis on th e budget constraint facing respondents, and on potential substitute uses of their budget? Is the payment vehicle used in the survey credible? Does the survey incorporate appropri ate language to ameliorate strategic bias, symbolic responses and “yea saying”? Is there any sign of systematic bias in the survey or presented information? If so, how can it be ameliorated?

Richard C. Bishop Section 6.6.1 shows that hypothetical bias has been thoroughly considered and addressed. I agree with EPA’s reluctance to introduce a cheap talk script. In my judgment the budget constraint and substitute reminders are more than adequate. Overdoing e fforts to provide budget constraint and substitute reminders risks insulting significant numbers of subject s. (“What? Do you think I don’t understand that if I spend more money on this, I will have less money for other goods? Do you think I am stupid?”) I discuss the payment vehicle below (see specific comment s relating to Page 21 paragraph 1). I think the survey meets current standards in dealing with the other potential sources of bias described in this question. Section 6.6.3 seems more than adequate in this regard.

Kevin J. Boyle The survey does a good job of addressing hypothetical bias, but this bias is not controlled to the extent that the review document claims. One additional im provement would be to make the decision rule by which the policy will be provided more explicit.

There is sufficient emphasis on the budget constraint, but I wonder how much it matters with such low cost amounts. In fact I am concerned that the co efficient on the cost variable may not ultimately be significant so the pilot study is imperative to test this hypothesis.

The payment vehicle is credible, but it may be better to acknowledge that there may be some tax effects.

The study does the best it can to address strategic bias, symbolic responses and year saying, but the design choice experiments opens up the possibility for all of these problems. Thus, I do not think that the controls are as strong as the study designers claim (see my comments below). I do not see a systematic bias in the information presentation, but do make some suggestions for improvements.

Timothy C. Haab The survey design minimizes the potential for incentiv e incompatible responses. I would like to see a more detailed description of the mechanism by which the ‘cost of living’ increases. For example, “Such regulations will no doubt increase the cost of producing power to the firm. These costs will most likely be passed on to consumers in the form of higher pri ces for electricity and other goods that heavily rely on power to be produced.

The budget constraint and substitute reminders are weak. They appear as three bullet points on one page. I would like to see some examples of the substitutes. The substitutes don’t have to be just environmental substitutes. “The proposed policy will result in higher co sts to you. This is money that could be spent on other things like eating out, recreation, gas, etc.”Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 21 James R. Kahn I don't believe that hypothetical bias is a real problem with correctly structured conjoint analysis studies, and I believe that this study is correctly structured. Respondents know they will be making a real world decision and that it will have real world conseque nces in terms of fish populations and the cost of electricity and other products. There is not a need to focus on "cheap talk" o r other mechanisms for alleviating hypothetical bias, because they are not asked about a hypothetical donation. They are asked about potential impacts on the day to day cost of living. Since the questions are structured with both costs and benefits, it is very difficult for a respondent to behave strategically. I see no potential for systematic bias, outside of what I noted in my discussion of section 3.2 (respondent fati gue) and section 4.1 (oversampling of the information literate in a way that is independent of typical demographic or socioeconomic classifications). The respondent fatigue issu e can be addressed by increasing the participation payment. The over-sampling of the information literate population is extremely difficult to address, and since appropriate discussion of the non-response surv eys are not contained in this document, I cannot determine the significance of this problem.

I also note potential problems associated with w ho answers the telephone and who is handed the telephone. The document does not specify a protocol for randomly choosing the household member who participates in the survey. In addition, there ar e potential biases that will develop if cell phone numbers are not included in the survey sample.

Alan J. Krupnick Some of these questions are also redundant to above. A new issue is: Payment vehicle: I think this payment vehicle is highly problematic. I don’t accept that respondents will accept that product prices will be higher and hence their costs will go up. I rec ognize that the paper disputes this from focus groups. Respondents in my experience will blame business and be lieve that they shouldn’t have to pay. They may dispute that prices will go up, or may think prices will go up by more than they should (rather than by less than the cost increase) because of monopoly business practices. Or respondents may believe that they can simply substitute away from commoditi es whose prices rise (admittedly, substituting from electricity wouldn’t be easy!). At the very least, debriefing questions would be needed to tease out whether the payment vehicle caused problems for respondents.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 22 8. Will the answers to this survey lend themselv es to straightforward economic interpretation of willingness to pay? Are the proposed econometric modeling approach and validity tests appropriate? Would you suggest an alternative modeling approach or alternative or additional validity tests?

Richard C. Bishop As I say below, I have some reservations about the payment vehicle, but otherwise the results should be amenable to straightforward interpretation. The plans for analysis, which involve not only a rather traditional approach, but also an effort to acc ount for heterogeneous preferences seems more than adequate for this stage in the project.

Kevin J. Boyle The study results can lead to “straight forward economic interpretation” if the sample weighting issues are resolved and state-of-the art estimation techniques are used. Neither of these points is clear in the current documentation and I say more on these points below.

There is only one validity test that I see in the design and that is the external test of scope, which I believe is fundamentally flawed (see my comments below).

Timothy C. Haab Unless I have missed some details, the proposed econometric technique fails to capture the panel nature of your responses. Each subject will make up to three choices. These responses mustbe treated as correlated for a given respondent. Failing to do so will bias parameter estimates and estimated standard errors. Models for repeated choices are readily av ailable in standard software packages like LIMDEP.

James R. Kahn The answers to this survey will le nd themselves to straightforward interpretation in the decision-making context. The validity tests seem appropriate, but as noted above, I am a bit confused by the econometric modeling. When talking about heterogeneity in secti on 4.3.3, the authors refer to using the Heckman correction method. However, this does not appear inthe econometric specification listed on page 30. This inconsistency leads the reader to wonder exac tly what econometric model is being employed.

Alan J. Krupnick I think that the sample size is larger than needed for the su rvey as currently designed. I suggest that the experimental design be made more differentiated, with different information treatments developed and tested to examine the effect of certain major choices in survey design on the results.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 23 9. The national benefits analysis for Phase III of the 316(b) rulemakings must be completed within the next several months and a final rule issued by June 1, 2006. Are there improvements to the survey and associated mo del that would be feasible to implement within the available time frame? If mo re time was available, how would your answer change?

Richard C. Bishop That is a very short time horizon to do the pilot, ma in survey, follow-up survey, and analysis. You are likely to have to live with some warts and loose e nds. More time would allow more attention to the suggestions coming out of this and other peer reviews, a thorough evaluation of the results of the pilot, a more careful statistical analysis of the main survey results, and a lot more thought about the follow-up survey. I urge you to add at least six months to the schedule.

Kevin J. Boyle I do not believe the study can be completed by June 1! Yes, if there was 9 months to a year available.

Timothy C. Haab The suggestions I have made in the above responses can all be implemented in a short time period. The overall concept and design of the survey is sound and with minor corrections will represent a solid piece of applied valuation.

James R. Kahn There are a few sections where the discussion needs to be beefed-up to make the reader more confident in the analysis. This includes the discussion of the stratified sample, nonresponse biases and the econometric specification. The primary change to be made in the survey would be the incorporation of an additional response to question 10, as mentioned in my respon se to question 4. Finally, since the survey is so lengthy, it is important to pay the respondents a hi gher payment, or run the risk of the respondents treating the later stages of the survey with less than their full attention.

Alan J. Krupnick I don’t know if the above idea (in 8.) is feasible within the time frame. Some ideas below are obviously easy to implement.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 24 V. SPECIFIC COMMENTS ON REPORT

Kevin J. Boyle Page 6, Paragraph 2 I have concerns about the 600 follow-up intervie ws with nonrespondents. No goal was set for the response rate. These are people who have already reject ed the mode of implementation of the survey or the specific survey for a variety of reasons. Whatis being provided as incentive to get these people to respond? If you meet the target of 600 completed interviews, but have a nonrespondent sample response rate of 10-20 percent, then this information will beof limited use; you have just added an additional sample selection problem. In addition, if you find a difference in values you will not be able to identify whether you have a sample selection effect, mode eff ect or both effects causing the difference. While you are making the follow-up as similar as possible to the or iginal survey implementation it is not identical.

Richard C. Bishop Page 6, Paragraph 2 More advantages of the Internet mode might be menti oned. Four come immediately to mind. First, mail surveys can run into difficulties with skip patterns that can easily be pr ogrammed into an Internet survey. Second, where a number of stated-choice questions are to be asked back-to-back, it is possible to tailor later questions based on answers to earlier ones. Suppose a respondent chooses Alternative A over Alternative B when costs are $15 and $5, respectively. It would not be very useful to ask that same respondent a second question where attribute levels are the same except that Altern ative A has a cost level of $10. Mail surveys cannot be tailored leave out such redundant questions. Third, voice over to help respondents through complicated or extra long materials can be added. Fourth, links to more detailed information can be added for those respondents who think they need it without complicating the main presentation for everyone. Also, regarding existing studies, do you want to mention the Viscuzi et al. (2004) study on water quality that was done for EPA and which was cited later in this document? I believe an Internet mode was used there.

Kevin J. Boyle Page 6, Paragraph 3 Where are you going to obtain “historical response rates of individuals in that group for similar types of surveys?” You make the argument in this report, which I agree with, that no work of this type exists. This assertion nullifies you ability to make such an adjustment in the sample design.

Alan J. Krupnick Page 6, Paragrapgh 3 I don’t see the need for oversampling. Rather samp ling weights can later be used to correct for differences between the population and the sample. I am not sure that doing this up front in sample selection is worth it. Maybe it will introduce unknown bi ases relative to a more neutral attempt to enlist respondents.

Kevin J. Boyle Page 7, Top I think the objectives are good and appropriate for the policy issue you wish to value.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 25 James R. Kahn Page 7, Previous Research (Section 1.2) I concur with the authors that previous studies do not ex ist that contain an estimation of the social benefits of nationwide changes in populations of fish species that include species utilized in sport and commercial fisheries, as well as species that are not sought but are important forage fish. Although there are a plethora of valuation studies of fisheries including those atthe national level, these tend to focus on specific species such as salmon, or recreational species in gene ral. Additionally, these studies generally target the use values of these species and do not measure values su ch as altruistic value or existence values. The authors are correct in there assertion that general studies of the value of water quality improvements as a whole measure of different suite of services than the proposed §316(B) decision will affect. An implication of the authors findings which should be exp licitly stated in this section is that it is simply not possible to evaluate the benefits of the§316(B) Phase III program with a benefits transfer study.

Alan J. Krupnick Page 8, Paragraph 2 It is one thing to assert that the values from this study will only apply to fish, rather than ecosystems; another to make people believe that. In my e xperience, people know very well about food webs and won’t surrender this idea easily. From the report, I am not sure that the idea should be surrendered. Is it true that there are not ecosystem level effects of entrainment?

James R. Kahn Page 8, Consultations (Section 1.3) The consultations, focus groups and pilot survey di scussed in this section are consistent with the standards of the literature. This reviewer has no further suggestions in this regard.

James R. Kahn Page 9, Survey Objectives (Section 2.1) The survey objectives are clearly stated and c onsistent with developing the knowledge necessary to inform decision-making with regard to the §316(B) Phase III program. No modifications need to be made to this section.

Alan J. Krupnick Page 10, Second Bullet I am not clear from reading the report exactly how un certainties are going to be treated in the survey.

James R. Kahn Page 10, Key Variables (Section 2.2) The authors do an excellent job in terms of tying key policy variables to the structure of the choice experiment framework (conjoint analysis). The literatu re cited is appropriate (Adamowicz, et al, Louviere et al, Bennett and Blamey and Batement).

Kevin J. Boyle Page 10, Bottom It is good and necessary for welfare calculations to havean opt-out option in the choice questions, not all choice question designs include this design feature.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 26 James R. Kahn Page 11, Survey Support (Section 2.3) This reviewer is very familiar with Abt Associat es, Inc., and has great respect for their work. This reviewer is not familiar with the work of KnowledgeNetworks, Inc., but presumes that the authors have chosen a reputable sampling firm.

James R. Kahn Page 11, Collection Method (Section 3.1) The proposed method of an internet-based survey is the best method that can be implemented given the time constraints that EPA faces. The internet-base su rvey has the advantage of being able to supply information in a fashion that is more difficult inface-to-face or mail surveys, as well as avoiding the biased associated with the use of a telephone survey. I have some concerns that the use of the internetbased survey can lead to some bi ases that can not be corrected through advanced statistical methods. This will be further elaborated in the discussion of biases. Nonetheless, the proposed method is undoubtedly the best direction to go for the type of inform ation that is needed to support this decision.

Kevin J. Boyle Page 11, Paragraph 1 of Section 3.1 I do not believe that the last sentence of this paragraph is true. A panel member could give access to another member of their household without the su rvey administrator’s knowledge. You have no more control here than you do with a mail survey; the recipient can allow another to respond.

Richard C. Bishop Page 11, Paragraph 3 It is probably worth mentioning explicitly here or elsewhere that the KN panel includes people who do not have Internet access. The WebTV option is mentioned, but not explained. This is a concern that, for example, Dillman (2000) mentions with regard to Internet surveys and I think KN has covered this one pretty well. Also, those who complete surveys ar e rewarded in an effort to reduce attrition.

Kevin J. Boyle Page 12, Top What if computer technology is the reason people optednot to participate in the panel? This follow-up procedure will do nothing to bring this type of person into the study.

Richard C. Bishop Page 12, Paragraph 2 This paragraph is hard to follow. For example, “Respondents will receive an incentive fee of $5 to $10 for completing this survey.” Which survey, the follow-up or the main survey or both?

James R. Kahn Page 12, Survey Response and Follow-up (Section 3.2) I am somewhat concerned about the length of time that is required to complete this survey. Between the review of the background materials and the actual completion of the survey, EPA estimates that 40 minutes are required. Although this length of engagement is consistent w ith conjoint analysis experiments in the test-marketing of new market goods, these marketing research firms pay the respondents substantially more than $5 to $10 do llars. I have had previous discussi ons with marketing professors who suggest that $30 to $50 dollars is mo re appropriate. The longer the time of the survey, the greater must be the payment to guard against the onset of res pondent fatigue and meaningless responses. Although the survey contains some follow-up questions to test th e validity of responses, it might make sense to increasePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 27 the payment to users as much as the research budget will permit, up to about $30.

Alan J. Krupnick Page 12, Paragraph 3 A 40-minute average survey is longer than usual fo r such surveys (30 minutes is usually the target).

Richard C. Bishop Page 12, Paragraph 3 It is better not to promise too much. “The survey administration format will be designed to ensure that individuals who agree to participate in the follow- up interviews must review all background information before completing the survey.” I don’t know of any way to make sure people actually read and digest such information. Also, here and elsewhere it is a little misleading to call these surveys interviews. Interviews imply person-to-person contact, either in person or at least over the phone. Unless I missed something, only the Internet will be used.

James R. Kahn Page 12, Pretests and Pilot Tests (Section 3.3) The pretest focus group and proposed pilot study w ith 500 responses seem to be appropriate. The second paragraph of page 13 indicates that "Results from the final two focus groups (one including cognitive interviews) provided convincing evidence that the majo rity of respondents answer the stated preference survey in ways appropriate for stated preference WTP estimates, and that their responses do not reflect biases noted above." The October 18 memo/report fro m Abt Associates that documents the focus group research provides evidence that supports this conclusion.

Richard C. Bishop Page 12, Paragraph 4 At least for the studies I am involved in, regular KN panelists do not receive direct cash payments, but rather points that can be redeemed for rewards. Have special arrangements been made to provide cash payment is this case? If so, it would be worthwhileto test for the effects of cash versus points.

Richard C. Bishop Page 12, Paragraph 5 What is EPA ICR #2155.01? I am not sure I unders tand the difference between cognitive interviews, think aloud analyses, and verbal protocol analyses. It is always hard for an outsider to evaluate what happened in such sessions, but at least the exact proce dures could be spelled out. Where were they held? How were subjects recruited (e.g., were people from a wide range of socioeconomic backgrounds successfully recruited)? Who moderated? What pr ocedures were followed in each group or set of groups (e.g., how did the subjects “complete the survey,” on-lin e or paper versions)? Did participants often say things that indicated that they hold use and/or nonu se values for the proposal? If so, what sorts of reasons did they give? Since participants filled out the survey, how about some tables with results?

Kevin J. Boyle Page 13, First Full Paragraph In assessing the design it would have been much more helpful if the text reported the issues/concerns identified and how they were specifically addressed in th e survey design. This type of claim is not helpful and only raises questions in my mind.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 28 James R. Kahn Page 13, Target Population (Section 4.1.1) This is a standard approach

.

James R. Kahn Page 14, Sampling Frame (Section 4.1.2) I have several concerns about this section. Th e document does not make clear whether the telephone numbers in the sample are land-lines or both land-lines and cell phones. If cell phones are not included, there is a serious sampling problem because young pe ople (18 to 25) generally do not have land-lines unless they are living in the parents' home. (I do not have statistics on this, just my general observations of my children, their friends and all the university students with whom I have contact). I also believe that there are biases introduced by who answers the phone in the household. If the survey is implemented with the person who answers the phone, then lazy husbands will tend to be underrepresented in the sample. On the other hand, the phone could be given to the pe rson in the household who most cares about fishing. This would introduce a different type of bias in the sample. This section does not state if there is a procedure for randomly choosing a person within the household.

James R. Kahn Page 14, Sample Size (Section 4.1.3) The authors use standard practice.

Alan J. Krupnick Page 14, Paragraph 2 (Section 4.1.3) If the precision achieved is “more than adequate” than one should do what is adequate and spend the rest of the money doing additional treatments to shore up unavoidable issues with the survey.

James R. Kahn Page 14, Stratification Variables (4.1.4) Knowledge Network is capable of creating a st ratified sample, which adequately corresponds to representation of different socio-economic groups as re presented in the U.S. Census. My concern here is since people are being asked to engage in an onlin e survey, the group of people who are computer literate (and have online access) may give different responses th an the group of people who decline to participate because they are made uncomfortable by information technology. There two groups may have radically different WTP for environmental improvement, and th ere is no way to correct this potential bias through over-representing a particular demographic or socio- economic segment of the population in the sample. The document briefly touches on this type of issu e when talking about the 600 individual non-response survey, but does not say how they intend to be guide d by the information that is developed from the nonresponse survey.
Kevin J. Boyle Page 15, Paragraph 1 What basis do you have for identifying “groups that tend to provide inconsistent or invalid answers” to attribute-based choice questions? I do not know of any information in the literature to support this sample design.

Kevin J. Boyle Page 15, Table 2 How does this stratification by region compare to the percent of the national population living in each region?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 29 James R. Kahn Page 16, Sampling Method (Section 4.1.5) This is a fairly standard approach but does not alleviate my concerns with section 4.1.4.

James R. Kahn Page 16, Response Rate (4.2.1) The response rate seems appropriate, but my origin al concerns about the potential for biases associated with information literacy remain.

James R. Kahn Page 17, Precision Targets (4.2.2) The level of the precision targets is fine, but the document provides not a single clue as to how they were computed, or chosen based on sta ndard references in the literature.

Kevin J. Boyle Page 17, Table 3 The 90 percent confidence intervals for a “typical bi nary variable” are meaningless in this context; what we care about is the error bounds on the welf are estimates derived from conjoint responses.

Kevin J. Boyle Page 18, Paragraph 1 I disagree with the statement that the “use of such methods to identify protest responses is well-established in the literature.” Some studies do nothing to identify protest responses, others use questions as you are proposing and others use statisti cal procedures to identify data outliers. There is no established procedure and all are ad hoc . There is nothing wrong with doing what you propose it is just that the conceptual and empirical foundations are not as solid as you portray them. My experience suggests that a carefully designed protest protocol will result in many respondents providing contradictory evidence, a single person can provide information that indicates a valid response while another of their responses may indicate a protest; what do you do with this respondent.

James R. Kahn Page 18, Non-Sampling Errors (Section 4.3) The discussion of the potential problems associated with protest bids, and the questions included in the survey to identify potential problems with protest bids are adequate.

The section also discusses potential problems with a nd solutions to noresponse errors and unobserved heterogeneity. The actual explanation of the solutions is left to sections 4.3.1, 4.3.2, and 4.3.3.

James R. Kahn Page 18, Stratified Survey Sample (Section 4.3.1) This section looks at the problem of nonresponse bias. If a particular groups of the population are more prone to decline to participate, then over-representi ng this group in the initial sampling effort can correct this problem, and help to generate a complete d sample population with the same demographic and socioeconomic characteristics as the population as a whol e. In theory, this is correct, but the document does not specify a procedure to identify the degree of over-targeting the various demographic and socioeconomic groups.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 30 Richard C. Bishop Pages 18 and 19 and Elsewhere Let me say at the outset that—pending clear evid ence to the contrary—I find arguments for the representativeness of KN panels convincing. Plans for th is study relating to adjustments in the sample for the main survey seem sound to me. On the other hand, there are those who are holding fast to the conventional wisdom that validity requires probability sampling and a high response rate. Folks at OMB are a case in point and, as you are well aware, they con tinue to be all powerful. In this context, the follow-up survey plan does not seem to me to be suffi ciently convincing to carry the day. A total of 600 people from various stages of the recruitment process w ill be sampled and asked to complete an Internet survey. It seems unrealistic to hope for much of a response from people who have refused to get involved with Knowledge Networks in the first place and an in centive of a few dollars is not likely to help. Not only are they likely to be put off by the new contact but many of them will not have an Internet connection. And not one citation is provided to bolster confidence in what is being proposed. Has KN assured you that they have successfully done this be fore? If so, a few statistics would be incredibly helpful here.

This is a thorny problem to be sure. It was one of my reasons for recommending the employment of a prominent survey expert (see part C under Genera l Comments). Dealing with nonresponse bias has, of course, been an issue in survey research from the beginning. How have survey experts addressed the problem? How can this study capitalize on what has b een learned in the past? Beyond that, we need to continue to hammer away at the simplistic emphasi s on response rates and focus attention on the basic issue: representativeness. It is also important to st ress that there is no such thing as a perfect sample or survey mode. A good starting point is Krosnick’s “Survey Research” in the 1999 issue of the Annual Review of Psychology(http://www.unt.edu/rss/class/survey/survey_review.pdf). He points out there (p. 451) that “when probability sampling methods are used, it is no longer sensible to presume that lower response rates necessarily signal lower representativ eness.” He even cites studies where a higher response rate caused bias. None of the studies he cit es fit your needs here exactly, but there is a lot of food for thought. I understand that Krosnick brought a similar message to a recent EPA workshop.

I don’t claim to have a complete solution, but here ar e a couple of ideas. First, one thing we are doing is adding questions from a general population survey dea ling with environmental attitudes. You could add some of these to the main survey. Then, you can show that your KN respondents match up with national demographics, but also hopefully that they match up with the environmental attitudes of the general population. Sorry I don’t have an exact citation at my finger tips for you, but I can get it for you if you want it.

Second, and this builds on the first, rather than hoping to get meaningful results from nonrespondents on the whole survey using the Internet, you could try doing an abbreviated tele phone survey. I have done this with nonrespondents to a mail survey by calling th em up, noting that they had not responded to the survey, and asking them if they would be willing to answer a few questions. Some said that they would still be willing to fill out the survey and, if so, we ask whether they needed a new copy. Whether they said they would fill it out or not, everyone who was willing to participate “for just a few minutes” got the short list of questions including some on socio-demogr aphics and environmental attitudes. Then we were able to adjust for nonresponse using the telephone survey responses along with the main data set. Obviously, your situation is somewhat different, but it seems like this approach could be adapted.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 31 James R. Kahn Page 19, Nonresponse follow-up interviews (Section 4.3.2) This section is also inadequate in its explanation of the procedures to be used to incorporate results from the non-response survey into the final analysis.

Kevin J. Boyle Page 19, Paragraph 2 I do not know of any articles in the literature wher e a Heckman selection model has been merged with mixed logit (Page 25, 2nd Paragraph). This is an im portant estimation issue you will need to address as you will need to consider the correlated responses of individual respondents and the literature is strongly supporting heterogeneous preferen ces through random parameters. You can not run a Heckman on a simple specification and then address these issues in a second model, which seems to be implicitly implied in the write up.

James R. Kahn Page 19, Statistical Correction for Unobserved Heterogeneity (Section 4.3.3) There seems to be an inconsistency between the discussion in this section in the second and third paragraphs of page 19 and the econometric specification of the desired model at the top of page 30. I do not see a Heckman correction coefficient in this mode l. This lack of consistency leads to doubts about how thoroughly the authors have thought through the correction for unobserved heterogeneity. This must be addressed.

Alan J. Krupnick Page 19 through 20 Heckman’s approach is referenced but not explained.

Richard C. Bishop Page 21, Paragraph 1 and Elsewhere I am somewhat concerned about the cost of living vehicle. Are there any precedents for using this wording in the literature? What does it mean exactly , and how does it relate to conventional definitions of willingness to pay? How would cost of living work into a traditional indirect utility function like that on p. 23? We are, of course, looking for a reduction in income that would wipe out the gain in utility from a given policy. Will respondents understand that poi nt? After all, I only have so much income. If my cost of living goes up, it doesn’t mean that I will have that much less money to spend on goods and services (unless I reduce savings by an equivalent amount). Cost of living has to do with not being able to buy as many goods and services. I suppose one might fall back on the idea that an increase in the cost of living is a reduction in real income, but I am not sure the average c itizen thinks in those terms. I think going with “Increased cost to you household because of higher prices” or something like that would be safer. The focus group memo is reassuring in this re gard. Subjects seemed to understand that the choice questions were referring to costs to them. Neverthe less, why not be completely explicit by referring to “the prices you pay for goods and services produced at f actories and power plants that use cooling water from ocean, lakes, and streams”?

While on this topic, let me also question the effort to convince subjects that taxes won’t increase. Given that prices and taxes are often combined in payment vehicles, I am not sure what the rationale is for avoiding taxes here. Perhaps it is an attempt to avoi d vehicle bias. Yes, some vehicle bias may creep in, but I still like to include taxes in a case like this fo r two reasons. First, that taxes would increase may seem more plausible to subjects. After all, the government buys lots of these goods too (e.g., paper and electricity). Is it plausible that my household costs would go up but the costs of government would not?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 32 Surely either taxes will go up or some people, at least, will suffer losses in government services. Second, dislike of taxes may lead to some vehicle bias, but it also serves as a wakeup call for subjects. Here is a cost that would be real and that you can’t avoid. If someone tells me that they are willing to pay more taxes for an environmental improvement, I am more inc lined to believe them. Also the direction of the bias, if any, is known. (Admittedly, choice of paym ent vehicle is something that various researchers in this area disagree about. It is still very much a judgment call. I am just giving you my opinion.)

Additionally, pages 15 and 16 of the focus group memo do not provide much evidence that taxes would be problematical in this case.

Richard C. Bishop Page 21, Paragraph 1 and Elsewhere The statement about not adding up or comparing programs is probably okay, but I am not sure your average person will get the point. I like to be more direct with something like, “At this point, it is not clear what the alternatives will actually cost, so we are asking about a range of costs. Also, we are asking about a range of effects so that our results can be adapted as scientists learn more.”

Kevin J. Boyle Page 21, Paragraph 1 It is good to tell respondents to consider each question i ndependently, but this is not likely to be sufficient (see Holmes and Boyle, Land Economics, Vol. 81, No . 1, 2005). This interac tion with how respondents answer the sequential questions leaves questions about what statistical information should be used for policy purposes.

Kevin J. Boyle Page 21, Paragraph 2 The certainty questions of Champ et al. (2004) are not very useful. It will always be possible to map two welfare estimates through some type of integer or c ontinuous preference scale. The question that remains is whether the specific point on the uncertainty scale is unique to the specific study or can be generalized to other types of applications. The Champ study is sile nt on this issue. There is no clear approach in the literature to “successfully control for hypothetical bias.” This does not mean that you should not try, it just means that you may not make the right adjustment and should not make the claim that you have adjusted for this bias; your resulting value may overestimate or underestimate welfare.

Kevin J. Boyle Page 23, Paragraph 1 I suggest in considering data quality you consider so me of the information that Berrens and colleagues have explored with internet surveys, e.g., time spent viewing slide show, time spent answering the valuation questions, etc. This inform ation could be interacted with respondent characteristics that you are collecting, e.g., environmental group membership, education, etc.

James R. Kahn Page 23, Analysis of Stated Preference Data (Section 6.1) This discussion is pretty standard and accepted within the environmental economics community.

James R. Kahn Page 23, Paragraph 2 The random utility model pre-dates both Hanemann and McConnell and shoul d be attributed to McFadden (1974).Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 33 Kevin J. Boyle Page 25, Second Complete Paragraph This is the type of model you need to use and I expect ed it to be stated more explicitly earlier in the text.

James R. Kahn Page 25, Advantages of Choice Experiments (Section 6.2) Again, this is standard and consistent with the literature.

Alan J. Krupnick Page 27, Paragraph 2 Here is the justification for the CE. It is not compelling.

James R. Kahn Page 27, Comment of Survey Preparation and Pretesting (Section 6.3) The EPA has done a good job in pretesting of the survey and in survey preparation.

James R. Kahn Page 28, Econometric Specification (Section 6.4) As mentioned above, the Heckman correction method th at was referenced section 4.3.3 is not mentioned in this section. This is a major problem and must beaddressed so that it is clearly understood exactly what econometric specification is being estimated.

Kevin J. Boyle Page 29, Paragraph 1 I am confused about how the “baseline population leve ls” are going to be implemented in the survey. I see on page 31 (bottom) the levels. Does this mean that the baseline will only be varied for the “Population Change” attribute? Is this part of the orthogonal design? I seem to be missing something from the explanation in the text.

Kevin J. Boyle Page 30, Model I suggest considering alternative-specific constants in the model. There are many variants in the published literature, but I have one specific recommendation. This recommendation is a binary variable that equals “1” if one of the two policy alternatives is chosen and “0” otherwise. This will give you an idea of whether people are “in the market” for reasons other than the attributes of your design that you explicitly control for in the estimation. There are a variety of reasons why the coefficient on this variable may be positive and significant and you cannot identify specific effects, but potential explanations include all of the reasons a person might be “incorrectly in the mark et”. You could try interacting some of your protest probe variables with this ASC.

James R. Kahn Page 30, Experimental Design (Section 6.5) This is a well done section, following the standardliterature. EPA has done a good job of eliminating pairs of choice sets which could lead to undesirable reactions by the respondents, such as a choice set which completely dominates another in all attributes, or nonsensical choice sets (high levels of saved fish but low levels of fish population). Attachment 4, wh ich further explains the experimental design appears to be on the mark.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 34 Richard C. Bishop Page 30, Experimental Design (Section 6.5) Experimental design is an important step in choice experiments. I am not an expert, but from what I know about it, this step has been more than adequate ly addressed in this section and in Attachment 4. My only suggestion is that perhaps there are too many le vels of the attributes. Unless you saw clear focus group evidence to the contrary, I would be tempted to drop 8% for the population change, 2% for catches, and $2 and $4 for costs. For this kind of broad polic y, I find it hard to believe that people will be that sensitive. (Below, in discussing the survey, I reco mmend that you not use monthly costs. If you use annual costs as the main cost attribute, maybe my advi ce about cost levels is less relevant. As it is, you are talking about pocket change.) Remember that if you can bracket the eventual outcomes you want to value in the benefit estimates, interpolation is an option.

Kevin J. Boyle Page 31, Attribute Levels The dollar amounts seem trivially small. These are unlikely to impinge on most respondents’ budgets (Page 34 2nd Paragraph), which makes me skeptical of discussions about how focus group respondents indicated that they are considering their budget constr aints (Page 34, bottom). You really need the pilot to make sure the estimated coefficient on cost will be significant.

Kevin J. Boyle Page 32, Paragraph 1 It is good to exclude dominated alternatives.

Alan J. Krupnick Page 32, Paragraph 1 The survey team recognizes the possibility that some pa irs will not make sense. But this raises the issue of interdependencies across the attributes.

James R. Kahn Page 32, Information Provision (Section 6.6) Any time that a valuation experiment is done, the r esearchers must provide information to the respondents concerning the implications of the changes they ar e being asked to assess. However, the researcher can be in a difficult situation because too much informati on or the wrong kind of information can dramatically influence a person's preference for one scenario over a nother. The researcher must try to create a balance between providing the information allowing for the measurement of the respondent's preferences, without creating a new set of preferences for the respondents. For example, showing pictures of cute baby seals in an oil spill scenario could unduly influence the results.

My impression is that in this case, EPA has done a good job in presenting important information in a "value neutral" fashion. I do not think they have cr ossed the line between being "estimators" of value and "creators" of value.

Alan J. Krupnick Page 33, Paragraph 2 In spite of the focus group sessions, there is no way EPAcan “ensure” that the graphics don’t lead to bias. The use of color and pretty pictures that are rou tine in this survey is a bad idea, in my view.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 35 James R. Kahn Page 33, Last Paragraph Note the use of a double negative, "not unavoidable."

James R. Kahn Page 33, Amelioration of Hypothetical Bias (Section 6.6.1) My personal opinion is that the concern about hypothetical bias is a result of a combination of some sloppy work by researchers in the early years of CVM, and a bias on the part of some economists against giving credence to values that do not originate in r eal world markets. If survey questions are designed appropriately, hypothetical bias need not be a problem. Conjoint analysis is a technique that is less prone to hypothetical bias than CVM, and the EPA has done a good job with its focus groups and survey design. I strongly agree with the conclusion in the second ha lf of the second paragraph on page 35 that "cheap talk" is neither necessary nor desirable in this survey . I think the decision framework of conjoint analysis obviates the need for cheap talk.

Kevin J. Boyle Page 34, First Complete Paragraph The incentive compatibility you refer to for a referendum format applies to a single, binary choice, not the application you have where respondents choosing over two alternatives and do nothing and answering multiple, sequential choice questions. A single choice qu estion with three answer options does allow for free riding and other strategic behavior, as do es answering a sequence of choice questions.

Kevin J. Boyle Page 34, Paragraph 3 Increasing the “consequentiality of choice” is important to the design, especially in regard to hypothetical choice, but may lead to protest responses as well.

Kevin J. Boyle Page 35, Paragraph 1 The statement that EPA finds little evidence to suggest that hypothetical bias should be significant in the proposed survey instrument” is naive and silly . The empirical evidence is quite clear that stated-preference methods can provide useful polic y information, but that value estimates tend to be overstated relative to actual monetary transactions. In recent work that I have done with Taylor and Morrison, which is funded by EPA, we find that welf are estimates from a three alternative attribute-based experiment can be 20 to 40 percent higher that comp arable cash values. More importantly, only 5 percent of respondents entered the market when cash was i nvolved and over 60 percent of respondents entered the hypothetical market. This is not an issue that you can design or assert away.

James R. Kahn Page 35, Amelioration of Symbolic and Warm-Glow Effects (Section 6.6.2): I agree with the EPA that they have appropriately eliminated warm glows, biases towards big businesses, and so on. My chief concern in this section is whether "slide 52" is believable. Many people are aware of the impacts of thermal pollution on estuaries and othe r water bodies, and they could believe that one of the technology options (re-circulating cooling water) c ould have less of an impact on water temperatures. Also, might people believe that reducing impingement of fish could also reduce impingement of reptiles, amphibians and birds? This may have been addressed in the work with survey groups, but a discussion is not contained in the document.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 36 Alan J. Krupnick Page 36, Paragraph 1 In my experience, symbolic concerns are always an issue.

Kevin J. Boyle Page 36, Slide 52 I think that you can still expect people to think about more than fish. I think people realize that birds eat fish and there are other ecosystem effects. The ASC th at I suggested above is one way to partially control for these and other unwanted c onsiderations by respondents.

Kevin J. Boyle Page 37, Assessing Scope Sensitivity (Section 6.6.3) I do not think that you have a true external test of scope. If your scope test follows the regular choice questions, which comprise your internal test, you have already “trained” respondents to consider the differences and this is not a true, external, between-s ubjects test. However, I am not convinced that the external test of scope is necessary.

James R. Kahn Page 37, Assessing Scope Sensitivity (Section 6.6.3) The EPA adequately address the issue of Scope Sensitivity

Richard C. Bishop Page 37, Assessing Scope Sensitivity (Section 6.6.3) This section is not always easy to follow, but it seems to make the question of scope harder than it needs to be. The internal scope test is adequately described except that on p. 38, paragraph 2, EPA states that the internal scope test assesses “sensitivity to scope acr oss choices of a single individual.” Surely looking at regression coefficients on attributes for the expected signs and significance does not involve single individuals. This is an internal scope test onl y because each respondent sees more than one stated-choice question. Scope-test purists worry that havi ng thought about earlier questions will influence how respondents look at later ones. This is easier to see with a simple contingent valuation exercise where asking a single set of respondents two questions wher e the scope of the environmental improvement is varied would be considered an internal scope. It would be “weaker” because such respondents might take the question about the small change as a cure that th ey should be willing to pay more for the large change or vice versa. The same problem—if it is a problem—could occur as subjects are asked a number of stated-choice questions, thus making the scope test using all the stat ed-choice questions an internal test. Particularly in an Internet survey (where you can control the order of question presentation and be notified when respondents skip the first stated-choice question or co me back and change their answers), one ought to be able to design the first question of each version of the survey so that respon ses can be used in a splitsample scope test. Provided sample sizes are large enough, the coefficients from the random utility model run on various first questions could be checked for si gn and significance. As I said, I am not sure I follow the scope test as planned in this session, but my approach seems simpler.

James R. Kahn Page 39, Communicating Uncertainty to Respondents (Section 6.6.4) On the whole, this is well done, but I have real probl ems with slide 30 in the third paragraph of page 40. The second sentence of this slide (Common sense dict ates....) really dismisses any scientific evidence about these relationships. The scientif ic knowledge should not be disreg arded in an appeal to common sense. The question should talk more specifically a bout how scientists can pin the outcomes to a rangePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 37 with a degree of confidence, and by the way, if y ou think about it this way, preventing the loss of young fish and fish eggs will certainly increase fish populations in the future.

Kevin J. Boyle Page 40, Slide 53 I believe addressing uncertainty is important and t oo many statedpreference studies overlook the fact that environmental outcomes are not known with certain ty. Having said this, I am concerned that your treatment of uncertainty may raise more problems than it helps. I understand the issue of cognitive overload, but other studies have incorporated speci fic information about uncertainty in the design, especially those valuing health outcomes. Your cu rrent design raises the issue without any focusing resolution so that each respondent is left to make their own assumptions regarding the likelihood of outcomes from the Phase III policy. Your “Champ style” question convolutes uncertainty about outcomes with uncertainty about responses. It is possibl e to have a scenario when outcomes are known with certainty and respondents are still uncertain about their response and vice versa.

James R. Kahn Page 41, Last Paragraph I agree with the conclusion.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 38 VI. SPECIFIC COMMENTS ON INFORMATIONAL SLIDE SHOW

Alan Krupnick General There are too many graphics and photos, any of which could introduce unknown biases. The researchers appear to have designed the information for maximum interest, which is good, but at the expense of a dry neutrality, which is necessary

Kevin J. Boyle Informational Slide Show I believe this is a very good mechanis m for conveying the background information.

Kevin J. Boyle Cover Page of Slide Show (Slide 1) You state that it is a survey of northeast residents, but it is far from clear whether respondents should just value the northeast or all of the U.S. Cons ider the following points in the slide show.

Slide 2: You say “protect fish stocks in the Northeast US waters.” What if people take trips to fish in other regions of the U.S. care about what happens there? For example, when I have gone fishing on the Great Lakes I know that some of the best fish ing is by these facilities, but when I fish in the northeast I know that my fishing is not near any facilities. Slide 10, 11 and 12: You talk about facilities, but not north east facilities. Won’t respondents wonder about controlling facilities in other parts of the country? Slide 13: The focus returns to the northeast. Slides 18 and 19: There is no mention of the northeast on these slides where you talk about fish losses. Slide 21: You talk about government policies, but no region is mentioned. Most respondents will know that regulation is done at the state and national levels. Slides 26 and 27: The first bullet on slide 26 states: “Redu ce fish losses in Northeast waters.” The following slide (#27) states: “EPA needs to know whether US households are willing to pay the cost ...” These two slides take you respondent both direc tions. Will northeast residents think that people from other regions of the country will also be paying for the northeast regional benefits? Finally, the choice questions do no specify the region of the effect s or who is voting. I think this inconsistency in information is bound to confuse some respondents.

Finally, the choice questions do no specify the region of the effects or who is voting. I think this inconsistency in information is bound to confuse some respondents.

Alan J. Krupnick Slide 2 “Protect fish stocks.” This seems like a biased statement.

Alan J. Krupnick Slides 5 and 6 “part of a larger ecological system.” So which is it? Are the fish an indicator of ecological effects or is the issue just fish? One information slide gives one picture, while another gives the opposite impression.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 39 Alan J. Krupnick Slide 9 “Many fish stocks have been declining.” This app ears to establish a declining baseline, but this idea is not carried forward to the choice questions or later in the text.

Richard C. Bishop Slide 9 Thoughtful subjects will see a glitch in your story. Here you say ecosystem functions are affected. Later you say its only fish and all other effects are negligible.

Richard C. Bishop Slide 10 I am not sure this diagram will make much sense to subjects at this stage. Also, “commercial facilities” seems vague. Could you say something like “paper mills, factories, and power plants?” Perhaps you could even combine this slide and the next one.

Richard C. Bishop Slide 12 “Aquatic organisms”? I thought this was about fish. Also the diagram might be explained with a couple of sentences in the audio. This is a very important slide. What does “intake flow” mean? What is this boxy looking thing in the middle of the picture, what I will call the “screen” here? Perhaps you should label the pipe walls. What is a “juvenile”? Is this really about juveniles and adults or large fish and small fish? Do fish that are small as adults only get impi nged and not entrained? Seeing the fish next to the screen labeled “adult” seems to imply that big (“adult” ) fish just turn and swim out. Nothing seems to be caught on the screen. Given the importance of what is being said here, you may need a better diagram unless focus group participants were very clear on wh at this diagram portrays. Are eggs only “fish” eggs? If so, they should be labeled as such. Did no one in the focus groups raise questions about other organisms like plankton? If so, you may need to clar ify why such effects are ne gligible. Does fish mean only finfish as implied here or are shellfish affect ed, say through destruction of eggs. If not, you may need to tell people why. Do you mean to imply that “screens” do or do not work? If not, why not? Is there any possibility of inserting a video clip here i nvolving either animation or a real intake structure?

Kevin J. Boyle Slide 13 The focus returns to the northeast.

Alan J. Krupnick Slide 15 “About 1/3 are types...” So does the 1/3 refer to species or to numbers of fish? Not clear.

Richard C. Bishop Slide 17 The second sentence of audio needs to be more clearly written.

Richard C. Bishop Slide 18 In the last line of the script, “is” should be “are”.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 40 Alan J. Krupnick Slide 19 The table is very complicated and should not be included like this in the information treatment.

Richard C. Bishop Slide 19 Not sure I follow the biology here. Take striped bass, for example. Presumably fish caught by fishermen are from the adult population. Do you mean that 1.8 million adult striped bass are lost or are sufficient sub-adult bass lost to I&E to equal that many adults or are you comparing apples and oranges (bass of all sizes in intake structures versus adult bass taken by fishermen)? The last possibility seems a bit misleading.

Richard C. Bishop Slide 20 I wouldn’t pick on commercial fishermen here, particularly given that recreational and commercial catches are combined in the preceding slide. Why makeyourselves targets of industry needlessly? In the next sentence, “it has contributed to a measurable dec line in some fish stocks, in some locations.” Sounds like government-speak to me. Subjects are likely to ask, what about the species that matter to me? Can you tell them more?

Richard C. Bishop Slide 21 I don’t like the term “technologies” here and elsewhere. I don’t know what it means to people. And “cooling water intake structure” and “air burst system” are not much better. A first cut at alternatives might include “equipment,” “structures used to pull water from water bodies for cooling,” and “equipment that uses blasts of air.” I hope the picture is clearer in the actual slide show. And those “aquatic organisms” get mentioned again.

Alan J. Krupnick Slides 22 thru 24 Pie charts are not very helpful when nearly all the chart is one color.

Richard C. Bishop Slide 22 “Close-cycle cooling”? What does that mean? Also this looks like you are mixing information. What does total fish losses have to do with different “lev els of technology”? I would work the total losses of 1.8 million fish into the pie chart on the next slide and explain the pie chart in the audio. Again, this is worth some time given how fundamental it is to the goals of the study.

Alan J. Krupnick Slide 23 “reduce fish losses by about 25%.” Doesn’t this set up an expectation about the probable size of fish loss reductions? If so, when the choice questions vary this amount, how will respondents interpret the difference?

Richard C. Bishop Slide 23 I don’t follow the diagram with the striped bass. What do the grid and the X mean?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 41 Kevin J. Boyle Slides 24 and 25 The information on these slides is confusing. You state that some facilities are using technologies that “reduce fish losses up to 98%.” You also state: “Theproposed policies would require similar technologies at all facilities ...” Does that mean that facilities usi ng the best technology might be required to use a less efficient technology?

Richard C. Bishop Slide 24 Better explain in a sentence or two what a closed-cyc le cooling tower does and how it works. Again the diagram (bass, arrow, X) may not communicate. Also, I’d give them a sentence or two explaining the pie chart.

Richard C. Bishop Slide 25 I have the impression from somewhere that the alread y regulated facilities are only power plants. If so, say so, and indicate that other kinds of facilities as well as some power plants are not currently regulated.

Alan J. Krupnick Slide 25 “Technologies...already in use at 52% of facilities.” This phrase is problematic. If we already have such experience, why is there uncertainty? And why is th e government regulating again? What is the current regulatory environment? What isn’t working now?

Alan J. Krupnick Slide 26 “Some costs would unavoidably be passed on.” Some, but not all. So would shareholders benefit? What is the respondent supposed to think? Passing on costs is not “unavoidable,” if the markets, for instance, are international and competitive, the manufacturer would have to bear the costs.

Richard C. Bishop Slide 26 For the second bullet, I would say something like, “increase the costs of producing goods and services, including those you consume.”

Alan J. Krupnick Slide 27 “EPA needs to know....” This is an odd phrase. We need to know how strongly people feel about reducing fish losses, but people are not being asked to do a benefit-cost analysis.

Richard C. Bishop Slide 29 and later ones I am not sure about the terms “policy” and “polic y options” here. It sounds too much like government speak. I would mention “regulations of factories and power plants that use cooling water from water bodies” when I absolutely had to (maybe a little thought would produce something shorter). Otherwise, I would use “regulations” when I thought I could get away with it and otherwise maybe “regulation of industries that use cooling water.”Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 42 Richard C. Bishop Slide 30 Good. This starts to take care of why different co mbinations of attributes will be presented. Reminders may be needed later. I would try to shorten the third bullet or make the last part a separate one, something like: Remember that, while saving more fisheggs and young fish will likely lead to more adult fish, scientists are still uncertain when they try to predict how many. One of my frustrations with the Internet mode is that, in mail surveys, I like to limit bullets to two lines or two and a half at most and that becomes very hard when using the Internet format. So far, I have not found a way around this disadvantage of Internet.

Alan J. Krupnick Slide 30 “Common sense...” I would say no. Common sense could refer to sustainable populations that are considerably below current levels and preventing loss of adult fish here could merely lead to equal numbers of death from some other direction.

Richard C. Bishop Slide 31 This is a good start toward legitimizing choosing a ch eaper alternative or no action. I would be even more explicit, adding another bullet at the top: We want to know how you personally really feel about the alternatives. (By the way, don’t be afraid to use unde rscoring to draw the eye toward a central theme in a longer bullet.)

Alan J. Krupnick Slide 31 Status Quo. Best practice, according to Adamowicz, is to portray the baseline as an additional column in the diagram, rather than just checking off a status quo response. To do so requires stating the ecological baseline, something that the survey team has not wanted to do so far.

Richard C. Bishop Slide 32 Maybe it’s just a matter of taste, but I am not sure your diagram helps much and may even confuse. In this case, will they get the point of the light bulb? I would add a few words of text under the pictures. On left “Cheaper Goods.” In the middle, “OR.” A nd on the right “More Fish.” And add a sentence of explanation to the script.

Richard C. Bishop Slide 33 Add to the script something like, “This slide shows th e form that some important questions in the survey will take. It shows two options, Option A and Option B. On the first row it shows how many fish would be saved, 1.3 billion under Option A versus only 980 million under Option B.

Richard C. Bishop Slides 34, 35, and 36 are okay as an introduction. Just tell the subject that more explanation will be coming soon. Otherwise, many subjects are going to think they are supposed to understand and remember what they are seeing here.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 43 Richard C. Bishop Slide 38 Did focus group members do okay with this explanati on of carrying capacity? Could you refer to levels in 1950 or 1900 or prior to European settlement or something? 1950 would make this match with changes in catch.

Richard C. Bishop Slide 40 I am not sure about asking them to assume something out of the blue. Maybe you could soften it by saying that “Scientist are still debating about exactly how current conditions compare to the 100% level, but 60% would be a safe assumption for purposes of this survey. If no new regulations are placed on cooling water intakes, fish populations are expected to continue at the 60% level in future years.” Did people in focus groups ever ask what about stronger po llution laws or fishing regulations? It is tough to know how to describe baseline with so many forces impinging on fish stocks, but do you dare say nothing about the future of other regulations? Even worse, did anyone question whether cooling water regulations are the cheapest way to improve things?

Richard C. Bishop Slide 45 I am evolving into a slide show designer rather than a review, so let me stop here. I hope these comments help to highlight why I think the slide show needs a lot of work.

Kevin J. Boyle Slide 40 You do not tell respondents what will happen if they chose not to vote for either option. You have to assume that that they assume that current conditions will be maintained, but some may assume further declines given the information you presented earlier. You need to specifically tell respondents what you want them to assume about the baseline. This applies to the other baseline conditions for other attributes as well.

Richard C. Bishop Slide 41 Be careful of the word “between” here. Do y ou mean to give a range of outcomes, between no improvement and an improvement to 75% of historical levels? That is not what the cell in the matrix on Slide 33 says. It seems to imply a point estimate, not a range of outcomes. Also there is a mismatch between this slide and 33, where Alternative A leads to the 70% level. Or is 70% the point estimate for the range between 60% and 75%? If so, this is very confusing and will not, in my opinion, help underscore scientific uncertainty effectively.

Richard C. Bishop Slides 42 thru 44 Similar comments apply.

Richard C. Bishop Slide 45 I am not sure that the point you are trying to make above the pictures will be apparent to a large share of respondents. Better try a sentence or two more in th e audio. The reference to water quality below the picture seems strange. Several factors other than wate r quality affect fish, as you have already explained. Why not say current conditions in the Northeast?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 44 Alan J. Krupnick Slide 52 “Significant impacts.” One slide says that there “are no other si gnificant impacts,” implying that the fish kills are a significant impact. This may introduce bias.

Kevin J. Boyle Slide 60 It is possible that there will be some tax consequen ces if some of the costs are written off. This could result in decreased tax revenues. I have found that a more believable payment vehicle allows prices and taxes to change.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 45 VII. COMMENTS ON FULL TEXT OF STATED PREFERENCE SURVEY

James R. Kahn Question 1 through 3 (Concern for policy issues) These questions are right on the mark and accomplishe d the intended goal of framing the policy context.

Richard C. Bishop Question 2 Having gone through the slide show, I am not sure I could answer this question correctly. I remember cooling water is not the most important but .... I think you are asking for trouble here. What if 20% or 30% get this wrong? Would you conclude that they di d not understand the material? The battery of truefalse questions I recommended earlier would serve you be tter. And don’t be too hard or tricky. Examples of true-false questions: 1) Cooling water withdrawals are the most important single cause of declines in fish populations in the Northeast. 2) Cooling water intakes are already regulated at many power plants. Etc. Then see how respondents do in the pretest.

Richard C. Bishop Question 3 I think this question errors in the other direction. It is never a good idea to insult your respondents. When 95% get this right, what will you conclude, that that percentage was not asleep or drunk during the slide show?

James R. Kahn Section II Instructions and Questions These represent a good set of instructions. Again, the report/memo of October 18 from Abt Associates provides documentation that these instructions are well-understood by the sample population.

James R. Kahn Questions 4 through 6 These are well-defined choice sets constructed in a fash ion that is consistent with the literature. The attribute levels are well-chosen, and provide suffici ent variation to reveal WTP and to conduct basic scope tests.

Richard C. Bishop Question 4 and following Two things: First, we are formatting this sort of qu estion differently. Rather than leaving the no-action alternative as one line at the bottom of the questi on as is done here, we are making it one of three alternatives. We have three options and Alternative A is always the no-action-at-zero cost alternative. Options B and C are like the ones you have here. We also ask not only for their most-preferred, but also their least-preferred alternative, so that we get a full ranking of the alternatives for each stated-choice question. This is not fully tested and published approachyet, but we think it is promising. It gives a lot of information about preferences. It raises the no- action alternative to the status of a full-fledged alternative, rather than hiding it at the bottom, whic h may imply to some subjects that you really expect most respondents to go for one of the alternatives in the matrix. And it would remind subjects of theattributes of the status quo. Second, there is not much precedent for using monthly payments in statedpreference studies. Of course this is how we pay for lots of stuff, but most researchers worry about coming across as a salesperson. “Do I have a deal for you! For a mere $2 a month you can save 450 million fish per year, increasing long-term fish populations to 65% of carrying capacity ....” I would go with the annual payment only.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 46 James R. Kahn Questions 7 through 9 Good debriefing questions.

Richard C. Bishop Question 8 Question 8 could be converted to true-false.

Richard C. Bishop Question 9 Question 9 is on the right track, but seems a little self-serving. Most people are going to know you expect them to take the first choice. Here is one of the places where a survey expert could help you get at what you want using more neutral questions.

Richard C. Bishop Question 10 Same as Question 9

.

James R. Kahn Question 10 Needs an additional response category as follows:

___I thought there were impacts on ecosystems in additi on to those created by changes in fish populations and partially based my answers on these impacts.

Richard C. Bishop Question 11 through 13 These are okay, but might work better with Likert scal es, which work very well in an Internet survey format. Ranking involves a larger respondent burden and may not yield answers that are as helpful when you construct covariates.

Richard C. Bishop Question 16 Since they will be a long way beyond the slide show , you might insert a map here or use some other means to remind them of what you mean by the Northeastern States. I like to use a longer time span in questions like this (last three year or last five y ears). Also I would give them a range: never, once or twice a year, etc. Some people seem to regard an open-ended question like this as requiring an exact answer. My way will also be more amenable to the Internet format. Also, is fishing from a boat fishing, boating, or both?

Richard C. Bishop Question 17 Question 17 will also need to be reformatted and adap ted for the Internet mode. You can ask open-ended question, but it will slow data entry and cost more.

Richard C. Bishop Question 18 Question 18 cries out for a Likert scale.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 47 Richard C. Bishop Question 20-29 Don’t ask respondents to answer questions that they have already answered when they were enrolled in the KN panel. Note also that town is an ambiguous term. In my home state a town is a rural unit of government below the county level and outside of inco rporated villages and cities. If you ask them about numbers in their household, be sure to tell them whether they are to count themselves or not.

James R. Kahn Questions The rest of the survey consists of standard ques tions, some focusing on debriefing issues, some seeking to measure intensity of preferences and some looki ng to define the demographic and socioeconomic attributes of the respondent. There are no problems here.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 48 VIII. ADDITIONAL REFERENCES

Richard C. Bishop A few are embedded above.

James R. Kahn Boxall, P.C. W.l. Adamawicz, M. Williams and J. Louviere, 1996: A comparison of stated preference models for environmental valu ation Ecological Economics 19, 243.

Casey, James, 2006, Modeling of farmer preferences for agroforestry systems, in the Handbook of Contingent Valuation, (Alberini a nd Kahn, eds.) Edward Elgar, 340-353

Carlsoonn, Fredrik, P Frykom and C. Lieejstope ( 2003) Valuing Wetland Attributes : An Application of Choice Experiments, Ecological Economcsi 47, 95-103.

Farber, Stephen and B. Griner, 2000, Valuing waters hed quality improvements using conjoint analysis, Ecological Economics, 34.

Hanley, Wright and Adamawicz, 1998 Using Choice expe riments to value the environment, Environ and Resource Economics, 11: 413

Holmes, Thomas R and W.L. Adamowicz (2003) Attribute based methods: A Primer on Non-Market Valuation, Champ, Brown and Boyle (eds) Dordrech t, the Netherlands, Kluwer Academic Publishers.

Kahn, James and Steven Stewart, 2006, An introducti on to choice modeling for non-market valuation, in A Handbook of Contingent Valuation, Kahn and Alberini, eds, Edward Elgar Publishers.

McFadden, 1974 Conditional Logit Analysis of Qua litative Choice Behavior in P Sarembeka (Ed), Frontiers in Econometrics, New York: Academic Press.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-1 APPENDIX A

REVIEWER COMMENTSPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-2 PEER REVIEWER COMMENTS

Richard C. BishopPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-3 PEER REVIEW OF

“PEER REVIEW PACKAGE FOR WILLINGNESS TO PAY SURVEY INSTRUMENT FOR §316(B) PHASE III COOLING WATER INTAKE STRUCTURES ”

Comments Submitted by: Richard C. Bishop

I. GENERAL COMMENTS

Before beginning my comments, let me suggest that y ou take them with a grain of salt in the following sense. I will definitely focus on potential problems, fo rgetting to tell you about all the things that are done well. Much hard work is evident in what I saw here and it has produced a lot of progress. Also, survey design is as much an art as a science and we who work in the field frequently disagree about best practice. In the end, those working on this study will have to use their own judgment about which of my comments and suggestions to focus on in final revisions before the pretest. I am kind of like a dentist doing a checkup. Indeed, reading my comments may seem like a trip to the dentist! I like to probe around, searching for the survey equivalent of cavities. I hope that some of my comments will help you do an excellent study. But you will and indeed shoul d decide to ignore others either because you are not sure they are helpful or you can’t take the time or expend the energy to fix what may be low priority problems. With these caveats, here are my general comments:

A. The work is being conducted in a total valuati on framework, which is appropriate to the case. A narrower framework that ignores nonuse values would ri sk neglecting most of the benefits of the regulations. B. So far as I know, the researchers are correct wh en they suggest that no previous studies have valued changes in fish stocks including forage , recreational and commercial species over a broad geographic area. This will be a path-breaking study in that regard. C. To some extent, I suppose I am passing the buck, but I would recommend that a survey expert be employed as a consultant to help with this study. A heavy hitter like Stanley Presser or Don Dillman or Jon Krosnick would be great if you can get one of them; another possibility would be Nora Cate Shaeffer of UW-Madison. Nora is less well known in environmental economics, but her credentials in survey methodology are imp eccable. I have personally worked with her on stated-preference studies (e.g., she is a co-author of the paper cited as Heberlein, 2005, in the document I was sent). Such a person could he lp evaluate and remedy problems with the instrument including those listed below. In add ition, for reasons stated below, I am concerned about the plan for addressing nonresponse issues. The sort of person I am recommending might be able to help a lot. D. I am particularly concerned that the slide show is not ready for prime time. In studies like this one, conveying information is crucial and very difficult. A large number of nitty-gritty issues are probed under Specific Comments. Let me raise some broader issues here. a. The document that was sent to me for this review was for the most part very clearly written. Most of the time, I was able to figure out what had been accomplished and what the plans were. Perhaps the biggest s hortcoming—if I missed it somewhere, I apologize—was the lack of a description of the procedures that would be used in the slide show. In what follows, I will assume that r espondents will be required to view the slides before they are allowed to open the survey proper; that the pace through the slide show will be controlled through Knowle dge Networks, and not at the discretion of the subject; that the script in Attachment 2 will be applied as voice-over with each slide; and thatPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-4 there will be no way for subjects to review slides after they have gone through the slide show. I will come back to this last issue in a moment. b. The slide show and the survey are currently planned as two separate steps. This is a major part of the research strategy and merits very careful consideration. There is no easy formula for conveying all the information needed by subjects in a study like this. Admittedly, there are some precedents in th e literature for setting up the information as a separate step, as noted in EPA’s write-up. A nice clean break between information and survey questions may work better, but after c onsidering this approach I think I still prefer to mix information and questions together, which is how I am tackling the information problem these days. In the Internet surveys I am working on, subjects are free to go back and forth between frames. Information is presented in modest doses. A frame or two of information is given and then a question pops up and they can go back to the information if they want to. Indeed, properly framed qu estions are an incentive to look back if some parts of the information are not yet clear. c. Frankly, the amount of information to be communicates all at once seems huge. Given detail in some of the slides, would it be unr easonable to assume it will take subjects close half an hour (30 seconds per slide) to view the slide show? If so, that is half an hour before they answer one survey question. The total process might take 45 minutes to an hour. That is asking a lot even of Knowle dge Networks subjects. If you do a separate slide show, it should be cut to 15 minutes or 20 minutes maximum and the pace should be set to be conducive to learning. Has having a separate slide show perhaps made it too tempting to add more and more slides? d. If you stick with the slide show, I would r ecommend that you make it interactive with the survey so that subjects can go back and revi ew information if they want. Try putting links in the survey so they can go back to one or more slides and then return to the place in the survey where they stopped. Also, they need to be able to turn the slide show on and off. (What if the phone rings or the ki ds start yelling in the other room?) E. Again, I apologize if I missed it somewhere, but I was a little surprised to find that the slide show and survey only referred to the Northeast. I woul d infer that subjects from different regions will get versions tailored to their region. Maybe th at would make sense, but I would be tempted to explain it in the slideshow to avoid confusion. Tell them that this is a national policy but that they need to be concerned only about their re gion since people in other regions will be paying costs there. If a national sample will be cons idering only changes that are proposed for the Northeast, then there is more explaining to be done. What about the fish in my region? F. As a general comment on design, except for wa ndering out to Wisconsin and Oregon for a couple of groups, all the qualitative work was done in a fairly tight area of the East. Why? I often find it instructive to have a wider geographic range. Also, the focus group memo points out that while people are generally more interested in their home regions, some care about other areas. Focusing on regions could cause some problems in this study. If I live in Maryland, for example, is Maine more important to me than Virginia? If I care somewhat about other regions, shouldn’t I be informed about them so I can include them in my values? (Perhaps I am not appreciating the difficulty of expanding the information to cover the whole country. Also, limiting people to their home region might be supported as a way of making value estimates conservative.)Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-5 II. CHARGE QUESTIONS

Question 1: Is the choice experiment survey an appr opriate stated preference survey format for the estimation of marginal willingness to pay for the characterized polic ies? Will the survey likely provide useful insights into pref erences, tradeoffs, and willingness to pay associated with the described policies?

Comments: Provided that stated preference surveys are well designed and executed, they have wide support for applications like this, both in the scientific literature, as cited in the materials provided to this reviewer and by govern mental agencies, including not only USEPA, 2000, as cited, but also the Office of Mana gement and Budget, Regulatory Analysis. Circular No. A-4. U.S. Office of Management and Budget, 2003. http://www.whitehouse.gov/omb/inforeg/circular_a4.pdf . Having worked in the field for many years, I am convinced that stated-preference methods are capable of providing useful insights into preferences, tradeoffs , and values in cases like this. The choice experiments being designed here are in keeping with one of the major branches of the literature that has many respected proponents.

Question 2: Does the effort used in developing and testing the proposed survey meet current standards for stated preference su rveys reflected in the literature?

Comments: Those designing the study display exce llent knowledge of the scientific literature on stated-preference methods; experts on the biol ogical side have obviously been consulted; attention has been given to theoretical issu es; survey materials were developed; focus groups were conducted and the survey materi als revised; and a peer review is being conducted now to gain additiona l input before the pilot study is implemented. Except perhaps for gaining more input from an expert in survey methods (See General Comment C above) and some specific questions raised below that may warrant further attention, these procedures meet current standards.

Question 3: Can the policy options be sufficiently well defined such that the survey is likely to provide useful insights into preferences, tradeoffs and willingness to pay?

Comments: The breadth of the impacts of I&E regul ations in terms of both geography and affected species makes this a challenging study. It is impossible to develop a scenario that touches on all the aspects that may be of relevance to all subjects. And, like so many other environmental economic studies, scientif ic uncertainty about both the baseline and the effects of the regulations add to the di fficulties. We need to recognize the limitations of stated-preference methods in such a context. For example, if it were possible to tailor surveys to the effects of the regulations in each subject’s home area, their answers might change. Still, useful—albeit imprecise—insi ghts into preferences, tradeoffs and WTP can probably be gained with this study. More comments drawing on the results of the focus groups might increase the level of confid ence. Did most participants seem to “get it” when they talked about what they were being asked to think through? Could most talk meaningful ways about the tradeoffs involved and why they might care about them? Did some members of the groups say they didn’t car e? (Not everyone needs to care about the fish affected by use of water for cooling and other such problems. I find it reassuring when members of groups I am conducting have learned enough and felt comfortable enough to say that they are not interested.)Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-6 Question 4: Does the information provided in the survey on the baseline ecological conditions, the expected changes, and the expected costs a nd their incidence seem sufficient, appropriate, unbiased and accurate? Does the survey provideappropriate and sufficient introductory and debriefing questions to enable identif ication of protest responses and symbolic responses, to test retention and understandi ng of information, etc.? Would you suggest any additional debriefing questions?

Comments: As I have stated, I think the slide show can be improved, but the basic information on baselines and changes is there and seems to be presented in an unbiased way. I have suggested some improvements in the debrie fing questions in my detailed comments on the survey (see below), but they involve fine tuning, rather than major changes. The introductory questions designed to assess whether the respondent understood the information (now Questions 2 a nd 3) should be beefed up. I like to use a series of truefalse questions when I am trying to evaluate what people have learned from the information provided. When I introduce the true-false questions, the instructions clearly say that the goal is not to test the partic ipant’s knowledge, like in school, but to test how well we did in communicating information to them. This allows me to go into more depth than Questions 2 and 3 in the current version of this instrument. And saying that respondents averaged 80% or 90% in answering the questions (which is my goal in information provision) is a good way to sile nce those who might otherwise try to argue that people didn’t read or didn’t understand the information. Let me hasten to add that not everyone agrees that this is a good idea. I think they are afraid of turning off respondents, thus contributing to overall and item nonresponse. But I have checked this approach out in focus groups and othe r qualitative research and so long as people understand the intent, I do not detect much resistance. I say more about the specifics below.

Additionally, Question 2 does not seem like a very good test for “Testing for Appropriate Understanding of the Introductory Materials.” Slide 8 of the introductory materials states that there are many causes of fish decline. Slide 20 says (emphasis in original) that intake structures “are usually smallerthan the effects of commercial fishing.” The voiceover for Slide 40 makes a vague reference to “a wide range of past human activities that have caused fish stocks to decline.” Unl ess I missed something buried in the slide show, respondents do not have a basis in the introductory information to answer Question 2. To the extent that they get the ranking correct, they must be falling back on their general knowledge.

Question 5: Does the survey sufficiently and appr opriately characterize the policy options under consideration? Does the survey place sufficient and appropriate emphasis on the scope of resource changes? Does the survey appropria tely communicate the scientific uncertainties as well as uncertainties associated with the policies in question?

Comments: See my responses to Question 3.

Question 6: What are your expectations regardi ng the potential validity and reliability of benefit estimates that might be provided by the proposedsurvey, compared to the current state of the art in the stated preference literature?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-7 Comments: I have stated some reservations about details both above and in the specific comments below. Much attention to the design of the s lide show and the survey is still needed, but I think this study is well along toward being state-of-the-art for one that must deal with such a broad regulatory initiative.

Question 7: Does the survey take appropriate step s to minimize hypothetical bias? Does the survey place sufficient and appropriate emphasis on the budget constraint facing respondents, and on potential substitute uses of their budget? Is the payment vehicle used in the survey credible? Does the survey incorporate appropriate language to ameliorate strategic bias, symbolic responses and ““yea saying””? Is ther e any sign of systematic bias in the survey or presented information? If so, how can it be ameliorated?

Comments: Section 6.6.1 shows that hypothetical bias has been thoroughly considered and addressed. I agree with EPA’s reluctance to introduce a cheap talk script. In my judgment the budget constraint and substitute reminders are more than adequate. Overdoing efforts to provide budget constraint and substitute reminders risks insulting significant numbers of subjects. (“What? Do you think I don’t unders tand that if I spend more money on this, I will have less money for other goods? Do you think I am stupid?”) I discuss the payment vehicle below (see specific comments re lating to Page 21 paragraph 1). I think the survey meets current standards in dea ling with the other potential sources of bias described in this question. Section 6.6.3 seems more than adequate in this regard.

Question 8: Will the answers to this survey lend themselves to straightforward economic interpretation of willingness to pay? Are the proposed econometric modeling approach and validity tests appropriate? Would you s uggest an alternative modeling approach or alternative or additional validity tests?

Comments: As I say below, I have some reservations about the payment vehicle, but otherwise the results should be amenable to straightforw ard interpretation. The plans for analysis, which involve not only a rather traditional approach, but also an effort to account for heterogeneous preferences seems more than ad equate for this stage in the project.

Question 9: The national benefits analysis for Phase III of the 316(b) rulemakings must be completed within the next several months and a final rule issued by June 1, 2006. Are there improvements to the survey and associated model that would be feasible to implement within the available time frame? If more time was available, how would your answer change?

Comments: That is a very short time horizon to do the pilot, main survey, follow-up survey, and analysis. You are likely to have to live with some warts and loose ends. More time would allow more attention to the suggestions coming out of this and other peer reviews, a thorough evaluation of the results of the pilot, a more careful statistical analysis of the main survey results, and a lot more thought about the follow-up survey. I urge you to add at least six months to the schedule.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-8 III. SPECIFIC COMMENTS

P. 6, paragraph 2: More advantages of the Internet mode might be mentioned. Four come immediately to mind. First, mail surveys can run into difficulties w ith skip patterns that can easily be programmed into an Internet survey. Second, where a number of stat ed-choice questions are to be asked back-to-back, it is possible to tailor later questions based on answ ers to earlier ones. Suppose a respondent chooses Alternative A over Alternative B when costs are $15 and $5, respectively. It would not be very useful to ask that same respondent a second question where attri bute levels are the same except that Alternative A has a cost level of $10. Mail surveys cannot be tailo red leave out such redundant questions. Third, voice over to help respondents through complicated or extra l ong materials can be added. Fourth, links to more detailed information can be added for those respondents who think they need it without complicating the main presentation for everyone. Also, regarding existing studies, do you want to mention the Viscuzi et al. (2004) study on water quality that was done for EPA and which was cited later in this document? I believe an Internet mode was used there.

P. 11, paragraph 3: It is probably worth mentioni ng explicitly here or elsewhere that the KN panel includes people who do not have Internet access. The WebTV option is mentioned, but not explained. This is a concern that, for example, Dillman (2000) me ntions with regard to Internet surveys and I think KN has covered this one pretty well. Also, those w ho complete surveys are rewarded in an effort to reduce attrition.

P. 12, paragraph 2: This para graph is hard to follow. For example, “Respondents will receive an incentive fee of $5 to $10 for completing this survey.” Which survey, the follow-up or the main survey or both?

P. 12, paragraph 3: It is better not to promise too much. “The survey administration format will be designed to ensure that individuals who agree to participate in the follow-up interviews must review all background information before completing the survey.” I don’t know of any way to make sure people actually read and digest such information. Also, here and elsewhere it is a little misleading to call these surveys interviews. Interviews imply person-to-pers on contact, either in person or at least over the phone. Unless I missed something, only the Internet will be used.

P.12, paragraph 4: At least for the studies I am involved in, regular KN panelists do not receive direct cash payments, but rather points that can be redeemedfor rewards. Have special arrangements been made to provide cash payment is this case? If so, it would beworthwhile to test for the effects of cash versus points.

P. 12, paragraph 5: What is EPA ICR #2155.01? I am not sure I understand the difference between cognitive interviews, think aloud analyses, and verbal prot ocol analyses. It is always hard for an outsider to evaluate what happened in such sessions, but at l east the exact procedures could be spelled out. Where were they held? How were subjects recruited (e .g., were people from a wide range of socioeconomic backgrounds successfully recruited)? Who moderated? What procedures were followed in each group or set of groups (e.g., how did the subjects “complete the survey,” on-line or paper versions)? Did participants often say things that indicated that they hold use and/or nonuse values for the proposal? If so, what sorts of reasons did they give? Since participants filled out the survey, how about some tables with results?Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-9 Pp. 18 and 19 and elsewhere: Let me say at the outset that—pending clear evidence to the contrary—I find arguments for the representativeness of KN panels convincing. Plans for this study relating to adjustments in the sample for the ma in survey seem sound to me. On the other hand, there are those who are holding fast to the conventional wisdom that validity requires probability sampling and a high response rate. Folks at OMB are a case in point and, as you are well aware, they continue to be all powerful. In this context, the follow-up survey plan does not seem to me to be sufficiently convincing to carry the day. A total of 600 people from various st ages of the recruitment process will be sampled and asked to complete an Internet su rvey. It seems unrealistic to hope for much of a response from people who have refused to get involved w ith Knowledge Networks in the first place and an incentive of a few dollars is not likely to help. Not only are they likely to be put off by the new contact but many of them will not have an Internet connection. And not one citation is provided to bolster confidence in what is being proposed. Has KN assured you that they have su ccessfully done this before? If so, a few statistics would be incredibly helpful here.

This is a thorny problem to be sure. It was one of my reasons for recommending the employment of a prominent survey expert (see part C under Genera l Comments). Dealing with nonresponse bias has, of course, been an issue in survey research from the beginning. How have survey experts addressed the problem? How can this study capitalize on what has b een learned in the past? Beyond that, we need to continue to hammer away at the simplistic emphasi s on response rates and focus attention on the basic issue: representativeness. It is also important to st ress that there is no such thing as a perfect sample or survey mode. A good starting point is Krosnick’s “Survey Research” in the 1999 issue of the Annual Review of Psychology (http://www.unt.edu/rss/class/survey/survey_review.pdf). He points out there (p. 451) that “when probability sampling methods are used, it is no longer sensible to presume that lower response rates necessarily signal lower representativ eness.” He even cites studies where a higher response rate caused bias. None of the studies he cit es fit your needs here exactly, but there is a lot of food for thought. I understand that Krosnick brought a similar message to a recent EPA workshop.

I don’t claim to have a complete solution, but here ar e a couple of ideas. First, one thing we are doing is adding questions from a general population survey dea ling with environmental attitudes. You could add some of these to the main survey. Then, you can show that your KN respondents match up with national demographics, but also hopefully that they match up with the environmental attitudes of the general population. Sorry I don’t have an exact citation at my finger tips for you, but I can get it for you if you want it.

Second, and this builds on the first, rather than hoping to get meaningful results from nonrespondents on the whole survey using the Internet, you could try doing an abbreviated tele phone survey. I have done this with nonrespondents to a mail survey by calling th em up, noting that they had not responded to the survey, and asking them if they would be willing to answer a few questions. Some said that they would still be willing to fill out the survey and, if so, we ask whether they needed a new copy. Whether they said they would fill it out or not, everyone who was willing to participate “for just a few minutes” got the short list of questions including some on socio-demogr aphics and environmental attitudes. Then we were able to adjust for nonresponse using the telephone survey responses along with the main data set. Obviously, your situation is somewhat different, but it seems like this approach could be adapted.

Page 21, paragraph 1 and elsewhere: I am somewhat concerned about the cost of living vehicle. Are there any precedents for using this wording in the literature? What does it mean exactly, and how does it relate to conventional definitions of willingness to pay? How would cost of living work into a traditional indirect utility function like that on p. 23? We are, of course, looking for a reduction in income that would wipe out the gain in utility from a given polic y. Will respondents understand that point? After all,Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-10 I only have so much income. If my cost of living go es up, it doesn’t mean that I will have that much less money to spend on goods and services (unless I reduce savi ngs by an equivalent amount). Cost of living has to do with not being able to buy as many goods and services. I suppose one might fall back on the idea that an increase in the cost of living is a reduc tion in real income, but I am not sure the average citizen thinks in those terms. I think going with“Increased cost to you household because of higher prices” or something like that would be safer. The focus group memo is reassuring in this regard. Subjects seemed to understand that the choice questions were referring to costs to them. Nevertheless, why not be completely explicit by referring to “the prices you pay for goods and services produced at factories and power plants that use coo ling water from ocean, lakes, and streams”?

While on this topic, let me also question the effort to convince subjects that taxes won’t increase. Given that prices and taxes are often combined in payment vehicles, I am not sure what the rationale is for avoiding taxes here. Perhaps it is an attempt to avoi d vehicle bias. Yes, some vehicle bias may creep in, but I still like to include taxes in a case like this fo r two reasons. First, that taxes would increase may seem more plausible to subjects. After all, the government buys lots of these goods too (e.g., paper and electricity). Is it plausible that my household costs would go up but the costs of government would not? Surely either taxes will go up or some people, at least, will suffer losses in government services. Second, dislike of taxes may lead to some vehicle bias, but it also serves as a wakeup call for subjects. Here is a cost that would be real and that you can’t avoid. If someone tells me that they are willing to pay more taxes for an environmental improvement, I am more inc lined to believe them. Also the direction of the bias, if any, is known. (Admittedly, choice of paym ent vehicle is something that various researchers in this area disagree about. It is still very much a judgment call. I am just giving you my opinion.)

Additionally, pages 15 and 16 of the focus group memo do not provide much evidence that taxes would be problematical in this case.

Same paragraph: The statement about not adding up or comparing programs is probably okay, but I am not sure your average person will get the point. I lik e to be more direct with something like, “At this point, it is not clear what the alternatives will actually co st, so we are asking about a range of costs. Also, we are asking about a range of effects so that our results can be adapted as scientists learn more.”

Pp. 30ff, Section 6.5: Experimental design is an important step in choice experiments. I am not an expert, but from what I know about it, this step has been mo re than adequately addressed in this section and in Attachment 4. My only suggestion is that perhaps there are too many levels of the attributes. Unless you saw clear focus group evidence to the contrary, I woul d be tempted to drop 8% for the population change, 2% for catches, and $2 and $4 for costs. For this kind of broad policy, I find it hard to believe that people will be that sensitive. (Below, in discussing the surv ey, I recommend that you not use monthly costs. If you use annual costs as the main cost attribute, maybe my advice about cost levels is less relevant. As it is, you are talking about pocket change.) Remember that if you can bracket the eventual outcomes you want to value in the benefit estimates, interpolation is an option.

Pp. 37ff, Section 6.6.3: This section is not always easy to follow, but it seems to make the question of scope harder than it needs to be. The internal scopetest is adequately described except that on p. 38, paragraph 2, EPA states that the internal scopetest assesses “sensitivity to scope across choices of a single individual.” Surely looking at regression coefficients on attributes for the expected signs and significance does not involve single individuals. This is an internal scope test only because each respondent sees more than one stated-choice question. Scope-test purists worry that having thought about earlier questions will influence how respondents look at later ones. This is easier to see with a simple contingent valuation exercise where asking a single set of respondents two questions where the scope ofPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-11 the environmental improvement is varied would be c onsidered an internal scope. It would be “weaker” because such respondents might take the question about the small change as a cure that they should be willing to pay more for the large change or vice versa. The same problem—if it is a problem—could occur as subjects are asked a number of stated-choice questions, thus making the scope test using all the stated-choice questions an internal test. Particularly in an Internet survey (where you can control the order of question presentation and be notified when respondents skip the first stated-choice question or come back and change their answers), one ought to beable to design the first question of each version of the survey so that responses can be used in a split-sample scope test. Provided sample sizes are large enough, the coefficients from the random utility model run on various first questions could be checked for sign and significance. As I said, I am not sure I follow the scope test as planned in this session, but my approach seems simpler.

Comments on slides: A. Slide 9: Thoughtful subjects will see a glitch in your story. Here you say ecosystem functions are affected. Later you say its onl y fish and all other effects are negligible. B. Slide 10: I am not sure this diagram will make much sense to subjects at this stage. Also, “commercial facilities” seems vague. Could you say something like “paper mills, factories, and power plants?” Perhaps you could ev en combine this slide and the next one. C. Slide 12: “Aquatic organisms”? I thought this was about fish. Also the diagram might be explained with a couple of sentences in the audio. This is a very important slide. What does “intake flow” mean? What is this boxy looking thing in the middle of the picture, what I will call the “screen” here? Perhaps you should label the pipe walls. What is a “juvenile”? Is this really about juveniles and adults or large fish and small fish? Do fish that are small as adults only get impinged and not entrained? Seeing the fish next to the screen labeled “adult” seems to imply that big (“adult”) fish just turn and swim out. Nothing seems to be caught on the screen. Given the importance of what is being said here, you may need a better diagram unless focus group participants were very clear on what this diagram portrays. Are eggs only “fish” eggs? If so, they shoul d be labeled as such. Did no one in the focus groups raise questions about other organisms like plankton? If so, you may need to clarify why such effects are negligible. Does fish mean only finf ish as implied here or are shellfish affected, say through destruction of eggs. If not, you may need to tell people why. Do you mean to imply that “screens” do or do not work? If not, why not? Is there any possibility of inserting a video clip here involving either animation or a real intake structure? D. Slide 17: The second sentence of audio needs to be more clearly written. E. Slide 18: In the last line of the script, “is” should be “are”. F. Slide 19: Not sure I follow the biology here. Take striped bass, for example. Presumably fish caught by fishermen are from the adult population. Do you mean that 1.8 million adult striped bass are lost or are sufficient sub-adult bass lost to I&E to equal that many adults or are you comparing apples and oranges (bass of all sizes in intake structures versus adult bass taken by fishermen)? The last possibility seems a bit misleading. G. Slide 20: I wouldn’t pick on commercial fisherme n here, particularly given that recreational and commercial catches are combined in the pr eceding slide. Why make yourselves targets of industry needlessly? In the next sentence, “it has contributed to a measurable decline in some fish stocks, in some locations.” Sounds like government-speak to me. Subjects are likely to ask, what about the species that matter to me? Can you tell them more? H. Slide 21: I don’t like the term “technologies” here and elsewhere. I don’t know what it means to people. And “cooling water intake structure” and “air burst system” are not much better. A first cut at alternatives might incl ude “equipment,” “structures used to pull waterPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-12 from water bodies for cooling,” and “equipment that uses blasts of air.” I hope the picture is clearer in the actual slide show. And t hose “aquatic organisms” get mentioned again. I. Slide 22: “Close-cycle cooling”? What does that mean? Also this looks like you are mixing information. What does total fish losses have to do with different “levels of technology”? I would work the total losses of 1.8 million fish in to the pie chart on the next slide and explain the pie chart in the audio. Again, this is worth some time given how fundamental it is to the goals of the study. J. Slide 23: I don’t follow the diagram with the striped bass. What do the grid and the X mean? K. Slide 24: Better explain in a sentence or tw o what a closed-cycle cooling tower does and how it works. Again the diagram (bass, arrow, X) may not communicate. Also, I’d give them a sentence or two explaining the pie chart. L. Slide 25: I have the impression from somewhere that the already regulated facilities are only power plants. If so, say so, and indicate that other kinds of facilities as well as some power plants are not currently regulated. M. Slide 26: for the second bullet, I would say something like, “increase the costs of producing goods and services, including those you consume.” N. Slide 29 and later ones: I am not sure about th e terms “policy” and “policy options” here. It sounds too much like government speak. I woul d mention “regulations of factories and power plants that use cooling water from wate r bodies” when I absolutely had to (maybe a little thought would produce something shorter). Otherwise, I would use “regulations” when I thought I could get away with it and otherwi se maybe “regulation of industries that use cooling water.” O. Slide 30: Good. This starts to take care of why different combinations of attributes will be presented. Reminders may be needed later. I would try to shorten the third bullet or make the last part a separate one, something like: Re member that, while saving more fish eggs and young fish will likely lead to more adult fish, scientists are still uncertain when they try to predict how many. One of my frustrations with the Internet mode is that, in mail surveys, I like to limit bullets to two lines or two and a half at most and that becomes very hard when using the Internet format. So far, I ha ve not found a way around this disadvantage of Internet. P. Slide 31: This is a good start toward legitimiz ing choosing a cheaper alternative or no action. I would be even more explicit, adding another bullet at the top: We want to know how you personally really feel about the alternatives. (By the way, don’t be afraid to use underscoring to draw the eye toward a central theme in a longer bullet.) Q. Slide 32: Maybe it’s just a matter of taste, but I am not sure your diagram helps much and may even confuse. In this case, will they get the point of the light bulb? I would add a few words of text under the pictures. On left “Cheaper Goods.” In the middle, “OR.” And on the right “More Fish.” And add a sentence of explanation to the script. R. Slide 33: Add to the script something like, “T his slide shows the form that some important questions in the survey will take. It shows tw o options, Option A and Option B. On the first row it shows how many fish would be sav ed, 1.3 billion under Option A versus only 980 million under Option B. S. Slides 34, 35, and 36 are okay as an introduction. Just tell the subject that more explanation will be coming soon. Otherwise, many subjects are going to think they are supposed to understand and remember what they are seeing here. T. Slide 37: The script talks about scales while the pictures talk in terms of percents. You need to be consistent. My guess is that people will grasp percents quicker than scales, particularly scales with large numbers of points like 100.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-13 U. Slide 38: Did focus group members do okay with this explanation of carrying capacity? Could you refer to levels in 1950 or 1900 or prior to European settlement or something? 1950 would make this match with changes in catch. V. Slide 40, I am not sure about asking them to assume something out of the blue. Maybe you could soften it by saying that “Scientist are still debating about exactly how current conditions compare to the 100% level, but 60% would be a safe assumption for purposes of this survey. If no new regulations are placed on cooling water intakes, fish populations are expected to continue at the 60% level in futu re years.” Did people in focus groups ever ask what about stronger pollution laws or fishi ng regulations? It is tough to know how to describe baseline with so many forces impinging on fish stocks, but do you dare say nothing about the future of other regulations? Even worse, did anyone question whether cooling water regulations are the cheapest way to improve things? W. Slide 41: Be careful of the word “between” he re. Do you mean to give a range of outcomes, between no improvement and an improvement to 75% of historical levels? That is not what the cell in the matrix on Slide 33 says. It seems to imply a point estimate, not a range of outcomes. Also there is a mismatch betweenthis slide and 33, where Alternative A leads to the 70% level. Or is 70% the point estimate for the range between 60% and 75%? If so, this is very confusing and will not, in my opi nion, help underscore scientific uncertainty effectively. X. Slides 42-44: Similar comments apply. Y. Slide 45: I am not sure that the point you are trying to make above the pictures will be apparent to a large share of respondents. Better try a sentence or two more in the audio. The reference to water quality below the picture seem s strange. Several factors other than water quality affect fish, as you have already explai ned. Why not say current conditions in the Northeast? Z. I am evolving into a slide show designer rath er than a review, so let me stop here. I hope these comments help to highlight why I thi nk the slide show needs a lot of work.

Comments on the survey:

A. Q2: Having gone through the slide show, I am not sure I could answer this question correctly. I remember cooling water is not the most important but .... I think you are asking for trouble here. What if 20% or 30% get this wrong? Would you conclude that they did not understand the material? The battery of true-false questions I recommended earlier would serve you better. And don’t be too hard or tricky. Exam ples of true-false questions: 1) Cooling water withdrawals are the most important single cause of declines in fish populations in the Northeast. 2) Cooling water intakes are already regulated at many power plants. Etc. Then see how respondents do in the pretest. B. Q3: I think this question errors in the other direction. It is never a good idea to insult your respondents. When 95% get this right, what w ill you conclude, that that percentage was not asleep or drunk during the slide show? C. Q4 and following: Two things: First, we are formatting this sort of question differently. Rather than leaving the no-action alternative as one line at the bottom of the question as is done here, we are making it one of three alternatives. We have three options and Alternative A is always the no-action-at-zero cost altern ative. Options B and C are like the ones you have here. We also ask not only for their most-preferred, but also their least-preferred alternative, so that we get a full ranking of the alternatives for each stated-choice question. This is not fully tested and published approach yet, but we think it is promising. It gives a lot of information about preferences. It raises th e no-action alternative to the status of a full-Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-14 fledged alternative, rather than hiding it at th e bottom, which may imply to some subjects that you really expect most respondents to go for oneof the alternatives in the matrix. And it would remind subjects of the attributes of the status quo. Second, there is not much precedent for using monthly payments in stated-p reference studies. Of course this is how we pay for lots of stuff, but most researchers wo rry about coming across as a salesperson. “Do I have a deal for you! For a mere $2 a month you can save 450 million fish per year, increasing long-term fish populations to 65% of carrying capacity ....” I would go with the annual payment only. D. Q8 could be converted to true-false. E. Q9 is on the right track, but seems a little self-serving. Most people are going to know you expect them to take the first choice. Here is one of the places where a survey expert could help you get at what you want using more neutral questions. F. Q10: ditto. G. Q11through Q13: these are okay, but might work better with Likert scales, which work very well in an Internet survey format. Ranking involves a larger respondent burden and may not yield answers that are as helpful when you construct covariates. H. Q16: Since they will be a long way beyond the slide show, you might insert a map here or use some other means to remind them of what you mean by the Northeastern States. I like to use a longer time span in questions like this (last three year or last five years). Also I wouldgive them a range: never, once or twice a year, etc. Some people seem to regard an openended question like this as requiring an exact answ er. My way will also be more amenable to the Internet format. Also, is fishing from a boat fishing, boating, or both? I. Q17 will also need to be reformatted and ad apted for the Internet mode. You can ask openended question, but it will slow data entry and cost more. J. Q18 cries out for a Likert scale. K. Q20-29, don’t ask respondents to answer questi ons that they have already answered when they were enrolled in the KN panel. Note also that town is an ambiguous term. In my home state a town is a rural unit of government below the county level and outside of incorporated villages and cities. If you ask them about numbe rs in their household, be sure to tell them whether they are to count themselves or not.

IV. ADDITIONAL REFFERENCES

A few are embedded above.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-15 PEER REVIEWER COMMENTS

Kevin J. BoylePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-16 Comments on:

“Willingness to Pay Survey Instrument for §316(b) Phase III Cooling Water Intake Structures”

By:

Kevin J. Boyle Professor and Head Department of Agricultural and Applied Economics Virginia Tech

February 2, 2006

Answers to the Peer-Review Questions:

1. The choice experiment is an appropriate me thodology and is the best methodology available to estimate values for the Phase III policy.

The survey results will provide useful informati on, but perhaps not as strong as asserted in the peerreview document (see my comments below).

2. The design effort does meet current standards, but the inferences from this design process are too strong (see my comments below).

3. Yes, the policy options can be sufficiently well defined. I have made some specific recommendations that should be addressed and the limitations/assu mptions should be more explicitly stated.

4. The ecological condition information is good. R espondents need to be told what to assume about baseline conditions. I presume that you want them to assume that current conditions will be maintained if they do not choose a policy alternative but this in not clear in the information presented to respondents. Limitations, like no information on sp ecific species, need to be explicitly stated and recognized in the study documentation.

I think the information is well balanced.

I have noted below that I am skeptical of debriefi ng questions in general a nd do not have any specific recommendations.

5. I find this question hard to answer. First, the ultimate ecological effects are not known. Thus I do not know if the selected baseline conditions and attribut e levels will be able to address the actual policy question. Second, while the analysis discussion talk s about linear and quadratic specifications, with a small number of attribute levels the best that you may be able to estimate is a piece-wise approximation to a nonlinear function, which leav es question if the actual policy conditions fall on intermediate levels.

I think the scope of the change is well communicat ed with words, numbers and graphics, except for the region of focus.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-17 Uncertainty is not addressed well (see my comments below).

6. I believe that with further revisions this will bea state-of-the-art instrument and data collection process. I have concerns that the data collec tion process may not end up being as sophisticated as proposed. The data analysis section is too vague at this point to address whether this portion of the study will be state-of-the-art.

7. The survey does a good job of addressing hypothetical bias, but this bias is not controlled to the extent that the review document claims. One add itional improvement would be to make the decision rule by which the policy will be provided more explicit.

There is sufficient emphasis on the budget constraint, but I wonder how much it matters with such low cost amounts. In fact I am concerned that th e coefficient on the cost variable may not ultimately be significant so the pilot study is imperative to test this hypothesis.

The payment vehicle is credible, but it may be be tter to acknowledge that there may be some tax effects.

The study does the best it can to address strategic bias, symbolic responses and yea saying, but the design choice experiments opens up the possibility for all of these problems. Thus, I do not think that the controls are as strong as the study designers claim (see my comments below).

I do not see a systematic bias in the information presentation, but do make some suggestions for improvements.

8. The study results can lead to “straight forwar d economic interpretation” if the sample weighting issues are resolved and state-of-the art estimation techniques are used. Neither of these points is clear in the current documentation and I say more on these points below.

There is only one validity test that I see in the desi gn and that is the external test of scope, which I believe is fundamentally flawed (see my comments below). 9. I do not believe the study can be completed by June 1! Yes, if there was 9 months to a year available.

General Comments:

1. There are many global statements about valuation and the design of this specific study that are purely assertions and are not based on any hard empirical research or theoretical foundations. These statements detract from what is really a basically sound research design. Examples of the former are statements with respect to the incentive compatibility of choice questions and hypothetical bias, and examples of the latter are extrapolations from what was learned from the focus groups.

2. The entire discussions of protest responses, hypothetical bias, and other problematic responses appear to select items from the literature at face value. The key point is that you did try to address these issues, but you probably have not in the design nor will not be able to in the estimation do as much to control for these problems as you imply in the review document. The key point really is how much subsets of people in these categories actually affect your welfare estimates. My reading of the literature is that the only major concern in this regard in hypothetical bias and you are going to do as much as you can to control this issue, but not to the extent you claim.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-18 3. You talk about a sample design that includes multiple weighting schemes and there is no discussion about how you are going to resolve these different levels of weights nor how you are going to investigate how weights affect your welfare estimates. One key issue is that you, like many others, assume that making the sample look like the U.S. population will solve all nonrespone biases, but if sample selection arises from something other than Census demographics you have not corrected the problem, you have an endogenous weighting problem.

4. There is no discussion of how you are going to a dd up the values from different regions to get a benefit estimate for national policy. What if r espondents value an area larger than their sampling region? What if respondents think that people fro m other regions will pay for their region? See my comments below.

5. You assume that the type/mix of fish does not matter to people. This is a strong and perhaps necessary assumption. You must make it explic it because information on specific species while substantially increasing the information load fo r respondents may also change values estimates.

6. I think this is a well designed study, but there is definitely a need for improvement. While considerable effort has gone into the scenario design, the design is not as tight as the authors of the review document purport it to be. You have done a very good job, but by over-selling your design you take away from the good features and make the entire design more controversial.

Specific Comments:

p. 6, 1 stcomplete para. – I have concerns about th e 600 follow-up interviews with nonrespondents. No goal was set for the response rate. These are people who have already rejected the mode of implementation of the survey or the specific survey for a variety of reasons. What is being provided as incentive to get these people to respond? If you meet the target of 600 completed interviews, but have a nonrespondent sample response rate of 10-20 percent, th en this information will be of limited use; you have just added an additional sample selection problem . In addition, if you find a difference in values you will not be able to identify whether you have a sample selection effect, mode effect or both effects causing the difference. While you are making the fo llow-up as similar as possible to the original survey implementation it is not identical.

p. 6, 3 rdpara. – Where are you going to obtain “historical response rates of individuals in that group for similar types of surveys”? You make the argument in this report, which I agree with, that no work of this type exists. This assertion nullifies you ability to make such an adjustment in the sample design.

p. 7, top – I think the objectives are good and appropriate for the policy issue you wish to value.

p. 10, bottom – It is good and necessary for welfare calcu lations to have an opt-out option in the choice questions, not all choice question designs include this design feature.

p. 11, 1 stpara. of Section 3.1 – I do not believe that the l ast sentence of this paragraph is true. A panel member could give access to another member of their household without the survey administrator’s knowledge. You have no more contro l here than you do with a mail survey; they recipient can allow another to respond.

p. 12, top – What if computer technology is the reas on people opted not to participate in the panel? This follow-up procedure will do nothing to bring this type of person into the study.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-19 p. 13, 1 stfull para. – In assessing the design it would have beenmuch more helpful if the text reported the issues/concerns identified and how they were specifica lly addressed in the survey design. This type of claim is not helpful and only raises questions in my mind.

p. 15, bottom of 1 stincomplete para. – What basis do you have for identifying “groups that tend to provide inconsistent or invalid answers” to attribute-based choice questions? I do not know of any information in the literature to support this sample design.

p. 15, Table 2 – How does this stratification by re gion compare to the percent of the national population living in each region?

p. 17, Table 3 – The 90 percent confidence intervals fo r a “typical binary variable” are meaningless in this context; what we care about is the error bounds on th e welfare estimates derived from conjoint responses.

p. 18, 1 stpara. – I disagree with the statement that the “use of such methods to identify protest responses is well-established in the literature.” Some studies do nothing to identify protest responses, others use questions as you are proposing and others use statistical procedures to identify data outliers. There is no established procedure and all are ad hoc. There is nothing wrong with doing what you propose it is just that the conceptual and empirical foundations are not as solid as you portray them. My experience suggests that a carefully designed protest protocol will result in many respondents providing contradictory evidence, a single person can provide information that indicates a valid response while another of their responses may indicate a protest; what do you do with this respondent.

p. 19, 2 ndpara. – I do not know of any articles in the literature where a Heckman selection model has been merged with mixed logit (p. 25, 2 ndpara.). This is an important estimation issue you will need to address as you will need to consider the correlated res ponses of individual respondents and the literature is strongly supporting heterogeneous preferences through random parameters. You can not run a Heckman on a simple specification and then address these issu es in a second model, which seems to be implicitly implied in the write up.

p. 21, bottom of 1 stincomplete para. – It is good to te ll respondents to consider each question independently, but this is not likely to be suffi cient (see Holmes and Boyle, Land Economics, Vol. 81, No. 1, 2005). This interaction w ith how respondents answer the sequential questions leaves questions about what statistical information should be used for policy purposes.

p. 21, bottom – The certainty questions of Champ et al. (2004) are not very useful. It will always be possible to map two welfare estimates through some type of integer or continuous preference scale. The question that remains is whether the specific point on the uncertainty scale is unique to the specific study or can be generalized to other types of applications. The Champ study is silent on this issue. There is no clear approach in the literature to “successfully cont rol for hypothetical bias.” This does not mean that you should not try, it just means that you may not make the right adjustment and should not make the claim that you have adjusted for this bias; your resulting value may overestimate or underestimate welfare.

p. 23, top – I suggest in considering data quality you consider some of the information that Berrens and colleagues have explored with internet surveys, e. g., time spent viewing slide show, time spent answering the valuation questions, etc. This information could be interacted with respondent characteristics that you are collecting, e.g., environmental group membership, education, etc.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-20 p. 25, 2 ndcomplete para. – This is the type of model you need to use and I expected it to be stated more explicitly earlier in the text.

p. 29, top – I am confused about how the “baseline population levels” are going to be implemented in the survey. I see on page 31 (bottom) the levels. Does this mean that the baseline will only be varied for the “Population Change” attribute? Is this part of the orthogonal design? I seem to be missing something from the explanation in the text.

p. 30, top – I suggest considering alternative-specific constants in the model. There are many variants in the published literature, but I have one specific recommendation. This recommendation is a binary variable that equals “1” if one of the two policy alte rnatives is chosen and “0” otherwise. This will give you an idea of whether people are “in the market” for reasons other than the attributes of your design that you explicitly control for in the estimation. There are a variety of reasons why the coefficient on this variable may be positive and significant and you cannot identify specific effects, but potential explanations include all of the r easons a person might be “incorrectly in the market”. You could try interacting some of your protest probe variables with this ASC.

p. 31, attribute levels – The dollar amounts seem trivially small. These are unlikely to impinge on most respondents’ budgets (p. 34 2 ndpara.), which makes me skeptical of discussions about how focus group respondents indicated that they are considering thei r budget constraints (p. 34, bottom). You really need the pilot to make sure the estimated coefficient on cost will be significant.

p. 32, 1 stpara. – It is good to exclude dominated alternatives.

p. 34, 1 stcomplete para. – The incentive compatibility you refer to for a referendum format applies to a single, binary choice, not the application you havewhere respondents choosing over two alternatives and do nothing and answering multiple, sequential choice questions. A single choice question with three answer options does allow for free riding and other strategic behavior, as does answering a sequence of choice questions.

p. 34, 3 rdpara. – Increasing the “consequentiality of choi ce” is important to the design, especially in regard to hypothetical choice, but may lead to protest responses as well.

p. 35, 1 stpara. – The statement that EPA finds little evidence to suggest that hypothetical bias should be significant in the proposed survey instrument” is na ive and silly. The empirical evidence is quite clear that stated-preference methods can provide useful po licy information, but that value estimates tend to be overstated relative to actual monetary transactions. In recent work that I have done with Taylor and Morrison, which is funded by EPA, we find that welf are estimates from a three alternative attribute-based experiment can be 20 to 40 percent higher that compar able cash values. More importantly, only 5 percent of respondents entered the market when cash was i nvolved and over 60 percent of respondents entered the hypothetical market. This is not an issue that you can design or assert away.

p.36, slide 52 – I think that you can still expect peopl e to think about more than fish. I think people realize that birds eat fish and there are other ecosyst em effects. The ASC that I suggested above is one way to partially control for these and othe r unwanted considerations by respondents.

p. 37, Section 6.6.3 – I do not think that you have a true external test of scope. If your scope test follows the regular choice questions, which comprise your inte rnal test, you have already “trained” respondents to consider the differences and this is not a true, external, between-subjects test. However, I am notPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-21 convinced that the external test of scope is necessary.

p. 40, slide 53 – I believe addressing uncertainty is important and too many stated-preference studies overlook the fact that environmental outcomes are not known with certainty. Having said this, I am concerned that your treatment of uncertainty may raise more problems than it helps. I understand the issue of cognitive overload, but other studies have in corporated specific information about uncertainty in the design, especially those valuing health outcomes. Your current design raises the issue without any focusing resolution so that each respondent is left to make their own assumptions regarding the likelihood of outcomes from the Phase III policy. Your “Champ style” question convolutes uncertainty about outcomes with uncertainty about responses. It is possible to have a scenario when outcomes are known with certainty and respondents are still uncertain about their response and vice versa.

p. 51, Informational Slide Show – I believe this is a very good mechanism for conveying the background information.

Cover page of slide show (slide 1) – You state that it is a survey of northeast residents, but it is far from clear whether respondents should just value the north east or all of the U.S. Consider the following points in the slide show.

• Slide 2 – You say “protect fish stocks in the Northeast US waters.” What if people take trips to fish in other regions of the U.S. care about what happens there? For example, when I have gone fishing on the Great Lakes I know that some of th e best fishing is by these facilities, but when I fish in the northeast I know that my fishing is not near any facilities.

• Slide 10, 11 & 12 – You talk about facilities, but not northeast facilities. Won’t respondents wonder about controlling facilities in other parts of the country? • Slide 13 – The focus returns to the northeast. • Slides 18 & 19 – There is no mention of the northeast on these slides where you talk about fish losses. • Slide 21 – You talk about government policies, but no region is mentioned. Most respondents will know that regulation is done at the state and national levels. • Slides 26 and 27 – The first bullet on slide 26 states: “Reduce fish losses in Northeast waters.” The following slide (#27) states: “EPA needs to know whether US households are willing to pay the cost ...” These two slides take you respondent both directions. Will northeast residents think that people from other regions of the country will also be paying for the northeast regional benefits?

Finally, the choice questions do no specify the region of the effects or who is voting. I think this inconsistency in information is bound to confuse some respondents.

Slides 24 & 25 – The information on these slides is conf using. You state that some facilities are using technologies that “reduce fish losses up to 98%.” You also state: “The proposed policies would require similar technologies at all facilities ...” Does that mean that facilities using the best technology might be required to use a less efficient technology?

Slide 40 – You do not tell respondents what will happen if they chose not to vote for either option. You have to assume that that they assume that current conditions will be maintained, but some may assume further declines given the information you presented earlier. You need to specifically tell respondents what you want them to assume about the baseline. This applies to the other baseline conditions for other attributes as well.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-22 Slide 60 – It is possible that there will be some tax c onsequences if some of the costs are written off. This could result in decreased tax revenues. I have f ound that a more believable payment vehicle allows prices and taxes to change.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-23 PEER REVIEWER COMMENTS

Timothy C. HaabPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-24 Peer Review of Willingness to Pay Survey Instrument for 316(B) Phase III Cooling Water Intake Structures

By Tim Haab

General Comments and Overall Impression of the Scientific Merit of the Document:

Thank you for the opportunity to review this document and survey. I found the document to be informative, well-written and easy to follow. Th e document does a good job of outlining the procedures to be used and for the most part follows acceptable procedures for the valuation of increased fish stocks from cooling water intake regulation. Particularly, the document provides thorough documentation of the planned sampling scheme and procedures for ensuri ng a valid representative sample from an internet based survey. Many of the fears decision makers face in regard to internet surveys should be allayed. Clearly the authors of the document—in conjuncti on with the experts at Knowledge Network—have spent considerable time designing a sampling scheme a nd estimation strategy for correcting any potential sampling bias created by the internet panel.

Keeping in mind my overall positive impression of the ju stification and survey, I do have some concerns that can be readily addressed. For the most part these are minor concerns that can be addressed quickly and should not cause a significant delay in the administration of the survey. My comments and concerns are detailed below in response to the charge questions provided by Versar. Perhaps my most significant concern deals with the analysis of the survey r esponses. It seems as though the authors have proposed a fairly basic method of analysis of the choice e xperiment response and have ignored some important econometric issues that should be addressed. Further, I have concerns about the aggregation of fish stocks into a single number in the survey responses. I will provide details below.

Again, I want to emphasize that overall I think this survey is well-conceived, meets standard academic practices and when administered will provide useful and accurate information to EPA.

Responses to Charge Questions:



### 1. Is the choice experiment survey an appropriate stated preference survey format for the estimation of marginal willingness to pay for the characterized policies? Will the survey likely provide useful insights into preferences, tradeoffs, and willingness to pay associated with the described policies? In short my answer is yes. The choice experiment is the obvious and most appropriate choice for the valuation of multi-attribute policy changes—such as 316(B). I can think of no other method that will allow for the valuation of multiple attributes in such a clean manner. As designed the survey collects:
• background information on perceptions towards policies in general • trade-offs among fish stock attributes • debriefing information to establish the validity of responses • demographic information



### Combined, all of this information will allow the authors to provide a detailed description and estimate of the values respondents place on fish stock increases dues to Phase III regulations.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-25 

### The use of the Knowledge Networks internet panel is accepted in the academic community and provides an accurate but less expensive alternative to in-person surveys. The reviewed documents do a nice job of describing the potential pitfalls with internet surveys and the appropriate solutions.

I have a few concerns about question ordering and wording in the survey that I will outline below.

2. Does the effort used in developing and testing th e proposed survey meet current standards for stated preference surveys reflected in the literature? 

### Yes, the use of 12 focus groups and cognitive interviews for survey development is appropriate and warranted. The 500 individual pilot survey will provide EPA with valuable insights into how the public is interpreting the survey. I woul d encourage the survey administrator to follow up with pilot survey subjects to get open ended feedback on the instrument itself. Such feedback will prove valuable in making sure subjects are interpreting questions appropriately.

3. Can the policy options be sufficiently well defined such that the survey is likely to provide useful insights into preferences, tradeoffs and willingness to pay? 

### I have a number of concerns here. First, the slide show and survey spend considerable time trying to convey that the numbers presented are ‘uncertain.’ I’m not sure this is successful and it is critical that it be. In particular, the choice experiment questions themselves seem to convey ‘certain’ information—“1360 billion fish saved”, “Increase to 70%”. There is no uncertainty conveyed. In the blow-ups to the slide-show some ambiguity arises in the wording: “Policies may increase stocks to between 60 and 75%” How is the respondent to interpret the actual survey question? Will it definitely increase to 70%? Will it increase to somewhere between 60% and 70%? Did your focus groups show any confusion over this? On a similar note, do respondents all have a common baseline for comparison? They are told that “100%=maximum historical fish levels”. Is that with or without human intervention? Does everyone have an accurate assessment of what that baseline will be, or are respondents comparing apples to oranges? Again, these are questions that were probably addressed in the focus groups but without more information it’s not clear what the respondent is valuing. Is it an absolute change or a relative change? Do subjects know what catch levels were in 1950? There seems to be a non-sequitur in the slide show. It wasn’t clear to me that the choice experiment was an example until after I read through everything once. Nowhere does it say, “Here is a sample of what one of these questions will look like.” Also, I have concerns that the sample questions may bias the answers to the actual choice questions. For the same reason you want to control for potential order effects in the repeated paired comparisons, you may need to control for the values reported in the sample problems. The respondent may be tempted to compare the actual choice policies to the sample policies. I’m concerned about aggregating all fish species into a single aggregate population change. The slide-show give subjects information on the species affected and the table gives detailed information on changes in fish species. Are subjects supposed to assume proportional increasesPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

1 

### Green, D., K. Jacowitz, D. Kahnemann, and D. McFadden (1998) “Referendum contingent valuation, anchoring a nd willingness to pay for public goods,” Resource and Energy Economics, 20: 85-116.

2/06 A-26 

### in stocks for all species? Or is it assumed that just the total matters? The National Marine Fisheries Service has detailed estimates of the value of various species to recreational anglers.Those numbers clearly show that anglers value species differently. Is it the same for nonanglers? I’m not sure you can answer this with the current survey.

4. Does the information provided in the survey on the baseline ecological conditions, the expected changes, and the expected costs and their inci dence seem sufficient, appropriate, unbiased and accurate? Does the survey provide appropriate and sufficient introductory and debriefing questions to enable identification of protest responses and symbolic responses, to test retention and understanding of information, etc.? Would y ou suggest any additional debriefing questions? 

### For the first part of the question, see the answer above. For the second part, I have concerns about the placement of the introductory questions in the survey. The first two questions are designed to assess general attitudes towards policy and the rankings will be severely compromised by the slide-show. The slide show provides 15 minutes of information on the negative environmental effects of power plant operations and then subjects are asked to rank environmental objectives among a group of other policies. It seems like this question should come before the slide show.

5. Does the survey sufficiently and appropriately char acterize the policy options under consideration? Does the survey place sufficient and appropriate em phasis on the scope of resource changes? Does the survey appropriately communicate the scientific un certainties as well as uncertainties associated with the policies in question? 

### See response to question 3.
6. What are your expectations regarding the poten tial validity and reliability of benefit estimates that might be provided by the proposed survey, compared to the current state of the art in the stated preference literature? I feel the proposed survey takes the necessary precautions to provide a reliable, conservative estimate of willingness to pay. While the authors argue agai nst cheap talk in the documentation, the survey implements a watered-down version cheap talk by st rongly encouraging respondents to take the valuation questions seriously. Further, by reminding subjects th at the survey results will be used to inform the policy decision, the responses are taken from the realm of purely inconsequential to possibly consequential. Many would argue that this is enough to meet the c onsequntiality criterion Carson, Groves and Machina and Green, Jacow itz, Kahnemann and McFadden 1 set out for incentive compatibility.

7. Does the survey take appropriate steps to mini mize hypothetical bias? Does the survey place sufficient and appropriate emphasis on the budget constraint facing respondents, and on potential substitute uses of their budget? Is the payment vehicle used in the su rvey credible? Does the survey incorporate appropriate language to ameliora te strategic bias, symbolic responses and “yea saying”? Is there any sign of systematic bias in the survey or presented information? If so, how canPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-27 it be ameliorated? 

### The survey design minimizes the potential for incentive incompatible responses. I would like to see a more detailed description of the mechanism by which the ‘cost of living’ increases. For example, “Such regulations will no doubt increase the cost of producing power to the firm. These costs will most likely be passed on to consumers in the form of higher prices for electricity and other goods that heavily rely on power to be produced. The budget constraint and substitute reminders are weak. They appear as three bullet points on one page. I would like to see some examples of the substitutes. The substitutes don’t have to be just environmental substitutes. “The proposed policy will result in higher costs to you. This is money that could be spent on other things like eating out, recreation, gas, etc.”

8. Will the answers to this survey lend themselves to straightforward economic interpretation of willingness to pay? Are the proposed econom etric modeling approach and validity tests appropriate? Would you suggest an alternative modeling approac h or alternative or additional validity tests? 

### Unless I have missed some details, the proposed econometric technique fails to capture the panel nature of your responses. Each subject will make up to three choices. These responses mustbe treated as correlated for a given respondent. Failing to do so will bias parameter estimates and estimated standard errors. Models for repeated choices are readily available in standard software packages like LIMDEP.

9. The national benefits analysis for Phase III of the 316( b) rulemakings must be completed within the next several months and a final rule issued by J une 1, 2006. Are there improvements to the survey and associated model that would be feasible to im plement within the available time frame? If more time was available, how would your answer change? The suggestions I have made in the above responses can all be implemented in a short time period. The overall concept and design of the survey is sound and with minor corrections will represent a solid piece of applied valuation.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-28 PEER REVIEWER COMMENTS

James R. KahnPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-29 

## Peer Review for Willingness to Pay survey Instrument for §316(B) Phase III Cooling Water Intake Structures

Report to Versar, Inc. (job #110709.4007.033)

by

James R. Kahn, Ph.D. John F. Hendon Professor of Economics Director, Environmental Studies Program Washington and Lee University and Collaborating Professor Universidade Federal do Amazonas (Brazil)

January 16, 2006Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-30 Introduction:

The U.S. Environmental Protection Agency's plan for implementation of a willingness to pay survey instrument to assess the benefits of §316( B) Phase III Cooling Water Intake Structures is a well-conceived plan that should produce results suffi cient to guide the decision-making process. After reading the document explaining the survey plan, I fi nd that the authors are quite knowledgeable of stated preference valuation measures, the controversies asso ciated with their use, and the best means of implementation. I am very pleased that the EPA decide d to use the choice experiment (conjoint analysis) stated preference approach, as I believe this appro ach to be less subject to biases and more conducive to the evaluation of alternative policy options in comparis on to contingent valuation. The justification of the approach, the background information slide show, and the survey itself are all first rate work. If the research plan is implemented as proposed, the EPA will be in a good position to make a decision relative to §316(B) Phase III Cooling Water Intake Structures and to defend the decision that is made. Although my overall assessment of the proposed survey and implem entation plan is quite favorable, I have several suggestions that the EPA may want to consider before implementing the plan.

This peer review will begin with a section-by-secti on overview of the peer review package, organized by section. This will be followed by responses to th e specific questions posed in the "Charge to Peer Reviewers." The final section of the review summari zes the review and highlights changes that could be made to fine tune the explanatory document, bac kground information slide show, and the review itself.

Previous Research (Section 1.2)

I concur with the authors that previous studies do not ex ist that contain an estimation of the social benefits of nationwide changes in populations of fish species that include species utilized in sport and commercial fisheries, as well as species that are not sought but are important forage fish. Although there are a plethora of valuation studies of fisheries including those atthe national level, these tend to focus on specific species such as salmon, or recreational species in gene ral. Additionally, these studies generally target the use values of these species and do not measure values su ch as altruistic value or existence values. The authors are correct in there assertion that general studies of the value of water quality improvements as a whole measure of different suite of services than the proposed §316(B) decision will affect. An implication of the authors findings which should be exp licitly stated in this section is that it is simply not possible to evaluate the benefits of the§316(B) Phase III program with a benefits transfer study.

Consultations (section 1.3)

The consultations, focus groups and pilot survey di scussed in this section are consistent with the standards of the literature. This reviewer has no further suggestions in this regard.

Survey Objectives (section 2.1)

The survey objectives are clearly stated and c onsistent with developing the knowledge necessary to inform decision-making with regard to the §316(B) Phase III program. No modifications need to be made to this section.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-31 Key Variables (section 2.2)

The authors do an excellent job in terms of tying key policy variables to the structure of the choice experiment framework (conjoint analysis). The literatu re cited is appropriate (Adamowicz, et al, Louviere et al, Bennett and Blamey and Batement).

Survey Support (section 2.3)

This reviewer is very familiar with Abt Associat es, Inc., and has great respect for their work. This reviewer is not familiar with the work of KnowledgeNetworks, Inc., but presumes that the authors have chosen a reputable sampling firm.

Collection Method (section 3.1)

The proposed method of an internet-based survey is the best method that can be implemented given the time constraints that EPA faces. The internet-base su rvey has the advantage of being able to supply information in a fashion that is more difficult inface-to-face or mail surveys, as well as avoiding the biased associated with the use of a telephone survey. I have some concerns that the use of the internetbased survey can lead to some bi ases that can not be corrected through advanced statistical methods. This will be further elaborated in the discussion of biases. Nonetheless, the proposed method is undoubtedly the best direction to go for the type of inform ation that is needed to support this decision.

Survey Response and Follow-up (section 3.2):

I am somewhat concerned about the length of time that is required to complete this survey. Between the review of the background materials and the actual completion of the survey, EPA estimates that 40 minutes are required. Although this length of engagement is consistent w ith conjoint analysis experiments in the test-marketing of new market goods, these marketing research firms pay the respondents substantially more than $5 to $10 do llars. I have had previous discussi ons with marketing professors who suggest that $30 to $50 dollars is mo re appropriate. The longer the time of the survey, the greater must be the payment to guard against the onset of res pondent fatigue and meaningless responses. Although the survey contains some follow-up questions to test th e validity of responses, it might make sense to increase the payment to users as much as the research budget will permit, up to about $30.

Pretests and Pilot Tests (section 3.3):

The pretest focus group and proposed pilot study w ith 500 responses seem to be appropriate. The second paragraph of page 13 indicates that "Results from the final two focus groups (one including cognitive interviews) provided convincing evidence that the majo rity of respondents answer the stated preference survey in ways appropriate for stated preference WTP estimates, and that their responses do not reflect biases noted above." The October 18 memo/report fro m Abt Associates that documents the focus group research provides evidence that supports this conclusion.

Target Population (section 4.1.1):

This is a standard approach

.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-32 Sampling Frame (section 4.1.2):

I have several concerns about this section. Th e document does not make clear whether the telephone numbers in the sample are land-lines or both land-lines and cell phones. If cell phones are not included, there is a serious sampling problem because young pe ople (18 to 25) generally do not have land-lines unless they are living in the parents' home. (I do not have statistics on this, just my general observations of my children, their friends and all the university students with whom I have contact). I also believe that there are biases introduced by who answers the phone in the household. If the survey is implemented with the person who answers the phone, then lazy husbands will tend to be underrepresented in the sample. On the other hand, the phone could be given to the pe rson in the household who most cares about fishing. This would introduce a different type of bias in the sample. This section does not state if there is a procedure for randomly choosing a person within the household.

Sample Size (section 4.1.3):

The authors use standard practice.

Stratification Variables (4.1.4):

Knowledge Network is capable of creating a st ratified sample, which adequately corresponds to representation of different socio-economic groups as re presented in the U.S. Census. My concern here is since people are being asked to engage in an onlin e survey, the group of people who are computer literate (and have online access) may give different responses th an the group of people who decline to participate because they are made uncomfortable by information technology. There two groups may have radically different WTP for environmental improvement, and th ere is no way to correct this potential bias through over-representing a particular demographic or socio- economic segment of the population in the sample. The document briefly touches on this type of issu e when talking about the 600 individual non-response survey, but does not say how they intend to be guide d by the information that is developed from the nonresponse survey.

Sampling Method (section 4.1.5)

This is a fairly standard approach but does not alleviate my concerns with section 4.1.4.

Response rate (4.2.1):

The response rate seems appropriate, but my origin al concerns about the potential for biases associated with information literacy remain.

Precision Targets (4.2.2):

The level of the precision targets is fine, but the document provides not a single clue as to how they were computed, or chosen based on sta ndard references in the literature.

Non-Sampling Errors (4.3)

The discussion of the potential problems associated withprotest bids, and the questions included in the survey to identify potential problems with protest bids are adequate.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-33 The section also discusses potential problems with a nd solutions to noresponse errors and unobserved heterogeneity. The actual explanation of the solutions is left to sections 4.3.1, 4.3.2, and 4.3.3.

Stratified Survey Sample (section 4.3.1):

This section looks at the problem of nonresponse bias. If a particular groups of the population are more prone to decline to participate, then over-representi ng this group in the initial sampling effort can correct this problem, and help to generate a complete d sample population with the same demographic and socioeconomic characteristics as the population as a whol e. In theory, this is correct, but the document does not specify a procedure to identify the degree of over-targeting the various demographic and socioeconomic groups.

Nonresponse follow-up interviews (section 4.3.2):

This section is also inadequate in its explanation of the procedures to be used to incorporate results from the non-response survey into the final analysis.

Statistical Correction for Unobserved Heterogeneity (section 4.3.3):

There seems to be an inconsistency between the discussion in this section in the second and third paragraphs of page 19 and the econometric specification of the desired model at the top of page 30. I do not see a Heckman correction coefficient in this mode l. This lack of consistency leads to doubts about how thoroughly the authors have thought through the correction for unobserved heterogeneity. This must be addressed.

Questionnaire Design (section 5):

The discussion of the survey is gr ouped by the section of the survey.

Section 1: Concern for policy issues— These questions are right on the mark and accomplishe d the intended goal of framing the policy context.

Section 2: Voting for regulations to prevent fish losses—

These represent a good set of instructions. Again, the report/memo of October 18 from Abt Associates provides documentation that these instructions are well-understood by the sample population.

Questions 4-6---

These are well-defined choice sets constructed in a fash ion that is consistent with the literature. The attribute levels are well-chosen, and provide suffici ent variation to reveal WTP and to conduct basic scope tests.

Questions 7-9

Good debriefing questions.

Question 10- Needs an additional response category as follows:Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-34 ___I thought there were impacts on ecosystems in additi on to those created by changes in fish populations and partially based my answers on these impacts.

The rest of the survey consists of standard ques tions, some focusing on debriefing issues, some seeking to measure intensity of preferences and some looki ng to define the demographic and socioeconomic attributes of the respondent. There are no problems here.

Analysis of Stated Preference Data (Section 6.1):

This discussion is pretty standard and accepted within the environmental economics community.

Second paragraph-page 23-The random utility m odel pre-dates both Hanemann and McConnell and should be attributed to McFadden (1974).

Advantages of Choice Experiments (Section 6.2):

Again, this is standard and consistent with the literature.

Comment of Survey Preparation and Pretesting (Section 6.3):

The EPA has done a good job in pretesting of the survey and in survey preparation.

Econometric Specification (section 6.4)

As mentioned above, the Heckman correction method th at was referenced section 4.3.3 is not mentioned in this section. This is a major problem and must beaddressed so that it is clearly understood exactly what econometric specification is being estimated.

Experimental Design (section 6.5):

This is a well done section, following the standardliterature. EPA has done a good job of eliminating pairs of choice sets which could lead to undesirable reactions by the respondents, such as a choice set which completely dominates another in all attributes, or nonsensical choice sets (high levels of saved fish but low levels of fish population). Attachment 4, wh ich further explains the experimental design appears to be on the mark.

Information Provision (section 6.6)

Any time that a valuation experiment is done, the r esearchers must provide information to the respondents concerning the implications of the changes they ar e being asked to assess. However, the researcher can be in a difficult situation because too much informati on or the wrong kind of information can dramatically influence a person's preference for one scenario over a nother. The researcher must try to create a balance between providing the information allowing for the measurement of the respondent's preferences, without creating a new set of preferences for the respondents. For example, showing pictures of cute baby seals in an oil spill scenario could unduly influence the results.

My impression is that in this case, EPA has done a good job in presenting important information in a "value neutral" fashion. I do not think they have cr ossed the line between being "estimators" of value and "creators" of value.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-35 Amelioration of Hypothetical Bias (section 6.6.1):

Note the use of a double negative, "not unavoi dable," in the last paragraph on page 33.

My personal opinion is that the concern about hypothetical bias is a result of a combination of some sloppy work by researchers in the early years of CVM, and a bias on the part of some economists against giving credence to values that do not originate in r eal world markets. If survey questions are designed appropriately, hypothetical bias need not be a problem. Conjoint analysis is a technique that is less prone to hypothetical bias than CVM, and the EPA has done a good job with its focus groups and survey design. I strongly agree with the conclusion in the second ha lf of the second paragraph on page 35 that "cheap talk" is neither necessary nor desirable in this survey . I think the decision framework of conjoint analysis obviates the need for cheap talk.

Amelioration of Symbolic and Warm-Glow Effects (section 6.6.2):

I agree with the EPA that they have appropriately eliminated warm glows, biases towards big businesses, and so on. My chief concern in this section is whether "slide 52" is believable. Many people are aware of the impacts of thermal pollution on estuaries and othe r water bodies, and they could believe that one of the technology options (re-circulating cooling water) c ould have less of an impact on water temperatures. Also, might people believe that reducing impingement of fish could also reduce impingement of reptiles, amphibians and birds? This may have been addressed in the work with survey groups, but a discussion is not contained in the document.

Assessing Scope Sensitivity (section 6.6.3):

The EPA adequately address the issue of Scope Sensitivity

Communicating Uncertainty to Respondents (section 6.6.4):

On the whole, this is well done, but I have real probl ems with slide 30 in the third paragraph of page 40. The second sentence of this slide (Common sense dict ates....) really dismisses any scientific evidence about these relationships. The scientif ic knowledge should not be disreg arded in an appeal to common sense. The question should talk more specifically a bout how scientists can pin the outcomes to a range with a degree of confidence, and by the way, if y ou think about it this way, preventing the loss of young fish and fish eggs will certainly increase fish populations in the future.

I agree with the conclusion in the last paragraph on page 41.

ANSWERS TO QUESTIONS CONTAINED IN THE "CHARGE TO REVIEWERS"

(1) Not only is the choice experiment survey an appropriate stated preference survey format for the estimation of marginal willingness to pay for the charact erized policies, it is the best stated preference approach for this purpose. I believe the survey to be well-constructed and it w ill provide useful insights into preferences, trade-offs and the willingness to pay associated with the described policies. Based on everything I have read in the document, I am confid ent that the U.S. EPA will have the best possible information on which to base these decisions.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-36 (2) Yes, the focus groups and pre-testing more than meets current standards that have been discussed in the literature.

(3) Yes, the survey does this in qu ite a nice fashion. Since the policies have varying levels of cost and impacts on fish, they naturally fit into choice set scenarios.

(4) In general, I think the survey provided appropriate information on baseline ecological conditions. The introductory and debriefing questions also seemed appropriate to identify protest responses and symbolic responses. However, as I noted in my discussion above of 6.6.2, people might believe that reducing impingement of fish would also reduce impingement of birds, reptiles and amphibians, and that some technologies might be associated with less thermal pollution than other technologies. The potential existence of this problem could be tested by adding an additional response to question ten, that would be something like this:

___I though there were impacts on ecosystems in additi on to those created by changes in fish populations and partially based my answers on these impacts.

(5) The survey adequately characterizes the polic y options under consideration and places appropriate emphasis on the scope of resource change. I think thatfor the most part, the survey adequately addresses scientific uncertainty, although as noted above, I did not like the choice of words ("common sense") in slide 30. I thought the use of these words cheapen ed the quality of the scientific information.

(6) My expectations are that the validity and re liability of the benefit estimates are very strong in comparison to the current state of the stated preferen ce valuation literature. I would suspect that after the dust settles from the rule-making process, the authors w ill be able to publish this study in a top journal, such as the Journal of Environmental Economics and Management, Ecological Economics, or Environment and Resource Economics.

(7) I don't believe that hypothetical bias is a real problem with correctly structured conjoint analysis studies, and I believe that this study is correctly structured. Respondents know they will be making a real world decision and that it will have real world conse quences in terms of fish populations and the cost of electricity and other products. There is not a need to focus on "cheap talk" or other mechanisms for alleviating hypothetical bias, because they are not asked about a hypothetical donation. They are asked about potential impacts on the day to day cost of living. Since the questions are structured with both costs and benefits, it is very difficult for a respondent to behave strategically. I see no potential for systematic bias, outside of what I noted in my discussion of section 3.2 (respondent fati gue) and section 4.1 (oversampling of the information literate in a way that is independent of typical demographic or socioeconomic classifications). The respondent fatigue issu e can be addressed by increasing the participation payment. The over-sampling of the information literate population is extremely difficult to address, and since appropriate discussion of the non-response surv eys are not contained in this document, I cannot determine the significance of this problem.

I also note potential problems associated with w ho answers the telephone and who is handed the telephone. The document does not specify a protocol for randomly choosing the household member who participates in the survey. In addition, there ar e potential biases that will develop if cell phone numbers are not included in the survey sample.

(8) The answers to this survey will lend themselves to straightforward interpretation in the decisionmaking context. The validity tests seem appropriate, but as noted above, I am a bit confused by thePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-37 econometric modeling. When talking about heterogeneity in section 4.3.3, the authors refer to using the Heckman correction method. However, this does not appear in the econometric specification listed on page 30. This inconsistency leads the reader to wonder exactly what econometric model is being employed.

(9) There are a few sections where the discussion n eeds to be beefed-up to make the reader more confident in the analysis. This includes the discu ssion of the stratified sample, nonresponse biases and the econometric specification. The primary change to be ma de in the survey would be the incorporation of an additional response to question 10, as mentioned in my response to question 4. Finally, since the survey is so lengthy, it is important to pay the respondents a higher payment, or run the risk of the respondents treating the later stages of the survey with less than their full attention.

Conclusions:

The proposed survey looks like it will accomplish th e desired purposes, and is extremely well done. However, some small changes can make it better. In addition, there are several sections, particularly the sections on biases and econometric specification that n eed more exposition in order to make the final results of the survey believable. Moreover, the refe rence list is not very impressive and does not provide an adequate survey of existing applied studies us ing this technique. This unimpressive list gives an impression that the researchers have not done all of their homework. For example, I did a Google Scholar search on the keywords "choice experiments environmen tal valuation" and 25 of the first 40 hits were directly relevant. A partial list of additional references is supplied.

In summary, with a few modifications, the surv ey and supporting documentation will provide the appropriate type of information withthe appropriate level of confidence in the empirical results. I am very pleased that EPA has taken this route, instead of rely ing on benefits transfer or a contingent valuation
survey.

Additional References: Boxall, P.C. W.l. Adamawicz, M. Williams and J. Louviere, 1996: A comparison of stated preference models for environmental valu ation Ecological Economics 19, 243.

Casey, James, 2006, Modeling of farmer preferences for agroforestry systems, in the Handbook of Contingent Valuation, (Alberini a nd Kahn, eds.) Edward Elgar, 340-353

Carlsoonn, Fredrik, P Frykom and C. Lieejstope ( 2003) Valuing Wetland Attributes : An Application of Choice Experiments, Ecological Economcsi 47, 95-103.

Farber, Stephen and B. Griner, 2000, Valuing waters hed quality improvements using conjoint analysis, Ecological Economics, 34.

Hanley, Wright and Adamawicz, 1998 Using Choice expe riments to value the environment, Environ and Resource Economics, 11: 413

Holmes, Thomas R and W.L. Adamowicz (2003) Attribute based methods: A Primer on Non-Market Valuation, Champ, Brown and Boyle (eds) Dordrech t, the Netherlands, Kluwer Academic Publishers.

Kahn, James and Steven Stewart, 2006, An introducti on to choice modeling for non-market valuation, inPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-38 A Handbook of Contingent Valuation, Kahn and Alberini, eds, Edward Elgar Publishers.

McFadden, 1974 Conditional Logit Analysis of Qua litative Choice Behavior in P Sarembeka (Ed), Frontiers in Econometrics, New York: Academic Press.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-39 PEER REVIEWER COMMENTS

Alan J. KrupnickPeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-40 Comments on Peer Review Package for WTP Survey Instrument for 316(B) Phase III Cooling Water Intake Structures Work Assignment 3-33 Versar Job Number: 110709.4077.033 By Alan J. Krupnick

The charge asks reviewers to organize comments into general comments, responses to the charge questions, specific comments about the report, and a dditional references. As the bulk of my comments are about the survey itself and the charge questions are largely overlapping, I answer the charge questions very briefly below and then provide detailed comments organized by problems with survey design. This is followed by a few comments on the report (I have to say though, that the report is not very important relative to the survey itself.)

General Comments

I commend the survey team for their tremendous amount of hard work and boldness in attempting to capture preferences for a pretty ephemeral commodity – fish kills that have no ecological consequences. I also commend EPA for providing what must be a huge survey budget for 5,000 completes. While I actually think this is excessive for what is planned, it is heartening to see the agency make a commitment to such a major study. At the same time, in a study of this complexity there are inevitable tradeoffs that have to made, say between consequentiality and bias, where the literature is insufficient as a guide. There are many such choices here and, given my experience, I think I would have made some different choices, or at least altered the design to develop a strategy of protecting the survey against the possibility that some of the choices were bad ones, primarily through developing additional treatments.

Change Questions

1a. The choice experiment (CE) approach is not necessarily the best way of obtaining WTP estimates. It is more complex than the CV alternative. The mo st compelling justification for the former over the latter is if the attributes affected by entrainment need to be unbundled, i.e., if separate shadow prices (marginal WTP) need to be estimated for each of them, presumably so they can be used in benefit transfer exercises to apply in locations where the improvements in these attributes are different. A case can be made here to avoid false precision over these effects that may come from a CE and treat them as bundled in a CV, using several different treatments to capture a relevant range of attribute differences.

1b. Whether the survey produces useful values depends on all its features ex ante as well as how well the analysis turns out. On balance, I think there is some reason to be concerned about its features, as detailed below, and one can’t know until the survey r esults have been analyzed if they will withstand scrutiny.

2. The effort with 12 focus groups seems reasonable to me, certainly not excessive for a “commodity” as complex as this one. In our survey on the WTP to improve the Adirondack ecosystem, we used over 30 focus groups. This probably was excessive, in retrospect.

3. This question is really about whether the leve ls of the attributes are appropriate. I have no information to make such a judgment.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-41 4. a. The information on baseline ecological conditions is lacking, as the materials readily indicate. I think it will be hard to believe the results for two r easons that come up here: one, an effort is made to eliminate in people’s minds the links with fish kills to ecological damage and, two, there is no baseline for total fish stocks. Concerning the fo rmer, in my experience, respondents will default to believing there is a link and just telling them ther e is not is not enough. At a minimum, add a debriefing question to see if they bought this restri ction. Concerning the latter reason, having the baseline defined by I&E losses seems wrong to me. The baseline should be fish populations. If these are not known I would recommend making them up, treating this as a parameter and trying alternative information treatments varying this base line to see if responses are sensitive to it. b. I think the survey goes overboard on consequen tiality giving too little attention to its flip side – bias and warm glow. The questions to test for th is are very complicated and one can never know if these perceptions are being fully eliminated.

5. Mostly redundant to above questi ons. The new part is about uncertainty. In my work I find that when a survey admits that effect s are uncertain, respondents tend to reject the survey (and therefore, biasing estimates downwards, unless such people are eliminated from the sample) and think the survey makers don’t know what they are talking about. The paper discounts this effect; but we see this effect very strongly in our work.

6. I think getting credible marginal WTP values by attributes is unlikely because the attributes aren’t independent (see more on this below).

7. Some of these questions are also redundant to above . A new issue is: Payment vehicle: I think this payment vehicle is highly problematic. I don’t accept that respondents will accept that product prices will be higher and hence their costs will go up. I recognize that the paper disputes this from focus groups. Respondents in my experience will blame business and believe that they shouldn’t have to pay. They may dispute that prices will go up, or may think prices will go up by more than they should (rather than by less than the cost increase) because of monopoly business practices. Or respondents may believe that they can simply substitute away from commodities whose prices rise (admittedly, substituting from electricity wouldn’t beeasy!). At the very least, debriefing questions would be needed to tease out whether the payment vehicle caused problems for respondents.

8. I think that the sample size is larger than needed for the survey as currently designed. I suggest that the experimental design be made more differentiate d, with different information treatments developed and tested to examine the effect of certainmajor choices in survey design on the results.

9. I don’t know if the above idea (in 8.) is feasible within the time frame. Some ideas below are obviously easy to implement.

Detail on major problems.

1. There is no baseline or the baseline is poorly defi ned. There are three ecological attributes in the valuation exercise: numbers of fish saved, eff ect on long-term fish populations and effect on recreational and commercial catch. The first h as no baseline, except relative to the total fish currently “lost” in cooling water intakes. Whilesome individuals might care about fish death per se, I would imagine this is not a large fraction of the population. Most would want to know if this amount of death is large relative to the overall populations of various fish species. They might want to know this to gauge whether fish populations are at risk, which is really dealt with by the second attribute or just about having a number to scale the deaths to. If it is the former, then thePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-42 number of fish saved attribute is redundant and s hould be dropped. If it is the latter, then some number is needed. In this case, my advice is to make up a few different baselines and then use them in several different treatments to test whether the baseline that was made up makes any difference to WTP. I suspect it will, but in that way can be corrected or adjusted for. If something like this isn’t done, I wouldn’t know how to interpret any results from the survey.

The second attribute’s baseline is poorly defined. The scale gives an endpoint of 100%, while some screens fail to show what this 100% refers to. I recognize that the text may define this, but I think it needs to be on the screen. The text defi nition refers to the maximum historic levels of fish. This is poorly defined in my view. The maximum may not be the optimum or even be sustainable. I think that some sentence like: Scientists and ecologists have determined that appropriate fish populations cons istent with the health of the ecosystems affected. The 100% refers to population levels consis tent with that determination.

Finally, the third attribute baseline is also poorly defined. Why is the catch in 1950 significant? We are not told on the screen. Again, if this is interpreted as peak catch then this may not be optimal at all. Indeed, some respondents might interpret an increase in catch as a bad thing, thereby screwing up the scope tests.

I am also troubled by what we term “expansiv e priors” about the baseline. The documentation says that (at least one) respondents may think that the fish populations are declining. Their response to a declining baseline situation will certain ly be different than that if they think the baseline is static or increasing.

Overall, much more care is needed to define b aselines and test for baseline effects when different (or made up) definitions are used.

2. Definitions of the commodities . One of my biggest concerns is that the commodity is defined in terms of fish, not as an indicator of ecosystem health, but as the fish themselves. Respondents may well substitute their own views on ecosystem linkages with fish kills and you won’t know what they are thinking.

And not just fish, but commercial fish (in part ) which respondents should have no logical reason to be willing to pay for. Respondents may ask themselves whether prices of fish will come down with the options being considered, thereby canceli ng out some or all of their increased costs from the option itself. So there are really two issues: whether respondents are willing to ignore ecosystem health concerns and what the role play ed by commercial fishing is in their thinking. On the latter I would stipulate that scientists have determined that there will be no effect on commercial fish availability or prices because suchfish come from all over the world, and drop any further mention of it. I would leave in recreational fishing as there may be option and use value associated with it and it is placed-based. Because recreational fishing is placed-based, the text would need to clarify how patchy the e ffects would be across substitutable recreational fishing sites.

A broader issue is that the three attributes y ou list are not independent, but are related to one another in complex ways. Probably, fewer kills will result in larger long-term fish populations and greater recreation and commercial catch. Bu t, your orthogonal design varies attribute levels for these attributes independently (as it should), wh ich means that fish kills could be going down a little while long term populations could be going up by a lot – and that doesn’t really makePeer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-43 sense from any sort of simple story. Indeed, increases in catch could have effects on long-term fish populations, irrespective of fish kills fro m entrainment, so the interdependencies are everywhere. The documentation says that some inadmissible combinations of levels were dropped from the design space, so maybe this problem has been at least partly addressed. What is not addressed, though, is that if interdependencies exist in people’s minds then the marginal WTP estimates attached to each attribute will not be reliable.

Another issue with the commodity is aggregation over species. The survey lumps all types of fish together and talks about average effects over all sp ecies. I would want to be sure that this does not create problems for the respondent in interpretation.

Another issue is uncertainty in the commodity . Our research shows that admitting uncertainty results in a downward bias and increased variance. People think that if you are uncertain about effects either the effect can’t be very important or large or that you don’t know what you’re talking about so why should they take the surv ey seriously. Both tend to push WTP down.

The authors want to have it both ways. They want to invoke uncertainty, so they can claim they took it into account, and then ignore it in the choice questions because respondents thought that presenting intervals of improvements was too confus ing. I don't think they can have it both ways without making changes in the treatment of uncertainty.

As noted elsewhere, my suggestion is to vary inseparate treatments how uncertainty is handled, both about the baseline and about the changes.

3. Overall experimental design . With such a large sample size being planned I am surprised that there are no plans for treatments, e.g., on information, uncertainty, baselines, etc. At the same time, the large sample is being used to get a better understanding of how WTP varies regionally. Why is such regional precision required? Fu rther, why is the commodity being limited regionally? In our work, the payment vehicle definition required a limit to the state, but no such limit is needed here as there are no taxes involve d. Indeed, one could argue that a regional limitation is unrealistic. Market effects will bethe result of cumulative changes across the country (to a large degree), so by not discussi ng them the survey may seem less credible and WTP may be biased.

4. Consequentiality versus neutrality . My view may be controversial, but I think the survey errs too far towards consequentiality (and to far away from neutrality (the absence of bias)). Every survey has to make choices about whether or not to state the sponsor of the survey, its real purpose, and the implications of voting. By being truthful, one may easily create bias. To me, this survey faces the same problem we faced in wh ether to tell respondents that they are voting on a question about air pollution controls (with costs passed on to them) to improve conditions in the Adirondacks versus trying to find an biased way of getting at preferences for cleanup and in so doing having to be untruthful and perhaps lose so me consequentiality. If people know that the USEPA is doing this survey, they will assume that voting for the options is encouraged. Saying that EPA will use these results to assess values be gs the public to vote on a warm glow basis rather than their true preferences.

5. Payment Vehicle . The payment vehicle begs respondents to wonder why they should have to pay when business is causing the problem. The notion that increased costs result in higher prices is not one understood by most people, particularly for costs incurred by powerful and perceived,Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-44 high profit industries, such as electric power. They may think that industry will make them pay, but they don’t think they should have to pay a nd may therefore vote against such a plan rather than reveal their true preferences for the fish.

6. Extent of the Market . I don’t understand why the survey is geographically limited in the way it is. Since the payment vehicle is a change in market prices, these changes can be realized by anyone in an affected market. For some products, this is the entire nation; for others, perhaps effects would be more regional. For power plants, regiona l effects are conceivable. But these effects for the Northeast, would surely include Pennsylvani a (which is excluded now) because this state is part of a regional power grid that includes states labeled as northeastern states. From the benefits side, the extent of the market is even more cloudy . Effects of entrainment are likely to affect an entire watershed, so that is the extent of the market. Again, Pennsylvania and Maryland share a common watershed, given the Susquehanna that runs through both states; yet PA is excluded and MD is included in the definition of the Northeast region.

7. More Minor Comments: SLIDE SHOW

a. There are too many graphics and photos, any of which could introduce unknown biases. The researchers appear to have designed the information for maximum interest, which is good, but at the expense of a dry neutrality, which is necessary b. “Protect fish stocks.” This seems like a biased statement. c. “part of a larger ecological system.” So whic h is it? Are the fish an indicator of ecological effects or is the issue just fish? One informati on slide gives one picture, while another gives the opposite impression. d. “Many fish stocks have been declining.” This appears to establish a declining baseline, but this idea is not carried forward to the choice questions or later in the text. e. “About 1/3 are types...” So does the 1/3 refer to species or to numbers of fish? Not clear. f. Table. The table is very complicated and s hould not be included like this in the information
treatment. g. Pie charts are not very helpful when nearly all the chart is one color. h. “reduce fish losses by about 25%.” Doesn’t this set up an expectation about the probable size of fish loss reductions? If so, when the choice qu estions vary this amount, how will respondents interpret the difference? i. “Technologies...already in use at 52% of facilities.” This phrase is problematic. If we already have such experience, why is there uncertainty ? And why is the government regulating again? What is the current regulatory environment? What isn’t working now? j. “Some costs would unavoidably be passed on.” Some, but not all. So would shareholders benefit? What is the respondent supposed to thi nk? Passing on costs is not “unavoidable,” if the markets, for instance, are international and comp etitive, the manufacturer would have to bear the
costs. k. “EPA needs to know....” This is an odd phrase. We need to know how strongly people feel about reducing fish losses, but people are not being asked to do a benefit-cost analysis. l. “Common sense...” I would say no. Common sense could refer to sustainable populations that are considerably below current levels and preventi ng loss of adult fish here could merely lead to equal numbers of death from some other direction. m. Status Quo. Best practice, according to Adamow icz, is to portray the baseline as an additional column in the diagram, rather than just checking off a status quo response. To do so requires stating the ecological baseline, something that the survey team has not wanted to do so far.Peer Review for “Willingness to Pay Survey Instrument for §316(B) Phase III Cooling Water Intake Structures”

2/06 A-45 n. “Significant impacts.” One slide says that th ere “are no other significant impacts,” implying that the fish kills are a significant impact. This may introduce bias.

Text Comments

Pg. 6. I don’t see the need for oversampling. Rather sampling weights can later be used to correct for differences between the population and the sample. I am not sure that doing this up front in sample selection is worth it. Maybe it will introduce unknown bi ases relative to a more neutral attempt to enlist respondents.

Pg. 8. 2 ndParagraph It is one thing to assert that the values from this study will only apply to fish, rather than ecosystems; another to make people believe th at. In my experience, people know very well about food webs and won’t surrender this idea easily. From the report, I am not sure that the idea should be surrendered. Is it true that there are not ecosystem level effects of entrainment?

Pg. 10 (second bullet). I am not clear from reading the report exactly how uncertainties are going to be treated in the survey.

Pg. 12, 3 rdParagraph A 40-minute average survey is l onger than usual for such surveys (30 minutes is usually the target)

Pg. 14. 2 ndParagraph If the precision achieved is “more than adequate” than one should do what is adequate and spend the rest of the money doing a dditional treatments to shore up unavoidable issues with the survey.

Pg. 19-20. Heckman’s approach is referenced but not explained.

Pg. 27 2 ndParagraph Here is the justificati on for the CE. It is not compelling.

Pg. 32 1 stParagraph The survey team recognizes the possib ility that some pairs will not make sense. But this raises the issue of interd ependencies across the attributes.

Pg. 33 2 ndParagraph In spite of the focus group sessions , there is no way EPA can “ensure” that the graphics don’t lead to bias. The use of color and pretty pictures that are routine in this survey is a bad idea, in my view.

Pg. 36 1 stParagraph In my experience, symbolic concerns are always an issue.