COMMENTS SUMMARY REPORT

PEER REVIEW PACKAGE FOR “WILLINGNESS TO PAY SURVEY INSTRUMENT FOR

§316(B) PHASE III COOLING WATER INTAKE STRUCTURES”

Prepared for:

U.S. Environmental Protection Agency

Office of Water

Office of Science and Technology

Health and Ecological Criteria Division

1301 Constitution Ave., N.W.

Washington, D.C. 20004

Prepared by:

Versar, Inc.

6850 Versar Center

Springfield, Virginia 22151

February 2006
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

i 2/06

TABLE OF CONTENTS

I. INTRODUCTION ............................................................ 1

II. CHARGE TO THE PEER REVIEWERS ........................................... 3

III. GENERAL COMMENTS ...................................................... 5

IV. RESPONSE TO CHARGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

V. SPECIFIC COMMENTS ON REPORT ........................................... 25

VI. SPECIFIC COMMENTS ON INFORMATIONAL SLIDE SHOW ..................... 39

VII. COMMENTS ON FULL TEXT OF STATED PREFERENCE SURVEY ................ 46

VIII. ADDITIONAL REFERENCES ................................................. 49

APPENDIX A: REVIEWER COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Richard C. Bishop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2

Kevin J. Boyle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-16

Timothy C. Haab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-24

James R. Kahn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-28

Alan J. Krupnick ........................................................... A-38
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

1 2/06

I. INTRODUCTION

The U.S. Environmental Protection Agency is developing regulations under section 316(b) of the Clean

Water Act (33 U.S.C. 1251 et seq). Section 316(b) requires that the location, design, construction and

capacity of cooling water intake structures reflect the best technology available for minimizing adverse

environmental impact. The EPA estimates that more than 1,500 industrial facilities use large volumes of

water from lakes, rivers, estuaries or oceans, or other waters of the U.S. for cooling, including steam

electric power plants, pulp and paper makers, chemical manufacturers, petroleum refiners, manufacturers

of primary metals like iron, steel and aluminum, and manufacturers of other products. 

Cooling water intake structures pull aquatic organisms including fish, shellfish, and their eggs and young,

into cooling systems (entrainment). There, the organisms may be killed or injured by heat, physical

stress, or by chemicals used to clean the cooling system. Larger organisms may be killed or injured when

trapped against screens or other structures at the front of an intake structure (impingement). EPA

estimates that 533,378,000 (age-one equivalent) aquatic organisms are impinged and entrained annually at

Phase III facilities. The impacts, if any, of these losses on population levels and ecosystem functions are

not known.

Under a court-ordered schedule, EPA is developing section 316(b) regulations in three phases. In

December 2001, EPA published final regulations addressing all new power plants and facilities that

withdraw more than two million gallons of water daily from waters of the U.S. (Phase I). The Phase II

rule for implementing section 316(b) was promulgated in February 2004 and applies to most existing

power-producing facilities that utilize a cooling water intake structure and withdraw 50 million gallons

per day from waters of the U.S. The Phase III proposed regulation was published for comment in

November 2004 and applies to existing manufacturers and power plants using smaller volumes of cooling

water (i.e., power plants that withdraw less than 50 million gallons per day). EPA must take final action

on these regulations by June 1, 2006. The following website contains information about the three phases

of the regulations and the supporting analysis: http://www.epa.gov/waterscience/316b/basic.htm

As required under Executive Order 12866, EPA is conducting economic impact and cost-benefit analyses

for the section 316(b) regulation for Phase III facilities. A comprehensive estimate of total resource value

should include both use and non-use values, such that the resulting total social benefit estimates may be

compared to total social cost. Developing comprehensive quantified benefit estimates for the section

316(b) regulation requires consideration of non-use values because nearly all (96 percent) of impingement

and entrainment losses at CWIS consist of either forage species, or non-landed recreational and

commercial species that do not have direct uses or, as a result, direct use values. Although individuals do

not use these resources directly they may nevertheless be affected by changes in resource status or

quality, such that they would be willing to pay to maintain these resources. It is generally accepted that

non-use values may be substantial in some cases, and that failure to recognize such values may lead to

improper inferences regarding policy benefits. Many public comments on the proposed section 316(b)

regulation for Phase II facilities and the Phase II Notice of Data Availability suggested that a properly

designed and conducted stated preference, or contingent valuation (CV), survey would be the most

appropriate and acceptable method to estimate the total (including use and non-use) benefits of the rule. 

Stated preference survey methodology is the generally accepted means to estimate total values. Stated

preference surveys use carefully designed questions to elicit respondents' willingness to pay (WTP) for

particular ecological improvements, based on their responses to either discrete choice or open-ended
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

2 2/06

questions regarding hypothetical resource improvements or programs. Such improvements may include

increased protection of aquatic habitats or species with particular attributes.

To assess public policy significance or importance of the ecological gains from the section 316(b)

regulation for Phase III facilities, EPA proposes to conduct a stated preference study to measure non-use

benefits of reduced fish losses at CWIS due to the regulation. The study would focus on a broad range of

aquatic species, including forage fish and a variety of fish species harvested by commercial and

recreational fishermen. The findings from this study would be used in developing estimates of the

economic benefits of the section 316(b) regulation for Phase III facilities. These findings would also be

pertinent to economists and policy makers studying changes in fish populations and aquatic habitat

improvements.

The survey will ask respondents to choose how they would vote, if presented with two different

hypothetical regulatory options characterized by [a] changes in annual impingement and entrainment

losses of fish and other organisms, [b] effects on long-term fish populations, [c] effects on recreational

and commercial catch, and [d] an unavoidable cost of living increase for the respondent's household. 

Respondents will be allowed to "vote" for one of the presented regulatory options, or to choose not to

vote for either option. The survey will also ask respondents to answer questions about their reasons for

voting, their level of concern about various policy issues, and their affiliations and recreational activities. 

Peer review is a process for enhancing a scientific or technical work document so that the decision or

position taken by the Agency, based on the technical document, has a sound, credible basis. The goal of

the Agency’s Peer Review Policy is to ensure that scientific and technical work products receive

appropriate levels of critical scrutiny from scientific and technical experts as part of the overall decision

making process. This document was reviewed by Dr. Richard C. Bishop, Dr. Kevin J. Boyle, Dr.

Timothy C. Haab, Dr. James R. Kahn, and Dr. Alan J. Krupnick. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

3 2/06

II. CHARGE TO THE PEER REVIEWERS

1. Is the choice experiment survey an appropriate stated preference survey format for the estimation

of marginal willingness to pay for the characterized policies? Will the survey likely provide

useful insights into preferences, tradeoffs, and willingness to pay associated with the described

policies?

2. Does the effort used in developing and testing the proposed survey meet current standards for

stated preference surveys reflected in the literature?

3. Can the policy options be sufficiently well defined such that the survey is likely to provide useful

insights into preferences, tradeoffs and willingness to pay? As structured, can respondents – by

taking the survey – report values for the actual environmental improvement, where impacts may

be large from the perspective of forage fish only or impingement and entrainment, but small from

the perspective of only recreational and commercial fish species? Will the survey likely provide

useful insights into preferences, tradeoffs, and willingness to pay associated with the range of

likely policies?

4. Does the information provided in the survey on the baseline ecological conditions, the expected

changes, and the expected costs and their incidence seem sufficient, appropriate, unbiased and

accurate? Does the survey provide appropriate and sufficient introductory and debriefing

questions to enable identification of protest responses and symbolic responses, to test retention

and understanding of information, etc.? Would you suggest any additional debriefing questions?

5. Does the survey sufficiently and appropriately characterize the policy options under

consideration? Does the survey place sufficient and appropriate emphasis on the scope of

resource changes? Does the survey appropriately communicate the scientific uncertainties as well

as uncertainties associated with the policies in question?

6. What are your expectations regarding the potential validity and reliability of benefit estimates that

might be provided by the proposed survey, compared to the current state of the art in the stated

preference literature?

7. Does the survey take appropriate steps to minimize hypothetical bias? Does the survey place

sufficient and appropriate emphasis on the budget constraint facing respondents, and on potential

substitute uses of their budget? Is the payment vehicle used in the survey credible? Does the

survey incorporate appropriate language to ameliorate strategic bias, symbolic responses and “yea

saying”? Is there any sign of systematic bias in the survey or presented information? If so, how

can it be ameliorated?

8. Will the answers to this survey lend themselves to straightforward economic interpretation of

willingness to pay? Are the proposed econometric modeling approach and validity tests

appropriate? Would you suggest an alternative modeling approach or alternative or additional

validity tests?

9. The national benefits analysis for Phase III of the 316(b) rulemakings must be completed within

the next several months and a final rule issued by June 1, 2006. Are there improvements to the

survey and associated model that would be feasible to implement within the available time frame? 

If more time was available, how would your answer change?
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

4 2/06

III. GENERAL COMMENTS

Richard C. Bishop

Before beginning my comments, let me suggest that you take them with a grain of salt in the following

sense. I will definitely focus on potential problems, forgetting to tell you about all the things that are

done well. Much hard work is evident in what I saw here and it has produced a lot of progress. Also,

survey design is as much an art as a science and we who work in the field frequently disagree about best

practice. In the end, those working on this study will have to use their own judgment about which of my

comments and suggestions to focus on in final revisions before the pretest. I am kind of like a dentist

doing a checkup. Indeed, reading my comments may seem like a trip to the dentist! I like to probe

around, searching for the survey equivalent of cavities. I hope that some of my comments will help you

do an excellent study. But you will and indeed should decide to ignore others either because you are not

sure they are helpful or you can’t take the time or expend the energy to fix what may be low priority

problems. With these caveats, here are my general comments:

• The work is being conducted in a total valuation framework, which is appropriate to the case. A

narrower framework that ignores nonuse values would risk neglecting most of the benefits of the

regulations. 

• So far as I know, the researchers are correct when they suggest that no previous studies have valued

changes in fish stocks including forage, recreational and commercial species over a broad geographic

area. This will be a path-breaking study in that regard.

• To some extent, I suppose I am passing the buck, but I would recommend that a survey expert be

employed as a consultant to help with this study. A heavy hitter like Stanley Presser or Don Dillman

or Jon Krosnick would be great if you can get one of them; another possibility would be Nora Cate

Shaeffer of UW-Madison. Nora is less well known in environmental economics, but her credentials

in survey methodology are impeccable. I have personally worked with her on stated-preference

studies (e.g., she is a co-author of the paper cited as Heberlein, 2005, in the document I was sent). 

Such a person could help evaluate and remedy problems with the instrument including those listed

below. In addition, for reasons stated below, I am concerned about the plan for addressing

nonresponse issues. The sort of person I am recommending might be able to help a lot.

• I am particularly concerned that the slide show is not ready for prime time. In studies like this one,

conveying information is crucial and very difficult. A large number of nitty-gritty issues are probed

under Specific Comments. Let me raise some broader issues here.

< The document that was sent to me for this review was for the most part very clearly written. 

Most of the time, I was able to figure out what had been accomplished and what the plans were. 

Perhaps the biggest shortcoming—if I missed it somewhere, I apologize—was the lack of a

description of the procedures that would be used in the slide show. In what follows, I will

assume that respondents will be required to view the slides before they are allowed to open the

survey proper; that the pace through the slide show will be controlled through Knowledge

Networks, and not at the discretion of the subject; that the script in Attachment 2 will be applied

as voice-over with each slide; and that there will be no way for subjects to review slides after they

have gone through the slide show. I will come back to this last issue in a moment.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

5 2/06

< The slide show and the survey are currently planned as two separate steps. This is a major part of

the research strategy and merits very careful consideration. There is no easy formula for

conveying all the information needed by subjects in a study like this. Admittedly, there are some

precedents in the literature for setting up the information as a separate step, as noted in EPA’s

write-up. A nice clean break between information and survey questions may work better, but

after considering this approach I think I still prefer to mix information and questions together,

which is how I am tackling the information problem these days. In the Internet surveys I am

working on, subjects are free to go back and forth between frames. Information is presented in

modest doses. A frame or two of information is given and then a question pops up and they can

go back to the information if they want to. Indeed, properly framed questions are an incentive to

look back if some parts of the information are not yet clear.

< Frankly, the amount of information to be communicates all at once seems huge. Given detail in

some of the slides, would it be unreasonable to assume it will take subjects close half an hour (30

seconds per slide) to view the slide show? If so, that is half an hour before they answer one

survey question. The total process might take 45 minutes to an hour. That is asking a lot even of

Knowledge Networks subjects. If you do a separate slide show, it should be cut to 15 minutes or

20 minutes maximum and the pace should be set to be conducive to learning. Has having a

separate slide show perhaps made it too tempting to add more and more slides?

< If you stick with the slide show, I would recommend that you make it interactive with the survey

so that subjects can go back and review information if they want. Try putting links in the survey

so they can go back to one or more slides and then return to the place in the survey where they

stopped. Also, they need to be able to turn the slide show on and off. (What if the phone rings or

the kids start yelling in the other room?)

• Again, I apologize if I missed it somewhere, but I was a little surprised to find that the slide show and

survey only referred to the Northeast. I would infer that subjects from different regions will get

versions tailored to their region. Maybe that would make sense, but I would be tempted to explain it

in the slideshow to avoid confusion. Tell them that this is a national policy but that they need to be

concerned only about their region since people in other regions will be paying costs there. If a

national sample will be considering only changes that are proposed for the Northeast, then there is

more explaining to be done. What about the fish in my region?

• As a general comment on design, except for wandering out to Wisconsin and Oregon for a couple of

groups, all the qualitative work was done in a fairly tight area of the East. Why? I often find it

instructive to have a wider geographic range. Also, the focus group memo points out that while

people are generally more interested in their home regions, some care about other areas. Focusing on

regions could cause some problems in this study. If I live in Maryland, for example, is Maine more

important to me than Virginia? If I care somewhat about other regions, shouldn’t I be informed about

them so I can include them in my values? (Perhaps I am not appreciating the difficulty of expanding

the information to cover the whole country. Also, limiting people to their home region might be

supported as a way of making value estimates conservative.) 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

6 2/06

Kevin J. Boyle

• There are many global statements about valuation and the design of this specific study that are purely

assertions and are not based on any hard empirical research ortheoretical foundations. These

statements detract from what is really a basically sound research design. Examples of the former are

statements with respect to the incentive compatibility of choice questions and hypothetical bias, and

examples of the latter are extrapolations from what was learned from the focus groups.

• The entire discussions of protest responses, hypothetical bias, and other problematic responses appear

to select items from the literature at face value. The key point isthat you did try to address these

issues, but you probably have not in the design norwill not be able to in the estimation do as much to

control for these problems as you imply in the review document. The key point really is how much

subsets of people inthese categories actually affect your welfare estimates. My reading of the

literature is that the only major concern in this regard in hypothetical bias and you are going to do as

much as you can to control this issue, but not to the extent you claim.

• You talk about a sample design that includes multiple weighting schemes and there is no discussion

about how you are going to resolve these different levels of weights nor how you are going to

investigate how weights affect your welfare estimates. One key issue is that you, like many others,

assume that making the sample look like the U.S.population will solve all nonrespone biases, but if

sample selection arises from something other than Census demographics you have not corrected the

problem, you have an endogenous weighting problem.

• There is no discussion of how you are going to add up the values from different regions to get a

benefit estimate for national policy. What if respondents value an area larger than their sampling

region? What if respondents think that people from other regions will pay for their region? See my

comments below.

• You assume that the type/mix of fish does not matter to people. This is a strong and perhaps

necessary assumption. You must make it explicit because information on specific species while

substantially increasing the information load for respondents may also change values estimates.

• I think this is a well designed study, but there is definitely a need for improvement. While

considerable effort has gone into the scenario design, the design is not as tight as the authors of the

review document purport it to be. You have done a very good job, but by over-selling your design

you take away from the good features and make the entire design more controversial.

Timothy C. Haab

Thank you for the opportunity to review this document and survey. I found the document to be

informative, well-written and easy to follow. The document does a good job of outlining the procedures

to be used and for the most part follows acceptable procedures for the valuation of increased fish stocks

from cooling water intake regulation. Particularly, the document provides thorough documentation of the

planned sampling scheme and procedures for ensuring a valid representative sample from an internet

based survey. Many of the fears decision makers face in regard to internet surveys should be allayed. 

Clearly the authors of the document—in conjunction with the experts at Knowledge Network—have

spent considerable time designing a sampling scheme and estimation strategy for correcting any potential

sampling bias created by the internet panel. 

Keeping in mind my overall positive impression of the justification and survey, I do have some concerns

that can be readily addressed. For the most part these are minor concerns that can be addressed quickly
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

7 2/06

and should not cause a significant delay in the administration of the survey. My comments and concerns

are detailed below in response to the charge questions provided by Versar. Perhaps my most significant

concern deals with the analysis of the survey responses. It seems as though the authors have proposed a

fairly basic method of analysis of the choice experiment response and have ignored some important

econometric issues that should be addressed. Further, I have concerns about the aggregation of fish

stocks into a single number in the survey responses. I will provide details below. 

Again, I want to emphasize that overall I think this survey is well-conceived, meets standard academic

practices and when administered will provide useful and accurate information to EPA. 

James R. Kahn

The U.S. Environmental Protection Agency's plan for implementation of a willingness to pay survey

instrument to assess the benefits of §316(B) Phase III Cooling Water Intake Structures is a wellconceived plan that should produce results sufficient to guide the decision-making process. After reading

the document explaining the survey plan, I find that the authors are quite knowledgeable of stated

preference valuation measures, the controversies associated with their use, and the best means of

implementation. I am very pleased that the EPA decided to use the choice experiment (conjoint analysis)

stated preference approach, as I believe this approach to be less subject to biases and more conducive to

the evaluation of alternative policy options in comparison to contingent valuation. The justification of the

approach, the background information slide show, and the survey itself are all first rate work. If the

research plan is implemented as proposed, the EPA will be in a good position to make a decision relative

to §316(B) Phase III Cooling Water Intake Structures and to defend the decision that is made. Although

my overall assessment of the proposed survey and implementation plan is quite favorable, I have several

suggestions that the EPA may want to consider before implementing the plan. 

Conclusions:

The proposed survey looks like it will accomplish the desired purposes, and is extremely well done.

However, some small changes can make it better. In addition, there are several sections, particularly the

sections on biases and econometric specification that need more exposition in order to make the final

results of the survey believable. Moreover, the reference list is not very impressive and does not provide

an adequate survey of existing applied studies using this technique. This unimpressive list gives an

impression that the researchers have not done all of their homework. For example, I did a Google Scholar

search on the keywords "choice experiments environmental valuation" and 25 of the first 40 hits were

directly relevant. A partial list of additional references is supplied.

In summary, with a few modifications, the survey and supporting documentation will provide the

appropriate type of information with the appropriate level of confidence in the empirical results. I am very

pleased that EPA has taken this route, instead of relying on benefits transfer or a contingent valuation

survey.

Alan J. Krupnick

I commend the survey team for their tremendous amount of hard work and boldness in attempting to

capture preferences for a pretty ephemeral commodity – fish kills that have no ecological consequences. I

also commend EPA for providing what must be a huge survey budget for 5,000 completes. While I

actually think this is excessive for what is planned, it is heartening to see the agency make a commitment

to such a major study. At the same time, in a study of this complexity there are inevitable tradeoffs that

have to made, say between consequentiality and bias, where the literature is insufficient as a guide. There

are many such choices here and, given my experience, I think I would have made some different choices,

or at least altered the design to develop a strategy of protecting the survey against the possibility that
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

8 2/06

some of the choices were bad ones, primarily through developing additional treatments.

Detail on major problems:

• There is no baseline or the baseline is poorly defined. There are three ecological attributes in the

valuation exercise: numbers of fish saved, effect on long-term fish populations and effect on

recreational and commercial catch. The first has no baseline, except relative to the total fish currently

“lost” in cooling water intakes. While some individuals might care about fish death per se, I would

imagine this is not a large fraction of the population. Most would want to know if this amount of

death is large relative to the overall populations of various fish species. They might want to know

this to gauge whether fish populations are at risk, which is really dealt with by the second attribute or

just about having a number to scale the deaths to. If it is the former, then the number of fish saved

attribute is redundant and should be dropped. If it is the latter, then some number is needed. In this

case, my advice is to make up a few different baselines and then use them in several different

treatments to test whether the baseline that was made up makes any difference to WTP. I suspect it

will, but in that way can be corrected or adjusted for. If something like this isn’t done, I wouldn’t

know how to interpret any results from the survey.

The second attribute’s baseline is poorly defined. The scale gives an endpoint of 100%, while some

screens fail to show what this 100% refers to. I recognize that the text may define this, but I think it

needs to be on the screen. The text definition refers to the maximum historic levels of fish. This is

poorly defined in my view. The maximum may not be the optimum or even be sustainable. I think

that some sentence like: Scientists and ecologists have determined that appropriate fish populations

consistent with the health of the ecosystems affected. The 100% refers to population levels consistent

with that determination.

Finally, the third attribute baseline is also poorly defined. Why is the catch in 1950 significant? We

are not told on the screen. Again, if this is interpreted as peak catch then this may not be optimal at

all. Indeed, some respondents might interpret an increase in catch as a bad thing, thereby screwing up

the scope tests.

I am also troubled by what we term “expansive priors” about the baseline. The documentation says

that (at least one) respondents may think that the fish populations are declining. Their response to a

declining baseline situation will certainly be different than that if they think the baseline is static or

increasing. 

Overall, much more care is needed to define baselines and test for baseline effects when different (or

made up) definitions are used.

• Definitions of the commodities. One of my biggest concerns is that the commodity is defined in terms

of fish, not as an indicator of ecosystem health, but as the fish themselves. Respondents may well

substitute their own views on ecosystem linkages with fish kills and you won’t know what they are

thinking.

And not just fish, but commercial fish (in part) which respondents should have no logical reason to be

willing to pay for. Respondents may ask themselves whether prices of fish will come down with the

options being considered, thereby canceling out some or all of their increased costs from the option

itself. So there are really two issues: whether respondents are willing to ignore ecosystem health

concerns and what the role played by commercial fishing is in their thinking. On the latter I would

stipulate that scientists have determined that there will be no effect on commercial fish availability or
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

9 2/06

prices because such fish come from all over the world, and drop any further mention of it. I would

leave in recreational fishing as there may be option and use value associated with it and it is placedbased. Because recreational fishing is placed-based, the text would need to clarify how patchy the

effects would be across substitutable recreational fishing sites.

A broader issue is that the three attributes you list are not independent, but are related to one another

in complex ways. Probably, fewer kills will result in larger long-term fish populations and greater

recreation and commercial catch. But, your orthogonal design varies attribute levels for these

attributes independently (as it should), which means that fish kills could be going down a little while

long term populations could be going up by a lot – and that doesn’t really make sense from any sort

of simple story. Indeed, increases in catch could have effects on long-term fish populations,

irrespective of fish kills from entrainment, so the interdependencies are everywhere. The

documentation says that some inadmissible combinations of levels were dropped from the design

space, so maybe this problem has been at least partly addressed. What is not addressed, though, is

that if interdependencies exist in people’s minds then the marginal WTP estimates attached to each

attribute will not be reliable.

Another issue with the commodity is aggregation over species. The survey lumps all types of fish

together and talks about average effects over all species. I would want to be sure that this does not

create problems for the respondent in interpretation. 

Another issue is uncertainty in the commodity. Our research shows that admitting uncertainty results

in a downward bias and increased variance. People think that if you are uncertain about effects either

the effect can’t be very important or large or that you don’t know what you’re talking about so why

should they take the survey seriously. Both tend to push WTP down. 

The authors want to have it both ways. They want to invoke uncertainty, so they can claim they took

it into account, and then ignore it in the choice questions because respondents thought that presenting

intervals of improvements was too confusing. I don't think they can have it both ways without

making changes in the treatment of uncertainty.

As noted elsewhere, my suggestion is to vary in separate treatments how uncertainty is handled, both

about the baseline and about the changes.

• Overall experimental design. With such a large sample size being planned I am surprised that there

are no plans for treatments, e.g., on information, uncertainty, baselines, etc. At the same time, the

large sample is being used to get a better understanding of how WTP varies regionally. Why is such

regional precision required? Further, why is the commodity being limited regionally? In our work,

the payment vehicle definition required a limit to the state, but no such limit is needed here as there

are no taxes involved. Indeed, one could argue that a regional limitation is unrealistic. Market effects

will be the result of cumulative changes across the country (to a large degree), so by not discussing

them the survey may seem less credible and WTP may be biased. 

• Consequentiality versus neutrality. My view may be controversial, but I think the survey errs too far

towards consequentiality (and to far away from neutrality (the absence of bias)). Every survey has to

make choices about whether or not to state the sponsor of the survey, its real purpose, and the

implications of voting. By being truthful, one may easily create bias. To me, this survey faces the

same problem we faced in whether to tell respondents that they are voting on a question about air

pollution controls (with costs passed on to them) to improve conditions in the Adirondacks versus
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

10 2/06

trying to find an biased way of getting at preferences for cleanup and in so doing having to be

untruthful and perhaps lose some consequentiality. If people know that the USEPA is doing this

survey, they will assume that voting for the options is encouraged. Saying that EPA will use these

results to assess values begs the public to vote on a warm glow basis rather than their true

preferences. 

• Payment Vehicle. The payment vehicle begs respondents to wonder why they should have to pay

when business is causing the problem. The notion that increased costs result in higher prices is not

one understood by most people, particularly for costs incurred by powerful and perceived, high profit

industries, such as electric power. They may think that industry will make them pay, but they don’t

think they should have to pay and may therefore vote against such a plan rather than reveal their true

preferences for the fish. 

• Extent of the Market. I don’t understand why the survey is geographically limited in the way it is. 

Since the payment vehicle is a change in market prices, these changes can be realized by anyone in an

affected market. For some products, this is the entire nation; for others, perhaps effects would be

more regional. For power plants, regional effects are conceivable. But these effects for the

Northeast, would surely include Pennsylvania (which is excluded now) because this state is part of a

regional power grid that includes states labeled as northeastern states. From the benefits side, the

extent of the market is even more cloudy. Effects of entrainment are likely to affect an entire

watershed, so that is the extent of the market. Again, Pennsylvania and Maryland share a common

watershed, given the Susquehanna that runs through both states; yet PA is excluded and MD is

included in the definition of the Northeast region.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

11 2/06

1. Is the choice experiment survey an appropriate stated preference survey format for the

estimation of marginal willingness to pay for the characterized policies? Will the survey

likely provide useful insights into preferences, tradeoffs, and willingness to pay associated

with the described policies?

IV. RESPONSE TO CHARGE

Richard C. Bishop

Provided that stated preference surveys are well designed and executed, they have wide support for

applications like this, both in the scientific literature, as cited in the materials provided to this reviewer

and by governmental agencies, including not only USEPA, 2000, as cited, but also the Office of

Management and Budget, Regulatory Analysis. Circular No. A-4. U.S. Office of Management and

Budget, 2003. http://www.whitehouse.gov/omb/inforeg/circular_a4.pdf. Having worked in the field

for many years, I am convinced that stated-preference methods are capable of providing useful insights

into preferences, tradeoffs, and values in cases like this. The choice experiments being designed here are

in keeping with one of the major branches of the literature that has many respected proponents.

Kevin J. Boyle

The choice experiment is an appropriate methodology and is the best methodology available to estimate

values for the Phase III policy.

The survey results will provide useful information, but perhaps not as strong as asserted in the

peer-review document (see my comments below).

Timothy C. Haab

In short my answer is yes. The choice experiment is the obvious and most appropriate choice for the

valuation of multi-attribute policy changes—such as 316(B). I can think of no other method that will allow

for the valuation of multiple attributes in such a clean manner. As designed the survey collects: 

• background information on perceptions towards policies in general

• trade-offs among fish stock attributes

• debriefing information to establish the validity of responses

• demographic information

Combined, all of this information will allow the authors to provide a detailed description and estimate of the

values respondents place on fish stock increases dues to Phase III regulations. 

The use of the Knowledge Networks internet panel is accepted in the academic community and provides an

accurate but less expensive alternative to in-person surveys. The reviewed documents do a nice job of

describing the potential pitfalls with internet surveys and the appropriate solutions. 

I have a few concerns about question ordering and wording in the survey that I will outline below. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

12 2/06

James R. Kahn

Not only is the choice experiment survey an appropriate stated preference survey format for the

estimation of marginal willingness to pay for the characterized policies, it is the best stated preference

approach for this purpose. I believe the survey to be well-constructed and it will provide useful insights

into preferences, trade-offs and the willingness to pay associated with the described policies. Based on

everything I have read in the document, I am confident that the U.S. EPA will have the best possible

information on which to base these decisions.

Alan J. Krupnick

a. The choice experiment (CE) approach is not necessarily the best way of obtaining WTP estimates. It

is more complex than the CV alternative. The most compelling justification for the former over the

latter is if the attributes affected by entrainment need to be unbundled, i.e., if separate shadow prices

(marginal WTP) need to be estimated for each of them, presumably so they can be used in benefit

transfer exercises to apply in locations where the improvements in these attributes are different. A

case can be made here to avoid false precision over these effects that may come from a CE and treat

them as bundled in a CV, using several different treatments to capture a relevant range of attribute

differences. 

b. Whether the survey produces useful values depends on all its features ex ante as well as how well the

analysis turns out. On balance, I think there is some reason to be concerned about its features, as detailed

below, and one can’t know until the survey results have been analyzed if they will withstand scrutiny.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

13 2/06

2. Does the effort used in developing and testing the proposed survey meet current standards

for stated preference surveys reflected in the literature?

Richard C. Bishop

Those designing the study display excellent knowledge of the scientific literature on stated-preference

methods; experts on the biological side have obviously been consulted; attention has been given to

theoretical issues; survey materials were developed; focus groups were conducted and the survey

materials revised; and a peer review is being conducted now to gain additional input before the pilot study

is implemented. Except perhaps for gaining more input from an expert in survey methods (See General

Comments above) and some specific questions raised below that may warrant further attention, these

procedures meet current standards.

Kevin J. Boyle

The design effort does meet current standards, but the inferences from this design process are too strong

(see my comments below).

Timothy C. Haab

Yes, the use of 12 focus groups and cognitive interviews for survey development is appropriate and

warranted. The 500 individual pilot survey will provide EPA with valuable insights into how the public

is interpreting the survey. I would encourage the survey administrator to follow up with pilot survey

subjects to get open ended feedback on the instrument itself. Such feedback will prove valuable in

making sure subjects are interpreting questions appropriately.

James R. Kahn

Yes, the focus groups and pre-testing more than meets current standards that have been discussed in the

literature.

Alan J. Krupnick

The effort with 12 focus groups seems reasonable to me, certainly not excessive for a “commodity” as

complex as this one. In our survey on the WTP to improve the Adirondack ecosystem, we used over 30

focus groups. This probably was excessive, in retrospect.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

14 2/06

3. Can the policy options be sufficiently well defined such that the survey is likely to provide

useful insights into preferences, tradeoffs and willingness to pay? 

Richard C. Bishop

The breadth of the impacts of I&E regulations in terms of both geography and affected species makes this

a challenging study. It is impossible to develop a scenario that touches on all the aspects that may be of

relevance to all subjects. And, like so many other environmental economic studies, scientific uncertainty

about both the baseline and the effects of the regulations add to the difficulties. We need to recognize the

limitations of stated-preference methods in such a context. For example, if it were possible to tailor

surveys to the effects of the regulations in each subject’s home area, their answers might change. Still,

useful—albeit imprecise—insights into preferences, tradeoffs and WTP can probably be gained with this

study. More comments drawing on the results of the focus groups might increase the level of confidence. 

Did most participants seem to “get it” when they talked about what they were being asked to think

through? Could most talk meaningful ways about the tradeoffs involved and why they might care about

them? Did some members of the groups say they didn’t care? (Not everyone needs to care about the fish

affected by use of water for cooling and other such problems. I find it reassuring when members of

groups I am conducting have learned enough and felt comfortable enough to say that they are not

interested.)

Kevin J. Boyle

Yes, the policy options can be sufficiently well defined. I have made some specific recommendations that

should be addressed and the limitations/assumptions should be more explicitly stated.

Timothy C. Haab

I have a number of concerns here. First, the slide show and survey spend considerable time trying to

convey that the numbers presented are ‘uncertain.’ I’m not sure this is successful and it is critical that it

be. In particular, the choice experiment questions themselves seem to convey ‘certain’

information—“1360 billion fish saved”, “Increase to 70%”. There is no uncertainty conveyed. In the

blow-ups to the slide-show some ambiguity arises in the wording: “Policies may increase stocks to

between 60 and 75%” How is the respondent to interpret the actual survey question? Will it definitely

increase to 70%? Will it increase to somewhere between 60% and 70%? Did your focus groups show

any confusion over this?

On a similar note, do respondents all have a common baseline for comparison? They are told that

“100%=maximum historical fish levels”. Is that with or without human intervention? Does everyone

have an accurate assessment of what that baseline will be, or are respondents comparing apples to

oranges? Again, these are questions that were probably addressed in the focus groups but without more

information it’s not clear what the respondent is valuing. Is it an absolute change or a relative change? 

Do subjects know what catch levels were in 1950? 

There seems to be a non-sequitur in the slide show. It wasn’t clear to me that the choice experiment was

an example until after I read through everything once. Nowhere does it say, “Here is a sample of what

one of these questions will look like.”
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

15 2/06

Also, I have concerns that the sample questions may bias the answers to the actual choice questions. For

the same reason you want to control for potential order effects in the repeated paired comparisons, you

may need to control for the values reported in the sample problems. The respondent may be tempted to

compare the actual choice policies to the sample policies. 

I’m concerned about aggregating all fish species into a single aggregate population change. The slideshow give subjects information on the species affected and the table gives detailed information on

changes in fish species. Are subjects supposed to assume proportional increases in stocks for all species? 

Or is it assumed that just the total matters? The National Marine Fisheries Service has detailed estimates

of the value of various species to recreational anglers. Those numbers clearly show that anglers value

species differently. Is it the same for non-anglers? I’m not sure you can answer this with the current

survey. 

James R. Kahn

Yes, the survey does this in quite a nice fashion. Since the policies have varying levels of cost and

impacts on fish, they naturally fit into choice set scenarios.

Alan J. Krupnick

This question is really about whether the levels of the attributes are appropriate. I have no information to

make such a judgment.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

16 2/06

4. Does the information provided in the survey on the baseline ecological conditions, the

expected changes, and the expected costs and their incidence seem sufficient, appropriate,

unbiased and accurate? Does the survey provide appropriate and sufficient introductory

and debriefing questions to enable identification of protest responses and symbolic

responses, to test retention and understanding of information, etc.? Would you suggest any

additional debriefing questions?

Richard C. Bishop

As I have stated, I think the slide show can be improved, but the basic information on baselines and

changes is there and seems to be presented in an unbiased way. I have suggested some improvements in

the debriefing questions in my detailed comments on the survey (see below), but they involve fine tuning,

rather than major changes. The introductory questions designed to assess whether the respondent

understood the information (now Questions 2 and 3) should be beefed up. I like to use a series of truefalse questions when I am trying to evaluate what people have learned from the information provided. 

When I introduce the true-false questions, the instructions clearly say that the goal is not to test the

participant’s knowledge, like in school, but to test how well we did in communicating information to

them. This allows me to go into more depth than Questions 2 and 3 in the current version of this

instrument. And saying that respondents averaged 80% or 90% in answering the questions (which is my

goal in information provision) is a good way to silence those who might otherwise try to argue that people

didn’t read or didn’t understand the information. Let me hasten to add that not everyone agrees that this

is a good idea. I think they are afraid of turning off respondents, thus contributing to overall and item

nonresponse. But I have checked this approach out in focus groups and other qualitative research and so

long as people understand the intent, I do not detect much resistance. I say more about the specifics

below.

Additionally, Question 2 does not seem like a very good test for “Testing for Appropriate Understanding

of the Introductory Materials.” Slide 8 of the introductory materials states that there are many causes of

fish decline. Slide 20 says (emphasis in original) that intake structures “are usually smaller than the

effects of commercial fishing.” The voice-over for Slide 40 makes a vague reference to “a wide range of

past human activities that have caused fish stocks to decline.” Unless I missed something buried in the

slide show, respondents do not have a basis in the introductory information to answer Question 2. To the

extent that they get the ranking correct, they must be falling back on their general knowledge.

Kevin J. Boyle

The ecological condition information is good. Respondents need to be told what to assume about baseline

conditions. I presume that you want them to assume that current conditions will be maintained if they do

not choose a policy alternative but this in not clear in the information presented to respondents. 

Limitations, like no information on specific species, need to be explicitly stated and recognized in the

study documentation. I think the information is well balanced. I have noted below that I am skeptical of

debriefing questions in general and do not have any specific recommendations.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

17 2/06

Timothy C. Haab

For the first part of the question, see the answer to question above.

For the second part, I have concerns about the placement of the introductory questions in the survey. The

first two questions are designed to assess general attitudes towards policy and the rankings will be severely

compromised by the slide-show. The slide show provides 15 minutes of information on the negative

environmental effects of power plant operations and then subjects are asked to rank environmental objectives

among a group of other policies. It seems like this question should come before the slide show.

James R. Kahn

In general, I think the survey provided appropriate information on baseline ecological conditions. The

introductory and debriefing questions also seemed appropriate to identify protest responses and symbolic

responses. However, as I noted in my discussion of 6.6.2, people might believe that reducing

impingement of fish would also reduce impingement of birds, reptiles and amphibians, and that some

technologies might be associated with less thermal pollution than other technologies. The potential

existence of this problem could be tested by adding an additional response to question ten, that would be

something like this:

___I thought there were impacts on ecosystems in addition to those created by changes in fish populations

and partially based my answers on these impacts.

Alan J. Krupnick

a. The information on baseline ecological conditions is lacking, as the materials readily indicate. I think

it will be hard to believe the results for two reasons that come up here: one, an effort is made to

eliminate in people’s minds the links with fish kills to ecological damage and, two, there is no

baseline for total fish stocks. Concerning the former, in my experience, respondents will default to

believing there is a link and just telling them there is not is not enough. At a minimum, add a

debriefing question to see if they bought this restriction. Concerning the latter reason, having the

baseline defined by I&E losses seems wrong to me. The baseline should be fish populations. If these

are not known I would recommend making them up, treating this as a parameter and trying alternative

information treatments varying this baseline to see if responses are sensitive to it.

b. I think the survey goes overboard on consequentiality giving too little attention to its flip side – bias

and warm glow. The questions to test for this are very complicated and one can never know if these

perceptions are being fully eliminated.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

18 2/06

5. Does the survey sufficiently and appropriately characterize the policy options under

consideration? Does the survey place sufficient and appropriate emphasis on the scope of

resource changes? Does the survey appropriately communicate the scientific uncertainties

as well as uncertainties associated with the policies in question?

Richard C. Bishop

See my responses to Question 3.

Kevin J. Boyle

I find this question hard to answer. First, the ultimate ecological effects are not known. Thus I do not

know if the selected baseline conditions and attribute levels will be able to address the actual policy

question. Second, while the analysis discussion talks about linear and quadratic specifications, with a

small number of attribute levels the best that you may be able to estimate is a piece-wise approximation to

a nonlinear function, which leaves question if the actual policy conditions fall on intermediate levels. 

I think the scope of the change is well communicated with words, numbers and graphics, except for the

region of focus.

Uncertainty is not addressed well (see my comments below).

Timothy C. Haab

See response to question 3.

James R. Kahn

The survey adequately characterizes the policy options under consideration and places appropriate

emphasis on the scope of resource change. I think that for the most part, the survey adequately addresses

scientific uncertainty, although as noted above, I did not like the choice of words ("common sense") in

slide 30. I thought the use of these words cheapened the quality of the scientific information.

Alan J. Krupnick

Mostly redundant to above questions. The new part is about uncertainty. In my work I find that when a

survey admits that effects are uncertain, respondents tend to reject the survey (and therefore, biasing

estimates downwards, unless such people are eliminated from the sample) and think the survey makers

don’t know what they are talking about. The paper discounts this effect; but we see this effect very

strongly in our work. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

1

 Green, D., K. Jacowitz, D. Kahnemann, and D. McFadden (1998) “Referendum contingent valuation,

anchoring and willingness to pay for public goods,” Resource and Energy Economics, 20: 85-116.

19 2/06

6. What are your expectations regarding the potential validity and reliability of benefit

estimates that might be provided by the proposed survey, compared to the current state

of the art in the stated preference literature?

Richard C. Bishop

I have stated some reservations about details both above and in the specific comments below. Much

attention to the design of the slide show and the survey is still needed, but I think this study is well along

toward being state-of-the-art for one that must deal with such a broad regulatory initiative.

Kevin J. Boyle

I believe that with further revisions this will be a state-of-the-art instrument and data collection process. I

have concerns that the data collection process may not end up being as sophisticated as proposed. The

data analysis section is too vague at this point to address whether this portion of the study will be

state-of-the-art.

Timothy C. Haab

I feel the proposed survey takes the necessary precautions to provide a reliable, conservative estimate of

willingness to pay. While the authors argue against cheap talk in the documentation, the survey

implements a watered-down version cheap talk by strongly encouraging respondents to take the valuation

questions seriously. Further, by reminding subjects that the survey results will be used to inform the

policy decision, the responses are taken from the realm of purely inconsequential to possibly

consequential. Many would argue that this is enough to meet the consequntiality criterion Carson, Groves

and Machina and Green, Jacowitz, Kahnemann and McFadden1

 set out for incentive compatibility.

James R. Kahn

My expectations are that the validity and reliability of the benefit estimates are very strong in comparison

to the current state of the stated preference valuation literature. I would suspect that after the dust settles

from the rule-making process, the authors will be able to publish this study in a top journal, such as the

Journal of Environmental Economics and Management, Ecological Economics, or Environment and

Resource Economics.

Alan J. Krupnick

I think getting credible marginal WTP values by attributes is unlikely because the attributes aren’t

independent (see more on this below).
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

20 2/06

7. Does the survey take appropriate steps to minimize hypothetical bias? Does the survey

place sufficient and appropriate emphasis on the budget constraint facing respondents, and

on potential substitute uses of their budget? Is the payment vehicle used in the survey

credible? Does the survey incorporate appropriate language to ameliorate strategic bias,

symbolic responses and “yea saying”? Is there any sign of systematic bias in the survey or

presented information? If so, how can it be ameliorated?

Richard C. Bishop

Section 6.6.1 shows that hypothetical bias has been thoroughly considered and addressed. I agree with

EPA’s reluctance to introduce a cheap talk script. In my judgment the budget constraint and substitute

reminders are more than adequate. Overdoing efforts to provide budget constraint and substitute

reminders risks insulting significant numbers of subjects. (“What? Do you think I don’t understand that if

I spend more money on this, I will have less money for other goods? Do you think I am stupid?”) I

discuss the payment vehicle below (see specific comments relating to Page 21 paragraph 1). I think the

survey meets current standards in dealing with the other potential sources of bias described in this

question. Section 6.6.3 seems more than adequate in this regard.

Kevin J. Boyle

The survey does a good job of addressing hypothetical bias, but this bias is not controlled to the extent

that the review document claims. One additional improvement would be to make the decision rule by

which the policy will be provided more explicit.

There is sufficient emphasis on the budget constraint, but I wonder how much it matters with such low

cost amounts. In fact I am concerned that the coefficient on the cost variable may not ultimately be

significant so the pilot study is imperative to test this hypothesis.

The payment vehicle is credible, but it may be better to acknowledge that there may be some tax effects.

The study does the best it can to address strategic bias, symbolic responses and year saying, but the

design choice experiments opens up the possibility for all of these problems. Thus, I do not think that the

controls are as strong as the study designers claim (see my comments below). I do not see a systematic

bias in the information presentation, but do make some suggestions for improvements.

Timothy C. Haab

The survey design minimizes the potential for incentive incompatible responses. I would like to see a

more detailed description of the mechanism by which the ‘cost of living’ increases. For example, “Such

regulations will no doubt increase the cost of producing power to the firm. These costs will most likely

be passed on to consumers in the form of higher prices for electricity and other goods that heavily rely on

power to be produced. 

The budget constraint and substitute reminders are weak. They appear as three bullet points on one page. 

I would like to see some examples of the substitutes. The substitutes don’t have to be just environmental

substitutes. “The proposed policy will result in higher costs to you. This is money that could be spent on

other things like eating out, recreation, gas, etc.”
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

21 2/06

James R. Kahn

I don't believe that hypothetical bias is a real problem with correctly structured conjoint analysis studies,

and I believe that this study is correctly structured. Respondents know they will be making a real world

decision and that it will have real world consequences in terms of fish populations and the cost of

electricity and other products. There is not a need to focus on "cheap talk" o r other mechanisms for

alleviating hypothetical bias, because they are not asked about a hypothetical donation. They are asked

about potential impacts on the day to day cost of living. Since the questions are structured with both costs

and benefits, it is very difficult for a respondent to behave strategically. I see no potential for systematic

bias, outside of what I noted in my discussion of section 3.2 (respondent fatigue) and section 4.1 (oversampling of the information literate in a way that is independent of typical demographic or socioeconomic classifications). The respondent fatigue issue can be addressed by increasing the participation

payment. The over-sampling of the information literate population is extremely difficult to address, and

since appropriate discussion of the non-response surveys are not contained in this document, I cannot

determine the significance of this problem.

I also note potential problems associated with who answers the telephone and who is handed the

telephone. The document does not specify a protocol for randomly choosing the household member who

participates in the survey. In addition, there are potential biases that will develop if cell phone numbers

are not included in the survey sample.

Alan J. Krupnick

Some of these questions are also redundant to above. A new issue is: Payment vehicle: I think this

payment vehicle is highly problematic. I don’t accept that respondents will accept that product prices will

be higher and hence their costs will go up. I recognize that the paper disputes this from focus groups. 

Respondents in my experience will blame business and believe that they shouldn’t have to pay. They

may dispute that prices will go up, or may think prices will go up by more than they should (rather than

by less than the cost increase) because of monopoly business practices. Or respondents may believe that

they can simply substitute away from commodities whose prices rise (admittedly, substituting from

electricity wouldn’t be easy!). At the very least, debriefing questions would be needed to tease out

whether the payment vehicle caused problems for respondents.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

22 2/06

8. Will the answers to this survey lend themselves to straightforward economic interpretation

of willingness to pay? Are the proposed econometric modeling approach and validity tests

appropriate? Would you suggest an alternative modeling approach or alternative or

additional validity tests?

Richard C. Bishop

As I say below, I have some reservations about the payment vehicle, but otherwise the results should be

amenable to straightforward interpretation. The plans for analysis, which involve not only a rather

traditional approach, but also an effort to account for heterogeneous preferences seems more than

adequate for this stage in the project. 

Kevin J. Boyle

The study results can lead to “straight forward economic interpretation” if the sample weighting issues are

resolved and state-of-the art estimation techniques are used. Neither of these points is clear in the current

documentation and I say more on these points below.

There is only one validity test that I see in the design and that is the external test of scope, which I believe

is fundamentally flawed (see my comments below).

Timothy C. Haab

Unless I have missed some details, the proposed econometric technique fails to capture the panel nature of

your responses. Each subject will make up to three choices. These responses must be treated as

correlated for a given respondent. Failing to do so will bias parameter estimates and estimated standard

errors. Models for repeated choices are readily available in standard software packages like LIMDEP.

James R. Kahn

The answers to this survey will lend themselves to straightforward interpretation in the decision-making

context. The validity tests seem appropriate, but as noted above, I am a bit confused by the econometric

modeling. When talking about heterogeneity in section 4.3.3, the authors refer to using the Heckman

correction method. However, this does not appear in the econometric specification listed on page 30. This

inconsistency leads the reader to wonder exactly what econometric model is being employed.

Alan J. Krupnick

I think that the sample size is larger than needed for the survey as currently designed. I suggest that the

experimental design be made more differentiated, with different information treatments developed and

tested to examine the effect of certain major choices in survey design on the results.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

23 2/06

9. The national benefits analysis for Phase III of the 316(b) rulemakings must be completed

within the next several months and a final rule issued by June 1, 2006. Are there

improvements to the survey and associated model that would be feasible to implement

within the available time frame? If more time was available, how would your answer

change?

Richard C. Bishop

That is a very short time horizon to do the pilot, main survey, follow-up survey, and analysis. You are

likely to have to live with some warts and loose ends. More time would allow more attention to the

suggestions coming out of this and other peer reviews, a thorough evaluation of the results of the pilot, a

more careful statistical analysis of the main survey results, and a lot more thought about the follow-up

survey. I urge you to add at least six months to the schedule.

Kevin J. Boyle

I do not believe the study can be completed by June 1! Yes, if there was 9 months to a year available.

Timothy C. Haab

The suggestions I have made in the above responses can all be implemented in a short time period. The

overall concept and design of the survey is sound and with minor corrections will represent a solid piece

of applied valuation. 

James R. Kahn

There are a few sections where the discussion needs to be beefed-up to make the reader more confident in

the analysis. This includes the discussion of the stratified sample, nonresponse biases and the econometric

specification. The primary change to be made in the survey would be the incorporation of an additional

response to question 10, as mentioned in my response to question 4. Finally, since the survey is so

lengthy, it is important to pay the respondents a higher payment, or run the risk of the respondents

treating the later stages of the survey with less than their full attention.

Alan J. Krupnick

I don’t know if the above idea (in 8.) is feasible within the time frame. Some ideas below are obviously

easy to implement.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

24 2/06

V. SPECIFIC COMMENTS ON REPORT

Kevin J. Boyle

Page 6, Paragraph 2

I have concerns about the 600 follow-up interviews with nonrespondents. No goal was set for the

response rate. These are people who have already rejected the mode of implementation of the survey or

the specific survey for a variety of reasons. What is being provided as incentive to get these people to

respond? If you meet the target of 600 completed interviews, but have a nonrespondent sample response

rate of 10-20 percent, then this information will be of limited use; you have just added an additional

sample selection problem. In addition, if you find a difference in values you will not be able to identify

whether you have a sample selection effect, mode effect or both effects causing the difference. While you

are making the follow-up as similar as possible to the original survey implementation it is not identical.

Richard C. Bishop

Page 6, Paragraph 2

More advantages of the Internet mode might be mentioned. Four come immediately to mind. First, mail

surveys can run into difficulties with skip patterns that can easily be programmed into an Internet survey. 

Second, where a number of stated-choice questions are to be asked back-to-back, it is possible to tailor

later questions based on answers to earlier ones. Suppose a respondent chooses Alternative A over

Alternative B when costs are $15 and $5, respectively. It would not be very useful to ask that same

respondent a second question where attribute levels are the same except that Alternative A has a cost level

of $10. Mail surveys cannot be tailored leave out such redundant questions. Third, voice over to help

respondents through complicated or extra long materials can be added. Fourth, links to more detailed

information can be added for those respondents who think they need it without complicating the main

presentation for everyone. Also, regarding existing studies, do you want to mention the Viscuzi et al.

(2004) study on water quality that was done for EPA and which was cited later in this document? I

believe an Internet mode was used there. 

Kevin J. Boyle

Page 6, Paragraph 3

Where are you going to obtain “historical response rates of individuals in that group for similar types of

surveys?” You make the argument in this report, which I agree with, that no work of this type exists. This

assertion nullifies you ability to make such an adjustment in the sample design.

Alan J. Krupnick

Page 6, Paragrapgh 3

I don’t see the need for oversampling. Rather sampling weights can later be used to correct for

differences between the population and the sample. I am not sure that doing this up front in sample

selection is worth it. Maybe it will introduce unknown biases relative to a more neutral attempt to enlist

respondents. 

Kevin J. Boyle

Page 7, Top

I think the objectives are good and appropriate for the policy issue you wish to value.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

25 2/06

James R. Kahn

Page 7, Previous Research (Section 1.2)

I concur with the authors that previous studies do not exist that contain an estimation of the social benefits

of nationwide changes in populations of fish species that include species utilized in sport and commercial

fisheries, as well as species that are not sought but are important forage fish. Although there are a plethora

of valuation studies of fisheries including those at the national level, these tend to focus on specific

species such as salmon, or recreational species in general. Additionally, these studies generally target the

use values of these species and do not measure values such as altruistic value or existence values. The

authors are correct in there assertion that general studies of the value of water quality improvements as a

whole measure of different suite of services than the proposed §316(B) decision will affect. An

implication of the authors findings which should be explicitly stated in this section is that it is simply not

possible to evaluate the benefits of the§316(B) Phase III program with a benefits transfer study.

Alan J. Krupnick

Page 8, Paragraph 2 

It is one thing to assert that the values from this study will only apply to fish, rather than ecosystems;

another to make people believe that. In my experience, people know very well about food webs and

won’t surrender this idea easily. From the report, I am not sure that the idea should be surrendered. Is it

true that there are not ecosystem level effects of entrainment? 

James R. Kahn

Page 8, Consultations (Section 1.3)

The consultations, focus groups and pilot survey discussed in this section are consistent with the

standards of the literature. This reviewer has no further suggestions in this regard.

James R. Kahn

Page 9, Survey Objectives (Section 2.1)

The survey objectives are clearly stated and consistent with developing the knowledge necessary to

inform decision-making with regard to the §316(B) Phase III program. No modifications need to be made

to this section.

Alan J. Krupnick

Page 10, Second Bullet

I am not clear from reading the report exactly how uncertainties are going to be treated in the survey.

James R. Kahn

Page 10, Key Variables (Section 2.2)

The authors do an excellent job in terms of tying key policy variables to the structure of the choice

experiment framework (conjoint analysis). The literature cited is appropriate (Adamowicz, et al, Louviere

et al, Bennett and Blamey and Batement).

Kevin J. Boyle

Page 10, Bottom 

It is good and necessary for welfare calculations to have an opt-out option in the choice questions, not all

choice question designs include this design feature.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

26 2/06

James R. Kahn

Page 11, Survey Support (Section 2.3)

This reviewer is very familiar with Abt Associates, Inc., and has great respect for their work. This

reviewer is not familiar with the work of Knowledge Networks, Inc., but presumes that the authors have

chosen a reputable sampling firm.

James R. Kahn

Page 11, Collection Method (Section 3.1)

The proposed method of an internet-based survey is the best method that can be implemented given the

time constraints that EPA faces. The internet-base survey has the advantage of being able to supply

information in a fashion that is more difficult in face-to-face or mail surveys, as well as avoiding the

biased associated with the use of a telephone survey. I have some concerns that the use of the internetbased survey can lead to some biases that can not be corrected through advanced statistical methods. This

will be further elaborated in the discussion of biases. Nonetheless, the proposed method is undoubtedly

the best direction to go for the type of information that is needed to support this decision.

Kevin J. Boyle

Page 11, Paragraph 1 of Section 3.1 

I do not believe that the last sentence of this paragraph is true. A panel member could give access to

another member of their household without the survey administrator’s knowledge. You have no more

control here than you do with a mail survey; the recipient can allow another to respond.

Richard C. Bishop

Page 11, Paragraph 3

It is probably worth mentioning explicitly here or elsewhere that the KN panel includes people who do

not have Internet access. The WebTV option is mentioned, but not explained. This is a concern that, for

example, Dillman (2000) mentions with regard to Internet surveys and I think KN has covered this one

pretty well. Also, those who complete surveys are rewarded in an effort to reduce attrition. 

Kevin J. Boyle

Page 12, Top 

What if computer technology is the reason people opted not to participate in the panel? This follow-up

procedure will do nothing to bring this type of person into the study.

Richard C. Bishop

Page 12, Paragraph 2

This paragraph is hard to follow. For example, “Respondents will receive an incentive fee of $5 to $10

for completing this survey.” Which survey, the follow-up or the main survey or both? 

James R. Kahn

Page 12, Survey Response and Follow-up (Section 3.2)

I am somewhat concerned about the length of time that is required to complete this survey. Between the

review of the background materials and the actual completion of the survey, EPA estimates that 40

minutes are required. Although this length of engagement is consistent with conjoint analysis experiments

in the test-marketing of new market goods, these marketing research firms pay the respondents

substantially more than $5 to $10 dollars. I have had previous discussions with marketing professors who

suggest that $30 to $50 dollars is more appropriate. The longer the time of the survey, the greater must be

the payment to guard against the onset of respondent fatigue and meaningless responses. Although the

survey contains some follow-up questions to test the validity of responses, it might make sense to increase
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

27 2/06

the payment to users as much as the research budget will permit, up to about $30.

Alan J. Krupnick

Page 12, Paragraph 3

A 40-minute average survey is longer than usual for such surveys (30 minutes is usually the target).

Richard C. Bishop

Page 12, Paragraph 3

It is better not to promise too much. “The survey administration format will be designed to ensure that

individuals who agree to participate in the follow-up interviews must review all background information

before completing the survey.” I don’t know of any way to make sure people actually read and digest

such information. Also, here and elsewhere it is a little misleading to call these surveys interviews. 

Interviews imply person-to-person contact, either in person or at least over the phone. Unless I missed

something, only the Internet will be used.

James R. Kahn

Page 12, Pretests and Pilot Tests (Section 3.3)

The pretest focus group and proposed pilot study with 500 responses seem to be appropriate. The second

paragraph of page 13 indicates that "Results from the final two focus groups (one including cognitive

interviews) provided convincing evidence that the majority of respondents answer the stated preference

survey in ways appropriate for stated preference WTP estimates, and that their responses do not reflect

biases noted above." The October 18 memo/report from Abt Associates that documents the focus group

research provides evidence that supports this conclusion. 

Richard C. Bishop

Page 12, Paragraph 4

At least for the studies I am involved in, regular KN panelists do not receive direct cash payments, but

rather points that can be redeemed for rewards. Have special arrangements been made to provide cash

payment is this case? If so, it would be worthwhile to test for the effects of cash versus points.

Richard C. Bishop

Page 12, Paragraph 5

What is EPA ICR #2155.01? I am not sure I understand the difference between cognitive interviews,

think aloud analyses, and verbal protocol analyses. It is always hard for an outsider to evaluate what

happened in such sessions, but at least the exact procedures could be spelled out. Where were they held? 

How were subjects recruited (e.g., were people from a wide range of socioeconomic backgrounds

successfully recruited)? Who moderated? What procedures were followed in each group or set of groups

(e.g., how did the subjects “complete the survey,” on-line or paper versions)? Did participants often say

things that indicated that they hold use and/or nonuse values for the proposal? If so, what sorts of reasons

did they give? Since participants filled out the survey, how about some tables with results?

Kevin J. Boyle

Page 13, First Full Paragraph 

In assessing the design it would have been much more helpful if the text reported the issues/concerns

identified and how they were specifically addressed in the survey design. This type of claim is not helpful

and only raises questions in my mind.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

28 2/06

James R. Kahn

Page 13, Target Population (Section 4.1.1)

This is a standard approach.

James R. Kahn

Page 14, Sampling Frame (Section 4.1.2)

I have several concerns about this section. The document does not make clear whether the telephone

numbers in the sample are land-lines or both land-lines and cell phones. If cell phones are not included,

there is a serious sampling problem because young people (18 to 25) generally do not have land-lines

unless they are living in the parents' home. (I do not have statistics on this, just my general observations

of my children, their friends and all the university students with whom I have contact). I also believe that

there are biases introduced by who answers the phone in the household. If the survey is implemented with

the person who answers the phone, then lazy husbands will tend to be underrepresented in the sample. On

the other hand, the phone could be given to the person in the household who most cares about fishing.

This would introduce a different type of bias in the sample. This section does not state if there is a

procedure for randomly choosing a person within the household.

James R. Kahn

Page 14, Sample Size (Section 4.1.3)

The authors use standard practice.

Alan J. Krupnick

Page 14, Paragraph 2 (Section 4.1.3)

If the precision achieved is “more than adequate” than one should do what is adequate and spend the rest

of the money doing additional treatments to shore up unavoidable issues with the survey.

James R. Kahn

Page 14, Stratification Variables (4.1.4)

Knowledge Network is capable of creating a stratified sample, which adequately corresponds to

representation of different socio-economic groups as represented in the U.S. Census. My concern here is

since people are being asked to engage in an online survey, the group of people who are computer literate

(and have online access) may give different responses than the group of people who decline to participate

because they are made uncomfortable by information technology. There two groups may have radically

different WTP for environmental improvement, and there is no way to correct this potential bias through

over-representing a particular demographic or socio-economic segment of the population in the sample.

The document briefly touches on this type of issue when talking about the 600 individual non-response

survey, but does not say how they intend to be guided by the information that is developed from the nonresponse survey.

Kevin J. Boyle

Page 15, Paragraph 1 

What basis do you have for identifying “groups that tend to provide inconsistent or invalid answers” to

attribute-based choice questions? I do not know of any information in the literature to support this sample

design.

Kevin J. Boyle

Page 15, Table 2 

How does this stratification by region compare to the percent of the national population living in each

region?
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

29 2/06

James R. Kahn

Page 16, Sampling Method (Section 4.1.5)

This is a fairly standard approach but does not alleviate my concerns with section 4.1.4.

James R. Kahn

Page 16, Response Rate (4.2.1)

The response rate seems appropriate, but my original concerns about the potential for biases associated

with information literacy remain.

James R. Kahn

Page 17, Precision Targets (4.2.2)

The level of the precision targets is fine, but the document provides not a single clue as to how they were

computed, or chosen based on standard references in the literature.

Kevin J. Boyle

Page 17, Table 3 

The 90 percent confidence intervals for a “typical binary variable” are meaningless in this context; what

we care about is the error bounds on the welfare estimates derived from conjoint responses.

Kevin J. Boyle

Page 18, Paragraph 1

I disagree with the statement that the “use of such methods to identify protest responses is

well-established in the literature.” Some studies do nothing to identify protest responses, others use

questions as you are proposing and others use statistical procedures to identify data outliers. There is no

established procedure and all are ad hoc. There is nothing wrong with doing what you propose it is just

that the conceptual and empirical foundations are not as solid as you portray them. My experience

suggests that a carefully designed protest protocol will result in many respondents providing

contradictory evidence, a single person can provide information that indicates a valid response while

another of their responses may indicate a protest; what do you do with this respondent.

James R. Kahn

Page 18, Non-Sampling Errors (Section 4.3)

The discussion of the potential problems associated with protest bids, and the questions included in the

survey to identify potential problems with protest bids are adequate.

The section also discusses potential problems with and solutions to noresponse errors and unobserved

heterogeneity. The actual explanation of the solutions is left to sections 4.3.1, 4.3.2, and 4.3.3.

James R. Kahn

Page 18, Stratified Survey Sample (Section 4.3.1)

This section looks at the problem of nonresponse bias. If a particular groups of the population are more

prone to decline to participate, then over-representing this group in the initial sampling effort can correct

this problem, and help to generate a completed sample population with the same demographic and

socioeconomic characteristics as the population as a whole. In theory, this is correct, but the document

does not specify a procedure to identify the degree of over-targeting the various demographic and

socioeconomic groups.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

30 2/06

Richard C. Bishop

Pages 18 and 19 and Elsewhere

Let me say at the outset that—pending clear evidence to the contrary—I find arguments for the

representativeness of KN panels convincing. Plans for this study relating to adjustments in the sample for

the main survey seem sound to me. On the other hand, there are those who are holding fast to the

conventional wisdom that validity requires probability sampling and a high response rate. Folks at OMB

are a case in point and, as you are well aware, they continue to be all powerful. In this context, the

follow-up survey plan does not seem to me to be sufficiently convincing to carry the day. A total of 600

people from various stages of the recruitment process will be sampled and asked to complete an Internet

survey. It seems unrealistic to hope for much of a response from people who have refused to get involved

with Knowledge Networks in the first place and an incentive of a few dollars is not likely to help. Not

only are they likely to be put off by the new contact but many of them will not have an Internet

connection. And not one citation is provided to bolster confidence in what is being proposed. Has KN

assured you that they have successfully done this before? If so, a few statistics would be incredibly

helpful here.

This is a thorny problem to be sure. It was one of my reasons for recommending the employment of a

prominent survey expert (see part C under General Comments). Dealing with nonresponse bias has, of

course, been an issue in survey research from the beginning. How have survey experts addressed the

problem? How can this study capitalize on what has been learned in the past? Beyond that, we need to

continue to hammer away at the simplistic emphasis on response rates and focus attention on the basic

issue: representativeness. It is also important to stress that there is no such thing as a perfect sample or

survey mode. A good starting point is Krosnick’s “Survey Research” in the 1999 issue of the Annual

Review of Psychology (http://www.unt.edu/rss/class/survey/survey_review.pdf). He points out there (p.

451) that “when probability sampling methods are used, it is no longer sensible to presume that lower

response rates necessarily signal lower representativeness.” He even cites studies where a higher

response rate caused bias. None of the studies he cites fit your needs here exactly, but there is a lot of

food for thought. I understand that Krosnick brought a similar message to a recent EPA workshop. 

I don’t claim to have a complete solution, but here are a couple of ideas. First, one thing we are doing is

adding questions from a general population survey dealing with environmental attitudes. You could add

some of these to the main survey. Then, you can show that your KN respondents match up with national

demographics, but also hopefully that they match up with the environmental attitudes of the general

population. Sorry I don’t have an exact citation at my finger tips for you, but I can get it for you if you

want it. 

Second, and this builds on the first, rather than hoping to get meaningful results from nonrespondents on

the whole survey using the Internet, you could try doing an abbreviated telephone survey. I have done

this with nonrespondents to a mail survey by calling them up, noting that they had not responded to the

survey, and asking them if they would be willing to answer a few questions. Some said that they would

still be willing to fill out the survey and, if so, we ask whether they needed a new copy. Whether they

said they would fill it out or not, everyone who was willing to participate “for just a few minutes” got the

short list of questions including some on socio-demographics and environmental attitudes. Then we were

able to adjust for nonresponse using the telephone survey responses along with the main data set. 

Obviously, your situation is somewhat different, but it seems like this approach could be adapted.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

31 2/06

James R. Kahn

Page 19, Nonresponse follow-up interviews (Section 4.3.2)

This section is also inadequate in its explanation of the procedures to be used to incorporate results from

the non-response survey into the final analysis.

Kevin J. Boyle

Page 19, Paragraph 2

I do not know of any articles in the literature where a Heckman selection model has been merged with

mixed logit (Page 25, 2nd Paragraph). This is an important estimation issue you will need to address as

you will need to consider the correlated responses of individual respondents and the literature is strongly

supporting heterogeneous preferences through random parameters. You can not run a Heckman on a

simple specification and then address these issues in a second model, which seems to be implicitly

implied in the write up.

James R. Kahn

Page 19, Statistical Correction for Unobserved Heterogeneity (Section 4.3.3)

There seems to be an inconsistency between the discussion in this section in the second and third

paragraphs of page 19 and the econometric specification of the desired model at the top of page 30. I do

not see a Heckman correction coefficient in this model. This lack of consistency leads to doubts about

how thoroughly the authors have thought through the correction for unobserved heterogeneity. This must

be addressed.

Alan J. Krupnick

Page 19 through 20

Heckman’s approach is referenced but not explained.

Richard C. Bishop

Page 21, Paragraph 1 and Elsewhere

I am somewhat concerned about the cost of living vehicle. Are there any precedents for using this

wording in the literature? What does it mean exactly, and how does it relate to conventional definitions

of willingness to pay? How would cost of living work into a traditional indirect utility function like that

on p. 23? We are, of course, looking for a reduction in income that would wipe out the gain in utility

from a given policy. Will respondents understand that point? After all, I only have so much income. If

my cost of living goes up, it doesn’t mean that I will have that much less money to spend on goods and

services (unless I reduce savings by an equivalent amount). Cost of living has to do with not being able

to buy as many goods and services. I suppose one might fall back on the idea that an increase in the cost

of living is a reduction in real income, but I am not sure the average citizen thinks in those terms. I think

going with “Increased cost to you household because of higher prices” or something like that would be

safer. The focus group memo is reassuring in this regard. Subjects seemed to understand that the choice

questions were referring to costs to them. Nevertheless, why not be completely explicit by referring to

“the prices you pay for goods and services produced at factories and power plants that use cooling water

from ocean, lakes, and streams”?

While on this topic, let me also question the effort to convince subjects that taxes won’t increase. Given

that prices and taxes are often combined in payment vehicles, I am not sure what the rationale is for

avoiding taxes here. Perhaps it is an attempt to avoid vehicle bias. Yes, some vehicle bias may creep in,

but I still like to include taxes in a case like this for two reasons. First, that taxes would increase may

seem more plausible to subjects. After all, the government buys lots of these goods too (e.g., paper and

electricity). Is it plausible that my household costs would go up but the costs of government would not? 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

32 2/06

Surely either taxes will go up or some people, at least, will suffer losses in government services. Second,

dislike of taxes may lead to some vehicle bias, but it also serves as a wakeup call for subjects. Here is a

cost that would be real and that you can’t avoid. If someone tells me that they are willing to pay more

taxes for an environmental improvement, I am more inclined to believe them. Also the direction of the

bias, if any, is known. (Admittedly, choice of payment vehicle is something that various researchers in

this area disagree about. It is still very much a judgment call. I am just giving you my opinion.) 

Additionally, pages 15 and 16 of the focus group memo do not provide much evidence that taxes would

be problematical in this case.

Richard C. Bishop

Page 21, Paragraph 1 and Elsewhere

The statement about not adding up or comparing programs is probably okay, but I am not sure your

average person will get the point. I like to be more direct with something like, “At this point, it is not

clear what the alternatives will actually cost, so we are asking about a range of costs. Also, we are asking

about a range of effects so that our results can be adapted as scientists learn more.” 

Kevin J. Boyle

Page 21, Paragraph 1

It is good to tell respondents to consider each question independently, but this is not likely to be sufficient

(see Holmes and Boyle, Land Economics, Vol. 81, No. 1, 2005). This interaction with how respondents

answer the sequential questions leaves questions about what statistical information should be used for

policy purposes.

Kevin J. Boyle

Page 21, Paragraph 2

The certainty questions of Champ et al. (2004) are not very useful. It will always be possible to map two

welfare estimates through some type of integer or continuous preference scale. The question that remains

is whether the specific point on the uncertainty scale is unique to the specific study or can be generalized

to other types of applications. The Champ study is silent on this issue. There is no clear approach in the

literature to “successfully control for hypothetical bias.” This does not mean that you should not try, it

just means that you may not make the right adjustment and should not make the claim that you have

adjusted for this bias; your resulting value may overestimate or underestimate welfare.

Kevin J. Boyle

Page 23, Paragraph 1 

I suggest in considering data quality you consider some of the information that Berrens and colleagues

have explored with internet surveys, e.g., time spent viewing slide show, time spent answering the

valuation questions, etc. This information could be interacted with respondent characteristics that you are

collecting, e.g., environmental group membership, education, etc.

James R. Kahn

Page 23, Analysis of Stated Preference Data (Section 6.1)

This discussion is pretty standard and accepted within the environmental economics community.

James R. Kahn

Page 23, Paragraph 2

The random utility model pre-dates both Hanemann and McConnell and should be attributed to

McFadden (1974). 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

33 2/06

Kevin J. Boyle

Page 25, Second Complete Paragraph 

This is the type of model you need to use and I expected it to be stated more explicitly earlier in the text.

James R. Kahn

Page 25, Advantages of Choice Experiments (Section 6.2)

Again, this is standard and consistent with the literature.

Alan J. Krupnick

Page 27, Paragraph 2

Here is the justification for the CE. It is not compelling.

James R. Kahn

Page 27, Comment of Survey Preparation and Pretesting (Section 6.3)

The EPA has done a good job in pretesting of the survey and in survey preparation.

James R. Kahn

Page 28, Econometric Specification (Section 6.4)

As mentioned above, the Heckman correction method that was referenced section 4.3.3 is not mentioned

in this section. This is a major problem and must be addressed so that it is clearly understood exactly what

econometric specification is being estimated.

Kevin J. Boyle

Page 29, Paragraph 1 

I am confused about how the “baseline population levels” are going to be implemented in the survey. I

see on page 31 (bottom) the levels. Does this mean that the baseline will only be varied for the

“Population Change” attribute? Is this part of the orthogonal design? I seem to be missing something

from the explanation in the text.

Kevin J. Boyle

Page 30, Model

I suggest considering alternative-specific constants in the model. There are many variants in the published

literature, but I have one specific recommendation. This recommendation is a binary variable that equals

“1” if one of the two policy alternatives is chosen and “0” otherwise. This will give you an idea of

whether people are “in the market” for reasons other than the attributes of your design that you explicitly

control for in the estimation. There are a variety of reasons why the coefficient on this variable may be

positive and significant and you cannot identify specific effects, but potential explanations include all of

the reasons a person might be “incorrectly in the market”. You could try interacting some of your protest

probe variables with this ASC.

James R. Kahn

Page 30, Experimental Design (Section 6.5)

This is a well done section, following the standard literature. EPA has done a good job of eliminating

pairs of choice sets which could lead to undesirable reactions by the respondents, such as a choice set

which completely dominates another in all attributes, or nonsensical choice sets (high levels of saved fish

but low levels of fish population). Attachment 4, which further explains the experimental design appears

to be on the mark.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

34 2/06

Richard C. Bishop

Page 30, Experimental Design (Section 6.5)

Experimental design is an important step in choice experiments. I am not an expert, but from what I

know about it, this step has been more than adequately addressed in this section and in Attachment 4. My

only suggestion is that perhaps there are too many levels of the attributes. Unless you saw clear focus

group evidence to the contrary, I would be tempted to drop 8% for the population change, 2% for catches,

and $2 and $4 for costs. For this kind of broad policy, I find it hard to believe that people will be that

sensitive. (Below, in discussing the survey, I recommend that you not use monthly costs. If you use

annual costs as the main cost attribute, maybe my advice about cost levels is less relevant. As it is, you

are talking about pocket change.) Remember that if you can bracket the eventual outcomes you want to

value in the benefit estimates, interpolation is an option.

Kevin J. Boyle

Page 31, Attribute Levels 

The dollar amounts seem trivially small. These are unlikely to impinge on most respondents’ budgets

(Page 34 2nd Paragraph), which makes me skeptical of discussions about how focus group respondents

indicated that they are considering their budget constraints (Page 34, bottom). You really need the pilot to

make sure the estimated coefficient on cost will be significant.

Kevin J. Boyle

Page 32, Paragraph 1

It is good to exclude dominated alternatives.

Alan J. Krupnick

Page 32, Paragraph 1 

The survey team recognizes the possibility that some pairs will not make sense. But this raises the issue

of interdependencies across the attributes.

James R. Kahn

Page 32, Information Provision (Section 6.6)

Any time that a valuation experiment is done, the researchers must provide information to the respondents

concerning the implications of the changes they are being asked to assess. However, the researcher can be

in a difficult situation because too much information or the wrong kind of information can dramatically

influence a person's preference for one scenario over another. The researcher must try to create a balance

between providing the information allowing for the measurement of the respondent's preferences, without

creating a new set of preferences for the respondents. For example, showing pictures of cute baby seals in

an oil spill scenario could unduly influence the results.

My impression is that in this case, EPA has done a good job in presenting important information in a

"value neutral" fashion. I do not think they have crossed the line between being "estimators" of value and

"creators" of value.

Alan J. Krupnick

Page 33, Paragraph 2

In spite of the focus group sessions, there is no way EPA can “ensure” that the graphics don’t lead to bias. 

The use of color and pretty pictures that are routine in this survey is a bad idea, in my view.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

35 2/06

James R. Kahn

Page 33, Last Paragraph

Note the use of a double negative, "not unavoidable."

James R. Kahn

Page 33, Amelioration of Hypothetical Bias (Section 6.6.1)

My personal opinion is that the concern about hypothetical bias is a result of a combination of some

sloppy work by researchers in the early years of CVM, and a bias on the part of some economists against

giving credence to values that do not originate in real world markets. If survey questions are designed

appropriately, hypothetical bias need not be a problem. Conjoint analysis is a technique that is less prone

to hypothetical bias than CVM, and the EPA has done a good job with its focus groups and survey design.

I strongly agree with the conclusion in the second half of the second paragraph on page 35 that "cheap

talk" is neither necessary nor desirable in this survey. I think the decision framework of conjoint analysis

obviates the need for cheap talk.

Kevin J. Boyle

Page 34, First Complete Paragraph 

The incentive compatibility you refer to for a referendum format applies to a single, binary choice, not the

application you have where respondents choosing over two alternatives and do nothing and answering

multiple, sequential choice questions. A single choice question with three answer options does allow for

free riding and other strategic behavior, as does answering a sequence of choice questions.

Kevin J. Boyle

Page 34, Paragraph 3

Increasing the “consequentiality of choice” is important to the design, especially in regard to hypothetical

choice, but may lead to protest responses as well.

Kevin J. Boyle

Page 35, Paragraph 1

The statement that EPA finds little evidence to suggest that hypothetical bias should be significant in the

proposed survey instrument” is naive and silly. The empirical evidence is quite clear that

stated-preference methods can provide useful policy information, but that value estimates tend to be

overstated relative to actual monetary transactions. In recent work that I have done with Taylor and

Morrison, which is funded by EPA, we find that welfare estimates from a three alternative attribute-based

experiment can be 20 to 40 percent higher that comparable cash values. More importantly, only 5 percent

of respondents entered the market when cash was involved and over 60 percent of respondents entered the

hypothetical market. This is not an issue that you can design or assert away.

James R. Kahn

Page 35, Amelioration of Symbolic and Warm-Glow Effects (Section 6.6.2):

I agree with the EPA that they have appropriately eliminated warm glows, biases towards big businesses,

and so on. My chief concern in this section is whether "slide 52" is believable. Many people are aware of

the impacts of thermal pollution on estuaries and other water bodies, and they could believe that one of

the technology options (re-circulating cooling water) could have less of an impact on water temperatures.

Also, might people believe that reducing impingement of fish could also reduce impingement of reptiles,

amphibians and birds? This may have been addressed in the work with survey groups, but a discussion is

not contained in the document.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

36 2/06

Alan J. Krupnick

Page 36, Paragraph 1

In my experience, symbolic concerns are always an issue.

Kevin J. Boyle

Page 36, Slide 52 

I think that you can still expect people to think about more than fish. I think people realize that birds eat

fish and there are other ecosystem effects. The ASC that I suggested above is one way to partially control

for these and other unwanted considerations by respondents.

Kevin J. Boyle

Page 37, Assessing Scope Sensitivity (Section 6.6.3)

I do not think that you have a true external test of scope. If your scope test follows the regular choice

questions, which comprise your internal test, you have already “trained” respondents to consider the

differences and this is not a true, external, between-subjects test. However, I am not convinced that the

external test of scope is necessary.

James R. Kahn

Page 37, Assessing Scope Sensitivity (Section 6.6.3)

The EPA adequately address the issue of Scope Sensitivity

Richard C. Bishop

Page 37, Assessing Scope Sensitivity (Section 6.6.3)

This section is not always easy to follow, but it seems to make the question of scope harder than it needs

to be. The internal scope test is adequately described except that on p. 38, paragraph 2, EPA states that

the internal scope test assesses “sensitivity to scope across choices of a single individual.” Surely looking

at regression coefficients on attributes for the expected signs and significance does not involve single

individuals. This is an internal scope test only because each respondent sees more than one stated-choice

question. Scope-test purists worry that having thought about earlier questions will influence how

respondents look at later ones. This is easier to see with a simple contingent valuation exercise where

asking a single set of respondents two questions where the scope of the environmental improvement is

varied would be considered an internal scope. It would be “weaker” because such respondents might take

the question about the small change as a cure that they should be willing to pay more for the large change

or vice versa. The same problem—if it is a problem—could occur as subjects are asked a number of

stated-choice 

questions, thus making the scope test using all the stated-choice questions an internal test. Particularly in

an Internet survey (where you can control the order of question presentation and be notified when

respondents skip the first stated-choice question or come back and change their answers), one ought to be

able to design the first question of each version of the survey so that responses can be used in a splitsample scope test. Provided sample sizes are large enough, the coefficients from the random utility model

run on various first questions could be checked for sign and significance. As I said, I am not sure I follow

the scope test as planned in this session, but my approach seems simpler.

James R. Kahn

Page 39, Communicating Uncertainty to Respondents (Section 6.6.4)

On the whole, this is well done, but I have real problems with slide 30 in the third paragraph of page 40.

The second sentence of this slide (Common sense dictates....) really dismisses any scientific evidence

about these relationships. The scientific knowledge should not be disregarded in an appeal to common

sense. The question should talk more specifically about how scientists can pin the outcomes to a range
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

37 2/06

with a degree of confidence, and by the way, if you think about it this way, preventing the loss of young

fish and fish eggs will certainly increase fish populations in the future.

Kevin J. Boyle

Page 40, Slide 53 

I believe addressing uncertainty is important and too many statedpreference studies overlook the fact that

environmental outcomes are not known with certainty. Having said this, I am concerned that your

treatment of uncertainty may raise more problems than it helps. I understand the issue of cognitive

overload, but other studies have incorporated specific information about uncertainty in the design,

especially those valuing health outcomes. Your current design raises the issue without any focusing

resolution so that each respondent is left to make their own assumptions regarding the likelihood of

outcomes from the Phase III policy. Your “Champ style” question convolutes uncertainty about outcomes

with uncertainty about responses. It is possible to have a scenario when outcomes are known with

certainty and respondents are still uncertain about their response and vice versa.

James R. Kahn

Page 41, Last Paragraph

I agree with the conclusion.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

38 2/06

VI. SPECIFIC COMMENTS ON INFORMATIONAL SLIDE SHOW

Alan Krupnick

General

There are too many graphics and photos, any of which could introduce unknown biases. The researchers

appear to have designed the information for maximum interest, which is good, but at the expense of a dry

neutrality, which is necessary

Kevin J. Boyle

Informational Slide Show 

I believe this is a very good mechanism for conveying the background information.

Kevin J. Boyle

Cover Page of Slide Show (Slide 1) 

You state that it is a survey of northeast residents, but it is far from clear whether respondents should just

value the northeast or all of the U.S. Consider the following points in the slide show.

Slide 2: You say “protect fish stocks in the Northeast US waters.” What if people take trips to fish in

other regions of the U.S. care about what happens there? For example, when I have gone fishing on

the Great Lakes I know that some of the best fishing is by these facilities, but when I fish in the

northeast I know that my fishing is not near any facilities.

Slide 10, 11 and 12: You talk about facilities, but not northeast facilities. Won’t respondents wonder

about controlling facilities in other parts of the country?

Slide 13: The focus returns to the northeast.

Slides 18 and 19: There is no mention of the northeast on these slides where you talk about fish

losses.

Slide 21: You talk about government policies, but no region is mentioned. Most respondents will

know that regulation is done at the state and national levels.

Slides 26 and 27: The first bullet on slide 26 states: “Reduce fish losses in Northeast waters.” The

following slide (#27) states: “EPA needs to know whether US households are willing to pay the cost

...” These two slides take you respondent both directions. Will northeast residents think that people

from other regions of the country will also be paying for the northeast regional benefits? Finally, the

choice questions do no specify the region of the effects or who is voting. I think this inconsistency in

information is bound to confuse some respondents.

Finally, the choice questions do no specify the region of the effects or who is voting. I think this

inconsistency in information is bound to confuse some respondents.

Alan J. Krupnick

Slide 2

“Protect fish stocks.” This seems like a biased statement. 

Alan J. Krupnick

Slides 5 and 6

“part of a larger ecological system.” So which is it? Are the fish an indicator of ecological effects or is

the issue just fish? One information slide gives one picture, while another gives the opposite impression.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

39 2/06

Alan J. Krupnick

Slide 9

“Many fish stocks have been declining.” This appears to establish a declining baseline, but this idea is

not carried forward to the choice questions or later in the text. 

Richard C. Bishop

Slide 9

Thoughtful subjects will see a glitch in your story. Here you say ecosystem functions are affected. Later

you say its only fish and all other effects are negligible.

Richard C. Bishop

Slide 10

I am not sure this diagram will make much sense to subjects at this stage. Also, “commercial facilities”

seems vague. Could you say something like “paper mills, factories, and power plants?” Perhaps you

could even combine this slide and the next one.

Richard C. Bishop

Slide 12

“Aquatic organisms”? I thought this was about fish. Also the diagram might be explained with a couple

of sentences in the audio. This is a very important slide. What does “intake flow” mean? What is this

boxy looking thing in the middle of the picture, what I will call the “screen” here? Perhaps you should

label the pipe walls. What is a “juvenile”? Is this really about juveniles and adults or large fish and small

fish? Do fish that are small as adults only get impinged and not entrained? Seeing the fish next to the

screen labeled “adult” seems to imply that big (“adult”) fish just turn and swim out. Nothing seems to be

caught on the screen. Given the importance of what is being said here, you may need a better diagram

unless focus group participants were very clear on what this diagram portrays. Are eggs only “fish”

eggs? If so, they should be labeled as such. Did no one in the focus groups raise questions about other

organisms like plankton? If so, you may need to clarify why such effects are negligible. Does fish mean

only finfish as implied here or are shellfish affected, say through destruction of eggs. If not, you may

need to tell people why. Do you mean to imply that “screens” do or do not work? If not, why not? Is

there any possibility of inserting a video clip here involving either animation or a real intake structure?

Kevin J. Boyle

Slide 13 

The focus returns to the northeast.

Alan J. Krupnick

Slide 15

“About 1/3 are types...” So does the 1/3 refer to species or to numbers of fish? Not clear.

Richard C. Bishop

Slide 17

The second sentence of audio needs to be more clearly written.

Richard C. Bishop

Slide 18

In the last line of the script, “is” should be “are”.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

40 2/06

Alan J. Krupnick

Slide 19

The table is very complicated and should not be included like this in the information treatment. 

Richard C. Bishop

Slide 19

Not sure I follow the biology here. Take striped bass, for example. Presumably fish caught by fishermen

are from the adult population. Do you mean that 1.8 million adult striped bass are lost or are sufficient

sub-adult bass lost to I&E to equal that many adults or are you comparing apples and oranges (bass of all

sizes in intake structures versus adult bass taken by fishermen)? The last possibility seems a bit

misleading.

Richard C. Bishop

Slide 20

I wouldn’t pick on commercial fishermen here, particularly given that recreational and commercial

catches are combined in the preceding slide. Why make yourselves targets of industry needlessly? In the

next sentence, “it has contributed to a measurable decline in some fish stocks, in some locations.” Sounds

like government-speak to me. Subjects are likely to ask, what about the species that matter to me? Can

you tell them more?

Richard C. Bishop

Slide 21

I don’t like the term “technologies” here and elsewhere. I don’t know what it means to people. And

“cooling water intake structure” and “air burst system” are not much better. A first cut at alternatives

might include “equipment,” “structures used to pull water from water bodies for cooling,” and

“equipment that uses blasts of air.” I hope the picture is clearer in the actual slide show. And those

“aquatic organisms” get mentioned again.

Alan J. Krupnick

Slides 22 thru 24

Pie charts are not very helpful when nearly all the chart is one color. 

Richard C. Bishop

Slide 22

“Close-cycle cooling”? What does that mean? Also this looks like you are mixing information. What

does total fish losses have to do with different “levels of technology”? I would work the total losses of

1.8 million fish into the pie chart on the next slide and explain the pie chart in the audio. Again, this is

worth some time given how fundamental it is to the goals of the study.

Alan J. Krupnick

Slide 23

“reduce fish losses by about 25%.” Doesn’t this set up an expectation about the probable size of fish loss

reductions? If so, when the choice questions vary this amount, how will respondents interpret the

difference?

Richard C. Bishop

Slide 23

I don’t follow the diagram with the striped bass. What do the grid and the X mean?
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

41 2/06

Kevin J. Boyle

Slides 24 and 25 

The information on these slides is confusing. You state that some facilities are using technologies that

“reduce fish losses up to 98%.” You also state: “The proposed policies would require similar technologies

at all facilities ...” Does that mean that facilities using the best technology might be required to use a less

efficient technology?

Richard C. Bishop

Slide 24

Better explain in a sentence or two what a closed-cycle cooling tower does and how it works. Again the

diagram (bass, arrow, X) may not communicate. Also, I’d give them a sentence or two explaining the pie

chart. 

Richard C. Bishop

Slide 25

I have the impression from somewhere that the already regulated facilities are only power plants. If so,

say so, and indicate that other kinds of facilities as well as some power plants are not currently regulated.

Alan J. Krupnick

Slide 25

“Technologies...already in use at 52% of facilities.” This phrase is problematic. If we already have such

experience, why is there uncertainty? And why is the government regulating again? What is the current

regulatory environment? What isn’t working now?

Alan J. Krupnick

Slide 26

“Some costs would unavoidably be passed on.” Some, but not all. So would shareholders benefit? What

is the respondent supposed to think? Passing on costs is not “unavoidable,” if the markets, for instance,

are international and competitive, the manufacturer would have to bear the costs.

Richard C. Bishop

Slide 26

For the second bullet, I would say something like, “increase the costs of producing goods and services,

including those you consume.”

Alan J. Krupnick

Slide 27

“EPA needs to know....” This is an odd phrase. We need to know how strongly people feel about

reducing fish losses, but people are not being asked to do a benefit-cost analysis.

Richard C. Bishop

Slide 29 and later ones

I am not sure about the terms “policy” and “policy options” here. It sounds too much like government

speak. I would mention “regulations of factories and power plants that use cooling water from water

bodies” when I absolutely had to (maybe a little thought would produce something shorter). Otherwise, I

would use “regulations” when I thought I could get away with it and otherwise maybe “regulation of

industries that use cooling water.” 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

42 2/06

Richard C. Bishop

Slide 30

Good. This starts to take care of why different combinations of attributes will be presented. Reminders

may be needed later. I would try to shorten the third bullet or make the last part a separate one,

something like: Remember that, while saving more fish eggs and young fish will likely lead to more adult

fish, scientists are still uncertain when they try to predict how many. One of my frustrations with the

Internet mode is that, in mail surveys, I like to limit bullets to two lines or two and a half at most and that

becomes very hard when using the Internet format. So far, I have not found a way around this

disadvantage of Internet.

Alan J. Krupnick

Slide 30

“Common sense...” I would say no. Common sense could refer to sustainable populations that are

considerably below current levels and preventing loss of adult fish here could merely lead to equal

numbers of death from some other direction.

Richard C. Bishop

Slide 31

This is a good start toward legitimizing choosing a cheaper alternative or no action. I would be even

more explicit, adding another bullet at the top: We want to know how you personally really feel about the

alternatives. (By the way, don’t be afraid to use underscoring to draw the eye toward a central theme in a

longer bullet.)

Alan J. Krupnick

Slide 31

Status Quo. Best practice, according to Adamowicz, is to portray the baseline as an additional column in

the diagram, rather than just checking off a status quo response. To do so requires stating the ecological

baseline, something that the survey team has not wanted to do so far. 

Richard C. Bishop

Slide 32

Maybe it’s just a matter of taste, but I am not sure your diagram helps much and may even confuse. In

this case, will they get the point of the light bulb? I would add a few words of text under the pictures. On

left “Cheaper Goods.” In the middle, “OR.” And on the right “More Fish.” And add a sentence of

explanation to the script.

Richard C. Bishop

Slide 33

Add to the script something like, “This slide shows the form that some important questions in the survey

will take. It shows two options, Option A and Option B. On the first row it shows how many fish would

be saved, 1.3 billion under Option A versus only 980 million under Option B.

Richard C. Bishop

Slides 34, 35, and 36

are okay as an introduction. Just tell the subject that more explanation will be coming soon. Otherwise,

many subjects are going to think they are supposed to understand and remember what they are seeing

here.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

43 2/06

Richard C. Bishop

Slide 38

Did focus group members do okay with this explanation of carrying capacity? Could you refer to levels

in 1950 or 1900 or prior to European settlement or something? 1950 would make this match with

changes in catch.

Richard C. Bishop

Slide 40

I am not sure about asking them to assume something out of the blue. Maybe you could soften it by

saying that “Scientist are still debating about exactly how current conditions compare to the 100% level,

but 60% would be a safe assumption for purposes of this survey. If no new regulations are placed on

cooling water intakes, fish populations are expected to continue at the 60% level in future years.” Did

people in focus groups ever ask what about stronger pollution laws or fishing regulations? It is tough to

know how to describe baseline with so many forces impinging on fish stocks, but do you dare say nothing

about the future of other regulations? Even worse, did anyone question whether cooling water regulations

are the cheapest way to improve things?

Richard C. Bishop

Slide 45

I am evolving into a slide show designer rather than a review, so let me stop here. I hope these comments

help to highlight why I think the slide show needs a lot of work. 

Kevin J. Boyle

Slide 40 

You do not tell respondents what will happen if they chose not to vote for either option. You have to

assume that that they assume that current conditions will be maintained, but some may assume further

declines given the information you presented earlier. You need to specifically tell respondents what you

want them to assume about the baseline. This applies to the other baseline conditions for other

attributes as well.

Richard C. Bishop

Slide 41

Be careful of the word “between” here. Do you mean to give a range of outcomes, between no

improvement and an improvement to 75% of historical levels? That is not what the cell in the matrix on

Slide 33 says. It seems to imply a point estimate, not a range of outcomes. Also there is a mismatch

between this slide and 33, where Alternative A leads to the 70% level. Or is 70% the point estimate for

the range between 60% and 75%? If so, this is very confusing and will not, in my opinion, help

underscore scientific uncertainty effectively.

Richard C. Bishop

Slides 42 thru 44

Similar comments apply. 

Richard C. Bishop

Slide 45

I am not sure that the point you are trying to make above the pictures will be apparent to a large share of

respondents. Better try a sentence or two more in the audio. The reference to water quality below the

picture seems strange. Several factors other than water quality affect fish, as you have already explained. 

Why not say current conditions in the Northeast? 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

44 2/06

Alan J. Krupnick

Slide 52

“Significant impacts.” One slide says that there “are no other significant impacts,” implying that the fish

kills are a significant impact. This may introduce bias.

Kevin J. Boyle

Slide 60 

It is possible that there will be some tax consequences if some of the costs are written off. This could

result in decreased tax revenues. I have found that a more believable payment vehicle allows prices and

taxes to change.

 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

45 2/06

VII. COMMENTS ON FULL TEXT OF STATED PREFERENCE SURVEY

James R. Kahn

Question 1 through 3 (Concern for policy issues)

These questions are right on the mark and accomplished the intended goal of framing the policy context.

Richard C. Bishop

Question 2

Having gone through the slide show, I am not sure I could answer this question correctly. I remember

cooling water is not the most important but .... I think you are asking for trouble here. What if 20% or

30% get this wrong? Would you conclude that they did not understand the material? The battery of truefalse questions I recommended earlier would serve you better. And don’t be too hard or tricky. Examples

of true-false questions: 1) Cooling water withdrawals are the most important single cause of declines in

fish populations in the Northeast. 2) Cooling water intakes are already regulated at many power plants. 

Etc. Then see how respondents do in the pretest.

Richard C. Bishop

Question 3

I think this question errors in the other direction. It is never a good idea to insult your respondents. 

When 95% get this right, what will you conclude, that that percentage was not asleep or drunk during the

slide show?

James R. Kahn

Section II Instructions and Questions

These represent a good set of instructions. Again, the report/memo of October 18 from Abt Associates

provides documentation that these instructions are well-understood by the sample population.

James R. Kahn

Questions 4 through 6

These are well-defined choice sets constructed in a fashion that is consistent with the literature. The

attribute levels are well-chosen, and provide sufficient variation to reveal WTP and to conduct basic

scope tests.

Richard C. Bishop

Question 4 and following

Two things: First, we are formatting this sort of question differently. Rather than leaving the no-action

alternative as one line at the bottom of the question as is done here, we are making it one of three

alternatives. We have three options and Alternative A is always the no-action-at-zero cost alternative. 

Options B and C are like the ones you have here. We also ask not only for their most-preferred, but also

their least-preferred alternative, so that we get a full ranking of the alternatives for each stated-choice

question. This is not fully tested and published approach yet, but we think it is promising. It gives a lot

of information about preferences. It raises the no-action alternative to the status of a full-fledged

alternative, rather than hiding it at the bottom, which may imply to some subjects that you really expect

most respondents to go for one of the alternatives in the matrix. And it would remind subjects of the

attributes of the status quo. Second, there is not much precedent for using monthly payments in statedpreference studies. Of course this is how we pay for lots of stuff, but most researchers worry about

coming across as a salesperson. “Do I have a deal for you! For a mere $2 a month you can save 450

million fish per year, increasing long-term fish populations to 65% of carrying capacity ....” I would go

with the annual payment only.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

46 2/06

James R. Kahn

Questions 7 through 9

Good debriefing questions.

Richard C. Bishop

Question 8

Question 8 could be converted to true-false.

Richard C. Bishop

Question 9

Question 9 is on the right track, but seems a little self-serving. Most people are going to know you expect

them to take the first choice. Here is one of the places where a survey expert could help you get at what

you want using more neutral questions.

Richard C. Bishop

Question 10

Same as Question 9.

James R. Kahn

Question 10

Needs an additional response category as follows:

___I thought there were impacts on ecosystems in addition to those created by changes in fish populations

and partially based my answers on these impacts.

Richard C. Bishop

Question 11 through 13

These are okay, but might work better with Likert scales, which work very well in an Internet survey

format. Ranking involves a larger respondent burden and may not yield answers that are as helpful when

you construct covariates. 

Richard C. Bishop

Question 16

Since they will be a long way beyond the slide show, you might insert a map here or use some other

means to remind them of what you mean by the Northeastern States. I like to use a longer time span in

questions like this (last three year or last five years). Also I would give them a range: never, once or

twice a year, etc. Some people seem to regard an open-ended question like this as requiring an exact

answer. My way will also be more amenable to the Internet format. Also, is fishing from a boat fishing,

boating, or both?

Richard C. Bishop

Question 17

Question 17 will also need to be reformatted and adapted for the Internet mode. You can ask open-ended

question, but it will slow data entry and cost more.

Richard C. Bishop

Question 18

Question 18 cries out for a Likert scale.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

47 2/06

Richard C. Bishop

Question 20-29

Don’t ask respondents to answer questions that they have already answered when they were enrolled in

the KN panel. Note also that town is an ambiguous term. In my home state a town is a rural unit of

government below the county level and outside of incorporated villages and cities. If you ask them about

numbers in their household, be sure to tell them whether they are to count themselves or not.

James R. Kahn

Questions

The rest of the survey consists of standard questions, some focusing on debriefing issues, some seeking to

measure intensity of preferences and some looking to define the demographic and socioeconomic

attributes of the respondent. There are no problems here.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

48 2/06

VIII. ADDITIONAL REFERENCES

Richard C. Bishop

A few are embedded above.

James R. Kahn

Boxall, P.C. W.l. Adamawicz, M. Williams and J. Louviere, 1996: A comparison of stated preference

models for environmental valuation Ecological Economics 19, 243.

Casey, James, 2006, Modeling of farmer preferences for agroforestry systems, in the Handbook of

Contingent Valuation, (Alberini and Kahn, eds.) Edward Elgar, 340-353

Carlsoonn, Fredrik, P Frykom and C. Lieejstope (2003) Valuing Wetland Attributes : An Application of

Choice Experiments, Ecological Economcsi 47, 95-103.

Farber, Stephen and B. Griner, 2000, Valuing watershed quality improvements using conjoint analysis,

Ecological Economics, 34.

Hanley, Wright and Adamawicz, 1998 Using Choice experiments to value the environment, Environ and

Resource Economics, 11: 413

Holmes, Thomas R and W.L. Adamowicz (2003) Attribute based methods: A Primer on Non-Market

Valuation, Champ, Brown and Boyle (eds) Dordrecht, the Netherlands, Kluwer Academic Publishers.

Kahn, James and Steven Stewart, 2006, An introduction to choice modeling for non-market valuation, in

A Handbook of Contingent Valuation, Kahn and Alberini, eds, Edward Elgar Publishers.

McFadden, 1974 Conditional Logit Analysis of Qualitative Choice Behavior in P Sarembeka (Ed),

Frontiers in Econometrics, New York: Academic Press.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-1 2/06

APPENDIX A

REVIEWER COMMENTS
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-2 2/06

PEER REVIEWER COMMENTS

Richard C. Bishop
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-3 2/06

PEER REVIEW OF 

“PEER REVIEW PACKAGE FOR WILLINGNESS TO PAY SURVEY INSTRUMENT FOR

§316(B) PHASE III COOLING WATER INTAKE STRUCTURES”

Comments Submitted by: Richard C. Bishop

I. GENERAL COMMENTS

Before beginning my comments, let me suggest that you take them with a grain of salt in the following

sense. I will definitely focus on potential problems, forgetting to tell you about all the things that are

done well. Much hard work is evident in what I saw here and it has produced a lot of progress. Also,

survey design is as much an art as a science and we who work in the field frequently disagree about best

practice. In the end, those working on this study will have to use their own judgment about which of my

comments and suggestions to focus on in final revisions before the pretest. I am kind of like a dentist

doing a checkup. Indeed, reading my comments may seem like a trip to the dentist! I like to probe

around, searching for the survey equivalent of cavities. I hope that some of my comments will help you

do an excellent study. But you will and indeed should decide to ignore others either because you are not

sure they are helpful or you can’t take the time or expend the energy to fix what may be low priority

problems. With these caveats, here are my general comments:

A. The work is being conducted in a total valuation framework, which is appropriate to the case. A

narrower framework that ignores nonuse values would risk neglecting most of the benefits of the

regulations. 

B. So far as I know, the researchers are correct when they suggest that no previous studies have

valued changes in fish stocks including forage, recreational and commercial species over a broad

geographic area. This will be a path-breaking study in that regard.

C. To some extent, I suppose I am passing the buck, but I would recommend that a survey expert be

employed as a consultant to help with this study. A heavy hitter like Stanley Presser or Don

Dillman or Jon Krosnick would be great if you can get one of them; another possibility would be

Nora Cate Shaeffer of UW-Madison. Nora is less well known in environmental economics, but

her credentials in survey methodology are impeccable. I have personally worked with her on

stated-preference studies (e.g., she is a co-author of the paper cited as Heberlein, 2005, in the

document I was sent). Such a person could help evaluate and remedy problems with the

instrument including those listed below. In addition, for reasons stated below, I am concerned

about the plan for addressing nonresponse issues. The sort of person I am recommending might

be able to help a lot.

D. I am particularly concerned that the slide show is not ready for prime time. In studies like this

one, conveying information is crucial and very difficult. A large number of nitty-gritty issues are

probed under Specific Comments. Let me raise some broader issues here.

a. The document that was sent to me for this review was for the most part very clearly

written. Most of the time, I was able to figure out what had been accomplished and what

the plans were. Perhaps the biggest shortcoming—if I missed it somewhere, I

apologize—was the lack of a description of the procedures that would be used in the slide

show. In what follows, I will assume that respondents will be required to view the slides

before they are allowed to open the survey proper; that the pace through the slide show

will be controlled through Knowledge Networks, and not at the discretion of the subject;

that the script in Attachment 2 will be applied as voice-over with each slide; and that
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-4 2/06

there will be no way for subjects to review slides after they have gone through the slide

show. I will come back to this last issue in a moment.

b. The slide show and the survey are currently planned as two separate steps. This is a

major part of the research strategy and merits very careful consideration. There is no

easy formula for conveying all the information needed by subjects in a study like this. 

Admittedly, there are some precedents in the literature for setting up the information as a

separate step, as noted in EPA’s write-up. A nice clean break between information and

survey questions may work better, but after considering this approach I think I still prefer

to mix information and questions together, which is how I am tackling the information

problem these days. In the Internet surveys I am working on, subjects are free to go back

and forth between frames. Information is presented in modest doses. A frame or two of

information is given and then a question pops up and they can go back to the information

if they want to. Indeed, properly framed questions are an incentive to look back if some

parts of the information are not yet clear.

c. Frankly, the amount of information to be communicates all at once seems huge. Given

detail in some of the slides, would it be unreasonable to assume it will take subjects close

half an hour (30 seconds per slide) to view the slide show? If so, that is half an hour

before they answer one survey question. The total process might take 45 minutes to an

hour. That is asking a lot even of Knowledge Networks subjects. If you do a separate

slide show, it should be cut to 15 minutes or 20 minutes maximum and the pace should be

set to be conducive to learning. Has having a separate slide show perhaps made it too

tempting to add more and more slides?

d. If you stick with the slide show, I would recommend that you make it interactive with the

survey so that subjects can go back and review information if they want. Try putting

links in the survey so they can go back to one or more slides and then return to the place

in the survey where they stopped. Also, they need to be able to turn the slide show on

and off. (What if the phone rings or the kids start yelling in the other room?) 

E. Again, I apologize if I missed it somewhere, but I was a little surprised to find that the slide show

and survey only referred to the Northeast. I would infer that subjects from different regions will

get versions tailored to their region. Maybe that would make sense, but I would be tempted to

explain it in the slideshow to avoid confusion. Tell them that this is a national policy but that

they need to be concerned only about their region since people in other regions will be paying

costs there. If a national sample will be considering only changes that are proposed for the

Northeast, then there is more explaining to be done. What about the fish in my region?

F. As a general comment on design, except for wandering out to Wisconsin and Oregon for a couple

of groups, all the qualitative work was done in a fairly tight area of the East. Why? I often find it

instructive to have a wider geographic range. Also, the focus group memo points out that while

people are generally more interested in their home regions, some care about other areas. 

Focusing on regions could cause some problems in this study. If I live in Maryland, for example,

is Maine more important to me than Virginia? If I care somewhat about other regions, shouldn’t I

be informed about them so I can include them in my values? (Perhaps I am not appreciating the

difficulty of expanding the information to cover the whole country. Also, limiting people to their

home region might be supported as a way of making value estimates conservative.) 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-5 2/06

II. CHARGE QUESTIONS

Question 1: Is the choice experiment survey an appropriate stated preference survey format for the

estimation of marginal willingness to pay for the characterized policies? Will the survey

likely provide useful insights into preferences, tradeoffs, and willingness to pay

associated with the described policies?

Comments: Provided that stated preference surveys are well designed and executed, they have wide

support for applications like this, both in the scientific literature, as cited in the materials

provided to this reviewer and by governmental agencies, including not only USEPA,

2000, as cited, but also the Office of Management and Budget, Regulatory Analysis.

Circular No. A-4. U.S. Office of Management and Budget, 2003. 

http://www.whitehouse.gov/omb/inforeg/circular_a4.pdf. Having worked in the field for

many years, I am convinced that stated-preference methods are capable of providing

useful insights into preferences, tradeoffs, and values in cases like this. The choice

experiments being designed here are in keeping with one of the major branches of the

literature that has many respected proponents.

Question 2: Does the effort used in developing and testing the proposed survey meet current

standards for stated preference surveys reflected in the literature?

Comments: Those designing the study display excellent knowledge of the scientific literature on

stated-preference methods; experts on the biological side have obviously been consulted;

attention has been given to theoretical issues; survey materials were developed; focus

groups were conducted and the survey materials revised; and a peer review is being

conducted now to gain additional input before the pilot study is implemented. Except

perhaps for gaining more input from an expert in survey methods (See General Comment

C above) and some specific questions raised below that may warrant further attention,

these procedures meet current standards.

Question 3: Can the policy options be sufficiently well defined such that the survey is likely to

provide useful insights into preferences, tradeoffs and willingness to pay?

Comments: The breadth of the impacts of I&E regulations in terms of both geography and affected

species makes this a challenging study. It is impossible to develop a scenario that

touches on all the aspects that may be of relevance to all subjects. And, like so many

other environmental economic studies, scientific uncertainty about both the baseline and

the effects of the regulations add to the difficulties. We need to recognize the limitations

of stated-preference methods in such a context. For example, if it were possible to tailor

surveys to the effects of the regulations in each subject’s home area, their answers might

change. Still, useful—albeit imprecise—insights into preferences, tradeoffs and WTP

can probably be gained with this study. More comments drawing on the results of the

focus groups might increase the level of confidence. Did most participants seem to “get

it” when they talked about what they were being asked to think through? Could most talk

meaningful ways about the tradeoffs involved and why they might care about them? Did

some members of the groups say they didn’t care? (Not everyone needs to care about the

fish affected by use of water for cooling and other such problems. I find it reassuring

when members of groups I am conducting have learned enough and felt comfortable

enough to say that they are not interested.)
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-6 2/06

Question 4: Does the information provided in the survey on the baseline ecological conditions, the

expected changes, and the expected costs and their incidence seem sufficient, appropriate,

unbiased and accurate? Does the survey provide appropriate and sufficient introductory

and debriefing questions to enable identification of protest responses and symbolic

responses, to test retention and understanding of information, etc.? Would you suggest

any additional debriefing questions?

Comments: As I have stated, I think the slide show can be improved, but the basic information on

baselines and changes is there and seems to be presented in an unbiased way. I have

suggested some improvements in the debriefing questions in my detailed comments on

the survey (see below), but they involve fine tuning, rather than major changes. The

introductory questions designed to assess whether the respondent understood the

information (now Questions 2 and 3) should be beefed up. I like to use a series of truefalse questions when I am trying to evaluate what people have learned from the

information provided. When I introduce the true-false questions, the instructions clearly

say that the goal is not to test the participant’s knowledge, like in school, but to test how

well we did in communicating information to them. This allows me to go into more

depth than Questions 2 and 3 in the current version of this instrument. And saying that

respondents averaged 80% or 90% in answering the questions (which is my goal in

information provision) is a good way to silence those who might otherwise try to argue

that people didn’t read or didn’t understand the information. Let me hasten to add that

not everyone agrees that this is a good idea. I think they are afraid of turning off

respondents, thus contributing to overall and item nonresponse. But I have checked this

approach out in focus groups and other qualitative research and so long as people

understand the intent, I do not detect much resistance. I say more about the specifics

below.

Additionally, Question 2 does not seem like a very good test for “Testing for Appropriate

Understanding of the Introductory Materials.” Slide 8 of the introductory materials states

that there are many causes of fish decline. Slide 20 says (emphasis in original) that

intake structures “are usually smaller than the effects of commercial fishing.” The voiceover for Slide 40 makes a vague reference to “a wide range of past human activities that

have caused fish stocks to decline.” Unless I missed something buried in the slide show,

respondents do not have a basis in the introductory information to answer Question 2. To

the extent that they get the ranking correct, they must be falling back on their general

knowledge.

Question 5: Does the survey sufficiently and appropriately characterize the policy options under

consideration? Does the survey place sufficient and appropriate emphasis on the scope of

resource changes? Does the survey appropriately communicate the scientific uncertainties

as well as uncertainties associated with the policies in question?

Comments: See my responses to Question 3.

Question 6: What are your expectations regarding the potential validity and reliability of benefit

estimates that might be provided by the proposed survey, compared to the current state of

the art in the stated preference literature?
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-7 2/06

Comments: I have stated some reservations about details both above and in the specific comments

below. Much attention to the design of the slide show and the survey is still needed, but I

think this study is well along toward being state-of-the-art for one that must deal with

such a broad regulatory initiative.

Question 7: Does the survey take appropriate steps to minimize hypothetical bias? Does the survey

place sufficient and appropriate emphasis on the budget constraint facing respondents,

and on potential substitute uses of their budget? Is the payment vehicle used in the survey

credible? Does the survey incorporate appropriate language to ameliorate strategic bias,

symbolic responses and ““yea saying””? Is there any sign of systematic bias in the survey

or presented information? If so, how can it be ameliorated?

Comments: Section 6.6.1 shows that hypothetical bias has been thoroughly considered and addressed. 

I agree with EPA’s reluctance to introduce a cheap talk script. In my judgment the

budget constraint and substitute reminders are more than adequate. Overdoing efforts to

provide budget constraint and substitute reminders risks insulting significant numbers of

subjects. (“What? Do you think I don’t understand that if I spend more money on this, I

will have less money for other goods? Do you think I am stupid?”) I discuss the

payment vehicle below (see specific comments relating to Page 21 paragraph 1). I think

the survey meets current standards in dealing with the other potential sources of bias

described in this question. Section 6.6.3 seems more than adequate in this regard.

Question 8: Will the answers to this survey lend themselves to straightforward economic

interpretation of willingness to pay? Are the proposed econometric modeling approach

and validity tests appropriate? Would you suggest an alternative modeling approach or

alternative or additional validity tests?

Comments: As I say below, I have some reservations about the payment vehicle, but otherwise the

results should be amenable to straightforward interpretation. The plans for analysis,

which involve not only a rather traditional approach, but also an effort to account for

heterogeneous preferences seems more than adequate for this stage in the project. 

Question 9: The national benefits analysis for Phase III of the 316(b) rulemakings must be completed

within the next several months and a final rule issued by June 1, 2006. Are there

improvements to the survey and associated model that would be feasible to implement

within the available time frame? If more time was available, how would your answer

change?

Comments: That is a very short time horizon to do the pilot, main survey, follow-up survey, and

analysis. You are likely to have to live with some warts and loose ends. More time

would allow more attention to the suggestions coming out of this and other peer reviews,

a thorough evaluation of the results of the pilot, a more careful statistical analysis of the

main survey results, and a lot more thought about the follow-up survey. I urge you to

add at least six months to the schedule.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-8 2/06

III. SPECIFIC COMMENTS

P. 6, paragraph 2: More advantages of the Internet mode might be mentioned. Four come immediately to

mind. First, mail surveys can run into difficulties with skip patterns that can easily be programmed into

an Internet survey. Second, where a number of stated-choice questions are to be asked back-to-back, it is

possible to tailor later questions based on answers to earlier ones. Suppose a respondent chooses

Alternative A over Alternative B when costs are $15 and $5, respectively. It would not be very useful to

ask that same respondent a second question where attribute levels are the same except that Alternative A

has a cost level of $10. Mail surveys cannot be tailored leave out such redundant questions. Third, voice

over to help respondents through complicated or extra long materials can be added. Fourth, links to more

detailed information can be added for those respondents who think they need it without complicating the

main presentation for everyone. Also, regarding existing studies, do you want to mention the Viscuzi et

al. (2004) study on water quality that was done for EPA and which was cited later in this document? I

believe an Internet mode was used there. 

P. 11, paragraph 3: It is probably worth mentioning explicitly here or elsewhere that the KN panel

includes people who do not have Internet access. The WebTV option is mentioned, but not explained. 

This is a concern that, for example, Dillman (2000) mentions with regard to Internet surveys and I think

KN has covered this one pretty well. Also, those who complete surveys are rewarded in an effort to

reduce attrition. 

P. 12, paragraph 2: This paragraph is hard to follow. For example, “Respondents will receive an

incentive fee of $5 to $10 for completing this survey.” Which survey, the follow-up or the main survey

or both? 

P. 12, paragraph 3: It is better not to promise too much. “The survey administration format will be

designed to ensure that individuals who agree to participate in the follow-up interviews must review all

background information before completing the survey.” I don’t know of any way to make sure people

actually read and digest such information. Also, here and elsewhere it is a little misleading to call these

surveys interviews. Interviews imply person-to-person contact, either in person or at least over the phone. 

Unless I missed something, only the Internet will be used.

P.12, paragraph 4: At least for the studies I am involved in, regular KN panelists do not receive direct

cash payments, but rather points that can be redeemed for rewards. Have special arrangements been made

to provide cash payment is this case? If so, it would be worthwhile to test for the effects of cash versus

points.

P. 12, paragraph 5: What is EPA ICR #2155.01? I am not sure I understand the difference between

cognitive interviews, think aloud analyses, and verbal protocol analyses. It is always hard for an outsider

to evaluate what happened in such sessions, but at least the exact procedures could be spelled out. Where

were they held? How were subjects recruited (e.g., were people from a wide range of socioeconomic

backgrounds successfully recruited)? Who moderated? What procedures were followed in each group or

set of groups (e.g., how did the subjects “complete the survey,” on-line or paper versions)? Did

participants often say things that indicated that they hold use and/or nonuse values for the proposal? If

so, what sorts of reasons did they give? Since participants filled out the survey, how about some tables

with results?
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-9 2/06

Pp. 18 and 19 and elsewhere: Let me say at the outset that—pending clear evidence to the contrary—I

find arguments for the representativeness of KN panels convincing. Plans for this study relating to

adjustments in the sample for the main survey seem sound to me. On the other hand, there are those who

are holding fast to the conventional wisdom that validity requires probability sampling and a high

response rate. Folks at OMB are a case in point and, as you are well aware, they continue to be all

powerful. In this context, the follow-up survey plan does not seem to me to be sufficiently convincing to

carry the day. A total of 600 people from various stages of the recruitment process will be sampled and

asked to complete an Internet survey. It seems unrealistic to hope for much of a response from people

who have refused to get involved with Knowledge Networks in the first place and an incentive of a few

dollars is not likely to help. Not only are they likely to be put off by the new contact but many of them

will not have an Internet connection. And not one citation is provided to bolster confidence in what is

being proposed. Has KN assured you that they have successfully done this before? If so, a few statistics

would be incredibly helpful here. 

This is a thorny problem to be sure. It was one of my reasons for recommending the employment of a

prominent survey expert (see part C under General Comments). Dealing with nonresponse bias has, of

course, been an issue in survey research from the beginning. How have survey experts addressed the

problem? How can this study capitalize on what has been learned in the past? Beyond that, we need to

continue to hammer away at the simplistic emphasis on response rates and focus attention on the basic

issue: representativeness. It is also important to stress that there is no such thing as a perfect sample or

survey mode. A good starting point is Krosnick’s “Survey Research” in the 1999 issue of the Annual

Review of Psychology (http://www.unt.edu/rss/class/survey/survey_review.pdf). He points out there (p.

451) that “when probability sampling methods are used, it is no longer sensible to presume that lower

response rates necessarily signal lower representativeness.” He even cites studies where a higher

response rate caused bias. None of the studies he cites fit your needs here exactly, but there is a lot of

food for thought. I understand that Krosnick brought a similar message to a recent EPA workshop. 

I don’t claim to have a complete solution, but here are a couple of ideas. First, one thing we are doing is

adding questions from a general population survey dealing with environmental attitudes. You could add

some of these to the main survey. Then, you can show that your KN respondents match up with national

demographics, but also hopefully that they match up with the environmental attitudes of the general

population. Sorry I don’t have an exact citation at my finger tips for you, but I can get it for you if you

want it. 

Second, and this builds on the first, rather than hoping to get meaningful results from nonrespondents on

the whole survey using the Internet, you could try doing an abbreviated telephone survey. I have done

this with nonrespondents to a mail survey by calling them up, noting that they had not responded to the

survey, and asking them if they would be willing to answer a few questions. Some said that they would

still be willing to fill out the survey and, if so, we ask whether they needed a new copy. Whether they

said they would fill it out or not, everyone who was willing to participate “for just a few minutes” got the

short list of questions including some on socio-demographics and environmental attitudes. Then we were

able to adjust for nonresponse using the telephone survey responses along with the main data set. 

Obviously, your situation is somewhat different, but it seems like this approach could be adapted.

Page 21, paragraph 1 and elsewhere: I am somewhat concerned about the cost of living vehicle. Are

there any precedents for using this wording in the literature? What does it mean exactly, and how does it

relate to conventional definitions of willingness to pay? How would cost of living work into a traditional

indirect utility function like that on p. 23? We are, of course, looking for a reduction in income that

would wipe out the gain in utility from a given policy. Will respondents understand that point? After all,
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-10 2/06

I only have so much income. If my cost of living goes up, it doesn’t mean that I will have that much less

money to spend on goods and services (unless I reduce savings by an equivalent amount). Cost of living

has to do with not being able to buy as many goods and services. I suppose one might fall back on the

idea that an increase in the cost of living is a reduction in real income, but I am not sure the average

citizen thinks in those terms. I think going with “Increased cost to you household because of higher

prices” or something like that would be safer. The focus group memo is reassuring in this regard. 

Subjects seemed to understand that the choice questions were referring to costs to them. Nevertheless,

why not be completely explicit by referring to “the prices you pay for goods and services produced at

factories and power plants that use cooling water from ocean, lakes, and streams”?

While on this topic, let me also question the effort to convince subjects that taxes won’t increase. Given

that prices and taxes are often combined in payment vehicles, I am not sure what the rationale is for

avoiding taxes here. Perhaps it is an attempt to avoid vehicle bias. Yes, some vehicle bias may creep in,

but I still like to include taxes in a case like this for two reasons. First, that taxes would increase may

seem more plausible to subjects. After all, the government buys lots of these goods too (e.g., paper and

electricity). Is it plausible that my household costs would go up but the costs of government would not? 

Surely either taxes will go up or some people, at least, will suffer losses in government services. Second,

dislike of taxes may lead to some vehicle bias, but it also serves as a wakeup call for subjects. Here is a

cost that would be real and that you can’t avoid. If someone tells me that they are willing to pay more

taxes for an environmental improvement, I am more inclined to believe them. Also the direction of the

bias, if any, is known. (Admittedly, choice of payment vehicle is something that various researchers in

this area disagree about. It is still very much a judgment call. I am just giving you my opinion.)

Additionally, pages 15 and 16 of the focus group memo do not provide much evidence that taxes would

be problematical in this case. 

Same paragraph: The statement about not adding up or comparing programs is probably okay, but I am

not sure your average person will get the point. I like to be more direct with something like, “At this

point, it is not clear what the alternatives will actually cost, so we are asking about a range of costs. Also,

we are asking about a range of effects so that our results can be adapted as scientists learn more.” 

Pp. 30ff, Section 6.5: Experimental design is an important step in choice experiments. I am not an expert,

but from what I know about it, this step has been more than adequately addressed in this section and in

Attachment 4. My only suggestion is that perhaps there are too many levels of the attributes. Unless you

saw clear focus group evidence to the contrary, I would be tempted to drop 8% for the population change,

2% for catches, and $2 and $4 for costs. For this kind of broad policy, I find it hard to believe that people

will be that sensitive. (Below, in discussing the survey, I recommend that you not use monthly costs. If

you use annual costs as the main cost attribute, maybe my advice about cost levels is less relevant. As it

is, you are talking about pocket change.) Remember that if you can bracket the eventual outcomes you

want to value in the benefit estimates, interpolation is an option.

Pp. 37ff, Section 6.6.3: This section is not always easy to follow, but it seems to make the question of

scope harder than it needs to be. The internal scope test is adequately described except that on p. 38,

paragraph 2, EPA states that the internal scope test assesses “sensitivity to scope across choices of a

single individual.” Surely looking at regression coefficients on attributes for the expected signs and

significance does not involve single individuals. This is an internal scope test only because each

respondent sees more than one stated-choice question. Scope-test purists worry that having thought about

earlier questions will influence how respondents look at later ones. This is easier to see with a simple

contingent valuation exercise where asking a single set of respondents two questions where the scope of
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-11 2/06

the environmental improvement is varied would be considered an internal scope. It would be “weaker”

because such respondents might take the question about the small change as a cure that they should be

willing to pay more for the large change or vice versa. The same problem—if it is a problem—could

occur as subjects are asked a number of stated-choice questions, thus making the scope test using all the

stated-choice questions an internal test. Particularly in an Internet survey (where you can control the

order of question presentation and be notified when respondents skip the first stated-choice question or

come back and change their answers), one ought to be able to design the first question of each version of

the survey so that responses can be used in a split-sample scope test. Provided sample sizes are large

enough, the coefficients from the random utility model run on various first questions could be checked for

sign and significance. As I said, I am not sure I follow the scope test as planned in this session, but my

approach seems simpler.

Comments on slides:

A. Slide 9: Thoughtful subjects will see a glitch in your story. Here you say ecosystem

functions are affected. Later you say its only fish and all other effects are negligible.

B. Slide 10: I am not sure this diagram will make much sense to subjects at this stage. Also,

“commercial facilities” seems vague. Could you say something like “paper mills, factories,

and power plants?” Perhaps you could even combine this slide and the next one.

C. Slide 12: “Aquatic organisms”? I thought this was about fish. Also the diagram might be

explained with a couple of sentences in the audio. This is a very important slide. What does

“intake flow” mean? What is this boxy looking thing in the middle of the picture, what I will

call the “screen” here? Perhaps you should label the pipe walls. What is a “juvenile”? Is this

really about juveniles and adults or large fish and small fish? Do fish that are small as adults

only get impinged and not entrained? Seeing the fish next to the screen labeled “adult” seems

to imply that big (“adult”) fish just turn and swim out. Nothing seems to be caught on the

screen. Given the importance of what is being said here, you may need a better diagram

unless focus group participants were very clear on what this diagram portrays. Are eggs only

“fish” eggs? If so, they should be labeled as such. Did no one in the focus groups raise

questions about other organisms like plankton? If so, you may need to clarify why such

effects are negligible. Does fish mean only finfish as implied here or are shellfish affected,

say through destruction of eggs. If not, you may need to tell people why. Do you mean to

imply that “screens” do or do not work? If not, why not? Is there any possibility of inserting

a video clip here involving either animation or a real intake structure?

D. Slide 17: The second sentence of audio needs to be more clearly written.

E. Slide 18: In the last line of the script, “is” should be “are”.

F. Slide 19: Not sure I follow the biology here. Take striped bass, for example. Presumably

fish caught by fishermen are from the adult population. Do you mean that 1.8 million adult

striped bass are lost or are sufficient sub-adult bass lost to I&E to equal that many adults or

are you comparing apples and oranges (bass of all sizes in intake structures versus adult bass

taken by fishermen)? The last possibility seems a bit misleading.

G. Slide 20: I wouldn’t pick on commercial fishermen here, particularly given that recreational

and commercial catches are combined in the preceding slide. Why make yourselves targets

of industry needlessly? In the next sentence, “it has contributed to a measurable decline in

some fish stocks, in some locations.” Sounds like government-speak to me. Subjects are

likely to ask, what about the species that matter to me? Can you tell them more?

H. Slide 21: I don’t like the term “technologies” here and elsewhere. I don’t know what it

means to people. And “cooling water intake structure” and “air burst system” are not much

better. A first cut at alternatives might include “equipment,” “structures used to pull water
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-12 2/06

from water bodies for cooling,” and “equipment that uses blasts of air.” I hope the picture is

clearer in the actual slide show. And those “aquatic organisms” get mentioned again.

I. Slide 22: “Close-cycle cooling”? What does that mean? Also this looks like you are mixing

information. What does total fish losses have to do with different “levels of technology”? I

would work the total losses of 1.8 million fish into the pie chart on the next slide and explain

the pie chart in the audio. Again, this is worth some time given how fundamental it is to the

goals of the study.

J. Slide 23: I don’t follow the diagram with the striped bass. What do the grid and the X mean?

K. Slide 24: Better explain in a sentence or two what a closed-cycle cooling tower does and

how it works. Again the diagram (bass, arrow, X) may not communicate. Also, I’d give

them a sentence or two explaining the pie chart. 

L. Slide 25: I have the impression from somewhere that the already regulated facilities are only

power plants. If so, say so, and indicate that other kinds of facilities as well as some power

plants are not currently regulated.

M. Slide 26: for the second bullet, I would say something like, “increase the costs of producing

goods and services, including those you consume.”

N. Slide 29 and later ones: I am not sure about the terms “policy” and “policy options” here. It

sounds too much like government speak. I would mention “regulations of factories and

power plants that use cooling water from water bodies” when I absolutely had to (maybe a

little thought would produce something shorter). Otherwise, I would use “regulations” when

I thought I could get away with it and otherwise maybe “regulation of industries that use

cooling water.” 

O. Slide 30: Good. This starts to take care of why different combinations of attributes will be

presented. Reminders may be needed later. I would try to shorten the third bullet or make

the last part a separate one, something like: Remember that, while saving more fish eggs and

young fish will likely lead to more adult fish, scientists are still uncertain when they try to

predict how many. One of my frustrations with the Internet mode is that, in mail surveys, I

like to limit bullets to two lines or two and a half at most and that becomes very hard when

using the Internet format. So far, I have not found a way around this disadvantage of

Internet.

P. Slide 31: This is a good start toward legitimizing choosing a cheaper alternative or no action. 

I would be even more explicit, adding another bullet at the top: We want to know how you

personally really feel about the alternatives. (By the way, don’t be afraid to use underscoring

to draw the eye toward a central theme in a longer bullet.)

Q. Slide 32: Maybe it’s just a matter of taste, but I am not sure your diagram helps much and

may even confuse. In this case, will they get the point of the light bulb? I would add a few

words of text under the pictures. On left “Cheaper Goods.” In the middle, “OR.” And on the

right “More Fish.” And add a sentence of explanation to the script.

R. Slide 33: Add to the script something like, “This slide shows the form that some important

questions in the survey will take. It shows two options, Option A and Option B. On the first

row it shows how many fish would be saved, 1.3 billion under Option A versus only 980

million under Option B.

S. Slides 34, 35, and 36 are okay as an introduction. Just tell the subject that more explanation

will be coming soon. Otherwise, many subjects are going to think they are supposed to

understand and remember what they are seeing here.

T. Slide 37: The script talks about scales while the pictures talk in terms of percents. You need

to be consistent. My guess is that people will grasp percents quicker than scales, particularly

scales with large numbers of points like 100. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-13 2/06

U. Slide 38: Did focus group members do okay with this explanation of carrying capacity? 

Could you refer to levels in 1950 or 1900 or prior to European settlement or something? 

1950 would make this match with changes in catch.

V. Slide 40, I am not sure about asking them to assume something out of the blue. Maybe you

could soften it by saying that “Scientist are still debating about exactly how current

conditions compare to the 100% level, but 60% would be a safe assumption for purposes of

this survey. If no new regulations are placed on cooling water intakes, fish populations are

expected to continue at the 60% level in future years.” Did people in focus groups ever ask

what about stronger pollution laws or fishing regulations? It is tough to know how to

describe baseline with so many forces impinging on fish stocks, but do you dare say nothing

about the future of other regulations? Even worse, did anyone question whether cooling

water regulations are the cheapest way to improve things?

W. Slide 41: Be careful of the word “between” here. Do you mean to give a range of outcomes,

between no improvement and an improvement to 75% of historical levels? That is not what

the cell in the matrix on Slide 33 says. It seems to imply a point estimate, not a range of

outcomes. Also there is a mismatch between this slide and 33, where Alternative A leads to

the 70% level. Or is 70% the point estimate for the range between 60% and 75%? If so, this

is very confusing and will not, in my opinion, help underscore scientific uncertainty

effectively.

X. Slides 42-44: Similar comments apply. 

Y. Slide 45: I am not sure that the point you are trying to make above the pictures will be

apparent to a large share of respondents. Better try a sentence or two more in the audio. The

reference to water quality below the picture seems strange. Several factors other than water

quality affect fish, as you have already explained. Why not say current conditions in the

Northeast?

Z. I am evolving into a slide show designer rather than a review, so let me stop here. I hope

these comments help to highlight why I think the slide show needs a lot of work. 

Comments on the survey:

A. Q2: Having gone through the slide show, I am not sure I could answer this question correctly. 

I remember cooling water is not the most important but .... I think you are asking for trouble

here. What if 20% or 30% get this wrong? Would you conclude that they did not understand

the material? The battery of true-false questions I recommended earlier would serve you

better. And don’t be too hard or tricky. Examples of true-false questions: 1) Cooling water

withdrawals are the most important single cause of declines in fish populations in the

Northeast. 2) Cooling water intakes are already regulated at many power plants. Etc. Then

see how respondents do in the pretest.

B. Q3: I think this question errors in the other direction. It is never a good idea to insult your

respondents. When 95% get this right, what will you conclude, that that percentage was not

asleep or drunk during the slide show?

C. Q4 and following: Two things: First, we are formatting this sort of question differently. 

Rather than leaving the no-action alternative as one line at the bottom of the question as is

done here, we are making it one of three alternatives. We have three options and Alternative

A is always the no-action-at-zero cost alternative. Options B and C are like the ones you

have here. We also ask not only for their most-preferred, but also their least-preferred

alternative, so that we get a full ranking of the alternatives for each stated-choice question. 

This is not fully tested and published approach yet, but we think it is promising. It gives a lot

of information about preferences. It raises the no-action alternative to the status of a full-
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-14 2/06

fledged alternative, rather than hiding it at the bottom, which may imply to some subjects that

you really expect most respondents to go for one of the alternatives in the matrix. And it

would remind subjects of the attributes of the status quo. Second, there is not much

precedent for using monthly payments in stated-preference studies. Of course this is how we

pay for lots of stuff, but most researchers worry about coming across as a salesperson. “Do I

have a deal for you! For a mere $2 a month you can save 450 million fish per year,

increasing long-term fish populations to 65% of carrying capacity ....” I would go with the

annual payment only.

D. Q8 could be converted to true-false.

E. Q9 is on the right track, but seems a little self-serving. Most people are going to know you

expect them to take the first choice. Here is one of the places where a survey expert could

help you get at what you want using more neutral questions.

F. Q10: ditto.

G. Q11through Q13: these are okay, but might work better with Likert scales, which work very

well in an Internet survey format. Ranking involves a larger respondent burden and may not

yield answers that are as helpful when you construct covariates. 

H. Q16: Since they will be a long way beyond the slide show, you might insert a map here or

use some other means to remind them of what you mean by the Northeastern States. I like to

use a longer time span in questions like this (last three year or last five years). Also I would

give them a range: never, once or twice a year, etc. Some people seem to regard an openended question like this as requiring an exact answer. My way will also be more amenable to

the Internet format. Also, is fishing from a boat fishing, boating, or both?

I. Q17 will also need to be reformatted and adapted for the Internet mode. You can ask openended question, but it will slow data entry and cost more.

J. Q18 cries out for a Likert scale.

K. Q20-29, don’t ask respondents to answer questions that they have already answered when

they were enrolled in the KN panel. Note also that town is an ambiguous term. In my home

state a town is a rural unit of government below the county level and outside of incorporated

villages and cities. If you ask them about numbers in their household, be sure to tell them

whether they are to count themselves or not.

IV. ADDITIONAL REFFERENCES

A few are embedded above.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-15 2/06

PEER REVIEWER COMMENTS

Kevin J. Boyle
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-16 2/06

Comments on:

“Willingness to Pay Survey Instrument for §316(b) Phase III Cooling Water Intake Structures”

By:

Kevin J. Boyle

Professor and Head

Department of Agricultural and Applied Economics

Virginia Tech

February 2, 2006

Answers to the Peer-Review Questions:

1. The choice experiment is an appropriate methodology and is the best methodology available to

estimate values for the Phase III policy. 

The survey results will provide useful information, but perhaps not as strong as asserted in the peerreview document (see my comments below).

2. The design effort does meet current standards, but the inferences from this design process are too

strong (see my comments below).

3. Yes, the policy options can be sufficiently well defined. I have made some specific recommendations

that should be addressed and the limitations/assumptions should be more explicitly stated.

4. The ecological condition information is good. Respondents need to be told what to assume about

baseline conditions. I presume that you want them to assume that current conditions will be

maintained if they do not choose a policy alternative but this in not clear in the information presented

to respondents. Limitations, like no information on specific species, need to be explicitly stated and

recognized in the study documentation. 

I think the information is well balanced. 

I have noted below that I am skeptical of debriefing questions in general and do not have any specific

recommendations.

5. I find this question hard to answer. First, the ultimate ecological effects are not known. Thus I do not

know if the selected baseline conditions and attribute levels will be able to address the actual policy

question. Second, while the analysis discussion talks about linear and quadratic specifications, with a

small number of attribute levels the best that you may be able to estimate is a piece-wise

approximation to a nonlinear function, which leaves question if the actual policy conditions fall on

intermediate levels. 

I think the scope of the change is well communicated with words, numbers and graphics, except for

the region of focus. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-17 2/06

Uncertainty is not addressed well (see my comments below).

6. I believe that with further revisions this will be a state-of-the-art instrument and data collection

process. I have concerns that the data collection process may not end up being as sophisticated as

proposed. The data analysis section is too vague at this point to address whether this portion of the

study will be state-of-the-art.

7. The survey does a good job of addressing hypothetical bias, but this bias is not controlled to the

extent that the review document claims. One additional improvement would be to make the decision

rule by which the policy will be provided more explicit. 

There is sufficient emphasis on the budget constraint, but I wonder how much it matters with such

low cost amounts. In fact I am concerned that the coefficient on the cost variable may not ultimately

be significant so the pilot study is imperative to test this hypothesis. 

The payment vehicle is credible, but it may be better to acknowledge that there may be some tax

effects. 

The study does the best it can to address strategic bias, symbolic responses and yea saying, but the

design choice experiments opens up the possibility for all of these problems. Thus, I do not think that

the controls are as strong as the study designers claim (see my comments below). 

I do not see a systematic bias in the information presentation, but do make some suggestions for

improvements.

8. The study results can lead to “straight forward economic interpretation” if the sample weighting

issues are resolved and state-of-the art estimation techniques are used. Neither of these points is clear

in the current documentation and I say more on these points below. 

There is only one validity test that I see in the design and that is the external test of scope, which I

believe is fundamentally flawed (see my comments below).

9. I do not believe the study can be completed by June 1! Yes, if there was 9 months to a year available.

General Comments:

1. There are many global statements about valuation and the design of this specific study that are purely

assertions and are not based on any hard empirical research or theoretical foundations. These

statements detract from what is really a basically sound research design. Examples of the former are

statements with respect to the incentive compatibility of choice questions and hypothetical bias, and

examples of the latter are extrapolations from what was learned from the focus groups.

2. The entire discussions of protest responses, hypothetical bias, and other problematic responses appear

to select items from the literature at face value. The key point is that you did try to address these

issues, but you probably have not in the design nor will not be able to in the estimation do as much to

control for these problems as you imply in the review document. The key point really is how much

subsets of people in these categories actually affect your welfare estimates. My reading of the

literature is that the only major concern in this regard in hypothetical bias and you are going to do as

much as you can to control this issue, but not to the extent you claim.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-18 2/06

3. You talk about a sample design that includes multiple weighting schemes and there is no discussion

about how you are going to resolve these different levels of weights nor how you are going to

investigate how weights affect your welfare estimates. One key issue is that you, like many others,

assume that making the sample look like the U.S. population will solve all nonrespone biases, but if

sample selection arises from something other than Census demographics you have not corrected the

problem, you have an endogenous weighting problem.

4. There is no discussion of how you are going to add up the values from different regions to get a

benefit estimate for national policy. What if respondents value an area larger than their sampling

region? What if respondents think that people from other regions will pay for their region? See my

comments below.

5. You assume that the type/mix of fish does not matter to people. This is a strong and perhaps

necessary assumption. You must make it explicit because information on specific species while

substantially increasing the information load for respondents may also change values estimates.

6. I think this is a well designed study, but there is definitely a need for improvement. While

considerable effort has gone into the scenario design, the design is not as tight as the authors of the

review document purport it to be. You have done a very good job, but by over-selling your design

you take away from the good features and make the entire design more controversial.

Specific Comments:

p. 6, 1st complete para. – I have concerns about the 600 follow-up interviews with nonrespondents. No

goal was set for the response rate. These are people who have already rejected the mode of

implementation of the survey or the specific survey for a variety of reasons. What is being provided as

incentive to get these people to respond? If you meet the target of 600 completed interviews, but have a

nonrespondent sample response rate of 10-20 percent, then this information will be of limited use; you

have just added an additional sample selection problem. In addition, if you find a difference in values

you will not be able to identify whether you have a sample selection effect, mode effect or both effects

causing the difference. While you are making the follow-up as similar as possible to the original survey

implementation it is not identical.

p. 6, 3rd para. – Where are you going to obtain “historical response rates of individuals in that group for

similar types of surveys”? You make the argument in this report, which I agree with, that no work of this

type exists. This assertion nullifies you ability to make such an adjustment in the sample design.

p. 7, top – I think the objectives are good and appropriate for the policy issue you wish to value.

p. 10, bottom – It is good and necessary for welfare calculations to have an opt-out option in the choice

questions, not all choice question designs include this design feature.

p. 11, 1st para. of Section 3.1 – I do not believe that the last sentence of this paragraph is true. A panel

member could give access to another member of their household without the survey administrator’s

knowledge. You have no more control here than you do with a mail survey; they recipient can allow

another to respond. 

p. 12, top – What if computer technology is the reason people opted not to participate in the panel? This

follow-up procedure will do nothing to bring this type of person into the study.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-19 2/06

p. 13, 1st full para. – In assessing the design it would have been much more helpful if the text reported the

issues/concerns identified and how they were specifically addressed in the survey design. This type of

claim is not helpful and only raises questions in my mind.

p. 15, bottom of 1st incomplete para. – What basis do you have for identifying “groups that tend to

provide inconsistent or invalid answers” to attribute-based choice questions? I do not know of any

information in the literature to support this sample design.

p. 15, Table 2 – How does this stratification by region compare to the percent of the national population

living in each region?

p. 17, Table 3 – The 90 percent confidence intervals for a “typical binary variable” are meaningless in this

context; what we care about is the error bounds on the welfare estimates derived from conjoint responses.

p. 18, 1st para. – I disagree with the statement that the “use of such methods to identify protest responses

is well-established in the literature.” Some studies do nothing to identify protest responses, others use

questions as you are proposing and others use statistical procedures to identify data outliers. There is no

established procedure and all are ad hoc. There is nothing wrong with doing what you propose it is just

that the conceptual and empirical foundations are not as solid as you portray them. My experience

suggests that a carefully designed protest protocol will result in many respondents providing

contradictory evidence, a single person can provide information that indicates a valid response while

another of their responses may indicate a protest; what do you do with this respondent.

p. 19, 2nd para. – I do not know of any articles in the literature where a Heckman selection model has been

merged with mixed logit (p. 25, 2nd para.). This is an important estimation issue you will need to address

as you will need to consider the correlated responses of individual respondents and the literature is

strongly supporting heterogeneous preferences through random parameters. You can not run a Heckman

on a simple specification and then address these issues in a second model, which seems to be implicitly

implied in the write up.

p. 21, bottom of 1st incomplete para. – It is good to tell respondents to consider each question

independently, but this is not likely to be sufficient (see Holmes and Boyle, Land Economics, Vol. 81,

No. 1, 2005). This interaction with how respondents answer the sequential questions leaves questions

about what statistical information should be used for policy purposes.

p. 21, bottom – The certainty questions of Champ et al. (2004) are not very useful. It will always be

possible to map two welfare estimates through some type of integer or continuous preference scale. The

question that remains is whether the specific point on the uncertainty scale is unique to the specific study

or can be generalized to other types of applications. The Champ study is silent on this issue. There is no

clear approach in the literature to “successfully control for hypothetical bias.” This does not mean that

you should not try, it just means that you may not make the right adjustment and should not make the

claim that you have adjusted for this bias; your resulting value may overestimate or underestimate

welfare.

p. 23, top – I suggest in considering data quality you consider some of the information that Berrens and

colleagues have explored with internet surveys, e.g., time spent viewing slide show, time spent answering

the valuation questions, etc. This information could be interacted with respondent characteristics that you

are collecting, e.g., environmental group membership, education, etc.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-20 2/06

p. 25, 2nd complete para. – This is the type of model you need to use and I expected it to be stated more

explicitly earlier in the text.

p. 29, top – I am confused about how the “baseline population levels” are going to be implemented in the

survey. I see on page 31 (bottom) the levels. Does this mean that the baseline will only be varied for the

“Population Change” attribute? Is this part of the orthogonal design? I seem to be missing something

from the explanation in the text.

p. 30, top – I suggest considering alternative-specific constants in the model. There are many variants in

the published literature, but I have one specific recommendation. This recommendation is a binary

variable that equals “1” if one of the two policy alternatives is chosen and “0” otherwise. This will give

you an idea of whether people are “in the market” for reasons other than the attributes of your design that

you explicitly control for in the estimation. There are a variety of reasons why the coefficient on this

variable may be positive and significant and you cannot identify specific effects, but potential

explanations include all of the reasons a person might be “incorrectly in the market”. You could try

interacting some of your protest probe variables with this ASC.

p. 31, attribute levels – The dollar amounts seem trivially small. These are unlikely to impinge on most

respondents’ budgets (p. 34 2nd para.), which makes me skeptical of discussions about how focus group

respondents indicated that they are considering their budget constraints (p. 34, bottom). You really need

the pilot to make sure the estimated coefficient on cost will be significant.

p. 32, 1st para. – It is good to exclude dominated alternatives.

p. 34, 1st complete para. – The incentive compatibility you refer to for a referendum format applies to a

single, binary choice, not the application you have where respondents choosing over two alternatives and

do nothing and answering multiple, sequential choice questions. A single choice question with three

answer options does allow for free riding and other strategic behavior, as does answering a sequence of

choice questions.

p. 34, 3rd para. – Increasing the “consequentiality of choice” is important to the design, especially in

regard to hypothetical choice, but may lead to protest responses as well.

p. 35, 1st para. – The statement that EPA finds little evidence to suggest that hypothetical bias should be

significant in the proposed survey instrument” is naive and silly. The empirical evidence is quite clear

that stated-preference methods can provide useful policy information, but that value estimates tend to be

overstated relative to actual monetary transactions. In recent work that I have done with Taylor and

Morrison, which is funded by EPA, we find that welfare estimates from a three alternative attribute-based

experiment can be 20 to 40 percent higher that comparable cash values. More importantly, only 5 percent

of respondents entered the market when cash was involved and over 60 percent of respondents entered the

hypothetical market. This is not an issue that you can design or assert away.

p.36, slide 52 – I think that you can still expect people to think about more than fish. I think people

realize that birds eat fish and there are other ecosystem effects. The ASC that I suggested above is one

way to partially control for these and other unwanted considerations by respondents.

p. 37, Section 6.6.3 – I do not think that you have a true external test of scope. If your scope test follows

the regular choice questions, which comprise your internal test, you have already “trained” respondents to

consider the differences and this is not a true, external, between-subjects test. However, I am not
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-21 2/06

convinced that the external test of scope is necessary.

p. 40, slide 53 – I believe addressing uncertainty is important and too many stated-preference studies

overlook the fact that environmental outcomes are not known with certainty. Having said this, I am

concerned that your treatment of uncertainty may raise more problems than it helps. I understand the

issue of cognitive overload, but other studies have incorporated specific information about uncertainty in

the design, especially those valuing health outcomes. Your current design raises the issue without any

focusing resolution so that each respondent is left to make their own assumptions regarding the likelihood

of outcomes from the Phase III policy. Your “Champ style” question convolutes uncertainty about

outcomes with uncertainty about responses. It is possible to have a scenario when outcomes are known

with certainty and respondents are still uncertain about their response and vice versa.

p. 51, Informational Slide Show – I believe this is a very good mechanism for conveying the background

information.

Cover page of slide show (slide 1) – You state that it is a survey of northeast residents, but it is far from

clear whether respondents should just value the northeast or all of the U.S. Consider the following

points in the slide show.

• Slide 2 – You say “protect fish stocks in the Northeast US waters.” What if people take trips to

fish in other regions of the U.S. care about what happens there? For example, when I have gone

fishing on the Great Lakes I know that some of the best fishing is by these facilities, but when I

fish in the northeast I know that my fishing is not near any facilities.

• Slide 10, 11 & 12 – You talk about facilities, but not northeast facilities. Won’t respondents

wonder about controlling facilities in other parts of the country?

• Slide 13 – The focus returns to the northeast.

• Slides 18 & 19 – There is no mention of the northeast on these slides where you talk about fish

losses.

• Slide 21 – You talk about government policies, but no region is mentioned. Most respondents

will know that regulation is done at the state and national levels.

• Slides 26 and 27 – The first bullet on slide 26 states: “Reduce fish losses in Northeast waters.” 

The following slide (#27) states: “EPA needs to know whether US households are willing to pay

the cost ...” These two slides take you respondent both directions. Will northeast residents think

that people from other regions of the country will also be paying for the northeast regional

benefits?

Finally, the choice questions do no specify the region of the effects or who is voting. I think this

inconsistency in information is bound to confuse some respondents.

Slides 24 & 25 – The information on these slides is confusing. You state that some facilities are using

technologies that “reduce fish losses up to 98%.” You also state: “The proposed policies would

require similar technologies at all facilities ...” Does that mean that facilities using the best

technology might be required to use a less efficient technology?

Slide 40 – You do not tell respondents what will happen if they chose not to vote for either option. You

have to assume that that they assume that current conditions will be maintained, but some may assume

further declines given the information you presented earlier. You need to specifically tell respondents

what you want them to assume about the baseline. This applies to the other baseline conditions for

other attributes as well.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-22 2/06

Slide 60 – It is possible that there will be some tax consequences if some of the costs are written off. This

could result in decreased tax revenues. I have found that a more believable payment vehicle allows

prices and taxes to change.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-23 2/06

PEER REVIEWER COMMENTS

Timothy C. Haab
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-24 2/06

Peer Review of Willingness to Pay Survey Instrument for 316(B) Phase III Cooling Water Intake

Structures

By

Tim Haab

General Comments and Overall Impression of the Scientific Merit of the Document:

Thank you for the opportunity to review this document and survey. I found the document to be

informative, well-written and easy to follow. The document does a good job of outlining the procedures

to be used and for the most part follows acceptable procedures for the valuation of increased fish stocks

from cooling water intake regulation. Particularly, the document provides thorough documentation of the

planned sampling scheme and procedures for ensuring a valid representative sample from an internet

based survey. Many of the fears decision makers face in regard to internet surveys should be allayed. 

Clearly the authors of the document—in conjunction with the experts at Knowledge Network—have

spent considerable time designing a sampling scheme and estimation strategy for correcting any potential

sampling bias created by the internet panel. 

Keeping in mind my overall positive impression of the justification and survey, I do have some concerns

that can be readily addressed. For the most part these are minor concerns that can be addressed quickly

and should not cause a significant delay in the administration of the survey. My comments and concerns

are detailed below in response to the charge questions provided by Versar. Perhaps my most significant

concern deals with the analysis of the survey responses. It seems as though the authors have proposed a

fairly basic method of analysis of the choice experiment response and have ignored some important

econometric issues that should be addressed. Further, I have concerns about the aggregation of fish

stocks into a single number in the survey responses. I will provide details below. 

Again, I want to emphasize that overall I think this survey is well-conceived, meets standard academic

practices and when administered will provide useful and accurate information to EPA. 

Responses to Charge Questions:

1. Is the choice experiment survey an appropriate stated preference survey format for the

estimation of marginal willingness to pay for the characterized policies? Will the survey

likely provide useful insights into preferences, tradeoffs, and willingness to pay associated

with the described policies?

In short my answer is yes. The choice experiment is the obvious and most appropriate choice

for the valuation of multi-attribute policy changes—such as 316(B). I can think of no other

method that will allow for the valuation of multiple attributes in such a clean manner. As

designed the survey collects: 

• background information on perceptions towards policies in general

• trade-offs among fish stock attributes

• debriefing information to establish the validity of responses

• demographic information

Combined, all of this information will allow the authors to provide a detailed description and

estimate of the values respondents place on fish stock increases dues to Phase III regulations. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-25 2/06

The use of the Knowledge Networks internet panel is accepted in the academic community and

provides an accurate but less expensive alternative to in-person surveys. The reviewed

documents do a nice job of describing the potential pitfalls with internet surveys and the

appropriate solutions. 

I have a few concerns about question ordering and wording in the survey that I will outline

below. 

2. Does the effort used in developing and testing the proposed survey meet current standards for stated

preference surveys reflected in the literature?

Yes, the use of 12 focus groups and cognitive interviews for survey development is appropriate

and warranted. The 500 individual pilot survey will provide EPA with valuable insights into

how the public is interpreting the survey. I would encourage the survey administrator to follow

up with pilot survey subjects to get open ended feedback on the instrument itself. Such feedback

will prove valuable in making sure subjects are interpreting questions appropriately.

3. Can the policy options be sufficiently well defined such that the survey is likely to provide useful

insights into preferences, tradeoffs and willingness to pay? 

I have a number of concerns here. First, the slide show and survey spend considerable time

trying to convey that the numbers presented are ‘uncertain.’ I’m not sure this is successful and it

is critical that it be. In particular, the choice experiment questions themselves seem to convey

‘certain’ information—“1360 billion fish saved”, “Increase to 70%”. There is no uncertainty

conveyed. In the blow-ups to the slide-show some ambiguity arises in the wording: “Policies

may increase stocks to between 60 and 75%” How is the respondent to interpret the actual

survey question? Will it definitely increase to 70%? Will it increase to somewhere between

60% and 70%? Did your focus groups show any confusion over this?

On a similar note, do respondents all have a common baseline for comparison? They are told

that “100%=maximum historical fish levels”. Is that with or without human intervention? Does

everyone have an accurate assessment of what that baseline will be, or are respondents

comparing apples to oranges? Again, these are questions that were probably addressed in the

focus groups but without more information it’s not clear what the respondent is valuing. Is it an

absolute change or a relative change? Do subjects know what catch levels were in 1950? 

There seems to be a non-sequitur in the slide show. It wasn’t clear to me that the choice

experiment was an example until after I read through everything once. Nowhere does it say,

“Here is a sample of what one of these questions will look like.”

Also, I have concerns that the sample questions may bias the answers to the actual choice

questions. For the same reason you want to control for potential order effects in the repeated

paired comparisons, you may need to control for the values reported in the sample problems. 

The respondent may be tempted to compare the actual choice policies to the sample policies. 

I’m concerned about aggregating all fish species into a single aggregate population change. The

slide-show give subjects information on the species affected and the table gives detailed

information on changes in fish species. Are subjects supposed to assume proportional increases
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

1

 Green, D., K. Jacowitz, D. Kahnemann, and D. McFadden (1998) “Referendum

contingent valuation, anchoring and willingness to pay for public goods,” Resource and

Energy Economics, 20: 85-116.

A-26 2/06

in stocks for all species? Or is it assumed that just the total matters? The National Marine

Fisheries Service has detailed estimates of the value of various species to recreational anglers. 

Those numbers clearly show that anglers value species differently. Is it the same for nonanglers? I’m not sure you can answer this with the current survey. 

4. Does the information provided in the survey on the baseline ecological conditions, the expected

changes, and the expected costs and their incidence seem sufficient, appropriate, unbiased and

accurate? Does the survey provide appropriate and sufficient introductory and debriefing questions

to enable identification of protest responses and symbolic responses, to test retention and

understanding of information, etc.? Would you suggest any additional debriefing questions?

For the first part of the question, see the answer above.

For the second part, I have concerns about the placement of the introductory questions in the

survey. The first two questions are designed to assess general attitudes towards policy and the

rankings will be severely compromised by the slide-show. The slide show provides 15 minutes

of information on the negative environmental effects of power plant operations and then subjects

are asked to rank environmental objectives among a group of other policies. It seems like this

question should come before the slide show.

5. Does the survey sufficiently and appropriately characterize the policy options under consideration? 

Does the survey place sufficient and appropriate emphasis on the scope of resource changes? Does

the survey appropriately communicate the scientific uncertainties as well as uncertainties associated

with the policies in question?

See response to question 3.

6. What are your expectations regarding the potential validity and reliability of benefit estimates that

might be provided by the proposed survey, compared to the current state of the art in the stated

preference literature?

I feel the proposed survey takes the necessary precautions to provide a reliable, conservative estimate of

willingness to pay. While the authors argue against cheap talk in the documentation, the survey

implements a watered-down version cheap talk by strongly encouraging respondents to take the valuation

questions seriously. Further, by reminding subjects that the survey results will be used to inform the

policy decision, the responses are taken from the realm of purely inconsequential to possibly

consequential. Many would argue that this is enough to meet the consequntiality criterion Carson, Groves

and Machina and Green, Jacowitz, Kahnemann and McFadden1

 set out for incentive compatibility.

7. Does the survey take appropriate steps to minimize hypothetical bias? Does the survey place

sufficient and appropriate emphasis on the budget constraint facing respondents, and on potential

substitute uses of their budget? Is the payment vehicle used in the survey credible? Does the survey

incorporate appropriate language to ameliorate strategic bias, symbolic responses and “yea

saying”? Is there any sign of systematic bias in the survey or presented information? If so, how can
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-27 2/06

it be ameliorated?

The survey design minimizes the potential for incentive incompatible responses. I would like to

see a more detailed description of the mechanism by which the ‘cost of living’ increases. For

example, “Such regulations will no doubt increase the cost of producing power to the firm. 

These costs will most likely be passed on to consumers in the form of higher prices for

electricity and other goods that heavily rely on power to be produced. 

The budget constraint and substitute reminders are weak. They appear as three bullet points on

one page. I would like to see some examples of the substitutes. The substitutes don’t have to be

just environmental substitutes. “The proposed policy will result in higher costs to you. This is

money that could be spent on other things like eating out, recreation, gas, etc.”

8. Will the answers to this survey lend themselves to straightforward economic interpretation of

willingness to pay? Are the proposed econometric modeling approach and validity tests

appropriate? Would you suggest an alternative modeling approach or alternative or additional

validity tests?

Unless I have missed some details, the proposed econometric technique fails to capture the panel

nature of your responses. Each subject will make up to three choices. These responses must be

treated as correlated for a given respondent. Failing to do so will bias parameter estimates and

estimated standard errors. Models for repeated choices are readily available in standard software

packages like LIMDEP.

9. The national benefits analysis for Phase III of the 316(b) rulemakings must be completed within the

next several months and a final rule issued by June 1, 2006. Are there improvements to the survey

and associated model that would be feasible to implement within the available time frame? If more

time was available, how would your answer change?

The suggestions I have made in the above responses can all be implemented in a short time period. The

overall concept and design of the survey is sound and with minor corrections will represent a solid piece

of applied valuation. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-28 2/06

PEER REVIEWER COMMENTS

James R. Kahn
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-29 2/06

Peer Review for Willingness to Pay survey Instrument for §316(B)

Phase III Cooling Water Intake Structures

Report to Versar, Inc. (job #110709.4007.033)

by

James R. Kahn, Ph.D.

John F. Hendon Professor of Economics

Director, Environmental Studies Program

Washington and Lee University

and

Collaborating Professor

Universidade Federal do Amazonas (Brazil)

January 16, 2006
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-30 2/06

Introduction:

The U.S. Environmental Protection Agency's plan for implementation of a willingness to 

pay survey instrument to assess the benefits of §316(B) Phase III Cooling Water Intake Structures is a

well-conceived plan that should produce results sufficient to guide the decision-making process. After

reading the document explaining the survey plan, I find that the authors are quite knowledgeable of stated

preference valuation measures, the controversies associated with their use, and the best means of

implementation. I am very pleased that the EPA decided to use the choice experiment (conjoint analysis)

stated preference approach, as I believe this approach to be less subject to biases and more conducive to

the evaluation of alternative policy options in comparison to contingent valuation. The justification of the

approach, the background information slide show, and the survey itself are all first rate work. If the

research plan is implemented as proposed, the EPA will be in a good position to make a decision relative

to §316(B) Phase III Cooling Water Intake Structures and to defend the decision that is made. Although

my overall assessment of the proposed survey and implementation plan is quite favorable, I have several

suggestions that the EPA may want to consider before implementing the plan. 

This peer review will begin with a section-by-section overview of the peer review package, organized

by section. This will be followed by responses to the specific questions posed in the "Charge to Peer

Reviewers." The final section of the review summarizes the review and highlights changes that could be

made to fine tune the explanatory document, background information slide show, and the review itself.

Previous Research (Section 1.2)

I concur with the authors that previous studies do not exist that contain an estimation of the social benefits

of nationwide changes in populations of fish species that include species utilized in sport and commercial

fisheries, as well as species that are not sought but are important forage fish. Although there are a plethora

of valuation studies of fisheries including those at the national level, these tend to focus on specific

species such as salmon, or recreational species in general. Additionally, these studies generally target the

use values of these species and do not measure values such as altruistic value or existence values. The

authors are correct in there assertion that general studies of the value of water quality improvements as a

whole measure of different suite of services than the proposed §316(B) decision will affect. An

implication of the authors findings which should be explicitly stated in this section is that it is simply not

possible to evaluate the benefits of the§316(B) Phase III program with a benefits transfer study.

Consultations (section 1.3)

The consultations, focus groups and pilot survey discussed in this section are consistent with the

standards of the literature. This reviewer has no further suggestions in this regard.

Survey Objectives (section 2.1)

The survey objectives are clearly stated and consistent with developing the knowledge necessary to

inform decision-making with regard to the §316(B) Phase III program. No modifications need to be made

to this section.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-31 2/06

Key Variables (section 2.2)

The authors do an excellent job in terms of tying key policy variables to the structure of the choice

experiment framework (conjoint analysis). The literature cited is appropriate (Adamowicz, et al, Louviere

et al, Bennett and Blamey and Batement).

Survey Support (section 2.3)

This reviewer is very familiar with Abt Associates, Inc., and has great respect for their work. This

reviewer is not familiar with the work of Knowledge Networks, Inc., but presumes that the authors have

chosen a reputable sampling firm.

Collection Method (section 3.1)

The proposed method of an internet-based survey is the best method that can be implemented given the

time constraints that EPA faces. The internet-base survey has the advantage of being able to supply

information in a fashion that is more difficult in face-to-face or mail surveys, as well as avoiding the

biased associated with the use of a telephone survey. I have some concerns that the use of the internetbased survey can lead to some biases that can not be corrected through advanced statistical methods. This

will be further elaborated in the discussion of biases. Nonetheless, the proposed method is undoubtedly

the best direction to go for the type of information that is needed to support this decision.

Survey Response and Follow-up (section 3.2):

I am somewhat concerned about the length of time that is required to complete this survey. Between the

review of the background materials and the actual completion of the survey, EPA estimates that 40

minutes are required. Although this length of engagement is consistent with conjoint analysis experiments

in the test-marketing of new market goods, these marketing research firms pay the respondents

substantially more than $5 to $10 dollars. I have had previous discussions with marketing professors who

suggest that $30 to $50 dollars is more appropriate. The longer the time of the survey, the greater must be

the payment to guard against the onset of respondent fatigue and meaningless responses. Although the

survey contains some follow-up questions to test the validity of responses, it might make sense to increase

the payment to users as much as the research budget will permit, up to about $30.

Pretests and Pilot Tests (section 3.3):

The pretest focus group and proposed pilot study with 500 responses seem to be appropriate. The second

paragraph of page 13 indicates that "Results from the final two focus groups (one including cognitive

interviews) provided convincing evidence that the majority of respondents answer the stated preference

survey in ways appropriate for stated preference WTP estimates, and that their responses do not reflect

biases noted above." The October 18 memo/report from Abt Associates that documents the focus group

research provides evidence that supports this conclusion. 

Target Population (section 4.1.1):

This is a standard approach.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-32 2/06

Sampling Frame (section 4.1.2):

I have several concerns about this section. The document does not make clear whether the telephone

numbers in the sample are land-lines or both land-lines and cell phones. If cell phones are not included,

there is a serious sampling problem because young people (18 to 25) generally do not have land-lines

unless they are living in the parents' home. (I do not have statistics on this, just my general observations

of my children, their friends and all the university students with whom I have contact). I also believe that

there are biases introduced by who answers the phone in the household. If the survey is implemented with

the person who answers the phone, then lazy husbands will tend to be underrepresented in the sample. On

the other hand, the phone could be given to the person in the household who most cares about fishing.

This would introduce a different type of bias in the sample. This section does not state if there is a

procedure for randomly choosing a person within the household.

Sample Size (section 4.1.3):

The authors use standard practice.

Stratification Variables (4.1.4):

Knowledge Network is capable of creating a stratified sample, which adequately corresponds to

representation of different socio-economic groups as represented in the U.S. Census. My concern here is

since people are being asked to engage in an online survey, the group of people who are computer literate

(and have online access) may give different responses than the group of people who decline to participate

because they are made uncomfortable by information technology. There two groups may have radically

different WTP for environmental improvement, and there is no way to correct this potential bias through

over-representing a particular demographic or socio-economic segment of the population in the sample.

The document briefly touches on this type of issue when talking about the 600 individual non-response

survey, but does not say how they intend to be guided by the information that is developed from the nonresponse survey.

Sampling Method (section 4.1.5)

This is a fairly standard approach but does not alleviate my concerns with section 4.1.4.

Response rate (4.2.1):

The response rate seems appropriate, but my original concerns about the potential for biases associated

with information literacy remain.

Precision Targets (4.2.2):

The level of the precision targets is fine, but the document provides not a single clue as to how they were

computed, or chosen based on standard references in the literature.

Non-Sampling Errors (4.3)

The discussion of the potential problems associated with protest bids, and the questions included in the

survey to identify potential problems with protest bids are adequate.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-33 2/06

The section also discusses potential problems with and solutions to noresponse errors and unobserved

heterogeneity. The actual explanation of the solutions is left to sections 4.3.1, 4.3.2, and 4.3.3.

Stratified Survey Sample (section 4.3.1):

This section looks at the problem of nonresponse bias. If a particular groups of the population are more

prone to decline to participate, then over-representing this group in the initial sampling effort can correct

this problem, and help to generate a completed sample population with the same demographic and

socioeconomic characteristics as the population as a whole. In theory, this is correct, but the document

does not specify a procedure to identify the degree of over-targeting the various demographic and

socioeconomic groups.

Nonresponse follow-up interviews (section 4.3.2):

This section is also inadequate in its explanation of the procedures to be used to incorporate results from

the non-response survey into the final analysis.

Statistical Correction for Unobserved Heterogeneity (section 4.3.3):

There seems to be an inconsistency between the discussion in this section in the second and third

paragraphs of page 19 and the econometric specification of the desired model at the top of page 30. I do

not see a Heckman correction coefficient in this model. This lack of consistency leads to doubts about

how thoroughly the authors have thought through the correction for unobserved heterogeneity. This must

be addressed.

Questionnaire Design (section 5):

The discussion of the survey is grouped by the section of the survey.

Section 1: Concern for policy issues—

These questions are right on the mark and accomplished the intended goal of framing the policy context.

Section 2: Voting for regulations to prevent fish losses—

These represent a good set of instructions. Again, the report/memo of October 18 from Abt Associates

provides documentation that these instructions are well-understood by the sample population.

Questions 4-6---

These are well-defined choice sets constructed in a fashion that is consistent with the literature. The

attribute levels are well-chosen, and provide sufficient variation to reveal WTP and to conduct basic

scope tests.

Questions 7-9

Good debriefing questions.

Question 10- Needs an additional response category as follows:
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-34 2/06

___I thought there were impacts on ecosystems in addition to those created by changes in fish populations

and partially based my answers on these impacts.

The rest of the survey consists of standard questions, some focusing on debriefing issues, some seeking to

measure intensity of preferences and some looking to define the demographic and socioeconomic

attributes of the respondent. There are no problems here.

Analysis of Stated Preference Data (Section 6.1):

This discussion is pretty standard and accepted within the environmental economics community.

Second paragraph-page 23-The random utility model pre-dates both Hanemann and McConnell and

should be attributed to McFadden (1974). 

Advantages of Choice Experiments (Section 6.2):

Again, this is standard and consistent with the literature.

Comment of Survey Preparation and Pretesting (Section 6.3):

The EPA has done a good job in pretesting of the survey and in survey preparation.

Econometric Specification (section 6.4)

As mentioned above, the Heckman correction method that was referenced section 4.3.3 is not mentioned

in this section. This is a major problem and must be addressed so that it is clearly understood exactly what

econometric specification is being estimated.

Experimental Design (section 6.5):

This is a well done section, following the standard literature. EPA has done a good job of eliminating

pairs of choice sets which could lead to undesirable reactions by the respondents, such as a choice set

which completely dominates another in all attributes, or nonsensical choice sets (high levels of saved fish

but low levels of fish population). Attachment 4, which further explains the experimental design appears

to be on the mark.

Information Provision (section 6.6)

Any time that a valuation experiment is done, the researchers must provide information to the respondents

concerning the implications of the changes they are being asked to assess. However, the researcher can be

in a difficult situation because too much information or the wrong kind of information can dramatically

influence a person's preference for one scenario over another. The researcher must try to create a balance

between providing the information allowing for the measurement of the respondent's preferences, without

creating a new set of preferences for the respondents. For example, showing pictures of cute baby seals in

an oil spill scenario could unduly influence the results.

My impression is that in this case, EPA has done a good job in presenting important information in a

"value neutral" fashion. I do not think they have crossed the line between being "estimators" of value and

"creators" of value.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-35 2/06

Amelioration of Hypothetical Bias (section 6.6.1):

Note the use of a double negative, "not unavoidable," in the last paragraph on page 33.

My personal opinion is that the concern about hypothetical bias is a result of a combination of some

sloppy work by researchers in the early years of CVM, and a bias on the part of some economists against

giving credence to values that do not originate in real world markets. If survey questions are designed

appropriately, hypothetical bias need not be a problem. Conjoint analysis is a technique that is less prone

to hypothetical bias than CVM, and the EPA has done a good job with its focus groups and survey design.

I strongly agree with the conclusion in the second half of the second paragraph on page 35 that "cheap

talk" is neither necessary nor desirable in this survey. I think the decision framework of conjoint analysis

obviates the need for cheap talk.

Amelioration of Symbolic and Warm-Glow Effects (section 6.6.2):

I agree with the EPA that they have appropriately eliminated warm glows, biases towards big businesses,

and so on. My chief concern in this section is whether "slide 52" is believable. Many people are aware of

the impacts of thermal pollution on estuaries and other water bodies, and they could believe that one of

the technology options (re-circulating cooling water) could have less of an impact on water temperatures.

Also, might people believe that reducing impingement of fish could also reduce impingement of reptiles,

amphibians and birds? This may have been addressed in the work with survey groups, but a discussion is

not contained in the document.

Assessing Scope Sensitivity (section 6.6.3):

The EPA adequately address the issue of Scope Sensitivity

Communicating Uncertainty to Respondents (section 6.6.4):

On the whole, this is well done, but I have real problems with slide 30 in the third paragraph of page 40.

The second sentence of this slide (Common sense dictates....) really dismisses any scientific evidence

about these relationships. The scientific knowledge should not be disregarded in an appeal to common

sense. The question should talk more specifically about how scientists can pin the outcomes to a range

with a degree of confidence, and by the way, if you think about it this way, preventing the loss of young

fish and fish eggs will certainly increase fish populations in the future.

I agree with the conclusion in the last paragraph on page 41.

ANSWERS TO QUESTIONS CONTAINED IN THE "CHARGE TO REVIEWERS"

(1) Not only is the choice experiment survey an appropriate stated preference survey format for the

estimation of marginal willingness to pay for the characterized policies, it is the best stated preference

approach for this purpose. I believe the survey to be well-constructed and it will provide useful insights

into preferences, trade-offs and the willingness to pay associated with the described policies. Based on

everything I have read in the document, I am confident that the U.S. EPA will have the best possible

information on which to base these decisions.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-36 2/06

(2) Yes, the focus groups and pre-testing more than meets current standards that have been discussed in

the literature.

(3) Yes, the survey does this in quite a nice fashion. Since the policies have varying levels of cost and

impacts on fish, they naturally fit into choice set scenarios.

(4) In general, I think the survey provided appropriate information on baseline ecological conditions. The

introductory and debriefing questions also seemed appropriate to identify protest responses and symbolic

responses. However, as I noted in my discussion above of 6.6.2, people might believe that reducing

impingement of fish would also reduce impingement of birds, reptiles and amphibians, and that some

technologies might be associated with less thermal pollution than other technologies. The potential

existence of this problem could be tested by adding an additional response to question ten, that would be

something like this:

___I though there were impacts on ecosystems in addition to those created by changes in fish populations

and partially based my answers on these impacts.

(5) The survey adequately characterizes the policy options under consideration and places appropriate

emphasis on the scope of resource change. I think that for the most part, the survey adequately addresses

scientific uncertainty, although as noted above, I did not like the choice of words ("common sense") in

slide 30. I thought the use of these words cheapened the quality of the scientific information.

(6) My expectations are that the validity and reliability of the benefit estimates are very strong in

comparison to the current state of the stated preference valuation literature. I would suspect that after the

dust settles from the rule-making process, the authors will be able to publish this study in a top journal,

such as the Journal of Environmental Economics and Management, Ecological Economics, or

Environment and Resource Economics.

(7) I don't believe that hypothetical bias is a real problem with correctly structured conjoint analysis

studies, and I believe that this study is correctly structured. Respondents know they will be making a real

world decision and that it will have real world consequences in terms of fish populations and the cost of

electricity and other products. There is not a need to focus on "cheap talk" or other mechanisms for

alleviating hypothetical bias, because they are not asked about a hypothetical donation. They are asked

about potential impacts on the day to day cost of living. Since the questions are structured with both costs

and benefits, it is very difficult for a respondent to behave strategically. I see no potential for systematic

bias, outside of what I noted in my discussion of section 3.2 (respondent fatigue) and section 4.1 (oversampling of the information literate in a way that is independent of typical demographic or socioeconomic classifications). The respondent fatigue issue can be addressed by increasing the participation

payment. The over-sampling of the information literate population is extremely difficult to address, and

since appropriate discussion of the non-response surveys are not contained in this document, I cannot

determine the significance of this problem.

I also note potential problems associated with who answers the telephone and who is handed the

telephone. The document does not specify a protocol for randomly choosing the household member who

participates in the survey. In addition, there are potential biases that will develop if cell phone numbers

are not included in the survey sample.

(8) The answers to this survey will lend themselves to straightforward interpretation in the decisionmaking context. The validity tests seem appropriate, but as noted above, I am a bit confused by the
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-37 2/06

econometric modeling. When talking about heterogeneity in section 4.3.3, the authors refer to using the

Heckman correction method. However, this does not appear in the econometric specification listed on

page 30. This inconsistency leads the reader to wonder exactly what econometric model is being

employed.

(9) There are a few sections where the discussion needs to be beefed-up to make the reader more

confident in the analysis. This includes the discussion of the stratified sample, nonresponse biases and the

econometric specification. The primary change to be made in the survey would be the incorporation of an

additional response to question 10, as mentioned in my response to question 4. Finally, since the survey is

so lengthy, it is important to pay the respondents a higher payment, or run the risk of the respondents

treating the later stages of the survey with less than their full attention.

Conclusions:

The proposed survey looks like it will accomplish the desired purposes, and is extremely well done.

However, some small changes can make it better. In addition, there are several sections, particularly the

sections on biases and econometric specification that need more exposition in order to make the final

results of the survey believable. Moreover, the reference list is not very impressive and does not provide

an adequate survey of existing applied studies using this technique. This unimpressive list gives an

impression that the researchers have not done all of their homework. For example, I did a Google Scholar

search on the keywords "choice experiments environmental valuation" and 25 of the first 40 hits were

directly relevant. A partial list of additional references is supplied.

In summary, with a few modifications, the survey and supporting documentation will provide the

appropriate type of information with the appropriate level of confidence in the empirical results. I am very

pleased that EPA has taken this route, instead of relying on benefits transfer or a contingent valuation

survey.

Additional References:

Boxall, P.C. W.l. Adamawicz, M. Williams and J. Louviere, 1996: A comparison of stated preference

models for environmental valuation Ecological Economics 19, 243.

Casey, James, 2006, Modeling of farmer preferences for agroforestry systems, in the Handbook of

Contingent Valuation, (Alberini and Kahn, eds.) Edward Elgar, 340-353

Carlsoonn, Fredrik, P Frykom and C. Lieejstope (2003) Valuing Wetland Attributes : An Application of

Choice Experiments, Ecological Economcsi 47, 95-103.

Farber, Stephen and B. Griner, 2000, Valuing watershed quality improvements using conjoint analysis,

Ecological Economics, 34.

Hanley, Wright and Adamawicz, 1998 Using Choice experiments to value the environment, Environ and

Resource Economics, 11: 413

Holmes, Thomas R and W.L. Adamowicz (2003) Attribute based methods: A Primer on Non-Market

Valuation, Champ, Brown and Boyle (eds) Dordrecht, the Netherlands, Kluwer Academic Publishers.

Kahn, James and Steven Stewart, 2006, An introduction to choice modeling for non-market valuation, in
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-38 2/06

A Handbook of Contingent Valuation, Kahn and Alberini, eds, Edward Elgar Publishers.

McFadden, 1974 Conditional Logit Analysis of Qualitative Choice Behavior in P Sarembeka (Ed),

Frontiers in Econometrics, New York: Academic Press.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-39 2/06

PEER REVIEWER COMMENTS

Alan J. Krupnick
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-40 2/06

Comments on

Peer Review Package for WTP Survey Instrument for 316(B)

Phase III Cooling Water Intake Structures 

Work Assignment 3-33

Versar Job Number: 110709.4077.033

By

Alan J. Krupnick

The charge asks reviewers to organize comments into general comments, responses to the charge

questions, specific comments about the report, and additional references. As the bulk of my comments

are about the survey itself and the charge questions are largely overlapping, I answer the charge questions

very briefly below and then provide detailed comments organized by problems with survey design. This

is followed by a few comments on the report (I have to say though, that the report is not very important

relative to the survey itself.)

General Comments

I commend the survey team for their tremendous amount of hard work and boldness in attempting to

capture preferences for a pretty ephemeral commodity – fish kills that have no ecological consequences. I

also commend EPA for providing what must be a huge survey budget for 5,000 completes. While I

actually think this is excessive for what is planned, it is heartening to see the agency make a commitment

to such a major study. At the same time, in a study of this complexity there are inevitable tradeoffs that

have to made, say between consequentiality and bias, where the literature is insufficient as a guide. There

are many such choices here and, given my experience, I think I would have made some different choices,

or at least altered the design to develop a strategy of protecting the survey against the possibility that

some of the choices were bad ones, primarily through developing additional treatments.

Change Questions

1a. The choice experiment (CE) approach is not necessarily the best way of obtaining WTP estimates. It

is more complex than the CV alternative. The most compelling justification for the former over the

latter is if the attributes affected by entrainment need to be unbundled, i.e., if separate shadow prices

(marginal WTP) need to be estimated for each of them, presumably so they can be used in benefit

transfer exercises to apply in locations where the improvements in these attributes are different. A

case can be made here to avoid false precision over these effects that may come from a CE and treat

them as bundled in a CV, using several different treatments to capture a relevant range of attribute

differences. 

1b. Whether the survey produces useful values depends on all its features ex ante as well as how well the

analysis turns out. On balance, I think there is some reason to be concerned about its features, as

detailed below, and one can’t know until the survey results have been analyzed if they will withstand

scrutiny.

2. The effort with 12 focus groups seems reasonable to me, certainly not excessive for a “commodity” as

complex as this one. In our survey on the WTP to improve the Adirondack ecosystem, we used over

30 focus groups. This probably was excessive, in retrospect.

3. This question is really about whether the levels of the attributes are appropriate. I have no

information to make such a judgment.
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-41 2/06

4. a. The information on baseline ecological conditions is lacking, as the materials readily indicate. I

think it will be hard to believe the results for two reasons that come up here: one, an effort is made to

eliminate in people’s minds the links with fish kills to ecological damage and, two, there is no

baseline for total fish stocks. Concerning the former, in my experience, respondents will default to

believing there is a link and just telling them there is not is not enough. At a minimum, add a

debriefing question to see if they bought this restriction. Concerning the latter reason, having the

baseline defined by I&E losses seems wrong to me. The baseline should be fish populations. If these

are not known I would recommend making them up, treating this as a parameter and trying alternative

information treatments varying this baseline to see if responses are sensitive to it.

b. I think the survey goes overboard on consequentiality giving too little attention to its flip side –

bias and warm glow. The questions to test for this are very complicated and one can never know if

these perceptions are being fully eliminated.

5. Mostly redundant to above questions. The new part is about uncertainty. In my work I find that

when a survey admits that effects are uncertain, respondents tend to reject the survey (and therefore,

biasing estimates downwards, unless such people are eliminated from the sample) and think the

survey makers don’t know what they are talking about. The paper discounts this effect; but we see

this effect very strongly in our work. 

6. I think getting credible marginal WTP values by attributes is unlikely because the attributes aren’t

independent (see more on this below).

7. Some of these questions are also redundant to above. A new issue is: Payment vehicle: I think this

payment vehicle is highly problematic. I don’t accept that respondents will accept that product prices

will be higher and hence their costs will go up. I recognize that the paper disputes this from focus

groups. Respondents in my experience will blame business and believe that they shouldn’t have to

pay. They may dispute that prices will go up, or may think prices will go up by more than they

should (rather than by less than the cost increase) because of monopoly business practices. Or

respondents may believe that they can simply substitute away from commodities whose prices rise

(admittedly, substituting from electricity wouldn’t be easy!). At the very least, debriefing questions

would be needed to tease out whether the payment vehicle caused problems for respondents.

8. I think that the sample size is larger than needed for the survey as currently designed. I suggest that

the experimental design be made more differentiated, with different information treatments developed

and tested to examine the effect of certain major choices in survey design on the results.

9. I don’t know if the above idea (in 8.) is feasible within the time frame. Some ideas below are

obviously easy to implement.

Detail on major problems.

1. There is no baseline or the baseline is poorly defined. There are three ecological attributes in the

valuation exercise: numbers of fish saved, effect on long-term fish populations and effect on

recreational and commercial catch. The first has no baseline, except relative to the total fish

currently “lost” in cooling water intakes. While some individuals might care about fish death per

se, I would imagine this is not a large fraction of the population. Most would want to know if this

amount of death is large relative to the overall populations of various fish species. They might

want to know this to gauge whether fish populations are at risk, which is really dealt with by the

second attribute or just about having a number to scale the deaths to. If it is the former, then the
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-42 2/06

number of fish saved attribute is redundant and should be dropped. If it is the latter, then some

number is needed. In this case, my advice is to make up a few different baselines and then use

them in several different treatments to test whether the baseline that was made up makes any

difference to WTP. I suspect it will, but in that way can be corrected or adjusted for. If

something like this isn’t done, I wouldn’t know how to interpret any results from the survey.

The second attribute’s baseline is poorly defined. The scale gives an endpoint of 100%, while

some screens fail to show what this 100% refers to. I recognize that the text may define this, but I

think it needs to be on the screen. The text definition refers to the maximum historic levels of

fish. This is poorly defined in my view. The maximum may not be the optimum or even be

sustainable. I think that some sentence like: Scientists and ecologists have determined that

appropriate fish populations consistent with the health of the ecosystems affected. The 100%

refers to population levels consistent with that determination.

Finally, the third attribute baseline is also poorly defined. Why is the catch in 1950 significant? 

We are not told on the screen. Again, if this is interpreted as peak catch then this may not be

optimal at all. Indeed, some respondents might interpret an increase in catch as a bad thing,

thereby screwing up the scope tests.

I am also troubled by what we term “expansive priors” about the baseline. The documentation

says that (at least one) respondents may think that the fish populations are declining. Their

response to a declining baseline situation will certainly be different than that if they think the

baseline is static or increasing. 

Overall, much more care is needed to define baselines and test for baseline effects when different

(or made up) definitions are used.

2. Definitions of the commodities. One of my biggest concerns is that the commodity is defined in

terms of fish, not as an indicator of ecosystem health, but as the fish themselves. Respondents

may well substitute their own views on ecosystem linkages with fish kills and you won’t know

what they are thinking.

And not just fish, but commercial fish (in part) which respondents should have no logical reason

to be willing to pay for. Respondents may ask themselves whether prices of fish will come down

with the options being considered, thereby canceling out some or all of their increased costs from

the option itself. So there are really two issues: whether respondents are willing to ignore

ecosystem health concerns and what the role played by commercial fishing is in their thinking. 

On the latter I would stipulate that scientists have determined that there will be no effect on

commercial fish availability or prices because such fish come from all over the world, and drop

any further mention of it. I would leave in recreational fishing as there may be option and use

value associated with it and it is placed-based. Because recreational fishing is placed-based, the

text would need to clarify how patchy the effects would be across substitutable recreational

fishing sites.

A broader issue is that the three attributes you list are not independent, but are related to one

another in complex ways. Probably, fewer kills will result in larger long-term fish populations

and greater recreation and commercial catch. But, your orthogonal design varies attribute levels

for these attributes independently (as it should), which means that fish kills could be going down

a little while long term populations could be going up by a lot – and that doesn’t really make
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-43 2/06

sense from any sort of simple story. Indeed, increases in catch could have effects on long-term

fish populations, irrespective of fish kills from entrainment, so the interdependencies are

everywhere. The documentation says that some inadmissible combinations of levels were

dropped from the design space, so maybe this problem has been at least partly addressed. What is

not addressed, though, is that if interdependencies exist in people’s minds then the marginal WTP

estimates attached to each attribute will not be reliable.

Another issue with the commodity is aggregation over species. The survey lumps all types of fish

together and talks about average effects over all species. I would want to be sure that this does

not create problems for the respondent in interpretation. 

Another issue is uncertainty in the commodity. Our research shows that admitting uncertainty

results in a downward bias and increased variance. People think that if you are uncertain about

effects either the effect can’t be very important or large or that you don’t know what you’re

talking about so why should they take the survey seriously. Both tend to push WTP down. 

The authors want to have it both ways. They want to invoke uncertainty, so they can claim they

took it into account, and then ignore it in the choice questions because respondents thought that

presenting intervals of improvements was too confusing. I don't think they can have it both ways

without making changes in the treatment of uncertainty.

As noted elsewhere, my suggestion is to vary in separate treatments how uncertainty is handled,

both about the baseline and about the changes.

3. Overall experimental design. With such a large sample size being planned I am surprised that

there are no plans for treatments, e.g., on information, uncertainty, baselines, etc. At the same

time, the large sample is being used to get a better understanding of how WTP varies regionally. 

Why is such regional precision required? Further, why is the commodity being limited

regionally? In our work, the payment vehicle definition required a limit to the state, but no such

limit is needed here as there are no taxes involved. Indeed, one could argue that a regional

limitation is unrealistic. Market effects will be the result of cumulative changes across the

country (to a large degree), so by not discussing them the survey may seem less credible and

WTP may be biased. 

4. Consequentiality versus neutrality. My view may be controversial, but I think the survey errs

too far towards consequentiality (and to far away from neutrality (the absence of bias)). Every

survey has to make choices about whether or not to state the sponsor of the survey, its real

purpose, and the implications of voting. By being truthful, one may easily create bias. To me,

this survey faces the same problem we faced in whether to tell respondents that they are voting on

a question about air pollution controls (with costs passed on to them) to improve conditions in the

Adirondacks versus trying to find an biased way of getting at preferences for cleanup and in so

doing having to be untruthful and perhaps lose some consequentiality. If people know that the

USEPA is doing this survey, they will assume that voting for the options is encouraged. Saying

that EPA will use these results to assess values begs the public to vote on a warm glow basis

rather than their true preferences. 

5. Payment Vehicle. The payment vehicle begs respondents to wonder why they should have to pay

when business is causing the problem. The notion that increased costs result in higher prices is

not one understood by most people, particularly for costs incurred by powerful and perceived,
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-44 2/06

high profit industries, such as electric power. They may think that industry will make them pay,

but they don’t think they should have to pay and may therefore vote against such a plan rather

than reveal their true preferences for the fish. 

6. Extent of the Market. I don’t understand why the survey is geographically limited in the way it is. 

Since the payment vehicle is a change in market prices, these changes can be realized by anyone

in an affected market. For some products, this is the entire nation; for others, perhaps effects

would be more regional. For power plants, regional effects are conceivable. But these effects for

the Northeast, would surely include Pennsylvania (which is excluded now) because this state is

part of a regional power grid that includes states labeled as northeastern states. From the benefits

side, the extent of the market is even more cloudy. Effects of entrainment are likely to affect an

entire watershed, so that is the extent of the market. Again, Pennsylvania and Maryland share a

common watershed, given the Susquehanna that runs through both states; yet PA is excluded and

MD is included in the definition of the Northeast region.

7. More Minor Comments: SLIDE SHOW

a. There are too many graphics and photos, any of which could introduce unknown biases. The

researchers appear to have designed the information for maximum interest, which is good, but at

the expense of a dry neutrality, which is necessary

b. “Protect fish stocks.” This seems like a biased statement. 

c. “part of a larger ecological system.” So which is it? Are the fish an indicator of ecological

effects or is the issue just fish? One information slide gives one picture, while another gives the

opposite impression.

d. “Many fish stocks have been declining.” This appears to establish a declining baseline, but this

idea is not carried forward to the choice questions or later in the text. 

e. “About 1/3 are types...” So does the 1/3 refer to species or to numbers of fish? Not clear.

f. Table. The table is very complicated and should not be included like this in the information

treatment. 

g. Pie charts are not very helpful when nearly all the chart is one color. 

h. “reduce fish losses by about 25%.” Doesn’t this set up an expectation about the probable size of

fish loss reductions? If so, when the choice questions vary this amount, how will respondents

interpret the difference?

i. “Technologies...already in use at 52% of facilities.” This phrase is problematic. If we already

have such experience, why is there uncertainty? And why is the government regulating again? 

What is the current regulatory environment? What isn’t working now?

j. “Some costs would unavoidably be passed on.” Some, but not all. So would shareholders

benefit? What is the respondent supposed to think? Passing on costs is not “unavoidable,” if the

markets, for instance, are international and competitive, the manufacturer would have to bear the

costs.

k. “EPA needs to know....” This is an odd phrase. We need to know how strongly people feel

about reducing fish losses, but people are not being asked to do a benefit-cost analysis.

l. “Common sense...” I would say no. Common sense could refer to sustainable populations that

are considerably below current levels and preventing loss of adult fish here could merely lead to

equal numbers of death from some other direction.

m. Status Quo. Best practice, according to Adamowicz, is to portray the baseline as an additional

column in the diagram, rather than just checking off a status quo response. To do so requires

stating the ecological baseline, something that the survey team has not wanted to do so far. 
Peer Review for “Willingness to Pay Survey Instrument for §316(B) 

Phase III Cooling Water Intake Structures”

A-45 2/06

n. “Significant impacts.” One slide says that there “are no other significant impacts,” implying that

the fish kills are a significant impact. This may introduce bias.

Text Comments

Pg. 6. I don’t see the need for oversampling. Rather sampling weights can later be used to correct for

differences between the population and the sample. I am not sure that doing this up front in sample

selection is worth it. Maybe it will introduce unknown biases relative to a more neutral attempt to enlist

respondents. 

Pg. 8. 2nd Paragraph It is one thing to assert that the values from this study will only apply to fish, rather

than ecosystems; another to make people believe that. In my experience, people know very well about

food webs and won’t surrender this idea easily. From the report, I am not sure that the idea should be

surrendered. Is it true that there are not ecosystem level effects of entrainment? 

Pg. 10 (second bullet). I am not clear from reading the report exactly how uncertainties are going to be

treated in the survey.

Pg. 12, 3rd Paragraph A 40-minute average survey is longer than usual for such surveys (30 minutes is

usually the target)

Pg. 14. 2nd Paragraph If the precision achieved is “more than adequate” than one should do what is

adequate and spend the rest of the money doing additional treatments to shore up unavoidable issues with

the survey.

Pg. 19-20. Heckman’s approach is referenced but not explained.

Pg. 27 2nd Paragraph Here is the justification for the CE. It is not compelling.

Pg. 32 1st Paragraph The survey team recognizes the possibility that some pairs will not make sense. But

this raises the issue of interdependencies across the attributes.

Pg. 33 2nd Paragraph In spite of the focus group sessions, there is no way EPA can “ensure” that the

graphics don’t lead to bias. The use of color and pretty pictures that are routine in this survey is a bad

idea, in my view.

Pg. 36 1st Paragraph In my experience, symbolic concerns are always an issue.