1

Response to “Comments of American Chemistry

Council, American Forest & Paper Association,

American Petroleum Institute, and Utility Water Act

Group on ICR for Willingness to Pay Survey for

Section 316(b) Existing Facilities Cooling Water Intake

Structures” (Federal Register Vol. 75, No. 139: 42438-

42440; July 21, 2010) and EPA Supporting Documents

DOCKET ID No, EPA-HQ-OW-2010-0595

Responses to the American Chemistry Council, American Forest &

Paper Association, American Petroleum Institute, and Utility Water

Act Group Comment

Executive Summary

This section briefly summarizes main issues presented in “Comments of American Chemistry Council,

American Forest & Paper Association, American Iron and Steel Institute, American Petroleum Institute,

American Public Power Association, and Utility Water Act Group on EPA’s Proposed Information

Collection Request on Willingness to Pay Survey for § 316(b) Existing Facilities Cooling Water Intake

Structures.” EPA responses are provided below. EPA responses are organized to correspond to the

individual ordered section of the comment.

Section I: Introduction

The section provides background on the proposed EPA survey based on the information provided in the

Agency’s Information Collection Request (ICR), lists objectives of the report, and introduces organization

of the report. No response is necessary.

Section II: Benefit Categories and Assessment Guidelines for Developing Reliable

Contingent Valuation Studies and Previous Attempts to Measure Non-Use Values

The Agency notes that the majority of this section does not address the specific survey materials or

information collection approaches described in the Agency’s ICR. Instead, these pages provide a

summary of benefit categories using EPA Guideline for Preparing Economic Analysis (EPA 2000,

hereafter Guidelines). The commenter states that general categories of benefits potentially relevant to

§316(b) regulatory assessment include: market benefits, non-market direct use benefits, non-market

indirect use benefits and non-use benefits. The commenter then argues that non-use values for fish

protection are likely not to be significant. The commenter points out that two economic criteria must be

met if non-use values for fish protection are likely to be significant (1) the resource is unique and (2) the

loss would be irreversible or subject to a long recovery period.
2

EPA disagrees with the commenter that non-use values for fish protection are likely to be insignificant

and notes that the economic literature advises against the use of ad hoc, expert assumptions regarding the

magnitude of non-use values—particularly assumptions that non-use values are trivial. Because values

are inherently subjective, Bateman et al. (2002, p. 75) emphasize that “it would be wrong for experts to

assume that one resource is a perfectly good substitute for another” [and hence that non-use values are

trivial or small]. As further noted by Bateman et al. (2002, p. 75), “there are, therefore, no easy rules for

determining at the outset” whether non-use values are likely to be significant or non-significant. Based

on this clear guidance, EPA believes that empirical analysis should be used to determine the magnitude of

non-use values in the 316(b) regulation for existing facilities.

Furthermore, results from the focus groups and cognitive interviews indicated that some individuals hold

substantial non-use values for protecting all fish species including forage fish. Participants in the EPA

focus groups and interviews cited a variety of motivations for preventing fish losses, including the

satisfaction of knowing that the fish exist, the desire to bequeath healthy fish populations to future

generations, and the desire to protect the functioning of aquatic ecosystems.

Section III: Guidelines for Developing Reliable Contingent Valuation Studies and

Previous Attempts to Measure Non-Use Values

The Agency notes that the majority of this section does not address the specific survey materials or

information collection approaches described in the Agency’s Information Collection Request (ICR).

Instead, these pages provide a summary of contingent valuation methods in general, along with a

selection of guidelines chosen from references of varying ages and degrees of obsolescence. Given that

many of these pages do not refer directly to the Agency’s proposed survey or methods, specific responses

to these sections are largely unnecessary. The Agency will focus responses on areas in which the

comments refer to the specifics of the proposed information collection request, including the proposed

stated preference methods. Given that some of the Commenter’s later points refer back to this general

introductory material, however, the Agency feels that it is important to place these materials and

associated claims in a broader context of the stated preference literature.

Finally, some of the following responses refer to the extensive focus groups used by the Agency to

develop and test the proposed survey. The focus groups and interviews took place in the following

locations and on these dates:1

1) East Providence, RI, July 21, 2010

2) Waltham, MA, July 29, 2010

3) St. Louis, MO, August 4, 2010

4) East Providence, RI, August 12, 2010

5) Bethesda, MD, August 18, 2010

6) Charlotte, NC, September 8, 2010

1

In this response, we use the citation format (#X-Z to refer to page Z of the transcript for focus group X. Citations to

cognitive interviews use a similar format, (#X/Y-Z), where Y represents the interview number.
3

7) Philadelphia, PA, September 23, 2010

III.A. Overview of Contingent Valuation

As a precursor to later comments, the Agency notes that this comment uses largely obsolete, and perhaps

misleading, technical language to describe the proposed methods. The broader category of survey-based

valuation—in which values are based on respondents’ answers to questions involving hypothetical

commodities or policy changes—is most commonly denoted “stated preference valuation.” The

contemporary literature generally uses phrase “contingent valuation,” in contrast, to refer only to

questions involving open-ended valuation questions. In contrast, the methods proposed by the Agency

apply choice experiment (or choice modeling) methods (Bennett and Blamey 2001); these are not the

same as open-ended contingent valuation questions, which involve “directly asking respondents their

willingness to pay (WTP) for a particular commodity or environmental change” (comment p. 8). These

distinctions are critical, as the methods and characteristics of the two methods are different, and the two

methods have been clearly shown to differ with regard to the prevalence of certain biases and response

anomalies. This distinction is reflected as far back as the original guidelines of the NOAA Blue Ribbon

Panel (Arrow et al. 1993), which recommends choice-based methods. The Agency notes that the

comment frequently obfuscates of the distinction between open-ended contingent valuation and choice

experiments, leading to comments or implications that might be considered misleading—particularly to

those unfamiliar with the proposed methods.

This general comment aside, the Agency agrees, in general, with most of the broad description of stated

preference methods presented in this section. The majority of this Commenter’s points in this section are

not specific to the proposed stated preference methods, but rather provide a general introduction to stated

preference methods in general. As such, and given that nearly all of this material is well-established in

the literature, the Agency sees no specific need to comment. As a precursor to more detailed responses

later, however, the Agency notes that it is fully aware of the broad literature on stated preference methods.

The Agency also believes that the introductory material tends to emphasize language and position taken

by skeptics rather than practitioners of the approach; that is it provides a rather one-sided perspective on

the literature. For example, while noting accurately that “[s]ome economists take a dim view of the

ability of CV surveys to reliably measure the “non-use” values” (p. 8), the comment fails to reflect the

fact that many economists also support the ability of stated preference surveys to measure these values,

and that stated preference surveys are widely used in the published literature for such purposes. Agency

guidelines also support the use of stated preference surveys to quantify non-use values (US EPA 2000).

Also, the Commenter’s language in this section (particularly at the top of page 9) seems to imply that

there is a single set of universal guidelines that apply to all stated preference surveys regardless of type or

context. For example, consider the phrase “any CV study that is to be used in a policy decision should

both be well-designed (i.e., should follow the proposed guidelines)...” (p. 9, emphasis added). This

misleadingly implies that there is a single set of guidelines. While there are many agreed-upon “best

practices” for stated preference survey design and implementation (see, for example, Bateman et al. 2002

and Louviere et al. 2000), the Agency also notes that there are many other areas in which the literature

has not agreed upon these guidelines.

Finally, the Agency believes that the discussion of the cost of stated preference survey implementation vis

à vis the assumed size of nonuse values is also misleading. First, the majority of the costs of the proposed

survey—those involved with survey development and focus group testing—have already been incurred
4

and hence are most appropriately viewed as sunk costs, and therefore irrelevant to future decisions. More

importantly, this comment is based on an unjustified presumption that nonuse values in the present policy

context are likely to be small. EPA emphasizes that the magnitude of nonuse benefits associated with

changes in aquatic habitat is an empirical question—and that the potential significance of such benefits

can only be assessed using appropriate empirical methods. EPA is aware of no research—empirical or

otherwise—that demonstrates that individuals have limited existence motives for the types of fish that

would be protected by policies at issue. The Agency notes that the same issue was raised by the

Commenter in responses to a prior Agency ICR (American Chemistry Council, American Forest & Paper

Association, American Iron and Steel Institute, American Petroleum Institute, American Public Power

Association, and Utility Water Act Group on EPA’s Proposed Information Collection Request on

Willingness to Pay Survey for § 316(b) Phase III Cooling Water Intake Structures). The Agency refers

the Commenter to our original responses to the same issue in the earlier submitted comments.

III.B. Guidelines for Contingent Valuation Studies

As noted above, the material in this section largely summarizes selected guidelines for stated preference

surveys found in a variety of documents. As this section draws no conclusions regarding the ICR, the

Agency has few comments here. The Agency, however, notes that the majority of guidance reflected in

this section is drawn from documents that are now more than fifteen years old, and hence do not reflect

more recent advances in stated preference design or testing. This section is also drawn from a very small

portion of the available literature and guidance related to stated preference methodology. Hence, the

Agency emphasizes that the comments in this section are most appropriately viewed as a selective

perspective on a small, and often fairly old, segment of the relevant stated preference literature.

III.C. Internal Validity Tests

This section neither draws conclusions regarding the ICR nor offers concrete suggestions. Rather, this

section summarizes a variety of validity tests that might be conducted to assess the reliability of stated

preference surveys. As these comments are not specific to the survey or ICR in question, the Agency has

only a few comments.

Scope Tests: While noting the clear limitations in naïve scope tests discussed by Heberlein et al. (2005),

the Agency agrees that it is important that survey responses in this case show sensitivity to scope. This is

one of the primary reasons for the use of choice experiment methodology, which is better able to capture

WTP differentials related to changes in resource scope (Bateman et al. 2002). Unlike open-ended

questions, in which scope insensitivity is a primary concern, EPA emphasizes that choice experiments

generally have shown much less difficulty with respondents reacting appropriately to the scope and scale

of resource changes. Moreover, as clearly noted by Bennett and Blamey (2001, p. 231), “internal scope

tests are automatically available from the results of a [choice modeling] exercise.” That is within choice

experiments, sensitivity to scope is indicated by the statistical significance and sign of parameter

estimates associated with program attributes (Bennett and Blamey 2001). Internal scope sensitivity will

therefore be tested through model results for choice experiment attributes.

In addition to internal scope tests implicit in all choice experiment statistical analysis, EPA will also

conduct external scope tests (cf. Giraud and Loomis 1999). The primary difference between internal and

external tests is that the former assess sensitivity to scope across choices of a single respondent, while the

latter involves split-sample assessments across different respondents. Within a choice modeling context,
5

external scope tests are generally considered stronger, although also more likely to be confounded by

differences in the implied choice frame (Bennett and Blamey 2001). A variety of options for external

scope tests exist, depending on the structure of stated choice questions under consideration. In the present

case, attribute-by-attribute external scope tests will be conducted over a split sub-sample of respondents

considering a specific set of choices, with all held constant across the considered choices except the scope

of the attribute for which the test is to be conducted. For example, to conduct an external scope test for

reductions in annual fish losses, one would consider a set of choices that is identical over two respondent

groups, except that one considers a choice with a greater reduction in fish losses. Assessing the choices

over this split sample allows for an external test of scope. The Agency emphasizes that the formal

applicability of the above-noted scope test is contingent upon the specific choice frame implied by levels

of other attributes in the choice question. This is a characteristic of nearly all external scope tests applied

in choice experiment framework (Bennett and Blamey 1999).

Split-sample tests such as those proposed above often require the addition of question versions to the

experimental design, to accommodate the specific structural needs of the attribute-by-attribute external

scope test. Otherwise, confounding effects of other varying attributes can render results of scope tests

ambiguous. In the present case, the proposed tests would require the addition of new profiles to the

experimental design. While small numbers of questions added to the experimental design should have

minimal impacts on overall efficiency, larger numbers may have a more significant impact. Hence, given

constraints on the total number of survey respondents, there is an empirical tradeoff between the number

of external scope tests that may be conducted and the efficiency of the experimental design and statistical

analysis.

The “Adding-Up” Test: The “adding up” test is based on an oversimplified perspective on embedding

effects that ignores the influence of task frame and order on behavior. Economic theory clearly predicts

that “the simultaneous availability of other projects” will generally affect the value of any other policy in

question (Carson et al. 1999). Moreover, as noted by Hanemann and Kanninen (1999), “evidence in

market research as well as in environmental valuation overwhelmingly rejects procedural or task

invariance.” This issue does not relate to any type of survey bias or difficulty with non-market valuation,

but rather is a fundamental theoretical issue related to the potential substitutability among goods provided

by government policies. Within the specific issue of embedding in stated preference surveys, it is also

well known that such behavioral patterns may occur equally in market and non-market settings (Carson et

al. 1999; Hanemann 1994). As stated by Hanemann (1994), “it should come as no surprise that the value

of one commodity changes when the quantity of another varies: in other words, that willingness-to-pay

depends on economic context.” The “adding up” test also presumes an identical choice frame when

assessing WTP for Good B (presuming that Good A has already been provided) and assessing WTP for

simultaneous provision of Goods A and B. The fact that this assumption is almost certainly invalid

likewise invalidates the associated test. Given these unavoidable concerns, the Agency does not view the

“adding up” test as an appropriate validation of the proposed survey instruments.

Anchoring Tests: The Agency notes that classical anchoring tests applied to dichotomous choice

contingent valuation, as described by the comment, are not strictly applicable to choice experiments.

Econometrically Estimating the Relationship between WTP Answers and Various Factors: As a

clarifying note, the Agency first emphasizes that the proposed choice experiments will not solicit “WTP

Answers,” as implied by the comment. This seems to reflect the confusion between open-ended

contingent valuation and choice experiments noted above. However, the Agency does plan to conduct an
6

extensive evaluation of choice responses, including an evaluation of the impact of socioeconomic and

other measurable factors on implicit prices.

III.D. Prior Efforts to Measure Non-Use Value Using Stated Preference Methods

The Agency strongly rejects the claim that “[p]rior efforts to measure non-use values using stated

preference methods cast substantial doubt on the reliability of the method, contrary to what EPA

contends” (p. 17). The Agency notes that this entire section is drawn from an unsystematic discussion of

only two past stated preference studies, both of which: (1) were conducted nearly two decades ago, (2)

used methods different than those proposed by the Agency in the present ICR, including some approaches

that are now considered rudimentary and/or largely obsolete, (3) addressed resource types distinct from

those addressed in the present ICR. Given the breadth of the contemporary stated preference literature—

including thousands of published works—the Agency strongly rejects the argument that any conclusive

insights regarding the ability to estimate non-use values may be drawn from an ad hoc discussion of two

research studies conducted using methods different from those proposed by the Agency within the current

ICR.

Section IV: Likely Biases in the Proposed EPA Survey

As a precursor to more detailed comments below, the Agency rejects the claim that the suggested

“sources of bias...would render the proposed EPA survey unreliable” (p. 19). Responses to specific

issues are listed below.

IV.A. Bias Due to Creation of Preferences

The first comment in this section is that “the benefits estimated from the EPA survey would be

conditioned on the information included in the survey.” Specifically, the comment suggests that “people

had little or no prior awareness of CWIS, much less knowledge about cooling water intakes and the

purported effects on fish populations” (p. 19). The Commenter claims that this creates a significant

concern that “EPA has created preferences, where none existed before.” and that EPA has ignored the

“substantial psychological literature on constructed preferences” (p. 19). The Agency notes that these

comments are similar in substance to earlier comments received from the same Commenter in response to

an earlier version of the survey (American Chemistry Council, American Forest & Paper Association,

American Iron and Steel Institute, American Petroleum Institute, American Public Power Association,

and Utility Water Act Group on EPA’s Proposed Information Collection Request on Willingness to Pay

Survey for § 316(b) Phase III Cooling Water Intake Structures). The Agency finds these comments

similarly unconvincing.

With regard to the general comment that respondents’ answers will be informed by, and conditional on,

the information provided in the survey, EPA believes this result is entirely expected, and is consistent

with the academic literature. It is well known that information can influence economic values (Bergstrom

and Stoll 1989; Bergstrom et al. 1989; Hoehn and Randall 2002; Cameron and Englin 1997). The

Agency emphasizes that the sensitivity of values (and behavior) to information is true of both market and

non-market values (Bateman et al. 2002, p. 298), and in no way invalidates the proposed non-market

values that would be estimated through the proposed stated preference approaches. Cameron and Englin

(1997), for example, provide a detailed discussion of the various types of information that may be

possessed by or provided to respondents, and how this information may influence values in a legitimate
7

neoclassical benefit cost framework. It is also interesting to note that Cameron and Englin’s (1997)

analysis focuses on a policy topic that is fairly similar to the one addressed here—WTP for a doubling of

trout in the northeastern United States. Hence, the Agency feels that this analysis and discussion is highly

relevant to the proposed stated preference survey instrument and analysis.

More specifically, however, this comment reflects two distinct but related claims. The first is that the

presentation of significant information to survey respondents is inappropriate. The second is that the

presentation of information in a survey instrument—particularly when extensive and paired with policy

issues of which respondents may have limited prior knowledge—will likely result in “constructed

preferences” inconsistent with neoclassical benefit cost analysis. Based on its review of the economics

literature and standard practice in benefit cost analysis, the Agency finds neither of these arguments to be

particularly convincing, and notes that such perspectives—while indeed found in some psychological

literature—are not reflected in the guidance provided by the mainstream valuation and benefit-cost

literature.

With regard to the appropriateness of information provision in stated preference surveys, it is common

practice in such surveys to provide substantial information to survey respondents. In contrast to the

claims of the Commenter, the broader literature tends to treat with skepticism those surveys that do NOT

provide sufficient information to respondents. As stated by Bateman et al. (2002, p. 309), “respondents

should be given enough information in a clear-enough fashion so that they should be comfortable making

a decision.” It is well established in the stated preference literature that valid stated preference value

estimation requires that respondents are provided with an adequate, accurate, and detailed representation

of changes to natural resources. For example, the NOAA Blue Ribbon Panel on Contingent Valuation

(Arrow et al. 1993, p. 4605) state clearly that “if CV surveys are to elicit useful information about

willingness to pay, respondents must understand exactly what it is they are being asked to value (or vote

upon)...” The pane further indicates that surveys providing “sketchy details” about the results of

proposed policies call “into question the estimates derived there from,” and hence suggest a high degree

of detail and richness in the descriptions of scenarios. Similar guidance is provided by other key sources

in the CVM literature (e.g., Mitchell and Carson 1989; Louviere et al. 2000; Hanemann 1994). Given this

clear guidance to provide detailed, specific, and quantitative information, the Agency is not convinced by

the argument that the provision of substantial amounts of information is inappropriate or will lead to a

biased expression of value among respondents. To the contrary, the Agency believes, following the

literature, that the provision of minimal information in the survey would encourage exactly the type of

symbolic responses and embedding that the Commenter discusses in subsequent sections. Multiple focus

groups and cognitive interviews conducted by the Agency were conducted to determine, among other

things, the type and quantity of information that was required by respondents in order to make an

informed, comfortable decision, as recommended by Batemen et al. (2002) and many other sources (e.g.,

Mitchell and Carson 1989). These focus group and interview results were used to determine the type and

quantity of information provided in the survey.

With regard to the argument that the substantial information provided by the survey indicates that the

survey is “creating preferences” among respondents, the Agency emphasizes that this argument is one

that has been leveled at stated preference methods for over 15 years. This argument was not judged to be

an overriding argument against stated preference valuation by the NOAA Blue Ribbon panel (Arrow et al.

1993), nor it is generally considered to be convincing by the benefit cost literature. Moreover, as noted

by Bateman et al. (2002, p. 298), arguments regarding constructed preferences in stated preference

valuation are often based on a misconception concerning that which economic theory “does and does not
8

say about preferences. Contrary to common criticism, economic theory does not dictate that individuals

have prior-formulated values for all possible options that might be offered to them... [T]he neoclassical

economic view is that ... survey responses must be driven by preferences which are consistent with

economic theory...” Similar to the context found in many stated preference instruments addressing goods

that may be unfamiliar to some respondents, consumers in actual markets are frequently presented with

new market goods for which they have no experience. The fact that consumers may have little knowledge

of new market goods prior to their introduction does not invalidate the ability of markets to reveal values;

nor does it invalidate the ability of stated preference methods to estimate non-market values (Bateman et

al. 2002).2

The Commenter continues by claiming that the “such a blank slate on the part of most, if not all of the

survey respondents, makes the information presented problematic” (p. 19). This comment suggests that

the Commenter has confused the multiattribute good being valued (attributes associated with fish losses

and associated ecological effectsw) and the policy context in which it is being valued. Surely, the

Commenter cannot be suggesting that people have no familiarity with fish. Rather, many people are

relatively unfamiliar with a particular type of human effect on fish (i.e., that caused by CWIS). This

parallel is quite similar to that found in Cameron and Englin’s (1997) analysis, which assessed WTP to

avoid effects of acid rain on trout populations. The fact that many individuals were not familiar (prior to

the survey) with the effects of acid rain on trout did not invalidate WTP measures in this published case.

Indeed, in the view of the Agency, the survey at issue in the present ICR exemplifies the type of valuation

context addressed by Cameron and Englin, in which “respondents themselves will have acquired varying

levels of experience with the commodity in question, or at least with closely related commodities,” but in

which the survey provides additional information used by respondents to formulate their valuation

responses.

IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation

This comment suggests that survey materials prepared by the Agency contain inaccurate statements and

invalid comparisons that will lead respondents to believe that the benefits of increased regulation of

cooling water intakes would be substantially greater than is actually the case. The Agency rejects this

claim, and emphasizes that the survey is explicitly designed to provide respondent with an understanding

of the proposed policies that is as accurate as possible given the best available ecological science.

The initial paragraphs of this section draw no conclusions regarding the ICR. As a result, the Agency has

no comments on these paragraphs. The Agency’s responses begin with the misleading claim that

“Beginning with the title page of the survey, EPA provides information to respondents that leads them to

believe that they are valuing a policy with much wider-reaching effects than can be supported by the

relevant science” (p. 20). The Agency can find no examples in the revised (post focus group) survey of

“skewed description of the relative importance of the effects of I&E on fish populations.” For example,

the revised survey clearly states “Cooling water use is not the largest cause of fish loss in most areas

(fishing causes greater losses), but has affected some fish populations.” Moreover, the survey now

provides explicit information comparing losses of age-1 equivalents to total estimated populations for

representative species. These and other devices in the revised survey are designed to provide a clear and

2

In an example provided by Bateman et al. (2002, p. 341), “prior the introduction of the video recorder individuals

could not have values for such a good. This did not prevent them from determining such values when such a

choice became available.”
9

unbiased perspective on the magnitude and effects of entrainment and impingement. The Agency

emphasizes that the estimates of impingement and entrainment losses used in the survey are scientific

estimates specific to the policy options valued in this study. The policies considered by EPA in the

context of entrainment and impingement reductions would prevent the loss of a potentially large number

of age-one equivalent fish, both regionally and nationwide. The data presented in the survey represent the

best scientific estimates of fish losses and population changes available.

Other comments in this section are similarly unsupported by either the content of the survey instrument

itself or the results of focus groups. For example, the claim “EPA’s survey materials substantially

overstate the effects that are likely to occur on recreational and commercial fishing as a result of the

proposed policy. After reading the materials, people are much more likely to think that the majority of the

impacts will be on the recreationally and commercially significant species” is explicitly contradicted both

by direct statements in the survey and by the comments of focus group participants. The survey explicitly

states that the majority of affected species are “forage species not caught by humans.” This is clearly

understood by respondents. This is reflected, for example, in the statement of one focus group

respondent: “You know it was educational that there’s a lot more forage fish that there are fish that are

actually caught and consumed by humans” (#7-7).

Similar claims of Barnthouse (2010)—attached to the submitted comment—are also unsupported by the

Agency’s focus groups and interviews used to test respondent comprehension of the proposed survey

instrument. First, numbers in the survey are already presented as Age-1 equivalents—a fact apparently

misunderstood by Barnthouse (2010). This seems clear due to the statement that “[t]he survey materials

should, like the case study documents, express E&I losses in terms of age-one equivalents” Barnthouse

(2010, p. 1). More broadly, the Agency’s focus groups clearly suggested that respondents generally

understood that CWIS primary affects “young fish” which may eventually develop into adult fish. Many

participants stated that the graphic on Page 2 of the survey (“How Fish are Affected by Water Intake”)

helped them understand how CWIS impacts fish and how it differentially affects eggs, juvenile, and adult

fish: “I was thinking more fish is going to get through the screen if they’re smaller. The smaller, the more

eggs and the more fish you’re going to lose” (#6-8). Moreover, despite the Commenter’s claims to the

contrary, focus group participants generally understood that in most cases the quantity of fish saved

would be small relative to the total fish population. For example:

Moderator: When you look at that fish saved per year, how did you think about that in terms of

all the fish out there in the water?

Respondent: I guess probably a pretty small part [#6-7].

Participants generally felt that the survey provided useful information about fish losses attributable to

CWIS, other sources of mortality, and population sizes. Participants generally understood based on the

introductory material that CWIS is not the largest source of fish mortality. The draft survey was also

revised to include pie charts comparing young fish losses from CWIS to estimated adult fish populations

for example species. This presentation was deemed important in order to address participant comments

on intermediate draft survey versions which did not specifically include examples to this effect.

In summary, the Agency devoted considerable attention to this issue when designing and testing the

survey instrument. The suggestion that the presentation of the magnitude of fish losses is “misleading”

does not appear to be supported by focus group results. More broadly and importantly, however, the
10

proposed comment suggests a lack of understanding of the capacity of choice experiments to isolate the

marginal effects of different attributes on stated choices and hence on WTP—a methodology clearly

described by numerous sources (e.g., Louviere et al. 2000). The choice experiment survey and

experimental design are explicitly designed such that the model can incorporate a wide-range of potential

outcomes, including zero changes in population and harvest. That is, the choice scenarios (or profiles)

explicitly evaluate respondents choices and implied values for cases in which changes in ecological

outcomes (including changes in fish losses) is both large and small. Unlike older forms of stated

preference analysis in which WTP is contingent upon a single specification of ecological effects, choice

experiments solicit responses under a large number of possible attribute configurations. This will allow

the Agency, for example, to estimate WTP for policies in which the broader ecological effects of I&E

reductions—beyond direct influences on the number of age-1 equivalent fish lost—are zero.

In summary, the Agency finds the comments in this section to be unsupported by evidence from either the

survey instrument, broader ecological science or focus group results. The comments also suggest a

misunderstanding of the valuation approach proposed by the ICR, and its capacity to estimate WTP for a

broad range of possible ecological outcomes.

IV.C. Bias Due to the Proposed EPA Survey’s Failure to Characterize Accurately Substitutes and

Alternatives

The comments offered in section IV.C.1 draw no conclusions regarding the ICR. As a result, the Agency

has few comments on these paragraphs. However, the Agency would like to highlight the fact that the

quoted guidelines universally relate to the requirement that stated preference surveys provide an accurate

description of the commodity to be valued. None of these guidelines or the associated quotes address a

“failure to characterize accurately substitutes and alternatives,” (p. 21) as implied by the comment. The

Agency agrees that valid stated preference valuation requires that the survey provide an accurate

characterization of the commodity to be valued, as suggested by the quoted sources.

Section IV.C.2 is entirely contingent upon a distinct claim that is not supported by any of the quoted

guidelines above—that “respondents must understand not only the program or resource in question, but

also the alternatives or substitutes to the program or resource in question” (emphasis added). As stated

by Carson et al. (1999, p. 120-121) regarding the potential impact of substitutes on estimated values, “the

value of [a project] may become arbitrarily small if enough projects are considered and the projects are

substitutes with respect to valuation.” They clearly note also that “this is a purely theoretical issue which

is not related to contingent valuation or passive-use values per se.” In doing so, Carson et al. (1999) note

that “researchers need only consider other [substitute projects] that will actually be undertaken at the

same time, rather than all potential projects.” They also recommend that the set of substitutes presented

to respondents be limited to “close substitutes of which respondents are unaware,” and criticize “extreme

position[s]” such as that of Kemp and Maxwell’s (1993) Exxon-sponsored study in which respondents

were asked to allocate expenditures to as many as 32 different categories of public expenditure. To the

contrary, Carson et al. (1999) notes that “in most cases, the number of potential relevant and unknown

substitutes is small and as such represents little problem in designing and administering the survey.”

The Agency also notes that the entirety of section IV.C.2 relates to potential broader effects of closed

cycle cooling. This is based on a misleading and incorrect presumption that the purpose of the Agency’s

proposed survey is to evaluate WTP for cooling towers. As stated clearly by the ICR, this is not the case.

The clearly stated purpose of the stated preference survey is to “elicit values for changes in fish mortality
11

and the associated ecological gains from a section 316(b) regulation covering Phase II and Phase III

facilities” (EPA ICR No. 2402.01). Given that the purpose of the survey is not to evaluate WTP for

cooling towers and their impacts, the broader ecological impacts of cooling towards and closed cycle

cooling are not relevant to the ICR.

The Agency acknowledges that there may be a range of other welfare-relevant effects associated with the

operation of closed-cycle cooling. Evaluation of these benefits and costs, however, is not the purpose of

the ICR, nor is it required for appropriate assessment of WTP for changes in fish mortality and the

associated ecological effects. Hence, the material in this section is neither relevant for the stated

preference survey under consideration, nor is the related information required in order to provide an

accurate description of the commodity to be valued.

IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes

The comments in this section address a number of only loosely related issues, few of which are related to

“inherent collinearity in the ... attributes respondents are valuing.” As a set, the Agency finds the

comments in this section to be unclear and unconvincing. Moreover, they may suggest some lack of

understanding of the choice experiment methodology that is proposed.

The first comment in this section may be summarized by the concluding sentences: “While supposedly

people are thinking that the only fish being saved are forage fish, if they are really considering

recreationally and commercially significant fish in answering the questions, then there is no independence

among the attributes. Consequently, the values that would be attributed to forage fish are really values for

some other fish species.” (p. 26) The Agency is puzzled by this comment, as it seems to reflect a lack of

understanding of both the proposed survey scenarios and the proposed choice experiment methodology.

The survey instrument developed and revised as a result of the Agency’s focus groups includes

independent and separate ecological attributes communicating changes in (1) fish saved (all species), (2)

fish populations (all species), (3) commercial and recreational fish populations, and (4) the health of

aquatic ecosystems. These were chosen as a result of extensive focus group testing and consultations

with Agency scientists. Given that these are presented in choice experiment scenarios as independent

attributes, the Agency sees little likelihood that respondents will somehow confuse the meaning of these

attributes, as claimed in this part of the comment. Focus group results strongly support this contention,

suggesting that respondents were able to distinguish the attributes communicating changes in all fish

species and those communicating changes in only commercially and recreationally significant species.

The following focus group quote is typical, showing a clear distinction between “commercial fish

populations” and “fish populations (all fish)”: “[...] specifically the fish that are more generally used as

food for people versus some of these other fish that are just fish [...]” (#2-5). Moreover, repeated

questioning in focus groups showed that respondents found it useful and instructive to have different

attributes to represent effects on broader fish populations (of all fish) and populations of recreationally

and commercially significant species. The following quote is typical: “[...] it was nice to know that the

differentiation between commercial fish and recreational fish and the forage fish ... having that

information kind of influenced some of my votes I would say” (#7-10). This evidence suggests little of the

confusion claimed within this comment.

From a technical perspective, the Agency notes that statistical independence among different choice

attributes is assured by the experimental design. In fact, the choice attributes are nearly orthogonal within

the Agency’s proposed design, providing much lower collinearity (approximately zero) than is possible in
12

virtually any other type of economic valuation. This standard practice within choice experiments allows

implicit prices associated with each attribute to be statistically isolated. The Agency has used a Defficiency criterion for the experimental design, which optimizes the design in terms of such factors as

orthogonality and balance (Kuhfeld and Tobias 2005; Lusk and Norwood 2005), thereby minimizing the

influence of any possible collinearity on model results. This approach, when applied appropriately,

explicitly minimizes collinearity among attributes, again suggesting that Commenter’s concerns in this

area are unfounded.

It is unclear what is meant by the Commenter’s claim that “respondents are supposed to ignore levels and

changes in the various scenarios.” To the contrary, the survey explicitly asks respondents to consider

attribute levels and changes in order to answer choice scenarios. Hence, there seems to be little basis for

this contention. Similarly, regarding the implied comparisons across different choice questions, the

survey includes an entire page devoted to instructions in this area. These instructions are as follows

(survey page 12):

“Now you will be asked to consider a new set of policy options for Northeast waters. As you

vote, please remember—

 Questions 5 and 6 present new sets of policy options. These options require a different

mix of technologies in different areas.

 Each question is a separate vote. Questions 5 and 6 cannot be directly compared to each

other, or to Question 4.

 Do not add up effects or costs across different questions.

 Policy costs and effects depend on many factors. Saving more fish does not necessarily

mean that all effects will improve.”

Focus groups clearly indicated that respondents read and understood these instructions. Participants

understood the choice task correctly and generally believed that the survey decision scenario was clear.

Nearly all respondents indicated that they followed the survey’s instruction to evaluate each policy

question on its own and did not compare policy options across pages. A small minority of participants

stated that they did look page across the choice questions when completing the participants looked back

across pages but these participants generally indicated that this was to help them ensure consistency

across their responses. As stated by one participant, “I did look back, not to compare, but I did look back

just to see how I answered [...] to make sure that I’m thinking about this right and being consistent. So I

don’t know that I compared, I just wanted to make – I think it helped me understand” (#3-7). When

probed it was clear that they were answering the choice questions independently. For example:

Moderator: ... were you looking at it in reference to other questions, [or] answering them as a

group?

Response 1: No, on its own.

Response 2: On its own.

Response 3: On its own. [#3-7]

This combined evidence provides no suggestion of the type of claims made by the Commenter in this

section.
13

The final comment in this section relates to the general claim that “The indexes are confusing and overly

broad, especially the index of the health of the overall aquatic environment.” With regard to these

indices, most have been changed substantially from those reviewed for this comment, based on input from

both Agency scientists and from focus groups. The revised indices are both more narrow and more clear.

This includes revisions to the description of the Aquatic Ecological Condition Score, based on the

responses of focus groups and cognitive interviews. Moreover, some of the more speculative suggestions

of this comment (e.g., inclusion of a water quality as a component of the index may cause respondents to

believe that it is being “influenced in some way” by the proposed regulations) have no basis in focus

group or interview results. The Agency placed particular emphasis on this attribute in focus group and

interview testing to assess exactly the type of concerns raised by the Commenter in this section. Results

suggest that these concerns are largely unfounded. Respondents’ comments suggested a generally

appropriate understanding of the Aquatic Ecological Condition Score and why it might change as a result

of the proposed policies. For example, when discussing why changes in “Fish Saved” were different than

changes in “Aquatic Ecological Condition,” one respondent stated “Fish are only part of it, this

ecosystem. You got plants, you got ocean ... all sorts of life so that's why they – it's a different ...

percentage is because fish are only a part of it”(#3-11). This comment was typical of respondents’

understanding of this score. Focus group responses suggested that respondents were acutely aware that

many stressors and changes affect ecosystems, that that changes in regulations on CWIS are only one of

many factors that can influence ecosystems.

In summary, the Agency finds no convincing arguments in this section to support the speculation that

“willingness-to-pay estimates are for something other than protecting forage fish and are inherently

flawed and biased” (p. 26).

IV.E. Hypothetical Bias

The first three pages of section two do not address the specific survey materials or information collection

approaches described in the Agency’s Information Collection Request (ICR). Instead, these pages

provide a summary of selected publications that address the general issue of potential hypothetical bias in

stated preference survey instruments. The Agency notes that this is a well-known concern with stated

preference surveys, accompanied with decades of research addressing its severity and extent, as well as

the effectiveness of approaches for its mitigation. Hence, the majority of this Commenter’s points in this

section are most appropriately viewed not as specific critiques of the proposed survey instrument, but

rather as a comment on stated preference methods in general. The Agency has no comment on this

literature review.

As a precursor to more detailed comments, the Agency notes that it is fully aware of the research

addressing the potential for hypothetical bias in stated preference welfare estimation. The Agency

emphasizes that the majority of the citations used within this section were published at least ten years ago,

prior to a substantial body of research addressing potential mechanisms for hypothetical bias mitigation.

The Agency also believes that the Commenter, in summarizing this literature, has provided a somewhat

one-sided perspective on the relevant published research. For example, even after reviewing much of the

older information on hypothetical bias cited by the comment, the NOAA Blue Ribbon Panel stated that,

“[t]he Panel concludes that under [conditions described in the report], CV studies convey useful

information. We think it is fair to describe such information as reliable by the standards that seem to be

implicit in similar contexts, like market analysis for new and innovative products...” (Arrow et al. 1993,

p. 4610). The Panel also states that “[hypothetical bias] may be offset or even over-corrected” through
14

conservative design practices. A broad review and meta-analysis of the hypothetical bias literature is also

provided by Murphy et al. (2005), who find a median ratio of hypothetical to real WTP of only 1.35. The

Agency believes that this broader, more comprehensive analysis shows are more accurate picture of the

extent of hypothetical bias than the narrow selection of the literature reflected in the comment. Moreover,

Murphy et al. (2005) clearly indicate that the types of findings cited in this comment—including

hypothetical WTP estimates that are “four to six times” actual WTP—are best described as outliers in the

literature. In fact, one of the conclusions of Murphy et al. (2005, p. 323), that “the overwhelming majority

of observations have relatively low [hypothetical bias], possibly suggesting that hypothetical bias may not

be as significant a problem in stated preference analyses as is often thought.” Hence, while the Agency

agrees that hypothetical bias is a concern—and has taken this very seriously in survey development and

testing—it does not agree that substantial hypothetical bias is unavoidable.

A more comprehensive review of the recent literature would also have revealed methodological advances

provide various means for the amelioration of hypothetical bias. For example, many researchers have had

success using cheap talk methodology, certainty adjustments, and other methods shown to have the

promise to largely eliminate or at a minimum reduce hypothetical biases (e.g., Champ et al. 1997, 2004;

Cummings and Taylor 1999; Ready et al. 2010). Recently published findings of Ready et al. (2010, p.

363), for example, show that contemporary certainly adjustments “successfully mitigated hypothetical

bias.” While the evidence regarding the ability of researchers to ameliorate hypothetical biases is mixed,

there is no evidence that such biases are unavoidable. To the contrary, as noted recently by Murphy et al.

(2010), there is a “small but growing literature that suggesting the absence of hypothetical bias in

aggregate in induced value settings...”

The Agency notes that many of research efforts that provide evidence for the presence of hypothetical

biases are carried out in simulated markets, and hence do not reflect true assessments of criterion validity

(Bateman et al. 2002). While simulated markets—experimental or otherwise—might generate unbiased

WTP estimates, they are often conducted in institutional settings unfamiliar to respondents. Such settings

may generate novel or unusual behavior, confusion, distrust, or other behaviors that may lead to

misstatements of true, underlying WTP. For such reasons Bateman et al. (2002) argue that comparisons

involving WTP estimates obtained through simulated or experimental markets do not qualify as true tests

of criterion validity (i.e., they do not provide unambiguous measures of true willingness to pay in actual

markets). The most common alternative—the use of individual or group donation mechanisms (e.g.,

Foster et al. 1997; Seip and Strand 1992)—suffers from confounding effects of free-riding and the

associated lack of incentive compatibility (Bateman et al. 2002; Shläpfer et al. 2004) As a result, as

stated by Murphy and Stevens (2004, p. 185), one cannot unambiguously claim that values estimated

using laboratory experiments are any more representative of true willingness to pay than values estimates

using hypothetical markets.

While critics of stated preference methods often claim insurmountable divergences between stated and

real market behavior, evidence from various comparisons of actual versus hypothetical behavior

contradicts the claim that hypothetical bias is unavoidable, particularly in the aggregate. For example, as

conceded in the comment to the Agency, numerous comparisons of hypothetical versus binding referenda

find close correspondence between choices in stated preference surveys and voting behavior in binding,

consequential referenda (Bateman et al.2002, p. 318; Schläpfer et al. 2004; Johnston 2006; Vossler et al.

2003). Hence, in these true tests of criterion validity, responses in hypothetical and real contexts comport

quite closely, suggesting that stated preference surveys can provide useful indications of that which

individuals “are willing to forgo to enjoy a particular benefit” (OMB Circular A-4).
15

After conceding that many comparisons of hypothetical and binding referenda have not shown evidence

of hypothetical bias, the Commenter goes on to claim that these findings are not applicable to the Agency’s

proposed survey efforts, largely due to the fact that the Agency’s survey provides “substantial information to

the respondents” (p. 29). These arguments, however, are speculative and are not supported by systematic

findings in the published literature; the Agency is aware of no published literature which suggests a

systematic linkage between information provision and hypothetical bias of the type claimed within the

comment. Moreover, the comment fails to mention that the survey has been explicitly designed to

maximize the consequentiality and salience of choice experiment questions, through direct linkages to

proposed Agency regulatory efforts. Hence, according to the criteria proposed by the comment, there are

numerous aspects of the proposed survey that should in fact decrease the potential for hypothetical biases

(e.g., salience).

The Agency, however, recognizes that hypothetical bias is an important consideration in stated preference

design, and takes this concern very seriously. Because of this, extensive survey testing in focus groups

and cognitive interviews (cf. Mitchell and Carson 1989; Desvousges and Smith 1988; Desvousges et al.

1984; Johnston et al. 1995; Powe 2007) was conducted to assess the extent to which such biases might be

prevalent. Evidence from this extensive qualitative testing provided little evidence of hypothetical bias.

For example, focus group and interview transcripts show that, when asked explicitly, respondents almost

universally indicated that their answers to choice questions in the survey instrument would be identical if

the same questions were encountered in a binding referendum. Moreover, the survey explicitly

incorporates elements to enable mitigation of any remaining hypothetical bias, such as a certainty followup questions (Ready et al. 2010).

The Agency recognizes the possibility of hypothetical bias—and indeed takes this possibility very

seriously. As noted in the ICR, the Agency will follow the published literature in designing

mitigation strategies to eliminate or at a minimum reduce the potential for hypothetical bias.

Section V: Assessment of the Need for the Proposed EPA Survey

V.A: Non-Use Benefits Not Likely to Be Significant in This Case

In this section, the commenter argues that the review of the both existing literature and relevant economic

theory indicates that none of the rationales for including non-use values in the benefits analysis of the

316(b) regulation offered by EPA are particularly compelling. The commenter further states that “nonuse values are expected to be significant only when the resource in question is unique and will not recover

for an extended period of time.”

As noted above, EPA disagrees with the commenter that non-use values for fish protection are likely to be

insignificant and notes that the economic literature advises against the use of ad hoc, expert assumptions

regarding the magnitude of non-use values—particularly assumptions that non-use values are trivial

(Bateman et al. 2002). Based on the guidance from economic literature (e.g., Bateman 2002), EPA

believes that empirical analysis should be used to determine the magnitude of non-use values in the

316(b) regulation for existing facilities. Furthermore, results from the focus groups and cognitive

interviews indicated that some individuals hold substantial non-use values for protecting all fish species

including forage fish. See EPA’s response to Section II for more detail.
16

In addition, the commenter argues that existence motives for non-use value for fish are very limited. In

particular, “whether the forage fish, which are primarily consumed by other large fish would be among

the fish population that people care about, is subject to debate at best.” The Agency notes that participants

in the EPA focus groups and interviews cited a variety of motivations for preventing fish losses, including

the satisfaction of knowing that the fish exist, the desire to bequeath healthy fish populations to future

generations, and the desire to protect the functioning of aquatic ecosystems. The Agency further notes

that the referendum format of the survey allows respondents who do not hold non-use values for fish to

choose policy options that reflect their preferences; respondents need not value the existence of fish, nor

do they need to be able to separate their use and non-use values.

V.B: The Proposed EPA Survey Cannot Isolate Non-Use Values

In this section, the commenter argues that the proposed EPA Survey cannot isolate non-use values. EPA

points out that the purpose of the survey is to measure total WTP for the changes discussed in the

survey—including both use and non-use values. When the results of the survey are used to estimate

benefits in the context of the 316(b) regulation, EPA will approximate non-use values associated with the

policy changes based on total WTP of non-users—a method characterized as “...one of the more

defensible means of distinguishing use and nonuse values in a stated preference context” (Johnston et al.

2005). A primary advantage of this approach is the general conceptual acceptance of nonuser values,

even among those highly critical of surveys that attempt to isolate nonuse values among the general

population (Cummings and Harrison 1995). .

Additional comments in this section suggest additional confusion among the commenter with regard to

the appropriate measurement of values in the 316(b) policy case. For example, the commenter states

“even a respondent who does not use resource directly may reveal preferences for large fish populations

and sustainable commercial fisheries because they like to eat fish” and that these respondents would be

incorrectly included as nonuse. As noted above, however, it is perfectly valid for nonusers to maintain

nonuse values for recreational and commercial fish that are in addition to the use values that might be

realized by users. That is fact is not noted by the commenter suggests, again, a fundamental flaw in

reasoning and understanding of accepted conceptual models for benefit cost analysis (Boardman et al.

2001).

V.C: There exist Reliable Methods for Measuring the Value of Forage Fish and Unharvested Commercial

and Recreational Fish

This section of the Comment argues that: (1) “unharvested recreational benefits are included within the

category of non-market (direct use) benefits” and (2) “forage fish benefits are included within the

category of indirect (use) benefits.” Therefore, there is no need for EPA to develop a proposed survey to

include these benefits in its evaluation 316(b) alternatives. These statements show the commenter’s

confusion about the multiple ways in which commercially and recreationally valuable fish may influence

the utility of individuals. Specifically, it fails to recognize that there are two legitimate sources of

economic value related to these fish: use benefits and non-use benefits. As noted in the Comment,

commercial and recreational anglers benefit from catching these species. However, only a small

percentage of the recreationally and commercially valuable fish whose losses would be prevented by the

316(b) regulation will be caught by anglers. The remaining fish would enter individuals’ utility functions

by providing non-use benefits—through their existence—that have little or no relationship to the role of

these species in recreational and commercial fisheries. EPA emphasizes the well-known tenet of
17

economic theory which states that benefits or costs from multiple direct (primary) effects on markets

should be counted within a benefit-cost analysis; this is not double counting, and is indeed appropriate

(Just et al. 1982). Comprehensive estimates of total resource value should include both use and non-use

values, such that the resulting total value estimates may be compared to total social cost . “Non-use

values, like use values, have their basis in the theory of individual preferences and the measurement of

welfare changes. According to theory, use values and non-use values are additive” (Freeman, 2003, p.

154).

Both the market valuation and revealed preference methods used to estimate benefits of improved

commercial and recreational harvest allow EPA to estimate direct use values only. These methods are not

suitable for estimating total values (use and nonuse) of harvested species and therefore would

underestimate total value (Freeman 2003). Both recreational and commercial and forage fish attributes

may enter individuals’ utility functions directly, providing non-use benefits that bear little or no

relationship to (and are not subsumed by) use benefits generated by recreational and commercial fish.

EPA refers to Freeman (2003), who provides a clear distinction between use benefits (indirect and direct)

and true non-use benefits.

Finally, EPA re-iterates that nearly all (98 percent) of impingement and entrainment losses at CWIS

consist of either forage species, or non-landed recreational and commercial species that do not have direct

uses or, as a result, direct use values. Therefore, EPA believes that (1) only a small fraction of potential

benefits associated with the 316(b) regulation is quantified in EPA’s benefit cost analysis of the proposed

316(b) regulation for the Phase III facilities and (2) developing comprehensive quantified benefit

estimates for the section 316(b) regulation requires consideration of non-use values.

Appendix 1: Detailed Assessment of Survey Design Elements in EPA’s Proposed Survey

This section of the comment includes a variety of claims regarding the validity of the survey instrument.

As a precursor to specific responses below, the Agency notes that almost all of these issues have been

subjected to extensive testing in focus groups. Where problems were found, the survey has been revised

accordingly. Hence, many of the survey design elements noted below have already been removed or

changed. In many other instances, however, focus groups showed little cause for concern, suggesting that

many of the speculative claims below have little basis in fact. Specific responses are as follows:

1. Summary of Comment: “The cover page titles the survey as Fish and Aquatic Habitat. This is

likely to cause the respondent to think about all fish and all habitats that support fish. ... At an

absolute minimum, EPA should add questions to assess the extent that people think the fish

impacted by I&E are more likely to be commercial and recreational fish. The Agency should also

add probes on the likelihood that people think overfishing would also be limited by the policy and

that habitat would be improved in meaningful ways.” Agency Response: These issues were

probed extensively in focus groups. As noted in responses above, focus groups showed little

evidence of the type of broad confusion claimed by this comment. In contrast, focus groups often

included respondent-initiated discussions of such issues as the specific technologies used to

mitigate entrainment and impingement, the graphics used to communicate entrainment and

impingement dynamics, and other issues which demonstrated relatively clear understanding of the

policy issues at hand. There was no evidence that respondents thought that the proposed policies

would limit overfishing. Cognitive interviews and focus groups also probed reactions to each

page of the survey, and no concerns were raised about the title page. The revised survey also
18

includes numerous debriefing questions to assess the motivations for choice experiment

responses.

2. Summary of Comment: The second comment in this section is related to a claim that the survey

design provides deceptive information on the policy context and scope of the resource that would

be affected. Agency Response: See the Agencies response to IV.B. Bias Due to Lack of Match

Between Survey Information and the Regulatory Situation; the same responses apply here.

3. Summary of Comment: “the map is very hard to read, making it difficult for respondents to have

any clear idea of what water bodies are likely to be impacted by the rule.” Agency Response:

The interpretation and clarity of the map was probed extensively in focus groups and was

changed as a result of focus group responses (Besedin and Stapler 2010). The specific map in

question is no longer present in the revised survey instrument. The revised map is designed to

clearly identify facilities using cooling water intake structures, as suggested in this comment.

4. Summary of Comment: “What Kinds of Fish Are Affected?” This discussion is very confusing.

Agency Response: See the Agencies response to IV.B. Bias Due to Lack of Match Between

Survey Information and the Regulatory Situation; the same responses apply here. These issues

were probed extensively in focus groups to ensure a sufficient match between respondents’

understanding of the policy context and the effects of entrainment and impingement as

characterized by Agency data. This section was also revised significantly as a result of focus

groups, to minimize any confusion that occurred.

5. Summary of Comment: Question 1. Significant clarification is needed here. Agency Response:

This question is no longer present in the survey instrument.

6. Summary of Comment: The number of fish lost should not be given out of context. (This refers to

page entitled “How Many Fish Are Affected? A Northeast Coastal Example” in the original

survey.) Agency Response: This entire page has been extensively reworked to provide improved

context on the magnitude of fish losses, based on the input of focus groups. Almost none of the

original text has been retained in the revised page in question. See also the Agency’s response to

IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the

same responses apply here.

7. Summary of Comment: What does this fish loss really mean? Is the health and sustainability of

the fish stocks affected? (This refers to page entitled “How Many Fish Are Affected? A

Northeast Coastal Example” in the original survey.) Agency Response: This entire page and the

survey as a whole has been extensively reworked to provide improved context on the magnitude

and impact of fish losses, based on the input of focus groups. Almost none of the original text

has been retained in the revised page in question. See also the Agency’s response to IV.B. Bias

Due to Lack of Match Between Survey Information and the Regulatory Situation; the same

responses apply here.

8. Summary of Comment: The flounder example is deceptive. (This refers to page entitled “How

Many Fish Are Affected? A Northeast Coastal Example” in the original survey.) Agency

Response: The flounder example has been removed from the revised survey.

9. Summary of Comment: There is not enough information provided about the proposals for people

to determine whether or not the policy is worth it. − How much do these modifications cost?

Agency Response: This comment confuses the cost of a policy with the benefit of a policy. It is

well-established within economic theory and practice that the cost of a commodity is entirely
19

unrelated to the potential benefit provided by that commodity (Just et al. 2004). This is the

reason that stated preference surveys in the literature omit information on commodity cost when

evaluating WTP. The Agency has followed this standard practice here—as clearly supported by

economic theory.

10. Summary of Comment: Not mentioned anywhere in this survey is the fact that there could be

other environmental policies that would provide larger benefits to fish than regulating the cooling

systems of the plants and potentially at a lower social cost Agency Response: The purpose of the

proposed stated preference survey is stated clearly in the ICR and supporting statements: the

survey instrument “will be used to elicit values for changes in fish mortality and the associated

ecological gains from a section 316(b) regulation covering Phase II and Phase III facilities.” The

purpose of the survey is not to estimate benefits of “other environmental policies that would

provide larger benefits to fish.” While the Agency agrees that there are other possible policies

that might benefit fish, these are not relevant in the present ICR or associated survey instrument.

11. Summary of Comment: “Would You Vote for Policies to Help Protect Fish and Fish Habitat?”

Again the title implies that the benefits of the policies are more extensive than just fish loss.

Habitat will not be affected by the proposed policy. Agency Response: The capacity of habitats

affected by CWIS operations to support aquatic life is affected by entrainment and impingement

effects. Because fish habitat is influenced by entrainment and impingement effects, the proposed

policies do influence fish habitat. Moreover, there is no evidence from focus groups that

respondents broadly misunderstood the benefits of the proposed policies. See also the Agency’s

response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory

Situation; the same responses apply here.

12. Summary of Comment: Question 2. These are very broad and significant categories of issues that

the respondents are asked to rate Agency Response: This question is no longer present in the

survey instrument.

13. Summary of Comment: ““This Survey is Similar to a Public Vote”: There does not seem to any

evidence to support [the attribute levels included in the survey].” Agency Response: The

commercial fish sustainability score is no longer included in the Agency’s survey. Hence,

comments regarding this score are no longer relevant. With regard to attributes associated with

fish populations, the comment claims that “EPA needs to provide some evidence that fish

populations are affected to include this in the survey as part of the scenario.” The same claim is

made for the condition of aquatic ecosystems. The Agency disagrees with this argument, and

notes that the proposed comment suggests a lack of understanding of the capacity of choice

experiments to isolate the marginal effects of different attributes on stated choices and hence on

WTP—a methodology clearly described by numerous sources (e.g., Louviere et al. 2000). The

choice experiment survey and experimental design are explicitly designed such that the model

can incorporate a wide-range of potential outcomes, including zero changes in fish populations

and ecological condition. That is, the choice scenarios (or profiles) explicitly evaluate

respondents choices and implied values for cases in which changes in ecological outcomes

(including changes in fish populations) are both large and small. Unlike older forms of stated

preference analysis in which WTP is contingent upon a single specification of ecological effects,

choice experiments solicit responses under a large number of possible attribute configurations.

This will allow the Agency, for example, to estimate WTP for policies in which the broader

ecological effects of I&E reductions—beyond direct influences on the number of age-1

equivalent fish lost—are zero. Hence, contrary to the claim of this comment, it is not true that

the Agency needs to provide prior evidence “that fish populations are affected to include this in

the survey as part of the scenario.”
20

14. Summary of Comment: “1. Commercial Fish Sustainability Score”: The idea of using a

commercial fishing sustainability score is inappropriate. Agency Response: The commercial fish

sustainability score is no longer included in the Agency’s survey.

15. Summary of Comment: “2. Fish Population Score”: Using fish population numbers without any

scientific data to support it is inappropriate. Agency Response: The Agency notes that the ranges

for this attribute are grounded in the best available data. Responses in #13 above also apply here.

16. Summary of Comment: “The Aquatic Ecosystems Score”: This index is completely unrelated to

fish loss from intakes. Agency Response: The score is determined by many factors including

water temperature, the presence and abundance of aquatic species, including species affected by

cooling water intakes, and habitat conditions. Responses in #13 above also apply here.

17. Summary of Comment: Only category one and five can truly be linked to the policy the

respondents are considering in the scenarios: fish loss prevention and cost to household. Including

the effects on the sustainability of commercial fishing and fish populations and aquatic ecosystem

misleads the respondent to think that such impacts are on the same order of magnitude as

potential changes in the numbers of forage fish. Agency Response: There is no information in

this section that implies, in any way, that “such impacts are on the same order of magnitude as

potential changes in the numbers of forage fish.” Information provided in the choice questions

clearly demonstrates that the effects on attributes may differ. Focus group responses clearly

indicated that respondents understood these differences. The following quote is typical, and

shows the respondent clearly distinguishing between the magnitude of impacts on fish losses and

impacts on other attributes: “[...] when I compared A and B, they’re virtually the same statistics

except on fish saved per year, where there were a lot more fish saved” (#5-6). See also the

Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the

Regulatory Situation; the same responses apply here.

18. Summary of Comment: All of the categories the respondents are asked consider appear to be

correlated. Therefore, the respondent would find it hard to not rank the first four at the same level.

Agency Response: Comments related to correlation of ecological attributes are addressed in the

Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes.

Focus groups show little evidence of the type of difficulties claimed within this comment.

19. Summary of Comment: The relationship between commercial fishing sustainability and fish

populations does not make sense. Agency Response: The commercial fish sustainability score is

no longer included in the Agency’s survey.

20. Summary of Comment: “As You Vote Please Remember”: The third bullet is confusing:

“Depending on the type of technology required and other factors, effects on fish and ecosystems

may be different – even if the annual reduction in fish losses is similar.” Agency Response: The

meaning of this statement is clear and self-evident. Focus groups showed no evidence of

confusion related to this statement.

21. Summary of Comment: Attribute levels in questions 4, 5 and 6 are likely to be confusing to

respondents. Agency Response: Focus group responses do not support the Commenter’s claims.

The Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying

Attributes address these comments. Moreover, the majority of focus group participants indicated

that they were confident with their responses to questions 4, 5 and 6—again suggesting that the

claims of respondent confusion here are overstated. For example, participants made statements
21

such as: “Yeah, I’m pretty much confident” (#3-8), and “If I believed that it was gonna affect

regulations, I think I would have voted the same exact way.” (#7-9).

22. Summary of Comment: Comments on question #8 and on National vs. Northeast Policies Agency

Response: This material is no longer included in the revised survey.

Appendix 2: The Proposed EPA Survey Failed to Adequately Respond to Peer Reviewer

Comments

As a precursor to later comments, the Agency highlights that the survey proposed in the ICR, and tested

in focus groups, bears little direct resemblance to the survey that was peer reviewed in 2005. Hence,

many of the peer review comments are no longer relevant. The Agency, however, takes all peer review

comments very seriously, and has accounted for these comments in all survey revisions and in

development of the present stated preference survey.

Appendix 2.A. Background on Peer Review Process

This section does not address the specific survey materials or information collection approaches described

in the Agency’s Information Collection Request (ICR). The Agency has no comment on this section.

Appendix 2. B. Evaluation of the Current EPA Proposed Survey Based on Peer Review Comments

Most of the introductory paragraphs in this deal with revisions that the Agency has already made in direct

response to peer review comments. Claims of remaining limitations are almost universally related to

issues that have been comprehensively addressed above. These are as follows:

1. Summary of Comment: Removal of photos from the survey is a step in the right direction for

reducing bias; however, it does not go far enough as the survey continues to lead the respondent

to believe that I&E has farther reaching effects than science can support Agency Response: This

issue has been dealt with comprehensively above. See the Agency’s response to IV.B. Bias Due

to Lack of Match Between Survey Information and the Regulatory Situation; the same responses

apply here.

2. Summary of Comment: The overall tenor of the information is one in which uncertainty is

minimized relative to the actual uncertainty associated with the effects. Agency Response: The

revised survey is explicit about the lack of certainty, and this was clearly acknowledged by focus

group participants. Participants understood that the ecological changes described in the survey

were uncertain, and most participants were comfortable making decisions in the presence of this

uncertainty. Their responses indicated that they understood this uncertainty based on the

information presented in the introductory material and considered it when evaluating policy

options. For example, one participant referred to Page 4, which addresses scientific uncertainty

as, “That one kind of the disclaimer page” (#7-19). Some participants also expressed that they

weren’t surprised at this uncertainty, with statements such as “I wasn’t surprised at that” (#4/6-

7). Multiple participants expressed that they perceived the values as “best guess”: “My guess is

that it did come from studies but I have a healthy dose of skepticism about the accuracy of it. I

don’t think it’s been in any way skewed purposefully, but I know that this is a best guess,

reasonable guess perhaps” (#5-10), “it shows me that they are being honest for the most part.

You know, you can't obviously be accurate on everything, but this is a kind of a best guess” (#4/1-

3), “[...] They had more numbers on the commercial fish population. The rest was more of a

guesstimate” (#5-15), and “you don’t know the exact number and nobody knows” (#4/3-4).
22

Overall, participants felt comfortable making decisions in the presence of uncertainty, but some

participants had differing opinions on how it impacted their choices: “No it didn’t – it didn’t

either way” (#4/4-7) and “I’m assuming if you’re a scientist and you are doing studies, I’m

taking you at your word. I still don’t think that anyone’s capable of complete accuracy in this

mater – I guess it did affect my answers because I think that taking them that they can do it pretty

decently” (#4/5-3). Neither the content of the survey nor focus group responses support the claim

that “uncertainty is minimized.”

3. Summary of Comment: “The addition of habitat quality to the attributes in the choice sets, in an

attempt to address that issue in the mind of respondents...” Agency Response: There is no

“habitat quality” index or attribute in the proposed survey instrument.

4. Summary of Comment: Finally, we do not think that EPA has addressed the peer reviewers’

concerns about the definition of baseline. Based on our review, baseline continues to be poorly

defined. Agency Response: Baselines are clearly defined for each of the attributes on page 7 in

the revised survey, in both ecological and common language terms. The Agency notes that these

attributes are distinct from those in the original survey version reviewed for this comment, and

that the baselines are different. Therefore, this comment is not directly relevant to the current

survey.

Responses to Specific Peer Review Comments in Appendix 2, Table 2

1. Summary of Comment: “There is no baseline or baseline is poorly defined.” Agency Response:

All of the attributes have been changed significantly from the original survey version, including a

more clear definition of all baselines. These include significant changes to the attributes as

defined in the version of the survey reviewed by the Commenter. For example, the commercial

fish sustainability score is no longer included in the Agency’s survey, so related comments are no

longer relevant. The baseline for fish losses is clearly and unambiguously stated on page 7 of the

revised survey. The fish losses attribute is described as: “A score between 0 and 100 showing the

reduction in young fish lost because of the new policy. A score of 100 means that no fish are lost

in cooling water intakes (all fish are saved). The current score in Northeast waters is 0; this

represents the status quo with no new policies.” The commercial fishing attribute has been

revised entirely, with a different definition and new baseline. Hence the comment on the

originally-defined attribute is no longer relevant.

2. Summary of Comment: “The commodity is defined simply as fish.” Agency Response: The

commodity is no longer simply defined as fish. The revised survey incorporates four ecological

attributes, communicating effects on commercial and recreational fish populations, all fish

populations, and aquatic ecological condition. This directly responds to the peer review comment

of Krupnick in this section.

3. Summary of Comment: “Uncertainty is not handled properly.” Agency Response: See the

Agency’s response to Appendix 2. B. Evaluation of the Current EPA Proposed Survey Based on

Peer Review Comments, comment #2. This comment details the Agency’s treatment of

uncertainty in the revised survey and related focus group responses. Beyond this, the choice

experiment survey and experimental design are explicitly designed such that the model can

incorporate a wide-range of potential outcomes, including zero changes in population and harvest.

In this way, the model can account for uncertainty, allowing sensitivity analysis of cases in which

I&E reductions result in zero, small, or large changes in various attributes.
23

4. Summary of Comment: “The attributes are interdependent in complex ways.” Agency Response:

The Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying

Attributes address these comments. See also the related Response to Appendix 1: Detailed

Assessment of Survey Design Elements in EPA’s Proposed Survey, comment #21.

5. Summary of Comment: “The survey may create bias based on the sponsor..” Agency Response:

The Commenter argues here that the identification of EPA as the sponsor of the survey somehow

contributes to an inappropriate bias. The Agency rejects the suggestion that this information

should be withheld from respondents, and notes that there is nothing in the literature that suggests

that such information should be omitted. To the contrary, the literature clearly emphasizes the

importance of mechanisms that reinforce the consequentiality of stated preference surveys and

survey responses (Carson and Groves 2007). The emphasis on a specific government agency

within survey materials is clearly consistent with Carson and Groves (2007) arguments

concerning the important role of consequentiality in stated preference surveys, defined as a case

in which “survey results are seen by the agent as potentially influencing agency action” and in

which the agent treats the survey as “an opportunity to influence those actions.” Recent work by

Johnston (2006) also clearly emphasizes the potential benefits of consequentiality in stated

preference survey instruments, and recommends identifying government sponsors of policy issues

addressed by survey instruments as a means to eliminate hypothetical bias. Moreover, the

Agency has found nothing in the literature to suggest that identifying the government agency

associated with particular policy actions will bias WTP responses.

6. Summary of Comment: “Payment vehicle may cause rejection.” Agency Response: the cost of

hypothetical programs are clearly described following standard practice in stated preference

surveys. Given this—and the results of focus groups—the Agency can find no evidence that the

payment vehicle will lead to scenario rejection. The current format of the payment vehicle is the

result of focus groups on the original version of the 316(b) survey instrument in 2005, with

additional testing in the more recent 2010 focus groups. These focus groups and discussions with

both OMB and NCEE in 2005 ultimately led to the current description of the payment vehicle.

Focus groups showed no difficulty understanding the format of the payment vehicle, and

respondents clearly understood the cost as an ongoing cost. The Agency also notes that a similar,

ongoing cost of living payment vehicle has been used by prior EPA-sponsored research, with no

apparent difficulties (Magat et al. 2000). Participants in 2010 focus groups clearly understood

that selecting Options A or B would result in increased costs to their household. They understood

that costs to industry would be passed on to consumers, with one participant stating, “So my

thought process is that the household stuff is being passed down to consumers from the cost of the

added regulation they have” (#3-4). The draft survey was also revised to improve clarity

regarding how costs would be passed to consumers by adding a clause on Page 1 stating that

goods and services with costs would include “electricity and common household products”. The

definition on Page 7 was also revised to include “services” purchased by consumers. These

revisions greatly reduced the potential for respondent confusion when evaluating policy options.

Respondents also appeared to understand the ongoing, monthly nature of the payments. Despite

the Commenter’s suggestions to the contrary, there is little evidence from focus groups and

interviews that respondents misunderstood the payment vehicle, or that there were any instances

of payment vehicle rejections. A few respondents expressed that companies should be bearing

the costs, rather than consumers. For example, a participant stated, “Instead of passing these

charges to us, why aren’t they passing them to these companies [...] that are responsible” (#2-4).

However, there was a general understanding among most participants that “[...] ultimately it’s all

us consumers that ultimately pay for everything” (#2-4).
24

7. Summary of Comment: “Wording of payment vehicle could be improved.” Agency Response:

See response to comment #6 directly above.

8. Summary of Comment: “Wording of payment vehicle could be improved.” Agency Response:

See response to comment #6 directly above.

9. Summary of Comment: “Dollar amounts are too small.” Agency Response: Focus group

responses show that most participants considered their budget when choosing between specific

policy options, and that the stated costs were of sufficient magnitude to be relevant. For example,

some of them stated “So, I don’t go out to eat twice (#5-12)”, “[...] if I was working at my old

job making the money that I was making, I would probably have chose a higher amount” (#4/5-

12), and “the cost per month came out to only $3 per month, which I feel is very affordable” (#2-

6). Some participants stated that household cost was the most important factor in their decision:

“Yeah. Absolutely” (#4/4-8). Some respondents didn’t view the household cost amounts under

the policy options as significant relative to their personal budget constraints: “[...] $2 per month

doesn’t seem like a whole lot of money. So, you almost feel guilty like 2 bucks a month, I’ll save

some fish” (#7-4). Other participants indicated that they would have to think harder about budget

trade-off if the amounts were higher. For example, one respondent stated, “If you put a zero

behind one of the numbers, I’d have to think about it very harder than, you know, 24 dollars a

year” (#5-12). Many other respondents, however, indicated that the payment vehicle amounts

were sufficient to change their responses: “See, I’m unemployed right now, so I’m looking at

every little dollar, where it’s going. That’s what affected me” (#6-5).

10. Summary of Comment: “The graphics could lead to bias.” Agency Response: Since the peer

review, all of the graphics in the original survey have been changed. Given that this comment

regarded graphics that are no longer in the survey instrument, it is no longer relevant. All new

graphics were tested extensively in focus groups. Participants endorsed the use of graphics in the

survey booklet. Most felt that the graphics helped them to visualize how fish are entrained and

impinged, technological solutions, facilities locations, and ecosystem effects. For example, some

respondents stated: “I mean, the filter thing, where you can see X number of fish on one side and

then only the little ones are getting through, it kind of tells you what’s going on”(#1-5), “After

reading the previous page and then reading this picture clicked [...] I got it” (#7-10), and “I think

it was good to visualize the problem in the graphic” (#7-10). Participants also felt that the

graphics improved their understanding of these issues and tended to think of them as “providing

information” (#6-14), rather than pushing them to answer a certain way. Many respondents

indicated that they understood the meaning of the graphics: “I think they were easy to follow. It

was pretty straightforward” (#1-10) and “It looks clear” (#5-5). However, some respondents

provided feedback regarding aspects of the various graphics which they found confusing and on

how they might be improved. Participants’ comments during the focus groups were used to refine

the graphics within the draft survey in order to improve clarity and reduce the potential for

respondent confusion. Participants’ comments also indicated that they had little emotional

reaction to the image on the cover of the survey and that it did not influence their survey

responses: “It’s a nice picture, but it didn’t do anything, you know, wow, spectacular” (#4/5-3), “I

noticed it. I didn’t – as far – kind of gave me an idea what the topic was going to be about” (#4/1-

2), and “... looking at it without knowing any of this inside, I think my impression was, it’s

colorful and kind of blurry” (#4/8-10).

11. Summary of Comment: “Region of concern is unclear..” Agency Response: The survey now

deals only with one region. This region is clearly identified on the map.
25

12. Summary of Comment: “Protect fish stocks is a biased statement.” Agency Response: As noted

by the Commenter, this statement has been removed as requested by the peer review.

13. Summary of Comment: “The Indication of a declining baseline” may influence responses. Agency

Response: As noted by the Commenter, this statement has been removed as requested by the peer

review. The survey no longer mentions a declining baseline.

14. Summary of Comment: “Language regarding lost fish is vague..” Agency Response: All of the

language in question has been revised in the survey instrument. See the Agency’s response to

IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the

same responses apply here.

15. Summary of Comment: “Table on Slide 19 is confusing.” Agency Response: This graphic is no

longer in the survey.

16. Summary of Comment: “Pie charts are not very helpful when nearly all of the chart is one color.”

Agency Response: The charts in the survey have all been revised, and use two colors as

requested.

17. Summary of Comment: ““Technologies” is unclear.” Agency Response: The survey now clearly

identifies and describes the technologies in question on page 5. This is entirely new material that

has been added to the survey to address this peer review comment.

18. Summary of Comment: “Technologies are already in place needs to be explained” Agency

Response: The survey now clearly addresses this issue on page 5. This is new material that has

been added to the survey to address this peer review comment.

19. Summary of Comment: “Status Quo needs to be a separate choice” Agency Response: The

Agency agrees. The status quo is a separate choice in the revised survey.

References

Arrow, K. , R. Solow, E. Leamer, P. Portney, R. Rander, and H. Schuman. 1993. “Report of the NOAA

Panel on Contingent Valuation.” Federal Register 58(10): 4602-4614.

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S.

Mourato, E. Ozdemiroglu, D.W. Pearce, R. Sugden, and J. Swanson. 2002. Economic Valuation

with Stated Preference Surveys: A Manual. Northampton, MA: Edward Elgar.

Bennett, J. and R. Blamey, eds. 2001. The Choice Modeling Approach to Environmental Valuation.

Northampton, MA: Edward Elgar.

Carson, R.T. and T. Groves. 2007. Incentive and Informational Properties of Preference Questions.

Environmental and Resource Economics 37(1):181–210.

Carson, R.T., N.E. Flores, and R.C. Mitchell, The Theory and Measurement of Passive-Use Value, in

Valuing Environmental Preferences: Theory and Practice of the Contingent Valuation Method in
26

the US, EU, and Developing Countries, edited by I.J. Bateman and K.G. Willis, Oxford

University Press, Oxford, UK, 1999.

Champ, P.A., and R.C. Bishop. 2001. “Donation payment mechanisms and contingent valuation: An

empirical study of hypothetical bias.” Environmental and Resource Economics 19:383–402.

Champ, P.A., R.C. Bishop, T.C. Brown, and D.W. McCollum. 1997. Using donation mechanisms to

value non-use benefits from public goods. Journal of Environmental Economics and Management

33(2):151–162.

Champ, P.A., R. Moore, R.C. Bishop. 2004. “Hypothetical Bias: The Mitigating Effects of Certainty

Questions and Cheap Talk.” Selected Paper prepared for presentation at the American

Agricultural Economics Association Annual Meeting, Denver, Colorado.

Collins, J.P. and C.A. Vossler. 2009. Incentive Compatibility Tests of Choice Experiment Value

Elicitation Questions. Journal of Environmental Economics and Management 58(2): 226-235.

Cummings, R. and L. Taylor. 1999. Unbiased Value Estimates for Environmental Goods: A Cheap Talk

Design for the Contingent Valuation Method. American Economic Review 89:649-665.

Desvousges, W.H. and V.K Smith. 1988. Focus Groups and Risk Communication: the Science of

Listening to Data. Risk Analysis 8:479-484.

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in

Designing a contingent Valuation Survey to Measure the Benefits of Hazardous Waste

Management Regulations. Research Triangle Institute: Research Triangle Park, NC.

Dillman, D.A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley

and Sons.

Foster, V., I. J. Bateman, and D. Harley. 1997. Real and Hypothetical Willingness to Pay for

Environmental Preservation: A Non-experimental Comparison. Journal of Agricultural

Economics 48(2):123-138.

Giraud, K.L., J.B. Loomis and R.L. Johnson. 1999. Internal and external scope in willingness-to-pay

estimates for threatened and endangered wildlife. Journal of Environmental Management 56:

221-229.

Hanemann, W.M. 1994. Valuing the Environment Through Contingent Valuation. Journal of

Environmental Perspectives 8(4): 19-43.

Hanemann, W.M. and B. Kanninen. 1999. “The Statistical Analysis of Discrete-Response CV Data.” In

Valuing Environmental Preferences: Theory and Practice of the Contingent Valuation Method in

the US, EU, and Developing Countries. Edited by I.J. Bateman and K.G. Willis, Oxford

University Press, Oxford, UK.

Heberlein, T.A., M.A. Wilson, R.C. Bishop, and N.C. Schaeffer. 2005. Rethinking the Scope Test as a

Criterion in Contingent Valuation. Journal of Environmental Economics and Management 50(1):

1-22.
27

Johnston, R.J. 2006. Is Hypothetical Bias Universal? Validating Contingent Valuation Responses Using a

Binding Public Referendum. Journal of Environmental Economics and Management 52(1): 469-

481.

Johnston, R.J., T.F. Weaver, L.A. Smith and S.K. Swallow. 1995. Contingent Valuation Focus Groups:

Insights From Ethnographic Interview Techniques. Agricultural and Resource Economics

Review 24(1): 56-69.

Kemp, M.A. and C. Maxwell. 1993. Exploring a Budget Context for Contingent Valuation Estimates, in

Hausman, J.A. ed., Contingent Valuation: A Critical Assessment. New York: North Holland,

pp. 217-269.

Kuhfeld, W.F. and Tobias, R.D., 2005. Large factorial designs for product engineering and marketing

research applications. Technometrics 47: 132-141.

Louviere, J.J., D.A. Hensher, and J.D. Swait. 2000. Stated Preference Methods: Analysis and

Application. Cambridge, UK: Cambridge University Press.

Lusk, J.L. and F.B. Norwood. 2005. Effect of Experimental Design on Choice-Based Conjoint Valuation

Estimates. American Journal of Agricultural Economics 87(3):771-785.

Magat, W.A., J. Huber, K.W. Viscusi and J. Bell. 2000. An Iterative Choice Approach to Valuing Clean

Lakes, Rivers, and Streams. Journal of Risk and Uncertainty 21(1): 7-43.

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation

Method. Resources for the Future, Washington, D.C.

Murphy, J.J., P.G. Allen, T.H. Stevens, and D. Weatherhead. 2005. A Meta-Analysis of Hypothetical Bias

in Stated Preference Valuation. Environmental and Resource Economics 30(3): 313–25.

Murphy, J.J. and T.H. Stevens. 2004. Contingent Valuation, Hypothetical Bias, and Experimental

Economics. Agricultural and Resource Economics Review 33(2): 182-192.

Murphy, J. J. T.H. Stevens and L. Yadev. 2010. A Comparison of Induced Value and Home-Grown

Value Experiments to Test for Hypothetical Bias in Contingent Valuation. Environmental and

Resource Economics 47(1): 111-123.

Powe, N.A. 2007. Redesigning Environmental Valuation: Mixing Methods within Stated Preference

Techniques. Cheltenham, UK: Edward Elgar.

Ready, R.C., P.A. Champ and J.L. Lawton. 2010. Using Respondent Uncertainty to Mitigate Hypothetical

Bias in a Stated Choice Experiment. Land Economics 86(2): 363-381.

Schläpfer, F., A. Roschewitz, and N. Hanley. 2004. Validation of Stated Preferences for Public Goods:

A Comparison of Contingent Valuation Survey Response and Voting Behavior. Ecological

Economics 51(1): 1-16.

Seip, K. and Strand, J. 1992. Willingness to Pay for Environmental Goods in Norway: A Contingent

Valuation Study with Real Payments. Environmental and Resource Economics 2: 91-106.
28

U.S. EPA. 2000. Guidelines for Preparing Economic Analyses. USEPA 240-R-00-003. U.S. EPA,

Office of the Administrator, Washington, DC, September 2000.

Vossler, C.A. and M.F. Evans. 2009. Bridging the Gap Between the Field and the Lab: Environmental

Goods, Policy Maker Input and Consequentiality. Journal of Environmental Economics and

Management 58(3): 338-345.

Vossler, C.A., J. Kerkvliet. 2003. “A Criterion Validity Test of the Contingent Valuation Method:

Comparing Hypothetical and Actual Voting Behavior for a Public Referendum.” Journal of

Environmental Economics and Management 45(3): 631-649.