Response to “Comments by Electric Power Research

Institute (EPRI) on the USEPA’s Supporting Statement

for Information Collection Request for Willingness to

Pay Survey for §316(b) Existing Facilities Cooling

Water Intake Structures: Instrument, Pre-Test, and

Implementation” (Federal Register Vol. 75, No. 139:

42438-42440; July 21, 2010) and EPA Supporting

Documents

DOCKET ID No, EPA-HQ-OW-2010-0595

Responses to the EPRI Comment

Introduction

In the introduction, the commenter briefly summarizes information provided in the Supporting Statement

for Information Collection Request for Willingness to Pay Survey for Section 316(b) Existing Facilities

Cooling Water Intake Structures: Instrument, Pre-test, and Implementation, (henceforth, Supporting

Document), including survey and sample design. In addition, the commenter summarizes the main

issues discussed in the comment including:

 hypothetical bias;

 providing an accurate description of the resource impacts resulting from I&E;

 addressing non-response bias;

 statistical uncertainty;

 experimental design;

 weighting and extrapolation approach; and

 respondent comprehension

Response to Section 2, Hypothetical Bias

The first five pages of section two do not address the specific survey materials or information collection

approaches described in the Agency’s Information Collection Request (ICR). Instead, these pages

provide a summary of selected publications that address the general issue of potential hypothetical bias in

stated preference survey instruments. As noted in the comment, “the difference between stated intentions

and actual behavior constitutes hypothetical bias.” The Agency notes that this is a well-known concern

with stated preference surveys, accompanied with decades of research addressing its severity and extent,

as well as the effectiveness of approaches for its mitigation. Hence, the majority of this Commenter’s

points in this section are most appropriately viewed not as specific critiques of the proposed survey

instrument, but rather as a comment on stated preference methods in general.
As a precursor to more detailed comments, the Agency notes that it is fully aware of the research

addressing the potential for hypothetical bias in stated preference welfare estimation. The Agency

emphasizes that the majority of the citations used within this section were published at least ten years ago,

prior to a substantial body of research addressing potential mechanisms for hypothetical bias mitigation.

The Agency also believes that the Commenter, in summarizing this literature, has provided a somewhat

one-sided perspective on the relevant published research. For example, the comment provides a quote

from the NOAA Blue Ribbon Panel (Arrow et al. 1993, p. 4610) that “hypothetical markets tend to

overstate willingness-to-pay.” However, the comment suppresses comments within the same report

that support the use of well-designed stated preference methods, even considering the possibility of

hypothetical bias. These include the relatively conclusive statement, “[t]he Panel concludes that under

[conditions described in the report], CV studies convey useful information. We think it is fair to describe

such information as reliable by the standards that seem to be implicit in similar contexts, like market

analysis for new and innovative products...” (Arrow et al. 1993, p. 4610). The Panel also states that

“[hypothetical bias] may be offset or even over-corrected” through conservative design practices. In a

second example of the selective representation of the literature within this comment, the Commenter cites

Murphy et al. (2005) who find a median ratio of hypothetical to real WTP of approximately 1.35.

However, the comment fails to note one of the primary conclusions of Murphy et al. (2005, p. 323), that

“the overwhelming majority of observations have relatively low [hypothetical bias], possibly suggesting

that hypothetical bias may not be as significant a problem in stated preference analyses as is often

thought.” Hence, while the Agency agrees that hypothetical bias is a concern—and has taken this very

seriously in survey development and testing—it does not agree that the literature suggests that

“unrealistically high values” (p. 2-5) are unavoidable.

A more comprehensive review of the recent literature would also have revealed methodological advances

provide various means for the amelioration of hypothetical bias. For example, many researchers have had

success using cheap talk methodology, certainty adjustments, and other methods shown to have the

promise to largely eliminate or at a minimum reduce hypothetical biases (e.g., Champ et al. 1997, 2004;

Cummings and Taylor 1999; Ready et al. 2010). Recently published findings of Ready et al. (2010, p.

363), for example, show that contemporary certainly adjustments “successfully mitigated hypothetical

bias.” While the evidence regarding the ability of researchers to ameliorate hypothetical biases is mixed,

there is no evidence that such biases are unavoidable. To the contrary, as noted recently by Murphy et al.

(2010), there is a “small but growing literature that suggesting the absence of hypothetical bias in

aggregate in induced value settings...”

The Agency notes that many of research efforts that provide evidence for the presence of hypothetical

biases are carried out in simulated markets, and hence do not reflect true assessments of criterion validity

(Bateman et al. 2002). While simulated markets—experimental or otherwise—might generate unbiased

WTP estimates, they are often conducted in institutional settings unfamiliar to respondents. Such settings

may generate novel or unusual behavior, confusion, distrust, or other behaviors that may lead to

misstatements of true, underlying WTP. For such reasons Bateman et al. (2002) argue that comparisons

involving WTP estimates obtained through simulated or experimental markets do not qualify as true tests

of criterion validity (i.e., they do not provide unambiguous measures of true willingness to pay in actual

markets). The most common alternative—the use of individual or group donation mechanisms (e.g.,

Foster et al. 1997; Seip and Strand 1992)—suffers from confounding effects of free-riding and the

associated lack of incentive compatibility (Bateman et al. 2002; Shläpfer et al. 2004) As a result, as

stated by Murphy and Stevens (2004, p. 185), one cannot unambiguously claim that values estimated
using laboratory experiments are any more representative of true willingness to pay than values estimates

using hypothetical markets.

While critics of stated preference methods often claim insurmountable divergences between stated and

real market behavior, evidence from various comparisons of actual versus hypothetical behavior

contradicts the claim that hypothetical bias is unavoidable, particularly in the aggregate. For example, as

conceded in the comment to the Agency, numerous comparisons of hypothetical versus binding referenda

find close correspondence between choices in stated preference surveys and voting behavior in binding,

consequential referenda (Bateman et al.2002, p. 318; Schläpfer et al. 2004; Johnston 2006; Vossler et al.

2003). Hence, in these true tests of criterion validity, responses in hypothetical and real contexts comport

quite closely, suggesting that stated preference surveys can provide useful indications of that which

individuals “are willing to forgo to enjoy a particular benefit” (OMB Circular A-4).

After conceding that comparisons of hypothetical and binding referenda “do not, generally, reflect

hypothetical bias,” (p. 2-3) the Commenter goes on to argue that these findings are not applicable to

the Agency’s proposed survey efforts, largely due to the claimed nature of the good under

consideration. These arguments, however, are speculative and are not supported by systematic

findings in the published literature. Moreover, the evidence in this section appears to be chosen

selectively to support the Commenter’s claim. For example, regarding arguments that greater

“salience and familiarity” of a commodity are related to a closer correspondence between hypothetical and

actual choices, the comment argues that the commodity in question within the proposed Agency survey is

“complex and likely unfamiliar” (p. 2-5). However, the comment fails to mention that the survey has

been explicitly designed to maximize the consequentiality and salience of choice experiment questions,

through direct linkages to proposed Agency regulatory efforts. Hence, according to the criteria proposed

by the comment, there are numerous aspects of the proposed survey that should in fact decrease the

potential for hypothetical biases (e.g., salience).

The Agency, however, recognizes that hypothetical bias is an important consideration in stated preference

design, and takes this concern very seriously. Because of this, extensive survey testing in focus groups

and cognitive interviews (cf. Mitchell and Carson 1989; Desvousges and Smith 1988; Desvousges et al.

1984; Johnston et al. 1995; Powe 2007) was conducted to assess the extent to which such biases might be

prevalent. The focus groups and interviews took place in the following locations and on these dates:1

1) East Providence, RI, July 21, 2010

2) Waltham, MA, July 29, 2010

3) St. Louis, MO, August 4, 2010

4) East Providence, RI, August 12, 2010

5) Bethesda, MD, August 18, 2010

6) Charlotte, NC, September 8, 2010

1

In this response, we use the citation format (#X-Z to refer to page Z of the transcript for focus group X. Citations to

cognitive interviews use a similar format, (#X/Y-Z), where Y represents the interview number.
7) Philadelphia, PA, September 23, 2010

Evidence from this extensive qualitative testing provided little evidence of hypothetical bias. For

example, focus group and interview transcripts show that, when asked explicitly, respondents almost

universally indicated that their answers to choice questions in the survey instrument would be identical if

the same questions were encountered in a binding referendum. Moreover, the survey explicitly

incorporates elements to enable mitigation of any remaining hypothetical bias, such as a certainty followup questions (Ready et al. 2010).

In summary, while the Agency recognizes the possibility of hypothetical bias—and indeed takes this

possibility very seriously—it is unconvinced by the claim that “hypothetical bias toward unrealistically

high non-use values is likely for the proposed EPA study.” The Agency believes that this claim rests

on a selective representation of the literature, and does not account for approaches that are now

available to mitigate such biases in the published literature. As noted in the ICR, the Agency will

follow the published literature in designing mitigation strategies to eliminate or at a minimum reduce

the potential for hypothetical bias.

The Referendum Approach

Within the broader section on hypothetical bias, the Commenter includes a sub-section arguing that

the Agency’s proposed “choice experiment format may not be sufficiently similar to a CV

referendum” to realize the benefits of a referendum format. These benefits can include incentive

compatibility under conditions described by Carson and Groves (2007). The majority of this subsection (pages 2-6 to 2-8) is devoted to statements regarding differences between choice experiment

surveys and binding public referenda. The Agency emphasizes, however, that these arguments are

largely immaterial given the recent literature that explicitly addresses the issue of incentive

compatibility within choice experiments. That is, recent research provides explicit evidence

regarding the incentive properties of choice experiments, so that arguments of general “similarity” to

binding referenda—such as those provided in this comment—are no longer directly relevant. This

recent literature appears to have been overlooked by the Commenter when writing this section.

Incentive compatible stated preference surveys provide no incentive for non-truthful preference revelation

(Carson and Groves 2007). The literature is clear regarding the importance of incentive compatibility in

stated preference value elicitation and the role of both question format and scenario consequentiality in

ensuring this property (Carson et al. 2000; Carson and Groves 2007; Collins and Vossler 2009; Herriges

et al. 2010; Johnston 2006; Vossler and Evans 2009). It has been established that referendum-type stated

preference choices are incentive compatible given that certain conditions are met, including the condition

that responses are believed by respondents to be consequential, or potentially influencing public policy

decisions (Carson and Groves 2007; Herriges et al. 2010). More recent literature has explored whether

choice experiments, which typically involve multiple choice questions and more than two choice options

within each question, share the incentive compatibility properties of simple “one shot” discrete choice

questions (Collins and Vossler 2009). Results show that choice experiments maintain these desirable

incentive properties under a wide range of conditions and identify conditions under which this occurs

(Collins and Vossler 2009). For example, Collins and Vossler (2009) conclude the following in their

analysis of incentive compatibility in multiple option choice experiments: “there is a low frequency of

deviations from theoretical predictions for both two- and three-option choice experiment value elicitation
questions,” suggesting the desirable incentive properties of appropriately designed three-option choice

experiments.

The comment also goes on to speculate regarding the consequentiality of the proposed survey instrument.

These arguments are based largely on the characteristics of the policy and payment vehicle. This

speculation notwithstanding, the Agency emphasizes that consequentiality is related to whether “the agent

answering a preference survey question [views] their responses as potentially influencing the agency’s

actions” (Carson and Groves 2007, p. 183). That is, consequentiality is related to the perceptions of

survey respondents. The Agency devoted significant attention to this issue in focus groups and cognitive

interviews, and found strong evidence that respondents viewed the survey instrument as consequential.

Participants took the survey questions seriously and indicated that they thought that their choices would

be used to influence policy. For example, when specifically asked, respondents indicated clearly that they

believed that the government would consider the results of the survey when choosing a course of action

(see, for example, focus group transcripts #3-28, #5-16 and #6-13). Moreover, there were no responses,

in any of the focus groups, suggesting that even a single respondent viewed the survey as nonconsequential. In the face of this strong evidence to the contrary, the Agency is unconvinced by the

Commenter’s claims that the survey should not be viewed as consequential.

The Certainty Question

This portion of the comment summarizes the literature on certainty adjustments. It makes the correct

observation that economic theory does not identify an appropriate adjustment; hence the choice of the cutoff point for certainty adjustments (i.e., when ‘yes’ responses are recoded as ‘no’ responses) is

empirically determined. The Agency agrees with the comment (p. 2-9) that “there is no universal

certainty adjustment that the EPA can apply.” Despite this, the Agency believes that there is sufficient

evidence from the published literature to proceed with a reliable protocol for certainty adjustments. First,

following standard practice in Agency benefit cost analysis (US EPA 2000), the Agency will conduct

sensitivity analysis to determine the impact of different reasonable certainty adjustments on estimated

WTP. Second, and more importantly, the combined literature provides a relatively high degree of

consistency regarding findings of certainty adjustments necessary to equilibrate hypothetical and actual

WTP. These typically fall between 7 and 8 on a ten-point scale (e.g., Champ et al. 2004; Poe et al. 2002;

Ready et al. 2010). Given the Agency’s 5-point scale, this would correspond most closely to a certainty

value of 4 (Agree), below which responses would be recoded. In summary, in the absence of a “universal

certainty adjustment,” the Agency will draw on the combined evidence from the empirical literature,

much of which is cited in the comment.

Suggested Changes in the Questionnaire and Study Design

The Agency recognizes that the validity of stated preference results depends on appropriate survey and

methodological design. Hence, when designing the proposed stated preference methods, the Agency has

relied upon the accepted good practice guidelines in the published literature, for example as found in

Bateman et al. (2002), Louviere et al. (2000), the older work of Mitchell and Carson (1989) and other

sources cited previously in this response. To the best knowledge of the Agency, the Commenter’s

suggestions for changes in study design do not comport with this standard guidance, and appear to be

largely ad hoc. Moreover, as noted above, the Agency has subjected the survey to extensive testing in

focus groups and cognitive interviews, with particular emphasis on the issues raised in this portion of the
comment. Responses in these focus groups and testing fail to support the types of substantial problems or

biases claimed within the comment.

Response to Section 3, Providing an Accurate Description of the Resource Impacts

Resulting from I&E

Section 3 of the EPRI comment focuses on EPA's description of resource impacts resulting from

IM&EM. Specific comments highlight several shortcomings in the draft survey instrument. In many

cases, EPA identified similar or identical shortcomings in the survey questionnaire as a consequence of

focus group and cognitive interview testing, and previously altered the survey in ways that addressed

many of the specific comments.

The first two pages of section three do not directly address the survey materials described in the Agency’s

Information Collection Request (ICR). Instead, these pages provide a case study of a single generating

facility on the Connecticut River (Connecticut Yankee, hereafter CY) which opened in 1968 and ceased

electrical production in 1996. As noted in the first two pages of the comment, a great deal of long-term

ecological research occurred in the vicinity of CY, some of which occurred following plant shut-down in

1996. Although EPA recognizes that case-studies may play a valuable role in the identification of broader

patterns, the ability to extrapolate from such reports depends on the degree to which facilities highlighted

in case-studies are representative of in-scope facilities. The CY facility is not representative of in-scope

316(b) facilities, either in the Northeast region, or in the country as a whole. For example, as the

commenter notes, the lower Connecticut River is ecologically unique. Additionally, the lower

Connecticut River valley remains largely undeveloped, has benefitted from a program to restore

diadromous fish to the watershed (Gephard and McMenemy 2004), and is protected through numerous

conservation designations including its status as an American Heritage River, a Nature Conservancy "Last

Great Place", and its inclusion within the Silvio O. Conte National Fish and Wildlife Refuge (Jacobsen et

al., 2004). Moreover, the water quality of the lower Connecticut River has greatly improved over the past

30 years as a consequence of measures undertaken under the Clean Water Act (Mullaney 2004). Thus,

although the CY case-study is a good example of in-depth ecological monitoring, the generality of the

points raised in this section are questionable at the national scale.

EPA believes the CY case study referenced in the comment is most appropriately viewed as a best-case

scenario for the long-term ecological consequences of a facility's operation in a relatively stable and wellprotected ecosystem, and not as a relevant critique of the proposed survey instrument. For example, the

commenter quotes Jacobson et al (2004) who state that, "there is no evidence of any long-term impact of

the operation of CY on the ecology of the lower Connecticut River". Although EPA recognizes that the

long-term resilience of aquatic systems following long-term disturbance may be substantial, much of the

resilience observed near CY is likely to be a consequence of the special setting of the facility, improved

water quality, and 7 years of recovery following the facility's closure. EPA emphasizes that the ability of

aquatic systems to recover from disturbance does not indicate that immediate ecological impacts of

IM&EM are minor, particularly in areas with degraded ecosystems and with multiple, cumulative

impacts. Additionally, because the survey instrument is designed to estimate the total value (i.e. use and

non-use) held by respondents on the ongoing effects of IM&EM on fish and fish populations, EPA

believes that the well-documented ability of aquatic ecosystems to recover from long-term disturbance is

not relevant to the present survey.
Within the first page of the comment, EPRI asserts that direct effects of I&E on fish populations, fish

community structure and function have not been demonstrated, despite intensive study within the

Northeast Region. EPA disagrees with these claims. EPA has previously determined in its NPDES

Permitting Determinations for Brayton Point Station's Thermal Discharge and Cooling Water Intake that

losses from impingement and entrainment significantly contributed to the collapse of the indigenous

community of fish in Mount Hope Bay (EPA-New England 2002; Gibson 2002). The direct effects of

IM&EM on fish populations have also been documented in the Hudson River (NY State Department of

Environmental Conservation, 2003) and the Southern California Bight (Murdoch et al 1989).

Additionally, a recent publication of the Atlantic States Marine Fisheries Commission (ASFMC)

explicitly lists water withdrawal facilities as a threat to Atlantic Coast diadromous fish habitat (Greene et

al 2009). Finally, fish behavior – including feeding rates and times - is known to change as a consequence

of once-through cooling systems (Massengill 1973). Thus, documentation exists to show that IM&EM

has direct effects on fish populations and fish community structure.

In its comment, EPRI that IM&EM is unlikely to be central to the issue of resource sustainability. EPA

agrees that the overall sustainability of commercially fished species is driven not by IM&EM, but by

fishing pressure, pollution, climate change, and other large-scale ecological pressures. Despite the general

observation that IM&EM are minor when compared to these effects, IM&EM has been shown to have

measurable impacts on local fish populations and communities. As such, EPA believes that existing,

published scientific information is not inconsistent with its suggestion in the survey that populations of

some fish species will improve with reduced IM&EM losses. Importantly, increases in fisheries

sustainability and fish population values proposed in the survey instrument are small. As such, the

Agency does not believe it is overstating the potential for improvements to fish populations as a

consequence of reduced IM&EM.

Additionally, EPA emphasizes that changes in fish populations are not the only criteria for quantifying

relevant adverse ecological impacts that occur to fish and fish populations as a consequence of IM&EM.

Although resource sustainability is one possible nonuse value that may be held by survey respondents,

EPA notes that respondents may place a value on individual fish killed, human interaction with

ecosystems, and the potential for cumulative impacts on ecosystems independent of resource

sustainability.

In the second portion of Section 3, the commenter notes several technical issues about the information

provided in the proposed study questionnaire. These comments will be addressed individually:

Question 1a. EPRI disagrees with the statement that scientists know the precise number of fish killed by

IM&EM on an annual basis.

Response: EPA agrees that there is substantial uncertainty about the number of fish killed on an annual

basis by IM&EM. The phrase in question was previously altered as a response to survey focus groups to

better indicate the uncertainty of this estimate.

Question 1b: The estimated value of fish killed (1.1 billion) appears to be based on analysis from the

original Phase II rule, which is itself based on sampling from the 1970s and 1980s. Due to changes in fish

populations, this estimate may substantially overestimate current losses due to IM&EM.
Response: The estimated number of age-1 equivalent fish killed (1.1 billion) is based on an updated

analysis similar to that used for the Phase II rule. IM&EM sampling in this analysis includes values from

the 1970s and 1980s, 1990s and the early 2000s. EPA recognizes that there are many sources of

uncertainty in its estimates, including changes in fish populations (with increases and decreases in

species-specific stock estimates), and the language used in updated versions of the survey instrument

reflect this uncertainty. The nature of these uncertainties, however, does not inherently bias the point

estimate either up or down. Instead, model parameters are likely to be distributed above and below true

parameter values, leading to imprecision, rather than inaccuracy. Thus, EPA believes the estimate of 1.1

billion fish killed provides a good estimate of average annual IM&EM losses.

Question 1c. The EPA questionnaire does not explain what a "young fish" is. This may lead respondents

to believe that "young fish" are likely to become members of the adult stock, despite the fact that natural

fish mortality rates are high.

Response: The commenter has identified a weakness in the survey instrument. The term "young fish" is

now defined in the revised document. This definition explicitly takes into account the fact that not all eggs

and larvae survive to adulthood.

Question 1d. EPA underlined "1.1 billion young fish" which adds emphasis to this number and may bias

lay respondents understanding of the magnitude of these losses.

Response: The wording of this section has been substantially altered in revisions of the survey instrument,

and the estimated number of IM&EM losses is no longer highlighted.

Question 2a. Comparing the number of age-1 equivalent fish lost to the number of adult fish provides

little information of value. It would be more meaningful to inform respondents as to the likelihood that a

measurable change would be observed if I&E losses were eliminated or reduced.

Response: EPA agrees that comparing age-1 equivalents to adult fish is not a good way to convey

information about the potential population-level impacts of IM&EM. However, EPA disagrees that

reporting the likelihood that a measurable change would be observed if IM&EM were eliminated is a

suitable alternative. This is because the power of fisheries statistics to detect changes in fish populations

is low. Instead, EPA has modified the survey to compare age-1 equivalent losses due to IM&EM with

age-1 equivalent populations. Thus, survey respondents are able to directly compare the magnitude of

relative losses, while accounting for both natural and fisheries-related mortality.

Question 2b: EPA states the annual loss of young winter flounder is 7.5 million. This is likely based on

data from the 1970s and 1980s when flounder stocks were high.

Response: EPA has significantly altered the way in which this section is presented. Additionally, as noted

above, the estimated number of fish losses is based on an updated analysis that includes values from the

1970s and 1980s, 1990s and the early 2000s.

Question 3a: The commenter believes that the statement "many types of fish" implies a bias towards a

greater loss of young fish.

Response: Although EPA disagrees with the comment, this section of the survey instrument has been

significantly altered. As such, this question is no longer relevant.
Question 4: EPRI believes that there are several technical issues related to the information conveyed to

the respondent about the risk, and related uncertainties, to resources of value. Specifically, EPRI believes

relevant information has been omitted, including:

4a: The Atlantic States Marine Fisheries Commission (ASMFC) does not include cooling water

withdrawals as a major agent causing depressed fish populations in its management plans, but

instead focuses on overfishing and habitat alterations

4b: EPA does not present historical data from studies of I&E impacts on fisheries resources, none

of which have documented a decline in fish populations directly attributable to cooling water

withdrawals.

Response: EPA believes that all fish species impacted by IM&EM have value, and that a the risk to

resources of value (such as fish species regulated by the ASMFC) is not the only way that cooling water

use adversely impacts fish and fish populations. For example, survey respondents may inherently value all

fish, or value policies that limit anthropogenic impacts whether or not they result in measurable impacts

to commercially-important resources. Accordingly, EPA disagrees with EPRI's focus on commercial

fisheries resources.

4a: EPA recognizes that overfishing and habitat alterations are likely to play larger roles than

IM&EM on fish populations. This is highlighted on page 3 of the survey, where EPA states that

"Cooling water use is not the largest cause of fish loss in most areas (fishing causes greater

losses), but has affected some fish populations" (emphasis in the original). Additionally, the lack

of emphasis on IM&EM in fisheries management plans does not in any way imply that these

losses do not cause adverse environmental impacts that have value. Moreover, EPA notes that

ASMFC specifically highlights water withdrawal facilities as a threat to Atlantic Coast

diadromous fish habitat (Greene et al 2009).

4b. EPA disagrees that there have not been documented cases of fish populations declining as a

direct result of cooling water withdrawals (EPA-New England 2002; Gibson 2002; Murdoch et al

1989; NYSDEC 2003). Moreover, EPA never claims that the population-level impacts of

IM&EM are substantial. Instead, the survey instrument states that "Although scientists can

predict the number of fish saved each year, the effect on fish populations is uncertain." Thus,

EPA has been careful to ensure that survey respondents are not misinformed about the potential

population-level effects of IM&EM on fish stocks.

Response to Section 4, Non-Response Bias

In this section, the commenter makes an argument that unknown differences between non-respondents

and the people who completed the survey represent the most serious concern about non-response bias.

According to the commenter, the unobserved heterogeneity between respondents and non-respondents is

would arguably yield results that cannot be clearly interpreted.

The commenter further notes that the NOAA Panel defines 70 percent as a high response rate and that the

Office of Management and Budget (OMB) guidelines for conducting surveys (2006) established a

threshold response rate of 80 percent. Analytical assessments are necessary for surveys that have a
response rate of less than 80 percent. The commenter indicated that EPA anticipates a response rate of 40

percent for its proposed study and, as a result, EPA plans to conduct a non-response study.

The commenter further states that the description of the non-response study provided in the Supporting

Document is very brief and unclear and suggests to provide a more detailed discussion of the planned

non-response study. In response to this comment, EPA provided more detailed description of the planned

study in the revised Supporting Document.

The commenter argues that the initial telephone survey that EPA plans to use as the basis for its nonresponse analysis not provide sufficient information for conducting non-response analysis. Although he

initial telephone survey would provide demographic information, participation in recreation activities, and

some basic opinions on several policy issues for both the respondents and the non-respondents may be of

limited use in addressing non-response if their not the determinants of WTP.

The comment also appears to suggest that the Agency ask numerous questions regarding the “detailed

opinions of the respondents” regarding the commodities in question (e.g., fish) as a means to correct for

non-response bias. As a basis for this suggestion, the comment offers findings from Champ and Bishop

(2001) that such opinions—in the case of wind energy—are correlated with WTP. The Agency believes

that this comment is misguided for a number of different reasons; these are detailed below.

First, the Agency emphasizes that non-response questions are asked of individuals who would (or who

have already) refused to answer the primary survey. These individuals are also typically unwilling to

answer lengthy questions regarding their environmental opinions. As a result, non-response questions

must be very brief, to encourage responses even among those who refuse to answer the broader survey.

For example, within the non-response assessment of Cameron and DeShazo (2005), the analysis is based

responses to a single question. This requirement for brevity precludes detailed questioning regarding

environmental opinions (i.e., “information ... likely to explain household WTP for fish saved”), as

suggested by the Commenter on page 4-1.

The comment also appears to confuse published evidence regarding the systematic correlations between

various types of opinions and estimated WTP (e.g., as in Champ and Bishop 2001) with evidence

regarding the potential for non-response biases related to these opinions. There are many latent and nonlatent factors which may be correlated with WTP estimates. This does not imply that all of these factors

should or can be measured when assessing non-response. The Agency has proposed standard approaches

for non-response assessments and calibrations, for example following Cameron and DeShazo (2005),

Whitehead et al. (1993) and Yoo and Yang (2001) in proposing tests and corrections for non-response

based on a small number of attitudinal and behavioral questions, combined with demographic

characteristics. These currently reflect standard practice within the literature.

Response to Section 5, Statistical Uncertainty

The commenter begins by summarizing basic information on sample selection and compares the

estimated sample size provided in the Supporting document (i.e., 2,000 households) with the number of

survey respondents required for the EPA survey based on Cochran’s formula. Using the Cohran’s

approach the commenter arrived at a recommended sample size of 1984. The commenter noted a small

difference between this estimate and the estimated sample size provided in the Supporting Document.

EPA notes that the Cochrane’s formula was used for estimating the initial sample size (1984). Additional
16 observations were added to the Inland region to account for a minimum sample size (288 observations)

required by experimental design. The Agency regrets that it was not more clear in the Supporting

Statement.

Beginning at the bottom of page 5-1, the comment addresses a number of loosely related topics related to

the methods used to estimate the distribution of WTP from choice experiment data. Although presented

together, many of these comments address distinct issues. As a precursor to more detailed responses

below, the Agency emphasizes that methods for WTP estimation from ecological choice experiment

data—of exactly the type proposed by the Agency—are very well established in the published literature.

For example, the information in Haab and McConnell (2002) is well established and is implicit in the

methods proposed by the Agency. More broadly, when designing the proposed methods, the Agency

closely followed accepted contemporary methods in the published literature for the estimation of WTP

distributions under statistical uncertainty. Hence, when considering the arguments of the Commenter, the

Agency views them within the context of this literature.

With regards to sample sizes required for choice experiments and WTP calculation, the Agency follows

the guidance of Louviere et al. (2000, p. 261-265), which explicitly accounts for the nature of choice

experiment data. Nearly identical guidance is found in Bateman et al. (2002). As an aside, the Agency

notes that the sample sizes typically required to obtain large sample statistical properties given choice

experiment designs are often greater than those required to estimate population proportions at a desired

degree of accuracy and probability. So, sample sizes calculated using Cochran’s formula or found in

Dillman (2000) are considered only minimum samples (Louviere et al. 2000). In following guidance

from the established choice experiment literature, the sample sizes to be used by the Agency fully

account for the fact that “WTP is a ratio of two random variables” (p. 5-2) and that WTP “is calculated

as a complicated function of estimated choice models” (p. 5-2). The Agency refers the Commenter to

the references cited above for additional details. Given the clear basis of the proposed sample sizes in

the established choice experiment literature, the Agency is unconvinced by the claim that “additional

effort should be undertaken to determine the appropriate sample size and uncertainty calculations prior

to its proposed study.”

The stated preference literature also provides extensive guidance on methods for hypothesis tests that

account for the statistical properties of WTP estimates. These were consulted by the Agency when

developing the proposed methods. The Agency refers the commenter, for example, to Poe et al. (2005)

and Hensher and Greene (2003), who both provide explicit guidance related to the characterization of

aggregate WTP distributions from stated preference data and the use of these distributions for

hypothesis tests. The comment’s discussion of Neyman and Pearson (1933), while perhaps interesting

from an historical or philosophical perspective, has no direct bearing or consequence on methods

proposed by the Agency. The Agency has proposed standard approaches for stated preference

hypothesis testing as reflected in the contemporary literature (e.g., Poe et al. 2005), together with sample

sizes that are sufficient to enable the proposed tests, as well as WTP estimation with desired levels of

accuracy and probability. The Agency notes that the proposed methods incorporate the “Krinsky and

Robb” approach suggested by the comment; a fact that seems to have been overlooked within this

section. In summary, the Agency fully agrees that “is important for EPA to consider the level of

significance associated with an estimate of aggregate WTP,” (p. 5-3), and has proposed methods for

these calculations that correspond fully with recommended approaches from the relevant economic

literature.
Regarding the choice of sampling weights, the Agency appreciates the recognition that the proposed

approach is “the standard approach” (p. 5-4). The Agency notes that the alternative approach suggested

in the comment is speculative, and has no identified basis in the peer reviewed choice experiment

literature. Moreover, the ambiguously described “simulation [that] would include variations of an

experimental design, and result in simulated responses, model estimation and extrapolation” provides

insufficient guidance for development of a viable alternative to the standard approaches proposed by the

Agency. In the absence of concrete and established alternatives, the Agency has proposed a more

conservative approach of reliance on accepted and standard methods from the stated preference

literature.

Finally, the Agency notes that the discussion of the graded-pairs methodology is not directly relevant, as

this method has not been accepted as an established method for WTP estimation (US EPA 2000;

Freeman 2003). Regardless of the cognitive properties of this method, it’s inability to provide welldefined economic welfare estimates precludes it’s use for benefit estimation.

Response to Section 6, Experimental Design

With regard to experimental design, the Agency appreciates the Commenter’s familiarity with the

tradeoffs in developing any choice experiment design—for example between the efficiency in estimating

main effects and an ability to estimate desired interactions. While the comment discusses this issue solely

in terms of the ability to estimate certain effects of interest, the Agency also highlights the issue of

aliasing, in which the estimation of main effects can be biased by the omission of confounded interactions

within the experimental design (Louviere et al. 2000). The inclusion of two-way interactions within the

experimental design reduces the possibility for aliasing of this type. The Agency appreciates the

suggestion that a D-efficiency criterion be used for the experimental design, and notes that this is the

criterion that was in fact used, following Kuhfeld and Tobias (2005) and Lusk and Norwood (2005).

Hence, the Agency is already using the approach proposed by the Commenter. The Agency regrets that

this was not more clear in the ICR materials.

The comment goes on to discuss various tests of response consistency. These include internal and

external scope tests, such as those already proposed by the Agency. Additional tests that are discussed

include dominant and graded-pair tests. The Agency notes that there are dozens of tests that might be

conducted on stated preference survey responses to assess validity, many of these are already proposed by

the Agency and are incorporated within the survey and experimental design. The Agency also notes—as

also indicated in the comment itself—that there are tradeoffs involved in each of these tests. In the

present case, an ability to conduct dominant and graded-pair tests would require the introduction of

significant inefficiencies within the experimental design. These would reduce the information available

from a given total sample size. The Agency notes that the Commenter has already expressed concern

with sample sizes (see response to Section 5 above); for these and other reasons the Agency agrees that it

is important to obtain the most possible information from a given sample (i.e., to increase design

efficiency to the extent feasible in the current context). Given this, and given the substantial battery of

tests already proposed for the survey response data, the Agency is hesitant to introduce known correlation

and inefficiencies into the experimental design for the sole purpose of conducting additional and often

redundant tests.

While the comment notes that “non-market researchers in the SP arena currently take advantage of the

repetitive, introspective nature of SP data by employing experimental designs that support a battery of
statistical tests” (p. 6-2), the Agency emphasizes that the present survey already supports a far greater

battery of tests than most other surveys present in the published literature. Moreover, many of the

suggestions of the comment would involve substantial tradeoffs in data quality. For example,

respondents in the Agency’s focus groups clearly suggested that additions to the survey’s length should

be avoided. Given this, the Agency views the suggestion to have respondents “[answer] more than three

[choice] questions” (p. 6-4) as infeasible. In summary, the Agency believes that the additional insight

that might be gained through the additional tests suggested by this comment do not justify the

unavoidable reductions in data quality that would be required to enable these tests.

Response to Section 7, Weighting and Extrapolation Approach

The Agency notes that the discussion of the “weighting and extrapolation approach” is not directly

relevant, as this method is speculative and has not been accepted as an established method for WTP

estimation (US EPA 2000; Freeman 2003). The Agency believes—following guidance in the literature

and the Agency’s own guidance documents (Arrow et al. 1993; US EPA 2000)—that established stated

preference methods are capable of estimating reliable and accurate welfare measures, if surveys and

approaches are appropriately designed. The alternative approach suggested by the Commenter has

neither been accepted by the Agency as a valid approach for stated preference benefit estimation, nor is

it broadly accepted in the stated preference literature. The Agency can find little basis for the proposed

“illustrative SP nesting structure” in the established choice experiment literature. Hence, the Agency

does not consider it to be a valid alternative for benefit estimation in the present case. The Agency’s

proposed use of these methods for nonuse WTP estimation reflects the standards of the broader literature,

which considers the use of such methods appropriate.

Response to Section 8, Respondent Comprehension

The Agency agrees with the Commenter that a stated preference survey should “provide information to

respondents about the hypothetical commodity so that they understand and accept it and can give

meaningful answers to the valuation questions” (p. 8-1). Given the importance of commodity

comprehension, the Agency devoted considerable attention to this issue in the focus groups and cognitive

interviews listed above. The number of focus groups and interviews conducted for this survey—and for

the closely related survey tested in 2005-06—far exceeds the number conducted for most stated

preference surveys found in the published literature. Among the primary areas of emphasis in these focus

groups and pretests was comprehension of the hypothetical commodity and related issues (e.g.,

understanding of the payment vehicle). Hence, as suggested, the Agency has taken extra “care during its

... pretesting activities to ensure consistent respondent interpretation of the commodity and full

understanding of the information delivered within the instrument” (p. 8-1).

Payment Vehicle

As noted by the comment, the cost of hypothetical programs are “described as a cost per year” (p. 8-1).

This follows standard practice in many stated preference surveys in the published literature, and is an

accepted format for the payment vehicle. There are hundreds if not thousands of stated preference

surveys reflected in the literature that use payment vehicles reflecting recurring, ongoing payments of

exactly the type proposed by the Agency. In nearly all of these cases, the ongoing nature of these

payments is implied clearly by the nature of the policy context, as it is in the present survey. Given this—
and the results of focus groups—the Agency is not convinced that respondents require still additional

information regarding the details of the payment vehicle.

The current format of the payment vehicle is the result of focus groups on the original version of the

316(b) survey instrument in 2005, with additional testing in the more recent 2010 focus groups. These

focus groups and discussions with both OMB and NCEE in 2005 ultimately led to the current description

of the payment vehicle. Focus groups showed no difficulty understanding the format of the payment

vehicle, and respondents clearly understood the cost as an ongoing cost. The Agency also notes that a

similar, ongoing cost of living payment vehicle has been used by prior EPA-sponsored research, with no

apparent difficulties (Magat et al. 2000).

Participants in 2010 focus groups clearly understood that selecting Options A or B would result in

increased costs to their household. They generally understood that costs to industry would be passed on to

consumers, with one participant stating, “So my thought process is that the household stuff is being

passed down to consumers from the cost of the added regulation they have” (#3-4). The draft survey was

also revised to improve clarity regarding how costs would be passed to consumers by adding a clause on

Page 1 stating that goods and services with costs would include “electricity and common household

products”. The definition of “cost per year” on Page 7 was also revised to include “services” purchased by

consumers. These revisions greatly reduced the potential for respondent confusion when evaluating policy

options. Respondents also appeared to understand the ongoing, monthly nature of the payments. Despite

the Commenter’s suggestions to the contrary, there is no evidence from focus groups and interviews that

respondents had differing perceptions regarding the length of time over which the payments would last.

Similarly, there is no suggestion from focus groups that a “one shot” payment would be more easily

understood—and in fact a one-shot payment would be less realistic given the nature of the policies under

consideration. It is the goal of the Agency to evaluate WTP based on characterization of the policy

changes that is as realistic as possible given the constraints of the stated preference survey format.

Description of the payment vehicle as a “one-time payment (which would likely be characterized as a tax

increase to be credible)” (p. 8-2) has no justification in terms of the actual mechanisms through which

costs would be passed on to consumers, and would require considerable deception on the part of the

survey instrument. The Agency can find no justification in the published literature or elsewhere for

changing the payment vehicle in this manner—particularly given the successful performance of the

proposed payment vehicle in focus groups.

The comment also concludes that “it is appropriate for the EPA to explicitly discuss not only the inherent

uncertainty of projected ecological effects but also the accompanying uncertainty in the costs” (p. 8-2).

While the basis for this “conclusion” remains unstated, the Agency notes that cost uncertainty is already

noted in the survey instrument. For example, bullet points on survey page 10 state: “You will be shown

different questions, with different combinations of technology and different costs,” and “Depending on

the policies chosen, costs to your household could range from $0 per month to a maximum of $6 per

month.” Page 12 also states “Policy costs and effects depend on many factors.” In sum, the survey

already incorporates clear and repeated text that communicates the potential variability in costs.

Ecological Changes

The Agency appreciates the comment’s acknowledgements of the challenges involved in presenting

realistic policy changes to respondents, particularly when those changes involve complex ecological
systems. As noted by the Commenter, the Agency is relying upon the most recent research in this area

to support its efforts. As a result of the extensive focus group and cognitive interview testing of the

original survey instrument reviewed by the Commenter, the survey has already undergone substantial

changes in the way it communicates ecological information. Hence, many of the comments in this

section are obsolete, as the elements in question are no longer components of the revised survey

instrument. The Agency also emphasizes that the crucial element when testing the communication of

ecological information in stated preference surveys is the reaction and understanding of potential “real

world” respondents. Hence, when determining whether particular survey attributes are appropriate, the

Agency gives substantial weight to focus group and pretest responses.

Aside from assuring accurate respondent understanding of the policy changes in question, it is very

important to the Agency—following Johnston et al. (2010)—that the changes reflected in the survey are

ecologically accurate and (to the extent possible) measurable. Following Johnston et al. (2010) and

standard approaches in the ecological literature, all attributes are presented relative to a clear reference

condition, reflecting the best possible outcome in a given ecosystem or the level of an indicator that

would be expected in an undisturbed system. The comment expresses concern at the “eye catching” (p.

8-3) nature of the changes in the number of fish saved, and also speculates that the non-zero starting

points for other attributes “may inadvertently send signals that these other attributes are acceptable at

current levels, because they are not zero.”

With regard to these concerns, the Agency first highlights a fact acknowledged by the Commenter—

“this treatment of the fish saved [is] technically correct” (p. 8-3). The Agency, however, is also fully

aware of the differences in the ecological ranges of attribute effects which occur in the realistic policy

contexts shown by the survey, and so has conducted repeated assessments of respondents’ reactions in

focus groups. In contrast to the speculation of the Commenter, focus groups showed clear variation in

the weight given to fish saved compared to other ecological attributes. Overall, focus group and

interview participants’ statements implied different opinions about the importance of preventing fish

losses versus increasing fish population or improving the condition of aquatic ecosystems. Participants’

choices and statements implied different opinions about whether changes in fish mortality were of

importance without large increases in the long-term fish population or the condition of aquatic

ecosystems. Some participants stated that they choose a policy option because of the difference in fish

saved, “I went with B because I saw that was a higher percentage fish saved” (#1-11). Comments from

some of these participants indicated that they preferred to avoid losses because they place some values

on fish in their own right, and clearly recognize the differences in ecological effects: “[...] looking

down, some of them are not as much of a difference. But I’m all for trying to save as many as we can”

(#4/6-2). Some other participants indicated that effects on populations or ecosystem condition were

most important, and that they felt uncomfortable spending money to prevent fish mortality if there was

uncertainty in whether it would improve these effects: “I have to admit, I voted generally for the higher

plan until we got to question 6, where the condition of the ecosystem didn’t change” (#5-11).

In addition, while the percentage change in fish saved is indeed larger than changes for other attributes,

focus group participants generally understood that in most cases the quantity of fish saved would be

small relative to the total fish population. For example:

Moderator: When you look at that fish saved per year, how did you think about that in terms of

all the fish out there in the water?
Respondent: I guess probably a pretty small part [#6-7].

Participants generally felt that the survey provided useful information about fish losses attributable to

CWIS, other sources of mortality, and population sizes. Participants generally understood based on the

introductory material that CWIS is not the largest source of fish mortality. One participant stated he

initially thought that CWIS losses are a very large problem but the statement on Page 3 that “Cooling

water use is not the large cause of fish loss in most areas” “kind of tempers it a little bit” (#7-12).

Participants were very aware, however, that the magnitude of fish losses is most appropriately assessed

relative to broader baselines: “All this information, all this research, but I mean, how many salmons to

Americans eat a year, you know? I mean, I don’t know in comparison to 17,000 paddlefish?” (#3-21).

The draft survey was revised to include pie charts comparing young fish losses from CWIS to estimated

adult fish populations for example species. This presentation was deemed important in order to address

participant comments on intermediate draft survey versions which did not specifically include examples

to this effects.

In summary, the Agency devoted considerable attention to this issue when designing and testing the

survey instrument. The suggestion that the presentation of this attribute is “misleading” does not appear

to be supported by focus group results. The survey provides extensive information to help respondents

understand both the cardinal and relative meaning of all ecological attributes. While recognizing the

Commenter’s concerns in this area, the Agency notes that focus group responses, in general, do not

appear to support these concerns. For example, among all focus group and interview responses, there

was not a single instance in which a respondent thought “that because no fish are saved, they will be

become extinct, in accordance with the interpretation of zero for the fish population score” (p. 8-3).

Neither were there suggestions from focus groups that the attribute levels implied that “saving fish

must be critically important”; in fact many respondents noted explicitly that the broader ecological

effects of saving fish seemed to be small. While there are always challenges associated with the

communication of complex ecological information in stated preference surveys, the Agency has given

this issue substantial attention in survey development and testing. Focus group and cognitive

interview results do not suggest that the presentation of the “Fish Saved” attribute is misleading in the

ways suggested by this comment.

The Agency agrees with the Commenter that the the Commercial Fish Sustainability Score was

potentially misleading. These difficulties were highlighted by focus groups. As a result, this attribute has

been removed from the survey.

The Agency has also significantly revised the description of the Aquatic Ecological Condition Score,

based on the responses of focus groups and cognitive interviews. Many of the “trigger words” mentioned

in the comment have been eliminated (e.g., the survey no longer mentions fish tissue contamination).

Moreover, some of the more speculative suggestions of this comment (e.g., inclusion of a water quality

discussion may raise concerns for some respondents about the safety of their drinking water) have no

basis in focus group or interview results. The Agency placed particular emphasis on this attribute in focus

group and interview testing to assess exactly the type of concerns raised by the Commenter in this section.

Results suggest that these concerns are largely unfounded. Respondents’ comments suggested a generally

appropriate understanding of the Aquatic Ecological Condition Score and why it might change as a result

of the proposed policies. For example, when discussing why changes in “Fish Saved” were different than

changes in “Aquatic Ecological Condition,” one respondent stated “Fish are only part of it, this

ecosystem. You got plants, you got ocean ... all sorts of life so that's why they – it's a different ...
percentage is because fish are only a part of it”(#3-11). This comment was typical of respondents’

understanding of this score.

Finally, focus group responses suggested that respondents were acutely aware that many stressors and

changes affect ecosystems, that that changes in regulations on CWIS are only one of many factors that

can influence ecosystems. There did not appear to be any confusion over the fact that scores of 100 for

various attributes are generally unattainable through reductions in CWIS fish losses alone.

Quiz Questions

The proposed survey already includes a number of questions to assess patterns and rationales underlying

responses. Moreover, as noted above, respondents in the Agency’s focus groups clearly suggested that

additions to the survey’s length should be avoided. In addition, the issues implied by the suggested

questions have already been extensively studied in focus groups.

For example, focus group responses clearly indicate that respondents understand “that implementing the

policy does nothing to address the largest source of fish losses (harvesting by humans).” To the contrary,

focus groups included numerous discussions of the detailed mechanisms through which the proposed

regulations would affect fish. Participants generally understood that CWIS primary affects “young fish”

which may eventually develop into adult fish. Many participants stated that the graphic on Page 2 of the

survey (“How Fish are Affected by Water Intake”) helped them understand how CWIS impacts fish and

how it differentially affects eggs, juvenile, and adult fish: “I was thinking more fish is going to get

through the screen if they’re smaller. The smaller, the more eggs and the more fish you’re going to lose”

(#6-8). The draft survey was also revised to further clarify the graphic with arrows linking it to supporting

text. As said by a participant, “I like the arrows. Large fish are easier to kill. The arrow points to a larger

adult fish. Juvenile fish and eggs it points to juvenile fish and eggs” (#7-10). Many participants showed an

understanding of the meaning of “young fish” and the relationship between young fish and adult fish. One

stated, “[...] still young eggs and fish are important too [...], but an egg is an egg which grows to an

adult.” (#3-21), and another described juvenile fish as “the kids” (#6-8). These and other detailed

discussions showed no evidence of the type of broad-scale confusion implied by the suggested quiz

question.

Moreover, as noted above, there is no evidence whatsoever from focus groups or interviews that

“respondents think that 0% of fished saved in the status quo policy means that fish species will become

extinct.” Similarly, the Agency conducted extensive focus group testing of the payment vehicle, and

found absolutely no evidence that respondents believed that they could avoid cost increases because they

“do not buy that many of the affected products.” Participants clearly understood that selecting Options A

or B would result in increased costs to their household. They generally understood that costs to industry

would be passed on to consumers, with one participant stating, “So my thought process is that the

household stuff is being passed down to consumers from the cost of the added regulation they have” (#3-

4).

In summary, the Agency recognizes that there are dozens of possible “quiz questions” which one might

suggest for a survey such as this. The fact that one can suggest a quiz question does not mean that its

inclusion is justified—particularly in the face of focus group responses which suggest that such areas of

confusion are, in general, not a major concern.
References

Arrow, K. , R. Solow, E. Leamer, P. Portney, R. Rander, and H. Schuman. 1993. “Report of the NOAA

Panel on Contingent Valuation.” Federal Register 58(10): 4602-4614.

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S.

Mourato, E. Ozdemiroglu, D.W. Pearce, R. Sugden, and J. Swanson. 2002. Economic Valuation

with Stated Preference Surveys: A Manual. Northampton, MA: Edward Elgar.

Cameron, T.A. and J.R. DeShazo. 2005. Comprehensive selectivity assessment for a major consumer

panel: attitudes toward government regulation of environment, health and safety risks.

Department of Economics, University of Oregon.

Carson, R. T., T. Groves, and M. J. Machina. 2000. “Incentive and Informational Properties of

Preference Questions.” Working Paper, Department of Economics, University of California, San

Diego.

Carson, R.T. and T. Groves. 2007. Incentive and Informational Properties of Preference Questions.

Environmental and Resource Economics 37(1):181–210.

Champ, P.A., and R.C. Bishop. 2001. “Donation payment mechanisms and contingent valuation: An

empirical study of hypothetical bias.” Environmental and Resource Economics 19:383–402.

Champ, P.A., R.C. Bishop, T.C. Brown, and D.W. McCollum. 1997. Using donation mechanisms to

value non-use benefits from public goods. Journal of Environmental Economics and Management

33(2):151–162.

Champ, P.A., R. Moore, R.C. Bishop. 2004. “Hypothetical Bias: The Mitigating Effects of Certainty

Questions and Cheap Talk.” Selected Paper prepared for presentation at the American

Agricultural Economics Association Annual Meeting, Denver, Colorado.

Collins, J.P. and C.A. Vossler. 2009. Incentive Compatibility Tests of Choice Experiment Value

Elicitation Questions. Journal of Environmental Economics and Management 58(2): 226-235.

Cummings, R. and L. Taylor. 1999. Unbiased Value Estimates for Environmental Goods: A Cheap Talk

Design for the Contingent Valuation Method. American Economic Review 89:649-665.

Desvousges, W.H. and V.K Smith. 1988. Focus Groups and Risk Communication: the Science of

Listening to Data. Risk Analysis 8:479-484.

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in

Designing a contingent Valuation Survey to Measure the Benefits of Hazardous Waste

Management Regulations. Research Triangle Institute: Research Triangle Park, NC.

Dillman, D.A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley

and Sons.

EPA–New England (2002) Clean Water Act NPDES Permitting Determinations for Thermal Discharge

and Cooling Water Intake from Brayton Point Station in Somerset, MA (NPDES Permit No. MA

0003654), July 22, 2002.
Foster, V., I. J. Bateman, and D. Harley. 1997. Real and Hypothetical Willingness to Pay for

Environmental Preservation: A Non-experimental Comparison. Journal of Agricultural

Economics 48(2):123-138.

Freeman III, A.M. 2003. The Measurement of Environmental and Resource Values: Theory and

Methods. Washington, DC: Resources for the Future.

Gephard, S. and McMenemy, J. (2004). An overview of the program to restore Atlantic Salmon and other

diadromous fishes to the Connecticut River with notes on the current status of these species in the

river. In: Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill,

eds. The Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower

Connecticut River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland,

pp. 525-538

Gibson, M. (2002) Winter Flounder Abundance Near Brayton Point Station, Mt. Hope Bay Revisited:

Separating Local from Regional Impacts using Long Term Abundance Data. RI Division of Fish

and Wildlife Research Reference Document 02/1

Greene, K. E., J. L. Zimmerman, R. W. Laney, and J. C. Thomas-Blate. (2009). Atlantic coast

diadromous fish habitat: A review of utilization, threats, recommendations for conservation, and

research needs. Atlantic States Marine Fisheries Commission Habitat Management Series No. 9,

Washington, D.C.

Haab, T.C., and K.E. McConnell. 2002. Valuing Environmental and Natural Resources: The

Econometrics of Non-Market Valuation. Cheltenham, UK: Edward Elgar.

Hensher, D.A. and W.H. Greene. 2003. The Mixed Logit Model: The State of Practice. Transportation

30(2): 133–176.

Herriges, J., C. Kling, C. Liu and J. Tobias. 2010. What are the Consequences of Consequentiality?

Journal of Environmental Economics and Management 59(1): 67-81.

Jacobson, P.M., Leggett, W.C., Marcy, B.C. Jr., Massengill, R.R. (2004) Summary and Conclusion. In:

Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill, eds. The

Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower Connecticut

River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland, pp. 525-538

Johnston, R.J. 2006. Is Hypothetical Bias Universal? Validating Contingent Valuation Responses Using a

Binding Public Referendum. Journal of Environmental Economics and Management 52(1): 469-

481.

Johnston, R.J., E.T. Schultz, K. Segerson and E.Y. Besedin. 2010. Bioindicator-Based Stated Preference

Valuation for Aquatic Habitat and Ecosystem Service Restoration, in Bennett, J. ed. International

Handbook on Non-Marketed Environmental Valuation. Cheltenham, UK: Edward Elgar,

forthcoming.

Johnston, R.J., T.F. Weaver, L.A. Smith and S.K. Swallow. 1995. Contingent Valuation Focus Groups:

Insights From Ethnographic Interview Techniques. Agricultural and Resource Economics

Review 24(1): 56-69.
Krinsky, I. and A.L. Robb. 1986. On Approximating the Statistical Properties of Elasticities. Review of

Economics and Statistics 68(4): 715-719.

Kuhfeld, W.F. and Tobias, R.D., 2005. Large factorial designs for product engineering and marketing

research applications. Technometrics 47: 132-141.

Louviere, J.J., D.A. Hensher, and J.D. Swait. 2000. Stated Preference Methods: Analysis and

Application. Cambridge, UK: Cambridge University Press.

Lusk, J.L. and F.B. Norwood. 2005. Effect of Experimental Design on Choice-Based Conjoint Valuation

Estimates. American Journal of Agricultural Economics 87(3):771-785.

Magat, W.A., J. Huber, K.W. Viscusi and J. Bell. 2000. An Iterative Choice Approach to Valuing Clean

Lakes, Rivers, and Streams. Journal of Risk and Uncertainty 21(1): 7-43.

Massengill R (1973) Change in feeding and body condition of brown bullheads overwintering in the

heated effluent of a power plant. Chesapeake Science 14: 138-141

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation

Method. Resources for the Future, Washington, D.C.

Mullaney, J.R. (2004) Summary of water quality trends in the Connecticut River 1968-1998. In:

Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill, eds. The

Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower Connecticut

River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland, pp. 525-538

Murdoch, W.W., R.C. Fay, and B.J. Mechalas (1989). Final Report of the Marine Review Committee to

the California Coastal Commission. August 1989, MRC Document No. 89–02.

Murphy, J.J., P.G. Allen, T.H. Stevens, and D. Weatherhead. 2005. A Meta-Analysis of Hypothetical Bias

in Stated Preference Valuation. Environmental and Resource Economics 30(3): 313–25.

Murphy, J.J. and T.H. Stevens. 2004. Contingent Valuation, Hypothetical Bias, and Experimental

Economics. Agricultural and Resource Economics Review 33(2): 182-192.

Murphy, J. J. T.H. Stevens and L. Yadev. 2010. A Comparison of Induced Value and Home-Grown

Value Experiments to Test for Hypothetical Bias in Contingent Valuation. Environmental and

Resource Economics 47(1): 111-123.

New York State Department of Environmental Conservation (NYSDEC). (2003). Final Environmental

Impact Statement: Concerning the Applications to Renew NYSPDES Permits for the Roseton 1

& 2, Bowline 1 & 2 and Indian Point 2 & 3 Steam Electric Generating Stations, Orange,

Rockland and Westchester Counties.

Neyman, J., and E.S. Pearson. 1933. On the problem of the most efficient tests of statistical hypotheses.

Phil. Trans. Roy. Soc. London, Ser. A 231:289–337.

Poe, G.L., J.E. Clark, D. Rondeau, and W.D. Schulze. 2002. Provision point mechanisms and field

validity tests of contingent valuation. Environmental and Resource Economics 23: 105–131.
Poe, G.L., K.L. Giraud and J.B. Loomis. 2005. Computational Methods for Measuring the Difference in

Empirical Distributions. American Journal of Agricultural Economics 87(2): 353-365.

Powe, N.A. 2007. Redesigning Environmental Valuation: Mixing Methods within Stated Preference

Techniques. Cheltenham, UK: Edward Elgar.

Ready, R.C., P.A. Champ and J.L. Lawton. 2010. Using Respondent Uncertainty to Mitigate Hypothetical

Bias in a Stated Choice Experiment. Land Economics 86(2): 363-381.

Schläpfer, F., A. Roschewitz, and N. Hanley. 2004. Validation of Stated Preferences for Public Goods:

A Comparison of Contingent Valuation Survey Response and Voting Behavior. Ecological

Economics 51(1): 1-16.

Seip, K. and Strand, J. 1992. Willingness to Pay for Environmental Goods in Norway: A Contingent

Valuation Study with Real Payments. Environmental and Resource Economics 2: 91-106.

U.S. EPA. 2000. Guidelines for Preparing Economic Analyses. USEPA 240-R-00-003. U.S. EPA,

Office of the Administrator, Washington, DC, September 2000.

Vossler, C.A. and M.F. Evans. 2009. Bridging the Gap Between the Field and the Lab: Environmental

Goods, Policy Maker Input and Consequentiality. Journal of Environmental Economics and

Management 58(3): 338-345.

Vossler, C.A., J. Kerkvliet. 2003. “A Criterion Validity Test of the Contingent Valuation Method:

Comparing Hypothetical and Actual Voting Behavior for a Public Referendum.” Journal of

Environmental Economics and Management 45(3): 631-649.

Whitehead, J. C., P. A. Groothuis and G. C. Blomquist. 1993. Testing for Non-Response and

Sample Selection Bias in Contingent Valuation: Analysis of a Combination Phone/Mail Survey.

Economics Letters 41: 215–220.

Yoo, S.H. and H.J. Yang. 2001. Application of Sample Selection Model to Double-Bounded

Dichotomous Choice Contingent Valuation Studies. Environmental and Resource Economics 20:

147–163.