# Response to “Comments by Electric Power Research

Institute (EPRI) on the USEPA’s Supporting Statement for Information Collection Request for Willingness to Pay Survey for §316(b) Existing Facilities Cooling Water Intake Structures: Instrument, Pre-Test, and Implementation” (Federal Register Vol. 75, No. 139: 42438-42440; July 21, 2010) and EPA Supporting
Documents

DOCKET ID No, EPA-HQ-OW-2010-0595

Responses to the EPRI Comment

Introduction

In the introduction, the commenter briefly summarizes information provided in the Supporting Statement for Information Collection Request for Willingness to Pay Survey for Section 316(b) Existing Facilities Cooling Water Intake Structures: Instrument, Pre-test, and Implementation, (henceforth, Supporting Document), including survey and sample design. In addition, the commenter summarizes the main issues discussed in the comment including:

 hypothetical bias;

 providing an accurate description of the resource impacts resulting from I&E;

 addressing non-response bias;

 statistical uncertainty;

 experimental design;

 weighting and extrapolation approach; and

 respondent comprehension

Response to Section 2, Hypothetical Bias

The first five pages of section two do not address the specific survey materials or information collection approaches described in the Agency’s Information Collection Request (ICR). Instead, these pages provide a summary of selected publications that address the general issue of potential hypothetical bias in stated preference survey instruments. As noted in the comment, “ the difference between stated intentions and actual behavior constitutes hypothetical bias.” The Agency notes that this is a well-known concern with stated preference surveys, accompanied with decades of research addressing its severity and extent, as well as the effectiveness of approaches for its mitigation. Hence, the majority of this Commenter’s points in this section are most appropriately viewed not as specific critiques of the proposed survey instrument, but rather as a comment on stated preference methods in general.As a precursor to more detailed comments, the Agency notes that it is fully aware of the research

addressing the potential for hypothetical bias in stated preference welfare estimation. The Agency emphasizes that the majority of the citations used within this section were published at least ten years ago, prior to a substantial body of research addressing potential mechanisms for hypothetical bias mitigation. The Agency also believes that the Commenter, in summarizing this literature, has provided a somewhat one-sided perspective on the relevant published research. For example, the comment provides a quote
from the NOAA Blue Ribbon Panel (Arrow et al. 1993, p. 4610) that “hypothetical markets tend to overstate willingness-to-pay.” However, the comment suppresses comments within the same report that support the use of well-designed stated preference methods, even considering the possibility of hypothetical bias. These include the relatively conclusive statement, “[t]he Panel concludes that under [conditions described in the report], CV studies convey useful information. We think it is fair to describe such information as reliable by the standards that seem to be implicit in similar contexts, like market analysis for new and innovative products...” (Arrow et al. 1993, p. 4610). The Panel also states that “[hypothetical bias] may be offset or even over-corrected” through conservative design practices. In a second example of the selective representation of the literature within this comment, the Commenter cites Murphy et al. (2005) who find a median ratio of hypothetical to real WTP of approximately 1.35. However, the comment fails to note one of the primary conclusions of Murphy et al. (2005, p. 323), that “the overwhelming majority of observations have relatively low [hypothetical bias], possibly suggesting that hypothetical bias may not be as significant a problem in stated preference analyses as is often thought.” Hence, while the Agency agrees that hypothetical bias is a concern—and has taken this very seriously in survey development and testing—it does not agree that the literature suggests that “unrealistically high values” (p. 2-5) are unavoidable.

A more comprehensive review of the recent literature would also have revealed methodological advances provide various means for the amelioration of hypothetical bias. For example, many researchers have had success using cheap talk methodology, certainty adjustments, and other methods shown to have the promise to largely eliminate or at a minimum reduce hypothetical biases (e.g., Champ et al. 1997, 2004; Cummings and Taylor 1999; Ready et al. 2010). Recently published findings of Ready et al. (2010, p. 363), for example, show that contemporary certainly adjustments “successfully mitigated hypothetical bias.” While the evidence regarding the ability of researchers to ameliorate hypothetical biases is mixed, there is no evidence that such biases are unavoidable. To the contrary, as noted recently by Murphy et al. (2010), there is a “small but growing literature that suggesting the absence of hypothetical bias in aggregate in induced value settings...”

The Agency notes that many of research efforts that provide evidence for the presence of hypothetical biases are carried out in simulated markets, and hence do not reflect true assessments of criterion validity (Bateman et al. 2002). While simulated markets—experimental or otherwise—might generate unbiased WTP estimates, they are often conducted in institutional settings unfamiliar to respondents. Such settings may generate novel or unusual behavior, confusion, distrust, or other behaviors that may lead to misstatements of true, underlying WTP. For such reasons Bateman et al. (2002) argue that comparisons involving WTP estimates obtained through simulated or experimental markets do not qualify as true tests of criterion validity (i.e., they do not provide unambiguous measures of true willingness to pay in actual markets). The most common alternative—the use of individual or group donation mechanisms (e.g., Foster et al. 1997; Seip and Strand 1992)—suffers from confounding effects of free-riding and the associated lack of incentive compatibility (Bateman et al. 2002; Shläpfer et al. 2004) As a result, as stated by Murphy and Stevens (2004, p. 185), one cannot unambiguously claim that values estimatedusing laboratory experiments are any more representative of true willingness to pay than values estimates

using hypothetical markets.

While critics of stated preference methods often claim insurmountable divergences between stated and real market behavior, evidence from various comparisons of actual versus hypothetical behavior contradicts the claim that hypothetical bias is unavoidable, particularly in the aggregate. For example, as conceded in the comment to the Agency, numerous comparisons of hypothetical versus binding referenda find close correspondence between choices in stated preference surveys and voting behavior in binding, consequential referenda (Bateman et al.2002, p. 318; Schläpfer et al. 2004; Johnston 2006; Vossler et al. 2003). Hence, in these true tests of criterion validity, responses in hypothetical and real contexts comport quite closely, suggesting that stated preference surveys can provide useful indications of that which individuals “are willing to forgo to enjoy a particular benefit” (OMB Circular A-4).

After conceding that comparisons of hypothetical and binding referenda “ do not, generally, reflect hypothetical bias,” (p. 2-3) the Commenter goes on to argue that these findings are not applicable to the Agency’s proposed survey efforts, largely due to the claimed nature of the good under consideration. These arguments, however, are speculative and are not supported by systematic findings in the published literature. Moreover, the evidence in this section appears to be chosen selectively to support the Commenter’s claim. For example, regarding arguments that greater “salience and familiarity” of a commodity are related to a closer correspondence between hypothetical and actual choices, the comment argues that the commodity in question within the proposed Agency survey is “complex and likely unfamiliar” (p. 2-5). However, the comment fails to mention that the survey has been explicitly designed to maximize the consequentiality and salience of choice experiment questions, through direct linkages to proposed Agency regulatory efforts. Hence, according to the criteria proposed by the comment, there are numerous aspects of the proposed survey that should in fact decrease the potential for hypothetical biases (e.g., salience).

The Agency, however, recognizes that hypothetical bias is an important consideration in stated preference design, and takes this concern very seriously. Because of this, extensive survey testing in focus groups and cognitive interviews (cf. Mitchell and Carson 1989; Desvousges and Smith 1988; Desvousges et al. 1984; Johnston et al. 1995; Powe 2007) was conducted to assess the extent to which such biases might be prevalent. The focus groups and interviews took place in the following locations and on these dates: 1

1) East Providence, RI, July 21, 2010

2) Waltham, MA, July 29, 2010

3) St. Louis, MO, August 4, 2010

4) East Providence, RI, August 12, 2010

5) Bethesda, MD, August 18, 2010

6) Charlotte, NC, September 8, 2010

1 In this response, we use the citation format (#X-Z to refer to page Z of the transcript for focus group X. Citations to cognitive interviews use a similar format, (#X/Y-Z), where Y represents the interview number.7) Philadelphia, PA, September 23, 2010

Evidence from this extensive qualitative testing provided little evidence of hypothetical bias. For example, focus group and interview transcripts show that, when asked explicitly, respondents almost universally indicated that their answers to choice questions in the survey instrument would be identical if the same questions were encountered in a binding referendum. Moreover, the survey explicitlyincorporates elements to enable mitigation of any remaining hypothetical bias, such as a certainty followup questions (Ready et al. 2010).

In summary, while the Agency recognizes the possibility of hypothetical bias—and indeed takes this possibility very seriously—it is unconvinced by the claim that “ hypothetical bias toward unrealistically high non-use values is likely for the proposed EPA study.” The Agency believes that this claim rests on a selective representation of the literature, and does not account for approaches that are now available to mitigate such biases in the published literature. As noted in the ICR, the Agency will follow the published literature in designing mitigation strategies to eliminate or at a minimum reduce the potential for hypothetical bias.

The Referendum Approach

Within the broader section on hypothetical bias, the Commenter includes a sub-section arguing that the Agency’s proposed “choice experiment format may not be sufficiently similar to a CV
referendum” to realize the benefits of a referendum format. These benefits can include incentivecompatibility under conditions described by Carson and Groves (2007). The majority of this subsection (pages 2-6 to 2-8) is devoted to statements regarding differences between choice experiment surveys and binding public referenda. The Agency emphasizes, however, that these arguments are largely immaterial given the recent literature that explicitly addresses the issue of incentive compatibility within choice experiments. That is, recent research provides explicit evidence regarding the incentive properties of choice experiments, so that arguments of general “similarity” to binding referenda—such as those provided in this comment—are no longer directly relevant. This recent literature appears to have been overlooked by the Commenter when writing this section.

Incentive compatible stated preference surveys provide no incentive for non-truthful preference revelation (Carson and Groves 2007). The literature is clear regarding the importance of incentive compatibility in stated preference value elicitation and the role of both question format and scenario consequentiality in ensuring this property (Carson et al. 2000; Carson and Groves 2007; Collins and Vossler 2009; Herriges et al. 2010; Johnston 2006; Vossler and Evans 2009). It has been established that referendum-type stated preference choices are incentive compatible given that certain conditions are met, including the condition that responses are believed by respondents to be consequential, or potentially influencing public policy decisions (Carson and Groves 2007; Herriges et al. 2010). More recent literature has explored whether choice experiments, which typically involve multiple choice questions and more than two choice options within each question, share the incentive compatibility properties of simple “one shot” discrete choice questions (Collins and Vossler 2009). Results show that choice experiments maintain these desirable incentive properties under a wide range of conditions and identify conditions under which this occurs (Collins and Vossler 2009). For example, Collins and Vossler (2009) conclude the following in their analysis of incentive compatibility in multiple option choice experiments: “there is a low frequency of deviations from theoretical predictions for both two- and three-option choice experiment value elicitationquestions,” suggesting the desirable incentive properties of appropriately designed three-option choice

experiments.

The comment also goes on to speculate regarding the consequentiality of the proposed survey instrument. These arguments are based largely on the characteristics of the policy and payment vehicle. This speculation notwithstanding, the Agency emphasizes that consequentiality is related to whether “the agent answering a preference survey question [views] their responses as potentially influencing the agency’s actions” (Carson and Groves 2007, p. 183). That is, consequentiality is related to the perceptions of survey respondents. The Agency devoted significant attention to this issue in focus groups and cognitive interviews, and found strong evidence that respondents viewed the survey instrument as consequential. Participants took the survey questions seriously and indicated that they thought that their choices would be used to influence policy. For example, when specifically asked, respondents indicated clearly that they believed that the government would consider the results of the survey when choosing a course of action (see, for example, focus group transcripts #3-28, #5-16 and #6-13). Moreover, there were no responses ,in any of the focus groups, suggesting that even a single respondent viewed the survey as nonconsequential. In the face of this strong evidence to the contrary, the Agency is unconvinced by the Commenter’s claims that the survey should not be viewed as consequential.

The Certainty Question

This portion of the comment summarizes the literature on certainty adjustments. It makes the correctobservation that economic theory does not identify an appropriate adjustment; hence the choice of the cutoff point for certainty adjustments (i.e., when ‘yes’ responses are recoded as ‘no’ responses) is empirically determined. The Agency agrees with the comment (p. 2-9) that “there is no universal certainty adjustment that the EPA can apply.” Despite this, the Agency believes that there is sufficient evidence from the published literature to proceed with a reliable protocol for certainty adjustments. First, following standard practice in Agency benefit cost analysis (US EPA 2000), the Agency will conduct sensitivity analysis to determine the impact of different reasonable certainty adjustments on estimated WTP. Second, and more importantly, the combined literature provides a relatively high degree of consistency regarding findings of certainty adjustments necessary to equilibrate hypothetical and actual WTP. These typically fall between 7 and 8 on a ten-point scale (e.g., Champ et al. 2004; Poe et al. 2002; Ready et al. 2010). Given the Agency’s 5-point scale, this would correspond most closely to a certainty value of 4 (Agree), below which responses would be recoded. In summary, in the absence of a “universal certainty adjustment,” the Agency will draw on the combined evidence from the empirical literature,
much of which is cited in the comment.

Suggested Changes in the Questionnaire and Study Design

The Agency recognizes that the validity of stated preference results depends on appropriate survey and methodological design. Hence, when designing the proposed stated preference methods, the Agency has relied upon the accepted good practice guidelines in the published literature, for example as found in Bateman et al. (2002), Louviere et al. (2000), the older work of Mitchell and Carson (1989) and other sources cited previously in this response. To the best knowledge of the Agency, the Commenter’s suggestions for changes in study design do not comport with this standard guidance, and appear to be largely ad hoc . Moreover, as noted above, the Agency has subjected the survey to extensive testing in focus groups and cognitive interviews, with particular emphasis on the issues raised in this portion of thecomment. Responses in these focus groups and testing fail to support the types of substantial problems or

biases claimed within the comment.

Response to Section 3, Providing an Accurate Description of the Resource Impacts Resulting from I&E

Section 3 of the EPRI comment focuses on EPA's description of resource impacts resulting from IM&EM. Specific comments highlight several shortcomings in the draft survey instrument. In many cases, EPA identified similar or identical shortcomings in the survey questionnaire as a consequence of focus group and cognitive interview testing, and previously altered the survey in ways that addressed many of the specific comments.

The first two pages of section three do not directly address the survey materials described in the Agency’s Information Collection Request (ICR). Instead, these pages provide a case study of a single generating facility on the Connecticut River (Connecticut Yankee, hereafter CY) which opened in 1968 and ceased electrical production in 1996. As noted in the first two pages of the comment, a great deal of long-term ecological research occurred in the vicinity of CY, some of which occurred following plant shut-down in 1996. Although EPA recognizes that case-studies may play a valuable role in the identification of broader patterns, the ability to extrapolate from such reports depends on the degree to which facilities highlighted in case-studies are representative of in-scope facilities. The CY facility is not representative of in-scope 316(b) facilities, either in the Northeast region, or in the country as a whole. For example, as the commenter notes, the lower Connecticut River is ecologically unique. Additionally, the lower Connecticut River valley remains largely undeveloped, has benefitted from a program to restore diadromous fish to the watershed (Gephard and McMenemy 2004), and is protected through numerous conservation designations including its status as an American Heritage River, a Nature Conservancy "Last Great Place", and its inclusion within the Silvio O. Conte National Fish and Wildlife Refuge (Jacobsen et al., 2004). Moreover, the water quality of the lower Connecticut River has greatly improved over the past 30 years as a consequence of measures undertaken under the Clean Water Act (Mullaney 2004). Thus, although the CY case-study is a good example of in-depth ecological monitoring, the generality of the points raised in this section are questionable at the national scale.

EPA believes the CY case study referenced in the comment is most appropriately viewed as a best-casescenario for the long-term ecological consequences of a facility's operation in a relatively stable and wellprotected ecosystem, and not as a relevant critique of the proposed survey instrument. For example, the commenter quotes Jacobson et al (2004) who state that, "there is no evidence of any long-term impact of the operation of CY on the ecology of the lower Connecticut River". Although EPA recognizes that the long-term resilience of aquatic systems following long-term disturbance may be substantial, much of the resilience observed near CY is likely to be a consequence of the special setting of the facility, improved water quality, and 7 years of recovery following the facility's closure. EPA emphasizes that the ability of aquatic systems to recover from disturbance does not indicate that immediate ecological impacts of IM&EM are minor, particularly in areas with degraded ecosystems and with multiple, cumulative impacts. Additionally, because the survey instrument is designed to estimate the total value (i.e. use and non-use) held by respondents on the ongoing effects of IM&EM on fish and fish populations, EPA believes that the well-documented ability of aquatic ecosystems to recover from long-term disturbance is not relevant to the present survey.Within the first page of the comment, EPRI asserts that direct effects of I&E on fish populations, fish

community structure and function have not been demonstrated, despite intensive study within the Northeast Region. EPA disagrees with these claims. EPA has previously determined in its NPDES Permitting Determinations for Brayton Point Station's Thermal Discharge and Cooling Water Intake that losses from impingement and entrainment significantly contributed to the collapse of the indigenous community of fish in Mount Hope Bay (EPA-New England 2002; Gibson 2002). The direct effects of IM&EM on fish populations have also been documented in the Hudson River (NY State Department of Environmental Conservation, 2003) and the Southern California Bight (Murdoch et al 1989). Additionally, a recent publication of the Atlantic States Marine Fisheries Commission (ASFMC) explicitly lists water withdrawal facilities as a threat to Atlantic Coast diadromous fish habitat (Greene et al 2009). Finally, fish behavior – including feeding rates and times - is known to change as a consequence of once-through cooling systems (Massengill 1973). Thus, documentation exists to show that IM&EM has direct effects on fish populations and fish community structure.

In its comment, EPRI that IM&EM is unlikely to be central to the issue of resource sustainability. EPA agrees that the overall sustainability of commercially fished species is driven not by IM&EM, but by fishing pressure, pollution, climate change, and other large-scale ecological pressures. Despite the general observation that IM&EM are minor when compared to these effects, IM&EM has been shown to have measurable impacts on local fish populations and communities. As such, EPA believes that existing, published scientific information is not inconsistent with its suggestion in the survey that populations of some fish species will improve with reduced IM&EM losses. Importantly, increases in fisheries sustainability and fish population values proposed in the survey instrument are small. As such, the Agency does not believe it is overstating the potential for improvements to fish populations as a consequence of reduced IM&EM.

Additionally, EPA emphasizes that changes in fish populations are not the only criteria for quantifying relevant adverse ecological impacts that occur to fish and fish populations as a consequence of IM&EM. Although resource sustainability is one possible nonuse value that may be held by survey respondents, EPA notes that respondents may place a value on individual fish killed, human interaction with ecosystems, and the potential for cumulative impacts on ecosystems independent of resource sustainability.

In the second portion of Section 3, the commenter notes several technical issues about the information provided in the proposed study questionnaire. These comments will be addressed individually:

Question 1a. EPRI disagrees with the statement that scientists know the precise number of fish killed by
IM&EM on an annual basis.

Response: EPA agrees that there is substantial uncertainty about the number of fish killed on an annual basis by IM&EM. The phrase in question was previously altered as a response to survey focus groups to better indicate the uncertainty of this estimate.

Question 1b: The estimated value of fish killed (1.1 billion) appears to be based on analysis from the original Phase II rule, which is itself based on sampling from the 1970s and 1980s. Due to changes in fish populations, this estimate may substantially overestimate current losses due to IM&EM.Response: The estimated number of age-1 equivalent fish killed (1.1 billion) is based on an updated

analysis similar to that used for the Phase II rule. IM&EM sampling in this analysis includes values from the 1970s and 1980s, 1990s and the early 2000s. EPA recognizes that there are many sources of uncertainty in its estimates, including changes in fish populations (with increases and decreases in species-specific stock estimates), and the language used in updated versions of the survey instrument reflect this uncertainty. The nature of these uncertainties, however, does not inherently bias the point estimate either up or down. Instead, model parameters are likely to be distributed above and below true parameter values, leading to imprecision, rather than inaccuracy. Thus, EPA believes the estimate of 1.1 billion fish killed provides a good estimate of average annual IM&EM losses.

Question 1c. The EPA questionnaire does not explain what a "young fish" is. This may lead respondents to believe that "young fish" are likely to become members of the adult stock, despite the fact that natural fish mortality rates are high.

Response: The commenter has identified a weakness in the survey instrument. The term "young fish" is now defined in the revised document. This definition explicitly takes into account the fact that not all eggs
and larvae survive to adulthood.

Question 1d. EPA underlined "1.1 billion young fish" which adds emphasis to this number and may bias lay respondents understanding of the magnitude of these losses.

Response: The wording of this section has been substantially altered in revisions of the survey instrument, and the estimated number of IM&EM losses is no longer highlighted.

Question 2a. Comparing the number of age-1 equivalent fish lost to the number of adult fish provides little information of value. It would be more meaningful to inform respondents as to the likelihood that a measurable change would be observed if I&E losses were eliminated or reduced.

Response: EPA agrees that comparing age-1 equivalents to adult fish is not a good way to convey information about the potential population-level impacts of IM&EM. However, EPA disagrees that reporting the likelihood that a measurable change would be observed if IM&EM were eliminated is a suitable alternative. This is because the power of fisheries statistics to detect changes in fish populations is low. Instead, EPA has modified the survey to compare age-1 equivalent losses due to IM&EM with age-1 equivalent populations. Thus, survey respondents are able to directly compare the magnitude of relative losses, while accounting for both natural and fisheries-related mortality.

Question 2b: EPA states the annual loss of young winter flounder is 7.5 million. This is likely based on data from the 1970s and 1980s when flounder stocks were high.

Response: EPA has significantly altered the way in which this section is presented. Additionally, as noted above, the estimated number of fish losses is based on an updated analysis that includes values from the 1970s and 1980s, 1990s and the early 2000s.

Question 3a: The commenter believes that the statement "many types of fish" implies a bias towards a greater loss of young fish.

Response: Although EPA disagrees with the comment, this section of the survey instrument has been significantly altered. As such, this question is no longer relevant.Question 4: EPRI believes that there are several technical issues related to the information conveyed to

the respondent about the risk, and related uncertainties, to resources of value. Specifically, EPRI believes relevant information has been omitted, including:

4a: The Atlantic States Marine Fisheries Commission (ASMFC) does not include cooling water withdrawals as a major agent causing depressed fish populations in its management plans, but instead focuses on overfishing and habitat alterations

4b: EPA does not present historical data from studies of I&E impacts on fisheries resources, none of which have documented a decline in fish populations directly attributable to cooling water
withdrawals.

Response: EPA believes that all fish species impacted by IM&EM have value, and that a the risk to resources of value (such as fish species regulated by the ASMFC) is not the only way that cooling water use adversely impacts fish and fish populations. For example, survey respondents may inherently value all fish, or value policies that limit anthropogenic impacts whether or not they result in measurable impacts to commercially-important resources. Accordingly, EPA disagrees with EPRI's focus on commercial
fisheries resources.

4a: EPA recognizes that overfishing and habitat alterations are likely to play larger roles than IM&EM on fish populations. This is highlighted on page 3 of the survey, where EPA states that "Cooling water use is not the largest cause of fish loss in most areas (fishing causes greater losses), but has affected some fish populations" (emphasis in the original). Additionally, the lack of emphasis on IM&EM in fisheries management plans does not in any way imply that these losses do not cause adverse environmental impacts that have value. Moreover, EPA notes that ASMFC specifically highlights water withdrawal facilities as a threat to Atlantic Coast diadromous fish habitat (Greene et al 2009).

4b. EPA disagrees that there have not been documented cases of fish populations declining as a direct result of cooling water withdrawals (EPA-New England 2002; Gibson 2002; Murdoch et al 1989; NYSDEC 2003). Moreover, EPA never claims that the population-level impacts of IM&EM are substantial. Instead, the survey instrument states that "Although scientists can predict the number of fish saved each year, the effect on fish populations is uncertain." Thus, EPA has been careful to ensure that survey respondents are not misinformed about the potential population-level effects of IM&EM on fish stocks.

Response to Section 4, Non-Response Bias

In this section, the commenter makes an argument that unknown differences between non-respondents and the people who completed the survey represent the most serious concern about non-response bias. According to the commenter, the unobserved heterogeneity between respondents and non-respondents is would arguably yield results that cannot be clearly interpreted.

The commenter further notes that the NOAA Panel defines 70 percent as a high response rate and that the Office of Management and Budget (OMB) guidelines for conducting surveys (2006) established a threshold response rate of 80 percent. Analytical assessments are necessary for surveys that have aresponse rate of less than 80 percent. The commenter indicated that EPA anticipates a response rate of 40

percent for its proposed study and, as a result, EPA plans to conduct a non-response study.

The commenter further states that the description of the non-response study provided in the Supporting Document is very brief and unclear and suggests to provide a more detailed discussion of the planned non-response study. In response to this comment, EPA provided more detailed description of the planned study in the revised Supporting Document.

The commenter argues that the initial telephone survey that EPA plans to use as the basis for its nonresponse analysis not provide sufficient information for conducting non-response analysis. Although he initial telephone survey would provide demographic information, participation in recreation activities, and some basic opinions on several policy issues for both the respondents and the non-respondents may be of limited use in addressing non-response if their not the determinants of WTP.

The comment also appears to suggest that the Agency ask numerous questions regarding the “detailed opinions of the respondents” regarding the commodities in question (e.g., fish) as a means to correct for non-response bias. As a basis for this suggestion, the comment offers findings from Champ and Bishop (2001) that such opinions—in the case of wind energy—are correlated with WTP. The Agency believes that this comment is misguided for a number of different reasons; these are detailed below.

First, the Agency emphasizes that non-response questions are asked of individuals who would (or who have already) refused to answer the primary survey. These individuals are also typically unwilling to answer lengthy questions regarding their environmental opinions. As a result, non-response questions must be very brief, to encourage responses even among those who refuse to answer the broader survey. For example, within the non-response assessment of Cameron and DeShazo (2005), the analysis is based responses to a single question. This requirement for brevity precludes detailed questioning regarding environmental opinions (i.e., “information ... likely to explain household WTP for fish saved”), as suggested by the Commenter on page 4-1.

The comment also appears to confuse published evidence regarding the systematic correlations between various types of opinions and estimated WTP (e.g., as in Champ and Bishop 2001) with evidenceregarding the potential for non-response biases related to these opinions. There are many latent and nonlatent factors which may be correlated with WTP estimates. This does not imply that all of these factors should or can be measured when assessing non-response. The Agency has proposed standard approaches for non-response assessments and calibrations, for example following Cameron and DeShazo (2005), Whitehead et al. (1993) and Yoo and Yang (2001) in proposing tests and corrections for non-response based on a small number of attitudinal and behavioral questions, combined with demographic characteristics. These currently reflect standard practice within the literature.

Response to Section 5, Statistical Uncertainty

The commenter begins by summarizing basic information on sample selection and compares the estimated sample size provided in the Supporting document (i.e., 2,000 households) with the number of survey respondents required for the EPA survey based on Cochran’s formula. Using the Cohran’s approach the commenter arrived at a recommended sample size of 1984. The commenter noted a small difference between this estimate and the estimated sample size provided in the Supporting Document. EPA notes that the Cochrane’s formula was used for estimating the initial sample size (1984). Additional16 observations were added to the Inland region to account for a minimum sample size (288 observations)

required by experimental design. The Agency regrets that it was not more clear in the Supporting

Statement.

Beginning at the bottom of page 5-1, the comment addresses a number of loosely related topics related to the methods used to estimate the distribution of WTP from choice experiment data. Although presented together, many of these comments address distinct issues. As a precursor to more detailed responses below, the Agency emphasizes that methods for WTP estimation from ecological choice experiment data—of exactly the type proposed by the Agency—are very well established in the published literature. For example, the information in Haab and McConnell (2002) is well established and is implicit in the methods proposed by the Agency. More broadly, when designing the proposed methods, the Agency closely followed accepted contemporary methods in the published literature for the estimation of WTP distributions under statistical uncertainty. Hence, when considering the arguments of the Commenter, the Agency views them within the context of this literature.

With regards to sample sizes required for choice experiments and WTP calculation, the Agency follows the guidance of Louviere et al. (2000, p. 261-265), which explicitly accounts for the nature of choice experiment data. Nearly identical guidance is found in Bateman et al. (2002). As an aside, the Agency notes that the sample sizes typically required to obtain large sample statistical properties given choice experiment designs are often greater than those required to estimate population proportions at a desired degree of accuracy and probability. So, sample sizes calculated using Cochran’s formula or found in Dillman (2000) are considered only minimum samples (Louviere et al. 2000). In following guidance from the established choice experiment literature, the sample sizes to be used by the Agency fully account for the fact that “WTP is a ratio of two random variables” (p. 5-2) and that WTP “is calculated as a complicated function of estimated choice models” (p. 5-2). The Agency refers the Commenter to the references cited above for additional details. Given the clear basis of the proposed sample sizes in the established choice experiment literature, the Agency is unconvinced by the claim that “additional effort should be undertaken to determine the appropriate sample size and uncertainty calculations prior to its proposed study.”

The stated preference literature also provides extensive guidance on methods for hypothesis tests that account for the statistical properties of WTP estimates. These were consulted by the Agency when developing the proposed methods. The Agency refers the commenter, for example, to Poe et al. (2005) and Hensher and Greene (2003), who both provide explicit guidance related to the characterization of aggregate WTP distributions from stated preference data and the use of these distributions for hypothesis tests. The comment’s discussion of Neyman and Pearson (1933), while perhaps interesting from an historical or philosophical perspective, has no direct bearing or consequence on methods proposed by the Agency. The Agency has proposed standard approaches for stated preference hypothesis testing as reflected in the contemporary literature (e.g., Poe et al. 2005), together with sample sizes that are sufficient to enable the proposed tests, as well as WTP estimation with desired levels of accuracy and probability. The Agency notes that the proposed methods incorporate the “Krinsky and Robb” approach suggested by the comment; a fact that seems to have been overlooked within this section. In summary, the Agency fully agrees that “is important for EPA to consider the level of significance associated with an estimate of aggregate WTP,” (p. 5-3), and has proposed methods for these calculations that correspond fully with recommended approaches from the relevant economic
literature.Regarding the choice of sampling weights, the Agency appreciates the recognition that the proposed

approach is “the standard approach” (p. 5-4). The Agency notes that the alternative approach suggested in the comment is speculative, and has no identified basis in the peer reviewed choice experiment literature. Moreover, the ambiguously described “simulation [that] would include variations of an experimental design, and result in simulated responses, model estimation and extrapolation” provides insufficient guidance for development of a viable alternative to the standard approaches proposed by the Agency. In the absence of concrete and established alternatives, the Agency has proposed a more conservative approach of reliance on accepted and standard methods from the stated preference
literature.

Finally, the Agency notes that the discussion of the graded-pairs methodology is not directly relevant, as this method has not been accepted as an established method for WTP estimation (US EPA 2000;Freeman 2003). Regardless of the cognitive properties of this method, it’s inability to provide welldefined economic welfare estimates precludes it’s use for benefit estimation.

Response to Section 6, Experimental Design

With regard to experimental design, the Agency appreciates the Commenter’s familiarity with the tradeoffs in developing any choice experiment design—for example between the efficiency in estimating main effects and an ability to estimate desired interactions. While the comment discusses this issue solely in terms of the ability to estimate certain effects of interest, the Agency also highlights the issue of aliasing, in which the estimation of main effects can be biased by the omission of confounded interactions within the experimental design (Louviere et al. 2000). The inclusion of two-way interactions within the experimental design reduces the possibility for aliasing of this type. The Agency appreciates the suggestion that a D-efficiency criterion be used for the experimental design, and notes that this is the criterion that was in fact used, following Kuhfeld and Tobias (2005) and Lusk and Norwood (2005). Hence, the Agency is already using the approach proposed by the Commenter. The Agency regrets that
this was not more clear in the ICR materials.

The comment goes on to discuss various tests of response consistency. These include internal and external scope tests, such as those already proposed by the Agency. Additional tests that are discussed include dominant and graded-pair tests. The Agency notes that there are dozens of tests that might be conducted on stated preference survey responses to assess validity, many of these are already proposed by the Agency and are incorporated within the survey and experimental design. The Agency also notes—as
also indicated in the comment itself—that there are tradeoffs involved in each of these tests. In the present case, an ability to conduct dominant and graded-pair tests would require the introduction of significant inefficiencies within the experimental design. These would reduce the information available from a given total sample size. The Agency notes that the Commenter has already expressed concern with sample sizes (see response to Section 5 above); for these and other reasons the Agency agrees that it is important to obtain the most possible information from a given sample (i.e., to increase design efficiency to the extent feasible in the current context). Given this, and given the substantial battery of tests already proposed for the survey response data, the Agency is hesitant to introduce known correlation and inefficiencies into the experimental design for the sole purpose of conducting additional and often
redundant tests.

While the comment notes that “non-market researchers in the SP arena currently take advantage of the repetitive, introspective nature of SP data by employing experimental designs that support a battery ofstatistical tests” (p. 6-2), the Agency emphasizes that the present survey already supports a far greater

battery of tests than most other surveys present in the published literature. Moreover, many of the suggestions of the comment would involve substantial tradeoffs in data quality. For example, respondents in the Agency’s focus groups clearly suggested that additions to the survey’s length should be avoided. Given this, the Agency views the suggestion to have respondents “[answer] more than three [choice] questions” (p. 6-4) as infeasible. In summary, the Agency believes that the additional insight that might be gained through the additional tests suggested by this comment do not justify the unavoidable reductions in data quality that would be required to enable these tests.

Response to Section 7, Weighting and Extrapolation Approach

The Agency notes that the discussion of the “weighting and extrapolation approach” is not directly relevant, as this method is speculative and has not been accepted as an established method for WTP estimation (US EPA 2000; Freeman 2003). The Agency believes—following guidance in the literature and the Agency’s own guidance documents (Arrow et al. 1993; US EPA 2000)—that established stated preference methods are capable of estimating reliable and accurate welfare measures, if surveys and approaches are appropriately designed. The alternative approach suggested by the Commenter has neither been accepted by the Agency as a valid approach for stated preference benefit estimation, nor is it broadly accepted in the stated preference literature. The Agency can find little basis for the proposed “illustrative SP nesting structure” in the established choice experiment literature. Hence, the Agency does not consider it to be a valid alternative for benefit estimation in the present case. The Agency’s proposed use of these methods for nonuse WTP estimation reflects the standards of the broader literature, which considers the use of such methods appropriate.

Response to Section 8, Respondent Comprehension

The Agency agrees with the Commenter that a stated preference survey should “provide information to respondents about the hypothetical commodity so that they understand and accept it and can give meaningful answers to the valuation questions” (p. 8-1). Given the importance of commodity comprehension, the Agency devoted considerable attention to this issue in the focus groups and cognitive interviews listed above. The number of focus groups and interviews conducted for this survey—and for the closely related survey tested in 2005-06—far exceeds the number conducted for most stated preference surveys found in the published literature. Among the primary areas of emphasis in these focus groups and pretests was comprehension of the hypothetical commodity and related issues (e.g., understanding of the payment vehicle). Hence, as suggested, the Agency has taken extra “care during its ... pretesting activities to ensure consistent respondent interpretation of the commodity and full understanding of the information delivered within the instrument” (p. 8-1).

Payment Vehicle

As noted by the comment, the cost of hypothetical programs are “described as a cost per year” (p. 8-1). This follows standard practice in many stated preference surveys in the published literature, and is an accepted format for the payment vehicle. There are hundreds if not thousands of stated preference surveys reflected in the literature that use payment vehicles reflecting recurring, ongoing payments of exactly the type proposed by the Agency. In nearly all of these cases, the ongoing nature of these payments is implied clearly by the nature of the policy context, as it is in the present survey. Given this—and the results of focus groups—the Agency is not convinced that respondents require still additional

information regarding the details of the payment vehicle.

The current format of the payment vehicle is the result of focus groups on the original version of the 316(b) survey instrument in 2005, with additional testing in the more recent 2010 focus groups. These focus groups and discussions with both OMB and NCEE in 2005 ultimately led to the current description of the payment vehicle. Focus groups showed no difficulty understanding the format of the payment vehicle, and respondents clearly understood the cost as an ongoing cost. The Agency also notes that a similar, ongoing cost of living payment vehicle has been used by prior EPA-sponsored research, with no apparent difficulties (Magat et al. 2000).

Participants in 2010 focus groups clearly understood that selecting Options A or B would result in increased costs to their household. They generally understood that costs to industry would be passed on to consumers, with one participant stating, “So my thought process is that the household stuff is being passed down to consumers from the cost of the added regulation they have” (#3-4). The draft survey was also revised to improve clarity regarding how costs would be passed to consumers by adding a clause on Page 1 stating that goods and services with costs would include “electricity and common household products”. The definition of “cost per year” on Page 7 was also revised to include “services” purchased by consumers. These revisions greatly reduced the potential for respondent confusion when evaluating policy options. Respondents also appeared to understand the ongoing, monthly nature of the payments. Despite the Commenter’s suggestions to the contrary, there is no evidence from focus groups and interviews that respondents had differing perceptions regarding the length of time over which the payments would last.

Similarly, there is no suggestion from focus groups that a “one shot” payment would be more easily understood—and in fact a one-shot payment would be less realistic given the nature of the policies under consideration. It is the goal of the Agency to evaluate WTP based on characterization of the policy changes that is as realistic as possible given the constraints of the stated preference survey format. Description of the payment vehicle as a “one-time payment (which would likely be characterized as a tax increase to be credible)” (p. 8-2) has no justification in terms of the actual mechanisms through which costs would be passed on to consumers, and would require considerable deception on the part of the survey instrument. The Agency can find no justification in the published literature or elsewhere for changing the payment vehicle in this manner—particularly given the successful performance of the proposed payment vehicle in focus groups.

The comment also concludes that “it is appropriate for the EPA to explicitly discuss not only the inherent uncertainty of projected ecological effects but also the accompanying uncertainty in the costs” (p. 8-2). While the basis for this “conclusion” remains unstated, the Agency notes that cost uncertainty is already noted in the survey instrument. For example, bullet points on survey page 10 state: “You will be shown different questions, with different combinations of technology and different costs,” and “Depending on the policies chosen, costs to your household could range from $0 per month to a maximum of $6 per

month.” Page 12 also states “Policy costs and effects depend on many factors.” In sum, the survey already incorporates clear and repeated text that communicates the potential variability in costs.

Ecological Changes

The Agency appreciates the comment’s acknowledgements of the challenges involved in presenting realistic policy changes to respondents, particularly when those changes involve complex ecologicalsystems. As noted by the Commenter, the Agency is relying upon the most recent research in this area

to support its efforts. As a result of the extensive focus group and cognitive interview testing of the original survey instrument reviewed by the Commenter, the survey has already undergone substantial changes in the way it communicates ecological information. Hence, many of the comments in this section are obsolete, as the elements in question are no longer components of the revised survey instrument. The Agency also emphasizes that the crucial element when testing the communication of ecological information in stated preference surveys is the reaction and understanding of potential “real world” respondents. Hence, when determining whether particular survey attributes are appropriate, the Agency gives substantial weight to focus group and pretest responses.

Aside from assuring accurate respondent understanding of the policy changes in question, it is very important to the Agency—following Johnston et al. (2010)—that the changes reflected in the survey are ecologically accurate and (to the extent possible) measurable. Following Johnston et al. (2010) and standard approaches in the ecological literature, all attributes are presented relative to a clear reference condition, reflecting the best possible outcome in a given ecosystem or the level of an indicator that would be expected in an undisturbed system. The comment expresses concern at the “eye catching” (p. 8-3) nature of the changes in the number of fish saved, and also speculates that the non-zero starting points for other attributes “may inadvertently send signals that these other attributes are acceptable at current levels, because they are not zero.”

With regard to these concerns, the Agency first highlights a fact acknowledged by the Commenter— “this treatment of the fish saved [is] technically correct” (p. 8-3). The Agency, however, is also fully aware of the differences in the ecological ranges of attribute effects which occur in the realistic policy contexts shown by the survey, and so has conducted repeated assessments of respondents’ reactions in focus groups. In contrast to the speculation of the Commenter, focus groups showed clear variation in the weight given to fish saved compared to other ecological attributes. Overall, focus group and interview participants’ statements implied different opinions about the importance of preventing fish losses versus increasing fish population or improving the condition of aquatic ecosystems. Participants’ choices and statements implied different opinions about whether changes in fish mortality were of importance without large increases in the long-term fish population or the condition of aquatic ecosystems. Some participants stated that they choose a policy option because of the difference in fish saved, “I went with B because I saw that was a higher percentage fish saved” (#1-11). Comments from some of these participants indicated that they preferred to avoid losses because they place some values on fish in their own right, and clearly recognize the differences in ecological effects: “[...] looking down, some of them are not as much of a difference. But I’m all for trying to save as many as we can” (#4/6-2). Some other participants indicated that effects on populations or ecosystem condition were most important, and that they felt uncomfortable spending money to prevent fish mortality if there was uncertainty in whether it would improve these effects: “I have to admit, I voted generally for the higher plan until we got to question 6, where the condition of the ecosystem didn’t change” (#5-11).

In addition, while the percentage change in fish saved is indeed larger than changes for other attributes, focus group participants generally understood that in most cases the quantity of fish saved would be small relative to the total fish population. For example:

Moderator: When you look at that fish saved per year, how did you think about that in terms of all the fish out there in the water?Respondent: I guess probably a pretty small part [#6-7].

Participants generally felt that the survey provided useful information about fish losses attributable to CWIS, other sources of mortality, and population sizes. Participants generally understood based on the introductory material that CWIS is not the largest source of fish mortality. One participant stated he initially thought that CWIS losses are a very large problem but the statement on Page 3 that “Cooling water use is not the large cause of fish loss in most areas” “ kind of tempers it a little bit” (#7-12). Participants were very aware, however, that the magnitude of fish losses is most appropriately assessed
relative to broader baselines: “ All this information, all this research, but I mean, how many salmons to Americans eat a year, you know? I mean, I don’t know in comparison to 17,000 paddlefish? ” (#3-21). The draft survey was revised to include pie charts comparing young fish losses from CWIS to estimated adult fish populations for example species. This presentation was deemed important in order to address participant comments on intermediate draft survey versions which did not specifically include examples
to this effects.

In summary, the Agency devoted considerable attention to this issue when designing and testing the survey instrument. The suggestion that the presentation of this attribute is “misleading” does not appear to be supported by focus group results. The survey provides extensive information to help respondents understand both the cardinal and relative meaning of all ecological attributes. While recognizing the Commenter’s concerns in this area, the Agency notes that focus group responses, in general, do not appear to support these concerns. For example, among all focus group and interview responses, there was not a single instance in which a respondent thought “that because no fish are saved, they will be become extinct, in accordance with the interpretation of zero for the fish population score” (p. 8-3). Neither were there suggestions from focus groups that the attribute levels implied that “saving fish must be critically important”; in fact many respondents noted explicitly that the broader ecological effects of saving fish seemed to be small. While there are always challenges associated with the communication of complex ecological information in stated preference surveys, the Agency has given this issue substantial attention in survey development and testing. Focus group and cognitive interview results do not suggest that the presentation of the “Fish Saved” attribute is misleading in the ways suggested by this comment.

The Agency agrees with the Commenter that the the Commercial Fish Sustainability Score was potentially misleading. These difficulties were highlighted by focus groups. As a result, this attribute has been removed from the survey.

The Agency has also significantly revised the description of the Aquatic Ecological Condition Score, based on the responses of focus groups and cognitive interviews. Many of the “trigger words” mentioned in the comment have been eliminated (e.g., the survey no longer mentions fish tissue contamination). Moreover, some of the more speculative suggestions of this comment (e.g., inclusion of a water quality discussion may raise concerns for some respondents about the safety of their drinking water) have no basis in focus group or interview results. The Agency placed particular emphasis on this attribute in focus group and interview testing to assess exactly the type of concerns raised by the Commenter in this section. Results suggest that these concerns are largely unfounded. Respondents’ comments suggested a generally appropriate understanding of the Aquatic Ecological Condition Score and why it might change as a result of the proposed policies. For example, when discussing why changes in “Fish Saved” were different than changes in “Aquatic Ecological Condition,” one respondent stated “Fish are only part of it, this ecosystem. You got plants, you got ocean ... all sorts of life so that's why they – it's a different ...percentage is because fish are only a part of it” (#3-11). This comment was typical of respondents’

understanding of this score.

Finally, focus group responses suggested that respondents were acutely aware that many stressors and changes affect ecosystems, that that changes in regulations on CWIS are only one of many factors that can influence ecosystems. There did not appear to be any confusion over the fact that scores of 100 for various attributes are generally unattainable through reductions in CWIS fish losses alone.

Quiz Questions

The proposed survey already includes a number of questions to assess patterns and rationales underlying responses. Moreover, as noted above, respondents in the Agency’s focus groups clearly suggested that additions to the survey’s length should be avoided. In addition, the issues implied by the suggested questions have already been extensively studied in focus groups.

For example, focus group responses clearly indicate that respondents understand “that implementing the policy does nothing to address the largest source of fish losses (harvesting by humans).” To the contrary, focus groups included numerous discussions of the detailed mechanisms through which the proposed regulations would affect fish. Participants generally understood that CWIS primary affects “young fish” which may eventually develop into adult fish. Many participants stated that the graphic on Page 2 of the survey (“How Fish are Affected by Water Intake”) helped them understand how CWIS impacts fish and how it differentially affects eggs, juvenile, and adult fish: “I was thinking more fish is going to get through the screen if they’re smaller. The smaller, the more eggs and the more fish you’re going to lose ” (#6-8). The draft survey was also revised to further clarify the graphic with arrows linking it to supporting text. As said by a participant, “I like the arrows. Large fish are easier to kill. The arrow points to a larger adult fish. Juvenile fish and eggs it points to juvenile fish and eggs ” (#7-10). Many participants showed an understanding of the meaning of “young fish” and the relationship between young fish and adult fish. One stated, “[...] still young eggs and fish are important too [...], but an egg is an egg which grows to an
adult.” (#3-21), and another described juvenile fish as “ the kids” (#6-8). These and other detailed discussions showed no evidence of the type of broad-scale confusion implied by the suggested quiz question.

Moreover, as noted above, there is no evidence whatsoever from focus groups or interviews that “respondents think that 0% of fished saved in the status quo policy means that fish species will become extinct.” Similarly, the Agency conducted extensive focus group testing of the payment vehicle, and found absolutely no evidence that respondents believed that they could avoid cost increases because they “do not buy that many of the affected products.” Participants clearly understood that selecting Options A or B would result in increased costs to their household. They generally understood that costs to industry would be passed on to consumers, with one participant stating, “So my thought process is that the household stuff is being passed down to consumers from the cost of the added regulation they have” (#3- 4).

In summary, the Agency recognizes that there are dozens of possible “quiz questions” which one might suggest for a survey such as this. The fact that one can suggest a quiz question does not mean that its inclusion is justified—particularly in the face of focus group responses which suggest that such areas of confusion are, in general, not a major concern.References

Arrow, K. , R. Solow, E. Leamer, P. Portney, R. Rander, and H. Schuman. 1993. “Report of the NOAA Panel on Contingent Valuation.” Federal Register 58(10): 4602-4614.

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S. Mourato, E. Ozdemiroglu, D.W. Pearce, R. Sugden, and J. Swanson. 2002. Economic Valuation with Stated Preference Surveys: A Manual. Northampton, MA: Edward Elgar.

Cameron, T.A. and J.R. DeShazo. 2005. Comprehensive selectivity assessment for a major consumer panel: attitudes toward government regulation of environment, health and safety risks. Department of Economics, University of Oregon.

Carson, R. T., T. Groves, and M. J. Machina. 2000. “Incentive and Informational Properties of Preference Questions.” Working Paper, Department of Economics, University of California, San Diego.

Carson, R.T. and T. Groves. 2007. Incentive and Informational Properties of Preference Questions. Environmental and Resource Economics 37(1):181–210.

Champ, P.A., and R.C. Bishop. 2001. “Donation payment mechanisms and contingent valuation: An empirical study of hypothetical bias.” Environmental and Resource Economics 19:383–402.

Champ, P.A., R.C. Bishop, T.C. Brown, and D.W. McCollum. 1997. Using donation mechanisms to value non-use benefits from public goods. Journal of Environmental Economics and Management 33(2):151–162.

Champ, P.A., R. Moore, R.C. Bishop. 2004. “Hypothetical Bias: The Mitigating Effects of Certainty Questions and Cheap Talk.” Selected Paper prepared for presentation at the American Agricultural Economics Association Annual Meeting, Denver, Colorado.

Collins, J.P. and C.A. Vossler. 2009. Incentive Compatibility Tests of Choice Experiment Value Elicitation Questions. Journal of Environmental Economics and Management 58(2): 226-235.

Cummings, R. and L. Taylor. 1999. Unbiased Value Estimates for Environmental Goods: A Cheap Talk Design for the Contingent Valuation Method. American Economic Review 89:649-665.

Desvousges, W.H. and V.K Smith. 1988. Focus Groups and Risk Communication: the Science of Listening to Data. Risk Analysis 8:479-484.

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in Designing a contingent Valuation Survey to Measure the Benefits of Hazardous Waste Management Regulations. Research Triangle Institute: Research Triangle Park, NC.

Dillman, D.A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley
and Sons.

EPA–New England (2002) Clean Water Act NPDES Permitting Determinations for Thermal Discharge and Cooling Water Intake from Brayton Point Station in Somerset, MA (NPDES Permit No. MA 0003654), July 22, 2002.Foster, V., I. J. Bateman, and D. Harley. 1997. Real and Hypothetical Willingness to Pay for

Environmental Preservation: A Non-experimental Comparison. Journal of Agricultural Economics 48(2):123-138.

Freeman III, A.M. 2003. The Measurement of Environmental and Resource Values: Theory and Methods. Washington, DC: Resources for the Future.

Gephard, S. and McMenemy, J. (2004). An overview of the program to restore Atlantic Salmon and other diadromous fishes to the Connecticut River with notes on the current status of these species in the
river. In : Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill, eds. The Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower Connecticut River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland, pp. 525-538

Gibson, M. (2002) Winter Flounder Abundance Near Brayton Point Station, Mt. Hope Bay Revisited: Separating Local from Regional Impacts using Long Term Abundance Data. RI Division of Fish
and Wildlife Research Reference Document 02/1

Greene, K. E., J. L. Zimmerman, R. W. Laney, and J. C. Thomas-Blate. (2009). Atlantic coast diadromous fish habitat: A review of utilization, threats, recommendations for conservation, and research needs. Atlantic States Marine Fisheries Commission Habitat Management Series No. 9, Washington, D.C.

Haab, T.C., and K.E. McConnell. 2002. Valuing Environmental and Natural Resources: The Econometrics of Non-Market Valuation. Cheltenham, UK: Edward Elgar.

Hensher, D.A. and W.H. Greene. 2003. The Mixed Logit Model: The State of Practice. Transportation 30(2): 133–176.

Herriges, J., C. Kling, C. Liu and J. Tobias. 2010. What are the Consequences of Consequentiality? Journal of Environmental Economics and Management 59(1): 67-81.

Jacobson, P.M., Leggett, W.C., Marcy, B.C. Jr., Massengill, R.R. (2004) Summary and Conclusion. In: Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill, eds. The Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower Connecticut River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland, pp. 525-538

Johnston, R.J. 2006. Is Hypothetical Bias Universal? Validating Contingent Valuation Responses Using a Binding Public Referendum. Journal of Environmental Economics and Management 52(1): 469-

481.

Johnston, R.J., E.T. Schultz, K. Segerson and E.Y. Besedin. 2010. Bioindicator-Based Stated Preference Valuation for Aquatic Habitat and Ecosystem Service Restoration, in Bennett, J. ed. International Handbook on Non-Marketed Environmental Valuation. Cheltenham, UK: Edward Elgar, forthcoming.

Johnston, R.J., T.F. Weaver, L.A. Smith and S.K. Swallow. 1995. Contingent Valuation Focus Groups: Insights From Ethnographic Interview Techniques. Agricultural and Resource Economics Review 24(1): 56-69.Krinsky, I. and A.L. Robb. 1986. On Approximating the Statistical Properties of Elasticities. Review of

Economics and Statistics 68(4): 715-719.

Kuhfeld, W.F. and Tobias, R.D., 2005. Large factorial designs for product engineering and marketing research applications. Technometrics 47: 132-141.

Louviere, J.J., D.A. Hensher, and J.D. Swait. 2000. Stated Preference Methods: Analysis and Application. Cambridge, UK: Cambridge University Press.

Lusk, J.L. and F.B. Norwood. 2005. Effect of Experimental Design on Choice-Based Conjoint Valuation Estimates. American Journal of Agricultural Economics 87(3):771-785.

Magat, W.A., J. Huber, K.W. Viscusi and J. Bell. 2000. An Iterative Choice Approach to Valuing Clean Lakes, Rivers, and Streams. Journal of Risk and Uncertainty 21(1): 7-43.

Massengill R (1973) Change in feeding and body condition of brown bullheads overwintering in the heated effluent of a power plant. Chesapeake Science 14: 138-141

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation Method. Resources for the Future, Washington, D.C.

Mullaney, J.R. (2004) Summary of water quality trends in the Connecticut River 1968-1998. In: Jacobson, P. M., D. A. Dixon, W. C. Leggett, B. C. Marcy, Jr., and R. R. Massengill, eds. The Connecticut River Ecological Study (1965-1973) revisited: ecology of the lower Connecticut River 1973-2003. American Fisheries Society, Monograph 9, Bethesda, Maryland, pp. 525-538

Murdoch, W.W., R.C. Fay, and B.J. Mechalas (1989). Final Report of the Marine Review Committee to the California Coastal Commission. August 1989, MRC Document No. 89–02.

Murphy, J.J., P.G. Allen, T.H. Stevens, and D. Weatherhead. 2005. A Meta-Analysis of Hypothetical Bias in Stated Preference Valuation. Environmental and Resource Economics 30(3): 313–25.

Murphy, J.J. and T.H. Stevens. 2004. Contingent Valuation, Hypothetical Bias, and Experimental Economics. Agricultural and Resource Economics Review 33(2): 182-192.

Murphy, J. J. T.H. Stevens and L. Yadev. 2010. A Comparison of Induced Value and Home-Grown Value Experiments to Test for Hypothetical Bias in Contingent Valuation. Environmental and Resource Economics 47(1): 111-123.

New York State Department of Environmental Conservation (NYSDEC). (2003). Final Environmental Impact Statement: Concerning the Applications to Renew NYSPDES Permits for the Roseton 1 & 2, Bowline 1 & 2 and Indian Point 2 & 3 Steam Electric Generating Stations, Orange,
Rockland and Westchester Counties.

Neyman, J., and E.S. Pearson. 1933. On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. Roy. Soc. London, Ser. A 231:289–337.

Poe, G.L., J.E. Clark, D. Rondeau, and W.D. Schulze. 2002. Provision point mechanisms and field validity tests of contingent valuation. Environmental and Resource Economics 23: 105–131.Poe, G.L., K.L. Giraud and J.B. Loomis. 2005. Computational Methods for Measuring the Difference in

Empirical Distributions. American Journal of Agricultural Economics 87(2): 353-365.

Powe, N.A. 2007. Redesigning Environmental Valuation: Mixing Methods within Stated Preference Techniques. Cheltenham, UK: Edward Elgar.

Ready, R.C., P.A. Champ and J.L. Lawton. 2010. Using Respondent Uncertainty to Mitigate Hypothetical Bias in a Stated Choice Experiment. Land Economics 86(2): 363-381.

Schläpfer, F., A. Roschewitz, and N. Hanley. 2004. Validation of Stated Preferences for Public Goods: A Comparison of Contingent Valuation Survey Response and Voting Behavior. Ecological Economics 51(1): 1-16.

Seip, K. and Strand, J. 1992. Willingness to Pay for Environmental Goods in Norway: A Contingent Valuation Study with Real Payments. Environmental and Resource Economics 2: 91-106.

U.S. EPA. 2000. Guidelines for Preparing Economic Analyses. USEPA 240-R-00-003. U.S. EPA, Office of the Administrator, Washington, DC, September 2000.

Vossler, C.A. and M.F. Evans. 2009. Bridging the Gap Between the Field and the Lab: Environmental Goods, Policy Maker Input and Consequentiality. Journal of Environmental Economics and Management 58(3): 338-345.

Vossler, C.A., J. Kerkvliet. 2003. “A Criterion Validity Test of the Contingent Valuation Method: Comparing Hypothetical and Actual Voting Behavior for a Public Referendum.” Journal of Environmental Economics and Management 45(3): 631-649.

Whitehead, J. C., P. A. Groothuis and G. C. Blomquist. 1993. Testing for Non-Response and

Sample Selection Bias in Contingent Valuation: Analysis of a Combination Phone/Mail Survey.
Economics Letters 41: 215–220.

Yoo, S.H. and H.J. Yang. 2001. Application of Sample Selection Model to Double-Bounded Dichotomous Choice Contingent Valuation Studies. Environmental and Resource Economics 20:

147–163.