1



# Response to “Comments of American Chemistry Council, American Forest & Paper Association, American Petroleum Institute, and Utility Water Act Group on ICR for Willingness to Pay Survey for Section 316(b) Existing Facilities Cooling Water Intake Structures” (Federal Register Vol. 75, No. 139: 42438- 42440; July 21, 2010) and EPA Supporting Documents

DOCKET ID No, EPA-HQ-OW-2010-0595

Responses to the American Chemistry Council, American Forest & Paper Association, American Petroleum Institute, and Utility Water Act Group Comment

Executive Summary

This section briefly summarizes main issues presented in “Comments of American Chemistry Council, American Forest & Paper Association, American Iron and Steel Institute, American Petroleum Institute, American Public Power Association, and Utility Water Act Group on EPA’s Proposed Information Collection Request on Willingness to Pay Survey for § 316(b) Existing Facilities Cooling Water Intake Structures.” EPA responses are provided below. EPA responses are organized to correspond to the

individual ordered section of the comment.

Section I: Introduction

The section provides background on the proposed EPA survey based on the information provided in the Agency’s Information Collection Request (ICR), lists objectives of the report, and introduces organization of the report. No response is necessary.

Section II: Benefit Categories and Assessment Guidelines for Developing Reliable Contingent Valuation Studies and Previous Attempts to Measure Non-Use Values

The Agency notes that the majority of this section does not address the specific survey materials or information collection approaches described in the Agency’s ICR. Instead, these pages provide a summary of benefit categories using EPA Guideline for Preparing Economic Analysis (EPA 2000, hereafter Guidelines). The commenter states that general categories of benefits potentially relevant to § 316(b) regulatory assessment include: market benefits, non-market direct use benefits, non-market indirect use benefits and non-use benefits. The commenter then argues that non-use values for fish protection are likely not to be significant. The commenter points out that two economic criteria must be met if non-use values for fish protection are likely to be significant (1) the resource is unique and (2) the loss would be irreversible or subject to a long recovery period.2

EPA disagrees with the commenter that non-use values for fish protection are likely to be insignificant and notes that the economic literature advises against the use of ad hoc, expert assumptions regarding the magnitude of non-use values—particularly assumptions that non-use values are trivial. Because values are inherently subjective, Bateman et al. (2002, p. 75) emphasize that “it would be wrong for experts to assume that one resource is a perfectly good substitute for another” [and hence that non-use values are trivial or small]. As further noted by Bateman et al. (2002, p. 75), “there are, therefore, no easy rules for determining at the outset” whether non-use values are likely to be significant or non-significant. Based on this clear guidance, EPA believes that empirical analysis should be used to determine the magnitude of non-use values in the 316(b) regulation for existing facilities.

Furthermore, results from the focus groups and cognitive interviews indicated that some individuals hold substantial non-use values for protecting all fish species including forage fish. Participants in the EPA focus groups and interviews cited a variety of motivations for preventing fish losses, including the satisfaction of knowing that the fish exist, the desire to bequeath healthy fish populations to future generations, and the desire to protect the functioning of aquatic ecosystems.

Section III: Guidelines for Developing Reliable Contingent Valuation Studies and Previous Attempts to Measure Non-Use Values

The Agency notes that the majority of this section does not address the specific survey materials or information collection approaches described in the Agency’s Information Collection Request (ICR). Instead, these pages provide a summary of contingent valuation methods in general, along with a selection of guidelines chosen from references of varying ages and degrees of obsolescence. Given that many of these pages do not refer directly to the Agency’s proposed survey or methods, specific responses to these sections are largely unnecessary. The Agency will focus responses on areas in which the comments refer to the specifics of the proposed information collection request, including the proposed stated preference methods. Given that some of the Commenter’s later points refer back to this general introductory material, however, the Agency feels that it is important to place these materials and associated claims in a broader context of the stated preference literature.

Finally, some of the following responses refer to the extensive focus groups used by the Agency to develop and test the proposed survey. The focus groups and interviews took place in the following

locations and on these dates: 1

1) East Providence, RI, July 21, 2010

2) Waltham, MA, July 29, 2010

3) St. Louis, MO, August 4, 2010

4) East Providence, RI, August 12, 2010

5) Bethesda, MD, August 18, 2010

6) Charlotte, NC, September 8, 2010

1 In this response, we use the citation format (#X-Z to refer to page Z of the transcript for focus group X. Citations to cognitive interviews use a similar format, (#X/Y-Z), where Y represents the interview number.3

7) Philadelphia, PA, September 23, 2010

III.A. Overview of Contingent Valuation

As a precursor to later comments, the Agency notes that this comment uses largely obsolete, and perhaps misleading, technical language to describe the proposed methods. The broader category of survey-based valuation—in which values are based on respondents’ answers to questions involving hypothetical commodities or policy changes—is most commonly denoted “stated preference valuation.” The contemporary literature generally uses phrase “contingent valuation,” in contrast, to refer only to questions involving open-ended valuation questions. In contrast, the methods proposed by the Agency apply choice experiment (or choice modeling) methods (Bennett and Blamey 2001); these are not the same as open-ended contingent valuation questions, which involve “directly asking respondents their willingness to pay (WTP) for a particular commodity or environmental change” (comment p. 8). These distinctions are critical, as the methods and characteristics of the two methods are different, and the two methods have been clearly shown to differ with regard to the prevalence of certain biases and response anomalies. This distinction is reflected as far back as the original guidelines of the NOAA Blue Ribbon Panel (Arrow et al. 1993), which recommends choice-based methods. The Agency notes that the comment frequently obfuscates of the distinction between open-ended contingent valuation and choice experiments, leading to comments or implications that might be considered misleading—particularly to those unfamiliar with the proposed methods.

This general comment aside, the Agency agrees, in general, with most of the broad description of stated preference methods presented in this section. The majority of this Commenter’s points in this section are not specific to the proposed stated preference methods, but rather provide a general introduction to stated preference methods in general. As such, and given that nearly all of this material is well-established in the literature, the Agency sees no specific need to comment. As a precursor to more detailed responses later, however, the Agency notes that it is fully aware of the broad literature on stated preference methods. The Agency also believes that the introductory material tends to emphasize language and position taken by skeptics rather than practitioners of the approach; that is it provides a rather one-sided perspective on the literature. For example, while noting accurately that “[s]ome economists take a dim view of the ability of CV surveys to reliably measure the “non-use” values” (p. 8), the comment fails to reflect the fact that many economists also support the ability of stated preference surveys to measure these values, and that stated preference surveys are widely used in the published literature for such purposes. Agency guidelines also support the use of stated preference surveys to quantify non-use values (US EPA 2000).

Also, the Commenter’s language in this section (particularly at the top of page 9) seems to imply that there is a single set of universal guidelines that apply to all stated preference surveys regardless of type or context. For example, consider the phrase “any CV study that is to be used in a policy decision should both be well-designed (i.e., should follow the proposed guidelines)...” (p. 9, emphasis added). This misleadingly implies that there is a single set of guidelines. While there are many agreed-upon “best practices” for stated preference survey design and implementation (see, for example, Bateman et al. 2002 and Louviere et al. 2000), the Agency also notes that there are many other areas in which the literature has not agreed upon these guidelines.

Finally, the Agency believes that the discussion of the cost of stated preference survey implementation vis à vis the assumed size of nonuse values is also misleading. First, the majority of the costs of the proposed survey—those involved with survey development and focus group testing—have already been incurred4

and hence are most appropriately viewed as sunk costs, and therefore irrelevant to future decisions. More importantly, this comment is based on an unjustified presumption that nonuse values in the present policy context are likely to be small. EPA emphasizes that the magnitude of nonuse benefits associated with changes in aquatic habitat is an empirical question—and that the potential significance of such benefits can only be assessed using appropriate empirical methods. EPA is aware of no research—empirical or otherwise—that demonstrates that individuals have limited existence motives for the types of fish that would be protected by policies at issue. The Agency notes that the same issue was raised by the Commenter in responses to a prior Agency ICR (American Chemistry Council, American Forest & Paper Association, American Iron and Steel Institute, American Petroleum Institute, American Public Power Association, and Utility Water Act Group on EPA’s Proposed Information Collection Request on Willingness to Pay Survey for § 316(b) Phase III Cooling Water Intake Structures). The Agency refers the Commenter to our original responses to the same issue in the earlier submitted comments.

III.B. Guidelines for Contingent Valuation Studies

As noted above, the material in this section largely summarizes selected guidelines for stated preference surveys found in a variety of documents. As this section draws no conclusions regarding the ICR, the Agency has few comments here. The Agency, however, notes that the majority of guidance reflected in this section is drawn from documents that are now more than fifteen years old, and hence do not reflect more recent advances in stated preference design or testing. This section is also drawn from a very small portion of the available literature and guidance related to stated preference methodology. Hence, the Agency emphasizes that the comments in this section are most appropriately viewed as a selective perspective on a small, and often fairly old, segment of the relevant stated preference literature.

III.C. Internal Validity Tests

This section neither draws conclusions regarding the ICR nor offers concrete suggestions. Rather, this section summarizes a variety of validity tests that might be conducted to assess the reliability of stated preference surveys. As these comments are not specific to the survey or ICR in question, the Agency has only a few comments.

Scope Tests : While noting the clear limitations in naïve scope tests discussed by Heberlein et al. (2005), the Agency agrees that it is important that survey responses in this case show sensitivity to scope. This is one of the primary reasons for the use of choice experiment methodology, which is better able to capture WTP differentials related to changes in resource scope (Bateman et al. 2002). Unlike open-ended questions, in which scope insensitivity is a primary concern, EPA emphasizes that choice experiments generally have shown much less difficulty with respondents reacting appropriately to the scope and scale of resource changes. Moreover, as clearly noted by Bennett and Blamey (2001, p. 231), “internal scope tests are automatically available from the results of a [choice modeling] exercise.” That is within choice experiments, sensitivity to scope is indicated by the statistical significance and sign of parameter estimates associated with program attributes (Bennett and Blamey 2001). Internal scope sensitivity will therefore be tested through model results for choice experiment attributes.

In addition to internal scope tests implicit in all choice experiment statistical analysis, EPA will also conduct external scope tests (cf. Giraud and Loomis 1999). The primary difference between internal and external tests is that the former assess sensitivity to scope across choices of a single respondent, while the latter involves split-sample assessments across different respondents. Within a choice modeling context,5

external scope tests are generally considered stronger, although also more likely to be confounded by differences in the implied choice frame (Bennett and Blamey 2001). A variety of options for external scope tests exist, depending on the structure of stated choice questions under consideration. In the present case, attribute-by-attribute external scope tests will be conducted over a split sub-sample of respondents considering a specific set of choices, with all held constant across the considered choices except the scope of the attribute for which the test is to be conducted. For example, to conduct an external scope test for reductions in annual fish losses, one would consider a set of choices that is identical over two respondent groups, except that one considers a choice with a greater reduction in fish losses. Assessing the choices over this split sample allows for an external test of scope. The Agency emphasizes that the formal applicability of the above-noted scope test is contingent upon the specific choice frame implied by levels of other attributes in the choice question. This is a characteristic of nearly all external scope tests applied in choice experiment framework (Bennett and Blamey 1999).

Split-sample tests such as those proposed above often require the addition of question versions to the experimental design, to accommodate the specific structural needs of the attribute-by-attribute external scope test. Otherwise, confounding effects of other varying attributes can render results of scope tests ambiguous. In the present case, the proposed tests would require the addition of new profiles to the experimental design. While small numbers of questions added to the experimental design should have minimal impacts on overall efficiency, larger numbers may have a more significant impact. Hence, given constraints on the total number of survey respondents, there is an empirical tradeoff between the number of external scope tests that may be conducted and the efficiency of the experimental design and statistical analysis.

The “Adding-Up” Test : The “adding up” test is based on an oversimplified perspective on embedding effects that ignores the influence of task frame and order on behavior. Economic theory clearly predicts that “the simultaneous availability of other projects” will generally affect the value of any other policy in question (Carson et al. 1999). Moreover, as noted by Hanemann and Kanninen (1999), “evidence in market research as well as in environmental valuation overwhelmingly rejects procedural or task invariance.” This issue does not relate to any type of survey bias or difficulty with non-market valuation, but rather is a fundamental theoretical issue related to the potential substitutability among goods provided by government policies. Within the specific issue of embedding in stated preference surveys, it is also well known that such behavioral patterns may occur equally in market and non-market settings (Carson et al. 1999; Hanemann 1994). As stated by Hanemann (1994), “it should come as no surprise that the value of one commodity changes when the quantity of another varies: in other words, that willingness-to-pay depends on economic context.” The “adding up” test also presumes an identical choice frame when assessing WTP for Good B (presuming that Good A has already been provided) and assessing WTP for simultaneous provision of Goods A and B. The fact that this assumption is almost certainly invalid likewise invalidates the associated test. Given these unavoidable concerns, the Agency does not view the “adding up” test as an appropriate validation of the proposed survey instruments.

Anchoring Tests: The Agency notes that classical anchoring tests applied to dichotomous choice contingent valuation, as described by the comment, are not strictly applicable to choice experiments.

Econometrically Estimating the Relationship between WTP Answers and Various Factors: As a clarifying note, the Agency first emphasizes that the proposed choice experiments will not solicit “WTP Answers,” as implied by the comment. This seems to reflect the confusion between open-ended contingent valuation and choice experiments noted above. However, the Agency does plan to conduct an6

extensive evaluation of choice responses, including an evaluation of the impact of socioeconomic and other measurable factors on implicit prices.

III.D. Prior Efforts to Measure Non-Use Value Using Stated Preference Methods

The Agency strongly rejects the claim that “[p]rior efforts to measure non-use values using stated preference methods cast substantial doubt on the reliability of the method, contrary to what EPA contends” (p. 17). The Agency notes that this entire section is drawn from an unsystematic discussion of only two past stated preference studies, both of which: (1) were conducted nearly two decades ago, (2) used methods different than those proposed by the Agency in the present ICR, including some approaches that are now considered rudimentary and/or largely obsolete, (3) addressed resource types distinct from those addressed in the present ICR. Given the breadth of the contemporary stated preference literature— including thousands of published works—the Agency strongly rejects the argument that any conclusive insights regarding the ability to estimate non-use values may be drawn from an ad hoc discussion of two research studies conducted using methods different from those proposed by the Agency within the current

ICR.

Section IV: Likely Biases in the Proposed EPA Survey

As a precursor to more detailed comments below, the Agency rejects the claim that the suggested “sources of bias...would render the proposed EPA survey unreliable” (p. 19). Responses to specific

issues are listed below.

IV.A. Bias Due to Creation of Preferences

The first comment in this section is that “the benefits estimated from the EPA survey would be conditioned on the information included in the survey.” Specifically, the comment suggests that “people had little or no prior awareness of CWIS, much less knowledge about cooling water intakes and the purported effects on fish populations” (p. 19). The Commenter claims that this creates a significant concern that “EPA has created preferences, where none existed before.” and that EPA has ignored the “substantial psychological literature on constructed preferences” (p. 19). The Agency notes that these comments are similar in substance to earlier comments received from the same Commenter in response to an earlier version of the survey (American Chemistry Council, American Forest & Paper Association, American Iron and Steel Institute, American Petroleum Institute, American Public Power Association, and Utility Water Act Group on EPA’s Proposed Information Collection Request on Willingness to Pay Survey for § 316(b) Phase III Cooling Water Intake Structures). The Agency finds these comments similarly unconvincing.

With regard to the general comment that respondents’ answers will be informed by, and conditional on, the information provided in the survey, EPA believes this result is entirely expected, and is consistent with the academic literature. It is well known that information can influence economic values (Bergstrom and Stoll 1989; Bergstrom et al. 1989; Hoehn and Randall 2002; Cameron and Englin 1997). The Agency emphasizes that the sensitivity of values (and behavior) to information is true of both market and non-market values (Bateman et al. 2002, p. 298), and in no way invalidates the proposed non-market values that would be estimated through the proposed stated preference approaches. Cameron and Englin (1997), for example, provide a detailed discussion of the various types of information that may be possessed by or provided to respondents, and how this information may influence values in a legitimate7

neoclassical benefit cost framework. It is also interesting to note that Cameron and Englin’s (1997) analysis focuses on a policy topic that is fairly similar to the one addressed here—WTP for a doubling of trout in the northeastern United States. Hence, the Agency feels that this analysis and discussion is highly relevant to the proposed stated preference survey instrument and analysis.

More specifically, however, this comment reflects two distinct but related claims. The first is that the presentation of significant information to survey respondents is inappropriate. The second is that the presentation of information in a survey instrument—particularly when extensive and paired with policy issues of which respondents may have limited prior knowledge—will likely result in “constructed preferences” inconsistent with neoclassical benefit cost analysis. Based on its review of the economics literature and standard practice in benefit cost analysis, the Agency finds neither of these arguments to be particularly convincing, and notes that such perspectives—while indeed found in some psychological literature—are not reflected in the guidance provided by the mainstream valuation and benefit-cost

literature.

With regard to the appropriateness of information provision in stated preference surveys, it is common practice in such surveys to provide substantial information to survey respondents. In contrast to the claims of the Commenter, the broader literature tends to treat with skepticism those surveys that do NOT provide sufficient information to respondents. As stated by Bateman et al. (2002, p. 309), “respondents should be given enough information in a clear-enough fashion so that they should be comfortable making a decision.” It is well established in the stated preference literature that valid stated preference value estimation requires that respondents are provided with an adequate, accurate, and detailed representation of changes to natural resources. For example, the NOAA Blue Ribbon Panel on Contingent Valuation (Arrow et al. 1993, p. 4605) state clearly that “if CV surveys are to elicit useful information about willingness to pay, respondents must understand exactly what it is they are being asked to value (or vote upon)...” The pane further indicates that surveys providing “sketchy details” about the results of proposed policies call “into question the estimates derived there from,” and hence suggest a high degree of detail and richness in the descriptions of scenarios. Similar guidance is provided by other key sources in the CVM literature (e.g., Mitchell and Carson 1989; Louviere et al. 2000; Hanemann 1994). Given this clear guidance to provide detailed, specific, and quantitative information, the Agency is not convinced by the argument that the provision of substantial amounts of information is inappropriate or will lead to a biased expression of value among respondents. To the contrary, the Agency believes, following the literature, that the provision of minimal information in the survey would encourage exactly the type of symbolic responses and embedding that the Commenter discusses in subsequent sections. Multiple focus groups and cognitive interviews conducted by the Agency were conducted to determine, among other things, the type and quantity of information that was required by respondents in order to make an informed, comfortable decision, as recommended by Batemen et al. (2002) and many other sources (e.g., Mitchell and Carson 1989). These focus group and interview results were used to determine the type and quantity of information provided in the survey.

With regard to the argument that the substantial information provided by the survey indicates that the survey is “creating preferences” among respondents, the Agency emphasizes that this argument is one that has been leveled at stated preference methods for over 15 years. This argument was not judged to be an overriding argument against stated preference valuation by the NOAA Blue Ribbon panel (Arrow et al. 1993), nor it is generally considered to be convincing by the benefit cost literature. Moreover, as noted by Bateman et al. (2002, p. 298), arguments regarding constructed preferences in stated preference valuation are often based on a misconception concerning that which economic theory “does and does not8

say about preferences. Contrary to common criticism, economic theory does not dictate that individuals have prior-formulated values for all possible options that might be offered to them... [T]he neoclassical economic view is that ... survey responses must be driven by preferences which are consistent with economic theory...” Similar to the context found in many stated preference instruments addressing goods that may be unfamiliar to some respondents, consumers in actual markets are frequently presented with new market goods for which they have no experience. The fact that consumers may have little knowledge of new market goods prior to their introduction does not invalidate the ability of markets to reveal values; nor does it invalidate the ability of stated preference methods to estimate non-market values (Bateman et al. 2002). 2

The Commenter continues by claiming that the “such a blank slate on the part of most, if not all of the survey respondents, makes the information presented problematic” (p. 19). This comment suggests that the Commenter has confused the multiattribute good being valued (attributes associated with fish losses and associated ecological effectsw) and the policy context in which it is being valued. Surely, the Commenter cannot be suggesting that people have no familiarity with fish. Rather, many people are relatively unfamiliar with a particular type of human effect on fish (i.e., that caused by CWIS). This parallel is quite similar to that found in Cameron and Englin’s (1997) analysis, which assessed WTP to avoid effects of acid rain on trout populations. The fact that many individuals were not familiar (prior to the survey) with the effects of acid rain on trout did not invalidate WTP measures in this published case. Indeed, in the view of the Agency, the survey at issue in the present ICR exemplifies the type of valuation context addressed by Cameron and Englin, in which “respondents themselves will have acquired varying levels of experience with the commodity in question, or at least with closely related commodities,” but in which the survey provides additional information used by respondents to formulate their valuation

responses.

IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation

This comment suggests that survey materials prepared by the Agency contain inaccurate statements and invalid comparisons that will lead respondents to believe that the benefits of increased regulation of cooling water intakes would be substantially greater than is actually the case. The Agency rejects this claim, and emphasizes that the survey is explicitly designed to provide respondent with an understanding of the proposed policies that is as accurate as possible given the best available ecological science.

The initial paragraphs of this section draw no conclusions regarding the ICR. As a result, the Agency has no comments on these paragraphs. The Agency’s responses begin with the misleading claim that “Beginning with the title page of the survey, EPA provides information to respondents that leads them to believe that they are valuing a policy with much wider-reaching effects than can be supported by the relevant science” (p. 20). The Agency can find no examples in the revised (post focus group) survey of “skewed description of the relative importance of the effects of I&E on fish populations.” For example, the revised survey clearly states “Cooling water use is not the largest cause of fish loss in most areas (fishing causes greater losses), but has affected some fish populations.” Moreover, the survey now provides explicit information comparing losses of age-1 equivalents to total estimated populations for representative species. These and other devices in the revised survey are designed to provide a clear and

2 In an example provided by Bateman et al. (2002, p. 341), “prior the introduction of the video recorder individuals could not have values for such a good. This did not prevent them from determining such values when such a

choice became available.”9

unbiased perspective on the magnitude and effects of entrainment and impingement. The Agency emphasizes that the estimates of impingement and entrainment losses used in the survey are scientific estimates specific to the policy options valued in this study. The policies considered by EPA in the context of entrainment and impingement reductions would prevent the loss of a potentially large number of age-one equivalent fish, both regionally and nationwide. The data presented in the survey represent the best scientific estimates of fish losses and population changes available.

Other comments in this section are similarly unsupported by either the content of the survey instrument itself or the results of focus groups. For example, the claim “EPA’s survey materials substantially overstate the effects that are likely to occur on recreational and commercial fishing as a result of the proposed policy. After reading the materials, people are much more likely to think that the majority of the impacts will be on the recreationally and commercially significant species” is explicitly contradicted both by direct statements in the survey and by the comments of focus group participants. The survey explicitly states that the majority of affected species are “forage species not caught by humans.” This is clearly understood by respondents. This is reflected, for example, in the statement of one focus group respondent: “You know it was educational that there’s a lot more forage fish that there are fish that are actually caught and consumed by humans” (#7-7).

Similar claims of Barnthouse (2010)—attached to the submitted comment—are also unsupported by the Agency’s focus groups and interviews used to test respondent comprehension of the proposed survey instrument. First, numbers in the survey are already presented as Age-1 equivalents—a fact apparently misunderstood by Barnthouse (2010). This seems clear due to the statement that “[t]he survey materials should, like the case study documents, express E&I losses in terms of age-one equivalents” Barnthouse (2010, p. 1). More broadly, the Agency’s focus groups clearly suggested that respondents generally understood that CWIS primary affects “young fish” which may eventually develop into adult fish. Many participants stated that the graphic on Page 2 of the survey (“How Fish are Affected by Water Intake”) helped them understand how CWIS impacts fish and how it differentially affects eggs, juvenile, and adult

fish: “I was thinking more fish is going to get through the screen if they’re smaller. The smaller, the more eggs and the more fish you’re going to lose” (#6-8). Moreover, despite the Commenter’s claims to the
contrary, focus group participants generally understood that in most cases the quantity of fish saved would be small relative to the total fish population. For example:

Moderator: When you look at that fish saved per year, how did you think about that in terms of all the fish out there in the water?

Respondent: I guess probably a pretty small part [#6-7].

Participants generally felt that the survey provided useful information about fish losses attributable to CWIS, other sources of mortality, and population sizes. Participants generally understood based on the introductory material that CWIS is not the largest source of fish mortality. The draft survey was also revised to include pie charts comparing young fish losses from CWIS to estimated adult fish populations for example species. This presentation was deemed important in order to address participant comments on intermediate draft survey versions which did not specifically include examples to this effect.

In summary, the Agency devoted considerable attention to this issue when designing and testing the survey instrument. The suggestion that the presentation of the magnitude of fish losses is “misleading” does not appear to be supported by focus group results. More broadly and importantly, however, the10

proposed comment suggests a lack of understanding of the capacity of choice experiments to isolate the marginal effects of different attributes on stated choices and hence on WTP—a methodology clearly described by numerous sources (e.g., Louviere et al. 2000). The choice experiment survey and experimental design are explicitly designed such that the model can incorporate a wide-range of potential outcomes, including zero changes in population and harvest. That is, the choice scenarios (or profiles) explicitly evaluate respondents choices and implied values for cases in which changes in ecological outcomes (including changes in fish losses) is both large and small. Unlike older forms of stated preference analysis in which WTP is contingent upon a single specification of ecological effects, choice experiments solicit responses under a large number of possible attribute configurations. This will allow the Agency, for example, to estimate WTP for policies in which the broader ecological effects of I&E reductions—beyond direct influences on the number of age-1 equivalent fish lost—are zero.

In summary, the Agency finds the comments in this section to be unsupported by evidence from either the survey instrument, broader ecological science or focus group results. The comments also suggest a misunderstanding of the valuation approach proposed by the ICR, and its capacity to estimate WTP for a broad range of possible ecological outcomes.

IV.C. Bias Due to the Proposed EPA Survey’s Failure to Characterize Accurately Substitutes and
Alternatives

The comments offered in section IV.C.1 draw no conclusions regarding the ICR. As a result, the Agency has few comments on these paragraphs. However, the Agency would like to highlight the fact that the quoted guidelines universally relate to the requirement that stated preference surveys provide an accurate description of the commodity to be valued . None of these guidelines or the associated quotes address a “failure to characterize accurately substitutes and alternatives,” (p. 21) as implied by the comment. The Agency agrees that valid stated preference valuation requires that the survey provide an accurate characterization of the commodity to be valued, as suggested by the quoted sources.

Section IV.C.2 is entirely contingent upon a distinct claim that is not supported by any of the quoted guidelines above—that “respondents must understand not only the program or resource in question, but

also the alternatives or substitutes to the program or resource in question ” (emphasis added). As stated by Carson et al. (1999, p. 120-121) regarding the potential impact of substitutes on estimated values, “the value of [a project] may become arbitrarily small if enough projects are considered and the projects are substitutes with respect to valuation.” They clearly note also that “this is a purely theoretical issue which is not related to contingent valuation or passive-use values per se.” In doing so, Carson et al. (1999) note that “researchers need only consider other [substitute projects] that will actually be undertaken at the same time, rather than all potential projects.” They also recommend that the set of substitutes presented to respondents be limited to “close substitutes of which respondents are unaware,” and criticize “extreme position[s]” such as that of Kemp and Maxwell’s (1993) Exxon-sponsored study in which respondents were asked to allocate expenditures to as many as 32 different categories of public expenditure. To the contrary, Carson et al. (1999) notes that “in most cases, the number of potential relevant and unknown substitutes is small and as such represents little problem in designing and administering the survey.”

The Agency also notes that the entirety of section IV.C.2 relates to potential broader effects of closed cycle cooling. This is based on a misleading and incorrect presumption that the purpose of the Agency’s proposed survey is to evaluate WTP for cooling towers. As stated clearly by the ICR, this is not the case. The clearly stated purpose of the stated preference survey is to “elicit values for changes in fish mortality11

and the associated ecological gains from a section 316(b) regulation covering Phase II and Phase III facilities” (EPA ICR No. 2402.01). Given that the purpose of the survey is not to evaluate WTP for cooling towers and their impacts, the broader ecological impacts of cooling towards and closed cycle cooling are not relevant to the ICR.

The Agency acknowledges that there may be a range of other welfare-relevant effects associated with the operation of closed-cycle cooling. Evaluation of these benefits and costs, however, is not the purpose of the ICR, nor is it required for appropriate assessment of WTP for changes in fish mortality and the associated ecological effects. Hence, the material in this section is neither relevant for the stated preference survey under consideration, nor is the related information required in order to provide an accurate description of the commodity to be valued.

IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes

The comments in this section address a number of only loosely related issues, few of which are related to “inherent collinearity in the ... attributes respondents are valuing.” As a set, the Agency finds the comments in this section to be unclear and unconvincing. Moreover, they may suggest some lack of understanding of the choice experiment methodology that is proposed.

The first comment in this section may be summarized by the concluding sentences: “While supposedly people are thinking that the only fish being saved are forage fish, if they are really considering recreationally and commercially significant fish in answering the questions, then there is no independence among the attributes. Consequently, the values that would be attributed to forage fish are really values for some other fish species.” (p. 26) The Agency is puzzled by this comment, as it seems to reflect a lack of understanding of both the proposed survey scenarios and the proposed choice experiment methodology. The survey instrument developed and revised as a result of the Agency’s focus groups includes independent and separate ecological attributes communicating changes in (1) fish saved (all species), (2) fish populations (all species), (3) commercial and recreational fish populations, and (4) the health of aquatic ecosystems. These were chosen as a result of extensive focus group testing and consultations with Agency scientists. Given that these are presented in choice experiment scenarios as independent attributes, the Agency sees little likelihood that respondents will somehow confuse the meaning of these attributes, as claimed in this part of the comment. Focus group results strongly support this contention, suggesting that respondents were able to distinguish the attributes communicating changes in all fish species and those communicating changes in only commercially and recreationally significant species. The following focus group quote is typical, showing a clear distinction between “commercial fish populations” and “fish populations (all fish)”: “[...] specifically the fish that are more generally used as food for people versus some of these other fish that are just fish [...]” (#2-5). Moreover, repeated questioning in focus groups showed that respondents found it useful and instructive to have different attributes to represent effects on broader fish populations (of all fish) and populations of recreationally and commercially significant species. The following quote is typical: “[...] it was nice to know that the differentiation between commercial fish and recreational fish and the forage fish ... having that information kind of influenced some of my votes I would say” (#7-10). This evidence suggests little of the

confusion claimed within this comment.

From a technical perspective, the Agency notes that statistical independence among different choice attributes is assured by the experimental design. In fact, the choice attributes are nearly orthogonal within the Agency’s proposed design, providing much lower collinearity (approximately zero) than is possible in12

virtually any other type of economic valuation. This standard practice within choice experiments allowsimplicit prices associated with each attribute to be statistically isolated. The Agency has used a Defficiency criterion for the experimental design, which optimizes the design in terms of such factors as orthogonality and balance (Kuhfeld and Tobias 2005; Lusk and Norwood 2005), thereby minimizing the influence of any possible collinearity on model results. This approach, when applied appropriately, explicitly minimizes collinearity among attributes, again suggesting that Commenter’s concerns in this

area are unfounded.

It is unclear what is meant by the Commenter’s claim that “respondents are supposed to ignore levels and changes in the various scenarios.” To the contrary, the survey explicitly asks respondents to consider attribute levels and changes in order to answer choice scenarios. Hence, there seems to be little basis for this contention. Similarly, regarding the implied comparisons across different choice questions, the survey includes an entire page devoted to instructions in this area. These instructions are as follows (survey page 12):

“Now you will be asked to consider a new set of policy options for Northeast waters. As you vote, please remember—

 Questions 5 and 6 present new sets of policy options. These options require a different mix of technologies in different areas.

 Each question is a separate vote. Questions 5 and 6 cannot be directly compared to each other, or to Question 4.

 Do not add up effects or costs across different questions.

 Policy costs and effects depend on many factors. Saving more fish does not necessarily mean that all effects will improve.”

Focus groups clearly indicated that respondents read and understood these instructions. Participants understood the choice task correctly and generally believed that the survey decision scenario was clear. Nearly all respondents indicated that they followed the survey’s instruction to evaluate each policy question on its own and did not compare policy options across pages. A small minority of participants stated that they did look page across the choice questions when completing the participants looked back across pages but these participants generally indicated that this was to help them ensure consistency across their responses. As stated by one participant, “I did look back, not to compare, but I did look back just to see how I answered [...] to make sure that I’m thinking about this right and being consistent. So I don’t know that I compared, I just wanted to make – I think it helped me understand” (#3-7). When probed it was clear that they were answering the choice questions independently. For example:

Moderator: ... were you looking at it in reference to other questions, [or] answering them as a group?

Response 1: No, on its own.

Response 2: On its own.

Response 3: On its own. [#3-7]

This combined evidence provides no suggestion of the type of claims made by the Commenter in this

section.13

The final comment in this section relates to the general claim that “The indexes are confusing and overly broad, especially the index of the health of the overall aquatic environment.” With regard to these indices, most have been changed substantially from those reviewed for this comment, based on input from both Agency scientists and from focus groups. The revised indices are both more narrow and more clear. This includes revisions to the description of the Aquatic Ecological Condition Score, based on the responses of focus groups and cognitive interviews. Moreover, some of the more speculative suggestions of this comment (e.g., inclusion of a water quality as a component of the index may cause respondents to believe that it is being “influenced in some way” by the proposed regulations) have no basis in focus group or interview results. The Agency placed particular emphasis on this attribute in focus group and interview testing to assess exactly the type of concerns raised by the Commenter in this section. Results suggest that these concerns are largely unfounded. Respondents’ comments suggested a generally appropriate understanding of the Aquatic Ecological Condition Score and why it might change as a result of the proposed policies. For example, when discussing why changes in “Fish Saved” were different than changes in “Aquatic Ecological Condition,” one respondent stated “Fish are only part of it, this ecosystem. You got plants, you got ocean ... all sorts of life so that's why they – it's a different ... percentage is because fish are only a part of it” (#3-11). This comment was typical of respondents’ understanding of this score. Focus group responses suggested that respondents were acutely aware that many stressors and changes affect ecosystems, that that changes in regulations on CWIS are only one of many factors that can influence ecosystems.

In summary, the Agency finds no convincing arguments in this section to support the speculation that “willingness-to-pay estimates are for something other than protecting forage fish and are inherently flawed and biased” (p. 26).

IV.E. Hypothetical Bias

The first three pages of section two do not address the specific survey materials or information collection approaches described in the Agency’s Information Collection Request (ICR). Instead, these pages provide a summary of selected publications that address the general issue of potential hypothetical bias in stated preference survey instruments. The Agency notes that this is a well-known concern with stated preference surveys, accompanied with decades of research addressing its severity and extent, as well as the effectiveness of approaches for its mitigation. Hence, the majority of this Commenter’s points in this section are most appropriately viewed not as specific critiques of the proposed survey instrument, but rather as a comment on stated preference methods in general. The Agency has no comment on this

literature review.

As a precursor to more detailed comments, the Agency notes that it is fully aware of the research addressing the potential for hypothetical bias in stated preference welfare estimation. The Agency emphasizes that the majority of the citations used within this section were published at least ten years ago, prior to a substantial body of research addressing potential mechanisms for hypothetical bias mitigation. The Agency also believes that the Commenter, in summarizing this literature, has provided a somewhat one-sided perspective on the relevant published research. For example, even after reviewing much of the older information on hypothetical bias cited by the comment, the NOAA Blue Ribbon Panel stated that

, “[t]he Panel concludes that under [conditions described in the report], CV studies convey useful information. We think it is fair to describe such information as reliable by the standards that seem to be implicit in similar contexts, like market analysis for new and innovative products...” (Arrow et al. 1993, p. 4610). The Panel also states that “[hypothetical bias] may be offset or even over-corrected” through14

conservative design practices. A broad review and meta-analysis of the hypothetical bias literature is also provided by Murphy et al. (2005), who find a median ratio of hypothetical to real WTP of only 1.35 . The Agency believes that this broader, more comprehensive analysis shows are more accurate picture of the extent of hypothetical bias than the narrow selection of the literature reflected in the comment. Moreover, Murphy et al. (2005) clearly indicate that the types of findings cited in this comment—including hypothetical WTP estimates that are “ four to six times” actual WTP—are best described as outliers in the

literature. In fact, one of the conclusions of Murphy et al. (2005, p. 323), that “the overwhelming majority of observations have relatively low [hypothetical bias], possibly suggesting that hypothetical bias may not be as significant a problem in stated preference analyses as is often thought.” Hence, while the Agency agrees that hypothetical bias is a concern—and has taken this very seriously in survey development and testing—it does not agree that substantial hypothetical bias is unavoidable.

A more comprehensive review of the recent literature would also have revealed methodological advances provide various means for the amelioration of hypothetical bias. For example, many researchers have had success using cheap talk methodology, certainty adjustments, and other methods shown to have the promise to largely eliminate or at a minimum reduce hypothetical biases (e.g., Champ et al. 1997, 2004; Cummings and Taylor 1999; Ready et al. 2010). Recently published findings of Ready et al. (2010, p. 363), for example, show that contemporary certainly adjustments “successfully mitigated hypothetical bias.” While the evidence regarding the ability of researchers to ameliorate hypothetical biases is mixed, there is no evidence that such biases are unavoidable. To the contrary, as noted recently by Murphy et al. (2010), there is a “small but growing literature that suggesting the absence of hypothetical bias in aggregate in induced value settings...”

The Agency notes that many of research efforts that provide evidence for the presence of hypothetical biases are carried out in simulated markets, and hence do not reflect true assessments of criterion validity (Bateman et al. 2002). While simulated markets—experimental or otherwise—might generate unbiased WTP estimates, they are often conducted in institutional settings unfamiliar to respondents. Such settings may generate novel or unusual behavior, confusion, distrust, or other behaviors that may lead to misstatements of true, underlying WTP. For such reasons Bateman et al. (2002) argue that comparisons involving WTP estimates obtained through simulated or experimental markets do not qualify as true tests of criterion validity (i.e., they do not provide unambiguous measures of true willingness to pay in actual markets). The most common alternative—the use of individual or group donation mechanisms (e.g., Foster et al. 1997; Seip and Strand 1992)—suffers from confounding effects of free-riding and the associated lack of incentive compatibility (Bateman et al. 2002; Shläpfer et al. 2004) As a result, as stated by Murphy and Stevens (2004, p. 185), one cannot unambiguously claim that values estimated using laboratory experiments are any more representative of true willingness to pay than values estimates using hypothetical markets.

While critics of stated preference methods often claim insurmountable divergences between stated and real market behavior, evidence from various comparisons of actual versus hypothetical behavior contradicts the claim that hypothetical bias is unavoidable, particularly in the aggregate. For example, as conceded in the comment to the Agency, numerous comparisons of hypothetical versus binding referenda find close correspondence between choices in stated preference surveys and voting behavior in binding, consequential referenda (Bateman et al.2002, p. 318; Schläpfer et al. 2004; Johnston 2006; Vossler et al. 2003). Hence, in these true tests of criterion validity, responses in hypothetical and real contexts comport quite closely, suggesting that stated preference surveys can provide useful indications of that which individuals “are willing to forgo to enjoy a particular benefit” (OMB Circular A-4).15

After conceding that many comparisons of hypothetical and binding referenda have not shown evidence of hypothetical bias, the Commenter goes on to claim that these findings are not applicable to the Agency’s proposed survey efforts, largely due to the fact that the Agency’s survey provides “substantial information to the respondents” (p. 29). These arguments, however, are speculative and are not supported by systematic findings in the published literature; the Agency is aware of no published literature which suggests a systematic linkage between information provision and hypothetical bias of the type claimed within the comment. Moreover, the comment fails to mention that the survey has been explicitly designed to maximize the consequentiality and salience of choice experiment questions, through direct linkages to proposed Agency regulatory efforts. Hence, according to the criteria proposed by the comment, there are numerous aspects of the proposed survey that should in fact decrease the potential for hypothetical biases (e.g., salience).

The Agency, however, recognizes that hypothetical bias is an important consideration in stated preference design, and takes this concern very seriously. Because of this, extensive survey testing in focus groups and cognitive interviews (cf. Mitchell and Carson 1989; Desvousges and Smith 1988; Desvousges et al. 1984; Johnston et al. 1995; Powe 2007) was conducted to assess the extent to which such biases might be prevalent. Evidence from this extensive qualitative testing provided little evidence of hypothetical bias. For example, focus group and interview transcripts show that, when asked explicitly, respondents almost universally indicated that their answers to choice questions in the survey instrument would be identical if the same questions were encountered in a binding referendum. Moreover, the survey explicitlyincorporates elements to enable mitigation of any remaining hypothetical bias, such as a certainty followup questions (Ready et al. 2010).

The Agency recognizes the possibility of hypothetical bias—and indeed takes this possibility very seriously. As noted in the ICR, the Agency will follow the published literature in designing mitigation strategies to eliminate or at a minimum reduce the potential for hypothetical bias.

Section V: Assessment of the Need for the Proposed EPA Survey

V.A: Non-Use Benefits Not Likely to Be Significant in This Case

In this section, the commenter argues that the review of the both existing literature and relevant economic theory indicates that none of the rationales for including non-use values in the benefits analysis of the316(b) regulation offered by EPA are particularly compelling. The commenter further states that “nonuse values are expected to be significant only when the resource in question is unique and will not recover for an extended period of time.”

As noted above, EPA disagrees with the commenter that non-use values for fish protection are likely to be insignificant and notes that the economic literature advises against the use of ad hoc, expert assumptions regarding the magnitude of non-use values—particularly assumptions that non-use values are trivial (Bateman et al. 2002). Based on the guidance from economic literature (e.g., Bateman 2002), EPA believes that empirical analysis should be used to determine the magnitude of non-use values in the 316(b) regulation for existing facilities. Furthermore, results from the focus groups and cognitive interviews indicated that some individuals hold substantial non-use values for protecting all fish species including forage fish. See EPA’s response to Section II for more detail.16

In addition, the commenter argues that existence motives for non-use value for fish are very limited. In particular, “whether the forage fish, which are primarily consumed by other large fish would be among the fish population that people care about, is subject to debate at best.” The Agency notes that participants in the EPA focus groups and interviews cited a variety of motivations for preventing fish losses, including the satisfaction of knowing that the fish exist, the desire to bequeath healthy fish populations to future generations, and the desire to protect the functioning of aquatic ecosystems. The Agency further notes that the referendum format of the survey allows respondents who do not hold non-use values for fish to choose policy options that reflect their preferences; respondents need not value the existence of fish, nor do they need to be able to separate their use and non-use values.

V.B: The Proposed EPA Survey Cannot Isolate Non-Use Values

In this section, the commenter argues that the proposed EPA Survey cannot isolate non-use values. EPA points out that the purpose of the survey is to measure total WTP for the changes discussed in the survey—including both use and non-use values. When the results of the survey are used to estimate benefits in the context of the 316(b) regulation, EPA will approximate non-use values associated with the policy changes based on total WTP of non-users—a method characterized as “...one of the more defensible means of distinguishing use and nonuse values in a stated preference context” (Johnston et al. 2005). A primary advantage of this approach is the general conceptual acceptance of nonuser values, even among those highly critical of surveys that attempt to isolate nonuse values among the general population (Cummings and Harrison 1995). .

Additional comments in this section suggest additional confusion among the commenter with regard to the appropriate measurement of values in the 316(b) policy case. For example, the commenter states “even a respondent who does not use resource directly may reveal preferences for large fish populations and sustainable commercial fisheries because they like to eat fish” and that these respondents would be incorrectly included as nonuse. As noted above, however, it is perfectly valid for nonusers to maintain nonuse values for recreational and commercial fish that are in addition to the use values that might be realized by users. That is fact is not noted by the commenter suggests, again, a fundamental flaw in reasoning and understanding of accepted conceptual models for benefit cost analysis (Boardman et al. 2001).

V.C: There exist Reliable Methods for Measuring the Value of Forage Fish and Unharvested Commercial

and Recreational Fish

This section of the Comment argues that: (1) “unharvested recreational benefits are included within the category of non-market (direct use) benefits” and (2) “forage fish benefits are included within the category of indirect (use) benefits.” Therefore, there is no need for EPA to develop a proposed survey to include these benefits in its evaluation 316(b) alternatives. These statements show the commenter’s confusion about the multiple ways in which commercially and recreationally valuable fish may influence the utility of individuals. Specifically, it fails to recognize that there are two legitimate sources of economic value related to these fish: use benefits and non-use benefits. As noted in the Comment, commercial and recreational anglers benefit from catching these species. However, only a small percentage of the recreationally and commercially valuable fish whose losses would be prevented by the 316(b) regulation will be caught by anglers. The remaining fish would enter individuals’ utility functions by providing non-use benefits—through their existence—that have little or no relationship to the role of these species in recreational and commercial fisheries. EPA emphasizes the well-known tenet of17

economic theory which states that benefits or costs from multiple direct (primary) effects on markets should be counted within a benefit-cost analysis; this is not double counting, and is indeed appropriate (Just et al. 1982). Comprehensive estimates of total resource value should include both use and non-use values, such that the resulting total value estimates may be compared to total social cost . “Non-use values, like use values, have their basis in the theory of individual preferences and the measurement of welfare changes. According to theory, use values and non-use values are additive” (Freeman, 2003, p. 154).

Both the market valuation and revealed preference methods used to estimate benefits of improved commercial and recreational harvest allow EPA to estimate direct use values only. These methods are not suitable for estimating total values (use and nonuse) of harvested species and therefore would underestimate total value (Freeman 2003). Both recreational and commercial and forage fish attributes may enter individuals’ utility functions directly, providing non-use benefits that bear little or no relationship to (and are not subsumed by) use benefits generated by recreational and commercial fish. EPA refers to Freeman (2003), who provides a clear distinction between use benefits (indirect and direct)

and true non-use benefits.

Finally, EPA re-iterates that nearly all (98 percent) of impingement and entrainment losses at CWIS consist of either forage species, or non-landed recreational and commercial species that do not have direct uses or, as a result, direct use values. Therefore, EPA believes that (1) only a small fraction of potential benefits associated with the 316(b) regulation is quantified in EPA’s benefit cost analysis of the proposed 316(b) regulation for the Phase III facilities and (2) developing comprehensive quantified benefit estimates for the section 316(b) regulation requires consideration of non-use values.

Appendix 1: Detailed Assessment of Survey Design Elements in EPA’s Proposed Survey

This section of the comment includes a variety of claims regarding the validity of the survey instrument. As a precursor to specific responses below, the Agency notes that almost all of these issues have been subjected to extensive testing in focus groups. Where problems were found, the survey has been revised accordingly. Hence, many of the survey design elements noted below have already been removed or changed. In many other instances, however, focus groups showed little cause for concern, suggesting that many of the speculative claims below have little basis in fact. Specific responses are as follows:

1. Summary of Comment: “The cover page titles the survey as Fish and Aquatic Habitat. This is likely to cause the respondent to think about all fish and all habitats that support fish. ... At an absolute minimum, EPA should add questions to assess the extent that people think the fish impacted by I&E are more likely to be commercial and recreational fish. The Agency should also add probes on the likelihood that people think overfishing would also be limited by the policy and that habitat would be improved in meaningful ways.” Agency Response: These issues were probed extensively in focus groups. As noted in responses above, focus groups showed little evidence of the type of broad confusion claimed by this comment. In contrast, focus groups often included respondent-initiated discussions of such issues as the specific technologies used to mitigate entrainment and impingement, the graphics used to communicate entrainment and impingement dynamics, and other issues which demonstrated relatively clear understanding of the policy issues at hand. There was no evidence that respondents thought that the proposed policies would limit overfishing. Cognitive interviews and focus groups also probed reactions to each page of the survey, and no concerns were raised about the title page. The revised survey also18

includes numerous debriefing questions to assess the motivations for choice experiment

responses.

2. Summary of Comment: The second comment in this section is related to a claim that the survey design provides deceptive information on the policy context and scope of the resource that would
be affected. Agency Response: See the Agencies response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

3. Summary of Comment: “the map is very hard to read, making it difficult for respondents to have any clear idea of what water bodies are likely to be impacted by the rule.” Agency Response: The interpretation and clarity of the map was probed extensively in focus groups and was changed as a result of focus group responses (Besedin and Stapler 2010). The specific map in question is no longer present in the revised survey instrument. The revised map is designed to clearly identify facilities using cooling water intake structures, as suggested in this comment.

4. Summary of Comment: “What Kinds of Fish Are Affected?” This discussion is very confusing. Agency Response: See the Agencies response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here. These issues were probed extensively in focus groups to ensure a sufficient match between respondents’ understanding of the policy context and the effects of entrainment and impingement as characterized by Agency data. This section was also revised significantly as a result of focus groups, to minimize any confusion that occurred.

5. Summary of Comment: Question 1. Significant clarification is needed here. Agency Response: This question is no longer present in the survey instrument.

6. Summary of Comment: The number of fish lost should not be given out of context. (This refers to page entitled “ How Many Fish Are Affected? A Northeast Coastal Example” in the original survey.) Agency Response: This entire page has been extensively reworked to provide improved context on the magnitude of fish losses, based on the input of focus groups. Almost none of the original text has been retained in the revised page in question. See also the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

7. Summary of Comment: What does this fish loss really mean? Is the health and sustainability of the fish stocks affected? (This refers to page entitled “ How Many Fish Are Affected? A Northeast Coastal Example” in the original survey.) Agency Response: This entire page and the survey as a whole has been extensively reworked to provide improved context on the magnitude and impact of fish losses, based on the input of focus groups. Almost none of the original text has been retained in the revised page in question. See also the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

8. Summary of Comment: The flounder example is deceptive. (This refers to page entitled “ How Many Fish Are Affected? A Northeast Coastal Example” in the original survey.) Agency Response: The flounder example has been removed from the revised survey.

9. Summary of Comment: There is not enough information provided about the proposals for people

t o d et e rmi n e wh eth er o r no t th e p o l i c y i s w o r th i t

. − H o w m

u c h do th e s e mo di f

i c at i o n s c o s t ? Agency Response: This comment confuses the cost of a policy with the benefit of a policy. It is well-established within economic theory and practice that the cost of a commodity is entirely19

unrelated to the potential benefit provided by that commodity (Just et al. 2004). This is the reason that stated preference surveys in the literature omit information on commodity cost when evaluating WTP. The Agency has followed this standard practice here—as clearly supported by economic theory.

10. Summary of Comment: Not mentioned anywhere in this survey is the fact that there could be other environmental policies that would provide larger benefits to fish than regulating the cooling systems of the plants and potentially at a lower social cost Agency Response: The purpose of the proposed stated preference survey is stated clearly in the ICR and supporting statements: the survey instrument “will be used to elicit values for changes in fish mortality and the associated ecological gains from a section 316(b) regulation covering Phase II and Phase III facilities.” The purpose of the survey is not to estimate benefits of “other environmental policies that would provide larger benefits to fish.” While the Agency agrees that there are other possible policies that might benefit fish, these are not relevant in the present ICR or associated survey instrument.

11. Summary of Comment: “Would You Vote for Policies to Help Protect Fish and Fish Habitat?” Again the title implies that the benefits of the policies are more extensive than just fish loss. Habitat will not be affected by the proposed policy. Agency Response: The capacity of habitats affected by CWIS operations to support aquatic life is affected by entrainment and impingement effects. Because fish habitat is influenced by entrainment and impingement effects, the proposed policies do influence fish habitat. Moreover, there is no evidence from focus groups that respondents broadly misunderstood the benefits of the proposed policies. See also the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

12. Summary of Comment: Question 2. These are very broad and significant categories of issues that the respondents are asked to rate Agency Response: This question is no longer present in the survey instrument.

13. Summary of Comment: “ “This Survey is Similar to a Public Vote”: There does not seem to any evidence to support [the attribute levels included in the survey].” Agency Response: The commercial fish sustainability score is no longer included in the Agency’s survey. Hence, comments regarding this score are no longer relevant. With regard to attributes associated with fish populations, the comment claims that “EPA needs to provide some evidence that fish populations are affected to include this in the survey as part of the scenario.” The same claim is made for the condition of aquatic ecosystems. The Agency disagrees with this argument, and notes that the proposed comment suggests a lack of understanding of the capacity of choice experiments to isolate the marginal effects of different attributes on stated choices and hence on WTP—a methodology clearly described by numerous sources (e.g., Louviere et al. 2000). The choice experiment survey and experimental design are explicitly designed such that the model can incorporate a wide-range of potential outcomes, including zero changes in fish populations and ecological condition. That is, the choice scenarios (or profiles) explicitly evaluate respondents choices and implied values for cases in which changes in ecological outcomes (including changes in fish populations) are both large and small. Unlike older forms of stated preference analysis in which WTP is contingent upon a single specification of ecological effects, choice experiments solicit responses under a large number of possible attribute configurations. This will allow the Agency, for example, to estimate WTP for policies in which the broader ecological effects of I&E reductions—beyond direct influences on the number of age-1 equivalent fish lost—are zero. Hence, contrary to the claim of this comment, it is not true that the Agency needs to provide prior evidence “that fish populations are affected to include this in the survey as part of the scenario.”20

14. Summary of Comment: “1. Commercial Fish Sustainability Score”: The idea of using a commercial fishing sustainability score is inappropriate. Agency Response: The commercial fish sustainability score is no longer included in the Agency’s survey.

15. Summary of Comment: “2. Fish Population Score”: Using fish population numbers without any scientific data to support it is inappropriate. Agency Response: The Agency notes that the ranges for this attribute are grounded in the best available data. Responses in #13 above also apply here.

16. Summary of Comment: “The Aquatic Ecosystems Score”: This index is completely unrelated to
fish loss from intakes. Agency Response: The score is determined by many factors including water temperature, the presence and abundance of aquatic species, including species affected by cooling water intakes, and habitat conditions. Responses in #13 above also apply here.

17. Summary of Comment: Only category one and five can truly be linked to the policy the respondents are considering in the scenarios: fish loss prevention and cost to household. Including the effects on the sustainability of commercial fishing and fish populations and aquatic ecosystem misleads the respondent to think that such impacts are on the same order of magnitude as potential changes in the numbers of forage fish. Agency Response: There is no information in this section that implies, in any way, that “such impacts are on the same order of magnitude as potential changes in the numbers of forage fish.” Information provided in the choice questions clearly demonstrates that the effects on attributes may differ. Focus group responses clearly indicated that respondents understood these differences. The following quote is typical, and shows the respondent clearly distinguishing between the magnitude of impacts on fish losses and impacts on other attributes: “[...] when I compared A and B, they’re virtually the same statistics except on fish saved per year, where there were a lot more fish saved” (#5-6). See also the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

18. Summary of Comment: All of the categories the respondents are asked consider appear to be correlated. Therefore, the respondent would find it hard to not rank the first four at the same level. Agency Response: Comments related to correlation of ecological attributes are addressed in the Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes. Focus groups show little evidence of the type of difficulties claimed within this comment.

19. Summary of Comment: The relationship between commercial fishing sustainability and fish populations does not make sense. Agency Response: The commercial fish sustainability score is no longer included in the Agency’s survey.

20. Summary of Comment: “As You Vote Please Remember”: The third bullet is confusing: “Depending on the type of technology required and other factors, effects on fish and ecosystems may be different – even if the annual reduction in fish losses is similar.” Agency Response: The meaning of this statement is clear and self-evident. Focus groups showed no evidence of
confusion related to this statement.

21. Summary of Comment: Attribute levels in questions 4, 5 and 6 are likely to be confusing to respondents. Agency Response: Focus group responses do not support the Commenter’s claims. The Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes address these comments. Moreover, the majority of focus group participants indicated that they were confident with their responses to questions 4, 5 and 6—again suggesting that the claims of respondent confusion here are overstated. For example, participants made statements21

such as: “Yeah, I’m pretty much confident” (#3-8), and “If I believed that it was gonna affect regulations, I think I would have voted the same exact way.” (#7-9).

22. Summary of Comment: Comments on question #8 and on National vs. Northeast Policies Agency Response: This material is no longer included in the revised survey.

Appendix 2: The Proposed EPA Survey Failed to Adequately Respond to Peer Reviewer
Comments

As a precursor to later comments, the Agency highlights that the survey proposed in the ICR, and tested in focus groups, bears little direct resemblance to the survey that was peer reviewed in 2005. Hence, many of the peer review comments are no longer relevant. The Agency, however, takes all peer review comments very seriously, and has accounted for these comments in all survey revisions and in development of the present stated preference survey.

Appendix 2.A. Background on Peer Review Process

This section does not address the specific survey materials or information collection approaches described in the Agency’s Information Collection Request (ICR). The Agency has no comment on this section.

Appendix 2. B. Evaluation of the Current EPA Proposed Survey Based on Peer Review Comments

Most of the introductory paragraphs in this deal with revisions that the Agency has already made in direct response to peer review comments. Claims of remaining limitations are almost universally related to issues that have been comprehensively addressed above. These are as follows:

1. Summary of Comment: Removal of photos from the survey is a step in the right direction for reducing bias; however, it does not go far enough as the survey continues to lead the respondent to believe that I&E has farther reaching effects than science can support Agency Response: This issue has been dealt with comprehensively above. See the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

2. Summary of Comment: The overall tenor of the information is one in which uncertainty is minimized relative to the actual uncertainty associated with the effects. Agency Response: The revised survey is explicit about the lack of certainty, and this was clearly acknowledged by focus group participants. Participants understood that the ecological changes described in the survey were uncertain, and most participants were comfortable making decisions in the presence of this uncertainty. Their responses indicated that they understood this uncertainty based on the information presented in the introductory material and considered it when evaluating policy options. For example, one participant referred to Page 4, which addresses scientific uncertainty

as, “That one kind of the disclaimer page” (#7-19). Some participants also expressed that they weren’t surprised at this uncertainty, with statements such as “I wasn’t surprised at that” (#4/6- 7). Multiple participants expressed that they perceived the values as “best guess” : “My guess is that it did come from studies but I have a healthy dose of skepticism about the accuracy of it. I don’t think it’s been in any way skewed purposefully, but I know that this is a best guess, reasonable guess perhaps” (#5-10), “it shows me that they are being honest for the most part. You know, you can't obviously be accurate on everything, but this is a kind of a best guess” (#4/1- 3), “[...] They had more numbers on the commercial fish population. The rest was more of a guesstimate” (#5-15), and “you don’t know the exact number and nobody knows” (#4/3-4).22

Overall, participants felt comfortable making decisions in the presence of uncertainty, but some participants had differing opinions on how it impacted their choices: “No it didn’t – it didn’t either way” (#4/4-7) and “I’m assuming if you’re a scientist and you are doing studies, I’m taking you at your word. I still don’t think that anyone’s capable of complete accuracy in this mater – I guess it did affect my answers because I think that taking them that they can do it pretty decently” (#4/5-3). Neither the content of the survey nor focus group responses support the claim that “uncertainty is minimized.”

3. Summary of Comment: “The addition of habitat quality to the attributes in the choice sets, in an attempt to address that issue in the mind of respondents...” Agency Response: There is no “habitat quality” index or attribute in the proposed survey instrument.

4. Summary of Comment: Finally, we do not think that EPA has addressed the peer reviewers’ concerns about the definition of baseline. Based on our review, baseline continues to be poorly
defined. Agency Response: Baselines are clearly defined for each of the attributes on page 7 in the revised survey, in both ecological and common language terms. The Agency notes that these attributes are distinct from those in the original survey version reviewed for this comment, and that the baselines are different. Therefore, this comment is not directly relevant to the current

survey.

Responses to Specific Peer Review Comments in Appendix 2, Table 2

1. Summary of Comment: “There is no baseline or baseline is poorly defined.” Agency Response: All of the attributes have been changed significantly from the original survey version, including a more clear definition of all baselines. These include significant changes to the attributes as defined in the version of the survey reviewed by the Commenter. For example, the commercial fish sustainability score is no longer included in the Agency’s survey, so related comments are no longer relevant. The baseline for fish losses is clearly and unambiguously stated on page 7 of the revised survey. The fish losses attribute is described as: “A score between 0 and 100 showing the reduction in young fish lost because of the new policy. A score of 100 means that no fish are lost in cooling water intakes (all fish are saved). The current score in Northeast waters is 0; this represents the status quo with no new policies.” The commercial fishing attribute has been revised entirely, with a different definition and new baseline. Hence the comment on the originally-defined attribute is no longer relevant.

2. Summary of Comment: “The commodity is defined simply as fish.” Agency Response: The commodity is no longer simply defined as fish. The revised survey incorporates four ecological attributes, communicating effects on commercial and recreational fish populations, all fish populations, and aquatic ecological condition. This directly responds to the peer review comment of Krupnick in this section.

3. Summary of Comment: “Uncertainty is not handled properly.” Agency Response: See the Agency’s response to Appendix 2. B. Evaluation of the Current EPA Proposed Survey Based on Peer Review Comments, comment #2. This comment details the Agency’s treatment of uncertainty in the revised survey and related focus group responses. Beyond this, the choice experiment survey and experimental design are explicitly designed such that the model can incorporate a wide-range of potential outcomes, including zero changes in population and harvest. In this way, the model can account for uncertainty, allowing sensitivity analysis of cases in which I&E reductions result in zero, small, or large changes in various attributes.23

4. Summary of Comment: “The attributes are interdependent in complex ways.” Agency Response: The Agency’s responses to IV.D. Biases Related to Inherent Collinearity in the Underlying Attributes address these comments. See also the related Response to Appendix 1: Detailed Assessment of Survey Design Elements in EPA’s Proposed Survey, comment #21.

5. Summary of Comment: “The survey may create bias based on the sponsor..” Agency Response: The Commenter argues here that the identification of EPA as the sponsor of the survey somehow contributes to an inappropriate bias. The Agency rejects the suggestion that this information should be withheld from respondents, and notes that there is nothing in the literature that suggests that such information should be omitted. To the contrary, the literature clearly emphasizes the importance of mechanisms that reinforce the consequentiality of stated preference surveys and survey responses (Carson and Groves 2007). The emphasis on a specific government agency within survey materials is clearly consistent with Carson and Groves (2007) arguments concerning the important role of consequentiality in stated preference surveys, defined as a case in which “survey results are seen by the agent as potentially influencing agency action” and in which the agent treats the survey as “an opportunity to influence those actions.” Recent work by Johnston (2006) also clearly emphasizes the potential benefits of consequentiality in stated preference survey instruments, and recommends identifying government sponsors of policy issues addressed by survey instruments as a means to eliminate hypothetical bias. Moreover, the Agency has found nothing in the literature to suggest that identifying the government agency associated with particular policy actions will bias WTP responses.

6. Summary of Comment: “Payment vehicle may cause rejection.” Agency Response: the cost of hypothetical programs are clearly described following standard practice in stated preference surveys. Given this—and the results of focus groups—the Agency can find no evidence that the payment vehicle will lead to scenario rejection. The current format of the payment vehicle is the result of focus groups on the original version of the 316(b) survey instrument in 2005, with additional testing in the more recent 2010 focus groups. These focus groups and discussions with both OMB and NCEE in 2005 ultimately led to the current description of the payment vehicle. Focus groups showed no difficulty understanding the format of the payment vehicle, and respondents clearly understood the cost as an ongoing cost. The Agency also notes that a similar, ongoing cost of living payment vehicle has been used by prior EPA-sponsored research, with no apparent difficulties (Magat et al. 2000). Participants in 2010 focus groups clearly understood that selecting Options A or B would result in increased costs to their household. They understood that costs to industry would be passed on to consumers, with one participant stating, “So my thought process is that the household stuff is being passed down to consumers from the cost of the added regulation they have” (#3-4). The draft survey was also revised to improve clarity regarding how costs would be passed to consumers by adding a clause on Page 1 stating that goods and services with costs would include “electricity and common household products”. The definition on Page 7 was also revised to include “services” purchased by consumers. These revisions greatly reduced the potential for respondent confusion when evaluating policy options. Respondents also appeared to understand the ongoing, monthly nature of the payments. Despite the Commenter’s suggestions to the contrary, there is little evidence from focus groups and interviews that respondents misunderstood the payment vehicle, or that there were any instances of payment vehicle rejections. A few respondents expressed that companies should be bearing the costs, rather than consumers. For example, a participant stated, “ Instead of passing these charges to us, why aren’t they passing them to these companies [...] that are responsible ” (#2-4). However, there was a general understanding among most participants that “ [...] ultimately it’s all us consumers that ultimately pay for everything ” (#2-4).24

7. Summary of Comment: “Wording of payment vehicle could be improved.” Agency Response: See response to comment #6 directly above.

8. Summary of Comment: “Wording of payment vehicle could be improved.” Agency Response: See response to comment #6 directly above.

9. Summary of Comment: “Dollar amounts are too small.” Agency Response: Focus group responses show that most participants considered their budget when choosing between specific policy options, and that the stated costs were of sufficient magnitude to be relevant. For example,
some of them stated “So, I don’t go out to eat twice (#5-12)”, “[...] if I was working at my old job making the money that I was making, I would probably have chose a higher amount” (#4/5- 12), and “the cost per month came out to only $3 per month, which I feel is very affordable” (#2- 6). Some participants stated that household cost was the most important factor in their decision: “Yeah. Absolutely” (#4/4-8). Some respondents didn’t view the household cost amounts under the policy options as significant relative to their personal budget constraints: “[...] $2 per month doesn’t seem like a whole lot of money. So, you almost feel guilty like 2 bucks a month, I’ll save some fish” (#7-4). Other participants indicated that they would have to think harder about budget trade-off if the amounts were higher. For example, one respondent stated, “If you put a zero behind one of the numbers, I’d have to think about it very harder than, you know, 24 dollars a year” (#5-12). Many other respondents, however, indicated that the payment vehicle amounts were sufficient to change their responses: “See, I’m unemployed right now, so I’m looking at every little dollar, where it’s going. That’s what affected me” (#6-5).

10. Summary of Comment: “The graphics could lead to bias.” Agency Response: Since the peer review, all of the graphics in the original survey have been changed. Given that this comment regarded graphics that are no longer in the survey instrument, it is no longer relevant. All new graphics were tested extensively in focus groups. Participants endorsed the use of graphics in the survey booklet. Most felt that the graphics helped them to visualize how fish are entrained and impinged, technological solutions, facilities locations, and ecosystem effects. For example, some respondents stated: “I mean, the filter thing, where you can see X number of fish on one side and then only the little ones are getting through, it kind of tells you what’s going on ”(#1-5), “ After reading the previous page and then reading this picture clicked [...] I got it ” (#7-10), and “I think it was good to visualize the problem in the graphic ” (#7-10). Participants also felt that the graphics improved their understanding of these issues and tended to think of them as “ providing information ” (#6-14), rather than pushing them to answer a certain way. Many respondents indicated that they understood the meaning of the graphics: “ I think they were easy to follow. It was pretty straightforward” (#1-10) and “It looks clear” (#5-5). However, some respondents provided feedback regarding aspects of the various graphics which they found confusing and on how they might be improved. Participants’ comments during the focus groups were used to refine the graphics within the draft survey in order to improve clarity and reduce the potential for respondent confusion. Participants’ comments also indicated that they had little emotional reaction to the image on the cover of the survey and that it did not influence their survey responses: “It’s a nice picture, but it didn’t do anything, you know, wow, spectacular ” (#4/5-3), “I noticed it. I didn’t – as far – kind of gave me an idea what the topic was going to be about ” (#4/1- 2), and “ ... looking at it without knowing any of this inside, I think my impression was, it’s colorful and kind of blurry ” (#4/8-10).

11. Summary of Comment: “Region of concern is unclear..” Agency Response: The survey now deals only with one region. This region is clearly identified on the map.25

12. Summary of Comment: “Protect fish stocks is a biased statement.” Agency Response: As noted by the Commenter, this statement has been removed as requested by the peer review.

13. Summary of Comment: “The Indication of a declining baseline” may influence responses. Agency Response: As noted by the Commenter, this statement has been removed as requested by the peer review. The survey no longer mentions a declining baseline.

14. Summary of Comment: “Language regarding lost fish is vague..” Agency Response: All of the language in question has been revised in the survey instrument. See the Agency’s response to IV.B. Bias Due to Lack of Match Between Survey Information and the Regulatory Situation; the same responses apply here.

15. Summary of Comment: “Table on Slide 19 is confusing.” Agency Response: This graphic is no longer in the survey.

16. Summary of Comment: “Pie charts are not very helpful when nearly all of the chart is one color.” Agency Response: The charts in the survey have all been revised, and use two colors as requested.

17. Summary of Comment: ““Technologies” is unclear.” Agency Response: The survey now clearly identifies and describes the technologies in question on page 5. This is entirely new material that has been added to the survey to address this peer review comment.

18. Summary of Comment: “Technologies are already in place needs to be explained” Agency Response: The survey now clearly addresses this issue on page 5. This is new material that has been added to the survey to address this peer review comment.

19. Summary of Comment: “Status Quo needs to be a separate choice” Agency Response: The Agency agrees. The status quo is a separate choice in the revised survey.

References

Arrow, K. , R. Solow, E. Leamer, P. Portney, R. Rander, and H. Schuman. 1993. “Report of the NOAA Panel on Contingent Valuation.” Federal Register 58(10): 4602-4614.

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S. Mourato, E. Ozdemiroglu, D.W. Pearce, R. Sugden, and J. Swanson. 2002. Economic Valuation with Stated Preference Surveys: A Manual. Northampton, MA: Edward Elgar.

Bennett, J. and R. Blamey, eds. 2001. The Choice Modeling Approach to Environmental Valuation. Northampton, MA: Edward Elgar.

Carson, R.T. and T. Groves. 2007. Incentive and Informational Properties of Preference Questions. Environmental and Resource Economics 37(1):181–210.

Carson, R.T., N.E. Flores, and R.C. Mitchell, The Theory and Measurement of Passive-Use Value, in Valuing Environmental Preferences: Theory and Practice of the Contingent Valuation Method in26

the US, EU, and Developing Countries, edited by I.J. Bateman and K.G. Willis, Oxford University Press, Oxford, UK, 1999.

Champ, P.A., and R.C. Bishop. 2001. “Donation payment mechanisms and contingent valuation: An empirical study of hypothetical bias.” Environmental and Resource Economics 19:383–402.

Champ, P.A., R.C. Bishop, T.C. Brown, and D.W. McCollum. 1997. Using donation mechanisms to value non-use benefits from public goods. Journal of Environmental Economics and Management 33(2):151–162.

Champ, P.A., R. Moore, R.C. Bishop. 2004. “Hypothetical Bias: The Mitigating Effects of Certainty Questions and Cheap Talk.” Selected Paper prepared for presentation at the American Agricultural Economics Association Annual Meeting, Denver, Colorado.

Collins, J.P. and C.A. Vossler. 2009. Incentive Compatibility Tests of Choice Experiment Value Elicitation Questions. Journal of Environmental Economics and Management 58(2): 226-235.

Cummings, R. and L. Taylor. 1999. Unbiased Value Estimates for Environmental Goods: A Cheap Talk Design for the Contingent Valuation Method. American Economic Review 89:649-665.

Desvousges, W.H. and V.K Smith. 1988. Focus Groups and Risk Communication: the Science of Listening to Data. Risk Analysis 8:479-484.

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in Designing a contingent Valuation Survey to Measure the Benefits of Hazardous Waste Management Regulations. Research Triangle Institute: Research Triangle Park, NC.

Dillman, D.A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley

and Sons.

Foster, V., I. J. Bateman, and D. Harley. 1997. Real and Hypothetical Willingness to Pay for Environmental Preservation: A Non-experimental Comparison. Journal of Agricultural Economics 48(2):123-138.

Giraud, K.L., J.B. Loomis and R.L. Johnson. 1999. Internal and external scope in willingness-to-pay estimates for threatened and endangered wildlife. Journal of Environmental Management 56:

221-229.

Hanemann, W.M. 1994. Valuing the Environment Through Contingent Valuation. Journal of Environmental Perspectives 8(4): 19-43.

Hanemann, W.M. and B. Kanninen. 1999. “The Statistical Analysis of Discrete-Response CV Data.” In Valuing Environmental Preferences: Theory and Practice of the Contingent Valuation Method in the US, EU, and Developing Countries. Edited by I.J. Bateman and K.G. Willis, Oxford University Press, Oxford, UK.

Heberlein, T.A., M.A. Wilson, R.C. Bishop, and N.C. Schaeffer. 2005. Rethinking the Scope Test as a Criterion in Contingent Valuation. Journal of Environmental Economics and Management 50(1):

1-22.27

Johnston, R.J. 2006. Is Hypothetical Bias Universal? Validating Contingent Valuation Responses Using a Binding Public Referendum. Journal of Environmental Economics and Management 52(1): 469-

481.

Johnston, R.J., T.F. Weaver, L.A. Smith and S.K. Swallow. 1995. Contingent Valuation Focus Groups: Insights From Ethnographic Interview Techniques. Agricultural and Resource Economics Review 24(1): 56-69.

Kemp, M.A. and C. Maxwell. 1993. Exploring a Budget Context for Contingent Valuation Estimates, in Hausman, J.A. ed., Contingent Valuation: A Critical Assessment. New York: North Holland, pp. 217-269.

Kuhfeld, W.F. and Tobias, R.D., 2005. Large factorial designs for product engineering and marketing research applications. Technometrics 47: 132-141.

Louviere, J.J., D.A. Hensher, and J.D. Swait. 2000. Stated Preference Methods: Analysis and Application. Cambridge, UK: Cambridge University Press.

Lusk, J.L. and F.B. Norwood. 2005. Effect of Experimental Design on Choice-Based Conjoint Valuation Estimates. American Journal of Agricultural Economics 87(3):771-785.

Magat, W.A., J. Huber, K.W. Viscusi and J. Bell. 2000. An Iterative Choice Approach to Valuing Clean Lakes, Rivers, and Streams. Journal of Risk and Uncertainty 21(1): 7-43.

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation Method. Resources for the Future, Washington, D.C.

Murphy, J.J., P.G. Allen, T.H. Stevens, and D. Weatherhead. 2005. A Meta-Analysis of Hypothetical Bias in Stated Preference Valuation. Environmental and Resource Economics 30(3): 313–25.

Murphy, J.J. and T.H. Stevens. 2004. Contingent Valuation, Hypothetical Bias, and Experimental Economics. Agricultural and Resource Economics Review 33(2): 182-192.

Murphy, J. J. T.H. Stevens and L. Yadev. 2010. A Comparison of Induced Value and Home-Grown Value Experiments to Test for Hypothetical Bias in Contingent Valuation. Environmental and Resource Economics 47(1): 111-123.

Powe, N.A. 2007. Redesigning Environmental Valuation: Mixing Methods within Stated Preference Techniques. Cheltenham, UK: Edward Elgar.

Ready, R.C., P.A. Champ and J.L. Lawton. 2010. Using Respondent Uncertainty to Mitigate Hypothetical Bias in a Stated Choice Experiment. Land Economics 86(2): 363-381.

Schläpfer, F., A. Roschewitz, and N. Hanley. 2004. Validation of Stated Preferences for Public Goods: A Comparison of Contingent Valuation Survey Response and Voting Behavior. Ecological Economics 51(1): 1-16.

Seip, K. and Strand, J. 1992. Willingness to Pay for Environmental Goods in Norway: A Contingent Valuation Study with Real Payments. Environmental and Resource Economics 2: 91-106.28

U.S. EPA. 2000. Guidelines for Preparing Economic Analyses. USEPA 240-R-00-003. U.S. EPA, Office of the Administrator, Washington, DC, September 2000.

Vossler, C.A. and M.F. Evans. 2009. Bridging the Gap Between the Field and the Lab: Environmental Goods, Policy Maker Input and Consequentiality. Journal of Environmental Economics and Management 58(3): 338-345.

Vossler, C.A., J. Kerkvliet. 2003. “A Criterion Validity Test of the Contingent Valuation Method: Comparing Hypothetical and Actual Voting Behavior for a Public Referendum.” Journal of Environmental Economics and Management 45(3): 631-649.