## Environment and Resource Division

Date May 23, 2006

To Julie Hewitt, OW/OST/EAD

From Elena Besedin, Abt Associates Inc

Subject Development of Willingness to Pay Survey Instrument for Section 316(b) Phase III Cooling Water Intake Structures

As required under Executive Order 1286 6, EPA conducted economic impact and cost-benefit analyses for the section 316(b) regulation for Phase III fac ilities. Cost-benefit analys is should include a comprehensive estimate of total social benefits, in cluding both use and non-use values. “Non-use values, like use values, have their basis in the theory of individual preferences and the measurement of welfare changes. According to theory, use and non-use values are additive” (Freeman, 2003). It is generally accepted in the economic literature that non-use values may be substantial in some cases. Additionally, when small per capita non-use values are held by a substa ntial fraction of the population, they can be very large in the aggregate. Therefore, failure to r ecognize such values may lead to improper inferences regarding policy benefits. As stated by Freeman ( 2003, p. 138), “[i]f non-use values are large, ignoring them in natural resource policymaking could lead to serious errors and resource misallocations.” Both EPA’s own Guidelines to for Preparing Economic Analysis and OMB’s Circular A-4, governing Regulatory Analysis, support the need to assess non- use values (U.S. EPA, 2000a; U.S. OMB, 2003).

Consideration of potential non-use values is partic ularly important in the case of the section 316(b) regulation because nearly all (97 to 98 percent) impi ngement and entrainment losses at CWIS consist of either forage species, or non-landed recreational a nd commercial species that do not have direct uses. Although individuals do not use these resources directly, they may nevertheless be affected by changes in the resource status or quality, such that they woul d be willing to pay to maintain these resources. Although economic theory clearly allows for the possi bility of significant non-use values in this case, it does not shed light on whether such values ar e large or small in the aggregate.

EPA attempted to measure non-use benefits of the 316( b) regulation in monetary terms, as suggested by EPA’s own guidance and OMB’s (U.S. EPA, 2000; U. S. OMB, 2003). Using the benefits transfer technique requires adequate empirical valuation studi es. No empirical studies were found that estimated non-use values for impacts on fish alone. Thus, EPA began designing a stated preference survey to separately estimate total value (including non-use value) of fish impacts when work on this rule began.

Stated preference surveys use carefully designed questions to elicit respondents’ willingness to pay (WTP) for particular ecological improvements, based on their responses to either discrete choice or openended questions regarding hypothe tical resource improvements or programs. Such improvements may include increased protection of aquatic habitats or speci es with particular attributes. Stated preference survey methodology is the generally accepted mean s to estimate total (including use and non-use) 

# memorandum2 resource values. Moreover, stated preference surv ey methods “are likely to offer the only feasible approaches to estimating non-use values” (Freeman, 2003, p. 154).

EPA received OMB approval in August, 2005 of an Information Collection Request to conduct focus groups for survey design (EPA ICR #2155.01). EPA has conducted a series of twelve focus groups in different geographic locations to assist in the deve lopment of this stated preference survey. Based on findings from the focus groups, EPA designed a survey using state-of-the-art valuation methods, and conducted an external peer review of the survey in strument and analysis plan, completed in February, 2006 (Versar, 2006). While peer reviewers agreed that the effort was state-of-the-art, they also provided suggestions to improve the reliability of the results; these suggestions could not be incorporated in time for today’s action. As a result, EPA was unable to monetize non-use benefits of the policy options considered in development of the final 316(b) regulation for Phase III facilities.

The remainder of this memorandum, summarizes findi ngs from focus groups, outlines EPA’s plan for the pilot study, and presents experts’ conclusions from peer review of the survey instrument.

1. Focus Groups

To assist in the development of this stated prefer ence survey, EPA conducted a series of twelve focus groups with a total of 96 respondents. These fo cus groups and cognitive in terviews were conducted under EPA ICR #2155.01, “Willingness To Pay Survey for Section 316(b) Phase III Cooling Water Intake Structures: Instrument, Pre-Test, and Implem entation.” The focus groups and interviews took place in the following locations on these dates:

1) Providence, RI; May 4, 2004 2) Providence, RI; May 11, 2004 3) North Haven, CT; May 20, 2004 4) Baltimore, MD; June 1, 2004 5) Baltimore, MD; June 10, 2004 6) Portland, OR; August 15, 2005 7) Baltimore, MD; August 23, 2005 8) Green Bay, WI; August 29, 2005 9) East Providence, RI; September 2, 2005 [12 cognitive interviews] 10) Bethesda, MD; September 7, 2005 11) Providence, RI; September 15, 2005 [8 cognitive interviews] 12) Baltimore, MD; September 23, 2005

These focus groups were conducted following standa rd, accepted practices in the stated preference literature, as outlined by Mitchell and Carson (1989), Desvousges et al. (1984), and Johnston et al. (1995). Some of the focus groups incorporated indi vidual cognitive interviews, as detailed by (Kaplowicz et al. 2004). The focus groups and cognitive interv iews allowed EPA to bette r understand the public's perceptions and attitudes concerning fishery resources, to frame and define survey questions, to pretest draft survey questions, to test for and reduce pot ential biases that may be associated with stated preference methodology, and to ensure that both resear chers and respondents have similar interpretations of survey language and scenarios. In particular, c ognitive interviews allowed in-depth exploration of the cognitive processes used by respondents to answer survey questions, without the potential for interpersonal dynamics to sway respondents’ comments (Kaplowicz et al. 2004).

Each focus group was composed of seven to nine i ndividuals that were randomly selected by the focus group facilities from panels of focus group participants maintained by each facility. Participants were3 eligible if they were eighteen years of age or older, if they were not full-time students, and if they had not participated in a survey or focus group in the last si x months. Each of the focus groups included a mix of individuals with diverse socioeconomic backgrounds , based on characteristics such as income, marital status, age, race, education, occupation, and gender. Participants were compensated for their participation in the focus groups.

The format of the focus group sessions varied. In gene ral, a moderator led a group through a discussion of specific topics to discover individuals’ attitudes and op inions. For example, discussion topics included:

• general questions about participan ts’ knowledge and opinions about fish and what role fish play in the environment,

• general questions about participants’ use of wa ter resources for recreation and their recreational experiences;

• questions about participants’ knowledge of the current health/quality of fish populations/resources in their local area;

• questions about participants’ familiarity with any pa rticular threats to fish and fish habitats from human activities;

• questions about participants’ motives (if any) for protecting fish and other aquatic organisms; and

• questions about participants’ opinions about pr ograms that would reduce fish mortality from manmade causes, the geographic scope of these pr ograms, and the long-term effects of those programs on fish populations.

In some focus groups, participants were also asked to respond to several questions from the draft survey instrument. This was followed by a debriefing sessi on. During the debriefing sessions, the moderator asked focus group participants about their reactions to the survey format and cont ent, whether the survey questions were clear, whether the background info rmation presented in the survey or introductory materials was sufficient and how participants interpre ted it, what went through participants’ minds when they read survey questions, and what their motivati ons were for responding they way they did to the survey. In Focus Groups 4 to 12, the moderator firs t read introductory materials accompanied by graphics and pictures and then handed out draft survey questions.

Each session was one to two hours long. All sessions were audio taped and transcribed. For more detailed discussion of results from the focus groups, refer to the focus group reports:

• Besedin et al, 2005. Findings from 2005 Focus Groups Conducted Under EPA ICR #2155.0 2. Memorandum to Erik Helm, EPA/OW/O ST/EAD, dated October 18, 2005.

• Besedin et al, 2004. Findings from Focus Groups . Memorandum to Erik Helm, EPA/OW/OST/EAD, dated June 30, 2004

The appendices of these two reports also provide fu ll transcripts, presentation slide shows and scripts, survey instruments, and fact sheets for each focus group and set of cognitive interviews.

2. Findings from Focus Groups

Focus groups are often described as “informal sessi ons in which a skilled moderator leads a group of individuals through a discussion of specific topics to discover their attitudes and opinions” (Desvousges et al., 1984, p. 2-1; cited in Johnston et al., 1995, p. 56). Focus groups are among the most significant qualitative research tools used in social sciences (B ateman et al., 2002); observations from focus groups or similar qualitative tools may provide information allowing one to interpret or validate previously obtained quantitative results (Responsive Management , 1992). EPA conducted 12 fo cus groups meetings4 to obtain insight into potential WTP values for reduc ing I&E at cooling water intakes and motivations underlying respondents’ stated willingness to pay (W TP) values (Desvousges et al., 1984; 1988; Mitchell and Carson, 1989; Johnston et al., 1995).

Because focus groups rarely represent a truly random sample of particular populations, it is generally considered inappropriate to subs titute focus group data for WTP estim ates derived from large-sample survey instruments (Bateman et al., 2002, p. 153) . Nonetheless, it is well known that observations from focus groups or similar qualitative tools may provide information allowing one to interpret or validate previously obtained results (Responsive Management , 1992). Pre- or post-implementation focus groups may provide supporting evidence of the validity of stated preference survey (or related benefit transfer) results and the assumptions that underlie such an alyses (e.g., Arrow et al., 1993). Table 1 provides summary statistics from 7 focus group meetings.

Based on the findings from 12 focus groups, the Agen cy found that both user and non-users of the affected aquatic resources are likely to hold values for reducing I&E mortality of the affected fish species. 74% of focus group participants cited bequest and existence values as one th e main motives for their WTP for reducing I&E mortality. Only 39% particip ants cited use motives (e.g., preserving fish as food source for people or participation in recreational fishing) as one the main motives for reducing I&E mortality.

Table 1: Summary of Focus Group Responses

by Motives for Reducing Fish Losses a

Motive Type for Answering “Yes” to Program Option Choices Number of Respondents Answering “Yes” b, c Percentage of All Program Option “Yes” Answers

Non-Use Motives 113.5 74%

Bequest Value 37 24%

Existence Value 76.5 50%

Use Motives 59 39%

Source of Food for People 33 22%

Commercial and Recreational Fishing Industry 21 14%

Personal Angler Use 5 3%

Other Factors 6.5 4%

Total “Yes” Answers to Program Options 153 100%

Total Observations 190

a Each question is regarded as one observation. Observations include focus groups for the following regions: Pacific Mountain, Northeast, and Great Lakes. The Mid-Atlantic Region is excluded from the count because relevant questions were not included in these focus groups.

b Double counting may occur if a respondent answered “yes” to multiple option choices in one question or ranked motives options; therefore, the number of answers for each choice may not sum to the total number of program option “yes” answers.

c If motive options were ranked, a weight was applied to each answer.5 Focus group participants agreed that all fish species (including forage fish) play an important role in the affected ecosystems and thus also have significant values. For example, when one respondent was asked about what fish species he was considering when he made his choice between the two stated preference (SP) options, he answered:

“I think that I am more worried about the types of fish that are not directly used by us. I think that the types of fish that are directly used by us — I th ink people will make sure there are plenty of them. Probably because of the money aspect of it. I am mo re concerned about the types of fish that are not related to us because people will think of it as someth ing that we don’t need and they will allow them to go extinct.”

(Besedin et al., 2004a, Focus Group # 4, p. 7)

Another respondent, when asked if he had specific fish species in mind when he was participating in the focus group or did he find all types of fish important, said:

“ No, I don't have any specific in mind. I think every fish will be equally important for one thing or another, whether it's for food or I guess some fish are food for other fish. Or even just to enjoy, you know. They have a right to live like anything else, you know? ”

(Besedin et al., 2005, Focus Group # 11, Interview # 5, p. 3)

Focus groups also showed that 74% of focus group participants have non-zero values for reducing I&E mortality. Based on focus group findings the mean va lue for reducing I&E mortality range from $12 to $54 per year. As pointed above, it is inappropriate to substitute focu s group data for mean WTP estimates derived from large-sample survey instruments. Ho wever, this finding suggests that WTP for reducing I&E mortality of all species is likely to be positive.

Additional detail on EPA’s findings from the focus group meetings is provided in the Focus Group Reports (Besedin et al., 2004a; Besedin et al., 2005).

3. Pilot Study

In addition to conducting focu s groups EPA has developed a detailed plan for the pilot test of the survey instrument prior to conducting the main study. The purpose of the pilot study is to assess the performance of the proposed survey instrument and slide show under various treatments. It would also allow the Agency to determine whether answers reflect expect ed neoclassical tradeoffs suggested by focus groups, or whether responses are strongly influenced by misi nterpretations of survey information, symbolic effects, cost-effectiveness heuristics, and other pattern s that are generally less conducive to appropriate welfare estimation. Findings of the pilot study w ould allow the Agency the cap acity to adjust the fullscale study for any problems that are evident.

The Agency developed a list of hypothesis tests and quantitative assessments for the pilot version of the survey instrument. These tests were designed to assess the performance of the survey instrument in important areas (e.g., sensitivity to scope for included model attributes, the statistical significance of baseline fish populations, attribute order effects, and assessments of respon se consistency and item non-6 response), and allow for adjustments to the survey —if necessary—to address any limitations or problems discovered. Other tests are designed to assess the im plications of different survey treatments on choices and model performance.

EPA provides a detailed plan for the pilot test phase in a memorandum entitled “ List of Hypothesis Tests and Quantitative Assessments for EPA 316(b) Analysis ”, February 13, 2006 (Besedin and Johnston, 2006).

4. Peer Review Results

EPA conducted an external peer review of the surv ey instrument and analysis plan, completed in February, 2006 (Versar, 2006). The goal of the Agency’s Peer Review Policy is to ensure that scientific and technical work products receive appropriate levels of critical scru tiny from scientific and technical experts as part of the overall decision making process. This survey instrument and analysis plan were reviewed by well recognized experts in the field of resource economics, including Dr. Richard C. Bishop, Dr. Kevin J. Boyle, Dr. Timothy C. Haab, Dr. James R. Kahn, and Dr. Alan J. Krupnick.

Peer reviewers were asked to comment on whether the effort used in developing and testing the proposed survey meet current standards for stated preference surveys reflected in the literature and whether the survey likely provide useful insights into prefer ences, tradeoffs, and willingness to pay associated with the described policies. The review ers were also asked to comment on specific survey components (e.g., whether the survey sufficiently and appropriately ch aracterize the policy options under consideration) and to provide suggestions for improvements to the survey instrument and the associated model. For detail on specific questions to peer reviewers see EPA’s charge to peer reviewers in the peer review report for willingness to pay survey instrument for §316(b) Ph ase III cooling water intake structures (Versar, 2006).

The majority of reviewers noted that “those designing the study display excellent knowledge of the scientific literature on stated-preference methods” and th at “the overall concept and design of the survey is sound and with minor corrections will represent a solid piece of applied valuation.” (Versar, 2006, p. 13 and p.23) Four of the five reviewers were confid ent that “the survey results will provide useful information” to decision makers (Versar, 2006, p.13).

Peer reviewers provided a number of suggestions to improve the reliability of the results, including revisions to some debriefing questions (e.g., incor porating additional response options) and slide show materials. EPA notes that the Agency carefully cons idered suggested improvements to the slide show and the survey instrument and began incorporating revision s suggested by R. Bishop, J. Kahn, K. Boyle, and T. Haab. Although the reviewers characterized their suggestions for improvement as “minor corrections” to the survey instrument that “can all be implemente d in a short time” they urged EPA to add between six to 12 months to the schedule to ensure “thorough eval uation of the results of the pilot, a more careful statistical analysis of the main survey results, and a lot more thought about the follow-up survey.” (Peer review Report, p.23). As a result, EPA was unable to implement the stated pref erence survey instrument within the regulatory schedule and to monetize non-u se benefits of the final 316(b) regulation for the

Phase III facilities.

Additional detail on peer review findings is provi ded in the Peer Review Report (Versar, 2006).7 5. References

Arrow, K., R. Solow, P. Portney, E. Leamer, R. Radner, and H. Schuman. 1993. Report of the NOAA panel on contingent valuation. Federal Register 58(10):4602-4614.

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S. Mourato, E. Ozdemiroglu, D.W. Pierce, R. Sugden, and J. Swanson. 2002. Economic Valuation with Stated Preference Surveys: A Manual . Edward Elgar, Northampton, MA.

Besedin E. and R. Johston. 2006. List of Hypothesis Tests and Quan titative Assessments for EPA 316(b) Analysis . Memorandum to Julie Hewitt U.S. EPA/OW, da ted February 13, 2006, Cambridge, MA.

Besedin, E., M. Ranson, and R. Johnston. 2004. Findings from Focus Groups . Memorandum to Erik Helm, U.S. EPA/OW, dated June 30 2004. Appendices I-III, Focus Group Transcripts.

Besedin, E., R. Johnston, M. Ranson, and J. Ahlen. 2005. Findings from 2005 Focus Groups Conducted
Under EPA ICR #2155.02 . Memorandum to Erik Helm, U.S. EPA/OW, dated October 18 2005, Cambridge, MA.

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in Designing a contingent Valuation Survey to Measu re the Benefits of Hazardous Waste Management Regulations. Research Triangle Ins titute, Research Triangle Park, NC.

Freeman III, A.M. 2003. The Measurement of Environmental and Resource Values: Theory and Methods

. Resources for the Future, Washington, DC.

Johnston, R.J., T.F. Weaver, L.A. Smith, and S.K. Swallow. 1995. Contingent valuation focus groups: Insights from ethnographic interview techniques. Agricultural and Resource Economics Review 24(1):56-

69.

Kaplowicz, M.D., F. Lupi, and J. P. Hoehn. 2004. Multiple methods for developing and evaluating a stated-choice questionnaire to value wetlands. In Mehtods for Testing and Evaluating Survery Questionnaires , S. Presser, J. M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer (eds.). John Wiley & Sons, Inc., New Jersey, pp. 503-524.

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation

Method . Resources for the Future, Washington, DC.

Responsive Management. 1992. Responsive Management Report: Focus on Focus Groups. Responsive Management , Tallahassee, FL.

U.S. EPA. 2000a. Guidelines for Preparing Economic Analyses . EPA 240-R-00-003. U.S. Environmental Protection Agency, Washington, DC. September, Cambridge, MA.

U.S. OMB. 2003. Circular A-4 . U.S. Office of Management and Budget. September 17, 2003. http://www.whitehouse.gov/OMB/circulars/a004/a-4.pdf

.8 Versar. 2006. Peer Review Report for “Willingness to Pay Su rvey Instrument for Section 316(b) Phase III Cooling Water Intake Structures.” Prepared for U.S. EPA Office of Water's Office of Science and Technology by Versar, Inc., Springfield, VA.