Environment and Resource Division 

Date May 23, 2006

To Julie Hewitt, OW/OST/EAD

From Elena Besedin, Abt Associates Inc

Subject Development of Willingness to Pay Survey Instrument for Section 316(b) Phase III 

Cooling Water Intake Structures

As required under Executive Order 12866, EPA conducted economic impact and cost-benefit analyses 

for the section 316(b) regulation for Phase III facilities. Cost-benefit analysis should include a 

comprehensive estimate of total social benefits, including both use and non-use values. “Non-use values, 

like use values, have their basis in the theory of individual preferences and the measurement of welfare 

changes. According to theory, use and non-use values are additive” (Freeman, 2003). It is generally 

accepted in the economic literature that non-use values may be substantial in some cases. Additionally, 

when small per capita non-use values are held by a substantial fraction of the population, they can be very 

large in the aggregate. Therefore, failure to recognize such values may lead to improper inferences 

regarding policy benefits. As stated by Freeman (2003, p. 138), “[i]f non-use values are large, ignoring 

them in natural resource policymaking could lead to serious errors and resource misallocations.” Both 

EPA’s own Guidelines to for Preparing Economic Analysis and OMB’s Circular A-4, governing 

Regulatory Analysis, support the need to assess non-use values (U.S. EPA, 2000a; U.S. OMB, 2003). 

Consideration of potential non-use values is particularly important in the case of the section 316(b) 

regulation because nearly all (97 to 98 percent) impingement and entrainment losses at CWIS consist of 

either forage species, or non-landed recreational and commercial species that do not have direct uses. 

Although individuals do not use these resources directly, they may nevertheless be affected by changes in 

the resource status or quality, such that they would be willing to pay to maintain these resources. 

Although economic theory clearly allows for the possibility of significant non-use values in this case, it 

does not shed light on whether such values are large or small in the aggregate. 

EPA attempted to measure non-use benefits of the 316(b) regulation in monetary terms, as suggested by 

EPA’s own guidance and OMB’s (U.S. EPA, 2000; U.S. OMB, 2003). Using the benefits transfer 

technique requires adequate empirical valuation studies. No empirical studies were found that estimated 

non-use values for impacts on fish alone. Thus, EPA began designing a stated preference survey to 

separately estimate total value (including non-use value) of fish impacts when work on this rule began. 

Stated preference surveys use carefully designed questions to elicit respondents’ willingness to pay 

(WTP) for particular ecological improvements, based on their responses to either discrete choice or openended questions regarding hypothetical resource improvements or programs. Such improvements may 

include increased protection of aquatic habitats or species with particular attributes. Stated preference 

survey methodology is the generally accepted means to estimate total (including use and non-use) 

memorandum 
2

resource values. Moreover, stated preference survey methods “are likely to offer the only feasible 

approaches to estimating non-use values” (Freeman, 2003, p. 154). 

EPA received OMB approval in August, 2005 of an Information Collection Request to conduct focus 

groups for survey design (EPA ICR #2155.01). EPA has conducted a series of twelve focus groups in 

different geographic locations to assist in the development of this stated preference survey. Based on 

findings from the focus groups, EPA designed a survey using state-of-the-art valuation methods, and 

conducted an external peer review of the survey instrument and analysis plan, completed in February, 

2006 (Versar, 2006). While peer reviewers agreed that the effort was state-of-the-art, they also provided 

suggestions to improve the reliability of the results; these suggestions could not be incorporated in time 

for today’s action. As a result, EPA was unable to monetize non-use benefits of the policy options 

considered in development of the final 316(b) regulation for Phase III facilities. 

The remainder of this memorandum, summarizes findings from focus groups, outlines EPA’s plan for the 

pilot study, and presents experts’ conclusions from peer review of the survey instrument. 

1. Focus Groups 

To assist in the development of this stated preference survey, EPA conducted a series of twelve focus 

groups with a total of 96 respondents. These focus groups and cognitive interviews were conducted 

under EPA ICR #2155.01, “Willingness To Pay Survey for Section 316(b) Phase III Cooling Water 

Intake Structures: Instrument, Pre-Test, and Implementation.” The focus groups and interviews took place 

in the following locations on these dates: 

1) Providence, RI; May 4, 2004 

2) Providence, RI; May 11, 2004 

3) North Haven, CT; May 20, 2004 

4) Baltimore, MD; June 1, 2004 

5) Baltimore, MD; June 10, 2004 

6) Portland, OR; August 15, 2005 

7) Baltimore, MD; August 23, 2005 

8) Green Bay, WI; August 29, 2005 

9) East Providence, RI; September 2, 2005 [12 cognitive interviews] 

10) Bethesda, MD; September 7, 2005 

11) Providence, RI; September 15, 2005 [8 cognitive interviews] 

12) Baltimore, MD; September 23, 2005 

These focus groups were conducted following standard, accepted practices in the stated preference 

literature, as outlined by Mitchell and Carson (1989), Desvousges et al. (1984), and Johnston et al. 

(1995). Some of the focus groups incorporated individual cognitive interviews, as detailed by (Kaplowicz 

et al. 2004). The focus groups and cognitive interviews allowed EPA to better understand the public's 

perceptions and attitudes concerning fishery resources, to frame and define survey questions, to pretest 

draft survey questions, to test for and reduce potential biases that may be associated with stated 

preference methodology, and to ensure that both researchers and respondents have similar interpretations 

of survey language and scenarios. In particular, cognitive interviews allowed in-depth exploration of the 

cognitive processes used by respondents to answer survey questions, without the potential for 

interpersonal dynamics to sway respondents’ comments (Kaplowicz et al. 2004). 

Each focus group was composed of seven to nine individuals that were randomly selected by the focus 

group facilities from panels of focus group participants maintained by each facility. Participants were 
3

eligible if they were eighteen years of age or older, if they were not full-time students, and if they had not 

participated in a survey or focus group in the last six months. Each of the focus groups included a mix of 

individuals with diverse socioeconomic backgrounds, based on characteristics such as income, marital 

status, age, race, education, occupation, and gender. Participants were compensated for their participation 

in the focus groups. 

The format of the focus group sessions varied. In general, a moderator led a group through a discussion of 

specific topics to discover individuals’ attitudes and opinions. For example, discussion topics included: 

• general questions about participants’ knowledge and opinions about fish and what role fish play 

in the environment, 

• general questions about participants’ use of water resources for recreation and their recreational 

experiences; 

• questions about participants’ knowledge of the current health/quality of fish 

populations/resources in their local area; 

• questions about participants’ familiarity with any particular threats to fish and fish habitats from 

human activities; 

• questions about participants’ motives (if any) for protecting fish and other aquatic organisms; and 

• questions about participants’ opinions about programs that would reduce fish mortality from manmade causes, the geographic scope of these programs, and the long-term effects of those 

programs on fish populations. 

In some focus groups, participants were also asked to respond to several questions from the draft survey 

instrument. This was followed by a debriefing session. During the debriefing sessions, the moderator 

asked focus group participants about their reactions to the survey format and content, whether the survey 

questions were clear, whether the background information presented in the survey or introductory 

materials was sufficient and how participants interpreted it, what went through participants’ minds when 

they read survey questions, and what their motivations were for responding they way they did to the 

survey. In Focus Groups 4 to 12, the moderator first read introductory materials accompanied by graphics 

and pictures and then handed out draft survey questions. 

Each session was one to two hours long. All sessions were audio taped and transcribed. For more 

detailed discussion of results from the focus groups, refer to the focus group reports: 

• Besedin et al, 2005. Findings from 2005 Focus Groups Conducted Under EPA ICR #2155.02. 

Memorandum to Erik Helm, EPA/OW/OST/EAD, dated October 18, 2005. 

• Besedin et al, 2004. Findings from Focus Groups. Memorandum to Erik Helm, 

EPA/OW/OST/EAD, dated June 30, 2004 

The appendices of these two reports also provide full transcripts, presentation slide shows and scripts, 

survey instruments, and fact sheets for each focus group and set of cognitive interviews. 

2. Findings from Focus Groups 

Focus groups are often described as “informal sessions in which a skilled moderator leads a group of 

individuals through a discussion of specific topics to discover their attitudes and opinions” (Desvousges 

et al., 1984, p. 2-1; cited in Johnston et al., 1995, p. 56). Focus groups are among the most significant 

qualitative research tools used in social sciences (Bateman et al., 2002); observations from focus groups 

or similar qualitative tools may provide information allowing one to interpret or validate previously 

obtained quantitative results (Responsive Management, 1992). EPA conducted 12 focus groups meetings 
4

to obtain insight into potential WTP values for reducing I&E at cooling water intakes and motivations 

underlying respondents’ stated willingness to pay (WTP) values (Desvousges et al., 1984; 1988; Mitchell 

and Carson, 1989; Johnston et al., 1995). 

Because focus groups rarely represent a truly random sample of particular populations, it is generally 

considered inappropriate to substitute focus group data for WTP estimates derived from large-sample 

survey instruments (Bateman et al., 2002, p. 153). Nonetheless, it is well known that observations from 

focus groups or similar qualitative tools may provide information allowing one to interpret or validate 

previously obtained results (Responsive Management, 1992). Pre- or post-implementation focus groups 

may provide supporting evidence of the validity of stated preference survey (or related benefit transfer) 

results and the assumptions that underlie such analyses (e.g., Arrow et al., 1993). Table 1 provides 

summary statistics from 7 focus group meetings. 

Based on the findings from 12 focus groups, the Agency found that both user and non-users of the 

affected aquatic resources are likely to hold values for reducing I&E mortality of the affected fish species. 

74% of focus group participants cited bequest and existence values as one the main motives for their 

WTP for reducing I&E mortality. Only 39% participants cited use motives (e.g., preserving fish as food 

source for people or participation in recreational fishing) as one the main motives for reducing I&E 

mortality. 

Table 1: Summary of Focus Group Responses 

by Motives for Reducing Fish Losses a

Motive Type for Answering “Yes” to 

Program Option Choices 

Number of Respondents 

Answering “Yes” b, c

Percentage of All Program 

Option “Yes” Answers

Non-Use Motives 113.5 74% 

Bequest Value 37 24% 

Existence Value 76.5 50% 

Use Motives 59 39% 

Source of Food for People 33 22% 

Commercial and Recreational Fishing Industry 21 14% 

Personal Angler Use 5 3% 

Other Factors 6.5 4% 

Total “Yes” Answers to Program Options 153 100% 

Total Observations 190 

a

 Each question is regarded as one observation. Observations include focus groups for the following regions: Pacific 

Mountain, Northeast, and Great Lakes. The Mid-Atlantic Region is excluded from the count because relevant 

questions were not included in these focus groups. 

b

 Double counting may occur if a respondent answered “yes” to multiple option choices in one question or ranked 

motives options; therefore, the number of answers for each choice may not sum to the total number of program 

option “yes” answers. 

c

 If motive options were ranked, a weight was applied to each answer. 
5

Focus group participants agreed that all fish species (including forage fish) play an important role in the 

affected ecosystems and thus also have significant values. For example, when one respondent was asked 

about what fish species he was considering when he made his choice between the two stated preference 

(SP) options, he answered: 

“I think that I am more worried about the types of fish that are not directly used by us. I think that the 

types of fish that are directly used by us — I think people will make sure there are plenty of them. 

Probably because of the money aspect of it. I am more concerned about the types of fish that are not 

related to us because people will think of it as something that we don’t need and they will allow them to 

go extinct.” 

(Besedin et al., 2004a, Focus Group # 4, p. 7) 

Another respondent, when asked if he had specific fish species in mind when he was participating in the 

focus group or did he find all types of fish important, said: 

“No, I don't have any specific in mind. I think every fish will be equally important for one thing or 

another, whether it's for food or I guess some fish are food for other fish. Or even just to enjoy, you 

know. They have a right to live like anything else, you know?” 

(Besedin et al., 2005, Focus Group # 11, Interview # 5, p. 3) 

Focus groups also showed that 74% of focus group participants have non-zero values for reducing I&E 

mortality. Based on focus group findings the mean value for reducing I&E mortality range from $12 to 

$54 per year. As pointed above, it is inappropriate to substitute focus group data for mean WTP estimates 

derived from large-sample survey instruments. However, this finding suggests that WTP for reducing 

I&E mortality of all species is likely to be positive. 

Additional detail on EPA’s findings from the focus group meetings is provided in the Focus Group 

Reports (Besedin et al., 2004a; Besedin et al., 2005). 

3. Pilot Study 

In addition to conducting focus groups EPA has developed a detailed plan for the pilot test of the survey 

instrument prior to conducting the main study. The purpose of the pilot study is to assess the performance 

of the proposed survey instrument and slide show under various treatments. It would also allow the 

Agency to determine whether answers reflect expected neoclassical tradeoffs suggested by focus groups, 

or whether responses are strongly influenced by misinterpretations of survey information, symbolic 

effects, cost-effectiveness heuristics, and other patterns that are generally less conducive to appropriate 

welfare estimation. Findings of the pilot study would allow the Agency the capacity to adjust the fullscale study for any problems that are evident. 

The Agency developed a list of hypothesis tests and quantitative assessments for the pilot version of the 

survey instrument. These tests were designed to assess the performance of the survey instrument in 

important areas (e.g., sensitivity to scope for included model attributes, the statistical significance of 

baseline fish populations, attribute order effects, and assessments of response consistency and item non-
6

response), and allow for adjustments to the survey—if necessary—to address any limitations or problems 

discovered. Other tests are designed to assess the implications of different survey treatments on choices 

and model performance. 

EPA provides a detailed plan for the pilot test phase in a memorandum entitled “List of Hypothesis Tests 

and Quantitative Assessments for EPA 316(b) Analysis”, February 13, 2006 (Besedin and Johnston, 

2006). 

4. Peer Review Results

EPA conducted an external peer review of the survey instrument and analysis plan, completed in 

February, 2006 (Versar, 2006). The goal of the Agency’s Peer Review Policy is to ensure that scientific 

and technical work products receive appropriate levels of critical scrutiny from scientific and technical 

experts as part of the overall decision making process. This survey instrument and analysis plan were 

reviewed by well recognized experts in the field of resource economics, including Dr. Richard C. Bishop, 

Dr. Kevin J. Boyle, Dr. Timothy C. Haab, Dr. James R. Kahn, and Dr. Alan J. Krupnick. 

Peer reviewers were asked to comment on whether the effort used in developing and testing the proposed 

survey meet current standards for stated preference surveys reflected in the literature and whether the 

survey likely provide useful insights into preferences, tradeoffs, and willingness to pay associated with 

the described policies. The reviewers were also asked to comment on specific survey components (e.g., 

whether the survey sufficiently and appropriately characterize the policy options under consideration) and 

to provide suggestions for improvements to the survey instrument and the associated model. For detail on 

specific questions to peer reviewers see EPA’s charge to peer reviewers in the peer review report for 

willingness to pay survey instrument for §316(b) Phase III cooling water intake structures (Versar, 2006). 

The majority of reviewers noted that “those designing the study display excellent knowledge of the 

scientific literature on stated-preference methods” and that “the overall concept and design of the survey 

is sound and with minor corrections will represent a solid piece of applied valuation.” (Versar, 2006, p. 13 

and p.23) Four of the five reviewers were confident that “the survey results will provide useful 

information” to decision makers (Versar, 2006, p.13). 

Peer reviewers provided a number of suggestions to improve the reliability of the results, including 

revisions to some debriefing questions (e.g., incorporating additional response options) and slide show 

materials. EPA notes that the Agency carefully considered suggested improvements to the slide show and 

the survey instrument and began incorporating revisions suggested by R. Bishop, J. Kahn, K. Boyle, and 

T. Haab. Although the reviewers characterized their suggestions for improvement as “minor corrections” 

to the survey instrument that “can all be implemented in a short time” they urged EPA to add between six 

to 12 months to the schedule to ensure “thorough evaluation of the results of the pilot, a more careful 

statistical analysis of the main survey results, and a lot more thought about the follow-up survey.” (Peer 

review Report, p.23). As a result, EPA was unable to implement the stated preference survey instrument 

within the regulatory schedule and to monetize non-use benefits of the final 316(b) regulation for the 

Phase III facilities. 

Additional detail on peer review findings is provided in the Peer Review Report (Versar, 2006). 
7

5. References 

Arrow, K., R. Solow, P. Portney, E. Leamer, R. Radner, and H. Schuman. 1993. Report of the NOAA 

panel on contingent valuation. Federal Register 58(10):4602-4614. 

Bateman, I.J., R.T. Carson, B. Day, M. Hanemann, N. Hanley, T. Hett, M. Jones-Lee, G. Loomes, S. 

Mourato, E. Ozdemiroglu, D.W. Pierce, R. Sugden, and J. Swanson. 2002. Economic Valuation with 

Stated Preference Surveys: A Manual. Edward Elgar, Northampton, MA. 

Besedin E. and R. Johston. 2006. List of Hypothesis Tests and Quantitative Assessments for EPA 316(b) 

Analysis. Memorandum to Julie Hewitt U.S. EPA/OW, dated February 13, 2006, Cambridge, MA. 

Besedin, E., M. Ranson, and R. Johnston. 2004. Findings from Focus Groups. Memorandum to Erik 

Helm, U.S. EPA/OW, dated June 30 2004. Appendices I-III, Focus Group Transcripts. 

Besedin, E., R. Johnston, M. Ranson, and J. Ahlen. 2005. Findings from 2005 Focus Groups Conducted 

Under EPA ICR #2155.02. Memorandum to Erik Helm, U.S. EPA/OW, dated October 18 2005, 

Cambridge, MA. 

Desvousges, W.H., V.K. Smith, D.H. Brown, and D.K. Pate. 1984. The Role of Focus Groups in 

Designing a contingent Valuation Survey to Measure the Benefits of Hazardous Waste Management 

Regulations. Research Triangle Institute, Research Triangle Park, NC. 

Freeman III, A.M. 2003. The Measurement of Environmental and Resource Values: Theory and Methods. 

Resources for the Future, Washington, DC. 

Johnston, R.J., T.F. Weaver, L.A. Smith, and S.K. Swallow. 1995. Contingent valuation focus groups: 

Insights from ethnographic interview techniques. Agricultural and Resource Economics Review 24(1):56-

69. 

Kaplowicz, M.D., F. Lupi, and J. P. Hoehn. 2004. Multiple methods for developing and evaluating a 

stated-choice questionnaire to value wetlands. In Mehtods for Testing and Evaluating Survery 

Questionnaires, S. Presser, J. M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer 

(eds.). John Wiley & Sons, Inc., New Jersey, pp. 503-524. 

Mitchell, R.C. and R.T. Carson. 1989. Using Surveys to Value Public Goods: The Contingent Valuation 

Method. Resources for the Future, Washington, DC. 

Responsive Management. 1992. Responsive Management Report: Focus on Focus Groups. Responsive 

Management, Tallahassee, FL. 

U.S. EPA. 2000a. Guidelines for Preparing Economic Analyses. EPA 240-R-00-003. U.S. Environmental 

Protection Agency, Washington, DC. September, Cambridge, MA. 

U.S. OMB. 2003. Circular A-4. U.S. Office of Management and Budget. September 17, 2003. 

http://www.whitehouse.gov/OMB/circulars/a004/a-4.pdf. 
8

Versar. 2006. Peer Review Report for “Willingness to Pay Survey Instrument for Section 316(b) Phase III 

Cooling Water Intake Structures.” Prepared for U.S. EPA Office of Water's Office of Science and 

Technology by Versar, Inc., Springfield, VA.