Sensitivity to Scope: Evidence from a CVM Study of Wetlands Daniel J. Mullarkey Economic Research Service, USDA 1800 M Street NW, Room 4087 Washington, DC 20036

Phone: 202-649-5599 Fax: 202-649-5756 Email: danielm@econ.ag.gov Richard C. Bishop, Chair Dept. of Agricultural and Applied Economics University of Wisconsin-Madison 427 Lorch Street Madison, WI 53706 Phone: 608-262-8966 Fax: 608-262-4376 Email: bishop@aae.wisc.edu

Presented at: American Agricultural Economics Association Annual Meeting Nashville, TN, August 8-11, 1999 Presenter and primary contact: Daniel J. Mullarkey 1

Abstract: Wetlands valuation is a situation in which CVM studies might be expected to fail scope tests. This paper reports results from a split-sample CVM study of Wisconsin wetlands. The survey employed a multiple-bounded, polychotomous-choice format, and compared WTP distributions using the method of convolutions. The survey demonstrated sensitivity to scope.

1 Daniel J. Mullarkey is an Economist at the Economic Research Service, USDA, and Richard C. Bishop is chair of the Department of Agricultural and Applied Economics at the University of Wisconsin-Madison. The views expressed in this article do not necessarily represent the policies or views of the U.S. Department of Agriculture. Copyright 1999 by Daniel J. Mullarkey and Richard C. Bishop. All rights reserved. Readers may make verbatim copies of this document for non-commercial purposes by any means, provided that this copyright notice appears on all such copies.Introduction The debate over whether the Contingent Valuation Method (CVM) can provide valid estimates of economic value focuses in part on whether willingness to pay (WTP) estimates are sensitive to the scope of the good being valued. According to economic theory, if good A contains all of the attributes of good B (in at least the same levels), and also includes positive levels of at least one additional attribute, then good A should be valued higher than good B. This allows a testable hypothesis for CVM applications, which can be used to provide evidence on the theoretical construct validity of the survey in question (Bishop, Champ and Mullarkey, 1995). The issue of scope insensitivity has been championed by critics of CV (Kahneman and Knetsch, 1992; Diamond et al., 1993; Desvousges et al., 1993; and Schkade and Payne, 1994). These studies have been criticized elsewhere (Smith, 1992; Hanemann, 1994; Carson, 1995; Mullarkey, 1997) as lacking content validity, with primary objections being that the goods to be valued are poorly defined, and that survey design features and econometric techniques have not been state-of-the-art. In reviewing past studies, Carson (1995) finds considerable evidence of sensitivity to scope, with the exceptions being the studies cited above. Despite the flaws in these surveys, sensitivity to scope is a legitimate issue to raise, and the increased attention will result in

better CV studies. Wetlands valuation represents a perfect situation in which to explore this issue. Wetlands valuation has historically been piecemeal and incomplete (Mullarkey, 1997), and there is interest in determining whether CVM can provide better information about total values for the services of particular wetlands. 2 However, values for many wetlands will contain components of nonuse

2 Good information on values for wetlands remains a priority despite recent reductions in the rate of loss of wetlands. One of the major tools in this push toward “no net loss” of wetlandsand indirect use values; the diverse services provided by wetlands are relatively unfamiliar to many people; and there is considerable scientific uncertainty surrounding the ecological functions performed by wetlands. These are factors that have been identified as being troublesome for CVM, leading to the suggestion that survey participants will express their values for wetlands in general rather than the wetlands of interest to a particular survey. This paper reports on a CVM survey designed to test sensitivity to scope, with the application being a wetlands preservation project.

Case Study The case study chosen for this research was a highway expansion project in northwest Wisconsin that required the filling of 110 acres of wetlands. The Wisconsin Department of Transportation (WDOT) had decided to expand a 44-mile stretch of U.S. Highway 53 (USH 53). Plans called for the two new lanes to traverse 47 acres of tributary system wetlands and 63 acres

of isolated basins. The USH 53 wetlands were located in an area that has an abundance of wetlands, and were not known to contain any threatened or endangered species. These are classic examples of the type of unspectacular wetlands that continue to be lost in small but steady increments, and about which we have very little valuation information. These losses are being mitigated by the creation of 220 acres of isolated basins at different sites along the highway. In accordance with the cooperative agreement between the WDOT and the Wisconsin Department of Natural Resources, a 2-1 replacement ratio was chosen to reflect the somewhat imperfect success of wetland creation attempts to replace all of the lost

is mitigation, where wetlands are restored or created in one location to offset loss of wetlands in a nearby location. In such cases, where the goal is a fair trade of wetland services, it is importantwetland services. This project was selected in large part because the Environmental Impact Statement prepared by the WDOT contained the most detailed ecological data set on the services provided by any wetlands in Wisconsin that faced the current threat of development. A representative subset of the 130 wetlands was evaluated on a service-by-service basis using the Wetland Evaluation Technique (WET). Sixteen wetlands were rated for eleven services: groundwater recharge, groundwater discharge, floodflow alteration, sediment stabilization, sediment and toxicant retention, nutrient removal and transformation, primary production export, aquatic diversity and abundance, wildlife breeding habitat, wildlife migration habitat, and wildlife wintering habitat. WET returns a qualitative rating of high, medium, low or uncertain for each service evaluated. A rating of “ high ” for a particular function implies that there is a high probability that the wetland provides that service, rather than implying that the wetland provides a large quantity of that service. Reflecting the diversity amongst even wetlands in the same geographic area, the WET ratings for the USH 53 wetlands varied quite a bit. There were clear differences between the isolated basins and the tributary system wetlands, which necessitated describing the two types separately. There were also differences within each type, which complicated the task of providing a general description of the quality of each service. Providing useful information for eleven services and two types of wetlands was the central challenge to developing a survey with strong content validity. 3 After three focus groups, a pretest, and consultations with WDOT wetland specialists, we condensed the eleven services to six. In response to feedback from focus group participants, we used one table with columns for each type of wetland, rather than having

to understand both the value of the original wetlands and the value of the mitigation wetlands.separate table for each type of wetland. The majority of focus group participants wanted quantitative information, thus we provided percentages of each type of wetland rating high, moderate, or low for each service. The final information set of reference levels for the wetland services in provided in Figure 1.

Figure 1: Table of Wetland Services BENEFIT RIVER SYSTEM ISOLATED BASINS

Floodflow Alteration These wetlands are good at slowing flood flows. This reduces erosion and damage to downstream areas. Isolated basins are also good at slowing floods, absorbing water, and reducing the potential for flood damage.

Water Purification Some river system wetlands purify water by filtering out pollutants (sediment, nutrients, and some toxic substances). The isolated basins effectively protect surface and groundwater supplies by removing pollutants (sediment, nutrients, and some toxic substances).

Fish and Other Aquatic Animals 60% of the river system wetlands can support fish and aquatic species, but the other 40% cannot. The isolated basins generally are not suitable habitat for fish and other aquatic populations.

Wildlife Breeding

Habitat These river system wetlands are good breeding habitat for a variety of wildlife. For breeding habitat, 45% of the isolated basins rate low, 22% medium, and 33% high. Bird Migration

Habitat These wetlands provide good habitat for migrating birds. For migration habitat, 56% of the isolated basins rate low, 22% medium and 22% high.

Bird Wintering

Habitat Most of these river system wetlands are good wintering habitat; the 30% that freeze over are not. These isolated basins freeze in the winter, which makes them poor habitat for wintering birds.

3 Developing a survey that was perceived as neutral between the choice of development or preservation was also challenging, and the focus groups were quite useful in this endeavor.One decision that needs to be made in every CVM survey is which value elicitation device, or question format, to employ. Different types of valuation questions can produce different welfare estimates (Welsh and Poe, 1998). The question format selected for this survey was the multiple-bounded polychotomous-choice (MB-PC) framework (see Welsh and Bishop, 1993, and Welsh and Poe, 1998). Multiple-bounded questions are an extension of single- and double-bounded referendum questions. Participants are asked whether or not they would vote to pay for a project over a range of costs. For example, this study used 16 cost levels, ranging from ten cents to $150. Each participant is asked how they would vote at each amount, thus avoiding the anchoring bias problems associated with double-bounded questions. 4 Compared to singlebounded questions, having participants evaluate a wide range of costs may reduce yea-saying since they are not faced with only one chance to support an environmental improvement. Unless the participant answers “ yes ” to every amount, or “ no ” to every amount, the multiple-bounded approach will provide fairly tight upper and lowers bounds on each participants ’ WTP. The polychotomous-choice format allows respondents to indicate how certain or uncertain they are that they would vote yes or no to any particular amount. For example, the five voting choices offered in this survey were “ Definitely Yes ” , “ Probably Yes ” , “ Not Sure ” ,

“ Probably No ” , and “ Definitely No ” . Allowing respondents to express uncertainty may provide insight into the valuation processes of respondents. When combined with multiple-bounded questions, the researcher can observe the point on the individual ’ s demand curve where WTP starts to become uncertain. This format seems appropriate for wetlands valuation studies since

4 There may be biases associated with the range of costs covered and intervals between costs, or with whether listing costs in ascending or descending order affects welfare estimation. Hopefully thorough use of qualitative design tools, such as focus groups, will avoid or reduce these potential biases; however, they certainly warrant further research.respondents typically will not have had a lot of experience searching the preferences for wetlands. As with other formats that do not estimate point values, MB-PC is less statistically efficient than the open-ended question format. However, since it does provide fairly tight upper and lower bounds for most respondents, it is more efficient that single- or double-bounded dichotomous choice questions. This gain in statistical precision is welcome in any study, but seems particularly important for studies that explicitly seek to test sensitivity to scope. Using a mail survey, a random sample of 550 Wisconsin residents was asked to consider a hypothetical alternative alignment of the two new lanes that would travel around the existing wetlands instead of through them. The survey specified that the alternative alignment would have the same safety and economic benefits as the planned alignment, but would preserve the original wetlands. Survey participants were also informed that the alternative alignment would cost more to construct, and were asked about their willingness to pay higher state income taxes in order to preserve the original wetlands. After three rounds of mailing and a follow-up postcard, the survey had a response rate of 60%. Removing those who did not properly complete the valuation question left 117 respondents in the base group and 122 in the scope group. A True-False quiz was included on the inside cover of the questionnaire booklet. This type of quiz serves two purposes. First, it presents respondents with an opportunity to review to review the information provided, which leads to more informed responses. Second, it provides the researcher with quantitative evidence of content validity. The quiz in this survey differed between the two groups. Base group respondents were asked eight questions, six of which asked whether scientists expected the mitigation wetlands to replace each of the six wetland services. The other two questions matched two for the scope group, who answered six other questions as well, many of which focused on the quality of the original wetlands. Overall, the base groupcorrectly answered 83.9% of the questions, and the scope group answered 88.4% correctly. Of the fourteen questions asked, all but three had correct response rates in excess of 80%, with the lowest being 69%. These results provide encouraging evidence of content validity. The test of sensitivity to scope was accomplished by splitting the sample into two groups. The base group was informed about which services the mitigation attempt was expected to replace. The scope group, on the other hand, was not informed that a mitigation attempt was being conducted. The two groups were thus asked to value projects that differed in the net services provided (i.e., that differed in scope), since one group stands to have some services replaced by the mitigation attempt, but the other group does not. Economic theory predicts that the scope group ’ s WTP will exceed that of the base group since the scope group stands to lose more wetland services if the alternative alignment is not adopted. The null hypothesis of the scope test, therefore, is that WTP will be equal across groups, and the alternative hypothesis is that the scope group ’ s WTP is greater than the base group ’ s WTP. Given the structure of this scope test, a one-tailed test is appropriate.

Scope Test Results To conduct the scope test, sample means need to be estimated, and then some method of comparison needs to be determined. In order to estimate sample means from polychotomous-choice data, the data first need to be converted to dichotomous-choice data. For the data in the USH 53 survey, three levels of WTP certainty were used to dichotomize the data.

The “ Definitely Yes ” dichotomizing scheme counts only the responses in column A (see Figure 4.1) as Yes votes, and the rest as No votes. The “ Probably Yes ” scheme considers responses in columns A or B to be Yes votes, and the rest to be No votes. Finally, the “ Not Sure ” schemecounts responses in columns A, B, or C as Yes votes, and all others as No votes. Sample means will, of course, increase as decreasing levels of WTP certainty are allowed to count as Yes

votes. 5 The ambiguity over how to interpret answers to polychotomous-choice questions need not be viewed as negative. This value elicitation format allows the researcher to present a range of WTP estimates, and can be interpreted as providing upper and lower bounds on WTP. 6 When a more specific estimate is required, the conservative “ Definitely Yes ” level is in keeping with the NOAA panel ’ s suggestion that when choices need to be made, CV researchers should lean toward the conservative. As will be seen, the scope test results are robust across certainty levels. To further explore robustness, we examined three data sets that differed in their handling of potential outliers. Four respondents in the base group, and six in the scope group, answered

“ Definitely Yes ” to every bid offered in the WTP question, including the highest bid of $150. These responses might be considered outliers. They may well represent the individual's true valuation, or they may be a form of protest bid. Determining which respondents are being truthful and which are expressing inflated valuations is, of course, a subjective and essentially arbitrary process. If we accept the principle of consumer sovereignty, then it seems we should accept the answers as being truthful unless we have evidence to the contrary. Such evidence would have to come from answers to other survey questions. The types of evidence that might be indicative of

5 The use of a polychotomous-choice question format is not strictly compatible with the random utility model, which assumes that the individual knows his or her utility (and therefore WTP) with certainty. A more complex statistical model that incorporates uncertainty on the part of the respondent might allow for more of the polychotomous-choice WTP information to be utilized than does the present dichotomization scheme. The handling of WTP uncertainty in the context of CV is clearly an area worthy of further research.

6 Welsh and Poe (1998) compare the various certainty levels of the MB-PC question to single-bounded, payment card and open-ended question formats for the same environmental amenity, which helps illuminate how respondents deal with uncertainty in these latter formats.untruthful WTP responses are rejection of key elements of the scenario, skepticism about the valuation exercise, and answers that indicate either a lack of support for the alternative alignment or a lack of interest in the environment. The ten potential outliers were screened with these considerations in mind. Scope test results are presented below for three data sets. The full data set contains all 239 participants; the “ no outliers ” data set removed all ten potential outliers, and the

“best ” data set has only three of the outliers removed (two from the base group and one from the scope group) based on answers to other questions that identified them as potential protest bids. As will be seen, scope tests results are robust across all three data sets. Sample means were estimated using maximum likelihood estimation, with a logistic cumulative density function specified for the WTP distribution. As expected, sample means were higher for the scope group than for the base group. Sample means for the base group were $13.68, $24.07, and $37.38 for the three certainty levels, respectively. Sample means for the scope group were $20.77, $41.84, and $57.83, respectively. One way to compare these sample means is to use the method of convolutions describedby Poe, Severance-Lossin and Welsh (1994). The convolutions approach allows for a semiparametric comparison of two distributions of WTP estimates. Empirical WTP distributions can

be obtained from estimates of the mean and variance-covariance matrices of the estimated coefficients by using resampling techniques, such as Monte Carlo or bootstrapping. The method of convolutions then provides an exact statistical test of the difference between these two empirical distributions. This approach essentially computes a third distribution that is the difference between the two empirical WTP distributions, chops off the tail(s) according to the specified significance level, and sees if the remaining distribution includes zero. If not, the WTP distributions are significantly different.The Monte Carlo resampling technique was used to generate 1000 estimates of mean WTP for each group. Next, the P-value of the convolution of these two empirical distributions was calculated, and this two-step procedure was repeated the procedure 100 times. Results for all three data sets are presented in Table 1. Looking first at the “best ” data, average P-values for the three dichotomizing schemes were .0097, .0005, and .0038, respectively. Thus the null hypothesis of equal WTP across treatments can be rejected at the 1% significance level for all three degrees of WTP certainty.

Table 1: One-Tailed Method of Convolutions Scope Tests WTP Certainty Level Full Data Best Data No Outliers Definitely Yes .0419 .0097 .0597 Probably Yes .0017 .0005 .0023 Not Sure .0092 .0038 .0123 N (base, scope) (117,122) (115,121) (113,116)

Recall that the “best ” data omitted two “ outliers ” from the base group, but only one from the scope group. This serves to mildly accentuate the difference between the two groups. When scope tests are run on the full data, average P-values are naturally higher. However, the scope test still rejects the null hypothesis of equality at the 5% significance level for all three WTP certainty levels, and at the 1% level for both the “ Probably Yes ” and “ Not Sure ” levels. Thus the conclusion is the same - that WTP estimates from this survey are sensitive to scope. Similarly, scope tests were performed on the “ No Outlier ” data, which omitted four observations from the base group and six from the scope group. This data should reduce the difference in WTP between the two groups. Here again, the null hypothesis can be rejected forthe “ Definitely Yes ” scheme at the 10% significance level, for the “ Probably Yes ” scheme at the 1% level, and for the “ Not Sure ” scheme at the 5% level. Even with all ten potential “ outliers ” removed, the WTP estimates are clearly sensitive to the scope of the change in wetland services. Likelihood ratio tests could also be used to conduct the scope test. In this context, likelihood ratio tests determine whether the estimated coefficients are equal across samples by comparing value of the log likelihood function (LL) for each the two samples to the value of the log likelihood function for a pooled sample. The likelihood ratio test statistic (LR) is calculated as LR = -2 * [LL(base) + LL(scope) - LL(pooled)], and compared to the chi-square distribution with degrees of freedom equal to the number of estimated parameters. These are two-tailed tests, which allow us to test whether the distribution of WTP is the same across samples, but do not test for the direction of any difference between the samples. In order to compare the statistical power of likelihood ratio tests to that of the convolutions method, a two-tailed convolutions test would be required. Table 2 presents such a comparison, where the results presented in the three

columns labeled “ LR ” are likelihood ratio statistics, and the results in the three columns labeled

“ P-value ” are the average P-values of 100 sets of convolutions (as described above), using a twotailed truncation scheme. Table 2: Comparison of Two-Tailed Scope Test Results Between the Method of Convolutions and Likelihood Ratio Statistics Certainty Level Full Data Best Data No Outlier Data LR P-value LR P-value LR P-value Definitely Yes 3.65 .0837 7.44** .0194 3.06 .1194 Probably Yes 11.24*** .0033 15.14*** .0009 11.55*** .0046 Not Sure 6.27** .0183 8.11** .0075 5.92* .0246 N (base, scope) 117, 122 117, 122 115, 121 115, 121 113, 116 113, 116 * Significant at the 10% level; ** Significant at 5%; *** Significant at 1%.As noted, a one-tail test is more appropriate given the hypothesis we seek to test, but this two-tailed comparison is presented in order to demonstrate that the convolutions approach is statistically more powerful than likelihood ratio tests. Note the three shaded boxes in Table 2. In each of these cases, the convolutions method would reject the null hypothesis at lower standard significance levels than would the likelihood ratio test. For example, using the full data set and looking at the “ Definitely Yes ” level of WTP certainty, the likelihood ratio test would fail to reject the null hypothesis at the 10% significance level, whereas the method of convolutions would reject the null hypothesis. This could obviously lead to very different and erroneous policy recommendations. Thus if the researcher or analyst wishes to assess differences in estimates of mean WTP (or any empirical distribution), s/he should use the convolutions approach to provide a more powerful test.

Conclusions This study has shown that CVM studies can demonstrate sensitivity to scope, even for complex, relatively unfamiliar goods that are dominated by nonuse values. This result is robust across certainty levels and across data sets with potential outliers removed. The three factors that contributed to this finding were careful attention to establishing strong content validity, the use of a state-of-the-art value elicitation device, and use of powerful econometric techniques. Given the difficult valuation context, this finding is encouraging for the use of CVM. It is also good news for the valuation of wetlands, which would benefit from a more holistic valuation methodology.References Bishop, R.C., P.A. Champ and D.J. Mullarkey. 1995. “ Contingent Valuation. ” In Handbook of Environmental Economics . Ed. Daniel W. Bromley. Oxford: Blackwell Publishers. 629-654. Carson, R.T. 1995. “ Contingent Valuation Surveys and Tests of Insensitivity to Scope. ” Draft manuscript. Desvousges, W.H., F.R. Johnson, R.W. Dunford, K.J. Boyle, S.P. Hudson and K.N. Wilson. 1993. “ Measuring Natural Resource Damages with Contingent Valuation: Tests of Validity and Reliability. ” In “ Contingent Valuation: A Critical Assessment” , J.A. Hausman ed. NorthHolland, Amsterdam. Diamond, P.A., J.A. Hausman, F.R. Johnson, R.W. Dunford, K.J. Boyle, S.P. Hudson, and K.N. Wilson. 1993. “ Measuring Natural Resource Damages with Contingent Valuation: Tests of Validity and Reliability. ” In “ Contingent Valuation: A Critical Assessment” , J.A. Hausman ed. North-Holland, Amsterdam. Hanemann, W.M. 1994. “ Valuing the Environment Through Contingent Valuation. ” Journal of Economic Perspectives. 8(4):19-43. Kahneman, D. and J.L. Knetsch. 1992. “ Valuing Public Goods: The Purchase of Moral Satisfaction. ” Journal of Environmental Economics and Management. 22(1):57-70. Mullarkey, D.J. 1997. “ Contingent Valuation of Wetlands: Testing Sensitivity to Scope. ” Ph.D. dissertation, University of Wisconsin, Madison. Poe, G.L., E.K. Severance-Lossin and M.P. Welsh. 1994. “ Measuring the Difference (X-Y) of Simulated Distributions: A Convolutions Approach. ” American Journal of Agricultural Economics. 76:904-915. Schkade, D.A. and J.W. Payne. 1994. “ How People Respond to Contingent Valuation Questions: A Verbal Protocol Analysis of Willingness to Pay for an Environmental Regulation. ” Journal of Environmental Economics and Management. 26:88-109. Smith, V.K. 1992. “ Comment: Arbitrary Values, Good Causes, and Premature Verdicts. ” Journal of Environmental Economics and Management . 22(1):71-89. Welsh, M.P. and R.C. Bishop. 1993. “ Multiple Bounded Discrete Choice Models. ” In J.C. Bergstrom, ed., Benefits & Costs Transfer in Natural Resource Planning. Western Regional Research Publication, W-133, Sixth Interim Report. 331-352. Welsh, M.P. and G.L. Poe. 1998. “ Elicitation Effects in Contingent Valuation: Comparisons to a Multiple Bounded Discrete Choice Approach. ” Journal of Environmental Economics and Management . 36:170-185.