MARKET MODELS, PROTEST BIDS, AND OUTLIERS IN 

CONTINGENT VALUATION 

By Greg Lindsey ~ 

ABS'mACT: Responses to willingness-to-pay questions in contingent valuation (CV) 

surveys are being used with increasing frequency to obtain estimates of the value 

of public goods, such as improved water quality, for use in benefit-cost analyses. 

Analysts must decide whether the appropriate conceptual model for any particular 

application is a private or a political market. When estimating willingness-to-pay 

and total benefits, analysts must decide whether to censor outliers and certain zero 

responses, called protest bids, from data sets. Issues associated with identification 

of protest bids are reviewed, and alternate methods for the treatment of outliers 

are discussed. The significance of these methodological decisions is illustrated with 

an example involving willingness to pay for new storm water programs in Baltimore 

County, Md. to help achieve Chesapeake Bay nutrient-reduction objectives. It is 

argued that decisions concerning analysis of troublesome responses should be consistent with the choice of market model for the application. 

INTRODUCTION 

Since the U.S. Water Resources Council (Economic 1983), the Army 

Corps of Engineers (Moser and Dunning 1986), and other agencies have 

sanctioned the use of contingent valuation (CV) methodology in benefitcost applications, surveys have been used with increasing frequency to obtain 

estimates of benefits for public goods, such as water quality. Recent, wellknown examples include Carson and Mitchell (1986) and Smith and Desvousges (1986). Some researchers, however, are skeptical about the usefulness of the information collected in surveys, simply because it is so difficult 

to describe all the relevant parameters of the item to be valued (Fischhoff 

and Furby 1988). Administrators have long recognized, for example, that 

willingness to pay (WTP) for flood control is greater after a significant flood 

event (Stitt 1986). It is questionable whether preflood surveys can capture 

postflood WTP. Similarly, pre- and postdrought surveys about WTP for a 

reliable water supply might obtain very different results. Hobbs et al. (1989) 

concludes that CV is at the initial implementation state, and that many 

methodologic problems must be solved before these tools can be routinely 

used but that even attempts to place values on human lives can be useful. 

This paper addresses one of the methodologic considerations that confront 

analysts who use CV, namely, how decisions to treat outliers and dertain 

zero bids can affect estimates of total benefits. Analysts know that some 

respondents will be unwilling or unable to participate meaningfully in surveys and will provide unreasonable or arbitrary responses to questions about 

WTP (Smith and Desvousges 1986). Hence, they frequently censor so-called 

"protest zeroes" and outliers. Protest zeroes are statements of zero WTP 

from individuals who presumably value the good positively. Outliers are 

any bids that seem unlikely given presumptions about the underlying distAsst. Prof., Ping. Program, School of Public and Envir. Affairs, Indiana Univ., 

BS 4068, 801 West Michigan, Indianapolis, IN 46202. 

Note. Discussion open until July 1, 1994. To extend the closing date one month, 

a written request must be filed with the ASCE Manager of Journals. The manuscript 

for this technical note was submitted for review and possible publication on April 

23, 1992. This technical note is part of the Journal of Water Resources Planning and 

Management, Vol. 120, No. 1, January/February, 1994. 9 ISSN 0733-950X/ 

94/0001-0121/$1.00 + $.15 per page. Technical note No. 3926. 

121 
tribution of bids. Techniques used to treat these bids are important because 

they can affect estimates of aggregate WTP (Milon 1989). It is argued here 

that the analyst's choice of techniques should follow from the conceptual 

model for the CV application. 

MODELS FOR CONTINGENT VALUATION APPLICATIONS 

The conceptual model that has guided the development of CV has been 

the private market, and CV exercises typically have been designed to emulate markets for private goods and services. The premise of CV is that 

surveys can be used to elicit accurately maximum WTP for public goods; it 

is assumed that bids represent what the good is worth to the respondents. 

Individual bids must be aggregated; the sample mean bid is usually extrapolated to the total population to obtain estimates of total WTP. A decision 

rule, such as positive net present value in which total benefits are based on 

an estimate of mean WTP, implies that projects valued highly by a small 

minority of the population could be approved. If, however, certain bids do 

not represent true valuations, the model breaks down, and the troublesome 

bids must be censored. This has led to the development of techniques for 

differentiating protest zeroes from valid zeroes and to debate over the treatment of outliers. 

An alternate conceptual model for CV experiments is a political market 

(Mitchell and Carson 1989). This is generally more conservative: the appropriate decision rule is the median bid, which, given the shape of most 

WTP distributions, means that estimated WTP will be lower. The numbers 

of zero bids and outliers still are important because they can affect the 

estimate of the median. Conceptually, however, the analyst is faced with 

somewhat different problems. Rather than being concerned mainly whether 

a zero response represents an individual's true valuation of a public good, 

the analyst must determine whether a zero bid accurately reflects how a 

respondent might react if confronted in a referendum with a tax proposal. 

Individuals may be unwilling to pay more for an environmental program-- 

even though they value that program highly--because, as is the case with 

most public goods, they have no direct control over the establishment of 

priorities or the allocation of funds to individual items in the bundle of goods 

paid for with tax dollars. Hence, in a political market, more zeroes will be 

accepted at their face value. With outliers, because the exact value of high 

bids at the tail of the distribution is irrelevant to the estimation of the median 

bid, the method of treatment becomes less important. 

The political and private market models can be thought of as end points 

on a continuum; estimates of WTP consistent with each establish bounds 

on the range of benefits. From a methodological perspective, the implications of choosing a political market as a model are intuitively clear and 

desirable: the problems of controlling for protest bids and outliers are reduced. The importance of consistency between the analysts' conceptual 

models and choice of techniques is illustrated in the following application. 

APPLICATION: WTP FOR STORM-WATER MANAGEMENT 

A CV survey was administered by mail in August 1989 to 1,600 homeowners in Baltimore County, Md. to determine their WTP for storm-water 

programs to help achieve Chesapeake Bay nutrient-reduction Objectives. 

The recipients were selected randomly from 133,344 owner-occupants listed 

in county property-tax files. Fifty-eight percent of the questionnaires were 

122 
returned. The response rates to individual questions on the completed questionnaires ranged from 77% to 97%. The sample is generally believed to 

be representative of homeowners in Baltimore County. Details of the survey 

and additional information about results are provided in Lindsey (1992). 

The elicitation question was phrased: What is the most you would be 

willing to pay each year in storm-water utility charges before you would 

vote no in the referendum? The answer option consisted of a matrix of 

values ranging from $0.00 to $1,000.00 called a payment card. Approximately 89% of the respondents answered the WTP question. The distribution of WTP (n = 824) responses is nonnormal (Lilliefors statistic = 

0.298), positively skewed (skewness coefficient = 5.742), and leptokurtic 

(kurtosis coefficient = 52.607). The unadjusted, mean annual bid is $42.00 

and the median is $20.00. The mode is $0.00 (33% of the responses). The 

highest bid was $1,000.00; 26 bids were above $200.00. 

TREATMENT OF PROTEST ZEROES AND OUTLIERS 

To identify protest zeroes, researchers ask respondents to explain their 

zero bids. Choices of answers typically include: "that is what it is worth to 

me"; "did not want to place dollar value"; or "am opposed to all new 

taxes." Zeroes explained by reasons other than "that is what it is worth to 

me" or "all I can afford" are often classified as protest bids. In this application, homeowners were provided eight alternative explanations for zero 

bids. A plausible approach to classification of each under both private and 

political market models is presented in Table 1. For some, the appropriate 

classification is not obvious. 

Explanations 02, 03, and 06 (did not want to place dollar value, objected 

to way question was presented, and not enough information) each imply 

that respondents might have answered differently had they more information 

or if the question had been structured differently. Hence, an analyst could 

TABLE 1. Survey Question: If You Are Not Willing to Pay any New Charges to 

Control Storm Water and Improve Water Quality in Chesapeake Bay, What Is the 

One Most important Reason You Answered the Way You Did? 

lumber 

(1) 

01 

02 

03 

04 

05 

06 

07 

08 

Number of 

Choice of answers answers Percent 

(2) (3) (4) 

That is what it is worth 9 3 No 

to me 

Did not want to place 15 6 Yes 

dollar value 

Objected to way question 3 1 Yes 

was presented 

Too much waste in 75 28 Perhaps 

government 

Am opposed to all new 51 19 Perhaps 

taxes 

Not enough information 13 5 Yes 

Industry should pay 66 25 Perhaps 

Other (please explain) 36 13 Perhaps 

Total responses 268 a 100 -- 

Classified as Protest Bid? 

In model of 

private-goods In model of 

market political market 

(5) (6) 

No 

Perhaps 

Perhaps 

No 

No 

Perhaps 

No 

No 

aThere were 274 bids of zero WTP; 98% of the respondents who stated zero willingness to pay 

answered this question. 

123 
conclude that zeroes explained by these answers are not accurate representations of true WTP and that, regardless of the conceptual model, these 

should be classified as protest zeroes. Zero, however, may be the most likely 

bid from an individual who is uninformed or uncomfortable with a question 

on a referendum ballot. Hence, the rationale for second-guessing the respondent is weaker if the conceptual model is a political market. 

Twenty-five percent of the respondents bid zero because "industry should 

pay" (answer 07). Interpretation of this explanation is more troublesome 

for it provides no insight into whether respondents value the good, only 

that they believe someone else should pay. Classification of these bids as 

protest zeroes seems plausible if the conceptual model is a private market 

and the objective of the analysis is to determine true WTP. It seems inappropriate, however, if the model is a political market and one believes 

that individuals who say that someone else should pay for a program would 

vote zero because of their convictions. 

Almost half (47%) explained their zero bids by responding that there is 

"too much waste in government" or that they are "opposed to all new 

taxes" (answers 04 and 05, respectively). These responses also provide no 

information about true valuation of storm-water programs, but they do 

provide information about preferences for more government programs. 

Treatment of these as protest zeroes may be acceptable if the conceptual 

model is a private market. If the model is a referendum, however, there is 

little justification for censoring them. 

The least ambiguous response is "that is what it is worth to me" (answer 

01). Only nine individuals explained their bids with this answer. This is 

somewhat troubling: a literal interpretation is that just 3% of the zero bids 

are accurate. 

Given the eight possible explanations, there are 256 (28 ) possible subsets 

of protest zeroes and potential estimates of WTP. The effects of different 

approaches to classification are illustrated with four subsets in Table 2. The 

benchmark from which these approaches are evaluated involves extrapolation of the sample mean ($42.00) and median ($20.00) without adjustment 

for protest zeroes. Estimates of total annual WTP corresponding to these 

point estimates are $5,600,000 and $2,700,000, respectively. 

Another option would be to classify as protest bids only those respondents 

who stated explicitly that they did not want to value the good, that they 

objected to the question, or that they lacked sufficient information (answers 

02, 03, 06, respectively). Deletion of these 31 bids increases the estimates 

TABLE 2. Effects of Deleting Protest Bids on WTP 

Distribution statistics 

(t) 

Number of cases 

Median ($) 

Mean ($) 

Variance 

Standard error 

Total WTP: extrapolating median 

Total WTP: extrapolating mean 

POLITICAL MARKET PRIVATE MARKET 

Answers Deleted as Protest Bids 

None 

(benchmark) 

(2) 

824 

20 

42.00 

5,724 

2.6 

$2,700,000 

$5,600,000 

02-03.06 02-03.06-07 

(3) (4) 

793 727 

25 

43.60 47.60 

5,878 6,223 

2.7 2.9 

$3,300,000 

$5,800,000 $6,300,00 

02-08 

(5) 

665 

61.20 a 

7172 

3.0 

$8,200,000 

aSignificantly different from benchmark and other cases at 95% confidence interval. 

124 
of mean and median WTP to $43.60 and $25.00, respectively (Table 2). 

Corresponding estimates of total WTP increase to $5,800,000 and $3,300,000. 

A third approach would be to delete, in addition to these, the 66 zero 

bids explained by the response "industry should pay." This approach (which 

requires the analyst to exercise greater judgment and, as noted previously, 

seems appropriate only for a private market model), increases estimates of 

annual mean and total WTP to $47.60 and $6,300,000, respectively (Table 

2). 

An approach that implies strict adherence to a private goods model involves deletion of all the zero bids, except those explained by "that is what 

it is worth to me." This increases estimates of the annual mean and total 

WTP to $61.20 and $8,200,000, respectively (Table 2). The estimate is 

significantly different from the estimates in the other cases, the differences 

among which are not significant. What is clear from this example is that 

analysts whose frame of reference is a political market are required to make 

fewer interpretations of data. 

Outliers pose comparable problems. Analysts must first define and then 

decide what to do with them. Researchers have identified outliers by 

ad hoc, arbitrary standards [e.g., as bids greater than 10 standard deviations 

from the sample mean bid (Rowe et al. 1980) or as bids greater than some 

percentage of a respondent's annual household income]; through the use 

of a robust statistical estimator, such as the a-trimmed mean (Mitchell and 

Carson 1989); or through the use of regression diagnostics, such as the 

DFBETA statistic (Smith and Desvousges 1986), or the analysis of studentized residuals. 

Use of a-trimmed mean procedures (Appendix I) with CV data poses 

several problems, mainly because bids are constrained to be zero or greater. 

It is not uncommon that, as in this case, a bid of zero is the mode. To use 

the procedure, then, requires judgments about the zeroes to be deleted. 

Another problem involves setting the a-level. Analysts typically recommend 

values of 5-10%, although this is subjective. Other problems occur due to 

clusters of bids at round numbers. 

The DFBETA statistic is a measure of the influence of an observation 

on the coefficients in an ordinary least-squares (OLS) regression (Belsley 

et al. 1980). The approach, which involves calculating the change in each 

estimated coefficient that results from deletion of an observation, can become cumbersome if a regression equation includes more than one independent variable (Moser and Dunning 1986). In addition, the overall effect 

of eliminating observations is difficult to determine because the elements 

of the statistic are not commensurable (Fox 1984). 

Another regression procedure for determining outliers involves analysis 

of the studentized residuals from a bid function (Belsley et al. 1980; Fox 

1984; Wilkinson 1988). Studentized residuals are estimated by standardizing 

the residuals from OLS regression and "successively omitting each observation, calculating its residual based on the regression coefficients obtained 

for the remaining sample, and dividing the resulting residual by its standard 

error" (Fox 1984) (see Appendix II). With this approach, model specification and methods of compensating for item nonresponse are important 

procedural considerations. 

Once outliers have been identified, the problem of what to do with them 

remains. Most analysts censor them. It is not difficult, however, to envision 

cases in which the choice of technique could become a factor in whether a 

hypothesis is accepted or rejected or whether a project is judged efficient. 

125 
In controversial applications, if outliers are deleted simply because the residuals from a regression are unusually large, critics can charge that the 

analyst has censored from the sample those individuals who do not conform 

to narrow conceptions of economic behavior. 

As with protest bids, one perspective is that treatment of outliers is more 

important when the conceptual model for the application is a private market 

(i.e., when the point estimate of interest is the mean rather than the median). 

Adoption of a political market minimizes the problem of outliers if, as will 

be the case in most applications, the outliers comprise high bids at the end 

of the distribution and it is assumed that the bids accurately reflect some 

positive WTP greater than the median value. This is because the absolute 

magnitudes of bids greater than the median do not influence the estimate 

of the median value. 

The effects of three different approaches to controlling outliers in Baltimore County stormwater survey are illustrated in Table 3. The ad hoc 

procedure of culling bids greater than 10 standard deviations above the 

mean (or of bids greater than 0.5% of reported income) results in deletion 

of just one $1,000 bid. The resultant mean ($40.80) is not significantly 

different than the benchmark mean ($42.00, given in Table 3). Selection of 

the cutoff is arbitrary, however. 

Use of the a-trimmed mean approach (a = 4%) results in deletion of 33 

bids greater than $150.00 (and 33 zero bids). The a-level of 4% was determined by inspection; zero bids were deleted randomly. The resultant estimate of mean willingness to pay ($32.30) is significantly different than the 

mean estimated from the uncensored data (Table 3). 

The third approach involves identification of outliers based on the distribution of studentized residuals from an OLS regression. Analysis of studentized residuals for the bid function presented in Table 4 results in the 

identification of 26 outliers, each of which is greater than or equal to $200.00. 

Deletion of these bids yields a mean annual WTP of $32.50, which is also 

significantly different than the benchmark (Table 3). As noted, however, 

the decision to censor responses that do not fit a hypothesized model can 

be problematic. In this case, it is conceivable that some individuals would 

be willing to pay $200,00 or more annually to clean up urban storm water. 

Under a private-market model, a reasonable approach with these data 

might be to retain all zero bids but censor outliers identified in a regression 

framework. This results in an estimate of annual mean WTP of $32.50, 

TABLE 3. Effects of Deleting Outliers on WTP 

Distribution 

statistics 

(1) 

PRIVATE MARKET 

Technique Used to Identify Outliers 

Political 

market 

(2) 

,Bench mark 

(3) 

Outliers = 

bids > 10 SD 

> Mean WTP 

(4) 

cx-trimmed 

mean (4%) 

(5) 

Studentized 

residuals 

(6) 

Number of cases 824 824 823 759 798 

Median/mean 20 42.00 40.80 32.30 ~ 32.50 a 

Variance -- 5,724 4,613 1,502 1,709 

Standard error -- 2.6 2.4 1.4 1.5 

Total WTP $2,700,000 $5,600,000 $5,400,000 $4,300,000 $4,300,000 

aSignificantly different from benchmark at 95% confidence interval. 

126 
TABLE 4. Ordinary Least-Squares Bid Function for Identifying Studentized Residuals 

Variable a Coefficient T-Statistic 

(1) (2) (3) 

Constant 

Household income 

Years of education 

Use of the bay for outdoor recreation (dummy) 

Ranking of bay water quality 

Age over 65 (dummy) 

Rating of likelihood of success of bay programs 

Goal statement (dummy) 

User charge payment vehicle (dummy) 

Payment reminder (dummy) 

Questionnaire return date 

-45.1498 

0.0005 

2.7081 

7.5159 

-0.1932 

-4.0902 

4.8946 

1.3508 

- 1.8356 

- 8.5265 

0.0227 

-3.11 

5.60 

3.22 

1.62 

-0.17 

- 0.69 

4.86 

0.30 

-0.41 

-1.91 

0.10 

"Dependent variable is the willingness to pay for storm-water programs. 

Note: adjusted R 2 = 0.104; F statistic = 11.75; and N = 924. 

which corresponds to an estimate of annual benefits for all owner-occupants 

of $4,300,000. The rationale for retaining all zeroes is that, except for the 

extreme case in which all zeroes are censored, except those explained by 

answer 01 ("that is what it is worth to me"), estimates of WTP do not differ 

significantly. This judgment is pragmatic and involves balancing the value 

of the "better" estimate that might be obtained if some subset of zeroes 

(e.g., those associated with answers 02, 03, and 06) were censored against 

the loss of information that occurs with the decision to censor. A rationale 

for using regression to identify outliers is that the approach is consistent 

with a private-market model. 

If, alternatively, a political market is assumed, the issues posed by protest 

zeroes and outliers are reduced. If no outliers are censored, the median bid 

is $20.00 or $25.00 per year, depending on assumptions made about zero 

bids. Corresponding estimates of total WTP are $2,700,000 and $3,300,000, 

respectively. 

SOME OBSERVATIONS 

It is clear that the analyst's frame of reference or conceptual model for 

analysis may dictate use of procedures that, when implemented, significantly 

affect estimates of WTP. Adherence to a private-market model rather than 

a political market model will generally result in a higher estimate of WTP. 

Adoption of a political market will require the analyst to make fewer interpretations of ambiguous responses. Although there always will be ~imbiguity 

in how to classify certain responses, the issue in some applications may be 

moot if alternate classifications do not significantly change estimates of 

WTP. Careful presentation of both assumptions and methodology is essential to inform decision-makers adequately. Planners should take care that 

analytic techniques are consistent with conceptual models for any particular 

CV application. 

127 
APPENDIX I. a-TRIMMED MEAN 

The a-trimmed mean is defined by 

= S(nct)+l -~ . . . 71- X[i,t_(not)] 

n - 2(mx) ................................ (1) 

where Xi = set of ordered WTP observations; i = 1 . . . n; n = number 

of observations in the sample; and ct = percentage; 0.0 -< et < 0.50. 

APPENDIX II. STUDENTIZED RESIDUALS 

Studentized residuals follow a t-distribution with N-k-2 degrees of freedom and can be estimated by 

~/n - k- 2 

E7 = L .................................. (2) 

~/n - ~ - 1 - E~ 

where/~i = least-squares residual standarized by dividing each least-squares 

residual by its estimated standard deviation; E* = studentized residual; n 

= number of observations; and k = number of regressors excluding the 

constant. 

APPENDIX III. REFERENCES 

Belsley, D. A., Kuh, E., and Welsch, R. E. (1980). Regression diagnostics: identifying 

influential data and sources of colinearity. John Wiley and Sons, Inc., New York, 

N.Y. 

Carson, R. T., and Mitchell, R. C. (1986). "The value of clean water: the public's 

willingness to pay for boatable, fishable, and swimmable quality water." Discussion 

paper QE85-08, Resources for the Future, Washington, D.C. 

Economic and environmental principles and guidelines for water and related land 

resources implementation studies. (1983). U.S. Water Resour. Council, Washington, D.C. 

Fischhoff, B., and Furby, L. (1988). "Measuring values: a conceptual framework 

for interpreting transactions with special reference to contingent valuation of visibility." J. Risk and Uncertainty, 1(2), 147-184. 

Fox, J. (1984). Linear statistical models and related methods. John Wiley and Sons, 

Inc., New York, N.Y. 

Hobbs, 13. F., Stakhiv, E. Z., and Grayman, W. M. (1989). "Impact evaluation 

procedures: theory, practice, and needs." J. of Water Resour. Ping. and Mgmt., 

115(1), 2-21. 

Lindsey, G. (1992). "An experiment in contingent valuation: willingness to pay for 

storm water management," PhD thesis, The Johns Hopkins Univ., Baltimore, Md. 

Milon, J. W. (1989). "Contingent valuation experiments for strategic behavior." J. 

Envir. Economics and Mgmt., 17(3), 293-308. 

Mitchell, R. C., and Carson, R. T. (1989). Using surveys to value public goods: the 

contingent valuation method. Resources for the Future, Washington, D.C. 

Moser, D. A., and Dunning, M. C. (1986). "Volume II: a guide for using the 

contingent value methodology in recreation studies." National economic development procedures manual--recreation, U.S. Army Corps of Engrs., Inst. for 

Water Resour., Fort Belvoir, Va. 

Rowe, R. D., d'Arge, R. C., and Brookshire, D. S. (1980). "An experiment on the 

economic value of visibility." Envir. Economics and Mgmt., 7(1), 1-19. 

128 
Smith, V. K., and Desvousges, W. H. (1986). Measuring water quality benefits. 

Kluwer-Nijhoff Publishing, Boston, Ma. 

Stitt, T. A. (1986). "Establishing a storm water management utility." Public Works, 

117(9), 126-127, 150, 152, 154, 156. 

Wilkinson, L. (1988). SYSTAT: the system for statistics. SYSTAT, Inc., Evanston, 

IU. 

129