From: bks@alfa.berkeley.edu (Bradley K. Sherman)
Newsgroups: talk.politics.misc,alt.politics.elections,alt.politics.bush,alt.politics.clinton,alt.politics.perot
Subject: The Mathematics of Error
Date: 15 Oct 1992 22:50:57 GMT
Organization: Psephologists R Us
Distribution: world
NNTP-Posting-Host: alfa.berkeley.edu


I asked a friend (Brad Palmquist, blp@hotel.berkeley.edu) this question:

If you saw a poll result like this:
   Clinton   39%
   Bush      33%
   Perot     14%
   Undecided 14%
and the legend said Margin of Error +- 4%, would you say that Clinton's
lead was outside the margin of error?

Here is (with permission) his response:

|It's simpler first to deal with the two-candidate situation,
|let's say Clinton/Bush only.  As an example pick the same spread of 6%
|so we have:
|
| Clinton 54%
| Bush    48%
|
|This is a binomial distribution which is specified by a single parameter, P,
|namely the probability of "success" (in the non-normative sense where we
|can arbitrarily designate one outcome as success).  Let us call the probability
|of getting a Clinton supporter in our sample P.
|
|Then the variance of the population is P(1-P).  The variance of the sample
|mean (call it little P, or p) is then P(1-P)/n, where n is the sample size.
|This can be approximated by the sample estimate of the variance.  Plugging
|in p, the sample estimate of P, and adjusting for a slight bias we get
|p(1-p)/(n - 1) as the unbiased estimate of the variance of p.
|
|You can see why so many surveys report "margins of error" of +-3%.
|Many of the proportions being estimated fall close to .5 and many of the sample
|sizes are roughly 1000. To get +-.03 exactly set the sample size to be 1112
|and say true P is .5.  Then the sample variance of the mean is: .5(.5)/1112.
|The standard error (square root of the variance) is .5/33.3.  Recall binomial
|distribution approaches normal pretty quickly as long as P is bounded away from
|0 or 1.  In fact 30 or 40 cases is usually enough, so 1000 some will be very
|well approximated by the normal distribution.  In that case we choose two
|standard deviations to get a .95 confidence interval which is:
|2(.5)/33.3 = .03.
|
|
|For the poll you cite the confidence interval was +-4%.  I need to know the
|sample size, but I can make a guess at it by working backwards.
|Since they only provided one "margin of error" I'll assume they were
|basing it on a sample estimate of 50%.  (In fact the margin of error is
|less for Perot than for the other two because .14 is farther away from
|.5 where the variance is at its maximum.) 
|	
|	n = (1 / .04) ^ 2 = 625
|
|In a two-way race the question can be phrased as "Is the difference
|p - q statistically significant?"  Now p - q = p - (1 - p) = 2p - 1.
|The variance of this quantity is 4 times the variance of p alone and
|its standard error is twice as big.  Hence we could say that the difference
|between Clinton and Bush is 6% +- 8%.  This is not significant at the .95 level.
|
|It is interesting to see what the "p-value" of this difference is.
|The p-, or prob-, value is the probability of finding the sample value actually
|found, assuming the null hypothesis to be true.  Here H sub 0 is that the
|difference is 0, that the two are tied.  If this were really true the
|probability of finding a difference as large as 6% is .13.  So if somebody
|set the significance level at .9, the difference would still be insignificant.
|At .87 it would be significant.  Now really, our best guess is, of course,
|that Clinton is ahead.  Still, even if they were tied we would expect to
|get one or the other ahead by as much as 6% in about 13 polls out of 100.
|
|Now moving on to the multi-candidate situation, the sample distribution is
|a multinomial. The distribution is characterized by k-1 parameters where
|k is the number of classes (in this case candidates).  Obviously the
|binomial is a multinomial with 2 classes, and hence, 1 parameter.
|In addition to variances we now have to be concerned with covariances.
|
|For a simple pairwise comparison, e.g. the difference between Clinton (39%)
|and Bush (33%), we can use a general variance formula for linear functions
|of two variables, namely Var(aX + bY) = a^2 Var(X) + b^2 Var(Y) + 2ab Cov(X,Y).
|The covariance of the proportions in two classes is -P sub 1 * P sub 2 /n
|and its unbiased sample estimate is -p sub 1 * p sub 2 /(n - 1).
|
|Let C stand for Clinton's proportion and B for Bush's:
|
|	(n - 1) * Var(C - B) = .39(.61) + .33(.67) + 2(.39)(.33) = .7164
|
|So Var(C - B) = .00114807 and SE(C - B) = .033388.  Taking 2 standard errors
|gives a confidence interval of +-.068.  The p-value is .077.
|
|
|Note that if you just threw away the results from respondents who didn't
|choose either Bush or Clinton you would have a sample size of 450,
|C = .54, B = .46, C - B = .08 +-.094, p-value = .089.
|
|    --Brad Palmquist (blp@hotel.berkeley.edu)

[Please send only MATHEMATICAL quibbles to Palmquist.  Insults to me. --bks]
----

    --Brad Sherman

God make the integers, all else is the work of man.  --Kronecker
Xref: sparky talk.politics.misc:51139 alt.politics.elections:16263 alt.politics.bush:8747 alt.politics.clinton:9888