Statistical Foundations of Election Poll
By: Tony Peng
Tags: Binomial Distribution, Confidence, Poll, Statistics, Data Science

Statistical Foundations of Election Poll

Photo by Wojciech Portnicki on Unsplash

In this blog, I try to answer the question,

“How can a poll from a sample of, say, 1,000 likely votes, estimate the support of a candidate in a nationwide race?”

by discussing the following topics,

The crucial goal/assumption of a poll of being able to select a random sample of voters, From the concepts of a binomial experiment to the final equations that describe the sampling distribution of the sample proportion, and finally, The adequate interpretations of a poll’s results with confidence level and its associated margin of error.

I will go through and explain all the concepts above and derive all relevant equations if appropriate.

A crucial assumption of a binomial experiment (or Bernoulli experiment) is that the chance of something happening is a constant in an experiment. For example, the flip of a fair coin has a 50% chance of landing on the head as well as on the tail. And this chance does not change between flips. If we perform a test consisting of 1,000 trials (flips), we could expect that close to half of the coins would land on the head. In a test, we may see 482 times a coin on the head, and the other 518 times on the tail (with a head proportion of 0.482). If we repeat the test again the result may be a little different, say, with 505 times on the head and the other 495 times on the tail (with a head proportion of 0.505). Let’s ask a question:

“How can we figure out the true chance of a coin landing on the head in a single flip (=0.5) by the results of the above flipping tests?”

The answer is, as long as the chance of landing on the head in each flip is a constant (= 50%), the more the number of tests (each test consisting of 1,000 flips) we perform, although the head proportion of each test will be different, we could expect that the mean of the head proportions of all tests will approach the true chance of 0.5 when the number of tests increases to a very large number (this is an example of the so-called central limit theorem). On the other hand, we could also expect that, if we only perform one test but increase the number of flips to a very, very large number, the head proportion could also approach the true chance of 0.5 (this is the so-called law of large numbers).

Ok, but how is the above relevant to a poll? Let us look at an analogy between the test of flipping a fair coin and that of withdrawing a ball from a bag that contains a large number of white balls (= coins landing on the head) and red balls (= coins landing on the tail) whose numbers are equal. The task now is to estimate the true white-ball proportion (= 0.5) by sticking your hand inside this imaginary bag that cannot be seen through and picking one ball out at a time (= flip a coin) to reveal its color until certain sample size (= the total number of flips in a test) is reached. Ideally, we are supposed to put the ball back to the bag after a pick so that the ball proportion is not changed by a pick. However, if the number of balls that are taken out from the bag (= sample size) is much smaller than the total number of balls in the bag this requirement can be relaxed.

In order for the statistics of the flipping of a fair coin to be fully applicable in the case of withdrawing a ball from the bag, a crucial assumption is that the balls in the bag need to be completely and randomly mixed so that a random ball can be chosen each time from the bag. In another word, no matter which portion of the bag you stick your hand to, the balls surrounding your hand available for choice are always 50% white and 50% red, which corresponds to the true white-ball proportion of 50% in the bag.

Here, estimating the true white-ball proportion in a bag by withdrawing one ball at a time is just like estimating the true support proportion of a candidate in the entire body of eligible voters by polling one voter at a time.

We assume that a voter has made up her/his mind to either support or not support a candidate, so the outcome is binary. If we were to be able to select a random sample of voters in a poll, the chance of any voter in this poll who supports a candidate would be a constant, which is equal to the true support proportion of this candidate in the entire body of eligible voters.

It can be seen that, if the balls in the bag are completely and randomly mixed, the random sample assumption is always valid regardless of the total number of balls in the bag, may it be 10,000, or 100,000, or one million, etc. This is the reason why, if a random sample of voters can be selected, a poll conducted on 1,000 voters will be able to estimate a candidate’s support in a nationwide election with, say, 240 million eligible voters.

With the random sample assumption in mind, let’s examine the statistical foundations of a poll by first looking into a binomial experiment (Bernoulli experiment) in detail.

As discussed earlier, a binomial experiment has only two possible outcomes, e.g., a coin can only land on the head or on the tail after a flip, or a voter in an election can either support or not support a certain candidate. Another property of a binomial experiment is that the outcome of one trial does not affect the outcome of another trial, i.e., the chance of a specific outcome of a trial is a constant. For example, for a fair coin, the chance of landing on the head (a success) as well as on the tail (a failure) after a flip is 50% (p = 0.5), and this chance is the same for each flip. In the case of a poll, there is a true support proportion (= p) of a candidate in the entire population of eligible voters that we hope to estimate by a poll. Let’s assume p = 0.6 just for the sake of discussions. As was mentioned earlier, if we were to be able to randomly sample a voter from the entire population of voters, then the chance of this voter supporting a candidate (a success) is 60% (since p = 0.6). We also assume that the number of voters we select for a poll is much smaller than the total number of eligible voters so that this support proportion of 60% does not change when we poll another voter.

Let’s define a random variable Y that represents the outcome of a single binomial trial. If a coin lands on the head (chance = p) after a flip then Y = 1 and if it lands on the tail (chance = 1-p) then Y = 0. Or if a voter supports a candidate (chance = p) then Y = 1 and if a voter does not support the candidate (chance = 1-p) then Y = 0. From the earlier discussions, we know that if the trial is repeated many, many times the long-term proportion of success would approach the true chance of p. The mean of Y (= μ, the long-term expected value of Y), can be calculated using the chance of happening of each outcome as,

where yi is the possible outcomes of Y (y1 = 1 and y2= 0) and p1 the chance of y1 happening (= p) and p2 the chances of y2 happening (= 1-p). Here again, μ is the long-term expected value of Y after many trials.

The spread, or variance, of Y (Y = 1 or Y = 0) is defined as the expected value of the squared deviation of Y from its mean μ (= p), which can be expressed as,

where yi is the possible outcomes of Y (y1 = 1 and y2= 0) and p1 the chance of y1 happening (= p) and p2 the chances of y2 happening (= 1-p). The variance of Y expressed by Equation 2 is, again, the long-term expected value of the spread of Y after many trials.

As can be seen in Figure 1 below, which shows the variance of Y as a function of p, the variance is the largest when p = 0.5 and decreases symmetrically when p is away from this value.

Figure 1: Variance of Y as a function of p

So far we have discussed the repetition of one Bernoulli trial many times. Let’s now expand it to the repetition of, say, n Bernoulli trials as a group, many times. We are interested in the number of successes (defined here as a new random variable X) after a total of n Bernoulli trial. X could be the number of heads after a coin-flipping test that consists of n flips or the number of voters who support a candidate after polling a voter sample of size n. For example, if a coin is flipped 2 times (n = 2), then the number of heads (= X) could be either 0 (= no head), 1 (= one head and one tail), or 2 (= two heads) after a single test of 2 flips.

How about the mean of X? That is, what is the long-term expected value of X after a great number of tests or samples (each test/sample comprising n Bernoulli trials)? Since each Bernoulli trial is independent of each other, the expected value of X is just the sum of all the expected value of Y in n Bernoulli trials,

where E[Y] = μ = p from Equation 1.

Similarly, since the variance of a sum of independent random variables is the sum of the individual variances, the variance of X (= Var(X)) can simply be expressed as,

where Var[Y] = p(1-p) from Equation 2. Let’s check the validity of Equation 4 using a simple example of flipping 2 fair coins A and B as a group (n = 2) many times. When A = Head and B = Head then X = 2, and the long-term probability of this happening is 0.25 (=0.5 * 0.5). There are two cases when X could be 1, where A = Head and B = Tail, or A = Tail and B = Head, and the long-term probability of this happening is 0.5 (= 0.5 * 0.5 + 0.5 * 0.5). Lastly, X = 0 when A and B are both on the tail with a long-term probability of 0.25 (=0.5 * 0.5). As a result, Var[Y] can be calculated from the first principle as,

where Xi = 2, 1, and 0 and pi = 0.25, 0.5, and 0.25 when i =1, 2, and 3, respectively, n = 2, and p = 0.5. It can be seen that the Var(X) of 0.5 calculated using Equation 5 is the same as the one calculated using Equation 4, which is also 0.5 (= 2 * 0.5 *(1–0.5)).

The measure of X’s actual spread from its mean (= np), i.e., its standard deviation (= SD), which has the same unit as the number of trials, can be expressed as,

where Var(X) = np(1-p) from Equation 4.

Now we know the mean and standard deviation of X after a great number of tests/samples, each with a test/sample size of n. But what does this discrete binomial distribution of X actually look like? It is safe to assume that, when n is not small or p is not close to 0 or 1, a normal distribution, with the same mean and standard deviation derived in Equations 3 and 6, respectively, describes quite precisely the binomial distribution of X. For example, if p = 0.6, we can safely describe the discrete binomial distribution of X as a normal distribution when the n value is equal to or larger than 13. The n is derived by the criteria equations np ≧ 5 and n(1-p) ≧ 5. If p = 0.5 then the threshold value of n = 10. Since the number of likely voters in a poll is much larger than the above n values it is safe to use a continuous normal distribution to approximate the discrete binomial distribution of X in our discussions.

Photo by Jaime Dantas on Unsplash

So far we have treated the distribution as a function of X, the total number of successes after n trials. What if our interest is to know the mean and standard deviation of the proportion of successes rather than the total number of successes? For example, we are interested in the proportion of voters who support a candidate rather than the total number of these voters in a poll. Quite simply, we just need to rescale the x-axis of the distribution by dividing X by the sample size of n to get a new proportion axis Xp (= X/n). Similarly, the mean in Equation 3 and the standard deviation in Equation 6, since they have the same unit as X, can be rescaled as,

where μp is the mean and σp the standard deviation of the proportion of likely voters that support a candidate in a poll. We can call this new distribution the sampling distribution of the sample proportion.

Let’s look at an example of this new distribution. First, let’s assume that we somehow know that 60% of the voters among the entire population of registered voters support a certain candidate (p = 0.6). Second, we assume that we were to be able to randomly select a sample with a sample size of 1,000 voters in a poll from which a value of the random variable Xp can be calculated. We then (using the same polling method) repeat the poll for, say, 10,000 times, to obtain many different values of Xp so that a trace of relative frequency, i.e., probability (= frequency of occurrence at Xp divided by 10,000), as a function of Xp can be plotted. From Equation 7 we know that the mean (= μp) of this plot is 0.6 and the standard deviation (= σp) is 0.015 (= sqrt(0.6 * 0.4 / 1000)) and the distribution can be approximated using a normal distribution. The plot would look somewhat like the one shown in Figure 2 below.

Figure 2: Sampling distribution of the sample proportion

For a normal distribution, we know that there is a 95% chance (we call this probability the confidence level, that is, we are 95% confident) that a sample (= Xp) will be within ± 2 standard deviations (= σp) from the center at p, that is,

If we multiply Equation 8 by -1 and rearrange the order of the terms, Equation 8 can be expressed as,

Equation 9 implies that we can interpret the statement “There is a 95% chance that a sample (= Xp) will be within ± 2 σp from the true population proportion p.” as “There is a 95% chance that the true population proportion p will be within ± 2σp (= margin of error) from the Xp of a single sample.” The new interpretation is great since as long as we have some way of estimating the value of σp without first knowing p, we will be able to state, with a certain degree of confidence, the location of true p relative to the Xp from a single random sample we collect.

Now, how do we estimate σp using Equation 7 without first knowing the true proportion value of p? In fact, we can safely replace the true proportion of p in Equation 7 with the proportion from a single sample we collect (= Xp) to approximate σp as,

where SExp is called the Standard Error (SE) we calculate using the proportion Xp from a single sample. When n is large enough, the SExp calculated using the above Equation approximates the actual standard deviation σp calculated using Equation 7 quite nicely.

Let’s look at an example (using the trace in Figure 2) to compare the SE calculated using Equation 10 and the σp calculated using Equation 7 when two quite extreme values of Xp at ± 3 standard deviations (= σp) are used. We know that for a normal distribution there is a 99.7% chance that a sample (= Xp) will be within ± 3 standard deviations (= σp) from the center at p (= 0.6). That is, there is only a 0.3% chance that Xp is either smaller than 0.555 (= 0.6 – 3 * 0.0155) or greater than 0.645 (= 0.6 + 3 * 0.0155). With Xp at the lower limit of 0.555 the SExp calculated using Equation 10 is 0.0157 (= ± 2 SExp error margin of ± 3.14%) and with Xp at the upper limit of 0.645 the SExp is 0.0151 (= ± 2 SExp error margin of ± 3.02%), which are both quite close to the real σp value of 0.0155 (= ± 2 σp error margin of ± 3.10%).

The confidence level of 95% in the statement “There is a 95% chance that the true population p will be within ± 2 SExp from the Xp of a single random sample.” basically states that if we repeat the poll for, say, 100 times, this method produces margin-of-error intervals (= ± 2 SExp) that capture the true population proportion (= p) in 95% of the samples, i.e., in 95 polls. Figure 3 below visualizes this concept of confidence level.

Figure 3: Visualization of the confidence level where the margin of error of a sample that either capture or miss the true population proportion of p = 0.60

In Figure 3, a dot represents the Xp value of a single random sample and its associated bar represents the interval that corresponds to the 95% confidence level margin of error (= ± 2 SExp, where SExp is calculated using Equation 10 with the Xp of the sample). The true population proportion at p (= 0.6) is what we want to estimate.

There are two colors of dots and their associated error bars, i.e., green and red. As can be seen in Figure 3, most of the error bars, which are all in green, do capture p, while there are only two bars in red that miss p.

95% confidence level says that we are confident that if we repeat the sampling for, say, 100 times, in about 95 samples the error bars (the green bars) will capture p, while only in about 5 samples the error bars (the red bars) will miss p.

Whether an error bar capture p or not, we do not have any confidence about the exact location of p with respect to the error bar. For example, if in a random sample of 1,000 likely voters we found that a candidate has a 57% support proportion (Xp = 0.57). Then using Equation 10 we can calculate the SExp as 0.0157. If we choose a confidence level of 95%, then the margin of error (= ± 2SExp) can be calculated to be ± 3.14%. So, we are 95% confident that the true support proportion (= p) of this candidate is between about 53.9% (= 57-3.14 %) and 60.1% (= 57+3.14 %). On the other hand, there is a 5% chance that the candidate’s true support (= p) is either less than 53.9% or greater than 60.1%. In any of the above cases, we really do not have any confidence of the exact location of the true support of this candidate (= p) with respect to this single random sample of Xp (= 57%) or the confidence interval from 53.9% to 60.1%.

In conclusion, the closer the sample of a poll to a random one, the more applicable the statistical interpretations derived from Binomial trials. With only one random sample of, say, 1,000 voters, we could have a certain degree of confidence that the true support proportion of this candidate among the entire body of eligible voters, regardless of how big the body is, is within a margin of error that is calculated using the support proportion from the sample of this single poll alone.

In this blog, the concept of “sample proportion” is discussed. If you are also interested in its counterpart “sample mean” you can found the discussions here.

I learn a lot by writing this blog and I hope it benefits you too.