JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
, VOL. , NO. , –, Applications and Case Studies
https://doi.org/./..

Disentangling Bias and Variance in Election Polls
Houshmand Shirani-Mehra , David Rothschildb , Sharad Goela , and Andrew GelmancaDepartment of Management Science & Engineering, Stanford University, Stanford, CA; b Microsoft Research, New York, NY; c Department of Statistics
and Department of Political Science, Columbia University, New York, NY

ABSTRACT

It is well known among researchers and practitioners that election polls suffer from a variety of sampling
and nonsampling errors, often collectively referred to as total survey error. Reported margins of error typically only capture sampling variability, and in particular, generally ignore nonsampling errors in defining thetarget population (e.g., errors due to uncertainty in who will vote). Here, we empirically analyze 4221 polls
for 608 state-level presidential, senatorial, and gubernatorial elections between 1998 and 2014, all of whichwere conducted during the final three weeks of the campaigns. Comparing to the actual election outcomes,
we find that average survey error as measured by root mean square error is approximately 3.5 percentage
points, about twice as large as that implied by most reported margins of error. We decompose survey error
into election-level bias and variance terms. We find that average absolute election-level bias is about 2 percentage points, indicating that polls for a given election often share a common component of error. This
shared error may stem from the fact that polling organizations often face similar difficulties in reaching various subgroups of the population, and that they rely on similar screening rules when estimating who will
vote. We also find that average election-level variance is higher than implied by simple random sampling,in part because polling organizations often use complex sampling designs and adjustment procedures. We
conclude by discussing how these results help explain polling failures in the 2016 U.S. presidential election,and offer recommendations to improve polling practice.

1. Introduction
Election polling is arguably the most visible manifestation of
statistics in everyday life, and embodies one of the great successstories of the field: random sampling. Through the 1930s and1940s, pollsters aimed to achieve representative samples throughstrict demographic quotas. But as recounted in so many textbooks, all the major polls that used quota sampling in the 1948
presidential election incorrectly predicted Dewey would beatTruman. This high-profile reality check on the statistical methods of the time spurred pollsters to move to random sampling,ushering in the age of modern polling.
It has long been known that the margins of errors provided
by survey organizations, and reported in the news, understate
the total survey error. This is an important topic in sampling
but is difficult to address in general for two reasons. First, welike to decompose error into bias and variance, but this can only
be done with any precision if we have a large number of surveys
and outcomes—not merely a large number of respondents in an
individual survey. Second, assessment of error requires a groundtruth for comparison, which is typically not available, as the reason for conducting a sample survey in the first place is to estimate some population characteristic that is not already known.
In the present paper, we decompose survey error in a large setof state-level pre-election polls. This dataset resolves both of the

ARTICLE HISTORY

Received June 
Revised February KEYWORDS

Margin of error;
Nonsampling error; Polling
bias; Total survey error

problems just noted. First, the combination of multiple elections
and many states gives us a large sample of polls. Second, we cancompare the polls to actual election results.
1.1. Background
Election polls typically survey a random sample of eligible or
likely voters, and then generate population-level estimates bytaking a weighted average of responses, where the weights aredesigned to correct for known differences between sample andpopulation.1 This general analysis framework yields both a point
estimate of the election outcome, and also an estimate of theerror in that prediction due to sample variance which accounts
for the survey weights (Lohr 2009). In practice, however, pollingorganizations often use the weights only in computing estimates,
ignoring them when computing standard errors and insteadreporting 95% margins of error based on the formula for simple
random sampling (SRS)—for example, ±3.5 percentage points
for an election survey with 800 people. Appropriate correction
for the “design effect” corresponding to unequal weights would
increase margins of error (see, for example, Mercer (2016)). The
increase in margin of error depends on the poll, as some surveys
have self-weighting designs (i.e., the sampling is constructed sothat no weights are used in the analysis), while others weight on

CONTACT Sharad Goel
scgoel@stanford.edu
Department of Management Science & Engineering,  Via Ortega, Stanford University, Stanford, CA, .
Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/r/JASA.
 One common technique for setting survey weights is raking, in which weights are defined so that the weighted distributions of various demographic features (e.g., age,sex, and race) of respondents in the sample agree with the marginal distributions in the target population (Voss, Gelman, and King ).
©  American Statistical Association

608

H. SHIRANI-MEHR ET AL.

many factors. For some leading polls, standard errors should be
increased by a factor of 30% to account for the weighting.2
Though this approach to quantifying polling error is popular and convenient, it is well known by both researchersand practitioners that discrepancies between poll results andelection outcomes are only partially attributable to sample
variance (Ansolabehere and Belin 1993). As observed in theextensive literature on total survey error (Biemer 2010; Grovesand Lyberg 2010), there are at least four additional types of errorthat are not reflected in the usually reported margins of error:
frame, nonresponse, measurement, and specification. Frameerror occurs when there is a mismatch between the samplingframe and the target population. For example, for phone-based
surveys, people without phones would never be included in any
sample. Of particular import for election surveys, the samplingframe includes many adults who are not likely to vote, which
pollsters recognize and attempt to correct for using likely votersscreens, typically estimated with error from survey questions.
Nonresponse error occurs when missing values are systematically related to the response. For example, supporters of thetrailing candidate may be less likely to respond to surveys (Gelman et al. 2016). With nonresponse rates exceeding 90% forelection surveys, this is a growing concern (Pew ResearchCenter 2016). Measurement error arises when the survey
instrument itself affects the response, for example, due to ordereffects (McFarland 1981) or question wording (Smith 1987).
Finally, specification error occurs when a respondent’s interpretation of a question differs from what the surveyor intends toconvey (e.g., due to language barriers). In addition to these fourtypes of error common to nearly all surveys, election polls sufferfrom an additional complication: shifting attitudes. Whereas
surveys typically seek to gauge what respondents will do onelection day, they can only directly measure current beliefs.
In contrast to errors due to sample variance, it is difficult—and perhaps impossible—to build a useful and general statisticaltheory for the remaining components of total survey error.
Moreover, even empirically measuring total survey error canbe difficult, as it involves comparing the results of repeatedsurveys to a ground truth obtained, for example, via a census.
For these reasons, it is not surprising that many survey organizations continue to use estimates of error based on theoreticalsampling variation, simply acknowledging the limitations of
the approach. Indeed, Gallup (2007) explicitly states that their
methodology assumes “other sources of error, such as nonresponse, by some members of the targeted sample are equal,” andfurther notes that “other errors that can affect survey validity
include measurement error associated with the questionnaire,such as translation issues and coverage error, where a part orparts of the target population ... have a zero probability of being
selected for the survey.”
 For a sampling of  polls for  senate elections, only  reported margins oferror higher than what one would compute using the SRS formula, and  of these
exceptions were accounted for by YouGov, a polling organization that explicitly
notes that it inflates variance to adjust for the survey weights. Similarly, for asampling of  state-level polls for the  presidential election, only  reported
higher-than-SRS margins of error. Complete survey weights are available for ABC, CBS, and Gallup surveys conducted during the  election and deposited
into Roper Center’s iPOLL. To account for the weights in these surveys, standard
errors should on average be multiplied by . (with an interquartile range of .–.
across the surveys), compared to the standard errors assuming SRS.

1.2. Our Study
Here, we empirically and systematically study error in election
polling, taking advantage of the fact that multiple polls are typically conducted for each election, and that the election outcome
can be taken to be the ground truth. We investigate 4221 polls for608 state-level presidential, senatorial, and gubernatorial elections between 1998 and 2014, all of which were conducted in the
final three weeks of the election campaigns. By focusing on the
final weeks of the campaigns, we seek to minimize the impact oferrors due to changing attitudes in the electorate, and hence toisolate the effects of the remaining components of survey error.
We find that the average difference between poll results andelection outcomes—as measured by root mean square error(RMSE)—is 3.5 percentage points, about twice the error impliedby most reported confidence intervals.3 To decompose this survey error into election-level bias and variance terms, we carryout a Bayesian meta-analysis. We find that average absoluteelection-level bias is about 2 percentage points, indicating thatpolls for a given election often share a common componentof error. This result is likely driven in part by the fact thatmost polls, even when conducted by different polling organizations, rely on similar likely voter models, and thus surprises
in election day turnout can have comparable effects on all thepolls. Moreover, these correlated frame errors extend to thevarious elections—presidential, senatorial, and gubernatorial—
across the state.

2. Data
2.1. Data Description
Our primary analysis is based on 4221 polls completed during
the final three weeks of 608 state-level presidential, senatorial,and gubernatorial elections between 1998 and 2014. Polls are
typically conducted over the course of several days, and following convention, we throughout associate the “date” of the pollwith the last date during which it was in the field. We do notinclude house elections in our analysis since polling is only available for a small and nonrepresentative subset of such races.
To construct this dataset, we started with the 4154 state-level
polls for elections in 1998–2013 that were collected and madeavailable by FiveThirtyEight, all of which were completed during the final three weeks of the campaigns. We augment thesepolls with the 67 corresponding ones for 2014 posted on Pollster.com, where for consistency with the FiveThirtyEight data,
we consider only those completed in the last three weeks of thecampaigns. In total, we end up with 1646 polls for 241 senatorial elections, 1496 polls for 179 state-level presidential elections,and 1079 polls for 188 gubernatorial elections.
In addition to our primary dataset described above, we also
consider 7040 polls completed during the last 100 days of 314
state-level presidential, senatorial, and gubernatorial elections
between 2004 and 2012. All polls for this secondary dataset were Most reported margins of error assume estimates are unbiased, and report %

confidence intervals of approximately ±3.5 percentage points for a sample of 
respondents. This in turn implies the RMSE for such a sample is approximately .
percentage points, about half of our empirical estimate of RMSE. As discussed inFootnote , many polling organizations do not adjust for survey weights when
computing uncertainty estimates, which in part explains this gap.

JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION

609

Difference between poll results and election outcomes
Senatorial

Gubernatorial

Presidential

Number of polls

300

200

100

0

10%

5%

0%

5%

10%

10%

5%

0%

5%

10%

10%

5%

0%

5%

10%

Figure . The distribution of polling errors (Republican share of two-party support in the poll minus Republican share of the two-party vote in the election) for state-levelpresidential, senatorial, and gubernatorial election polls between  and . Positive values indicate the Republican candidate received more support in the poll thanin the election. For comparison, the dashed lines show the theoretical distribution of polling errors assuming each poll is generated via SRS.

obtained from Pollster.com and RealClearPolitics.com. Whereas
this complementary set of polls covers only the more recent elections, it has the advantage of containing polls conducted earlier
in the campaign cycle.
2.2. Data Exploration
For each poll in our primary dataset (i.e., polls conducted during
the final three weeks of the campaign), we estimate total survey
error by computing the difference between: (1) support for theRepublican candidate in the poll; and (2) the final vote share forthat candidate on election day. As is standard in the literature, we
consider two-party poll and vote share: we divide support for theRepublican candidate by total support for the Republican and
Democratic candidates, excluding undecideds and supporters of
any third-party candidates.
Figure 1 shows the distribution of these differences, where
positive values on the x-axis indicate the Republican candidate
received more support in the poll than in the election. We repeat
this process separately for senatorial, gubernatorial, and presidential polls. For comparison, the dotted lines show the theoretical distribution of polling errors assuming SRS. Specifically,for each senate poll i, we first simulate an SRS polling result bydrawing a sample from a binomial distribution with parameters
ni and v r[i] , where ni is the number of respondents in poll i who
express a preference for one of the two major-party candidates,and v r[i] is the final two-party vote share of the Republican candidate in the corresponding election r[i]. The dotted lines in theleft-hand panel of Figure 1 show the distribution of errors acrossthis set of synthetic senate polls. Theoretical SRS error distributions are generated analogously for gubernatorial and presidential polls.
The plot highlights two points. First, for all three political offices, polling errors are approximately centered at zero.
Thus, at least across all the elections and years that we consider, polls are not systematically biased toward either party.
Indeed, it would be surprising if we had found systematic error,
since pollsters are highly motivated to notice and correct forany such aggregate bias. Second, the polls exhibit substantiallylarger errors than one would expect from SRS. For example,
it is not uncommon for senatorial and gubernatorial polls to

miss the election outcome by more than 5 percentage points, anevent that would rarely occur if respondents were simple random draws from the electorate.
We quantify these polling errors in terms of the RMSE.4
The senatorial and gubernatorial polls, in particular, havesubstantially larger RMSE (3.7% and 3.9%, respectively) than
SRS (2.0% and 2.1%, respectively). In contrast, the RMSE forstate-level presidential polls is 2.5%, not much larger than one
would expect from SRS (2.0%). Because reported margins oferror are typically derived from theoretical SRS error rates,
the traditional intervals are too narrow. Namely, SRS-based
95% confidence intervals cover the actual outcome for only
73% of senatorial polls, 74% of gubernatorial polls, and 88% ofpresidential polls. It is not immediately clear why presidential
polls fare better, but one possibility is that turnout in suchelections is easier to predict and so these polls suffer less fromsuch error; in addition, presidential polls have higher visibility
and so the organizations that conduct such surveys may invest
more resources into their sampling and adjustment procedures.
We have thus far focused on polls conducted in the three
weeks prior to election day, in an attempt to minimize the effects
of error due to changing attitudes in the electorate. To examine
the robustness of this assumption, we now turn to our secondary
polling dataset and, in Figure 2, plot average poll error as a function of the number of days to the election. Due to the relatively
small number of polls conducted on any given day, we include
in each point in the plot all the polls completed in a seven-day
window centered at the focal date (i.e., polls completed within
three days before or after that day). As expected, polls early inthe campaign season indeed exhibit more error than those taken
near election day. Average error, however, appears to stabilize
in the final weeks, with little difference in RMSE one month
before the election versus one week before the election. Thus,
the polling errors that we see during the final weeks of the campaigns are likely not driven by changing attitudes, but rather
result from nonsampling error, particularly frame and nonresponse error. As noted earlier, measurement and specification
 Assuming N to be the number of polls, for each poll i ∈ {1, . . . , N}, let y denoteithe two-party support for the Republican candidate, and let v r[i] denote the final
two-party vote share
the Republican candidate in the corresponding election! of"2
r[i]. Then, RMSE is N1 N
i=1 (yi − v r[i] ) .

610

H. SHIRANI-MEHR ET AL.

Root mean square poll error over time
8%
Senatorial

6%
4%
2%

apparent that the empirical poll averages are substantially morevariable, particularly for senatorial and gubernatorial elections.
Figure 3 shows that even a substantial amount of excess variance in polls cannot fully explain our empirical observations,and thus points to the importance of accounting for electionlevel bias.

0%

3. A Model for Election Polls

8%
Gubernatorial

6%
4%
2%
0%
8%

Presidential

6%
4%
2%
0%

90

80

70

60

50

40

30

20

10

0

Days to Election

Figure . Poll error, as measured by RMSE, over the course of elections. The RMSE oneach day x indicates the average error for polls completed in a seven-day window
centered at x. The dashed vertical line at the three-week mark shows that poll error
is relatively stable during the final stretches of the campaigns, suggesting that thediscrepancies we see between poll results and election outcomes are by and large
not due to shifting attitudes in the electorate.

errors also likely play a role, though election polls are arguably
less susceptible to such forms of error.
In principle, Figure 1 is consistent with two possibilities. On
the one hand, election polls may typically be unbiased but havelarge variance; on the other hand, polls in an election may generally have nonzero bias, but in aggregate these biases cancel toyield the depicted distribution. Our goal is to quantify the structure of polling errors. But, before formally addressing this task,
we carry out the following simple analysis to build intuition. For
each election r, we first compute the average poll estimate
1 #
yi ,
ȳr =
|Sr | i∈Srwhere Sr is the set of polls in that election, and yi is the two-party
support for the Republican candidate in the ith poll. Figure 3(left) shows the difference between ȳr and the election outcome(i.e., the difference between the two-party poll average and thetwo-party Republican vote share), where each point in the plot
is an election. For comparison, Figure 3(middle) shows the same
quantities for synthetic SRS polls, generated as above. It is visually apparent that the empirical poll averages are significantly
more dispersed than expected under SRS, whereas Figure 1 indicates that individual polls are over-dispersed, Figure 3 showsthat poll averages also exhibit considerable over-dispersion.
Finally, Figure 3(right) plots results for synthetic polls that are
unbiased but that have twice the variance as SRS. Specifically,
we simulate a polling result by drawing a sample from a binomial
distribution with parameters v r (the election outcome) and ni /2(half the number of respondents in the real poll), since halving
the size of the poll doubles the variance. Doubling poll varianceincreases the dispersion of poll averages, but it is again visually

We now present and fit a statistical model to shed light on the
structure of polling results. The bias term in our model capturessystematic errors shared by all polls in an election (e.g., due toshared frame errors). The variance term captures residual dispersion, from traditional sampling variation as well as variation
due to differing survey methodologies across polls and pollingorganizations. Our approach can be thought of as a Bayesianmeta-analysis of survey results.
For each poll i in election r[i] conducted at time ti , let yi
denote the two-party support for the Republican candidate (asmeasured by the poll), where the poll has ni respondents withpreference for one of the two major-party candidates. Let v r[i]denote the final two-party vote share for the Republican candidate. Then, we model the poll outcome yi as a random draw froma normal distribution parameterized as follows:yi ∼ N(pi , σi2 )logit(pi ) = logit(v r[i] ) + αr[i] + βr[i]tipi (1 − pi )2+ τr[i]
.
σi2 =
ni

(1)

Here, αr[i] + βr[i]ti is the bias of the ith poll (positive values indicate the poll is likely to overestimate support for the Republican candidate), where we allow the bias to change linearly overtime.5 The possibility of election-specific excess variance (rel2term. Such
ative to SRS) in poll results is captured by the τr[i]excess variance may, for example, result from complex sampling
designs and adjustment procedures. Estimating excess varianceis statistically and computationally tricky, and there are many
possible ways to model it. For simplicity, we use an additive term,
and note that our final results are robust to natural alternatives;
for example, we obtain qualitatively similar results if we assume
a multiplicative relationship.
When modeling poll results in this way, one must decide
which factors to include as affecting the mean pi rather than thevariance σi2 . For example, in our current formulation, systematic differences between polling firms (Silver 2017) are not modeled as part of pi , and so these “house effects” implicitly enter inthe σi2 term. There is thus no perfect separation between bias
and variance, as explicitly accounting for more sources of variation when modeling the mean increases estimates of bias while
simultaneously decreasing estimates of variance. Nevertheless,as our objective is to understand the election-level structure ofpolls, our decomposition above seems natural and useful.
 To clarify our notation, we note that for each poll i, r[i] denotes the election forwhich the poll was conducted, and αr[i] , βr[i] , and τr[i] denote the corresponding
coefficients for that election. Thus, for each election j, there is one (α j , β j , τ j )triple. Our model allows for a linear time trend (β j ), but we note that our empiricalresults are qualitatively similar even without this term.

JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION

611

Figure . Difference between polling averages and election outcomes (i.e., Republican share of the two-party vote), where each point is an election. The left panel shows
results for the real polling data; the middle panel shows results for a synthetic dataset of SRS polls; and the right panel shows results for a synthetic dataset of polls that are
unbiased but that have twice the variance of SRS.

To partially pool information across elections, we place ahierarchical structure on the parameters (Gelman and Hill2007). We specifically set
α j ∼ N(µα , σα2 )β j ∼ N(µβ , σβ2 )τ j2 ∼ N+ (0 , στ2 ),where N+ denotes the half-normal distribution. Finally, weakly
informative priors are assigned to the hyperparamaters µα , σα ,µβ , σβ , and στ . Namely, µα ∼ N(0, 0.22 ), σα ∼ N+ (0, 0.22 ),µβ ∼ N(0, 0.22 ), σβ ∼ N+ (0, 0.22 ), and στ ∼ N+ (0, 0.052 ).
Our priors are weakly informative in that they allow for a large,
but not extreme, range of parameter values. In particular, thougha 5 percentage point (which is roughly equivalent to 0.2 on thelogit scale) poll bias or excess dispersion would be substantial, it
is of approximately the right magnitude. We note that while an
inverse gamma distribution is a traditional choice of prior forvariance parameters, it rules out values near zero (Gelman et al.
2006); our use of half-normal distributions is thus more consistent with our decision to select weakly informative priors. In
Section 4.3, we experiment with alternative prior structures andshow that our results are robust to the exact specification.

4. Results

The fitted model lets us estimate three key quantities. First,
we define average election-level absolute bias µb bykwhere k is the total number of elections in consideration (acrossall years and states), and br is the bias for election r. Specifically,
br is defined by1 #br =(pi − v r ),|Sr | i∈Srwhere Sr is the set of polls in election r, and pi is computed via
Equation (1). That is, to compute br , we average the bias for eachpoll in the election. Second, we define the average absolute bias
on election day µb0 bykµb0 =where qr is defined by1#|qr − v r |,k r=1

logit(qr ) = logit(v r ) + αr .
That is, we define µb0 by setting ti to zero in Equation (1). Finally,
we define average election-level standard deviation µσ byk4.1. Preliminaries
We fit the above model separately for senatorial, presidential, and gubernatorial elections. Posterior distributions for theparameters are obtained via Hamiltonian Monte Carlo (Hoffman and Gelman 2014) as implemented in Stan, an open-source
modeling language for full Bayesian statistical inference.

1#|br |,
k r=1

µb =

µσ =
where
σr =

1#
σr ,
k r=1

1 #
σi .
|Sr | i∈Sr

612

H. SHIRANI-MEHR ET AL.

Estimated election level absolute bias

Number of Elections

Senatorial

Gubernatorial

Presidential

30

20

10

0
0%

2%

4%

6%

8%

0%

2%

4%

6%

8%

0%

2%

4%

6%

8%

6%

8%

Estimated election level standard deviation

Number of Elections

Senatorial

Gubernatorial

Presidential

75

50

25

0
0%

2%

4%

6%

8%

0%

2%

4%

6%

8%

0%

2%

4%

Figure . Model estimates of election-level absolute bias (top plots) and election-level standard deviation (bottom plots).

To check that our modeling framework produces accurate
estimates, we first fit it on synthetic data generated via SRS, preserving the empirically observed election outcomes, the number
and date of polls in each election, and the size of each poll. On
this synthetic dataset, we find the mean posterior estimates µ̂band µ̂b0 are approximately 0.2 percentage points (i.e., approximately two-tenths of 1 percentage point), nearly identical to
the theoretically correct answer of zero. We further find thatthe posterior mean µ̂σ is approximately 2.1 percentage points,
closely aligned with the theoretically correct answer of 2.0.
4.2. Empirical Results
Table 1 summarizes the results of fitting the model on our primary polling dataset. The results show elections for all three
offices exhibit substantial average election-level absolute bias,
approximately 2 percentage points for senatorial and gubernatorial elections and 1 percentage point for presidential elections.
The poll bias is about as big as the theoretical sampling variationfrom SRS. The full distribution of election-level mean posterior
estimates is shown in Figure 4. The top panel in the plot shows
the distribution of |b̂r |, and the bottom panel shows σ̂r .
Why do polls exhibit nonnegligible election-level bias? Weoffer three possibilities. First, as discussed above, polls in a given
election often have similar sampling frames. As an extreme
example, telephone surveys, regardless of the organization that
conducts them, will miss those who do not have a telephone.

More generally, polling organizations are likely to undercountsimilar, hard-to-reach groups of people (though post-sampling
adjustments can in part correct for this). Relatedly, projections about who will vote—often based on standard likely voter
screens—do not vary much from poll to poll, and as a consequence, election day surprises (e.g., an unexpectedly high number of minorities or young people turning out to vote) affect allpolls similarly. Second, since polls often apply similar methods
to correct for nonresponse, errors in these methods can again
affect all polls in a systematic way. For example, it has recently
been shown that supporters of the trailing candidate are lesslikely to respond to polls, even after adjusting for demographics (Gelman et al. 2016). Since most polling organizations do notcorrect for such partisan selection effects, their polls are all likely
Table . Mean posterior estimates of election-level poll bias and standard deviation,with the standard deviation of the posterior distribution given in parentheses. Biasand standard deviation are higher than would be expected from SRS. Under SRS,
the average election-level standard deviation would be . percentage points forsenatorial and presidential polls, and . percentage points for gubernatorial polls;
the bias would be zero.

Average election-level absolutebias (µ̂b )Average election-level absolutebias on election day (µ̂b )0Average election-level standard
deviation (µ̂σ )Senatorial

Gubernatorial

Presidential

.% (.%).% (.%).% (.%).% (.%).% (.%).% (.%).% (.%).% (.%).% (.%)JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION

Table . Mean posterior estimates for various choices of priors. Our results are qualitatively similar regardless of the priors selected.

Average absolute bias
4%

PriorsGubernatorial
Senatorial

3%

2%

1%
Presidential
0%

2000

2004

2008

613

2012

Figure . Model-based estimates of average absolute bias show no consistent timetrends across election cycles.

to be systematically skewed. Finally, respondents might misreport their vote intentions, perhaps because of social desirability bias (if they support a polarizing candidate) or acquiescencebias (if they believe the poll to be leaning against their preferredcandidate).
Figure 5 shows how the average absolute election-level bias
changes from one election cycle to the next. To estimate averageabsolute bias for each year, we average the estimated absoluteelection bias for all elections that year. While there is noticeable
year-to-year variation, the magnitude is consistent over time,
providing further evidence that the effects we observe are realand persistent. We note that one might have expected to see arise in poll bias over time given that survey response rates haveplummeted—from an average of 36% in 1998 to 9% in 2012 (Pew
Research Center 2012). One possibility is that pre- and postsurvey adjustments to create demographically balanced samples
mitigate the most serious issues associated with falling response
rates, while doing little to correct for the much harder problem
of uncertainty in turnout.
Finally, Figure 6 shows the relationship between electionlevel bias in elections for different offices within a state. Eachpoint corresponds to a state, and the panels plot estimated bias
for the two elections indicated on the axes. Overall, we findmoderate correlation in bias for elections within the state: 0.45
for gubernatorial versus senatorial, 0.50 for presidential versus
senatorial, and 0.39 for gubernatorial versus presidential.6 Suchcorrelation again likely comes from a combination of frameand nonresponse errors. For example, since party-line voting
is relatively common, an unusually high turnout of Democrats
on election day could affect the accuracy of polling in multiple
races. This correlated bias in turn leads to correlated errors, andillustrates the importance of treating polling results as correlated
rather than independent samples of public sentiment.
4.3. Sensitivity Analysis
We conclude our analysis by examining the robustness of ourresults to the choice of priors in the model. In our primary
analysis, we consider a 5 percentage point (equivalent to 0.2
on the logit scale) standard deviation for the bias and variance To calculate these numbers, we removed an extreme outlier that is not shown in

Figure , which corresponds to polls conducted in Utah in . There are only
two polls in the dataset for each race in Utah in .

MeasureSen. Gov. Pres.

µα , µβ ∼ N(0, 12 )σα , σβ ∼ N+ (0, 12 )στ ∼ N+ (0, 0.22 )Absolute bias
.% .% .%Election day absolute bias .% .% .%Standard deviation
.% .% .%µα , µβ ∼ N(0, 0.042 )σα , σβ ∼ N+ (0, 0.042 )στ ∼ N+ (0, 0.012 )Absolute bias
.% .% .%Election day absolute bias .% .% .%Standard deviation
.% .% .%µα , µβ ∼ N(0, 0.22 )Absolute bias
.% .% .%σα , σβ ∼ Gamma−1 (3.6, 0.4) Election day absolute bias .% .% .%Standard deviation
.% .% .%στ ∼ Gamma−1 (3.6, 0.1)

hyperparameters. In this section, we consider three alternative
choices. First, we change the standard deviation defined for
all hyperparameters to approximately 25 percentage points,
corresponding to a prior that is effectively flat over the feasible
parameter region. Second, we change the standard deviation toapproximately 1 percentage point, corresponding to an informative prior that constrains the bias and excess variance to berelatively small. Finally, we replace the half-normal prior on thevariance hyperparameters with an inverse gamma distribution;
α and β were chosen so that the resulting distribution has mean
and variance approximately equal to that of the half-normal
distribution in the original setting. Table 2 shows the results ofthis sensitivity analysis. Our posterior estimates are qualitatively
similar in all cases, regardless of which priors are used. While
the posterior estimates for absolute bias are nearly identical,
inverse gamma priors on the variance hyperparameters result insomewhat higher estimated election-level standard deviation.

5. Discussion
Researchers and practitioners have long known that traditional
margins of error understate the uncertainty of election polls, but
by how much has been hard to determine, in part because of alack of data. By compiling and analyzing a large collection of historical election polls, we find substantial election-level bias andexcess variance. We estimate average absolute bias is 2.1 percentage points for senate races, 2.3 percentage points for gubernatorial races, and 1.2 percentage point for presidential races. At thevery least, these findings suggest that care should be taken whenusing poll results to assess a candidate’s reported lead in a competitive race. Moreover, in light of the correlated polling errorsthat we find, close poll results should give one pause not only forpredicting the outcome of a single election, but also for predicting the collective outcome of related races. To mitigate the recognized uncertainty in any single poll, it has become increasingly
common to turn to aggregated poll results, whose nominal variance is often temptingly small. While aggregating results is generally sensible, it is particularly important in this case to remember that shared election-level poll bias persists unchanged, evenwhen averaging over a large number of surveys.
The 2016 U.S. presidential election offers a timely exampleof how correlated poll errors can lead to spurious predictions.
Up through the final stretch of the campaign, nearly all pollsters
declared Hillary Clinton the overwhelming favorite to win the

614

H. SHIRANI-MEHR ET AL.

Figure . Comparison of election-level polling bias in various pairs of state-level elections. Each point indicates the estimated bias in two different elections in the samestate in the same year. The plots show modest correlations, suggesting a mix of frame, and nonresponse errors.

election. The New York Times, for example, placed the probability of a Clinton win at 85% on the day before the election.
Donald Trump ultimately lost the popular vote, but beat forecasts by about 2 percentage points. He ended up carrying nearly
all the key swing states, including Florida, Iowa, Pennsylvania,
Michigan, and Wisconsin, resulting in an electoral college win
and the presidency. Because of shared poll bias—both for multiple polls forecasting the same state-level race, and also for polls
in different states—even modest errors significantly impact win
estimates. Such correlated errors might arise from a variety ofsources, including frame errors due to incorrectly estimating
the turnout population. For example, a higher-than-expected
turnout among white men, or other Republican-leaning groups,
may have skewed poll predictions across the nation.
Our analysis offers a starting point for polling organizations
to quantify the uncertainty in predictions left unmeasured bytraditional margins of errors. Instead of simply stating that thesecommonly reported metrics miss significant sources of error,
which is the status quo, these organizations could—and we feelshould—start quantifying and reporting the gap between theory
and practice. Indeed, empirical election-level bias and variancecould be directly incorporated into reported margins of error.
Though it is hard to estimate these quantities for any particularelection, historical averages could be used as proxies.
Large election-level bias does not afflict all estimated quantities equally. For example, it is common to track movements insentiment over time, where the precise absolute level of support
is not as important as the change in support. A stakeholder may
primarily be interested in whether a candidate is on an up or
downswing rather than his or her exact standing. In this case,the bias terms—if they are constant over time—cancel out.
Given the considerable influence election polls have on
campaign strategy, media narratives, and popular opinion, it
is important to have both accurate estimates of candidate support and also accurate accounting of the uncertainty in thoseestimates. Looking forward, we hope our analysis and methodological approach provide a framework for understanding,incorporating, and reporting errors in election polls.

Acknowledgments
The survey weights discussed in Footnote 2 are based on polls obtained
from the iPOLL Databank provided by the Roper Center for Public Opinion

Research at Cornell University. The data and code to replicate our results areavailable online at https://github.com/5harad/polling-errors.

Funding
This work is supported in part by DARPA under agreement number
D17AC00001, and ONR grant N00014-15-1-2541.

References
Ansolabehere, S., and Belin, T. R. (1993), “Poll Faulting,” Chance, 6, 22–28.
[608]
Biemer, P. P. (2010), “Total Survey Error: Design, Implementation, and Evaluation,” Public Opinion Quarterly, 74, 817–848. [608]Gallup (2007), “Gallup World Poll Research Design.” available at http://
media.gallup.com/WorldPoll/PDF/WPResearchDesign091007bleeds.
pdf. [608]Gelman, A., and Hill, J. (2007), Data Analysis Using Regression and Multilevel/Hierarchical Models, New York: Cambridge University Press. [611]Gelman, A., Goel, S., Rivers, D., and Rothschild, D. (2016), “The Mythical Swing Voter,” Quarterly Journal of Political Science, 11, 103–130.
[608,612]
Gelman, A. (2006), “Prior Distributions for Variance Parameters in Hierarchical Models (Comment on Article by Browne and Draper),” Bayesian
Analysis, 1, 515–534. [611]Groves, R. M., and Lyberg, L. (2010), “Total Survey Error: Past, Present, andFuture,” Public Opinion Quarterly, 74, 849–879. [608]Hoffman, M. D., and Gelman, A. (2014), “The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo,” Journal of
Machine Learning Research, 15, 1593–1623. [611]Lohr, S. (2009), Sampling: Design and Analysis, Boston, MA: Nelson Education. [607]McFarland, S. G. (1981), “Effects of Question Order on Survey Responses,”
Public Opinion Quarterly, 45, 208–215. [608]Mercer, A. (2016), “5 Key Things to Know about the Margin of Error inElection Polls,” available at http://www.pewresearch.org/fact-tank/2016/09/08/understanding-the-margin-of-error-in-election-polls/. [607]
Pew Research Center (2012), “Assessing the Representativeness of Public
Opinion Surveys,” available at http://www.people-press.org/2012/05/15/
assessing-the-representativeness-of-public-opinion-surveys,”. [613]——— (2016), “Our Survey Methodology in Detail,” available at
http://www.people-press.org/methodology/our-survey-methodologyin-detail. [608]Silver, N. (2017), “Fivethirtyeight’s Pollster Ratings,” available athttps://projects.fivethirtyeight.com/pollster-ratings/. [610]Smith, T. W. (1987), “That Which we Call Welfare by any Other Name
Would Smell Sweeter: An Analysis of the Impact of Question Wording on Response Patterns,” Public Opinion Quarterly, 51, 75–83. [608]Voss, D. S., Gelman, A., and King, G. (1995), “Pre-Election Survey Methodology: Details from Nine Polling Organizations, 1988 and 1992,” Public
Opinion Quarterly, 59, 98–132. [607]