Why traditional polling keeps getting it wrong
By: Don Vaughn
Tags: Politics, Bias, Trump, Polling, Election 2020

Polling in the presence of bias

How did Invisibly predict the outcome of the presidential election more accurately than traditional polls? Our technology minimized this “shy Trump” bias by creating psychological distance through anonymity.

Our Realtime Research surveys are displayed to people on webpages (Figure 2A). This type of interaction seems more anonymous than a phone call, and I believe respondents feel much more comfortable voicing their true intentions. Additionally, our surveys are fairly nondescript and completely voluntary, so people see a casual opportunity to voice their opinion, and some of them opt in. This method has proven successful in many of the previous races we’ve predicted (Figure 2B/C). To be clear: I do not claim that our methodology is bias-free — just that it’s less biased than traditional polling.

Figure 2: Survey examples and past political performances. A. Questions are shown to the user on webpages. Prior to the 2020 presidential election, Invisibly surveys predicted several political outcomes, including B. Iowa’s 4th District 2020 Republican primary and C. Texas’s 13th District 2020 Republican primary.————————————————————————————————

Now, traditional pollsters happily let us know that standard practice in the field is absolutely not to allow anyone to respond voluntarily (i.e., a “convenience sample”), but rather to dial random phone numbers. I teach statistics, so I understand that this is an optimal approach… in theory. But outside the ivory tower of well-controlled lab experiments, this method focuses far too much on finding a sample with near-perfect representation and then employing statistical corrections to make the data 100% representative across multiple demographic variables. It ignores the potential value in sampling only those who want to participate. After all, people choose to vote; they are not chosen to vote by the government randomly dialing phone numbers.

Our technology also increases precision by collecting larger samples — often at least an order of magnitude larger than traditional phone polling methods. I don’t doubt that this focus on scale, rather than the purity of the representative sample, also underlies our success. I’ll trust 35,000 interested and willing respondents over 800 robocalls any day.

While the biased polling phenomenon has the most obvious effects on political campaign strategies, it has secondary effects on the rest of us. The media displays long lists of polls, almost all biased in the same direction, and this exposure divorces our expectations from reality. When CNN consistently showed Biden up in the months preceding the election — by as much as 16 points — we are all shocked on election night when the race is actually very close. This discrepancy can undermine our trust in the results and may even affect the election’s outcome if enough people mistakenly believe that their vote isn’t “needed” because their candidate is so far ahead. More succinctly: bad polling is bad for democracy.

Dr. Don Vaughn, Ph.D.

Head of Product

don.vaughn@invisibly.com

*Results as of November 8, 2020; electoral vote count assumes North Carolina goes to Trump and Georgia to Biden.

[1] “Majorities of Democrats (52%), independents (59%) and Republicans (77%) all agree they have political opinions they are afraid to share.” https://www.cato.org/publications/survey-reports/poll-62-americans-say-they-have-political-views-theyre-afraid-share#liberals-are-divided-political-expression

[2] “Nearly two-thirds of Republicans and Republican-leaning independents (64%) think that ‘Democrats in this country are very comfortable to freely and openly express their political views,’ but only about a quarter (26%) think Republicans around the nation experience that same level of comfort.” https://www.pewresearch.org/fact-tank/2019/06/24/republicans-see-a-national-political-climate-comfortable-for-democrats-but-less-so-for-gop/