Problems in Political Polling
By: Mike Aga
Tags: Hillary Clinton, Joe Biden, Donald Trump, Polling, Politics

There is an odd pheromone in democratic circles, that became palpable within hours of the major news networks calling the race for Joe Biden. Even though Biden had won and ushered in a new era of good governance and “not-Trump”, there was a definitive tangible mood of letdown and regret. The expectation were so high for democrats that even though Biden won the presidency with a historic popular vote count, there was a sense that even with a win for the presidency, there was not a clear repudiation of the outgoing president and his populist demagoguery. For many, a clear win for democrats looked like a unified government, with Democrats controlling all three branches of government. The resulting election ended with a democratic president, but a house with a significantly weakened democratic majority and a senate that is for democrats at best a 50/50 tie and for democrats at worst a 52/48 body with them in the minority. And while politics has often been defined by unrealistic expectations, the 2020 results fell far below even some of the most conservative estimates.

That leads even the most casual of political observers with a multitude of lingering questions. The purpose of this article is to walk you through the basic questions you might have surrounding political polling in the last two presidential cycles, answering questions where we can and explaining why the answer may not be so simple when we can’t. Anyone should be able to read this article and come away with an understanding of the problems in political polling.

What happened that made expectations so high for democrats? What does the trend of more discrepancies in polling results mean? Not only for the media companies providing the polls but also the faith in democracy for those who build their expectations around polling. If polling can’t be trusted as a reliable rubric, what can political campaigns do to correctly target their resources and maximize chances of succeeding?

The initial focus will be on the just past election of 2020. In that election, centrist democrat Joe Biden who had previously served as the Vice President was seeking to overthrow the at time current President, Donald Trump who is a republican. The election was defined not by policy preferences of the candidates or more traditional campaign metrics, instead, it was defined largely by the character of the two men running. Joe Biden was largely viewed as a good man, but that image was attacked by republicans with allegations of inappropriate physical contact and allegations of financial misdeeds mainly about his son Hunter Biden. While the republican Donald Trump was attacked mainly for his own personal character and her governmental failings particularly on the spread of the coronavirus pandemic. Throughout the general election, which is the point in the campaign between the selecting of the two nominees and the end of the campaign in November, the two campaigns were incredibly consistent. Well, there were the obvious October surprises like the death of Justice Ruth Bader Ginsburg and the subsequent appointment of her replacement Justice Barrett and the New York Times reporting on president trump only paying $750 in federal taxes, unlike most modern campaigns the race consistently showed Joe Biden with a sizable lead nationwide.

So, what does have a sizable lead nationwide actually mean? Who are the people actually determining that Joe Biden is winning the election at any one point in time before November 3rd?

Those groups would be known as pollsters being a pollster as a professor that has been around and prominent as long as there have been politics in a way to actually contact people in essence their job is to predict the winner of outcomes. Some of the time that can be as simple as calling individuals and households and simply asking who you are going to be voting for in this next upcoming election, but it can be as complex as doing a full statistical breakdown on entire populations and randomly drawing a sample from that population and asking them who they’re going to be voting for and then using that data to extrapolate predictive results. That is pulling at its most simple and throughout history there have been multiple times where its effectiveness has been brought into question.

But never before in modern history have polling been so spectacularly wrong in two cycles in a row, at a certain point conversations around statistical significance is beginning to look insignificant. In political circles, you can easily strike up an hours-long conversation that could turn into a shouting match simply by asking, what do you think went wrong with polling in ’16 and ’20?

Everyone in politics and media has an opinion on polling and most are a little bit right and a little bit wrong, and well you could have an argument about the different factors that could be weighted differently, or the merits of internet-based polling vs live calling, a more useful discussion should revolve around the larger trends of why political polling is failing more and more often.

Politics dividing Americans is no new advancement, but the way the fourth estate deals with politics has taken on new brutality as a twenty-first-century blood sport. Americans have been increasingly divided and media is one of the largest factors in that division. This danger emerges when the media begins to evaluate the state not as a democracy or as a government but as a sport. This style can be seen quite easily when taking into account partisan bias media. The Cable News Network (CNN) is a great example of this sportsmanship. Jeff Zucker, who made a name for himself creating shows like Fear Factor, The Apprentice, and Pardon the Interruption, is the current president of the network. He has said in the past “the idea that politics is [a] sport is undeniable, and we understood that and approached it that way”. His words are very reflective of reality — much of CNN’s news programming, debates, and elections are framed in the same context as a sporting event like a boxing match or soccer game.

When watching a sport, the most important thing to understand is how to determine the score. In politics, people frequently equate polls to the score. Conflating these two is incredibly dangerous for our democracy, as the fourth estate is now dealing with an epidemic of analysts giving mediocre interpretations of bad polling data.

If you are not familiar with American politics, media outlets often conduct polls asking citizens various questions ranging from “who do you intend to vote for” to “where should the president spend thanksgiving.” Polling data is important for anyone who has a responsibility to the public, as conducting a poll ethically can give individuals and policymakers a sense of public opinion. Polls are normally conducted and designed to extrapolate information and opinions by asking a series of questions or having people rank points of information.

The first poll that successfully predicted the outcome of a political race in the United States took place during the presidential election of 1824 between John Quincy Adams and Andrew Jackson. The poll was conducted in Pennsylvania and was a simple straw poll of roughly five hundred civilians that were eligible to vote. Jackson won the poll three hundred and thirty-five to Adam’s one hundred and sixty-five, which correctly predicted that Jackson would win the popular vote by an almost two to one margin. The poll was successful, as it correctly predicted Andrew Jackson would win the popular vote by a wide margin. Since the was the most accurate poll that had been conducted at that point, this model of a straw poll became extremely popular over the next two centuries.

Polls remained a largely local phenomenon; however, in 1916 The Literary Digest conducted the first national poll that correctly predicting that the governor of New Jersey, Woodrow Wilson, would be elected president of the United States. They did so by mailing out postcards to the populous. The Digest then went on to correctly predict the victory of Presidents Hoover, Harding, Coolidge, and Franklin Roosevelt. In the ensuing decades, four polling methods were widely considered to be the most accurate: opinion, tracking, brushfire, benchmark. As a brief disclaimer, this essay will primarily focus on the American government and American polling methods.

There is a misconception among the American public that polling is an exact science, or for that matter, anything in the American government is an absolute science. Instead, everything in the American government is some shade of gray. In polling, there are never absolutes; however, that is not how politics are often presented in America. In reality, they are often portrayed as a scoring system in sports.

In a sporting event, such as a soccer match, if there is a score of 6 to 4 in the fourth quarter as an observing spectator we can easily conclude that the team with fifteen points has no chance of winning the game anymore. Yes, it is technically possible, but highly improbable and bordering on impossible. This is the method of thought that people typically apply to poll — a score for a, particularly high stakes game. However, polling simply is not that black and white. It is not a score; rather, it is a probability. In reality, polls are more like a flood warning. If you live in an area that has a fifteen percent chance to flood, you would understandably be wary of being in danger and take the necessary precautions. Most people outside of politics might fail to grasp just how fluid numbers in politics can be. For instance, if a pre-election poll has one candidate at 60% and their opponent at 40% most people would expect a blowout, but in politics, you should think of a 60/40 poll as a focus group of 10 people, change one person’s mind and suddenly the race is a dead heat.

In 2016, most trusted polls and media outlets said that the Democratic nominee for president, Hillary Clinton, would win and assume the presidency over the Republican nominee, Donald Trump. They predicted that the likelihood of Hillary Clinton winning the presidency was at roughly eighty-five percent, and Donald Trump at fifteen — do those numbers sound familiar? Joe Biden led then President Trump in the next cycle, even though the race was still won by Joe Biden it was significantly closer than anyone expected. This number was not the exact results of a poll as we typically think of them but was rather a prediction made based on polling data, this is a somewhat nuanced distinction that for some was not worth making.

Throughout the last three decades, the United States has shifted towards increasingly partisan media. Partisan media is media that has the goal, intentionally or unintentionally, of catering to one particular type of political identity. A good example of this is how FOX News appealed to Republicans in American and CNN appealed to Democrats. These new outlets thrive off having an “other” to sensationalize events and thus frame their side “winning” and the other “losing.” This is turning politics into an increasingly brutal game of sport. As media outlets become more reliant on viewership and ratings, the presence of an “other” helps raises the stakes and encourage viewership. These strategies have been effective at cementing viewer bases, but detrimental to American politics. This has resulted in American news outlets that needed to satisfy their viewers either by portraying them as triumphant victors or the embattled resistance.

So, what happens when viewers find themselves in the opposite group of what they wish to be, such as when their party does not have the support of the general public and is in the minority? Viewers wish to appear as the majority. The solution to this problem is simple — show them evidence that they are what they say they are. Essentially, the media has to change the score of the game. Now, this essay is not claiming that FOX or CNN ever knowing told lies to the public. Rather, news outlets such as these may have at one time or another “cherry-picked” data to show realities that were not false but were not necessarily true either. While big news outlets are particularly guilty of this, they are not the only ones. Non-partisan groups, political parties, campaigns, and others all bear a responsibility for propagating bad polling, interpreting data in poor ways, and presenting the results as scientific facts while in reality the results merely say what they wish the data to show. This may be a harsh sentiment, but many groups are culpable for the strain that polls have put on our democracy. Polls are flawed on many levels, and many different groups have used them to their advantage or to advance an agenda. The rest of this section will focus on how exactly polls can be manipulated and why those polls are a detriment to our society.

Polls can typically be manipulated on two major axis: how the question is asked and how the data is interpreted. Just because data can be misrepresented does not inherently mean that it is misrepresented everywhere. But the ugly truth of the matter is that this type of manipulation is not uncommon. Often this manipulation is not malicious or purposeful. The New York Times did an excellent experiment on this theory that poll results change based on who interprets the data. The New York Times found that between five respected pollsters they received four different results. Though the methods are similar they ended up with mostly different results.

Often this manipulation is not malicious or purposeful. The New York Times did an excellent experiment on this theory in 2016 that poll results change based on who interprets the data. The New York Times found that between five respected pollsters they received four different results. Though the methods are similar they ended up with mostly different results.

The New York Times describes their methods in the following way:

Charles Franklin: Clinton +3

Marquette Law

Mr. Franklin approximated the approach of a traditional pollster and did not use any of the information on the voter registration file. He weighed the sample to an estimate of the demographic composition of Florida’s registered voters in 2016, based on census data, by age, sex, education, gender, and race. Mr. Franklin’s likely voters were those who said they were “almost certain” to vote.

Patrick Ruffini: Clinton +1

Echelon Insights

Mr. Ruffini weighted the sample by voter file data on age, race, gender, and party registration. He next added turnout scores: an estimate for how likely each voter is to turn out, based exclusively on their voting history. He then weighted the sample to the likely turnout profile of both registered and likely voters — basically making sure that there was the right number of likely and unlikely voters in the voter file. This is probably the approach most similar to the Upshot/Siena methodology, so it is not surprising that it also is the closest result.

Sam Corbett-Davies, Andrew Gelman and David Rothschild: Trump +1

Stanford University/Columbia University/Microsoft Research

Long story short: They built a model that tries to figure out what characteristics predict support for Mrs. Clinton and Mr. Trump based on many of the same variables used for weighting. They then predicted how every person in the state would vote, based on that model. It’s the same approach we used to make the pretty dot maps of Florida. The likely electorate was determined exclusively by vote history, not self-reported voice choice. They included 2012 voters — which is why their electorate has more black voters than the others — and then included newly registered voters according to a model of voting history based on registration.

Margie Omero, Robert Green, Adam Rosenblatt Clinton +4

Penn Schoen Berland Research

The sample was weighted to state voter file data for party registration, gender, race, and ethnicity. They then excluded the people who said they were unlikely to vote. These self-reported unlikely voters were 7 percent of the sample, so this is the most permissive likely voter screen of the groups. In part, as a result, it’s also Mrs. Clinton’s best performance. In an email, Ms. Omero noted that every scenario they examined showed an advantage for Clinton.

The wording of questions is also vital when conducting polls. Poorly worded questions can lead people on and cause them to answer in a disingenuous way, especially when multiple questions are combined into one or recipients' answers are limited by the available responses. For example, a Fox News poll in 2010 asked the question “Do you agree or disagree with the following statement: the federal government has gotten totally out of control and threatens our basic liberties unless we clear house and commit to drastic change.” A professor pointed out that there were four different assertions in the question that the respondent had to agree with to answer affirmatively, which could have easily skewed the results.

It should be stated somewhere in a paper about polling that at this point in history, polls drive money just as much as money drives polls. Polls make for great television and they get eyeballs for the exact reason this paper has said fourteen times now — polls boil complex issues and race into a score like one would see at a sporting event. Because of this, bad or misleading polling data, or bad analysis, can lead to substantial shifts in policy. 2012 is a fantastic tangible example of this phenomenon in which polling data, that was largely accurate but egregiously overhyped, claimed that President Obama had been re-elected purely on the Latino vote with immigration reform being a key issue. This data was largely taken out of context when analysts looked at exit polls from 2012. Republicans took notice of the fact that then-President Obama won the states of Nevada Colorado, and New Mexico, three states that up to that point had been reliably voting with Republicans. They were also three states for whom immigration reform was a popular issue among Latino voters who make a significant portion of the states’ electorate. Since the Republican Party is largely anti-immigration those Latino voters largely turned out for Democrats which did contribute to President Obama winning those states somewhat unexpectedly. The headline in the ensuing weeks was that Latino voters had won the presidency for Barack Obama. In reality, the polling data showed that Latino voters were a major factor in President Obama winning those three states but not a major factor in Obama winning the election as a whole. Without those three states, the President would have still won the popular vote and electoral vote by a considerable margin. Yet that is not how it was presented to the masses, simply because “Latinos win the presidency for Barack Obama” is a headline with more gravitas than “Latinos play a factor in Obama winning three states that he did not need.” In this case, accurate polling data with scientific integrity was warped by the media.

Republicans somewhat got swept up in this message and, in an attempt to win back these Latino voters, made comprehensive immigration reform a priority for the Republican Party leadership.

Like it or not, money in politics largely flows towards the winner which builds momentum. If a candidate is shown to be leading in the polls, they often get far more donations than their opponents regardless of party. That is why campaigns put such a strong emphasis on polling data if they are winning and ignore it publicly if they are losing. This also allows corporations to more easily buy influence in elections. If corporations have a way of knowing who is going to win, they know who to donate money to buy influence. Campaigns know that corporations will be far more likely to donate to them if corporations have reason to believe that they would win their electoral race. Thus the demand for favorable polls is put into place and it is more important that these be favorable rather than accurate.

At this point the question of “If polls are so bad why do we still have them?” is probably rattling around in your mind. The answer is threefold. First, the only people that seem to care are academics kicking up dust for very good reasons, Dr. Robert Wuthnow and the Brookings Institute both have excellent series and books on polling that don’t get many eyeballs because it is a far less exciting issue than an impending government shutdown or a trade war. As priorities go, reforming polling is relatively low on the list. Second, polling is entertaining and gives an easy way for anyone to understand complex political realities in a numerical way. It is also presented as an exact science. For better or for worse, this is the reality. It gives people that are not involved in politics an easy way to understand politics but it often gets oversimplified or manipulated. Finally, there is not really a real way to ban or restrict polling as it gives any idiot with a clipboard and enough time away to gauge public opinion. I would know — I’m one of those idiots with a clipboard. This essay only outlined a few methods of manipulating polls nefariously and didn’t even touch on how polls maybe just are conducted incompetently and all the impacts that may have on their outcomes. In reality, there are far more ways polling data can be manipulated and consumers of polling data should remain vigilant.

Overall polling is an issue that is quietly warping American politics in unforeseen ways. Polling is far too easy to manipulate and is presented as a science when it simply is not. Calling polling a pseudoscience is not even accurate because polling was never an exact science, to begin with. It was only an educated guess that was presented as a scientifically exact way of keeping score. While first steps towards fixing the problem may involve decreasing our reliance on polling and understanding what the information from polls really represents, ultimately there really is not a good way to move forward. Perhaps a better method of surveying public opinion will be developed in the future, but for now, we are unfortunately stuck with polls. As a population, it is up to us to start taking polls with a grain of salt. For non-politicos if you remember nothing else from this article remember the unofficial motto for the Obama 08’ campaign, “polls are shit get to work”.