The Most Fundamental and Controversial Debate in Statistics and Data Science
By: Andre Ye
Tags: Machine Learning, Data Science, Data Analysis, Statistics, Computer Science

The Most Fundamental and Controversial Debate in Statistics and Data Science

The Frequentist-Bayesian Dispute

Statistics is an ever-evolving field, constantly catching up with the exponentially rising number of applications and sources of data. As statistics and data science — encompassing everything from machine learning and AI to the figures politicians state matter-of-factly to prove their point — expand rapidly, a 250-year-old divide on the fundamental nature of statistics is widening just as quickly.

The Bayesians and the Frequentists reflect different perspectives on the process of doing science. Because the dispute is so fundamental and split at the root of statistics and data science, controversy between the two schools of thought breaks out in many places, from the world of A/B testing to social research to general figures (what’s the average height in the US? You can’t measure everyone). In fact, take any well-known statistics problem like the German Tank Problem and there will likely be two approaches/solutions: a Frequentist one and a Bayesian one.

Broadly speaking, the 19th century was dominated by Bayesian statistics, whereas the 20th century was Frequentist. Where the 21st-century lands on the Bayesian-Frequentist scale, with its exponentially rising demand for massive quantities of data to be sifted and processed, will see that view of statistics and data science amplified on a magnitude never imagined before.

Bayesian statistics is better-suited to using all information at its disposal to make rapid progress — in pursuing scientific progress, Bayesians tend to be more aggressive and optimistic in their modelling assumptions, sacrificing rigor in confirming the underlying premises of a model in order to give it more perspective (pulling in data from other sources). Frequentist data scientists, on the other hand, are more cautious and defensive, aiming for universally acceptable conclusions that will stand up to harsh adversarial scrutiny. The Frequentist model of science will sacrifice some gain that might be able to improve the model if it risks the academic and statistical integrity.

Often, the most clean-cut difference used to differentiate the two groups is the openness with which they accept new and external data, which may make the model more aware but also may flaunt or even violate the underlying assumptions of the model. Frequentists feel most comfortable using only data and the parameters of the current experiment, because it can be compared apples-to-apples, even if it does not give the model much perspective. Bayesians will attempt to include data from, say, different external experiments, which they argue will improve the model such that it doesn’t matter if the data cannot be put on a strict apples-to-apples correspondence. A Bayesian method of thinking will constantly update a model or calculation with new information and data, whereas a Frequentist will settle for constant parameters.

This controversial comic pokes fun at the Frequentist’s conservatism with the introducing of external data in the current experiment:

A controversial Frequentist-Bayesian comic. Source. Free to share.

The current parameters of the experiment dictate that:

Each die has a one-sixth chance of landing. There are two dice, so there are 36 total possible combinations. If the two dice fall into one of the 36 possible combinations, the detector is lying. Rephrased, it has a 1/36 chance of lying.

Given the parameters of the experiment, the Frequentist finds the p-value, a measure of how likely the experiment yielded false results (think 5 out of 100 patients in a clinical trial not supporting the hypothesis, the p-value is 0.05), to be less than 0.05. This p < 0.05 standard is so set in stone in the academic world that entire fields of p-hacking — distorting your results such that the p-value standard can be met and the study can be published in an academic journal — have grown tremendous popularity. Because the chance that the study’s finding is a fluke is less than 0.05 and the neutrino detector declares that the sun has exploded, the Frequentist proclaims that the sun has indeed exploded! to which the Bayesian quips, “Bet you $50 it hasn’t.” (In reference to Bayesian scholar Bruno de Finetti, who extensively used bets in his thought experiments.)

Of course, this is silly. The Frequentist doesn’t factor in the fact that the probability of the sun exploding in the first place is extremely small — much, much smaller than 1/36 — meaning that, even when factoring in the 1/36 chance of lying, the sun still has a very small chance of exploding. In fact, the Bayesian equation, used to calculate the probability of an event occurring given another event is true, can be used in this scenario:

In the Bayesian equation and in Bayesian statistics, external information is fed in the form of a prior. In this case, the prior comes in the form of the chance that the sun has exploded (one in 4.38×¹⁰¹³), which is combined with the parameters and conditions of the current experiment.

Although this contentious comic is a humorous stab at the differences between Bayesians and Frequentists, there’s plenty that is left out. Even Randall Munroe, the author of the comic, has acknowledged:

I seem to have stepped on a hornet’s nest, though, by adding “Frequentist” and “Bayesian” titles to the panels. This came as a surprise to me, in part because I actually added them as an afterthought, along with the final punchline. … The truth is, I genuinely didn’t realize Frequentists and Bayesians were actual camps of people — all of whom are now emailing me. I thought they were loosely-applied labels — perhaps just labels appropriated by the books I had happened to read recently — for the standard textbook approach we learned in science class versus an approach which more carefully incorporates the ideas of prior probabilities.

The ‘experiment’ of the sun exploding is not repeatable, an important factor to consider when comparing the two approaches. However, the underlying issue with this contrast is that it is simple and makes perfect sense to integrate the prior into the calculations (all for the punchline, I suppose). Unfortunately, statistics and data science problems in the real world are never as clean-cut.

Consider, for instance, the primary battleground of the Bayesian-Frequentist conflict: the A/B test, which is commonly used by companies to compare two groups (A and B) where everything is kept the same except for one variable. The difference between the two groups can then (usually) be attributed to the change in that variable; for example, changing the color palette from light to a little bit darker may significantly increase interaction with the website. Every time you visit a website from a reasonably large company — Google and Microsoft, for example — you are probably engaged in dozens of A/B tests. Successful A/B tests have made their operators billions of dollars (literally).

Say our company, Macrosoft, which sells gigantic computers, wants to perform an A/B test to determine if we can increase the number of people that buy products off the site by changing the font. Whenever a user logs on to Macrosoft, they are split into either the A or B category and are shown content in the corresponding font.

Created by Author.

1000 people participated (unknowingly, of course) in the A/B experiment. 500 were in group A and 500 in group B, where admission into a group was randomly decided. The A/B test yields the following results:

20% of users in Group A bought products. Hence, if a user was in Group A, they had a 20% chance of buying a product.

25% of users in Group B bought products. Hence, if a user was in Group B, they had a 25% chance of buying a product.

At this point, the Frequentist would conclude that changing the font to that of in Group B, the Macrosoft would see a 5% increase in those products purchased through the site. To be more and more confident of this change, the Frequentist would add the number of people in the experiment, expecting the result to inch closer and closer to the real value as the number of subjects increases. Known as the Law of Large Numbers, this idea is the foundation of Frequentist statistics.

The Bayesian wouldn’t be so sure of that conclusion. The Bayesian retrieves data from a study that found no significant emotional or mental difference between different sets of commonly-used fonts and factors it into his or her calculation. When this study is factored in, the Bayesian finds no difference between group A and group B — the 5% difference was just a random fluke. Had the company spent hours changing the website fonts, nothing would have changed. This reflects the primary goal of Bayesian data science: to attempt to put more context behind the face value of a result.

The Frequentist isn’t willing to accept the Bayesian’s conclusion. “The study has different premises than ours,” he/she says. “It drew from a different population size, had different contexts, and different goals. It’s not an apples-to-apples comparison and hence shouldn’t even be factored into the calculation.” The Bayesian fires back, “What does it matter? It’s better than taking these results at face value, which are inevitably biased.”

There’s no clear answer as to which perspective to trust, although often statistics or data science teams at universities, companies, and other institutions often choose and stick with one to be consistent. Yes, the Bayesian may give the model more perspective, but from the Frequentist’s view, adding external data will cause more problems — introducing new dimensions of bias and underlying assumptions — than it solves. This centuries-old debate will carry even greater quantities of importance as the need to make smart decisions from overflowing rivers of data carry billion-dollar stakes.

Let’s see where 21st-century data science lands.