Statistics and It’s Love-Hate Relationship with Science
By: Sruthi Korlakunta
Tags: Data Science, Big Data, Business Intelligence, Books, Statistics

Statistics and It’s Love-Hate Relationship with Science

Statistical significance and Science. Source: Giphy

Why does the toothpaste commercial want you to know that 9 out of 10 dentists approve their product?

This article addresses statistical significance, a popular convention in modern empirical science and statistics.

Although numbers form the heart of statistics, only when an interpreter converts the problems of math to usable formulations in language, it can be put to any real use. This makes statistics a double-edged sword. On the one hand, numbers have a concrete meaning. One slip of interpretation, they turn your world upside down.

If you are a business intelligence expert, you definitely have experienced it at work. Absolute numbers tell you less than percentages and percentages tell you less than the percentages compared to last year.

The numbers don’t change, but the story you tell with them does every time you see a new perspective.

In a way, you can be glad because numbers can be automated but interpretations as of now, need human finesse. But human finesse is associated with a multitude of cognitive biases [Thinking Fast and Slow, Daniel Kanheman]. Though heavily mathematically equipped, we are not wired to be mathematical creatures. we as of now don’t know how to feel 38% happy with something [Fooled by randomness, Naseem Nicolas Taleb]. We are either happy or sad.

The problem with Empirical Sciences

Empirical sciences and statistics have a love-hate relation with each other. Empirical scientists are eager to achieve the magic threshold of statistical significance to prove their propositions. Statistics on the other hand is a reluctant participant in establishing cause and effect.

For instance, statistics can measure a correlation of 100% between your toilet breaks and the number of goals scored against your favorite team. But it is merely an unwitting participant in your conclusion to never pee again.

While empirical scientists are eager to prove and disprove theories and toss them into black and white piles, only the white pile grows larger each day. The “disproves” don’t get a paper to their name because it is unexciting to read a paper titled “The color of your car is irrelevant to your probability of getting cancer”.

On the other hand, it is very exciting to publish a statistical significance report on how every time Miley Cyrus drops a new single, the Himalayan tectonic plate moves a centimeter closer to Eurasia. (This is not a fact)

Statistical significance and the null hypothesis

What is “Statistically Significant”?

If you are testing the impact of a cause on an effect by observation, a statistically significant proof is when there is substantial evidence from observations that the observed “effect” is not by mere chance.

Null Hypothesis, source: Giphy

Say you are testing a drug for causing happiness. You want to claim that the drug is effective. So you will start by saying “Assume that the drug has no impact on happiness whatsoever”. And then you will go on to prove that this is not correct.

For you to claim that your drug is effective, you have to prove that

There were enough number of days where you were happy after taking the drug There are enough days you were NOT happy when you did not take the drug.

The “enough” number is the statistical significance threshold. If the opposite is true, that is if the drug actually has no bearing on your happiness, it established the null hypothesis.

How do we measure statistical significance?

Photo by Charles Deluvio on Unsplash

The value measured to prove statistical significance is called the p-value. It is the probability value of the observed results, assuming that the null hypothesis is true. The threshold accepted in the research community for a statistically significant result is P ≤ 0.05.

Is it a god-given number found in nature? Is it an observed value, like acceleration due to gravity =9.8 m/s²?, like the meaning of life = 42? (Hitchhiker’s Guide to the galaxy, cheers if you got the reference). This value, like many things in statistics, was established, I assume, per intuition by F.A Fischer in the early twentieth century.

Let’s see how it works:

Consider the following excellent example by Jordan Ellenberg. for you to check for both conditions 1 and 2 above for your drug, you take 100 bunnies and test their happiness after giving 50 of them the drug and 50 of them a placebo. But it is not enough to show that the 50 drug-bunnies are happy. You should also show that the non-drug bunnies are relatively unhappy.

Imagine that out of 50 non-drug bunnies, in 5 days, 45 are happy. the probability of finding a happy bunny if the drug would have no effect at all after 5 days (as is the case in the non-drug bunnies), is therefore 45/50 = 90%. Now look in your drug-bunny lot. You see that all bunnies are happy.

The probability that a bunny is happy when one does nothing is 90% of the total, as observed from the placebo lot.

The probability that a second bunny is also happy is 90% of the remaining 90%, 0.9*0.9 = 0.81

The probability that the next bunny is also happy is 90% of the remaining, 0.9*0.9*0.9, and so on

With each new happy bunny, the probability that the next bunny is happy falls. The probability, therefore, of all bunnies being happy is (0.9) multiplied to itself 50 times, 0.005, which is one in 200 times. The observed number of 50 out of 50 happy bunnies is, therefore, an event of slim probability under the assumption of the null hypothesis. Thus you statistically proved that your drug has an impact. The tinier your p-value, the happier you can be.

Can you trust statistical significance?

The above example is more than convincing. If your results are that glaringly significant, maybe it means that they are correct. Then what is all the noise about?

Charles Darwin himself mused that if his observations about evolution are a mere chance, It would be a wonder that so many observations in the natural world fit his theory.

Source: Giphy

Unfortunately, this convention of meeting the statistical significance threshold has caused a furor in many scientific communities. Statistical significance is only as powerful an instrument as the data set it measures.

If a bigger dataset or a larger demographic or a different area is chosen, it is very likely that the significance results differ vastly. At the same time, It is very unlikely that some cases like drug tests will even reveal a null hypothesis. The odds that the null hypothesis is exactly found are very less when you massively alter your body chemistry with a drug.

In the words of Jordan Ellenberg, it is more likely that something will cause or prevent cancer than do nothing! this extreme unlikelihood of the null hypothesis leaves many loopholes for desperate researchers to botch results using clever language.

As a result, it is technically scientific to splash tomatoes against a wall and tell the future with the patterns they make because you had it correct about 100 times. The 1000 tomato-splashers who don’t get it right don’t write a paper. If a thousand tomato splashers try splashing tomatoes a thousand times, it is not a wonder if one gets the future right 100 times by mere chance.

Does it mean that all scientists false-report?

So are we to believe that scientists and statisticians are unfaithful to their science? Is there no true scientific inquiry?

To be fair, science and philosophy share one common pitfall. Both the scientist and the philosopher believe that they are in pursuit of the truth. But it is very easy to stray from the path and end up pursuing evidence for what one already believes is the truth without realizing it.

Add to it the pressure of having to publish for your employer, publish for your sponsor, find evidence for something (anything!), the stringent course of time experiments need for validating, and the possibility that the research is outdated in the market after the n years it takes to do it. Truth is by no means a trivial pursuit.

As a reader, it is essential to know that one has to take results and their meaning with a pinch of salt. Accept that hardly anything is black-and-white. The more parameters involved in coming to a conclusion, the better. But the more parameters involved, the more complex is the question and its less it’s ability to be answered by observation alone. It is also a great practice to look for the repeatability of the results. Did 10 other researchers find the same thing? did 10 researchers all come from the same demographic and time-period? Such are the questions you should ask to make your interpretation of results more meaningful.