Statistical Musings - Part I. I am a data scientist and an avid…
By: Apurva Misra
Tags: Hypothesis Testing, Randomization, Probability, Black Swan

Patterns

Humans have an innate ability to find patterns, it was an important skill from our hunter-gatherer days. We are hardwired to remember negative experiences and form patterns. The berries from this tree almost choked me to death or that weird looking animal was able to overpower my friend the other day, so it would be wise to stay away from them.

Looking for patterns is a useful skill, until it is not. The ability of humans to find patterns in completely random processes can draw us into many false leads. Humans have a hard time accepting a process to be completely random. If a normal coin is being tossed and we get heads three times in a row, we fallaciously think that the next one “must be tails to even things out”, even though both heads and tails are equally likely on any one toss, no matter what the previous pattern may have been.

Earthquakes: A completely random process, has a number of predictive models created trying to forecast the next tremor but till date no research has ever been able to predict the next disaster but that doesn’t stop us from putting our lab coats on.

Market: The same applies to the stock market, in this case I will give an example to prove that it is unpredictable. Let us assume a group of people have observed a pattern that a particular stock rises every Monday and gradually decreases over the week, what would they do? The obvious next step would be to buy this stock in high volumes on a friday to sell it on the coming monday for higher value but by doing so they themselves raise the price of the stock on friday, hence canceling the effect that they observed. In general, if you think you have found a pattern in the stock market and you are the coolest and are going to make a lot of money then you are basically wrong. If you have observed a pattern then fellow humans have also observed the same pattern and market will self-correct itself as shown above.

Fact: 65% of Americans think they are smarter than average, changing the very definition of what average means.

What other thing do humans have a hard time grasping?

Improbable events aka Black swans: Their name comes from the rarity of black swans in nature. A person who hasn’t encountered a single black swan in his/her life can just never state that black swans don’t exist unless he/ she has encountered the whole swan population. One single encounter with a black swan, however, proves the “No black swans exist” argument wrong. Improbable events are those which seem quite far-fetched cause you haven’t encountered them yet or they happened pretty far off in history and the world today is quite different so they end up in the “least expected” bucket. It needs one instance of it happening to prove the current belief wrong.

Availability Bias: Examples of swans that come to mind when we think about swans are representative of the population. (Lack of a black swan picture lead to above artistic experience)

Now is the time to introduce the limitations to Hypothesis testing and p-values. Let me briefly explain what hypothesis testing is before jumping on to the pros and cons. It is a statistical technique popular to compare two hypotheses one corresponding to the status quo and the other that the researcher is trying to prove right. For example, in the simplest form Hypothesis0: the vaccine has no effect. Hypothesis1: the vaccine provides immunity. A controlled random trial is done in which a randomly [VERY IMPORTANT, more on this below] distributed bunch of people(race, gender, location, etc.) are divided into two groups and one is administered a vaccine and the other is not. The difference in the reactions of the two groups against the particular disease is measured and a series of mathematical formulae are used to come up with the number “p” which is the probability of Hypothesis0 being true. If the number p is less than 0.05 then it is considered “statistically significant” that Hypothesis1 is true.

The first problem with this method of hypothesis testing is the hard threshold at 0.05 which is good to remove any sort of ambiguity but also leads to p-value hacking by scientists by manipulating the data, for example, classifying a relevant data point as an outlier to manage to get the p-value below

0.05. Why do this? Because if not, the research is going to go into the file drawer- journals are not really keen on publishing failed experiments and this creates a massive incentive problem because a study with a p-value of greater than 0.05 is considered to have NO information relevant to mankind. Failed experiments can help guide a research scientist the same way as a successful one can, but only if they are ever published.

The second problem, what does a successful hypothesis test mean? It does NOT mean that hypothesis 1 is true, it just means there is some potential and it should be investigated further. What if the experimenter just got lucky? Skepticism is a necessity while working with statistics.

Randomization: Consider a scenario in which you are trying to investigate whether mice prefer seeds/grains over a pizza, and you administer a survey to the mouse population in New York, it is more likely to result in mice preferring junk over other food items. Hence, biased results due to incomplete randomization which didn’t account for the population outside of New York.

Correlation is not Causation

Everything is correlated with everything else, the question is whether it is significant? Your favorite baseball team may win whenever you do your lucky dance, but does you wiggling your butt cause them to win?

If X and Y are correlated it does not mean X causes Y or Y causes X, there might be a Z that causes both X and Y. If drug use is correlated with tattoos that does not mean nudging people to go through a painful tattoo removal procedure will solve your drug problems. There might be a third factor like unemployment that causes both. Looking for the right problem to solve is always more work than coming up with the right solution.

So, where else is this relevant?

Information theory: Whenever we want to have a safe communication between two endpoints there are two things we are optimising for

Increasing the amount of information (decreasing correlation) transferred. Avoiding information loss.

One solution can be repeating each information bit twice so if the receiver finds a non-repeated pattern somewhere that would indicate a damaged information string. It can be repeated thrice to make it more robust but this isn’t optimizing for the amount of information that can be transferred. Correlation plays an important factor here to reach the optimal between the two needs.

Compression: “jpeg” an extension seen with pictures is a compression format, what does it do? In simple terms it just gets rid of the repeated bits in a picture, for example a wall in the background and compresses the whole picture down to the unique colours and brightness along with where they should be repeated.