Hypothesis testing, a simple illustration of the superiority of the Bayesian framework | by Alain Tanguy | Towards Data Science

Usually hypothesis testing is not the first topic that comes to mind when we talk about bayesianism. The current trend is indeed more about incorporating this approach to Machine Learning models, and less about reviewing old-fashion statistics.
Nevertheless, more and more studies1 reveal the extent of the failure of so many reasearch papers that can be attributed to the misunderstanding/mis-use of “elementary” statistic methods. Given the extent of this phenomenom, it is quite obvious that such a is not the result of scientists personal lack of serious in that regard, but more of a global systemic anomaly which has been going on for quite a long time now. In light of this, it seems that more intuitive and easy to interpret methods should be considered.
Misconceptions over what p-values really lie at the root of this situation, or at least embody this reality the most. Most data-scientists are nowadays well educated when it comes to Machine learning, to the detriment of even the most basic statistic concepts ; hypothesis testing is among those.
We are now going to explore one of the simplest test a statistician may have to run, and from there see how to build a bayesian approach, that allow both a simple and forthright conclusion.
We’ll consider one of the simplest experiments there is in order not to lose sight of the point we want to make with unnecessary convolutions: Coin-flipping. The problem is defined this way:
A coin can be flipped a limited amount of times.
Our objective is to find if this coin is fair or not. in other words, we want to know how likely is our hypothesis (H0) “the coin is fair” to be false.
Pretty simple right?
The design of the experiment should be made considering the statistical method that will be used to answer our question. A student t-test appears as an obvious choice for our hypothesis testing. We will compare the mean of our sample to the value 0.5.
Good practice wants us to calculate the statistical power when designing our study in order to estimate the appropriate sample size, or at to least tell us about the limitations of our study if sample size is already fixed (it is commonly required to have at least a power of 0.8)
The power of a binary hypothesis test is the probability that the test rejects the null hypothesis (H0) when a specific alternative hypothesis (H1) is true.
In other words, the power of the test is the probability of getting a p-value under a specified threshold when the coin is actually unfair.
For convenience, we will only consider an alternative hypothesis (H1) “the coin follows a Bernoulli law of parameter p = 0.4”, which will allow us to easily compute the power of our test.
A look at the ROC curve of this test will help us understand what is going on. For a given significance level α, we can derive both the True and False Positive Rates, analytically or from simulation; or in other words, the statistical Power of the test and the significance level itself.
Usually α= 0.05 is the limit accepted, and for this threshold a minimum amount of 192 flips is required to reach our objective (and since flipping coins is not the 1st thing that comes to mind when one thinks about entertaining activities, we will limit ourselves to flipping this coin 192 times!).
Here comes our problem now. Assume we found that the coin is unfair with a p-value = 0.01, what can we conclude now?
Is there a 1% chance that the Null hypothesis is True?
Are we 99% sure that our coin is unfair?
None of these as you probably have already concluded. Actually, at this point, there is no way to give a probability of our coin being unfair... Quite annoying, especially given our original question: How likely is our coin to be unfair? The only conclusion we can draw is, had the coin been fair, we would only have a probability of 0.01 to observe such extreme results.
In fact Bayes theorem gives us a way to compute this value, but we need more information: priors. With H0 = “coin is unfair”, H1= “coin is fair”, and S = “test succesful with p-value < 0.05”, we have:
P(H0 | S) = P(S | H0) * P(H0) / (P(S | H0) * P(H0) + P(S | H1) * P(H1))
Initially assuming our coin more likely to be fair, we set P(H1) = 0.9 and P(H0) = 0.1. Thus we have
P(H0 | S) = 0.8 * 0.1 / (0.8 * 0.1 + 0.05 * 0.9)
P(H0 | S) = 0.64
So only 64% chance for our coin to be unfair even if our test came positive. Quite far from the 95% we would have hoped for... Yet this probability carries much more insights than a simple p-value, and luckily we can go further. This was just a first step into the bayesian world, and as we are going to see now there is much more to unfold to improve our test!
For this part, we are going to use the PyMC3 library for python, allowing us to easily build Bayesian nonparametric models.
We model the sampling of an unfair coin (p = 0.4) and compute the resulting posterior distributions. Though a non-trivial Markov chain Monte Carlo (MCMC) algorithm is used, the very idea behind it is simply the application of Bayes theorem.
Now that we understood how to transpose our regular hypothesis testing in bayesian terms, we can adjust the prior distributions to reflect our true belief of the initial state of the system: A beta distribution centered in 0.5, would highlight the fact that we believe that if the coin is biased it must not be as extreme as 0 or 1.
import pymc3 as pmN = 192 #Sampling sizewith pm.Model() as model:    p = pm.Uniform("p", 0, 1)    observations = pm.Binomial("observations", N, 0.4)    obs = pm.Binomial("obs", N, p, observed=observations)    step = pm.Metropolis()    start = pm.find_MAP()    trace = pm.sample(100000, step=step, start=start)    burned_trace=trace[1000:]    p_samples = burned_trace["p"]
We can also provide non-informative priors, as some many bayesianists would recommend. For example, we could assume a uniform prior for p over [0; 1].
p = pm.distributions.continuous.Beta("p", 2, 2)
In each case, we can directly compute the probability that p < 0.5 from the posterior distributions.
print(np.mean(p_samples<0.5))
We get respectively 0.98 and 0.97 with the uniform prior and the beta(2,2) prior, informing us of the degree of confidence we have that the coin we observe is actually fair or not.
In his paper Bayesian estimation supersedes the t-test (BEST)2, John Kruschke suggests looking at what he calls Highest Density Intervals, consisting of the smallest intervals containing 95% of the posterior density in order to define our tests criteria.
For example in our case, we would want to know if these intervals overlap with the value we are testing against 0.5, and conclude that (H0) is rejected if 0.5 was not included in those.
Reasoning with densities enables us to exploit all of the information we have access to. Better, it allows us to capture all the nuances from the data and draw non-ambiguous conclusions.
One could argue that our analysis is biased given that we had to pick a prior. I will not discuss here the difference of paradigm between frequentism and bayesianism, but will simply recall that not choosing a prior is often equivalent to implicitly picking one.
The Bayesian framework provides an easy to perform and easy to read alternative to classic statistical tests.
Posterior densities provide us with mathematical concepts much more convenient to grasp and use in comparison with notions such as p-values which have proven to be misleading countless times.
Eventually, the concept of the Bayesian network allows us to conceive much more complex experiments and to test any hypothesis by simply considering posterior distributions, as we observe with the case of A/B testing.
T.V. Pereira and J.P.A. Ioannidis. “Statistically significant meta- analyses of clinical trials have modest credibility and inflated effects.” Journal of Clinical Epidemiology 64, no. 10 (2011): 1060– 1069. DOI: 10.1016/j.jclinepi.2010.12.012.Kruschke, John. (2012) Bayesian estimation supersedes the t-test. Journal of Experimental Psychology: General.
T.V. Pereira and J.P.A. Ioannidis. “Statistically significant meta- analyses of clinical trials have modest credibility and inflated effects.” Journal of Clinical Epidemiology 64, no. 10 (2011): 1060– 1069. DOI: 10.1016/j.jclinepi.2010.12.012.
Kruschke, John. (2012) Bayesian estimation supersedes the t-test. Journal of Experimental Psychology: General.