Statistics is dead, long live statistics!  | Towards Data Science

Estimating confidence intervals and hypothesis testing are two common statistical tasks. They have an air of mystery around them as they rely on math accompanied by complex and case-specific assumptions. While this is still the way many people do stats, I argue it is not the best way. Modern computing power allows us to take advantage of resampling methods that are easier to understand and free of assumptions and approximations.
Back in the old days, when computing power was a fraction of what we have today, statisticians had no choice but to turn to mathematical functions such as the t-distribution to approximate sampling distributions. You will see the t-statistics everywhere, from assessing regression coefficients’ significance, through calculating confidence intervals, to A/B testing. These days, however, we can turn away from this assumptions-heavy approach in favor of the resampling methods.
Resampling boils down to repeatedly sampling values from the data in order to assess random variability in statistics calculated from these data. It comes in two main flavors:
Bootstrapping, which is used to assess the reliability of an estimate or prediction. It can be used to compute confidence intervals or prediction intervals (don’t confuse the two!), to improve the accuracy of machine learning models (as in bagging, or bootstrap aggregating, the basis of random forests and many boosting models), or to estimate the uncertainty from imputing missing data.
Permuting, which is used for hypothesis testing.
Let’s look at each of them and how to use them to solve common problems in a math-and-assumption-free, standardized, error-prone way.
Bootstrapping simply means taking many samples from the original data with replacement. These are called bootstrap samples, and since we are drawing with replacement, the same data point may appear multiple times in a single bootstrap sample. The point of this is to mimic obtaining many samples from a hypothetical population so that we can observe sampling uncertainty. We then perform the desired calculations on each bootstrap sample in isolation and combine the results into a distribution of some statistic of interest that reflects its variability.(quote from my article on bootstrapping confidence intervals)
The bootstrap is a unified and standardized approach. Whatever it is you want to achieve, you always follow the same steps: draw many samples with replacement, calculate whatever you care about on each sample, combine the results from all samples into a distribution, and use this distribution to learn about the variability in your estimate. Let’s look at some examples.
Bootstrap takes samples with replacement from the data to measure sampling uncertainty. It has a myriad of use cases.
Are the houses in south California more expansive than in the northern part of the state? Let’s run an A/B test based on the California housing data!
To do this, we can split the data by the median latitude and compare the average house price in the south ($222k) to the average price in the north ($191k), to get the difference of $31k. Note the dataset, and so the prices, are from 1997 — wish these were today’s prices, don’t we?
Is this difference significant? Or maybe, had we collected another portion of data, we would get a different number, perhaps even a negative one? Confidence interval is the answer. A 95% CI tells us that if we had collected many other data sets on houses in California and had run such an A/B test on each of them, in 95% of the cases the true average difference between the southern and northern prices (which we would know should we have data on all houses in California) would be covered by the interval.
Here is how to compute the 95% CI for our A/B test via bootstrapping:
Mean diff: 0.3080677771202604695% CI: [0.28223704 0.33397099]
We can be 95% sure that the interval $28k — $33k contains the true difference, so yes, houses in the south are pretty likely more expensive. And thanks to the bootstrap, we can say so without assuming that prices follow a normal distribution (which they obviously don’t). This is due to the magic of the Central Limit Theorem — bootstrap samples are random, hence independent, and so the difference in their means is normally distributed.
Let’s continue playing with the California housing data to find what was the mean house age in the state back in 1997, and what was its standard error. We can calculate the average house age in the data to be 28.6 years, but how accurate is this sample estimate with regards to the Californian houses in general? Let’s bootstrap to see!
Average mean: 28.6383311627907Standard error: 0.08733193966359917
The bootstrap estimate of the average house age corresponds to what we observe in our data, and we also got a bootstrapped standard error of 0.08 years, which is close to one month. This suggests our estimate of the mean age is pretty accurate!
In my last article, I have written about the difference between confidence and prediction intervals. The post includes code to bootstrap both in the context of regression models — don’t hesitate to take a detour to check it out!
towardsdatascience.com
It is a common practice, when tackling incomplete data, to fill in the missing values using some imputation method or another, and then run the analyses or train the models on such a filled-in dataset. This is a terrible practice!
Such an approach completely ignores the fact that imputed values are just estimates that come with some uncertainty and this uncertainty transfers to any model or analysis built on top of imputed data. In other words, any model prediction or statistical estimate that we report should be accompanied by some variability measure that incorporates the uncertainty from imputation. I have written about it here (an R example) and here (general remarks on handling missing data).
And once again, bootstrap to the rescue! We need to take bootstrap samples from the incomplete dataset, impute each of them in isolation, train a model on each of them separately, and then combine the predictions from all these models into a distribution. We can then compute intervals or standard deviations based on this distribution to quantify the uncertainty in the predictions that partly comes from imputation.
Bootstrapping seems to work like magic. It lets us approximate any quantity we want, even those for which classical statistics knows no formulas, and informs us about the uncertainty in its estimates, all of this assumption-free.
But don’t forget that the bootstrap doesn’t create more data or compensate for a small sample. It only works if the data we have is representative of the population of interest. If that’s the case, then bootstrapping tells us what would have been, had we collected more data.
Next to bootstrapping, the second resampling method is permuting. In probability theory, a permutation of some set of items is simply a reordering of the items in this set. For example, the set of numbers [1, 2, 3] can be permuted in five different ways to obtain the following sets: [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 1, 2] and [3, 2, 1]. This simple idea enables us to perform hypothesis testing in a formula-free manner, in a way that is easy to interpret and understand and without relying on distributional assumptions, which is typically the case in the classical approach to testing.
To understand how permutation testing works, we need a quick refresher on classical hypothesis testing (for a more in-depth elaboration, check out this article). To stay down-to-eath, we will look at the California housing data again. This time, let’s test if the house prices in the east and in the west of the state are the same.
The test that we want to conduct is a test for the difference in means: we want to know whether the mean house price in the east ($215k in the data) is the same as the mean house price in the west ($199k in the data). Is the observed difference of $16k a result of random chance, or were the houses in the east really more expensive, on average? This use case is well-served by the t-test. Let’s walk through the testing procedure step by step.
First, we need to set up our hypotheses. The null hypothesis will be that the average house price in the east is equal to the average house price in the west, and we will try to reject it in favor of the alternative hypothesis that they are not equal.Next, we need something called the test statistic. A test statistic is a single number calculated according to some formula, specific to the test we are conducting. It has to fulfill two conditions: it needs to be calculatable from the data and we need to know its distribution assuming the null hypothesis is true. Luckily, the case of our test for the difference in means has been well-researched ages ago and we know that if the average prices are equal (the null hypothesis is true) and we observe some difference in the averages only due to chance, then the following test statistic has a t-distribution.
First, we need to set up our hypotheses. The null hypothesis will be that the average house price in the east is equal to the average house price in the west, and we will try to reject it in favor of the alternative hypothesis that they are not equal.
Next, we need something called the test statistic. A test statistic is a single number calculated according to some formula, specific to the test we are conducting. It has to fulfill two conditions: it needs to be calculatable from the data and we need to know its distribution assuming the null hypothesis is true. Luckily, the case of our test for the difference in means has been well-researched ages ago and we know that if the average prices are equal (the null hypothesis is true) and we observe some difference in the averages only due to chance, then the following test statistic has a t-distribution.
3. The Xs with dashes on top of them denote the mean prices in the two groups (east and west), the s2 are the variances in the prices, and n is the number of observations, assumed the same in both groups. We can use this formula to compute our test statistic.
t-stat: 9.627149556121305
4. Finally, we need to use the fact that we know that if the null hypothesis is true, then our t-statistic has a t-distribution with (2*n)-2 degrees of freedom. It looks like this:
From this, we conclude that if the null hypothesis was true (if the average prices were the same in the east and the west), then the value of the t-statistic we have obtained would be nearly impossible to get. This means that most likely, the null is false and we should reject it: the average prices differ.
5. We can quantify this decision by calculating the p-value. The p-value is the probability that, given the null is true, we can get the test statistic that we actually got from the data (or even a more extreme one). In other words, it is the percentage of the blue mass that is located to the right of the dashed line, which in this case seems to be close to zero. Let’s compute it, this time using a scipy function. Calling scipy.stats.ttest_ind(price_east, price_west) gives us the following output:
Ttest_indResult(    statistic=9.627692430826853,     pvalue=6.791637088718439e-22)
You can see that the test statistic of 9.627 matches our manual calculation and that the p-value is virtually zero. Hence, we reject the null hypothesis. Phew!
Notice how we have relied on the approximating formula for the t-statistic, which is based on multiple assumptions. We have implicitly assumed that the variances and the number of observations are the same for western and eastern prices. We have also assumed that these two data subsamples are independent and normally distributed. And finally, that the sample size is large enough. We are at five assumptions already, some of them clearly not holding (prices are rarely normally distributed!).
Classical hypothesis tests come with tonnes of case-specific assumptions. In more complex cases, ready formulas might be unavailable. Permutation testing fixes both these issues.
Sure, you can adjust the formula to meet your needs, but firstly, another formula will come with another set of hidden assumptions, and secondly, in cases more complex than a simple t-test, closed-form formulas might not even be known for your specific use case. Enters permutation testing!
The general idea behind permutation testing is to take all available data and permute it randomly. This corresponds to the null hypothesis that there is no effect. Next, we repeatedly draw samples (without replacement) from this combined and permuted dataset, calculate the quantities of interest and combine them into a distribution. We know that this distribution embodies variation that is produced by chance alone due to the permutation step. Finally, we compare the quantity of interest we observe in our data to the distribution of its random variation. If it is located well within this distribution, we haven’t proved anything — there is no evidence to reject the null hypothesis since our result might have been produced by chance. If, however, our observed value finds itself far away from the distribution, we have grounds to say it’s unlikely that it was produced by chance and we reject the null hypothesis. We can also calculate the p-value, just like in classical testing. Let’s see it in practice!
In our east-west price test, we will need to combine all prices, repeatedly permute them, create two groups of sizes matching the original groups, and compute the differences in average prices. This way, we get the distribution of the differences produced by chance. Let’s compare it to our observed difference of $16k.
The difference in average prices between the east and the west that we observed is much larger than what might be produced by chance, so we will reject the null hypothesis. Let’s quantify it by calculating the p-value. Since the p-value is the proportion of the blue mass to the right of the red line, we can easily compute it as follows.
In this pretty evident case, the p-value is of course zero, just like in the classical test.
In a pretty similar fashion, we can run A/B/C testing known as the analysis of variance, or ANOVA. Say we are testing different color themes for our website with the goal to engage the users.
As part of the experiment, we have randomly shown the users different website variants and we calculated the time they spent on the website. Below are the data for 15 users (five were shown the yellow version, another five the blue one, and the last five the green one). Naturally, for such an experiment to be valid, we would need many more than 15 users, this is just a demonstration. We are interested in the extent to which the differences between the means are larger than what might have been produced by random chance. If they are significantly larger, then we might conclude that the color indeed impacts the time spent on the website.(quote from my article on probability distributions)
While we could do it based on the statistical theory and the F-distribution (check the F-distribution section in this article to see how to do this), we can also use permutation testing.
p-value: 0.00844
Let’s visualize the observed variance on top of the permuted ones.
You will find that the p-value is very close to what we would get had we followed the classical F-test approach. Since it is rather small, we conclude that the color does impact the website’s stickiness.
The χ2 or chi-square test is in some sense a version of ANOVA for discrete data. Consider the three colors of the website once again. This time, instead of time spent on the site (a continuous variable), we have measured the number of purchases made through the site (a discrete variable). We want to know which color leads to the most purchases. Each color has been displayed to 1000 users, and here are the results:
We could run a classical chi2 test (see how to do it here) but an assumption-free permutation test might be a better option.
The first step is to assume that the three website versions generate the same number of purchases, on average (our null hypothesis). If that was true, we would expect the same number of purchases for each version, and it would be (17+9+14)/3, or 13.33 purchases.(quote from my article on probability distributions)
Hence, in our permutation test, we generate a distribution of chance-generated statistics by assuming 40 purchases and 2960 non-purchases across all website colors, and we sample from this. The chi2-statistic we are interested in is the sum of squared differences between the permuted purchases and the expected purchases.
p-value: 0.270826
The high p-value leads us to conclude that the differences in the number of purchases for differently-colored websites can be caused by random chance alone.
Interestingly, the χ2 distribution used in a classical chi-2 test is actually only an approximation of the real distribution of the test statistic. For this reason, the resampled p-value value from our permutation test is slightly different than the one we could get from a classical test (we got 0.3 there). This is yet another piece of evidence in favor of the resampling approach — no approximations are needed here.
Some of the ideas and code in this article are based on:Bruce, Bruce & Gedeck (2020), Practical Statistics for Data Scientists, 2nd edition, O’Reilly
Thanks for reading!
If you liked this post, why don’t you subscribe for email updates on my new articles? And by becoming a Medium member, you can support my writing and get unlimited access to all stories by other authors and myself.
Need consulting? You can ask me anything or book me for a 1:1 here.
You can also try one of my other articles. Can’t choose? Pick one of these: