Bootstrapping the Standard Deviation | by Ben Bogart | Towards Data Science

Bootstrapping is a resampling method that allows us to infer statistics about a population from a sample. It is also easy to perform and understand, which makes it so darn cool. Practitioners who use bootstrap or fully appreciate its potential know that they can use it to estimate various population statistics, yet nearly all examples I could find online only use bootstrapping to estimate the population's mean. I think it's time to change that.
In this short article, I will review the bootstrap method and how to execute it in python. Then we’ll estimate the confidence intervals for the population's standard deviation using this method to alleviate any confusion around how to bootstrap population statistics other than the mean from a sample. We’ll do a little visualization to understand better what we learned and experiment with drawing a larger number of samples to see how this affects the outcome.
Let's dive in.
If you prefer, you can follow along in the Jupyter Notebook here.
Start by importing all the packages we will need.
import pandas as pdimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlineimport seaborn as snsimport scipy.stats as st
Now, let's generate a fictitious “population.” I made up the mean and standard deviation. You can make up your own if you wish.
# generate a ficticious population with 1 million valuespop_mean = 53.21pop_std = 4.23population = np.random.normal(pop_mean, pop_std, 10**6)# plot the populationsns.distplot(population)plt.title('Population Distribution')
We’ve now created a “population” with one million values, a mean of 52.21, and a standard deviation of 4.23.
We want to draw a small sample from the population to use for bootstrapping an approximation of the population parameters. In practice, we would only have the sample.
# Draw 30 random values from the populationsample = np.random.choice(population, size=30, replace=False)
The variable sample now contains 30 randomly drawn values from the population.
I’m going to go quickly here. If you want a more in-depth look at the bootstrap method, check out my previous article Estimating Future Online Event Donation Revenue for Musicians and Nonprofits — Bootstrap estimation of confidence intervals with python.
All of the magic with bootstrapping happens as a result of sampling with replacement. Replacement means that when we draw a sample, we record that number then return that number to the source so that it has the same chance of being selected again. This is what creates the variation we are analyzing to make inferences about the population.
We can think of this as drawing numbers from a hat (or a boot if you prefer).
Count the number of records in your hatDraw a random record from your hatRecord the value of that record and put the record back in the hatRepeat until you have recorded the same number of values as you have records in the hatCalculate a metric on that sample, such as mean or standard deviation, and record itRepeat the process 1000 or so timesCalculate quantiles to get your confidence intervals
Count the number of records in your hat
Draw a random record from your hat
Record the value of that record and put the record back in the hat
Repeat until you have recorded the same number of values as you have records in the hat
Calculate a metric on that sample, such as mean or standard deviation, and record it
Repeat the process 1000 or so times
Calculate quantiles to get your confidence intervals
Let's do this with numpy which is much faster than drawing numbers from a hat.
bs = np.random.choice(sample, (len(sample), 1000), replace=True)
Now we have a 30x1000 numpy array with each of 1000 rows representing a new sample.
To infer the population mean, we can take the mean of all rows and then the mean of those means.
bs_means = bs.mean(axis=0)bs_means_mean = bs_means.mean()bs_means_mean
54.072626327248315
That's not bad. Remember the population mean was 53.21.
But that’s not the useful part. Had we taken the mean of our sample, we would have gotten similar results.
sample.mean()
54.12110644012915
The real value is in the confidence intervals we can generate from bootstrapping. We’ll go back and use our bs_means variable, which contains the mean of each of our 1000 resampled samples. To calculate a 95% confidence interval, we can take the 0.025 and 0.975 quantiles of those means. 95% of the means of our resampled samples are between these two quantiles, giving us our confidence interval.
lower_ci = np.quantile(bs_means, 0.025)upper_ci = np.quantile(bs_means, 0.975)lower_ci, upper_ci
(52.41754898532445, 55.602661654526315)
Amazing, right? If this same sample had been drawn from a large number of different possible populations, 95/100 of those populations would have a mean between 52.54 and 55.77.
That's the point where most bootstrap examples end, but we want to see how well we can recreate our population, so we will also need to calculate a standard deviation.
The good news is the process is the same, except we take the standard deviation of each resampled sample instead of the mean. In the same we previously took the mean of the means, we can now calculate the mean of the standard deviations.
bs_stds = bs.std(axis=0)bs_std_mean = bs_stds.mean()bs_std_mean
4.398626063763372
And we can calculate the confidence intervals for the population standard deviation by finding the quantiles as before.
lower_ci = np.quantile(bs_stds, 0.025)upper_ci = np.quantile(bs_stds, 0.975)lower_ci, upper_ci
(3.3352503395296598, 5.387739215237817)
The 95% confidence interval for the population standard deviation is 3.34 to 5.39.
Let’s plot that out to get a better idea of what it means.
# plot the distributions with std at the confidence intervalsdist_low = st.norm(loc=bs_means_mean, scale=std_qtl_025)y_low = dist_low.pdf(x)dist_high = st.norm(loc=bs_means_mean, scale=std_qtl_975)y_high = dist_high.pdf(x)#plotfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))plt.sca(ax1)sns.distplot(population, ax=ax1)plt.plot(x,y_low, color='red')plt.plot(x,y_high, color='red')plt.title('Variation in std at Confidence Intervals')# plot the distributions with mean at the confidence intervalsdist_low = st.norm(loc=means_qtl_025, scale=bs_std_mean)y_low = dist_low.pdf(x)dist_high = st.norm(loc=means_qtl_975, scale=bs_std_mean)y_high = dist_high.pdf(x)#plotplt.sca(ax2)sns.distplot(population)plt.plot(x,y_low, color='red')plt.plot(x,y_high, color='red')plt.title('Variation in mean at Confidence Intervals')
The first plot shows that 95/100 times the population distribution would fall between the two red distributions, representing the standard deviation's upper and lower confidence intervals. The second plot shows that 95/100 times the population distribution would fall between the two red distributions, representing the mean's upper and lower confidence intervals. Now we have an idea of the ranges of possible population parameters that are likely given our sample.
You probably have one more question itching at you: Would we get more precise results if we resampled more than 1000 times?
Let's find out by resampling 10 million times instead of 1,000 times
bs_big = np.random.choice(sample, (len(sample), 10**7), replace=True)
Calculate the mean.
bs_means = bs_big.mean(axis=0)bs_means_mean = bs_means.mean()bs_means_mean
54.12096064254814
The previous mean was 54.12110644012915. That's not much of a difference, and it's a little farther from the ground truth of 53.21.
means_qtl_025 = np.quantile(bs_means, 0.025)means_qtl_975 = np.quantile(bs_means, 0.975)means_qtl_025, means_qtl_975
(52.535753878990164, 55.76684250100114)
Previous values were (52.41754898532445, 55.602661654526315)
bs_stds = bs_big.std(axis=0)bs_std_mean = bs_stds.mean()bs_std_mean
4.414164449356725
The previous value was 4.398626063763372
std_qtl_025 = np.quantile(bs_stds, 0.025)std_qtl_975 = np.quantile(bs_stds, 0.975)std_qtl_025, std_qtl_975
(3.3352503395296598, 5.387739215237817)
The previous values were (3.356752806683829, 5.460033411850985)
There doesn’t appear to be much difference. Let's plot it out again.
x = np.linspace(30, 80, 1000)# plot the distributions with std at the confidence intervalsdist_low = st.norm(loc=bs_means_mean, scale=std_qtl_025)y_low = dist_low.pdf(x)dist_high = st.norm(loc=bs_means_mean, scale=std_qtl_975)y_high = dist_high.pdf(x)#plotfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))plt.sca(ax1)sns.distplot(population, ax=ax1)plt.plot(x,y_low, color='red')plt.plot(x,y_high, color='red')plt.title('Variation in std at Confidence Intervals\n10 million re-samples')# plot the distributions with mean at the confidence intervalsdist_low = st.norm(loc=means_qtl_025, scale=bs_std_mean)y_low = dist_low.pdf(x)dist_high = st.norm(loc=means_qtl_975, scale=bs_std_mean)y_high = dist_high.pdf(x)#plotplt.sca(ax2)sns.distplot(population)plt.plot(x,y_low, color='red')plt.plot(x,y_high, color='red')plt.title('Variation in mean at Confidence Intervals\n10 million re-samples')
The previous plot resampling 1,000 times:
I’d say those results are basically indiscernible, and certainly, there was no advantage to resampling more than 1000 times in our case.
So there you go, the untold but well-known story of bootstrapping both mean and standard deviation to estimate population parameters.