Bootstrap and Statistical Inference in Python | by Leihua Ye, PhD | Towards Data Science

The complete Jupyter Notebook is available at my Github.
Bootstrap is a resampling strategy with replacement that requires no assumptions about the data distribution. It is a powerful tool that allows us to make inferences about the population statistics (e.g., mean, variance) when we only have a finite number of samples. Even when we only have one sample, the bootstrap method provides a good enough approximation to the true population statistics.
Amazing!
This is a follow-up post on the bootstrap method. I’ve introduced the fundamentals (e.g., the what, when, and why) and applications in another post. Please check it out for detailed walkthrough.
towardsdatascience.com
In today’s post, we shift gears and primarily focus on its utilities in statistical inference and Python implementation.
Bootstrap is a resampling process of the original sample with the same sample size and replacement for a lot of iterations.
This is a straightforward question, which fits the requirements of the bootstrap method (from a single sample to population statistics). We will use a simulated example for this question. Hypothetically, we have collected data on body weights of half a million male adults in America, and it ranges from 100 to 240.
# step 1: simulate population distribution
array([209, 226, 166, ..., 197, 175, 121]# weight_pop169.633012 # population mean40.39571209680374 # population std
First, we simulate a population distribution from 100 to 240, with a mean of 169 and a standard deviation of 40.
Realistically speaking, the population parameters (mean and standard deviation) should remain unknown. Here, we play God and reserve these population parameters for later-on comparisons with the bootstrapped results.
# step 2: take a random sample from the population
169.127 # sample_mean40.466218886869086 # sample std
Second, we take a random sample of the population and obtain the sample mean 169.127, and sample standard error, 40.466.
step 3: bootstrap for 10,000 times
array([169.023, 168.843, 168.031, ..., 169.98 , 167.204, 172.015])
Third, we repeat the resampling process for 10K times and construct a bootstrapped sampling data, which should follow a normal distribution (Central Limit Theorem).
# step 4: analysis and interpretation 
169.13262949999998 # bootstrapped sample means169.633012 # recall: true population mean1.2729552820228016 # bootstrapped stdarray([166.627975, 171.60705 ]) # 95% C.I.
Here are the breakdowns.
After 10K iterations, the bootstrapped sample mean is 169.13, which is very close to the true population mean, 169.6. The bootstrapped sample standard error is 1.27, and the 95% Confidence Interval is [166, 171].
In other words, 95% of the time, the true population mean lies between 166 and 171. Actually, this is true if you check the population parameters obtained in Step 1.
Distribution and C.I. in Graph:
As you can see here, we have successfully used the bootstrap method and “reverse-engineered” the population parameters from one single sample.
This is amazing!
Let’s see how it can solve more realistic questions.
This is a real Data Science interview question that I was asked by a FAANG company. It is a super smart question and I love it.
Here is why.
First, it is subtle. We probably do not think about using bootstrap in the first place, but the question asks us to perform an inference on a small dataset.
What is the method that allows us to make such inferences?
Bingo! Bootstrap!
Also, this question contains limited information, and we have to ask clarification questions, like if a confidence interval is needed, how many samples we should do before we move on to solutions. It tests good communication besides testing our statistics.
# step 0: create a population distribution of 100 random numbers from 0 to 200
At ground zero, we randomly generate a population distribution of 100 random numbers from 0 to 200.
# step 1: take a random sample of size 10
array([134, 139,   3, 153,  51, 146,  48, 139,   3, 168])
Then, we take a random sample of the population and obtain a small sample of 10 numbers.
# step 2: what’s the median of this sample (sample_1)?
115.0
n = len(some): returns the length of some.
some.sort(): for any arrays, we have to sort the values before we calculate the median value.
if n%2==0: if the length of some is even, then we calculate the median as such.
else: if the length is odd, then we calculate the median as such.
After calculations, sample_1 has a median value is 115.
Alternatively, we can use numpy.median(some) to obtain the median value, if the interviewers allow you to import Python libraries. Otherwise, we can still obtain the median value using the above self-defined function.
# step 3: repeatedly sample from the sample (sample_1) with replacement, aka. bootstrap
In step 3, we introduce bootstrap to repeatedly resample from the previous sample (sample_1) for 10,000 times and obtain a resampling distribution of the median values, called boot_sample_medians.
Let’s break down the for loop.
boot_sample_medians = []: create an empty list to store values
for i in range(10000): iterate over 10K times
boot_sample = np.random.choice(sample_1, replace =True, size = 10): collect a resample of 6 values from sample_1 with replacement.
boot_median: calculate the median value for each iteration
boot_sample_medians.append(boot_median): attach the median value to the list.
# step 4: stand error and confidence interval
36.24544558530768 #std106.5054 # the average value of repeated samples' median valuesarray([ 27., 146.]) #95% C.I.
As the results show, the repeated samples has a standard error of 36, an average value of 106.5, and its 95% confidence interval is [27, 146]. Recall, the original sample (sample_1) has a median of 115. So, the bootstrap methods offers a really close approximation to the original distribution.
The reason why the C.I, is so wide is because of the small sample size and considerably large variations among numbers. In each iteration, any different selections of numbers would result in a different median value, which explains the variation and the wide C.I.
If the interview question asks you to do any form of statistical inferences with a limited sample size (e.g., one sample), the bootstrap comes in handy.
To disentangle the Python programming part, the key is to understand the statistical procedures well and code it up with each step.
Ask for clarification questions before doing any programming, and ask for hints or help if got stuck. It’s highly encouraged to do so in interviews.
Good Luck!
Understand the questions well before proposing solutions.
Ask for clarification questions if key information is missing.
Practice these advanced topics before your next Data Science interview.
Medium recently evolved its Writer Partner Program, which supports ordinary writers like myself. If you are not a subscriber yet and sign up via the following link, I’ll receive a portion of the membership fees.
leihua-ye.medium.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Please find me on LinkedIn and Youtube.
Also, check my other posts on Artificial Intelligence and Machine Learning.