Bootstrapping using Python and R. Estimating a sampling distribution... | by Michael Grogan | Towards Data Science

All too often, the attempt to determine the characteristics of a population is constrained by the fact that we must rely on a sample to determine characteristics of that population.
When analysing data, we would like to be able to estimate a sampling distribution in order to perform hypothesis tests and calculate confidence intervals.
One way of attempting to solve this issue is a method called bootstrapping, whereby results for a wider population are inferred from repeated sampling.
For instance, in order to determine if a different sample of the same population would have yielded similar results, the ideal would be to obtain new data samples. However, given that this may not be possible — an alternative is to randomly sample from the existing data.
For this example, we will look at the distribution of average daily rates (ADR) among a series of hotel customers. Specifically, a subset of ADR figures are used in conjunction with a bootstrapping technique to analyse what the distribution would be expected to look like given a larger sample size.
When generating samples using bootstrapping, this is done with replacement. This means that it is possible to select the same element from the sample more than once.
Using a sample of 300 ADR values for hotel customers as randomly sampled from the dataset provided by Antonio, Almeida, and Nunes, we are going to generate 5,000 bootstrap samples of size 300.
Specifically, numpy is used as below to generate 300 samples with replacement, and a for loop is used to generate 5,000 iterations of 300 samples at a time.
my_samples = []for _ in range(5000):    x = np.random.choice(sample, size=300, replace=True)    my_samples.append(x.mean())
Here is a histogram of the bootstrapped samples:
Here is a histogram of the original 300 samples that were randomly selected:
We can see that the original sample is significantly positively skewed. However, the histogram of the bootstrapped samples approximates a normal distribution — in line with the assumption of the central limit theorem that as the sample size increases, the underlying data distribution can be expected to approximate a normal one regardless of the shape of the original distribution.
A primary use of bootstrapping is to estimate the confidence interval of the population mean. For instance, a 95% confidence interval means we are 95% confident that the mean lies within a particular range.
The confidence interval for the bootstrapped sample is as follows:
>>> import statsmodels.stats.api as sms>>> sms.DescrStatsW(my_samples).tconfint_mean()(94.28205060553655, 94.4777529411301)
According to the above, we can be 95% confident that the true population mean lies between 94.28 and 94.47.
Using the boot library in R, a similar form of analysis can be conducted.
Again, assume that only 300 samples are available:
adr<-sample(adr, 300)
Using random sampling with replacement, 5,000 replications of these 300 samples are generated.
x<-sample(adr, 300, replace = TRUE, prob = NULL)> mean_results <- boot(x, boot_mean, R = 5000)ORDINARY NONPARAMETRIC BOOTSTRAPCall:boot(data = x, statistic = boot_mean, R = 5000)Bootstrap Statistics :    original      bias    std. errort1* 93.00084 -0.02135855    2.030718
Additionally, here are the confidence interval calculations:
> boot.ci(mean_results)BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONSBased on 5000 bootstrap replicatesCALL : boot.ci(boot.out = mean_results)Intervals : Level      Normal              Basic         95%   (89.04, 97.00 )   (88.98, 96.82 )Level     Percentile            BCa          95%   (89.18, 97.02 )   (89.36, 97.32 )  Calculations and Intervals on Original Scale
A histogram of the bootstrapped samples are generated:
When running in R, we can see that the confidence intervals are wider than those obtained when running using Python. While the reason for this is unclear, the analysis still indicates that the sample mean obtained is representative of the population as a whole, and is unlikely to have been obtained by chance.
While bootstrapping can be quite a useful tool for inferring population characteristics from a sample, a common error is to assume that bootstrapping is a cure for small sample sizes.
As explained on Cross Validated, a small sample size is likely to be more volatile and prone to significant deviations from that of the broader population. Therefore, one should ensure that the sample being analysed is sufficiently large in order to capture population characteristics adequately.
Moreover, another issue with small sample sizes is that the bootstrap distribution itself may end up being narrower — which in turn would result in a narrower confidence interval that could deviate significantly from the real value.
In this article, you have seen:
How bootstrapping can allow us to estimate a sampling distribution using repeated sampling
Importance of replacement when random sampling
How to generate bootstrap samples using Python and R
Limitations of bootstrapping and why larger sample sizes are preferable
Many thanks for your time, and any questions or feedback are greatly appreciated. You can find more of my data science content at michael-grogan.com.
Antonio, Almeida, Nunes (2019). Hotel Booking Demand Dataset
Cross Validated: Can bootstrap be seen as a “cure” for the small sample size?
docs.scipy.org: scipy.stats.bootstrap
Julien Beaulieu: Sampling Distributions
Statistics By Jim: Introduction to Bootstrapping in Statistics with an Example
University of Toronto Coders - Resampling Techniques in R: Bootstrapping and Permutation Testing
Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as professional advice. The findings and interpretations in this article are those of the author and are not endorsed by or affiliated with any third-party mentioned in this article. The author has no relationship with any third parties mentioned in this article.