A Practical Guide to Bootstrap in R | by Leihua Ye, PhD | Towards Data Science

updated on August–11–2020
Bootstrap is a resampling method with replacement.
It allows us to estimate the distribution of the population even from a single sample.
In Machine Learning, bootstrap estimates the prediction performance while applying to unobserved data.
For the dataset and R code, please check my Github (link).
Bootstrap is a resampling method where large numbers of samples of the same size are repeatedly drawn, with replacement, from a single original sample.
Here is the English translation. Normally, it is not possible to infer the population parameter from a single, or a finite number of, sample.The uncertainty of the population originates from sampling variability: there is one value with a sample, while we obtain another value if we collect another sample.
The next question is:
How to eliminate the variability and approximate the population parameter as closely as possible?
By repeatedly sampling with replacement, bootstrap creates the resulting samples distribution a Gaussian distribution, which makes statistical inference (e.g., constructing a Confidence Interval) possible.
Bootstrap breaks down into the following steps:
decide how many bootstrap samples to performwhat is the sample size?for each bootstrap sample:
decide how many bootstrap samples to perform
what is the sample size?
for each bootstrap sample:
draw a sample with replacement with the chosen size
calculate the statistic of interest for that sample
4. calculate the mean of the calculated sample statistics
These procedures may seem a little bit daunting, but fortunately we don’t have to manually run the calculations by hand. Modern programming languages (e.g., R or Python) handle the dirty work for us.
Before answering the why, let’s look at some common inference challenges that we face as data scientists.
After A/B tests, to what extent can we trust a small sample size (say, 100) would represent the true population?
If sample repeatedly, will the estimate of interest vary? If they do vary, what does the distribution look like?
Is it possible to make valid inferences when the distribution of the population is too complicated or unknown?
Making valid statistical inference is a major part of data scientists’ daily routine. However, there are strict assumptions and prerequisites for valid inference, which may not hold or remain unknown.
Ideally, we would like to collect data on the entire population and save us some troubles of statistical inference. Obviously, this is an expensive and not-so-recommended approach, considering time and monetary expenses.
For example, it’s not feasible to survey the entire American population for their political view but a small portion of the entirety, say 10k Americans, and ask for their political preferences is doable.
The challenge with the approach is that we may get slightly different results each time we collect a sample. Theoretically, the standard deviation of a point estimate could be considerably large for repeated samplings from the population, which may bias the estimate.
Here is the punch line:
As a non-parametric estimation method, bootstrap comes in handy and quantifies the uncertainty of an estimate involved with the standard deviation.
For the following scenarios, bootstrap is a desirable approach:
When the distribution of a statistic is unknown or complicated.
When the distribution of a statistic is unknown or complicated.
Reason: bootstrap is a non-parametric approach and does not ask for specific distributions).
2. When the sample size is too small to draw a valid inference.
Reason: bootstrap is a resampling method with replacement and re-creates any number of resamples if needed).
3. You need a pilot study to feel the water before pouring all of your resources in the wrong direction.
Reason: related to #2 point, bootstrap can create the population’s distribution, and we can check the variance).
Since Sport is back and DraftKings is trending recently, I’ll use the NBA post-season game 6 between the Celtics and Heat (Github).
mydata <- read.csv("Celtics_Heat_Game_6_Actual.csv")mydata
We are interested in the correlation between two variables: the percentage that players have been drafted (X.Drafted) and the Fantasy Points (FPTS) scored.
To calculate the correlation for one game is simple but with less practical value. We have a general impression that top players (e.g. Jayson Tatum, Jimmy Butler) are going to the most drafted and also score the most points in a game.
What potentially could help pick players in the next game is:
If we draw repeated samples 10,000 times, what is the range of the correlation between these two variables?
In R, there are two steps for bootstrapping.
Install the package boot if you haven’t.
Step 0: Package preparation
install.packages('boot',dep=TRUE)library(boot)
Step 1: Define a function that calculates the metric of interest
function_1 <- function(data, i){ d2 <- data[i,]  return(cor(d2$X.Drafted, d2$FPTS))}
The above function has two arguments: data and i. The first argument, data, is the dataset to be used, and i is the vector index of which rows from the dataset will be picked to create a bootstrap sample.
Step 2: Apply the boot() function
set.seed(1)bootstrap_correlation <- boot(mydata,function_1,R=10000)
Remember to set seed to get reproducible results. The boot() function has three parameters: the dataset, which function, and the number of bootstrap samples.
bootstrap_correlation
The original correlation between the two variables is 0.892, with a standard error of 0.043.
summary(bootstrap_correlation)
The returned value is an object of a class called ‘boot,’ which contains the following parts:
t0: values of our statistic in original dataset
t: a matrix with sum(R) rows, each of which is a bootstrap replicate of the result of calling statistic
...
(Please refer to the R documentation for more information. )
class(bootstrap_correlation)[1] "boot"range(bootstrap_correlation$t)[1] 0.6839681 0.9929641mean(bootstrap_correlation$t)[1] 0.8955649sd(bootstrap_correlation$t)[1] 0.04318599
Some other common statistics of bootstrap samples: range, mean, and standard deviation, shown above.
boot.ci(boot.out=bootstrap_correlation,type=c(‘norm’,’basic’,’perc’,’bca’))
This is how we calculate 4 types of confidence intervals for bootstrapped samples.
In the above example, we are interested in the correlation between two variables: X.Drafted and FPTS. However, this is only one sample, and the finite sample size makes it difficult to generalize the finding at the population level. In other words, can we extend the discovery from one sample to all other cases?
In Game 6, the correlation coefficient between the percentage of players drafted and the fantasy points scored at DraftKings stands at 0.89. We bootstrap the sample 10000 times and find the following sample distribution:
Range of the correlation coefficient: [0.6839681, 0.9929641].Mean: 0.8955649Standard deviation: 0.0431859995% confidence interval: [0.8041,0.9734]
Range of the correlation coefficient: [0.6839681, 0.9929641].
Mean: 0.8955649
Standard deviation: 0.04318599
95% confidence interval: [0.8041,0.9734]
As we can see, the range of the coefficient is quite wide from 0.68 to 0.99, and the 95% CI is from 0.8 to 0.97. We may get a 0.8 correlation coefficient next time.
Both statistics suggest that these two variables are from moderately to strongly correlated, and we may get a less strongly correlated relationship between these two players (e.g., because of the regression to the mean for top players). Practically, we shall be especially careful while drafting the top-performing players.
Medium recently evolved its Writer Partner Program, which supports ordinary writers like myself. If you are not a subscriber yet and sign up via the following link, I’ll receive a portion of the membership fees.
leihua-ye.medium.com
For the dataset and R code, please check my Github (link).
towardsdatascience.com
Please find me on LinkedIn and Youtube.
Also, check my other posts on Artificial Intelligence and Machine Learning.