Linear Regression With Bootstrapping | by James Andrew Godwin | Towards Data Science

This article builds on my Linear Regression and Bootstrap Resampling pieces.
For the literary-minded among my readers, the subtitle is a quote from ‘Ulysses’ 1922, by James Joyce! The origin of the term “bootstrap” is in literature, though not from Joyce. The usage denotes: to better oneself by one’s own efforts — further evolving to encompass metaphors for a series of self-sustaining processes that proceed without external help, the context we are likely most familiar with.
For data scientists and machine learning engineers, this bootstrapping context is an important tool for sampling data. For this reason, it is one of the most important to consider what underlies the variation of numbers, the variation of distributions. We use resampling when we have a small amount of data as it allows us to see how much variation there would have been.
The following content helps to explain two types of bootstrapping as applied to linear regression models!
We treat the data sample we have as the only representation of the population that we have. Then to get more datasets from it, we resample the data with replacement.
For example, say we polled a classroom of 20 students on how old the teacher is. Their 20 answers are simulated below:
import numpy as npfrom scipy import statsimport matplotlib.pyplot as pltfrom sklearn.utils import resample%matplotlib inline# 20 student guesses of the teacher’s ageteacher_age = [24, 40, 27, 33, 31, 33, 35, 32, 29, 34, 39, 40, 41, 36, 34, 35, 29, 30, 35, 98]# expected age could be the averageteacher_avg = np.mean(teacher_age)print(‘Average of teacher age guesses: {} years old’.format(teacher_avg))teacher_std = np.std(teacher_age)print(‘Std_Dev of teacher age guesses: {0:.2f} years’.format(teacher_std))print(len(teacher_age))
Considering the guesses, only 5 of them are above the average. Visually inspecting the data we see that this is because of one outlier of 98 years. While we may consider dropping this outlier to get a better guess, we can also use bootstrap resampling to get more data that approaches the parent distribution. Here we will repeatedly sample with replacement to get a set of subsamples. Each subsample will also have 20 data points, the same as the original.
N.B: Remember that when sampling with replacement, it is possible — but unlikely, to get a subsample that has the outlier picked several times! We set the replace=True flag in np.random.choice .
n_sets = 100n_samples = len(teacher_age)def generate_samples(dataset, n): return list(np.random.choice(dataset, size=n, replace=True))boot_samples = [generate_samples(teacher_age, n_samples) for _ in range(n_sets)]print(‘Here are the top 3 samples generated:’)print(‘{}, ...’.format(boot_samples[0:3]))
Now we calculate the mean and standard deviation of each sample and then output the average of the averaged means and the average of the 100 subsampled standard deviations
sample_means = [np.mean(x) for x in boot_samples]sample_stdev = [np.std(x) for x in boot_samples]# take the average of all the meansset_mean = np.mean(sample_means)# average of all the std_devsset_stdev = np.mean(sample_stdev)print(‘Average of the sample averages: {0:.2f}’.format(set_mean))print(‘Average of the sample st. devs: {0:.2f}’.format(set_stdev))
Hmm — so far bootstrapping has barely changed anything. The sample averages of the sets are very similar (which we expect), but we also have not done much with the 100 subsamples we created.
Let us create a function that drops the lowest and highest N elements from the subsamples and apply this function to each sample and then investigate the averages and standard deviations of all the bootstrapped samples:
def trim_mean(dataset, n): # sort dataset in ascending order sorted_list = sorted(dataset) # trim the list (each list is of size: 20 - 2 = 18) trimmed_list = sorted_list[n:(len(sorted_list) — n)] return(trimmed_list)# apply function to all n (100) samplesboot_trims = [trim_mean(x, 1) for x in boot_samples]# each sample mean and st. dev.trimmed_means = [np.mean(x) for x in boot_trims]trimmed_stdevs = [np.std(x) for x in boot_trims]# average of all the meanstrimmed_mean_avg = np.mean(trimmed_means)# And the average of all the standard deviationstrimmed_stdev = np.mean(trimmed_stdevs)print(‘Average of the sample averages: {:.2f}’.format(trimmed_mean_avg))print(‘Average of the sample std_devs: {:.2f}’.format(trimmed_stdev))
import seaborn as snsbins = [x for x in range(25, 55)]ax = sns.distplot(sample_means, bins, label=’Original Bootstrap’)ax = sns.distplot(trimmed_means, bins, label=’Trimmed Bootstrap’)ax.legend(loc=’upper right’)plt.show()
The multi-modality (multiple peaks) are an artifact of the subsampling — they are heavily influenced by the random data subsampled, but overall the data appear to be normally distributed.
How do the distributions differ if we trim more data points? Let us find out:
# create different trimmed meansboot_trim1 = [trimmed_mean(x, 1) for x in boot_samples]boot_trim2 = [trimmed_mean(x, 2) for x in boot_samples]boot_trim3 = [trimmed_mean(x, 3) for x in boot_samples]boot_trim4 = [trimmed_mean(x, 4) for x in boot_samples]# sample each mean and std devtrimmed_mean1 = [np.mean(x) for x in boot_trim1]trimmed_stdev1 = [np.std(x) for x in boot_trim1]trimmed_mean2 = [np.mean(x) for x in boot_trim2]trimmed_stdev2 = [np.std(x) for x in boot_trim2]trimmed_mean3 = [np.mean(x) for x in boot_trim3]trimmed_stdev3 = [np.std(x) for x in boot_trim3]trimmed_mean4 = [np.mean(x) for x in boot_trim4]trimmed_stdev4 = [np.std(x) for x in boot_trim4]# plot the meansbins = [x for x in range(25, 55)]ax = sns.distplot(trimmed_mean1, bins, label=’Trimmed 1') #trim most #extreme min and max valuesax = sns.distplot(trimmed_mean2, bins, label=’Trimmed 2')ax = sns.distplot(trimmed_mean3, bins, label=’Trimmed 3')ax = sns.distplot(trimmed_mean4, bins, label=’Trimmed 4') # trim 4 #most extreme min and max valuesax.legend(loc=’upper right’)plt.show()
Notice how there is less and less multi-modality as more and more data is trimmed. Taken to the limit this trimming function would return only the value closest to the median. The students guesses are now closely distributed around what we would consider the “true mean,” which is the teachers age!
The bootstrap method can be applied to regression models. Bootstrapping a regression model gives insight into how variable the model parameters are. It is useful to know how much random variation there is in regression coefficients simply because of small changes in data values.
As with most statistics, it is possible to bootstrap almost any regression model. However, since bootstrap resampling uses a large number of subsamples, it can be computationally intensive.
N.B: For large-scale problems, it is necessary to use other resampling methods like k-fold cross-validation. Happy Data Science Land (nee Physics Land) is not the same grey world of floating-point inaccuracies and compromises need to be made.
Let us create our data and fit the linear regression:
import numpy as npimport matplotlib.pyplot as pltimport pandas as pdimport statsmodels.formula.api as sm# synthetic sample datan_points = 25x = np.linspace(0, 10, n_points)y = x + (np.random.rand(len(x)) * 5)data_df = pd.DataFrame({‘x’: x, ‘y’: y})ols_model = sm.ols(formula = ‘y ~ x’, data=data_df)results = ols_model.fit()# coefficientsprint(‘Intercept, x-Slope : {}’.format(results.params))y_pred = ols_model.fit().predict(data_df[‘x’])# plot resultsplt.scatter(x, y)plt.plot(x, y_pred, linewidth=2)plt.grid(True)plt.xlabel(‘x’)plt.ylabel(‘y’)plt.title(‘x vs y’)plt.show()
The first type of bootstrapping to perform is parametric resampling. Parametric because our model has a parameter, the mean, which we are trying to estimate.
The pandas sample function will help us with this process:
# resample with replacement each rowboot_slopes = []boot_interc = []n_boots = 100plt.figure()for _ in range(n_boots): # sample the rows, same size, with replacement sample_df = data_df.sample(n=n_points, replace=True) # fit a linear regression ols_model_temp = sm.ols(formula = ‘y ~ x’, data=sample_df) results_temp = ols_model_temp.fit()  # append coefficients boot_interc.append(results_temp.params[0]) boot_slopes.append(results_temp.params[1])  # plot a greyed out line y_pred_temp = ols_model_temp.fit().predict(sample_df[‘x’]) plt.plot(sample_df[‘x’], y_pred_temp, color=’grey’, alpha=0.2)# add data pointsplt.scatter(x, y)plt.plot(x, y_pred, linewidth=2)plt.grid(True)plt.xlabel(‘x’)plt.ylabel(‘y’)plt.title(‘x vs y’)plt.show()
The bands above are positioned more around the center of the data than at the edges because more data are concentrated at the center than at the edges.
Distribution of slope and intercept coefficients:
sns.distplot(boot_slopes)plt.show()
sns.distplot(boot_interc)plt.show()
The data above paints a pretty picture from parametric bootstrapping. However, if by chance, and this is more likely, we had sparse data, there may be a chance that our random selection of points are entirely in one area and not in another — recall the mention that the outlier can be sampled several times despite being a single outlying point.
To fight this, we can apply a different type of bootstrapping, called ‘non-parametric bootstrapping’ whereby we apply bootstrapping on the residuals, and not the parameter itself.
The idea is that we want to observe a point at every x-position that we have a point in the original data. Therefore, we fit the original data and re-shuffle the residuals at every point.
N.B: The assumption underlying our ability to do this is that the errors are independent and identically distributed (IID) among our data.
# create synthetic sample datan_points = 25x = np.linspace(0, 10, n_points)y = x + (np.random.rand(len(x)) * 5)data_df = pd.DataFrame({‘x’: x, ‘y’: y})# fit linear regression modelols_model = sm.ols(formula = ‘y ~ x’, data=data_df)results = ols_model.fit()# get predicted valuesy_pred = results.predict(data_df[‘x’])# get residual valuesresids = results.resid
Now that we fitted the values and residuals from our least-squares fit, we generate N (100) new sets of data by sampling the residuals and adding them to the fitted values:
boot2_slopes = []boot2_interc = []n_boots = 100plt.figure()for _ in range(n_boots): # create a sampling of the residuals with replacement boot_resids = np.random.choice(resids, n_points, replace=True) y_temp = [y_pred_i + resid_i for y_pred_i, resid_i in zip(y_pred, boot_resids)]  sample_df = pd.DataFrame({‘x’: list(x), ‘y’: y_temp}) # Fit a linear regression ols_model_temp = sm.ols(formula = ‘y ~ x’, data=sample_df) results_temp = ols_model_temp.fit()  # get coefficients boot2_interc.append(results_temp.params[0]) boot2_slopes.append(results_temp.params[1])  # plot a greyed out line y_pred_temp = ols_model_temp.fit().predict(sample_df[‘x’]) plt.plot(sample_df[‘x’], y_pred_temp, color=’grey’, alpha=0.2)# add data pointsplt.scatter(x, y)plt.plot(x, y_pred, linewidth=2)plt.grid(True)plt.xlabel(‘x’)plt.ylabel(‘y’)plt.title(‘x vs y’)plt.show()
Again we check the distributions of the parameters:
sns.distplot(boot2_slopes)plt.show()
sns.distplot(boot2_interc)plt.show()
This helps us understand ther error in our parameters with linear regression. In fact, this type of bootstrapping can be extended to most linear and non-linear models that have a unique, optimal solution. Using this methodology allows us to get an error distribution for every parameter!
Bootstrap is not the only resampling method, there are several, but IMO it is the among the best for production models because it makes minimal assumptions about the parent distribution and is well implemented into many languages and packages.
In this article we looked at applying bootstrapping techniques to linear regression in two ways:
Parametric bootstrapping — resampling from all of the points:
Sample the data with replacement numerous times (100)Fit a linear regression to each sampleStore the coefficients (intercept and slopes)Plot a histogram of the parameters
Sample the data with replacement numerous times (100)
Fit a linear regression to each sample
Store the coefficients (intercept and slopes)
Plot a histogram of the parameters
Non-parametric boostrapping — resampling on the residuals with an uneven distribution of feature values:
Find the optimal linear regression on all the original dataExtract the residuals from the fitCreate new y-values using the residual samplesFit the linear regression with the new y-valuesStore the slope and interceptsPlot a histogram of the parameters
Find the optimal linear regression on all the original data
Extract the residuals from the fit
Create new y-values using the residual samples
Fit the linear regression with the new y-values
Store the slope and intercepts
Plot a histogram of the parameters
Like Ulysses himself in his “Odessy” — I hope I have created a simple narrative that helps illustrate bootstrap resampling in the context of linear regressions. Next I will continue writing about regression but with the added wrinkle of talking about overfitting and using regularization help with that!
Find me on Linkedin
Physicist cum Data Scientist- Available for new opportunity | SaaS | Sports | Start-ups | Scale-ups