Resampling Methods for Inference Analysis
By: Idil Ismiguzel
Tags: Data Science, Statistics, Python, Education, Exploratory Data Analysis

1. Bootstrapping

Bootstrapping is a method to create samples with replacement from the original sample. Since it is done with replacement each data point has an equal probability of being picked subsequently. We can bootstrap n times (say 1000), calculate and record the desired estimate (i.e. mean) for each of these n samples. In the end, we can find the distribution for the desired estimate.

Image by author

📢General use cases of bootstrapping are finding robust estimates of confidence intervals, standard errors of population parameters such as mean, median, correlation coefficient, regression coefficient, etc.

To illustrate Bootstrapping in Python, I will be using Fish market dataset from Kaggle. The dataset consists of 7 common different fish species in fish market sales and I will be focusing on the “Roach” fish. I will also be selecting “Length1” as attribute.

data = pd.read_csv('./Fish.csv')

data = data.loc[data["Species"] == "Roach"]["Length1"] pd.DataFrame({'values': data.describe()}).reset_index()

As above we can calculate the sample mean with data.mean(), 95% confidence intervals with np.percentile(data, [2.5,97.5]).

However, by going from sample distribution to the population distribution, we are making several hidden assumptions such as: distribution of the “Length1” values are normally distributed and confidence intervals are symmetric. In most cases, these assumptions might not be correct.

mean = data.mean()

confidence_int = np.percentile(data, [2.5, 97.5])

Sample Mean= 20.65, 95% Confidence Interval = [14.61, 27.36]

Distribution of Length1 Values with the 95% Confidence Intervals

For this reason, we should bootstrap to find mean and confidence intervals.👇

mean_lengths, n = [], 1000 for i in range(n):

sample = np.random.choice(data["Length1"],

replace=True,

size=len(data))

sample_mean = sample.mean()

mean_lengths.append(sample_mean)





boot_mean = np.mean(mean_lengths)

boot_ci = np.percentile(mean_lengths, [2.5, 97.5])

Bootstrapped Mean= 20.60, 95% Confidence Interval = [19.22 22.07]

Distribution of Bootstrapped Sample Mean Values with the 95% Confidence Intervals

2. Jackknife Resampling

Jackknife resampling technique is based on creating samples by systematically leaving one observation out in the original dataset. We should calculate the desired value for each sample and then find the distribution of all values.

As an example, if we are given a sample with size n, then the Jackknife estimate is found by aggregating the estimated statistic of each (n-1) samples.

Image by author

📢General use cases of Jackknife resampling is finding variance and bias.

mean_lengths, n = [], len(data)

index = np.arange(n) for i in range(n):

jk_sample = data[index != i]

mean_lengths.append(jk_sample.mean()) mean_lengths_jk = np.mean(np.array(mean_lengths))

jk_variance = (n-1)*np.var(mean_lengths)

Jackknife estimate of the mean = 20.64

Jackknife estimate of the variance = 0.59

3. Permutation Test

Permutation test is a type of statistical significance test, which tries to obtain the distribution of the test statistic (under the null hypothesis) without making strong assumptions about data.

Imagine that we are given 2 samples and we would like to test whether if these 2 samples have different characteristics (i.e. mean). We could calculate the mean of the 2 samples and decide whether they are different or not. But analyzing in this way wouldn’t be statistically significant since we do not have the data of entire population, but only subsets from it.

Instead, we can perform permutation test, which works as follows:

Combine two samples into a single dataset. Shuffle the combined dataset and randomly resample it into 2 datasets (sized same as prior samples). Calculate the test statistics (i.e. the difference between means) and record the value. Repeat the steps above n times (say 10000 times). Compare the original test statistic with the recorded values, if the original test statistic lies well with the permuted records then test statistics are indeed not different and the difference between sample means is only due to chance. If the original statistic lies outside of the most permuted records than it means that there is a significant difference between the two samples.

To illustrate Permutation test in Python, I will be using again the Fish market dataset from Kaggle. This time I will be focusing on the “Roach” fish by selecting “Length1” & “Length2” attributes.

💬I will be investigating if the mean values of these two values differ significantly or due to chance.

First 5 rows from the dataset

data["Length1"].mean()

data["Length2"].mean()

Mean of Length1: 20.65

Mean of Length2: 22.27

We are investigating whether the difference between two sample means (-1.62) is by chance or statistically significant.

--> Step 1: Combine datasets

sample1 = data["Length1"]

sample2 = data["Length2"]

data = np.concatenate([sample1, sample2]) --> Step 2: Shuffle and randomly resample into 2 datasets

perm = np.array([np.random.permutation(len(sample1) + len(sample2)) for i in range(10000)])

permuted_1_db = data[perm[:, :len(sample1)]]

permuted_2_db = data[perm[:, len(sample1):]] --> Step 3: Calculate the test statistics

samples = np.mean(permuted_1_db, axis=1) - np.mean(permuted_2_db, axis=1) --> Step 4: Repeat it n times

Done in step 2 with for loop --> Step 5: Compare the original test stat with the recorded values

test_stat = np.mean(sample1) - np.mean(sample2)

p_val = 2*np.sum(samples >= np.abs(test_stat))/10000

print("p-value = {}".format(p_val))

test_statistic = -1.62

p-value = 0.15

The p-value tells us 15% of the time, we should expect to reach the similar mean difference between the 2 samples.

If you want to learn more about permutation tests and hypothesis testing, you can also read my article A guide to A/B testing (attached below)⚡