The Power of Permutation Testing. Getting creative when running... | by Kieran | Towards Data Science

Hypothesis testing has been around for decades, with well-established methods of determining whether the results that have been observed are significant. Yet sometimes it can be easy to lose track of which testing approach to use or whether it can be reliably applied to your situation.
Companies often use surveys to track their NPS (Net Promoter Score), a measure that’s designed to reflect customer loyalty and potential for growth.
The calculation is simple, ask customers how likely they are to recommend your business on a scale of 0–10, then subtract the percentage of negative responses from the percentage of positive.
What’s less simple, however, is knowing when two scores are significantly different from each other. Traditional hypothesis testing is usually answering the question of whether two means, proportions, or variances are meaningfully different from one another. NPS does not slot neatly into those tests because it’s actually measuring the difference in two correlated proportions.
What test statistic do you use for that?
Some kind of solution is needed. If we’re conducting an experiment where a random sample of customers receives a new type of service, it’s not good enough to simply eye-ball our survey results, we want to know whether it has had a genuine impact on their NPS.
Enter the permutation test, a technique that is so useful because it can be applied to any test statistic we define and makes no assumptions about the underlying distribution being normal.
Permutation testing starts off with the null hypothesis that there is no difference between the two groups we are measuring. If that assumption is true, we should be able to randomly shuffle all of the observed values between the two groups without influencing their overall statistics.
To check whether the null hypothesis can be rejected, the data is randomly shuffled, again and again, and the difference between the two randomly created groups is recorded each time. At the end of it all, the actual difference observed between the two groups can be compared with the thousands of simulations performed to see how surprising/unlikely it would be under the null hypothesis.
In using this approach, we can dispense with any complex formulae and instead follow an intuitive process that captures what a null hypothesis and p-value really mean.
Let’s see how this can be done in Python.
Firstly, assume we have a dataset that includes the NPS survey responses of customers in two different groups:
import pandas as pddf = pd.read_csv('nps_data.csv')df.head()| group   | response || ------- | -------- || control |        7 || test    |        3 || ...     |      ... || test    |        5 |
A handy technique I borrowed from Displayr shows that we don’t have to calculate the % of promoters and detractors each time in order to subtract one from the other. We can instead replace promoter scores with 100, detractor scores with -100, and neutrals with 0, so that taking the mean represents the overall NPS score.
df.loc[df['response'].between(0, 6), 'recoded'] = -100  # Detractorsdf.loc[df['response'].between(7, 8), 'recoded'] = 0  # Neutralsdf.loc[df['response'].between(9, 10), 'recoded'] = 100  # Promoters
Next, we need a process that can randomly shuffle all of the data and record the difference in NPS.
import numpy as npdef simulate_difference_in_nps(nps_data: pd.DataFrame) -> float:    """    Shuffle all of the responses and reassign them to each group,     before calculating the absolute difference in NPS between the     two groups in this simulated scenario.    Parameters    ----------    nps_data : pandas DataFrame        Dataset containing the NPS scores of survey respondents from         two different groups.    Returns    -------    float        The absolute difference in NPS scores between the two groups         once all of the responses had been shuffled and reassigned         to each group.    """    responses_in_single_group = nps_data.value_counts('group')[0]    randomly_shuffled = np.random.permutation(        nps_data['recoded'].values    )    nps_1 = randomly_shuffled[:responses_in_single_group].mean()    nps_2 = randomly_shuffled[responses_in_single_group:].mean()    return abs(nps_2 - nps_1)
The process will be repeated again and again to build a robust sample of simulations given that it would generally be impractical, even for a powerful computer, to explore every possible permutation of the results. This approach of taking a fixed number of samples rather than all possible permutations is known as the Monte Carlo method.
num_simulations = 100000simulated_results = []for _ in range(num_simulations):    simulated_results.append(        simulate_difference_in_nps(nps_data=df)    )    simulated_results = np.array(simulated_results)
The final step is to determine whether we can reject the null hypothesis after comparing what we observed to the thousands of simulations run with random permutations of the data.
# This would be calculated from the original data, but let's # hard-code a value for simplicityobserved_difference_in_nps = 1.6# Use 5% level of significancesignificance_level = 0.05# How often did we see a difference as extreme, or more extreme, # than what was observedsimulations_as_or_more_extreme = sum(    simulated_results >= observed_difference_in_nps)# The observed result is statistically significant if its p-value is # less than the pre-defined significance level # i.e. it was very unlikely to occur under the null hypothesisnum_simulations = simulated_results.shape[0]p_value = simulations_greater_than_observed / num_simulations# Boolean which is True if significant, False otherwisesignificant_or_not = p_value < significance_level
As a nice bonus, with all of those random simulations performed, we can even visualise how unusual our observed result would be under the null hypothesis.
import matplotlib.pyplot as pltimport seaborn as sns# Plot permutation simulationsdensity_plot = sns.kdeplot(simulated_results, shade=True)density_plot.set(    xlabel='Absolute Difference in NPS Scores Between Groups',    ylabel='Proportion of Simulations',    title='How Surprising Is Our Observed Result?')# Add a line to show the actual difference observed in the datadensity_plot.axvline(    x=observed_difference_in_nps,     color='red',     linestyle='--')plt.legend(    labels=['Observed Difference', 'Simulated'],     loc='upper right')plt.show()
And that’s that!
We now have a template for performing a two-tailed test on NPS scores. It wouldn’t take much effort to refactor the code for one-tailed tests; simply record the difference for each permutation rather than the absolute difference.
The implementation above helps explain the steps needed, but it’s a touch clunky and inefficient.
Check out my GitHub repo to see how this process can be placed within a more efficient, unit-tested, helper class that can be parallelised across multiple cores when repeating the permutation process.
The same repo also contains the demonstration below on how to use the code.
This article focuses on NPS for illustration, but permutation testing can be applied to any domain, particularly ones in which we’re unsure what the distribution of the variable we are measuring looks like.
As with any statistical approach, however, there are a few things to consider before using permutation testing.
We still need to start with an appropriate sample size. If we only have 3 respondents in each group, any form of hypothesis testing is probably a pointless exercise.The null hypothesis states that the values for each group are exchangeable, so we are technically testing whether the values from Group 1 come from the same distribution as Group 2. Therefore permutation testing still has an assumption that the two groups have the same underlying variance, which may not always be valid1.Knowing how many permutations to perform can be tricky. In simple terms, it’s best to err on the side of caution and choose as many as our machine can handle within the amount of time we’re willing to wait. There is plenty of further reading on picking the number in advance2 3.
We still need to start with an appropriate sample size. If we only have 3 respondents in each group, any form of hypothesis testing is probably a pointless exercise.
The null hypothesis states that the values for each group are exchangeable, so we are technically testing whether the values from Group 1 come from the same distribution as Group 2. Therefore permutation testing still has an assumption that the two groups have the same underlying variance, which may not always be valid1.
Knowing how many permutations to perform can be tricky. In simple terms, it’s best to err on the side of caution and choose as many as our machine can handle within the amount of time we’re willing to wait. There is plenty of further reading on picking the number in advance2 3.
Keep those considerations in mind and you’ll safely add a whole new testing approach to your data science toolkit.
Feel free to check out my other articles:
A simple trick to avoid running out of memory
How to Find the Right Number of Clusters
How to Manage a Junior Data Scientist
Making Your Loss Function Count
[1] Rand R. Wilcox, 8 — COMPARING TWO INDEPENDENT GROUPS, Editor(s): Rand R. Wilcox, Applying Contemporary Statistical Techniques, Academic Press, 2003, Pages 237–284, ISBN 9780127515410, https://doi.org/10.1016/B978-012751541-0/50029-8.(https://www.sciencedirect.com/science/article/pii/B9780127515410500298)