Hypothesis Testing. What it is and how to do it in Python | by Arthur Mello | Towards Data Science

A hypothesis is a claim or a premise that we want to test. Hypothesis testing is a way of backing up your conclusions with data, in a more “scientific” way. It is useful not only to scientists, but actually important in so many areas, ranging from marketing to web design to pharmaceutical trials and much more. Since it’s so useful and grounded on basic statistical concepts, learning when to apply the most typical tests will improve your data science skills, even if you are not a data scientist but still work with data in some way. In this article, we seek to explain the usefulness of the most common hypothesis tests and when to use them. It concludes with a Python code illustration of how the tests are executed in a data science project.
So, what is a statistical hypothesis? It is a hypothesis that can be tested using data. For instance, if you believe the “Add to cart” button in your e-commerce should be red instead of blue, because that would make people buy more, you could formulate the following hypothesis: “A red button will generate more sales than a blue button”.
Now, how can you test that? The most “obvious” approach would be to create two versions of the website, run both for a while, randomly assigning users to one of them, and then see which one generated more sales. If the “blue” version generates $1 million in a month while the “red” version generates $10 million, you got your answer, right? What if the “red” version generates $2 million? What if it generates $1million and 1 cent? When is the difference big enough to be significant? The problem with this approach is that it doesn’t take into account the natural variability we have when dealing with randomness. Hypothesis testing, on the other hand, not only considers randomness, but it also takes into account how confident we want to be in our conclusions.
Hypothesis testing therefore is a procedure through which we can evaluate the plausibility of our conclusion, using not only our sample data, but knowing how much risk we are willing to take, and the types of errors that we’re likely to encounter in the process.
The null hypothesis is the default claim, premise, value or parameter that is generally accepted. The alternative hypothesis is a research or investigative hypothesis that includes the claim to be tested which could hopefully replace the null hypothesis, in case it is proved to be true. The various tests below help to figure out to what extent information from the sample data can be extrapolated to the population from which it was drawn.
In our example above, the null hypothesis would be that there is no difference between the two colors, while the alternative hypothesis could be that there is a difference between them (or, more specifically, that the red button will generate more sales than a blue button).
This is a useful type of test used to compare one numeric feature (as in height or weight) between two groups in a dataset. It determines if there is a significant difference between the means of two groups, by yielding a p-value, which tells you how likely the difference observed in the sample between those two sample groups could have happened if there was no difference at the population level (your null hypothesis). If the p-value is less than a certain threshold (usually 0.05) then there’s a statistically significant difference between the two groups under investigation.
An important issue with the t-test is that it assumes a normal distribution and equal variances in both groups. Ok, but how can I know if my distributions are normal? Well, with a Shapiro-Wilk test, for instance.
Named after Samuel Sanford Shapiro and Martin Wilk, the Shapiro-Wilk test is generally used to test normality in frequentist statistics. You use it when you want to tell if a random sample comes from a normal distribution or not. The smaller the W value returned by the test, the more confident you can be that the sample is not normally distributed. This result can lead to the rejection of the null hypothesis you started with in the first place. You can also think about this in terms of a p-value: when it is less than or equal to a certain threshold (usually 0.05), then you can reject the null hypothesis.
Right, but what if my distributions are not normal? I just can’t compare them? Well, you can, but this time using the Mann-Whitney test.
This is also called the Mann-Whitney U test in statistics. This test becomes useful when you want to look at the difference between two independent groups, especially when the dependent variable is either ordinal or continuous, but not normally distributed.
Both types of tests we saw to compare groups were done by using 2 groups, but what if we have more than two groups?
ANOVA tests become handy when we want to compare multiple categories in a dataset. ANOVA (sometimes referred to as analysis of variance) is particularly useful when testing for differences in a numeric variable across more than 2 categories at the same time. ANOVA also relies on 3 main assumptions: samples are independent, come from a normally distributed population and have the same variance.
But what if both of your variables are categorical, and you want to check if they behave as if they were independent? Then try a Pearson’s chi-squared test on your data.
This is useful when dealing with categorical variables, to see how likely it would be to observe your data’s distribution if those variables were independent. For instance, let’s say you want to know if being a man or a woman makes you more or less likely to be a smoker. If those two variables were independent, you could expect the same percentage of men and women among both smokers and non-smokers. Pearson’s chi-squared test will tell you how confident you can be that there is such a difference, or not.
We’ll be working with a dataset containing information on Black Friday sales, and running the tests we just saw (the entire code is available here). The original dataset is available here and has one row per order, but we have changed it to contain one row per customer, to make some of our analysis more intuitive.
Our final dataset has, then, information on customer’s gender, marital status, occupation, city and their purchase amount.
First of all, let’s charge all the functions we need for the tests from the scipy library (one for each test):
[IN]:from scipy.stats import ttest_ind, shapiro, mannwhitneyu, f_oneway, chi2_contingency
Next, we’ll define our p-value threshold (0.05) and create a function to determine whether we should reject or not the null hypothesis:
[IN]:def test_result(p, ref_pvalue=0.05):    if p<ref_pvalue:        result=”Reject null hypothesis”    else:        result=”Don’t reject null hypothesis”    return result
Why do we always choose 0.05 as the threshold for p-value? This is mostly a convention and it might depend on the domain you work in (marketing, biostatistics, etc.) The smaller the threshold, the more confident you can be if you decide to reject the null hypothesis, but the less often you’ll reject it.
Now, let’s say we want to answer a very basic question: did men and women spend the same amount? At first, this looks like a job for the t-test, so let’s try it out:
[IN]:men = df[df['Gender']=='M']women = df[df['Gender']=='F']ttest_results = ttest_ind(men['Purchase'], women['Purchase'])print('T-test result: {}'.format(test_result(ttest_results[1])))[OUT]:T-test result: Reject null hypothesis
The null hypothesis here is that the mean purchase values for men and women are the same, so rejecting it implies that there might be differences between those values. However, it seems that we forgot to check for a very important assumption before running our test: are those samples normally distributed? In order to verify this, we’ll take a look at the distribution of our data and then run Shapiro-Wilk tests:
Looking at the distribution of purchase values for men and women we can tell that this is probably not normally distributed, given its high skewness (asymmetry). Still, let’s run our test to see if our intuition is right:
[IN]:shapiro_results_men = shapiro(men['Purchase'])print('Shapiro-wilk test results for men: {}'.format(    test_result(shapiro_results_men[1]))    )shapiro_results_women = shapiro(women['Purchase'])print('Shapiro-wilk test results for women: {}'.format(    test_result(shapiro_results_women[1]))    )[OUT]:Shapiro-wilk test results for men: Reject null hypothesisShapiro-wilk test results for women: Reject null hypothesis
Here, rejecting the null hypothesis means rejecting the hypothesis that the observations are normally distributed. So, indeed, our previous t-test result was not valid. We should, instead, try a Mann-Whitney test:
[IN]:mannwhitney_results = mannwhitneyu(men['Purchase'],women['Purchase'])print('Mann-Whitney test results: {}'.format(test_result(mannwhitney_results[1])))[OUT]:Mann-Whitney test results: Reject null hypothesis
Again, we reject the hypothesis that there is no difference between the sample means (but this time, using the appropriate test).
Now, let’s try to answer a different question: is there a difference in the purchase values across different city categories? This time, there are 3 different categories, so neither t-test or Mann-Whitney will do. Instead, we can use ANOVA, which, again, requires normal distributions. There is a smart trick we can use to make our data closer to a normal distribution: using a log-transformation. We’ll then use this trick here, before running our ANOVA test.
[IN]:a = df[df['City_Category']=='A']b = df[df['City_Category']=='B']c = df[df['City_Category']=='C']a['Purchase'] = a['Purchase'].apply(lambda x: np.log(x))b['Purchase'] = b['Purchase'].apply(lambda x: np.log(x))c['Purchase'] = c['Purchase'].apply(lambda x: np.log(x))anova_results = f_oneway(a['Purchase'], b['Purchase'], c['Purchase'])print('ANOVA test results: {}'.format(test_result(anova_results[1])))[OUT]:ANOVA test results: Reject null hypothesis
Again, we are rejecting the null hypothesis that there is no difference between the means of our city categories.
Now, to the last question, this time involving 2 categorical variables: are marital status and occupation related? This time, it’s not about comparing means, but about comparing frequencies. So, first we create a contingency table to calculate frequencies of all combinations of occupation and marital status and then we run our test:
[IN]:contingency_table = pd.crosstab(df['Marital_Status'],df['Occupation'])chisquare_results = chi2_contingency(contingency_table)print('Chi-squared contingency test results: {}'.format(test_result(chisquare_results[1])))[OUT]:Chi-squared contingency test results: Don't reject null hypothesis
This time we did not reject the null hypothesis, meaning there is no reason to believe there is some dependency between marital status and occupation: they behave more or less as if they were independent.
Please note that we did not verify all the assumptions needed for our tests in the code. The goal was to show how to perform them in Python and how to interpret their results. It would be impractical to find a dataset with variables that meet the criteria for all of the tests at once. In reality, though, you should always verify that those assumptions hold before running your tests.
There are many hypothesis tests in statistics and this list is far from comprising them all. These tests are, however, the most commonly used, and should cover most basic use cases. If you want to learn more about hypothesis testing or applied statistics, a great book on the subject is “Practical Statistics for Data Scientists”. Another good resource for delving deeper into statistics is LearnMore365 (their courses are usually not free, though). This article on Laerd details a bit more on the Mann-Whitney test.
Most tests make some sort of assumption about the data, so read carefully about what these assumptions are and how to verify them (you can usually find that in the Python function documentation).
If you like reading about applied statistics, you might like this article:
towardsdatascience.com
This article was written by Arthur Mello and Kingsley Essegbey.
Feel free to reach out to Arthur or Kingsley on LinkedIn if you would like to discuss further, it would be a pleasure (honestly).