A Complete Guide to Hypothesis Testing | by Christina | Towards Data Science

Hypothesis testing is a method of statistical inference that considers the null hypothesis H0 vs. the alternative hypothesis Ha, where we are typically looking to assess evidence against H0. Such a test is used to compare data sets against one another, or compare a data set against some external standard. The former being a two sample test (independent or matched pairs), and the latter being a one sample test. For example, “does group A have a higher pain tolerance than group B?” or “is the mean age of the control group 21?”, respectively. A hypothesis test ends with a decision based on a pre-specified level of significance α–either to reject the null hypothesis when we have strong enough evidence against it, or fail to reject the null.
Now the question is, how do we know if we have strong enough evidence against this null hypothesis? To answer this, we must first understand the different types of errors when it comes to testing.
Either errors is considered bad, and so when concluding a decision for the hypothesis test, we wish to minimize the probability of such errors. Typically, we denote α = P(Type I Error) and β = P(Type II Error).
To reiterate, Type I Error consists of incorrectly rejecting the null hypothesis when, in fact, the null is true. If you are familiar with hypothesis testing, you have probably seen that statisticians typically set the level of significance at α = 0.05, and reject the null hypothesis when the p-value is less than or equal to α. That is, if the null is true, then we have a small probability of rejecting it (Type I Error). As a side note, the p-value corresponds to how likely we are to obtain similar results if H0 is true.
On the other hand, Type II Error consists of not rejecting the null hypothesis when it is actually false. As stated previously, β = P(Type II Error). The probability of not making a Type II Error is called power, and power = 1 - β. Note that we want a low β, so high power is good. Power depends on the value of the true parameter θ and on sample size.
How do we estimate power? Consider the following example: suppose we test H0: μ = 9 vs. Ha: μ ≠ 9, where μ is the population mean, but actually, μ = 8. This means H0 is incorrect and we would want to reject it. In order to determine how likely that is to happen, we also need the population standard deviation, say we have σ = 3 from a previous study, and the sample size, say n = 20. We then estimate power by repeatedly drawing random samples from the true distribution and testing whether its mean is 9. If we ran the test on 1000 random samples and correctly rejected H0 say 357 times, the estimated power would be of 0.357.
Another use for power is that, often, researchers will use it to determine the sample size they need for their study. In other words, how big of a sample is needed to achieve a certain power?
A hypothesis test can be a one sample test, a two sample test, or a matched pairs test. In addition, the these tests can be one-sided or two-sided. The difference between these lie in what we wish to test.
A one sample test consists of comparing the data to an external value. This is not comparing two groups, but one group to said external standard.
A two independent samples test compares one group to another group in the data, where the two groups are independent. That is, the observations recorded for these groups are independent, and then we are comparing then with each other.
We have matched pairs, or two dependent samples, when we have two measurements per subject. Typically, we see this when subjects are re-tested after a certain period of time.
A one-sided test occurs when we wish to determine in which direction the difference between the compared groups, or compared to the external standard, is. In this case, the objective of the test consists only of examining one end, and ignores what happens on the other end. An example would be to test wether group A performed better than group B on the exam. Here, we notice that we are only concerned with when the difference between A and B is positive.
A two-sided test consists of testing whether there exists a difference between the compared groups, or compared to the external standard. In this case, the difference can be either positive or negative.
The main assumption of the t-test is that the data is normally distributed. To be more precise, we need the sampling distribution of the sample mean to be approximately normal... but lets ignore this for now as this requires some knowledge on bootstrap distributions, and simply consider approximate normality of the data. Typically, for a large sample size, we have approximate normality by central limit theorem. However, if our sample is not that big, it would be a good idea to assess normality before proceeding with a t-test.
Some simple ways to check for normally distributed data is by examining a boxplot or a histogram representation of the data. A quick way to get such plots in Python is through the Seaborn library. For example, here’s a quick way to get these plots, where x represents the sample of the data we want to evaluate.
import numpy as np import seaborn as snsfrom matplotlib import pyplot as plt# generate normal datasns.set()np.random.seed(100)x = np.random.normal(size=100)# put plots side by sidefig, ax = plt.subplots(1, 2, figsize=(15, 6))# histogramsns.distplot(x, kde=False, ax=ax[0]).set_title("Histogram")# boxplotsns.boxplot(x=x, orient="v", ax=ax[1]).set_title("Boxplot")
Assessing normality from each of the above two plots is quite simple. For a histogram, we want to see symmetry about the mean and fairly even tails. For a boxplot, we want fairly even whisker lengths (which are associated to the tails of a normal distribution). In addition, in both cases, we do not want to see any outliers, nor skewed data. If the data is right skewed, then we will notice that most of bars on the histogram will be condensed to the left, and that there will be a longer upper whisker on the boxplot. In our example, considering that the data used to generate the above plots was directly sampled from a normal distribution, it makes sense that the generated plots meet normality.
Once we have concluded that our data is approximately normal, we need to decide which two sample t-test to use as there are further assumptions to check. This decision is based on whether the compared two groups have equal spreads/variances or not. To evaluate this, we will look at boxplots of both groups. If the spreads are the same, the interquartile range (IQR) for both boxplots will be approximately the same.
The most common hypothesis test is a t-test. However, a t-test (similarly to other tests) comes with assumptions, and if these assumption are not met, the results will not make sense. We must chose the appropriate test to use based on the distribution of the data, in addition to the number of groups being compared in the data. Most of the tests mentioned bellow are accessible through Python’s Scipy library.
If we conclude that the data has approximate normality, we can then use a t-test. There is the one-sample t-test and two types of two sample t-tests: pooled t-test (for equal variance) and Welch-Satterthwaite t-test (for unequal variance). Another case of a t-test is when we have two samples that are related to each other i.e. the matched pairs situation, where we will use a paired t-test.
For a one sample t-test, the null hypothesis H0 is “the mean is equal to θ0” vs. the alternative hypothesis Ha that “the mean is not equal to/less than/greater θ0”.
from scipy import stats# one sample t-test (two-sided)stats.ttest_1samp(group1, null_mean)
For a two sample t-test (where the samples are independent), H0 is “the mean of group A is equal to the mean of group B” vs. the alternative hypothesis Ha that “the mean of group A is different/greater/less than the mean of group B”. In the two sample case, each of the groups needs to meet approximate normality. Moreover, we need to assess the variances of both groups to decide which two sample t-test to use. The default is for unequal variance.
from scipy import stats# two sample t-test (two-sided)stats.ttest_ind(group1, group2, equal_var=True) #for Pooled
For a matched pairs t-test, H0 is “the mean of both groups are equal” or “ the true difference between the means of both groups is 0”. Here, the difference between both groups must be approximately normal.
from scipy import stats# matched pairs t-teststats.ttest_rel(group1, group2)
When each independent group is approximately normal, and are comparing three or more groups of independent observations, we must use an analysis of variance (ANOVA). ANOVA has as H0 “all groups have the same mean” vs. Ha “at least one group has a different mean from the others”.
Again, as with t-test, we need to check for equal variance between the groups. If the spreads between the groups are equal, we can proceed with regular one-way ANOVA. If the spreads between the groups are not equal, we will need to use Welch ANOVA. This latter test is available through Python’s Pingouin library.
from scipy import stats# one-way ANOVAstats.f_oneway(group1, group2, group3, ...)
Notice that this test will not let us know where the difference is, and so we must do further testing to find which group differs from the rest, if any. In the case of equal variance, we will continue the analysis with TukeyHSD test. And in the case of unequal variance, Games-Howell is the way to go.
If we conclude that the data is not approximately normal, we cannot get any significant nor reliable results from using a t-test. We must then resort to a test that does not rely on the symmetry of the distribution. For a one sample test, we should resort to the sign test. Though when testing for two or more groups, we will use Mood’s median test. These test evaluate association between groups based on the median of the data instead of the mean. It is important to note that these median-based tests do not assume anything about the underlying distribution of the data.
The sign test has H0 “the median is equal to θ0“, while the null hypothesis for Mood’s median test is “the groups have the same (grand) median” vs. alternative hypothesis “one of the groups has median different than the grand median”.
from scipy import stats# Mood's median teststats.median_test(group1, group2, ..., ties)
Mood’s median test can be used for comparing two or more groups. So if we are comparing more than two groups, similarly to ANOVA, we need to find where the difference in medians occurs (if any). We do so with a pairwise median test.