Understanding Hypothesis Testing. A simple yet detailed dive into all the... | by Shubhangi Hora | Towards Data Science

Hypothesis testing is an important mathematical concept that’s used in the field of data science. While it’s really easy to call a random method from a python library that’ll carry out the test for you, it’s both necessary and interesting to know what is actually happening behind the scenes! What is hypothesis testing, why do we need it, what does it do and how does it really work? Let’s find out!
Hypothesis testing is a statistical method to determine whether a hypothesis that you have holds true or not. The hypothesis can be with respect to two variables within a dataset, an association between two groups or a situation.
The method evaluates two mutually exclusive statements (two events that cannot occur simultaneously) to determine which statement is best supported by the sample data and make an informed decision.
The building blocks of hypothesis testing are:
Hypotheses — null and alternateStatistical testsProbability distributionsTest statisticsCritical valuesLevel of significance (alpha)P-value
Hypotheses — null and alternate
Statistical tests
Probability distributions
Test statistics
Critical values
Level of significance (alpha)
P-value
The process of hypothesis testing involves two hypotheses — a null hypothesis and an alternate hypothesis.
The null hypothesis is a statement that assumes there is no relationship between two variables, no association between two groups or no change in the current situation — hence ‘null’. It is denoted by H0.
The alternate hypothesis is the opposite of the null hypothesis because it assumes that there is some relationship between two variables or there is some change in the current situation — hence ‘alternate’. It is denoted by Ha or H1.
The conclusion of hypothesis testing is whether the null hypothesis is not rejected (so you accept that there is no change) or is rejected (so you reject that there is no change).
Let’s understand this with an example.
There’s a dataset that contains information on patients who’ve been admitted to a hospital in the past year. Two features present in this dataset are location and total amount spent. We believe there’s a difference in the total amount spent during admission based on where a person lives — this statement is your alternate hypothesis, since it assumes a relationship between location and the amount spent in the hospital.
Hence, the null hypothesis will be that there is no difference in the amount spent at the hospital based on where a person lives.
Now we need to figure out whether we should or should not reject the null hypothesis. How do we do that?
Statistical tests help us in rejecting or not rejecting our null hypothesis by providing us with a test statistic (a single numeric value) from our sample data. This test statistic measures the extent to which the sample data agrees with the null hypothesis — if it agrees then the null hypothesis is not rejected, else it is rejected.
There are multiple statistical tests that exist for different use cases. The selection of which statistical test to use your hypothesis testing depends on several factors, such as — the distribution of the sample (whether it is normally distributed (follows the normal distribution)), what the sample size is, whether the variance is known, the type of data that you have, amongst some other things. Each test has its own test statistic based on the probability model/distribution that the test follows.
For example —
Statistical tests are of two types — one-tailed and two-tailed.
The concept of tails is related to probability distributions.
A probability distribution is a statistical function whose output is the probabilities of the occurrences of different events (possible outcomes) of an experiment.
For example:
A probability distribution is both in the form of data and a graph, but the graph is a better way to understand what tails are.
The tails of a distribution are the endings of the curve / graph on the sides of the distribution. The right end is known as the right tail or the upper tail, and the left end as the left tail or the lower tail.
It is not necessary for a distribution to have both tails in all situations, it could have just one. The number of tails is dependent on your null and alternate hypotheses.
Thus we have one-tailed distributions and two-tailed distributions.
So, a one-tailed statistical test is one whose distribution has only one tail — either the left (left-tailed test) or the right (right-tailed test). A two-tailed statistical test is one whose distribution has two tails — both left and right.
The purpose of a tail in statistical tests is to see whether the test statistic obtained falls within the tail or outside it. The tail region is known as the region of rejection — which makes sense since the null hypothesis is rejected if the test statistic falls within this area. If it falls outside it then the null hypothesis is accepted.
Determining where the tail begins is done with the help of a parameter known as the level of significance — alpha. It is the probability of rejecting the null hypothesis when the null hypothesis is true. The most common value for alpha is 5%, i.e. 0.05. This means that you face a 5% risk of rejecting the null hypothesis, i.e. believing there is a difference in the current situation when there actually isn’t.
In terms of our distribution, alpha is the area of the distribution that makes up the region of rejection — hence it is the tail(s). Since the tails are at the end of the distribution, we are literally saying, with the level of significance, how far away from the null hypothesis our test statistic needs to be for us to actually reject the null hypothesis.
So, if we were to select a right-tailed test and our level of significance was 5%, then the 5% of the area of distribution on the right end of the curve would be our region of rejection. Using this, we calculate the value on the x-axis which demarcates where the region of rejection begins — this is known as the critical value.
In the case of a two-tailed test, we would divide the level of significance by half — 2.5% — and this would be the area of each region of rejection on either side of the curve. Then we would have two critical values and the regions of rejection would also be known as the critical regions.
One-tailed tests are directional as there is only one region of rejection. You are looking for a test statistic that falls within that sole region of rejection. Two-tailed tests are nondirectional since there are two regions of rejection and if the test statistic falls in either of them then the null hypothesis is rejected.
For example, the sample mean age of patients at a dentist’s clinic is 18 — this is your null hypothesis (since this is the current scenario and you are assuming there is no change). Another dentist joins the practice and you believe that now the sample mean of patients will be greater than 18 — this is your alternate hypothesis. It is directional due to the ‘greater than’ — only if the sample mean is greater than 18 will you reject the null hypothesis. If it is less than 18, your null hypothesis will not be rejected.
If it were a two-tailed test, your alternate hypothesis would simply be that the sample mean age of patients is not equal to 18. The sample mean could be greater than or less than 18, either way the null hypothesis would be rejected, hence it is nondirectional.
So far, we have hypotheses, a statistical test based on a probability distribution, a level of significance from which we get the region of rejection from which we get the critical value, and a test statistic that falls somewhere within our distribution.
As mentioned before, the test statistic is compared with the critical value. The p-value is compared with the level of significance.
P-value is the probability that the test statistic you have obtained is by random chance which would mean the null hypothesis is actually true. If the p-value is lower than the level of significance then it is statistically significant and the null hypothesis is rejected, because it means that there is a less than 5% probability that the null hypothesis is true. If it is higher than the level of significance then the null hypothesis is not rejected.
In terms of the graphical representation of our probability distribution, the p-value is the area of the curve on the right of the test statistic.
A big plus point of the p-value is that it doesn’t need to be recalculated for different levels of significance since it is directly comparable. In the case of test statistics, they have to be compared with critical values which do need to be recalculated for different levels of significance (if the area of the regions of rejection change then the x-axis value for where the regions begin will also change). That’s why it’s always better to work with the p-value in hypothesis testing.
Despite the above mentioned advantage though, the p-value is still probabilistic. Hence there is a chance that the null hypothesis will be rejected even though it is true. Alpha (the level of significance) is the likelihood that this will happen. This error is known as a type I error or a false positive.
The opposite to the above error would be not rejecting the null hypothesis even though it is not true. This is known as a type II error or a false negative.
Both of these errors are inversely related, i.e. if you try to reduce one the other will increase. So how do you get the perfect situation? You don’t, you just have to decide which error is less risky in your situation. For example, it’s less risky to tell someone they are covid-19 positive when they’re actually negative — this is a false positive (type I error). A false negative would be saying someone doesn’t exhibit fraudulent behavior when they actually do.
Finally we have all the building blocks of hypothesis testing sorted so the process is also pretty clear. The steps for hypothesis testing are:
Formulating your hypothesesChoosing a level of significanceIdentifying a statistical testCalculating the test statisticCalculating the p-value and comparing it with the level of significanceConcluding — rejecting or not rejecting your null hypothesis
Formulating your hypotheses
Choosing a level of significance
Identifying a statistical test
Calculating the test statistic
Calculating the p-value and comparing it with the level of significance
Concluding — rejecting or not rejecting your null hypothesis
Let’s go through these steps with an example that we used above about the mean age of patients that visit a dentist. We know that the mean age is 18 and the standard deviation is 8. Let’s assume the data follows a normal distribution. We think that the mean age isn’t 18, but is in fact larger than 18.
Since we have directionality in our alternate hypothesis, we need a right-tailed statistical test.
We are comparing the sample mean age, so we can use the z-test to determine whether our alternate hypothesis is probable or not.
Our level of significance is 5% and we are performing a right-tailed test so the right tail of our curve is the region of rejection.
Our sample data is 30 patients at random and we calculate the sample mean age.
So we have:
Now we need to calculate our test statistic, which in the case of the z-test is the z-score. The formula for z-score is
So our z-score is
The critical value of the normal distribution with alpha as 0.05 is 1.645. So we can already see that our test statistic is greater than our critical value, hence we can reject the null hypothesis. But as mentioned before, the p-value is the best way to determine whether or not we should reject the null hypothesis, so let’s calculate that too.
The probability of getting a value of 2.05 on the normal distribution is 0.9798. Since the p-value is the area to the right of the test statistic, it is calculated by subtracting 0.9798 from 1.
A p-value of 0.0202 is less than the 0.05 and hence we can reject our null hypothesis and conclude that it is more probable for the sample mean to be greater than 18.
Let’s actually implement all of this in python now!
We’re going to use a covid-19 datatset. It contains region wise confirmed cases and deaths from early 2020. It also contains temperature, humidity, latitude and longitude.
The first step is to import all the required packages:
import pandas as pdimport statsmodels.api as sm
We’ll be using pandas to read our data and stasmodels to actually carry out our hypothesis test.
Read the data into a dataframe:
covid = pd.read_csv(“Covid-19.csv”)covid = covid.drop([“Unnamed: 0”], axis=1)covid.head()
I believe that there is a relationship between temperature and the number of confirmed covid-19 cases — is the average temperature in regions with confirmed covid-19 cases greater than 12 degrees?
The first step of hypothesis testing is to formulate the hypotheses. Ours would be:
Since our alternate hypothesis is directional, we will be performing a one-tailed statistical test (a right-tailed test) — the z-test.
The level of significance will be 0.05, the test statistic is the z-score and the critical value is 1.645.
It’s really easy to calculate the test statistic and the p-value using the statsmodels ztest method. Let’s wrap this functionality in a function of our own and add the comparison between alpha and the p-value.
def hyptest(data, value, alternate, alpha):    z, p = sm.stats.ztest(data, value=value, alternative=alternate)    result = 'null hypothesis rejected!' if p < alpha else 'null hypothesis not rejected'    return z, p, result
Now we call our function — pass the temperature column (since that’s the variable present in our hypothesis), the number 12 as value since that is the metric we are testing for, and “larger” in the alternative parameter since our alternate hypothesis is that the average is larger than the value.
hyptest(covid.Temperature, 12, "larger", 0.05)(1.0033989581188794, 0.15783420330389508, 'null hypothesis not rejected')
The results are the z-score, the p-value and the conclusion respectively. Since the p-value is greater than 0.05, we do not reject our null hypothesis.
And that’s how you carry out a hypothesis test!
Some key things to remember are:
hypothesis testing is done on a sample of the whole population — hence sample data. (if you want to determine whether age impacts the likelihood of getting admitted to the hospital, the ‘population’ would be every single person on earth. A ‘sample’ is a number of people chosen at random from this population)
the output of hypothesis testing is probabilistic since it is performed on a sample and not a population. So you can never say with complete certainty that the null hypothesis is true or not. You simply reject it or don’t reject it.
the null hypothesis is always written with equality — =, ≤ or ≥
the alternate hypothesis is always written with ≠, < or >
the p-value is affected by the sample size — if the sample size increases then the p-value will decrease if the null hypothesis is to be rejected.
Please do let me know your thoughts about this article in the comments below or reach out to me on LindkedIn and we can connect!
The raw data has been taken from John Hopkins’ GitHub repository and then modified to get the temperature of the region. The code used to perform the test and to create the graphs can be found here.