Hypothesis Testing & p-Value. A Simplified Approach | by Suraj Gurav | Towards Data Science

Recently, I implemented a new methodology at my work. I did some experiments, analyzed results and with some statistical calculations I proved, the new methodology is better than the existing one...
Yess !! I did the Hypothesis Testing !!!
I believe in keeping the things simple, and this is the simplified approach to understand the Hypothesis Testing...
Hypothesis Testing...
Being human, we always have questions about almost everything. It can be about buying a house, a car or simply eating an ice-cream. We always think, whether chocolate ice-cream is better than vanilla or a sports car is better than a simple pickup van. Similarly, businesses and researchers also have questions such as new methodology is better or not, whether the new product will generate more revenue than the existing one and so on...
To be able to answer such questions, we need to translate them into something known as Hypothesis. Then we need to collect the data through experiments, surveys, and processes. After performing some statistical tests, in the end, we can answer the posed questions.
Hypothesis testing is a statistical method which is used to make decision about entire population, with the help of only sample data. To make this decision, we come up with a value called as p-value...
Hypothesis testing helps the businesses and researchers, to make better data-based decisions.
Let us go step by step...
In this whole article, let us consider an example, which makes things easy to understand.
There is a company ABC, who wants to know if the new design of the welcome web page results in more number of website subscriptions or not. So, let us consider the following notation.
N_new = Average number of users subscribing to the website after receiving a new design of welcome web page
N_old = Average number of users subscribing to the website after receiving an old design of welcome web page
Hypothesis tests are used to understand the population parameters such as mean and standard deviation.
Step 1: Translate the Question into the Hypothesis
Hypothesis is an argument, made as a basis for research...
The question to be answered is translated into 2 competing and non-overlapping hypothesis. We must make the following two types of hypotheses.
Null Hypothesis (H0)
This is the argument which we believe to be true even before we collect any data
In this argument, two groups that we want to test are considered to be equal or having no difference between them. for example, chocolate ice-cream is as tasty as vanilla or a new methodology gives poor or same results as existing methodology.
Hence, this argument usually contains mathematical operators such as =, ≤ or ≥.
In our example, H0 : N_new ≤ N_old
Alternative Hypothesis (H1)
This is the argument which we would like to prove to be true. for example, chocolate ice-cream tastes better than vanilla or new methodology generates better results than the existing one.
Hence, this argument often contains mathematical operators such as ≠< or >.
In our example, H1 : N_new > N_old
Step 2: Determine the Significance Level
The significance level is a probability of rejecting the null hypothesis when it is actually true. It is denoted by alpha ( α ). It is determined before conducting the experiment.
In many statistical analyses, we observe something written as — “ with a 95% confidence interval”. This statement, actually tells us about the significance level, such that,
Confidence Interval = 1 — Significance Level
so, here “95% confidence interval” tells us that Significance Level is 0.05. This means there is a 5% probability that there is a difference in the groups being compared when in reality there is no difference i.e. 5% is the probability of rejecting the null hypothesis.
In our example, there will be a 5% probability that N_new > N_old , when in reality N_new ≤ N_old
Hence, the significance level is used to determine, whether the Null Hypothesis should be rejected or retained. To reject the null hypothesis, the observed effect of the data has to be significant. And when the observed effect is significant, it simply means that we are confident that it is real and not obtained by chance.
Usually, the significance level of 1% is considered for the medical field and 5% for research and business.
Step 3: Calculate the p-Value
the p-value is a probability of observing the results of the Null Hypothesis. The significance level is the target value, which should be achieved if we want to retain the Null Hypothesis.
Hence, as long as the p-value is less than the significance level, we must reject the null hypothesis. Lower p-value means, the population or the entire data has strong evidence against the null hypothesis.
The p-value is calculated based on the sample data. It evaluates how well the sample data support the null hypothesis. Hence, a higher p-value, indicates that the sampled data is really supporting the null hypothesis. In other words, there is no enough evidence in sampled data to reject the null hypothesis.
However, researchers always look for p-value lower than the significance level. p — value ≤ Significance Level, means that the sampled data provide enough evidence to reject the null hypothesis. In other words, the alternative hypothesis, which we want to prove true, wins the battle. The observed effect in the data is statistically significant.
In inferential statistics, p-value helps to decide the significance of observed effect in relation to null hypothesis. It gives researcher the confidence to uphold or reject the null hypothesis.
Graphically, the p-value for a statistically significant observed effect lies in the shaded area in the probability distribution above. The regions of ‘very unlikely observations’ indicate the observed result which is not in favor of the null hypothesis. In other words, the p-value is the probability of obtaining observed statistic in favor of the alternative hypothesis H1, when the null hypothesis H0 is true.
More notes about the actual calculation of p-value can be found here.
Step 4: Make Decision
As we have all the pieces of the puzzle in hand, it is time to make a decision.
To determine which hypothesis to retain, the p-value is compared with the significance level.
If p - value ≤ significance level, we reject the null hypothesisIf p - value > significance level, we fail to reject the null hypothesis
Rejecting the null hypothesis means we accept the alternative hypothesis.
As we are observing the sampled data, we might make mistakes while making the decision to retain or reject the null hypothesis. These wrong decisions are translated into something called Errors. We will not go into details of these errors here, but this is an overview of two types of errors.
Type I Error :
This error occurs when the decision to reject the null hypothesis goes wrong. In this error, the alternative hypothesis H1 is chosen, when the null hypothesis H0 is true. This is also called False Positive.
Type I error is often denoted by alpha α i.e. significance level. alpha α is the threshold for the percentage of Type I errors we are willing to commit.
Type II Error :
This error occurs when the decision to retain the null hypothesis goes wrong. In this error, the null hypothesis H0 is chosen, when the alternative hypothesis H1 is true. This error is also called False Negative.
However, these errors are always present in the statistical tests and must be kept in mind while interpreting the results.
Thanks for reading !!
I hope, I presented the hypothesis testing in simpler terms. If you like my writing, giving some claps to this post will keep me motivated to write more. You are free to share this on Facebook, LinkedIn, and other social-professional networking platforms so that someone in need might get helped.
I am always open to your suggestion and questions. You can connect with me on LinkedIn, Twitter, and GitHub.