Hypothesis Testing using R explained (Part 1)
By: Paresh Khandelwal
Tags: R, Data Science, Hypothesis Testing, Product Management, Data Visualization

Photo by Jon Tyson on Unsplash

Hypothesis testing is one of the most important and interesting topics of the Inferential statistics. Hypothesis testing is nothing but a statistical model designed to test the validity of a hypothesis or claim. For example, suppose you and your cousins decide to play a game in which you all roll a dice. Now each one of you has a fair and equal probability of 1/6 of getting a 6. You all roll the dice 5 times and every time it’s your youngest cousin who got 6 and you all start to think if he is cheating somehow. How do you statistically test that hypothesis is the process of hypothesis testing.

Exploring the terminologies

Null Hypothesis — The null hypothesis is the status quo or it’s an assumption that there is no change in the things. They are likely to be the same. This is denoted by H0.

The null hypothesis is the status quo or it’s an assumption that there is no change in the things. They are likely to be the same. This is denoted by H0. Alternate Hypothesis — The alternative hypothesis which is also denoted as H1 or HA sometimes, is the assumption there are significant pieces of evidence available in the data that lead to the conclusion that the null hypothesis can be rejected.

The alternative hypothesis which is also denoted as H1 or HA sometimes, is the assumption there are significant pieces of evidence available in the data that lead to the conclusion that the null hypothesis can be rejected. P-Value — P-Value stands for probabilistic value. In layman terms, it is the probability of getting results as extreme as the null hypothesis if the null hypothesis is true.

P-Value stands for probabilistic value. In layman terms, it is the probability of getting results as extreme as the null hypothesis if the null hypothesis is true. Parameter — Parameter is the quantity that summarizes the population data. Eg — Mean, Median, Standard Deviation, Variance, etc.

Parameter is the quantity that summarizes the population data. Eg — Mean, Median, Standard Deviation, Variance, etc. Statistic — Statistics refer to the summary of population sample data.

Statistics refer to the summary of population sample data. Criteria/ Significance Level — Significance level denoted as α is the threshold that decides whether the hypothesis has to be rejected or the pieces of evidence are weak to reject the hypothesis (in such cases we say, we fail to reject the hypothesis instead of saying accepting the hypothesis). Generally in behavioral sciences, the significance level is kept at 0.05 while other available criteria are 0.01, 0.05, 0.10. Now let us understand what does the significance level mean? Significance level (for example of) 0.05 or 5% says that there are 5% chances of getting results as same as the null hypothesis. If the p-value, in any case, is below 0.05 then the shreds of evidence are strong and we can reject the null hypothesis but if the p-value is more than 5% then it is considered to be statistically significant.

Significance level denoted as is the threshold that decides whether the hypothesis has to be rejected or the pieces of evidence are weak to reject the hypothesis (in such cases we say, we fail to reject the hypothesis instead of saying accepting the hypothesis). Generally in behavioral sciences, the significance level is kept at 0.05 while other available criteria are 0.01, 0.05, 0.10. Now let us understand what does the significance level mean? Significance level (for example of) 0.05 or 5% says that there are 5% chances of getting results as same as the null hypothesis. If the p-value, in any case, is below 0.05 then the shreds of evidence are strong and we can reject the null hypothesis but if the p-value is more than 5% then it is considered to be statistically significant. Methods — After setting up the criteria and collecting data, all you have to do is calculate the statistic using various methods available of hypothesis testing such as — T-Test. Z-Test, P-Value, Bayesian Tests, etc.

Calculating the P-Value using T-Statistics

Let’s look at how we can compute P-value and T-Score manually also using R. Also we will have a look at the interpretation of the results.

Formulae to calculate P-Value, and T-Statistic

P-Value = (x̅ — μ)/(σ√n)

T- Score = (x̅ — μ)/(s/√n)

Degree of Freedom = In case of a single-tailed test, df will be n-1, and in case of a two-tailed test, df will be n-2.

Where —

x̅ = Sample mean

μ = Population mean

σ = Population standard deviation

s = Sample standard deviation

n = Number of sample

Example of T-Test using R

Let’s take an example of an unpaired two-sample T-test. Suppose you’re a Product Manager and you are working on optimizing an online application journey. You made some changes in the journey and now you’re comparing time spent by users. The hypothesis is that customers will spend less time filling up the application on the new journey.

time_new<- c(38.9, 61.2, 73.3, 21.8, 63.4, 64.6, 48.4, 48.8, 48.5) time_old<- c(67.8, 60, 63.4, 76, 89.4, 73.3, 67.3, 61.3, 62.4)

Let’s put the data in a dataset called my_data

# Data in two numeric vectors

time_new <- c(38.9, 61.2, 73.3, 21.8, 63.4, 64.6, 48.4, 48.8, 48.5)

time_old <- c(67.8, 60, 63.4, 76, 89.4, 73.3, 67.3, 61.3, 62.4)

# Create a data frame

my_data <- data.frame(

group = rep(c(“new”, “old”), each = 9),

weight = c(time_new, time_old)

)

Plot a boxplot to check the data distribution

# Plot weight by group and color by group

boxplot(time_new, time_old,

main = “Boxploting showing data distribution”,

names = c(“time_new”, “time_old”),

col = c(“orange”, “red”),

border = “yellow”,

horizontal = TRUE

)

The boxplot will look like this and we can say the data is normally distributed.

Alternatively, to visualize the data distribution, we can plot a distribution curve too —

#Visual assessment of data distribution

d <- density(time_new)

plot(d, maxin=”Normality check — Visually”)

polygon(d, col=”red”, border=”blue”)

As a penultimate step, let’s assess the homogeneity of variance. The basic assumption of an independent t-test states that the variance should not be unequal. In case, they are, then chances of type — 1 error (incorrectly reject the null hypothesis) increase substantially. To check the homogeneity of variance, we will run an F-test. An F-test is used to compare the variance of two datasets.

#F-test to test homogenity between two datasets

res.ftest <- var.test(weight ~ group, data = my_data)

res.ftest

The output is given below states p-value = 0.1714 which is higher than the alpha 0.05 so we can conclude that there is no inequality in variance.

F test to compare two variances



data: weight by group

F = 2.7675, num df = 8, denom df = 8, p-value = 0.1714

alternative hypothesis: true ratio of variances is not equal to 1

95 percent confidence interval:

0.6242536 12.2689506

sample estimates:

ratio of variances

2.767478

Assumption checklist before running T-Test

The following three assumptions should be marked checked before running the t-test:

Are the two samples independent? Yes, they are.

Is the data in both the groups normally distributed? Yes, it is.

Is the variance in two populations the same? Yes, it is assumed to be an equal basis for the F-test results.

Alright, we have checked all the assumptions, let’s run the test and see whether the changes committed by you in the application journey has done some magic or not.

Running the T-test

# Compute t-test

res <- t.test(time_new, time_old, var.equal = TRUE)

res

The output will be:

Two Sample t-test



data: time_new and time_old

t = -2.7842, df = 16, p-value = 0.01327

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

-29.748019 -4.029759

sample estimates:

mean of x mean of y

52.10000 68.98889

Here in the results, t stands for t-test statistics, and df means the degree of freedom. In this result, we can clearly see the p-value = 0.01327 is below alpha 0.05 which means even if the null hypothesis is true, there is roughly a 1% chance of getting results as similar to that. We have already read it that if the p-value observed from the test is below alpha (in our case it was 0.05), the null hypothesis is rejected.

Errors in hypothesis testing

Majorly, there are two types of errors in hypothesis testing, namely, Type I and II errors.

Type I error

Incorrectly rejecting the null hypothesis even when it is true. This error is also known as False positive.

Type II error

Type II error is when the researcher incorrectly fails to reject the null hypothesis even when it is false. This error is also known as false negative.

Conclusion

So this was an introduction to the one technique of non-bayesian hypothesis testing. Apart from T-Statistics, Z-Score, and Bayesian analysis are other important and popular techniques oof hypothesis testing. We will cover Z Statistics and Bayesian analysis in upcoming blogs.