Hypothesis Testing using Welch’s t-test
By: Fernando Aguilar
Tags: Flatiron School, Hypothesis Testing, T Test, Business, Data Science

Introduction

For my module 2 final project, I used Northwind’s SQL Database to find out ways to improve its profitability. I performed hypothesis testing to uncover the effect of certain variables that would help Northwind either decrease costs or increase revenue. Throughout this blog post, I will look at the effect of discounts over product quantities ordered. This is relevant for Northwind since discounts are a marketing and sales tool designed to increase revenues, and it is critical to know if this strategy is resulting in higher sales.

Hypothesis Testing

Hypothesis testing is the most important statistical tool to test conventional wisdom. It must include two mutually exclusive hypothesis, the null and the alternate hypothesis. The null hypothesis is usually the one we want to reject in favor of our alternate hypothesis. In this specific project, for example, the null hypothesis is that discounts do not have an effect in sales, meaning average quantities ordered are the same for orders with and without discounts. On the other hand, the alternate hypothesis is that discounts do have an effect in quantities ordered, hence the mean quantities ordered amongst orders with and without discounts are not the same.

Analysis Plan

Given that I am trying to find the differences between the means of the quantities ordered amongst orders with and without discounts. I will use the two sample t-test to carry out the hypothesis testing. For this particular scenario, the null and alternate hypothesis are stated as:

H₀: μ₀ = μ1. Discounts do not have an effect on the average quantities ordered.

H₁: μ₀ ≠ μ₁. Discounts have an effect on the average quantities ordered.

To test this hypothesis I will choose a confidence level of 95% (significance level of 5%). This means that I am willing to accept a 5% probability of making a Type I Error. A Type I Error happens when the null hypothesis is rejected when in fact the null hypothesis is true.

Significance Level (alpha): α = 0.05. Probability of a Type I Error.

Northwind’s Data

Northwind’s order related data is stored in its SQL database. In order to extract the requiered data, first, let’s import the SQLAlchemy’s library to create an engine to be able to query the SQL database. All the information I need for this test is in the ‘OrderDetail’ table. I will also need libraries ti visualize, manipulate data and conduct Exploratory Data Analysis. I will query the database and store orders with and without discount each in its own pandas dataframe.

Import required libraries to query the SQL database and perform EDA.

Initial data visualization

The key takeaways from this visualization is that we can notice that neither of our samples are normally distributed, they do however have similar distributions. Another one is that by offering discounts the mean quantity ordered has jumped by almost 25%.

Two-samples t-test (difference in means)

One of the assumptions of the two sample t-test is that both samples are normally distributed. I can however, thanks to the central limit theorem, bypass this assumption given that both sample sizes are relatively large, with the smaller sample size being 838. The central limit theorem states that given a sufficiently large sample size from a population (usually more than 30) with a finite level of variance, the mean of all samples from the same population will be approximately equal to the mean of the population.

Another assumption is that the variances. of the two groups you are measuring are equal in the population. To test this assumption i will use the Levene test, which its null hypothesis is that all input samples are from populations with equal variances.

stats.levene(df_orderd.Quantity,df_ordernd.Quantity)

With a significance level of 5% (alpha), I reject the null hypothesis and accept that the samples are not from populations from equal variances. Fortunately, there is a variation of the two sample t-test when samples do not conform to the equal variance assumption. It’s called Welch’s t-test, and it is more reliable when the two samples have unequal variances and/or unequal sample sizes. Since the samples have unequal variances and unequal sample sizes, this is the test I will use to perform hypothesis testing.

Function to calculate the t-statistic and p-value

welch1 = welch_t(df_orderd.Quantity,df_ordernd.Quantity)

--------------------------------------------------------

Welch's t-test = 6.239069142123973

p-value: 5.65641429030433e-10

DoF: 1561.7678231942134

Decision: Reject the null hypothesis

I successfully reject the null hypothesis in favor of the alternate hypothesis. Hence, I accept that orders with discounts have a higher average order quantity than orders without discounts. Now, I have to find out the effect size of the discounts. This is important because the larger the effect size, the less likely it is to be random error.

Effect Size: Cohen’s d

Effect size is the magnitude of the difference between groups. Hence, it is imperative to find out if discounts provide a large enough effect to justify their implementation.

Function to obtain Cohen’s d

d = cohen_d(df_orderd.Quantity,df_ordernd.Quantity)

---------------------------------------------------

Cohens'd: 0.28641058366376687

Effect Size: Small

Now that we now that the effect size of discounts is small, it might be worth exploring the individual effect size of each of the discounts offered by Northwind since larger effect sizes are preferred.

Effect size by discount

Now, I will split the data according to the discount offered and conduct the same hypothesis testing methodology as above, but now for each discount level.

Table with effect sizes per discount level

From the results, I find it surprising to see that the effect size is not perfectly correlated to discount levels. I would have expected incremental effect sizes as discount levels increase. The least effect size was shown at the 10% discount level, a 5% or even offering a 15% discount gives a much larger effect size so they can be thought of as much better alternatives to boost sales.

Conclusion

Hypothesis testing is a strong and very important statistical tool that enable us to test conventional knowledge using sample data. Most parametric testing assume normality, but thanks to the central limit theorem, it is possible to bypass this assumption with large enough samples. Moreover, a variation of the two sample t-test called Welch’s t-test enables us to compare two sample means with unequal variances.

Resources and Further Reading