Statistical Tests in Python | Jul, 2021  | Towards Data Science

Imagine you are working on Exploratory Data Analysis or deciding on variables for feature engineering. However, you are not sure if there are significant differences among the groups in the variables to confirm particular hypotheses or warrant the variables to be used for modeling. Worse still, you may grapple with the appropriate tests to use. Sounds familiar?
Indeed, The main driver for this article stems from an effort to deepen understanding and note down for future reference, some of the common statistical methods and their application context. For each of the methods, the following are outlined
a brief introduction of the statistical test
worked example using dummy data
interpretation of the results
discussion on variations in use case (if any) & implementations.
Which type of statistical tests to use is influenced by the variables & their measurement levels. Variables can either be:
Numerical (measurements can be discrete or continuous; intervals or in ratio)
Categorical (measurement are discrete; either nominal or ordinal)
Note:
Nominal — no hierarchical sequence in the measurement levels.
Ordinal — having a specific sequence in measurement levels.
Thinking about the independent variables (predictors) and dependent variable (target), the statistical tests may be arranged as such:
# Import librariesimport pandas as pdimport numpy as npimport randomimport scipy.stats as statsimport statsmodels.api as smimport statsmodels.stats.contingency_tables as ctfrom statsmodels.graphics.gofplots import qqplotfrom statsmodels.stats.weightstats import ztestimport matplotlib.pyplot as plt plt.style.use('default')plt.style.use ('seaborn-darkgrid')
The Chi-square test of independence determines if there is a significant relationship between two categorical (i.e. nominal or ordinal) (with two or more mutually exclusive groups) variables. Hence, the null hypothesis is that no relationship exists between the two variables.
Non-parametric; does not require assumptions about population parameters
Compares difference in population proportions between groups
Contingency table of observed values is required.
It is a test to indicate if the variables are related (i.e. correlated). If they are correlated, we can use the phi coefficient to further determine the strength of the correlation.
The Chi-square test is sensitive to sample size (i.e. asymptotic)
As the sample size increases, the absolute differences become a smaller proportion of the expected value.
The outcome is that a strong association may not surface if sample size is small.
In large sample sizes, statistical significance may surface while the association is not substantial (i.e. very weak)
Given the data, is there any relationship between obesity and being diabetic?
# Create contingency table OD = pd.DataFrame({'notObese': [173, 259],                   'Obese' : [27, 61]},index ={'nonDiabetic','Diabetic'})OD
# chi-square testchiRes = stats.chi2_contingency(OD)# Detailsprint(f'chi-square statistic: {chiRes[0]}')print(f'p-value: {chiRes[1]}')print(f'degree of freedom: {chiRes[2]}')print('expected contingency table') print(chiRes[3])
Interpretation of results:
With a p-value of more than 0.05, we fail to reject the null hypothesis (H0: no relationship between the 2 variables). Obesity is not associated with being diabetic. The expected contingency table shows the expected values if the null hypothesis is True. The details may be examined from the difference between the values of the expected contingency table (should obesity be strongly associated with being diabetic) and the initial contingency table.
Given the data, is there any relationship between Gender and Eye color?
# Create the contigency tableGE = pd.DataFrame({'Blue' :[370, 359],                   'Brown':[352, 290],                   'Green':[198, 110],                   'Hazel':[187, 169]},index={'Female','Male'})GE
# chi-square testchiRes2 = stats.chi2_contingency(GE)# Detailsprint(f'chi-square statistic: {chiRes2[0]}')print(f'p-value: {chiRes2[1]}')print(f'degree of freedom: {chiRes2[2]}')print('expected contingency table') print(chiRes2[3])
# An alternative approach usng stats model api to derive expected table# statsmodel table objecttable = sm.stats.Table(GE)table.fittedvalues
# Can derive the direction of change between initial and expected table# resid_pearson method returns a table of residuals# positives - more observations than expected# negatives - less observations than expectedtable.resid_pearson
Interpretation of results:
With a p-value less than 0.05, we reject the null hypothesis (H0: no relationship between the 2 variables). Between the initial contingency table and expected contingency table, gender and eye color are dependent on each other.
Useful to determine the strength of a relationship between categorical variables. Linking the discussion back to Chi-square, assuming a low p-value; presence of dependence(statistically significant dependence between variables). The Cramer V’s helps indicate the strength of this dependence. Would be useful in situations where a statistically significant chi-square could be the result of large sample size instead of substantive relationship between the variables.
robust against large sample size
compares the strength of relationship between two categorical variables
variables should have two or more unique values per category
if there are only two unique measurement levels, it is the same as Phi Coefficient
the coefficient ranges from 0 to 1
What is the strength of association (i.e. dependence) between gender and eye color?
# chi-square testprint(f'chi-square statistic: {chiRes2[0]}')print(f'p-value: {chiRes2[1]}')# chi-square test statistic (alternative method to derive chi-sq stat)chisq_stat = stats.chi2_contingency(GE, correction=False)[0]# sample sizen = np.sum(np.sum(GE))# minimum of rows & columnsminshape = min(GE.shape)-1# Cramer's vV_ = np.sqrt( (chisq_stat/n)/minshape)print(f"Cramer' V: {V_}")
Interpretation of results:
With a p-value less than 0.05, we reject the null hypothesis (H0: no relationship between the 2 variables). Looking at Cramer’s V, the strength of the dependence is low.
compares the strength of the relationship between two categorical variables
variables only two unique measurement levels
the coefficient ranges from 0 to 1
What is the strength of association (i.e. dependence) between gender and eye color (brown & green only)?
# Phi coefficient; two by by contigency tableGEsubset = GE.drop(columns=['Blue', 'Hazel'])GEsubset
chiRes3 = stats.chi2_contingency(GEsubset)# chi-square testprint(f'chi-square statistic: {chiRes3[0]}')print(f'p-value: {chiRes3[1]}')# chi-square test statistic (alternative method to derive chi-sq stat)chisq_stat = stats.chi2_contingency(GEsubset, correction=False)[0]# sample sizen = np.sum(np.sum(GEsubset))# minimum of rows & columnsminshape = min(GEsubset.shape)-1# PhiPhi_ = np.sqrt( (chisq_stat/n)/minshape)print(f"Phi Coefficient: {Phi_}")
Interpretation of results:
With a p-value less than 0.05, we reject the null hypothesis (H0: no relationship between the 2 variables). Looking at Phi Coefficient, the strength of the dependence is low.
Also known as Mann Whitney Wilcoxon Test or Wilcoxon Rank Sum Test.
Compares differences between two variables when the dependent variable is either ordinal or continuous, & not normally distributed.
independent variable should consist of two categorical, independent groups.
Given the data is not normally distributed, is there a difference in rating between ramen with seafood and non-seafood?
# create dataframe# for 'hasSeafood', 1 stands for noodles with seafood.ramenRating = pd.DataFrame({'hasSeafood':[1,1,1,0,0,0,1,0,1,0,1,0],                            'Stars':[5.0,2.5,1.5,3.5,4.75,3.0,4.0,3.0,3.0,2.75,1.0,1.0]})ramenRating
# mann-whitney u teststats.mannwhitneyu(x=ramenRating['hasSeafood'],                   y=ramenRating['Stars'])
Interpretation of results:
With a p-value less than 0.05, we can reject the null hypothesis that there is no difference in rating between noodles with seafood and without seafood.
Determines if there is a statistically significant difference (note: not dependence!) between paired categorical or dichotomous variables. This sets it apart from the chi-square test of independence.
Non-parametric
measures difference between variable groups
constrained to a two-by-two table (i.e. contingency table for chi-square test may be n-by-two depending on the ordinal groups in the variable).
Tip: to apply the test, input the data as a nested array
Is there a difference between pain relief before and after taking of particular test medication?
# Create tableMed = pd.DataFrame({'postMedpain_No': [400, 800],                    'postMedpain_Yes' : [10, 40]},index={'priorMedpain_No','priorMedpain_Yes'})Med
# convert to arrayframe = Med.to_numpy()frame
# McNemar's Testprint(ct.mcnemar(frame, exact=False))
Interpretation of results:
With a p-value less than 0.05, there is a statistical difference in the number of patients experiencing pain before & after medication.
An extension of the McNemar Test. If there are only two unique values for the variables, Cochran’s Q test is the same as McNemar’s test.
The null hypothesis for the test is that the probability of success for different treatments is the same in the population. If the null hypothesis is rejected, a post hoc analysis is recommended. This is because while we can only infer that there are differences in proportions between the groups, Cochran's Q test doesn’t inform on which specific groups are different.
The dependent variable is dichotomous.
The Independent variable is categorical and has three or more related groups. Related groups are allowed in the sense that it is possible to have the same test subject in each group as it is measured on multiple occasions on the same dependent variable.
The cases are randomly sampled from the population.
The sample size is sufficiently large. As a rule of thumb, the number of subjects for which the responses are not all 0’s or 1’s, n, should be ≥ 4 and nk should be ≥ 24. This is not required for the exact binomial McNemar test.
Additional note: an often overlooked aspect of Cochran’s Q test implementation in Python is that the data format and structure to be passed in needs to be as it would appear in the data records; not in contingency table format.
Is there a difference in depression cases over the 3 subsequent courses of therapy?
# three columns for 3 sessions# 1 represents depression# 0 represents no more depressioncases = np.array([[0, 0, 0],                  [1, 0, 0],                  [1, 1, 0],                  [1, 1, 1]])                      count = np.array([ 6,  16, 4,  2])data = np.repeat(cases, count, 0)# Cochran's Q Testprint(ct.cochrans_q(data))
Interpretation of results:
With a p-value less than 0.05, there is a statistical difference in the batches of patients experiencing depression and no depression between the different number of sessions.
Used to measure the strength and direction of the association
between continuous & categorical variable
What is the strength in the association between the test scores and having studied for a test or not?
# initialize random number generatorrandom.seed(36)# Assume y is studied (1)/ not studied (0) # X is the test score, max 20 marksy = [random.randint(0, 1) for num in range(0, 10)]print(f'y values: {y}')X = []for i in y:    if i != 0:        ele = random.randint(11, 20)            else:        ele = random.randint(1, 10) # 1-10 not studied    X.append(ele)print(f'X values: {X}')
# point-biserial correlation# output is a tupleresult = stats.pointbiserialr(X, y)print(f'correlation between X and y: {result[0]:.2f}')print(f'p-value: {result[1]:.2g}')
Interpretation of results:
With a p-value less than 0.05, we reject the null hypothesis. X and y are dependent; with a strong association between them.
Compares the variance of continuous measures among multiple independent groups of variables
Categorical & continuous variable
can be as One-way or Two-way ANOVA
One-way ANOVA for instances where a variable consists of three or more independent levels of measurements.
Is there a significant difference in income between the three rankings?
# Create a dataframe with artifical values# Sector Ranking, SR1 to SR9, SR1 ranked highest (more than 3 groups)# Salary; continuous variableSRs = ['SR1', 'SR9', 'SR2', 'SR8', 'SR3', 'SR7', 'SR4', 'SR6', 'SR5',       'SR1', 'SR9'] Salary = [10000, 5000, 8000, 5500, 7500, 6000, 7000, 6500, 6800,          11000, 5100]rawData = pd.DataFrame({'ranks':SRs,                        'sal':Salary })
# derive the groups (i.e. sector ranks)groups = pd.unique(frame.ranks.values)# Create a dictionary to hold respective salary of sector rank groupsstatData = {grp: rawData['sal'][rawData['ranks'] == grp] for grp in groups}stats.f_oneway(statData['SR1'], statData['SR2'], statData['SR9'])
Interpretation of results:
The null hypothesis is rejected, with a p-value less than 0.05. There is a significant difference between the income of the analyzed groups of ranks.
Compares if two samples’ mean are different
not used if the sample size is less than 30
assumes sample are independent; follows normal distribution
mean may be unknown, variance is known
Is there a difference in the mean salary of the two business groups?
# create dataframe of salary(K) for 2 business groupsnp.random.seed(123)marketing = np.random.normal(106,2,40) # mean 106, std dev 2, sample 40finance = np.random.normal(108,1,40) # mean 108, std dev 1, sample 40marketingLst = marketing.tolist()financeLst = finance.tolist()salData = pd.DataFrame({'marketing':marketingLst,                         'finance':financeLst})print('marketing median sal:', salData['marketing'].median(), '\n'       'finance median sal:', salData['finance'].median())
# combine the 2 lists to get overall salary distributionmarketingLst.extend(financeLst)salFrame = pd.DataFrame(marketingLst,columns=['salary'])# check normalityfig, ax = plt.subplots(figsize=(4,4))qqplot(salFrame['salary'], fit=True, line='s',ax=ax)plt.show()
The distribution is normal if most points are within the red line. There are Slight skews at both ends. Salary is normally distributed for the most part. z-test is suited for normally distributed data.
# ztest# z-score, p-valuezScore, pValue = ztest(salData['marketing'], salData['finance'])print('zscore:', zScore, 'p-value:', pValue)
Interpretation of results:
The null hypothesis is rejected, with a p-value less than 0.05. There is a significant difference in the mean salary between the two groups.
Similar to z-test, but is applicable for when sample size is less than 30
little difference between z-test & t-test if sample sizes are large
assumes sample follows a normal distribution with unknown mean & variance
One sample t-test compares the mean with a population value
Two sample t-test tests if the means of two samples are equal
Differentiated by equal variance t-test and non-equal variance t-test (i.e. Welch’s test)
Is there a difference between the mean of marketing group salary with the company mean salary?
# Create the data (2 sets of test scores)np.random.seed(42)marketing2 = np.random.normal(106,2,20) # mean 106, std dev 2, sample 20finance2 = np.random.normal(108,2,20) # mean 108, std dev 1, sample 20# Normality can be checked by shew and kurtosis besides qqplot# For normally distributed data, the skewness should be about zero.print('marketing2 skew:', stats.skew(marketing2), '\n'      'finance2 skew:', stats.skew(finance2))# value of kurtosis measure for normal distr is 3print('marketing2 kurtosis:', stats.kurtosis(marketing2), '\n'      'finance2 kurtosis:', stats.kurtosis(finance2))
# 1 sample t-test, tests population mean salary 106stats.ttest_1samp(a=marketing2, popmean=106)
Interpretation of results:
With a p-value greater than 0.05, fail to reject the null hypothesis of the mean salary of the population being 106.
Is there a difference in the mean salaries between the two groups?
# check if variance similarnp.var(marketing2)/np.var(finance2)
With the ratio of the variance is less than 4:1, perform the t-test, else, perform Welch’s t-test (i.e. set equal_var to False.)
# Can assume equal variancestats.ttest_ind(a=marketing2, b=finance2, equal_var=True)
Is there a difference in the mean salaries between the two groups?
# Create the data (2 sets of test scores)np.random.seed(123)marketing3 = np.random.normal(108,2.5,20) # mean 90, std dev 2.5, sample 20finance3 = np.random.normal(108,1,20) # mean 108, std dev 1, sample 20# check if variance similarnp.var(marketing3)/np.var(finance3)
With the ratio of the variance is more than 4:1, perform Welch’s t-test (i.e. set equal_var to False.)
stats.ttest_ind(a=marketing3, b=finance3, equal_var=False)
Interpretation of results:
With a p-value of more than 0.05, fail to reject the null hypothesis of the two salaries mean being equal.
Correlation coefficients give a measure of the linear relationship between the two variables. If using .corr() via pandas, the default correlation method is Pearson.
The range between negative one to one (negative to positive correlation respectively)
the formula for calculating the correlation coefficient standardizes the variables, changes in scale or units of measurement will not affect its value.
Pearson correlation is a measure of the strength and direction of the linear relationship between two continuous variables. It relies on four key assumptions:
Data is interval or ratio. These types of continuous data are important for how correlation assumes values in variables are related. Ordinal or Categorical variable coding won’t work.
Pearson only works with linear data.
No outliers in data.
Data is normally distributed.
Data must be homoscedastic. Homoscedasticity assumes that data is equally distributed about the regression line
Spearman correlation is non-parametric. It relies on nearly all the same assumptions as Pearson, but it doesn’t rely on normality. The scores on one variable must be monotonically related to the other variable. Can be used on pairs of continuous (that don’t follow linear relationship) or ordinal variables.
Kendall rank correlation is non-parametric. It measures the strength of dependence between two variables. It can be used with ordinal or continuous data.
Effect strength: Cohen’s standard may be applied to the coefficient to determine the strength of the effect.
What is the correlation between continuous variables X1, X2, and y?
X1 = [500,400,300,200,100]X2 = [200,400,600,800,1000]y = [100,200,300.5,400,500.5]df = pd.DataFrame({'X1':X1,                   'X2':X2,                   'y' :y})# correlation matrixdf.corr()
Interpretation of results:
Assume y refers to the housing sales price, X1 Sibor rate, and X2 distance(km) from the city. From the coefficients, X1 is strongly negatively correlated with X2 and y. X2 is strongly positively correlated with y.
Determines if there is a statistically significant difference between paired continuous variables.
between two dependent samples (e.g. test scores before and after)
tests whether the changes tend to be in the positive or negative direction
Is there a significant difference between the mid-term and final-term test scores?
# compare the two test scores for each individual student.# create two score samples; one mid-term exam, the other end-of-term examnp.random.seed(123)midTerm = np.random.normal(59.45, 1.5, 100)endTerm = np.random.normal(60.05, 1.5, 100)# Paired (related) testt,p = stats.ttest_rel(endTerm, midTerm)# ttest_rel is 2-tailed, so half the resulting p-value to get a 1-tailed p-valuep1 = '%f' % (p/2)print("t-statistic:" + str(t))print("p-value:" + str(p1))
Interpretation of results:
With a p-value less than 0.1 (90% CI), we reject the null hypothesis. There is a significant difference between the 2 exam scores.
That’s all for now. Hopefully, this has been a useful primer/guide towards your journey through the various kinds of statistical tests.
A concise resource for further reading on variables and statistical approaches: http://web.pdx.edu/~newsomj/pa551/lectur23.htm
Jul2021: Initial Draft
26Oct2021: Updated with new tests, corrected typo errors