Statistical Tests for Comparing Machine Learning and Baseline Performance
By: Stacey Ronaghan
Tags: Machine Learning, Data Science, Hypothesis Testing, Statistics

When comparing a machine learning approach with the current solution, I wish to understand if any observed difference is statistically significant; that it is unlikely to be simply due to chance or noise in the data.

The appropriate test to evaluate statistical significance varies depending on what your machine learning model is predicting, the distribution of your data, and whether or not you’re comparing predictions on the subjects. This post highlights common tests and where they are suitable.

Null Hypothesis Statistical Testing

A null hypothesis statistical test is used to compare two samples of data and calculate statistical confidence that a perceived difference in a mathematical feature (e.g. different mean values) will be observed in the wider population.

Although there is ongoing criticism of null hypothesis significance testing, it is still the most commonly used technique for this validation.

Null Hypothesis

The null hypothesis is the statement that a difference does not exist between the distribution of the two samples of data; that any variance is seen due to noise or chance.

In terms of machine learning, our null hypothesis is that the performance metrics are equal, that any small gain, or loss, observed is not statistically significant.

P-Value

The test indicates if we should reject, or fail to reject, the null hypothesis that there is no difference. The statistical tests discussed below tend to provide a test statistic and a p-value.

If the p-value is below a given threshold (often 0.05), we can reject the null hypothesis and infer that the difference is statistically significant.

Business Interpretation

If you’re comparing your machine learning solution with the current baseline and you’re seeing improved performance, the fact that the improvement is significant is great news!

Similarly, if you’re using these techniques to test if a particularly model performs better than another, this can guide you to decide which to move forward with.

If you cannot reject the null hypothesis (p-value greater than or equal to the threshold), any observed effect or difference between the two data sets is not statistically significant. This can be positive news if you’re trying to automate a task where matching the current manual performance is sufficient. However, if the success criteria is to out-perform the current scenario, no statistically significance difference means that it is unlikely you will see benefit in taking this model to production.

Assumptions

Data Distribution

When using hypothesis tests for regression models, it is important to distinguish if you should use a parametric or non-parametric test. Parametric tests are preferred as they usually have more statistical power than non-parametric test; this means they’re more likely to detect a statistically significant effect if one exists. However, these tests assume that the data is normally distributed; if this assumption does not hold, a non-parametric test must be used.

If you plot error values, it is likely they will be of a non-normal distribution

When you’re comparing error values, it is highly likely that they are skewed to the left: the majority of errors are small and the frequency decreases as the error value increases. If you were to plot your data you may be able to visually interpret this but there are also normality tests you can use to mathematically confirm.

Paired or Unpaired Observations

If data has been collected from the same subjects for each model, this is referred to as “paired”. In machine learning, this means that the test data for the baseline and the trained model are the same.

Data collected from two independent groups is referred to as “unpaired”. This is common in scientific research, or A/B testing, when you are comparing a control group with a treatment group.

Appropriate Statistical Tests

Regression

If your machine learning model is predicting numerical values, the error metric tends to be one of the following:

Mean Square Error (MSE)

Mean Absolute Error (MAE)

Mean Absolute Percentage Error (MAPE)

Weighted Mean Absolute Percentage Error (WMAPE)

When using a hypothesis test to compare average errors, you are looking at the likelihood that the errors come from the same distribution; if the likelihood is sufficiently low, the difference between the two mean error values are statistically significance.

For normally distributed data the two appropriate tests are either Student’s t-test or Welch’s t-test. They both test that two populations have the same mean but Welch’s t-test is more reliable in there is a difference in variance or sample sizes. There are two implementations depending on whether the observations are paired or unpaired.

The two non-parametric tests are either Mann-Whitney U test or Wilcoxon signed-rank test, depending again if the observations are paired or not.

Classification

If your machine learning model is predicting which class an instance belongs to, the error metrics tend to be one of the following:

Accuracy

Precision

Recall

F1-score

Although your business must decide which metric should be prioritized, the most common techniques used with classification models simply evaluate whether the proportions of errors are significantly different. Further research is required if you explicitly wish to test the statistical significance of a precision or F1-score.

Classification models can either be binary (e.g. churned or not churned), or multi-class (e.g. sports, politics or science). For binary classification model comparison, the appropriate test is McNemar’s test, whilst for multi-class it is is the Stuart-Maxwell test.

Algorithm Comparison & Hyper-parameter Tuning

The above metrics are used when evaluating the results of one model vs. the baseline(or alternatively another model).

When evaluating different algorithms, or hyper-parameters, it is common to use k-fold cross validation. This allows you to train and test k models and collect their evaluation metrics, using their mean as an indication of the algorithms performance. If you do this for two algorithms, you can use a statistical test comparing the two means.

For example, if you want to compare a logistic regression model with a random forest model. You could split the data into 10-folds and train 10 logistic regression models and 10 random forest models. The two models will each have 10 corresponding performance metric values (e.g. accuracy) which can then be passed to a statistical test to see if any observed difference is statistically significant.

As k-fold cross validation re-samples the data, there is overlap between the training sets. This means that the values are not independent which is often an assumption for statistical hypothesis tests. Consequently, the two most common techniques are a 5x2 cross-validation with a modified t-test and McNemar’s test.