Testing Machine Learning Models
Tags: Machine Learning, Artificial Intelligence, Big Data, AI, Programming

Model Evaluation in Machine Learning Testing

Usually, software testing includes:

Unit tests. The program is broken down into blocks, and each element (unit) is tested separately.

The program is broken down into blocks, and each element (unit) is tested separately. Regression tests. They cover already tested software to see if it doesn’t suddenly break.

They cover already tested software to see if it doesn’t suddenly break. Integration tests. This type of testing observes how multiple components of the program work together.

Moreover, there are certain rules that people follow: Don’t merge the code before it passes all the tests; always test newly introduced blocks of code: when fixing bugs, write a test that captures the bug.

Machine learning adds more actions to your to-do list. You still need to follow ML’s best practices. Moreover, every ML model needs to not only be tested but evaluated. Your model should generalize well. This is not what we usually understand by testing, but evaluation is needed to make sure that the performance is satisfactory.

First of all, you split the database into three non-overlapping sets. You use a training set to train the model. Then, to evaluate the performance of the model, you use two sets of data:

Validation set. Having only a training set and a testing set is not enough if you do many rounds of hyperparameter-tuning (which is always). And that can result in overfitting. To avoid that, you can select a small validation data set to evaluate a model. Only after you get maximum accuracy on the validation set do you make the testing set come into the game.

Having only a training set and a testing set is not enough if you do many rounds of hyperparameter-tuning (which is always). And that can result in overfitting. To avoid that, you can select a small validation data set to evaluate a model. Only after you get maximum accuracy on the validation set do you make the testing set come into the game. Test set (or holdout set). Your model might fit the training dataset perfectly well. But where are the guarantees that it will do equally well in real life? In order to assure that, you select samples for a testing set from your training set — examples that the machine hasn’t seen before. It is important to remain unbiased during selection and draw samples at random. Also, you should not use the same set many times to avoid training on your test data. Your test set should be large enough to provide statistically meaningful results and be representative of the data set as a whole.

But just like test sets, validation sets wear out when used repeatedly. The more times you use the same data to make decisions about hyperparameter settings or other model improvements, the less confident you are that the model will generalize well on new, unseen data. So it is a good idea to collect more data to freshen up the test set and validation set.

Cross-validation

Cross-validation is a model evaluation technique that can be performed even on a limited dataset. The training set is divided into small subsets, and the model is trained and validated on each of these samples.

k-fold cross-validation

The most common cross-validation method is called k-fold cross-validation. To use it, you need to divide the dataset into k subsets (also called folds) and use them k times. For example, by breaking the dataset into ten subsets, you will perform a tenfold cross-validation. Each subset must be used as the validation set at least once.

This method is useful to test the skill of the machine learning model on unseen data. It is so popular because it is simple to apply, it works well even with relatively small datasets, and the results you get are generally quite accurate. If you want to learn more about how to cross-validate the model, check out a more detailed explanation on Medium.

Leave-one-out cross-validation

In this method, we train the model on all the data samples in the set except for one data point that is used to test the model. By repeating this process iteratively, each time leaving a different data point as a testing set, you get to test the performance for all the data.

The benefit of the method is low bias since all the data points are used. However, it also leads to higher variation in testing because we are testing the model against just one data point each time.

Cross-validation provides for more efficient use of the data and helps to better assess the accuracy of the model.