Proper Model Selection through Cross Validation
By: Günter Röhrich
Tags: Machine Learning, Data Science, K Fold Cross Validation, Crossvalidation, Regression

The Considerations Behind Cross Validation

So, what is cross validation? Recalling my post about model selection, where we saw that it may be necessary to split data into three different portions, one for training, one for validation (to choose among models) and eventually measure the true accuracy through the last data portion. This procedure is one viable way to choose the best among several models. Cross validation (CV) is not too different from this idea, but deals with the model training/validation in quite a smart way.

Photo by Anthony Intraversato on Unsplash

For CV we use a larger combined training and validation data set, followed by a testing dataset. A common way is to split data in 80/20% portions. Make sure to randomly select the data! The CV algorithm will use training/validation set for running multiple folds of cross validation — this is where the name “k-Fold Cross Validation” comes from.

This means, that training/validation data is implicitly split into several portions. To make it more tangible, we assume a k of 5 where the first 4 parts are used for training, the last part is then used for validation. In the next step, we are using the first 3 and the 5th whereas the 4th part is used for validation. You see where this is going.

What you need to remember: When we are creating 5 models, each data block is used k-1 times for training and once for validation. If this sounds confusing, just keep going. Have a look at the below illustration, showing different parts used per run.

Example: k = 5 Fold CV

Now having 5 different models for a k=5, the question is, what accuracy should be used? The answer is simple, for k=5 the overall model accuracy is the average of all five models. This is necessary to mention, in case we tested different parameters resulting in several different model-combinations.

After having decided which model (under which k) seems to work best, we run in once again on the whole test/train data (no split this time). Having our fitted model in hands, we then are required to test the the true performance on the test data. This step should sound familiar, it is basically the same as I have set out in the test-validation-training process.

Caveats

It is important that the model has never seen the test data before! Also note, if the models performance is not doing well on the test data, consider this to be your (potentially not desirable) result. It is not meaningful to tweak your model as you likely end up overfitting the model. The reason for this? You guessed right, random effects.

One more thing. Have you heard of Leave One Out CV? This is another form of the k-fold CV. In LOOCV we use a k value of n (the row dimension) — hence, we use all the data and create n models. From what I observed, LOOCV is often giving good results, however may take considerably longer than the usual k=10.

Do you have to write the CV function yourself? No, usually not. Referring to R programming language, you will often find functions that sound like the model you are about to fit, but having “cv” somewhere incorporated in its name. If you intended to create a simple linear model (lm), cv.lm is doing exactly what we were looking at above. Similar functions can be found for Python and Julia. Bottom line here, use a search engine to browse through your options — there are plenty.

Conclusion

Applying machine learning algorithms and statistical models has become quite straightforward. There are loads of available documentation and tutorials that guide you through the model fitting process. Why this rather “not so exciting” topic is quite important to me is simple: choosing the right data to create the model might have a sustainable influence on the model’s quality. Not considering the topic of CV and data that is used to create a model will likely result in overfitted and less accurate models overall.

Ending up with an “overestimated model” is nothing one should go with, and most certainly will come back to the creator sooner or later.

Equipped with knowledge about how and when to apply cross validation, hence reducing the risk of overfitting the model, you are well prepared to create all kind of machine learning models based on the right data and its implicit effects.

{See you next time.}