Proper Model Selection through Cross Validation | by Günter Röhrich | Nov, 20 | Towards Data Science

In this article, I will outline the basics of cross validation (CV), how it compares to random sampling, how (and if) ensemble learner need to use CV and how we should build models when we only have a few data points.
In a previous post, I introduced a sound process to split data in order to reduce the risk of fitting the model to random patterns. To briefly recap these two different types of effects:
Real effects: These effects are the one we would like to fit our model to. They show a pattern we can hopefully reproduce / estimate through our model.
Random effects: You might have guessed it, they just happen randomly. We can safely assume that every real world dataset will have those effects, hence fitting a model to 99% of all data points, will lead to the problem that the model will perform bad on the unknown 1% portion, the test data.
I you want to catch up on this topic, follow me here:
towardsdatascience.com
So, what is cross validation? Recalling my post about model selection, where we saw that it may be necessary to split data into three different portions, one for training, one for validation (to choose among models) and eventually measure the true accuracy through the test data portion. This procedure is a more stable (and preferred) way to choose the best among several models. Cross validation is not too different from this idea, but deals with the model training/validation in a smarter and computationally more efficient way:
For CV we use a full dataset, hence we do no longer require a split into train/test data portions beforehand, but rather take advantage of processing the full dataset and iteratively use different portions. These sampled sub-portions are commonly referred to as folds. The folds are sampled randomly and most commonly, the dataset is divided into 10 equal sized folds. For every iteration in the CV procedure, we use k-1 folds to train the model and use the remaining fold as a test portion. The 10 folds are not resampled for every iteration, but remain as initially drawn.
The CV algorithm will calculate the CV error, which relates to the testing error of the model and averages this error over all k runs. This results in an averaged error metric that allows us to infer a more reliable model accuracy.
To contrast train-test-split procedures, CV provides a more robust and reliable error score, as the data is not trained on only one dataset and tested through another, but rather trained on k different combinations of data, while being independently tested for each of this k cycles. Keep in mind, this is especially useful, when there is only a small dataset available.
What you need to remember: When we are creating 5 models, each data block is used k-1 times for training and once for validation:
Remarks on averaging results: As described, if we built 5 models, that is k equals 5, what accuracy score should be considered? One idea would be to select the best or worst score out of all sub-models, but this wouldn’t provide a reliable metric, being overly optimistic or pessimistic. The simple answer is, we define model accuracy as the average of all k models. It is important to understand that we do NOT average the parameters of the (sub) models (e.g. coefficients). If the best model is selected under CV, we fit the model to the entire dataset and use this model for further proceedings.
In random sampling (when comparing several models), we usually set aside a validation dataset. Of course, this can also be done for CV: For example, we could use a random subset of 90% of the data to build our models using CV and validate the best model with the lowest CV-error through the validation dataset — I wouldn’t be too surprised however, if the results is similar to the CV error metric.
Leave-one-out-CV (LOOCV) is a special case of CV, rather than picking a number of folds (e.g. k=10), we use k=N. This means we build N models (one for every row in our dataset), but always remove the i-th row from the model training and use this one for testing. Keep in mind that this is probably infeasible for very large datasets — and less computationally efficient than k-fold CV, in general.
You might be confronted with the situation where there is no pre-built cross validation functionality available for your model algorithm — e.g. Splines in R do not offer cross validation functionality. An easy, yet powerful tool to achieve a more realistic error metric is to do random sub sampling.
MCCV is computationally expensive, so make sure to apply it only if the dataset is not too large: Initialize an empty array that stores the CV error for every run, then run B loops (e.g. 100) and for each loop calculate the respective error score. After all runs, calculate an average error metric. The pseudo code is as follows:
results = array([  ])for loop in 100:    train, test = data.split # split data randomly, e.g. 80/20    error = model(train).get_CV_error()    results.append(error)mean(results) # gets us the average error score of all 100 runs
Hardly any machine learning topic can be discussed without having the bias-variance-trade-off in mind. As for CV, keep in mind that larger values of k usually tend to increase the variance of our response, however decrease the bias.
Tree-based ensemble learners use different (random) subsets of the same data to train a model. Given the fact that ensemble learner create a variety of similar models based on different data subsets, there is no need to use calculate a test error through the test data.
As Hastie, Tibshirani, & Friedman point out in their book “The Elements of Statistical Learning” (2009 p. 593), the out of bag error is almost identical to the CV error, meaning there is no need to implement another layer of CV for ensemble methods, like the random forest.
Train-test splits help obtaining a test error by using different datasets. This approach is obviously better than using only one dataset — which results in entirely overfitted models — but provides less stable results than cross validation.
To the question, do you have to write the CV function yourself? No, usually not —and keep in mind you can always consider applying MCCV. Referring to R programming language, you will often find functions that sound like the model you are about to fit, but having “cv” somewhere incorporated in its name. If you intended to create a simple linear model (lm), cv.lm is doing exactly what we were looking at above. Similar functions can be found for Python and Julia. Bottom line here, use a search engine to browse through your options — there are plenty.
Other functions provide the option of CV as a parameter as in this example using 10-fold CV:
Applying machine learning algorithms and statistical models has become quite straightforward. There are loads of available documentation and tutorials that guide you through the model fitting process. Why this rather “not so exciting” topic is quite important to me is simple: how to choose the right the model (among several). Not considering the topic of CV and data that is used to create a model will likely result in overfitted and less accurate models overall.
{See you next time}