What is Cross-Validation?. Testing your machine learning models... | by Mohammed Alhamid | Towards Data Science

Cross-Validation (CV) is one of the key topics around testing your learning models. Although the subject is widely known, I still find some misconceptions cover some of its aspects. When we train a model, we split the dataset into two main sets: training and testing. The training set represents all the examples that a model is learning from, while the testing set simulates the testing examples as in Figure 1.
CV provides the ability to estimate model performance on unseen data not used while training.
Data scientists rely on several reasons for using cross-validation during their building process of Machine Learning (ML) models. For instance, tuning the model hyperparameters, testing different properties of the overall datasets, and iterate the training process. Also, in cases where your training dataset is small, and the ability to split them into training, validation, and testing will significantly affect training accuracy. The following main points can summarize the reason we use a CV, but they overlap. Hence, the list is presented here in a simplified way:
One of the critical pillars of validating a learning model before putting them in production is making accurate predictions on unseen data. The unseen data is all types of data that a model has never learned before. Ideally, the testing data is supposed to flow directly to the model in many testing iterations. However, in reality, access to such data is limited or not yet available in a new environment.
The typical 80–20 rule of splitting data into training and testing can still be vulnerable to accidentally ending up in a perfect split that boosts the model accuracy while limiting it from performing the same in a real environment. Sometimes, the accuracy calculated this way is mostly a matter of luck! The 80–20 is not an actual rule per se, and you will find alternative ratios that range between 25~30% for testing and 70~75% for training.
Finding the best combination of model parameters is a common step to tune an algorithm toward learning the dataset’s hidden patterns. But, doing this step on a simple training-testing split is typically not recommended. The model performance is usually very sensitive to such parameters, and adjusting those based on a predefined dataset split should be avoided. It can cause the model to overfit and reduce its ability to generalize.
For parameter tuning and avoid model overfitting, you might find some recommendations around splitting the dataset into three partitions: training, testing, and validation. For instance, 70% of the data is used for training, 20% for validation, and the remaining 10% is used for testing. In cases where the actual dataset is small, we might need to invest in using the maximum amount of data to train the model. In other instances, splitting into such three partitions could create bias in the training process where some significant examples are kept in training or validation splits. Hence, CV can become handy to resolve this issue.
Sometimes the splits of training-testing data can be very tricky. The properties of the testing data are not similar to the properties of the training. Although randomness ensures that each sample can have the same chance to be selected in the testing set, the process of a single split can still bring instability when the experiment is repeated with a new division.
Cross-Validation has two main steps: splitting the data into subsets (called folds) and rotating the training and validation among them. The splitting technique commonly has the following properties:
Each fold has approximately the same size.
Data can be randomly selected in each fold or stratified.
All folds are used to train the model except one, which is used for validation. That validation fold should be rotated until all folds have become a validation fold once and only once.
Each example is recommended to be contained in one and only one fold.
K-fold and CV are two terms that are used interchangeably. K-fold is just describing how many folds you want to split your dataset into. Many libraries use k=10 as a default value representing 90% going to training and 10% going to the validation set. The next figure describes the process of iterating over the picked ten folds of the dataset.
Figure 2 shows how one fold in each step iteratively is held out for testing while the remaining folds are used to build the model. Each step calculates the prediction error. Upon the completion of the final step, a list of computed error are produced of which we can take the mean.
Figure 3 shows the change in the training and validation sets’ size when using different values for k. The training set size increases whenever we increase the number of folds while the validation set decreases. Typically, the smaller the validation set, the likelihood that the randomness rises with a high noise variation. Increasing the training set size will increase such randomness and bring more reliable performance metrics.
One of the advantages of CV is observing the model predictions against all the instances in the dataset. It ensures that the model has been tested on the full data without testing them simultaneously. Variations are expected in each step of the validation; therefore, computing the mean and standard deviation can reduce the information into a few comparing values.
There are other techniques on how to implement cross-validation. Let’s jump into some of those:
LOOCV is the an exhaustive holdout splitting approach that k-fold enhances. It has one additional step of building k models tested with each example. This approach is quite expensive and requires each holdout example to be tested using a model. It also might increase the overall error rate and becomes computationally costly if the dataset size is large. Usually, it is recommended when the dataset size is small. Figure 4 demonstrates the overall process of a simple dataset containing ten examples.
It is another exhaustive technique for performing CV. We can define in each iteration how many examples shall be used for testing the model and how many shall be left for training. The process is repeated for all possible combinations of pairs in the dataset. Note that LOOCV is a LOPCV technique where p =1.
Let’s refresh our minds on how to split the data using the Sklearn library. The following code divides the dataset into two splits: training and testing. We defined here that 1/3 of the dataset should be used for testing.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)
We will then build a function to show how k-fold works compared to the single split in the previous code.
def k_Fold_Split(data, n_splits, shuffle, random_state, verbose=False):    # Creating k-fold splitting     kfold = KFold(n_splits, shuffle, random_state)    sizes = [0 , 0]    for train, test in kfold.split(data):        if verbose:            print('train: indexes %s, val: indexes %s, size (training) %d, (val) %d' % (train, test, train.shape[0], test.shape[0]))        sizes[0] += train.shape[0]        sizes[1] += test.shape[0]            return int(sizes[0]/n_splits), int(sizes[1]/n_splits)
The `k_fold_split` function shows the indexes and the size of each partition. It enables you to track how much data is dedicated for each fold and the chosen indexes.
Using LOOCV as a splitting strategy is pretty straight forward. We will use again Sklearn library to perform the cross-validation.
from sklearn.model_selection import LeaveOneOutcv_strategy = LeaveOneOut()# cross_val_score will evaluate the model scores = cross_val_score(estimator, X, y, scoring='accuracy', cv=cv_strategy, n_jobs=-1)
Cross-validation is a great way to ensure the training dataset does not have an implicit type of ordering. However, some cases require the order to be preserved, such as time-series use cases. We can still use cross-validation for time-series datasets using some other technique such as time-based folds.
Dealing with cross-validation in an unbalanced dataset can be tricky as well. If oversampling is used, the leaking of the oversampled examples can mislead the CV results. One way to consider is the use of stratified sampling instead of splitting randomly. Here is a fantastic blog article discussing how to handle this situation.
CV measures the generalization of the model, but it cannot avoid bias altogether. Nested cross-validation focuses on ensuring the model’s hyperparameters are not overfitting the dataset. The nested keyword comes to hint at the use of double cross-validation on each fold. The hyperparameter tuning validation is achieved using another k-fold splits on the folds used to train the model.
As mentioned earlier, CV is used to measure if a learning model can generalize on unseen data. We rely on using a validation dataset for early stopping and parameter tuning different from the testing set during the building of models. There are a couple of research papers recently suggesting the use of CV to measure overfitting as well. In this case, the direct application would be the use of CV as a validation set for a learning model.
Cross-validation is a procedure to evaluate the performance of learning models. Datasets are typically split in a random or stratified strategy. The splitting technique can be varied and chosen based on the data’s size and the ultimate objective. Also, the computational cost plays a role in implementing the CV technique. Regression and classification problems can use CV, but careful consideration must be paid for some types of datasets such as time-series. A complete example of the code can be found on my Github.