Cross Validation — Why & How. Importance Of Cross Validation In... | by Amitrajit Bose | Towards Data Science

So, you have been working on an imbalanced data set for a few days now and trying out different machine learning models, training them on a part of your data set, testing their accuracy and you are ecstatic to see the score going above 0.95 every-time. Do you really think you have achieved 95% accuracy with your model?
I’m assuming that you have performed top-notch pre-processing on your data-set, also you have removed any missing values or categorical values and noise. Whatsoever state-of-the-art algorithm you have used to build your hypothesis function and train the machine learning model, you have to evaluate its performance before moving forward with it.
Now the easiest and fastest method to evaluate a model is to split the data set into training and testing set, train the model using the training set data and check its accuracy with the accuracy. And do not forget to shuffle the data set before performing the split. But this method is not at all an assurance, in simple words you cannot rely on this approach while finalizing a model. You might be wondering — Why?
Let us consider, you are working on a spam emails data set, which contains 98% of spam emails and 2% of non-spam valid emails. In this case, even if you do not create any model but just classify every input as spam, you will be getting 0.98 accuracy. This condition is called accuracy paradox.
Imagine what would happen if this was a model for classification of tumour cells or chest X-rays and you had pushed a 98% accurate model to the market. Maybe this would have killed hundreds of patients, you never know.
Do not worry and grab a cup of something warm. In the article below, I will be explaining the entire process of evaluation of your machine learning model. All you need to know is some basic Python syntax as a prerequisite.
We initially split up the entire data we have got into two sets, one is used to train the model upon and the other is kept as a holdout set which is used to check how the model behaves with completely unseen data. The figure below summarises the entire idea of performing the split.
Please note, that the train test ratio can be anything like 80:20, 75:25, 90:10, etc. This is a decision that the machine learning engineer has to take based on the amount of data. A good rule of thumb is to use 25% of the data-set for testing.
You can do this with ease using some Python and the open source Sci-kit Learn API.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42, shuffle = True, stratify = y)
X is the original entire set of features and y is the entire set of corresponding true labels. The above function splits the entire set into train and test set with a ratio of 0.3 assigned for the test set. The parameter shuffle is set to true, thus the data set will be randomly shuffled before the split. The parameter stratify is recently added to Sci-kit Learn from v0.17 , it is essential when dealing with imbalanced data sets, such as the spam classification example. It makes a split so that the proportion of values in the sample produced will be the same as the proportion of values provided to the parameter stratify. For example, if the variable y is a binary categorical variable with values 0 and 1 and there are 10% of zeros and 90% of ones, stratify=y will make sure that your random split has 10% of 0's and 90% of 1's.
As we were discussing, only checking how many examples from the test set were classified correctly is not a useful metric for checking model performance because of factors such as class imbalance. We need a more robust and nuanced metric.
Say hello to the confusion matrix. An easy and popular method of diagnosing model performance. Let us understand this with our scenario of spam email classification. The confusion matrix would look like this.
There are several metrics that can be deduced from the confusion matrix, such as —
Accuracy = (TP + TN) /(TP + TN + FP + FN)Precision = (TP) / (TP + FP)Recall = (TP) / (TP + FN)F1 Score = (2 x Precision x Recall) / (Precision + Recall)—  where TP is True Positive, FN is False Negative and likewise for the rest.
Precision is basically all the things that you said were relevant whereas Recall is all the things that are actually relevant. In other words, recall is also referred to as the sensitivity of your model, whereas precision is referred to as Positive Predicted Value. Here’s a one-pager cheat sheet to summarise it all.
Now that you have grasped the concept, let's understand how to do it with ease using the Sci-kit Learn API and a few lines of Python.
from sklearn.metrics import confusion_matrix, classification_reporty_pred = model.predict (X_test)print(confusion_matrix(y_test, y_pred))print(classification_report(y_test, y_pred))
Assuming that you have prepared the model using the .fit() method on the training set (on which I shall write probably some other day), you then calculate the predicted set of labels using the .predict() method of the model. You have the original labels for these in y_test and you pass the two arrays into the above two functions. What you’ll get will be a two-by-two confusion matrix (because spam classification is binary classification) and a classification report which returns all the above-discussed metrics.
Note: The true values are passed as the first parameter and the predicted values is the second parameter.
Cross validation is a technique for assessing how the statistical analysis generalises to an independent data set.It is a technique for evaluating machine learning models by training several models on subsets of the available input data and evaluating them on the complementary subset of the data. Using cross-validation, there are high chances that we can detect over-fitting with ease.
There are several cross validation techniques such as :-1. K-Fold Cross Validation2. Leave P-out Cross Validation3. Leave One-out Cross Validation4. Repeated Random Sub-sampling Method5. Holdout Method
In this post, we will discuss the most popular method of them i.e the K-Fold Cross Validation. The others are also very effective but less common to use.
So let’s take a minute to ask ourselves why we need cross-validation — We have been splitting the data set into a training set and testing set (or holdout set). But, the accuracy and metrics are highly biased upon how the split was performed, it depends upon whether the data set was shuffled, which part was taken for training and testing, how much, so on. Moreover, it is not representative of the model’s ability to generalize. This leads us to cross validation.
First I would like to introduce you to a golden rule — “Never mix training and test data”. Your first step should always be to isolate the test data-set and use it only for final evaluation. Cross-validation will thus be performed on the training set.
Initially, the entire training data set is broken up in k equal parts. The first part is kept as the hold out (testing) set and the remaining k-1 parts are used to train the model. Then the trained model is then tested on the holdout set. The above process is repeated k times, in each case we keep on changing the holdout set. Thus, every data point get an equal opportunity to be included in the test set.
Usually, k is equal to 3 or 5. It can be extended even to higher values like 10 or 15 but it becomes extremely computationally expensive and time-consuming. Let us have a look at how we can implement this with a few lines of Python code and the Sci-kit Learn API.
from sklearn.model_selection import cross_val_scoreprint(cross_val_score(model, X_train, y_train, cv=5))
We pass the model or classifier object, the features, the labels and the parameter cv which indicates the K for K-Fold cross-validation. The method will return a list of k accuracy values for each iteration. In general, we take the average of them and use it as a consolidated cross-validation score.
import numpy as npprint(np.mean(cross_val_score(model, X_train, y_train, cv=5)))
Although it might be computationally expensive, cross-validation is essential for evaluating the performance of the learning model.
Feel free to have a look at the other cross-validation score evaluation methods which I have included in the references section, at the end of this article.
The accuracy requirement of a machine learning model varies across industry, domain, requirement and problem statement. But confirming a final model should never be performed without evaluation of all essential metrics.
By the way, once you are done with evaluation and finally confirming your machine learning model, you should re-use the test data that was initially isolated for testing purposes only and train your model with the complete data you have so as to increase the chances for a better prediction.
Thanks for reading. This was a high-level overview of the topic, I tried to put my best efforts to explain the concepts at hand in an easy way. Please feel free to comment, criticize and suggest improvements to the article. Also, claps highly encourage me to write more! Stay tuned for more articles.
Have a look at this friendly introduction to neural networks with PyTorch.
[1]Leave P-out Cross Validation[2]Leave One-out Cross Validation[3]Repeated Random Sub-sampling Method[4]Holdout Method[5]Cross Validation[6]Stanford’s MOOC — Statistical Learning With R — Course notes