Increase the Accuracy of Your CNN by Following These 5 Tips I Learned From the Kaggle Community
By: Patrick Kalkman
Tags: TensorFlow, Convolutional Network, Kaggle, Python, Deep Learning

2. Use K-Fold Cross-Validation

Until now, we split the images into a training and a validation set. So we don’t use the entire training set as we are using a part for validation.

Another method for splitting your data into a training set and validation set is K-Fold Cross-Validation. This method was first mentioned by Stone M in 1977.

With K-Fold Cross-Validation, you divide the images into K parts of equal size. You then train your model K number of times with a different training and validation set.

This way, you make optimal use of all your training data.

K-Fold Cross Validation (k = 5), image by the author

It is crucial to note that you will train many models, one for each fold. This means changing the way we make predictions. We have the following options.

Use a single model, the one with the highest accuracy or loss.

Use all the models. Create a prediction with all the models and average the result. This is called an ensemble.

Retrain an alternative model using the same settings as the one used for the cross-validation. But now use the entire dataset.

Implementing K-Fold Cross-Validation

The scikit-learn library contains objects to help us. Two objects can help us with dividing our training data into folds. These are KFold and StratifiedKFold .

KFold

The KFold object splits our training data into k consecutive folds. When creating the object, you choose the number of folds. If you then call split on the object, it returns two arrays. The first array contains indices from our training data for training. The second array contains indices from our training data for validation.

Using KFold from scikit-learn

In row five, we create the KFold object and instruct it to create five different folds and shuffle the data. Then in row eight, we start a loop that will run five times. Each run returns the train_index and val_index array that contains indexes into the train_data data frame.

We then create two arrays, training_data , and validation_data that we use with the ImageDataGenerator .

StratifiedKFold

StratifiedKFold differs from normal KFold in that it makes sure that each fold has the same percentage of samples of each class. This is especially useful if your training data is not uniformly distributed.

Number of training images per disease category, image by the author

This is the case with our training data. So, we use StratifiedKFold. The implementation is the same as with KFold.

Using StratifiedKFold from scikit-learn

We iterate over all folds and use each training and validation set to train a model. Each model gets a unique filename. We use that filename when saving the model with the lowest loss.

We then add the history object that TensorFlow returns from the fit method to an array. We use this array to create a graph of each fold at the end of the training. The pre-trained model that I used for each fold was EfficientNet-B3.

The individual graphs did not show an increase in validation accuracy, as you can see in the charts of fold 1 and 2.