Do and don’t when using transformation to improve CNN deep learning model | by Chien Vu | Towards Data Science

According to wikipedia [1]
Data transformation is the process of converting data from one format or structure into another format or structure.
In computer vision, Data Augmentation is very important to regularize your network and increase the size of your training set. There are many Data transformation techniques (rotation, flip, crop, etc...) that can change the images’ pixel values but still keep almost the whole information of the image, so that human could hardly tell whether it was augmented or not. This forces the model to be more flexible with the large variation of object inside the image, regarding less on position, orientation, size, color,...Models trained with data augmentation usually generalize better but the remaining question is to what extend the Data Transformation can improve the performance of CNN model on image dataset.
Let’s take a look at some Data transformation techniques supporting by Pytorch in Torchvision that can help you to conduct augmentation on your ơn images. I also provide an experiment with real image dataset to show the performance when applying different approaches.
torchvision.transforms.CenterCrop(size): It’s similar to zooming in the center of image. It will crop the given image to a desired output size and position (size can be square or rectangle).
torchvision.transforms.Pad(padding): It equals to zooming out the image. It will create a pad outside the given image on all sides with the certain value.
torchvision.transforms.RandomCrop(size, padding): this function will crop the given image at random locations to create a bunch of images for training.
torchvision.transforms.RandomHorizontalFlip(p): This function will flip horizontally the given image randomly with a given probability.
torchvision.transforms.RandomVerticalFlip(p):This function will flip the given image vertically in random with a given probability.
torchvision.transforms.RandomPerspective(distortion_scale, p): This unction will perform perspective transformation of the given image randomly given a probability. It reduces the effect of perspective for model learning by distorting whole the image.
torchvision.transforms.Grayscale(num_output_channels): Convert image to grayscale. It’s sometimes helpful for CNN model to train faster with single channel and to learn more easily the pattern of images.
torchvision.transforms.ColorJitter(brightness, contrast, saturation, hue):I can randomly change the brightness, contrast and saturation of an image
torchvision.transforms.Normalize(mean, std): Normalize a tensor image with mean and standard deviation. It will help the CNN model to easily convert to global minimum or quickly reduce the loss. If you want to know why data normalization is essential and how to conduct normalization to improve the performance of machine learning model, you can refer to the below blog.
towardsdatascience.com
torchvision.transforms.RandomErasing(p, scale, ratio, value) randomly selects a rectangle region in the image and erases its pixels. This method penalizes the CNN model and helps to prevent the over-fitting phenomenon when training.
Augmentation can also be applied for NLP to improve performance. For example: randomly insert, swap, replace words in the sentences; using an intermediate language in translation (Back translation), shuffle sentences.
There are several libraries providing excellent augmentation modules such as Albumentations, NLPAug... Below is a great article for further exploration from neptune.ai.
neptune.ai
Let’s discuss the effect of Transformation technique on CNN model. The dataset used in this experiment is Dog Breed Identification from Kaggle competition which provided a strictly canine subset of ImageNet in order to practice fine-grained image categorization. This dataset includes 120 breeds of dogs with total 10,222 images. We split data to train/valid at 0.8/0.2 ratio.
In this article, pre-trained ResNet [2] (Residual Networks) model will be used as backbone for Classification CNN model. ResNet is one of the SOTA pre-trained models that reveals good results on a wide range of Computer Vision tasks. Instead of learning input features of image directly with a function H(x) (A stacking of multi-layer perceptron networks), the main idea of ResNet is to provide a residual function that can reframe H(x) = F(x)+x, where F(x) and x represent the stacked non-linear layers and the identity function (input=output) respectively. This idea can solve the degradation problem of deep neuron networks (vanishing gradient) because it is easier to optimize the residual mapping function F(x) than the original one.
First, we will load the original data.
# Create a custom Dataset class to read in data from dataframeclass BreedDogDataset(Dataset):    def __init__(self, dataframe, root_dir, transform=None):        self.dataframe = dataframe        self.root_dir = root_dir        self.transform = transform            def __len__(self):        return (len(self.dataframe))        def __getitem__(self, idx):        img_name = os.path.join(self.root_dir,                  self.dataframe.iloc[idx, 0])        image = Image.open(img_name)        target = self.dataframe.iloc[idx, 1]        target_processed = breeds_processed_dict[target]        if self.transform:            image = self.transform(image)        sample = (image, target_processed)        return sampletransform = transforms.Compose([transforms.Resize((255, 255)),                                transforms.ToTensor()])train_ds = BreedDogDataset(df_train,TRAIN_DIR, transform=transform)val_ds = BreedDogDataset(df_val,TRAIN_DIR, transform=transform)
Then we create a model like this structure:
# load pre-trained modelconv_base = models.resnet50(pretrained=True)# create a new model classclass Model(nn.Module):    def __init__(self, base_model, output):        super(Model, self).__init__()        self.base_model = base_model        self.output = output        self.fc1 = nn.Linear(n_outs, 512)        self.fc2 = nn.Linear(512, output)        self.dropout = nn.Dropout(0.5)        def forward(self, image):        x = self.base_model(image)        x = self.dropout(x)        x = F.relu(self.fc1(x))        x = self.dropout(x)        outs = self.fc2(x)        return outs
One of the most important hyperparameters to tune in a deep learning model is the learning rate. Picking the right learning rate is very important, if it’s too high, the model will face with diverge problem but if it’s too low, the model will take a lot of time to converge. The idea is that we can find a good learning rate by training the model for a few hundred iterations and then based on the learning curve, we pick up a good learning rate. There are a lot of Learning Rate Scheduling techniques, for example, Power scheduling, Exponential scheduling, 1Cycle Scheduling, constant scheduling,... In this experiment, we will apply 1Cycle Scheduling. In Leslie Smith’s paper, this technique was proved to push up remarkably the training time. You can check out this blog for more explanation.
towardsdatascience.com
According to the above learning curve result, it shows that the lower bound is 0.9E-04, I rounded values to 1.0E-03. Now, I can fine-tune the pre-trained ResNet50 on the new image dataset by defining a range of learning rates using the rule of thumb: starting at a rate that is one order lower than the learning rate where loss is minimum and ending at the “secure” rate 1E-03.
After training with 5 epochs, we got the accuracy in valid set is 83.284 %
Train loss and valid loss are shown below. It’s seen that both values decreased. From the 4th to 5th epoch, both loss and accuracy were almost constant. It means the model has learned relatively everything from data, all the good information that helps the model to distinguish breed dog was captured.
Then we check the top 6 highest loss images. We can see that one of the reasons for the model misclassifies some images is that the dog breed prediction is similar to other different breed samples, this is even impossible for humans to differentiate them. To solve this problem, the Transformation technique is very promising to help the model to learn deeply the pattern of each breed. Another issue is that some pictures have 2 different dogs within but the label shows only one. A wrong label is also another issue. These problems cannot be solved and it seems to be an outlier case.
Let’s apply some Transformation technique in this dataset.
mean = [0.4766, 0.4527, 0.3926]std = [0.2275, 0.2224, 0.2210]# define transform functiontransforms_train = transforms.Compose([    transforms.Pad(25, padding_mode='symmetric'),    transforms.RandomHorizontalFlip(),     transforms.RandomRotation(10),    transforms.ToTensor(),    transforms.Normalize(mean, std),    transforms.RandomErasing(p=0.75,scale=(0.02, 0.1),value=1.0, inplace=False)])transforms_val = transforms.Compose([    transforms.Resize((255, 255)),    transforms.ToTensor(),    transforms.Normalize(mean, std)])# load in the temporary dataset from the original dataframe with transformtrain_ds = BreedDogDataset(df_train,TRAIN_DIR, transform=transforms_train)val_ds = BreedDogDataset(df_val,TRAIN_DIR, transform=transforms_val)
After training with 5 epochs on the transformed dataset, we got the accuracy in valid set reached 0.817, a a slight degrading compared to the original image.
The new train loss and valid loss are shown below. The distance from train loss and valid loss increased vs the above model while the accuracy in the 4th and 5th epoch slightly dropped. This indicates that the new model faced an underfitting problem. Generally, transformation does help the model to prevent the overfitting issue. However in this case, because the previous model didn’t have the overfitting issue, and the training loss was even higher than valid loss, when applying the transformation, there was no improvement.
Let’s see the top 6 images that have the highest loss. We can see that some confusing images now could be classified but some haven’t. However, this problem cannot be solved and some outliers still affect the performance of model.
The main purpose of this blog is to introduce some techniques to transform images. Fundamentally, this will help to avoid the overfitting problem as well as to increase the number of training data for the CNN model. But it doesn’t mean that Transformation will always improve the model’s accuracy, it must depend on the input image (satellite, bacteria, animal, objects,..) and model structure. Many Deep-ML practitioners meet this problem, they usually did transform before checking how well their model can learn from their original image. If the model is underfitting the original image, transforming data wasn’t help to solve the problem, in other words, it will decrease the accuracy of your model. Therefore, we have to do step by step to diagnose our model and analysis the results visualize via the worst predictions.
You can contact me if you want further discussion. Here is my LinkedIn
Enjoy!!! 👦🏻
[1] https://en.wikipedia.org/wiki/Data_transformation
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep Residual Learning for Image Recognition, 2015.