Forgetting in Deep Learning
By: Mingyue Wei
Tags: Deep Learning, Catastrophic Forgetting

Problem Statement

Neural network models suffer from the phenomenon of catastrophic forgetting: a model can drastically lose its generalization ability on a task after being trained on a new task. This usually means a new task will likely override the weights that have been learned in the past (see Figure 1), and thus degrade the model performance for the past tasks. Without fixing this problem, a single neural network will not be able to adapt itself to a continuous learning scenario, because it forgets the existing information/knowledge when it learns new things.

For realistic applications of deep learning, where continual learning can be crucial, catastrophic forgetting would need to be avoided. However, there is only limited study about catastrophic forgetting and its underlying causes. In this project, we will explore how commonly used deep learning methods mitigate or exacerbate the degree of forgetting (e.g. batch-norm, dropout, data augmentation, weight decay, etc.). Further, we would like to select one or several methods and try to learn about the cause of effects.

Literature Review

For forgetting measurement, the main approach is to revisit a task after training on later tasks and compare the accuracy before and after². With different task settings, Kemker et al. proposed a method specifically for incremental class learning³ (for each new section, data of a single class will be learned), and Arora et al. suggest using scaled accuracy when comparing different models⁴.

There have been studies of ways to reduce the forgetting effect, and some of the proposed or accepted methods include: adding dropout layers⁵, max pooling, decreasing number of layers, and decreasing learning rates.

Data

The dataset we use is CIFAR-10. This dataset consists of 60,000 color images in total, evenly divided into 10 classes. For each class, 5,000 images are in the train set and the remaining are in the test set. The images are of size 32x32. The 10 classes of the dataset are:

airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck

Experiment Setup

We went with the approach of comparing before and after accuracy to measure forgetting effects. Our experiment workflow is shown in Figure 2.

Figure 2: Experiment workflow. Image by Author

The whole dataset is first evenly split into task 1 and task 2, each consisting of 5 classes of images. We train the model consecutively on task 1 and task 2, and after that, we re-evaluate the model on task 1 data by only refitting the output layer.

The two accuracy scores were then compared.

Accuracy 1: Accuracy on task 1 test before task 2 training

Accuracy 2: Accuracy on task 2 test after task 2 training Forgetting = Accuracy 1 - Accuracy 2

Task 1 and Task 2 Splits

We used 2 sets of splits throughout the experiments.

The first set was used during the initial modeling and exploration stage, which consists of 5 different splits, (split 0 to split 4), all are randomly divided.

We formalized the second set of splits after some experiments, which consists of one semantic split based on image similarity from human perspective and two random splits. This set was used for extended experiments of random shift and constant learning rate experiments. Following are the task 1 classes for each split:

Semantic split: airplane, automobile, bird, ship, truck

Random split 1: airplane, deer, frog, ship, truck

Random split 2: automobile, cat, deer, frog, ship

Baseline Model Structure: Customized CNN

Figure 3: Baseline model structure. Image by Author

Figure 3 shows the structure of our model. It is a customized CNN with decent performance across various task splits and trials, achieving consistent Accuracy 1 around 90%.

Other than the input and output layer, the model consists of 4 intermediate convolutional blocks, each with 2 convolutional layers, 1 batch normalization layer, 1 max pooling layer, and 1 dropout layer with 0.2 dropout rate.

Experiment Overview

The main objective of our experiments was to confirm whether the proposed methods of mitigating forgetting effects could work. We had two major directions: the first consists of experiments with various data augmentation methods, and the second dealt with training settings, including learning rate, weight decay, and training epochs.

After initial exploration of these approaches, we specifically extended our experiments on random shifting in data augmentation, and learning rate in training settings.

Data Augmentation Experiments

Basic Data Augmentation

We experimented with some basic data augmentation methods:

Horizontal/Vertical Flip Random Shift up to 10% and up to 20% Random Rotation up to 10°, 15°, 20°

These data augmentation methods were applied on both task 1 and task 2 training process. The test set did not involve any data augmentation. Their influences on forgetting comparing to the baseline are shown in Figure 4.

Figure 4: Percentage change in forgetting after each data augmentation method was applied, compared to the baseline model. Different colors represent results in different data splits. Image by Author

There is some variation between different data splits from the results. For most of the data augmentation methods, we cannot conclude if they have any effect on forgetting, as in some splits they increased forgetting while in some they did not, and the change was also generally not large.

However, among all the methods, random shift were clearly increasing forgetting, and the influence seems to increase when we shift more. We confirmed that Accuracy 1 of the models with random shifts were comparable to that of baseline without data augmentation, and as results were consistent across all 5 task splits, we believe that random shift does increase forgetting. This is interesting and quite surprising for us to learn more because random shift is a popular method applied to CIFAR-10 data to improve model performance.

Advance Data Augmentation: Mixup

Apart from basic data augmentation methods, we also experimented with mixup. Mixup is a popular method that can improve classification accuracy (as shown in Figure 5). Instead of feeding the network with raw images, mixup refers to take 2 images and do a linear combination of them using λ:

y_new = λ × (1 - y1) + (1 - λ) × y2 x_new = λ × (1 - x1) + (1 - λ) × x2

Figure 5: Percentage change in Accuracy 1 with different levels of mixup, compared to the baseline model. Image by Author

λ is a number between 0 and 1 and is drawn from Beta(𝛼, 𝛼) each time, with 𝛼 being a pre-defined hyper-parameter. Smaller 𝛼 creates less mixup effect, and mixup with large 𝛼 could lead to under-fitting. We experimented with 𝛼 values of [0.1, 0.2, 0.3, 0.4], all within the suggested range from Zhang et al.⁶

Figure 6: Percentage change in forgetting with different levels of mixup, compared to the baseline model. Different colors represent results in different data splits. Image by Author

Figure 6 shows results from mixup experiments. For most of the data splits and 𝛼 values, mixup did not mitigate forgetting. Impact of mixup on forgetting varied across different data splits as well, e.g. when 𝛼= 0.1, forgetting was decreased with two data splits but increased with the other.

More on Random Shift

After the initial experiments, we reached the following conclusions:

Random shift increases forgetting under the current setting. Mix up improves prediction accuracy but does not mitigate forgetting. The effect on forgetting of most data augmentation methods depends on data splits, thus generalization is not an easy get.

Based on these, we performed extended experiments with random shift using the second set of task splits.

In addition to the previous experiments applying random shift to both task 1 and task 2, we also did experiments where random shift was applied only to task 1, the results of which were presented in dashed lines in Figure 7. As the images are of size 32x32, we controlled the number of pixels up to which the image might shift in each experiment, with random shift of 4 pixels being the common practice in the field. All experiments were repeated 9 times and the average is presented.